git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: IPC Example


+ dev@

There's several examples of sending record batches from Java to Python
(and vice versa) over a network socket -- e.g. Jacques and I are
working on a prototype of a general purpose Arrow-native RPC framework
in Java and C++ respectively. Where there's some R&D needed is in Java
interactions with shared memory. So if you want to do a zero copy read
from a memory mapped file, then some development in the Arrow Java
libraries is required.

I'm not an expert but it seems like Netty has a mechanism to interact
with ByteBuffer, which should include MappedByteBuffer

https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffer/ReadOnlyUnsafeDirectByteBuf.java#L25

Correspondingly, an interface could be developed to enable the Java
IPC code path to write to a shared memory region.

To the Java developers, could we create some JIRA issues (if there are
not already) around Java shared memory IPC?

- Wes
On Sat, Sep 1, 2018 at 2:32 PM ALBERTO Bocchinfuso
<alberto_boc_94@xxxxxxxxxx> wrote:
>
> I want to reinforce this request. I am interested in the same topic.
>
> I’d like an example specially focused on the creation of a RecordBatch passed from a Java program to a python one and vice-versa.
>
>
>
> Thanks,
>
> Alberto
>
>
>
> ________________________________
> Da: Clive Cox <cc@xxxxxxxxx>
> Inviato: Saturday, September 1, 2018 6:12:29 PM
> A: user@xxxxxxxxxxxxxxxx
> Oggetto: IPC Example
>
> Hi,
>
>  Is there any example of how to do say Java - Python IPC? I'm not sure how to get started.
>
>  I'm thinking of using Arrow IPC to replace REST/gRPC APIs for communication when everything can be run on a single computer node and low-latency is the goal - hoping to remove the cost of serialization/deserialization and network costs. Would this make sense.
>
>  Thanks,
>
>  Clive
>