Re: IPC Example
There's several examples of sending record batches from Java to Python
(and vice versa) over a network socket -- e.g. Jacques and I are
working on a prototype of a general purpose Arrow-native RPC framework
in Java and C++ respectively. Where there's some R&D needed is in Java
interactions with shared memory. So if you want to do a zero copy read
from a memory mapped file, then some development in the Arrow Java
libraries is required.
I'm not an expert but it seems like Netty has a mechanism to interact
with ByteBuffer, which should include MappedByteBuffer
Correspondingly, an interface could be developed to enable the Java
IPC code path to write to a shared memory region.
To the Java developers, could we create some JIRA issues (if there are
not already) around Java shared memory IPC?
On Sat, Sep 1, 2018 at 2:32 PM ALBERTO Bocchinfuso
> I want to reinforce this request. I am interested in the same topic.
> I’d like an example specially focused on the creation of a RecordBatch passed from a Java program to a python one and vice-versa.
> Da: Clive Cox <cc@xxxxxxxxx>
> Inviato: Saturday, September 1, 2018 6:12:29 PM
> A: user@xxxxxxxxxxxxxxxx
> Oggetto: IPC Example
> Is there any example of how to do say Java - Python IPC? I'm not sure how to get started.
> I'm thinking of using Arrow IPC to replace REST/gRPC APIs for communication when everything can be run on a single computer node and low-latency is the goal - hoping to remove the cost of serialization/deserialization and network costs. Would this make sense.