git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: plan for Go implementation of Plasma


Neat! Thank you for the suggestions, I'll take a look into these other
approaches. Sticking with cgo does sound promising; I had dismissed it due
to needing to maintain a C interface, but if there's already some bindings
that might become official, that negates that issue.

On Wed, Dec 19, 2018 at 3:26 PM Philipp Moritz <pcmoritz@xxxxxxxxx> wrote:

> Hey Dustin,
>
> Thanks for getting in touch! Here are two additional ways to do it:
>
> 5. Native go client library: If Go has support to ship file descriptors
> over unix domain sockets (which I think it has, see
>
> https://github.com/opencontainers/runc/blob/master/libcontainer/utils/cmsg.go
> )
> and interact with memory mapped files it might also be possible to make a
> version of
> https://github.com/apache/arrow/blob/master/cpp/src/plasma/client.cc that
> is native go. The advantage is that it wouldn't need any additional
> compilation steps on the go side, the disadvantage is that it would need to
> be updated if the plasma client internals change (like they did recently
> with the removal of the release buffer).
>
> 6. GObject wrapper: Possibly one could use the GObject plasma bindings
> that kou and his team are managing to build a wrapper (not sure how
> feasible that is or if there is a mature GObject go implementation).
>
> I would encourage you to start by writing write down the ideal Go API for
> the client and then see how it can be implemented after that (to make sure
> the API, which is the most important piece, is not influenced by the
> implementation choice).
>
> Then, going the cgo route seems the most promising for me since that's I
> think the route that most go code interfaces with native libraries. There
> are some C bindings that have been written:
> https://github.com/plures/pxnd/tree/master/libplasma. If they are useful
> to
> you, we can make a plan to integrate them into the repo.
>
> Best,
> Philipp.
>
>
>
> On Wed, Dec 19, 2018 at 12:04 PM Dustin Long <dustmop@xxxxxx> wrote:
>
> > Hi all!
> >
> > I am a developer on qri <https://qri.io/>, a data-science tool built on
> > IPFS written in go. We're interested in integrating Arrow and especially
> > Plasma, in order to be able to share datasets with other apps like
> Jupyter
> > Notebook. Having this functionality is going to be key for how we plan to
> > integrate with existing frameworks.
> >
> > I've been investigating possible approaches for how to use Plasma in our
> > codecase. I realize that Plasma is still a work in progress, and doesn't
> > have stable API yet, but we're also a ways off from being ready to fully
> > integrate it on our side. Just figured it would be good to start this
> > conversation early in order to plan ahead for how development should
> > proceed.
> >
> > So, the prototypes I've been hacking on have revealed a few choices of
> how
> > to make our golang codebase call Plasma's C++, and I wanted to see what
> the
> > Plasma devs think about these approaches or if they have any preference
> for
> > how the go bindings should be behave.
> >
> > Here are the options in order of what seems to be least to most usable:
> >
> > 1. cgo
> >   Use go's builtin cgo facility to call the Plasma C++ implementation.
> cgo
> > is relatively easy to use, however it only can call C functions. So this
> > would require writing and maintaining a pure C language wrapper around
> the
> > C++ functionality we want to expose. A lot would be lost in translation
> and
> > the resulting go code would look nothing like the original library.
> >
> > 2. dlopen
> >   Install Plasma as a library on the user's system, then load the library
> > at run-time, looking up function calls and data structures as needed.
> > Removes the need for a static dependency, but still requires a lot of
> shim
> > code to be written to load the shared library calls. C++'s name mangling
> > gets in the way a lot.
> >
> > 3. Swig
> >   Wrap a swig interface file that exposes whatever functionality we want
> to
> > golang. The standard go tool has builtin swig support, which knows when
> to
> > invoke the swig generator, in order to create go bindings that resemble
> the
> > C++ original. The build process is relatively uninterrupted.
> >
> > I noticed there doesn't seem to be any swig in use currently in the arrow
> > codebase, which made me think there might have been a reason that it has
> > been avoided for other languages. I'm interested to hear any thoughts, or
> > see if there are other suggestions on how to proceed?
> >
> > Regards,
> > Dustin
> >
>