[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Adding CudaBuffer to pyarrow

Hi Wes,

Thanks for the hints!

I have created a very minimal GPU support to pyarrow, see

This includes:
1. introducing libarrow_gpu that is built into pyarrow.lib_gpu extension
module, currently only CudaBuffer is exposed
2. updating cmake configuration files to detect libarrow_gpu.{so,a}
3. updating with new options --with{,out}-arrow-gpu and
PYARROW_BUILD_ARROW_GPU env variable support.

Let me know if something in my approach needs to be changed.

Regarding packaging:
1. FYI, there is a cudatoolkit-dev package in conda-forge:

2. One approach is to introduce new conda packages arrow-cpp-gpu and
pyarrow-gpu that are build using -DARROW_GPU and --with-arrow-gpu,

Best regards,

On Wed, Aug 29, 2018 at 12:32 AM, Wes McKinney <wesmckinn@xxxxxxxxx> wrote:

> hi Pearu,
> Sounds good to me; I had wanted to do it already but got caught up
> working on other things. You'll probably want to create
> libarrow_gpu.pxd since this is an optional library. Otherwise it
> should be reasonably straightforward.
> some relevant issues
> -
> -
> If we wanted to ship Python wheels and conda packages with CUDA
> support built in, what would we have to do (currently these binaries
> are produced in Circle CI, Travis CI, and Appveyor)?
> - Wes
> On Tue, Aug 28, 2018 at 4:34 PM Pearu Peterson
> <pearu.peterson@xxxxxxxxxxxxx> wrote:
> >
> > Hi,
> >
> > Is there any interest in implementing CudaBuffer support in pyarrow? Are
> > there any obstacles that would prevent to do this?
> >
> > I would be willing to give it a try.
> >
> > As I understand, the entry point for this task would be to patch
> > pyarrow/includes/libarrow.pxd . Any hints for what to take into account
> are
> > appreciated.
> >
> > Best regards,
> > Pearu