git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Adding CudaBuffer to pyarrow


Hi Pearu,

Your approach seems ok on the principle.  Feel free to open a PR once
you have a basic API defined and exposed, so that we can iterate on that.

Regards

Antoine.


Le 29/08/2018 à 17:11, Pearu Peterson a écrit :
> Hi Wes,
> 
> Thanks for the hints!
> 
> I have created a very minimal GPU support to pyarrow, see
> 
> 
> https://github.com/apache/arrow/compare/master...Quansight:pearu-cuda-pyarrow?expand=1
> 
> This includes:
> 1. introducing libarrow_gpu that is built into pyarrow.lib_gpu extension
> module, currently only CudaBuffer is exposed
> 2. updating cmake configuration files to detect libarrow_gpu.{so,a}
> libraries
> 3. updating setup.py with new options --with{,out}-arrow-gpu and
> PYARROW_BUILD_ARROW_GPU env variable support.
> 
> Let me know if something in my approach needs to be changed.
> 
> Regarding packaging:
> 1. FYI, there is a cudatoolkit-dev package in conda-forge:
>   https://github.com/conda-forge/cudatoolkit-dev-feedstock
> 
> 2. One approach is to introduce new conda packages arrow-cpp-gpu and
> pyarrow-gpu that are build using -DARROW_GPU and --with-arrow-gpu,
> respectively.
> 
> Best regards,
> Pearu
> 
> On Wed, Aug 29, 2018 at 12:32 AM, Wes McKinney <wesmckinn@xxxxxxxxx> wrote:
> 
>> hi Pearu,
>>
>> Sounds good to me; I had wanted to do it already but got caught up
>> working on other things. You'll probably want to create
>> libarrow_gpu.pxd since this is an optional library. Otherwise it
>> should be reasonably straightforward.
>>
>> some relevant issues
>>
>> - https://issues.apache.org/jira/browse/ARROW-1424
>> - https://issues.apache.org/jira/browse/ARROW-2512
>>
>> If we wanted to ship Python wheels and conda packages with CUDA
>> support built in, what would we have to do (currently these binaries
>> are produced in Circle CI, Travis CI, and Appveyor)?
>>
>> - Wes
>> On Tue, Aug 28, 2018 at 4:34 PM Pearu Peterson
>> <pearu.peterson@xxxxxxxxxxxxx> wrote:
>>>
>>> Hi,
>>>
>>> Is there any interest in implementing CudaBuffer support in pyarrow? Are
>>> there any obstacles that would prevent to do this?
>>>
>>> I would be willing to give it a try.
>>>
>>> As I understand, the entry point for this task would be to patch
>>> pyarrow/includes/libarrow.pxd . Any hints for what to take into account
>> are
>>> appreciated.
>>>
>>> Best regards,
>>> Pearu
>>
>