git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Some initial GPU questions


Hi Anthony,

Le 26/06/2018 à 17:55, Anthony Scopatz a écrit :
> 
>    2. Should arrow compute assume that everything fits in memory? Arrow
>    seem to handle data that is larger than memory via the Buffer API. Are
>    there restrictions that using Buffers imply that we should be aware of?

A Buffer certainly assumes that your data can be accessed through its
linear address range; it doesn't assume that everything is present at a
given point in physical memory.  So the solution to larger than memory
data would be to use virtual memory facilities, which is how Plasma works.

As for CudaBuffer, I *think* it assumes that everything is in the
GPU-addressable range of memory it describes.  Whether or not the GPU
supports some notion of virtual memory depends on the GPU model, I guess?

On the topic of Buffers and GPUs, I'd like to point out the following
issue with the current APIs:
https://issues.apache.org/jira/browse/ARROW-2447

Regards

Antoine.