Yes my first reaction to both of these requests is - would dictionary-encoding work? - would a List<T> work?I think for the former the analogy is more clear, for the latter, technically a List encodes start and stop indices with an offset array rather than separate arrays for start and stop indices. Is there a reason an offset array wouldn't work for the OAMap use-case though?
Brian On 04/30/2018 04:55 PM, Antoine Pitrou wrote:
Actually, "pointer type" might just be another name for "dictionary type". Regards Antoine. Le 30/04/2018 à 22:08, Antoine Pitrou a écrit :Hi, Today I got the opportunity to talk with Jim Pivarski, the main developer on the OAMap project (*). Under the hood, he is doing something not unlike the Arrow representation of nested arrays: he stores and processes structured data as linear arrays, allowing very fast processing on seemingly irregular data (in Array parlance, think something like lists of lists of structs). It seems that OAMap data requires two kinds of logical types that Arrow misses : - a pointer type, where a physical array of ints is used to represent indices into another array (the logical value being of course the value pointed to) - a span type, where two physical arrays of ints are used to represent start and stop indices into another array (the logical value being the list of values delimited by the start / stop indices) Did such a feature request already come by? Is this something we should add to our roadmap or future wishlist? (*) https://github.com/diana-hep/oamap Regards Antoine.
|( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-arrow-development/msg04325.html on line 122|
|( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-arrow-development/msg04325.html on line 122|