git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: valid NaNs versus invalid NaNs?


Alternately Rhys, what Wes said. :)

Donald E. Foss | @DonaldFoss <https://twitter.com/DonaldFoss>
Never Stop Learning!
------ __o
----_`\<,_
---(_)/ (_)

> On Dec 10, 2018, at 11:23 AM, Donald Foss <donald.foss@xxxxxxxxx> wrote:
> 
> +1 on NaNs being an interop nightmare already, especially for those who work with multiple coding languages at the same time.
> 
> Issues regarding NaNs may be found at https://issues.apache.org/jira/browse/ARROW-2806?jql=text%20~%20%22NaN%22 <https://issues.apache.org/jira/browse/ARROW-2806?jql=text%20~%20%22NaN%22>. The last issue I see was from July 2018, with Python, and marked resolved 17 July 2018. The description may be helpful.
> 
> Regards,
> 
> Donald E. Foss | @DonaldFoss <https://twitter.com/DonaldFoss>
> Never Stop Learning!
> ------ __o
> ----_`\<,_
> ---(_)/ (_)
> 
>> On Dec 10, 2018, at 10:47 AM, Rhys Ulerich <Rhys.Ulerich@xxxxxxxxxxxx <mailto:Rhys.Ulerich@xxxxxxxxxxxx>> wrote:
>> 
>> 'Morning,
>> 
>> 
>> 
>> Regarding https://arrow.apache.org/docs/memory_layout.html <https://arrow.apache.org/docs/memory_layout.html>, how should is_valid be interpreted for primitive types that have their own notions of is_valid?
>> 
>> 
>> 
>> Concretely, how should folks interpret a "valid NaN" (is_valid 1 with float NaN) versus an "invalid NaN" (is valid 0 with float NaN)?  In RFC-ese, MUST individual NaNs be valid?  Or, MUST floats all be valid by omitting the validity bitset?
>> 
>> 
>> 
>> I ask because otherwise I can see a bunch of different systems interpreting this detail in many different ways.  That'd be an interop nightmare.  Especially since understanding why NaNs sneak into large datasets is already quite a hassle.
>> 
>> 
>> 
>> Anyhow, it seems worth addressing this gap at the written specification level.
>> 
>> 
>> 
>> (Apologies if this has been discussed previously-- I've found no searchable mailing list archives under http://mail-archives.apache.org/mod_mbox/arrow-dev/ <http://mail-archives.apache.org/mod_mbox/arrow-dev/> or https://cwiki.apache.org/confluence/display/ARROW <https://cwiki.apache.org/confluence/display/ARROW>.)
>> 
>> 
>> 
>> Thanks,
>> 
>> Rhys
>