git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Reading BigQuery Timestamps via python SDK returns error for dates < 1900


I don't see an easy workaround unfortunately. Basically, till this is fixed, BigQuery columns of type TIMESTAMP will not work for pre 1900 values. So you'll either have to change the values or type of this column.

Thanks,
Cham

On Wed, Apr 25, 2018 at 9:51 AM Yatish Gupta <ygupta@xxxxxxxxxx> wrote:
Thanks! In the meantime, is there a workaround for this?

Thanks!
Yatish

On Wed, Apr 25, 2018 at 12:20 AM Chamikara Jayalath <chamikara@xxxxxxxxxx> wrote:
Thanks for reporting. This seems to be due to a known Python bug: https://bugs.python.org/issue1777412

Seems like above bug has not been fixed for Python 2.x line. I created https://issues.apache.org/jira/browse/BEAM-4171 for tracking the Beam issue and possibly updating the Beam SDK to not use 'strftime'.

Thanks,
Cham

On Tue, Apr 24, 2018 at 2:33 PM Yatish Gupta <ygupta@xxxxxxxxxx> wrote:
Hey,

I am getting an exception when running beam on BigQuery rows that contain Timestamps less than 1900. I get a similar error when using the DirectRunner.

  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/nativebigqueryavroio.py", line 83, in _fix_field_value
    return dt.strftime('%Y-%m-%d %H:%M:%S.%f UTC')
ValueError: year=1851 is before 1900; the datetime strftime() methods require year >= 1900

How do I read these rows without changing my schema?

Thanks!



( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-beam-users/msg02164.html on line 72
Call Stack
#TimeMemoryFunctionLocation
10.0008368736{main}( ).../msg02164.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-beam-users/msg02164.html on line 72
Call Stack
#TimeMemoryFunctionLocation
10.0008368736{main}( ).../msg02164.html:0