git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Python3 and sensors module


Not a full patch to replace snakebite with libhdfs, but should temporarily
unblock ci/cd: https://github.com/apache/incubator-airflow/pull/3365

On Wed, May 16, 2018 at 10:00 AM, Bolke de Bruin <bdbruin@xxxxxxxxx> wrote:

> Snakebite is not py3 compatible. We should move to libhdfs. Happy to take
> patches!
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 16 mei 2018 om 18:57 heeft Cindy Rottinghuis <
> cindyrottinghuis@xxxxxxxxx> het volgende geschreven:
> >
> > Here is a sample of what I am seeing.
> > I am running on ubuntu, python 3.5.   This is a subset of the full
> message.  Once it went into the snakebite package, checked and the library
> wasn’t supported in python3.
> > ….
> > File "/usr/local/airflow/dags/dim_date.py", line 5, in <module>
> >    from airflow.operators.sensors import S3KeySensor
> >  File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py",
> line 34, in <module>
> >    from airflow.hooks.hdfs_hook import HDFSHook
> >  File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py",
> line 20, in <module>
> >    from snakebite.client import Client, HAClient, Namenode,
> AutoConfigClient
> >  File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py",
> line 1473
> >    baseTime = min(time * (1L << retries), cap);
> >                            ^
> >
> >
> >> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis <
> cindyrottinghuis@xxxxxxxxx> wrote:
> >>
> >> Hi,
> >>
> >> Yes, it looks like all of the other sensors will work under Python3,
> but not hdfs.  I am planning to use the s3 sensor, which is wrapped up in
> the sensors.py module.  My issue is that when I test my dag or airflow
> installation under python3, I get errors about the hdfs_hook, which I’m not
> using.   Other then creating my own version of the sensors.py file and
> removing the hdfs related functions/libraries, is there any thing else I
> can do to work around this?
> >>
> >>
> >>
> >>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko <fokko@xxxxxxxxxxxxxx>
> wrote:
> >>>
> >>> Hi Cindy,
> >>>
> >>> The other sensors should work under Python3. We try to support Python3
> as
> >>> much as possible, but sometimes libraries are used that are not
> compatible.
> >>> Could you describe what you are running into?
> >>>
> >>> Cheers, Fokko
> >>>
> >>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis <
> cindyrottinghuis@xxxxxxxxx>:
> >>>
> >>>> Hi,
> >>>>
> >>>> Are there any plans to update the HDFS_hook.py script to remove the
> >>>> reference to the snakebite python library? I’d like to run airflow on
> >>>> python3, and this is causing some issues.   The hdfs_hook script is
> >>>> referenced in the sensors module.
> >>>>
> >>>> Any suggestions?
> >>>>
> >>>> Thanks,
> >>>> Cindy
> >>
> >
>
>


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg03385.html on line 149
Call Stack
#TimeMemoryFunctionLocation
10.0033368632{main}( ).../msg03385.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg03385.html on line 149
Call Stack
#TimeMemoryFunctionLocation
10.0033368632{main}( ).../msg03385.html:0