git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Python3 and sensors module


Snakebite is not py3 compatible. We should move to libhdfs. Happy to take patches!

B.

Verstuurd vanaf mijn iPad

> Op 16 mei 2018 om 18:57 heeft Cindy Rottinghuis <cindyrottinghuis@xxxxxxxxx> het volgende geschreven:
> 
> Here is a sample of what I am seeing.   
> I am running on ubuntu, python 3.5.   This is a subset of the full message.  Once it went into the snakebite package, checked and the library wasn’t supported in python3.
> ….
> File "/usr/local/airflow/dags/dim_date.py", line 5, in <module>
>    from airflow.operators.sensors import S3KeySensor
>  File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py", line 34, in <module>
>    from airflow.hooks.hdfs_hook import HDFSHook
>  File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py", line 20, in <module>
>    from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
>  File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", line 1473
>    baseTime = min(time * (1L << retries), cap);
>                            ^
> 
> 
>> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis <cindyrottinghuis@xxxxxxxxx> wrote:
>> 
>> Hi,
>> 
>> Yes, it looks like all of the other sensors will work under Python3, but not hdfs.  I am planning to use the s3 sensor, which is wrapped up in the sensors.py module.  My issue is that when I test my dag or airflow installation under python3, I get errors about the hdfs_hook, which I’m not using.   Other then creating my own version of the sensors.py file and removing the hdfs related functions/libraries, is there any thing else I can do to work around this?
>> 
>> 
>> 
>>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko <fokko@xxxxxxxxxxxxxx> wrote:
>>> 
>>> Hi Cindy,
>>> 
>>> The other sensors should work under Python3. We try to support Python3 as
>>> much as possible, but sometimes libraries are used that are not compatible.
>>> Could you describe what you are running into?
>>> 
>>> Cheers, Fokko
>>> 
>>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis <cindyrottinghuis@xxxxxxxxx>:
>>> 
>>>> Hi,
>>>> 
>>>> Are there any plans to update the HDFS_hook.py script to remove the
>>>> reference to the snakebite python library? I’d like to run airflow on
>>>> python3, and this is causing some issues.   The hdfs_hook script is
>>>> referenced in the sensors module.
>>>> 
>>>> Any suggestions?
>>>> 
>>>> Thanks,
>>>> Cindy
>> 
>