git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: celery problem, AttributeError: async


I have submitted:

https://github.com/apache/incubator-airflow/pull/3388

--
Craig

On Mon, May 21, 2018 at 7:00 AM Naik Kaxil <k.naik@xxxxxxxxx> wrote:

> Thanks. Please do that.
>
> On 21/05/2018, 14:59, "Craig Rodrigues" <crodr001@xxxxxxxxx> wrote:
>
>     celery 4.1.1 was just released last night which has all the async
> problems fixed:
>
>     https://github.com/celery/celery/commits/v4.1.1
>
>     I'll test this out, and then submit a PR to bump airflow's celery
> version to 4.1.1
>
>     --
>     Craig
>
>     On 2018/05/21 07:20:50, Craig Rodrigues <crodr001@xxxxxxxxx> wrote:
>     > Hi,
>     >
>     > I used a requiremens.txt file with these three lines:
>     >
>     > git+
> https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
>     > celery>=4.2.0rc3
>     > kombu>=4.2.0
>     >
>     >
>     > I did
>     >
>     > pip install -r requirements.txt
>     >
>     > When I started my worker, I got:
>     >
>     > [2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor
> CeleryExecutor
>     > Traceback (most recent call last):
>     >   File "/bin/airflow", line 32, in <module>
>     >     args.func(args)
>     >   File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line
> 74, in wrapper
>     >     return f(*args, **kwargs)
>     >   File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line
> 959, in worker
>     >     worker.run(**options)
>     >   File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line
> 257, in run
>     >     **kwargs)
>     >   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py",
> line 101, in __init__
>     >     self.setup_instance(**self.prepare_args(**kwargs))
>     >   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py",
> line 124, in setup_instance
>     >     self.should_use_eventloop() if use_eventloop is None
>     >   File "/usr/lib/python2.7/site-packages/celery/worker/worker.py",
> line 243, in should_use_eve
>     >     self._conninfo.transport.implements.async and
>     >   File "/usr/lib/python2.7/site-packages/kombu/transport/base.py",
> line 125, in __getattr__
>     >     raise AttributeError(key)
>     > AttributeError: async
>     >
>     >
>     >
>     > I reported this as a bug in celery here:
>     >
>     > https://github.com/celery/celery/issues/4747
>     >
>     > It turns out that this is a known problem.  It looks like in kombu,
> they
>     > renamed a bunch of uses of "async" to "asynchronous", to not clash
> with the new
>     > 'async' keyword in Python 3.7.
>     >
>     > In celery, they are catching up and doing the same rename.
>     > Almost at the same time that I reported this bug, the issue was
> fixed in the master branch of celery:
>     >
>     >
> https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad
>     >
>     > Hopefully the versions of kombu and celery will stabilize so that it
> all works properly with airflow.
>     >
>     > --
>     > Craig
>     >
>     >
>     >
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.naik@xxxxxxxxx
> www.reply.com
>