git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

celery problem, AttributeError: async


Hi,

I used a requiremens.txt file with these three lines:

git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
celery>=4.2.0rc3
kombu>=4.2.0


I did

pip install -r requirements.txt

When I started my worker, I got:

[2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor CeleryExecutor
Traceback (most recent call last):
  File "/bin/airflow", line 32, in <module>
    args.func(args)
  File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 959, in worker
    worker.run(**options)
  File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 257, in run
    **kwargs)
  File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 101, in __init__
    self.setup_instance(**self.prepare_args(**kwargs))
  File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 124, in setup_instance
    self.should_use_eventloop() if use_eventloop is None
  File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 243, in should_use_eve
    self._conninfo.transport.implements.async and
  File "/usr/lib/python2.7/site-packages/kombu/transport/base.py", line 125, in __getattr__
    raise AttributeError(key)
AttributeError: async



I reported this as a bug in celery here:

https://github.com/celery/celery/issues/4747

It turns out that this is a known problem.  It looks like in kombu, they
renamed a bunch of uses of "async" to "asynchronous", to not clash with the new
'async' keyword in Python 3.7.

In celery, they are catching up and doing the same rename.  
Almost at the same time that I reported this bug, the issue was fixed in the master branch of celery:

https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad

Hopefully the versions of kombu and celery will stabilize so that it all works properly with airflow.

--
Craig