git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: celery problem: cannot override celery_broker_transport_options


Yes, you would need to duplicate a chunk of the default_celery in your copy right now. But you can just make it have the values you want - so it would be about 10 lines in total.

It seems that between AIRFLOW-966 and AIRFLOW-1840 things got a little bit out of sync in the default .cfg and the celery .py - the .py is looking for celery->ssl_* but the default config puts it in celery_broker_transport_options->ssl_*. Looking at the celery config option it looks like they aren't actually options for the other transports, it's just that they don't complain about the extra options. I think the fix is just to move them up to the celery section.

-ash


> On 24 May 2018, at 07:21, Craig Rodrigues <rodrigc@xxxxxxxxxxxxxx> wrote:
> 
> Ash,
> 
> Thanks!   You put me on the right track.
> Unfortunately, there is a lot of logic in
> airflow/config_templates/default_celery.py that I need,
> and if I was to come up with my own class to replace:
> 
> celery_config_options =
> airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
> 
> then I would basically have to rewrite my own version of default_celery.py
> 
> Also, in airflow/config_templates/default_airflow.cfg, there is this:
> 
> [celery_broker_transport_options]
> # The visibility timeout defines the number of seconds to wait for the
> worker
> # to acknowledge the task before the message is redelivered to another
> worker.
> # Make sure to increase the visibility timeout to match the time of the
> longest
> # ETA you're planning to use. Especially important in case of using Redis
> or SQS
> visibility_timeout = 21600
> 
> # In case of using SSL
> ssl_active = False
> ssl_key =
> ssl_cert =
> ssl_cacert =
> 
> 
> None of those options work if using an sqla backend for celery broker_url.
> 
> I need to think about this, but this needs to be cleaned up before
> Airflow 1.10 is released.  In airflow/config_templates/default_airflow.cfg
> there is this:
> 
> broker_url = sqla+mysql://airflow:airflow@localhost:3306/airflow
> 
> 
> So sqla is specified as the default broker_url for celery.
> A lot of people (including where I work) have used this template to set up
> Airflow + Celery, and even though sqla is an "experimental" broker_url,
> it actually works pretty well.
> 
> Now in airflow 1.10, something that was very easy to set up is now really
> complicated and unintuitive.
> 
> Would it be OK to change the airflow code so that
> in airflow/config_templates/default_airflow.cfg, all the options in
> [celery_broker_transport_options] are commented out?
> And if someone is running Redis, they would have to add those
> options in their own airflow.cfg file?
> 
> Bolke, do you have any comments?
> 
> --
> Craig
> 
> On Tue, May 22, 2018 at 1:50 AM Ash Berlin-Taylor <
> ash_airflowlist@xxxxxxxxxxxxxx> wrote:
> 
>> To use with the SQLA backend to celery you need to override the options
>> Airflow passes to Celery. Those come from
>> https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py
>> 
>> Since you don't want most/all of those options (and there is no way in the
>> config file to _remove_ a setting) you will have to point airflow to a
>> different file for the celery config:
>> 
>> This line in the config is what you will need to change:
>> 
>>    # Import path for celery configuration options
>>    celery_config_options =
>> airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
>> 
>> If you create something like config/celery_config.py containing:
>> 
>> 
>>    CELERY_CONFIG = {
>>        # Just the options you want to set
>>    }
>> 
>> 
>> (config/ should exist along side your dags/ folder, and I think it should
>> be added to the python path already). You can then set this in the config:
>> 
>>    celery_config_options = celery_config.CELERY_CONFIG
>> 
>> That should give you complete control
>> 
>>