git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Moving Airflow Config to Database.


> problem with
> this approach is these env variables wont behave correctly when we
> subshells


Can you explain what you mean by this?

-ash


> On 15 Nov 2018, at 12:03, Sai Phanindhra <phani8996@xxxxxxxxx> wrote:
> 
> Hi deng,
> I am currently using env variables for few airflow config variables which
> may differ across machines(airflow folder, log folder etc..,) Problem with
> this approach is these env variables wont behave correctly when we
> subshells. ( I faced issues when i added airflow jobs in supervisord).
>  Moving config to db not only addresses this issue, it will give provision
> to change config from UI.(most of the time, we cant give box access to all
> users. ). Config is db makes it easy to create/update config without
> touching code.
> 
> On Thu 15 Nov, 2018, 16:04 Deng Xiaodong <xd.deng.r@xxxxxxxxx wrote:
> 
>> A few solutions that may address your problem:
>> 
>> - Specify your configurations in environment variables, so it becomes much
>> easier to manage across machines
>> - use network attached storage to save your configuration file and mount it
>> to all your machines (this can address DAG file sync as well)
>> - ...
>> 
>> Personally I don’t see point moving configuration to DB.
>> 
>> 
>> XD
>> 
>> On Thu, Nov 15, 2018 at 18:29 Sai Phanindhra <phani8996@xxxxxxxxx> wrote:
>> 
>>> Hello Airflow users,
>>>      I recently encountered as issue with airflow. I am maintaining a
>>> airflow cluster, whenever i make a change in airflow configuration in one
>>> of the machine i have to consciously copy these changes to other machines
>>> in airflow cluster. Problem with this is it's a manual process and
>>> something users tend to forget to sync changes. I want move airflow
>> config
>>> to a database. I will be happy if you can share your valuable
>>> inputs/thoughts on this.
>>> 
>>> --
>>> Sai Phanindhra,
>>> Ph: +91 9043258999
>>> 
>>