git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Moving Airflow Config to Database.


Hi deng,
 I am currently using env variables for few airflow config variables which
may differ across machines(airflow folder, log folder etc..,) Problem with
this approach is these env variables wont behave correctly when we
subshells. ( I faced issues when i added airflow jobs in supervisord).
  Moving config to db not only addresses this issue, it will give provision
to change config from UI.(most of the time, we cant give box access to all
users. ). Config is db makes it easy to create/update config without
touching code.

On Thu 15 Nov, 2018, 16:04 Deng Xiaodong <xd.deng.r@xxxxxxxxx wrote:

> A few solutions that may address your problem:
>
> - Specify your configurations in environment variables, so it becomes much
> easier to manage across machines
> - use network attached storage to save your configuration file and mount it
> to all your machines (this can address DAG file sync as well)
> - ...
>
> Personally I don’t see point moving configuration to DB.
>
>
> XD
>
> On Thu, Nov 15, 2018 at 18:29 Sai Phanindhra <phani8996@xxxxxxxxx> wrote:
>
> > Hello Airflow users,
> >       I recently encountered as issue with airflow. I am maintaining a
> > airflow cluster, whenever i make a change in airflow configuration in one
> > of the machine i have to consciously copy these changes to other machines
> > in airflow cluster. Problem with this is it's a manual process and
> > something users tend to forget to sync changes. I want move airflow
> config
> > to a database. I will be happy if you can share your valuable
> > inputs/thoughts on this.
> >
> > --
> > Sai Phanindhra,
> > Ph: +91 9043258999
> >
>