git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Moving Airflow Config to Database.


Some of env variables are not getting reflected in supervisord
environment(sometimes new variables are not available, sometimes changes to
existing variables are not reflected)

On Thu 15 Nov, 2018, 17:37 Ash Berlin-Taylor <ash@xxxxxxxxxx wrote:

> > problem with
> > this approach is these env variables wont behave correctly when we
> > subshells
>
>
> Can you explain what you mean by this?
>
> -ash
>
>
> > On 15 Nov 2018, at 12:03, Sai Phanindhra <phani8996@xxxxxxxxx> wrote:
> >
> > Hi deng,
> > I am currently using env variables for few airflow config variables which
> > may differ across machines(airflow folder, log folder etc..,) Problem
> with
> > this approach is these env variables wont behave correctly when we
> > subshells. ( I faced issues when i added airflow jobs in supervisord).
> >  Moving config to db not only addresses this issue, it will give
> provision
> > to change config from UI.(most of the time, we cant give box access to
> all
> > users. ). Config is db makes it easy to create/update config without
> > touching code.
> >
> > On Thu 15 Nov, 2018, 16:04 Deng Xiaodong <xd.deng.r@xxxxxxxxx wrote:
> >
> >> A few solutions that may address your problem:
> >>
> >> - Specify your configurations in environment variables, so it becomes
> much
> >> easier to manage across machines
> >> - use network attached storage to save your configuration file and
> mount it
> >> to all your machines (this can address DAG file sync as well)
> >> - ...
> >>
> >> Personally I don’t see point moving configuration to DB.
> >>
> >>
> >> XD
> >>
> >> On Thu, Nov 15, 2018 at 18:29 Sai Phanindhra <phani8996@xxxxxxxxx>
> wrote:
> >>
> >>> Hello Airflow users,
> >>>      I recently encountered as issue with airflow. I am maintaining a
> >>> airflow cluster, whenever i make a change in airflow configuration in
> one
> >>> of the machine i have to consciously copy these changes to other
> machines
> >>> in airflow cluster. Problem with this is it's a manual process and
> >>> something users tend to forget to sync changes. I want move airflow
> >> config
> >>> to a database. I will be happy if you can share your valuable
> >>> inputs/thoughts on this.
> >>>
> >>> --
> >>> Sai Phanindhra,
> >>> Ph: +91 9043258999
> >>>
> >>
>
>