git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Moving Airflow Config to Database.


Just a guess, but do you need to reload supervisorctl itself before
restarting the service? If you add an env var to the supervisor config, and
then restart the supervisor-managed service, it will actually be running
with the old supervisor config file still. The supervisor daemon itself
must be reloaded.



On Thu, Nov 15, 2018 at 7:59 AM Sai Phanindhra <phani8996@xxxxxxxxx> wrote:

> Some of env variables are not getting reflected in supervisord
> environment(sometimes new variables are not available, sometimes changes to
> existing variables are not reflected)
>
> On Thu 15 Nov, 2018, 17:37 Ash Berlin-Taylor <ash@xxxxxxxxxx wrote:
>
> > > problem with
> > > this approach is these env variables wont behave correctly when we
> > > subshells
> >
> >
> > Can you explain what you mean by this?
> >
> > -ash
> >
> >
> > > On 15 Nov 2018, at 12:03, Sai Phanindhra <phani8996@xxxxxxxxx> wrote:
> > >
> > > Hi deng,
> > > I am currently using env variables for few airflow config variables
> which
> > > may differ across machines(airflow folder, log folder etc..,) Problem
> > with
> > > this approach is these env variables wont behave correctly when we
> > > subshells. ( I faced issues when i added airflow jobs in supervisord).
> > >  Moving config to db not only addresses this issue, it will give
> > provision
> > > to change config from UI.(most of the time, we cant give box access to
> > all
> > > users. ). Config is db makes it easy to create/update config without
> > > touching code.
> > >
> > > On Thu 15 Nov, 2018, 16:04 Deng Xiaodong <xd.deng.r@xxxxxxxxx wrote:
> > >
> > >> A few solutions that may address your problem:
> > >>
> > >> - Specify your configurations in environment variables, so it becomes
> > much
> > >> easier to manage across machines
> > >> - use network attached storage to save your configuration file and
> > mount it
> > >> to all your machines (this can address DAG file sync as well)
> > >> - ...
> > >>
> > >> Personally I don’t see point moving configuration to DB.
> > >>
> > >>
> > >> XD
> > >>
> > >> On Thu, Nov 15, 2018 at 18:29 Sai Phanindhra <phani8996@xxxxxxxxx>
> > wrote:
> > >>
> > >>> Hello Airflow users,
> > >>>      I recently encountered as issue with airflow. I am maintaining a
> > >>> airflow cluster, whenever i make a change in airflow configuration in
> > one
> > >>> of the machine i have to consciously copy these changes to other
> > machines
> > >>> in airflow cluster. Problem with this is it's a manual process and
> > >>> something users tend to forget to sync changes. I want move airflow
> > >> config
> > >>> to a database. I will be happy if you can share your valuable
> > >>> inputs/thoughts on this.
> > >>>
> > >>> --
> > >>> Sai Phanindhra,
> > >>> Ph: +91 9043258999
> > >>>
> > >>
> >
> >
>