git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Can a DAG be conditionally hidden from the UI?


Thanks all, following your advice I've decided to do this right now with a
decorator.  It looks crude so any suggestions are welcome.

The decorator function checks an environment variable that we set on our
servers and if it matches any of the passed in environment names, it'll
allow the DAG to be created.


def environments(env_names):
    def restrict_environment(func):
        def func_wrapper(*args, **kwargs):
*            if( os.getenv("environment") in env_names):*
                return DAG(*args, **kwargs)
        return func_wrapper
    return restrict_environment

*@environments(["env1","env2","production"])*
def get_DAG(*args, **kwargs):
    return None

dag = get_DAG('my_dag', default_args=default_args, schedule_interval="30 5
* * *")


Regards
Shah


On Mon, Oct 8, 2018 at 5:19 PM William Pursell <williamp@xxxxxxxxx.invalid>
wrote:

> This would be a very desirable feature.  It's not just an issue of
> differing environments, but also changing requirements.  A dag may be
> used for a time, and then removed from the environment.  But if it's
> still in the airflow db, it will get a row on the UI with a black "I'
> indicating that the dag is missing.  As far as I know, then only way
> to remove it is to manually edit the database.
> On Mon, Oct 8, 2018 at 9:43 AM Chris Palmer <chris@xxxxxxxxxxxx> wrote:
> >
> > I like James solution better, but the initial thought I had was to deploy
> > airflowignore files to the environments to filter out files that should
> not
> > be processed when filling the DagBag.
> >
> > Chris
> >
> > On Mon, Oct 8, 2018 at 10:22 AM James Meickle
> > <jmeickle@xxxxxxxxxxxxxx.invalid> wrote:
> >
> > > As long as the Airflow process can't find the DAG as a top-level
> object in
> > > the module, it won't be registered. For example, we have a function
> that
> > > returns DAGs; the function returns nothing if it's not in the right
> > > environment.
> > >
> > > On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <mendhak@xxxxxxxxx> wrote:
> > >
> > > > Hi all,
> > > >
> > > > tl;dr - Is it possible to conditionally hide a DAG from the UI based
> on
> > > an
> > > > environment variable?
> > > >
> > > >
> > > >
> > > > Our team has a single repo with several DAGs in it and we deploy it
> > > across
> > > > multiple 'environments' (think dev, test, and other integration
> > > > environments).  While most DAGs are meant to run everywhere, we do
> have
> > > > several which are meant to run in one and only one environment.  Of
> > > course
> > > > they are all paused, but it would be nice to declutter a bit for
> > > ourselves.
> > > >
> > > > My question then - is it possible to conditionally hide a DAG from
> the UI
> > > > based on an environment variable or some flag somewhere.
> > > >
> > > > This is just wishful thinking - the dev could do something like
> > > >
> > > > dag = get_dag(...),and get_dag() would have a decorator like
> > > > @only_run_in("integration4,dev,local")
> > > >
> > > > And that decorator returns some kind of None object or special DAG
> which
> > > > just doesn't appear in the list.
> > > >
> > > > Or perhaps some other way to accomplish this same effect - any ideas
> > > would
> > > > be appreciated.
> > > >
> > > > Thanks
> > > >
> > >
>
>