git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

feature request: push/save DAG objects directly


Hi AirFlow Developer,

I am Dr. Soma Dhavala, ML architect. At EkStep <https://ekstep.org/>, a
non-profit organisation, we are developing an open source ML workbench and
we are using AirFlow as one of the core components, on the top of which are
writing an application layer. The idea is to make this tool available to
millions of users who use our SunBird <http://www.sunbird.org/about/>
platform

We compose DAGs programmatically - that is an agent auto-composes a DAG
based on a specification (i.e., DAGS are not handcrafted but are generated
based a on template), so we dont have any python script that literally
constructs a DAG object. So it was less than ideal for us to drop .py files
into the DAG folder. An ability to save/write DAG objects directly to the
DB or make them available to the DagBag would ideal.

Right now, we drop a py script that will load pickled DAG objects and makes
them available to the DagBag upon its instantiation. The work around we are
using can be seen here
https://github.com/dhavala/airflow-exp

I am wondering if you have any plans to inject DAG objects via methods
available on the "dag" class or something similar.

thanks,
-soma


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg03481.html on line 94
Call Stack
#TimeMemoryFunctionLocation
10.0006363672{main}( ).../msg03481.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg03481.html on line 94
Call Stack
#TimeMemoryFunctionLocation
10.0006363672{main}( ).../msg03481.html:0