[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Wiring top-level DAGs together

Hello maintainers,

I need to have several identical (differing only in arguments) top-level
DAGs that can also be triggered together with following constraints /

   - Individual top-level DAGs will have schedule_interval=None as they
   will only need occasional manual triggering
   - The series (wiring-up) of DAGs, however, needs to run daily
   - Order and number of DAGs in series is fixed (known ahead of writing
   code) and changes rarely (probably once in a month or so)


What would be a good strategy to wire-up these top-level DAGs so that they
run can be run together as well as independently? (or is there something
fundamentally wrong with this idea?)


I can think of SubDagOperator (problematic) and TriggerDagRunOperator
(satisfactory) but neither appear very tidy. Is there a better way? I am
open to complete re-structuring of my workflow as I am yet to deploy it.


Here's the link
to my question on SO.
And here are some related links that I went through

   - Airflow, mark a task success or skip it before dag run
   - AIrflow - Splitting DAG definition across multiple files
   - Airflow DAG success callback
   - How to set dependencies between DAGs in Airflow?
   - Airflow: When Your DAG is Far Behind The Schedule

*Shubham Gupta*
Software Engineer