[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Fusing operators together

*[Please let me know if this is NOT the correct place for such a query]*

Hello maintainers and committers,
I've stumbled upon this design decision for my Airflow project. Any
pointers would be helpful.


   - I'm in the process of deploying Airflow and I've felt the need to
   merge groups of operators that form a single logical task (to clear the
   clutter in huge DAGs)
   - The most common use-case would be coupling an operator and the
   corresponding sensor. For instance, one might want to chain together the
   EmrStepOperator and EmrStepSensor


Possible approaches

   - This could be achieved by offloading actual logic to Hooks and then
   using as many hooks as needed within an operator
   - A hacky alternative (if at all) would be SubDagOperator



   - Are hooks the right tool for this problem?
   - Any other way to compose operators together?
   - Is it a good idea to combine operators at all?

Here's <> my complete (more
elaborate) question on StackOverflow


*Shubham Gupta*
Software Engineer