git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Fusing operators together


*[Please let me know if this is NOT the correct place for such a query]*

Hello maintainers and committers,
I've stumbled upon this design decision for my Airflow project. Any
pointers would be helpful.

Overview

   - I'm in the process of deploying Airflow and I've felt the need to
   merge groups of operators that form a single logical task (to clear the
   clutter in huge DAGs)
   - The most common use-case would be coupling an operator and the
   corresponding sensor. For instance, one might want to chain together the
   EmrStepOperator and EmrStepSensor


----

Possible approaches

   - This could be achieved by offloading actual logic to Hooks and then
   using as many hooks as needed within an operator
   - A hacky alternative (if at all) would be SubDagOperator


----

Questions

   - Are hooks the right tool for this problem?
   - Any other way to compose operators together?
   - Is it a good idea to combine operators at all?


Here's <https://stackoverflow.com/questions/53308306> my complete (more
elaborate) question on StackOverflow

Thanks

*Shubham Gupta*
Software Engineer
 zomato