Fusing operators together
*[Please let me know if this is NOT the correct place for such a query]*
Hello maintainers and committers,
I've stumbled upon this design decision for my Airflow project. Any
pointers would be helpful.
- I'm in the process of deploying Airflow and I've felt the need to
merge groups of operators that form a single logical task (to clear the
clutter in huge DAGs)
- The most common use-case would be coupling an operator and the
corresponding sensor. For instance, one might want to chain together the
EmrStepOperator and EmrStepSensor
- This could be achieved by offloading actual logic to Hooks and then
using as many hooks as needed within an operator
- A hacky alternative (if at all) would be SubDagOperator
- Are hooks the right tool for this problem?
- Any other way to compose operators together?
- Is it a good idea to combine operators at all?
Here's <https://stackoverflow.com/questions/53308306> my complete (more
elaborate) question on StackOverflow