[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

SubdagOperator and Pools


we have a SubdagOperator with lots of tasks in it. We want to limit the parallelism, with which these tasks execute. Therefore we created a pool and added the tasks within the SubdagOperator to this pool.

However, this setting is not respected (see image attached).

Now we am wondering why that is. In '' on the master branch there is a comment that

    "Airflow pool is not honored by SubDagOperator."

This comment is not in the file in v1.9.0 (which I am using).

So this means that Pools are not respected for Subdags?

On the other handside it states that Subdags use the SequentialExecutor, which *should* execute tasks sequentially?

Can anyone clarify this, please?
And if pools do not work, what options do we have to limit parallelism in a Subdag?

Thanks in advance,

( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg04169.html on line 99
Call Stack
10.0001357752{main}( ).../msg04169.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg04169.html on line 99
Call Stack
10.0001357752{main}( ).../msg04169.html:0