git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

The need for LocalTaskJob


Hi Max, Dan et al,

Currently, when a scheduled task runs this happens in three steps: 

1. Worker 
2. LocalTaskJob
3. Raw task instance

It uses (by default) 5 (!) different processes:

1. Worker 
2. Bash + Airflow
3. Bash + Airflow 

I think we can merge worker and LocalTaskJob as the latter seems exist only to track a particular task. This can be done within the worker without side effects. Next to thatI think we can limit the amount of (airflow) processes to 2 if we remove the bash dependency. I don’t see any reason to depend on bash.

Can you guys shed some light on what the thoughts were around those choices? Am I missing anything on why they should exist?

Cheers
Bolke

Verstuurd vanaf mijn iPad


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg04094.html on line 90
Call Stack
#TimeMemoryFunctionLocation
10.0006358472{main}( ).../msg04094.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg04094.html on line 90
Call Stack
#TimeMemoryFunctionLocation
10.0006358472{main}( ).../msg04094.html:0