git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: The need for LocalTaskJob


Alex (cc'd) brought this up to me about this a while ago too, and I agreed
with him. It is definitely something we should do, I remember there were
some things that were a bit tricky about removing the intermediate process
and would be a bit of work to fix (something about the tasks needing to
heartbeat the parent process maybe?).

TLDR: No blockers from me, just might be a bit of work to implement.


On Sat, Aug 4, 2018 at 9:15 AM Bolke de Bruin <bdbruin@xxxxxxxxx> wrote:

> Hi Max, Dan et al,
>
> Currently, when a scheduled task runs this happens in three steps:
>
> 1. Worker
> 2. LocalTaskJob
> 3. Raw task instance
>
> It uses (by default) 5 (!) different processes:
>
> 1. Worker
> 2. Bash + Airflow
> 3. Bash + Airflow
>
> I think we can merge worker and LocalTaskJob as the latter seems exist
> only to track a particular task. This can be done within the worker without
> side effects. Next to thatI think we can limit the amount of (airflow)
> processes to 2 if we remove the bash dependency. I don’t see any reason to
> depend on bash.
>
> Can you guys shed some light on what the thoughts were around those
> choices? Am I missing anything on why they should exist?
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg04096.html on line 106
Call Stack
#TimeMemoryFunctionLocation
10.0005363640{main}( ).../msg04096.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg04096.html on line 106
Call Stack
#TimeMemoryFunctionLocation
10.0005363640{main}( ).../msg04096.html:0