git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Delay between tasks executions issue


We use max_threads = number of scheduler cores.

On 2018/09/11 09:49:53, Chandu Kavar <cckavar@xxxxxxxxx> wrote: 
> Thanks Raman,
> 
> Understood.
> 
> We have around 500 DAGs. What value do you suggest for max_threads?
> 
> On Tue, Sep 11, 2018, 5:44 PM ramandumcs@xxxxxxxxx <ramandumcs@xxxxxxxxx>
> wrote:
> 
> > Hi Chandu,
> > How many dag files are there on the scheduler. As per my understanding
> > scheduler processes each file to trigger any dag/task run. It spawns number
> > of processes equivalent to "max_threads" count to parallelize file
> > processing. So you can try by increasing airflow config's max_threads count.
> >
> > Thanks,
> > Raman Gupta
> >
> > On 2018/09/11 07:11:23, Chandu Kavar <cckavar@xxxxxxxxx> wrote:
> > > Hi All,
> > >
> > > We are seeing there is a delay between triggering the next task. Please
> > > find the attached screenshot.
> > >
> > > First and the last task are very small. So, you can see the tiny object.
> > >
> > > [image: image.png]
> > > Do you suggest any airflow config can resolve this problem?
> > >
> > > Thanks,
> > > Chandu
> > >
> >
>