[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Scheduler gets slow with increasing number of Dags

Hi All,

We have a use case where there are 100s of DAGs in the scheduler's local dag folder but at a time only ~50 dags are active(Other dags are disabled). New dags keep on adding to the local Dag folder.
We are observing that scheduler is taking lot of time(around 20 minutes) in picking those newly created dags.
We are using following conf
Machine config: 8 core, 16 GB RAM
Executor: LocalExecutor
Parallelism: 50
max_thread: 4,
Dir list interval: 60.

Is there any recommended setting or config to optimise the scheduler's dag listing and parsing.