Scheduler gets slow with increasing number of Dags
We have a use case where there are 100s of DAGs in the scheduler's local dag folder but at a time only ~50 dags are active(Other dags are disabled). New dags keep on adding to the local Dag folder.
We are observing that scheduler is taking lot of time(around 20 minutes) in picking those newly created dags.
We are using following conf
Machine config: 8 core, 16 GB RAM
Dir list interval: 60.
Is there any recommended setting or config to optimise the scheduler's dag listing and parsing.