Large number of SQL Connection from airflow scheduler
I have a query. It will be great if somebody in the community can clarify it.
Basically, for each dag_processing process, I can see the orm being reconfigured and disposed towards the end. This is creating and closing a lot of connections on MySQL server, which crashes in scenarios where there are a lot of new dags to process.
I want to know if this is the intended behavior and why? Also what kind of improvement can be planned in this area so that we can reuse the connection pool amongst these processes.