git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Large number of SQL Connection from airflow scheduler


Hi Anand,
>From my experience that reusing connection pool across those parsing
processes would be dangerous( recall I got some connection corruption error
because of that). If you find that to be too many connection you can
probably tune down the max_threads
<https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L507>
to
reduce the number of parsing processes or make the scheduler parse slower
by tune down min_file_process_interval
<https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L464>

Cheers,
Kevin Y

On Thu, Dec 6, 2018 at 5:53 PM anandgupta1412@xxxxxxxxx <
anandgupta1412@xxxxxxxxx> wrote:

> HI,
>
> I have a query. It will be great if somebody in the community can clarify
> it.
>
> Basically, for each dag_processing process, I can see the orm being
> reconfigured and disposed towards the end. This is creating and closing a
> lot of connections on MySQL server, which crashes in scenarios where there
> are a lot of new dags to process.
>
> I want to know if this is the intended behavior and why? Also what kind of
> improvement can be planned in this area so that we can reuse the connection
> pool amongst these processes.
>
> Thanks,
> Anand
>