Re: Large number of SQL Connection from airflow scheduler
What might also cause a big number of connections is using Airflow
Variables at the top level of the parsed DAGs. There was a discussion
earlier in the devlist (
that with each variable a new database connection is established. So maybe
that's the root cause of the problem?
On Fri, Dec 7, 2018 at 4:20 AM Kevin Yang <yrqls21@xxxxxxxxx> wrote:
> Hi Anand,
> From my experience that reusing connection pool across those parsing
> processes would be dangerous( recall I got some connection corruption error
> because of that). If you find that to be too many connection you can
> probably tune down the max_threads
> reduce the number of parsing processes or make the scheduler parse slower
> by tune down min_file_process_interval
> Kevin Y
> On Thu, Dec 6, 2018 at 5:53 PM anandgupta1412@xxxxxxxxx <
> anandgupta1412@xxxxxxxxx> wrote:
> > HI,
> > I have a query. It will be great if somebody in the community can clarify
> > it.
> > Basically, for each dag_processing process, I can see the orm being
> > reconfigured and disposed towards the end. This is creating and closing a
> > lot of connections on MySQL server, which crashes in scenarios where
> > are a lot of new dags to process.
> > I want to know if this is the intended behavior and why? Also what kind
> > improvement can be planned in this area so that we can reuse the
> > pool amongst these processes.
> > Thanks,
> > Anand
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
We create human & business stories through technology.
Check out our projects! <https://www.polidea.com/our-work>
[image: Github] <https://github.com/Polidea> [image: Facebook]
<https://www.facebook.com/Polidea.Software> [image: Twitter]
<https://twitter.com/polidea> [image: Linkedin]
<https://www.linkedin.com/company/polidea> [image: Instagram]
<https://instagram.com/polidea> [image: Behance]