git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Neutron] memchached connections


I will apply the workaround and keep watching it.
Given comment #7, do we know if
https://review.opendev.org/#/c/742193/ is the right fix yet?

Thanks!
Tony
> -----Original Message-----
> From: RadosÅ?aw Piliszek <radoslaw.piliszek at gmail.com>
> Sent: Saturday, September 12, 2020 2:41 AM
> To: Tony Liu <tonyliu0592 at hotmail.com>
> Cc: openstack-discuss <openstack-discuss at lists.openstack.org>
> Subject: Re: [Neutron] memchached connections
> 
> I believe you are hitting [1].
> 
> We have recently worked around that in Kolla-Ansible [2].
> 
> We'd like to get clarifications on whether the workaround is the right
> choice but it seems to work. :-)
> 
> [1] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659
> [2] https://review.opendev.org/746966
> 
> -yoctozepto
> 
> On Sat, Sep 12, 2020 at 4:34 AM Tony Liu <tonyliu0592 at hotmail.com> wrote:
> >
> > I restarted neutron-server on all 3 nodes. Those memcached connections
> > from neutron-server are gone. Everything is back to normal. It seems
> > like that memcached connections are not closed properly in
> > neutron-server. The connections pile up over time. Is there any know
> > issue related? Could any Neutron experts comment here?
> >
> > Thanks!
> > Tony
> > > -----Original Message-----
> > > From: Tony Liu <tonyliu0592 at hotmail.com>
> > > Sent: Friday, September 11, 2020 2:26 PM
> > > To: openstack-discuss <openstack-discuss at lists.openstack.org>
> > > Subject: memchached connections
> > >
> > > Hi,
> > >
> > > Is there any guidance or experiences to estimate the number of
> > > memcached connections?
> > >
> > > Here is memcached connection on one of the 3 controllers.
> > > Connection number is the total established connections to all 3
> > > memcached nodes.
> > >
> > > Node 1:
> > > 10 Keystone workers have 62 connections.
> > > 11 Nova API workers have 37 connections.
> > > 6 Neutron server works have 4304 connections.
> > > 1 memcached has 4973 connections.
> > >
> > > Node 2:
> > > 10 Keystone workers have 62 connections.
> > > 11 Nova API workers have 30 connections.
> > > 6 Neutron server works have 3703 connections.
> > > 1 memcached has 4973 connections.
> > >
> > > Node 3:
> > > 10 Keystone workers have 54 connections.
> > > 11 Nova API workers have 15 connections.
> > > 6 Neutron server works have 6541 connections.
> > > 1 memcached has 4973 connections.
> > >
> > > Before I increase the connection limit for memcached, I'd like to
> > > understand if all the above is expected?
> > >
> > > How Neutron server and memcached take so many connections?
> > >
> > > Any elaboration is appreciated.
> > >
> > > BTW, the problem leading me here is memcached connection timeout,
> > > which results all services depending on memcached stop working
> properly.
> > >
> > >
> > > Thanks!
> > > Tony
> >
> >