git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Openstack] [Openstack-operators] Recovering from full outage


On 07/12/2018 08:20 AM, Torin Woltjer wrote:
> The neutron-metadata-agent service is running, the the agent is alive, 
> and it is listening on port 8775. However, new instances still do not 
> get any information like hostname or keypair. If I run `curl 
> 192.168.116.22:8775` from the compute nodes, I do get a response. The 
> metadata agent is running, listening, and accessible from the compute 
> nodes; and it worked previously.
> 
> I'm stumped.

There is also a metadata proxy that runs in the qrouter namespace, you 
can verify it's running and getting requests by looking at both iptables 
and netstat output.

$ sudo ip netns exec qrouter-$ID iptables-save -c | grep 169
[16:960] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p 
tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
[96:7968] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ 
-p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff

The numbers inside [] represent packets:bytes, so non-zero is good.

$ sudo ip netns exec qrouter-$ID netstat -anep | grep 9697
tcp        0      0 0.0.0.0:9697            0.0.0.0:* 
LISTEN      0          294339      4867/haproxy

If you have a running instance you can log into, running curl to the 
metadata IP would be helpful to try and diagnose since it would go 
through this entire path.

-Brian


> /*Torin Woltjer*/
> *Grand Dial Communications - A ZK Tech Inc. Company*
> *616.776.1066 ext. 2006*
> /*<http://www.granddial.com>www.granddial.com <http://www.granddial.com>*/
> 
> ------------------------------------------------------------------------
> *From*: �ர�ண� ��மார� (Arun Kumar) <thangam.arunx at gmail.com>
> *Sent*: 7/12/18 12:01 AM
> *To*: torin.woltjer at granddial.com
> *Cc*: "openstack at lists.openstack.org" <openstack at lists.openstack.org>, 
> openstack-operators at lists.openstack.org
> *Subject*: Re: [Openstack-operators] [Openstack] Recovering from full outage
> Hi Torin,
> 
>     If I run `ip netns exec qrouter netstat -lnp` or `ip netns exec
>     qdhcp netstat -lnp` on the controller, should I see anything
>     listening on the metadata port (8775)? When I run these commands I
>     don't see that listening, but I have no example of a working system
>     to check against. Can anybody verify this?
> 
> 
> Either on qrouter/qdhcp namespaces, you won't see port 8775, instead 
> check whether meta-data service is running on the neutron controller 
> node(s) and listening on port 8775? Aslo, you can verify metadata and 
> neturon services using following commands
> 
> service neutron-metadata-agent status
> neutron agent-list
> netstat -ntplua | grep :8775
> 
> 
> Thanks & Regards
> Arun
> 
> à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?
> �ன�ப��ன�
> �ர�ண�
> ந���பம� நம�ம�ழியில� தழ������ ��ய�வ�ம�
> <http://thangamaniarun.wordpress.com><http://thangamaniarun.wordpress.com>http://thangamaniarun.wordpress.com
> à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?à®?
> 
> 
> 
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 




( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/openstack/msg39926.html on line 151
Call Stack
#TimeMemoryFunctionLocation
10.0011368600{main}( ).../msg39926.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/openstack/msg39926.html on line 151
Call Stack
#TimeMemoryFunctionLocation
10.0011368600{main}( ).../msg39926.html:0