git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RuntimeError: OVS transaction timed out - After minor upgrade


Is your ovs-vswitchd process running on controllers? Sounds like agent
can talk to ovsdb (it was able to connect) but then times out when
waiting for response from vswitchd process.

Do these command run successfully on the problematic controller?
ovs-vsctl show
ovs-ofctl show br-int

Kuba

On 10/04/2019 14:20, Yedhu Sastri wrote:
> Dear All,
> 
> We did a minor upgrade on our OpenStack environment Newton HA cluster from
> 14.2.12 to 14.2.16 using the following link.
> 
> https://docs.openstack.org/openstack-ansible/newton/upgrade-guide/minor-upgrade.html
> 
> The upgrade was successful and we tested creation of VM's and it was also
> successful we were able to ssh into the VM's.
> 
> Then we rebooted the controllers one by one. But after that we can create
> the VM's but it is not getting IP from dhcp agent.
> 
> In the openvswitch-agent.log we are getting 'OVS transaction timeout'. Any
> help is much appreciated.
> 
> 2019-04-09 17:25:29.063 3504 INFO
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] port_unbound():
> net_uuid None not managed by VLAN manager
> 2019-04-09 17:25:29.088 3504 INFO neutron.agent.common.ovs_lib
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Port
> 04275c8a-2d12-477b-b435-f6483d418e93 not present in bridge br-int
> 2019-04-09 17:25:29.089 3504 INFO
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] port_unbound():
> net_uuid None not managed by VLAN manager
> 2019-04-09 17:25:29.198 3504 INFO
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] port_unbound():
> net_uuid None not managed by VLAN manager
> 2019-04-09 17:25:29.228 3504 INFO neutron.agent.common.ovs_lib
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Port
> 5c25b580-9dcb-4930-a2fe-9cab951114dd not present in bridge br-int
> 2019-04-09 17:25:29.228 3504 INFO
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] port_unbound():
> net_uuid None not managed by VLAN manager
> 2019-04-09 17:25:29.229 3504 INFO neutron.agent.securitygroups_rpc
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Remove device filter
> for [u'63cb0c6a-6e82-41a8-ad46-511ee84579ad',
> u'9764e98a-bb32-4d37-9460-5a546f351e5a',
> u'2c075f29-156c-4787-b40f-434c437164e4',
> u'f00de46f-301a-4789-add1-a8239ee6c859',
> u'e26c1806-8041-4ce2-aea0-74ca469add67',
> u'9fa17447-d25d-441b-9983-b81e43c6e6d2',
> u'3ef24ce6-19b5-47c8-9510-767e64d33e9f',
> u'c2d37add-65a0-4fc9-914c-2382af70b1ca',
> u'a8b7c7aa-a5d0-47b3-a944-7b6c4428eda8',
> u'bf6f7608-0573-43f7-822a-878d7708d985',
> u'046670cd-77dd-4ce8-bb4b-f1489264375b',
> u'b7aefa18-d768-47a8-bb5f-6b1e838deaeb',
> u'04275c8a-2d12-477b-b435-f6483d418e93',
> u'124eb5e6-2b8a-44a9-aa8b-f6d9bc87f13f',
> u'5c25b580-9dcb-4930-a2fe-9cab951114dd']
> 2019-04-09 17:27:09.740 3504 INFO neutron.agent.securitygroups_rpc
> [req-5984df4b-c30f-403c-8e87-86c4ccf50a9f - - - - -] Provider rule updated
> 2019-04-09 17:27:41.961 3504 INFO neutron.agent.securitygroups_rpc
> [req-67c36a44-f778-48f0-b7ae-0a5244ff95c1 - - - - -] Provider rule updated
> 2019-04-09 17:28:28.334 3504 INFO neutron.agent.securitygroups_rpc
> [req-aeffc24f-a354-412e-b458-950c1d4b52ef - - - - -] Provider rule updated
> 2019-04-09 17:28:46.276 3504 INFO neutron.agent.securitygroups_rpc
> [req-2dd51325-40c4-4ff0-a384-bccbe3ca0bfc - - - - -] Provider rule updated
> 2019-04-09 17:28:46.278 3504 INFO neutron.agent.securitygroups_rpc
> [req-6d48bfd9-0230-4a82-a477-d9e4171db519 - - - - -] Provider rule updated
> 2019-04-09 17:30:32.695 3504 INFO neutron.agent.securitygroups_rpc
> [req-01b9b7a8-84a4-42cc-aebb-db846eb2b8f0 - - - - -] Provider rule updated
> 2019-04-09 17:30:54.288 3504 INFO neutron.agent.securitygroups_rpc
> [req-de1755bd-0e21-4b75-9fde-db422ee9a79a - - - - -] Provider rule updated
> 2019-04-09 17:32:12.377 3504 INFO neutron.agent.securitygroups_rpc
> [req-26bc4547-edb7-4092-b4f4-6465c29dfbe4 - - - - -] Provider rule updated
> 2019-04-09 17:32:21.444 3504 WARNING
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Device
> 0d535260-dd98-4027-a2ba-4c5412d4eab0 not defined on plugin or binding failed
> 2019-04-09 17:32:21.461 3504 INFO
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Port
> d3baa557-a367-4a6d-8d58-7183df4243a6 updated. Details: {u'profile': {},
> u'network_qos_policy_id': None, u'qos_policy_id': None,
> u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id':
> u'680c9bdd-9cea-4059-8c8c-2928dd0ee48f', u'segmentation_id': 65606,
> u'device_owner': u'network:router_ha_interface', u'physical_network': None,
> u'mac_address': u'fa:16:3e:b3:9a:60', u'device':
> u'd3baa557-a367-4a6d-8d58-7183df4243a6', u'port_security_enabled': False,
> u'port_id': u'd3baa557-a367-4a6d-8d58-7183df4243a6', u'fixed_ips':
> [{u'subnet_id': u'5b49f8e5-991d-4ecd-a584-df605063aecb', u'ip_address':
> u'169.254.192.9'}], u'network_type': u'vxlan'}
> 2019-04-09 17:32:21.461 3504 INFO
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Assigning 1 as local
> vlan for net-id=680c9bdd-9cea-4059-8c8c-2928dd0ee48f
> 2019-04-09 17:32:35.477 3504 INFO oslo_messaging._drivers.amqpdriver [-] No
> calling threads waiting for msg_id : a9314d944ff8486abf909d4cc271f9a0
> 2019-04-09 17:32:39.016 3504 INFO neutron.agent.securitygroups_rpc
> [req-51453b2c-b454-45bd-b7f8-c78560437016 - - - - -] Provider rule updated
> 2019-04-09 17:32:54.141 3504 ERROR
> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Switch connection
> timeout
> 2019-04-09 17:32:57.964 3504 INFO
> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Bridge br-tun changed
> its datapath-ID from 6a740a5da349 to 00006a740a5da349
> 2019-04-09 17:32:58.015 3504 INFO
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Port
> 0d8dc3c9-c115-45f8-bbb2-cc715b7c1079 updated. Details: {u'profile': {},
> u'network_qos_policy_id': None, u'qos_policy_id': None,
> u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id':
> u'4ffb2c28-f9f1-4a25-82a0-df7f7a002434', u'segmentation_id': 625,
> u'device_owner': u'network:router_gateway', u'physical_network':
> u'outbound1', u'mac_address': u'fa:16:3e:aa:d7:56', u'device':
> u'0d8dc3c9-c115-45f8-bbb2-cc715b7c1079', u'port_security_enabled': False,
> u'port_id': u'0d8dc3c9-c115-45f8-bbb2-cc715b7c1079', u'fixed_ips':
> [{u'subnet_id': u'764f08fc-1979-402c-b93c-834d7148a8a5', u'ip_address':
> u'10.97.179.34'}], u'network_type': u'vlan'}
> 2019-04-09 17:32:58.016 3504 INFO
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Assigning 2 as local
> vlan for net-id=4ffb2c28-f9f1-4a25-82a0-df7f7a002434
> 2019-04-09 17:32:58.030 3504 INFO
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Port
> 1239fa6c-265a-4f16-ab00-c86fe40545d9 updated. Details: {u'profile': {},
> u'network_qos_policy_id': None, u'qos_policy_id': None,
> u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id':
> u'4ffb2c28-f9f1-4a25-82a0-df7f7a002434', u'segmentation_id': 625,
> u'device_owner': u'network:router_gateway', u'physical_network':
> u'outbound1', u'mac_address': u'fa:16:3e:4d:73:46', u'device':
> u'1239fa6c-265a-4f16-ab00-c86fe40545d9', u'port_security_enabled': False,
> u'port_id': u'1239fa6c-265a-4f16-ab00-c86fe40545d9', u'fixed_ips':
> [{u'subnet_id': u'764f08fc-1979-402c-b93c-834d7148a8a5', u'ip_address':
> u'10.97.179.99'}], u'network_type': u'vlan'}
> 2019-04-09 17:33:13.289 3504 ERROR neutron.agent.ovsdb.impl_idl
> [req-04b10d7a-3f2c-4eda-8d86-c36f994323d9 - - - - -] Traceback (most recent
> call last):
>   File
> "/openstack/venvs/neutron-14.2.16/lib/python2.7/site-packages/neutron/agent/ovsdb/native/connection.py",
> line 117, in run
>     txn.results.put(txn.do_commit())
>   File
> "/openstack/venvs/neutron-14.2.16/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py",
> line 91, in do_commit
>     raise RuntimeError(_("OVS transaction timed out"))
> RuntimeError: OVS transaction timed out
> 
> 
>