Ubuntu
openvswitch package

  1. Focal (20.04)
  2. Bug #1928031
  3. Activity log

Activity log for bug #1928031

Date Who What changed Old value New value Message
2021-05-11 03:01:00 Hemanth Nakkina bug added bug
2021-05-11 03:17:58 Hemanth Nakkina description neutron-ovn-metadata-agent not able to handle any metadata requests from the instances. Scenario: * Initially there is some intermittent connectivity issues that are descirbed in LP #1907686 https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/comments/9 * The fix for the above is available in python3-openvswitch package in ussuri-proposed pocket Installed the fix on all neutron-server and compute and restarted neutron-ovn-metadata-agent one by one * neutron-ovn-metadata-agent on one of the compute nodes not able to handle any metadata requests after restart. ( Please note the problem happened with only one ovn-metadata agent and rest of the agents are good on other compute nodes, so this is some race condition in IDL) Stacktrace shows both the workers 69188/69189 timed out on OVNSB IDL connection and hence sb_idl is never initialized. Stacktrace: 2021-04-15 22:27:03.803 69124 INFO neutron.common.config [-] /usr/bin/neutron-ovn-metadata-agent version 16.2.0 2021-04-15 22:27:03.832 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connecting... 2021-04-15 22:27:03.833 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connected 2021-04-15 22:27:03.949 69124 WARNING neutron.agent.ovn.metadata.agent [-] Can't read ovn-bridge external-id from OVSDB. Using br-int instead. 2021-04-15 22:27:03.950 69124 INFO oslo_service.service [-] Starting 2 workers 2021-04-15 22:27:03.985 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:03.986 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:04.005 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:04.006 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:04.033 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:04.061 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:06.129 69124 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/neutron_ovn_metadata_agent.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpgncr2rq7/privsep.sock'] 2021-04-15 22:27:06.757 69124 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2021-04-15 22:27:06.676 69211 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-04-15 22:27:06.678 69211 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none 2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep daemon running as pid 69211 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last): 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start() 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.546 69188 INFO eventlet.wsgi.server [-] (69188) wsgi starting up on http:/var/lib/neutron/metadata_proxy 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last): 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start() 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start() 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.555 69189 INFO eventlet.wsgi.server [-] (69189) wsgi starting up on http:/var/lib/neutron/metadata_proxy * However netstat shows the TCP connection to SB IDL is established for both the workers. So this could be some problem in data at IDL. $ grep 10.216.241.118 sosreport-cmp4az1cz20300kvh-00304240-2021-04-27-edidnbl/sos_commands/networking/netstat_-W_-neopa tcp 0 0 10.216.241.244:53284 10.216.241.118:6642 ESTABLISHED 0 407222779 69124/neutron-ovn-m off (0.00/0/0) tcp 1490260 0 10.216.241.244:53282 10.216.241.118:6642 ESTABLISHED 0 407288937 69189/neutron-ovn-m off (0.00/0/0) tcp 1490260 0 10.216.241.244:53280 10.216.241.118:6642 ESTABLISHED 0 407156544 69188/neutron-ovn-m off (0.00/0/0) It will be good if sb_idl is not initialised, the worker thread can be closed or neutron-ovn-metadata-agent should be stopped so that the problem will be notified to operator at the earliest. neutron-ovn-metadata-agent not able to handle any metadata requests from the instances. Scenario: * Initially there is some intermittent connectivity issues that are descirbed in LP #1907686   https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/comments/9 * The fix for the above is available in python3-openvswitch package in ussuri-proposed pocket   Installed the fix on all neutron-server and compute and restarted neutron-ovn-metadata-agent one by one * neutron-ovn-metadata-agent on one of the compute nodes not able to handle any metadata requests after restart. ( Please note the problem happened with only one ovn-metadata agent and rest of the agents are good on other compute nodes, so this is some race condition in IDL) Stacktrace shows both the workers 69188/69189 timed out on OVNSB IDL connection and hence sb_idl is never initialized. Stacktrace of Attribute error: ------------------------------ 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server Traceback (most recent call last): 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 67, in __call__ 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server instance_id, project_id = self._get_instance_and_project_id(req) 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 84, in _get_instance_and_project_id 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server ports = self.sb_idl.get_network_port_bindings_by_ip(network_id, 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl' 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server Stacktrace at the restart of neutron-ovn-metadata-agent process: ---------------------------------------------------------------- 2021-04-15 22:27:03.803 69124 INFO neutron.common.config [-] /usr/bin/neutron-ovn-metadata-agent version 16.2.0 2021-04-15 22:27:03.832 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connecting... 2021-04-15 22:27:03.833 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connected 2021-04-15 22:27:03.949 69124 WARNING neutron.agent.ovn.metadata.agent [-] Can't read ovn-bridge external-id from OVSDB. Using br-int instead. 2021-04-15 22:27:03.950 69124 INFO oslo_service.service [-] Starting 2 workers 2021-04-15 22:27:03.985 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:03.986 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:04.005 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:04.006 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:04.033 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:04.061 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:06.129 69124 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/neutron_ovn_metadata_agent.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpgncr2rq7/privsep.sock'] 2021-04-15 22:27:06.757 69124 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2021-04-15 22:27:06.676 69211 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-04-15 22:27:06.678 69211 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none 2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep daemon running as pid 69211 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last): 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start() 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.546 69188 INFO eventlet.wsgi.server [-] (69188) wsgi starting up on http:/var/lib/neutron/metadata_proxy 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last): 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start() 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start() 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.555 69189 INFO eventlet.wsgi.server [-] (69189) wsgi starting up on http:/var/lib/neutron/metadata_proxy * However netstat shows the TCP connection to SB IDL is established for both the workers. So this could be some problem in data at IDL. $ grep 10.216.241.118 sosreport-cmp4az1cz20300kvh-00304240-2021-04-27-edidnbl/sos_commands/networking/netstat_-W_-neopa tcp 0 0 10.216.241.244:53284 10.216.241.118:6642 ESTABLISHED 0 407222779 69124/neutron-ovn-m off (0.00/0/0) tcp 1490260 0 10.216.241.244:53282 10.216.241.118:6642 ESTABLISHED 0 407288937 69189/neutron-ovn-m off (0.00/0/0) tcp 1490260 0 10.216.241.244:53280 10.216.241.118:6642 ESTABLISHED 0 407156544 69188/neutron-ovn-m off (0.00/0/0) It will be good if sb_idl is not initialised, the worker thread can be closed or neutron-ovn-metadata-agent should be stopped so that the problem will be notified to operator at the earliest.
2021-05-11 03:25:20 Hemanth Nakkina description neutron-ovn-metadata-agent not able to handle any metadata requests from the instances. Scenario: * Initially there is some intermittent connectivity issues that are descirbed in LP #1907686   https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/comments/9 * The fix for the above is available in python3-openvswitch package in ussuri-proposed pocket   Installed the fix on all neutron-server and compute and restarted neutron-ovn-metadata-agent one by one * neutron-ovn-metadata-agent on one of the compute nodes not able to handle any metadata requests after restart. ( Please note the problem happened with only one ovn-metadata agent and rest of the agents are good on other compute nodes, so this is some race condition in IDL) Stacktrace shows both the workers 69188/69189 timed out on OVNSB IDL connection and hence sb_idl is never initialized. Stacktrace of Attribute error: ------------------------------ 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server Traceback (most recent call last): 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 67, in __call__ 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server instance_id, project_id = self._get_instance_and_project_id(req) 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 84, in _get_instance_and_project_id 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server ports = self.sb_idl.get_network_port_bindings_by_ip(network_id, 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl' 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server Stacktrace at the restart of neutron-ovn-metadata-agent process: ---------------------------------------------------------------- 2021-04-15 22:27:03.803 69124 INFO neutron.common.config [-] /usr/bin/neutron-ovn-metadata-agent version 16.2.0 2021-04-15 22:27:03.832 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connecting... 2021-04-15 22:27:03.833 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connected 2021-04-15 22:27:03.949 69124 WARNING neutron.agent.ovn.metadata.agent [-] Can't read ovn-bridge external-id from OVSDB. Using br-int instead. 2021-04-15 22:27:03.950 69124 INFO oslo_service.service [-] Starting 2 workers 2021-04-15 22:27:03.985 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:03.986 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:04.005 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:04.006 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:04.033 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:04.061 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:06.129 69124 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/neutron_ovn_metadata_agent.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpgncr2rq7/privsep.sock'] 2021-04-15 22:27:06.757 69124 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2021-04-15 22:27:06.676 69211 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-04-15 22:27:06.678 69211 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none 2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep daemon running as pid 69211 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last): 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start() 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.546 69188 INFO eventlet.wsgi.server [-] (69188) wsgi starting up on http:/var/lib/neutron/metadata_proxy 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last): 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start() 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start() 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.555 69189 INFO eventlet.wsgi.server [-] (69189) wsgi starting up on http:/var/lib/neutron/metadata_proxy * However netstat shows the TCP connection to SB IDL is established for both the workers. So this could be some problem in data at IDL. $ grep 10.216.241.118 sosreport-cmp4az1cz20300kvh-00304240-2021-04-27-edidnbl/sos_commands/networking/netstat_-W_-neopa tcp 0 0 10.216.241.244:53284 10.216.241.118:6642 ESTABLISHED 0 407222779 69124/neutron-ovn-m off (0.00/0/0) tcp 1490260 0 10.216.241.244:53282 10.216.241.118:6642 ESTABLISHED 0 407288937 69189/neutron-ovn-m off (0.00/0/0) tcp 1490260 0 10.216.241.244:53280 10.216.241.118:6642 ESTABLISHED 0 407156544 69188/neutron-ovn-m off (0.00/0/0) It will be good if sb_idl is not initialised, the worker thread can be closed or neutron-ovn-metadata-agent should be stopped so that the problem will be notified to operator at the earliest. neutron-ovn-metadata-agent not able to handle any metadata requests from the instances. Scenario: * Initially there is some intermittent connectivity issues that are descirbed in LP #1907686   https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/comments/9 * The fix for the above is available in python3-openvswitch package in ussuri-proposed pocket   Installed the fix on all neutron-server and compute and restarted neutron-ovn-metadata-agent one by one * neutron-ovn-metadata-agent on one of the compute nodes not able to handle any metadata requests after restart. ( Please note the problem happened with only one ovn-metadata agent and rest of the agents are good on other compute nodes, so this is some race condition in IDL) Stacktrace shows both the workers 69188/69189 timed out on OVNSB IDL connection and hence sb_idl is never initialized. Stacktrace of Attribute error: ------------------------------ 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server Traceback (most recent call last): 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 67, in __call__ 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server instance_id, project_id = self._get_instance_and_project_id(req) 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 84, in _get_instance_and_project_id 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server ports = self.sb_idl.get_network_port_bindings_by_ip(network_id, 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl' 2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server Stacktrace at the restart of neutron-ovn-metadata-agent process: ---------------------------------------------------------------- 2021-04-15 22:27:03.803 69124 INFO neutron.common.config [-] /usr/bin/neutron-ovn-metadata-agent version 16.2.0 2021-04-15 22:27:03.832 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connecting... 2021-04-15 22:27:03.833 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connected 2021-04-15 22:27:03.949 69124 WARNING neutron.agent.ovn.metadata.agent [-] Can't read ovn-bridge external-id from OVSDB. Using br-int instead. 2021-04-15 22:27:03.950 69124 INFO oslo_service.service [-] Starting 2 workers 2021-04-15 22:27:03.985 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:03.986 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:04.005 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting... 2021-04-15 22:27:04.006 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:04.033 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:04.061 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected 2021-04-15 22:27:06.129 69124 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/neutron_ovn_metadata_agent.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpgncr2rq7/privsep.sock'] 2021-04-15 22:27:06.757 69124 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2021-04-15 22:27:06.676 69211 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-04-15 22:27:06.678 69211 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none 2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep daemon running as pid 69211 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last): 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start() 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout 2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.546 69188 INFO eventlet.wsgi.server [-] (69188) wsgi starting up on http:/var/lib/neutron/metadata_proxy 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last): 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start() 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout 2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start() 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager idlutils.wait_for_change(self.idl, self.timeout) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Exception: Timeout 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred: 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start() 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__ 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager self.start_connection(connection) 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager raise connection_exception 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager 2021-04-15 22:30:08.555 69189 INFO eventlet.wsgi.server [-] (69189) wsgi starting up on http:/var/lib/neutron/metadata_proxy * However netstat shows the TCP connection to SB IDL is established for both the workers. So this could be some problem in data at IDL. $ grep 10.216.241.118 sosreport-*-2021-04-27-edidnbl/sos_commands/networking/netstat_-W_-neopa tcp 0 0 10.216.241.244:53284 10.216.241.118:6642 ESTABLISHED 0 407222779 69124/neutron-ovn-m off (0.00/0/0) tcp 1490260 0 10.216.241.244:53282 10.216.241.118:6642 ESTABLISHED 0 407288937 69189/neutron-ovn-m off (0.00/0/0) tcp 1490260 0 10.216.241.244:53280 10.216.241.118:6642 ESTABLISHED 0 407156544 69188/neutron-ovn-m off (0.00/0/0) It will be good if sb_idl is not initialised, the worker thread can be closed or neutron-ovn-metadata-agent should be stopped so that the problem will be notified to operator at the earliest.
2021-05-12 11:10:08 Hemanth Nakkina tags sts
2021-05-20 06:31:32 Frode Nordahl bug task added neutron (Ubuntu)
2021-05-20 06:31:50 Frode Nordahl neutron (Ubuntu): status New Triaged
2021-05-20 06:31:53 Frode Nordahl neutron (Ubuntu): importance Undecided High
2021-05-20 06:32:03 Frode Nordahl charm-ovn-chassis: status New Invalid
2021-06-16 08:45:55 Hemanth Nakkina bug task added neutron
2021-06-16 08:46:24 Hemanth Nakkina neutron: assignee Hemanth Nakkina (hemanth-n)
2021-06-16 09:33:25 OpenStack Infra neutron: status New In Progress
2021-06-30 07:02:06 Hemanth Nakkina tags sts ovn sts
2021-07-02 20:52:57 OpenStack Infra neutron: status In Progress Fix Released
2021-07-06 17:03:51 OpenStack Infra tags ovn sts in-stable-wallaby ovn sts
2021-07-06 17:10:08 OpenStack Infra tags in-stable-wallaby ovn sts in-stable-victoria in-stable-wallaby ovn sts
2021-07-08 07:24:19 OpenStack Infra tags in-stable-victoria in-stable-wallaby ovn sts in-stable-ussuri in-stable-victoria in-stable-wallaby ovn sts
2021-07-21 14:01:01 Nobuto Murata bug added subscriber Nobuto Murata
2021-08-06 12:38:01 Bernard Cafarelli tags in-stable-ussuri in-stable-victoria in-stable-wallaby ovn sts in-stable-ussuri in-stable-victoria in-stable-wallaby neutron-proactive-backport-potential ovn sts
2021-11-24 08:58:15 Hemanth Nakkina neutron (Ubuntu): status Triaged Fix Released
2022-01-05 03:51:30 Hemanth Nakkina bug task added openvswitch (Ubuntu)
2022-01-05 03:52:07 Hemanth Nakkina nominated for series Ubuntu Focal
2022-01-05 03:52:07 Hemanth Nakkina bug task added openvswitch (Ubuntu Focal)
2022-01-05 03:52:07 Hemanth Nakkina bug task added neutron (Ubuntu Focal)
2022-01-05 03:56:04 Hemanth Nakkina bug task added cloud-archive
2022-01-05 03:56:25 Hemanth Nakkina nominated for series cloud-archive/wallaby
2022-01-05 03:56:25 Hemanth Nakkina bug task added cloud-archive/wallaby
2022-01-05 03:56:25 Hemanth Nakkina nominated for series cloud-archive/ussuri
2022-01-05 03:56:25 Hemanth Nakkina bug task added cloud-archive/ussuri
2022-01-07 10:36:37 James Page openvswitch (Ubuntu): status New Fix Released
2022-01-10 03:42:51 Hemanth Nakkina nominated for series Ubuntu Hirsute
2022-01-10 03:42:51 Hemanth Nakkina bug task added openvswitch (Ubuntu Hirsute)
2022-01-10 03:42:51 Hemanth Nakkina bug task added neutron (Ubuntu Hirsute)
2022-03-23 09:09:04 Hemanth Nakkina neutron (Ubuntu Hirsute): status New Won't Fix
2022-03-23 09:09:55 Hemanth Nakkina openvswitch (Ubuntu Hirsute): status New Won't Fix