ovn: instance unable to retrieve metadata

Bug #1907686 reported by James Page
62
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
Ussuri
Fix Released
Undecided
Unassigned
Victoria
Won't Fix
Undecided
Unassigned
Wallaby
Fix Released
Undecided
Unassigned
charm-ovn-chassis
Invalid
Undecided
Unassigned
neutron
Invalid
Undecided
Unassigned
openvswitch (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
Hirsute
Fix Released
High
Unassigned

Bug Description

[Impact]
Cloud instances are unable to retrieve metadata on startup.

[Test Case]
Deploy OpenStack with OVN/OVS
Restart OVN central controllers
Create a new instance
Instance will fail to retrieve metadata with the message from the original bug report displayed in the metadata agent log on the local hypervisor

[Regression Potential]
The fix for this issue is included in the upstream 2.13.3 release of OVS.
The fix ensures that SSL related connection issues are correctly handling in python3-ovs avoiding an issue where the connection to the OVN SB IDL is reset and never recreated.
The OVN drivers use python3-ovsdbapp which in turn bases off code provided by python3-ovs.

[Original Bug Report]
Ubuntu:focal
OpenStack: ussuri
Instance port: hardware offloaded

instance created, attempts to access metadata - metadata agent can't resolve the port/network combination:

2020-12-10 15:00:18.258 4732 INFO neutron.agent.ovn.metadata.agent [-] Port d65418a6-d0e9-47e6-84ba-3d02fe75131a in datapath 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 bound to our ch
assis
2020-12-10 15:00:31.672 8062 ERROR neutron.agent.ovn.metadata.server [-] No port found in network 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 with IP address 10.5.1.155
2020-12-10 15:00:31.673 8062 INFO eventlet.wsgi.server [-] 10.5.1.155,<local> "GET /openstack HTTP/1.1" status: 404 len: 297 time: 0.0043790
2020-12-10 15:00:34.639 8062 ERROR neutron.agent.ovn.metadata.server [-] No port found in network 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 with IP address 10.5.1.155
2020-12-10 15:00:34.639 8062 INFO eventlet.wsgi.server [-] 10.5.1.155,<local> "GET /openstack HTTP/1.1" status: 404 len: 297 time: 0.0040138

Revision history for this message
James Page (james-page) wrote :

And another:

2020-12-10 15:00:21.013 4703 INFO neutron.agent.ovn.metadata.agent [-] Port f0cc41de-6f54-40c5-85cd-fdeb5249c154 in datapath 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 bound to our ch
assis
2020-12-10 15:00:34.017 7849 ERROR neutron.agent.ovn.metadata.server [-] No port found in network 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 with IP address 10.5.3.123
2020-12-10 15:00:34.018 7849 INFO eventlet.wsgi.server [-] 10.5.3.123,<local> "GET /openstack HTTP/1.1" status: 404 len: 297 time: 0.0043950
2020-12-10 15:00:36.664 7848 ERROR neutron.agent.ovn.metadata.server [-] No port found in network 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 with IP address 10.5.3.123
2020-12-10 15:00:36.665 7848 INFO eventlet.wsgi.server [-] 10.5.3.123,<local> "GET /openstack HTTP/1.1" status: 404 len: 297 time: 0.0045562

tags: added: ps5
Revision history for this message
James Page (james-page) wrote :
Download full text (4.9 KiB)

Port f0cc41de-6f54-40c5-85cd-fdeb5249c154 exists and has the correct IP

+-------------------------+---------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+---------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | ps5-rb2-n4.maas |
| binding_profile | capabilities='switchdev', pci_slot='0000:81:04.1', pci_vendor_info='15b3:101a', physical_network= |
| binding_vif_details | port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | direct |
| created_at | 2020-12-10T15:00:15Z |
| data_plane_status | None |
| description | |
| device_id | ff05ec43-8b15-4725-91eb-025b5dc37369 |
| device_owner | compute:AZ3 |
| dns_assignment | fqdn='ps5-bench-5.openstack.prodstack5.lan.', hostname='ps5-bench-5', ip_address='10.5.3.123' |
| dns_domain | |
| dns_name | ps5-bench-5 |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.5.3.123', subnet_id='6bb845b1-200d-4ee9-9a7e-bd9203cf0e6d' |
| id | f0cc41de-6f54-40c5-85cd-fdeb5249c154 |
| ip_allocation | immediate |
| mac_address | fa:16:3e:fb:75:59 |
| name | ps5-bench-5_port |
| network_id | 2864daa5-5336-419d-b0a5-beb...

Read more...

Revision history for this message
James Page (james-page) wrote :

Restarting the neutron-ovn-metadata-agent.service on the hypervisor cleared this issue so it looks like some sort of caching type problem in this service.

Revision history for this message
James Page (james-page) wrote :

Note that the fact the port/instance was hardware offloaded is not material here - I just tripped on the same issue with virtio ports.

summary: - instance unable to retrieve metadata
+ ovn: instance unable to retrieve metadata
Revision history for this message
Frode Nordahl (fnordahl) wrote :

Also see a few of these:
2021-01-12 06:48:21.947 52569 INFO eventlet.wsgi.server [-] 10.5.3.232,<local> "GET /openstack HTTP/1.1" status: 500 len: 362 time: 0.0006430
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server [-] Unexpected error.: AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl'
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server Traceback (most recent call last):
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 67, in __call__
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server instance_id, project_id = self._get_instance_and_project_id(req)
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 84, in _get_instance_and_project_id
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server ports = self.sb_idl.get_network_port_bindings_by_ip(network_id,
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl'
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server

Restarting the neutron-ovn-metadata-agent service resolves the issue.

tags: added: ovn
Revision history for this message
Seyeong Kim (seyeongkim) wrote :

The customer also faced this issue

tags: added: sts
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in neutron (Ubuntu):
status: New → Confirmed
Revision history for this message
James Troup (elmo) wrote :

We also saw this after a round of package upgrades on a cloud. python3-neutron went from 2:16.2.0-0ubuntu2~cloud0 to 2:16.2.0-0ubuntu3~cloud0 and ovn packages went from 20.03.1-0ubuntu1.1~cloud0 to 20.03.1-0ubuntu1.2~cloud0 (i.e. not an upstream version change).

Revision history for this message
Frode Nordahl (fnordahl) wrote :

While looking to reproduce this issue we found this in the logs of both the neutron-ovn-metadata-agent and neutron-server:

2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection [-] (111, 'ECONNREFUSED'): OpenSSL.SSL.SysCallError: (111, 'ECONNREFUSED')
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 106, in run
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection self.idl.run()
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 240, in run
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection self._session.run()
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 532, in run
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection self.__connect()
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 467, in __connect
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection error, self.stream = ovs.stream.Stream.open(name)
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/ovs/stream.py", line 192, in open
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection err = ovs.socket_util.check_connection_completion(sock)
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/ovs/socket_util.py", line 181, in check_connection_completion
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection sock.send("\0".encode(), socket.MSG_DONTWAIT)
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1737, in send
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection self._raise_ssl_error(self._ssl, result)
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection raise SysCallError(errno, errorcode.get(errno))
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection OpenSSL.SSL.SysCallError: (111, 'ECONNREFUSED')
2021-03-16 10:56:45.465 270932 ERROR ovsdbapp.backend.ovs_idl.connection

I suspect the log excerpts printed in comments above is fallout from this condition and that both issues will go away with a fixed python-ovs.

Revision history for this message
Frode Nordahl (fnordahl) wrote :

Fix proposed upstream:
http://patchwork.ozlabs.org/<email address hidden>/

Revision history for this message
Frode Nordahl (fnordahl) wrote :

With the proposed fix in place local validation allows me to successfully create instances just after a full cloud reboot as well as after synthetic test of restarting all the OVN ovsdb-server processes at once multiple times.

Revision history for this message
Frode Nordahl (fnordahl) wrote :

It was actually already fixed upstream in [0] and backported to branch-2.13 in [1] which is tagged in v2.13.3.

0: https://mail.openvswitch.org/pipermail/ovs-dev/2020-September/375050.html
1: https://github.com/openvswitch/ovs/commit/cdf2cf98e31ea4c99f62bf7a2c57a22050a71bbd

So this would be fixed by a new point release update in Ubuntu.

Changed in openvswitch (Ubuntu):
status: New → Triaged
importance: Undecided → High
Changed in neutron (Ubuntu):
status: Confirmed → Invalid
Changed in neutron:
status: New → Invalid
Changed in charm-ovn-chassis:
status: New → Invalid
Frode Nordahl (fnordahl)
no longer affects: neutron (Ubuntu)
no longer affects: neutron (Ubuntu Bionic)
no longer affects: neutron (Ubuntu Focal)
no longer affects: neutron (Ubuntu Groovy)
no longer affects: neutron (Ubuntu Hirsute)
Changed in openvswitch (Ubuntu Hirsute):
status: Triaged → Fix Released
Changed in openvswitch (Ubuntu Groovy):
status: New → Triaged
Changed in openvswitch (Ubuntu Focal):
status: New → Triaged
Changed in openvswitch (Ubuntu Bionic):
status: New → Triaged
Changed in openvswitch (Ubuntu Focal):
importance: Undecided → High
Changed in openvswitch (Ubuntu Groovy):
importance: Undecided → High
Changed in openvswitch (Ubuntu Bionic):
importance: Undecided → High
Revision history for this message
James Page (james-page) wrote :

2.13.3 uploaded for focal and groovy to:

  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690

for testing.

no longer affects: openvswitch (Ubuntu Bionic)
Revision history for this message
James Page (james-page) wrote :

The SRU for OVS 2.13.3 is tracked under bug 1920141

Revision history for this message
James Page (james-page) wrote :

Packages have now built in:

  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690

DPDK needed a fix for a regression introduced in the previous -update.

James Page (james-page)
description: updated
Revision history for this message
Edin S (exsdev) wrote :

For visibility: I've filed a bug for adding monitoring of the Metadata agent via the HTTP layer: LP#1921653

Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello James, or anyone else affected,

Accepted openvswitch into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvswitch/2.13.3-0ubuntu0.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openvswitch (Ubuntu Groovy):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-groovy
Changed in openvswitch (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Robie Basak (racb) wrote :

Hello James, or anyone else affected,

Accepted openvswitch into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvswitch/2.13.3-0ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (openvswitch/2.13.3-0ubuntu0.20.10.1)

All autopkgtests for the newly accepted openvswitch (2.13.3-0ubuntu0.20.10.1) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

netplan.io/0.101-0ubuntu3~20.10.1 (amd64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#openvswitch

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
David Ames (thedac) wrote :

@Robie,

I have been unable to validate as the amd64 package is failing to build:
https://launchpad.net/ubuntu/+source/openvswitch/2.13.3-0ubuntu0.20.04.1/+build/21397125

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (openvswitch/2.13.3-0ubuntu0.20.04.1)

All autopkgtests for the newly accepted openvswitch (2.13.3-0ubuntu0.20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

netplan.io/0.101-0ubuntu3~20.04.2 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#openvswitch

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
James Page (james-page) wrote : Please test proposed package

Hello James, or anyone else affected,

Accepted openvswitch into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-ussuri-needed
Revision history for this message
David Ames (thedac) wrote :

Verified focal-ussuri in a converged networking environment.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Drew Freiberger (afreiberger) wrote :

Verified bionic-ussuri-proposed python3-openvswitch package in production environment resolved the issue.

tags: added: verification-ussuri-done
removed: verification-ussuri-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Regression testing results for this were successful (apologies for the delay):

== focal-ussuri ==
21:38:43 ======
21:38:43 Totals
21:38:43 ======
21:38:43 Ran: 97 tests in 1663.4194 sec.
21:38:43 - Passed: 89
21:38:43 - Skipped: 8
21:38:43 - Expected Fail: 0
21:38:43 - Unexpected Success: 0
21:38:43 - Failed: 0
21:38:43 Sum of execute time for each test: 1151.4881 sec.

== bionic-ussuri ==
02:32:56 ======
02:32:56 Totals
02:32:56 ======
02:32:56 Ran: 97 tests in 960.0834 sec.
02:32:56 - Passed: 89
02:32:56 - Skipped: 8
02:32:56 - Expected Fail: 0
02:32:56 - Unexpected Success: 0
02:32:56 - Failed: 0
02:32:56 Sum of execute time for each test: 827.3194 sec.

== groovy-victoria ==
00:58:25 ======
00:58:25 Totals
00:58:25 ======
00:58:25 Ran: 97 tests in 1343.5539 sec.
00:58:25 - Passed: 89
00:58:25 - Skipped: 8
00:58:25 - Expected Fail: 0
00:58:25 - Unexpected Success: 0
00:58:25 - Failed: 0
00:58:25 Sum of execute time for each test: 1143.6704 sec.

== focal-victoria ==
04:39:56 ======
04:39:56 Totals
04:39:56 ======
04:39:56 Ran: 97 tests in 1445.2159 sec.
04:39:56 - Passed: 89
04:39:56 - Skipped: 8
04:39:56 - Expected Fail: 0
04:39:56 - Unexpected Success: 0
04:39:56 - Failed: 0
04:39:56 Sum of execute time for each test: 1045.0895 sec.

Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :
Download full text (3.7 KiB)

After applying the fix in bionic-proposed,
only one compute node exhibited the logs mentioned in #5
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server [-] Unexpected error.: AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl'

sb_idl is never initialised as the connection timed out during initialisation phase.

See error log:

2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start()
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout)
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException?
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start()
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connec...

Read more...

Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Typo in #26, tested in ussuri-proposed (not bionic-proposed)

Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Another customer hit this bug - waiting for release on uca ussuri

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello! Could we get this verified for groovy? Thanks!

Revision history for this message
James Page (james-page) wrote :

Yep on that today - thanks for the poke @sil2100

Revision history for this message
James Page (james-page) wrote :

Deployed groovy from proposed; restarted all ovn-central daemons post deployment and then started a new instance.

Instance booted and correctly retrieved metadata:

2021-05-13 12:46:06.269 2619 INFO eventlet.wsgi.server [-] 192.168.0.83,<local> "GET /openstack/2018-08-27/network_data.json HTTP/1.1" status: 200 len: 490 time: 0.0040092

tags: added: verification-done-groovy
removed: verification-needed-groovy
James Page (james-page)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 2.13.3-0ubuntu0.20.10.1

---------------
openvswitch (2.13.3-0ubuntu0.20.10.1) groovy; urgency=medium

  [ James Page ]
  * New upstream point release (LP: #1920141, LP: #1907686):
    - d/p/*: Refresh including drop of part of patch as equivalent
      changes included in point release.
    - Dropped security patches, included in release:
      + CVE-2015-8011.patch
      + CVE-2020-27827.patch
      + CVE-2020-35498.patch
  * Add BD on libdbus-1-dev to resolve linking issues for DPDK builds due
    to changes in DPDK.
  * d/control: Set minimum version of libdpdk-dev to avoid build
    failures with 19.11.6-0ubuntu0.20.10.1.

  [ Frode Nordahl ]
  * Fix recording of FQDN/hostname on startup (LP: #1915829):
    - d/p/ovs-dev-ovs-ctl-Allow-recording-hostname-separately.patch: Cherry
      pick of committed upstream fix to support skip of hostname
      configuration on ovs-vswitchd/ovsdb-server startup.
    - d/openvswitch-switch.ovs-record-hostname.service: Record hostname in
      Open vSwitch after network-online.target using new systemd unit.
    - d/openvswitch-switch.ovs-vswitchd.service: Pass `--no-record-hostname`
      option to `ovs-ctl` to delegate recording of hostname to the separate
      service.
    - d/openvswitch-switch.ovsdb-server.service: Pass `--no-record-hostname`
      option to `ovs-ctl` to delegate recording of hostname to the separate
      service.
    - d/openvswitch-switch.service: Add `Also` reference to
      ovs-record-hostname.service so that the service is enabled on install.
    - d/rules: Add `ovs-record-hostname.service` to package build.

 -- James Page <email address hidden> Wed, 24 Mar 2021 08:46:06 +0000

Changed in openvswitch (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for openvswitch has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 2.13.3-0ubuntu0.20.04.1

---------------
openvswitch (2.13.3-0ubuntu0.20.04.1) focal; urgency=medium

  [ James Page ]
  * New upstream point release (LP: #1920141, LP: #1907686).
  * Dropped security patches, included in release:
    - CVE-2015-8011.patch
    - CVE-2020-27827.patch
    - CVE-2020-35498.patch
  * Add BD on libdbus-1-dev to resolve linking issues for DPDK builds due
    to changes in DPDK.
  * d/control: Set minimum version of libdpdk-dev to avoid build
    failures with 19.11.6-0ubuntu0.20.04.1.

  [ Frode Nordahl ]
  * Fix recording of FQDN/hostname on startup (LP: #1915829):
    - d/p/ovs-dev-ovs-ctl-Allow-recording-hostname-separately.patch: Cherry
      pick of committed upstream fix to support skip of hostname
      configuration on ovs-vswitchd/ovsdb-server startup.
    - d/openvswitch-switch.ovs-record-hostname.service: Record hostname in
      Open vSwitch after network-online.target using new systemd unit.
    - d/openvswitch-switch.ovs-vswitchd.service: Pass `--no-record-hostname`
      option to `ovs-ctl` to delegate recording of hostname to the separate
      service.
    - d/openvswitch-switch.ovsdb-server.service: Pass `--no-record-hostname`
      option to `ovs-ctl` to delegate recording of hostname to the separate
      service.
    - d/openvswitch-switch.service: Add `Also` reference to
      ovs-record-hostname.service so that the service is enabled on install.
    - d/rules: Add `ovs-record-hostname.service` to package build.

 -- James Page <email address hidden> Wed, 24 Mar 2021 08:46:41 +0000

Changed in openvswitch (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Corey Bryant (corey.bryant) wrote :
Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

The verification of the Stable Release Update for openvswitch has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

This bug was fixed in the package openvswitch - 2.13.3-0ubuntu0.20.04.1~cloud0
---------------

 openvswitch (2.13.3-0ubuntu0.20.04.1~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 openvswitch (2.13.3-0ubuntu0.20.04.1) focal; urgency=medium
 .
   [ James Page ]
   * New upstream point release (LP: #1920141, LP: #1907686).
   * Dropped security patches, included in release:
     - CVE-2015-8011.patch
     - CVE-2020-27827.patch
     - CVE-2020-35498.patch
   * Add BD on libdbus-1-dev to resolve linking issues for DPDK builds due
     to changes in DPDK.
   * d/control: Set minimum version of libdpdk-dev to avoid build
     failures with 19.11.6-0ubuntu0.20.04.1.
 .
   [ Frode Nordahl ]
   * Fix recording of FQDN/hostname on startup (LP: #1915829):
     - d/p/ovs-dev-ovs-ctl-Allow-recording-hostname-separately.patch: Cherry
       pick of committed upstream fix to support skip of hostname
       configuration on ovs-vswitchd/ovsdb-server startup.
     - d/openvswitch-switch.ovs-record-hostname.service: Record hostname in
       Open vSwitch after network-online.target using new systemd unit.
     - d/openvswitch-switch.ovs-vswitchd.service: Pass `--no-record-hostname`
       option to `ovs-ctl` to delegate recording of hostname to the separate
       service.
     - d/openvswitch-switch.ovsdb-server.service: Pass `--no-record-hostname`
       option to `ovs-ctl` to delegate recording of hostname to the separate
       service.
     - d/openvswitch-switch.service: Add `Also` reference to
       ovs-record-hostname.service so that the service is enabled on install.
     - d/rules: Add `ovs-record-hostname.service` to package build.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.