| 2021-04-13 21:43:02 |
Michael Skalka |
bug |
|
|
added bug |
| 2021-04-13 21:46:22 |
Michael Skalka |
tags |
|
openstack-upgrade |
|
| 2021-04-13 21:47:14 |
Michael Skalka |
attachment added |
|
juju-crashdump-cd63074f-f189-4fa0-a532-9bdf6e6b0280.tar.xz https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1923668/+attachment/5487443/+files/juju-crashdump-cd63074f-f189-4fa0-a532-9bdf6e6b0280.tar.xz |
|
| 2021-04-13 21:59:25 |
Michael Skalka |
description |
While upgrading a cloud from Queens to Rocky I attempted to flush a hypervisor to avoid service disruption on the final unit of nova-compute using live-migrate. The action queues up in the dashboard however it completes with the instance remaining on the same host. Looking into the nova-compute logs from that instance it seems that the target host could not create the tap:
/var/log/nova/nova-compute.log:
2021-04-13 21:12:50.464 1286276 WARNING nova.compute.resource_tracker [req-b1cea8db-be1e-4252-9e31-c78d097ad671 - - - - -] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Instance not resizing, skipping migration.
2021-04-13 21:12:50.658 1286276 INFO nova.compute.resource_tracker [req-b1cea8db-be1e-4252-9e31-c78d097ad671 - - - - -] Final resource view: name=flagler.playground.solutionsqa phys_ram=32123MB used_ram=18432MB phys_disk=361GB used_disk=20GB total_vcpus=12 used_vcpus=1 pci_stats=[]
2021-04-13 21:13:02.025 1286276 ERROR nova.virt.libvirt.driver [req-06db27eb-b304-4969-b1e2-cbd0d80094ca d966ea789bfe431fb5863da1e72d6e49 80545c41a5db45d98d6adf7083c4914b - 9580fece017f4adf9b4ff1aa2bf836c8 9580fece017f4adf9b4ff1aa2bf836c8] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Live Migration failure: internal error: Unable to add port tap9c8d13c9-8a to OVS bridge br-int: libvirtError: internal error: Unable to add port tap9c8d13c9-8a to OVS bridge br-int
2021-04-13 21:13:02.187 1286276 ERROR nova.virt.libvirt.driver [req-06db27eb-b304-4969-b1e2-cbd0d80094ca d966ea789bfe431fb5863da1e72d6e49 80545c41a5db45d98d6adf7083c4914b - 9580fece017f4adf9b4ff1aa2bf836c8 9580fece017f4adf9b4ff1aa2bf836c8] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Migration operation has aborted
2021-04-13 21:13:02.364 1286276 INFO nova.compute.manager [req-06db27eb-b304-4969-b1e2-cbd0d80094ca d966ea789bfe431fb5863da1e72d6e49 80545c41a5db45d98d6adf7083c4914b - 9580fece017f4adf9b4ff1aa2bf836c8 9580fece017f4adf9b4ff1aa2bf836c8] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Swapping old allocation on 5a94928b-fb98-401f-bdd9-aa2f9f08602c held by migration 44727a6b-3417-4df3-9ca9-5b52e2e0f487 for instance
2021-04-13 21:13:04.381 1286276 WARNING nova.compute.manager [req-2f77835b-38ab-45b9-8acd-38a98ff3fcfc 6cad752c2b9744d6aac17fb26522004c d1aed1922a5a4a7899cae3e3afb6bc90 - c1a08b45ef134260be7501e96bc9ee3d c1a08b45ef134260be7501e96bc9ee3d] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Received unexpected event network-vif-unplugged-9c8d13c9-8a96-49e0-834a-3c512f1990cb for instance with vm_state active and task_state None.
2021-04-13 21:13:05.836 1286276 WARNING nova.compute.manager [req-66d4ddc6-4ac8-4c1a-8007-582d599da366 6cad752c2b9744d6aac17fb26522004c d1aed1922a5a4a7899cae3e3afb6bc90 - c1a08b45ef134260be7501e96bc9ee3d c1a08b45ef134260be7501e96bc9ee3d] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Received unexpected event network-vif-plugged-9c8d13c9-8a96-49e0-834a-3c512f1990cb for instance with vm_state active and task_state None.
Looking at the target unit the ovs-vsswitchd service is not even running on a number of the units: https://pastebin.ubuntu.com/p/YhdTQRRGb4/
Restarting the ovs-vsswitchd service on those hosts restores the ability to migrate. |
While upgrading a cloud from Queens to Rocky I attempted to flush a hypervisor to avoid service disruption on the final unit of nova-compute using live-migrate. The action queues up in the dashboard however it completes with the instance remaining on the same host. Looking into the nova-compute logs from that instance it seems that the target host could not create the tap:
/var/log/nova/nova-compute.log:
2021-04-13 21:12:50.464 1286276 WARNING nova.compute.resource_tracker [req-b1cea8db-be1e-4252-9e31-c78d097ad671 - - - - -] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Instance not resizing, skipping migration.
2021-04-13 21:12:50.658 1286276 INFO nova.compute.resource_tracker [req-b1cea8db-be1e-4252-9e31-c78d097ad671 - - - - -] Final resource view: name=flagler.playground.solutionsqa phys_ram=32123MB used_ram=18432MB phys_disk=361GB used_disk=20GB total_vcpus=12 used_vcpus=1 pci_stats=[]
2021-04-13 21:13:02.025 1286276 ERROR nova.virt.libvirt.driver [req-06db27eb-b304-4969-b1e2-cbd0d80094ca d966ea789bfe431fb5863da1e72d6e49 80545c41a5db45d98d6adf7083c4914b - 9580fece017f4adf9b4ff1aa2bf836c8 9580fece017f4adf9b4ff1aa2bf836c8] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Live Migration failure: internal error: Unable to add port tap9c8d13c9-8a to OVS bridge br-int: libvirtError: internal error: Unable to add port tap9c8d13c9-8a to OVS bridge br-int
2021-04-13 21:13:02.187 1286276 ERROR nova.virt.libvirt.driver [req-06db27eb-b304-4969-b1e2-cbd0d80094ca d966ea789bfe431fb5863da1e72d6e49 80545c41a5db45d98d6adf7083c4914b - 9580fece017f4adf9b4ff1aa2bf836c8 9580fece017f4adf9b4ff1aa2bf836c8] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Migration operation has aborted
2021-04-13 21:13:02.364 1286276 INFO nova.compute.manager [req-06db27eb-b304-4969-b1e2-cbd0d80094ca d966ea789bfe431fb5863da1e72d6e49 80545c41a5db45d98d6adf7083c4914b - 9580fece017f4adf9b4ff1aa2bf836c8 9580fece017f4adf9b4ff1aa2bf836c8] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Swapping old allocation on 5a94928b-fb98-401f-bdd9-aa2f9f08602c held by migration 44727a6b-3417-4df3-9ca9-5b52e2e0f487 for instance
2021-04-13 21:13:04.381 1286276 WARNING nova.compute.manager [req-2f77835b-38ab-45b9-8acd-38a98ff3fcfc 6cad752c2b9744d6aac17fb26522004c d1aed1922a5a4a7899cae3e3afb6bc90 - c1a08b45ef134260be7501e96bc9ee3d c1a08b45ef134260be7501e96bc9ee3d] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Received unexpected event network-vif-unplugged-9c8d13c9-8a96-49e0-834a-3c512f1990cb for instance with vm_state active and task_state None.
2021-04-13 21:13:05.836 1286276 WARNING nova.compute.manager [req-66d4ddc6-4ac8-4c1a-8007-582d599da366 6cad752c2b9744d6aac17fb26522004c d1aed1922a5a4a7899cae3e3afb6bc90 - c1a08b45ef134260be7501e96bc9ee3d c1a08b45ef134260be7501e96bc9ee3d] [instance: e341e106-5bec-4048-a76e-03ef0c70441c] Received unexpected event network-vif-plugged-9c8d13c9-8a96-49e0-834a-3c512f1990cb for instance with vm_state active and task_state None.
Looking at the target unit the ovs-vsswitchd service is not even running on a number of the units: https://pastebin.ubuntu.com/p/YhdTQRRGb4/
Restarting the ovs-vsswitchd service on those hosts restores the ability to migrate.
In each attempt the source of the instance was flagler and the destination was everitt which are machines 6 and 3 in the attached crashdump respectively. |
|
| 2021-04-14 13:26:25 |
Michael Skalka |
summary |
Upgrade from Queens to Rocky results in dead ovs-vsswitchd services |
Upgrade from Queens to Rocky results in dead ovs-vswitchd services |
|
| 2021-04-19 02:59:00 |
Billy Olsen |
attachment added |
|
crash file from system https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1923668/+attachment/5489633/+files/_usr_lib_openvswitch-switch_ovs-vswitchd.0.crash |
|
| 2021-06-18 02:21:21 |
Billy Olsen |
bug task added |
|
openvswitch (Ubuntu) |
|
| 2021-06-18 02:22:32 |
Billy Olsen |
nominated for series |
|
Ubuntu Focal |
|
| 2021-06-18 02:22:32 |
Billy Olsen |
bug task added |
|
openvswitch (Ubuntu Focal) |
|
| 2021-06-18 02:22:39 |
Billy Olsen |
openvswitch (Ubuntu Focal): status |
New |
Fix Released |
|
| 2021-06-18 02:22:49 |
Billy Olsen |
openvswitch (Ubuntu): status |
New |
Fix Released |
|
| 2021-06-18 02:25:49 |
Billy Olsen |
bug task added |
|
cloud-archive |
|
| 2021-06-18 02:27:45 |
Billy Olsen |
nominated for series |
|
cloud-archive/rocky |
|
| 2021-06-18 02:27:45 |
Billy Olsen |
bug task added |
|
cloud-archive/rocky |
|
| 2021-06-18 02:27:55 |
Billy Olsen |
cloud-archive: status |
New |
Triaged |
|
| 2021-06-18 02:27:57 |
Billy Olsen |
cloud-archive/rocky: status |
New |
Triaged |
|
| 2021-06-18 02:28:02 |
Billy Olsen |
charm-neutron-openvswitch: status |
New |
Invalid |
|
| 2021-06-18 02:28:09 |
Billy Olsen |
cloud-archive/rocky: importance |
Undecided |
High |
|
| 2021-10-25 08:07:12 |
Chris MacNaughton |
cloud-archive/rocky: assignee |
|
Chris MacNaughton (chris.macnaughton) |
|
| 2021-10-25 10:53:47 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chris.macnaughton/ubuntu/+source/openvswitch/+git/openvswitch/+merge/410730 |
|
| 2021-10-25 10:55:08 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chris.macnaughton/ubuntu/+source/openvswitch/+git/openvswitch/+merge/410731 |
|
| 2021-10-25 15:08:05 |
Chris MacNaughton |
cloud-archive/rocky: status |
Triaged |
Fix Committed |
|
| 2021-10-25 15:08:07 |
Chris MacNaughton |
tags |
openstack-upgrade |
openstack-upgrade verification-rocky-needed |
|
| 2023-05-26 13:47:53 |
Corey Bryant |
cloud-archive: status |
Triaged |
Invalid |
|
