Bugzilla reference: https://bugzilla.redhat.com/show_bug.cgi?id=2223997
this is similar to https://bugs.launchpad.net/neutron/+bug/2024160 and actually it is reproduced with the same tempest test, test_live_migration_with_trunk, but there are a few important differencies.
In LP2024160, the subport remained in status DOWN when the VM was created (before any migration!). And it happened always (the test always failed).
With this new bug, both subport and parent port successfully reach status=ACTIVE when the VM is created. Then, the VM is live-migrated to another compute. The test checks the VM status successfully changes to ACTIVE.
Finally, the test fails waiting for the subport status to change to ACTIVE: it never happens, it remains in status DOWN.
This failure doesn't happen always, so it seems it is due to a race condition.
Tempest logs when it fails (this job was retriggered/rechecked and the test passed):
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ac/887220/4/check/nova-live-migration/8ace3a8/testr_results.html
How often does this fail?
https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&branch=master&skip=0
24 jobs run on 11th and 12th July
7 of them failed - all these failures are due to this bug
i.e. ~30%
neutron server logs:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ac/887220/4/check/nova-live-migration/8ace3a8/controller/logs/screen-q-svc.txt
subport: e40925a3-4e71-4fa4-80ea-cc3a464101d6
parent port: 51e121cc-4e64-411e-b16a-e605cf967332
live-migration src host: np0034645395
live-migration dst host: np0034645398
It seems that due to a race condition, the following operations are processed in the wrong order.
The first one sets the parent port status to ACTIVE and changes revision number from 9 to 10:
Jul 11 15:45:48.620648 np0034645395 neutron-server[78430]: INFO neutron.tests.unit.plugins.ml2.drivers.mechanism_logger [None req-de6d27ae-5f07-41d8-90cb-e85d1228c53f None None] update_port_postcommit called with port settings {'id': '51e121cc-4e64-411e-b16a-e605cf967332', 'name': 'tempest-parent-2113033899', 'network_id': '6d3ce4b9-e66d-4137-a356-75e402377eb7', 'tenant_id': '4e4ea23ed96240ea8108fd937f2bd12b', 'mac_address': 'fa:16:3e:e6:51:04', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'cb3569cf-e7e0-4b01-8d56-aaf9fbcd41a8', 'device_owner': 'compute:nova', 'standard_attr_id': 173, 'fixed_ips': [{'subnet_id': '5ed321df-19d5-4c9b-8cf2-f27f0993fe12', 'ip_address': '10.1.0.5'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['ad92c344-fc82-4f88-bdc7-1d7845432a2f'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {'migrating_to': 'np0034645398'}, 'binding:host_id': 'np0034645395', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': True, 'trunk_details': {'trunk_id': '833832a8-c2bf-4ca9-a25a-9462f63b8926', 'sub_ports': [{'segmentation_id': 42, 'segmentation_type': 'vlan', 'port_id': 'e40925a3-4e71-4fa4-80ea-cc3a464101d6', 'mac_address': 'fa:16:3e:e5:55:14'}]}, 'tags': [], 'created_at': '2023-07-11T15:45:19Z', 'updated_at': '2023-07-11T15:45:48Z', 'revision_number': 10, 'project_id': '4e4ea23ed96240ea8108fd937f2bd12b'} (original settings {'id': '51e121cc-4e64-411e-b16a-e605cf967332', 'name': 'tempest-parent-2113033899', 'network_id': '6d3ce4b9-e66d-4137-a356-75e402377eb7', 'tenant_id': '4e4ea23ed96240ea8108fd937f2bd12b', 'mac_address': 'fa:16:3e:e6:51:04', 'admin_state_up': True, 'status': 'DOWN', 'device_id': 'cb3569cf-e7e0-4b01-8d56-aaf9fbcd41a8', 'device_owner': 'compute:nova', 'standard_attr_id': 173, 'fixed_ips': [{'subnet_id': '5ed321df-19d5-4c9b-8cf2-f27f0993fe12', 'ip_address': '10.1.0.5'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['ad92c344-fc82-4f88-bdc7-1d7845432a2f'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {'migrating_to': 'np0034645398'}, 'binding:host_id': 'np0034645395', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': True, 'trunk_details': {'trunk_id': '833832a8-c2bf-4ca9-a25a-9462f63b8926', 'sub_ports': [{'segmentation_id': 42, 'segmentation_type': 'vlan', 'port_id': 'e40925a3-4e71-4fa4-80ea-cc3a464101d6', 'mac_address': 'fa:16:3e:e5:55:14'}]}, 'tags': [], 'created_at': '2023-07-11T15:45:19Z', 'updated_at': '2023-07-11T15:45:48Z', 'revision_number': 9, 'project_id': '4e4ea23ed96240ea8108fd937f2bd12b'}) host np0034645395 (original host np0034645395) vif type ovs (original vif type ovs) vif details {'port_filter': True, 'connectivity': 'l2'} (original vif details {'port_filter': True, 'connectivity': 'l2'}) binding levels [{'bound_driver': 'ovn', 'bound_segment': {'id': '21392775-0d30-44c3-9f06-aa8348d69be8', 'network_type': 'geneve', 'physical_network': None, 'segmentation_id': 10205, 'network_id': '6d3ce4b9-e66d-4137-a356-75e402377eb7'}}] (original binding levels [{'bound_driver': 'ovn', 'bound_segment': {'id': '21392775-0d30-44c3-9f06-aa8348d69be8', 'network_type': 'geneve', 'physical_network': None, 'segmentation_id': 10205, 'network_id': '6d3ce4b9-e66d-4137-a356-75e402377eb7'}}]) on network {'id': '6d3ce4b9-e66d-4137-a356-75e402377eb7', 'name': 'tempest-LiveAutoBlockMigrationV225Test-1684440653-network', 'tenant_id': '4e4ea23ed96240ea8108fd937f2bd12b', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['5ed321df-19d5-4c9b-8cf2-f27f0993fe12'], 'standard_attr_id': 55, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'vlan_transparent': None, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2023-07-11T15:41:08Z', 'updated_at': '2023-07-11T15:41:10Z', 'revision_number': 2, 'project_id': '4e4ea23ed96240ea8108fd937f2bd12b', 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 10205} with segments to bind None
Jul 11 15:45:48.621081 np0034645395 neutron-server[78430]: DEBUG neutron.tests.unit.plugins.ml2.drivers.mechanism_logger [None req-de6d27ae-5f07-41d8-90cb-e85d1228c53f None None] update_port_postcommit:
Jul 11 15:45:48.621081 np0034645395 neutron-server[78430]: key[status], DOWN -> ACTIVE
Jul 11 15:45:48.621081 np0034645395 neutron-server[78430]: key[revision_number], 9 -> 10
The second operation wrongly sets the parent port status to DOWN again, and changes revision_number from 8 to 9, so apparently it should have been processed before:
Jul 11 15:45:49.283211 np0034645395 neutron-server[78430]: INFO neutron.tests.unit.plugins.ml2.drivers.mechanism_logger [None req-de6d27ae-5f07-41d8-90cb-e85d1228c53f None None] update_port_postcommit called with port settings {'id': '51e121cc-4e64-411e-b16a-e605cf967332', 'name': 'tempest-parent-2113033899', 'network_id': '6d3ce4b9-e66d-4137-a356-75e402377eb7', 'tenant_id': '4e4ea23ed96240ea8108fd937f2bd12b', 'mac_address': 'fa:16:3e:e6:51:04', 'admin_state_up': True, 'status': 'DOWN', 'device_id': 'cb3569cf-e7e0-4b01-8d56-aaf9fbcd41a8', 'device_owner': 'compute:nova', 'standard_attr_id': 173, 'fixed_ips': [{'subnet_id': '5ed321df-19d5-4c9b-8cf2-f27f0993fe12', 'ip_address': '10.1.0.5'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['ad92c344-fc82-4f88-bdc7-1d7845432a2f'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {'migrating_to': 'np0034645398'}, 'binding:host_id': 'np0034645395', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': True, 'trunk_details': {'trunk_id': '833832a8-c2bf-4ca9-a25a-9462f63b8926', 'sub_ports': [{'segmentation_id': 42, 'segmentation_type': 'vlan', 'port_id': 'e40925a3-4e71-4fa4-80ea-cc3a464101d6', 'mac_address': 'fa:16:3e:e5:55:14'}]}, 'tags': [], 'created_at': '2023-07-11T15:45:19Z', 'updated_at': '2023-07-11T15:45:48Z', 'revision_number': 9, 'project_id': '4e4ea23ed96240ea8108fd937f2bd12b'} (original settings {'id': '51e121cc-4e64-411e-b16a-e605cf967332', 'name': 'tempest-parent-2113033899', 'network_id': '6d3ce4b9-e66d-4137-a356-75e402377eb7', 'tenant_id': '4e4ea23ed96240ea8108fd937f2bd12b', 'mac_address': 'fa:16:3e:e6:51:04', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'cb3569cf-e7e0-4b01-8d56-aaf9fbcd41a8', 'device_owner': 'compute:nova', 'standard_attr_id': 173, 'fixed_ips': [{'subnet_id': '5ed321df-19d5-4c9b-8cf2-f27f0993fe12', 'ip_address': '10.1.0.5'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['ad92c344-fc82-4f88-bdc7-1d7845432a2f'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {'migrating_to': 'np0034645398'}, 'binding:host_id': 'np0034645395', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': True, 'trunk_details': {'trunk_id': '833832a8-c2bf-4ca9-a25a-9462f63b8926', 'sub_ports': [{'segmentation_id': 42, 'segmentation_type': 'vlan', 'port_id': 'e40925a3-4e71-4fa4-80ea-cc3a464101d6', 'mac_address': 'fa:16:3e:e5:55:14'}]}, 'tags': [], 'created_at': '2023-07-11T15:45:19Z', 'updated_at': '2023-07-11T15:45:43Z', 'revision_number': 8, 'project_id': '4e4ea23ed96240ea8108fd937f2bd12b'}) host np0034645395 (original host np0034645395) vif type ovs (original vif type ovs) vif details {'port_filter': True, 'connectivity': 'l2'} (original vif details {'port_filter': True, 'connectivity': 'l2'}) binding levels [{'bound_driver': 'ovn', 'bound_segment': {'id': '21392775-0d30-44c3-9f06-aa8348d69be8', 'network_type': 'geneve', 'physical_network': None, 'segmentation_id': 10205, 'network_id': '6d3ce4b9-e66d-4137-a356-75e402377eb7'}}] (original binding levels [{'bound_driver': 'ovn', 'bound_segment': {'id': '21392775-0d30-44c3-9f06-aa8348d69be8', 'network_type': 'geneve', 'physical_network': None, 'segmentation_id': 10205, 'network_id': '6d3ce4b9-e66d-4137-a356-75e402377eb7'}}]) on network {'id': '6d3ce4b9-e66d-4137-a356-75e402377eb7', 'name': 'tempest-LiveAutoBlockMigrationV225Test-1684440653-network', 'tenant_id': '4e4ea23ed96240ea8108fd937f2bd12b', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['5ed321df-19d5-4c9b-8cf2-f27f0993fe12'], 'standard_attr_id': 55, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'vlan_transparent': None, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2023-07-11T15:41:08Z', 'updated_at': '2023-07-11T15:41:10Z', 'revision_number': 2, 'project_id': '4e4ea23ed96240ea8108fd937f2bd12b', 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 10205} with segments to bind None
Jul 11 15:45:49.283646 np0034645395 neutron-server[78430]: DEBUG neutron.tests.unit.plugins.ml2.drivers.mechanism_logger [None req-de6d27ae-5f07-41d8-90cb-e85d1228c53f None None] update_port_postcommit:
Jul 11 15:45:49.283646 np0034645395 neutron-server[78430]: key[status], ACTIVE -> DOWN
Jul 11 15:45:49.283646 np0034645395 neutron-server[78430]: key[updated_at], 2023-07-11T15:45:43Z -> 2023-07-11T15:45:48Z
Jul 11 15:45:49.283646 np0034645395 neutron-server[78430]: key[revision_number], 8 -> 9
References:
* https:/
* https:/