[OVN][Trunk] subport doesn't reach status ACTIVE

thanks, opensearch link:
https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22line%20273,%20in%20test_live_migration_with_trunk%22'),sort:!())

OVN version bump was introduced with this patch:
https://review.opendev.org/c/openstack/neutron/+/883681

And it is related to port binding so suspicious: "Improve the ``PortBindingUpdateVirtualPortsEvent`` match filter"

Marking the failing tests as unstable:
https://review.opendev.org/c/openstack/tempest/+/886496

This issue is related to [0]. I'm investigating the patch proposed for this bug [1] that is solving this issue by updating the subport VIF type and the host ID, missing in the Trunk service, in the OVN driver.

[0]https://bugs.launchpad.net/neutron/+bug/2018289
[1]https://review.opendev.org/c/openstack/neutron/+/882581

Patch proposed: https://review.opendev.org/c/openstack/neutron/+/882581

This issue was fixed in the openstack/neutron 22.0.2 release.

This issue was fixed in the openstack/neutron 21.1.2 release.

I am seeing this test failing, may be on same things?

https://zuul.opendev.org/t/openstack/build/8ace3a8e9f3b4815950b40db262d621e/log/job-output.txt#33049

@ghanshyammann,
This bug was reproduced with a "live-migration" test, but notice the issue happened before any migrarion was performed:
"It's a test about live-migration, but it is important to notice that it fails before any live migration happens."
And it failed 100% times.

What you are reporting now looks similar, but it happens after live-migration (a few lines above, the test checked both subport and parent port are ACTIVE before live-migration):
https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_live_migration.py#L292
And apparently it doesn't fail always ("887220,4" was rechecked and the test passed):
https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&project=openstack/tempest

So, IMO, this is a different bug. Some things are common: OVN and trunk ports. Some things are different: happens after live-migration, doesn't fail 100% times.

After talking with Rodolfo, I have opened a new bug for the issue I described in the previous comment:
https://bugs.launchpad.net/neutron/+bug/2027605

Thanks eolivare.

It is failing 48 times after it was unskipped (https://review.opendev.org/c/openstack/tempest/+/887682).

may be first fux unhide this bug. thanks for reporting new bug, I agree with that.

https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2023-06-10T17:30:00.000Z',to:now))&_a=(columns:!(build_branch,build_change,build_name,project,build_status,build_queue),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:d,query:(language:lucene,query:'message:%22test_live_migration.py,%20line%20292,%20in%20test_live_migration_with_trunk%22%20'),sort:!())

This issue was fixed in the openstack/neutron 20.4.0 release.

This issue was fixed in the openstack/neutron 23.0.0.0b3 development milestone.

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/895152

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/895152
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/370f71e9877fad13883413225bf25051c0c9c391
Submitter: "Zuul (22348)"
Branch: master

commit 370f71e9877fad13883413225bf25051c0c9c391
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sat Sep 9 21:33:55 2023 +0000

    Mark "test_subport_connectivity_soft_reboot" as unstable

    This issue is related to:
    * https://bugs.launchpad.net/neutron/+bug/2033887: the patches
      previously merged in the Neutron repository [1][2] should be
      reverted.
    * https://bugs.launchpad.net/neutron/+bug/2024160: this issue
      is still present.

    [1]https://review.opendev.org/c/openstack/neutron/+/882581
    [2]https://review.opendev.org/c/openstack/neutron/+/888776

    Change-Id: I9d8d76810d15913c29610464ab1b8fa34863aaf6
    Related-Bug: #2024160
    Related-Bug: #2033887

I'm running into this and putting some notes for my troubleshooting, we're running OVN 23.03.0-1~cloud0 and running into this issue.

I'm dredging through this to see what I have, but I have a failing case that I'm looking here in front of me right now.

https://github.com/ovn-org/ovn/compare/v22.03.0...v22.03.3

So the thing that I noticed is that I assume there are no addresses assigned for the sub-ports, so what happens seems to be this in the `ovn-controller` logs:

```
2023-09-22T20:15:38.402Z|380055|binding|INFO|Claiming lport c6858836-2ef8-4259-bf30-e7d769d8c165 for this chassis.
2023-09-22T20:15:38.402Z|380056|binding|INFO|c6858836-2ef8-4259-bf30-e7d769d8c165: Claiming unknown
```

But I'm not seeing the same output as a usual virtual port...

```
2023-09-22T20:15:47.959Z|380057|pinctrl|INFO|Claiming virtual lport fa57424b-3228-4b29-9bb4-c6edb7927a5f for this chassis with the virtual parent cb15f805-943d-4efa-9646-7a700e6cd81b
2023-09-22T20:15:52.853Z|380058|pinctrl|INFO|Claiming virtual lport ced5a54f-bfab-40b2-8c40-82c97f3c2586 for this chassis with the virtual parent cb15f805-943d-4efa-9646-7a700e6cd81b
```

So I wonder what's up with that. It seems that OVN is happy if there is an IP assigned to parent and the sub-ports via Neutron, but it doesn't like it when there is no IP assigned (and port security enabled?)

So, running an `ovn-trace` for both the sub-port and the trunk, when port security is disabled, shows that it kills it early when it sees VLAN traffic. I suspect this is another bug:

```
❯ kubectl -n openstack exec pods/ovn-ovsdb-sb-2 -- ovn-trace --friendly-names --ovs neutron-ea05f3c2-b370-49e6-ab71-635376211fd5 '
  inport=="bffe6ba0-6bdc-4f99-b43e-a5dc9e5b6aea" &&
  vlan.vid==20 &&
  eth.src==52:54:00:3f:1d:a9 &&
  ip4.src==172.17.0.11 &&
  ip4.dst==172.17.0.100 &&
  ip.ttl == 64 &&
  icmp4.type == 8'
Defaulted container "ovsdb" out of: ovsdb, init (init)
# icmp,reg14=0xf,dl_vlan=20,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=52:54:00:3f:1d:a9,dl_dst=00:00:00:00:00:00,nw_src=172.17.0.11,nw_dst=172.17.0.100,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0

ingress(dp="zuul-ci-net-17dc44ba", inport="controller-ffd424a4-3b96-489a-84dd-e6d742e9cf8e-20")
-----------------------------------------------------------------------------------------------
 0. ls_in_check_port_sec (northd.c:8377): vlan.present, priority 100, uuid d24b30bb
    drop;
```

So in this case, it seems that for some reason, the maintenance task seems to 'unset' the state of the port:

2023-09-26 05:59:37.812 27 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-30eb5d90-11a4-422f-b017-a4c4099c8722 - - - - - -] Maintenance task: Fixing resource 34c341e5-3eb6-47d7-bc96-54a9a1aca61f (type: ports) at create/update check_for_inconsistencies /var/lib/openstack/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:384
2023-09-26 05:59:37.855 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(name=34c341e5-3eb6-47d7-bc96-54a9a1aca61f, resource={'id': '34c341e5-3eb6-47d7-bc96-54a9a1aca61f', 'name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'network_id': '16b4d11e-6734-4058-814e-7d22c4c9a8da', 'tenant_id': '4b633c451ac74233be3721a3635275e5', 'mac_address': 'fa:16:3e:30:c1:41', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '', 'device_owner': 'trunk:subport', 'standard_attr_id': 64968243, 'fixed_ips': [], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': '', 'binding:vif_type': 'unbound', 'binding:vif_details': {}, 'dns_name': '', 'dns_assignment': [], 'dns_domain': '', 'port_security_enabled': False, 'qos_policy_id': None, 'qos_network_policy_id': None, 'resource_request': None, 'ip_allocation': 'none', 'tags': [], 'created_at': '2023-09-26T05:54:30Z', 'updated_at': '2023-09-26T05:54:36Z', 'revision_number': 4, 'project_id': '4b633c451ac74233be3721a3635275e5'}, resource_type=ports, if_exists=True) do_commit /var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2023-09-26 05:59:37.856 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLSwitchPortCommand(lport=34c341e5-3eb6-47d7-bc96-54a9a1aca61f, external_ids_update=None, columns={'external_ids': {'neutron:port_name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'neutron:device_id': '', 'neutron:project_id': '4b633c451ac74233be3721a3635275e5', 'neutron:cidrs': '', 'neutron:device_owner': 'trunk:subport', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': '', 'neutron:network_name': 'neutron-16b4d11e-6734-4058-814e-7d22c4c9a8da', 'neutron:security_group_ids': '', 'neutron:revision_number': '4'}, 'parent_name': [], 'tag': [], 'options': {'requested-chassis': ''}, 'enabled': True, 'port_security': [], 'dhcpv4_options': [], 'dhcpv6_options': [], 'type': '', 'addresses': ['fa:16:3e:30:c1:41', 'unknown'], 'ha_chassis_group': []}, if_exists=False) do_commit /var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2023-09-26 05:59:37.857 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): PgDelPortCommand(port_group=neutron_pg_drop, lsp=['34c341e5-3eb6-47d7-bc96-54a9a1aca61f'], if_exists=False) do_commit /var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2023-09-26 05:59:38.117 11 DEBUG ovsdbapp.backend.ovs_idl.event [None req...

So I think the root cause is these events are causing the data to be cleared in the Neutron database, which then result in the port being removed after 5 minutes on the next maintenance cycle:

```
2023-09-26 05:54:36.201 11 DEBUG ovsdbapp.backend.ovs_idl.event [None req-c92127e4-2312-41be-9705-97f45cfcd374 - - - - - -] Matched UPDATE: PortBindingChassisEvent(events=('update',), table='Port_Binding', conditions=(('type', '=', 'chassisredirect'),), old_conditions=None), priority=20 to row=Port_Binding(parent_port=['7703365a-b55a-4856-a223-8ee5115e8400'], mac=['unknown'], chassis=[<ovs.db.idl.Row object at 0x7f3fe8ae89d0>], options={'requested-chassis': ''}, ha_chassis_group=[], type=, additional_encap=[], tag=[20], requested_additional_chassis=[], up=[True], tunnel_key=4, requested_chassis=[], additional_chassis=[], logical_port=34c341e5-3eb6-47d7-bc96-54a9a1aca61f, encap=[], gateway_chassis=[], external_ids={'name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'neutron:cidrs': '', 'neutron:device_id': '', 'neutron:device_owner': 'trunk:subport', 'neutron:network_name': 'neutron-16b4d11e-6734-4058-814e-7d22c4c9a8da', 'neutron:port_name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'neutron:project_id': '4b633c451ac74233be3721a3635275e5', 'neutron:revision_number': '2', 'neutron:security_group_ids': '', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': ''}, nat_addresses=[], virtual_parent=[], port_security=[], datapath=dc18c51f-d011-41ff-8c0e-1cd6a778bd28, mirror_rules=[]) old=Port_Binding(chassis=[], up=[False]) matches /var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/event.py:43
2023-09-26 05:54:36.259 14 DEBUG ovsdbapp.backend.ovs_idl.event [None req-dae54ec1-7359-47ff-bf53-34cdfcb55fdf - - - - - -] Matched DELETE: PortBindingUpdateVirtualPortsEvent(events=('update', 'delete'), table='Port_Binding', conditions=None, old_conditions=None), priority=20 to row=Port_Binding(parent_port=[], mac=['unknown'], chassis=[], options={'requested-chassis': ''}, ha_chassis_group=[], type=, additional_encap=[], tag=[], requested_additional_chassis=[], up=[False], tunnel_key=4, requested_chassis=[], additional_chassis=[], logical_port=34c341e5-3eb6-47d7-bc96-54a9a1aca61f, encap=[], gateway_chassis=[], external_ids={'name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'neutron:cidrs': '', 'neutron:device_id': '', 'neutron:device_owner': '', 'neutron:network_name': 'neutron-16b4d11e-6734-4058-814e-7d22c4c9a8da', 'neutron:port_name': 'compute-fa2ffcfd-a6d6-4baa-918e-cb5c67823f6a-20', 'neutron:project_id': '4b633c451ac74233be3721a3635275e5', 'neutron:revision_number': '1', 'neutron:security_group_ids': '', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': ''}, nat_addresses=[], virtual_parent=[], port_security=[], datapath=dc18c51f-d011-41ff-8c0e-1cd6a778bd28, mirror_rules=[]) old= matches /var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/event.py:43
```

More specifically, the DELETE on `PortBindingUpdateVirtualPortsEvent` is clearing out the binding details, so on the next maintenance run, Neutron kils it off.

OK, I am pretty sure the broken change is:

https://review.opendev.org/c/openstack/neutron/+/883681

I believe this is now firing this code for ALL ports instead of virtual ports..

There we go, I tracked it down to this commit:

https://github.com/ovn-org/ovn/commit/28d40c0a024227c9c845117087b7594723ff0ec7

The reality is that we 'fixed' Neutron by accommodating for this:

https://github.com/ovn-org/ovn/commit/ec933537f9b04b35ef4b79fb2b4743f9095da209

But we didn't notice we might be getting deletes for trunk ports (parent_port aka container). We need to accomodate for this.

Patch fixing a filtering issue in [1]: https://review.opendev.org/c/openstack/neutron/+/896590

[1]https://review.opendev.org/c/openstack/neutron/+/883681

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/896803

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/896804

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/896805

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896590
Committed: https://opendev.org/openstack/neutron/commit/1160aaa4716904fdba06a8dcfbc1d5a2ce419e72
Submitter: "Zuul (22348)"
Branch: master

commit 1160aaa4716904fdba06a8dcfbc1d5a2ce419e72
Author: Mohammed Naser <email address hidden>
Date: Tue Sep 26 20:02:55 2023 +0000

    [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent

    With newer versions of OVN, port bindings are removed and added
    for both virtual ports and container ports (aka trunk ports in
    Neutron world)[1][2].

    With this, Neutron recently landed a change in order to handle
    virtual ports for the update/delete steps[3], however, this has
    resulted in unexpected behaviour where the binding information
    is wiped for trunk ports, then subsequently removed on the
    next maintenance run.

    We should make sure that we only have row deletes for
    virtual ports specifically rather than all ports.

    [1]: https://github.com/ovn-org/ovn/commit/ec933537f9b04b35ef4b79fb2b4743f9095da209
    [2]: https://github.com/ovn-org/ovn/commit/28d40c0a024227c9c845117087b7594723ff0ec7
    [3]: https://review.opendev.org/c/openstack/neutron/+/883681

    Related-Bug: #2024160
    Change-Id: Ide3204748274cbab9731310c2e2bd8bd8c9a53ff

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896804
Committed: https://opendev.org/openstack/neutron/commit/2319e7c35f79798cba199f52a5a7c4a4ef7bc088
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 2319e7c35f79798cba199f52a5a7c4a4ef7bc088
Author: Mohammed Naser <email address hidden>
Date: Tue Sep 26 20:02:55 2023 +0000

    [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent

    With newer versions of OVN, port bindings are removed and added
    for both virtual ports and container ports (aka trunk ports in
    Neutron world)[1][2].

    With this, Neutron recently landed a change in order to handle
    virtual ports for the update/delete steps[3], however, this has
    resulted in unexpected behaviour where the binding information
    is wiped for trunk ports, then subsequently removed on the
    next maintenance run.

    We should make sure that we only have row deletes for
    virtual ports specifically rather than all ports.

    [1]: https://github.com/ovn-org/ovn/commit/ec933537f9b04b35ef4b79fb2b4743f9095da209
    [2]: https://github.com/ovn-org/ovn/commit/28d40c0a024227c9c845117087b7594723ff0ec7
    [3]: https://review.opendev.org/c/openstack/neutron/+/883681

    Related-Bug: #2024160
    Change-Id: Ide3204748274cbab9731310c2e2bd8bd8c9a53ff
    (cherry picked from commit 1160aaa4716904fdba06a8dcfbc1d5a2ce419e72)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896805
Committed: https://opendev.org/openstack/neutron/commit/0364f18848971446bb98619c94aa0c4024b33531
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 0364f18848971446bb98619c94aa0c4024b33531
Author: Mohammed Naser <email address hidden>
Date: Tue Sep 26 20:02:55 2023 +0000

    [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent

    With newer versions of OVN, port bindings are removed and added
    for both virtual ports and container ports (aka trunk ports in
    Neutron world)[1][2].

    With this, Neutron recently landed a change in order to handle
    virtual ports for the update/delete steps[3], however, this has
    resulted in unexpected behaviour where the binding information
    is wiped for trunk ports, then subsequently removed on the
    next maintenance run.

    We should make sure that we only have row deletes for
    virtual ports specifically rather than all ports.

    [1]: https://github.com/ovn-org/ovn/commit/ec933537f9b04b35ef4b79fb2b4743f9095da209
    [2]: https://github.com/ovn-org/ovn/commit/28d40c0a024227c9c845117087b7594723ff0ec7
    [3]: https://review.opendev.org/c/openstack/neutron/+/883681

    Related-Bug: #2024160
    Change-Id: Ide3204748274cbab9731310c2e2bd8bd8c9a53ff
    (cherry picked from commit 1160aaa4716904fdba06a8dcfbc1d5a2ce419e72)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/896803
Committed: https://opendev.org/openstack/neutron/commit/04e9b063ad526778ba80438fc851e03fa16e5085
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 04e9b063ad526778ba80438fc851e03fa16e5085
Author: Mohammed Naser <email address hidden>
Date: Tue Sep 26 20:02:55 2023 +0000

    [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent

    With newer versions of OVN, port bindings are removed and added
    for both virtual ports and container ports (aka trunk ports in
    Neutron world)[1][2].

    With this, Neutron recently landed a change in order to handle
    virtual ports for the update/delete steps[3], however, this has
    resulted in unexpected behaviour where the binding information
    is wiped for trunk ports, then subsequently removed on the
    next maintenance run.

    We should make sure that we only have row deletes for
    virtual ports specifically rather than all ports.

    [1]: https://github.com/ovn-org/ovn/commit/ec933537f9b04b35ef4b79fb2b4743f9095da209
    [2]: https://github.com/ovn-org/ovn/commit/28d40c0a024227c9c845117087b7594723ff0ec7
    [3]: https://review.opendev.org/c/openstack/neutron/+/883681

    Related-Bug: #2024160
    Change-Id: Ide3204748274cbab9731310c2e2bd8bd8c9a53ff
    (cherry picked from commit 1160aaa4716904fdba06a8dcfbc1d5a2ce419e72)