[sru] neutron-ovn-db-sync generates insufficient flow

Can you provide more detail? When you say insufficient flow:

1) What flows are missing in your opinion?
2) Are we talking about flows in the integration bridge?

Hello Miguel Lavalle.
When I use neutron-ovn-db-sync tool to generates OVN_NB and OVN_SB database, OVN-controller is based on OVN_SB database to create flow. Then, all port no security-group or disable port-security is not connected to network.
I think this is because the generated OVN_NB and OVN_SB are missing.

Reproduce:
- Create a VM
- Disable port security
- Remove NB & SB DB
- Run command neutron-ovn-db-sync-util to resync from neutron to NB database
neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair
- Restart ovn-controller
- VM with port disable security die

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/816328

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816328
Committed: https://opendev.org/openstack/neutron/commit/4511290b726f605384285228a28ad7b32a4b8c43
Submitter: "Zuul (22348)"
Branch: master

commit 4511290b726f605384285228a28ad7b32a4b8c43
Author: Nguyen Thanh Cong <email address hidden>
Date: Thu Nov 11 21:00:23 2021 +0700

    [OVN] Fix port disable security dead when run neutron-ovn-db-sync-util

    Port disable security should not have been in acl neutron_pg_drop, but
    when run neutron-ovn-db-sync-util, port disable security still added
    to ACL neutron_pg_drop. It because port disable security is not
    trusted port.

    Co-authored-by: archiephan <email address hidden>

    Closes-Bug: #1939723
    Change-Id: Iebce0929e3e68ac5be0acaf5cdac4f5833cb9f2f

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/818151

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/818152

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/818153

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818152
Committed: https://opendev.org/openstack/neutron/commit/78bc1f6fec8e216aca7cf6fbc6770631dacca0ee
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 78bc1f6fec8e216aca7cf6fbc6770631dacca0ee
Author: Nguyen Thanh Cong <email address hidden>
Date: Thu Nov 11 21:00:23 2021 +0700

    [OVN] Fix port disable security dead when run neutron-ovn-db-sync-util

    Port disable security should not have been in acl neutron_pg_drop, but
    when run neutron-ovn-db-sync-util, port disable security still added
    to ACL neutron_pg_drop. It because port disable security is not
    trusted port.

    Co-authored-by: archiephan <email address hidden>

    Closes-Bug: #1939723
    Change-Id: Iebce0929e3e68ac5be0acaf5cdac4f5833cb9f2f
    (cherry picked from commit 4511290b726f605384285228a28ad7b32a4b8c43)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818153
Committed: https://opendev.org/openstack/neutron/commit/32cc39663ae68fa14ba01bcd38c783c9a8004f40
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 32cc39663ae68fa14ba01bcd38c783c9a8004f40
Author: Nguyen Thanh Cong <email address hidden>
Date: Thu Nov 11 21:00:23 2021 +0700

    [OVN] Fix port disable security dead when run neutron-ovn-db-sync-util

    Port disable security should not have been in acl neutron_pg_drop, but
    when run neutron-ovn-db-sync-util, port disable security still added
    to ACL neutron_pg_drop. It because port disable security is not
    trusted port.

    Co-authored-by: archiephan <email address hidden>

    Closes-Bug: #1939723
    Change-Id: Iebce0929e3e68ac5be0acaf5cdac4f5833cb9f2f
    (cherry picked from commit 4511290b726f605384285228a28ad7b32a4b8c43)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818151
Committed: https://opendev.org/openstack/neutron/commit/cb083bdbcfc15a2c5f1ed0e0273b607752d7194e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit cb083bdbcfc15a2c5f1ed0e0273b607752d7194e
Author: Nguyen Thanh Cong <email address hidden>
Date: Thu Nov 11 21:00:23 2021 +0700

    [OVN] Fix port disable security dead when run neutron-ovn-db-sync-util

    Port disable security should not have been in acl neutron_pg_drop, but
    when run neutron-ovn-db-sync-util, port disable security still added
    to ACL neutron_pg_drop. It because port disable security is not
    trusted port.

    Co-authored-by: archiephan <email address hidden>

    Closes-Bug: #1939723
    Change-Id: Iebce0929e3e68ac5be0acaf5cdac4f5833cb9f2f
    (cherry picked from commit 4511290b726f605384285228a28ad7b32a4b8c43)

This issue was fixed in the openstack/neutron 19.1.0 release.

This issue was fixed in the openstack/neutron 17.3.0 release.

This issue was fixed in the openstack/neutron 18.2.0 release.

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/874979

Reviewed: https://review.opendev.org/c/openstack/neutron/+/874979
Committed: https://opendev.org/openstack/neutron/commit/b01ad2fa0913b087a7e8151eca7417b1d9b978ab
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit b01ad2fa0913b087a7e8151eca7417b1d9b978ab
Author: Nguyen Thanh Cong <email address hidden>
Date: Thu Nov 11 21:00:23 2021 +0700

    [OVN] Fix port disable security dead when run neutron-ovn-db-sync-util

    Port disable security should not have been in acl neutron_pg_drop, but
    when run neutron-ovn-db-sync-util, port disable security still added
    to ACL neutron_pg_drop. It because port disable security is not
    trusted port.

    Co-authored-by: archiephan <email address hidden>

    Closes-Bug: #1939723
    Change-Id: Iebce0929e3e68ac5be0acaf5cdac4f5833cb9f2f
    (cherry picked from commit 4511290b726f605384285228a28ad7b32a4b8c43)

The ubuntu package version with this fix has been uploaded to the focal unapproved queue for SRU team review. https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=neutron

Hello Son, or anyone else affected,

Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.4.2-0ubuntu6.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Son, or anyone else affected,

Accepted neutron into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

here are all my attempts today to reproduce the issue firstly - https://pastebin.ubuntu.com/p/vGr9RN57PX/

I am not yet sure if I have a successful reproducer before I can test the fix, but I could not ssh into the VM after the last step ,

"$ openstack server ssh testvm1 --private --login ubuntu" did not return, so I need to check if this is the expected behavior of the reproducer. The status however showed active and not failed,

+--------------------------------------+---------+--------+------------------------+-------+----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------+--------+------------------------+-------+----------+
| 94e0949a-5da2-4f3a-84ed-4673cf31fc9b | testvm1 | ACTIVE | private=192.168.21.129 | jammy | m1.small |
+--------------------------------------+---------+--------+------------------------+-------+----------+

I have emailed congnt95 and also will check with other openstack experts if these testing steps are correct, and if this is a workable reproducer, I will install the proposed package on the neutron-api node and check if the vm is reachable in that case after the last step of rebooting the ovn-central nodes after repairing the database using

# sudo neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf.copy --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair

Hello,

Now I haven't any openstack cluster for testing, nikhil please help me test it.

Step for reproduce and testing:
1. Create a VM (get neutron port_id, example xxxxx)
2. Check port_id in OVN northbound DB:
ovn-nbctl --no-leader-only lsp-list neutron-<neutron-network-id>
https://ibb.co/d7D9VT4
--> port_id in OVN is yyyyy

2. ovn-nbctl --no-leader-only list port_group neutron_pg_drop | grep yyyyy
--> port yyyyy in port_group neutron_pg_drop

3. Disable port security
4. ovn-nbctl --no-leader-only list port_group neutron_pg_drop | grep yyyyy
--> port yyyyy removed from port_group neutron_pg_drop

5. Remove NB & SB DB
6. Run command neutron-ovn-db-sync-util to resync from neutron to NB database
neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair
- Restart ovn-controller in all controller and compute
7. ovn-nbctl --no-leader-only list port_group neutron_pg_drop | grep yyyyy
--> port yyyyy appear in port_group neutron_pg_drop
--> Can't access to this VM (VM still active but can't ping or access to IP)

After this patch, in step 7: port yyyyy not appear in port_group neutron_pg_drop

I tried reproducing it, but cant see expected result in step 2,

Step for reproduce and testing:
1. Create a VM (get neutron port_id, example xxxxx)
2. ovn-nbctl --no-leader-only list port_group neutron_pg_drop | grep xxxxx
--> port xxxxx in port_group neutron_pg_drop

ubuntu@nkshirsagar-bastion:~/stsstack-bundles/openstack$ openstack server list
+--------------------------------------+----------------+--------+------------------------+---------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+----------------+--------+------------------------+---------+-----------+
| 9fdae43b-1bff-444d-9483-a5832732dfe2 | cirros2-060935 | ACTIVE | private=1XXX | cirros2 | m1.cirros |
| 838e7eba-1b7d-43b7-8ad7-eafd9c9d5087 | jammy-060831 | ERROR | | jammy | m1.small |
| b971c0a1-4344-4fa8-adba-ef6f7449626c | testneutron | ACTIVE | private=XXX | jammy | m1.small |
+--------------------------------------+----------------+--------+------------------------+---------+-----------+
ubuntu@nkshirsagar-bastion:~/stsstack-bundles/openstack$ openstack port list --server cirros2-060935
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+
| 5f03dc77-12d0-497e-888d-c0f1019da68d | |XXX | ip_address='XXX', subnet_id='ebeedda7-ee84-4f04-93ae-941e12daf9b7' | ACTIVE |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+

ubuntu@nkshirsagar-bastion:~/stsstack-bundles/openstack$ openstack port show 5f03dc77-12d0-497e-888d-c0f1019da68d
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | juju-5a7359-neutronsru-8.cloud.sts ...

I was able to reproduce this issue successfully. The reproducer is detailed at https://pastebin.canonical.com/p/wBT7Hdrg5k/

ubuntu@nkshirsagar-bastion:~/stsstack-bundles/openstack$ openstack server ssh cirros2-060935 --private --login ubuntu
ssh: connect to host XXX.XXX.21.240 port 22: Connection timed out

I will now attempt to validate the fix after upgrading to the neutron proposed package on the neutron-api node.

Hello,

I have tested this fix on focal/ussuri and verified it solves the issue. After the repair step, the port does not appear in port_group neutron_pg_drop, and the vm is also ping responsive.

The reproducer details are in the earlier comment, and the patch verification testing details are in https://pastebin.canonical.com/p/2J6C9tx4np/

I am marking the verified flags accordingly. Since the patch is exactly the same for the cloud archive, I am marking that done as well.

Regards,
Nikhil.

Hey Nikhil,

Thanks for performing the verifications in such a detailed manner.

I happened to be looking at this SRU/bug and had some observations:

> the patch verification testing details are in https://pastebin.canonical.com/p/2J6C9tx4np/

Please always keep the verification steps/logging (if needed) in the bug
(as comments or attachments, as appropriate), for documentation purposes.

Even though pastebins are useful, they might be removed, and in this case,
canonical's pastebin isn't available externally (ubuntu's pastebin is),
which restricts the availability to other people/community members.

> I have tested this fix on focal/ussuri [...]
> Since the patch is exactly the same for the cloud archive, I am marking that done as well.

Although this is ultimately up to the Cloud Archive's release team, I would like to suggest the Cloud Archive to be separately verified too.

The reason is, there is more involved in the verification than just the source code change alone.
Specially in the case of the Cloud Archive, there are different (more recent) package versions that may be pulled and are exercised as part of the verification steps (e.g., python dependencies or other libraries).
Thus in practice, the actually verified code is different, even if the patch is the same.

Thanks!
Mauricio

reproducer and verification details uploaded

When I tried to test the same fix also for the cloud archive, I ran into this issue if I tried it on focal/ussuri deployed using stsstack, (this is on a neutron-api leader node)

root@juju-408761-fixtestnew-ca-6:/etc/apt/sources.list.d# add-apt-repository cloud-archive:ussuri-proposed
 Ubuntu Cloud Archive for OpenStack Ussuri [proposed]
 More info: https://wiki.ubuntu.com/OpenStack/CloudArchive
Press [ENTER] to continue or Ctrl-c to cancel adding it.

cloud-archive for Ussuri-proposed only supported on bionic

And if I try on bionic, I run into several charms not supported, first of which is ovn itself which doesnt have latest/stable listed at https://charmhub.io/ovn-central

ubuntu@nkshirsagar-bastion:~/stsstack-bundles/openstack$ ./generate-bundle.sh -s bionic -r ussuri -n fixtest-bionic-new --run --ovn
Creating Juju model fixtest-bionic-new
Added 'fixtest-bionic-new' model on stsstack/stsstack with credential 'nkshirsagar' for user 'admin'

Created bionic-ussuri bundle and overlays:
  + glance.yaml
  + keystone.yaml
  + mysql.yaml
  + rabbitmq-source.yaml
  + neutron-ovn.yaml
  + neutron-ml2dns.yaml
  + placement.yaml
  + vault.yaml
  + vault-openstack-secrets.yaml
  + vault-openstack-certificates.yaml
  + vault-openstack-certificates-placement.yaml

Command to deploy:
juju deploy /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/openstack.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/glance.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/keystone.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/mysql.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/rabbitmq-source.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/neutron-ovn.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/neutron-ml2dns.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/placement.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/vault.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/vault-openstack-secrets.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/vault-openstack-certificates.yaml --overlay /home/ubuntu/stsstack-bundles/openstack/b/fixtest-bionic-new/o/vault-openstack-certificates-placement.yaml

Located charm "cinder" in charm-hub, channel ussuri/stable
Located charm "glance" in charm-hub, channel ussuri/stable
Located charm "keystone" in charm-hub, channel ussuri/stable
Located charm "percona-cluster" in charm-hub, channel 5.7/stable
Located charm "neutron-api" in charm-hub, channel ussuri/stable
Located charm "neutron-api-plugin-ovn" in charm-hub, channel ussuri/stable
Located charm "nova-cloud-controller" in charm-hub, channel ussuri/stable
Located charm "nova-compute" in charm-hub, channel ussuri/stable
ERROR cannot deploy bundle: cannot resolve charm or bundle "ovn-central": no releases found for channel "latest/stable"

So I am not sure how to actually test the cloud archive packages on focal/ussuri.

----

UPDATE - needed bionic to test the ussuri cloud arc...

I have tested the cloud archive packages successfully using bionic-ussuri. Detailed testing details uploaded to the LP bug.

root@juju-971eb3-bionicneutronfix-4:~# apt-cache policy neutron-server neutron-plugin-ml2
neutron-server:
  Installed: 2:16.4.2-0ubuntu6.3~cloud0
  Candidate: 2:16.4.2-0ubuntu6.3~cloud0
  Version table:
 *** 2:16.4.2-0ubuntu6.3~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status
     2:16.4.2-0ubuntu6.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages
     2:12.1.1-0ubuntu8.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
neutron-plugin-ml2:
  Installed: 2:16.4.2-0ubuntu6.3~cloud0
  Candidate: 2:16.4.2-0ubuntu6.3~cloud0
  Version table:
 *** 2:16.4.2-0ubuntu6.3~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status
     2:16.4.2-0ubuntu6.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages
     2:12.1.1-0ubuntu8.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

After the repair,

ubuntu@nkshirsagar-bastion:~/stsstack-bundles/openstack$ juju ssh ovn-central/2 sudo -s
root@juju-971eb3-bionicneutronfix-9:~# ovn-nbctl --no-leader-only list port_group neutron_pg_drop
_uuid : 1e69b7d7-7cc0-4322-9b4a-bdcd6d48facb
acls : [1ed4c1bc-f01c-4e1c-a9a1-089aa9b3f76f, 3869948c-ecc5-43b0-9cea-8499444bdcbe]
external_ids : {}
name : neutron_pg_drop
ports : []
root@juju-971eb3-bionicneutronfix-9:~# ovn-nbctl --no-leader-only lsp-list neutron-d9ce9deb-9d77-4369-b47f-c279790c348a
ec57bd37-9bf3-4d8d-8c63-0a1896ee0652 (10898fe3-39bf-49a6-a680-13ef376b7ee1)
a2c378c2-c523-493b-8ef0-4a2ea45052f2 (4b23642d-33aa-4af0-b940-6fbd13584c70)
fbdf43de-d22c-498f-a1ca-8438a8724c70 (e1cd77da-f905-48fc-83f7-907b7e25a819)
root@juju-971eb3-bionicneutronfix-9:~#

This bug was fixed in the package neutron - 2:16.4.2-0ubuntu6.3

---------------
neutron (2:16.4.2-0ubuntu6.3) focal; urgency=medium

  * d/p/check-subnet-in-remove-subnet-dhcp-options.patch: Ensure dhcp_options
    subnet check handles dictionary correctly (LP: #1948466).
  * d/p/ovn-fix-untrusted-port-security-enabled-check.patch: Fix logic for
    check that wraps adding of port to drop port group (LP: #1939723).

 -- Corey Bryant <email address hidden> Mon, 21 Aug 2023 15:29:46 -0400

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:16.4.2-0ubuntu6.3~cloud0
---------------

 neutron (2:16.4.2-0ubuntu6.3~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:16.4.2-0ubuntu6.3) focal; urgency=medium
 .
   * d/p/check-subnet-in-remove-subnet-dhcp-options.patch: Ensure dhcp_options
     subnet check handles dictionary correctly (LP: #1948466).
   * d/p/ovn-fix-untrusted-port-security-enabled-check.patch: Fix logic for
     check that wraps adding of port to drop port group (LP: #1939723).