neutron-ovs-cleanup runs on package upgrade

Some additional information regarding how the bug was found:

During the upgrade the ovs runs, deleting the ports. They don't come back, even after restarting the services, which suggests an issue within the neutron openvswitch agent in terms of re-creating ports that have been deleted.

from the ovsdb-tool -mm show-log tool on the neutron-gateway AFTER the upgrade.

```
record 44: 2020-06-25 19:10:48.024
  table Interface row "tap03282320-fb" (74bf6e22):
    delete row
  table Interface row "tap7e15b3c6-e3" (d331a9b6):
    delete row
  table Interface row "tap8e76995d-a6" (f8ca1e54):
    delete row
  table Port row "tap03282320-fb" (d891c2a5):
    delete row
  table Port row "tap7e15b3c6-e3" (8f105882):
    delete row
  table Port row "tap8e76995d-a6" (b5f389e0):
    delete row
  table Bridge row "br-int" (56555638):
    ports=[704ecf79-ff42-44bd-a72c-c586f59f433b, dcfd1c64-2e07-4bf7-8815-a96950384705, f38673c2-87c9-4ce1-b160-236637caf8d2]
  table Open_vSwitch row 70200fb1 (70200fb1):
    next_cfg=111
```

i.e. it was asked to be deleted.

from syslog (during upgrade):

```
Jun 25 19:10:44 juju-ed722c-mojo-14 systemd[1]: Stopping Openstack Neutron Open vSwitch Plugin Agent...
Jun 25 19:10:44 juju-ed722c-mojo-14 ovsdb-client: ovs|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)
Jun 25 19:10:46 juju-ed722c-mojo-14 systemd[1]: Stopped Openstack Neutron Open vSwitch Plugin Agent.
Jun 25 19:10:46 juju-ed722c-mojo-14 systemd[1]: Stopped OpenStack Neutron OVS cleanup.
Jun 25 19:10:46 juju-ed722c-mojo-14 systemd[1]: Stopping OpenStack Neutron OVS cleanup...
Jun 25 19:10:46 juju-ed722c-mojo-14 systemd[1]: Starting OpenStack Neutron OVS cleanup...
Jun 25 19:10:48 juju-ed722c-mojo-14 kernel: [ 6499.589055] device tap03282320-fb left promiscuous mode
Jun 25 19:10:48 juju-ed722c-mojo-14 kernel: [ 6499.611240] device tap7e15b3c6-e3 left promiscuous mode
Jun 25 19:10:48 juju-ed722c-mojo-14 kernel: [ 6499.612612] device tap8e76995d-a6 left promiscuous mode
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Started OpenStack Neutron OVS cleanup.
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Starting Openstack Neutron Open vSwitch Plugin Agent...
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Started Openstack Neutron Open vSwitch Plugin Agent.
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Reloading.
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Stopping Openstack Neutron Open vSwitch Plugin Agent...
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Stopped Openstack Neutron Open vSwitch Plugin Agent.
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Stopped OpenStack Neutron OVS cleanup.
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Stopping OpenStack Neutron OVS cleanup...
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Starting OpenStack Neutron OVS cleanup...
Jun 25 19:10:50 juju-ed722c-mojo-14 systemd[1]: Started OpenStack Neutron OVS cleanup.
Jun 25 19:10:50 juju-ed722c-mojo-14 systemd[1]: Starting Openstack Neutron Open vSwitch Plugin Agent...
Jun 25 19:10:50 juju-ed722c-mojo-14 systemd[1]: Started Openstack Neutron Open vSwitch Plugin Agent.
Jun 25 19:10:51 juju-ed722c-mojo-14 systemd[1]: Reloading.
Jun 25 19:10:51 juju-ed722c-mojo-...

Updated packages upload for groovy and for focal (for SRU team review).

This bug was fixed in the package neutron - 2:16.0.0-0ubuntu3

---------------
neutron (2:16.0.0-0ubuntu3) groovy; urgency=medium

  * Ensure OVS and linuxbridge cleanup systemd units are not restarted
    on upgrade or started on install (LP: #1885264):
    - d/rules: Pass --no-stop-on-upgrade to dh_install{systemd,init} for
      ovs and linuxbridge cleanup systemd units.
    - d/rules: Pass --no-start to dh_install{systemd,init} for
      ovs and linuxbridge cleanup systemd units.
    - d/rules: Also pass --no-restart-after-upgrade to deal with bug
      in older debhelper versions in Ubuntu Cloud Archive.
    - d/rules: Pass appropriate -X flags to each call to
      dh_install{systemd,init} to ensure that duplicate snippets are
      not generated in maintainer scripts.
  * d/p/bug1879717.patch: Cherry pick unit testing fix for updates into
    ovsdbapp.
  * d/rules,test-blacklist.txt: Blacklist misc failing tests on groovy
    until next set of snapshots.
  * d/p/misc-py38-p1.patch: Cherry pick fix for failing unit test.

 -- James Page <email address hidden> Thu, 02 Jul 2020 07:51:11 +0100

Hello James, or anyone else affected,

Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.0.0-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

focal testing:

neutron-ovs-cleanup (on first install from update pocket)

     Active: active (exited) since Wed 2020-07-08 16:36:12 UTC; 32s ago

neutron-ovs-cleanup (after package upgrade to -proposed version)

     Active: active (exited) since Wed 2020-07-08 16:36:12 UTC; 1min 18s ago

task not restarted as part of the package upgrade.

bionic-ussuri/proposed testing:

neutron-ovs-cleanup not restarted on package upgrade:

$ sudo apt install --reinstall neutron-openvswitch-agent
...

$ sudo systemctl status neutron-ovs-cleanup

● neutron-ovs-cleanup.service - OpenStack Neutron OVS cleanup
   Loaded: loaded (/lib/systemd/system/neutron-ovs-cleanup.service; enabled; vendor preset: enabled)
   Active: active (exited) since Fri 2020-07-10 12:56:07 UTC; 47min ago
 Main PID: 12976 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4703)
   CGroup: /system.slice/neutron-ovs-cleanup.service

This bug was fixed in the package neutron - 2:17.0.0~git2020070709.792f4efce9-0ubuntu1~cloud0
---------------

 neutron (2:17.0.0~git2020070709.792f4efce9-0ubuntu1~cloud0) focal-victoria; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 neutron (2:17.0.0~git2020070709.792f4efce9-0ubuntu1) groovy; urgency=medium
 .
   * New upstream snapshot for OpenStack Victoria.
   * d/control: Align (Build-)Depends with upstream.
   * d/py3dist-overrides: Created with win32 dep wmi.
   * d/*.upstart.in: Dropped. No longer needed.
   * d/control: Update Standards-Version to 4.5.0.
   * d/p/fix-neutron-ipset-cleanup-tool.patch: Dropped. Fixed upstream.
   * d/p/bug1879717.patch: Dropped. Fixed upstream.
   * d/p/misc-py38-p1.patch: Dropped. Fixed upstream.
   * d/p/skip-iptest.patch: Rebased.
   * d/p/flake8-legacy.patch: Dropped. No longer needed.
   * d/neutron-common.install: Remove netns-cleanup.filters along with
     upstream removal as part of migration to oslo.privsep.
   * d/rules: Set OS_* env variables for unit test execution in line with
     upstream tox.ini.
 .
 neutron (2:16.0.0-0ubuntu3) groovy; urgency=medium
 .
   * Ensure OVS and linuxbridge cleanup systemd units are not restarted
     on upgrade or started on install (LP: #1885264):
     - d/rules: Pass --no-stop-on-upgrade to dh_install{systemd,init} for
       ovs and linuxbridge cleanup systemd units.
     - d/rules: Pass --no-start to dh_install{systemd,init} for
       ovs and linuxbridge cleanup systemd units.
     - d/rules: Also pass --no-restart-after-upgrade to deal with bug
       in older debhelper versions in Ubuntu Cloud Archive.
     - d/rules: Pass appropriate -X flags to each call to
       dh_install{systemd,init} to ensure that duplicate snippets are
       not generated in maintainer scripts.
   * d/p/bug1879717.patch: Cherry pick unit testing fix for updates into
     ovsdbapp.
   * d/rules,test-blacklist.txt: Blacklist misc failing tests on groovy
     until next set of snapshots.
   * d/p/misc-py38-p1.patch: Cherry pick fix for failing unit test.
 .
 neutron (2:16.0.0-0ubuntu2) groovy; urgency=medium
 .
   * d/control: Add net-tools dependency to python3-neutron for netstat
     (LP: #1881852).
   * d/neutron-common.install: Add rootwrap.d/netns-cleanup.filters
     (LP: #1881852).
   * d/p/fix-neutron-ipset-cleanup-tool.patch: Fix Traceback (LP: #1881771).
 .
 neutron (2:16.0.0-0ubuntu1) groovy; urgency=medium
 .
   * d/watch: Scope to 16.x series.
   * New upstream release for OpenStack Ussuri (LP: #1877642).
   * d/p/monkey-patch-original-current-thread.patch: Dropped. Fixed
     in upstream release.

This bug was fixed in the package neutron - 2:16.0.0-0ubuntu0.20.04.2

---------------
neutron (2:16.0.0-0ubuntu0.20.04.2) focal; urgency=medium

  * Ensure OVS and linuxbridge cleanup systemd units are not restarted
    on upgrade or started on install (LP: #1885264):
    - d/rules: Pass --no-stop-on-upgrade to dh_install{systemd,init} for
      ovs and linuxbridge cleanup systemd units.
    - d/rules: Pass --no-start to dh_install{systemd,init} for
      ovs and linuxbridge cleanup systemd units.
    - d/rules: Also pass --no-restart-after-upgrade to deal with bug
      in older debhelper versions in Ubuntu Cloud Archive.
    - d/rules: Pass appropriate -X flags to each call to
      dh_install{systemd,init} to ensure that duplicate snippets are
      not generated in maintainer scripts.

 -- James Page <email address hidden> Thu, 02 Jul 2020 08:27:02 +0100

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.