[SRU] openvswitch 2.13.3 / linking changes in DPDK 19.11.x

The DPDK portion of this, which currently blocks the rebuild of OVS, is due to this issue [1].
I've reported and blocked this issue from 19.11.4, but it eventually crept into 19.11.6 as discussed here [2].

I'll discuss upstream if everyone would be ok with taking those conflicting changes out of 19.11.x (my preferred way) and until then provide you a preliminary DPDK build into the PPA [3].

[1]: http://mails.dpdk.org/archives/stable/2020-September/024796.html
[2]: http://mails.dpdk.org/archives/stable/2021-March/029418.html
[3]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690/+packages

Changes to remove:
f49248a990 Revert "Revert "build/pkg-config: prevent overlinking""
39586a4cf0 Revert "Revert "build/pkg-config: improve static linking flags""
906e935a1f Revert "Revert "build/pkg-config: output drivers first for static build""
deebf95239 Revert "Revert "build/pkg-config: move pkg-config file creation""
a3bd9a34bf Revert "Revert "build: always link whole DPDK static libraries""
d4bc124438 Revert "Revert "devtools: test static linkage with pkg-config""

This series changes linking behavior in 19.11.x and thereby should stay
out. It breaks - for example - a rebuild of OVS 2.13 or builds
of later OVS 2.13.x stable releases.

File moved on: deebf952 "build/pkg-config: move pkg-config file creation"
Later changes to buildtools/pkg-config/meson.build that we need to re-create.
3f33f5056 build: fix plugin load on static build
4a2baf099 build: fix linker flags on Windows
Those have to take place in meson.build after this is all reverted and gone.

I have sent the proposal to remove that to <email address hidden> as I want to get the buy in there to not re-re-re-re-revert later :-)

I uploaded a first shot at a focal/groovy revision of this as:
  dpdk_19.11.7-0ubuntu0.20.04.1~ppa1_source.changes
  dpdk_19.11.7-0ubuntu0.20.10.1~ppa1_source.changes
to the PPA [1].

That is the most recent released LTS version plus the reverts of the identified breaking changes as proposed to <email address hidden> [2].

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690/+packages
[2]: http://mails.dpdk.org/archives/stable/2021-March/029484.html

This was reviewed and tested by Jamespage (in regard to the OVS build) and by upstream (in regard to the changes on top of 19.11.7 and for the stable release itself).
Thereby I think this is ready to enter the SRU process.

Uploaded to F/G-unapproved
- dpdk_19.11.7-0ubuntu0.20.04.1_source.changes
- dpdk_19.11.7-0ubuntu0.20.10.1_source.changes

@SRU Team - OVS is now also uploaded. A versioned build-depends ensures that it works in the right order no matter how you accept it.

FYI: All upstream tests completed as well and pushed to the DPDK 19.11 stable branch. So it will surely be part of the next released version.

Hello James, or anyone else affected,

Accepted dpdk into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpdk/19.11.7-0ubuntu0.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello James, or anyone else affected,

Accepted dpdk into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpdk/19.11.7-0ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Tests on Focal LGTM
ubuntu@node-horsea:~/dpdk-testing$ tail -f dpdk-test.status
VIRT_SPAWNGUESTS
VIRT_TESTPMD
VIRT_L2FWD
BENCH_OVS
VUC_BENCH_OVSDPDK
VUC_ENDURANCE_STARTSTOP
VUC_ENDURANCE_ADDREMOVEPORT

1.0.0 (07:26:35): phys (BM) tests
  1.1.0 (07:26:35): initialize environment
    1.1.1 (07:29:19): testpmd => Pass
    1.1.2 (07:30:29): check testpmd output => Pass
2.0.0 (07:30:29): prep virtual test environment

1.0.0 (07:32:39): virt tests
  1.1.0 (07:32:39): initialize environment
3.0.0 (07:33:14): performance tests
  3.1.0 (07:33:14): prep benchmarks
  3.2.0 (07:34:53): performance tests
    3.2.1 (07:35:01): test guest-openvswitch for OVS-5CPU => Pass
    3.2.2 (07:53:52): test guest-dpdk-vhost-user-client-multiq for OVSDPDK-VUC => Pass
4.0.0 (08:14:19): VUC endurance checks
  4.1.0 (08:14:19): prep VUC endurance tests
    4.1.1 (08:24:56): start stop guests (client) => Pass
    4.1.2 (09:47:22): add/remove ports (client) => Pass
  4.2.0 (09:56:26): Final cleanup

Groovy

VIRT_SPAWNGUESTS
VIRT_TESTPMD
VIRT_L2FWD
BENCH_OVS
VUC_BENCH_OVSDPDK
VUC_ENDURANCE_STARTSTOP
VUC_ENDURANCE_ADDREMOVEPORT

1.0.0 (11:16:12): phys (BM) tests
  1.1.0 (11:16:12): initialize environment
    1.1.1 (11:18:35): testpmd => Pass
    1.1.2 (11:19:36): check testpmd output => Pass
2.0.0 (11:19:36): prep virtual test environment

1.0.0 (11:21:39): virt tests
  1.1.0 (11:21:39): initialize environment
3.0.0 (11:22:22): performance tests
  3.1.0 (11:22:22): prep benchmarks
  3.2.0 (11:22:42): performance tests
    3.2.1 (11:22:57): test guest-openvswitch for OVS-5CPU => Pass
    3.2.2 (11:41:32): test guest-dpdk-vhost-user-client-multiq for OVSDPDK-VUC => Pass
4.0.0 (12:00:39): VUC endurance checks
  4.1.0 (12:00:39): prep VUC endurance tests
    4.1.1 (12:09:59): start stop guests (client) => Pass
    4.1.2 (13:26:49): add/remove ports (client) => Pass
  4.2.0 (13:35:44): Final cleanup

This bug was fixed in the package dpdk - 19.11.7-0ubuntu0.20.10.1

---------------
dpdk (19.11.7-0ubuntu0.20.10.1) groovy; urgency=medium

  * Merge the latest upstream stable minor release 19.11.7-rc1
    19.11.7 at https://doc.dpdk.org/guides-19.11/rel_notes/release_19_11.html
  * fix linking back to pre be like 19.11.6 behavior (LP: #1920141)

 -- Christian Ehrhardt <email address hidden> Tue, 23 Feb 2021 11:03:38 +0100

The verification of the Stable Release Update for dpdk has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package dpdk - 19.11.7-0ubuntu0.20.04.1

---------------
dpdk (19.11.7-0ubuntu0.20.04.1) focal; urgency=medium

  * Merge the latest upstream stable minor release 19.11.7-rc1
    19.11.7 at https://doc.dpdk.org/guides-19.11/rel_notes/release_19_11.html
  * fix linking back to pre be like 19.11.6 behavior (LP: #1920141)

 -- Christian Ehrhardt <email address hidden> Tue, 23 Feb 2021 11:03:38 +0100

Hey @james-page, the customer confirmed that using temporary PPA [0] mentioned in [1] had fixed lp:1907686 [2], can you continue to merge PPA into Focal? thanks.

[0] https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690
[1] https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/comments/15
[2] https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/

There was a minor issue with the uploaded packages due to interim security updates - this has now been resolved - I'll ping someone in the SRU team to take a look again.

Corey is going to look at why debian/ltmain-whole-archive.diff and a corresponding d/rules change is being added in the Groovy upload, as this appears erroneous in that it isn't the same as in Focal, which now has the same version of DPDK.

Hello James, or anyone else affected,

Accepted openvswitch into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvswitch/2.13.3-0ubuntu0.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello James, or anyone else affected,

Accepted openvswitch into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvswitch/2.13.3-0ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted openvswitch (2.13.3-0ubuntu0.20.10.1) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

netplan.io/0.101-0ubuntu3~20.10.1 (amd64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#openvswitch

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted openvswitch (2.13.3-0ubuntu0.20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

netplan.io/0.101-0ubuntu3~20.04.2 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#openvswitch

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Regression testing results for this were successful (apologies for the delay):

== focal-ussuri ==
21:38:43 ======
21:38:43 Totals
21:38:43 ======
21:38:43 Ran: 97 tests in 1663.4194 sec.
21:38:43 - Passed: 89
21:38:43 - Skipped: 8
21:38:43 - Expected Fail: 0
21:38:43 - Unexpected Success: 0
21:38:43 - Failed: 0
21:38:43 Sum of execute time for each test: 1151.4881 sec.

== bionic-ussuri ==
02:32:56 ======
02:32:56 Totals
02:32:56 ======
02:32:56 Ran: 97 tests in 960.0834 sec.
02:32:56 - Passed: 89
02:32:56 - Skipped: 8
02:32:56 - Expected Fail: 0
02:32:56 - Unexpected Success: 0
02:32:56 - Failed: 0
02:32:56 Sum of execute time for each test: 827.3194 sec.

== groovy-victoria ==
00:58:25 ======
00:58:25 Totals
00:58:25 ======
00:58:25 Ran: 97 tests in 1343.5539 sec.
00:58:25 - Passed: 89
00:58:25 - Skipped: 8
00:58:25 - Expected Fail: 0
00:58:25 - Unexpected Success: 0
00:58:25 - Failed: 0
00:58:25 Sum of execute time for each test: 1143.6704 sec.

== focal-victoria ==
04:39:56 ======
04:39:56 Totals
04:39:56 ======
04:39:56 Ran: 97 tests in 1445.2159 sec.
04:39:56 - Passed: 89
04:39:56 - Skipped: 8
04:39:56 - Expected Fail: 0
04:39:56 - Unexpected Success: 0
04:39:56 - Failed: 0
04:39:56 Sum of execute time for each test: 1045.0895 sec.

I checked on the autopkgtest failures reported for focal and groovy - the failing test is related to general netplan.io linux bond testing and is unrelated to the OVS support in netplan.io.

This bug was fixed in the package openvswitch - 2.13.3-0ubuntu0.20.10.1

---------------
openvswitch (2.13.3-0ubuntu0.20.10.1) groovy; urgency=medium

  [ James Page ]
  * New upstream point release (LP: #1920141, LP: #1907686):
    - d/p/*: Refresh including drop of part of patch as equivalent
      changes included in point release.
    - Dropped security patches, included in release:
      + CVE-2015-8011.patch
      + CVE-2020-27827.patch
      + CVE-2020-35498.patch
  * Add BD on libdbus-1-dev to resolve linking issues for DPDK builds due
    to changes in DPDK.
  * d/control: Set minimum version of libdpdk-dev to avoid build
    failures with 19.11.6-0ubuntu0.20.10.1.

  [ Frode Nordahl ]
  * Fix recording of FQDN/hostname on startup (LP: #1915829):
    - d/p/ovs-dev-ovs-ctl-Allow-recording-hostname-separately.patch: Cherry
      pick of committed upstream fix to support skip of hostname
      configuration on ovs-vswitchd/ovsdb-server startup.
    - d/openvswitch-switch.ovs-record-hostname.service: Record hostname in
      Open vSwitch after network-online.target using new systemd unit.
    - d/openvswitch-switch.ovs-vswitchd.service: Pass `--no-record-hostname`
      option to `ovs-ctl` to delegate recording of hostname to the separate
      service.
    - d/openvswitch-switch.ovsdb-server.service: Pass `--no-record-hostname`
      option to `ovs-ctl` to delegate recording of hostname to the separate
      service.
    - d/openvswitch-switch.service: Add `Also` reference to
      ovs-record-hostname.service so that the service is enabled on install.
    - d/rules: Add `ovs-record-hostname.service` to package build.

 -- James Page <email address hidden> Wed, 24 Mar 2021 08:46:06 +0000

This bug was fixed in the package openvswitch - 2.13.3-0ubuntu0.20.04.1

---------------
openvswitch (2.13.3-0ubuntu0.20.04.1) focal; urgency=medium

  [ James Page ]
  * New upstream point release (LP: #1920141, LP: #1907686).
  * Dropped security patches, included in release:
    - CVE-2015-8011.patch
    - CVE-2020-27827.patch
    - CVE-2020-35498.patch
  * Add BD on libdbus-1-dev to resolve linking issues for DPDK builds due
    to changes in DPDK.
  * d/control: Set minimum version of libdpdk-dev to avoid build
    failures with 19.11.6-0ubuntu0.20.04.1.

  [ Frode Nordahl ]
  * Fix recording of FQDN/hostname on startup (LP: #1915829):
    - d/p/ovs-dev-ovs-ctl-Allow-recording-hostname-separately.patch: Cherry
      pick of committed upstream fix to support skip of hostname
      configuration on ovs-vswitchd/ovsdb-server startup.
    - d/openvswitch-switch.ovs-record-hostname.service: Record hostname in
      Open vSwitch after network-online.target using new systemd unit.
    - d/openvswitch-switch.ovs-vswitchd.service: Pass `--no-record-hostname`
      option to `ovs-ctl` to delegate recording of hostname to the separate
      service.
    - d/openvswitch-switch.ovsdb-server.service: Pass `--no-record-hostname`
      option to `ovs-ctl` to delegate recording of hostname to the separate
      service.
    - d/openvswitch-switch.service: Add `Also` reference to
      ovs-record-hostname.service so that the service is enabled on install.
    - d/rules: Add `ovs-record-hostname.service` to package build.

 -- James Page <email address hidden> Wed, 24 Mar 2021 08:46:41 +0000

The verification of the Stable Release Update for openvswitch has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package openvswitch - 2.13.3-0ubuntu0.20.04.1~cloud0
---------------

 openvswitch (2.13.3-0ubuntu0.20.04.1~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 openvswitch (2.13.3-0ubuntu0.20.04.1) focal; urgency=medium
 .
   [ James Page ]
   * New upstream point release (LP: #1920141, LP: #1907686).
   * Dropped security patches, included in release:
     - CVE-2015-8011.patch
     - CVE-2020-27827.patch
     - CVE-2020-35498.patch
   * Add BD on libdbus-1-dev to resolve linking issues for DPDK builds due
     to changes in DPDK.
   * d/control: Set minimum version of libdpdk-dev to avoid build
     failures with 19.11.6-0ubuntu0.20.04.1.
 .
   [ Frode Nordahl ]
   * Fix recording of FQDN/hostname on startup (LP: #1915829):
     - d/p/ovs-dev-ovs-ctl-Allow-recording-hostname-separately.patch: Cherry
       pick of committed upstream fix to support skip of hostname
       configuration on ovs-vswitchd/ovsdb-server startup.
     - d/openvswitch-switch.ovs-record-hostname.service: Record hostname in
       Open vSwitch after network-online.target using new systemd unit.
     - d/openvswitch-switch.ovs-vswitchd.service: Pass `--no-record-hostname`
       option to `ovs-ctl` to delegate recording of hostname to the separate
       service.
     - d/openvswitch-switch.ovsdb-server.service: Pass `--no-record-hostname`
       option to `ovs-ctl` to delegate recording of hostname to the separate
       service.
     - d/openvswitch-switch.service: Add `Also` reference to
       ovs-record-hostname.service so that the service is enabled on install.
     - d/rules: Add `ovs-record-hostname.service` to package build.