autopkgtest success rate dropped inhibiting proposed migration

Tests-in-lxd fails because LXD images ship invalid /etc/fstab again making systemd-remount-fs.service fail.
I'm marking systemd-fsckd test as flaky to not block others' work.

Thank you Rbalint.

Thanks to xnox I already found 1891716 and 1886886 which are sort of remotely related (to the next upload), I guess this bug will serve for the "marking systemd-fsckd test as flaky" task then.

Can we do something about this bad fstab in LXD or is there a bug to track?

Essentially I'd like to know when it overall is worth again to hit retry on my tests :-)

Adding update-excuse tag so that one can find this issue from update-excuses page.
To show up at the packages that fail I also add an qemu and kbd task (more might pile up)

Added another two packages that are blocked by this so they are marked/linked in update excuses :-/

Adding glib2.0 as it had a rebuild for libffi and will be blocke dby this as well.

@paelzer The fstab bug is LP: #1877078, I'm waiting for @vorlon's comment's about what his plan is after his revert.

[16:34] <rbalint> seb128, cpaelzer ahasenack sorry, i was out and apparently forgot setting that in my email. the last systemd upload should fix everything except for the livecd-rootfs revert that made fstab in lxd images invalide
[16:45] <cpaelzer> rbalint: " the last systemd upload" means groovy I guess, what about focal?
[16:47] <rbalint> cpaelzer, sru-s are usually prepared by ddstreet, but i guess you mean the flakiness
[16:49] <rbalint> cpaelzer, as i look at the last runs the latest focal systemd upload got flakier without a change in systemd so it may be an infra issue or needs more investigation
[16:50] <cpaelzer> rbalint: do you want a new bug about it for you and ddstreet then?
[16:50] <cpaelzer> to focus the discussion in one place I mean
[16:50] <rbalint> cpaelzer, LP: #1892358 is not open against focal so i think just adding focal would be enough

Task added

https://launchpad.net/ubuntu/+source/systemd/246.2-1ubuntu1 is still in proposed and I was looking forward to a bunch of custom triggers this morning.

But I found everything migrated this morning despite those new tests not being done.
https://autopkgtest.ubuntu.com/packages/s/systemd/groovy/amd64

Since the bug had no update about it I checked and found:
http://bazaar.launchpad.net/~ubuntu-release/britney/hints-ubuntu/revision/4957

Thanks rbalint - this certainly unblocks things for now.
Anything coming later should test against the new systemd version and then hopefully work.
Due to that I think we can set the formerly affected extra tasks to invalid (since they were only here for tracking of the update-excuses tag).

What is left is the similar situation in Focal (as mentioned before).
I have updated the tasks to reflect that properly and get update-excuse tagging there.

In Focal it is currently blocking qemu and build-essential SRUs and I wanted to ask if the plan is to do the same upload+test-reset there or anything else.

@ddstreet - rbalint said you are prepping the systemd SRUs, could you let us know what is the plan there?

Here an (monospace) overview of recent focal tests.

focal
  amd64
    networkd-testpy (F 5% S 0% B 5% => P 90%/) .........B.......F..
    boot-and-services (F 5% S 0% B 5% => P 90%/) .........B.......F..
    upstream (F 5% S 0% B 5% => P 90%/) .........B.......F..
    systemd-fsckd (F 95% S 0% B 5% => P 0%/) FFFFFFFFFBFFFFFFFFFF
  ppc64el
    systemd-fsckd (F 45% S 0% B 0% => P 55%/) FFFFFFF...F......F..
  s390x
    systemd-fsckd (F 35% S 0% B 0% => P 65%/) FFFFFFF.............

There also are some linux-meta uploads blocked by the same, but I don't know what task to add for them.

This bug was fixed in the package systemd - 246.2-1ubuntu1

---------------
systemd (246.2-1ubuntu1) groovy; urgency=medium

  * Merge from Debian unstable
  * Refresh patches
  * test: accept that char device 0/0 can now be created witout privileges.
    This fixes tests with Linux 5.8 and later (LP: #1891527)
    Files:
    - debian/patches/debian/Add-env-variable-for-machine-ID-path.patch
    - debian/patches/test-accept-that-char-device-0-0-can-now-be-created-witou.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=57909a0fa4999ba9ff918cfe6bb3fa53c230dd36
  * debian/tests/systemd-fsckd: Fail plymouth-start with overriding ExecStart=
    The 'process-killer' stopped working with plymouth 0.9.5 (LP: #1886886)
    File: debian/tests/systemd-fsckd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=efcc25b6011f23b9f90841f4ca0ee29ee674157f
  * cherry-pick from stable: network: only process non-error message
    File: debian/patches/network-only-process-non-error-message.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=dc97e3f5330ae64909ef640ad1ac401374f18c16
  * network: wait for previous address removal before configuring static addresses.
    Fixes systemd-networkd to establish connection after resume from suspend
    Files:
    - debian/patches/network-wait-for-previous-address-removal-before-configur.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=31390403bc000892ada9cb0cbedb198dd83fe7a2
  * network: do not fail if UseMTU=yes on DHCP lease lost
    File: debian/patches/network-do-not-fail-if-UseMTU-yes-on-DHCP-lease-lost.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=21c23966e8dc5fd0858664a5b89d6a9460c41f61
  * debian/systemd.postinst: Restart systemd-networkd.socket on incompatible change
    (Closes: #968589) (LP: #1891716)
    File: debian/systemd.postinst
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e144a644488c6ff52c2bb7cee71f50aa95c4efd8
  * debian/tests/control: Mark systemd-fsckd test as flaky.
    It is flaky on Ubuntu CI infrastructure (LP: #1892358)
    File: debian/tests/control
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8c55289624c6f44beb2c5bbae78aabc074d36b1a

systemd (246.2-1) unstable; urgency=medium

  * New upstream version 246.2
  * Remove resolvconf.conf drop-in, resolved integration moved to resolvconf
    package
  * Rebase patches
  * Add versioned Breaks against resolvconf (<< 1.83~) to systemd.
    The PathExists= directive was changed in v246 to match the documented
    behaviour but now causes resolvconf-pull-resolved.service to be
    continuously triggered by resolvconf-pull-resolved.path.
    This requires a fix in the resolvconf package, see #968015.
    (Closes: #967906)
  * Keep journal files compatible with older versions.
    Disable the KEYED-HASH journal feature by default and keep LZ4 (instead
    of ZSTD) as default compression for new journal files. Otherwise journal
    files are incompatible and can't be read by older journalctl
    implementation...

This seems to be also affecting Bionic: http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/amd64

We did upload new kernels, so it's difficult to isolate the issue to systemd, but the failures are similar to what we are seeing in Focal.

True @Kleber, here the stats of Bionic - at least on x86 you seem to have barely a chance.

bionic
  amd64
    boot-smoke (F 40% S 0% B 0% => P 60%/) ...F.....FFF.FF.FF..
    upstream (F 5% S 0% B 0% => P 95%/) ..F.................
    systemd-fsckd (F 90% S 0% B 0% => P 10%/) FFFFFFFFFFF.F.FFFFFF
  i386
    upstream (F 20% S 0% B 0% => P 80%/) .F.....F......F....F
    systemd-fsckd (F 20% S 0% B 0% => P 80%/) FF.F..F.............
  ppc64el
    upstream (F 15% S 0% B 0% => P 85%/) F..............FF...
    systemd-fsckd (F 10% S 0% B 0% => P 90%/) ...F...F............
  s390x
    no failures
  armhf
    no failures
  arm64
    logind (F 15% S 0% B 5% => P 80%/) .......B.....FF....F

Status changed to 'Confirmed' because the bug affects multiple users.

Util-linux SRU in Focal blocked as well, updating tags.

To match the open packages blocked on this in active releases I added linux-meta for BIonic (thanks Kleber for the hint) and Focal (Thanks Kelsey for the hint).

I think all of those had enough of retry-until-success and I'd ask again for how we should proceed there. Masking/Resetting the test via an override would throw away so much other coverage, but lacking other feedback I'll at least proposed the change - maybe this also brings more attention to here.

Last 20

focal
  amd64
    upstream (F 5% S 0% B 5% => P 90%/) F.................B.
    systemd-fsckd (F 95% S 0% B 5% => P 0%/) FFFFFFFFFFFFFFFFFFBF
  ppc64el
    systemd-fsckd (F 65% S 0% B 0% => P 35%/) FFFFFFFFFFFF...F....
  s390x
    systemd-fsckd (F 60% S 0% B 0% => P 40%/) FFFFFFFFFFFF........

bionic
  amd64
    boot-smoke (F 20% S 0% B 0% => P 80%/) ...........F.....FFF
    upstream (F 10% S 0% B 0% => P 90%/) .......F..F.........
    systemd-fsckd (F 85% S 0% B 0% => P 15%/) FFF.FF.FFFFFFFFFFFF.

They all seem to continue to be heavy on "systemd-fsckd"

If not going for marking the subtest flaky as in groovy, I have prepared force-reset-test as alternatives in:
- https://code.launchpad.net/~paelzer/britney/hints-ubuntu-focal-disable-systemd/+merge/390005
- https://code.launchpad.net/~paelzer/britney/hints-ubuntu-focal-disable-systemd/+merge/390004

hi @cpaelzer, sorry i missed reading this earlier, i'll give all the test failures a look to see what can be done.

@ddstret - any update how to proceed?
If you have no time yet, then the MPs to ignore the fail until we have a new version are up - just ack them and I guess the SRU Team will follow.

[15:17] <cpaelzer> ddstreet: any updates on 1892358?
[15:18] <ddstreet> @paelzer sorry not yet, i'll have systemd ready for upload next week including fixing the autopkgtests
[15:21] <cpaelzer> ok

Groovy
It seems this isn't over for groovy, it was mentioned that the issues in the subtest "tests-in-lxd" would be related to some fstab issues in those issues. But recently all architectures, but s390x recovered. I assume that the fstab issue is fixed but something else surfaces now.

groovy
  s390x
    tests-in-lxd (F 100% f 0% S 0% B 0% => P 0%/) FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
    boot-and-services (F 3% f 0% S 0% B 0% => P 96%/) .....F........................
    systemd-fsckd (F 26% f 73% S 0% B 0% => P 0%/) ffffffffffffffffffffFFFFfFFfFF

We see when systemd-fsckd was marked flaky (F->f), but tests-in-lxd continues to perma-stop any migration depending on systemd.

Currently this holds back a lot of things in groovy, it seems to be:
qemu/1:5.0-5ubuntu7 chiark-tcl/1.3.4ubuntu3 cmake-extras/1.5-1ubuntu1 cstore-fdw/1.7.0-1 dateparser/0.7.4-1 dpdk/19.11.4-1 gem2deb/1.2.1 glibc/2.32-0ubuntu2 gnome-desktop3/3.37.90.1-1ubuntu1 grep/3.4-1 libnss-nis/3.1-0ubuntu3 libnss-nisplus/1.3-0ubuntu3 link-grammar/5.8.0-1 mksh/59b-1 natsort/7.0.1-1 openconnect/8.10-1 orafce/3.13.4-1 pandas/1.0.5+dfsg-3 pango1.0/1.46.1-1 php-nesbot-carbon/2.32.2-1 php-shellcommand/1.6.1-1 php-twig/2.13.0-1 postgresql-12/12.4-1 postgresql-common/216 postgresql-multicorn/1.4.0-2 reprotest/0.7.15 snapd/2.46.1+20.10 symfony/4.4.11+dfsg-1ubuntu1 systemd/246.4-1ubuntu1 tracker/2.3.4-1build1 vis/0.6-3

Therefore re-opening the groovy task, this isn't over yet :-/
@rbalint please let us know if you know more details about it already.

@paelzer I agree that the fact, i.e. that systemd autpkgtest is failing in Groovy/s390x, but this is a real regression which is not due to flakiness of tests-in-lxd.
Please don't reopen this bug, but file a new one or find one already opened. As I see this is still LP: #1878225. I'm filing a hint for it for now.

@paelzer To correct myself this (s390x) regression is not due to the snapd seeding issue, but LP: #1895576 which may have been already fixed.

Thanks rbalint for pointing me to the other bug for the s390x fail we see now.
I'll continue there and consider the Groovy portion of this one closed again.

Seeing similar problems on Xenial and I suspect it's similar to the problems occurring with Bionic and Focal: http://autopkgtest.ubuntu.com/packages/s/systemd/xenial/amd64

Focal test hint was accepted tonight and things moved there.
@Kelsey - Steve denied the hint as a reset since recently a few results were good.
Maybe you want to submit the same as force-badtest and bring it up like that for X&B?

The results for xenial really look similarly bad ...

xenial
  amd64
    boot-and-services (F 45% f 0% S 0% B 5% => P 50%/) .FFFFFF...BFF.....F.
    boot-smoke (F 80% f 0% S 0% B 5% => P 15%/) FFFFF.FF..BFFFFFFFFF
    systemd-fsckd (F 65% f 0% S 0% B 5% => P 30%/) FF.FFF.F..B.FF.FFFFF
  ppc64el
    no failures
  s390x
    no failures

I know that ddstreet is considering the SRU updates for systemd to also mark those flaky.
I'm still +1 to mark them force-badtest until then but I'm not the one to decide.

The problem with Xenial is that there it isn't just flaky "systemd-fsckd" much more seems to be broken. I'll revise my Bionic MP, but for Xenial I'd like to hear from ddstreet/rbalint how they think we should go on there ...

And on Bionic we only need to do so for amd64&i386.
Here is a new MP for Bionic:
https://code.launchpad.net/~paelzer/britney/hints-ubuntu-bionic-disable-systemd-v2/+merge/390793

Thank you @christian. I marked the Xenial test cases under this same bug, though can open a new bug and update the hint bug references if needed. I'll check back to see when/if ddstreet/rbalint have a chance to review.

For the test results you're posting, I was wondering where you're getting those results from and an explanation of the results?

xenial
  amd64
    boot-and-services (F 45% f 0% S 0% B 5% => P 50%/) .FFFFFF...BFF.....F.
    boot-smoke (F 80% f 0% S 0% B 5% => P 15%/) FFFFF.FF..BFFFFFFFFF
    systemd-fsckd (F 65% f 0% S 0% B 5% => P 30%/) FF.FFF.F..B.FF.FFFFF
  ppc64el
    no failures
  s390x
    no failures

In the above, what does the F, f, B, & . stand for? Am I correct to assume the P *% is the pass rate? F for fail?

Thank you!

ok, very sorry for my delay in looking at this.

a quick scan of the results for x/b/f show:

ddstreet@smaug:~$ autopkgtest-manager --distro systemd -a all -r xenial -cv --since 2month --minimum 6
parsed '2month' as '2020-08-09T14:53:00.893145-04:00'
88: boot-smoke FAIL non-zero exit status 1
    (amd64:87 arm64:1)
39: systemd-fsckd FAIL non-zero exit status 1
    (amd64:39)
47: boot-and-services FAIL non-zero exit status 1
    (amd64:41 arm64:6)
ddstreet@smaug:~$ autopkgtest-manager --distro systemd -a all -r bionic -cv --since 2month --minimum 6
parsed '2month' as '2020-08-09T14:53:18.194358-04:00'
132: systemd-fsckd FAIL non-zero exit status 1
     (i386:14 amd64:117 ppc64el:1)
15: upstream FAIL non-zero exit status 1
    (i386:3 amd64:10 ppc64el:1 arm64:1)
29: boot-smoke FAIL non-zero exit status 1
    (i386:2 amd64:27)
ddstreet@smaug:~$ autopkgtest-manager --distro systemd -a all -r focal -cv --since 2month --minimum 6
parsed '2month' as '2020-08-09T14:56:04.065710-04:00'
96: systemd-fsckd FAIL non-zero exit status 137
    (amd64:68 ppc64el:20 s390x:8)
21: systemd-fsckd FAIL non-zero exit status 1
    (amd64:1 ppc64el:4 s390x:16)

I think the systemd-fsckd failures are issues with how the test is trying to detect 'good' and 'bad' service status, but in any case i rewrote some of the details of how the test works to be simpler and (hopefully) more accurate/reliable, I'm running it through tests from my ppa now, before I upload the test change.

I'll take a look at the other test failures; the boot-and-services test has typically had a lot of trouble with services randomly not starting ok, but i'll check on the details.

Hello Christian, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.43 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (237-3ubuntu10.43) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.4/5.4.0-52.57~18.04.1 (armhf)
lxc/3.0.3-0ubuntu1~18.04.1 (ppc64el)
openssh/1:7.6p1-4ubuntu0.3 (s390x, arm64, armhf, ppc64el, i386, amd64)
linux-aws-5.0/unknown (amd64)
gvfs/1.36.1-0ubuntu1.3.3 (arm64)
suricata/3.2-2ubuntu3 (i386)
nftables/0.8.2-1 (amd64)
libvirt/4.0.0-1ubuntu8.17 (i386)
netplan.io/0.99-0ubuntu3~18.04.3 (amd64)
systemd/237-3ubuntu10.43 (i386)
linux-hwe-5.0/5.0.0-62.67 (arm64, armhf)
docker.io/19.03.6-0ubuntu1~18.04.2 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hello Christian, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.8/5.8.0-25.26~20.04.1 (armhf)
python-dbusmock/0.19-1 (armhf)
lxc/1:4.0.2-0ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

autopkgtests pass for focal for all archs (except i386 which is expected to fail for focal)

autopkgtests pass for bionic for all archs, except once for i386, for which I opened bug 1902554; the i386 test passed on retry.

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package systemd - 245.4-4ubuntu3.3

---------------
systemd (245.4-4ubuntu3.3) focal; urgency=medium

  [ Rafael David Tinoco ]
  * d/p/lp1861941-dont-generate-disk-byuuid-for-bcache-uuid.patch:
    Reworded and reintroduced patch to fully explain delta is NOT a fix to
    LP: #1861941 if the bcache-tools patch exists, but should be kept anyway
    as the change makes sense for a better experience to end user.
    (LP: #1861941)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f8f64b3b58a04a83b1c426818b9affc41e0bff6c

  [ Dan Streetman ]
  * d/p/lp1882596-man-fix-some-manvolnum.patch:
    - fix some man section references (LP: #1882596)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3959ec95eff78d38ec4409807f151572afe83fe9
  * d/p/lp1895418-correct-resolved-conf-cache-default.patch:
    - fix resolved.conf default Cache= value (LP: #1895418)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ebe274a2b01658ee39b372d7033c35209510b028
  * d/p/lp1897744-resolve-enable-RES_TRUSTAD-towards-the-127.0.0.53-st.patch:
    - add resolv.conf 'trust-ad' option (LP: #1897744)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f6acc8c620b80adab7b048352d85e722b5ba8214
  * d/t/*:
    - Update tests to fix false negatives (LP: #1892358)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=cee6c31a6caec7888270c9fa8757105ab950ed0c
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=a1c1a2bb0ff27faf84fe94583631dfd0f1f4ed8f
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9417ce996766c133c2a33d4102ce1494f3166774

 -- Dan Streetman <email address hidden> Thu, 08 Oct 2020 16:14:56 -0400

This bug was fixed in the package systemd - 237-3ubuntu10.43

---------------
systemd (237-3ubuntu10.43) bionic; urgency=medium

  [ Guilherme G. Piccoli ]
  * d/p/lp1830746-bump-mlock-ulimit-to-64Mb.patch:
    - Bump the memlock limit to match Focal and newer releases (LP: #1830746)
      https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=61adb797642f3dd2e5c14f7914c2949c665cefe8

  [ Victor Manuel Tapia King ]
  * d/p/lp1896614-core-Avoid-race-when-starting-dbus-services.patch:
    - Fix race when starting dbus services (LP: #1896614)
      https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=373cb6ccd6978a7112bbfd7e5cf4f703a9f8448e

  [ Dan Streetman ]
  * d/t/*,
    d/p/lp1892358/0001-test-increase-qemu-timeout-for-TEST-08-and-TEST-09.patch,
    d/p/lp1892358/0002-test-increase-timeout-for-TEST-17-UDEV-WANTS.patch,
    d/p/lp1892358/0003-test-increase-qemu-timeout-for-TEST-18-and-TEST-19.patch:
    - Increase QEMU_TIMEOUT on 'upstream' autopkgtest tests
    - Pull latest tests from newer releases to fix false negatives
    - Blacklist flaky 'upstream' TEST-03
      (LP: #1892358)
      https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9fd8391c2499e163515b629a8ca5790898fc599d
      https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d1756b3e1c3e625ed7162cff4909e7a29c315051
      https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=37f8d73516a84e85e4057d6a92204b4a174af718
      https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=229ed2076eb773efc548035262b8b8009bf89207
      https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f2d7b1f952667316cc07a4b3c5010e66ace07a90
      https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=659befe61bbfeb7afc9efa24458c9745412d7c6d

 -- Victor Manuel Tapia King <email address hidden> Wed, 07 Oct 2020 16:30:03 -0400

I am a bit confused on this bug. I know it's a rather 'vague' bug describing overall flaky/troubling tests, but for instance it has been marked as Fix Released for all the series already (groovy as well, as this was devel back then). Now it's again present in the changelog entries for at least the groovy systemd SRU (probably for others as well). But because of this, I don't know what's the state of the fixes in say, hirsute? Are those backported from there? Would be nice if an explanation why this bug is being re-opened was stated before submitting the SRU.

Not saying we shouldn't re-use existing bugs, no. Just saying that this way not much information about the current situation is given.

@sil2100 Yes this bug has been recycled a bit too many times. We should open a new one when the tests become flaky. The fixes related to this bug are present in Hirsute.

Hello Christian, or anyone else affected,

Accepted systemd into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/246.6-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (246.6-1ubuntu1.1) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.46.1-1ubuntu1 (arm64, amd64)
prometheus/2.20.0+ds-1 (s390x)
netplan.io/0.100-0ubuntu5 (arm64)
flatpak/1.8.2-1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

autopkgtests pass for b/f/g

This bug was fixed in the package systemd - 246.6-1ubuntu1.1

---------------
systemd (246.6-1ubuntu1.1) groovy; urgency=medium

  [ Dan Streetman ]
  * d/t/boot-smoke: update test to avoid false negatives
    (LP: #1892358)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=205c30ca53b0e421db28bb56afaf5f88650ce592
  * d/t/boot-and-services: remove unneeded test lines
    (LP: #1892358)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=71853082af4e668996db574915c5a156f9897fd3
  * d/t/systemd-fsckd: rewrite test to try to fix false negatives
    (LP: #1892358)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6ae6be039ec582410769d2d6d131e12bdcd19a68
  * d/p/lp1905044-test-use-cap_last_cap-for-max-supported-cap-number-n.patch:
    test: use cap_last_cap() instead of capability_list_length()
    (LP: #1905044)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=84a4832f5f7d4f939c1c78c6be4c3f9e05cd7f59
  * d/p/lp1907306/0001-sd-dhcp-client-don-t-log-timeouts-if-already-expired.patch,
    d/p/lp1907306/0002-sd-dhcp-client-track-dhcp4-t1-t2-expire-times.patch,
    d/p/lp1907306/0003-sd-dhcp-client-add-RFC2131-retransmission-details.patch,
    d/p/lp1907306/0004-sd-dhcp-client-simplify-dhcp4-t1-t2-parsing.patch,
    d/p/lp1907306/0005-sd-dhcp-client-correct-dhcpv4-renew-rebind-retransmi.patch,
    d/p/lp1907306/0006-sd-dhcp-client-correct-retransmission-timeout-to-mat.patch,
    d/p/lp1907306/0007-test-network-increase-wait_online-timeout-to-handle-.patch,
    d/p/lp1907306/0008-sd-dhcp-client-fix-renew-rebind-timeout-calculation-.patch:
    Send correct number of dhcpv4 renew and rebind requests
    (LP: #1907306)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0a96dc16ac00e90cd3904e6d490d676b9bb98f1f
  * d/p/lp1902960-udev-re-assign-ID_NET_DRIVER-ID_NET_LINK_FILE-ID_NET.patch:
    Run net_setup_link on 'change' uevents (LP: #1902960)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7183e2ef4758ce47b152dec735e7d213d6003e37
  * d/t/root-unittests:
    Remove any corrupt journal files (LP: #1881947)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3d0ea66f0db4a204759fa0005f6f27579ee4195a

  [ Balint Reczey ]
  * d/t/systemd-fsckd: Plymouth-start stays active in 20.10 and later
    (LP: #1908067)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e3ddd09301c8bdaa59b4fe54d7906f609552370d

 -- Dan Streetman <email address hidden> Wed, 06 Jan 2021 15:40:39 -0500