systemd: networkd-test.py fails with newer kinetic kernels

Bug #2009859 reported by Nick Rosbrook
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Undecided
Unassigned
Jammy
New
Undecided
Unassigned
Kinetic
Won't Fix
Undecided
Unassigned
systemd (Ubuntu)
Invalid
Undecided
Unassigned
Jammy
Fix Committed
Low
Nick Rosbrook
Kinetic
Won't Fix
Undecided
Unassigned

Bug Description

[Impact]

systemd autopkgtests are failing in kinetic, and they are not considered regressions. The failing test is one within networkd-test.py:

======================================================================
ERROR: test_resolved_domain_restricted_dns (__main__.DnsmasqClientTest)
resolved: domain-restricted DNS servers
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.wTrLDQ/build.Oa5/src/test/networkd-test.py", line 680, in test_resolved_domain_restricted_dns
    out = subprocess.check_output(['resolvectl', 'query', 'math.lab'])
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['resolvectl', 'query', 'math.lab']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 35 tests in 161.770s

FAILED (errors=1, skipped=2)

The failures started without any changes to the systemd package, i.e. it was caused by another package upload. In particular, the failures started around 2022-11-17 with linux 5.19.0-24.25. The tests pass fine with linux 5.19.0-21.21, i.e. the version in kinetic-release.

This can be demonstrated locally with autopkgtest:

$ autopkgtest --shell-fail systemd --test-name networkd-test.py --pin-packages=kinetic=src:linux --setup-commands 'apt update' -- qemu /home/nr/ubuntu/autopkgtest-testbeds/autopkgtest-kinetic-amd64.img

Note in the above command I did not pass -U, so no packages in the test bed are upgraded. But, with:

$ autopkgtest --shell-fail systemd --test-name networkd-test.py -U -- qemu /home/nr/ubuntu/autopkgtest-testbeds/autopkgtest-kinetic-amd64.img

the kernel is upgraded, and the test fails.

This can also be done manually by grabbing the systemd source on a kinetic machine (e.g. pull-lp-source systemd kinetic), and running:

$ sudo python3 test/networkd-test.py DnsmasqClientTest.test_resolved_domain_restricted_dns

The attached journal.txt shows systemd-resolved logs from when the test is run. The failure is caused by the EHOSTUNREACH when trying to resolve math.lab.
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Mar 9 11:56 seq
 crw-rw---- 1 root audio 116, 33 Mar 9 11:56 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.23.1-0ubuntu3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.10
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:

Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 bochs-drmdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.19.0-35-generic root=UUID=7f0aa1bf-59bd-4d77-b092-7244f86e5566 ro console=tty0 console=hvc0 console=ttyS0
ProcVersionSignature: Ubuntu 5.19.0-35.36-generic 5.19.17
RelatedPackageVersions:
 linux-restricted-modules-5.19.0-35-generic N/A
 linux-backports-modules-5.19.0-35-generic N/A
 linux-firmware 20220923.gitf09bebf3-0ubuntu1.4
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: kinetic
Uname: Linux 5.19.0-35-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.release: 0.0
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.16.0-debian-1.16.0-4
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-kinetic
dmi.modalias: dmi:bvnSeaBIOS:bvr1.16.0-debian-1.16.0-4:bd04/01/2014:br0.0:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-kinetic:cvnQEMU:ct1:cvrpc-i440fx-kinetic:sku:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-kinetic
dmi.sys.vendor: QEMU

[Test Plan]

Examine relevant autopkgtest logs, and verify that networkd-test.py passes, and that test_resolved_domain_restricted_dns is skipped.

[Where problems could occur]

We are just skipping this specific flaky test.

This test has been flaky across several releases without a sure root cause identified. For now, we just want to skip the test (as has been done before)
to try and get a green baseline, and prevent real regressions going unnoticed.

Revision history for this message
Nick Rosbrook (enr0n) wrote :
Changed in systemd (Ubuntu):
status: New → Invalid
summary: - systemd: test-networkd.py fails with newer kinetic kernels
+ systemd: networkd-test.py fails with newer kinetic kernels
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2009859

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Changed in linux (Ubuntu Kinetic):
status: New → Incomplete
Revision history for this message
Nick Rosbrook (enr0n) wrote : CurrentDmesg.txt

apport information

tags: added: apport-collected kinetic
description: updated
Revision history for this message
Nick Rosbrook (enr0n) wrote : Lspci.txt

apport information

Revision history for this message
Nick Rosbrook (enr0n) wrote : Lspci-vt.txt

apport information

Revision history for this message
Nick Rosbrook (enr0n) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Nick Rosbrook (enr0n) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Nick Rosbrook (enr0n) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Nick Rosbrook (enr0n) wrote : ProcModules.txt

apport information

Revision history for this message
Nick Rosbrook (enr0n) wrote : UdevDb.txt

apport information

Revision history for this message
Nick Rosbrook (enr0n) wrote : WifiSyslog.txt

apport information

Revision history for this message
Nick Rosbrook (enr0n) wrote : acpidump.txt

apport information

Changed in linux (Ubuntu Kinetic):
status: Incomplete → Confirmed
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Andrei Gherzan (agherzan) wrote :
tags: added: sru-20230515
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu Kinetic):
status: New → Confirmed
Revision history for this message
Brian Murray (brian-murray) wrote :

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.

Changed in systemd (Ubuntu Kinetic):
status: Confirmed → Won't Fix
Changed in linux (Ubuntu Kinetic):
status: Confirmed → Won't Fix
Revision history for this message
Nick Rosbrook (enr0n) wrote :

Adding a jammy task for systemd because I see the error there now. At the very least I want to just skip the test in jammy to get a green baseline again.

Changed in systemd (Ubuntu Jammy):
status: New → Confirmed
importance: Undecided → Low
tags: added: systemd-sru-next
Nick Rosbrook (enr0n)
description: updated
Nick Rosbrook (enr0n)
Changed in systemd (Ubuntu Jammy):
assignee: nobody → Nick Rosbrook (enr0n)
status: Confirmed → In Progress
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote : Please test proposed package

Hello Nick, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/249.11-0ubuntu3.14)

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.14) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

docker.io-app/unknown (ppc64el)
dovecot/1:2.3.16+dfsg1-3ubuntu2.4 (s390x)
dpdk/unknown (ppc64el)
gpsd/unknown (ppc64el)
gvfs/unknown (ppc64el)
haproxy/2.4.24-0ubuntu0.22.04.1 (ppc64el)
indicator-session/17.3.20+21.10.20210613.1-0ubuntu1 (ppc64el)
init-system-helpers/unknown (ppc64el)
knot-resolver/unknown (ppc64el)
libqb/unknown (ppc64el)
linux-azure-6.8/6.8.0-1019.22~22.04.1 (arm64)
linux-gcp-6.8/6.8.0-1020.22~22.04.1 (arm64)
linux-gke/5.15.0-1070.76 (arm64)
linux-hwe-6.5/6.5.0-45.45~22.04.1 (arm64)
linux-hwe-6.8/6.8.0-50.51~22.04.1 (amd64)
linux-ibm-6.8/6.8.0-1017.17~22.04.1 (amd64)
linux-lowlatency-hwe-6.5/6.5.0-45.45.1~22.04.1 (amd64, arm64)
linux-lowlatency-hwe-6.8/6.8.0-50.51.1~22.04.1 (arm64)
linux-nvidia-6.5/6.5.0-1024.25 (arm64)
linux-nvidia-6.8/6.8.0-1020.22~22.04.1 (arm64)
linux-nvidia-tegra/5.15.0-1030.30 (arm64)
linux-oracle-6.5/6.5.0-1027.27~22.04.1 (amd64)
zfs-linux/2.1.5-1ubuntu6~22.04.4 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Chris Halse Rogers (raof) wrote : Proposed package upload rejected

An upload of systemd to jammy-proposed has been rejected from the upload queue for the following reason: "Prerm is missing the #DEBHELPER# substitution that would include any other scripts generated by debhelper. This doesn't seem intentional. Also, should this wait for the existing systemd in -proposed to be verot entangle this bug with the others? Feel free to reach out with any questions/answers :)".

Revision history for this message
Andreas Hasenack (ahasenack) wrote : Please test proposed package

Hello Nick, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/249.11-0ubuntu3.15)

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.15) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

apt/unknown (armhf)
bolt/unknown (armhf)
casync/unknown (armhf)
debspawn/0.5.2-1 (ppc64el)
flatpak/1.12.7-1ubuntu0.1 (amd64)
fluidsynth/unknown (armhf)
freedombox/unknown (armhf)
freeipa/unknown (armhf)
freeradius/unknown (armhf)
fwupd/unknown (armhf)
gamemode/unknown (armhf)
gdm3/unknown (armhf)
golang-github-coreos-go-systemd/unknown (armhf)
gpsd/unknown (armhf)
gvfs/unknown (armhf)
haproxy/unknown (armhf)
hwloc/unknown (armhf)
icinga2/unknown (armhf)
libsoup3/3.0.7-0ubuntu1 (amd64)
linux-gcp-6.8/6.8.0-1024.26~22.04.1 (amd64)
linux-gke/5.15.0-1077.83 (amd64)
linux-hwe-6.5/6.5.0-45.45~22.04.1 (amd64)
linux-hwe-6.5/unknown (armhf)
linux-hwe-6.8/unknown (armhf)
linux-lowlatency/5.15.0-135.146 (amd64)
linux-lowlatency-hwe-6.5/6.5.0-45.45.1~22.04.1 (amd64)
mir/unknown (armhf)
mkosi/unknown (armhf)
monitoring-plugins-systemd/unknown (armhf)
mpd/unknown (armhf)
multipath-tools/unknown (armhf)
munin/unknown (armhf)
mutter/unknown (armhf)
nagios-tang/unknown (armhf)
openssh/1:8.9p1-3ubuntu0.11 (amd64)
policykit-1/unknown (armhf)
polkit-qt-1/unknown (armhf)
postgresql-14/unknown (armhf)
prometheus-postgres-exporter/unknown (armhf)
prometheus-squid-exporter/unknown (armhf)
pulseaudio/unknown (armhf)
python-dbusmock/unknown (armhf)
python-systemd/unknown (armhf)
python-uinput/unknown (armhf)
pyudev/unknown (armhf)
qlcplus/unknown (armhf)
swupdate/2021.11-1 (amd64)
systemd-bootchart/unknown (armhf)
tgt/1:1.0.80-1ubuntu2 (amd64)
tpm2-abrmd/unknown (armhf)
tpm2-pkcs11/unknown (armhf)
ubuntu-drivers-common/unknown (armhf)
udisks2/unknown (armhf)
util-linux/unknown (armhf)
vlc/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Nick Rosbrook (enr0n) wrote :

This test passes now: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/s/systemd/20250228_041633_efb81@/log.gz

In particular, we see:

467s autopkgtest [03:25:44]: test networkd-test.py: [-----------------------

...SNIP...

600s test_resolved_domain_restricted_dns (__main__.DnsmasqClientTest)
600s resolved: domain-restricted DNS servers ... skipped 'LP: #2009859'
600s test_resolved_etc_hosts (__main__.DnsmasqClientTest)
600s resolved queries to /etc/hosts ... ok
600s test_route_only_dns (__main__.DnsmasqClientTest) ... ok
600s test_route_only_dns_all_domains (__main__.DnsmasqClientTest) ... ok
600s test_transient_hostname (__main__.DnsmasqClientTest)
600s networkd sets transient hostname from DHCP ... ok
600s test_transient_hostname_with_static (__main__.DnsmasqClientTest)
600s transient hostname is not applied if static hostname exists ... ok
600s test_basic_matching (__main__.MatchClientTest)
600s Verify the Name= line works throughout this class. ... ok
600s test_inverted_matching (__main__.MatchClientTest)
600s Verify that a '!'-prefixed value inverts the match. ... ok
600s test_coldplug_dhcp_ip4_only (__main__.NetworkdClientTest) ... ok
600s test_coldplug_dhcp_ip4_only_no_ra (__main__.NetworkdClientTest) ... ok
600s test_coldplug_dhcp_ip6 (__main__.NetworkdClientTest) ... skipped 'networkd does not have DHCPv6 server support'
600s test_coldplug_dhcp_yes_ip4 (__main__.NetworkdClientTest) ... ok
600s test_coldplug_dhcp_yes_ip4_no_ra (__main__.NetworkdClientTest) ... ok
600s test_dhcp_timezone (__main__.NetworkdClientTest)
600s networkd sets time zone from DHCP ... ok
600s test_dropin (__main__.NetworkdClientTest) ... ok
600s test_hotplug_dhcp_ip4 (__main__.NetworkdClientTest) ... ok
600s test_hotplug_dhcp_ip6 (__main__.NetworkdClientTest) ... skipped 'networkd does not have DHCPv6 server support'
600s test_route_only_dns (__main__.NetworkdClientTest) ... ok
600s test_route_only_dns_all_domains (__main__.NetworkdClientTest) ... ok
600s test_search_domains (__main__.NetworkdClientTest) ... ok
600s test_catchall_config (__main__.UnmanagedClientTest)
600s Verify link states with a catch-all config, hot-plug. ... ok
600s test_catchall_config_coldplug (__main__.UnmanagedClientTest)
600s Verify link states with a catch-all config, cold-plug. ... ok
600s test_unmanaged_setting (__main__.UnmanagedClientTest)
600s Verify link states with Unmanaged= settings, hot-plug. ... ok
600s test_unmanaged_setting_coldplug (__main__.UnmanagedClientTest)
600s Verify link states with Unmanaged= settings, cold-plug. ... ok
600s
600s ----------------------------------------------------------------------
600s Ran 35 tests in 132.384s
600s
600s OK (skipped=3)
600s autopkgtest [03:27:57]: test networkd-test.py: -----------------------]
600s networkd-test.py PASS

tags: added: verification-done-jammy
removed: verification-needed-jammy
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.