DHCPv4 (IAID+DUID) networking broken in LXC containers

Bug #1950794 reported by Lukas Märdian
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
lxd (Ubuntu)
New
Undecided
Unassigned
systemd (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

DHCPv4 networking does not work in the default IAID+DUID (ClientIdentifier=duid) mode in LXC containers, using systemd-networkd v249.5-2ubuntu1. Static configuration and DHCPv6 work without problem.

Reproducer:
$ lxc launch ubuntu-daily:jammy jj
$ lxc exec jj bash
# add-apt-repository ppa:ci-train-ppa-service/4704
# apt install systemd # install systemd 249.5-2ubuntu1
# cat /etc/systemd/network/00-test.network
[Match]
Name=eth0

[Network]
DHCP=ipv4
# systemctl restart systemd-networkd.service
# networkctl
IDX LINK TYPE OPERATIONAL SETUP
[...]
611 eth0 ether carrier failed

A workaround is to avoid IAID+DUID mode via:
[DHCPv4]
#ClientIdentifier=mac
ClientIdentifier=duid-only

Interesting logs:
Nov 12 14:10:48 jj systemd-networkd[174]: eth0: Requested to activate link
Nov 12 14:10:48 jj systemd-networkd[174]: eth0: DHCPv4 client: Failed to set IAID: Device or resource busy
Nov 12 14:10:48 jj systemd-networkd[174]: eth0: DHCP4 CLIENT: Failed to set IAID+DUID: Device or resource busy
Nov 12 14:10:48 jj systemd-networkd[174]: Failed to check link is initialized: Device or resource busy
Nov 12 14:10:48 jj systemd-networkd[174]: eth0: Failed

CVE References

Revision history for this message
Lukas Märdian (slyon) wrote :
summary: - DHCPv4 networking broken in LXC containers (IAID+DUID /
- ClientIdentifier=duid)
+ DHCPv4 (IAID+DUID) networking broken in LXC containers
Revision history for this message
Lukas Märdian (slyon) wrote :

Turns out dropping "debian/patches/units-Don-t-start-systemd-udev-trigger.service-in-a-conta.patch" (that we want to drop anyway) makes a difference here, i.e. removing the "ConditionVirtualization=!container" line from /usr/lib/systemd/service/systemd-udev-trigger.service.

# apt install systemd udev # upgrade systemd & udev to v249
# systemctl restart systemd-udev-trigger # run the 'udevadm trigger' commands
# systemctl restart systemd-networkd # restart networkd to re-run the DHCPv4 client.

That is even though the systemd-udev-trigger.service fails to execute successfully:
# systemctl status systemd-udev-trigger.service
● systemd-udev-trigger.service - Coldplug All udev Devices
     Loaded: loaded (/lib/systemd/system/systemd-udev-trigger.service; static)
     Active: active (exited) since Wed 2021-11-17 09:39:01 UTC; 37s ago
       Docs: man:udev(7)
             man:systemd-udevd.service(8)
    Process: 74 ExecStart=udevadm trigger --type=subsystems --action=add (code=exited, status=1/FAILURE)
    Process: 101 ExecStart=udevadm trigger --type=devices --action=add (code=exited, status=1/FAILURE)
   Main PID: 101 (code=exited, status=1/FAILURE)
        CPU: 160ms

Nov 17 09:39:01 jj2 udevadm[101]: nvme-delete-wq: Failed to write 'add' to '/sys/devices/virtual/workqueue/nvme-delete-wq/uevent': Permission denied
Nov 17 09:39:01 jj2 udevadm[101]: nvme-reset-wq: Failed to write 'add' to '/sys/devices/virtual/workqueue/nvme-reset-wq/uevent': Permission denied
Nov 17 09:39:01 jj2 udevadm[101]: nvme-wq: Failed to write 'add' to '/sys/devices/virtual/workqueue/nvme-wq/uevent': Permission denied
Nov 17 09:39:01 jj2 udevadm[101]: raid5wq: Failed to write 'add' to '/sys/devices/virtual/workqueue/raid5wq/uevent': Permission denied
Nov 17 09:39:01 jj2 udevadm[101]: scsi_tmf_0: Failed to write 'add' to '/sys/devices/virtual/workqueue/scsi_tmf_0/uevent': Permission denied
Nov 17 09:39:01 jj2 udevadm[101]: writeback: Failed to write 'add' to '/sys/devices/virtual/workqueue/writeback/uevent': Permission denied
Nov 17 09:39:01 jj2 udevadm[101]: dm-0: Failed to write 'add' to '/sys/devices/virtual/block/dm-0/uevent': Permission denied
Nov 17 09:39:01 jj2 udevadm[101]: dm-1: Failed to write 'add' to '/sys/devices/virtual/block/dm-1/uevent': Permission denied
Nov 17 09:39:01 jj2 udevadm[101]: dm-2: Failed to write 'add' to '/sys/devices/virtual/block/dm-2/uevent': Permission denied
Nov 17 09:39:01 jj2 udevadm[101]: dm-3: Failed to write 'add' to '/sys/devices/virtual/block/dm-3/uevent': Permission denied

Revision history for this message
Lukas Märdian (slyon) wrote :

Reverting this upstream commit seems to fix the problem: https://github.com/systemd/systemd/commit/0299deab53d2a087727a5d04c1500c322c48b63e

Lukas Märdian (slyon)
Changed in systemd (Ubuntu):
status: New → Fix Committed
Revision history for this message
Dan Streetman (ddstreet) wrote :

> Reverting this upstream commit seems to fix the problem:
> https://github.com/systemd/systemd/commit/0299deab53d2a087727a5d04c1500c322c48b63e

lxd and systemd have what I can only describe euphemistically as a horrible relationship. Instead of carrying another patch on systemd to get it working in lxd, could you try to work this out correctly, either by convincing upstream systemd to change or convincing lxd to change?

Long term, it does Ubuntu no favors by hacking up systemd because lxd doesn't conform to the systemd container interface.
https://systemd.io/CONTAINER_INTERFACE/

Revision history for this message
Lukas Märdian (slyon) wrote :

Yes.. LXD's system containers and systemd/udev have a special kind of relationship and we should strive to resolve that situation properly long-term. I already talked to stgraber about that, but I guess we should lay out a proper path forward as continuously patching more and more components of systemd is not a sustainable approach.

I'll leave that new revert-patch in for now, tho, as we do not have another short-term solution.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (13.8 KiB)

This bug was fixed in the package systemd - 249.5-2ubuntu1

---------------
systemd (249.5-2ubuntu1) jammy; urgency=medium

  * Merge to Ubuntu from Debian unstable
    - Dropped changes (applied upstream):
      + UBUNTU-units-disable-journald-watchdog.patch
      + test-Allow-running-only-a-subset-of-integration-tests-by-.patch
      + hwdb-Add-ProBook-to-use-micmute-hotkey.patch
      + d/p/lp1931578/*.patch (ActivationPolicy=)
      + lp1932352-hwdb-Add-mic-mute-key-mapping-for-HP-Elite-Dragonfly.patch
      + d/p/lp1858210/*.patch (tzdata.zi)
      + lp1914740-network-enable-DHCP-broadcast-flag-if-required-by-in.patch
      + lp1934981-correct-suspend-then-sleep-string.patch
      + CVE-2020-13529.patch
      + CVE-2021-33910.patch
      + Merge-pull-request-20199-from-ddstreet-unit_cgroup_catchu.patch
      + udev-fix-slot-based-network-names-on-s390.patch
      + udev-add-missing-initialization-to-fix-freeing-invalid-ad.patch
      + udev-allow-onboard-index-up-to-65535.patch
      + lp1940635.patch (networkd failed to acquire a DHCP6 lease, v249.5)
  * debian/gbp.conf: Update debian-branch to ubuntu-jammy
    File: debian/gbp.conf
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f35b4d6c0653c3125b2e0e5d962f84543733c3c0
  * Refresh Ubuntu-UseDomains-by-default.patch
    File: debian/patches/debian/Ubuntu-UseDomains-by-default.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6e3b2ffb1c2c2bf2223f91b350c1fd06b8903346
  * Refresh UBUNTU-test-test-functions-launch-qemu-with-vga-none.patch
    File: debian/patches/debian/UBUNTU-test-test-functions-launch-qemu-with-vga-none.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7142fb9381ab98d750a7c2473087ea7264be860d
  * Refresh UBUNTU-Revert-cgroup-Continue-unit-reset-if-cgroup-is-busy.patch
    File: debian/patches/debian/UBUNTU-Revert-cgroup-Continue-unit-reset-if-cgroup-is-busy.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3de0f444bfc0cf3b601c4a1ad8939d2cd69cdad6
  * Refresh UBUNTU-resolved-default-no-negative-caching.patch
    File: debian/patches/debian/UBUNTU-resolved-default-no-negative-caching.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d7b25651ea7ceae2ebfeafb84d9d65fb8284367a
  * Refresh Revert-network-if-sys-is-rw-then-udev-should-be-around.patch
    File: debian/patches/Revert-network-if-sys-is-rw-then-udev-should-be-around.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=61b004d7710b305d9996b3ad9bb47dc372fcbbac
  * Refresh UBUNTU-src-test-testmount-util.c-Skip-parts-of-test-mount-util-in-LXC.patch
    File: debian/patches/debian/UBUNTU-src-test-testmount-util.c-Skip-parts-of-test-mount-util-in-LXC.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=5fe343a0fc89771820f997c2d6621ac517d9070f
  * Refresh Merge-pull-request-20705-from-yuwata-test-oomd-util.patch
    File: debian/patches/Merge-pull-request-20705-from-yuwata-test-oomd-util.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e88125...

Changed in systemd (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.