Shutdown when triggering daemon-reload eary in boot

Bug #2037281 reported by Valentin David
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
New
High
Nick Rosbrook
Jammy
New
High
Nick Rosbrook

Bug Description

In Ubuntu Core 20, and Ubuntu Core 22, we are encountering an issue where if a service, started earlier than devices are processed by udev, does `systemctl daemon-reload`, the system shuts down. This is due to devices for mounted filesystem temporarily taken dead, which pulls most units down.

This was fixed by upstream in https://github.com/systemd/systemd/pull/23218.

But this was not backported to the versions used by Ubuntu packages for focal and jammy. The needed commit from that PR is the one with message `core/device: ignore DEVICE_FOUND_UDEV bit on switching root`.

This patch applies to 245.4-4ubuntu3.22 (focal) without rebasing needed. And I suppose it does also for jammy.

I have manually tested the fix with Ubuntu Core 20, and this fixes our issue.

We would like this patch to be backported to focal-updates and jammy-updates.

Thank you in advance.

[ Impact ]

If a user adds a service that calls `systemctl daemon-reload`, and if this service is started before systemd-udevd. And if the initrd is systemd (the case of Ubuntu Core), then most service will be stopped or cancel, and the machine will mostly shutdown everything and hang.

The fix has been backported down to 250 upstream. It is already on kinetic and later.

The fix only affects systems where systemd is used in initrd.

[ Test Plan ]

On Ubuntu Core 20 (with Core 22 kernel) or on Ubuntu Core 22. Or on any system that uses systemd in initrd.

Add a systemd service that calls `systemctl daemon-reload`.
The service should have `DefaultDependencies=no` in order to start as soon as possible and be enabled.

Restart the machine.

If fix is not applied, after the service is started, most of units with be shutdown, and the system will be unusable.

[ Where problems could occur ]

This should affect systems with systemd in initrd.

There are risks on systems that have an udev rule in initrd not present in the main system.

There are risks on systems that use db_persist in initrd where the device can potentially get dead state. Though this does not seem to happen on Ubuntu Core 22, even though we use db_persist for dev mapper devices. Regression is upstream bug #23429. Commits named "core/device: device_coldplug(): don't set DEVICE_DEAD" and "core/device: do not downgrade device state if it is already enumerated" could be applied as well.

Nick Rosbrook (enr0n)
Changed in systemd (Ubuntu Focal):
importance: Undecided → High
Changed in systemd (Ubuntu Jammy):
importance: Undecided → High
Changed in systemd (Ubuntu):
status: New → Fix Released
tags: added: systemd-sru-next
Nick Rosbrook (enr0n)
tags: added: foundations-todo
Changed in systemd (Ubuntu Focal):
assignee: nobody → Nick Rosbrook (enr0n)
Changed in systemd (Ubuntu Jammy):
assignee: nobody → Nick Rosbrook (enr0n)
Michael Vogt (mvo)
description: updated
description: updated
Changed in systemd (Ubuntu):
importance: Undecided → Critical
importance: Critical → High
importance: High → Undecided
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.