[Impact]
* Installation of latest systemd update in -security hangs with current versions of unattended-upgrades in supported releases. The u-u-side fix is tracked in LP: #1778219.
[Regression Potential]
* The daemons, shipped in deb:systemd, are not attempted to be restarted because despite package installation the system is in the middle of shutting down. This means that currently running daemons may be helding up open files on the filesystem, however all process are being stopped and killed as part of shutdown. Hence the worst possible regression from this, is an unclean shutdown, but even that shouldn't happen with this update.
[Test Case]
Reproduction:
rbalint@yogi:~$ lxc launch ubuntu:18.04 uu-systemd-onshutdown
Creating uu-systemd-onshutdown
Starting uu-systemd-onshutdown
rbalint@yogi:~$ lxc shell uu-systemd-onshutdown
mesg: ttyname failed: No such device
root@uu-systemd-onshutdown:~# apt update -qq
23 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@uu-systemd-onshutdown:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
root@uu-systemd-onshutdown:~# apt list --upgradable
Listing... Done
apport/bionic-updates 2.20.9-0ubuntu7.5 all [upgradable from: 2.20.9-0ubuntu7.4]
gettext-base/bionic-updates,bionic-security 0.19.8.1-6ubuntu0.1 amd64 [upgradable from: 0.19.8.1-6]
kmod/bionic-updates 24-1ubuntu3.1 amd64 [upgradable from: 24-1ubuntu3]
libglib2.0-0/bionic-updates 2.56.3-0ubuntu0.18.04.1 amd64 [upgradable from: 2.56.2-0ubuntu0.18.04.2]
libglib2.0-data/bionic-updates 2.56.3-0ubuntu0.18.04.1 all [upgradable from: 2.56.2-0ubuntu0.18.04.2]
libkmod2/bionic-updates 24-1ubuntu3.1 amd64 [upgradable from: 24-1ubuntu3]
libmspack0/bionic-updates,bionic-security 0.6-3ubuntu0.2 amd64 [upgradable from: 0.6-3ubuntu0.1]
libnss-systemd/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
libpam-systemd/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
libsystemd0/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
libudev1/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
lxd/bionic-updates 3.0.2-0ubuntu1~18.04.1 amd64 [upgradable from: 3.0.1-0ubuntu1~18.04.1]
lxd-client/bionic-updates 3.0.2-0ubuntu1~18.04.1 amd64 [upgradable from: 3.0.1-0ubuntu1~18.04.1]
openssh-client/bionic-updates,bionic-security 1:7.6p1-4ubuntu0.1 amd64 [upgradable from: 1:7.6p1-4]
openssh-server/bionic-updates,bionic-security 1:7.6p1-4ubuntu0.1 amd64 [upgradable from: 1:7.6p1-4]
openssh-sftp-server/bionic-updates,bionic-security 1:7.6p1-4ubuntu0.1 amd64 [upgradable from: 1:7.6p1-4]
python3-apport/bionic-updates 2.20.9-0ubuntu7.5 all [upgradable from: 2.20.9-0ubuntu7.4]
python3-distupgrade/bionic-updates 1:18.04.28 all [upgradable from: 1:18.04.27]
python3-problem-report/bionic-updates 2.20.9-0ubuntu7.5 all [upgradable from: 2.20.9-0ubuntu7.4]
systemd/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
systemd-sysv/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
ubuntu-release-upgrader-core/bionic-updates 1:18.04.28 all [upgradable from: 1:18.04.27]
udev/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
root@uu-systemd-onshutdown:~# reboot
Session terminated, terminating shell...Terminated
root@uu-systemd-
rbalint@yogi:~$
rbalint@yogi:~$ lxc shell uu-systemd-onshutdown
mesg: ttyname failed: No such device
root@uu-systemd-onshutdown:~# tail /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
Preparing to unpack .../libsystemd0_237-3ubuntu10.6_amd64.deb ...
Unpacking libsystemd0:amd64 (237-3ubuntu10.6) over (237-3ubuntu10.3) ...
Setting up libsystemd0:amd64 (237-3ubuntu10.6) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Setting up systemd (237-3ubuntu10.6) ...
Failed to try-restart systemd-networkd.service: Transaction is destructive.
See system logs and 'systemctl status systemd-networkd.service' for details.
Failed to try-restart systemd-resolved.service: Transaction is destructive.
See system logs and 'systemctl status systemd-resolved.service' for details.
root@uu-systemd-onshutdown:~# ps -ef | cat
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 15:34 ? 00:00:00 /lib/systemd/systemd --system --deserialize 22
root 53 1 0 15:34 ? 00:00:00 /lib/systemd/systemd-journald
systemd+ 153 1 0 15:34 ? 00:00:00 /lib/systemd/systemd-networkd
systemd+ 154 1 0 15:34 ? 00:00:00 /lib/systemd/systemd-resolved
message+ 194 1 0 15:34 ? 00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 572 1 0 15:35 ? 00:00:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown
root 586 572 0 15:35 ? 00:00:08 /usr/bin/python3 /usr/bin/unattended-upgrade
root 829 0 0 15:35 ? 00:00:00 su -l
root 830 829 0 15:35 ? 00:00:00 -su
root 1244 1 0 15:35 ? 00:00:00 /lib/systemd/systemd-udevd
root 1918 1 0 15:35 ? 00:00:00 /usr/sbin/sshd -D
root 1952 586 0 15:35 ? 00:00:00 /usr/bin/python3 /usr/bin/unattended-upgrade
root 2111 1952 0 15:35 pts/0 00:00:00 /usr/bin/dpkg --status-fd 10 --configure --pending
root 2125 2111 0 15:35 pts/0 00:00:00 /bin/sh /var/lib/dpkg/info/systemd.postinst configure 237-3ubuntu10.3
root 2165 2125 0 15:35 pts/0 00:00:00 systemctl try-restart systemd-journald.service
root 2170 0 0 15:36 ? 00:00:00 su -l
root 2171 2170 0 15:36 ? 00:00:00 -su
root 2197 2171 0 15:36 ? 00:00:00 ps -ef
root 2198 2171 0 15:36 ? 00:00:00 cat
root@uu-systemd-onshutdown:~# systemctl list-jobs
JOB UNIT TYPE STATE
323 reboot.target start waiting
405 basic.target stop waiting
359 sysinit.target stop waiting
433 cloud-init.service stop waiting
422 user.slice stop waiting
388 snapd.socket stop waiting
428 swap.target stop waiting
332 shutdown.target start waiting
412 slices.target stop waiting
415 systemd-networkd.service stop waiting
338 syslog.socket stop waiting
399 acpid.socket stop waiting
377 uuidd.socket stop waiting
416 systemd-networkd-wait-online.service stop waiting
373 acpid.path stop waiting
432 local-fs.target stop waiting
437 cryptsetup.target stop waiting
341 lxd.socket stop waiting
491 systemd-journald.service restart waiting
340 sockets.target stop waiting
330 var-lib-lxcfs.mount stop waiting
430 cloud-init-local.service stop waiting
453 paths.target stop waiting
402 iscsid.socket stop waiting
439 local-fs-pre.target stop waiting
392 dbus.socket stop waiting
440 systemd-resolved.service stop waiting
353 systemd-ask-password-console.path stop waiting
324 systemd-reboot.service start waiting
498 systemd-journal-flush.service restart waiting
325 final.target start waiting
414 systemd-sysctl.service stop waiting
331 run-user-0.mount stop waiting
435 network.target stop waiting
457 systemd-tmpfiles-setup.service stop waiting
458 systemd-ask-password-wall.path stop waiting
329 umount.target start waiting
369 unattended-upgrades.service stop running
434 network-pre.target stop waiting
436 systemd-tmpfiles-setup-dev.service stop waiting
456 systemd-update-utmp.service stop waiting
389 apport-forward.socket stop waiting
497 systemd-journald-audit.socket start waiting
43 jobs listed.
root@uu-systemd-onshutdown:~#
that's really really really bad!
_systemctl try-restart systemd-
_must_ not hang, it should either succeed or fail. that is the whole point of try-restart....
I wonder if `--no-block` would help here, but that's also a bandaid. Imho systemctl/systemd itself shouldn't be able to hang systems like that and like should reject this job submission if it will be impossible to complete.