"telinit u" under upstart (upstart's Restart command) with systemd-sysv installed runs systemd

Bug #1430479 reported by Timo Aaltonen
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Fix Released
Critical
Martin Pitt
upstart (Debian)
Fix Released
Unknown
upstart (Ubuntu)
Triaged
Medium
Unassigned

Bug Description

I've today had two machines kill my X session during dist-upgrade, and it happened during libc6 postinst. Dmesg shows that systemd got somehow started during the earlier phase of the upgrade. X logs show a failure with "(EE) Cannot establish any listening sockets" etc..

Adam can fill in the more detailed analysis of what's happening there.. but I think it should be fairly easy to reproduce by installing vivid from an earlier image and dist-upgrading to current, if needed.

Reproducer:
 - Start with a standard utopic VM (e. g. adt-buildvm-ubuntu-cloud -r utopic)
 - Switch apt sources to vivid
 - sudo apt install systemd-sysv
 - sudo telinit u # systemd command, forwards to upstart's /lib/sysvinit/telinit
 - Or directly: sudo /lib/sysvinit/telinit u

Revision history for this message
Adam Conrad (adconrad) wrote :

Fairly sure, without actually reproducing myself, that "telinit u" is starting a fresh systemd and taking over PID1, instead of either calling back to upstart (the current PID1) or doing nothing, either of which would be preferable.

Changed in systemd (Ubuntu):
importance: Undecided → Critical
Martin Pitt (pitti)
Changed in systemd (Ubuntu):
status: New → In Progress
assignee: nobody → Martin Pitt (pitti)
Revision history for this message
Martin Pitt (pitti) wrote :

Debugging notes:

 - Running telinit u on a current vivid system under upstart, but with systemd-sysv installed (i. e. select upstart from grub, using init=/sbin/upstart) is returning immediately. strace shows that it's connecting to the upstart socket. dmesg shows that upstart-{udev,file}-bridge restarts and otherwise not much is going on.

 - Running systemctl daemon-reexec on that system fails immediately with "Failed to get D-Bus connection: Operation not permitted" as the systemd D-Bus socket isn't available.

Revision history for this message
Martin Pitt (pitti) wrote :

 - Reconfiguring glibc on that system does run telinit u, but doesn't hang.

 - I tested/fixed upgrades from trusty and utopic in schroot a few weeks ago, but libc doesn't do the telinit u thing in chroots, so I missed that.

 - I can reproduce that upgrading utopic to vivid in a VM ends up with running systemd. That's clearly not intended.

Martin Pitt (pitti)
description: updated
description: updated
description: updated
summary: - switching init systems together with a libc upgrade kills X and disrupts
- the upgrade
+ "telinit u" under upstart with systemd-sysv installed runs systemd
Revision history for this message
Martin Pitt (pitti) wrote : Re: "telinit u" under upstart with systemd-sysv installed runs systemd

Stracing "telinit u" shows some interesting actions:

lstat("/run/systemd/system/", 0x7fff0ff73700) = -1 ENOENT (No such file or directory)

→ OK, that's the check whether systemd is running, it's expected to fail under upstart

execve("/lib/sysvinit/telinit", ["telinit", "u"], ... ) = 0

→ Hm, that's from upstart-bin, also looks ok

connect(3, {sa_family=AF_LOCAL, sun_path=@"/com/ubuntu/upstart"}, 22) = 0
[...]
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\3\1\0\0\0\0\1\0\0\0P\0\0\0\1\1o\0\23\0\0\0/com/ubuntu/Upstart\0\0\0\0\0\2\1s\0\25\0\0\0com.ubuntu.Upstart0_6\0\0\0\3\1s\0\7\0\0\0Restart\0", 96}, {"", 0}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 96
exit_group(0) = ?
+++ exited with 0 +++

Hm, nothing unusual here.

tags: added: systemd-boot
Revision history for this message
Martin Pitt (pitti) wrote :

An "interesting" variant of that is:

  sudo apt install upstart upstart-bin systemd-sysv
  sudo telinit u

While with utopic's upstart telinit returned immediately and systemd was run, with vivid's upstart the telinit now hangs eternally (but systemd still gets run)

Martin Pitt (pitti)
Changed in upstart (Ubuntu):
importance: Undecided → Critical
description: updated
summary: - "telinit u" under upstart with systemd-sysv installed runs systemd
+ "telinit u" under upstart (upstart's Restart command) with systemd-sysv
+ installed runs systemd
Revision history for this message
Martin Pitt (pitti) wrote :

That's because perform_reexec() just reexec's argv[0], which used to be /sbin/init. But that just changed pointing from upstart to pointing to systemd. So upstart's function isn't actually conceptually wrong, but perhaps we could run /sbin/upstart instead of argv[0] (which is likely /sbin/init)?

Revision history for this message
Martin Pitt (pitti) wrote :

So we either change upstart's perform_reexec() to run /sbin/upstart, or we change systemd's telinit to not forward an "u" request to upstart, and simply ignore the reload. The latter would break the intention of glibc's restart though (but it's probably not a big deal).

Revision history for this message
Adam Conrad (adconrad) wrote :

Fixing upstart to do the right thing is probably good anyway, but since one can't go back in time and fix it, that won't solve the upgrade issue. The only way to solve the upgrade issue is to make systemd not forward the request, I imagine. It could potentially make the first umount on reboot after the upgrade unclean, but I can't think of a better way around this.

Revision history for this message
Martin Pitt (pitti) wrote :

Adam points out that we would need to fix that in trusty's and utopic's upstart as well, so let's just work around that in systemd's telinit.

Changed in upstart (Ubuntu):
importance: Critical → Medium
Revision history for this message
Martin Pitt (pitti) wrote :
Changed in systemd (Ubuntu):
status: In Progress → Fix Committed
Changed in upstart (Ubuntu):
status: New → Triaged
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 219-4ubuntu5

---------------
systemd (219-4ubuntu5) vivid; urgency=medium

  * Merge from experimental branch:
    - /lib/lsb/init-functions.d/40-systemd: Don't ignore systemd unit
      dependencies in "degraded" mode. (LP: #1429734)
    - rules-ubuntu/71-power-switch-proliant.rules: Add support for HP ProLiant
      m400 Server Cartridge soft powerdown on Linux 3.16. (LP: #1428811)
  * systemctl: Don't forward telinit u to upstart. This works around upstart's
    Restart() always reexec'ing /sbin/init on Restart(), even if that changes
    to point to systemd during the upgrade. This avoids running systemd during
    a dist-upgrade. (LP: #1430479)
 -- Martin Pitt <email address hidden> Wed, 11 Mar 2015 11:01:36 +0100

Changed in systemd (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote :

As we have a workaround in systemd now, I untag this to clean up the bug list.

tags: removed: systemd-boot
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

I've verified that the upgrade works now.

Changed in upstart (Debian):
status: Unknown → New
Changed in upstart (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.