update fails on cloud server (invoke-rc.d restart failed) when using an unsupported kernel

Issue was first noted 3 weeks ago in the Linode Community forum:
https://www.linode.com/community/questions/23535/strange-new-error-failed-to-start-device-mapper-multipath-device-controller

Thanks for this bug report. This looks like a regression in caused by a security update uploaded to >= Focal. Status is Confirmed due to plenty of evidence in the Linode forum.

Thanks for flagging this Paride; my guess is this would have happened even for something trivial like a changelog-only update.

The default in Debian, for better or for worse, is to enable all services during installation. The default dh_installinit action appears to be failure if the service can't start.

There's a lot of frustrating things here:

- The Debian default feels like it made sense in 1997. I'm not sure it still does.

- dh_installinit probably shouldn't default to failing an installation if the service doesn't run. Recovering a broken apt is difficult for most people and frustrating for everyone. Recovering a broken service is no big deal, especially if you don't care about it.

- The Ubuntu server seed should not include multipathd: I have no firm numbers but I'd easily wager less than 1% of our ubuntu-server users can use it. It's wasted disk space, wasted memory, wasted startup time.

- The systemd service file went to great lengths to try to keep it running only where it might be useful, and prevent problems exactly like this one, but the systemd service language can't select for the things that actually matter: (a) does the kernel support the necessary interfaces (b) does the running system have hardware that can benefit from multipath.

- Perhaps an unconfigured or default configured daemon shouldn't fall over violently when there's nothing to actually multipath.

Linode users can add multipath=off to their kernel command line to instruct the systemd service to not try.
Linode users can mask multipathd.service.
Linode can add multipath=off to their default kernel command line.
Linode can mask multipathd.service.
Linode can add a policy-rc.d file to disable this service in their images. (I assume. I haven't tested that this ancient thing still works in a systemd world.)

Users stuck with this can probably use:
sudo dpkg --configure -a
sudo apt install -f

to try to unwedge apt after taking one of the above steps to disable the service.

Thanks

Oh yes, the equivs package can help fake up dependencies to placate apt. I'm not suggesting anyone do it but it might be something that someone somewhere would evaluate for themselves and decide if they'd like to do something so vastly confusing to their own systems.

As Seth mentioned, this isn't directly related to the security update. It looks like the multipath-tools packaging doesn't handle upgrades on an unsupported kernel without the required modules.

Thanks for finding the root cause of the issue. I guess it remains to be decided whether we want to consider this a bug in Ubuntu at all.

A few observations:

- invoke-rc.d shouldn't even be part of the equation here. I would consider this a bug in the Ubuntu package (or in the packaging helpers, generally) - this is a systemd system, /etc/init.d/multipath-tools should be shadowed by /lib/systemd/system/multipath-tools.service which is a symlink to multipathd.service, so calling invoke-rc.d on an Ubuntu system at all is wrong because in the *best case* it's duplicating the systemd unit handling already present via dh_systemd_start.
- maintainer scripts *should* error out if a service that was running fails to restart after an upgrade, because all the other options for error handling are worse. However, if a system was booted just fine and a given service was not running before upgrade, failing to start it shouldn't /necessarily/ result in the maintainer script failing. This is *probably* something we can have better semantics for with systemd than we did with init scripts, but I don't know that such a thing is implemented currently.
- multipath-tools should be installed as part of the default Ubuntu Server system (via the ubuntu-server metapackage), for the same reason that cryptsetup and lvm2 and mdadm and sundry other tools are installed by default even on systems that aren't using them - because hardware can change out from under an install, and you shouldn't need to install extra software when that happens.
- however, multipathd should *not* be running by default on systems where it's not needed, a pet peeve of mine (and many others). Memory usage on a typical (kinetic, in this case) VM:

1649760 root 20 0 1549644 112808 19324 S 0.0 2.8 58:55.08 lxd
 455051 root 20 0 844712 40964 26352 S 3.6 1.0 3949:52 jujud
3295101 1000000 20 0 734588 30796 8168 S 0.0 0.8 0:42.32 snapd
3287494 root 20 0 734588 30304 8792 S 0.0 0.8 1:03.02 snapd
1080861 root rt 0 290116 25764 7464 S 0.0 0.6 12:43.34 multipa+

consuming memory, consuming CPU time, doing nothing useful.

Ideally multipathd would be installed, but would only start if it was detected at runtime that it was needed. And then, as a side effect, it wouldn't be failing to restart on upgrade because it legitimately shouldn't be running.

It is non-trivial to get to that ideal state from where the package is today, but that *should* be the goal.

I mentioned invoke-rc.d due to the maintainer scripts calling it. I think when the invoke-rc.d fails, we hit the (pointless?) || exit 1, and then apt is very sad:

sarnold@wopr:/tmp/multipath $ ar x /srv/mirror/ubuntu/pool/main/m/multipath-tools/multipath-tools_0.8.8-1ubuntu2_amd64.deb
sarnold@wopr:/tmp/multipath $ tar xf control.tar.zst
sarnold@wopr:/tmp/multipath $ cat postinst
#!/bin/sh
set -e
# Automatically added by dh_systemd_enable/13.10.1ubuntu1
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then
 # The following line should be removed in trixie or trixie+1
 deb-systemd-helper unmask 'multipathd.service' >/dev/null || true

 # was-enabled defaults to true, so new installations run enable.
 if deb-systemd-helper --quiet was-enabled 'multipathd.service'; then
  # Enables the unit on first installation, creates new
  # symlinks on upgrades if the unit file has changed.
  deb-systemd-helper enable 'multipathd.service' >/dev/null || true
 else
  # Update the statefile to add new symlinks (if any), which need to be
  # cleaned up on purge. Also remove old symlinks.
  deb-systemd-helper update-state 'multipathd.service' >/dev/null || true
 fi
fi
# End automatically added section
# Automatically added by dh_systemd_start/13.10.1ubuntu1
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then
 if [ -z "${DPKG_ROOT:-}" ] && [ -d /run/systemd/system ]; then
  systemctl --system daemon-reload >/dev/null || true
  deb-systemd-invoke start 'multipathd.socket' >/dev/null || true
 fi
fi
# End automatically added section
# Automatically added by dh_installinit/13.10.1ubuntu1
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then
 if [ -z "${DPKG_ROOT:-}" ] && [ -x "/etc/init.d/multipath-tools" ]; then
  update-rc.d multipath-tools defaults >/dev/null
  if [ -n "$2" ]; then
   _dh_action=restart
  else
   _dh_action=start
  fi
  invoke-rc.d multipath-tools $_dh_action || exit 1
 fi
fi
# End automatically added section

Yes, invoke-rc.d is definitely called - I'm saying it *shouldn't* be in the sense that it's a bug that it is.

Ah! That makes sense, sorry I didn't catch the meaning. I think at this point we're mostly in agreement.

Picking this bug up.

I don't have a system right now to reproduce the failure, but I'm looking into getting said system. From looking at the failure, it seems this can be solved by the simple diff to d/rules in 0.8.8-1ubuntu1.22.04.1:

diff --git a/debian/rules b/debian/rules
index 5a32e8e..d2949c4 100755
--- a/debian/rules
+++ b/debian/rules
@@ -130,7 +130,7 @@ binary-arch: build install
        dh_installexamples -a
        dh_lintian -a
        dh_systemd_enable -pmultipath-tools multipathd.service
- dh_installinit -pmultipath-tools
+ dh_installinit -pmultipath-tools --no-start
        dh_installudev -pkpartx --priority=95
        dh_installudev -pkpartx --name=dm-parts --priority=56
        dh_installudev -pkpartx --name=del-part-nodes --priority=68

This would remove the restart/start operation from debhelper, making the dh_installinit automatically added section look like:

# Automatically added by dh_installinit/13.6ubuntu1
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then
 if [ -x "/etc/init.d/multipath-tools" ]; then
  update-rc.d multipath-tools defaults >/dev/null || exit 1
 fi
fi
# End automatically added section

FYI - mantic multipath-tools is in progress (https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2018051) and looks to be affected by this too, so if the above solution works I can sneak this in to the mantic version before release.

If we are patching debian/rules (which I see is quite manual and not up to current standards), why not drop the invocation of dh_installinit altogether? We don't need the init script installed.

This would require conffile removal handling as part of the upgrade. (debian/multipath-tools.maintscripts)

That's fair enough. I presume we have kept the init scripts in there to keep the debian diff low. I'll work on making that change!

I was able to reproduce this issue and test a fix. I encountered this error when using Linode's paravirtualized kernel 6.0.2-x86_64-linode157.

Do note, that when I created my Linode account and created a new Ubuntu 22.04 VM, the default kernel was Grub 2 which _does not_ suffer with this issue and you can down/upgrade freely between versions. I assume other Linode kernels suffer a similar issue, but it's not worth it to spend much time finding which ones are affected manually because the fix should be the same for all of them.

The fix is to simply remove all init handling in d/rules. I'll handle the SRU process for each release. I have already included the fix for the in-progress mantic multipath-tools merge[1].

[1] - https://code.launchpad.net/~mitchdz/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/446987

> the default kernel was Grub 2

Um?

Yeah... I didn't make a typo there. I'm honestly not familiar with Linode as a hosting platform but I assume there's a good reason that wording is used. When provisioning a system you select a drop down for "Select a Kernel" in the Boot Section. This section includes the paravirtualized kernel and other kernels all the way down to (deprecated) 4.0.2-x86_64-linode56.

The default kernel when starting an Ubuntu 22.04 VM in Linode with the "Grub 2" Kernel is 5.15.0-73-generic.

Marking kinetic as wont-fix as it is eol

Created a PPA/MP for each distribution and confirmed the fix for installing/upgrading. Just waiting for the fix to get into Mantic (expecting end of next week if things go well).

Just noting that this bug affects users who escaped security team QA by using a custom kernel, which isn't really supported. However, there's a significant enough set of users affected, and the underlying issue does seem to be a real issue in packaging that is trivially enough fixed that I think updating stable releases is probably OK.

I mentioned this bug to @sergiodj who IIUC is looking at multipath-tools and services which relates to Steve's comment 8. I think it would be useful to have his eyes look at this bug just to determine whether the situation and plans for Mantic might affect the SRUs in any way, and to wait on his ack before uploading the SRUs. It may be that he decides they're unrelated enough that the SRU of removing dh_installinit and rm_conffile (or however that ends up) on /etc/init.d/multipath-tools can land in SRUs right now anyway.

Yeah, the security update did not cause this. The problem code is in the postinst when invoke-rc.d is being called, so on any update/install of multipath-tools will cause this problem for users using the custom linode kernel right now.

This bug was fixed in the package multipath-tools - 0.9.4-5ubuntu1

---------------
multipath-tools (0.9.4-5ubuntu1) mantic; urgency=medium

  * Merge with Debian unstable (LP: #2018051). Remaining changes:
    - d/p/enable-find-multipaths.patch: re-enable find_multipaths by
      default -- see the removed 'add_find-multipaths.patch' (LP 1463046)
    - d/multipath.conf: Install friendly names multipath.conf by default,
      instead of generating it in every installer.
    - d/multipath-tools.dm-mpath-lvm.udev: Adjust initramfs integration
      for new udev rules
    - Remove d/initramfs/local-top (redundant with other initramfs scripts):
      + init-top: take over loading modules (dm-multipath and SCSI device
        handlers); move the missing dm-emc there (now scsi-dh-emc; see
        BTS 567014).
      + remove d/initramfs/local-top
    - d/initramfs/hooks: Add dm-queue-length: users may want to
      change from the default selector and should be able to do so.
      (LP 1673350)
    - multipath initramfs fixes for booting from multipathed devices:
      + d/initramfs/hooks: also copy wwids file on the installed
        system to ensure all paths come up on boot. (LP 1479929)
      + d/initramfs/hooks: install multipathd and required
        directories.
      + d/initramfs/hooks: copy multipath udev rules to initramfs
      + d/initramfs/hooks: do not copy kpartx rules to initramfs
      + d/initramfs/local-bottom: remember to stop multipathd.
      + d/initramfs/local-premount: wait for udev to settle before
        the call to resolve_device() in local_mount_root(), so the
        by-uuid/ symlinks have a chance to be updated by the
        multipath udev rules (LP 1503286).
      + d/initramfs/local-premount: Run multipath with -B so not to
        assign names nor change /etc/multipath/bindings during
        initramfs (LP 1561103)
    - debian/initramfs/local-bottom: wait for the multipathd unix
      socket to close, so to avoid multipathd.socket unit failure.
      (LP 1682178)
    - Split kpartx initramfs bits into kpartx-boot for dmraid (LP 941874)
      + d/kpartx-initramfs/hooks/kpartx
      + d/kpartx-boot.install
      + d/kpartx-boot.postinst
      + d/kpartx-boot.postrm
      + d/control: Add kpartx-boot package for dmraid
    - d/rules: Move udev rules to priority 95, because rules that load
      modules should be >90.
    - d/rules: remove -Bsymbolic-functions from LDFLAGS
      (https://github.com/opensvc/multipath-tools/issues/26)
    - Don't build the multipath-tools binary package on i386; only kpartx.
  * Dropped changes:
    - d/p/kpartx-Improve-finding-loopback-device-by-file.patch: Improve
      finding loopback devices (LP 1747044)
      [ Dropping due to LP: #1961633 ]
    - d/rules: copy udev rule after build.
      [ Included in debian version 0.9.4-5 ]
    - d/multipath-tools.install: install tmpfiles.d/multipath.conf
      [ Included in debian version 0.9.4-2 ]
    - SECURITY UPDATE: symlink attack CVE-2022-41973
      [ Applied upstream in 0.9.4 ]
    - SECURITY UPDATE: authorization bypass CVE-2022-41974
      [ Applied upstream in 0.9.4 ]
  * Added changes:
    - d/rules: do not install init scripts...

I tested this logic on Jammy which has the same logical change. The goal here is essentially a sanity check to ensure that removing /etc/init.d/multipath-tools doesn't cause things to go on fire. Nothing should be relying on that configuration file for us, and this is a check to confirm that.

NOTE: the laptop I used to test this is using Jammy if that is useful.
1. clone the test script repo
$ git clone https://github.com/canonical/server-test-scripts
$ cd server-test-scripts/ha/virsh

2. Setup test environment
$ export AGENT="multipath-tools-test"
$ ./setup-cluster.sh --iscsi-shared-device
// Wait here for a while

5. login to node 01 (can be any node)
$ ipaddr=$(sudo virsh domifaddr ha-agent-virsh-j-multipath-tools-test-node01 | grep ipv4 | head -n1 | awk '{print $4}' | sed 's/...$//')
$ ssh ubuntu@$ipaddr

6. Check that multipath is working properly
$ sudo multipath -ll
mpatha (36001405d94ad1e552f94b67b894b7794) dm-0 LIO-ORG,iscsi-disk01
size=1.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 7:0:0:0 sda 8:0 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 6:0:0:0 sdb 8:16 active ready running

7. check for init script
$ ll /etc/init.d/multipath-tools
-rwxr-xr-x 1 root root 2827 Feb 18 2022 /etc/init.d/multipath-tools*

8. update to the new multipath version
$ sudo add-apt-repository ppa:mitchdz/multipath-tools-2026881
$ sudo apt update -y && sudo apt upgrade -y
$ dpkg -l multipath-tools | grep multipath-tools | awk '{print $3}'
0.8.8-1ubuntu1.22.04.1
$ sudo apt upgrade
$ dpkg -l multipath-tools | grep multipath-tools | awk '{print $3}'
0.8.8-1ubuntu1.22.04.2~jammy5

9. check for init script
$ ll /etc/init.d/multipath-tools
ls: cannot access '/etc/init.d/multipath-tools': No such file or directory

10. Check multipath is still working fine after upgrade
$ sudo multipath -ll
mpatha (36001405d94ad1e552f94b67b894b7794) dm-0 LIO-ORG,iscsi-disk01
size=1.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 7:0:0:0 sda 8:0 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 6:0:0:0 sdb 8:16 active ready running

11. Reboot
wait for ssh
$ ssh ubuntu@$ipaddr

12. re-connect iscsi drives
$ sudo dhclient enp2s0 #enp2s0 is not in netplan for some reason
$ sudo iscsiadm -m node --login

13. Check multipath-tools is still working
$ sudo multipath -ll
mpatha (36001405d94ad1e552f94b67b894b7794) dm-0 LIO-ORG,iscsi-disk01
size=1.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:0 sda 8:0 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 7:0:0:0 sdb 8:16 active ready running

Doesn't mantic need a fix to remove the initscripts on upgrade?

Yes, there is a MP in progress right now for Mantic init rm_conffile - https://code.launchpad.net/~mitchdz/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/448287

For the Focal/Jammy/Lunar SRU - in my opinion the current autopkgtests are good enough to test for regressions with this change, so my testing will be focused on making sure the init conffile is removed, and upgrade/installs work in Linode using the paravirtual kernel.

Hello Aaron, or anyone else affected,

Accepted multipath-tools into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.8.8-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Aaron, or anyone else affected,

Accepted multipath-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.8.8-1ubuntu1.22.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Aaron, or anyone else affected,

Accepted multipath-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.8.3-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Sorry for delay on this. I haven't tested this package because I was waiting for the glibc migration. I see the package just went through to mantic, so going to promptly test all releases on Linode to confirm resolution in proposed pocket.

I have tested the proposed package on Lunar/Jammy/Focal

Tested packages:
Lunar: 0.8.8-1ubuntu2
Jammy: 0.8.8-1ubuntu1.22.04.2
Focal: 0.8.3-1ubuntu2.2

Just going to write the results down for Lunar, because they are almost identical; the only difference is you don't need to pin proposed for Jammy/Focal. That's just there because in Lunar proposed pocket is pinned below main (You can also use `apt -t lunar-proposed install multipath-tools` to install).

Steps:

1. Create VM in Linode - I used Nanode 1GB as my Plan

2. Wait for VM to boot

3. Stop VM

4. Change to paravirtual Kernel
In linode dashboard
* go to Configurations -> Edit -> Select a Kernel -> 6.0.2-x86_64-linode157
Save Changes

5. Power On Linode

6. Update packages
# apt update -y

7. Reproduce failure, we can do this by uninstalling the package and reinstalling
# apt remove -y multipath-tools
# apt install -y multipath-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  multipath-tools-boot
The following NEW packages will be installed:
  multipath-tools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 328 kB of archives.
After this operation, 1,233 kB of additional disk space will be used.
Get:1 http://mirrors.linode.com/ubuntu lunar/main amd64 multipath-tools amd64 0.8.8-1ubuntu2 [328 kB]
Fetched 328 kB in 0s (6,083 kB/s)
Selecting previously unselected package multipath-tools.
(Reading database ... 112654 files and directories currently installed.)
Preparing to unpack .../multipath-tools_0.8.8-1ubuntu2_amd64.deb ...
Unpacking multipath-tools (0.8.8-1ubuntu2) ...
Setting up multipath-tools (0.8.8-1ubuntu2) ...
multipathd.socket is a disabled or a static unit, not starting it.
Job for multipathd.service failed because the control process exited with error code.
See "systemctl status multipathd.service" and "journalctl -xeu multipathd.service" for details.
invoke-rc.d: initscript multipath-tools, action "restart" failed.
× multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/lib/systemd/system/multipathd.service; disabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2023-08-29 22:39:01 UTC; 7ms ago
TriggeredBy: ○ multipathd.socket
    Process: 37781 ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath (code=exited, status=1/FAILURE)
    Process: 37782 ExecStart=/sbin/multipathd -d -s (code=exited, status=1/FAILURE)
   Main PID: 37782 (code=exited, status=1/FAILURE)
     Status: "configure"
        CPU: 13ms

Aug 29 22:39:01 localhost systemd[1]: Starting multipathd.service - Device-Mapper Multipath Device Controller...
Aug 29 22:39:01 localhost multipathd[37782]: --------start up--------
Aug 29 22:39:01 localhost multipathd[37782]: read /etc/multipath.conf
Aug 29 22:39:01 localhost multipathd[37782]: DM multipath kernel driver not loaded
Aug 29 22:39:01 localhost multipathd[37782]: path checkers start up
Aug 29 22:39:01 localhost systemd[1]: multipathd.service: Main process exited, code=exited, status=1/FAILURE
Aug 29 22:39:01 localhost systemd[1]: multipathd.service: Failed with result 'exit-code'.
A...

I'm seeing the following when I upgrade from jammy to jammy-proposed: the multipathd service is not restarted.

The upgrade path calls:
 old-prerm upgrade 0.8.8-1ubuntu1.22.04.2
 old-postrm upgrade 0.8.8-1ubuntu1.22.04.2
 new-postinst configure 0.8.8-1ubuntu1.22.04.2

The old-prerm is this:
if [ -z "${DPKG_ROOT:-}" ] && [ "$1" = remove ] && [ -x "/etc/init.d/multipath-tools" ] ; then
    invoke-rc.d multipath-tools stop || exit 1
fi
if [ -z "${DPKG_ROOT:-}" ] && [ -d /run/systemd/system ]; then
    deb-systemd-invoke stop 'multipathd.socket' >/dev/null || true
fi

Notice how invoke-rc.d stops "multipath-tools", but deb-systemd-invoke only stops the socket. And only deb-systemd-invoke will be called, because invoke-rc.d is gated in $1=remove.

Old postrm has all actions gated in $1 being remove or purge, so nothing is done essentially.

Then new postinst, ignoring the various enable/update-state calls, we have a start:
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then
    if [ -z "${DPKG_ROOT:-}" ] && [ -d /run/systemd/system ]; then
        systemctl --system daemon-reload >/dev/null || true
        deb-systemd-invoke start 'multipathd.service' 'multipathd.socket' >/dev/null || true
    fi
fi

So why isn't this upgrade restarting (stop+start) multipathd?

I think it's because the old-prerm is only stopping the multipathd socket, not the service. THe journal log shows this during that upgrade:
Aug 31 14:31:29 j-mp-tools systemd[1]: Started Device-Mapper Multipath Device Controller.
Aug 31 14:31:48 j-mp-tools systemd[1]: multipathd.socket: Deactivated successfully.
Aug 31 14:31:48 j-mp-tools systemd[1]: Closed multipathd control socket.
Aug 31 14:31:48 j-mp-tools systemd[1]: Reloading.
Aug 31 14:31:48 j-mp-tools systemd[1]: multipathd.socket: Socket service multipathd.service already active, refusing.
Aug 31 14:31:48 j-mp-tools systemd[1]: Failed to listen on multipathd control socket.

I added a set -x to old-prerm:
Preparing to unpack .../multipath-tools_0.8.8-1ubuntu1.22.04.2_amd64.deb ...
+ FIXED=0.4.8-1
+ [ upgrade = failed-upgrade ]
+ [ -z ]
+ [ upgrade = remove ]
+ [ -z ]
+ [ -d /run/systemd/system ]
+ deb-systemd-invoke stop multipathd.socket
Unpacking multipath-tools (0.8.8-1ubuntu1.22.04.2) over (0.8.8-1ubuntu1.22.04.1) ...
Setting up multipath-tools (0.8.8-1ubuntu1.22.04.2) ...
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.
Removing obsolete conffile /etc/init.d/multipath-tools ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...

Notice the "Could not execute systemctl" message.

It failed to stop multipathd *service*, because it tried to stop the socket while the service was still active. But I can't reproduce this manually, systemctl commands are happy to stop the socket while the service is still running.

I'll troubleshoot a bit more later, but would be interested in your input. Your test case only shows that multipathd was still running after an upgrade, but I believe it was still the old process.

Thanks Andreas! I do definitely see the journalctl failure on upgrade and do concur with your findings. However, on upgrade I am prompted to restart the multipathd service. I'm wondering why you're not seeing that?

Attached my upgrade logs. The important thing is that at the end of apt update I see

Restarting services...
 systemctl restart multipathd.service

Andreas, can you confirm that you were not prompted to restart the multipathd service after upgrade? Also are you on an ubuntu minimal image or the like? I tested on an AWS EC2 instance, so I'm curious if the upgrade behavior might be different on systems which is causing this discrepancy.

Right, no prompt, that comes from needrestart, and we can't rely on that to restart services for us. The package has to take care of it itself.

I didn't finish the troubleshooting, something else came along, but I think the crux of the problem is that the old prerm script was doing invoke-rc.d stop multipath-tools, and deb-systemd-invoke stop multipath.socket (just the socket). Since we removed the invoke-rc.d part, that left just stopping the socket, which looks like is not really stopping the service. So the new postinst comes along and the "start" that it issues becomes a no-op, because the service was still running.

Maybe we need to add a conditional restart in postinst checking for the package versions, and handle this specific upgrade very specifically.

I'm changing the tag to verification-failed because multipathd was not restarted during this upgrade.

Thanks for this Andreas! I'll double check focal/jammy/lunar and ensure the same behavior is across all the releases.

If a user reboots/restarts the daemon that fixes the problem, but this should be solved on upgrade. I'll add logic to d/postinst to explicitly reload multipathd when upgrading from < this version.

Subsequent installations past this release will be fine because the new prerm handle closing both multipathd and the socket. e.g. sudo apt install multipath-tools --reinstall has no issue, so this isn't a problem going forward, it just needs to be addressed between the transition of having the init scripts and removing them.

This upgrade issue is not present on Focal, because the multipath-tools.prerm is different and has this added line:

# Automatically added by dh_installinit/12.10ubuntu1
if [ -x "/etc/init.d/multipath-tools" ]; then
 invoke-rc.d multipath-tools stop || exit 1
fi

So this only needs to be fixed for the transition in Jammy/Lunar.

While testing the fix for this multipathd.service restarting issue, I noticed a new issue relating to the multipathd.socket where the socket/service is not started in the right order in postinst. Noted that bug here - https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2034471

The migration into the main archive is stopped due to the following bugs:

https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2034471
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2035098

I'm working on getting a fix for these 2 new bugs in a new version right now. Once the new version for these 2 bugs get committed/merged then this bug can be set to merged.

For now, keep LL/JJ/FF as "Fix Committed" as it's in -devel, but not in the main archive.

All autopkgtests for the newly accepted multipath-tools (0.8.8-1ubuntu1.22.04.2) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

livecd-rootfs/2.765.20 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#multipath-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Looking at this during a patch-pilot shift and thought I'd add a quick note here, being somewhat (intimately) familiar with dh's systemd restart behaviour.

I concur with vorlon's observation that the d/rules in this package seems excessively "manual" (explicitly calling the various dh scripts). However, while I'd certainly say it's worth bringing the package up to modern standards the restart behaviour will need *very* careful testing. Some things to be aware of:

* dh systemd restart defaults change (in compat version 10 ... I think?) from stop->install->start to install->restart. Because this package doesn't explicitly select a restart behaviour it'll be affected by the default changing (if it moves beyond dh level 9).

* In LP: #1959054 we changed the behaviour of debhelper's stop->install->start method to run the stop phase in the preinst of the current version, rather than the prerm of the old version (see the ticket for more details). This will also potentially affect this package, should it upgrade (the change was in dh_installsystemd which this package doesn't currently use but which it will have to if it moves to dh-compat >9).

Again, this *shouldn't* affect this bug (I don't think? I've only skimmed the comments thus far though), but it's worth bearing in mind for future iterations.

Thanks for your comment Dave! Fortunately in Mantic the d/rules for multipath-tools was overhauled and doesn't call nearly as many manual steps, and now uses compat version 13, so a big improvement in my books.

For LL/JJ/FF I don't think we should change the compat version in an SRU, so we don't need to consider that, but it's definitely useful to keep in mind.

Hello Aaron, or anyone else affected,

Accepted multipath-tools into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.8.8-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Aaron, or anyone else affected,

Accepted multipath-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.8.8-1ubuntu1.22.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.