When upgrading to Mantic, it fails to install snap firmware-updater

Status changed to 'Confirmed' because the bug affects multiple users.

I tested an upgrade from Ubuntu 23.04 to Ubuntu 23.10 and was able to recreate this failure. After the upgrade was completed the `firmware-updater` snap was *not* installed and trying to install it again failed. However, after rebooting the upgraded system I was able to install the `firmware-updater` snap.

I also tested an upgrade from Ubuntu 23.04 to Ubuntu 23.10 where I installed the latest version of snapd (something do-release-upgrade does not require) and rebooted the Ubuntu 23.04 system. I then ran `do-release-upgrade -d` and the `firmware-updater` snap was not installed during the upgrade process.

On my up-to-date lunar system, I was able to 'snap install firmware-updater' without issue.

I was also able to install it in a lunar lxd container.

$ lxc launch ubuntu:lunar -- lp-2039268
Creating lp-2039268
Starting lp-2039268
$ lxc exec lp-2039268 -- bash
# snap info snapd | grep installed
  Start with 'snap list' to see installed snaps.
installed: 2.60.3 (20092) 42MB snapd
# snap install firmware-updater
[...]
# snap connections|grep :desktop-legacy
desktop-legacy firmware-updater:desktop-legacy :desktop-legacy -
#

On a live system booted with the lunar ISO, I also see no failures installing firmware-updater.

snapd snap in the image is at version 2.59.1.

the plug shows as connected after install.

And on a real installed system, I executed the release upgrade via update-manager -d and was unable to reproduce this issue.

I've tried multiple ways to stop this from happening:

1) upgrading the snapd snap to version 2.60.4 before starting the upgrade
2) modifying the upgrader so that firmware-updater is installed after gtk-common-themes

Neither of those resolved the issue and in my experience the error message changes occassionally.

Below is one message I've seen:

Automatically connect eligible plugs and slots of snap "firmware-updater" (internal error: auto-connect of &{"firmware-updater:icon-themes" "gtk-common-themes:icon-themes"} failed: snap "gtk-common-themes" has no slot named "icon-themes")

Another:

- Automatically connect eligible plugs and slots of snap "firmware-updater" (internal error: auto-connect of &{"firmware-updater:gnome-42-2204" "gnome-42-2204:gnome-42-2204"} failed: snap "gnome-42-2204" has no slot named "gnome-42-2204")

Also running `snap info snapd` I noticed that snapd is broken before rebooting after the upgrade.

opening a task on snapd for this

I have managed to reproduce this in a fresh lunar VM. Applied updates with update-manager, rebooted, ran the upgrade with update-manager -d.

Rerunning 'snap install firmware-updater' from the commandline also fails, though not always with the same slot failing.

Output of 'snap changes':

ID Status Spawn Ready Summary
1 Done 2023-04-18 today at 16:54 PDT Initialize system state
2 Done today at 16:53 PDT today at 16:53 PDT Initialize device
3 Done today at 17:46 PDT today at 17:46 PDT Switch "core22" snap to channel "stable"
4 Error today at 17:46 PDT today at 17:46 PDT Install "firmware-updater" snap from "stable/ubuntu-23.10" channel
5 Done today at 17:46 PDT today at 17:46 PDT Switch "gnome-42-2204" snap to channel "stable/ubuntu-23.10"
6 Done today at 17:46 PDT today at 17:46 PDT Switch "gtk-common-themes" snap to channel "stable/ubuntu-23.10"
7 Done today at 17:46 PDT today at 17:46 PDT Switch "snap-store" snap to channel "stable/ubuntu-23.10"
8 Done today at 17:46 PDT today at 17:46 PDT Switch "snapd-desktop-integration" snap to channel "stable/ubuntu-23.10"
9 Done today at 17:46 PDT today at 17:46 PDT Switch "firefox" snap to channel "stable/ubuntu-23.10"
10 Error today at 17:51 PDT today at 17:51 PDT Install "firmware-updater" snap
11 Error today at 17:51 PDT today at 17:51 PDT Install "firmware-updater" snap

output of 'snap tasks 4':

Status Spawn Ready Summary
Done today at 17:46 PDT today at 17:46 PDT Ensure prerequisites for "firmware-updater" are available
Undone today at 17:46 PDT today at 17:46 PDT Download snap "firmware-updater" (109) from channel "stable/ubuntu-23.10"
Done today at 17:46 PDT today at 17:46 PDT Fetch and check assertions for snap "firmware-updater" (109)
Undone today at 17:46 PDT today at 17:46 PDT Mount snap "firmware-updater" (109)
Undone today at 17:46 PDT today at 17:46 PDT Copy snap "firmware-updater" data
Undone today at 17:46 PDT today at 17:46 PDT Setup snap "firmware-updater" (109) security profiles
Undone today at 17:46 PDT today at 17:46 PDT Make snap "firmware-updater" (109) available to the system
Error today at 17:46 PDT today at 17:46 PDT Automatically connect eligible plugs and slots of snap "firmware-updater"
Hold today at 17:46 PDT today at 17:46 PDT Set automatic aliases for snap "firmware-updater"
Hold today at 17:46 PDT today at 17:46 PDT Setup snap "firmware-updater" aliases
Hold today at 17:46 PDT today at 17:46 PDT Run install hook of "firmware-updater" snap if present
Hold today at 17:46 PDT today at 17:46 PDT Run default-configure hook of "firmware-updater" snap if present
Hold today at 17:46 PDT today at 17:46 PDT Start snap "firmware-updater" (109) services
Hold today at 17:46 PDT today at 17:46 PDT Run configure hook of "firmware-updater" snap if present
Hold today at 17:46 PDT today at 17:46 PDT Run health check of "firmware-updater" snap

......................................................................
Automatically connect eligible plugs and slots of snap "fi...

mount showed /dev/vda2 mounted on /snap. 'sudo umount /snap' revealed the contents of /snap/snapd/18933.

Also for reference, the desktop system I upgraded where I could not reproduce this is ancient, continuously upgraded, and apparently does not have the snapd snap installed at all?

Before the upgrade I have a normal set of mount units for mount points under /snap.

After the upgrade, I get an additional 'snap.mount' unit that shows up and mounts on /snap.

# Ensure that snap mount directory is mounted "shared" so snaps can be refreshed correctly (LP: #1668759).
[Unit]
Description=Ensure that the snap directory shares mount events.
[Mount]
What=/snap
Where=/snap
Type=none
Options=bind,shared
[Install]
WantedBy=local-fs.target

The bug number in the comment appears to be incorrect.

This comes from the snapd mount generator.

The code to generate this mount unit is not new, it dates back to snapd 2.31.2.

But that unit was not there at boot.

Its generation gets triggered as part of the ubuntu-release-upgrade after the apt upgrade has finished, as part of the Quirks.manticPostUpgrade based on the timing in the log vs the timestamp of the mount unit file.

It does happen after the logging line 'DEBUG switch of snap core22 succeeded' (I guess because before upgrade it was incorrectly following the ubuntu-23.04 channel but should be following stable?)

I have confirmed that on a lunar system with snapd 2.59.1+23.04ubuntu1.1, the snap.mount does not exist and /snap is not bind-mounted.

The code of the generator says it's not supposed to be needed as a workaround if the root filesystem is already mounted as shared:, which is the case at the start of the upgrade.

After upgrade, the / line in /proc/self/mountinfo still shows:

28 1 252:2 / / rw,relatime shared:1 - ext4 /dev/vda2 rw

so I don't know why this mount unit winds up being output.

This is sufficient to reproduce the issue on a lunar desktop:

sudo sed -i -e's/lunar/mantic/' /etc/apt/sources.list && sudo apt update && sudo apt install systemd

$ lxc launch ubuntu:lunar lp-2039268
Creating lp-2039268
Starting lp-2039268
$ lxc exec lp-2039268 bash
# sed -i -e's/lunar/mantic/' /etc/apt/sources.list && apt update && apt install systemd
# ls -l /run/systemd/generator/snap.mount
-rw-r--r-- 1 root root 274 Oct 18 05:49 /run/systemd/generator/snap.mount
#

in a container the mount unit appears not to actually activate, but this is enough to reproduce the generation of the unit.

moving the generator aside and replacing it with the following script:

cd /lib/systemd
mv system-generators/snapd-generator .
cat > system-generators/snapd-generator <<EOF
#!/bin/sh

exec > /tmp/lp.2039268 2>&1
cat /proc/self/mountinfo

exec strace -ff -s256 -o /tmp/trace-2039268 /lib/systemd/snapd-generator "\$@"
EOF
chmod a+x system-generators/snapd-generator

It looks like this unit isn't actually generated by the snapd generator in the snapd deb, but maybe by the copy in the snapd snap? which could explain why /proc/self/mountinfo is different, but I don't understand how or why this would be executed.