[SRU] flash-kernel failure when upgrading f-k and kernel in the same cycle

The same issue (LP: #1861292) was fixed on focal, but the fix is not available after jammy. The patch for focal cannot be applied to lunar because of [1].

---
[1] https://salsa.debian.org/installer-team/flash-kernel/-/commit/30f64ebd28553e74986aa4a80963b42f478a4d3d

Verified the test flash-kernel package on a Xilinx ZCU board. With the test package, fitimage can be generated without any problem.

The attachment "flash-kernel-lunar.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Hmm, this is a tricky one and there's a lot I don't like here.

The first thing I don't like is I probably caused this issue back in focal :). I was fixing an issue of flash-kernel triggers getting "forgotten" during flash-kernel updates (LP: #1667742).

The second thing I don't like is that we apparently haven't noticed this (quite serious) issue in all the releases since. That's ... suspicious? I would at least have expected this to have cropped up for some people on the pi images which ought to be seeing something similar and are quite widely used.

The third thing I'm not particularly keen on is the proposed fix: it causes flash-kernel to exit *assuming* it'll be run in future without actually guaranteeing that (contrast with the trigger deferral logic near the top of main where flash-kernel triggers itself and then exits to ensure it will definitely be run later). It also doesn't necessarily account for the scenario where flash-kernel is being upgraded to fix an issue in flash-kernel's historic behaviour. If the current kernel needs re-installing anyway (because there was some defect in the way an old version of f-k handled things), even when a new kernel version will be installed later, that should still be done as we can't guarantee that the new kernel's initrd will be generated successfully.

Hence, the two things we really want to avoid:

* Exiting without error when an error has actually occurred (i.e. the initrd is missing, but it's *actually* missing not just "waiting to be generated")

* Guaranteeing we don't "skip" flash-kernel executions when flash-kernel itself is being upgraded, even when they might *seem* redundant (they're only redundant in the case that everything works correctly, but the error scenarios are valid edge cases)

I've spent a few hours digging into all the trigger logic that exists between the kernel images, initramfs-tools (responsible for generating the initrd), and flash-kernel. I'm reasonably convinced at this point that we can't *prevent* flash-kernel from running at a point where a new kernel isn't *completely* installed. The fact that flash-kernel *must* run when it itself is upgraded means we're always vulnerable to being run at that point.

However, flash-kernel is being run to install "the lastest version". It uses "linux-version list" to get the installed kernels. That operation lists the kernels which *exist* (as, say, /boot/vmlinuz-5.15.0-1018-xilinx-zynqmp) under /boot, but doesn't care whether the kernel package is actually *installed fully* by dpkg yet.

For example: /boot/vmlinuz-5.15.0-1018-xilinx-zynqmp exists because the linux-image-5.15.0-1018-xilinx-zynqmp has been unpacked, but it must still be in "triggers-awaiting" state because the initramfs-tools trigger has not yet run to generate the corresponding initrd.

So ... I think the root of the issue here is that flash-kernel is not being sufficiently discriminating of its selection of the "latest" kernel; it considers not-fully-installed kernels to be valid for installation. In fact, even if the initrd *does* exist, if triggers are still pending it still shouldn't consider that a valid candidate (this can occur if, for example, a linux-m...

Okay, I've now confirmed that this is indeed the rather more wide-ranging bug that I suspected it was (it also affects the Pi images, but in a different way). I'll try and explain the problem with reference to the log provided in comment:3. First, the scenario:

* the device currently has kernel 5.15.0-1015-xilinx-zynqmp
* kernel 5.15.0-1018-xilinx-zynqmp is available for upgrade
* the device currently has flash-kernel 3.104ubuntu14 installed
* flash-kernel 3.106ubuntu12test1 is available for upgrade

This in turn means that the following package states are currently true:

* linux-image-5.15.0-1015-xilinx-zynqmp is "installed"
* linux-image-5.15.0-1018-xilinx-zynqmp is "not installed"
* flash-kernel is "installed"

Now selected entries from the log, in chronological order:

  Preparing to unpack .../0-flash-kernel_3.106ubuntu12test1_arm64.deb ...
  Unpacking flash-kernel (3.106ubuntu12test1) over (3.104ubuntu14) ...
  ...
  Selecting previously unselected package linux-image-5.15.0-1018-xilinx-zynqmp.
  Preparing to unpack .../4-linux-image-5.15.0-1018-xilinx-zynqmp_5.15.0-1018.20_arm64.deb ...
  ...
  Setting up flash-kernel (3.106ubuntu12test1) ...

At this point:

* flash-kernel has now been upgraded but is now in "triggers-pending" as its upgrade triggers itself (this is the issue fixed in LP: #1667742)
* linux-image-5.15.0-1015-xilinx-zynqmp is still "installed"
* linux-image-5.15.0-1018-xilinx-zynqmp is now "unpacked"

Further entries:

  I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-1018-xilinx-zynqmp
  I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-1018-xilinx-zynqmp
  Setting up linux-image-xilinx-zynqmp (5.15.0.1018.19) ...
  Setting up linux-modules-5.15.0-1018-xilinx-zynqmp (5.15.0-1018.20) ...
  Setting up linux-xilinx-zynqmp (5.15.0.1018.19) ...

At this point, linux-image-5.15.0-1018-xilinx-zynqmp is now "triggers-awaiting". The kernel postinst has run, updating the /boot symlinks to point to the 1018 kernel. However, the /boot/initrd.img symlink is "dead" because the initrd for this version has not yet been generated. This doesn't matter though as the /boot kernel isn't the one actually used at boot time; it's just the one that flash-kernel copies to <wherever the platform needs>, and flash-kernel hasn't been called yet.

Carrying on:

  Processing triggers for initramfs-tools (0.140ubuntu13) ...
  initramfs-tools: update-initramfs -u
  update-initramfs: Generating /boot/initrd.img-5.15.0-1015-xilinx-zynqmp

The initramfs-tools hook is called for the *old* kernel, regenerating its initrd. This appears largely pointless (and in this case, actually is) but is necessary to cover the case of linux-modules-extra packages being added / removed, requiring a regeneration of the initrd (and possibly other scenarios I'm unaware of).

Carrying on:

  Processing triggers for flash-kernel (3.106ubuntu12test1) ...
  flash-kernel: installing version 5.15.0-1018-xilinx-zynqmp

Here's the root of the error. flash-kernel finally runs "properly" (no deferral), and uses "linux-version list" to determine the set of installed kernel versions. This simply looks for /boot/vmlinu[xz]-$ver files and reports the $ver suffixes. At this ...

I've prepared a patched version of flash-kernel for lunar in ppa:waveform/flash-kernel (version 3.106ubuntu13~ppa3 at https://launchpad.net/~waveform/+archive/ubuntu/flash-kernel), with test coverage of the new functions. I've tested that this fixes the issue illustrated on the Pi above which I believe has the same root cause as on the Xilinx boards but I would be most grateful if the fix could be tested on the Xilinx platform too.

To test:

* Install a stock lunar preinstalled server image
* sudo add-apt-repository ppa:waveform/flash-kernel
* sudo apt install linux-xilinx-zynqmp flash-kernel
* Ensure that upgrade proceeds without error
* sudo reboot
* Ensure system boots with new kernel version (and initrd!) successfully

As noted, the fix will result in a "redundant" run of flash-kernel instead of the skip, but this is safe and actually desirable for certain scenarios. I'll convert this to an SRU if verification on lunar is successful (and will forward the patch to Debian subsequently).

@Dave

I tried the test package[1]. It can generate a fitimage with old kernel without any problem. But, the fitimage for new kernel isn't generated after new initrd is ready.

Here is the log: (For detailed log, please see the attached file)
update-initramfs: Generating /boot/initrd.img-5.15.0-1018-xilinx-zynqmp
...
Ignoring old or unknown version 5.15.0-1018-xilinx-zynqmp (latest is 5.15.0-1015-xilinx-zynqmp)
/etc/kernel/postinst.d/zz-flash-kernel:
Ignoring old or unknown version 5.15.0-1018-xilinx-zynqmp (latest is 5.15.0-1015-xilinx-zynqmp)

---
[1] 3.106ubuntu13~ppa3, https://launchpad.net/~waveform/+archive/ubuntu/flash-kernel

Okay, attempt number 3! After another long dive into dpkg states, triggers, and the interactions around the kernel and initrd I think I've finally got a version that should:

1. Never install a kernel which isn't fully installed (unless --force is specified)

2. Avoids the race on waiting for a kernel to become fully installed during an apt run (when it discovers it's being asked to install a non-fully-installed kernel, it defers itself by retriggering)

3. Avoids duplicated runs (it does use the full version list from "linux-version list", but defers in the case that the latest isn't fully installed)

It's currently building the aforementioned PPA (https://launchpad.net/~waveform/+archive/ubuntu/flash-kernel) and the test procedure should be the same:

* Install a stock lunar preinstalled server image
* sudo add-apt-repository ppa:waveform/flash-kernel
* sudo apt install linux-xilinx-zynqmp flash-kernel
* Ensure that upgrade proceeds without error
* sudo reboot
* Ensure system boots with new kernel version (and initrd!) successfully

I've tested this several times with the raspi images (that displayed a related, but different symptom) in circumstances that do replicate the race condition without the fix and things are successful there, so I'm hoping that's also the case on Xilinx.

Hi Dave

With flash-kernel (3.106ubuntu13~ppa6), the issue is gone and a new fitimgae with new kernel is generated. Please see the attached file for detailed log.

This bug was fixed in the package flash-kernel - 3.106ubuntu13

---------------
flash-kernel (3.106ubuntu13) lunar; urgency=medium

  * Ensure that only kernels in fully "installed" state are considered for
    flashing (LP: #2007827)
  * test_functions: Add tests for include_only_flavors
  * d/control: Remove redundant awk dependency to appease lintian
  * Bump standards version

 -- Dave Jones <email address hidden> Mon, 27 Feb 2023 11:53:49 +0000

I've been generating kernel images with this script from the raspberry pi foundation kernels for some time:

https://gist.github.com/satmandu/a507c59d84737f6d29ff353395819d51

flash-kernel gives me this error now in lunar:

Using DTB: bcm2711-rpi-4-b.dtb
Installing /etc/flash-kernel/dtbs/bcm2711-rpi-4-b.dtb into /boot/dtbs/6.1.16-v8+/./bcm2711-rpi-4-b.dtb
Taking backup of bcm2711-rpi-4-b.dtb.
Installing new bcm2711-rpi-4-b.dtb.
dpkg-query: no packages found matching

I've had to modify /var/lib/dpkg/info/flash-kernel.postinst to have "exit 0" at the end, and I've also modified the beginning to replace "set -e" with "set -x" to get some more info on what's happening.

Trying to install the latest generated image, I get this error:

Setting up linux-image-6.1.16-v8+ (6.1.16-v8+-1) ...
Processing triggers for linux-image-6.1.16-v8+ (6.1.16-v8+-1) ...
/etc/kernel/postinst.d/dkms:
 * dkms: running auto installation service for kernel 6.1.16-v8+
   ...done.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.1.16-v8+
Using DTB: bcm2711-rpi-4-b.dtb
Installing /etc/flash-kernel/dtbs/bcm2711-rpi-4-b.dtb into /boot/dtbs/6.1.16-v8+/./bcm2711-rpi-4-b.dtb
Taking backup of bcm2711-rpi-4-b.dtb.
Installing new bcm2711-rpi-4-b.dtb.
flash-kernel: deferring update (trigger activated)
/etc/kernel/postinst.d/zz-flash-kernel:
Using DTB: bcm2711-rpi-4-b.dtb
Installing /etc/flash-kernel/dtbs/bcm2711-rpi-4-b.dtb into /boot/dtbs/6.1.16-v8+/./bcm2711-rpi-4-b.dtb
Taking backup of bcm2711-rpi-4-b.dtb.
Installing new bcm2711-rpi-4-b.dtb.
flash-kernel: deferring update (trigger activated)
Processing triggers for flash-kernel (3.106ubuntu13) ...
+ [ xtriggered = xtriggered ]
+ FLASH_KERNEL_NOTRIGGER=y flash-kernel
Using DTB: bcm2711-rpi-4-b.dtb
Installing /etc/flash-kernel/dtbs/bcm2711-rpi-4-b.dtb into /boot/dtbs/6.1.16-v8+/./bcm2711-rpi-4-b.dtb
Taking backup of bcm2711-rpi-4-b.dtb.
Installing new bcm2711-rpi-4-b.dtb.
dpkg-query: no packages found matching
+ exit 0

(without the exit 0, flash-kernel always quits with an error, leaving the install of the kernel broken.)

I can't tell if the flash-kernel breakage is connected to this change or not... but I can't understand why there is a dpkg-query error here.

I do see this:
 dpkg-query -l linux-image-6.1.16-v8+
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-======================-============-============-=================================
ii linux-image-6.1.16-v8+ 6.1.16-v8+-1 arm64 Linux kernel, version 6.1.16-v8+

(Once I've installed the kernel by manually copying the generated initrd.img and vmlinuz into /boot/firmware/ I can boot with the new kernel just fine, but the manual install wasn't needed a month ago.)

I am concerned that the way that this is being handled is significantly different from the well-exercised path used on x86. But I can't actually work out from the code how we are guarding against this with grub, despite the fact that I know we had a lot of design discussions about it previously. I'm cc:ing Andy Whitcroft here to see if he recalls what we're doing (if anything!) on x86 as I would like to see cross-architecture alignment here.

I'll try and explain my understanding of the differences between GRUB-based (x86) systems and flash-kernel systems (ARM, RISCV, x86 using u-boot, etc.), but I admit my knowledge of the GRUB side is considerably less than the flash-kernel side:

On all systems, the kernel packaging installs the kernel (and its device-trees, if they exist) into /boot, and initramfs-tools generates the initrd in /boot.

On GRUB-based systems, /boot may be part of the root partition (when root is not encrypted), or it may be its own partition (when root is encrypted) but in either case it's *typically* an ext-based file-system (ext2, ext4, etc). No further copying of the kernel or initrd is required.

On flash-kernel systems, /boot is always part of the root partition. It's "just another directory". The kernel or initrd then needs copying / flashing to some other location.

On GRUB-based systems, the boot (whether BIOS or EFI based) will eventually reach a GRUB stage which understands how to read the ext-based file-system, from where it'll load the kernel and initrd and proceed from there.

On flash-kernel systems, the bootloader (u-boot, pi-boot, aboot, etc.) is typically configured only to read FAT (some of these bootloaders *can* be configured to read ext-fs but aren't, usually due to performance / stability concerns). Hence, flash-kernel is called to copy artefacts from /boot to wherever they're required (some FAT file-system, or perhaps an EEPROM block).

Hence, on GRUB-based systems, there need not be a guard against the initrd not being generated before GRUB is updated. Even if the GRUB config is generated before the initrd exists, it doesn't matter because it will by the time we next boot (and the config just "points at" the file under /boot). However, on flash-kernel systems, the initrd under /boot is being actively transferred elsewhere and therefore it *does* matter that it exists prior to f-k being called.

At least, that's my understanding!

Dave, I don't agree that there's a relevant difference here between flash-kernel and grub.

Grub: initrd not yet generated; grub postinst is called and adds kernel as first entry in grub.cfg; power event happens before kernel postinst is called; system fails to (noninteractively) boot.

flash-kernel: initrd not yet generated; flash-kernel postinst is called and writes kernel (without initramfs) to the boot partition; power event happens before kernel postinst is called; system fails to boot.

Both scenarios need to be guarded against, and we should use the same model for both (and any other bootloaders on other archs).

*Unfortunately*, it turns out that while Andy and I whiteboarded this about 5 years back, what we discussed never wound up being implemented in the kernel packages. I believe the plan at the time was to use a dpkg-divert in the kernel preinst to divert /boot/vmlinuz-$ver to /boot/vmlinuz-$ver.dpkg-something, and then to un-divert it in the postinst only after the initramfs has been generated. grub already has code to ignore the pattern vmlinuz*.dpkg-* so no further changes would be required for grub; and it would be easy to make flash-kernel match.

However, none of this matters so long as the kernel doesn't implement such an interface.

In any case, I'm not happy with the proposed implementation in flash-kernel. I think rooting around in the dpkg database is bad form. Ultimately, though, it's not my opinion on this that matters - we really need input from the Kernel Team here wrt what interface we're going to support between the packages. I subscribed Andy to the bug, but he hasn't replied yet; adding Dimitri as well.

Nothing currently awaiting sponsorship, unsubscribing ~ubuntu-sponsors.

On Fri, Jun 23, 2023 at 05:40:57PM -0000, Steve Langasek wrote:
> Dave, I don't agree that there's a relevant difference here between
> flash-kernel and grub.
>
> Grub: initrd not yet generated; grub postinst is called and adds
> kernel as first entry in grub.cfg; power event happens before kernel
> postinst is called; system fails to (noninteractively) boot.
>
> flash-kernel: initrd not yet generated; flash-kernel postinst is
> called and writes kernel (without initramfs) to the boot partition;
> power event happens before kernel postinst is called; system fails to
> boot.
>
> Both scenarios need to be guarded against, and we should use the same
> model for both (and any other bootloaders on other archs).

The "system loses power while installing the kernel" is an extremely
variable story on the flash-kernel side of things. On some platforms
it's mostly guarded against, on others (including the Pi) it's not (and
never has been), and fixing this particular bug will not change that.

> *Unfortunately*, it turns out that while Andy and I whiteboarded this
> about 5 years back, what we discussed never wound up being implemented
> in the kernel packages. I believe the plan at the time was to use a
> dpkg-divert in the kernel preinst to divert /boot/vmlinuz-$ver to
> /boot/vmlinuz-$ver.dpkg-something, and then to un-divert it in the
> postinst only after the initramfs has been generated. grub already
> has code to ignore the pattern vmlinuz*.dpkg-* so no further changes
> would be required for grub; and it would be easy to make flash-kernel
> match.

I see; so "linux-version list" would be modified to ignore the
"vmlinuz*.dpkg-*" pattern, causing flash-kernel to ignore the diverting
kernel as a candidate.

> However, none of this matters so long as the kernel doesn't implement
> such an interface.
>
> In any case, I'm not happy with the proposed implementation in flash-
> kernel. I think rooting around in the dpkg database is bad form.
> Ultimately, though, it's not my opinion on this that matters - we
> really need input from the Kernel Team here wrt what interface we're
> going to support between the packages. I subscribed Andy to the bug,
> but he hasn't replied yet; adding Dimitri as well.

I agree rooting around in dpkg is bad form. I don't *like* this fix, and
I'll be ecstatic if I can chop (most of) it out in future.

So I guess the question is: how long until the kernel folk can implement
the dpkg-divert + linux-version changes, and depending on that answer,
is it worth SRU'ing this in the meantime given the issue has the
potential to break apt upgrades on affected platforms?

I very much appreciate Dave's work here, and Steve's input. It seems that we don't yet have consensus on the right way to fix this though? I think Dave's fix is currently in the SRU queue in Kinetic but blocked on consensus then, so setting that bug task to Incomplete.

satmandu's comments 12 and 13 also need addressing please. I'm not sure if using external kernel packages is something we'd support in Ubuntu, and perhaps we wouldn't consider this a regression, but if we're going to knowingly break them in an SRU, we should document that.

(oh, it's already Incomplete!)

Unfortunately, subscribing Andy and Dimitri hasn't elicited a reply from the Kernel Team on this. And as there are outstanding design/architecture questions, I don't think it's appropriate for us to land this SRU as-is. I'm going to reject the upload from the queue so the SRU team can stop polling it and leave it with you, Dave, to chase the Kernel Team to get consensus here.

An upload of flash-kernel to kinetic-proposed has been rejected from the upload queue for the following reason: "waiting for Kernel Team to weigh in".

Status changed to 'Confirmed' because the bug affects multiple users.

Recently being his by this on Xilinx ZynqMP while running full `apt upgrade` on vanilla system from the image https://people.canonical.com/~platform/images/xilinx/zcu-ubuntu-22.04/iot-limerick-zcu-classic-desktop-2204-x05-2-20221123-58.img.xz

We have about 20 people affected by this internally with potential to hit more. And this is going to repeat every time the `flash-kernel` is upgraded with the Kernel in the same upgrade flow.

With my kernel-team hat on I believe that we are miss-guided fixing this in flash-kernel (or any other consumer of /boot). We are working on a spec to ensure we only have "valid" kernels in /boot.

That said, in the short term though I have no objection to having a flash-kernel specific fix to paper over this.

Setting kinetic to invalid now it's end-of-life. Since the kernel team have now weighed in with no objections to a strictly short-term SRU for the purposes of working around this for existing users I'm going to re-upload the jammy branch and see if we can get this SRU'd.

In the code, you're setting locale variables, e.g.:
LANG=C.UTF-8 dpkg-query --search "$path" 2>/dev/null

I can't work out why this would be needed. What are you trying to change the behavior of here? LC_TYPE? LC_COLLATE? LC_MESSAGES?

In any case, there are several environment variables that take precedence over this setting, so it is probably not actually what you want.

Scanning the dpkg-query code I think the first instance (dpkg-query --search) is redundant. The history behind it is that the output from dpkg-query for *diverted files* is localized and in the code this originally came from I needed to avoid that localization. However, we're not concerned with diverted files here, and I see the output for non-diverted files doesn't involve localization so there's no need to avoid it.

The second instance (dpkg-query --show) is *possibly* redundant and has to do with whether the db:Status field is localized (or potentially could be) in output. If it's not, then it's redundant. Still, you are correct; I guess LC_ALL would be the better catch-all here (though there's something nagging in my memory about dangers of using LC_ALL)?

> though there's something nagging in my memory about dangers of using
> LC_ALL)?

I don't know of any dangers here. It's certainly safer than relying on
$LANG if your script behavior is dependent on the locale. So please fix :)