[UBUNTU 20.04] installkernel script does not symlink /boot/initrd.img which is required with the default zipl.conf

------- Comment From <email address hidden> 2020-05-06 08:06 EDT-------
To add, I'm pretty sure this broke in a very recent change as we were using
the installkernel script extensively just last week.
Also I believe there is something in the usual kernel upgrade process that
hides this issue as kernel upgrades via apt seem to work fine.

------- Comment From <email address hidden> 2020-05-13 09:53 EDT-------
I'm now getting the same behavior on a fresh Ubuntu 18.04 (including apt update && apt upgrade). This really makes testing kernels very frustrating especially with the Ubuntu config that unlike defconfig doesn't have the dasd and qeth drivers compiled in.

A couple of things were checked that could have causing this.
But, we for example cannot really see that 'image.postinst' is used while calling make install.
That script just seems to be incl. into deb package (as the name makes one believe).
Also "linux-update-symlinks" wasn't modified for a long time, so that doesn't seem to be the cause, too. zz-zipl postinstall just rewrites the boot loader...

I am not running into this with a normal update/upgrade - neither on bionic, not on focal.

Would you please let us know the exact steps you are using, when you run into this, so that we have a precise sequence that allows to recreate this issue?

We understood that you started with a fresh 20.04 install.
Then you probably cloned a kernel tree, like:
$ git clone https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal --branch master-next --single-branch focal-master-next
What were the next steps you did?

'Maybe' there is something bloken on your particular system - do you have a change to try on a different 20.04 installation, too?

------- Comment From <email address hidden> 2020-05-26 04:21 EDT-------
I've only seen this happen with manually using the installkernel script with the commands included in my first comment. I will have to try if it also occurs with "make install" in a kernel tree. I currently have a small vacation, just keeping the critical bugs progressing, but will try once I'm working again.
As for different setups, I've seen this on two separate installations one on LPAR and one in z/VM,

So you used installkernel directly rather than make install.
installkernel is shipped with debianutils
A quick look at the changelog (installkernel) didn't showed any changes this year.
So the script itself seems to be stable and unchanged...

You're obviously building and installing kernels in a different way that I do.
I compile and build deb packages directly, that I afterwards just install with apt ...

------- Comment From <email address hidden> 2020-06-02 07:57 EDT-------
I usually compile on a development machine and then
scp a tar archive of the modules and the vmlinuz e.g. to a z/VM
guest with less memory and CPU.
Previously this worked the same across all distros which makes
this very practical for us.

That said, I've now tried with a locally build Kernel
(from the focal Kernel repository master branch)
with the following sequence:

(As build dependencies I installed
build-essential, bison, flex, and libssl-dev)

$ make defconfig
(defconfig is what we usually use internally
but I don't think this changes anything)
$ make -j $(nproc)
$ sudo make modules_install
$ sudo make install

With this I also see the same issue /boot/vmlinuz is
freshly symlinked but /boot/initrd.img keeps pointing
to the previous kernel installed as a deb.

E.g. below that was the PPA Kernel from the other issue:

$ ls -l /boot/initrd.img
lrwxrwxrwx 1 root root 37 Jun 2 04:37 /boot/initrd.img -> initrd.img-5.4.0-31.35ubuntu2-generic
$ ls -l /boot/vmlinuz
lrwxrwxrwx 1 root root 14 Jun 2 05:57 /boot/vmlinuz -> vmlinuz-5.4.27

This is bad, it should be quite impossible to get /boot/initrd.img & /boot/vmlinuz symlinks out of sync!

The only relevant thing that I can think of is us dropping the /etc/kernel-img.conf file

Previously, /etc/kernel-img.conf contained

# Kernel image management overrides
# See kernel-img.conf(5) for details
do_symlinks = yes
do_bootloader = no
do_initrd = yes
link_in_boot = yes

on s390x, whilst "link_in_boot" default was no. We changed that, and make "link_in_boot" default to 1 everywhere and that's reflected in /usr/share/perl5/DebianLinux.pm

As an experiment, can you please create /etc/kernel-img.conf as shown above and let me know if that "fixes" things?

I'll try to trace this further myself as well.

$ sudo make modules_install
$ sudo make install

But that would not create initrd, would it?

I would expect for you to also call

$ update-initramfs -k 5.4.27

no?

To generate the initrd for the newly installed kernel, and update the symlink to point at it?

I also wonder, if we can stop using initrds on s390x. In most public clouds we build kernel config to include the storage modules, such that we can simply boot direct to rootfs without an initrd. This speeds up the boot process quite a bit.

Do you think we should support and/or default to initrd-less boot on s390x?

Hm, actually `make install` should do everything, if it calls debianutils provided installkernel I wonder if that is broken.

------- Comment From <email address hidden> 2020-06-02 10:15 EDT-------
"make install" also triggers initramfs creation as does manual
invocation of the installkernel script. In fact both even trigger
boot loader update/install.

This is the output:
$ sudo make install
sh -x ./arch/s390/boot/install.sh 5.4.27 arch/s390/boot/bzImage \
System.map "/boot"
+ [ -x /root/bin/installkernel ]
+ [ -x /sbin/installkernel ]
+ exec /sbin/installkernel 5.4.27 arch/s390/boot/bzImage System.map /boot
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.4.27 /boot/vmlinuz-5.4.27
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.4.27 /boot/vmlinuz-5.4.27
update-initramfs: Generating /boot/initrd.img-5.4.27
Using config file '/etc/zipl.conf'
Run /lib/s390-tools/zipl_helper.device-mapper /boot
Building bootmap in '/boot'
Building menu 'menu'
Adding #1: IPL section 'ubuntu' (default)
Adding #2: IPL section 'old'
Preparing boot device: dm-0.
Done.
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 5.4.27 /boot/vmlinuz-5.4.27
run-parts: executing /etc/kernel/postinst.d/update-notifier 5.4.27 /boot/vmlinuz-5.4.27
run-parts: executing /etc/kernel/postinst.d/zz-zipl 5.4.27 /boot/vmlinuz-5.4.27
Using config file '/etc/zipl.conf'
Run /lib/s390-tools/zipl_helper.device-mapper /boot
Building bootmap in '/boot'
Building menu 'menu'
Adding #1: IPL section 'ubuntu' (default)
Adding #2: IPL section 'old'
Preparing boot device: dm-0.
Done.

I did already have /etc/kernel-img.conf
with the following content:

# Kernel image management overrides
# See kernel-img.conf(5) for details
do_symlinks = yes
do_bootloader = yes
do_initrd = yes
link_in_boot = yes

Could this depend on the legacy installer?
Our automatically installed systems are currently all installed with it
and I also used that for this test LPAR (that has PCI Physical Functions)
since the legacy installer allowed a much nicer
manual network configuration using the HMC console.
Sadly that's also why I can't currently test on another fresh install
as our automatic installation system is currently being changed
to support the new installer and I couldn't get the previous workaround
to work today.

I'm not knowledgeable enough to give a good opinion on initrd less
boot but I would suspect that the deployments at our customers are much
more varied than the average cloud so I would be hesitant about defaulting
to it. Also in my experience s390 boot times are already quite good.

Afaik with our defconfig a lot of us developers regularly boot
without initrd or even any modules, so supporting it shouldn't be
a problem. As the defconfig doesn't include the
drivers for PCI devices into the Kernel I usually use both modules
and an initrd, which might be why I stumbled over this first.

Eoan and later d-i, new installer, curtin do not install /etc/kernel-img.conf.
Upgraded systems keep having it (ie. installed with bionic or xenial, and upgraded).

Can you please let me know if _removing_ /etc/kernel-img.conf breaks $ sudo make install, and if adding /etc/kernel-img.conf back fixes $ sudo make install?

Cause the expectation is that `/etc/kernel-img.conf` should not be there, yet everything should still work correctly.

I think somewhere something is reading "link_in_boot=yes" and was not updated with the new implicit default to always assume that on recent ubuntu.

------- Comment From <email address hidden> 2020-06-26 09:45 EDT-------
(In reply to comment #17)
> Eoan and later d-i, new installer, curtin do not install
> /etc/kernel-img.conf.
> Upgraded systems keep having it (ie. installed with bionic or xenial, and
> upgraded).
>
> Can you please let me know if _removing_ /etc/kernel-img.conf breaks $ sudo
> make install, and if adding /etc/kernel-img.conf back fixes $ sudo make
> install?
>
> Cause the expectation is that `/etc/kernel-img.conf` should not be there,
> yet everything should still work correctly.
>
> I think somewhere something is reading "link_in_boot=yes" and was not
> updated with the new implicit default to always assume that on recent ubuntu.

For me the installkernel script (including in the current case "sudo make install" from a mainline Linux tree) doesn't update the /boot/initrd.img symlink.

Interestingly a package upgrade for linux-generic does overwrite this symlink.

Not with the /etc/kernel-img.conf[0] and not without it either.
As before the /boot/vmlinuz link is always updated.

[0]
# Kernel image management overrides
# See kernel-img.conf(5) for details
do_symlinks = yes
do_bootloader = yes
do_initrd = yes
link_in_boot = yes

On Fri, 26 Jun 2020 at 15:01, bugproxy <email address hidden> wrote:
>
> ------- Comment From <email address hidden> 2020-06-26 09:45 EDT-------
> (In reply to comment #17)
> > Eoan and later d-i, new installer, curtin do not install
> > /etc/kernel-img.conf.
> > Upgraded systems keep having it (ie. installed with bionic or xenial, and
> > upgraded).
> >
> > Can you please let me know if _removing_ /etc/kernel-img.conf breaks $ sudo
> > make install, and if adding /etc/kernel-img.conf back fixes $ sudo make
> > install?
> >
> > Cause the expectation is that `/etc/kernel-img.conf` should not be there,
> > yet everything should still work correctly.
> >
> > I think somewhere something is reading "link_in_boot=yes" and was not
> > updated with the new implicit default to always assume that on recent ubuntu.
>
> For me the installkernel script (including in the current case "sudo
> make install" from a mainline Linux tree) doesn't update the
> /boot/initrd.img symlink.
>
> Interestingly a package upgrade for linux-generic does overwrite this
> symlink.

.deb package uses very different maintainer scripts / codepath, and is
not the same operation as "sudo make install".

I personally always build my kernels as debs, and install debs, rather
than doing "sudo make install". But I am a distribution developer, and
I care for .debs to work right. Kernel developers, I guess, are
inverse, and care for "upstream" $ sudo make install to work.

>
> Not with the /etc/kernel-img.conf[0] and not without it either.
> As before the /boot/vmlinuz link is always updated.
>

That is slightly concerning, as to how $ sudo make install, ever
worked before..... Or what has changed since.

Normally, $ sudo make install, should lookup if /sbin/installkernel is
available, and call that to "do what it has to do, on a given
distribution", and that script is shipped by the debianutils package
which is required and must be installed always. And it hasn't been
touched in ages.

I wonder which arguments are passed to /sbin/installkernel by $ sudo
make install. And whether the 4th argument is passed, and if it is
empty, /, or /boot. It should be either empty, or /boot.

--
Regards,

Dimitri.

The script responsible for updating the /boot/initrd.img symlink is called `linux-update-symlinks`. If a kernel is installed without that script being run, the symlink will not be updated.

The only place where `linux-update-symlinks` is run is inside each of our kernels, in `debian/templates/image.postinst.in`. This means that when our kernels are built as .deb packages, the appropriate machinery will be placed inside the .deb to thus update the /boot/initrd.img symlink.

I checked the change history to `installkernel` and could not find any point in time in which it ran `linux-update-symlinks`. Furthermore, the help info inside `linux-update-symlinks` contains the following note: "This command is intended to be called from the postinst and postrm maintainer scripts of Linux kernel packages." It looks like `linux-update-symlinks` was only ever intended to be used in postinst/postrm hooks for .deb packages (and not called by `installkernel`).

There are a few remedies to this problem I can see, but I'm not sure which is the correct solution:
1. Add a call to `linux-update-symlinks` into `installkernel`
or
2. Add a script into /etc/kernel/postinst.d to run `linux-update-symlinks`
or
3. Run `linux-update-symlinks` manually after installing a kernel directly via `installkernel`

------- Comment From <email address hidden> 2020-07-20 12:57 EDT-------
(In reply to comment #19)
> On Fri, 26 Jun 2020 at 15:01, bugproxy <email address hidden> wrote:
> >
> > (In reply to comment #17)
> > > Eoan and later d-i, new installer, curtin do not install
> > > /etc/kernel-img.conf.
> > > Upgraded systems keep having it (ie. installed with bionic or xenial, and
> > > upgraded).
> > >
> > > Can you please let me know if _removing_ /etc/kernel-img.conf breaks $ sudo
> > > make install, and if adding /etc/kernel-img.conf back fixes $ sudo make
> > > install?
> > >
> > > Cause the expectation is that `/etc/kernel-img.conf` should not be there,
> > > yet everything should still work correctly.
> > >
> > > I think somewhere something is reading "link_in_boot=yes" and was not
> > > updated with the new implicit default to always assume that on recent ubuntu.
> >
> > For me the installkernel script (including in the current case "sudo
> > make install" from a mainline Linux tree) doesn't update the
> > /boot/initrd.img symlink.
> >
> > Interestingly a package upgrade for linux-generic does overwrite this
> > symlink.
>
> .deb package uses very different maintainer scripts / codepath, and is
> not the same operation as "sudo make install".
>
> I personally always build my kernels as debs, and install debs, rather
> than doing "sudo make install". But I am a distribution developer, and
> I care for .debs to work right. Kernel developers, I guess, are
> inverse, and care for "upstream" $ sudo make install to work.
>
> >
> > Not with the /etc/kernel-img.conf[0] and not without it either.
> > As before the /boot/vmlinuz link is always updated.
> >
>
> That is slightly concerning, as to how $ sudo make install, ever
> worked before..... Or what has changed since.
>
> Normally, $ sudo make install, should lookup if /sbin/installkernel is
> available, and call that to "do what it has to do, on a given
> distribution", and that script is shipped by the debianutils package
> which is required and must be installed always. And it hasn't been
> touched in ages.
>
> I wonder which arguments are passed to /sbin/installkernel by $ sudo
> make install. And whether the 4th argument is passed, and if it is
> empty, /, or /boot. It should be either empty, or /boot.
>
> --
> Regards,
>
> Dimitri.
So when I built on a beefy development server I usually copy a tar of the modules + bzImage + System.map and e.g.:

# sudo tar -xavf 5.8.0-rc6-06703-g93c67d794cfc.tar -C /usr/
# sudo installkernel 5.8.0-rc6-06703-g93c67d794cfc bzImage System.map /boot

that works well on RHEL, Fedora, SUSE and I'm pretty sure it worked on Ubuntu only a week prior to me opening this Bugzilla. I also looked in the history
and didn't find anything so yeah I'm pretty confused how that
used to work on Ubuntu but I've even bisected like that (on Ubuntu) so have used that approach dozens of times.

Of course for us working with multiple distros there's a lot of value in
this working the same on all of them and I think that's the main point
of the installkernel script.

Could someone try to put the attached script into /etc/kernel/postinst.d and let me know whether that make installkernel work? Thanks.

------- Comment From <email address hidden> 2020-12-07 05:17 EDT-------
(In reply to comment #23)
> Could someone try to put the attached script into /etc/kernel/postinst.d and
> let me know whether that make installkernel work? Thanks.

I can confirm this works with installkernel and the previously described workflow on Ubuntu 20.04.1! Thanks!

Proposed update to linux-base for hirsute.

Updating the proposed change. This addresses the following:
- Only run from the trigger stage when called from dpkg
- Pass the same environmental flag as update-initramfs does
  (in theory there never should be need to generate an initrd)
- Handle special case of installkernel called multiple times
  with the same version

I have uploaded linux-base_4.5ubuntu1.3 (bionic) and linux-base_4.5ubuntu3.2 (focal).

This bug was fixed in the package linux-base - 4.5ubuntu5

---------------
linux-base (4.5ubuntu5) hirsute; urgency=medium

  * Add kernel postinst hook to update initrd softlinks to match the kernel
    version targets (LP: #1877088).

 -- Stefan Bader <email address hidden> Wed, 03 Feb 2021 16:05:08 +0100

Hello bugproxy, or anyone else affected,

Accepted linux-base into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-base/4.5ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello bugproxy, or anyone else affected,

Accepted linux-base into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-base/4.5ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted linux-base (4.5ubuntu1.3) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.4/5.4.0-72.80~18.04.1 (i386)
linux-hwe-5.0/5.0.0-65.71 (i386)
linux-oem-osp1/5.0.0-1071.77 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-base

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

------- Comment From <email address hidden> 2021-04-27 09:16 EDT-------
I just tested the installkernel script on Ubuntu 20.04 + Focal Proposed
with linux-base 4.5ubuntu3.2.

I performed the following testing on a s390x z15 LPAR:

# 5.12.0-07600-g75079231a966.tar is a tar of the modules compiled on another system
% ls
5.12.0-07600-g75079231a966.tar System.map bzImage
% tar -xavf 5.12.0-07600-g75079231a966.tar -C /usr/
% installkernel 5.12.0-07600-g75079231a966 bzImage System.map /boot

After that with the previous linux-base version I see the following:
% ls -la /boot
total 68556
drwxr-xr-x 2 root root 4096 Apr 27 12:56 .
drwxr-xr-x 20 root root 4096 Apr 27 08:33 ..
-rw-r--r-- 1 root root 4475596 Apr 27 12:56 System.map-5.12.0-07600-g75079231a966
-rw------- 1 root root 3099673 Apr 12 15:12 System.map-5.4.0-72-generic
-rw------- 1 root root 45056 Apr 27 12:56 bootmap
-rw-r--r-- 1 root root 90468 Apr 12 15:12 config-5.4.0-72-generic
lrwxrwxrwx 1 root root 27 Apr 27 08:34 initrd.img -> initrd.img-5.4.0-72-generic
-rw-r--r-- 1 root root 20356043 Apr 27 12:56 initrd.img-5.12.0-07600-g75079231a966
-rw-r--r-- 1 root root 24196892 Apr 27 12:46 initrd.img-5.4.0-72-generic
lrwxrwxrwx 1 root root 27 Apr 27 08:34 initrd.img.old -> initrd.img-5.4.0-72-generic
lrwxrwxrwx 1 root root 34 Apr 27 12:56 vmlinuz -> vmlinuz-5.12.0-07600-g75079231a966
-rw-r--r-- 1 root root 9744384 Apr 27 12:56 vmlinuz-5.12.0-07600-g75079231a966
-rw------- 1 root root 8172088 Apr 12 16:56 vmlinuz-5.4.0-72-generic
lrwxrwxrwx 1 root root 24 Apr 27 08:34 vmlinuz.old -> vmlinuz-5.4.0-72-generic

Note the wrong initrd.img symlink still pointing at initrd.img-5.4.0-72-generic

After the upate and performing the same steps again I get:

% ls -la /boot
total 117188
drwxr-xr-x 2 root root 4096 Apr 27 13:13 .
drwxr-xr-x 20 root root 4096 Apr 27 08:33 ..
-rw-r--r-- 1 root root 4475596 Apr 27 13:13 System.map-5.12.0-07600-g75079231a966
-rw-r--r-- 1 root root 4475596 Apr 27 12:56 System.map-5.12.0-07600-g75079231a966.old
-rw------- 1 root root 3099673 Apr 12 15:12 System.map-5.4.0-72-generic
-rw------- 1 root root 3100152 Apr 14 16:35 System.map-5.4.0-73-generic
-rw------- 1 root root 45056 Apr 27 13:13 bootmap
-rw-r--r-- 1 root root 90468 Apr 12 15:12 config-5.4.0-72-generic
-rw-r--r-- 1 root root 90468 Apr 14 16:35 config-5.4.0-73-generic
lrwxrwxrwx 1 root root 27 Apr 27 13:03 initrd.img -> initrd.img-5.4.0-73-generic
-rw-r--r-- 1 root root 20355627 Apr 27 13:13 initrd.img-5.12.0-07600-g75079231a966
-rw-r--r-- 1 root root 24196892 Apr 27 12:46 initrd.img-5.4.0-72-generic
-rw-r--r-- 1 root root 24204650 Apr 27 13:04 initrd.img-5.4.0-73-generic
lrwxrwxrwx 1 root root 37 Apr 27 13:03 initrd.img.old -> initrd.img-5.12.0-07600-g75079231a966
lrwxrwxrwx 1 root root 34 Apr 27 13:13 vmlinuz -> vmlinuz-5.12.0-07600-g75079231a966
-rw-r--r-- 1 root root 9744384 Apr 27 13:13 vmlinuz-5.12.0-07600-g75079231a966
-rw-r--r-- 1 root root 9744384 Apr 27 12:56 vmlinuz-5.12.0-07600-g75079231a966.old
-rw------- 1 root root 8172088 Apr 12 16:56 vmlinuz-5.4.0-72-generic
-rw------- 1 root root 8172...

The second attempt seems to have been made without undoing the files and links created by the old setup. The cases to handle with that mix get rather hard to handle. So if possible I would rather ask for to check whether the updated script works in a clean environment forward before marking this as failed.

------- Comment From <email address hidden> 2021-04-28 06:44 EDT-------
(In reply to comment #36)
> The second attempt seems to have been made without undoing the files and
> links created by the old setup. The cases to handle with that mix get rather
> hard to handle. So if possible I would rather ask for to check whether the
> updated script works in a clean environment forward before marking this as
> failed.

I'll redo the test with a prestine (despite update to Focal Proposed) system, it should be noted though that at least the initrd.img symlink at the second attempt
was created by the normal kernel update. What other links/files should I look into?

The kernel installation via debs always created the link. Just the installkernel command which is not part or the kernel packages had the issue of only moving around the kernel links but not the initrd ones. The case which I was working on was having a system which only had kernels install via .debs and then would run installkernel (with the new postinst.d hook installed).

------- Comment From <email address hidden> 2021-04-29 04:39 EDT-------
Redid the test on a freshly installed Focal + Proposed + "apt update && apt upgrade". Still the same behavior:

root@t46lp77:~/testkernel# installkernel 5.12.0-07607-g8029c2a40054 bzImage System.map /boot
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.12.0-07607-g8029c2a40054 /boot/vmlinuz-5.12.0-07607-g8029c2a40054
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.12.0-07607-g8029c2a40054 /boot/vmlinuz-5.12.0-07607-g8029c2a40054
update-initramfs: Generating /boot/initrd.img-5.12.0-07607-g8029c2a40054
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: sdn.
Done.
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 5.12.0-07607-g8029c2a40054 /boot/vmlinuz-5.12.0-07607-g8029c2a40054
run-parts: executing /etc/kernel/postinst.d/update-notifier 5.12.0-07607-g8029c2a40054 /boot/vmlinuz-5.12.0-07607-g8029c2a40054
run-parts: executing /etc/kernel/postinst.d/zz-zipl 5.12.0-07607-g8029c2a40054 /boot/vmlinuz-5.12.0-07607-g8029c2a40054
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: sdn.
Done.
root@t46lp77:~/testkernel# ls -l /boot
total 103288
-rw-r--r-- 1 root root 4475592 Apr 29 08:35 System.map-5.12.0-07607-g8029c2a40054
-rw------- 1 root root 3099673 Apr 12 15:12 System.map-5.4.0-72-generic
-rw------- 1 root root 3100152 Apr 14 16:35 System.map-5.4.0-73-generic
-rw------- 1 root root 45056 Apr 29 08:36 bootmap
-rw-r--r-- 1 root root 90468 Apr 12 15:12 config-5.4.0-72-generic
-rw-r--r-- 1 root root 90468 Apr 14 16:35 config-5.4.0-73-generic
lrwxrwxrwx 1 root root 27 Apr 29 08:19 initrd.img -> initrd.img-5.4.0-73-generic
-rw-r--r-- 1 root root 20355770 Apr 29 08:36 initrd.img-5.12.0-07607-g8029c2a40054
-rw-r--r-- 1 root root 24195082 Apr 29 08:19 initrd.img-5.4.0-72-generic
-rw-r--r-- 1 root root 24199441 Apr 29 08:19 initrd.img-5.4.0-73-generic
lrwxrwxrwx 1 root root 27 Apr 28 15:27 initrd.img.old -> initrd.img-5.4.0-72-generic
lrwxrwxrwx 1 root root 34 Apr 29 08:35 vmlinuz -> vmlinuz-5.12.0-07607-g8029c2a40054
-rw-r--r-- 1 root root 9744384 Apr 29 08:35 vmlinuz-5.12.0-07607-g8029c2a40054
-rw------- 1 root root 8172088 Apr 12 16:56 vmlinuz-5.4.0-72-generic
-rw------- 1 root root 8172088 Apr 14 16:37 vmlinuz-5.4.0-73-generic
lrwxrwxrwx 1 root root 24 Apr 29 08:19 vmlinuz.old -> vmlinuz-5.4.0-73-generic

The vmlinuz link is updated correctly but the initrd.img links (initrd.img and initrd.img.old) are not.

Need to take a deeper look into it later but I really think I have added some output about checking the links to the /etc/kernel/postinst.d script which does not show in the output of the last comment. I think it did not do either in the previous attempts but I missed that back then.

Hm, I think I know what went wrong. In my local tree the new script is executable but that seems to get lost when going via debdiff. Right now it likely will start to work doing a "chmod a+x /etc/kernel/postinst.d/xx-update-initrd-links".

Working on fixed up package(s)...

------- Comment From <email address hidden> 2021-04-29 10:52 EDT-------
(In reply to comment #41)
> Hm, I think I know what went wrong. In my local tree the new script is
> executable but that seems to get lost when going via debdiff. Right now it
> likely will start to work doing a "chmod a+x
> /etc/kernel/postinst.d/xx-update-initrd-links".

Great find! After doing the chmod a+x and redoing installkernel the symlink is correct.

------- Comment From <email address hidden> 2021-04-29 10:53 EDT-------
Oh and I see the following output:

Testing softlinks in /boot/...
/boot/initrd.img invalid target (initrd.img-5.4.0-73-generic)
Softlink: /boot/initrd.img -> initrd.img-5.12.0-07607-g8029c2a40054
/boot/initrd.img.old invalid target (initrd.img-5.4.0-72-generic)
Softlink: /boot/initrd.img.old -> initrd.img-5.12.0-07607-g8029c2a40054.old

Hello bugproxy, or anyone else affected,

Accepted linux-base into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-base/4.5ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello bugproxy, or anyone else affected,

Accepted linux-base into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-base/4.5ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted linux-base (4.5ubuntu1.4) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.4/5.4.0-74.83~18.04.1 (amd64)
makedumpfile/1:1.6.5-1ubuntu1~18.04.6 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-base

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted linux-base (4.5ubuntu3.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-oem-5.6/5.6.0-1057.61 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#linux-base

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

------- Comment From <email address hidden> 2021-05-17 10:50 EDT-------
I just did an "apt update && apt upgrade" on a KVM guest with Ubuntu 20.04 that I had previously used installkernel on with an early version of your updated script:

Setting up linux-image-5.4.0-73-generic (5.4.0-73.82) ...
I: /boot/vmlinuz.old is now a symlink to vmlinuz-5.12.0-rc8-07595-g446009324fa8
I: /boot/initrd.img.old is now a symlink to initrd.img-5.12.0-rc8-07595-g446009324fa8
I: /boot/vmlinuz is now a symlink to vmlinuz-5.4.0-73-generic
I: /boot/initrd.img is now a symlink to initrd.img-5.4.0-73-generic
Setting up alsa-utils (1.2.2-1ubuntu2.1) ...
alsa-restore.service is a disabled or a static unit not running, not starting it.
alsa-state.service is a disabled or a static unit not running, not starting it.
Setting up procps (2:3.3.16-1ubuntu2.1) ...
Setting up bind9-dnsutils (1:9.16.1-0ubuntu2.8) ...
Setting up cloud-init (21.1-19-gbad84ad4-0ubuntu1~20.04.2) ...
Setting up linux-modules-extra-5.4.0-73-generic (5.4.0-73.82) ...
Setting up linux-image-generic (5.4.0.73.76) ...
Setting up linux-generic (5.4.0.73.76) ...
Setting up python3-distupgrade (1:20.04.32) ...
Setting up ubuntu-release-upgrader-core (1:20.04.32) ...
Setting up python3-update-manager (1:20.04.10.7) ...
Setting up update-manager-core (1:20.04.10.7) ...
Setting up update-notifier-common (3.192.30.7) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Processing triggers for rsyslog (8.2001.0-1ubuntu1.1) ...
Processing triggers for systemd (245.4-4ubuntu3.6) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for dbus (1.12.16-2ubuntu2.1) ...
Processing triggers for initramfs-tools (0.136ubuntu6.4) ...
update-initramfs: Generating /boot/initrd.img-5.12.0-rc8-07595-g446009324fa8 <--- this is the currently running kernel installed with installkernel
Not invoking zipl: initrd doesn't exist yet
Processing triggers for linux-image-5.4.0-73-generic (5.4.0-73.82) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.4.0-73-generic
Not invoking zipl: initrd doesn't exist yet
/etc/kernel/postinst.d/zz-zipl:
Not invoking zipl: initrd doesn't exist yet <--- but above it said it generated ?it!?

So usually I would expect that upgrading the kernel after having used installkernel will afterwards use the new distro kernel. Am I correct in that assumption? It might also be that this system is somehow in a weird state but would like confirmation either way.

Hello bugproxy, or anyone else affected,

Accepted linux-base into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-base/4.5ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello bugproxy, or anyone else affected,

Accepted linux-base into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-base/4.5ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello bugproxy, or anyone else affected,

Accepted linux-base into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-base/4.5ubuntu1.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted linux-base (4.5ubuntu1.4) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.4/5.4.0-74.83~18.04.1 (amd64)
makedumpfile/1:1.6.5-1ubuntu1~18.04.6 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-base

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted linux-base (4.5ubuntu3.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-oem-5.6/5.6.0-1057.61 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#linux-base

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Possibly a regression is being reported
https://bugs.launchpad.net/ubuntu/+source/linux-base/+bug/1929255

Please investigate, and if that needs fixing, please take appropriate action. If determined to not be a regression, please remove the validation-failed tags.

Ubuntu Kernels in their postinst call linux-update-symlinks $change $version $image_path to setup correct links.

This was not done by installkernel script, and a broken xx-update-initrd-links script is being added to postinst.d that does not correctly handle link_in_boot setting and breaks upgrades.

imho installkernel script should call linux-update-symlinks just like our own linux-image kernels do in the postinst.

All autopkgtests for the newly accepted linux-base (4.5ubuntu3.4) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-aws-5.8/5.8.0-1035.37~20.04.1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#linux-base

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hello bugproxy, or anyone else affected,

Accepted linux-base into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-base/4.5ubuntu1.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello bugproxy, or anyone else affected,

Accepted linux-base into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-base/4.5ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello bugproxy, or anyone else affected,

Accepted linux-base into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-base/4.5ubuntu5.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

@Niklas, would you please give it another try?

All autopkgtests for the newly accepted linux-base (4.5ubuntu3.5) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.8/5.8.0-55.62~20.04.1 (amd64)
linux-gcp-5.8/5.8.0-1032.34~20.04.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#linux-base

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted linux-base (4.5ubuntu1.6) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-oem-osp1/5.0.0-1071.77 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-base

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

------- Comment From <email address hidden> 2021-06-14 09:51 EDT-------
(In reply to comment #58)
> @Niklas, would you please give it another try?

I can confirm the installkernel script now works as expected on current Ubuntu 20.4 with Proposed enabled. Thank you!

Thank you Niklas, I'm adjusting the tags accordingly...

See verifications on https://bugs.launchpad.net/ubuntu/+source/linux-base/+bug/1929255 which cover this bug too.

This bug was fixed in the package linux-base - 4.5ubuntu5.2

---------------
linux-base (4.5ubuntu5.2) hirsute; urgency=medium

  * Rewrite xx-update-initrd-links to use linux-update-symlinks. This will
    now make installkernel behaviour match the linux-image-*.postinst
    behaviour exactly with respect to creating & updating vmlinu? and
    initrd.img symlinks in / or /boot. LP: #1929255

linux-base (4.5ubuntu5.1) hirsute; urgency=medium

  * Make the new posinst hook executable (the correct bits were lost when
    adding the scripts via debdiff) (LP: #1877088).
  * Check for update-initramfs being installed before running the postinst
    hook which updates the softlinks (LP: #1928700).

 -- Dimitri John Ledkov <email address hidden> Wed, 02 Jun 2021 09:40:13 +0100

The verification of the Stable Release Update for linux-base has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package linux-base - 4.5ubuntu3.5

---------------
linux-base (4.5ubuntu3.5) focal; urgency=medium

  * Rewrite xx-update-initrd-links to use linux-update-symlinks. This will
    now make installkernel behaviour match the linux-image-*.postinst
    behaviour exactly with respect to creating & updating vmlinu? and
    initrd.img symlinks in / or /boot. LP: #1929255

linux-base (4.5ubuntu3.4) focal; urgency=medium

  * Check for update-initramfs being installed before running the postinst
    hook which updates the softlinks (LP: #1928700).

linux-base (4.5ubuntu3.3) focal; urgency=medium

  * Make the new posinst hook executable (the correct bits were lost when
    adding the scripts via debdiff) (LP: #1877088).

linux-base (4.5ubuntu3.2) focal; urgency=medium

  [ Stefan Bader ]
  * Add kernel postinst hook to update initrd softlinks to match the kernel
    version targets (LP: #1877088).

 -- Dimitri John Ledkov <email address hidden> Wed, 02 Jun 2021 09:43:06 +0100

This bug was fixed in the package linux-base - 4.5ubuntu1.6

---------------
linux-base (4.5ubuntu1.6) bionic; urgency=medium

  * Rewrite xx-update-initrd-links to use linux-update-symlinks. This will
    now make installkernel behaviour match the linux-image-*.postinst
    behaviour exactly with respect to creating & updating vmlinu? and
    initrd.img symlinks in / or /boot. LP: #1929255

linux-base (4.5ubuntu1.5) bionic; urgency=medium

  * Check for update-initramfs being installed before running the postinst
    hook which updates the softlinks (LP: #1928700).

linux-base (4.5ubuntu1.4) bionic; urgency=medium

  * Make the new posinst hook executable (the correct bits were lost when
    adding the scripts via debdiff) (LP: #1877088).

linux-base (4.5ubuntu1.3) bionic; urgency=medium

  [ Stefan Bader ]
  * Add kernel postinst hook to update initrd softlinks to match the kernel
    version targets (LP: #1877088).

 -- Dimitri John Ledkov <email address hidden> Wed, 02 Jun 2021 09:44:21 +0100

------- Comment From <email address hidden> 2021-06-29 04:33 EDT-------
IBM bugzilla status->closed, Fix Released for all requested distros