Unexpected / unwanted unattended-upgrades behaviour after kernel upgrade when Livepatch enabled

Status changed to 'Confirmed' because the bug affects multiple users.

How updates and optional reboot requests are surfaced is a pretty complex, I do not feel we can adequately address this in a bug report, it's part of a much wider story about the Pro client.

I'll go ahead and assign this to server as they own the pro client integration bits and are in the process of unifying that all, because I think they need to think about this for a bit.

If I may: I don't think this is a hugely technical issue. It comes down to unintended consequences caused by the fix for bug #1747499 - while its purpose was to resolve confusing user messaging, it probably took the wrong approach.

A straightforward fix would be to:
1. Revert the changes from bug #1747499.
2. Change the "System restart required" text applied by /usr/share/update-notifier/notify-reboot-required to something more specific and user-friendly, like "System restart required to finish applying updates."
3. Change the output from canonical-livepatch kernel-upgrade-required so that it doesn't reference restarting the system. (As we've seen this month - July 2023 - the message isn't always valid anyway.) For example the text could be simply: "Livepatch has fixed kernel vulnerabilities. Kernel upgrade recommended when available."

I believe that's all that is required, although I do understand that it's complicated somewhat by the functionality being split across different packages with different people responsible, and that it involves internationalised strings.

Thanks mev.

I wonder if there's a separate, deeper question here rather than just the messaging. How does one configure the system so that it automatically reboots when required, except when livepatch is sufficient? If that's not what you were raising, then perhaps we should have a separate bug to track that question.

Hi Robie -

I think that's a separate issue!

As far as I know, there's no way of measuring the sufficiency of Livepatch. A new kernel will usually have fixes for other bugs and less-impactful vulnerabilities on top of what Livepatch is taking care of, and it's impossible to know whether those fixes will be of significance to the user.

If a user wants to rely on Livepatch entirely (or 'until further notice'), unattended-upgrades can be set to blacklist automatic kernel upgrades.

I certainly don't see any value, in any circumstance, of installing a new kernel automatically but not alerting the user that it's not running, which is one of the things this bug is about, as well as failing to reboot automatically when unattended-upgrades has been set to do so.

(I'm not on the livepatch team)

> How does one configure the system so that it automatically reboots when required, except when livepatch is sufficient?

Livepatches for kernel flaws come after updated kernel packages: if you always reboot into a new kernel the day the kernel is delivered, you'll never install livepatches.

> If a user wants to rely on Livepatch entirely (or 'until further notice'), unattended-upgrades can be set to blacklist automatic kernel upgrades.

I would like to discourage this. Livepatch addresses only issues that (a) have had a CVE number assigned (b) have been scored as high or critical by our triage process. A lot of kernel security issues never get CVE numbers assigned. When we import new stable kernels from upstream developers, there may be security fixes that aren't actually called out as security fixes anywhere.

While it might seem wasteful to download a few hundred megabytes and rebuild an initramfs every two weeks regardless if you plan to reboot into it, unexpected reboots *do* happen, and it would be nice to get the most up-to-date kernel when that happens.

(Maybe an organization would rather have the entire fleet move along in a cadence, or only deploy kernels once they've been vetted through A/B testing or a staging environment, but that's an entirely separate choice to make. These organizations may disable unattended-upgrades entirely, or own their own repositories and control when packages are released into it, etc.)

Thanks

> Livepatches for kernel flaws come after updated kernel packages: if you always reboot into a new kernel the day the kernel is delivered, you'll never install livepatches.

Not always, and that's a problem with the current motd warning when Livepatch is operational, as described above. Last month (July 2023), for example, the Livepatch came before the availability of the kernel upgrade, at least on the Azure variant kernel (and possibly others).

Regardless I'm not sure if this provides any additional information/context, I faced this issue during using Cloudron.io as a paying customer (see forum thread https://forum.cloudron.io/topic/9720/reboot-for-security-updates-broken-on-livepatch/) on our Ubuntu 22 LTS server. So the current bug/change in behavior has a productive impact: Due to this, Cloudron doesn't detect anymore that a reboot is required for patching the OS and thus no notification is shown to the admin of the Cloudron instance. The workaround is to logon via shell onto the server and see the reboot hint in the welcome message.

See also bug 1998947.

FYI I'm trying to bring together all the people being involved with the last two changes in that area to avoid fixing this one case but thereby breaking yet another use-case. But it starts rather slowly ... :-/

There has been a discussion between the several engineering teams involved regarding a path forward on this bug. It's acknowledged that this is a regression which breaks the statement of user intent expressed by setting the Unattended-Upgrade::Automatic-Reboot option. However, this is an uncommon configuration (if you are configuring your system to automatically reboot on kernel upgrade, you are getting very little value out of livepatch on top of that since in the common case the kernel deb is made available to the system before the corresponding livepatch) so most users are not expected to be using both in combination. And we have agreed that we should be getting richer information from canonical-livepatch about the security bug coverage of available livepatches vs kernel debs that will allow a more nuanced decision about whether to tell the user a reboot is required. This work is planned for the 24.04 development cycle and we believe will cover this regression as well; so a targeted fix for this issue is not currently a priority.

Dropping the rls-ff-incoming in favor of adding a task (and one for jammy) so it is out of view until nn cycle.

Status changed to 'Confirmed' because the bug affects multiple users.