[FFE] increase /boot partition size

Hello Brian, or anyone else affected,

Accepted partman-auto into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/partman-auto/134ubuntu13.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I think we need to make sure to release-note it for 20.04.4 in case we decide to go with it. I accepted this into focal-proposed so that we can have dailies built with this change and some validation done.

I suppose we'll also need to get this new version pulled into ubiquity?

Hi Łukasz and Brian: I have been doing quite a bit of research in this area recently, and also am advocating to get kernel management and cleanup improved, especially for users who must have separate /boot partitions. This means professional users who are required to have full disk encryption according to company IT policies.

Using best practice to manage 3 kernel packages (e.g. oem, lowlatency-hwe, generic-hwe) results in the need to maintain 6 kernel file sets (latest and penultimate version for each kernel). Because of the way kernel cleaning is scheduled, additional space is also required for at least 2 additional images. You can see the entire reasoning and details at LP: #1960089.

Our conclusion is that 2.0 GB is the preferred target /boot partition size for professional systems. This is reinforced by my research which shows many pros recommend this same size when discussing partitioning (Just one example: https://adventures-in-tech.blogspot.com/2018/10/encrypted-ubuntu-installation-with.html)

Given disk space these days, this seems like a small price to pay to support best practice and (hopefully) a future unattended upgrades algorithm that honors them as well.

Status changed to 'Confirmed' because the bug affects multiple users.

Hello Brian, or anyone else affected,

Accepted ubiquity into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubiquity/20.04.15.19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Design Failure Mode Effects Analysis

System: Boot

Potential Failure Mode: /boot partition overfills

Effects of Failure: To return the system to a usable state, the user must have advanced knowledge and an available system recovery disk. The procedure involves disk mounting, chroot, package management, and deep file system knowledge. This is outside the range of most target users.

Severity: 10 - system is completely unusable until recovery, and recovery is time consuming and tedious for advanced users, or impossible for less skilled users.

Causes of Failure:
1. Multiple kernel updates between reboots will overfill the standard 705M /boot partition with over 3 kernels.
2. User installing generic and lowlatency hwe kernels. They may also have transient kernels when switching to another kernel like oem.

Preventative Activities:
1. There does not appear to be any tool to guide kernel selection for users or ensure the latest and penultimate versions are reserved for the kernels.
2. There does not appear to be any testing which prevents installation of a kernel that would over-fill the disk.
3. unattended-upgrades tries to clean images that fill up the /boot disk, but it does not consider disk space, and even when it work (which is a whole other issue), the /boot disk is required to hold 4 images at times which is not feasible with the current 705M size.

Occurrence: 5 (even users with a singe Kernel flavor encounter this)

Detection Rating: 5 - Preventative Activities are unlikely to be effective at all times

Risk Priority Number: Severity * Occurrence * Detection = 250

Take Away Points:

* The DFMEA indicates this is a severe issue that should be considered critical.
* When an overfull /boot disk occurs, the effect is catastrophic to the the average user, and for many is simple unrecoverable. This can and does drive users to abandon the OS when this occurs.
* Existing controls to prevent occurrence are inadequate (automated upgrades still allows disk to over fill when using a single kernel flavor, and does not consider disk space) or completely missing (users are not guided on the issue of kernel management). The popularity of forum posts over the years about this issue illustrates this is a substantial problem.
* This issue goes beyond /boot partition size, but the increasing it to handle all possible transient states is required for a complete solution.
* Disk space is cheap these days. On consumer desktop solutions, 2.0GB is a small price to pay to avoid a catastrophic failure which is otherwise unrecoverable for many users.

I tested an installation of the daily image of Ubuntu 20.04 from 2022-02-11 (using lvm and encryption) and my boot partition has a size of 1.44G. This falls within the range we expected so I'm setting this to verification-done.

@kfocus Thank you for the detailed assessment! For 20.04 (at least for 20.04.4) we will only bump the partition sizes slightly, as mentioned in the current description (and in the current SRU). We need to thread carefully as it's a point-release of an already released series. As for 22.04, we are certainly open to discussion and your findings will be very helpful to make the best decision.

Let's see how it goes.

@sil2100 @brian-murray: Thank you for your work and consideration on this. We addressed this issue Kubuntu Focus OEM ISO (https://kfocus.org/try) by increasing the /boot partition size and providing an app that monitors and removes unused kernels (https://kfocus.org/wf/tools#kclean). We want to share our findings upstream for the benefit of everyone that uses client Ubuntu with FDE. In summary, an additional 1.2G of disk space is a very small price to pay to avoid the extreme expense of guiding non-expert users to recover from an over-full partition, or losing a user because they aren't capable of recovery by themselves.

The verification of the Stable Release Update for partman-auto has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package partman-auto - 134ubuntu13.1

---------------
partman-auto (134ubuntu13.1) focal; urgency=medium

  * Bump minimum and maximum sizes for /boot partitions on default, x86, and
    EFI platforms to a minimum of 768Mb and maximum of 1536Mb. (LP: #1959971)

 -- Brian Murray <email address hidden> Thu, 03 Feb 2022 15:02:21 -0800

This bug was fixed in the package ubiquity - 20.04.15.19

---------------
ubiquity (20.04.15.19) focal; urgency=medium

  * Backport the ability to unhide passwords that are being typed-in during
    various stages of the installation (LP: #1960306).
  * Automatic update of included source packages: partman-auto
    134ubuntu13.1 (LP: #1959971).

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 08 Feb 2022 11:51:13 +0100

For Subiquity, this has been implemented:
https://github.com/canonical/subiquity/blob/caab859a4f07d0cf39baeb53512d61b632d7c362/subiquity/common/filesystem/manipulator.py#L46

I think this makes sense. We had a similar change done to focal already for 20.04.4 so jammy should be good in my eyes as well. Approved.

This bug was fixed in the package ubiquity - 22.04.8

---------------
ubiquity (22.04.8) jammy; urgency=medium

  * In partman-auto bump the minimum and maximum sizes for /boot partitions on
    default, x86, and EFI platforms to a minium of 1024Mb and a maximum of
    1848Mb. (LP: #1959971)
  * After copying log files to /var/log/installer set the permissions to 0o640
    and change the group owner to adm.
  * Resolve a crash in the KDE frontend when trying to install Kubuntu which
    was caused by changes to division. Thanks to LP user Cyril for the patch.
    (LP: #1963697)

 -- Brian Murray <email address hidden> Tue, 22 Mar 2022 10:35:07 -0700

ZFS installation using ubiquity still uses smaller 1GiB sized bpool (equivalent of /boot on zfs)

(In case anyone else finds this bug when trying to track down the current recommended partition sizes):

I noticed there was another commit on 2022-04-01 for Ubiquity/partman-auto which referenced this bug:

https://git.launchpad.net/ubiquity/commit/?id=9d1720f9d749885eb13e4bd9e23e5a832219a3ae
  In partman-auto further increase the minimum and maximum sizes for /boot partitions on default, x86, and
  EFI platforms to a minium of 1792Mb and a maximum of 2048Mb. (LP: #1959971)

A corresponding commit was made to Subiquity on 2022-4-11:

https://github.com/canonical/subiquity/commit/22fa581b66cd76baf7bb8c16b5c4e3ea01c0208f
  boot: further increase suggested sizing: min 1.7G

  Match the change implemented for Ubiquity 22.04.11,
  commit 9d1720f9d749885eb13e4bd9e23e5a832219a3ae.

(The 2GiB size is what Subiquity 23.04.2 proposed for for me today for my LVM install of Jammy.)

(And ZFS setup in Ubiquity was also updated to match:

https://git.launchpad.net/ubiquity/commit/scripts/zsys-setup?id=36b3970bfb903fa67186b09863c50ac55aaecc44

  zsys-setup: increase minimum bpool partition size to 1792
  This matches zfs bpool size to /boot size as produced by partman-auto.

https://git.launchpad.net/ubiquity/commit/?id=21ebbdf8c087da9e01394d80a3924c2d4f161753
  zsys: fix comment

  bpool minimum size was previously increased to 1.75G. Update comment to match.
)

I'm not sure if this helps to have another data point, but I think I'm running into this bug. I originally installed Ubuntu 20.04 on a 1 TB drive and chose to have it encrypted. It looks like the installation created a 704 MB /boot:

$ df -h /boot
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p2 704M 564M 89M 87% /boot

... which of course now is causing me all kinds of headaches. Today, the linux-firmware package is failing to install because there's not enough space in /boot.

This is all the more absurd given that I have a 1 TB hard drive. Surely the installation could take total drive size into account when creating the /boot partition?

This is also compounded by the fact that apt currently has 4 kernels installed. I thought the max was 3? But that's off-topic.