compress firmware in /lib/firmware

This bug was fixed in the package initramfs-tools - 0.140ubuntu8

---------------
initramfs-tools (0.140ubuntu8) jammy; urgency=medium

  * Cherry-pick merge request !26 to support xz compressed
    firmware. Uncompressed firmware is preffered, with a fallback to use
    compressed firmware files. LP: #1942260
  * Cherry-pick merge request !51 to decompress compressed kernel modules
    for boot speed performance. LP: #1932329
  * Cherry-pick partially merge request !50 to fix ftbfs with new
    shellcheck.
  * Fix shellcheck issues in ubuntu patches.

 -- Dimitri John Ledkov <email address hidden> Tue, 19 Oct 2021 11:49:38 +0100

Status changed to 'Confirmed' because the bug affects multiple users.

Fedora's individual binary firmware xz compression plans seem to be a success. They kept going and divided the package into many different firmware packages.

https://packages.fedoraproject.org/pkgs/linux-firmware/

https://packages.fedoraproject.org/pkgs/linux-firmware/intel-gpu-firmware/fedora-38-updates.html#files

Arch Linux has also done this since 2022-
https://archlinux.org/news/linux-firmware-202201190c6a7b3-2-requires-kernel-53-and-package-splitting/

Linux kernel 5.19 introduced ZSTD firmware compression-
https://wiki.gentoo.org/wiki/Linux_firmware#Optional:_Compression

All kernels need to support reading the compressed firmware files (even after partial upgrades). See bug #2028550 for the kernel side.

linux-firmware needs two patches to correctly produce compressed firmware files:

* https://lore.kernel.org<email address hidden>/T/#u
* https://lore.kernel.org<email address hidden>/T/#u

Attached a tested debdiff that could be uploaded once following points are addressed:
* support for zstd compressed firmware files in initramfs-tools (should be in 0.142ubuntu8)
* support for zstd compressed firmware in all relevant kernels (see bug #2028550)
* a "Breaks: initramfs-tools (<< 0.142ubuntu8~)" for the final uploaded version should be added

Both linux-firmware patches were accepted upstream. Upstream currently does not set a compression level and has no way to specify it.

Have you tried environment variables?

From the xz manpage:

ENVIRONMENT
       xz parses space-separated lists of options from the environment vari‐
       ables XZ_DEFAULTS and XZ_OPT, in this order, before parsing the options
       from the command line. Note that only options are parsed from the en‐
       vironment variables; all non-options are silently ignored. Parsing is
       done with getopt_long(3) which is used also for the command line argu‐
       ments.

       XZ_DEFAULTS
              User-specific or system-wide default options. Typically this is
              set in a shell initialization script to enable xz's memory usage
              limiter by default. Excluding shell initialization scripts and
              similar special cases, scripts must never set or unset XZ_DE‐
              FAULTS.

       XZ_OPT
              This is for passing options to xz when it is not possible to set
              the options directly on the xz command line. This is the case
              when xz is run by a script or tool, for example, GNU tar(1):

                     XZ_OPT=-2v tar caf foo.tar.xz foo

              Scripts may use XZ_OPT, for example, to set script-specific de‐
              fault compression options. It is still recommended to allow
              users to override XZ_OPT if that is reasonable. For example, in
              sh(1) scripts one may use something like this:

                     XZ_OPT=${XZ_OPT-"-7e"}
                     export XZ_OPT

From the zstd manpage:

   Environment Variables
       Employing environment variables to set parameters has security implica‐
       tions. Therefore, this avenue is intentionally limited. Only
       ZSTD_CLEVEL and ZSTD_NBTHREADS are currently supported. They set the
       compression level and number of threads to use during compression, re‐
       spectively.

       ZSTD_CLEVEL can be used to set the level between 1 and 19 (the "normal"
       range). If the value of ZSTD_CLEVEL is not a valid integer, it will be
       ignored with a warning message. ZSTD_CLEVEL just replaces the default
       compression level (3).

Then setting ZSTD_CLEVEL in debian/scripts/install-firmware should do the trick (needs testing!):

ZSTD_CLEVEL=19 ./copy-firmware.sh --zstd "${staging_dir}"

Unfortunately, that's not sufficient (include/exclude logic needs to be able to handle .zst files as well). Working on it.

With zstd compression, deb file size increases 267M -> 355M:
267M linux-firmware_20230731.git07f05b0c-0ubuntu1_amd64.deb
355M linux-firmware_20230731.git07f05b0c-0ubuntu1+zstd_amd64.deb

On-disk usage decreases 844M -> 367M:
844M /lib/firmware
367M /lib/firmware (w/ zstd)

Can you check what difference xz would make?

This bug was fixed in the package linux-firmware - 20230731.git07f05b0c-0ubuntu1

---------------
linux-firmware (20230731.git07f05b0c-0ubuntu1) mantic; urgency=medium

  * Miscellaneous Ubuntu changes
    - [Packaging] Compress all firmware license files
    - [Packaging] Check for /usr/bin/python3 in clean rule
    - Rebase to upstream commit 07f05b0c9750b4e3302473244400668dab70a91e
      Rebase against git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
      - qcom: Add Audio firmware for SC8280XP X13s
      - iwlwifi: update cc/Qu/QuZ firmwares for core80-39 release
      - iwlwifi: add new FWs from core80-39 release
      - check_whence: Check link targets are valid
      - Fix qcom ASoC tglp WHENCE entry
      - linux-firmware: Update firmware file for Intel Bluetooth AX201
      - linux-firmware: Update firmware file for Intel Bluetooth AX200
      - linux-firmware: Update firmware file for Intel Bluetooth AX210
      - linux-firmware: Update firmware file for Intel Bluetooth AX211
      - linux-firmware: Update firmware file for Intel Bluetooth AX211
      - linux-firmware: Update firmware file for Intel Bluetooth AX203
      - linux-firmware: Update firmware file for Intel Bluetooth AX203
      - linux-firmware: update firmware for MT7922 WiFi device
      - linux-firmware: update firmware for mediatek bluetooth chip (MT7922)
      - Partially revert "amdgpu: DMCUB updates for DCN 3.1.4 and 3.1.5"
      - cirrus: Add CS35L41 firmware for ASUS ROG 2023 Models
      - i915: update to GuC 70.8.0 and HuC 8.5.1 for MTL
      - i915: update DG2 GuC to v70.8.0
      - mediatek: Update mt8195 SCP firmware to support 10bit mode
      - wfx: update to firmware 3.16.1
      - Updated NXP SR150 UWB firmware
      - amdgpu: update DMCUB to v0.0.175.0 for various AMDGPU ASICs
      - amdgpu: update Picasso VCN firmware
      - amdgpu: update raven2 VCN firmware
      - amdgpu: update raven VCN firmware
      - amdgpu: update renoir VCN firmware
      - amdgpu: update green sardine VCN firmware
      - linux-firmware: Update AMD cpu microcode
      - linux-firmware: Update AMD fam17h cpu microcode
      - amdgpu: add initial GC 11.0.3 firmware
      - amdgpu: add initial PSP 13.0.10 firmware
      - amdgpu: add initial SDMA 6.0.3 firmware
      - amdgpu: add initial SMU 13.0.10 firmware
      - amdgpu: update VCN 4.0.0 firmware
      - qcom: sc8280xp: LENOVO: Remove execute bits
      - qcom: sc8280xp: LENOVO: remove directory sym link
      - copy-firmware: Fix test: unexpected operator
      - copy-firmware: Fix linking directories when using compression
      - cirrus: Add CS35L41 firmware for Dell Oasis Models
      - i915: Update ADLP DMC to v2.20
      - i915: Update MTL DMC to v2.13
      - ice: add LAG-supporting DDP package
    - SAUCE: Also check WHENCE.ubuntu for files to not compress
    - [Packaging] build-package: Drop sanity check
  * compress firmware in /lib/firmware (LP: #1942260)
    - SAUCE: Support additional compressor options
    - [Packaging] Enable zstd firmware compression
  * upgrading does not defer initramfs updates (LP: #1926608)
    - [Packaging] Use dpkg-trigger for updating the initramfs

 -- ...

This new linux-firmware is missing a "Breaks: initramfs-tools (<< 0.142ubuntu8~)" because initramfs-tools versions before 0.142ubuntu8 are missing support for zstd.

This bug was fixed in the package firmware-sof - 2.2.6-1ubuntu1

---------------
firmware-sof (2.2.6-1ubuntu1) mantic; urgency=medium

  * Compress sof firmware with zstd. LP: #1942260

 -- Dimitri John Ledkov <email address hidden> Tue, 29 Aug 2023 19:00:29 +0100