[SRU] CM4 NVMe boot fails due to old start4.elf / fixup4.dat

Status changed to 'Confirmed' because the bug affects multiple users.

Marking linux-firmware-raspi2 invalid for now; will be used if we decide to backport to focal. Currently prepping an upload for ppa:waveform/firmware with a test updated firmware. If that works well, will upload to lunar then start SRU to kinetic and jammy.

Test firmware packages now built in ppa:waveform/firmware, including tentative backports for kinetic and jammy:

https://launchpad.net/~waveform/+archive/ubuntu/firmware

I would be grateful if anyone could test these on their particular Pi setups, whether using NVMe or otherwise (to guard against regressions). I've tested locally on a jammy and kinetic setup without issue so far, booting one from USB and one from SD card but I don't have a CM4 NVMe setup to test with.

Theoretically we *should* be testing this on lunar, but there aren't any daily images for that yet and given this will need SRU'ing anyway (and the boot firmware is *relatively* disconnected from the release it happens to be installed upon) we can take jammy and kinetic tests as indicative.

I would recommend the following testing procedure (given we're messing with the boot partition), if you care about the content of your Pi (i.e. assuming this is anything other than an ephemeral installation):

* Shut down your Pi

* Connect your current storage to another computer, and backup the entirety of the boot partition. Assuming it mounts as /media/$USER/system-boot the following should suffice:

  $ tar -C /media/$USER/system-boot -cvzf ~/system-boot-backup.tar.gz .

* Safely remove the storage, connect it back to your Pi, then boot it

* Add the PPA specified above:

  $ sudo add-apt-repository -y ppa:waveform/firmware

* Upgrade the linux-firmware-raspi package and reboot:

  $ sudo apt install linux-firmware-raspi
  $ sudo reboot

* Please report success or failure here (including release tested, boot media used, and model of Pi)

* If everything breaks, the worst-case scenario (short of actually catching fire!) is your Pi cannot boot. To restore the old content of the boot partition, attach your storage to the machine on which you took the backup. Again, assuming the boot partition is mounted as /media/$USER/system-boot:

  $ tar -C /media/$USER/system-boot -xvzf ~/system-boot-backup.tar.gz

* Safely remove the storage, re-connect it to the Pi, and boot it

* Note at this point we've restored the boot partition, but the newer package is still installed, as is the PPA. This means any kernel upgrade (or anything that causes flash-kernel to run) will once again copy the new firmware over to the boot partition. We need to correct that:

  $ sudo add-apt-repository -r -y ppa:waveform/firmware
  $ sudo apt install linux-firmware-raspi/$(lsb_release -sc)

* At this point your Pi should be back to the currently shipping linux-firmware-raspi package on jammy/kinetic/lunar

I have some CM4 with NVMe, so I can try this week.

@ww8987 Excellent, thank you -- I look forward to hearing any results!

Hello,

I have tested nvme boot (one CM4 on tofu board - https://store.oratek.com/products/tofu). All is OK, boot successful :)

Thanks very much

Hi, I have just tested it on CM4 with NVMe. Everything works!!! Thanks a lot! I have tested on https://uplab.pro/2022/03/compute-blade-changelog/ if you interesting.

Excellent news; combined with the testing I've done here on a few boards & archs I'd say this is good to land. I'll upload to lunar and convert this ticket for SRU to kinetic & jammy. Many thanks for all the testing help!

This bug was fixed in the package linux-firmware-raspi - 8-0ubuntu1

---------------
linux-firmware-raspi (8-0ubuntu1) lunar; urgency=medium

  * New upstream boot firmware, 1.20220331
    - Fixes issues with NVME boot (LP: #1993937, LP: #1989713)
  * New upstream wifi firmware, 20210315-3+rpt7

    [ Serge Schneider ]
    - Add update-alternatives for cyfmac43455-sdio.bin

    [ Phil Elwell ]
    - Add license from Synaptics
    - Update SYN43436P firmware
      - The shipping firmware for the SYN43436P does not support 4-way
        handshake offloading. This new firmware (version string "Version:
        9.88.4.77 CRC: 143f9f15 Date: Thu 2022-03-31 17:25:16 CST Ucode
        Ver: 1043.20743 FWID: 01-3b307371") fixes that.
    - Updated SYN43436S firmware
      - The embedded clm_blob in the previous SYN43436S firmware did not offer
        any channels when the country code was set to KR (Korea). This
        firmware fixes that.
      - Full version string:
        Version: 7.45.96.141 (g53ba0094) CRC: 6a9f22a0 Date: Tue 2022-07-05
        21:30:23 CST Ucode Ver: 1043.20739 FWID: 01-eef9b3bf

  * New upstream bluetooth firmware release

    [ Phil Elwell ]

    - This updated Bluetooth firmware contains Braktooth fixes for CYW43455,
      addressing the following vulnerabilities:
      CVE-2021-34145
      CVE-2021-34146
      CVE-2021-34147
      CVE-2021-34148
    - Add Synaptics firmware:
      SYN43430A1.hcd and SYN43430B0.hcd are firmwares for the two Zero 2 W
      variants.
    - Add board-specific links for the Zero 2 W firmwares

 -- Dave Jones <email address hidden> Thu, 28 Jul 2022 13:50:22 +0100

That took a while to migrate -- sorry about the delay! I'll get on with prepping the SRUs this week.

Hello ww898, or anyone else affected,

Accepted linux-firmware-raspi into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware-raspi/8-0ubuntu1~22.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Dave - are we sure we can safely drop linux-firmware-raspi2 from jammy as well?

Hello,

I am using the patched firmware for several weeks on Ubuntu Jammy (CM4). I don't have any problem. I prefer a lot to use LTS release than non LTS ;)

Regards

> Dave - are we sure we can safely drop linux-firmware-raspi2 from jammy as well?

Oh damn, I did drop that in the backport, didn't I. No, it's not safe to drop that package -- it needs to be there for upgraders. Can you reject that upload, and I'll re-push it with the transitional package re-instated?

Sure thing! Rejecting ;)

Thanks! And thank you "git revert" for making it such a trivial task to re-instate :)

It should be re-uploaded now, with the transitional package re-instated.

Hello ww898, or anyone else affected,

Accepted linux-firmware-raspi into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware-raspi/8-0ubuntu1~22.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Thanks to @francois-bonelle and @ww8987 for all their testing help!

I've also tested on several Pi models here (both SD and SSD booting, and boot + wifi + bt functionality although I only tested the bluetooth bits on a couple of models but still should have covered both the 4343x and 4345x chipsets). Anyway, no issues here, so I'm going to mark the verification as done on kinetic and jammy at this point.

This bug was fixed in the package linux-firmware-raspi - 8-0ubuntu1~22.10

---------------
linux-firmware-raspi (8-0ubuntu1~22.10) kinetic; urgency=medium

  * New upstream boot firmware, 1.20220331
    - Fixes issues with NVME boot (LP: #1993937, LP: #1989713)
  * New upstream wifi firmware, 20210315-3+rpt7

    [ Serge Schneider ]
    - Add update-alternatives for cyfmac43455-sdio.bin

    [ Phil Elwell ]
    - Add license from Synaptics
    - Update SYN43436P firmware
      - The shipping firmware for the SYN43436P does not support 4-way
        handshake offloading. This new firmware (version string "Version:
        9.88.4.77 CRC: 143f9f15 Date: Thu 2022-03-31 17:25:16 CST Ucode
        Ver: 1043.20743 FWID: 01-3b307371") fixes that.
    - Updated SYN43436S firmware
      - The embedded clm_blob in the previous SYN43436S firmware did not offer
        any channels when the country code was set to KR (Korea). This
        firmware fixes that.
      - Full version string:
        Version: 7.45.96.141 (g53ba0094) CRC: 6a9f22a0 Date: Tue 2022-07-05
        21:30:23 CST Ucode Ver: 1043.20739 FWID: 01-eef9b3bf

  * New upstream bluetooth firmware release

    [ Phil Elwell ]

    - This updated Bluetooth firmware contains Braktooth fixes for CYW43455,
      addressing the following vulnerabilities:
      CVE-2021-34145
      CVE-2021-34146
      CVE-2021-34147
      CVE-2021-34148
    - Add Synaptics firmware:
      SYN43430A1.hcd and SYN43430B0.hcd are firmwares for the two Zero 2 W
      variants.
    - Add board-specific links for the Zero 2 W firmwares

 -- Dave Jones <email address hidden> Thu, 28 Jul 2022 13:50:22 +0100

The verification of the Stable Release Update for linux-firmware-raspi has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package linux-firmware-raspi - 8-0ubuntu1~22.04

---------------
linux-firmware-raspi (8-0ubuntu1~22.04) jammy; urgency=medium

  * New upstream boot firmware, 1.20220331
    - Fixes issues with NVME boot (LP: #1993937, LP: #1989713)
  * New upstream wifi firmware, 20210315-3+rpt7

    [ Serge Schneider ]
    - Add update-alternatives for cyfmac43455-sdio.bin

    [ Phil Elwell ]
    - Add license from Synaptics
    - Update SYN43436P firmware
      - The shipping firmware for the SYN43436P does not support 4-way
        handshake offloading. This new firmware (version string "Version:
        9.88.4.77 CRC: 143f9f15 Date: Thu 2022-03-31 17:25:16 CST Ucode
        Ver: 1043.20743 FWID: 01-3b307371") fixes that.
    - Updated SYN43436S firmware
      - The embedded clm_blob in the previous SYN43436S firmware did not offer
        any channels when the country code was set to KR (Korea). This
        firmware fixes that.
      - Full version string:
        Version: 7.45.96.141 (g53ba0094) CRC: 6a9f22a0 Date: Tue 2022-07-05
        21:30:23 CST Ucode Ver: 1043.20739 FWID: 01-eef9b3bf

  * New upstream bluetooth firmware release

    [ Phil Elwell ]

    - This updated Bluetooth firmware contains Braktooth fixes for CYW43455,
      addressing the following vulnerabilities:
      CVE-2021-34145
      CVE-2021-34146
      CVE-2021-34147
      CVE-2021-34148
    - Add Synaptics firmware:
      SYN43430A1.hcd and SYN43430B0.hcd are firmwares for the two Zero 2 W
      variants.
    - Add board-specific links for the Zero 2 W firmwares

 -- Dave Jones <email address hidden> Thu, 28 Jul 2022 13:50:22 +0100