Activity log for bug #1978785

Date Who What changed Old value New value Message
2022-06-15 07:41:44 Robert Liu bug added bug
2022-06-15 07:41:44 Robert Liu attachment added 2022-06-15 15-13-58 screenshot.png https://bugs.launchpad.net/bugs/1978785/+attachment/5597386/+files/2022-06-15%2015-13-58%20screenshot.png
2022-06-15 07:42:04 Robert Liu bug task added oem-priority
2022-06-15 07:43:52 Yuan-Chen Cheng tags oem-priority
2022-06-15 07:45:23 Robert Liu attachment added state.json https://bugs.launchpad.net/oem-priority/+bug/1978785/+attachment/5597394/+files/state.json
2022-06-20 10:48:37 Michael Vogt snapd: status New Incomplete
2022-06-22 13:30:32 Robert Liu description [Summary] When a UC20 system refreshes kernel and core20 snaps at the same time, if the kernel is bad, the system will not boot (this is expected). After restarting the system, it will use the previous kernel snap. Then the system will not only revert the kernel snap, but also the core20 snap. It will reboot automatically then fail to boot. (see the attached screenshot. [Step to reproduce] 1. use https://cdimage.ubuntu.com/ubuntu-core/20/dangerous-beta/20220614.1/ as the testing image 2. follow https://ubuntu.com/download/kvm to boot the system and finish first-boot process 3. login the system and run these commands: # to have a new kernel snap $ snap switch pc-kernel --channel 22/stable "pc-kernel" switched to the "22/stable" channel # to have a new core20 snap $ snap switch core20 --edge "core20" switched to the "latest/edge" channel $ snap refresh --list Name Version Rev Size Publisher Notes core20 20220614 1550 64MB canonical✓ base pc-kernel 5.15.0-35.36.1 1021 289MB canonical✓ kernel $ snap refresh snapd is about to reboot the system # make a bad kernel $ cd /run/mnt/ubuntu-boot/EFI/ubuntu/pc-kernel_1021.snap/ $ sudo sh -c 'cat /dev/null > kernel.efi' $ sudo reboot 4. after reboot, GRUB will fail to boot. Close the qemu window and restart it 5. The system will continue the boot process and later show: error: base snap "core20_1550.snap" does not exist on ubuntu-data [Expected result] The system should boot correctly with the previous kernel. (I wonder if it make sense to revert core20 as well) [Actual result] The system fails to boot and there is no way to recover. [Fail rate] 100% [Additional information] I also observed this issue on different platform. [Summary] When a UC20 system refreshes kernel and core20 snaps at the same time, if the kernel is bad, the system will not boot (this is expected). After restarting the system, it will use the previous kernel snap. Then the system will not only revert the kernel snap, but also the core20 snap. It will reboot automatically then fail to boot. (see the attached screenshot. [Step to reproduce] 1. use https://cdimage.ubuntu.com/ubuntu-core/20/dangerous-beta/20220622.1/ubuntu-core-20-amd64.img.xz as the testing image 2. follow https://ubuntu.com/download/kvm to boot the system and finish first-boot process $ qemu-system-x86_64 -smp 2 -m 2048 -net nic,model=virtio -net user,hostfwd=tcp::8022-:22,hostfwd=tcp::8090-:80 -vga qxl -drive file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=ubuntu-core-20-amd64.img,cache=none,format=raw,id=disk1,if=none -device virtio-blk-pci,drive=disk1,bootindex=1 -machine accel=kvm 3. login the system and make sure all snaps are up to date before testing. $ sudo snap refresh 4. run these commands: # to have a new kernel snap $ snap switch pc-kernel --channel 22/stable "pc-kernel" switched to the "22/stable" channel # to have a new core20 snap # depends on the release, the channel could be stable or edge $ snap switch core20 --stable "core20" switched to the "latest/edge" channel $ snap refresh --list Name Version Rev Size Publisher Notes core20 20220614 1550 64MB canonical✓ base pc-kernel 5.15.0-35.36.1 1021 289MB canonical✓ kernel $ snap refresh snapd is about to reboot the system # postponse the shutdown $ sudo shutdown -c # make a bad kernel $ sudo sh -c "cat /dev/null > /run/mnt/ubuntu-boot/EFI/ubuntu/try-kernel.efi" $ sudo reboot 4. after reboot, GRUB will fail to boot. Close the qemu window and restart it 5. The system will continue the boot process, reboot again and later show: error: base snap "core20_[rev].snap" does not exist on ubuntu-data [Expected result] The system should boot correctly with the previous kernel. (I wonder if it make sense to revert core20 as well) [Actual result] The system fails to boot and there is no way to recover. [Fail rate] 100% [Additional information] I also observed this issue on different platform.
2022-06-22 13:36:02 Robert Liu attachment added state.json https://bugs.launchpad.net/snapd/+bug/1978785/+attachment/5598911/+files/state.json
2022-06-22 13:41:32 Robert Liu description [Summary] When a UC20 system refreshes kernel and core20 snaps at the same time, if the kernel is bad, the system will not boot (this is expected). After restarting the system, it will use the previous kernel snap. Then the system will not only revert the kernel snap, but also the core20 snap. It will reboot automatically then fail to boot. (see the attached screenshot. [Step to reproduce] 1. use https://cdimage.ubuntu.com/ubuntu-core/20/dangerous-beta/20220622.1/ubuntu-core-20-amd64.img.xz as the testing image 2. follow https://ubuntu.com/download/kvm to boot the system and finish first-boot process $ qemu-system-x86_64 -smp 2 -m 2048 -net nic,model=virtio -net user,hostfwd=tcp::8022-:22,hostfwd=tcp::8090-:80 -vga qxl -drive file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=ubuntu-core-20-amd64.img,cache=none,format=raw,id=disk1,if=none -device virtio-blk-pci,drive=disk1,bootindex=1 -machine accel=kvm 3. login the system and make sure all snaps are up to date before testing. $ sudo snap refresh 4. run these commands: # to have a new kernel snap $ snap switch pc-kernel --channel 22/stable "pc-kernel" switched to the "22/stable" channel # to have a new core20 snap # depends on the release, the channel could be stable or edge $ snap switch core20 --stable "core20" switched to the "latest/edge" channel $ snap refresh --list Name Version Rev Size Publisher Notes core20 20220614 1550 64MB canonical✓ base pc-kernel 5.15.0-35.36.1 1021 289MB canonical✓ kernel $ snap refresh snapd is about to reboot the system # postponse the shutdown $ sudo shutdown -c # make a bad kernel $ sudo sh -c "cat /dev/null > /run/mnt/ubuntu-boot/EFI/ubuntu/try-kernel.efi" $ sudo reboot 4. after reboot, GRUB will fail to boot. Close the qemu window and restart it 5. The system will continue the boot process, reboot again and later show: error: base snap "core20_[rev].snap" does not exist on ubuntu-data [Expected result] The system should boot correctly with the previous kernel. (I wonder if it make sense to revert core20 as well) [Actual result] The system fails to boot and there is no way to recover. [Fail rate] 100% [Additional information] I also observed this issue on different platform. [Summary] When a UC20 system refreshes kernel and core20 snaps at the same time, if the kernel is bad, the system will not boot (this is expected). After restarting the system, it will use the previous kernel snap. Then the system will not only revert the kernel snap, but also the core20 snap. It will reboot automatically then fail to boot. (see the attached screenshot. [Step to reproduce] 1. use https://cdimage.ubuntu.com/ubuntu-core/20/dangerous-beta/20220622.1/ubuntu-core-20-amd64.img.xz as the testing image 2. follow https://ubuntu.com/download/kvm to boot the system and finish first-boot process $ qemu-system-x86_64 -smp 2 -m 2048 -net nic,model=virtio -net user,hostfwd=tcp::8022-:22,hostfwd=tcp::8090-:80 -vga qxl -drive file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=ubuntu-core-20-amd64.img,cache=none,format=raw,id=disk1,if=none -device virtio-blk-pci,drive=disk1,bootindex=1 -machine accel=kvm 3. login the system and make sure all snaps are up to date before testing. $ sudo snap refresh 4. run these commands: # to have a new kernel snap $ snap switch pc-kernel --channel 22/stable "pc-kernel" switched to the "22/stable" channel # to have a new core20 snap # depends on the release, the channel could be stable or edge $ snap switch core20 --stable "core20" switched to the "latest/edge" channel $ snap refresh --list Name Version Rev Size Publisher Notes core20 20220614 1550 64MB canonical✓ base pc-kernel 5.15.0-35.36.1 1021 289MB canonical✓ kernel $ snap refresh snapd is about to reboot the system # postponse the shutdown $ sudo shutdown -c # make a bad kernel $ sudo sh -c "cat /dev/null > /run/mnt/ubuntu-boot/EFI/ubuntu/try-kernel.efi" $ sudo reboot 4. after reboot, GRUB will fail to boot. Close the qemu window and restart it 5. The system will continue the boot process, reboot again and later show:   error: base snap "core20_[rev].snap" does not exist on ubuntu-data example input and output: https://pastebin.ubuntu.com/p/z2R8bGf7Gr/ [Expected result] The system should boot correctly with the previous kernel. (I wonder if it make sense to revert core20 as well) [Actual result] The system fails to boot and there is no way to recover. [Fail rate] 100% [Additional information] I also observed this issue on different platform.
2022-06-27 20:59:19 John Agosta bug added subscriber John Agosta
2022-07-07 11:01:20 Alfonso Sanchez-Beato snapd: status Incomplete In Progress
2022-07-07 11:01:20 Alfonso Sanchez-Beato snapd: assignee Alfonso Sanchez-Beato (alfonsosanchezbeato)
2022-07-15 13:51:03 Michael Vogt snapd: status In Progress Fix Committed
2022-09-14 03:49:13 Rex Tsai oem-priority: importance Undecided Critical
2022-09-14 03:49:16 Rex Tsai oem-priority: status New Fix Committed
2022-11-11 10:30:46 Michael Vogt snapd: status Fix Committed Fix Released