Ubuntu
livecd-rootfs package

GCE shielded VM integrity monitoring reports errors

Bug #1960564 reported by Ivan Kapelyukhin
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
livecd-rootfs (Ubuntu)
Fix Released
Undecided
Ivan Kapelyukhin
Focal
Fix Released
Undecided
Ivan Kapelyukhin
Impish
Fix Released
Undecided
Ivan Kapelyukhin
Jammy
Fix Released
Undecided
Ivan Kapelyukhin

Bug Description

[Impact]

 * GCE shielded VM instances created from official Ubuntu images starting with focal get integrity monitoring errors after second reboot without any actions or changes by the user.

 * This is due to `initrdless_boot_fallback_triggered` variable in  /boot/grub/grubenv being set to 0 after first boot. /boot/grub/grubenv is empty in the image prior to boot.

[Test Plan]

 * To reproduce the bug:
   1. Create a GCE shielded VM instance with integrity monitoring enabled:
     a) focal:
       gcloud compute instances create \
         integrity-test-focal \
         --machine-type "n2d-standard-2" \
         --zone "europe-west1-d" \
         --maintenance-policy=TERMINATE \
         --image-family=ubuntu-2004-lts \
         --image-project=ubuntu-os-cloud \
         --service-account YOUR_SERVICE_ACCOUNT_EMAIL \
         --scopes https://www.googleapis.com/auth/logging.read \
         --shielded-integrity-monitoring \
         --shielded-secure-boot
      b) impish:
        gcloud compute instances create \
          integrity-test-impish \
          --machine-type "n2d-standard-2" \
          --zone "europe-west1-d" \
          --maintenance-policy=TERMINATE \
          --image-family=ubuntu-2110 \
          --image-project=ubuntu-os-cloud \
          --service-account YOUR_SERVICE_ACCOUNT_EMAIL \
          --scopes https://www.googleapis.com/auth/logging.read \
          --shielded-integrity-monitoring \
          --shielded-secure-boot
      c) jammy:
        gcloud compute instances create \
          integrity-test-jammy \
          --machine-type "n2d-standard-2" \
          --zone "europe-west1-d" \
          --maintenance-policy=TERMINATE \
          --image-family=ubuntu-2204-lts \
          --image-project=ubuntu-os-cloud-devel \
          --service-account YOUR_SERVICE_ACCOUNT_EMAIL \
          --scopes https://www.googleapis.com/auth/logging.read \
          --shielded-integrity-monitoring \
          --shielded-secure-boot
   2. SSH into the instance and reboot it: `sudo reboot`
   3. After the instance is rebooted, check integrity monitoring logs:
     a) The easy way -- SSH into the instance and run:
       curl -sSf https://raw.githubusercontent.com/ikapelyukhin/gce-integrity-tester/master/integrity.sh | bash
     b) Alternatively, see the logs in the web console: https://console.cloud.google.com/logs/query

 * To verify the fix:
   1. Build a custom image with the fixed version of `livecd-rootfs`
   2. Upload it to GCE
   3. Register it in GCE with the same secureboot DBX as the official images
   4. Create an instance
   5. Reboot it
   6. Check integrity logs

[Where problems could occur]

 * Any code that expects `initrdless_boot_fallback_triggered` to be explicitly 0
 would break.

[Other Info]

 * I will build and register custom images the same way official images are  built and registered by CPC.
 * I can also spin up instances created from official/custom images and provide SSH access to them on request for bug reproduction/fix verification.

See original description
Ivan Kapelyukhin (ikapelyukhin)
description: updated
Changed in livecd-rootfs (Ubuntu):
assignee: nobody → Ivan Kapelyukhin (ikapelyukhin)
Ivan Kapelyukhin (ikapelyukhin)
Changed in livecd-rootfs (Ubuntu Impish):
assignee: nobody → Ivan Kapelyukhin (ikapelyukhin)
Changed in livecd-rootfs (Ubuntu Focal):
assignee: nobody → Ivan Kapelyukhin (ikapelyukhin)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.756

---------------
livecd-rootfs (2.756) jammy; urgency=medium

  * Unset `initrdless_boot_fallback_triggered` in /boot/grub/grubenv instead
    of setting it to 0 when the fallback is not triggered to prevent integrity
    monitoring errors on GCE. (LP: #1960564)

 -- Ivan Kapelyukhin <email address hidden> Thu, 10 Feb 2022 23:45:57 +0100

Changed in livecd-rootfs (Ubuntu Jammy):
status: New → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Ivan, or anyone else affected,

Accepted livecd-rootfs into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.742.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in livecd-rootfs (Ubuntu Impish):
status: New → Fix Committed
tags: added: verification-needed verification-needed-impish
Changed in livecd-rootfs (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Ivan, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.40 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ivan Kapelyukhin (ikapelyukhin) wrote :

TEST CASE:
1. Built Focal and Impish GCE images from respective `livecd-rootfs` packages with the fix
2. Started instances from those images, rebooted, ran the `integrity-tester` script
3. Ran `basic-ubuntu` and `google-secure-boot` test suites against those images

VERIFICATION DONE
Tests are passing, integrity tester shows no errors in integrity logs after instance reboot.

tags: added: verification-done verification-done-focal verification-done-impish
removed: verification-needed verification-needed-focal verification-needed-impish
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Since this is livecd-rootfs and image builds work, no need to wait the 7-day period. Releasing.

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.742.4

---------------
livecd-rootfs (2.742.4) impish; urgency=medium

  * Unset `initrdless_boot_fallback_triggered` in /boot/grub/grubenv instead
    of setting it to 0 when the fallback is not triggered to prevent integrity
    monitoring errors on GCE. (LP: #1960564)

 -- Ivan Kapelyukhin <email address hidden> Mon, 14 Feb 2022 21:24:22 +0100

Changed in livecd-rootfs (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.664.40

---------------
livecd-rootfs (2.664.40) focal; urgency=medium

  * Unset `initrdless_boot_fallback_triggered` in /boot/grub/grubenv instead
    of setting it to 0 when the fallback is not triggered to prevent integrity
    monitoring errors on GCE. (LP: #1960564)

 -- Ivan Kapelyukhin <email address hidden> Mon, 14 Feb 2022 21:32:01 +0100

Changed in livecd-rootfs (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.