UC20 (amd64) FDE/Secure Boot issues device after installation

Hi Chris, can you take a look at this?

Are you able to boot a live image on this hardware and then run the tcglog-check binary from https://github.com/canonical/tcglog-parser, pasting the output here?

Sure thing. Here you go:

ubuntu@ubuntu:~/tcglog-parser/tcglog-check$ sudo ./main
Computed TPM_ALG_SHA1 for PE image /cdrom/casper/vmlinuz - file:72699a4a532c4af545c0ea966d418c22c475c0ec, authenticode:db06d7caf036dd6fb9792b388a838f42e432ef67
Computed TPM_ALG_SHA256 for PE image /cdrom/casper/vmlinuz - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:de733c38e021e73fc90c78f9b4c9ffffff3aa3d62e60efdd81fc3813a4cfaa2a
Computed TPM_ALG_SHA1 for PE image /cdrom/EFI/BOOT/BOOTx64.EFI - file:2ad829a51ffd926f4b804d85d164a57f0f74995f, authenticode:21e79438580ec89df674dfe12653d77d132c3936
Computed TPM_ALG_SHA256 for PE image /cdrom/EFI/BOOT/BOOTx64.EFI - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:2ea4cb6a1f1eb1d3dce82d54fde26ded243ba3e18de7c6d211902a594fe56788
Computed TPM_ALG_SHA1 for PE image /cdrom/EFI/BOOT/grubx64.efi - file:268b8d2f57bb3df30cc966886a12e317974797b0, authenticode:64810b19fb377bd33aa94f6b4fbd458a39d8bec3
Computed TPM_ALG_SHA256 for PE image /cdrom/EFI/BOOT/grubx64.efi - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:fef2a72dc053a712f1f9d7d32d9598125e6cf261a3ed796c8a554b7b84fb439e
Computed TPM_ALG_SHA1 for PE image /cdrom/EFI/BOOT/mmx64.efi - file:2d3a7b1d5f79dd56ce865d6f0c4508b0ce68a3c9, authenticode:4c375598332928f352bd8064a17ec360c86acf73
Computed TPM_ALG_SHA256 for PE image /cdrom/EFI/BOOT/mmx64.efi - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:c2b6a68b7a4030efbae07aa781b5a7961ca0509d22ab8dcbc813b3c34e559779

*** FAIL ***: The following events contain event data that was not in the expected format and could not be decoded correctly:
        - Event 2 in PCR 4 (type: EV_EFI_BOOT_SERVICES_APPLICATION): unexpected EOF
This might be a bug in the firmware or bootloader code responsible for performing these measurements.

*** FAIL ***: The following events have digests that aren't consistent with the data recorded with them in the log:
        - Event 1 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA1) - expected (from data): 59b86e718fb296e9931ca823ab09bd0dab668429, got: bc1258cfa942833c7e758c8f2812b8687c67dfb6
        - Event 1 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA256) - expected (from data): b084d0b7807d13753bf6f1144884347c31695e300bedc2f16e1d2c0434b3cf27, got: bc17ac79004d2aa93ad983671676ebb90ccd8cb1d99c71ca0ae2a49aa99b95e0
        - Event 2 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA1) - expected (from data): ea15ae0858309833746ced9615f6c4a16a40704a, got: 3a424f3592621f46533b8b17ea4a4b6f8ce985d0
        - Event 2 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA256) - expected (from data): 70a208b567678205fb2c655a6cb52b65cda7e5b586a66cd2641e0c836fd83612, got: af626ba2565a19f353d4b87b10a24586db1c2df8ae101915fa96987e022c1f9f
        - Event 3 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA1) - expected (from data): 4eb0b1b25779173ed54466f7ed60d5fe15ce98e9, got: aa0d000a6d68e40eb9d70739d29c43252cf7d5e8
        - Event 3 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA256) - expected (from data): 79499fa40d498ac9d17464108afeda197a4c26dc...

Crap! Ignore this. I think I disabled this in the BIOS to get UC20 to boot. I'll post an updated version shortly.

Here's the output:

ubuntu@ubuntu:~/tcglog-parser/tcglog-check$ sudo ./main
Computed TPM_ALG_SHA1 for PE image /cdrom/casper/vmlinuz - file:72699a4a532c4af545c0ea966d418c22c475c0ec, authenticode:db06d7caf036dd6fb9792b388a838f42e432ef67
Computed TPM_ALG_SHA256 for PE image /cdrom/casper/vmlinuz - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:de733c38e021e73fc90c78f9b4c9ffffff3aa3d62e60efdd81fc3813a4cfaa2a
Computed TPM_ALG_SHA1 for PE image /cdrom/EFI/BOOT/BOOTx64.EFI - file:2ad829a51ffd926f4b804d85d164a57f0f74995f, authenticode:21e79438580ec89df674dfe12653d77d132c3936
Computed TPM_ALG_SHA256 for PE image /cdrom/EFI/BOOT/BOOTx64.EFI - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:2ea4cb6a1f1eb1d3dce82d54fde26ded243ba3e18de7c6d211902a594fe56788
Computed TPM_ALG_SHA1 for PE image /cdrom/EFI/BOOT/grubx64.efi - file:268b8d2f57bb3df30cc966886a12e317974797b0, authenticode:64810b19fb377bd33aa94f6b4fbd458a39d8bec3
Computed TPM_ALG_SHA256 for PE image /cdrom/EFI/BOOT/grubx64.efi - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:fef2a72dc053a712f1f9d7d32d9598125e6cf261a3ed796c8a554b7b84fb439e
Computed TPM_ALG_SHA1 for PE image /cdrom/EFI/BOOT/mmx64.efi - file:2d3a7b1d5f79dd56ce865d6f0c4508b0ce68a3c9, authenticode:4c375598332928f352bd8064a17ec360c86acf73
Computed TPM_ALG_SHA256 for PE image /cdrom/EFI/BOOT/mmx64.efi - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:c2b6a68b7a4030efbae07aa781b5a7961ca0509d22ab8dcbc813b3c34e559779

*** FAIL ***: The following events contain event data that was not in the expected format and could not be decoded correctly:
        - Event 2 in PCR 4 (type: EV_EFI_BOOT_SERVICES_APPLICATION): unexpected EOF
        - Event 3 in PCR 4 (type: EV_EFI_BOOT_SERVICES_APPLICATION): unexpected EOF
This might be a bug in the firmware or bootloader code responsible for performing these measurements.

*** FAIL ***: The following events have digests that aren't consistent with the data recorded with them in the log:
        - Event 1 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA1) - expected (from data): 8a2ccbac4b83284d0d6825a8991b42a2186e4869, got: 451a83f1357281c682b33b61419d226496930aa8
        - Event 1 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA256) - expected (from data): ead7ec0773e4f89b6b759090c27413709826fa98117e155da816d5cfd3fb12e1, got: c63fe495ed401b545144139ff241a406752238805fdbfa5624cf4c956372206b
        - Event 2 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA1) - expected (from data): ea15ae0858309833746ced9615f6c4a16a40704a, got: 3a424f3592621f46533b8b17ea4a4b6f8ce985d0
        - Event 2 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA256) - expected (from data): 70a208b567678205fb2c655a6cb52b65cda7e5b586a66cd2641e0c836fd83612, got: af626ba2565a19f353d4b87b10a24586db1c2df8ae101915fa96987e022c1f9f
        - Event 3 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA1) - expected (from data): a73819c1d90ac44255797d59c875d2effac81931, got: 3e739613a7668fe78c24d34eaeb379f4660361f8
        - Event 3 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TP...

Thanks for that.

It's quite likely that the firmware on this device doesn't implement the 2.0 series of the TCG EFI Protocol specification (https://trustedcomputinggroup.org/wp-content/uploads/EFI-Protocol-Specification-rev13-160330final.pdf), which shim depends on for measuring the PE image hash of grub and the kernel to PCR4 (using the HashLogExtendEvent function with the PE_COFF_IMAGE flag).

In the scenario when this protocol is missing, shim will fall back to using the 1.2 version of the TCG EFI Protocol (https://trustedcomputinggroup.org/wp-content/uploads/TCG_EFI_Protocol_1_22_Final-v05.pdf) if it is available, but this lacks the functionality required to measure the PE image hash of grub and the kernel to PCR4 - instead, it only measures the file hash.

As we precompute PCR values in order to create a PCR policy for sealing, this firmware limitation means that we compute the wrong values, resulting in the behaviour you see here. This isn't a configuration we support at the moment, although we probably should detect this and degrade more gracefully.

So I got another BIOS update installed, but now it seems completely different:

$ sudo ./tcglog-check
Computed TPM_ALG_SHA1 for PE image /boot/vmlinuz-5.13.0-27-generic - file:53872abdfb202b87a11413519f5eb5e2900d3378, authenticode:c862a2d1716f186468e54fed425316f8250a7f16
Computed TPM_ALG_SHA256 for PE image /boot/vmlinuz-5.13.0-27-generic - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:a7d5f761243ea773732e0239fdfae4729c9c5f17b76dc2c330469b84f3f7e909
Computed TPM_ALG_SHA384 for PE image /boot/vmlinuz-5.13.0-27-generic - file:38b060a751ac96384cd9327eb1b1e36a21fdb71114be07434c0cc7bf63f6e1da274edebfe76f65fbd51ad2f14898b95b, authenticode:b70b604d5c2b68bb5db40fdf437d36a8a97495ffa797dc2458615849897cee0524fbc29fbd74bf8f7ca692a9581ed0cc
Computed TPM_ALG_SHA512 for PE image /boot/vmlinuz-5.13.0-27-generic - file:cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e, authenticode:ee3cf73ca8db7248018a5232eeff43f8a27b0129d3285404d1a61a64f1d18154bee030222a69bf95bab3a605a7ff705266810b990b25ba5fef678192bd47c22b
panic: crypto: requested hash function #0 is unavailable

goroutine 1 [running]:
crypto.Hash.New(0x0, 0xc00007d980, 0xc000092be0)
        /usr/lib/go-1.13/src/crypto/crypto.go:89 +0x102
github.com/canonical/go-efilib.ComputePeImageDigest(0x0, 0x6e3da0, 0xc0000100a8, 0x9b0ea0, 0xc0000da708, 0x4, 0x4, 0x15c, 0x0)
        /<email address hidden>/pe.go:81 +0x35b
main.populatePeImageDataCache.func1.1(0xc0000100a8, 0xc000018530, 0x5, 0x8, 0xc000016a60, 0x1f)
        /home/mvip/tcglog-parser/tcglog-check/main.go:90 +0x150
main.populatePeImageDataCache.func1(0xc000010098, 0x6807a3, 0x5, 0xc0000cba98, 0xc000018530, 0x5, 0x8)
        /home/mvip/tcglog-parser/tcglog-check/main.go:102 +0x326
main.populatePeImageDataCache(0xc000018530, 0x5, 0x8)
        /home/mvip/tcglog-parser/tcglog-check/main.go:105 +0x241
main.run(0x0, 0x0)
        /home/mvip/tcglog-parser/tcglog-check/main.go:375 +0x369
main.main()
        /home/mvip/tcglog-parser/tcglog-check/main.go:506 +0x26

What do I make out of this, @Chris?

Just for context -- i've reached out and raised this with the supplier. Awaiting to hear back from them.

Still waiting to hear back from the supplier for the initial hardware. However, Intel send me a NUC (NUC8CCHKR) for evaluation, and the FDE appears to be broken on that device too.

$ sudo ./tcglog-check
Computed TPM_ALG_SHA1 for PE image /boot/vmlinuz-5.13.0-27-generic - file:53872abdfb202b87a11413519f5eb5e2900d3378, authenticode:c862a2d1716f186468e54fed425316f8250a7f16
Computed TPM_ALG_SHA256 for PE image /boot/vmlinuz-5.13.0-27-generic - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:a7d5f761243ea773732e0239fdfae4729c9c5f17b76dc2c330469b84f3f7e909
Computed TPM_ALG_SHA1 for PE image /boot/vmlinuz-5.13.0-28-generic - file:225fed1aac9a89688173bb21b3393c6fdc1dc430, authenticode:801484ec3dfdb30a1295c257ed4af6a6691d52f0
Computed TPM_ALG_SHA256 for PE image /boot/vmlinuz-5.13.0-28-generic - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:070690ae2dca73a79a778ef4add00ecf153a6a66ddd20f634dc7e64dea793fe7
Computed TPM_ALG_SHA1 for PE image /boot/grub/x86_64-efi/grub.efi - file:6278ce0b9cf820dc10e96a87c6b25152050d55fc, authenticode:7361fcefef711b0815b4d9028d98c085945a636f
Computed TPM_ALG_SHA256 for PE image /boot/grub/x86_64-efi/grub.efi - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:2a198128d5454aaabf55ac163af6a317657d7d973c2ee93e83e006629004647d
Computed TPM_ALG_SHA1 for PE image /boot/grub/x86_64-efi/core.efi - file:902ba3dd6a2488aac4ed02d477b22c442f6efd5d, authenticode:8fb3e780e85228a09bdcecc3606ed32c1caa2a6b
Computed TPM_ALG_SHA256 for PE image /boot/grub/x86_64-efi/core.efi - file:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, authenticode:f83e067df62a714c42c85068a4f1ee8478dec7eaa0fbb4f9d64d97e315d843d9

*** FAIL ***: The following events contain event data that was not in the expected format and could not be decoded correctly:
        - Event 3 in PCR 4 (type: EV_EFI_BOOT_SERVICES_APPLICATION): unexpected EOF
This might be a bug in the firmware or bootloader code responsible for performing these measurements.

*** FAIL ***: The following events have digests that aren't consistent with the data recorded with them in the log:
        - Event 1 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA1) - expected (from data): d9ccf90ea9113dcf35370ce080ecd0f66a540f06, got: 6679efba8336b19e98fc6195a8fd18bc505a22d7
        - Event 1 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA256) - expected (from data): 91313f69d0d9647d5da8f7b3fed1d0e5a56f3ca868973de094aca1e171e7d323, got: 3693def930a10e0944ac211fb38db82ad88589e24068a9bccf1c466ca134de9f
        - Event 2 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA1) - expected (from data): 790e2271e196cdd81005611aad9276d5e3d9bb61, got: c4c34451e886471d98f8acddf0677e145fbc457c
        - Event 2 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA256) - expected (from data): 6874a299ea1196d5f92c167f9f326a285c6de79f28e0383e5909946e1e35dcb7, got: 2e35174c23e89968791e7e2b05606fd13feeb2b1d9ce2fed678cacbb7e9a9b86
        - Event 3 in PCR 1 (type: EV_EFI_VARIABLE_BOOT, alg: TPM_ALG_SHA256) - expected (from data): 62c82d5d136e0405177e45ddcce5cb62fc3b17b92f926a77524fa45da9da...

I had a chat with Ondra about this, and he asked me to check if this worked on Windows 11.

After installing Windows 11 on the device, I was able to enable BitLocker with the TPM as the backend without any issues. Please see attached screenshot.

Just an update on this. I had a few conversations with various people at Ubuntu Summit (and also included this in my talk).

After trying Core22 (amd64 image), I can confirm that this is yet to be resolved. The error is manifested slightly different in Core22 though. It initially manages to seal the partition(s), but on the next reboot, it is asking for the recovery key ('Please enter the recovery key for disk /dev/disk[...]').

I am seeing the same issue on a different device (Compulab Fitlet2). The device has a fTPM - (Intel Atom x5-E3930).

I’ve tried a variety of channels and kernel combinations. It seems that the generic UC22 image for amd64 results in DA lockout after the first boot that sets up the encrypted partitions. There is no error message indicating a failure to seal keys to the TPM. Only messages saying information is being sealed at 0x188001/0x188002 handles. However, I’ve pulled power after the initial boot completes and checked the TPM state on classic ubuntu to confirm that the TPM is in DA lockout (counter/limit are both at 20) before ever attempting a normal core22 boot.

Alternatively when using the intel-iot kernel, DA lockout does not occur during setup but rather after a handful of failed iterations of boots/attempts to mount the two encrypted partitions.

My guess is that the keys aren’t properly sealed to the fTPM in either case. My plan is to add some debugging statements to snapd to see if it’s possible to find out how that key material is maybe not persisted during setup.

I built an image with a custom snapd to determine if SealKeys() in secboot_tpm.go was actually successfully sealing anything to the handles...

 if !tpm.DoesHandleExist(tpm2.Handle(pcrHandle), tpm.HmacSession()) {
  logger.Noticef("Nothing was saved to this handle: : %#x", pcrHandle)
 } else {
  logger.Noticef("Key material was found @ %#x", pcrHandle)
 }

Every call to SealKeys() after setting up the encrypted partitions resulted in "Nothing was saved to this handle: 0x..."

I've also printed out the lockout count (TPM_PT_LOCKOUT_COUNTER) during each call to SealKeys and it's consistently 0. So, I'm not quite sure how that value ramps up to 20 at the end of the initial boot.

Additionally I've tried configuring an FDE/secboot image using qemu, ran "snap recover --show-keys" after successful setup, transferred the qemu initialized image to the device, and supplied the known recovery key to allow the boot process to continue. This worked but the TPM was not properly restored from recovery key and each boot cycle still prompted for recovery key.

I have also tried fTPM and discrete TPM variations of the same device.

// Output from the discrete TPM device
$ sudo ./tcglog-check

Any guidance would be very much appreciated!

Regards,
Ben

Apparently DoesHandleExist() doesn't do what I think it does because it returns false during a successful QEMU boot as well. However, I've made some progress in identifying where things are going wrong for my device. I was able to get the thing to boot after setting up the encrypted partitions by setting the PCRProfile param as nil and the PCRPolicyCounterHandle to tpm2.HandleNull in secboot_tpm.go in snapd as shown below:

 creationParams := sb_tpm2.KeyCreationParams{
  PCRProfile: nil,
  PCRPolicyCounterHandle: tpm2.HandleNull,
  AuthKey: params.TPMPolicyAuthKey,
 }

Obviously this worked because there is no longer PCR authorization while unsealing the keys on subsequent boots so the boot is no longer secure. At least I've pinned the issue down to a PCR policy violation though and can begin figuring out why that's happening by tinkering with the policy.

@chrisccoulson, your comment above in response to viktor's tcglogcheck output is exactly what's happening in the case of my device (Fitlet2)! During my first few reads of this thread, I didn't really understand enough about how uefi secure boot works (or what a PE image digest was) for that explanation to sink in.

The measurements extending PCR 4 in my device are as follows:
* pre-OS event: using digest from event log
* bootx64.efi - PE image digest
* grubx64.efi - file digest
* grubx64.efi - file digest (why does grub extend twice?)
* kernel.efi - file digest

I understand it's part of the latest TCG spec but I'm super curious how extending via a file digest vs a PE image digest impacts security of the device or if there's some other reason for using PE image digest.

The function responsible for computing PCR 4 to become part of the key's PCR policy is "AddBootManagerProfile" in snapcore/efi/bootmanager_policy.go. It gets used in buildPCRProtectionProfile in snapd's secboot_tpm.go.

I wrote a POC "AddFitlet2BootManagerProfile"(below) that supports the PCR 4 calculation as performed by my device. It looks in /snap/pc and /snap/pc-kernel for boot/grub and kernel combinations that could be used on subsequent boots.

/*
 This is a customized AddBootManagerProfile to work with fitlet2. Fitlet2 takes the PE image digest of bootx64.efi and the file
 digest of everything after that. The normal AddBootManagerProfile expects everything to be a PE image digest.
*/
func AddFitlet2BootManagerProfile(profile *secboot_tpm2.PCRProtectionProfile, params *sb_efi.BootManagerProfileParams) error {
 env := params.Environment
 if env == nil {
  env = defaultEnv
 }

 // Load event log
 log, err := env.ReadEventLog()
 if err != nil {
  return xerrors.Errorf("cannot parse TCG event log: %w", err)
 }

 if !log.Algorithms.Contains(params.PCRAlgorithm) {
  return errors.New("cannot compute secure boot policy digests: the TCG event log does not have the requested algorithm")
 }

 // Replay the event log until we see the transition from "pre-OS" to "OS-present". The event log may contain measurements
 // for system preparation applications, and spec-compliant firmware should measure a EV_EFI_ACTION “Calling EFI Application
 // from Boot Option” event before the EV_SEPARATOR event, but not all firmware does this.
 var preOSevents []tpm2.Digest
 for _, event := range log.Events {
  if event.PCRIndex != bootManagerCodePCR {
   continue
  }
  preOSevents = append(preOSevents, tpm2.Digest(event.Digests[params.PCRAlgorithm]))
  if event.EventType == tcglog.EventTypeSeparator { // The first event (event 0) is an EV_SEPARATOR
   break
  }
 }

 bootAndGrubEFIs, err := getBootGrubEFIPairs() // Get boot and grub efi pairs from snaps in /snap/pc
 if err != nil {
  return err
 }

 kernelPaths, err := getKernelEFIs() // Get kernel.efi's from snaps in /snap/pc-kernel
 if err != nil {
  return err
 }

 // Build up a PCR profile for every possible boot/grub and kernel combination
 var subProfiles []*secboot_tpm2.PCRProtectionProfile
 for _, bootGrubPair := range bootAndGrubEFIs {
  f, err := os.Open(bootGrubPair[0]) // This one is not a file hash but is actually a legit computePEim...