Unable to deploy 16.04 w/ Xenial HWE to system with Secure Boot active

The "Installation" tab/log on the web UI shows this:

curtin: Installation started. (0.1.0~bzr480-0ubuntu1~ubuntu16.04.1)
blockdev: ioctl error on BLKRRPART: Device or resource busy
Creating new GPT entries.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
blockdev: ioctl error on BLKRRPART: Device or resource busy
/dev/sda missing partitions: 15
Unexpected error while running command.
Command: ['partition', '--format=uefi', '/dev/sda']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
curtin: Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'block-meta', 'simple']
Exit code: 3
Reason: -
Stdout: blockdev: ioctl error on BLKRRPART: Device or resource busy
        Creating new GPT entries.
        Warning: The kernel is still using the old partition table.
        The new table will be used at the next reboot or after you
        run partprobe(8) or kpartx(8)
        The operation has completed successfully.
        blockdev: ioctl error on BLKRRPART: Device or resource busy
        /dev/sda missing partitions: 15
        Unexpected error while running command.
        Command: ['partition', '--format=uefi', '/dev/sda']
        Exit code: 1
        Reason: -
        Stdout: ''

MAAS logs

Please attach the output of:

maas <user> machine get-curtin-config <system_id_of_machine>

I recommissioned and then tried redeploying thinking that may need to be done so MAAS knows it's a SB enabled system. So after re-commissioning successfully, again, the deployment seems to go fine, but on reboot, it can not boot. The error is a little more clear this time though:

Booting local disk...
error: no such device: /efi/ubuntu/shimx64.efi
error: File not found.

Press any key to continue...

  Failed to boot both default and fallback entries.

Press any key to continue...

And then after a few seconds it dumps to a grub screen that just has a "*Local" entry.

bladernr@critical-maas:~$ maas admin machine get-curtin-config pbrbtr
Success.
Machine-readable output follows:
apt:
  preserve_sources_list: false
  primary:
  - arches:
    - default
    uri: http://archive.ubuntu.com/ubuntu
  proxy: http://10.0.0.1:8000/
  security:
  - arches:
    - default
    uri: http://archive.ubuntu.com/ubuntu
  sources:
    cert_dev_4:
      key: '-----BEGIN PGP PUBLIC KEY BLOCK-----

        Version: SKS 1.1.6

        Comment: Hostname: keyserver.ubuntu.com

        mI0ESZRIGAEEALihBkS1+WPsmLIq/y9i1Gc5vyPy7dqJVUscP38cjFsCnG8O+4FeP6mD/uJf

        VNBTcR3NjQ1jxwLISaIKndGu4J6n8xVPLWb/sesYy75WjgwUKC5lQnA61UElBm4AJfydKOob

        WTMq2X4Gw71+uwtjdnujn3uJGYDmrARHYCkAT7snABEBAAG0JUxhdW5jaHBhZCBQUEEgZm9y

        IENoZWNrYm94IERldmVsb3BlcnOItgQTAQIAIAUCSZRIGAIbAwYLCQgHAwIEFQIIAwQWAgMB

        Ah4BAheAAAoJEBtLay0rvfK9XKAEAJqorKQxdRidJTxMEirFmhaVD9plAqO/5uNXYFsVU0kS

        EEsk5Ps4F4lvLqWoupbPHuha6ngtNJf5HFNoCivFWSLawTA4tfF9RGuPozDHEpTbdLPqfpOK

        pFd1I9KMiGEhzCrQDp8cYjZkdMDEPsO9A87S5BxpB9rwBj9cnYnpvbw5

        =JBdy

        -----END PGP PUBLIC KEY BLOCK-----'
      source: ppa:checkbox-dev/ppa
cloudconfig:
  maas-cloud-config:
    content: "#cloud-config\ndatasource:\n MAAS: {consumer_key: BhjZmmMatV9MEU8qQV,\
      \ metadata_url: 'http://10.0.0.1:5240/MAAS/metadata/',\n token_key: B3GFccudYtQ6u2VELA,\
      \ token_secret: 3XjW8hukDHa9Q3cWH5CZasxuG7p3v4Tf}\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_config.cfg
  maas-datasource:
    content: 'datasource_list: [ MAAS ]'
    path: /etc/cloud/cloud.cfg.d/90_maas_datasource.cfg
  maas-ubuntu-sso:
    content: '#cloud-config

      snappy: {email: <email address hidden>}

      '
    path: /etc/cloud/cloud.cfg.d/90_maas_ubuntu_sso.cfg
debconf_selections:
  maas: 'cloud-init cloud-init/datasources multiselect MAAS

    cloud-init cloud-init/maas-metadata-url string http://10.0.0.1:5240/MAAS/metadata/

    cloud-init cloud-init/maas-metadata-credentials string oauth_consumer_key=BhjZmmMatV9MEU8qQV&oauth_token_key=B3GFccudYtQ6u2VELA&oauth_token_secret=3XjW8hukDHa9Q3cWH5CZasxuG7p3v4Tf

    cloud-init cloud-init/local-cloud-config string apt:\n preserve_sources_list:
    false\n primary:\n - arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\n proxy:
    http://10.0.0.1:8000/\n security:\n - arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\n sources:\n cert_dev_4:
    {key: ''-----BEGIN PGP PUBLIC KEY BLOCK-----\n\n Version: SKS 1.1.6\n\n Comment:
    Hostname: keyserver.ubuntu.com\n\n\n mI0ESZRIGAEEALihBkS1+WPsmLIq/y9i1Gc5vyPy7dqJVUscP38cjFsCnG8O+4FeP6mD/uJf\n\n VNBTcR3NjQ1jxwLISaIKndGu4J6n8xVPLWb/sesYy75WjgwUKC5lQnA61UElBm4AJfydKOob\n\n WTMq2X4Gw71+uwtjdnujn3uJGYDmrARHYCkAT7snABEBAAG0JUxhdW5jaHBhZCBQUEEgZm9y\n\n IENoZWNrYm94IERldmVsb3BlcnOItgQTAQIAIAUCSZRIGAIbAwYLCQgHAwIEFQIIAwQWAgMB\n\n Ah4BAheAAAoJEBtLay0rvfK9XKAEAJqorKQxdRidJTxMEirFmhaVD9plAqO/5uNXYFsVU0kS\n\n EEsk5Ps4F4lvLqWoupbPHuha6ngtNJf5HFNoCivFWSLawTA4tfF9RGuPozDHEpTbdLPqfpOK\n\n pFd1I9KMiGEhzCrQDp8cYjZkdMDEPsO9A87S5BxpB9rwBj9cnYnpvbw5\n\n =JBdy\n\n -----END
    PGP PUBLIC KEY ...

After booting (with Secure Boot inactive), do you see the /boot/efi/EFI/ubuntu/shimx64.efi file? If not, then it sounds like curtin is failing to install it -- either not installing the shim-signed package itself or failing to copy shimx64.efi from that package to the ESP.

FWIW, during the install I DID see the apt messages showing where the shim was being unpacked...

I've been looking for how curtin should detect that it should install shim
and other secure boot related packages. It's certainly likely that curtin
needs to do something here, however I've not found out exactly how to
detect that.

https://bugs.launchpad.net/maas/+bug/1647827

https://bugs.launchpad.net/maas/+bug/1680917

If we can confirm (or point me to the right stuff) then we can
add those steps to detect when/where to install shim and other uefi related
packages.

On Tue, May 2, 2017 at 3:24 PM, Andres Rodriguez <email address hidden>
wrote:

> ** Changed in: maas
> Status: Invalid => Incomplete
>
> ** Changed in: maas
> Milestone: None => 2.2.0rc4
>
> --
> You received this bug notification because you are subscribed to curtin.
> Matching subscriptions: curtin-bugs-all
> https://bugs.launchpad.net/bugs/1687729
>
> Title:
> Unable to deploy 16.04 w/ Xenial HWE to system with Secure Boot active
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1687729/+subscriptions
>

That's a good point, Rod. So I disabled SB and rebooted, and was able to boot the system successfully.

ubuntu@xwing:~$ ll /boot/efi/EFI/ubuntu/
total 3468
drwxr-xr-x 2 root root 4096 May 2 20:32 ./
drwxr-xr-x 3 root root 4096 May 2 20:32 ../
-rwxr-xr-x 1 root root 72144 May 2 20:32 fbx64.efi*
-rwxr-xr-x 1 root root 201 May 2 20:32 grub.cfg*
-rwxr-xr-x 1 root root 1121144 May 2 20:32 grubx64.efi*
-rwxr-xr-x 1 root root 1168464 May 2 20:32 mmx64.efi*
-rwxr-xr-x 1 root root 1169992 May 2 20:32 shimx64.efi*

The shim is there, and as noted in comment #7 a moment ago, I did see the dpkg output showing the shim being unpacked during the install.

Also reflected in the dpkg log:

2017-05-02 20:32:29 install shim:amd64 <none> 0.9+1474479173.6c180c6-1ubuntu1
2017-05-02 20:32:29 status half-installed shim:amd64 0.9+1474479173.6c180c6-1ubuntu1
2017-05-02 20:32:29 status unpacked shim:amd64 0.9+1474479173.6c180c6-1ubuntu1
2017-05-02 20:32:29 status unpacked shim:amd64 0.9+1474479173.6c180c6-1ubuntu1
2017-05-02 20:32:29 install shim-signed:amd64 <none> 1.27~16.04.1+0.9+1474479173.6c180c6-1ubuntu1
2017-05-02 20:32:29 status half-installed shim-signed:amd64 1.27~16.04.1+0.9+1474479173.6c180c6-1ubuntu1
2017-05-02 20:32:29 status unpacked shim-signed:amd64 1.27~16.04.1+0.9+1474479173.6c180c6-1ubuntu1
2017-05-02 20:32:29 status unpacked shim-signed:amd64 1.27~16.04.1+0.9+1474479173.6c180c6-1ubuntu1
2017-05-02 20:32:29 trigproc libc-bin:amd64 2.23-0ubuntu7 <none>
2017-05-02 20:32:29 status half-configured libc-bin:amd64 2.23-0ubuntu7
2017-05-02 20:32:29 status installed libc-bin:amd64 2.23-0ubuntu7
2017-05-02 20:32:29 trigproc man-db:amd64 2.7.5-1 <none>
2017-05-02 20:32:29 status half-configured man-db:amd64 2.7.5-1
2017-05-02 20:32:29 status installed man-db:amd64 2.7.5-1
2017-05-02 20:32:29 startup packages configure
2017-05-02 20:32:29 configure libefivar0:amd64 0.23-2 <none>
2017-05-02 20:32:29 status triggers-pending libc-bin:amd64 2.23-0ubuntu7
2017-05-02 20:32:29 status unpacked libefivar0:amd64 0.23-2
2017-05-02 20:32:29 status half-configured libefivar0:amd64 0.23-2
2017-05-02 20:32:29 status installed libefivar0:amd64 0.23-2
2017-05-02 20:32:29 configure efibootmgr:amd64 0.12-4 <none>
2017-05-02 20:32:29 status unpacked efibootmgr:amd64 0.12-4
2017-05-02 20:32:29 status half-configured efibootmgr:amd64 0.12-4
2017-05-02 20:32:29 status installed efibootmgr:amd64 0.12-4
2017-05-02 20:32:29 configure grub-efi-amd64-bin:amd64 2.02~beta2-36ubuntu3.9 <none>
2017-05-02 20:32:29 status unpacked grub-efi-amd64-bin:amd64 2.02~beta2-36ubuntu3.9
2017-05-02 20:32:29 status half-configured grub-efi-amd64-bin:amd64 2.02~beta2-36ubuntu3.9
2017-05-02 20:32:29 status installed grub-efi-amd64-bin:amd64 2.02~beta2-36ubuntu3.9
2017-05-02 20:32:29 configure grub-efi-amd64:amd64 2.02~beta2-36ubuntu3.9 <none>
2017-05-02 20:32:29 status unpacked grub-efi-amd64:amd64 2.02~beta2-36ubuntu3.9
2017-05-02 20:32:29 status unpacked grub-efi-amd64:amd64 2.02~beta2-36ubuntu3.9
2017-05-02 20:32:29 status unpacked grub-efi-amd64:amd64 2.02~beta2-36ubuntu3.9
2017-05-02 20:32:29 status half-configured grub-efi-amd64:amd64 2.02~beta2-36ubuntu3.9
2017-05-02 20:32:34 status installed grub-efi-amd64:amd64 2.02~beta2-36ubuntu3.9
2017-05-02 20:32:34 configure grub-efi-amd64-signed:amd64 1.66.9+2.02~beta2-36ubuntu3.9 <none>
2017-05-02 20:32:34 status unpacked grub-efi-amd64-signed:amd64 1.66.9+2.02~beta2-36ubuntu3.9
2017-05-02 20:32:34 status half-configured grub-efi-amd64-signed:amd64 1.66.9+2.02~beta2-36ubuntu3.9
2017-05-02 20:32:34 status installed grub-efi-amd64-signed:amd64 1.66.9+2.02~beta2-36ubuntu3.9
2017-05-02 20:32:34 configure mokutil:amd64 0.3.0-0ubuntu3 <none>
2017-05-02 20:32:34 status unpacked mokutil:amd64 0.3.0-0ubuntu3
2017-05-02 20:32:34 status half-configured mokutil:amd64 0.3.0-0ubuntu3
2017-05-02 20:32:34 sta...

Jeff,

Could you try running this command:

efi-readvar

You may need to install the efitools package. When I tried this on a system I recently deployed via MAAS, it complained that "no efivarfs filesystem is mounted," so this might not work on your server; but booting a client image in "try before installing" mode should get it to work.

What you're looking for in the output is this:

db: List 0, type X509
    Signature 0, size 1572, owner 77fa9abd-0359-4d32-bd60-28f4e78f784b
        Subject:
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation UEFI CA 2011
        Issuer:
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation Third Party Marketplace Root

This is Microsoft's Secure Boot public key for third-party binaries. Canonical's Shim is signed with the matching private key. If this public key is missing from your computer's NVRAM, Canonical's Shim won't launch when Secure Boot is active. Note that there are likely to be a couple other Microsoft keys, too; the critical one is the "Microsoft Corporation Third Party Marketplace Root."

My hypothesis that this key is missing is a bit of a long shot, but I've heard of client systems in which this is the case, and it may be even more likely on servers than on client systems. There is a way to install the key, but it's a bit tedious:

http://www.rodsbooks.com/efi-bootloaders/controlling-sb.html

The system may be configured to boot grub directly instead of the shim first. Could you also post the output of

efibootmgr -v

Good call Lee, that looks like it might be the case:
ubuntu@xwing:~$ efibootmgr -v
BootCurrent: 0008
Timeout: 1 seconds
BootOrder: 0008,000D,000A,000E,0010,000C,000F,0006,0007
Boot0006* Generic Usb Device VenHw(99e275e7-75a0-4b37-a2e6-c5385e6c00cb)
Boot0007* CD/DVD Device VenHw(99e275e7-75a0-4b37-a2e6-c5385e6c00cb)
Boot0008* UEFI: IP4 Intel(R) I210 Gigabit Network Connection PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/MAC(4439c4456baa,0)/IPv4(0.0.0.0:0<->0.0.0.0:0,0,0)..BO
Boot000A* UEFI: IP4 Intel(R) I210 Gigabit Network Connection PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/MAC(4439c4456bab,0)/IPv4(0.0.0.0:0<->0.0.0.0:0,0,0)..BO
Boot000C UEFI: IP4 Intel(R) Ethernet Connection I217-LM PciRoot(0x0)/Pci(0x19,0x0)/MAC(fc4dd4f49811,0)/IPv4(0.0.0.0:0<->0.0.0.0:0,0,0)..BO
Boot000D* UEFI: IP6 Intel(R) I210 Gigabit Network Connection PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/MAC(4439c4456baa,0)/IPv6([::]:<->[::]:,0,0)..BO
Boot000E* UEFI: IP6 Intel(R) I210 Gigabit Network Connection PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/MAC(4439c4456bab,0)/IPv6([::]:<->[::]:,0,0)..BO
Boot000F UEFI: IP6 Intel(R) Ethernet Connection I217-LM PciRoot(0x0)/Pci(0x19,0x0)/MAC(fc4dd4f49811,0)/IPv6([::]:<->[::]:,0,0)..BO
Boot0010* ubuntu HD(1,GPT,ef8d114b-1908-4c26-ac2d-c9440a570607,0x800,0x100000)/File(\EFI\UBUNTU\GRUBX64.EFI)

Jeff, the efibootmgr output indicates that Boot0008 is first in the boot order. That's a PXE-boot entry, as are the next three in the BootOrder list, so unless PXE-booting is failing for some reason, the Boot0010 entry for Ubuntu's grubx64.efi should not come into play. You can test this by deleting the Boot0010 entry:

sudo efibootmgr -b 0010 -B

This should not affect booting if the system is PXE-booting, but if it's failing because of a direct boot of GRUB, this will cause the system to try the additional boot entries, which include additional PXE-boot entries, the USB drive, and the CD drive. If all of those fail it will probably launch the firmware setup utility, or maybe an EFI shell.

On Thu, May 4, 2017 at 9:09 AM, Rod Smith <email address hidden> wrote:
> Jeff, the efibootmgr output indicates that Boot0008 is first in the boot
> order. That's a PXE-boot entry, as are the next three in the BootOrder
> list, so unless PXE-booting is failing for some reason, the Boot0010
> entry for Ubuntu's grubx64.efi should not come into play. You can test
> this by deleting the Boot0010 entry:
>
> sudo efibootmgr -b 0010 -B
>
> This should not affect booting if the system is PXE-booting, but if it's
> failing because of a direct boot of GRUB, this will cause the system to
> try the additional boot entries, which include additional PXE-boot
> entries, the USB drive, and the CD drive. If all of those fail it will
> probably launch the firmware setup utility, or maybe an EFI shell.

That is true and yes PXE works fine. The sequence is: Hardware OpROMs
load, node PXE boots, MAAS tells it to boot locally, then it tries
loading the shim and can't find it.

As noted above, when I disable SB and boot into the OS, the shim is
clearly present in /boot

@rodsmith

the MSFT key is there:

ubuntu@xwing:~$ efi-readvar
Variable PK, length 862
PK: List 0, type X509
    Signature 0, size 834, owner 26dc4851-195f-4ae1-9a19-fbf883bbb35e
        Subject:
            CN=DO NOT TRUST - AMI Test PK
        Issuer:
            CN=DO NOT TRUST - AMI Test PK
Variable KEK, length 1560
KEK: List 0, type X509
    Signature 0, size 1532, owner 77fa9abd-0359-4d32-bd60-28f4e78f784b
        Subject:
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation KEK CA 2011
        Issuer:
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation Third Party Marketplace Root
Variable db, length 4867
db: List 0, type X509
    Signature 0, size 834, owner 26dc4851-195f-4ae1-9a19-fbf883bbb35e
        Subject:
            CN=Trust - Lenovo Certificate
        Issuer:
            CN=Trust - Lenovo Certificate
db: List 1, type X509
    Signature 0, size 834, owner 26dc4851-195f-4ae1-9a19-fbf883bbb35e
        Subject:
            CN=Trust - Lenovo Certificate
        Issuer:
            CN=Trust - Lenovo Certificate
db: List 2, type X509
    Signature 0, size 1515, owner 77fa9abd-0359-4d32-bd60-28f4e78f784b
        Subject:
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011
        Issuer:
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Root Certificate Authority 2010
db: List 3, type X509
    Signature 0, size 1572, owner 77fa9abd-0359-4d32-bd60-28f4e78f784b
        Subject:
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation UEFI CA 2011
        Issuer:
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation Third Party Marketplace Root
Variable dbx, length 76
dbx: List 0, type SHA256
    Signature 0, size 48, owner 26dc4851-195f-4ae1-9a19-fbf883bbb35e
        Hash:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Variable MokList has no entries

Jeff,

OK, the presence of the key rules out one possible cause of the failure. You might also want to check to be sure you've got the latest firmware installed. One of my systems (an Intel NUC) required a firmware upgrade to work correctly with Secure Boot and MAAS.

Based on Rod's suggestion, I'd suggest the firmware gets upgraded beforehand. This does seem like a firmware issue.

Since there has not been any further updates, and I have confirmed that in various other hardware, secure boot seems to have been broken without a proper firmware update. As such, I'm marking this bug as Invalid.

If, after upgrading your firmware, the problem still persists. Please re-open this bug report.