grub2-signed can not find the right device when there are two filesystems containing the file '.disk/info'.

We have to find /boot/grub somehow in the signed memdisk configuration (generated in debian/build-efi-images, if you're curious), but none of the solutions I initially thought of work here. The filesystems are identical because they were copied, so we can't leave any clues in the file names or contents. Filesystem labels are reserved for sysadmins to set - at least if you're at all sensible - and we can't realistically bake a filesystem UUID into the signed memdisk configuration because we'd then end up changing the UUID of the filesystem containing /boot/grub and that doesn't sound like a great plan.

So, all the cleverer search strategies seem off the table, but EFI does have the Loaded Image Protocol which lets you find out where the image you're running came from. And oh look - GRUB's EFI initialisation code already sets $root from that if it can. I suspect the reason I didn't rely on this is that GRUB only does this for hard disks (which might well be reasonable), but this is in the startup code used on CDs and I needed it to work there as well. I think the right answer is probably something along these lines:

-if ! search --file --set=root /.disk/info; then
- search --file --set=root /.disk/mini-info
+if [ -z "\$root" ]; then
+ if ! search --file --set=root /.disk/info; then
+ search --file --set=root /.disk/mini-info
+ fi
 fi

But I need to do some testing to check exactly how the boot process behaves in a few different cases, and possibly add more sanity checking there to make sure that ($root)/boot/grub exists before committing to it.

(Note to the enthusiastic: the above diff doesn't actually work so don't waste time building packages of it.)

The way we're calling grub-mkimage causes the prefix to be overridden to (memdisk)/boot/grub, which isn't terribly helpful. Switching the order of -m and -p on the command line improves things here, and then I think it may just be a matter of checking whether $prefix already refers to something that exists on the filesystem before going around searching for things.

I think I've sorted this out in my most recent bzr commit. However, I want to make sure I haven't broken hard disk booting of signed images; I'll do that by way of blocking the upload in saucy-proposed for manual testing.

This bug was fixed in the package grub2 - 2.00-18ubuntu3

---------------
grub2 (2.00-18ubuntu3) saucy; urgency=low

  * Fix recordfail-disabling patch to take account of the GRUB_PREFIX
    variable no longer existing.
 -- Colin Watson <email address hidden> Wed, 11 Sep 2013 17:41:22 +0100

Colin,

 I try to backport the patch into precise to make a test, so that I can verify if it could fix the issue in 12.04.

 I found the code is very big difference, the patch doesn't work for precise.

lp:~ubuntu-core-dev/ubuntu/saucy/grub2/saucy

 And I also try to build source from the latest grub2 in your lp in my pbuilder. Below is the log.

dpkg-source: error: can't build with source format '3.0 (quilt)': no upstream tarball found at ../grub2_2.00.orig.tar.{bz2,gz,lzma,xz}
dpkg-buildpackage: error: dpkg-source -b grub2 gave error exit status 255

@Colin,

 After download the grub2_2.00.orgi.tar.xz, it build success.

And is there any clue to let this fix in precise? Thanks!

I'll deal with the backport to precise once I have a successful test
report from saucy.

@Colin,

I downloaded the daily-live cdimage in 20130913.

1) Secure boot Enabled.
It prompt
Could not open "\EFI\BOOT\fallback.efi":14
then go into the recovery partition, show the restore UI.

2) Secure boot Disabled and set "UEFI First" in "UEFI/Legacy Boot Priority"
It prompt
Could not open "\EFI\BOOT\fallback.efi":14
then reboot the laptop ceaselessly.

3) Secure boot Disabled and set "Legacy First" in "UEFI/Legacy Boot Priority"
Startup into the installation UI, works fine.

Again, I tried the ubuntu-13.04-desktop-amd64.iso, just a little difference.

1) Secure boot Enabled.
Just go into the recovery partition, show the restore UI.

2) Secure boot Disabled and set "UEFI First" in "UEFI/Legacy Boot Priority"
It prompt
Secure boot not enabled
then reboot the laptop ceaselessly.

3) Secure boot Disabled and set "Legacy First" in "UEFI/Legacy Boot Priority"
Startup into the installation UI, works fine.

@Colin,

 What can I do for you to help locate the reason? Thanks!

The messages about \EFI\BOOT\fallback.efi are harmless noise from shim; you can disregard those.

Do you have any way to confirm that it's actually booting from the CD when in secure boot or UEFI-first mode (e.g. drive noise or activity lights)?

Colin,

 Sorry for reply late, and the boot from CD is very quick, just like checking CDRom, I'll check it today.

I have verified ubuntu-13.10-beta2-desktop-amd64.iso by the steps of the bug description and it works fine.

Hi $4, do we need this to be fixed in precise as well?

Yes, we do need this fix in precise as well if we want to deploy UEFI secure boot on OEM projects.

Collin,

 I just download the 20131016.1 daily live cdimage, and record it into DVD, then boot up from CDRom.

When secure boot is disabled, and UEFI first, it's just hang after the "error: variable 'root' isn't set."

When secure boot is enabled, it will boot up to recovery partition, I can hear the drive noise when select boot from DVD driver.
When it's in the UI of restore, I switch to tty1, you can find my recovery partition /dev/sda1 is mounted as /cdrom.

$ df -kh
Filesystem Size Used Avail Use% Mounted on
/cow 1.6G 31M 1.6G 2% /
udev 1.6G 4.0K 1.6G 1% /dev
tmpfs 649M 836K 648M 1% /run
/dev/sda1 1.5G 1.2G 279M 82% /cdrom
/dev/loop0 1.1G 1.1G 0 100% /rofs
tmpfs 1.6G 20K 1.6G 1% /tmp
none 5.0M 4.0K 5.0M 1% /run/lock
none 1.6G 76K 1.6G 1% /run/shm

And from dmesg log you can find the kernel's boot parameter.

    0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.5.0-36-generic (buildd@allspice) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #57sutton6-Ubuntu SMP Wed Sep 4 10:49:49 UTC 2013 (Ubuntu 3.5.0-36.57sutton6-generic 3.5.7.14)
[ 0.000000] Command line: BOOT_IMAGE=/casper/vmlinuz ubuntu-recovery/recovery_type=hdd boot=casper automatic-ubiquity noprompt quiet splash -- mem=4GB locale=zh_CN.UTF-8
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls

And I create a bootable USB with the same image, when boot up from USB, it works fine.

Filesystem Size Used Avail Use% Mounted on
/cow 1.8G 42M 1.8G 3% /
udev 1.8G 4.0K 1.8G 1% /dev
tmpfs 368M 1.2M 367M 1% /run
/dev/sdb1 4.0G 892M 3.2G 22% /cdrom
/dev/loop0 843M 843M 0 100% /rofs
none 4.0K 0 4.0K 0% /sys/fs/cgroup
tmpfs 1.8G 16K 1.8G 1% /tmp
none 5.0M 4.0K 5.0M 1% /run/lock
none 1.8G 144K 1.8G 1% /run/shm
none 100M 28K 100M 1% /run/user

And from dmesg log you can find the kernel's boot parameter.

[ 0.000000] Linux version 3.11.0-12-generic (buildd@allspice) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu7) ) #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 (Ubuntu 3.11.0-12.19-generic 3.11.3)
[ 0.000000] Command line: BOOT_IMAGE=/casper/vmlinuz.efi file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash --

@Collin,

And I also verify by 4$'s comment in #13, I can see every menu entry of grub with the suffix 'from recovery partition'.

Might be the original issue is fixed. But from the title, the issue from DVD image also caused by the partition which has .disk, so should I open a new one for the issue which boot up from DVD? Thanks!

And it's easy to reproduce.

* Reproduce Steps is almost same in description. from 1-7, you can't create a partition which have .disk and /boot/grub/grub.cfg

8. Use `brasero` to write saucy-desktop-amd64.iso to DVD disc.
9. Reboot and select the boot entry 'ATAPI CD0: PLDS DVD-RW DS8A9SH' from UEFI boot option menu.

* Expected Result
We shouldn't see every menu entry of grub with the suffix 'from recovery partition', cause we start from DVD drive.

* Actual Result
There is suffix 'from recovery partition' on menu entries of grub.

@BIn,

Can you use UEFI secure boot via DVD drive successfully?
I have tried two different laptops but they both failed.

First case:
IMG_20131018_134053.jpg
IMG_20131018_134235.jpg

Second case:
IMG_20131018_140231.jpg
IMG_20131018_140321.jpg

@Colin,

 I del the grub.cfg in recovery partition, after that it will enter grub shell.
 from the result, I thought the $prefix is zero or not exist.

grub> cat (memdisk)/grub.cfg

if [ -z "$prefix" -o ! -e "$prefix" ]; then
        if ! search --file --set=root /.disk/info; then
                search --file --set=root /.disk/mini-info
        fi
        set prefix=($root)/boot/grub
fi
if [ -e $prefix/$platform/grub.cfg ]; then
        source $prefix/$platform/grub.cfg
else
        source $prefix/grub.cfg
fi

@Colin,

 And another issue when I run 'search --file /.disk/info' in grub-shell, it will reboot my laptop. Is it a bug? Thanks!

@Bin Li: This sounds like it could be a firmware-specific issue. The prefix is supposed to be derived from the EFI Loaded Image Protocol, which is really the only way I'm aware of to implement the feature requested in this bug report in a reliable way. If it doesn't work on your firmware then we have a deeper problem. And yes, "search --file /.disk/info" rebooting your laptop is absolutely a bug, although one that should be filed separately. This could well be related to the broken/missing prefix.

One possibility is that this is a problem with EFI stack alignment; I mention that because it was recently fixed upstream. I expect to have a recent upstream snapshot in trusty in the next week or two, which would be worth testing. Until then, I think it would be best to leave verification of this bug to Shih-Yuan Lee, if that's OK, to avoid clouding this issue with what sounds like a separate problem.

Hello Shih-Yuan, or anyone else affected,

Accepted grub2 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/grub2/1.99-21ubuntu3.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

@Adam,

 After test it in precise-proposed, I found the signed efi image is still 3.10, while not ubuntu3.14. Could you help update this file? Thanks!

http://archive.ubuntu.com/ubuntu/dists/precise-proposed/main/installer-amd64/current/images/cdrom/debian-cd_info.tar.gz

And anyway, from the daily build image, we still met this issue.
Form #21, looks like the $prefix is zero or not exist, so I'll set it as verification-failed.

The daily build gets its GRUB binaries (at least in part) from the
latest version of debian-installer in precise-proposed, so this can't be
verified either way until that's updated. I've just uploaded an update
so we can do a proper build.

Today's daily build should have an up-to-date version of GRUB. Please retest.

Also, I would prefer somebody who was able to verify the fix in saucy to do the testing, to avoid clouding this with other issues. See my comment #23.

@Collin,

 Thanks a lot!

 And a question, is there a way to check the $prefix's value before search? And the $prefix was set by grub-mkimage with '-p' parameter? Thanks!

On Thu, Dec 12, 2013 at 11:24:31AM -0000, Bin Li wrote:
> @Collin,

My given name only has one "l", FYI.

> And a question, is there a way to check the $prefix's value before
> search?

I'm afraid the signed image business makes this very tedious. The
simplest method would probably be to take debian/build-efi-images from
the grub2 source package, extend its skeleton grub.cfg to print the
prefix and sleep for a while as the first thing it does, put the
resulting gcdx64.efi on a modified CD image, make sure you're in setup
mode if you're on a system with UEFI Secure Boot so that you don't have
to sign the image, and try booting it.

> And the $prefix was set by grub-mkimage with '-p' parameter?

Sort of. We pass a partial prefix using "-p /boot/grub". The effect of
this is to instruct the GRUB kernel to look for the device from which it
was loaded (on EFI, this uses the Loaded Image Protocol); if it finds
that device, it will set the prefix to (that-device)/boot/grub,
otherwise it will leave it unset.

An unset prefix at the stage you describe in comment #21 would imply
that GRUB entirely failed to get the loaded image location, or possibly
that it failed to determine a GRUB drive name from it. A nonexistent
prefix would perhaps imply that it picked the wrong GRUB drive.

If you're going to be building custom images anyway to diagnose this,
then it might be worth simply modifying GRUB to insert some grub_printf
and grub_millisleep calls so that we can see what it's doing when trying
to discover the loaded image location. Something like this:

=== modified file 'grub-core/kern/efi/init.c'
--- grub-core/kern/efi/init.c 2010-09-08 12:53:47 +0000
+++ grub-core/kern/efi/init.c 2013-12-12 12:06:24 +0000
@@ -68,10 +68,16 @@ grub_efi_set_prefix (void)

   if ((!device || device[0] == ',' || !device[0]) || !path)
     image = grub_efi_get_loaded_image (grub_efi_image_handle);
+ grub_printf ("image: %p\n", image);
+ grub_millisleep (1000);
   if (image)
     {
       if (!device)
- device = grub_efidisk_get_device_name (image->device_handle);
+ {
+ device = grub_efidisk_get_device_name (image->device_handle);
+ grub_printf ("device: %s\n", device);
+ grub_millisleep (1000);
+ }
       else if (device[0] == ',' || !device[0])
  {
    /* We have a partition, but still need to fill in the drive. */
@@ -113,6 +119,8 @@ grub_efi_set_prefix (void)
       prefix = grub_xasprintf ("(%s)%s", device, path);
       if (prefix)
  {
+ grub_printf ("setting prefix: %s\n", prefix);
+ grub_millisleep (1000);
    grub_env_set ("prefix", prefix);
    grub_free (prefix);
  }

I do think we should investigate this in a separate bug report, though;
and I'd like to know whether the fix currently in precise-proposed at
least improves things for Shih-Yuan Lee in the same way that the fix in
saucy did (comment #13).

http://paste.ubuntu.com/6565622/ is the test result from Ubuntu 12.04.3 LTS "Precise Pangolin" - Release amd64 (20131212).

Colin,

 Thanks a lot!

And I thought the new package have improvement, it's the direction which fix our issue, I mark as verificaiton-done.
Next I prepare open a new bug to find why the prefix not set correctly with the clue from #31.

Thanks all you guys!!

I'm going to release this since there are other important bugs fixed in this update and as far as I know it doesn't make this bug worse. This bug will probably be auto-closed, but I'll reopen it.

This bug was fixed in the package grub2 - 1.99-21ubuntu3.14

---------------
grub2 (1.99-21ubuntu3.14) precise; urgency=low

  * Handle FAT filesystems on non-512B disks (LP: #1065281).
  * Probe FusionIO devices (LP: #1237519).
  * On Linux, read partition start offsets from sysfs if possible
    (LP: #1237519).

grub2 (1.99-21ubuntu3.13) precise; urgency=low

  * Revamp hidden timeout handling by adding a new timeout_style environment
    variable and a corresponding GRUB_TIMEOUT_STYLE configuration key for
    grub-mkconfig. This controls hidden-timeout handling more simply than
    the previous arrangements, and pressing any hotkeys associated with menu
    entries during the hidden timeout will now boot the corresponding menu
    entry immediately (LP: #1178618). As part of merging this, radically
    simplify /etc/grub.d/30_os-prober; if it finds other OSes it can now
    just set timeout_style=menu and make sure the timeout is non-zero.
  * Fix mismerge of GRUB_RECOVERY_TITLE option in 1.99-21ubuntu3.12.

grub2 (1.99-21ubuntu3.12) precise; urgency=low

  * debian/build-efi-images: Where possible, make use of the device path
    derived from the EFI Loaded Image Protocol to compute the prefix
    (LP: #1097570).
  * Add GRUB_RECOVERY_TITLE option, to allow the controversial "recovery
    mode" text to be customised (LP: #1240360).
 -- Colin Watson <email address hidden> Thu, 05 Dec 2013 16:53:48 +0000

The verification of the Stable Release Update for grub2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Marking it as Fix Released, as it was already fix in 13.10

Now in Lenovo's laptop, we still meet this issue on trusty(14.04.2).

 I delete the grub.cfg in recovery partition, after that it will enter grub shell.
 from the result, I thought the $prefix is zero or not exist.

grub> cat (memdisk)/grub.cfg

if [ -z "$prefix" -o ! -e "$prefix" ]; then
        if ! search --file --set=root /.disk/info; then
                search --file --set=root /.disk/mini-info
        fi
        set prefix=($root)/boot/grub
fi
if [ -e $prefix/$platform/grub.cfg ]; then
        source $prefix/$platform/grub.cfg
else
        source $prefix/grub.cfg
fi

The file debian/build-efi-images of package grub2-2.02.

Grub executed the "search --file --set=root /.disk/info" first, try to find out the "root". Because .disk/info was exist on USB stick and HDD0 Partition2, so the result included 2 results. But the USB stick was the first. So the grub.cfg on USB stick was loaded and the system was booted from USB stick.

We still suffer from this issue for Ubuntu 14.04.

4$, do you get the exact same behavior in 14.04 as you initially reported for 12.04?

What I think I can infer from the matrix you posted before is that search doesn't prefer the current boot location if it finds a file in the search command on that disk. Otherwise, things tend to be listed with USB drive first, then hard disk, then CDROM (at least, here as well as on the pastebin you posted), which appears to be a static order in which things are detected by grub (in other words, unrelated to BIOS boot order, at least).

GRUB should follow the UEFI's BootCurrent to find the root instead of checking the existence of /.disk/info or /.disk/mini-info.

It looks like the followings. (pseudocode)

    if ! search --disk --set=root UEFI:BootCurrent; then
        if ! search --file --set=root /.disk/info; then
                search --file --set=root /.disk/mini-info
        fi
   fi
   set prefix=($root)/boot/grub

Or you can put UEFI's BootCurrent device or partition in the first order to check when running `search --file --set=root`.

In short, if GRUB never checks UEFI BootCurrent, the issue is impossible to fix.

Indeed you can use BootCurrent (or, as grub already provides that, use $cmdpath or some variation of it) to specify the right device as a basis for where to search for the files.

Rather than simply using search --file --set=root /.disk/info; I think we can make use of hints to specify where to search first. I've experimented with this over the past few days and it looks like it works properly. I've also prepared a patch to grub to provide an extra environment variable "$bootdrive" that would include just the disk specification for the device used for the EFI booting -- in other words, while $cmdpath contains the full path to the directory containing the EFI binaries used to boot ((hd2,gpt2)/EFI/ubuntu); it would contain just the drive without partition specification (hd2 in the same example).

Then, 'search --file --hint=$bootdrive --set=root /.disk/info' appears to return the right paths according to my tests.

I've sent the patch upstream (waiting to see that it was indeed received on the mailing list), and I'll do one final run of testing to make sure this behaves properly before considering an upload first to the current development release (since it does appear to be reproducible there still).

@Mathieu,

Your solution looks good. Thanks for your effort.
BTW, the GRUB script is embedded in grub2-signed so only the maintainers of grub2-signed can modify it.
We, downstream OEM projects, can not modify it because we need to support UEFI Secure Boot.

Yes, that will be fixed shortly, I was just waiting for review from the grub developers.

One question that remains is whether searching is in fact still necessary, which might further simplify the EFI images.

In my personal opinion, searching for .disk/info is unnecessary for UEFI BIOS if GRUB can find the right device from BootCurrent.
However I am wondering if legacy BIOS still needs it.

Oops, sorry for my previous bad comment.
Legacy BIOS won't use grub2-signed.

@Mathieu,

Is there any update for comment#44?

It's a bit hard to make sure I'm hitting all the right cases, but it did look (with a fairly recent grub) like searching was unnecessary. This would make the changes pretty small.

Shih-Yuan, could you try out a fixed-up grub2-signed image if I provide you with the right files for it? It would require at least replacing the grubx64.efi file on the EFI partition on a USB key. For a CD, it means re-building the CD with the right embedded images, something which seemed to be failing here. I'll prepare a new image for Trusty now and give rebuilding the CD another try (if it works, then I'll just get you an ISO image to use for testing)

Yes, I can help to verify it once I have the ISO image.

I've done my own testing on my side and unfortunately it seems like hinting the search command isn't sufficient to make this work -- there's definitely also the need to patch is some more to not use cache when hinting the command, so that things appear in the right order.

This said, I've had review on the $bootdrive patch and there's been some rightful pushback: there's a way to get the same thing via regexp, and we should otherwise make sure we're not unnecessarily duplicating variables (since there is already $cmdpath; though there is so logic to having a $grub_bootdev and $grub_bootpath separation).

Since it was suggested, I've looked into using regexp to retrieve the boot device name, it looks like it might work to remove the need to search for the right root device, but given that this is run in a UEFI context, there could be security implications to using regexp -- I've asked Colin Watson to weight in on that.

Milestoning to ubuntu-17.03 so it shows up better on my todo list. I haven't forgotten, but there's consistently been other tasks taking priority. With some similar work we've done for snappy, it should become easier to fix this issue.

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release