[regression] Cosmic daily images 20180606-11 install but boot only to grub prompt on EFI systems

Sorry I don't know what other packages to target.

Here's a diff between my current best known working image and the broken image.

The only thing that stands out between the working and broken images to me is:

< ii ubuntu-desktop 1.420 amd64 The Ubuntu desktop system
---
> ii ubuntu-desktop 1.421 amd64 The Ubuntu desktop system
1683c1684
< ii ubuntu-minimal 1.420 amd64 Minimal core of Ubuntu
---
> ii ubuntu-minimal 1.421 amd64 Minimal core of Ubuntu
1692c1693
< ii ubuntu-standard 1.420 amd64 The Ubuntu standard system
---
> ii ubuntu-standard 1.421 amd64 The Ubuntu standard system

Is this legit? Just a mistake?

ubuntu-meta (1.421) cosmic; urgency=medium

  * Refreshed dependencies
  * Moved open-iscsi to server-recommends (LP: #1630946)

 -- root <root@c.lxd> Wed, 30 May 2018 07:01:29 +0000

What daily image are we talking about?

How does "stuck in grub" present? Do you get thrown into a grub prompt, or do you just see nothing after the grub menu disappears?

We are talking about "cosmic daily images" http://cdimages.ubuntu.com/daily-live/

It's just the grub prompt.

I cannot reproduce this issue in a VM. Could you provide more details about the machines you tried to install on, is it UEFI hardware or with a specific setup?

It's seemingly got nothing to do with the hardware.

Today I have tested cosmic image 20180611 on two machines:

  * Lenovo X1 Carbon (gen 5)
  * Dell XPS 13 (9343)

and the bug persists. In both cases Ubuntu installs fine, but then the machine never boots.

I'm lucky in that I still have image 20180530 as a workaround.

Any chance I can get access to images 20180531-20180605 to better bisect the bug?

This is probably coincidence but the working images are slightly bigger than the non-working ones:

-rw------- 1 dan dan 1964244992 May 24 10:46 cosmic-desktop-amd64-20180522.iso
-rw------- 1 dan dan 1964244992 May 26 14:53 cosmic-desktop-amd64-20180525.iso
-rw------- 1 dan dan 1964244992 May 30 15:48 cosmic-desktop-amd64-20180530.iso
-rw------- 1 dan dan 1957183488 Jun 8 12:56 cosmic-desktop-amd64-20180606.iso
-rw------- 1 dan dan 1957183488 Jun 7 15:55 cosmic-desktop-amd64-20180607.iso
-rw------- 1 dan dan 1957183488 Jun 8 15:51 cosmic-desktop-amd64-20180608.iso
-rw------- 1 dan dan 1957183488 Jun 13 12:40 cosmic-desktop-amd64-20180611.iso

As already asked by Jean-Baptiste, could you give us more info on whether the installation has been done in UEFI or legacy BIOS mode? Is it broken for both cases? We did have a new grub2 released quite recently which might have potentially some effect here - but I am also unable to reproduce the issue when installing on a VM (tried ubuntu-server though). Let me check ubuntu-desktop maybe.

I am always using UEFI.

I don't think grub2 is the problem. This bug pre-dates that update by about 1 day. And image 20180606, which contains the bug, does not contain the new grub2:

ii grub-common 2.02-2ubuntu8 amd64 GRand Unified Bootloader (common files)
ii grub-gfxpayload-lists 0.7 amd64 GRUB gfxpayload blacklist
ii grub-pc 2.02-2ubuntu8 amd64 GRand Unified Bootloader, version 2 (PC/BIOS version)
ii grub-pc-bin 2.02-2ubuntu8 amd64 GRand Unified Bootloader, version 2 (PC/BIOS binaries)
ii grub2-common 2.02-2ubuntu8 amd64 GRand Unified Bootloader (common files for version 2)

See comment #2.

It's reproducible in a VM with a Cosmic desktop image and UEFI enabled.

Installer log on a UEFI VM

It seems the subiquity images are not affected. I have first tried the latest cosmic ubuntu-server-live image and was able to boot correctly into the system after installation. Then I have tried the old debian-installer based ubuntu-server images and was able to reproduce the issue, with the system displaying the grub prompt. This explains why I was not able to reproduce the issue before (I was using subiquity). Both on UEFI.

For now I'm just writing down some of my random thoughts, but looking at Daniel's diff between the working image and non-working, I don't see anything straightforward that could have caused it. That being said, Dimitri had a good idea that one of the livecd-rootfs changes could have caused it. I now see this which might be somehow related?

https://launchpad.net/ubuntu/+source/livecd-rootfs/2.528

Time-wise it could fit, as the package - even though uploaded on the 28th of May, it actually only migrated from cosmic-proposed to cosmic on the 5th of June (our images are built without -proposed enabled). Need to look into the change itself, but it does deal with grub somehow.

Yes, 'livecd-rootfs' is also a candidate since it doesn't appear in 'dpkg -l' so is hidden from my diff.

Comment #9 also suggests the problem might have arisen from a change to 'livecd-rootfs'

I am starting to doubt the theory that livecd-rootfs was responsible. I have built the livefs locally using livecd-rootfs 2.537 (so pre-dating the broken images), repacked the latest cosmic iso to include my squashfs in it, installed ubuntu-server on UEFI and am still experiencing the same grub prompt bug. Either my test-case is invalid or the issue lies somewhere else?

Looking at the change in livecd-rootfs it seems to be indeed not related, as the modified code-path is only executed for ubuntu-cpc projects.

Writing down some of our thoughts. Me and Jean-Baptiste sat down on the issue just now and noticed a strange thing happening in the ESP. We had 3 installs: A) cosmic ubuntu-server (debian-installer), B) cosmic ubuntu-server-live (subiquity) system and C) bionic ubuntu. A) is obviously broken, we get the grub2 prompt after booting. All cases but A) work, but even for A) it's possible still to boot the system by directly loading the config from the main partition.

We compared the ESP contents in A), B) and C). For B) the grub.cfg file (along with the .efi binaries) is located in the EFI/ubuntu/ directory on the efi partition. For C) there are 2 grub.cfg files present (and the corresponding binaries), one in EFI/ubuntu/ and one in EFI/grub/. Both of those systems boot. However, for A) the ESP only has the EFI/grub/ part. It looks like something that usually installed the required bits to ubuntu/ has changed and now, possibly, this is causing grub not proceeding with its boot process. Problem is we can't seem to see any changes in any of the installer packages that could have touched these parts.

It looks like the grub2-signed upload is the culprit, looks like the dependency switch we made caused in the d-i cases for grub2-efi-amd64 not to be installed, even though it's required. Will be consulting a fix.

I think it's because grub-efi-amd64 is missing. Installing it in /target at the end of installation (after remounting the efi partition) fixes the problem.

Since the grub2-signed changes that have introduced this regression are in fact needed, we will be looking into fixing this in the installer.

I would need someone to take a look and review my proposed fix for debian-installer. I have tested that on a custom ISO image and as a result I got my system booting. It did cause the grub prompt to appear (asking on which partition to install), but I suppose that's not anything new.

@sil2100, Colin already commented on a similar change you proposed against ubiquity https://code.launchpad.net/~sil2100/ubiquity/+git/ubiquity/+merge/348201/comments/909076

This bug was fixed in the package grub-installer - 1.128ubuntu9

---------------
grub-installer (1.128ubuntu9) cosmic; urgency=medium

  * grub-installer: install grub-pc for EFI setups and make sure we're
    not purging it earlier in case it got installed. It's needed to make sure
    the right maintainer scripts are run and the ESP populated. (LP: #1775743)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 19 Jun 2018 10:25:14 +0200

This bug was fixed in the package ubiquity - 18.10.4

---------------
ubiquity (18.10.4) cosmic; urgency=medium

  [ Chen-Han Hsiao (Stanley) ]
  * Add efivars to mountpoints loaded at bootloader install time.
    (LP: #1772374)

  [ Steve Langasek ]
  * scripts/plugininstall.py: don't hard-code a resume partition in
    /etc/initramfs-tools/conf.d/resume at install time. In bionic and later,
    initramfs-tools will autodetect an appropriate resume partition at
    initramfs generation time, so ubiquity's resume setting is redundant and
    possibly wrong. LP: #1768230.

  [ Łukasz 'sil2100' Zemczak ]
  * Make sure that grub-pc is not removed after installation for both EFI and
    legacy BIOS cases as we now technically need it even for EFI installs.
    (LP: #1775743)

  [ Mathieu Trudel-Lapierre ]
  * Automatic update of included source packages: grub-installer
    1.128ubuntu9.

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 25 Jun 2018 13:57:40 -0400

Not sure proposed fix is all of issue.

I boot 18.04 by default with UEFI and every test or experimental install in the past has overwritten my /EFI/ubuntu.
But install of Cosmic did not overwrite my /EFI/ubuntu but created /EFI/grub. It has a correct /EFI/grub/grub.cfg, but my UEFI is set to default boot using /EFI/ubuntu.
But booting grub UEFI entry booted me into my default install.
Issue is similar to previous bug reports where a change to GRUB_DISTRIBUTOR= creates a new UEFI entry, but internally either efi_distributor or perhaps just the code that defines where to find grub is hard coded to /EFI/ubuntu/grub.cfg.
It needs to use grub.cfg in the ESP folder it creates. And kluge on Kubuntu should finally be fixed.

Older version, not sure what you have now.

./.pc/install_arm64_naming.patch/util/grub-install.c: efi_distributor = "ubuntu";
./debian/patches/install_efi_ubuntu_flavours.patch:+ efi_distributor = "ubuntu";
./ChangeLog: GRUB_DISTRIBUTOR on efibootmgr line.
./debian/default/grub:GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
./debian/postinst.in: bootloader_id="$(config_item GRUB_DISTRIBUTOR | tr A-Z a-z | \
./debian/changelog: - Adjust UEFI installation to cope with Kubuntu setting GRUB_DISTRIBUTOR

On Tue, Jun 26, 2018 at 09:31:24PM -0000, Fred Palmer wrote:
> But install of Cosmic did not overwrite my /EFI/ubuntu but created
> /EFI/grub. It has a correct /EFI/grub/grub.cfg, but my UEFI is set to
> default boot using /EFI/ubuntu.

With which daily image did you see this?

AFAICS, no image has been released with both fixes yet:
  http://cdimages.ubuntu.com/daily-live/

> With which daily image did you see this?

The install of Cosmic has modified date of Mon 11 Jun 2018 07∶55∶53 AM CDT.
Not sure when I downloaded it.

AFAIK, all June images older than 20180627 will have the bug.

Please wait until the fix for this bug is released before retesting.

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1775743

Just saw one thread in Ubuntu Forums & one in AskUbuntu where with 18.04 the Boot-Repair output from efibootmgr -v shows no ubuntu entry and only has a grub entry.

See:
https://ubuntuforums.org/showthread.php?t=2396042
And:
https://askubuntu.com/questions/1053895/system-always-boots-to-grub-prompt?noredirect=1#comment1722812_1053895
This user already reported that copying /EFI/grub/grub.cfg to /EFI/ubuntu/grub.cfg fixed his boot issue.

From above comment

https://ubuntuforums.org/showthread.php?t=2396042

Installed from latest Ubuntu Studio 18.04 ISO.

Boots to grub> prompt

Also fixed by copying /boot/efi/EFI/grub/grub.cfg to /boot/efi/EFI/ubuntu/

This bug has been fixed for a while already and is about cosmic only (Ubuntu 18.10).

If you have any problems with 18.04 then please open a new bug.

Hello

I need to install Ubuntu 18.04.4 LTS in the following machine that has UEFI but I only get a blank screen after selecting the "intall" option or the "try Ubuntu live" option.

Processor Manufacturer Intel® Processor Type Core™ i5 Processor Model i5-1035G1 Processor Speed 1 GHz Processor Core Quad-core (4 Core™) Display & Graphics Graphics Controller Manufacturer Intel® Graphics Controller Model UHD Graphics Graphics Memory Technology DDR4 SDRAM Graphics Memory Accessibility Shared Memory Standard Memory 8 GB Memory Technology DDR4 SDRAM

I saw the solution suggested in the post#21: "I think it's because grub-efi-amd64 is missing. Installing it in /target at the end of installation (after remounting the efi partition) fixes the problem."

I wanted the mentioned solution but I do not know how to install grub-efi-amd64 in the USB stick that has the Ubuntu 18.04.4 LTS image that I want to install.

Does any of you know how this grub-efi-amd64 can be installed along the Ubuntu 18.04.4 LTS image?

Thanks