grub-installer fails due to alternate disk partition error - 14.04.3

Default Comment by Bridge

We should already have code in the version of grub-installer that will list PReP partitions and try to pick the one that matches the drive we're installing on. Could you please attach /var/log/syslog from the installer when it fails? This should include the exact messages from grub-installer, so it will tell us why it picks one particular drive.

------- Comment on attachment From <email address hidden> 2016-01-14 13:56 EDT-------

Sorry about the delay in getting this information posted. The bridge appeared to have eaten your last update until now.

I looked again, and I can confirm there is the code necessary to only wipe the device we've received as a target for grub (provided it's available), and otherwise resort to the first device that pops up from prep-bootdev (which can be any, in whatever order parted decides to hand them over to us).

Could you recreate the issue, but just before the installation of the bootloader, edit /usr/bin/grub-installer to add "set -x" at the top? I'd very much like to see what grub-installer thinks we've passed to it when it gets run, that could explain why we're seeing it find sdr1 rather than sdp from your install.

This also leads to seeing that there's another bug in that code (and I'll fix both together), we can't rely on parted giving us devices that are correctly mapped in /dev it seems, and should guard against the file not existing before running dd.

------- Comment on attachment From <email address hidden> 2016-01-14 17:27 EDT-------

The original system is not available for testing, but I was able to reproduce this behavior on an alternate test system. In this test, I altered the partition table on /dev/sda to match the reported scenario. I then attempted a 14.04 install on /dev/sdb. From my point of view, I do not want /dev/sda touched (it is not the droid you are looking for).

I just installed thrice; installed 16.10 on mpath0 and then mpath1; and finally installed 14.04 on mpath0. I was expecting to reproduce this bug by seeing mpath1 being wiped after being detected as the PReP bootdev, but it was not.

Could you please run '/usr/lib/grub-installer/prep-bootdev -l' on the system just before grub-installer runs?

The major problem is that we shouldn't see /dev/sda picked at all, since it's not the device that was selected for the install, and we see that grub-installer already knows it should try to match a partition to /dev/sdb in your logs:

Jan 14 22:16:41 main-menu[1409]: (process:72206): /usr/lib/grub-installer/prep-bootdev
Jan 14 22:16:41 main-menu[1409]: (process:72206): +
Jan 14 22:16:41 main-menu[1409]: (process:72206): wipe_bootdev=/dev/sda1
Jan 14 22:16:41 main-menu[1409]: (process:72206):
Jan 14 22:16:41 main-menu[1409]: (process:72206): +
Jan 14 22:16:41 main-menu[1409]: (process:72206): /usr/lib/grub-installer/prep-bootdev
Jan 14 22:16:41 main-menu[1409]: (process:72206): -l
Jan 14 22:16:41 main-menu[1409]: (process:72206):
Jan 14 22:16:41 main-menu[1409]: (process:72206): +
Jan 14 22:16:41 main-menu[1409]: (process:72206): [
Jan 14 22:16:41 main-menu[1409]: (process:72206): /dev/sda
Jan 14 22:16:41 main-menu[1409]: (process:72206): =
Jan 14 22:16:41 main-menu[1409]: (process:72206): /dev/sdb
Jan 14 22:16:41 main-menu[1409]: (process:72206): ]

There's still the issue of checking for the existence of the device node before running dd, to avoid crashes.

------- Comment From <email address hidden> 2016-01-22 17:46 EDT-------
~ # /usr/lib/grub-installer/prep-bootdev
/dev/sda1

~ # /usr/lib/grub-installer/prep-bootdev -l
/dev/sda1

From grub-installer:

...
case $ARCH:$grub_package in
ppc64el/*:grub-ieee1275)
# By default, use the first PReP partition found (prep-bootdev).
# If available, prefer a PReP partition in the same device (disc_offered).
# (the matching works on devices with/without disk-partition separator.)
wipe_bootdev=""
for prep_p in ; do
if [ "${prep_p%[0-9]*}" = "${disc_offered%[0-9]*}" ]; then
wipe_bootdev=$prep_p
break
fi
done
unset prep_p
...

The problem with the logic is that the install disk does not have a PReP partition in this case (because one is not needed in this particular environment). So, grub-installer appears to go out searching for some other victim when it should really just move along.

Still, we do add PReP partitions for every ppc64el system. This means you're doing manual partitioning, and specifically omitting the PReP partition, no?

In this case, is there any way we can recognize this particular system? Is it some chrp subtype that we should completely skip the PReP partition for? This would mean changes needed in grub-installer, as well as updates to partman-auto automatic partitioning recipes.

------- Comment on attachment From <email address hidden> 2016-01-14 13:56 EDT-------

Sorry about the delay in getting this information posted. The bridge appeared to have eaten your last update until now.

------- Comment From <email address hidden> 2016-01-25 11:07 EDT-------
(In reply to comment #24)
> Still, we do add PReP partitions for every ppc64el system. This means you're
> doing manual partitioning, and specifically omitting the PReP partition, no?

Actually, no. The partitioning is the the default chosen. All I did was to specify that all of /dev/sdb should be used.

> In this case, is there any way we can recognize this particular system? Is
> it some chrp subtype that we should completely skip the PReP partition for?
> This would mean changes needed in grub-installer, as well as updates to
> partman-auto automatic partitioning recipes.

I suspect that the installer is already recognizing that it does not need a PReP partition for this system (PowerNV/OPAL):

platform : PowerNV
model : 8247-42L
machine : PowerNV 8247-42L
firmware : OPAL v3

I think the original replication test system tripped across this issue because the system was being used in different test roles, and sharing disk resources between different boot environments. The alternate disk found with a PReP partition was being used by a PowerKVM guest when doing testing for that environment. This is what the PowerKVM guest environment reports:

platform : pSeries
model : IBM pSeries (emulated by qemu)
machine : CHRP IBM pSeries (emulated by qemu)

So, it looks like the installer partitioning code is correctly handling PowerNV vs CHRP. It's the grub bootloader logic that wants to overwrite the first PReP partition it finds (if one doesn't exist on the install drive) that is in question. Was there some historical reason for not wanting a PReP partition on some other drive?

Not that I can think of. Thing is, automatic partitioning should always set up an 8Mb PReP partition on the driver you're trying to install to. There are no special cases depending on the subtype for ppc64el; and you didn't boot with powerpc (which could have had special rules).

In 14.04, ppc64el uses the same rules for partitioning as ppc/chrp_ibm, which also always creates a prep partition.

In other words, even if it's not being used, the prep partition is always created on the systems; there are apparently no exceptions.

Looks like what is happening might be that due to the partitioning setup used (do you set up RAID too?), and probably in part because of bug 1487365, the wrong partitions are set with the prep flags, and sdb1 doesn't get it (even though it's already set up as a PReP partition of 8Mb size). When did you last try the install? On or after 2016-01-22?

------- Comment (attachment only) From <email address hidden> 2016-01-25 14:38 EDT-------

------- Comment From <email address hidden> 2016-01-25 14:41 EDT-------
The last install test was run on 2016-01-14. The system hasn't been needed, so it was left in the partial install state in case additional testing was needed. I blocked the installer's view of /dev/sda and let it complete so that additional post-install data could be gathered.

No RAID was used for this install. Once booted, the partition information shown for the new install disk is:

# parted /dev/sdb print
Model: IBM IPR-0 5DA5D300 (scsi)
Disk /dev/sdb: 1143GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
1 1049kB 8389kB 7340kB boot
2 8389kB 1096GB 1096GB ext4
3 1096GB 1143GB 46.2GB linux-swap(v1)

I think it would be necessary to try another install. Since the 14 (exactly on 2016-01-21) I uploaded fixes to partman-prep which might well improve on the situation here.

The jist of it, at least, is that there already is a PReP partition being created on your system (the boot partition listed in your comment), except the flags might not be set properly on the device.

------- Comment From <email address hidden> 2016-01-27 16:14 EDT-------
I did another install with latest netboot image (Jan 26). It did not change the behavior, and failed in the same way. I again deleted /dev/sda from the installer's view and let it complete. The attached sosreport is from that install test.

Is there some special install option I should be passing in order to get it to pick up your latest partman-prep update?

------- Comment (attachment only) From <email address hidden> 2016-01-27 16:11 EDT-------

------- Comment From <email address hidden> 2016-01-27 16:21 EDT-------
BTW: I did do another install with apt-setup/proposed=true in place, but it also failed in the same way.

------- Comment From <email address hidden> 2016-02-01 11:01 EDT-------
No partman-prep listed in syslog in this bug too.

------- Comment From <email address hidden> 2016-02-02 05:14 EDT-------
The discussion details from the other bug:
---

Theoretically, PowerNV doesn't need a PReP partition
(because the Petitboot bootloader directly searchs for, and reads, grub.cfg files)

However, the partman-prep udeb packages ships the prep_sync_flag script,
which scans for all PReP partitions and sets (syncs) up their PReP flag.

I believe that script shold help with this problem, as it seems the lack
of the PReP flag being set causes grub-installer to pick up an incorrect device.

------- Comment on attachment From <email address hidden> 2016-02-02 05:11 EDT-------

Hi @mathieu-tl

This patch has been verified to resolve this problem (thanks Kevin), and another one which is being mirrored.

The problem is partman-prep is currently not downloaded on PowerNV, but we need prep_sync_flag to set the PReP flag on the correct partition, which lines it up for grub-installer.

Details:
===

~ # tail /proc/cpuinfo
...
platform : PowerNV
model : 8286-42A
machine : PowerNV 8286-42A
firmware : OPAL v3

Without patch:
---

~ # archdetect
ppc64el/unknown

~ # grep partman-prep /var/log/syslog
~ #

With patch:
---

~ # archdetect
ppc64el/chrp_ibm

~ # grep partman-prep /var/log/syslog
Feb 1 13:11:34 anna[2929]: DEBUG: retrieving partman-prep 27ubuntu1
Feb 1 13:11:34 anna[2929]: 2016-02-01 13:11:34 URL:http://ports.ubuntu.com/ubuntu-ports//pool/main/p/partman-prep/partman-prep_27ubuntu1_ppc64el.udeb [13126/13126] -> "/var/cache/anna/partman-prep_27ubuntu1_ppc64el.udeb" [1]
~ #

Hi @mathieu-tl,

> This patch has been verified to resolve this problem (thanks Kevin), and another one which is being mirrored.

The other bug's mirroring has been cancelled. It's been identified as too similar to this one.

FYI, in case you may see it elsewhere, hope this description may save you some debug time:
- grub-installer fails because it picks an individual path of an multipath device; the lack of prep_sync_flag caused no PReP partition flag to be set on the multipath device (apparently, having the prep flag set underneath doesn't reflect it on the multipath device).

------- Comment From <email address hidden> 2016-02-02 11:30 EDT-------
*** Bug 136186 has been marked as a duplicate of this bug. ***

The attachment "Patch for libdebian-installer to detect PowerNV as powerpc/chrp_ibm" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

------- Comment From <email address hidden> 2016-02-04 11:18 EDT-------
*** Bug 136048 has been marked as a duplicate of this bug. ***

So this is in fact a bug in libdebian-installer, which got fixed in utopic... Marking Fix Released, and I'll open a task for trusty.

Hello bugproxy, or anyone else affected,

Accepted libdebian-installer into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libdebian-installer/0.88ubuntu5.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

------- Comment From <email address hidden> 2016-02-05 11:11 EDT-------
Since this is an install-time issue, we'll need an updated ubuntu-installer initrd.gz image to test (that's what we were using for our internal validation of the proposed patch). The files found at

http://ports.ubuntu.com/ubuntu-ports/dists/trusty-proposed/main/installer-ppc64el/current/images/wily-netboot/ubuntu-installer/ppc64el/

are still dated 26-Jan-2016

Please let us know when those netboot images have been updated.

Thanks.

------- Comment From <email address hidden> 2016-02-08 11:59 EDT-------
Hi
I still see the netboot images from 1/26, When will the images with the fix be available?

Hello bugproxy, or anyone else affected,

Accepted debian-installer into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debian-installer/20101020ubuntu318.35 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

------- Comment From <email address hidden> 2016-02-09 15:07 EDT-------
The latest updated netboot initrd image at http://ports.ubuntu.com/ubuntu-ports/dists/trusty-proposed/main/installer-ppc64el/current/images/wily-netboot/ubuntu-installer/ppc64el/ does not appear to contain the patch changes.

An install test using those new netboot images and specifying apt-setup/proposed=true still failed. archdetect is still returning "ppc64el/unknown"

~ # archdetect
ppc64el/unknown

------- Comment (attachment only) From <email address hidden> 2016-02-09 15:08 EDT-------

I would retest this; if you consistently use the "current" link you might not have retrieved the new image. It is necessary in this case to make sure the image retrieved is indeed the one from:

http://ports.ubuntu.com/ubuntu-ports/dists/trusty-proposed/main/installer-ppc64el/20101020ubuntu318.35/images/wily-netboot/

Setting back to verification-needed; I'm a little suspicious of the timing here since the images have been reportedly published at 15:39 (unless that was UTC), and I see the comment marked at 15:07 EDT...

------- Comment From <email address hidden> 2016-02-10 10:30 EDT-------
Hi Mathieu.

Our test team tested using the specific URL you provided and also continue to encounter the failure. I am currently re-running my own test for confirmation of previous timelines.

(In reply to comment #50)
> HI,
>
> Tried installing ubuntu14.04.4 using netboot install which is pointed to
> 'http://ports.ubuntu.com/ubuntu-ports/dists/trusty-proposed/main/installer-
> ppc64el/20101020ubuntu318.35/images/wily-netboot/' this netboot image and
> it still fails with same error as earlier on Habenaro system bmc-ray.
>
>
>
> ????????? [!!] Install the GRUB boot loader on a hard disk ??????????
> ? ?
> ? Installation step failed ?
> ? An installation step failed. You can try to run the failing item ?
> ? again from the menu, or skip it and choose something else. The ?
> ? failing step is: Install the GRUB boot loader on a hard disk ?
> ? ?
> ? <Continue> ?
> ? ?
> ?????????????????????????????????????????????????????????????????????
>
>
> Regards,
> Indira

------- Comment From <email address hidden> 2016-02-10 10:47 EDT-------
My retest succeeded, so I'm re-marking this bug back to verification-needed while I work on resolving why the test team's test is failing. Please stand by a bit.

------- Comment From <email address hidden> 2016-02-10 11:12 EDT-------
My current tests are good. The archdetect check is now returning the expected information:

~ # archdetect
ppc64el/chrp_ibm

The other test system has a proxy involved, so we are chasing internal issues related to that at the moment.

Marking this as "verification-done" based on the direct-connect tests done this morning.

This bug was fixed in the package libdebian-installer - 0.88ubuntu5.2

---------------
libdebian-installer (0.88ubuntu5.2) trusty; urgency=medium

  * Detect IBM PowerNV platform as powerpc/chrp_ibm. (LP: #1523637)

 -- Mauricio Faria de Oliveira <email address hidden> Thu, 04 Feb 2016 18:50:25 -0500

The verification of the Stable Release Update for libdebian-installer has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package debian-installer - 20101020ubuntu318.35

---------------
debian-installer (20101020ubuntu318.35) trusty; urgency=medium

  * No-change rebuild to catch the libdebian-installer fix for PowerNV
    support. (LP: #1523637)

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 08 Feb 2016 14:52:26 -0500