IBP: wrong variant of GBUB installed => booting from a huge hard drive (10+ TB) failed

Won't fix for the 6.0.x milestone as IBP has an experimental status there

Please add diagnostic snapshot.

Diagnostic snapshot in attachment.

According to logs, IBP didn't try to roll out the image. It was not able to find the proper block device.

It's not clear what happens.

fuel-agent log stops at
2015-06-03 17:08:14.734 9132 DEBUG fuel_agent.utils.utils [-] Trying to execute command: lvcreate -L 32768m -n swap os

any 'lvs'/'pvs'/another lvm related command gets stuck.

[root@bootstrap ~]# pvs
^C Interrupted...
  Giving up waiting for lock.
  /var/lock/lvm/V_os: flock failed: Interrupted system call
  Can't get lock for os
  Skipping volume group os
  PV VG Fmt Attr PSize PFree
  /dev/sda5 vm lvm2 a-- 10.82t 10.82t
[root@bootstrap ~]# pvs
^C Interrupted...
  Giving up waiting for lock.
  /var/lock/lvm/V_os: flock failed: Interrupted system call
  Can't get lock for os
  Skipping volume group os
  PV VG Fmt Attr PSize PFree
  /dev/sda5 vm lvm2 a-- 10.82t 10.82t
[root@bootstrap ~]# lvs

Also noticed that if 'provision' is executed in background just as 'provision &' it will fihish proreply.

I'm not sure how to properly reproduce lvm deadlock and how to avoid it.

> root 17655 0.0 0.0 25596 4612 pts/0 SN+ 17:17 0:00 lvcreate -L 32768m -n swap os

moving to mos-linux team to fix lvm or at least to understand why it happened

Many thanks Sergii Golovatuik for help.

we were able to strace lvcreate.

https://bugzilla.redhat.com/show_bug.cgi?id=997223

lvm is waiting for user's prompt when trying to wipe signature.

"WARNING: <signature name> signature detected on <device name>. Wipe it? [y/n]"

$ lvm dumpconfig |grep wipe
    wipe_signatures_when_zeroing_new_lvs=1

so in order to fix lvm locking, we should set it to 0 in /etc/lvm.conf.

LVM supports several command line options to override this behavior: --yes or --wipesignatures. Please use one of them in order to ensure needed behavior.

Related fix proposed to branch: master
Review: https://review.openstack.org/188356

lvcreate wasn't a root cause. it was entirely new bug.

with lvcreate patch applied, provisioning scripts was succeeded and was completed without any errors//traces/hangs.

it looks like it's GRUB2 issue. It only recognized 2 disks out of 3.

/boot partition was installed in /dev/sda (which is 11.5TB h/w array). GRUB didn't see /dev/sda at all.

As for ugly workaround, I can add some logic into agent to prevent choosing very large disks (2TB+) to be chosen for placing /boot on them.

Let mos-linux team take a look at it. Looks like completely GRUB related issue.

fuel-agent should install grub-efi instead of grub-pc on hard drives larger than 2 TB.

At the moment fuel-agent always installs grub-pc [1]. That version of grub uses BIOS INT13h to access hard drives.
In theory current BIOSes should support LBA48 INT 13h extension so virtually any drive should be readable.
However in practice all firmware suck so quite a number of BIOSes support at most LBA32 (which yields 2 TB size limit).
In order to avoid fighting the crappy firmware fuel-agent should install EFI based grub on hard drives larger than 2 TB.

[1] https://github.com/stackforge/fuel-web/blob/master/fuel_agent/fuel_agent/drivers/nailgun.py#L480

Just my 2 cents about how it all works.

If you need just a solution - they are written at the end of comment.

TL;DR

There are BIOS or UEFI boot shcemes and MBR or GPT disk schemes. You CANNOT use UEFI with MBR - it just will not work as long as GPT is mandatory UEFI part.
Also we have bootloader - in our case, grub (when I said 'grub', I do not mean grub legacy).
So, we can have next cases when user want to install operating system:

1. BIOS+MBR
In such case you cannot use (primarily) hard disks larger than 2TB - you can't address that with MBR partitioning scheme. Usually disk partitioning looks like:
- MBR
- MBR gap (your partitions will start from sector 63 due to old DOS restriction about partitioning on cylinder boundaries, so we have about 32 free unused kilobytes there)
- Your partitions (usually, /boot then /)

That thing - MBR gap is used by grub for store stage 1.5 bootloader (core.img in grub2, if I remember it right). So, your MBR point with grub stage 1 point to grub stage 1.5 and then grub stage 1.5 point to your /boot partition with grub stage 2.

2. BIOS+GPT
In this case you can use much larger disks - up to some insane numbers in PB. Usually disk partitioning looks like:
- Protected MBR
- GPT (mbr is part of it, actually)
- Special bios boot partition (usually about 1Mb in size)
- Your partitions (/boot and /)

As long as GPT doesn't have such requirement as partitioning by cylinder boundaries, we don't have 32 kilobytes space there - partitions start just after GPT itself. But we need to store out grub stage 1.5 loader somewhere. For such case we create SPECIAL partition with code ef02, named bios_boot. It replaces old MBR gap and grub stored it's stage 1.5 loader there. But it is an old good grub there, nothing actually new.

3. UEFI+GPT
As long as UEFI is absolutely new standard, old grub cannot be loaded by UEFI firmware. There are especial new one (actually, not - it just ported to meet UEFI specifications) grub-efi that should be installed. Usually disk partitioning looks like:
- Protected MBR (don't used in most of cases)
- GPT (mbr is part of it, actually)
- Analog of /boot partition for UEFI standard, code ef00 (EFI partition, some FAT-related fs with bootloader on it)
- Your partitions (/boot and /, you can place /boot in efi partition)

In this case there are no such thing as stage 1.5 for grub. GPT point right to your loader in EFI partition, nothing else. But it is new grub, incompatible with old one. Also you need to know that your /boot partition should be placed in first 2TB of your hard disk.

So, you can use next solutions there:
1. Don't support GPT and UEFI. No need to rewrite something, this bug will not closed.
2. Support GPT, don't support UEFI. You just need to rewrite your partition scheme, it's all. Fully compatible with old one.
3. Support GPT and UEFI. You will need to do some rather complex logic to understand what you need to do. Actually, we already have bug with UEFI support for 7.0, so I rather don't recommend do it now.

Fix proposed to branch: master
Review: https://review.openstack.org/189698

Stanislaw,

Rewriting partition scheme won't help because grub does not see huge disk no matter how you make partitions.

> 2. BIOS+GPT
> In this case you can use much larger disks - up to some insane numbers in PB.

Unless you've got a buggy (UEFI) firmware which does not support LBA48 in BIOS emulation mode.

> Usually disk partitioning looks like:
> - Protected MBR
> - GPT (mbr is part of it, actually)
> - Special bios boot partition (usually about 1Mb in size)
> - Your partitions (/boot and /)

Actually fuel-agent does not create bios boot partition (ef02), and certain firmwares refuse to boot from such a drive in a BIOS emulation mode.

> That thing - MBR gap is used by grub for store stage 1.5 bootloader (core.img in grub2, if I remember it right).
> So, your MBR point with grub stage 1 point to grub stage 1.5 and then grub stage 1.5 point to your /boot partition with grub stage 2.

There's no such thing as stage 1.5 in grub2, see http://www.gnu.org/software/grub/manual/grub.html#Images

> Actually fuel-agent does not create bios boot partition (ef02), and certain firmwares refuse to boot from such a drive in a BIOS emulation mode.

As far as I understand, we are talking about this small partition https://github.com/stackforge/fuel-web/blob/master/fuel_agent/fuel_agent/drivers/nailgun.py#L172-L175 So, Fuel agent does create this partition, but it's size 24 MB (for some reasons).

> Actually fuel-agent does not create bios boot partition (ef02)

I've filed a bug on this issue: https://bugs.launchpad.net/fuel/+bug/1463708

> As long as UEFI is absolutely new standard

UEFI 2.0 spec [1] is 9 years old, EFI version of grub2 is almost 7 years old.

[1] http://www.uefi.org/sites/default/files/resources/UEFI_Specification_2_and_Errata_Sept16_08.pdf

> I've filed a bug on this issue: https://bugs.launchpad.net/fuel/+bug/1463708

Apparently BIOS boot partition is here. However EFI system partition is missing (which might cause firmware misbehavior)

> grub rescue> insmod ext2
> grub rescue> ls (hd0,gpt1)/boot
> error: unknown filesystem

Actually the 1st GPT partition created by provisioning script is a BIOS boot partition (which is kind of wrong, but that's a different problem). That partition is not supposed to hold any filesystem (grub2 stores its core image into that partition). Therefore...

> No partition contains /boot filesystem.

... this conclusion is wrong.

> Trying to load : pxelinux.cfg/<MAC> ok
> Booting...
> error: no such device: <some uuid>.
> Entering rescue mode...
> grub rescue>

Cobbler might have sent a wrong configuration (root=UUID which does not actually exists, or something).
We need to investigate this bug more carefully. That is,
- check if another partition contains /boot filesystem
- if yes, check if grub is able to boot from that (with manually passed root= argument)

> However EFI system partition is missing (which might cause firmware misbehavior)

Alexei Sheplyakov, are you sure that absence of EFI partition could somehow interfere with not found disk by grub2? Could you clarify how it could help?

Some [U]EFI firmware in BIOS emulation mode refuses to boot from a hard drive which 1) has GPT, 2) has no EFI system partition
(such a hard drive is not available via BIOS int13h).

Reviewed: https://review.openstack.org/189698
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=059831145c657a5eaba563b18652351a459de566
Submitter: Jenkins
Branch: master

commit 059831145c657a5eaba563b18652351a459de566
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue Jun 9 15:56:26 2015 +0300

    IBP: /boot on small hard drive if possible

    On some hardware GRUB can not see huge hard drives (>2T)
    due to firmware bugs, so we'd better avoid placing
    /boot on such drives if it is possible.

    Change-Id: I71e0255820edef4725208db777bf5e0bf8d3958c
    Partial-Bug: #1461126

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/190642

Release note:

Currently, /boot partition is created on a disk which is smaller than 2T, if there is such a disk on a node. If a node has only huge disks (>2T) and if BIOS does not support huge disks (LBA48), the node will not be able to boot. If it is a bug of UEFI BIOS emulation mode, than it will be addressed in the future by implementing of UEFI support.

Reviewed: https://review.openstack.org/190642
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=fa44199924c630ce0356cfb74bcf4e94f6d9242e
Submitter: Jenkins
Branch: stable/6.1

commit fa44199924c630ce0356cfb74bcf4e94f6d9242e
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue Jun 9 15:56:26 2015 +0300

    IBP: /boot on small hard drive if possible

    On some hardware GRUB can not see huge hard drives (>2T)
    due to firmware bugs, so we'd better avoid placing
    /boot on such drives if it is possible.

    Change-Id: I71e0255820edef4725208db777bf5e0bf8d3958c
    Partial-Bug: #1461126

Vladimir, can you please update bug status? I see commits merged, is it right to assume that the bug is closed?

This commit fixes only this particular case when we have more than one disk on a node and at least one of them is smaller that 2T. The root cause of the issue is UEFI's BIOS emulation bug. It does not support LBA48. The correct way to address this issue is to implement uefi support.