LVM device unavailable after 18.04 to 20.04 upgrade Timed out waiting for device /dev/mapper/s5lp8--v g-home

Bug #1874381 reported by John George
32
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Confirmed
High
Unassigned
lvm2 (Ubuntu)
Confirmed
High
Unassigned
Focal
Confirmed
High
Unassigned

Bug Description

After upgrading an LPAR configured with LVM from 18.04 to 20.04 /home is no longer mounted.
During boot the console shows Timed out waiting for device /dev/mapper/s5lp8--vg-home

Please see the attached dist-upgrade logs and console output for more detail.

Revision history for this message
John George (jog) wrote :
Revision history for this message
John George (jog) wrote :
tags: added: rls-ff-incoming
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu-release-upgrader (Ubuntu):
status: New → Confirmed
Revision history for this message
Serhiy (xintx-ua) wrote :

See https://bugs.launchpad.net/bugs/1877473 for workarounds. While this report was first I was able to provide workarounds. Would you agree to mark this bug as a duplicate, please?

Changed in ubuntu-release-upgrader (Ubuntu):
importance: Undecided → High
Frank Heimes (fheimes)
tags: added: installer
Revision history for this message
Serhiy (xintx-ua) wrote :

Ok, copying my workarounds here and closing the other report:

Volume Group metadata had to be upgraded manually with

vgck --updatemetadata yourvgname

The VG name was somehow changed from lvm-name to just name and I had to edit /etc/crypttab and update-initramfs -u -k all

Also cryptsetup-initramfs package was deleted for some reason, had to re-install it in chroot. Maybe it happened because I have disabled installing recommended and suggested packages or maybe because none of *buntu-desktop packages are installed on my system. Not sure if I have to report it as a separate problem, please advise.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → Confirmed
importance: Undecided → High
tags: removed: installer
Changed in ubuntu-release-upgrader (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu-release-upgrader (Ubuntu) → lvm2 (Ubuntu)
tags: removed: rls-ff-incoming
Changed in lvm2 (Ubuntu Focal):
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Steve Langasek (vorlon) wrote :

John, are you able to confirm that a 'sudo vgck --updatemetadata s5lp8-vg' fixes this for you?

Also, before running that command, can you post the output of 'sudo vgdisplay' on this system?

Revision history for this message
Steve Langasek (vorlon) wrote :

You might also want to try 'sudo vgck -t s5lp8-vg' first before committing changes with --updatemetadata.

Changed in lvm2 (Ubuntu Focal):
milestone: none → ubuntu-20.04.1
tags: added: id-5ec6a06d9ddd891b690ef2e4
Revision history for this message
veribaka (veribaka) wrote :

I too tried to upgrade from 18.04 and ended up with a busybox, Ubuntu not finding my vg. Not sure how to enter those commands from a busybox Steve.

Re-imaging the device with 18.04 for the time being.

Revision history for this message
Serhiy (xintx-ua) wrote :

You need to either execute these commands after upgrade and _before_ rebooting or chroot from a LiveUSB boot.

Revision history for this message
veribaka (veribaka) wrote :

Reporting back:

"sudo vgck -t vg" reports that no changes would be made, it seems to be looking in /dev/sda rather than /dev/nvme0n1.

"sudo vgscan" first reports that /dev/sda open failed and then successfully detects the vg using metadata type lvm2.

Revision history for this message
Steve Langasek (vorlon) wrote :

awaiting feedback from the original reporter to confirm that these particular commands are applicable to his install.

Changed in lvm2 (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
veribaka (veribaka) wrote :

Is there a time frame to consider that the original reporter will not be returning to this discussion?

Should I open my own bug report with my logs?

Revision history for this message
Brian Murray (brian-murray) wrote : Re: [Bug 1874381] Re: LVM device unavailable after 18.04 to 20.04 upgrade Timed out waiting for device /dev/mapper/s5lp8--v g-home

On Fri, Jun 05, 2020 at 09:38:35AM -0000, veribaka wrote:
> Is there a time frame to consider that the original reporter will not be
> returning to this discussion?
>
> Should I open my own bug report with my logs?

Please do open your own bug report with the relevant log files and make
a comment in this bug report with the number of your new report.

Thanks!
--
Brian Murray

Revision history for this message
John George (jog) wrote :

I redeployed the system where this failures was original observed and recreated the issue.
Running 'sudo vgck --updatemetadata s5lp8-vg' did work around this bug, when run after the upgrade but prior to the reboot.

Here is the vgdisplay output requested by Steve:

ubuntu@s5lp8:~$ sudo vgdisplay
  --- Volume group ---
  VG Name s5lp8-vg
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 4
  VG Access read/write
  VG Status resizable
  MAX LV 0
  Cur LV 3
  Open LV 3
  Max PV 0
  Cur PV 1
  Act PV 1
  VG Size <64.00 GiB
  PE Size 4.00 MiB
  Total PE 16383
  Alloc PE / Size 16377 / 63.97 GiB
  Free PE / Size 6 / 24.00 MiB
  VG UUID 3B7wkv-AIvn-5A7N-pmWB-7y7h-ylkd-jUIfRd

Changed in lvm2 (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

So one solution to this would be to run "vgck --updatemetadata" on each vg in postinst... would there be downside to that?

Revision history for this message
veribaka (veribaka) wrote :
tags: added: fr-81
Changed in lvm2 (Ubuntu Focal):
milestone: ubuntu-20.04.1 → ubuntu-20.04.2
Changed in lvm2 (Ubuntu Focal):
milestone: ubuntu-20.04.2 → focal-updates
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

> So one solution to this would be to run "vgck --updatemetadata" on each vg in postinst... would there be downside to that?

how slow is it on very very large LVMs? Is it distructive? Maybe something to check with devops-y people, like I.S. / bootstack?

Revision history for this message
Frank Heimes (fheimes) wrote :

Sorry, I can't answer that for huge LVM installations.
But what I can say is that it is for s390x installations (multipath with LVMs on top) a kind of immediately responding.

One thought is to check/do this for example at the end of the dist-upgrade process, where a potential delay wouldn't be a very huge drawback (compared to a system that doesn't come up again after reboot).

Frank Heimes (fheimes)
Changed in lvm2 (Ubuntu):
assignee: Canonical Foundations Team (canonical-foundations) → nobody
Changed in ubuntu-z-systems:
assignee: Skipper Bug Screeners (skipper-screen-team) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.