ISST-LTE: root mpath device unavailable after installation

Default Comment by Bridge

Default Comment by Bridge

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1526984/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

------- Comment From <email address hidden> 2015-12-28 15:19 EDT-------
*** Bug 134385 has been marked as a duplicate of this bug. ***

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

------- Comment on attachment From <email address hidden> 2016-01-11 07:58 EDT-------

Hi Canonical @mathieu-tl,

This patch implements multipathd in the initramfs, and is reported to resolve the problem
(tested by developer and test team for more than 10 reboots each).

It's on top of the patch in LP 1432062 comment 10 (requirement, since multipathd -B might create devmaps w/ spaces in names).

Is it still in the window for 14.04.4?
Thanks!

Default Comment by Bridge

This bug was fixed in the package multipath-tools - 0.5.0-7ubuntu10

---------------
multipath-tools (0.5.0-7ubuntu10) xenial; urgency=medium

  * debian/patches/0052-readonly-bindings_multipathd.patch,
    debian/patches/0053-readonly-bindings_multipathd_prod.patch: support -B to
    allow multipathd to handle cases where the bindings file is read-only.
    (LP: #1526984)
  * debian/initramfs/hooks: install multipathd and required directories.
  * debian/initramfs/local-bottom, debian/rules: install local-bottom for
    initramfs.
  * debian/initramfs/local-premount: reload all maps to make sure they're
    indeed loaded and ready before we end premount.
  * debian/initramfs/local-top: run multipathd rather than a one-off call to
    multipath so that new paths can be correctly added as detected while we're
    still in the initramfs.
  * debian/initramfs/local-bottom: remember to stop multipathd.
  * debian/patches/git-kpartx-support-spaces-in-dev-names-b407050a.patch: deal
    with spaces in device names in kpartx too (LP: #1432062)

 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 27 Jan 2016 10:42:51 -0500

------- Comment From <email address hidden> 2016-02-02 17:47 EDT-------
*** Bug 136463 has been marked as a duplicate of this bug. ***

Hi @mathieu-tl,

Mathieu Trudel-Lapierre (mathieu-tl) on 2016-02-01
> description: updated
> Changed in multipath-tools (Ubuntu Trusty):
> status: New → Incomplete

Can you clarify the status 'incomplete' for this bug?
Is that indeed an information request for submitter (where, please) or just for some other change you'll do on the description field?

Thanks,

------- Comment From <email address hidden> 2016-02-04 11:54 EDT-------
Is multipath-tools - 0.5.0-7ubuntu10 going to be available in the repository anytime soon? We want to verify the problem is fixed before completing our testing for 14.04.4, which we will be done with early next week...

------- Comment From <email address hidden> 2016-02-04 12:13 EDT-------
Hi Mikhail,

(In reply to comment #60)
> Is multipath-tools - 0.5.0-7ubuntu10 going to be available in the repository
> anytime soon? We want to verify the problem is fixed before completing our
> testing for 14.04.4, which we will be done with early next week...

That version is for 16.04 - it is already fix released/available in the repository.

The version for 14.04.4 (something like 0.4.9-3ubuntu7.<something>) is not yet fix released/available.

There's no defined date for when it will be made available - but it is in the list of multipath fixes that Canonical's team is aware of and working on. Perhaps there will be single update w/ more than 1 bug fixed.

There's likely a number of tasks on their side as well, and this one is prioritized as Critical, so things should happen as best as possible.

------- Comment From <email address hidden> 2016-02-04 12:17 EDT-------
Got it, thanks! We'll see if the problem is gone on 16.04 next week.

Hello bugproxy, or anyone else affected,

Accepted multipath-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.4.9-3ubuntu7.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Mathieu,

As noted in LP:1538775, version 0.4.9-3ubuntu7.8 still isn't resolving the issue for me, as while my system does start (im not dropping to a rescue shell), the multipath daemon seems to be executed prior to the disks being discovered/udev starting and so it boots using sda for root instead of mpath. Running 'multipath -v5' post-boot shows that it wants to reconfigure pathing but cannot as the disks are in-use (understandably).

I've attached the dmesg log which shows when multipathd starts, the disks are detected, then udev starts.

Hum, that's kind of weird. You should see disks being picked up by multipath if they should, even if they are detected *after* multipath scripts run (because multipathd...). They wouldn't be detected before multipath runs, because then they would most likely be remapped anyway (unless something else claims them).

Could you file a new bug for this, including the output of multipath -v4 taken from the initramfs? You may stop the boot process by adding "break=pre-multipath" to break boot just before the multipath scripts. At that point, you'll need to modprobe dm_multipath; and then you should be able to run multipath -v4. If you exit from there boot should complete (hopefully succesfully, since there will have been some extra delay).

Hi @stu-g,

Thanks for your report.
I think the problem you're experiencing (handled in LP 1538775) is one we've been discussing/trying to get around for some time now.
Since scsi_wait_scan was removed, there's no way to really wait /on the SCSI scan/.
For that objective, udevadm settle won't help, as the SCSI scan does not generate any udev events.. but only the disks found in the process do..
So, if your environment hits that intermediary spot (ie, the SCSI scan only starts to add the disks to the kernel /after/ udevadm settle finished w/out more events to process), then you'd hit the problem nonetheless.

Unfortunately, there's not something we have came up with yet for that kind of problem. It's been a head scratcher for some time, as mentioned.

For this particular bug, the reporter verified the patches submitted (which implement multipathd in the initramfs) and it does handle some async discoveries nicely.
So, unfortunately it doesn't fix one other scenario (which seems hard to fix), but that doesn't make the fix wrong/unuseful on its own / the scenario it happens to fix/work on.

I verified this works correctly on 2 LPARs, including the original LPAR from the bug report. (Thanks, Michail)
Marking as verification-done.

Installed the version from -proposed:

 root@pinelp2:~# dpkg -l | grep 0.4.9-3ubuntu7.8
 ii kpartx 0.4.9-3ubuntu7.8 ppc64el create device mappings for partitions
 ii kpartx-boot 0.4.9-3ubuntu7.8 all Provides kpartx during boot
 ii multipath-tools 0.4.9-3ubuntu7.8 ppc64el maintain multipath block device access
 ii multipath-tools-boot 0.4.9-3ubuntu7.8 all Support booting from multipath devices

 root@pinelp3:~# dpkg -l | grep 0.4.9-3ubuntu7.8
 ii kpartx 0.4.9-3ubuntu7.8 ppc64el create device mappings for partitions
 ii kpartx-boot 0.4.9-3ubuntu7.8 all Provides kpartx during boot
 ii multipath-tools 0.4.9-3ubuntu7.8 ppc64el maintain multipath block device access
 ii multipath-tools-boot 0.4.9-3ubuntu7.8 all Support booting from multipath devices

Notice the 2 LPARs (pinelp2 and pinelp3) booted from an individual path.
On pinelp2, it was required to manually include the rootfs device's WWID (below).

 root@pinelp2:~# mount | grep ' / '
 /dev/sdq2 on / type ext4 (rw,errors=remount-ro)

 root@pinelp3:~# mount | grep ' / '
 /dev/sdg2 on / type ext4 (rw,errors=remount-ro)

Make sure the rootfs device is listed in /etc/multipath/wwids in the initramfs:

- This is required on pinelp2:

 root@pinelp2:~# multipath -c /dev/sdq
 /dev/sdq is not a valid multipath device path

 root@pinelp2:~# multipath -d -v3 | grep 'sdq: uid ='
 Feb 05 11:04:14 | sdq: uid = 36005076304ffc4410000000000000161 (callout)

 root@pinelp2:~# multipath -d -v3 | grep 'sdq: uid =' | awk '{ print "/" $8 "/" }' >> /etc/multipath/wwids

 root@pinelp2:~# multipath -c /dev/sdq
 /dev/sdq is a valid multipath device path

 root@pinelp2:~# update-initramfs -u
 update-initramfs: Generating /boot/initrd.img-4.2.0-27-generic

- Not required on pinelp3:

 root@pinelp3:~# multipath -c /dev/sdg
 /dev/sdg is a valid multipath device path

 Already in WWIDs, so this is likely a timing problem in the boot (works some of the times).

Test it on both LPARs.

 root@pinelp2:~# reboot
 <...>

 root@pinelp3:~# reboot
 <...>

The 2 LPARs booted successfully from a multipath device:

 root@pinelp2:~# mount | grep ' / '
 /dev/mapper/mpath3-part2 on / type ext4 (rw,errors=remount-ro)

 root@pinelp3:~# mount | grep ' / '
 /dev/mapper/mpath4-part2 on / type ext4 (rw,errors=remount-ro)

For details: console log for pinelp2:

 Notice:
 - The scsi disks are added /after/ multipathd is started.
 - There *IS* a wait of some seconds at this point.
   So, the udevadm settle is indeed working.

 Loading, please wait...
 [ 0.750797] systemd-udevd[206]: starting version 204
 Begin: Loading essential drivers ... done.
 Begin: Running /scripts/init-premount ... done.
 Begin: Mounting root file system ... Begin: Running /scr...

Well, to be honest there isn't anything we can do in multipath-tools or udev if the scans aren't generating events. The best option then is to look at the driver, if it permits some control of how the scans happen, and if not, why.

Default Comment by Bridge

Hello bugproxy, or anyone else affected,

Accepted multipath-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.4.9-3ubuntu7.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This was already verified successfully before; the additional update does not need reverification, only checking the regression in bug 1543430.

This bug was fixed in the package multipath-tools - 0.4.9-3ubuntu7.9

---------------
multipath-tools (0.4.9-3ubuntu7.9) trusty; urgency=medium

  * debian/patches/kpartx-support-device-names-with-spaces.patch: fix loopback
    files unmapping. (LP: #1543430)

multipath-tools (0.4.9-3ubuntu7.8) trusty; urgency=medium

  * debian/patches/kpartx-support-device-names-with-spaces.patch: deal with
    spaces in device names in kpartx too (LP: #1432062)
  * debian/initramfs/local-premount: wait for udev to settle before mounting
    so the by-uuid/ symlinks have a chance to be updated by udev rules.
    (LP: #1503286)
  * Allow device detection all through the initramfs: run multipathd instead
    of only scanning once for devices, so those that come up slower can still
    be used as a root device (LP: #1526984):
    - debian/patches/0050-readonly-bindings_prefix.patch,
      debian/patches/0051-readonly-bindings_multipath.patch,
      debian/patches/0052-readonly-bindings_multipathd.patch,
      debian/patches/0053-readonly-bindings_multipathd_prod.patch: support -B
      to allow multipathd to handle cases where the bindings file is read-only.
    - debian/initramfs/hooks: install multipathd and required directories.
    - debian/initramfs/local-premount: also reload all maps to make sure
      they're ready before we mount.
    - debian/initramfs/local-top: run multipathd rather than a one-off call to
      multipath so that new paths can be correctly added as detected while
      we're still in the initramfs.
    - debian/initramfs/local-bottom: remember to stop multipathd.
    - debian/initramfs/local-bottom, debian/rules: install local-bottom for
      initramfs.
  * debian/patches/lp1496210_add_IBM_XIV_defaults.patch: add support (default
    config values) for the IBM 2810XIV storage system. (LP: #1496210)
  * debian/patches/0054-kpartx-update-option.patch: run kpartx -u rather than
    kpartx -a, so as to remove old partition entries if the partition table
    has changed. (LP: #1473903)
  * debian/patches/multipath_enable_sync_support_1b8082c8.patch,
    debian/patches/kpartx_rely_on_udev_dev_creation_9a632fff.patch: synchronize
    udev, device-mapper and multipath, and let udev deal with creating device
    nodes and symlinks. (LP: #1486370)
  * debian/initramfs/local-top: drop scsi_wait_scan stanza, that module is no
    longer available. (LP: #1538775)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 09 Feb 2016 16:03:10 -0500

The verification of the Stable Release Update for multipath-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Default Comment by Bridge