software raid doesn't assemble before mount on boot

Thanks for your bugreport.

The bug for the backtrace is fixed now. The other bug about the degraded array is a bug with mdadm, I will create a taks for it.

Status changed to 'Confirmed' because the bug affects multiple users.

This bug was fixed in the package update-manager - 1:0.156.7

---------------
update-manager (1:0.156.7) precise; urgency=low

  [ Michael Vogt ]
  * DistUpgrade/DistUpgradeCache.py:
    - fix nvidia detection, thanks to Brian Murray
  * DistUpgrade/xorg_fix_proprietary.py:
    - fix crash when apt_pkg is not initialized (LP: #942106)
  * DistUpgrade/DistUpgradeController.py, DistUpgrade/mirrors.cfg:
    - support upgrades with commercial-ppas in the sources.list
      (LP: #939461) and add tests
  * UpdateManager/UpdateManager.py:
    - fix inaccurate string, thanks to JohnNapster (LP: #942590)

  [ Brian Murray ]
  * DistUpgrade/removal_blacklist.cfg:
    - Only blacklist unity-2d, not all packages whose names start with
      unity-2d. (LP: #940196)
 -- Michael Vogt <email address hidden> Mon, 05 Mar 2012 09:02:14 +0100

This patch to "/usr/share/initramfs-tools/scripts/local-premount/mdadm" solves the issue for me.

The attachment "mdadm.patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

An explanation: md detection and assembly is done by udev. Each time a linux_raid partition is detected, udev calls mdadm to do an incremental assembly.

But nothing is done to be sure incremental assembly is over before trying to mount the root partition.

This modification makes the "/usr/share/initramfs-tools/scripts/local-premount/mdadm" wait for the end of the incremental assembly before moving over.

I was able to test this patch using the following method: I added a printed line before and after the "wait_for_udev" command. I then rebooted a number of times and verified that the first line appeared from time to time before all MD device were assembled and that the last line always appeared after all MD device were assembled.

It is not just booting from a software raid, I have never tried that. For me, it was enough that I had a software raid in my computer. I was booting from an SSD.

I believe that any disk listed on the fstab failing to mount will lead to the dreaded initramfs console.

I'm still trying to reproduce/test this bug.
By the looks of it, adding wait_for_udev call is sensible.
On the other hand, udev incremental assembly has been disabled in a debian patch, which we inherited.
I am going to suggest dropping that patch & allow incremental udev assembly in mdadm package.

Setting multiple md arrays seems to be a good way to reproduce this bug
(see #682019). Or maybe just a big md array.

After that, it's just a timing issue. After rebooting several times, you will
probably run into this issue.

The Debian mdadm scripts uses an explicit assembly in initramfs and
seems to offer a lot more of fallback logic than we currently have.

Also, udev settle makes you wait for devices that might not be
necessary for boot.

The full Debian mdadm initramfs logic seems to be a good alternative
to the current logic.

This bug was fixed in the package mdadm - 3.2.5-1ubuntu1

---------------
mdadm (3.2.5-1ubuntu1) quantal; urgency=low

  * Merge from Debian testing. (LP: #920324) Remaining changes:
    - Call checks in local-premount to avoid race condition with udev
      and opening a degraded array.
    - d/initramfs/mdadm-functions: Record in /run when boot-degraded
      question has been asked so that it is only asked once
    - pass --test to mdadm to enable result codes for degraded arrays.
    - debian/control: we need udev and util-linux in the right version. We
      also remove the build dependency from quilt and docbook-to-man as both
      are not used in Ubuntus mdadm.
    - debian/initramfs/hook: kept the Ubuntus version for handling the absence
      of active raid arrays in <initramfs>/etc/mdadm/mdadm.conf
    - debian/initramfs/script.local-top.DEBIAN, debian/mdadm-startall,
      debian/mdadm.raid.DEBIAN: removed. udev does its job now instead.
    - debian/mdadm-startall.sgml, debian/mdadm-startall.8: documentation of
      unused startall script
    - debian/mdadm.config, debian/mdadm.postinst - let udev do the handling
      instead. Resolved merge conflict by keeping Ubuntu's version.
    - debian/mdadm.postinst, debian/mdadm.config, initramfs/init-premount:
      boot-degraded enablement; maintain udev starting of RAID devices;
      init-premount hook script for the initramfs, to provide information at
      boot
    - debian/mkconf.in is the older mkconf. Kept the Ubuntu version.
    - debian/rules: Kept Ubuntus version for installing apport hooks, not
      installing un-used startall script.
    - debian/install-rc, check.d/_numbers, check.d/root_on_raid: Ubuntu partman
      installer changes
    - debian/presubj: Dropped this unused bug reporting file. Instead use
      source_mdadm.py act as an apport hook for bug handling.
    - d/p/debian-changes-3.1.4-1+8efb9d1ubuntu4: mdadm udev rule
      incrementally adds mdadm member when detected. Starting such an
      array in degraded mode is possible by mdadm -IRs. Using mdadm
      -ARs without stopping the array first does nothing when no
      mdarray-unassociated device is available. Using mdadm -IRs to
      start a previously partially assembled array through incremental
      mode. Keeping the mdadm -ARs for assembling arrays which were for
      some reason not assembled through incremental mode (i.e through
      mdadm's udev rule).

  * Additional Ubuntu changes:
      - debian/initramfs/local-premount: add call wait_for_udev to wait a
      little longer for RAID devices to appear (LP: #942106)

  * Dropped Ubuntu changes:
    - Build udeb with -O2 on ppc64, working around a link error. Builds
      fine without it on debian.
    - rename debian/mdadm.vol_id.udev to debian/mdadm.mdadm-blkid.udev so
      that the rules file ends up with a more reasonable name. debian/rules
      changes for adding ubuntu's udev rule corresponding to mdadm. As we
      are now using 'upstream' udev rules see 3.2.3-2ubuntu2.

  * Changes to Ubuntu changes:
    - debian/source_mdadm.py: make apport hook python 2 and 3 compatible
      (LP: #1013171).

  * New upstream release closes this...

Although this is fix released, it's just an improvement by waiting for udev to settle.

Further improvements to RAID reliability will be done as part of:
https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-software-raid

Dmitrijs: What's the state of this SRU? will it make it to the point release?

...
* Upgrade to the -proposed package
...
This bug was fixed in the package mdadm-3.2.5-1ubuntu1
...

How to upgrade to a fixed version? install mdadm-3.2.5-1ubuntu1 from PPA?

Hello Håvar, or anyone else affected,

Accepted mdadm into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mdadm/3.2.5-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This has fixed the bug for me - thanks to the devs for doing the work.

Mark Ovens - could you please provide some detailed information regarding the testing that you did so that we can release the update to everyone? Thanks!

I only use the mdadm array for data, i.e. I don't boot from them. I was originally subscribed to one of the duplicate bugs. The problem I had was that booting would stop with a message about the disks not being ready and the option to 'S'kip or 'M'anual.

To work round this I added the `noauto` option to the /etc/fstab entry for the array and called mount(8) in /etc/rc.local

My testing of this fix has simply being to remove the `noauto` option from /etc/fstab and the call to mount(8) in /etc/rc.local.

I guess that this is not enough for those who boot from an array?

P.S. I am happy to try anything else you may require, but I'm away for 8 days from tomorrow.

Mark Ovens, you could do a test of:
1) connecting, assembling and mounting MD-ADM raid
2) $ sudo update-initramfs -u
3) reboot a couple of times and check if the boot succeeds each time.

I have had similar failure on all versions of precise on my Asus M4A785 with AMD X4 9850 with one drive on SiI 3132 Serial ATA Raid II Controller Boot from IDE drive and 6 SATA drives in raid5. Every boot failed with initramfs stop into busybox. Installing precise.proposed version of mdadm version 3.2.5-1ubuntu0.1 mentioned above fixed the problem with two successful boots with no problem.

mark@server7:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04 LTS
Release: 12.04
Codename: precise

mark@server7:~$ mdadm -V
mdadm - v3.2.5 - 18th May 2012

mark@server7:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdk[2] sdi[0] sdj[1] sdd[5] sdc[6](S) sdb[3]
      7814051840 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]

unused devices: <none>

Please ship it!

The 3.2.5-1ubuntu0.1 package fixes the problem for me through three reboots. I reported duplicate bug #776908 and was experiencing the problem consistently.

Hello Håvar, or anyone else affected,

Accepted mdadm into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mdadm/3.2.5-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Jean-Philippe, Håvar, are either of you able to confirm that the mdadm in precise-proposed fixes this issue for you?

It fixes the issue for me.

Ever since I have installed this version, this problem has disappeared. Previously, one over 3 boot did fail to assemble the software RAID volumes.

Thanks.

This does NOT fix this issue for me.

My system still boots up with some RAID arrays not running. Every single time.

This system has six SATA drives on two controllers. It contains seven RAID arrays, a mix of RAID 1, RAID 10, and RAID 6; all are listed in fstab. Some use 0.90.0 metadata and some use 1.2 metadata. The root filesystem is not on a RAID array (at least not any more, I got tired of that REAL fast) but everything else (including /boot and all swap) is on RAID. BOOT_DEGRADED is set. All partitions are GPT. Not using LUKS or LVM. All drives are 2TB and by various manufacturers, and I suspect some have 512B physical sectors and some have 2KB sectors. This is an AMD64 system with 8GB RAM.

This system has had about four different versions of Ubuntu on it, and has had multiple RAID arrays on it from the beginning. (This is why some of the arrays are still using 0.90.0 metadata.) RAID worked fine until the system was upgraded to Oneiric early in 2012 (no, it did not start with Precise).

I have carefully tested the system every time an updated kernel or mdadm has appeared, since the problem started with Oneiric. The behavior has gradually improved over the last several months. This latest version of mdadm (3.2.5) did not result in significant improvement; have rebooted four times since then and the behavior is consistent.

When the problem first started, on Oneiric, I had the root file system on RAID. This was unpleasant. I stopped using the system for a while, as I had another one running Maverick.

When I noticed some discussion of possibly related bugs on the Linux RAID list (I've been lurking there for years) I decided to test the system some more. By then Precise was out, so I upgraded. That did not help. Eventually I backed up all data onto another system and did a clean install of Precise on a non-RAID partition, which made the system tolerable. I left /boot on a RAID1 array (on all six drives), but that does not prevent the system from booting even if /boot does not start during Ubuntu startup (I assume because GRUB can find /boot even if Ubuntu later can't).

I started taking detailed notes in May (seven cramped pages so far). Have rebooted 23 times since then. On every boot, exactly two arrays did not start. Which arrays they were, varied from boot to boot; could be any of the arrays. No apparent correlation with metadata type or RAID level.

This mdadm 3.2.5 is the first time I have resorted to doing a forced upgrade from -proposed; before, I always just waited for a regular update. The most significant improvements happened with earlier regular updates. It has been a while since I had to wait for a degraded array to resync, or manually re-attach a component (usually a spare) that had become detached, or drop to the command line to zero a superblock before reattaching a component. It has been a while since an array containing swap has failed to start.

This issue has now become little more than an annoyance. I can now boot, wait for first array to not start, hit S, wait for the second, hit S, wait for the login screen, log in, wait for Unity desktop, start Disk Utility, manually start the two ...

@Doug Jones (djsdl)

This bug report is not meant to fix *all* issues with raid, with respect to booting. Previously booting would drop into initramfs too quickly, now we wait for udev events to be processed and then we drop to initramfs. So this particular bug fix is fixing just this issue. You do not describe dropping into initramfs & assembling arrays by hand, nor having RAID on root. So I do not believe that you are solely affected by this bug, but other bugs as well. So while not perfect, this -proposed upgrade does not regress any further for you (e.g. some raids were not assembled before, and still are not assembled), but it does improve the situation for others.

Did you see https://wiki.ubuntu.com/ReliableRaid specification and bugs linked from there? You may be familiar / affected by them.

I would be interested to see your setup, e.g. `$ sudo apport mdadm ` should create a bug with all relevant system info attached.

@Dmitrijs:

I must agree with your assessment. I cannot really tell which (of many) existing bug reports are most relevant to my system, so I've been chiming in on a bunch of them, just in case they provide clues helpful to others :-)

This latest mdadm update does not have any negative impact on me, and clearly helps others, so +1 from me.

Will do sudo apport mdadm. And thanks for the link to that spec.

@Dmitrijs:

sudo apport mdadm does nothing. I know that apport is installed and the service is running. Am I doing this wrong?

@Doug:

The correct command is 'sudo ubuntu-bug mdadm' or 'sudo apport-bug mdadm'.

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package mdadm - 3.2.5-1ubuntu0.2

---------------
mdadm (3.2.5-1ubuntu0.2) precise-proposed; urgency=low

  * Patch udev-md-raid.rules to not auto-start arrays based on detection
    of ddf or isw (imsm) signatures; this conflicts with dmraid usage in
    precise and requires more analysis before it can be enabled in SRU.
    LP: #1030292.

mdadm (3.2.5-1ubuntu0.1) precise-proposed; urgency=low

  * Stable Micro Point Bug Fix release:
    - SRU master bug (LP: #1009973)
    - Fixes segfault upon update-initramfs (LP: #969384)
    - Fixes does not allow internal bitmap on version 1.2 arrays (LP: #1022915)
  * Preserved previous behaviour, by reverting:
    - bitmap chunk size of 0 will continue to mean 512B
    - Modify the below fix, to accept previous syntax
      "b10c663 config: fix handing of 'homehost' in AUTO line."
  * Use upstream version of udev rules, instead of three slightly
    different ubuntu udev rules (LP: #968074) (LP: #1002357)
  * Add missing mdmon utility. This enables external metadata RAID
    formats: DDF and Intel Matrix Storage Management (IMSM). (LP: #957494)
  * Copy udev rule from /etc/udev/rules.d/ as well as the
    /lib/udev/rules.d/, to allow local administrator to override mdadm
    rules file (LP: #1017407)
  * debian/initramfs/local-premount: add call wait_for_udev to wait a
    little longer for RAID devices to appear. This improves
    boot reliability. (LP: #942106)

mdadm (3.2.5-1) unstable; urgency=low

  [ Michael Tokarev ]
  * new upstream (bugfix) release, fixing regression when --add'ing
    device to an array, introduced in 3.2.4, plus other minor fixes
    (Closes: #673104, #673344)
  * new patch: sha1-includes.diff to fix #include mess in new sha1.h
  * added a check into debian/checkarray to skip checking arrays created
    less than 2 weeks ago (Closes: #624273)

  [ Dmitrijs Ledkovs ]
  * Remove obsolete documentation dating back to ~etch release
  * Remove reference to obsolete documention from debconf templates
  * Update debconf templates translations
  * Remove compatability with acient initramfs-tools
  * Remove debian-specific mdadm-startall.8 in clean target

mdadm (3.2.4-1) unstable; urgency=low

  * new upstream (bugfix) release (Closes: #664088, #661552)
  * removed debian-run-udev.diff (applied upstream), and
    all RUNDIR handling from debian/rules (it is the default now)
  * add build-arch and build-indep targets to debian/rules, and
    bump Standards-Version to 3.9.3

mdadm (3.2.3-3) unstable; urgency=low

  * switch from topgit to plain 3.0 (quilt) layout, creating
    debian/patches. Don't build-depend on quilt as patching
    is done automatically by dpkg-source.
  * debian/patches/debian-run-udev.diff by Roger Leigh (Closes: #644319, #627774)
  * update debian/mdadm.logcheck.ignore.server to recognize "k" in
    addition of "blocks" in kernel messages. Thanks to Frédéric Brière
    for the patch (Closes: #656038)
 -- Steve Langasek <email address hidden> Fri, 03 Aug 2012 23:08:39 -0700

@Brian, Dmitrijs:

Thanks.

I have filed Bug # 1036366 to report the symptoms not resolved by this recent fix.

12.04.1 with all updated.This bug still affects me too!

My setup is the following: 4 LUKS encrypted partitions with mdadm RAID5 on top. The reason I encrypted first and then added them into the array was based on performance: 4 threads doing encryption/decryption instead of just 1. The max speed of this array vs Raid5 first, encrypt later is 169 MB/s vs 90 MB/s. This setup used to work perfectly in 9.10, but stopped working in 12.04. mdadm always thinks the array is degraded. I hope this bug is fixed!

BTW. My RAID 5 is perfectly healthy and can be assembled after booting process is finished. crypttab decrypts the devices automatically using key files.

The patch of https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/942106/comments/5 does not work in following case:
- Ubuntu 12.04
- partitionable RAID-1 on two complete disks sda and sdb, giving array md0. Root partition is md0p1, swap partition is md0p2.
- One disk physically removed before boot.
- boot degraded arrays is requested

The system reports Mount failed, /dev/disk/my-uuid/<uniqueid> busy.
This /dev/disk/my-uuid/<uniqueid> points to /dev/sda1.

Solution:
add a second wait_for_udev at the end of the local-premount/mdadm script.
This results in the script:
----------------
. /scripts/mdadm-functions
. /scripts/functions

wait_for_udev
degraded_arrays || exit 0
mountroot_fail || panic "Dropping to a shell."
wait_for_udev
-----------------

Hypothetical explanation:
mountroot_fail finds a degraded RAID-1 array and starts it with one disk, creating device /dev/md0. The function ends.
Next, the 'local' script tries to mount the root partition, but udev did not yet have time to detect the partitions and create the corresponding devices md0p1 and md0p1, and change the links in /dev/disk/by-uuid to these detected partitions.
The links in /dev/disk/by-uuid still point to /dev/sda[12], which are locked by mdadm for md0.

The added wait_for_udev lets udev find the partitions md0p1 and md0p2 and change the links in /dev/disk/by-uuid before
trying to mount the root partition.

Hi,
This fix does not work for me. I have a system with a single md raid 5 array with 5 disks, and root on a seperate non-raid disk. With the default bootdegraded=false my system frequently drops to the initramfs prompt with the usual timed out message. With BOOT_DEGRADED=true set in the mdadm initramfs config, the system frequently drops to the initramfs prompt complaining of 2 of 5, 3 of 5, or 4 of 5 disks failed to be added to the array.

I was able to fix the problem by horrible hack by adding a hard sleep after the udevadm settle call in wait_for_udev
(in /usr/share/initramfs-tools/scripts/functions)
# Wait for queued kernel/udev events
wait_for_udev()
{
        command -v udevadm >/dev/null 2>&1 || return 0
        udevadm settle ${1:+--timeout=$1}
        sleep 15
}

With this change the system reliably boots (albeit slightly slower). with the raid properly detected with all 5 drives.

This suggests to me that udevadm settle is not doing what it's supposed to be doing (waiting for the relevant controller/drives to fully come up.)

I am running Ubuntu 13.10 on an Asus M2N-E Motherboard with 6 disks attached to the onboard sata ports. (One with the root partition and the other 5 as part of the raid 5 array).

This board runs an older AMD-based Nvidia chipset and does not support ahci, so the sata drives run through the sata_nv module.

This doesnt look fixed for me too.

I run a software raid and the "solution" posted by Josh is the only one which is working in my case.
I have a software raid5 and my root partition is not part of the array.

I get the "Raid is degraded" warning in about 90% of my reboots.