Raid10 array not assembled at boot, all disks are marked as spare

I have exactly the same problem. The only quirk is that it is a three disk raid 10. The same disks (750, 500 and 250G) also hold a raid 0 array which works absolutely fine.

The /dev/mdadm/mdadm.conf file had an incorrect UUID after I'd had to rebuild the arrays. I have fixed this using mdadm --detail /dev/md0, to get the correct UUID, but others beware.

N.b. re-assembling, and re-creating the array (with assume clean) left all my data intact, and I could mount the ext4 LVM volumes.

here's a snip of my dmesg
[ 16.345255] md: bind<sda1>
[ 16.346671] mdadm[2286]: segfault at 0 ip 000000000040839b sp 00007fff6106a5d0 error 4 in mdadm[400000+2a000]
[ 16.384062] md: bind<sdb1>
[ 16.391313] mdadm[2305]: segfault at 0 ip 000000000040839b sp 00007fffa55e70d0 error 4 in mdadm[400000+2a000]
[ 16.432732] md: bind<sdc1>
[ 16.440382] mdadm[2306]: segfault at 0 ip 000000000040839b sp 00007fff1feb49a0 error 4 in mdadm[400000+2a000]

let me know how to help debug this.

I see an error just before the prompt:

exec of program '/sbin/' failed

I also noticed a /var/lock does not exist error, but it seemed to be there by the time the prompt came up (although var wasn't mounted due to the prior errors).

I have also noticed the following:

Before reboot:
md0 = sd[abc]1

After reboot:
md0 is gone
md_d0 is there with one or two of the original partitions.

To recover, I mdadm --stop the md_d0 bogus array, and recreate the original array, assume-clean, and it seems works fine. I also have to reactive the logical volume with lvchange -ay /dev/raid10/tank

I have just noticed that <code>blkid</code> does not mention the UUID of my dodgy /dev/md3 device. Three other functioning /dev/md devices are shown fine.

<code>vol_id</code> produces normal output, stating the UUID, but this doesn't feature in the blkid output.

I notice also that <code>mdadm --detail /dev/md3</code> shows the the array is local to this host, whereas the other arrays do not have this flag. I have been forced to recreate the array every reboot, so this is perhaps not significant. N.b. I am only booting into one system, not trying to mount the arrays from different OSes.

the same ?kernel bug exists with 2.6.29-rc8 which I got from the mainline ppa repository.

at the moment, this seems to be a raid 10 assembly bug. possibly only for three disk raid10, but raid 0 and raid 10 work for me in this setup.

four disk raid10 is affected as well

live Fedora 11 alpha CD: no problems with my raid 10 (although a different array is marked inactive...)

[liveuser@localhost ~]$ uname -a
Linux localhost.localdomain 2.6.29-0.66.rc3.fc11.x86_64 #1 SMP Thu Jan 29 14:44:32 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
[liveuser@localhost ~]$
[liveuser@localhost ~]$ more /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md_d2 : inactive sda8[0](S)
      70043264 blocks

md_d1 : active raid1 sda7[0] sdb7[1]
      68356416 blocks [2/2] [UU]

md_d4 : active raid0 sdc2[0] sde2[1]
      488287424 blocks 64k chunks

md_d3 : active raid10 sdc1[0] sde1[2] sdd1[1]
      366293824 blocks 64K chunks 2 near-copies [3/3] [UUU]

I have this problem as well ( 4 disks + hot spare with one raid10 /lvm) and one raid1 (/) array ).

Workaround:

Setting BOOT_DEGRADED=true in /etc/initramfs-tools/conf.d/mdadm works like a charm.

You need to do an
#sudo update-initramfs -u
afterwards.

Drawback: booting takes longer than normal, since mdadm waits 30s to assemble the (presumably) broken array and I'm not sure what happens if the arrays are *really* broken

with jaunty kernel 2.6.28-10, the raid is no longer assembled at all. The array can be created with mdadm --create --assume-clean, as before, apparently without any problems. this sounds okay if nothing has been written to the array yet, and since it isn't assembled, this should be fine. Unless the problem is in shutting down the array, but no segfault there, so probably not.

No joy with 2.6.28-11, although there are three md raid10 fixes, relating to recovery which I hoped might somehow impinge:
from upstream short changelog:
md: avoid races when stopping resync.
  * md/raid10: Don't call bitmap_cond_end_sync when we are doing recovery.
  * md/raid10: Don't skip more than 1 bitmap-chunk at a time during
    recovery.

this seems like a serious bug to me. if you are suffering, please add your comments; if it is just me and the reporter, please can someone help me work out why? N.b. a fedora kernel with ubuntu starts the array correctly, so this is a kernel or mdadm bug.

exactly the same bug on a completely different set-up. dmesg attached. n.b. SEGFAULTS in the middle.

this is a serious bug! raid10 is an installation option, not an obscure corner case, and it results in a non-booting system. no idea if this bug is also corrupting data silently.

There's no indication this is a(n upstream) kernel bug, marking that task invalid.

Riasing the priority and nominating for jaunty.

I am experiencing this bug with both two-disk raid10 arrays I use, I have to boot with --boot-degraded ever since I upgraded to Jaunty preview.

When I recreated the bug, I created the duplicate bug report bug 347747 with the apport crash information. The retracer has gone through and provided the stacktrace, which I'm attaching here. The rest of the apport crash information can be seen in my duplicate report.

Same Jaunty problem here.

I do not get the mdadm segfault with any of the other supported types (raid 0, 1, 4, 5, and 6, linear, multipath) so I believe only raid10 is affected by this.

If it's not going to be fixed for jaunty - shouldn't the default be BOOT_DEGRADED=TRUE? Otherwise all people running Raid 10 will get into some (fixable but rather annoying) trouble.

bad idea if there really is a degraded array...

needs fix or disable raid 10 (which is drastic). easier to fix than audit the many possible causes of data loss this might bring about if we left the bug in there. I don't understand why fedora kernels boot Ubuntu without this problem. Maybe someone clever could diff between the kernel i mention above and the ubuntu one: surely a clue there...

Disable raid 10 is not an option. But getting it fixed in a month is IMO questionable, too.
Alternatively perhaps it would be enough to display a warning message when updating to jaunty (and a Raid10 is present) and place a remark in the manpage (let's face it - almost no one is reading release notes).

Documented in the Beta technical overview:

  The mdadm package in Ubuntu 9.04 Beta will fail to assemble RAID10 arrays on boot. Other types of RAID are not affected; investigation of this issue is ongoing.

This is a high-priority issue which should be fixed for release, so I don't expect we'll need to document anything in the release notes.

With Dustin Kirkland's help, I tested out packages based on debian's 2.6.7.2 and 2.6.8 packages; 2.6.7.2 still causes mdadm to segfault for me when incrementally assembling a raid10 array, but 2.6.8 does not. We'll be digging in to find out what portion of 2.6.8 we need to cherry pick to fix this. Thanks for your patience!

Looking at the backtrace + the upstream diff, I think the attached patch is the bit that fixes it.

Thanks, slangasek.

sbeattie et al.-

I have prepared a new package for you to test, including *only* this
cherry picked patch. Would you please try out:
  * http://people.ubuntu.com/~kirkland/316670/
and report back?

I'll get this committed if you can confirm that this solves the issue...

:-Dustin

I'm attaching the debdiff here, for the sake of completeness (and since it only exists in my ramfs)...

:-Dustin

I can confirm Dustin's package fixes the segfault issue and lets a system with / on raid10 boot. I've also briefly verified that the other raid configurations still work properly. Thanks!

This bug was fixed in the package mdadm - 2.6.7.1-1ubuntu8

---------------
mdadm (2.6.7.1-1ubuntu8) jaunty; urgency=low

  Fix for LP: #316670
  * Incremental.c: cherry-pick upstream patch, "Fix NULL pointer oops"
    http://git.debian.org/?p=pkg-mdadm/mdadm.git;a=commit;h=43aaf431f66270080368d4b33378bd3dc0fa1c96

  Fix for LP: #349971 (Thanks, Steve Beattie!)
  * debian/source_mdadm.py: apport hooks for improved bug reporting

 -- Dustin Kirkland <email address hidden> Fri, 27 Mar 2009 18:52:51 -0500

great - works for me, BUT i had to update the UUID (mdadm --detail, not vol_id /dev/md...) in /etc/mdadm/mdadm.conf for it to assemble on boot, because I had had to recreate it each time I booted. This won't of course affect people who go straight to the fixed mdadm from intrepid.

i should have said: works for me with three disk raid10.

Why isn't the very simple regression test described done automatically?

Just adding my experience here, not sure if it is related.

I upgraded last night to the Jaunty Release Candidate.

Gdebi indicates that my installed mdadm is the same version as the AMD64 version from http://people.ubuntu.com/~kirkland/316670/

My root is on a regular sata disk, and I mount sdb1 and sdc1 as a Raid 1 mdadm+lvm2 (as /dev/md0 and /lvm/lvm-raid/homelv) on /home

The default upgrade did not find my md0, and instead puts sdc1 into a false md_d0.

Every time I boot, I have to drop into a console, execute mdadm --assemble --scan and manually mount the /dev/lvm-raid/homelv . These steps work, except my md0 is degraded to just one disk (sdb1)

I checked /proc/mdstat before the upgrade and everything was fine, so I'm wondering if this is the same bug or a different one?

What's the best route to fix this?

On Fri, Apr 17, 2009 at 12:59:18PM -0000, lunomad wrote:
> Just adding my experience here, not sure if it is related.
>
> I upgraded last night to the Jaunty Release Candidate.
>
> Gdebi indicates that my installed mdadm is the same version as the AMD64
> version from http://people.ubuntu.com/~kirkland/316670/
>
> My root is on a regular sata disk, and I mount sdb1 and sdc1 as a Raid 1
> mdadm+lvm2 (as /dev/md0 and /lvm/lvm-raid/homelv) on /home
>
> The default upgrade did not find my md0, and instead puts sdc1 into a
> false md_d0.
>
> Every time I boot, I have to drop into a console, execute mdadm
> --assemble --scan and manually mount the /dev/lvm-raid/homelv . These
> steps work, except my md0 is degraded to just one disk (sdb1)

Are you able to hot-add back /dev/sdc1 to the array (e.g. mdadm /dev/md_d0
--add /dev/sdc1)? If so, after successfully resync'ing, does the following
boot still drop you to the busybox shell? Also, before you're dropped
to the shell, are you being asked to boot in degraded mode (the timeout
while waiting for the root filesystem to come up is long, but the boot
degraded question timeout is quite short)?

> I checked /proc/mdstat before the upgrade and everything was fine, so
> I'm wondering if this is the same bug or a different one?

This is likely a separate bug, please open a new one (though feel
free to report the bug number here). Thanks!

--
Steve Beattie
<email address hidden>
http://NxNW.org/~steve/

During boot I was prompted with a warning, no valid partitions found on /dev/sdc, hit ctrl-D to continue. Because this isn't a root partition, I was able to continue booting with ctrl-D, which took me all the way to the gdm login. I then dropped to a shell with ctrl-alt-F1. There were no questions about degraded mode booting.

I fixed my problem with the following steps:

1. Stop the false md_d0 array
$ sudo mdadm --stop /dev/md_d0

2. Assemble the degraded md0 (only sdb was found) and mount it
$ sudo mdadm --assemble --scan
$ sudo mount /dev/lvm-raid/homelv /home

3. use mkconf to find the configuration option
$ sudo /usr/share/mdadm/mkconf

4. gedit /etc/mdadm/mdadm.conf to include the output from mkconf AND to use only sdb
DEVICE /dev/sdb*
# definitions of existing MD arrays
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=6a291d6f:27049e9f:b95e1135:52246c3f

5. Reboot --> ubuntu found and mounted my array just fine (mount info for LVM is my fstab)

6. Re-add sdc to md0
$ sudo mdadm /dev/md0 --re-add /dev/sdc1

7. Watch rebuild!
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[0] sdc1[2]
      976759936 blocks [2/1] [U_]
      [==>..................] recovery = 14.2% (139109312/976759936) finish=381.7min speed=36570K/sec

unused devices: <none>

I'm not sure why mdadm wanted to build the (incorrect) md_d0 and not the (correct) md0.

I had this issue on a RAID 10 with f2 layout with 3 disks. After upgrade from intrepid to jaunty (production) RAID(s) 1 were fine, but RAID 10 was inactive with only sdc(4) showing.

Fix was rather simple, only:
$ sudo mdadm --stop /dev/md_d2
$ sudo mdadm --assemble --scan

Result:
   /dev/md2: File exists
   mdadm: /dev/md/2 has been started with 3 drives.

and then:
$ sudo mount -a

and array was back...

In my previous post I forgot to write that previous procedure brought array back, but only until next reboot. To fix this I did the following:
$ sudo mdadm --detail /dev/md2

to display array characteristics
with that information I have updated /etc/mdadm/mdadm.conf
because md2 ARRAY was deleted from there.