mdadm - new added device is spare instead of active

Binary package hint: mdadm

hi.

it's possible that i forgot once to "--set-faulty" before removing a
device.
i added a new drive, it rebuilded the md1, and started rebuilding md3
until around 40% then stops and i get the below data.

i set-faulty /dev/sda3, remove failed, zero superblock, re-add the
device and stil get the same.
i also add a new device /dev/sdb3 and the same happens.
i even grow the raid to 4 disks and try to add 2 disks at the same time - no succes.

here is the data:

dmesg shows this over and over

[481073.717506] RAID1 conf printout:
[481073.717509] --- wd:1 rd:2
[481073.717512] disk 0, wo:0, o:1, dev:sdc3
[481073.717514] disk 1, wo:1, o:1, dev:sda3
[481073.717515] RAID1 conf printout:
[481073.717516] --- wd:1 rd:2
[481073.717518] disk 0, wo:0, o:1, dev:sdc3
[481073.717519] disk 1, wo:1, o:1, dev:sda3
[481074.099030] RAID1 conf printout:
[481074.099034] --- wd:1 rd:2
[481074.099037] disk 0, wo:0, o:1, dev:sdc3
[481074.099039] disk 1, wo:1, o:1, dev:sda3
[481074.099040] RAID1 conf printout:
[481074.099041] --- wd:1 rd:2
[481074.099043] disk 0, wo:0, o:1, dev:sdc3
[481074.099044] disk 1, wo:1, o:1, dev:sda3

dennis1 ~ $ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md3 : active raid1 sda3[2] sdc3[0]
      452550976 blocks [2/1] [U_]
      bitmap: 130/216 pages [520KB], 1024KB chunk

md1 : active raid1 sda1[1] sdc1[0]
      30716160 blocks [2/2] [UU]
      bitmap: 106/235 pages [424KB], 64KB chunk

unused devices: <none>

dennis1 ~ $ sudo mdadm --detail /dev/md3
/dev/md3:
        Version : 00.90
  Creation Time : Wed Dec 22 05:55:45 2010
     Raid Level : raid1
     Array Size : 452550976 (431.59 GiB 463.41 GB)
  Used Dev Size : 452550976 (431.59 GiB 463.41 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 3
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Jan 6 23:57:07 2011
          State : active, degraded
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

           UUID : 4c2769e4:568869bc:2a972c67:788b998b (local to host dennis1)
         Events : 0.726822

    Number Major Minor RaidDevice State
       0 8 35 0 active sync /dev/sdc3
       2 8 3 1 spare rebuilding /dev/sda3

dennis1 ~ $ sudo mdadm /dev/sda3 --examine
/dev/sda3:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 4c2769e4:568869bc:2a972c67:788b998b (local to host dennis1)
  Creation Time : Wed Dec 22 05:55:45 2010
     Raid Level : raid1
  Used Dev Size : 452550976 (431.59 GiB 463.41 GB)
     Array Size : 452550976 (431.59 GiB 463.41 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 3

    Update Time : Thu Jan 6 23:59:49 2011
          State : clean
Internal Bitmap : present
 Active Devices : 1
Working Devices : 2
 Failed Devices : 1
  Spare Devices : 1
       Checksum : a4452cd8 - correct
         Events : 726936

      Number Major Minor RaidDevice State
this 2 8 3 2 spare /dev/sda3

   0 0 8 35 0 active sync /dev/sdc3
   1 1 0 0 1 faulty removed
   2 2 8 3 2 spare /dev/sda3

dennis1 ~ $ sudo mdadm /dev/sdc3 --examine
/dev/sdc3:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 4c2769e4:568869bc:2a972c67:788b998b (local to host dennis1)
  Creation Time : Wed Dec 22 05:55:45 2010
     Raid Level : raid1
  Used Dev Size : 452550976 (431.59 GiB 463.41 GB)
     Array Size : 452550976 (431.59 GiB 463.41 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 3

    Update Time : Fri Jan 7 00:02:45 2011
          State : clean
Internal Bitmap : present
 Active Devices : 1
Working Devices : 2
 Failed Devices : 1
  Spare Devices : 1
       Checksum : a4452e86 - correct
         Events : 727046

      Number Major Minor RaidDevice State
this 0 8 35 0 active sync /dev/sdc3

   0 0 8 35 0 active sync /dev/sdc3
   1 1 0 0 1 faulty removed
   2 2 8 3 2 spare /dev/sda3

dennis1 ~ $ sudo mdadm /dev/md3 -a /dev/sda3
mdadm: Cannot open /dev/sda3: Device or resource busy

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: mdadm 2.6.7.1-1ubuntu15
ProcVersionSignature: Ubuntu 2.6.32-27.49-generic 2.6.32.26+drm33.12
Uname: Linux 2.6.32-27-generic x86_64
Architecture: amd64
Date: Fri Jan 7 01:02:00 2011
InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release amd64 (20100429)
MDadmExamine.dev.sda: Error: command ['/sbin/mdadm', '-E', '/dev/sda'] failed with exit code 1: mdadm: cannot open /dev/sda: Permission denied
MDadmExamine.dev.sda1: Error: command ['/sbin/mdadm', '-E', '/dev/sda1'] failed with exit code 1: mdadm: cannot open /dev/sda1: Permission denied
MDadmExamine.dev.sda2: Error: command ['/sbin/mdadm', '-E', '/dev/sda2'] failed with exit code 1: mdadm: cannot open /dev/sda2: Permission denied
MDadmExamine.dev.sda3: Error: command ['/sbin/mdadm', '-E', '/dev/sda3'] failed with exit code 1: mdadm: cannot open /dev/sda3: Permission denied
MDadmExamine.dev.sdb: Error: command ['/sbin/mdadm', '-E', '/dev/sdb'] failed with exit code 1: mdadm: cannot open /dev/sdb: Permission denied
MDadmExamine.dev.sdb1: Error: command ['/sbin/mdadm', '-E', '/dev/sdb1'] failed with exit code 1: mdadm: cannot open /dev/sdb1: Permission denied
MDadmExamine.dev.sdb2: Error: command ['/sbin/mdadm', '-E', '/dev/sdb2'] failed with exit code 1: mdadm: cannot open /dev/sdb2: Permission denied
MDadmExamine.dev.sdb3: Error: command ['/sbin/mdadm', '-E', '/dev/sdb3'] failed with exit code 1: mdadm: cannot open /dev/sdb3: Permission denied
MDadmExamine.dev.sdc: Error: command ['/sbin/mdadm', '-E', '/dev/sdc'] failed with exit code 1: mdadm: cannot open /dev/sdc: Permission denied
MDadmExamine.dev.sdc1: Error: command ['/sbin/mdadm', '-E', '/dev/sdc1'] failed with exit code 1: mdadm: cannot open /dev/sdc1: Permission denied
MDadmExamine.dev.sdc3: Error: command ['/sbin/mdadm', '-E', '/dev/sdc3'] failed with exit code 1: mdadm: cannot open /dev/sdc3: Permission denied
MDadmExamine.dev.sdd: Error: command ['/sbin/mdadm', '-E', '/dev/sdd'] failed with exit code 1: mdadm: cannot open /dev/sdd: Permission denied
MDadmExamine.dev.sdd1: Error: command ['/sbin/mdadm', '-E', '/dev/sdd1'] failed with exit code 1: mdadm: cannot open /dev/sdd1: Permission denied
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.32-27-generic root=/dev/mapper/sistem-volume1 ro crashkernel=384M-2G:64M,2G-:128M quiet splash
ProcEnviron:
 LANGUAGE=en
 LANG=en_US.utf8
 SHELL=/bin/bash
SourcePackage: mdadm
dmi.bios.date: 04/23/2008
dmi.bios.vendor: Intel Corp.
dmi.bios.version: JOQ3510J.86A.0882.2008.0423.1925
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: DQ35JO
dmi.board.vendor: Intel Corporation
dmi.board.version: AAD82085-804
dmi.chassis.type: 3
dmi.modalias: dmi:bvnIntelCorp.:bvrJOQ3510J.86A.0882.2008.0423.1925:bd04/23/2008:svn:pn:pvr:rvnIntelCorporation:rnDQ35JO:rvrAAD82085-804:cvn:ct3:cvr: