Setting up mdadm (3.2.5-5ubuntu3) freezes after calling grub1's update-grub from postinst

Since this is my home app server and testbed machine, I started uncomplicating my setup to see if that would help...

root@kami:~# lvremove /dev/KamiVG01/chippoke
Do you really want to remove and DISCARD active logical volume chippoke? [y/n]: y
  Logical volume "chippoke" successfully removed
root@kami:~# lvremove /dev/KamiVG01/nas-office
Do you really want to remove and DISCARD active logical volume nas-office? [y/n]: y
  Logical volume "nas-office" successfully removed
root@kami:~# lvscan
root@kami:~# vgscan
  Reading all physical volumes. This may take a while...
  Found volume group "KamiVG01" using metadata type lvm2
root@kami:~# vgremove KamiVG01
  Volume group "KamiVG01" successfully removed
root@kami:~# vgscan
  Reading all physical volumes. This may take a while...
  No volume groups found

dpkg --configure -a still froze at Found kernel: /boot/memtest86+.bin.

Also attached current menu.lst.

Interestingly, running update-grub manually seems to work fine (but walks over the kernels twice?):

root@kami:/boot/grub# /usr/sbin/update-grub
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-3.13.0-12-generic
Found kernel: /boot/vmlinuz-3.11.0-17-generic
Found kernel: /boot/vmlinuz-3.11.0-15-generic
Found kernel: /boot/memtest86+.bin
Found kernel: /boot/vmlinuz-3.13.0-12-generic
Found kernel: /boot/vmlinuz-3.11.0-17-generic
Found kernel: /boot/vmlinuz-3.11.0-15-generic
Found kernel: /boot/memtest86+.bin
Updating /boot/grub/menu.lst ... done

root@kami:/boot/grub#

Problem is still occurring. My server has been sitting in a half-upgraded unrebooted state since I filed this Bug. I've been killing the hung mdadm.postint script occasionally and re-running apt-get dist-upgrade.

Currently, the processes that are hanging are:

root@kami:~# ps auxww | grep dpkg
root 17854 0.0 0.2 17764 5308 pts/3 Ss+ 11:10 0:00 /usr/bin/dpkg --status-fd 47 --configure mdadm:amd64 libitm1:amd64 libgomp1:amd64 libasan0:amd64 libatomic1:amd64 libtsan0:amd64 libquadmath0:amd64 cpp-4.8:amd64 libgcc-4.8-dev:amd64 gcc-4.8:amd64 libstdc++-4.8-dev:amd64 g++-4.8:amd64 libpython3.3-minimal:amd64 python3.3-minimal:amd64 libpython3.3-stdlib:amd64 python3.3:amd64 bash-completion:all libsystemd-daemon0:amd64 systemd-shim:amd64 systemd-services:amd64 libpam-systemd:amd64 libpci3:amd64 pciutils:amd64 libsystemd-login0:amd64 imagemagick-common:all libasound2-data:all libasound2:amd64 libcups2:amd64 libcupsfilters1:amd64 libcupsimage2:amd64 libgdk-pixbuf2.0-common:all libgdk-pixbuf2.0-0:amd64 libmagickcore5:amd64 libmagickwand5:amd64 libmagickcore5-extra:amd64 libpolkit-agent-1-0:amd64 libpolkit-backend-1-0:amd64 libpython3.4-minimal:amd64 python3.4-minimal:amd64 libpython3.4-stdlib:amd64 python3.4:amd64 librados2:amd64 librbd1:amd64 libsnmp-base:all libsnmp30:amd64 php5-common:amd64 php5-mysql:amd64 php5-cli:amd64 php5-readline:amd64 libapache2-mod-php5:amd64 php5-gd:amd64 php5-curl:amd64 qemu-system-common:amd64 dh-python:all locales:all upstart:amd64 libisc95:amd64 libdns100:amd64 libisccc90:amd64 libisccfg90:amd64 libbind9-90:amd64 liblwres90:amd64 bind9utils:amd64 bind9:amd64 bind9-host:amd64 dnsutils:amd64 geoip-database:all libglib2.0-data:all lshw:amd64 memtest86+:amd64 openssh-client:amd64 openssh-sftp-server:amd64 openssh-server:amd64 python3-problem-report:all python3-apport:all python3-distupgrade:all ubuntu-release-upgrader-core:all python3-gi:amd64 shared-mime-info:amd64 ufw:all uuid-runtime:amd64 grub-common:amd64 apport:all apport-symptoms:all libdpkg-perl:all dpkg-dev:all imagemagick:amd64 libjs-sphinxdoc:all qemu-keymaps:all qemu-system-x86:amd64 qemu-kvm:amd64 libvirt0:amd64 libvirt-bin:amd64 linux-libc-dev:amd64 php5:all policykit-1:amd64 vim-puppet:all puppet-common:all puppet:all python-cinderclient:all python-gi:amd64 python-libvirt:amd64 python-lxml:amd64 python-novaclient:all python-secretstorage:all unattended-upgrades:all python-software-properties:all python-webtest:all qemu-utils:amd64 snmp:amd64 snmpd:amd64 ssh:all wpasupplicant:amd64 hardening-includes:all python-nova:all nova-common:all python-keystone:all python-swiftclient:all python-vm-builder:all ubuntu-vm-builder:all util-linux-locales:all
root 17855 0.1 0.7 63112 15192 pts/3 S+ 11:10 0:00 /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/mdadm.postinst configure 3.2.5-5ubuntu2
root 17867 0.0 0.0 4444 684 pts/3 S+ 11:10 0:00 /bin/sh /var/lib/dpkg/info/mdadm.postinst configure 3.2.5-5ubuntu2
root 18331 0.0 0.0 10464 880 pts/1 S+ 11:14 0:00 grep dpkg

And the culprit appears to be update-grub:

root@kami:~# ps auxww | grep grub
root 17854 0.0 0.2 17764 5308 pts/3 Ss+ 11:10 ...

Status changed to 'Confirmed' because the bug affects multiple users.

My experience is similar to Sean's although I have managed to keep my system in a usable state. mdadm and my raid seem to be functioning as normal, I just can't get through any apt related tasks without it getting stuck on configuring mdadm. I am also seeing update-grub run through my kernels twice when I run it. Not sure if that is significant. would like to find a workaround as upgrading packages will be a pain with this bug. Let me know if you need any info from me to squash this one.

Wondering if it would be safe to commet out the update-grub part of mdadm.postinst temporarily and then running update-grub manually after things finish as I can run it alone fine...

Does anyone have a usable workaround for this issue until we see 14.04.1?

Somehow, I was able to kill mdadm.postinit and dpkg and then sudo dpkg --configure -a seemed to finish. I haven't restarted yet, but my /proc/mdstat still shows everything up and assembled. We'll see.

I have the same problem, but have managed to complete the upgrade from 12.04 to 14.04 using:

sudo echo ``mdadm hold'' | dpkg --set-selections

followed by:

sudo dpkg --configure -a

I faced this when upgrading Ubuntu Desktop from 13.10 to 14.04 (via update-manager).

Following workaround worked fine for me:
I killed the update-grub process. That resumed the installer and it finally managed to complete successfully.
I also did a manual run of `update-grub` which succeeded without any freezing or other issues.

Having the same problem with an upgrade from 12.04 to 14.04 - had to fall back from grub2 to grub, since grub2 seems to have issues with software raid, then ran into this issue with mdadm freezing up during configuration.

If anyone runs into this bug, it would be great to get the "ps afx" output of the processes running at the time it hangs. And maybe as a bonus straces of the one.

Same bug here. For me solutions posted here https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1284514 solved the problem.

Here is the output of 'ps afx' on my system while it is hung on mdadm, 'dpkg --configure -a' or 'dpkg --configure mdadm'. This machine too, has been running since 2008 and upgraded successfully a zillion times.

Let me also point out, this is hanging with mdadm (3.2.5-5ubunutu4):

root@htpc:~# dpkg --configure -a
Setting up mdadm (3.2.5-5ubuntu4) ...
 Removing any system startup links for /etc/init.d/mdadm-raid ...
update-initramfs: deferring update (trigger activated)
update-grub is /usr/sbin/update-grub
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /vmlinuz-3.13.0-24-generic
Found kernel: /vmlinuz-3.11.0-20-generic
Found kernel: /vmlinuz-3.11.0-19-generic
<eternal hang here>

@Ryan (or anyone else who can reproduce). this hang:

What does:
# sh -x /usr/sbin/grub-mkconfig -o /boot/grub/grub.cfg
output?

To those who are affected, are you using grub1 or grub2?
Given the output above "found: /boot/grub/menu.lst" I'd guess grub1, not grub2. Thus it's wrong for mdadm to call `update-grub` command, as that's not needed.

Yes for me, but only because the step by step upgrades on this system over many years have never replaced it...

brian@schizo:~$ dpkg -l | grep grub
ii grub 0.97-29ubuntu66 i386 GRand Unified Bootloader (Legacy version)
ii grub-common 2.02~beta2-9 i386 GRand Unified Bootloader (common files)

This bug was fixed in the package mdadm - 3.2.5-5ubuntu5

---------------
mdadm (3.2.5-5ubuntu5) utopic; urgency=medium

  * Instead of calling update-grub in postinst, call update-grub2. (LP:
    #1285312) Although machines installed in this decade would have
    grub2's update-grub, this is not the case for machines upgraded into
    this decade from e.g. 8.04 LTS in which case update-grub would be
    grub1 with which this package doesn't integrate any more. All users
    who have not yet upgraded to grub2 are urged to upgrade to grub
    following instructions at:
    https://help.ubuntu.com/community/Grub2/Upgrading
 -- Dimitri John Ledkov <email address hidden> Wed, 30 Apr 2014 20:42:45 +0100

On 30 April 2014 19:22, Brian Burch <email address hidden> wrote:
> Yes for me, but only because the step by step upgrades on this system
> over many years have never replaced it...
>
> brian@schizo:~$ dpkg -l | grep grub
> ii grub 0.97-29ubuntu66 i386 GRand Unified Bootloader (Legacy version)
> ii grub-common 2.02~beta2-9 i386 GRand Unified Bootloader (common files)
>

Right. grub is no longer maintained. Together with other developers we
will discuss automatic upgrade to grub2. Whilst the fix in mdadm
packaging is in progress to be available in trusty you may want to in
the mean time install precise mdadm and upgrade to grub2, and then
complete upgrade to trusty.

--
Regards,

Dimitri.

I confirm Brian Burch's comment above about step-wise upgrades since 8.04. I do recall being "encouraged" to upgrade to grub2 but I had some peculiar config that made that risky. Of course, I don't recall what the risks were... Attached is the new grub.cfg created after running grub-mkconfig. Next I'll attach the console output produced.

Here's the console out from: "sh -x /usr/sbin/grub-mkconfig -o /boot/grub/grub.cfg".

Prior to running all this, I tried to purge mdadm and reinstall; no obvious effect.

After apt-get update, I purged then installed mdadm again, hoping I would get mdadm - 3.2.5-5ubuntu5 but, alas, I'm still getting the mdadm - 3.2.5-5ubuntu4 pkg which subsequently hangs.

I will attempt the upgrade from grub 0.97 --> 2 later as I'm out of time at the moment and the box is remotely administered (I can't monitor and play with the boot and chainloaders just now).

Hello Sean, or anyone else affected,

Accepted mdadm into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mdadm/3.2.5-5ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

@ Ryan you will not get 3.2.5-5ubuntu5 on trusty/14.04, instead you can get ubuntu4.1 from trusty-proposed as of now (or after a delay, until it reaches your mirror). Trusty-proposed is not available by default, see instructions to enable in the above comment. Alternatively you can wait until we release this update into trusty-updates pocket, which should be within a week or so.

After manually updating to grub2 (the grub-pc package) and removing legacy grub, I installed the proposed mdadm on my development trusty system. I haven't looked at the SRU verification tests yet. I realise the cryptsetup warning is not directly related to mdadm, but sdb is the second disk in my soft raid array and I don't have ANY encrypted volumes. This would make me nervous about upgrading my production system! Here is the console log:

brian@schizo:/etc/apt$ sudo apt-get install mdadm/trusty-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '3.2.5-5ubuntu4.1' (Ubuntu:14.04/trusty-proposed [i386]) for 'mdadm'
The following packages will be upgraded:
  mdadm
1 to upgrade, 0 to newly install, 0 to remove and 95 not to upgrade.
Need to get 362 kB of archives.
After this operation, 1,024 B of additional disk space will be used.
Get:1 http://mirror.ox.ac.uk/sites/archive.ubuntu.com/ubuntu/ trusty-proposed/main mdadm i386 3.2.5-5ubuntu4.1 [362 kB]
Fetched 362 kB in 1s (258 kB/s)
Preconfiguring packages ...
(Reading database ... 340057 files and directories currently installed.)
Preparing to unpack .../mdadm_3.2.5-5ubuntu4.1_i386.deb ...
 * Stopping MD monitoring service mdadm --monitor [ OK ]
Unpacking mdadm (3.2.5-5ubuntu4.1) over (3.2.5-5ubuntu4) ...
Processing triggers for man-db (2.6.7.1-1) ...
Processing triggers for doc-base (0.10.5) ...
Processing 6 changed doc-base files...
Registering documents with scrollkeeper...
Processing triggers for ureadahead (0.100.0-16) ...
ureadahead will be reprofiled on next reboot
Setting up mdadm (3.2.5-5ubuntu4.1) ...
 Removing any system startup links for /etc/init.d/mdadm-raid ...
update-initramfs: deferring update (trigger activated)
Generating grub configuration file ...
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 30372: /usr/sbin/grub-probe
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 30372: /usr/sbin/grub-probe
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 30423: /usr/sbin/grub-probe
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 30423: /usr/sbin/grub-probe
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 30474: /usr/sbin/grub-probe
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 30474: /usr/sbin/grub-probe
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 30525: /usr/sbin/grub-probe
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 30525: /usr/sbin/grub-probe
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 30701: /usr/sbin/grub-probe
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 30701: /usr/sbin/grub-probe
Found linux image: /boot/vmlinuz-3.13.0-24-generic
Found initrd image: /boot/initrd.img-3.13.0-24-generic
Found linux image: /boot/vmlinuz-3.11.0-19-generic
Found initrd image: /boot/initrd.img-3.11.0-19-generic
File descriptor 3 (pipe:[112701]) leaked on vgs invocation. Parent PID 31336: /usr/sbin/gr...

TEST CASE for mdadm 13.10 -> 14.04 SRU:
1. Upgrade mdadm while upgrading from Saucy to Trusty.
2. dpkg --configure mdadm will hang trying to update Grub1 and related processes must be killed.
3. Upgrade mdadm instead with the trusty-proposed mdadm - https://launchpad.net/ubuntu/+source/mdadm/3.2.5-5ubuntu4.1
3.a. apt-get install mdadm/trusty-proposed

VERIFICATION DONE:
- mdadm/trusty-proposed configuration completes successfully

This bug was fixed in the package mdadm - 3.2.5-5ubuntu4.1

---------------
mdadm (3.2.5-5ubuntu4.1) trusty; urgency=medium

  * Instead of calling update-grub in postinst, call update-grub2. (LP:
    #1285312) Although machines installed in this decade would have
    grub2's update-grub, this is not the case for machines upgraded into
    this decade from e.g. 8.04 LTS in which case update-grub would be
    grub1 with which this package doesn't integrate any more. All users
    who have not yet upgraded to grub2 are urged to upgrade to grub
    following instructions at:
    https://help.ubuntu.com/community/Grub2/Upgrading
 -- Dimitri John Ledkov <email address hidden> Wed, 30 Apr 2014 20:42:45 +0100

The verification of the Stable Release Update for mdadm has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

No more supported ubuntu versions use grub legacy, and the affected ones here have received fixes. Grub legacy upstream is also stopped, only receiving possible random fixes locally