Binary package hint: mountall
I upgraded a server from Jaunty (9.04) to Karmic (9.10) last night. After the upgrade, the server will no longer finish booting on its own. It appears to hang forever in mountall (version 1.0), saying "One or more of the mounts listed in /etc/fstab cannot yet be mounted" and listing all my filesystems.
I hit ESC for a recovery shell, fsck'ed every filesystem, and rebooted, but the same thing kept happening.
I was finally able to bring up the system by going into a recovery shell, remounting the (initially read-only) root via a "mount" command with the "-o remount,rw" option, mounting the other filesystems manually as well, and exiting the recovery shell via control-D. But when I restarted the server, the same thing happened again. So, I can reliably bring the system up by hand, but it won't come up by itself.
I'm using the standard Karmic server kernel (2.6.31-14-generic-pae). However, the problem persisted when I tried booting my previous kernel (2.6.28-16-server) after doing the Karmic upgrade, so it doesn't appear to be a kernel bug.
The only unusual thing I'm aware of about my server configuration is that all of its filesystems (including the root) are RAID1 mirrors handled by mdadm. The server did, however, boot up just fine (all the way, by itself) when it was running Jaunty, before the upgrade to Karmic. And when I go into a recovery shell, it's evident that the RAID1 mirrors are there -- "cat /proc/mdstat" shows everything as expected, and commands like "mount /dev/md_d2 /var" work fine. It's not clear to me that there should be any reason why mountall can't handle RAID1 mirror devices, and maybe the problem really lies elsewhere, but . . . .
Please note that I am *not* using UUID's to identify my filesystems in /etc/fstab. I mention this because I've seen a few people mention similar-sounding mountall problems that were resolved by editing their /etc/fstab to use the actual device names rather than UUID's; I'm already doing that, and I'm still having trouble.
I also did "aptitude update" and "aptitude dist-upgrade" again on the system, just to be sure that some part of the upgrade to Karmic might not have finished properly -- another thing that some people have suggested in connection with similar-sounding problems -- but this didn't fix the problem.
I think I may have found the solution (or, at least, a workaround). I replaced all instances of device names of the form /dev/md_dX with corresponding names of the form /dev/mdX (e.g., /dev/md_d1 -> /dev/md1). This involved editing /boot/grub/
There may still be a bug / misfeature in mountall, however: Why did mountall hang when the root was on /dev/md_d1, but it worked fine when the root was on the (supposedly equivalent) /dev/md1 (and similarly for other filesystems on other RAID1 mirrors)?