System frezes after fscking root

Bug #447817 reported by Jon Severinsson
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cryptsetup (Ubuntu)
New
Undecided
Unassigned
Nominated for Lucid by Michael Evans

Bug Description

Binary package hint: mountall

I have both / and /home on seperate luks encrypted partitions. / is decrypted using my passkey by the initramfs and /home is decrypted using a keyfile stored on / by the init scripts.

After upgrading from mountall 0.1.8 to 0.2.0 my system foze at boot. The last things it does before freezing is loading apparmor, fscking /boot and fscking /. It never gives any output related to decrypring or fscking /home.

After some trial and error I found a workaround:
In grub I replaced "ro quiet splash" with "text rw init=/bin/bash". I don't know if "text" is nessesary in the final version of the workaround, but during some previous trials I got a hung X with no ability to switch to text console.
Then I entered my passphrase for / as usual.
After I while I got a root prompt and issuedd the following commands:
mkdir /dev/shm /dev/pts # Don't know if this was nessesary, but df errored out becouse they was missing
cryptsetup luksOpen /dev/sdb2 faolain-home # This is my home partition
mount -a
mount -o remount,ro / # As the system otherwise hangs while fscking root
exec init
At this point the boot sequence resumes. First it fscks / (but not /boot or /home), then it tries to decrypt my three luks partions (/, swap and /home), but ofcourse / and /home fails (as they are already decrypted), and then the rest of the init scripts run and I get a login prompt (textual as that is what I requested in grub).

At this point I downloaded the mountall 0.1.8 deb using wget and installed it using dpkg -i, and after the next reboot it started working again.

I havn't quite figgured out what exactly caused the problem, but I think mountall tried to fsck /dev/mapper/faolain-home before cryptsetup got a chance to decrypt it. This is as it fscked /boot before running cryptsetup, and I believe /boot and /home should be fscked at the same time, as they have the same pass value in /etc/fstab

========================================
My /etc/crypttab
========================================
faolain-root UUID=3443152a-2b1c-405b-b904-e09d5c05d759 none luks
faolain-swap UUID=576ca4ba-940c-48c3-afc1-dd11e83e48d3 /etc/cryptkeys/faolain-swap luks
faolain-home UUID=a230afe6-c04d-4951-9ef8-2c796251906f /etc/cryptkeys/faolain-home luks
========================================

========================================
My /etc/fstab
========================================
# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
LABEL=faolain-swap none swap sw 0 0
LABEL=faolain-boot /boot ext2 relatime,nosuid,nodev 0 2
LABEL=faolain-root / ext3 relatime,barrier=1,user_xattr,acl,errors=remount-ro 0 1
LABEL=faolain-home /home ext3 relatime,barrier=1,user_xattr,acl,nosuid,nodev 0 2
tmpfs /tmp tmpfs nosuid,nodev,mode=1777 0 0
tmpfs /var/tmp tmpfs nosuid,nodev,mode=1777 0 0
========================================

ProblemType: Bug
Architecture: i386
Date: Sat Oct 10 08:16:36 2009
DistroRelease: Ubuntu 9.10
Package: mountall 0.1.8
ProcEnviron:
 LANGUAGE=sv_SE:sv:en_GB:en
 PATH=(custom, no user)
 LANG=sv_SE.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-13.43-generic
SourcePackage: mountall
Uname: Linux 2.6.31-13-generic i686

Revision history for this message
Jon Severinsson (jon-severinsson) wrote :
Revision history for this message
Scott James Remnant (Canonical) (canonical-scott) wrote :

it sounds like mountall is simply waiting for the encrypted disk to appear, and that you didn't get prompted for a passphrase means this is a cryptsetup issue

affects: mountall (Ubuntu) → cryptsetup (Ubuntu)
Revision history for this message
Jon Severinsson (jon-severinsson) wrote :

A small correction: Cryptsetup is not supposed to ask for a passphrase, as it should use the keyfile configured in /etc/crypttab. But that is most likely not relevant to the issue at hand.

However, please do note that cryptsetup 1.0.6+20090405.svn49-1ubuntu4 will set up the disk propperly when used together with mountall 01.8, but the exact same version off cryptsetup never does it's job when used together with mountall 0.2.0, presumably becouse mountall waits for the disk to appear indefenitely while somehow blocking cryptsetup from running at all until it mountall is done, a classic moment 22.

Revision history for this message
Jon Severinsson (jon-severinsson) wrote :

As I realized mountall 1.0 had been released I decided to try this again, but it still fails. This time, however, I'm getting an error message, rather than the system just freezing:

========================================
One or more of the mounts listed in /etc/fstab cannot yet be mounted:
swap: waiting for LABEL=faolain-swap
/home: waiting for LABEL=faolain-home
/var/tmp: waithing for tmpfs
Press ESC to enter a recovery shell
========================================

After pressing ESC I'm getting the following message:
========================================
key slot 0 unlocked.
Command successful.
General error mounting filesystems.
A maintenance shell will now be started.
CONTROL-D will terminate this shell and re-try.
root@faolain:~#
========================================

I used the maintenance shell to look around, and found that /dev/mapper/faolain-root exists and is mounted read-write as /, neither /dev/mapper/faolain-home nor /dev/mapper/faolain-swap exists, and that one of my tmpfs (/tmp) is mounted, but the other (/var/tmp) is not.

I also tried to start the cryptdisks-enable upstart script manually (by running "start cryptdisks-enable" in the shell) and this will create /dev/mapper/faolain-home and /dev/mapper/faolain-swap properly without asking for any password. Pressing Ctrl-D will however just restart usplash and gives me the same error massages again (see above). This time, however, pressing ESC will not give me another maintenance shell.

If I in addition to starting the cryptdisks-enable upstart script manually also mount all filesystems manually (by running "mount -a" in the shell) before pressing Ctr-D, then the system will resume the boot sequence and I'll get to my desktop eventually.

As cryptdisks-enable *does* do the right thing if started manually, I really do not believe the problem lies there, but rather with mountall.

This testing was done using mountall 1.0 and cryptsetup 1.0.6+20090405.svn49-1ubuntu7. Downgrading to mountall 0.1.8 while keeping cryptsetup 1.0.6+20090405.svn49-1ubuntu7 still solves the problem.

Revision history for this message
sunbird (sunbird) wrote :

I can confirm this problem on my system. My system is slightly different in that my / swap and /home are each crypted with a passphrase. Until recently (i.e. the last week), the system would prompt for passphrases for / and swap, and then throw an error message about waiting too long to mount fs (for /home) and ask if I wanted to enter a recovery shell. But if I just waited a few seconds more, it would prompt for my passphrase for /home and boot as normal.

As of this week, it will no longer boot past the prompt for the recovery shell. I have to use the shell to manually open the crypt and mount /home, then it continues.

This is with cryptsetup 1.0.6+20090405.svn49-1ubuntu7 and mountall 1.0. I will try to downgrade to mountall 0.1.8 to see if that fixes.

Revision history for this message
Michael Evans (mjevans1983) wrote :

I too am experiencing issues with multiple luks partitions. Under a single LVM I have three crypt-volumes. One for /, /home, and swap. All require passwords.

I have read the man page for crypttab and my entries are NOT using noearly; therefore all three partitions should be prompted for and asked in the INITRAMFS. Only / is asked for.

If I add break=bottom to the kernel parameters and manually invoke the proper cryptsetup command to create the name_crypt decrypted devices and then exit the recovery shell my system will boot.

Ideally I would like to configure /when/ I am asked for a pass-phrase so that I might control precisely when unlocking is attempted.

early=during initramfs, before any filesystem mounts, BEFORE RESUME IS ATTEMPTED
(???)=after / has been mounted, all paths relative to that container.
init=first thing after leaving the initramfs and invoking from /
(???)=After the first mount -a pass but before the second mount -a pass
manual=Prompted when started by the user
noauto=Prompted when started by the user

Revision history for this message
Michael Evans (mjevans1983) wrote :

This, BTW, was at one point working in Karmic, but recently broke (I don't remember exactly what updates that occurred with) and also showed up when I re-installed with just Lucid since it had otherwise become stable enough for near normal use (and more in depth testing).

Revision history for this message
Steve Langasek (vorlon) wrote :

> I have read the man page for crypttab and my entries are NOT using
> noearly; therefore all three partitions should be prompted for and asked
> in the INITRAMFS. Only / is asked for.

That's not what 'noearly' means. It refers to two passes after the root filesystem has been mounted. We don't currently implement that two-pass system correctly, but that's unrelated to any problem you would be experiencing with the filesystem layout as described.

If the passphrase for the swap partition isn't requested in the initramfs, that's a bug. Please file a separate bug report about this.

> Ideally I would like to configure /when/ I am asked for a pass-phrase
> so that I might control precisely when unlocking is attempted.

> early=during initramfs, before any filesystem mounts, BEFORE RESUME IS ATTEMPTED

There's no reason at all for this to be configurable. The only devices that are ever accessed in the initramfs are the root device and the swap partition (to check for a resume signature); typing in passphrases for other filesystems at this stage has no value, because if the system *does* resume from swap, the passphrase you typed won't be used.

> (???)=after / has been mounted, all paths relative to that container.
> init=first thing after leaving the initramfs and invoking from /

There's no reason to draw a line between these two cases.

> (???)=After the first mount -a pass but before the second mount -a pass

There are no "mount -a passes" at all in lucid.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.