Ubuntu
lvm2 package

/dev/dm-10 is not created for LVM volume

Bug #797226 reported by Arie Skliarouk
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
lvm2 (Fedora)
Fix Released
High
lvm2 (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

Binary package hint: udev

After I created 10th logical volume (shared - see below), udev did not create /dev/dm-10 device node:
root@mf:~# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 236 2011-06-05 09:04 control
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-devs -> ../dm-3
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-devs2 -> ../dm-5
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-devs3 -> ../dm-6
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-devs4 -> ../dm-8
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-dspp -> ../dm-1
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-lxc -> ../dm-0
lrwxrwxrwx 1 root root 7 2011-06-12 00:52 mf-mail -> ../dm-9
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-matchbox -> ../dm-4
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-muze -> ../dm-2
lrwxrwxrwx 1 root root 8 2011-06-14 17:18 mf-shared -> ../dm-10
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-tests -> ../dm-7
root@mf:~# ls -l /dev/dm*
brw-rw---- 1 root disk 251, 0 2011-06-05 09:04 /dev/dm-0
brw-rw---- 1 root disk 251, 1 2011-06-05 09:04 /dev/dm-1
brw-rw---- 1 root disk 251, 2 2011-06-05 09:04 /dev/dm-2
brw-rw---- 1 root disk 251, 3 2011-06-05 09:04 /dev/dm-3
brw-rw---- 1 root disk 251, 4 2011-06-05 09:04 /dev/dm-4
brw-rw---- 1 root disk 251, 5 2011-06-05 09:04 /dev/dm-5
brw-rw---- 1 root disk 251, 6 2011-06-05 09:04 /dev/dm-6
brw-rw---- 1 root disk 251, 7 2011-06-05 09:04 /dev/dm-7
brw-rw---- 1 root disk 251, 8 2011-06-05 09:04 /dev/dm-8
brw-rw---- 1 root disk 251, 9 2011-06-12 00:52 /dev/dm-9
root@mf:~#

Ubuntu 11.04 server amd64, kernel 2.6.38-8-server.
ii udev 167-0ubuntu3 rule-based device node and kernel event manager
ii lvm2 2.02.66-4ubuntu2 The Linux Logical Volume Manager
ii libdevmapper-event1.02.1 2:1.02.48-4ubuntu2 The Linux Kernel Device Mapper userspace library
ii libdevmapper1.02.1 2:1.02.48-4ubuntu2 The Linux Kernel Device Mapper userspace library

See original description
Revision history for this message
In , G. (g.-redhat-bugs) wrote :

I think I've found why none of my lvm mounts are working on boot up.

First the vgchange in fedora-storage-init never seems to get hit. And udev seems to have a problem setting up the drivers as well.

[root@liandra systemd]# /sbin/lvm vgchange -a y --sysinit
  10 logical volume(s) in volume group "LiandraStore" now active
  semid 32768: semop failed for cookie 0xd4d83af: incorrect semaphore state
  Failed to set a proper state for notification semaphore identified by cookie value 223183791 (0xd4d83af) to initialize waiting for incoming notifications.
  /dev/mapper/LiandraStore-VirtualMachines not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-Old--Liandra not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-Backups not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-LiveCD not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-iTunes_Media not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-DVD--Working not set up by udev: Falling back to direct node creation.
  The link /dev/LiandraStore/VirtualMachines should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/Old-Liandra should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/Backups should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/LiveCD should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/iTunes_Media should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/DVD-Working should had been created by udev but it was not found. Falling back to direct link creation.

Any ideas how I debug this further?

Revision history for this message
In , G. (g.-redhat-bugs) wrote :

This may be related to by dbus not starting as well in bug: 689861. Not sure.

Revision history for this message
In , Harald (harald-redhat-bugs) wrote :

are you sure udevd is running?

Revision history for this message
In , G. (g.-redhat-bugs) wrote :

I'm assuming it is:

[root@liandra ~]# ps -ef | grep udev
root 488 1 0 Mar23 ? 00:00:00 /sbin/udevd
root 18778 18319 0 09:12 pts/1 00:00:00 grep --color=auto udev

[root@liandra ~]# lvcreate -L 10G -n Test LiandraStore
  /dev/mapper/LiandraStore-Test not set up by udev: Falling back to direct node creation.
  The link /dev/LiandraStore/Test should had been created by udev but it was not found. Falling back to direct link creation.
  Logical volume "Test" created

[root@liandra ~]# lvremove LiandraStore/Test
Do you really want to remove active logical volume Test? [y/n]: y
  Logical volume "Test" successfully removed
  semid 98304: semop failed for cookie 0xd4d613a: incorrect semaphore state
  Failed to set a proper state for notification semaphore identified by cookie value 223174970 (0xd4d613a) to initialize waiting for incoming notifications.

Any further tests I ran run to verify it is running correctly?

Revision history for this message
In , Harald (harald-redhat-bugs) wrote :

any ideas from the lvm team, what's going wrong here?

Revision history for this message
In , Milan (milan-redhat-bugs) wrote :

This usually means, that you are using udev-enabled lvm (default in Fedora) but there are missing lvm udev rules (explicitly rule with "dmsetup udevcomplete" call, 95-dm-notify.rules).

lvm will fallback to manual node removal (I think that the warning is about semaphore state is some minor bug in lvm code but it is just consequence - if udev rules missing, your system is simply misconfigured.)

Verify, that device-mapper is properly installed.

To diagnose it, please run the lvm command with -vvvv and add output here.

Revision history for this message
In , G. (g.-redhat-bugs) wrote :

Created attachment 488442
Verbose lvcreate command

Revision history for this message
In , G. (g.-redhat-bugs) wrote :

Also verified the /lib/udev/rules.d/95-dm-notify.rules is the same in F15 as it is in other releases.

Revision history for this message
In , Peter (peter-redhat-bugs) wrote :

What's the kernel version used? Is this a stock Fedora kernel?

Revision history for this message
In , G. (g.-redhat-bugs) wrote :

Yes it's stock. 2.6.38-1.fc15.x86_64

Revision history for this message
In , Peter (peter-redhat-bugs) wrote :

Well, based on the log, uevent was not generated correctly:

  " Uevent not generated! Calling udev_complete internally to avoid process lock-up."

This decrements the value of the semaphore used for synchronization. However, if the event (for some reason) *is* generated despite the fact that the kernel function responsible for sending the uevent reports a failure and then we decrement the value once again from within the udev rules based on the generated event (/lib/udev/rules.d/95-dm-notify.rules), we can end up with that:

  "semid 32768: semop failed for cookie 0xd4d83af: incorrect semaphore state..."

The other messages about "... not set up by udev. Falling back to direct node creation" is just a consequence of failed synchronisation with udev (since we decremented semaphore value prematurely).

However I'm not quite sure now why this happens. It could be a bug in kernel code as well... (but I can't reproduce that)

Revision history for this message
In , Harald (harald-redhat-bugs) wrote :

you might also try the new udev version in F15

Revision history for this message
In , G. (g.-redhat-bugs) wrote :

New version in F15 seems to have solved the problems.

Arie Skliarouk (skliarie)
description: updated
Revision history for this message
Arie Skliarouk (skliarie) wrote :

Something weird happens:
The /dev/dm-10 is created on a freshly installed ubuntu 11.04. When I tried to create the logical volume again on the affected machine - it worked, albeit with a strange message:

  The link /dev/mf/lvmtest should had been created by udev but it was not found. Falling back to direct link creation.
  Logical volume "lvmtest" created

When I tried to remove it, I got an frightening message:

root@mf:~# lvremove /dev/mf/lvmtest
Do you really want to remove active logical volume lvmtest? [y/n]: y
  semid 524288: semop failed for cookie 0xd4d2311: incorrect semaphore state
  Failed to set a proper state for notification semaphore identified by cookie value 223159057 (0xd4d2311) to initialize waiting for incoming notifications.
  Logical volume "lvmtest" successfully removed

Googling for the keywords revealed similar problem on Fedora F15, which was solved by updated version of udev:
https://bugzilla.redhat.com/show_bug.cgi?id=690308

So looks like initial assignment of the bug to udev was correct after all.

Revision history for this message
Arie Skliarouk (skliarie) wrote :
Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :

Original reporter had tried to change it to lvm2 from udev as part of another report; merged as dup and changed.

affects: udev (Ubuntu) → lvm2 (Ubuntu)
Revision history for this message
Arie Skliarouk (skliarie) wrote :

I tried to understand the changes done to the udev-167-2 in F15 to fix the problem.
The commitdiffs can be see on the following page ("fixed udev-trigger.service"):
http://pkgs.fedoraproject.org/gitweb/?p=udev.git;a=shortlog

But looks like the bug already was fixed in the udev of ubuntu...

Dave, do you imply that the bug is really in lvm2 subsystem?

Revision history for this message
Arie Skliarouk (skliarie) wrote :

Until this is fixed, I wrote a shell script that looks in the /dev/mapper directory and recreates nodes that the udev missed.

Helps to me, might help to someone else.

Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :

Hi Arie,
  No I'm not sure if it's lvm2 or udev, just that you originally had two separate bug reports one for each and I just merged them and it looked like you'd made the 2nd one lvm2 hence why I changed this one.

Incidentally, I've seen that semaphore error when using cryptsetup luksOpen to open crypted fs; it doesn't seem to cause problems.

At the moment I seem to be getting all the entries in /dev/dm-* and /dev/mapper/* , but I've seen some errors/warnings on and off that I was attributing to problems with COW and similar lvm devices; but I could never really pin it down.

Dave

Revision history for this message
Arie Skliarouk (skliarie) wrote :

Here is the screenshot of the console when one can see failed attempts of udev to create the device nodes.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in lvm2 (Ubuntu):
status: New → Confirmed
Revision history for this message
Arie Skliarouk (skliarie) wrote :

The bug still exists in ubuntu 13.04.

Bug Watch Updater (bug-watch-updater)
Changed in lvm2 (Fedora):
importance: Unknown → High
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.