kolla-ansible

cinder volume require /lib/modules to load kernel module

Bug #1684378 reported by Jeffrey Zhang on 2017-04-20

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	Fix Released	Medium	Jeffrey Zhang	kolla-ansible pike-2 "pike"
	Ocata	Fix Released	Undecided	Jeffrey Zhang	kolla-ansible 4.0.2 "ocata"

Bug Description

Got following error when cinder creates snapshot.
The root cause is: lvcreate will load dm_* module when need. So /lib/modules should be mounted in cinder-volume container.

2017-04-20 01:46:19.332 30 ERROR cinder.brick.local_dev.lvm [req-289c6e61-87f0-45ed-b27c-518c952d2a05 2e6c6c47d4ab4d7dad5683045dbd3dac 41fb0654b66848028005751f9fb25f23 - default default] Cmd :sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvcreate --name _snapshot-30e4c32d-4312-4519-ad46-f0237629b690 --snapshot cinder-volumes/volume-ad33c384-6431-4b5c-9abc-aa1ea20a2138 -L 1.00g
2017-04-20 01:46:19.332 30 ERROR cinder.brick.local_dev.lvm [req-289c6e61-87f0-45ed-b27c-518c952d2a05 2e6c6c47d4ab4d7dad5683045dbd3dac 41fb0654b66848028005751f9fb25f23 - default default] StdOut :
2017-04-20 01:46:19.333 30 ERROR cinder.brick.local_dev.lvm [req-289c6e61-87f0-45ed-b27c-518c952d2a05 2e6c6c47d4ab4d7dad5683045dbd3dac 41fb0654b66848028005751f9fb25f23 - default default] StdErr :File descriptor 20 (/dev/urandom) leaked on lvcreate invocation. Parent PID 88: /var/lib/kolla/venv/bin/python
  WARNING: Not using lvmetad with older version.
  /usr/sbin/modprobe failed: 1
  snapshot: Required device-mapper target(s) not detected in your kernel.
  Run `lvcreate --help' for more information.

2017-04-20 01:46:19.334 30 DEBUG cinder.utils [req-289c6e61-87f0-45ed-b27c-518c952d2a05 2e6c6c47d4ab4d7dad5683045dbd3dac 41fb0654b66848028005751f9fb25f23 - default default] Failed attempt 3 _print_stop /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/utils.py:780
2017-04-20 01:46:19.334 30 DEBUG cinder.utils [req-289c6e61-87f0-45ed-b27c-518c952d2a05 2e6c6c47d4ab4d7dad5683045dbd3dac 41fb0654b66848028005751f9fb25f23 - default default] Have been at this for 8.04 seconds _print_stop /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/utils.py:782
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server [req-289c6e61-87f0-45ed-b27c-518c952d2a05 2e6c6c47d4ab4d7dad5683045dbd3dac 41fb0654b66848028005751f9fb25f23 - default default] Exception during message handling
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "<decorator-gen-239>", line 2, in create_snapshot
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 888, in create_snapshot
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server snapshot.save()
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 880, in create_snapshot
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server model_update = self.driver.create_snapshot(snapshot)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 447, in create_snapshot
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server self.configuration.lvm_type)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/utils.py", line 796, in _wrapper
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 212, in call
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server raise attempt.get()
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 247, in get
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2])
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 200, in call
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 595, in create_lv_snapshot
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server run_as_root=True)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/executor.py", line 49, in _execute
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/utils.py", line 123, in execute
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 400, in execute
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvcreate --name _snapshot-30e4c32d-4312-4519-ad46-f0237629b690 --snapshot cinder-volumes/volume-ad33c384-6431-4b5c-9abc-aa1ea20a2138 -L 1.00g
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server Exit code: 3
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server Stdout: u''
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server Stderr: u"File descriptor 20 (/dev/urandom) leaked on lvcreate invocation. Parent PID 88: /var/lib/kolla/venv/bin/python\n WARNING: Not using lvmetad with older version.\n /usr/sbin/modprobe failed: 1\n snapshot: Required device-mapper target(s) not detected in your kernel.\n Run `lvcreate --help' for more information.\n"
2017-04-20 01:46:19.385 30 ERROR oslo_messaging.rpc.server

Jeffrey Zhang (jeffrey4l) on 2017-04-20

Changed in kolla-ansible:
assignee:	nobody → Jeffrey Zhang (jeffrey4l)
milestone:	none → pike-1
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-20: Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.openstack.org/458342

Changed in kolla-ansible:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-20: Fix merged to kolla-ansible (master)

Reviewed: https://review.openstack.org/458342
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=13e5942a578f830d6eb94e668c91977ea366f1d3
Submitter: Jenkins
Branch: master

commit 13e5942a578f830d6eb94e668c91977ea366f1d3
Author: Jeffrey Zhang <email address hidden>
Date: Thu Apr 20 13:57:35 2017 +0800

Mount /lib/modules folder for cinder-volume container

    When using cinder lvm backend storage driver, lvcreate will load dm_*
    linux modules when need. So /lib/modules is needed by cinder_volume
    container.

Change-Id: I80985794d6aaa6b4ddc71680a97e0d07085dafa7
Closes-Bug: #1684378

Changed in kolla-ansible:
status:	In Progress → Fix Released

Revision history for this message

Jeffrey Zhang (jeffrey4l) wrote on 2017-05-03:

backport is commited by https://review.openstack.org/#/c/458430/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-06-09: Fix included in openstack/kolla-ansible 5.0.0.0b2

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b2 development milestone.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-06-19: Fix included in openstack/kolla-ansible 4.0.2

This issue was fixed in the openstack/kolla-ansible 4.0.2 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.