Problem with delete clusters

Bug #1777210 reported by admin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
magnum (Ubuntu)
New
Undecided
Unassigned

Bug Description

After forming a cluster of kubernetes machines
the process inserts & builds various LVM based volumes.
However the process seems flawed.

the Base volume is given a name say:
 volume-310ca8b5-980e-4d93-82d2-ddd9ccfabd9a cinder-volumes Vwi-aotz-- 5.00g cinder-volumes-pool

But a the process evolves the docker volumes that reside INSIDE the clusters are called:
 docker-pool docker twi-a-t--- 2.00g

a cluster of 2 machines produces the following:

 docker-pool docker twi-a-t--- 2.00g
 docker-pool docker twi---t-p- 2.00g

Two IDENTICALLY named units....
When the cluster is ready to be torn down with delete the process fails:
Stderr: u'File descriptor 20 (/dev/urandom) leaked on lvremove invocation. Parent PID 15349: /usr/bin/python2\n Logical volume cinder-volumes/volume-310ca8b5-980e-4d93-82d2-ddd9ccfabd9a is used by another device.\n'
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "<string>", line 2, in delete_volume
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server return f(*a, **k)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "<string>", line 2, in delete_volume
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 847, in delete_volume
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server new_status)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 829, in delete_volume
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server self.driver.delete_volume(volume)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 464, in delete_volume
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server self._delete_volume(volume)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 144, in _delete_volume
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server self.vg.delete(name)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 818, in _wrapper
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server raise attempt.get()
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2])
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 741, in delete
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server root_helper=self._root_helper, run_as_root=True)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 126, in execute
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvremove --config activation { retry_deactivation = 1} devices { ignore_suspended_devices = 1} -f cinder-volumes/volume-310ca8b5-980e-4d93-82d2-ddd9ccfabd9a
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server Exit code: 5
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server Stdout: u''
2018-06-16 11:08:06.884 4492 ERROR oslo_messaging.rpc.server Stderr: u'File descriptor 20 (/dev/urandom) leaked on lvremove invocation. Parent PID 15349: /usr/bin/python2\n Logical volume cinder-volumes/volume-310ca8b5-980e-4d93-82d2-ddd9ccfabd9a is used by another device.\n'

The delete is against the LVM NAMED root volume, it fails because the "docker" volumes STILL exist so it fails.

ALSO since multiple clusters can be instigated and they all build volumes called "docker-pool"
you cannot remove them with a simple:
lvremove -f docker/docker-pool
  Multiple VGs found with the same name: skipping docker
  Use --select vg_uuid=<uuid> in place of the VG name.

Magnum fails to correctly handle the naming of LVM volumes and fails to retrieve the UUID associated with the named LVM container. this leads to potential failure when mutiple devices exist during takedown of the container.

this then cascades up the stack

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.