kolla

cinder nova volume attach not working

Bug #1469323 reported by Jeff Peeler on 2015-06-26

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kolla	Fix Released	Critical	Ryan Hallisey	kolla liberty-2 "liberty-2"

Bug Description

log/debug provided by Ryan:

2015-06-26 21:46:19.148 1 INFO nova.compute.manager [req-da63d4c6-579e-416f-a6fa-cd246074df3e 5759e5c1587945c7a74d6a481285a580 102baa7f2b8749aba7f1cc80fe4520f4 - - -] [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Attaching volume 031f9e4b-d03c-4847-bbf9-649827e5b540 to /dev/vdb
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [req-da63d4c6-579e-416f-a6fa-cd246074df3e 5759e5c1587945c7a74d6a481285a580 102baa7f2b8749aba7f1cc80fe4520f4 - - -] [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Driver failed to attach volume 031f9e4b-d03c-4847-bbf9-649827e5b540 at /dev/vdb
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Traceback (most recent call last):
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 254, in attach
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] device_type=self['device_type'], encryption=encryption)
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1050, in attach_volume
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] self._connect_volume(connection_info, disk_info)
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1001, in _connect_volume
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] driver.connect_volume(connection_info, disk_info)
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] return f(*args, **kwargs)
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 434, in connect_volume
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] self._run_iscsiadm(iscsi_properties, ("--rescan",))
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 359, in _run_iscsiadm
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] check_exit_code=check_exit_code)
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 228, in execute
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] return processutils.execute(*cmd, **kwargs)
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 240, in execute
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] cmd=sanitized_cmd)
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] ProcessExecutionError: Unexpected error while running command.
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-031f9e4b-d03c-4847-bbf9-649827e5b540 -p 10.8.144.14:3260 --rescan
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Exit code: 21
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Stdout: u''
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Stderr: u'iscsiadm: No session found.\n'
2015-06-26 21:46:22.770 1 ERROR nova.virt.block_device [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8]
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [req-da63d4c6-579e-416f-a6fa-cd246074df3e 5759e5c1587945c7a74d6a481285a580 102baa7f2b8749aba7f1cc80fe4520f4 - - -] [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Failed to attach 031f9e4b-d03c-4847-bbf9-649827e5b540 at /dev/vdb
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Traceback (most recent call last):
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4443, in _attach_volume
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] do_check_attach=False, do_driver_attach=True)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 50, in wrapped
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] ret_val = method(obj, context, *args, **kwargs)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 263, in attach
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] connector)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] six.reraise(self.type_, self.value, self.tb)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 254, in attach
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] device_type=self['device_type'], encryption=encryption)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1050, in attach_volume
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] self._connect_volume(connection_info, disk_info)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1001, in _connect_volume
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] driver.connect_volume(connection_info, disk_info)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] return f(*args, **kwargs)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 434, in connect_volume
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] self._run_iscsiadm(iscsi_properties, ("--rescan",))
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 359, in _run_iscsiadm
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] check_exit_code=check_exit_code)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 228, in execute
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] return processutils.execute(*cmd, **kwargs)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 240, in execute
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] cmd=sanitized_cmd)
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] ProcessExecutionError: Unexpected error while running command.
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-031f9e4b-d03c-4847-bbf9-649827e5b540 -p 10.8.144.14:3260 --rescan
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Exit code: 21
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Stdout: u''
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8] Stderr: u'iscsiadm: No session found.\n'
2015-06-26 21:46:23.352 1 ERROR nova.compute.manager [instance: 09e5d23c-cfd9-41a9-8721-331b942efea8]
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher [req-da63d4c6-579e-416f-a6fa-cd246074df3e 5759e5c1587945c7a74d6a481285a580 102baa7f2b8749aba7f1cc80fe4520f4 - - -] Exception during message handling: Unexpected error while running command.
Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-031f9e4b-d03c-4847-bbf9-649827e5b540 -p 10.8.144.14:3260 --rescan
Exit code: 21
Stdout: u''
Stderr: u'iscsiadm: No session found.\n'
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher payload)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 335, in decorated_function
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 306, in decorated_function
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 363, in decorated_function
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 351, in decorated_function
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4433, in attach_volume
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher do_attach_volume(context, instance, driver_bdm)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4431, in do_attach_volume
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher bdm.destroy()
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4428, in do_attach_volume
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher return self._attach_volume(context, instance, driver_bdm)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4451, in _attach_volume
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher self.volume_api.unreserve_volume(context, bdm.volume_id)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4443, in _attach_volume
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher do_check_attach=False, do_driver_attach=True)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 50, in wrapped
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher ret_val = method(obj, context, *args, **kwargs)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 263, in attach
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher connector)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 254, in attach
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher device_type=self['device_type'], encryption=encryption)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1050, in attach_volume
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher self._connect_volume(connection_info, disk_info)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1001, in _connect_volume
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher driver.connect_volume(connection_info, disk_info)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 434, in connect_volume
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher self._run_iscsiadm(iscsi_properties, ("--rescan",))
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 359, in _run_iscsiadm
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher check_exit_code=check_exit_code)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/utils.py", line 228, in execute
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 240, in execute
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher cmd=sanitized_cmd)
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-031f9e4b-d03c-4847-bbf9-649827e5b540 -p 10.8.144.14:3260 --rescan
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher Exit code: 21
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher Stdout: u''
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher Stderr: u'iscsiadm: No session found.\n'
2015-06-26 21:46:23.553 1 ERROR oslo_messaging.rpc.dispatcher

Revision history for this message

Steven Dake (sdake) wrote on 2015-06-27:

While it would be nice if this problem could be solved prior to Monday, this won't be a blocker for our release. We are releasing Monday, assuming the dhcp issue with neutron can be resolved.

Changed in kolla:
status:	New → Triaged
importance:	Undecided → High

Steven Dake (sdake) on 2015-06-29

Changed in kolla:
milestone:	liberty-1 → liberty-2
importance:	High → Critical

Revision history for this message

Steven Dake (sdake) wrote on 2015-07-10:

Download full text (7.1 KiB)

More data from cinder-volume container:
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 252, in inner
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio     return f(*args, **kwargs)
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/lio.py", line 70, in _execute
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio     return utils.execute(*args, **kwargs)
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 152, in execute
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio     return processutils.execute(*cmd, **kwargs)
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio     cmd=sanitized_cmd)
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio ProcessExecutionError: Unexpected error while running command.
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio Command: cinder-rtstool create /dev/cinder-volumes/volume-1a41e8bc-de7f-4ec0-acaf-9661c07609c0 iqn.2010-10.org.openstack:volume-1a41e8bc-de7f-4ec0-acaf-9661c07609c0 CVDzf5gZa8W4nh9enmfR Ls4sJGQVvjCHp2Y4 False -p3260 -a192.168.1.100
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio Exit code: 1
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio Stdout: u''
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio Stderr: u'Traceback (most recent call last):\n  File "/usr/bin/cinder-rtstool", line 10, in <module>\n    sys.exit(main())\n  File "/usr/lib/python2.7/site-packages/cinder/cmd/rtstool.py", line 271, in main\n    **optional_args)\n  File "/usr/lib/python2.7/site-packages/cinder/cmd/rtstool.py", line 97, in create\n    % {\'msg_type\': msg_type, \'port\': portals_port, \'ip\': ip})\n  File "/usr/lib/python2.7/site-packages/oslo_i18n/_message.py", line 208, in __str__\n    raise UnicodeError(msg)\nUnicodeError\n'
2015-07-10 19:41:45.793 98 ERROR cinder.volume.targets.lio 
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager [req-d2479d31-1688-412d-8044-465ffee9137f 0823a3ed7aff4dd1b73c1213cc6747d6 39e82b5267dd4456ba4641f18752ee6e - - -] Create export for volume failed.
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager Traceback (most recent call last):
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1082, in initialize_connection
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager     volume)
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager     return f(*args, **kwargs)
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 643, in create_export
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager     volume_path)
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 220, in create_export
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager     **portals_config)
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/lio.py", line 135, in create_iscsi_target
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager     raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-1a41e8bc-de7f-4ec0-acaf-9661c07609c0.
2015-07-10 19:41:45.795 98 ERROR cinder.volume.manager 
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher [req-d2479d31-1688-412d-8044-465ffee9137f 0823a3ed7aff4dd1b73c1213cc6747d6 39e82b5267dd4456ba4641f18752ee6e - - -] Exception during message handling: Bad or unexpected response from the storage volume backend API: Create export for volume failed.
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1086, in initialize_connection
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher     raise exception.VolumeBackendAPIException(data=err_msg)
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Create export for volume failed.
2015-07-10 19:41:45.797 98 ERROR oslo_messaging.rpc.dispatcher 
2015-07-10 19:41:45.799 98 ERROR oslo_messaging._drivers.common [req-d2479d31-1688-412d-8044-465ffee9137f 0823a3ed7aff4dd1b73c1213cc6747d6 39e82b5267dd4456ba4641f18752ee6e - - -] Returning exception Bad or unexpected response from the storage volume backend API: Create export for volume failed. to caller
2015-07-10 19:41:45.799 98 ERROR oslo_messaging._drivers.common [req-d2479d31-1688-412d-8044-465ffee9137f 0823a3ed7aff4dd1b73c1213cc6747d6 39e82b5267dd4456ba4641f18752ee6e - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1086, in initialize_connection\n    raise exception.VolumeBackendAPIException(data=err_msg)\n', 'VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Create export for volume failed.\n']

OpenStack Infra (hudson-openstack) on 2015-07-15

Changed in kolla:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-17: Fix merged to kolla (master)

Reviewed: https://review.openstack.org/200139
Committed: https://git.openstack.org/cgit/stackforge/kolla/commit/?id=4008ca8619a83ac4ee9c56cb37f42b208d753823
Submitter: Jenkins
Branch: master

commit 4008ca8619a83ac4ee9c56cb37f42b208d753823
Author: Steven Dake <email address hidden>
Date: Thu Jul 9 07:49:28 2015 -0700

Start tgtd in cinder-volume container

    The tgtd process was not being started in the cinder-volume container.
    This results in iSCSI not being avialable for Cinder to operate
    against. Unfortunately cinder doesn't report the problem because LVM
    is responsible for the iSCSI mapping.

Co-Authored-By: Ryan Hallisey <email address hidden>

    Closes-Bug: 1473090
    Closes-Bug: 1469323
    Change-Id: Ie3403c5c3f81beb48960e838b4274e0c5796e387

Changed in kolla:
status:	In Progress → Fix Committed

Sam Yaple (s8m) on 2015-11-20

Changed in kolla:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.