OpenStack Compute (nova)

Bug #1318544
Comment #0

Comment 0 for bug 1318544

Revision history for this message

Thijs van Leeuwen (n-thijs) wrote on 2014-05-12:

Hi All,

I started building a Openstack cloud based on the new LTS version of Ubuntu (14.04). I installed both Control and Compute nodes as VM's on a XenServer. I selected the option 'Other Operating System' so this is running as HVM.

The cluster is up and running: i can start instances, create storage and allocate IP addresses. But the error i got is popping up when the instance is in spawning state. The error i attached is from the nova-compute node.

I use a IBM storwize backend with Cinder connection using iSCSI. I can see the instances being created, the storage being connected but the following error is keep coming back.

Locally on the XenServer SDB and SDC existed, which leaded to a earlier error in which HDB was HDC. I removed SDC locally from the XenServer using: "echo 1 > /sys/block/sdc/device/delete". This changed the HDC error message to HDB. The same thing i did for SDB. Both where virtual drives from DRAC. But still this message is keep popping up with HDB.

I read an other article about a similar error, this guy was pointed in the direction of a HVM vs PV solution because the VM is communicating with the DOM0 via it's kernel. I checked and after installing the XenServer tools the UUID is there and is readable.

While trying the solution above i also focused on the HDB reference, this was strange because with all the rewrite options in nova i would expect a reference to xvda: "xenapi_remap_vbd_dev=false"

Could you guys point me in the right direction or help me solve this problem? I think this isn't a mis-configuration because i tried all possible configurations and this message keeps popping up.

This is the last step towards a working openstack cloud with Ubuntu 14.04 and XenServer 6.2 all patches applied.

2014-05-12 09:00:30.918 6660 ERROR nova.compute.manager [req-595bc50b-e8b2-4de4-8c38-09887c8d4c82 f8609ec1f5254fcfb1fdf6df76876805 41971b840c7a405d9da5052a2506a1c9] [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] Error: Timeout waiting for device hdb to be created
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] Traceback (most recent call last):
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1311, in _build_instance
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] set_access_ip=set_access_ip)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 399, in decorated_function
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] return function(self, context, *args, **kwargs)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1723, in _spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] six.reraise(self.type_, self.value, self.tb)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1720, in _spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] block_device_info)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/driver.py", line 230, in spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] admin_password, network_info, block_device_info)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 357, in spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] network_info, block_device_info, name_label, rescue)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 526, in _spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] undo_mgr.rollback_and_reraise(msg=msg, instance=instance)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 830, in rollback_and_reraise
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] self._rollback()
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] six.reraise(self.type_, self.value, self.tb)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 509, in _spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] attach_devices_step(undo_mgr, vm_ref, vdis, disk_image_type)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 146, in inner
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] rv = f(*args, **kwargs)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 443, in attach_devices_step
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] attach_disks(undo_mgr, vm_ref, vdis, disk_image_type)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 390, in attach_disks
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] injected_files)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 595, in _attach_disks
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] flavor['root_gb'])
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 1037, in try_auto_configure_disk
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] _auto_configure_disk(session, vdi_ref, new_gb)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 1018, in _auto_configure_disk
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] with vdi_attached_here(session, vdi_ref, read_only=False) as dev:
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] return self.gen.next()
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 2200, in vdi_attached_here
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] _wait_for_device(dev)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 2155, in _wait_for_device
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] _('Timeout waiting for device %s to be created') % dev)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] StorageError: Timeout waiting for device hdb to be created
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]

Best,
- Thijs

Hi All,

I use a IBM storwize backend with Cinder connection using iSCSI. I can see the instances being created, the storage being connected but the following error is keep coming back.

While trying the solution above i also focused on the HDB reference, this was strange because with all the rewrite options in nova i would expect a reference to xvda: "xenapi_remap_vbd_dev=false"

Could you guys point me in the right direction or help me solve this problem? I think this isn't a mis-configuration because i tried all possible configurations and this message keeps popping up.

This is the last step towards a working openstack cloud with Ubuntu 14.04 and XenServer 6.2 all patches applied.

2014-05-12 09:00:30.918 6660 ERROR nova.compute.manager [req-595bc50b-e8b2-4de4-8c38-09887c8d4c82 f8609ec1f5254fcfb1fdf6df76876805 41971b840c7a405d9da5052a2506a1c9] [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] Error: Timeout waiting for device hdb to be created
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] Traceback (most recent call last):
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1311, in _build_instance
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     set_access_ip=set_access_ip)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 399, in decorated_function
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     return function(self, context, *args, **kwargs)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1723, in _spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     six.reraise(self.type_, self.value, self.tb)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1720, in _spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     block_device_info)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/driver.py", line 230, in spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     admin_password, network_info, block_device_info)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 357, in spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     network_info, block_device_info, name_label, rescue)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 526, in _spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     undo_mgr.rollback_and_reraise(msg=msg, instance=instance)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 830, in rollback_and_reraise
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     self._rollback()
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     six.reraise(self.type_, self.value, self.tb)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 509, in _spawn
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     attach_devices_step(undo_mgr, vm_ref, vdis, disk_image_type)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 146, in inner
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     rv = f(*args, **kwargs)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 443, in attach_devices_step
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     attach_disks(undo_mgr, vm_ref, vdis, disk_image_type)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 390, in attach_disks
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     injected_files)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 595, in _attach_disks
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     flavor['root_gb'])
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 1037, in try_auto_configure_disk
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     _auto_configure_disk(session, vdi_ref, new_gb)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 1018, in _auto_configure_disk
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     with vdi_attached_here(session, vdi_ref, read_only=False) as dev:
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     return self.gen.next()
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 2200, in vdi_attached_here
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     _wait_for_device(dev)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 2155, in _wait_for_device
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     _('Timeout waiting for device %s to be created') % dev)
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] StorageError: Timeout waiting for device hdb to be created
2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]

Best,
- Thijs