OpenStack Compute (nova)

VMware: Fail to take a snapshot of a VM

Bug #1441235 reported by Chinmaya Bharadwaj
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Low
Unassigned

Bug Description

With the latest code, cluster as the compute driver, Taking a snapshot is failing with following error :

ERROR oslo_messaging.rpc.dispatcher [req-9c49bfdd-641c-4022-90cc-aae697e387f3 admin admin] Exception during message handling: Error occurred while opening URL: https://10.1.214.7/nfc/52135ab3-7409-6a9c-30ba-65f00d264262/disk-1.vmdk for reading.
Cause: HTTPSConnectionPool(host='10.1.214.7', port=443): Max retries exceeded with url: /nfc/52135ab3-7409-6a9c-30ba-65f00d264262/disk-1.vmdk (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher payload)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 324, in decorated_function
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 295, in decorated_function
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 352, in decorated_function
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 340, in decorated_function
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 400, in decorated_function
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher instance=instance)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 390, in decorated_function
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher *args, **kwargs)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 3201, in snapshot_instance
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher task_states.IMAGE_SNAPSHOT)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 3231, in _snapshot_instance
2015-04-07 2

1:16:47.166 TRACE oslo_messaging.rpc.dispatcher update_task_state)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 506, in snapshot
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher self._vmops.snapshot(context, instance, image_id, update_task_state)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 833, in snapshot
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher vmdk_size=vmdk.capacity_in_bytes)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/images.py", line 457, in upload_image_stream_optimized
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher vmdk_size)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/rw_handles.py", line 500, in __init__
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher cacerts=cacerts)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/rw_handles.py", line 79, in _create_read_connection
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher raise exceptions.VimException(excep_msg, excep)
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher VimException: Error occurred while opening URL: https://10.1.214.7/nfc/52135ab3-7409-6a9c-30ba-65f00d264262/disk-1.vmdk for reading.
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher Cause: HTTPSConnectionPool(host='10.1.214.7', port=443): Max retries exceeded with url: /nfc/52135ab3-7409-6a9c-30ba-65f00d264262/disk-1.vmdk (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))
2015-04-07 21:16:47.166 TRACE oslo_messaging.rpc.dispatcher

git log -1 :
commit bf70df295b0529da9f4381f52fc94d328fa2fdb3
Merge: ad329b0 7560620
Author: Jenkins <email address hidden>
Date: Mon Apr 6 23:57:10 2015 +0000

    Merge "neutronv2: only create client once when adding/removing fixed IPs"

Environment : VC 5.5 U1, ESXi 5.5.0, 1834933

Is this an environment issue ? Export OVS template task is stuck for a long time and it fails, attached the snapshot.

See original description
Tags: vmware
Revision history for this message
Chinmaya Bharadwaj (acbharadwaj) wrote :
Revision history for this message
Chinmaya Bharadwaj (acbharadwaj) wrote :
description: updated
Revision history for this message
Sagar Ratnakara Nikam (sagar-r-nikam) wrote :

On further debugging, checking the ESXi hostd logs, contains this error

2015-04-17T06:28:49.081Z [FFD60920 verbose 'Vmsvc.vm:/vmfs/volumes/535f3d61-7c0442d6-56c9-62552a500042/OSTACK_SNAP_3dce2adb-fbe0-49ff-a606-aed25f69c875/OSTACK_SNAP_3dce2adb-fbe0-49ff-a606-aed25f69c875.vmx' opID=2a248111-4f-bb user=vpxuser] Is disk present message: Expected device (ide0:1) does not exist.

OSTACK_SNAP_3dce2adb-fbe0-49ff-a606-aed25f69c875 is the name of the OVF template which is being created as part of CreateSnapshot_Task

The OVF template does not get created and hence snapshot operation fails

Changed the adapter type to "lsiLogic" in glance image, still the same error

2015-04-17T07:02:25.686Z [2A440B70 verbose 'Vmsvc.vm:/vmfs/volumes/535f3d61-7c0442d6-56c9-62552a500042/OSTACK_SNAP_ce9e74ed-c847-4bc6-8d47-10b8e924d3a1/OSTACK_SNAP_ce9e74ed-c847-4bc6-8d47-10b8e924d3a1.vmx' opID=74009579-27-56 user=vpxuser] Is disk present message: Expected device (scsi0:0) does not exist.

The issue is occurring on devstack - nova master branch

Glance image property
chinmay@ChinmayDevstack:~/devstack$ glance image-show 9e4f5f08-364c-4f9e-ac4c-67642e228665
+-------------------------------+--------------------------------------+
| Property | Value |
 +-------------------------------+--------------------------------------+
| Property 'hw_vif_model' | e1000 |
| Property 'vmware_adaptertype' | lsiLogic |
| Property 'vmware_disktype' | sparse |
| checksum | 3fd19141ff969dcb4926b610769e5ba4 |
| container_format | bare |
| created_at | 2015-04-15T11:19:00.000000 |
| deleted | False |
| disk_format | vmdk |
| id | 9e4f5f08-364c-4f9e-ac4c-67642e228665 |
| is_public | True |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros-0.3.2-i386-disk |
| owner | e4cad1de0bb44f5fbfe1ad11fb24e3c2 |
| protected | False |
| size | 17104896 |
| status | active |
| updated_at | 2015-04-17T07:01:58.000000 |
 +-------------------------------+--------------------------------------+

Revision history for this message
Sagar Ratnakara Nikam (sagar-r-nikam) wrote :

The hostd log on ESXi host

Sylvain Bauza (sylvain-bauza)
Changed in nova:
importance: Undecided → Low
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.