OpenStack Compute (nova)

cold migration fails in VMware driver

Bug #1270332 reported by Sabari Murugesan
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Opinion
Wishlist
Radoslav Gerganov
OpenStack Compute (nova) next

Bug Description

With two compute nodes (on different hosts) configured to two different clusters in the same vCenter Server i.e :-

nova migrate <server> fails to migrate a server with the following error :-

2014-01-17 16:00:21.336 ERROR nova.openstack.common.rpc.amqp [req-0c587eb7-3a23-4790-b23d-f4ad005b5fe7 admin admin] Exception during message handling
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp **args)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp payload)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 73, in wrapped
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 244, in decorated_function
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp pass
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 230, in decorated_function
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 295, in decorated_function
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 272, in decorated_function
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 259, in decorated_function
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3163, in resize_instance
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp block_device_info)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 406, in migrate_disk_and_power_off
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp dest, flavor)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1182, in migrate_disk_and_power_off
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp self._session._wait_for_task(instance['uuid'], vm_clone_task)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 857, in _wait_for_task
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp ret_val = done.wait()
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp return hubs.get_hub().switch()
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp return self.greenlet.switch()
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp NovaException: Unable to access the virtual machine configuration: Unable to access file [datastore1]
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp
2014-01-17 16:00:21.497 DEBUG nova.openstack.common.rpc.amqp [-] Making synchronous call on conductor ... from (pid=28144) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:553
2014-01-17 16:00:21.498 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is b52829e908144155970ef23754eb46e3 from (pid=28144) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:556
2014-01-17 16:00:21.498 DEBUG nova.openstack.common.rpc.amqp [-] UNIQUE_ID is 393d7230809b4892a4d7834c5ab56c2f. from (pid=28144) _add_unique_id /opt/stack/nova/nova/openstack/common/rpc/amqp.py:341
2014-01-17 16:00:25.640 WARNING nova.virt.vmwareapi.driver [-] Task [CloneVM_Task] (returnval){
   value = "task-110"
   _type = "Task"
 } status: error Unable to access the virtual machine configuration: Unable to access file [datastore1]

Tags: vmware
Sabari Murugesan (smurugesan)
tags: added: vmware
Revision history for this message
Sabari Murugesan (smurugesan) wrote :

This is because vCenter server is trying to do the following :-

in relocate config spec :-

1. datastore is chosen from the target cluster.
2. host is set to null

The error is because vCenter tries to migrate the VM within the source host, but with a datastore available with the target cluster.

Ideally, the following should have happened in relocate config spec :-

1. datastore is chosen from the target cluster.
2. host is set to null.
3. resource pool is set to the target cluster.

Gary Kotton (garyk)
Changed in nova:
status: New → Confirmed
importance: Undecided → Medium
Sabari Murugesan (smurugesan)
Changed in nova:
assignee: nobody → Sabari Murugesan (smurugesan)
milestone: none → icehouse-3
OpenStack Infra (hudson-openstack)
Changed in nova:
status: Confirmed → In Progress
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-3 → icehouse-rc1
Tracy Jones (tjones-i)
Changed in nova:
milestone: icehouse-rc1 → next
tags: added: icehouse-rc-potential
Revision history for this message
Sabari Murugesan (smurugesan) wrote :

I missed adding the reviews to the bug https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1270332,n,z

Revision history for this message
Sabari Murugesan (smurugesan) wrote :

posting just the review https://review.openstack.org/#/c/76456/

Tracy Jones (tjones-i)
tags: added: icehouse-rc1-potential
removed: icehouse-rc-potential
Tracy Jones (tjones-i)
tags: removed: icehouse-rc1-potential
Sean Dague (sdague)
Changed in nova:
status: In Progress → Confirmed
assignee: Sabari Murugesan (smurugesan) → nobody
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Is this still a problem?

Changed in nova:
status: Confirmed → Incomplete
Revision history for this message
Tracy Jones (tjones-i) wrote :

rado is working on this

Changed in nova:
status: Incomplete → In Progress
assignee: nobody → Radoslav Gerganov (rgerganov)
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

@Radoslav:
Could you already make progress here?

Revision history for this message
Sivasathurappan Radhakrishnan (siva-radhakrishnan) wrote :

@Radoslav: Can you please let us know if you are still working on this bug ?

Revision history for this message
Radoslav Gerganov (rgerganov) wrote :

Yes, I am working on this. The cold migration shares common code with the live migration which is under review:

https://review.openstack.org/#/c/270116/

I was hoping to get some feedback there before pushing all the patches.

Revision history for this message
Augustina Ragwitz (auggy) wrote :

If this issue is addressed in that change, could you consider adding a "closes-bug" tag to the commit message?

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

Looks like this will be driven by the blueprint [1].

[1] https://blueprints.launchpad.net/nova/+spec/vmware-live-migration

Changed in nova:
status: In Progress → Opinion
importance: Medium → Wishlist
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.