VMware: attaching a volume to the VM failed

Hi,
Can you please try this again. The vmware cinder support was approved on Sept 5th.
In addition to this can you please indicate which cinder backend you used.
Thanks
Gary

Hi Gary,
I tried today. There is still errors when I attach the volume to the VM.

============================n-cpu=========================

2013-10-11 06:15:48.317 ERROR nova.openstack.common.rpc.amqp [req-1652a9a6-a236-49b0-8e83-063612b1f5a0 admin demo] Exception during me
ssage handling
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 461, in
_process_data
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp **args)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 17
2, in dispatch
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp payload)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 73, in wrapped
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 243, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp pass
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 229, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 271, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 258, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3640, in attach_vo
lume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp context, instance, mountpoint)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3635, in attach_vo
lume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp mountpoint, instance)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3681, in _attach_v
olume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp connector)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3671, in _attach_v
olume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp encryption=enc...

Hi Gary,
I tried it today. There is still errors when I attach the volume to the VM. The image I use is trend-tinyvm1-flat.vmdk.
============================n-cpu=========================
2013-10-11 06:15:48.317 ERROR nova.openstack.common.rpc.amqp [req-1652a9a6-a236-49b0-8e83-063612b1f5a0 admin demo] Exception during me
ssage handling
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 461, in
_process_data
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp **args)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 17
2, in dispatch
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp payload)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 73, in wrapped
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 243, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp pass
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 229, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 271, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 258, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3640, in attach_vo
lume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp context, instance, mountpoint)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3635, in attach_vo
lume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp mountpoint, instance)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3681, in _attach_v
olume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp connector)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3671, in _attach_v
olume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp encryption=encryption)
2013-10-11 06:15:48....

Hi Vincent,

Can you give some more details on you setup:
1. Have you configured with ESX/VC?
2. Which version of VMware server are you using.

Can you try powering off the instance and then perform a volume attach.

P.S: You could try searching for "The attempted operation cannot be performed in the current state (Powered on)" in http://kb.vmware.com/ if it helps.

The problem seems to arise from the fact that a sparse disk type is used. We are investigating

The issue is when the instance uses an IDE controller.

IDE controller does not support hot adding of a virtual disk http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1025883

However if an IDE controller is not being used for the primary disk of the virtual machine then the issue must not arise.

One possible fix would be to use a SCSI controller (one of http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.wssdk.apiref.doc%2Fvim.vm.device.VirtualSCSIController.html) for volume's disk irrespective of primary disk's controller.

These are configuration options:
vCenter: 5.5
vSphere web service SDK 5.5
Image:
# wget http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat.vmdk
# glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="thin" --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk

we get the error "2013-11-18 21:31:11.859 18737 TRACE nova.openstack.common.rpc.amqp NovaException: The attempted operation cannot be performed in the current state (Powered on)."

The root cause is that some disks are not hot-addable, same we met in bug [1]

The possible fix is stop instance before attach/detach volume. Note this will affect user's normal workload. like we did for bug [1], or mark this won't fix and suggest user to use SCSI controller in comment #6

 [1] https://bugs.launchpad.net/nova/+bug/1247556

This depends instance's image. Maybe we need raise a exception to tell user what 's happening.

   glance add name="ubuntuLTS" disk_format=vmdk container_format=ovf \
is_public=true vmware_adaptertype="lsiLogic" vmware_disktype="preallocated" \
vmware_ostype="ubuntu64Guest" < ubuntuLTS-flat.vmdk

you can use this command to import an image,

you can attach a volume in ACTIVE

Fix proposed to branch: master
Review: https://review.openstack.org/65799

all reviews are abandoned for this and it has not been updated in a long time. moving back to triaged. Please set to in progress if you plan on working on it

This is not a valid cinder bug. Nova vmware driver has to create or use an existing controller in the instance VM to attach the volume and it is unrelated to the controller in volume backing. Please re-open this if you disagree.

raising prio as it is blocking some of the cinder test cases.

The issue is pointed out above, i.e. ide controller does not support hotplugging. To fix this, an exception should be thrown when attempting to attach a volume to an active instance (as pointed out in the review by danpb).

Fix proposed to branch: master
Review: https://review.openstack.org/122251

Reviewed: https://review.openstack.org/122251
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e3fd9b377c082791670ce94ed972e0103f88d399
Submitter: Jenkins
Branch: master

commit e3fd9b377c082791670ce94ed972e0103f88d399
Author: Thang Pham <email address hidden>
Date: Wed Sep 17 17:14:15 2014 -0400

    VMware: Support volume hotplug

    When a volume is attached to an instance, the disk_type and
    adapter_type are found by making a get_dynamic_property vim
    call to the instance. If the adapter_type returned is
    "ide", the volume attach fails with "operation cannot be
    performed in the current state". This is because VMware IDE
    adapter does not support disk hotplug.

    Instead of making a get_dynamic_property vim call to the
    instance, it should be made to the volume (shadow VM). This
    properly indicates what adapter_type is needed by the volume.
    If the instance does not have a controller to support the
    volume adapter_type (e.g. lsiLogic), one will be added.
    If the volume adapter_type is "ide", a exception.Invalid will
    be thrown.

    Change-Id: Id826a33298885e9dbd4b52c9fe518ed2e6d50469
    Closes-Bug: #1226543