Attaching a volume to a "pending" instance claims success, but never attaches.

Bug #676819 reported by Kiall Mac Innes
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Eucalyptus
Fix Committed
Medium
Dmitrii Zagorodnov
eucalyptus (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

Attaching a volume to a "pending" instance claims success (via euca2ools, and the API) but never attaches to the instance. This may be a KVM only issue.

As its pretty easy to reproduce, I haven't attached much in the way of detail - let me know if you need any specific info.

Command Line usage, assuming i-307905AD is in the "pending" state:

$ euca-attach-volume -d /dev/vda -i i-307905AD vol-59820625
VOLUME vol-59820625

Also, this possibly only occurs very early (first 5 seconds) of the pending state. The same command succeeds if you wait for the instance to arrive at the "running" status.

Eucalyptus: Version: 2.0+bzr1241-0ubuntu4
Euca2ools: Version: 1.2-0ubuntu11

Tags: patch
Kiall Mac Innes (kiall)
description: updated
description: updated
Kiall Mac Innes (kiall)
description: updated
Changed in eucalyptus:
assignee: nobody → Dmitrii Zagorodnov (dmitrii)
Chuck Short (zulcss)
Changed in eucalyptus (Ubuntu):
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Luca Invernizzi (invernizzi) wrote :

This seems to be due to a incompatibility between eucalyptus-nc and libvirt

When attaching a disk (/dev/sdX or /dev/vdX), this is the output on the node controller:

[Thu Nov 24 23:33:56 2011][000571][EUCAINFO ] doAttachVolume() invoked (id=i-416607C2 vol=vol-5A24063B remote=//,192.35.222.25,iqn.2009-06.com.eucalyptus.secluster:store3,djsFVOsygc3D8GA6SAYtxMQRPLFlzdkXXPakJKalNo3lE8i9GKasvANekX4fETg3RSAv7zQgoRyQFOjc/vfnb8tqjdb9jJsInIzHuYHcDkihfXrYiSGmGxXzh2ijjpPb1BYAQZI2OwSM2Pux+0lwNJwGxl7zW2pVeSpFT9pulZahLH7eX/RcXaHWl/glKqMtxz/L3g06OlM/bneoFofCGZiPxBklDTiC2uz/HN3Ew+JF5u/PePv5qAFVlv61jJk4f80G7ZMM7+LZfjSmi4TVm/xAlRLzqiQbkmvabpXVY9utCBQS2qm46bFeZMFXaxf14iIIsDX3K0XD9SkOUAA5KQ== local=/dev/vda)
[Thu Nov 24 23:33:56 2011][000571][EUCAINFO ] connect_iscsi_target invoked (dev_string=//,192.35.222.25,iqn.2009-06.com.eucalyptus.secluster:store3,djsFVOsygc3D8GA6SAYtxMQRPLFlzdkXXPakJKalNo3lE8i9GKasvANekX4fETg3RSAv7zQgoRyQFOjc/vfnb8tqjdb9jJsInIzHuYHcDkihfXrYiSGmGxXzh2ijjpPb1BYAQZI2OwSM2Pux+0lwNJwGxl7zW2pVeSpFT9pulZahLH7eX/RcXaHWl/glKqMtxz/L3g06OlM/bneoFofCGZiPxBklDTiC2uz/HN3Ew+JF5u/PePv5qAFVlv61jJk4f80G7ZMM7+LZfjSmi4TVm/xAlRLzqiQbkmvabpXVY9utCBQS2qm46bFeZMFXaxf14iIIsDX3K0XD9SkOUAA5KQ==)
[Thu Nov 24 23:33:56 2011][000571][EUCADEBUG ] system_output(): [//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/connect_iscsitarget.pl //,192.35.222.25,iqn.2009-06.com.eucalyptus.secluster:store3,djsFVOsygc3D8GA6SAYtxMQRPLFlzdkXXPakJKalNo3lE8i9GKasvANekX4fETg3RSAv7zQgoRyQFOjc/vfnb8tqjdb9jJsInIzHuYHcDkihfXrYiSGmGxXzh2ijjpPb1BYAQZI2OwSM2Pux+0lwNJwGxl7zW2pVeSpFT9pulZahLH7eX/RcXaHWl/glKqMtxz/L3g06OlM/bneoFofCGZiPxBklDTiC2uz/HN3Ew+JF5u/PePv5qAFVlv61jJk4f80G7ZMM7+LZfjSmi4TVm/xAlRLzqiQbkmvabpXVY9utCBQS2qm46bFeZMFXaxf14iIIsDX3K0XD9SkOUAA5KQ==]
[Thu Nov 24 23:33:57 2011][000571][EUCAINFO ] Attached device: /dev/sdc
[Thu Nov 24 23:33:57 2011][000571][EUCAERROR ] libvirt: internal error unsupported driver name 'phy' for disk '/dev/sdc' (code=1)
[Thu Nov 24 23:33:57 2011][000571][EUCAERROR ] virDomainAttachDevice() failed (err=-1) XML=<disk type='block'><driver name='phy'/><source dev='/dev/sdc'/><target dev='vda'/></disk>
[Thu Nov 24 23:33:57 2011][000571][EUCAERROR ] ERROR: doAttachVolume() failed error=1

Please note the line: libvirt: internal error unsupported driver name 'phy' for disk '/dev/sdc' .
The driver name should be 'quemu'. Libvirt used to tolerate 'phy', but it doesn't anymore (as in https://bugzilla.redhat.com/show_bug.cgi?id=627143#c13).
I haven't had time to try it yet, but it appears that substituting "phy" with "quemu" in http://bazaar.launchpad.net/~eucalyptus-maintainers/eucalyptus/2.0.0-lp/view/head:/node/handlers_kvm.c would solve the problem.

Revision history for this message
Luca Invernizzi (invernizzi) wrote :

Obviously, I meant "qemu".

Revision history for this message
Luca Invernizzi (invernizzi) wrote :

As I suspected, the incorrect disk driver for kvm was the issue.
I've attached a patch in the debian patch format, that lets me successfully attach disks.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "31-handlers_kvm.patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch
Revision history for this message
Dmitrii Zagorodnov (dmitrii) wrote :

Luca - thanks for the patch! Looks like your solution is the right one for KVM. I'd like to ensure that volume attachments continue to work on Xen (driver name 'phy' is Xen-specific, which used to be ignored by QEMU until recently). So a bit of extra logic may be necessary to differentiate XML generated for Xen and KVM. I'll test and will get back to you.

Changed in eucalyptus:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Dmitrii Zagorodnov (dmitrii) wrote :

Ah, my bad, since it's a patch to handlers_kvm.c, Xen code path is not affected. I've confirmed that 'qemu' setting works with older Ubuntu releases, too. So, we'll be happy to take this as is.

Luca - are you OK with signing off the ownership of the patch to us?

Revision history for this message
Luca Invernizzi (invernizzi) wrote :

Sure.
Thanks for your help

Revision history for this message
graziano obertelli (graziano.obertelli) wrote :

Thanks Luca: assigning it to milestone 2.0.4.

Changed in eucalyptus:
milestone: none → 2.0.4
Changed in eucalyptus:
status: Confirmed → Fix Committed
Revision history for this message
Andy Grimm (agrimm) wrote :

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2742

Please watch that issue for further updates.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.