Ubuntu
eucalyptus package

[UEC 2.0+bzr1241-0ubuntu4.1] Unstable state for the iscsi daemon

Bug #778392 reported by Razique Mahroua
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
eucalyptus (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

Description: Ubuntu 10.10
Release: 10.10
UEC suite : 2.0+bzr1241-0ubuntu4.1

Description :
After doing tests of custom EMI's instancitation, the Node Controller went unstable. I was then unable to instanciate anymore instances. For the running instances with EBS attached, I was then unable to detach them, and for new instances, unable to attach a new EBS volume :

Detaching :

[Wed May 4 11:01:23 2011][020884][EUCAERROR ] ERROR: DetachVolume returned an error
[Wed May 4 11:01:23 2011] ERROR: doDetachVolume() returned FAIL

Attaching :
[Wed May 4 09:26:04 2011][013319][EUCAERROR ] libvirt: internal error '//,46.227.16.70,iqn.2009-06.com.eucalyptus.cluster1:store5,Itj/2Zbs1QNRlVzKROjF7k5nKBnck0NuFPlgm2hlf7F63sMKldO0W3vG4at6RE4PRe5gy6tKMPnkpZXAUQlymBOT+hAUmT4cBgvDYNdmAN98oLlUGXqkzZVI3SxBBfUeHwAi7v/tQHcZUg49ezVWpdXg4D55DUHT6BwsxLYmxXmbqRUH5RVeXw7r8Dhaplh1P5X5KWZHvdCrQDWbUF8zU37jxOsUANzzpTYUTfRl7VtHitvzQ2ebY06xRbABziBe6MY/p3r6qILyl8vDgot1MQ+OzWH3WizsDRu4g0uDWI3MVfKUh2UEz2mOA/IKSQF7Cfrv2zyIpUSGq+uDPhp/hg==' does not exist (code=1)

In order to be able to instanciate, I manually restarted the iscsi part :
$ service open-iscsi restart

which cleaned all the running sessions, but lead to :
[Wed May 4 09:34:32 2011][013319][EUCAERROR ] libvirt: operation failed: disk sde not found (code=9)
when I was trying to dettach disks.

Even by restarting all the services (NC, NC-publication, SC, SC-publication), UEC wouldn't clean the missing volumes. UEC kept in memory iscsi sessions and attachments (on the node controller (/dev/sdX and when euca2ools-describe-volumes invoked, the volumes were still present))

Expected behaviour :
- When all the components are restarted, the EBS sessions should be cleaned

What happened :
- Unable to get rid of "Attaching..." or "Dettaching..." status, due to missing iscsi volumes

Razique

Revision history for this message
Dave Walker (davewalker) wrote :

This issue needs to be reproduced in order to be confirmed.

Changed in eucalyptus (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in eucalyptus (Ubuntu):
status: New → Confirmed
Revision history for this message
Edmund Haselwanter (ehaselwanter) wrote :

Possible in a related state.

2012-06-18 08:59:13 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2012-06-18 08:59:13 TRACE nova.rpc.amqp Command: sudo nova-rootwrap iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000030 -p 192.168.124.87:3260 --op update -n node.startup -v manual
2012-06-18 08:59:13 TRACE nova.rpc.amqp Exit code: 255
2012-06-18 08:59:13 TRACE nova.rpc.amqp Stdout: ''
2012-06-18 08:59:13 TRACE nova.rpc.amqp Stderr: 'iscsiadm: no records found!\n'
2012-06-18 08:59:13 TRACE nova.rpc.amqp

root cause is a successful detach of the volume followed by another attempt after this successful detach (we try to create a reproduce able set of commands to get into this state)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.