Cinder

Failed to delete a never attached volume with lioadm helper

Bug #1300136 reported by Attila Fazekas
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Sergey Gotliv
Cinder 2014.2 "juno"
Icehouse
Fix Released
Medium
Attila Fazekas
Cinder 2014.1.1

Bug Description

/etc/cinder/cinder.conf:
[DEFAULT]
iscsi_helper = lioadm

As admin user, I am unable to delete a volume if it was not attached before.
the volume status became: error_deleting

$ cinder create 1
$ cinder delete daf0a29a-542b-4571-9d4d-5b9b59a7baf3
$ cinder list
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| daf0a29a-542b-4571-9d4d-5b9b59a7baf3 | error_deleting | None | 1 | None | false | |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+

../screen-logs/screen-c-vol.log:
2014-03-31 10:16:43.863 787 ERROR oslo.messaging.rpc.dispatcher [req-8f294457-08a3-4039-9626-ce63a2435bed 46a5bd04d46f49aabcf9edf8214df7a1 d587e4e2b14a41e4b38f62b07a130b3e - - -] Exception during message handling: No target id found for volume daf0a29a-542b-4571-9d4d-5b9b59a7baf3.
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/volume/manager.py", line 144, in lvo_inner1
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher return lvo_inner2(inst, context, volume_id, **kwargs)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/openstack/common/lockutils.py", line 233, in inner
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher retval = f(*args, **kwargs)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/volume/manager.py", line 143, in lvo_inner2
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher return f(*_args, **_kwargs)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/volume/manager.py", line 416, in delete_volume
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher {'status': 'error_deleting'})
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/openstack/common/excutils.py", line 68, in __exit__
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/volume/manager.py", line 400, in delete_volume
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher self.driver.remove_export(context, volume_ref)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/volume/drivers/lvm.py", line 540, in remove_export
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher self.target_helper.remove_export(context, volume)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/volume/iscsi.py", line 232, in remove_export
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher volume['id'])
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/db/api.py", line 234, in volume_get_iscsi_target_num
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher return IMPL.volume_get_iscsi_target_num(context, volume_id)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/db/sqlalchemy/api.py", line 119, in wrapper
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/db/sqlalchemy/api.py", line 1344, in volume_get_iscsi_target_num
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher raise exception.ISCSITargetNotFoundForVolume(volume_id=volume_id)
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher ISCSITargetNotFoundForVolume: No target id found for volume daf0a29a-542b-4571-9d4d-5b9b59a7baf3.
2014-03-31 10:16:43.863 787 TRACE oslo.messaging.rpc.dispatcher

When the lioadm is the iscsi helper the target just exists when the volume attached.
I should be able to delete a volume what was never attached.

The volumes which was attached to a vm before the delete request are deletable.

See original description
Tags: lio
Attila Fazekas (afazekas)
description: updated
Revision history for this message
Eric Harney (eharney) wrote :

This may have the same root cause as bug 1300148. Was this with or without this fix in your tree?

04c0579 Merge "driver.create/remove_export() require elevated context"
d09d12a driver.create/remove_export() require elevated context

Revision history for this message
Attila Fazekas (afazekas) wrote :

It was "without", since bug report is older than the referenced changes.
The issue is reproduce-able with the latest cinder version (fed5843972ced19d0c04d99a3b100a6108152063). "with"

I have used this version of devstack https://review.openstack.org/#/c/77630/, with CINDER_ISCSI_HELPER=lioadm in the local.conf (localrc).

It is not permission issue, the other bug is related to permissions.
I mentioned the fact I did it as admin user, because as demo user I was not able to attache the volume (the other bug), but if the volume ever gets attached to a server the volume becomes delete-able.

Not working sequence:
# cinder create 1
# cinder delete <volid>

The volume deletion failed.

Working sequence:
# cinder create 1
# nova boot server --image cirros-0.3.1-x86_64-uec --flavor 42
# nova volume-attach <server_id> <volume_id> /dev/vdc
# nova delete <server_id>
# cinder delete <vol_id>

Volume deletion succeeded, at least not listed by the cinder list.

The simple create/delete sequence is the not working one.

Revision history for this message
Eric Harney (eharney) wrote :

The problem here is that when using LIO, the iscsi_targets table does not get an entry until an attach is attempted (which is when the LIO target is created for that volume).

The delete path expects there to always be a target id in that table tied to that volume, but that is not guaranteed to the be the case.

LioAdm's remove_export tries to handle this via "except exception.NotFound:" but that is referencing cinder.brick.exception.NotFound rather than cinder.exception.NotFound so the error is not skipped.

Sergey Gotliv (sgotliv)
Changed in cinder:
assignee: nobody → Sergey Gotliv (sgotliv)
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/92472

Changed in cinder:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/92472
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=115b8447618073d3383bbd612ebd2fc68c8c5d28
Submitter: Jenkins
Branch: master

commit 115b8447618073d3383bbd612ebd2fc68c8c5d28
Author: Sergey Gotliv <email address hidden>
Date: Wed May 7 00:27:46 2014 +0300

    Fix wrong exception reference

    iscsi.py should catch cinder.exception.NotFound instead of
    cinder.brick.exception.NotFound.

    Change-Id: I77134683712e9706619a7f7fe82caed42ede5887
    Closes-Bug: #1300136

Changed in cinder:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/93029

Mike Perez (thingee)
tags: added: icehouse-backport-potential lio
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/icehouse)

Reviewed: https://review.openstack.org/93029
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=bc0442a8edf4fd059f0a94dd27c20597138fc3c2
Submitter: Jenkins
Branch: stable/icehouse

commit bc0442a8edf4fd059f0a94dd27c20597138fc3c2
Author: Sergey Gotliv <email address hidden>
Date: Wed May 7 00:27:46 2014 +0300

    Fix wrong exception reference

    iscsi.py should catch cinder.exception.NotFound instead of
    cinder.brick.exception.NotFound.

    Change-Id: I77134683712e9706619a7f7fe82caed42ede5887
    Closes-Bug: #1300136
    (cherry picked from commit 115b8447618073d3383bbd612ebd2fc68c8c5d28)

tags: added: in-stable-icehouse
Alan Pevec (apevec)
Changed in cinder:
importance: Undecided → Medium
tags: removed: icehouse-backport-potential in-stable-icehouse
Thierry Carrez (ttx)
Changed in cinder:
milestone: none → juno-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.