OpenStack Compute (nova)

Nova not detaching Cinder volume is volume is available

Bug #1988751 reported by Gorka Eguileor on 2022-09-05

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Confirmed	Undecided	Unassigned

Bug Description

It is known that Cinder and Nova can sometime be out of sync, where one of them thinks that a volume is attached and the other doesn't.

With the Cinder attachment API an operator can now fix the issue where Nova says the volume is detached and Cinder says it's not. The operator just needs to run the "cinder attachment-delete" command on the specific attachment.

The opposite situation, Cinder says it's available and Nova says it's attached, cannot be currently fixed without modifying the Nova database and making manual os-brick calls or the appropriate CLI calls to detach the volume.

Ideally Nova should be able to call os-brick using the BDM information to locally detach the volume (using the force option which can lose data) and then not call cinder to do the detach since it already says the volume is not mapped.

One way to reproduce this is:

- Create a VM
- Create a volume
- Attach the volume
- Delete the attachment in cinder with "cinder attachment-delete <attach_id>"
- Trying to detach the volume in nova

The error we'll see is something like:

ERROR (BadRequest): Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-ec02147a-6b5b-40d2-991c-3d49207f5c9b) (HTTP 400) (Request-ID: req-d8ab82c5-cb32-446e-a8e9-fd8e30be0995)

Revision history for this message

Uggla (rene-ribaud) wrote on 2022-09-13:

Download full text (5.6 KiB)

stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 attachment-list
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| ID | Volume ID | Status | Server ID |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| 66824067-5b10-49fc-b3fa-84aa56ff0d55 | e6ed1a54-a57e-438d-8e0c-0289367a2a97 | attached | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
| 7001e8eb-b12d-4e5c-bfa5-2aeda7fbbb9b | 87b8bfd0-1470-4ebf-a13c-50df99ffed80 | attached | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+

stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 attachment-delete 66824067-5b10-49fc-b3fa-84aa56ff0d55
stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 attachment-list
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| ID | Volume ID | Status | Server ID |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| 7001e8eb-b12d-4e5c-bfa5-2aeda7fbbb9b | 87b8bfd0-1470-4ebf-a13c-50df99ffed80 | attached | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+

stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 attachment-list
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| ID                                   | Volume ID                            | Status   | Server ID                            |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| 66824067-5b10-49fc-b3fa-84aa56ff0d55 | e6ed1a54-a57e-438d-8e0c-0289367a2a97 | attached | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
| 7001e8eb-b12d-4e5c-bfa5-2aeda7fbbb9b | 87b8bfd0-1470-4ebf-a13c-50df99ffed80 | attached | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+

stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 attachment-delete 66824067-5b10-49fc-b3fa-84aa56ff0d55
stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 attachment-list
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| ID                                   | Volume ID                            | Status   | Server ID                            |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| 7001e8eb-b12d-4e5c-bfa5-2aeda7fbbb9b | 87b8bfd0-1470-4ebf-a13c-50df99ffed80 | attached | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+

stack@openstack:~/devstack$ nova list
nova CLI is deprecated and will be a removed in a future release
+--------------------------------------+------+--------+------------+-------------+-----------------------------------------------------------------------+
| ID                                   | Name | Status | Task State | Power State | Networks                                                              |
+--------------------------------------+------+--------+------------+-------------+-----------------------------------------------------------------------+
| d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 | demo | ACTIVE | -          | Running     | private=10.0.0.33, fdf9:5223:5eb6:0:f816:3eff:fe7e:c053, 172.24.4.181 |
+--------------------------------------+------+--------+------------+-------------+-----------------------------------------------------------------------+

stack@openstack:~/devstack$ cinder list
+--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+
| ID                                   | Status    | Name        | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+
| 87b8bfd0-1470-4ebf-a13c-50df99ffed80 | in-use    |             | 10   | lvmdriver-1 | true     | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
| e6ed1a54-a57e-438d-8e0c-0289367a2a97 | available | demo-volume | 1    | lvmdriver-1 | false    |                                      |
+--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+
stack@openstack:~/devstack$ nova volume-detach d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 e6ed1a54-a57e-438d-8e0c-0289367a2a97
nova CLI is deprecated and will be a removed in a future release
ERROR (BadRequest): Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-bd2c8b9e-b7e9-4a93-adef-28c866282be0) (HTTP 400) (Request-ID: req-a48405c0-5612-475b-b40d-cc68551846a1)

Changed in nova:
status:	New → Confirmed

Revision history for this message

Uggla (rene-ribaud) wrote on 2022-09-13:

Above lof done 20220913 using Zed.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.