| 2021-05-26 10:18:48 |
Felix Huettner |
description |
Hi everyone,
the attachment_create api has the possibility to create volume attachments with and without a already specified connector. If no connector is specified the api will create a volume attachment db entry with the status "reserved" and put the volume to the "reserved" state. If a connector is specified then first the api creates the "reserved" information as above but then calls cinder-volume to actually combine the attachment with the connector leaving a "in-use" volume afterwards.
If we now assume that cinder-volume failes during the above step we will have a "reserved" volume attachment in the database and the volume will be in "reserved" state. This looks as follows:
* the volume is in "reserved"
* a volume-attachment in "reserved" state exists
This does not prevent the caller from trying again to attach the volume to the same instance as the "_attachment_reserve" method just creates an additional volume attachment record. If we assume cinder-volume has no issues this time (because it recovered in any way from the previous error) we now have the following state:
* the volume is in "in-use"
* a volume-attachment with the connector information exists and is "in-use"
* the old volume-attachment in "reserved" from the first step still exists
If the user now detaches the volume again cinder will clean up the currently active attachment, but the old "reserved" attachment still exists. Cinder will therfor put the volume in "reserved" state as well. This then looks as follows:
* the volume is in "reserved"
* the volume-attachment with the connector information is marked as deleted
* the old volume-attachment in "reserved" from the first step still exists
At this point the user requires admin intervention to attach the volume again to another instance. Deleting the volume is however possible.
My proposed solution would be to delete the attachment in the first step again if the call to "attachment_update" fails. |
Hi everyone,
the attachment_create api has the possibility to create volume attachments with and without a already specified connector. If no connector is specified the api will create a volume attachment db entry with the status "reserved" and put the volume to the "reserved" state. If a connector is specified then first the api creates the "reserved" information as above but then calls cinder-volume to actually combine the attachment with the connector leaving a "in-use" volume afterwards.
If we now assume that cinder-volume failes during the above step we will have a "reserved" volume attachment in the database and the volume will be in "reserved" state. This looks as follows:
* the volume is in "reserved"
* a volume-attachment in "reserved" state exists
This does not prevent the caller from trying again to attach the volume to the same instance as the "_attachment_reserve" method just creates an additional volume attachment record. If we assume cinder-volume has no issues this time (because it recovered in any way from the previous error) we now have the following state:
* the volume is in "in-use"
* a volume-attachment with the connector information exists and is "in-use"
* the old volume-attachment in "reserved" from the first step still exists
If the user now detaches the volume again cinder will clean up the currently active attachment, but the old "reserved" attachment still exists. Cinder will therfor put the volume in "reserved" state as well. This then looks as follows:
* the volume is in "reserved"
* the volume-attachment with the connector information is marked as deleted
* the old volume-attachment in "reserved" from the first step still exists
At this point the user requires admin intervention to attach the volume again to another instance. Deleting the volume is however possible.
The above can especially happen when there is a issue with cinder-volume during the _pre_live_migration of nova. Then nova will create the above reserved attachments that will not be cleaned up later.
My proposed solution would be to delete the attachment in the first step again if the call to "attachment_update" fails. |
|
