Simultaneous volume creation with the same image in multi-attach mode returns error.

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/805072

As far as I can see, this is more a feature than a bug. We don't have anything in the documentation [1] saying that this is not possible, we probably need to add this to the known issues section.

[1] https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiattach.html

I'm adding this to next week cinder meeting.

Hi,

Is the glance backend cinder in this case?
and which openstack release are you using?

Hi,

yes, glance backend is cinder.
Openstack version - Victoria.

We just had a discussion in #openstack-cinder [1] about a problem that I believe shares the same root cause as this bug report. The logs I'd like to share were too big for pastebin, so I'm attaching them here.

[1] https://meetings.opendev.org/irclogs/%23openstack-cinder/%23openstack-cinder.2022-04-07.log.html#t2022-04-07T08:13:58

Hi Bence,

NOTE: This is more of a glance issue than cinder as the changes made are part of glance cinder store.

I did the same image save operation 10 times concurrently and was able to save all 10 images successfully.

stack@devstack-VirtualBox:~/devstack$ glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file one.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file two.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file three.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file four.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file five.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file six.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file seven.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file eight.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file nine.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file ten.qcow2;

stack@devstack-VirtualBox:~/devstack$ ll *.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:20 eight.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:19 five.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:19 four.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:21 nine.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:17 one.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:20 seven.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:19 six.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:21 ten.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:18 three.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:18 two.qcow2

Glance image:

stack@devstack-VirtualBox:~/devstack$ glance image-show 18dc5568-1a47-44ff-a724-5ef8d44d9e08
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | b874c39491a2377b8490f5f1e89761a4 |
| container_format | bare |
| created_at | 2022-04-18T08:14:24Z |
| direct_url | cinder://lvmdriver-1/8851784d-5bef-4379-8c37-dd178fff8e37 |
| disk_format | qcow2 |
| id | 18dc5568-1a47-44ff-a724-5ef8d44d9e08 |
| locations | [{"url": "cinder://lvmdriver-1/8851784d-5bef-4379-8c37-dd178fff8e37", |
| | "metadata": {"store": "lvmdriver-1"}}] |
| min_disk | 0 |
| min_ram | 0 |
| name | test-2-multiattach-lvm ...

Sorry about my last comment, i was performing the operation serially.

Performing the operation in parallel, I was able to reproduce the issue and the root cause is that the retry interval is very small (1 second). I've reported a similar bug in glance store launchpad page[1] and proposed a fix[2]. Thanks for reporting the issue.

[1] https://bugs.launchpad.net/glance-store/+bug/1969373
[2] https://review.opendev.org/c/openstack/glance_store/+/838326

From cinder's perspective this is not a bug in our API.
We only accept a single attach/detach operation at any given time for a given volume. Callers supporting multi-attach need to be aware and add the appropriate retry mechanism.

There won't be any code changes in cinder for this bug, but leaving the bug open to add the appropriate documentation explicitly stating this.

Change abandoned by "Gorka Eguileor <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/805072
Reason: Thank you for working on this and for helping us to robustify the feature, but from cinder's perspective this is not a bug in our API.

We only accept a single attach/detach operation at any given time for a given volume. Callers supporting multi-attach need to be aware and add the appropriate retry mechanism.

Patch on the glance_store side will fix this issue: https://review.opendev.org/c/openstack/glance_store/+/838326