OpenStack Compute (nova)

volume_attachement entries are not getting deleted from DB

Bug #1998083 reported by Hemant Sonawane
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Incomplete
Medium
Unassigned

Bug Description

Hello
I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually to make it work.

For e.g I shelved some instance via openstack horizon to reproduce the same scenario.

Here are the detailed attachment entries you can see for the shelved instance.

+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·

| id | volume_id | attached_host | instance_uuid | attach_status | connector │·

                                                                                                                                                                                            | │·

+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·

| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | nfv1compute43 | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | attached | {"platform": "x86_64", "os_type": "linux│·

", "ip": "10.42.168.87", "host": "nfv1compute43", "multipath": false, "do_local_attach": false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint": "/dev/vda"} | │·

| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved | NULL │·

                                                                                                                                                                                            | │·

+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·

2 rows in set (0.00 sec)

If I would like to unshelve this instance it wont work as it has a duplicate entry in cinder db for the attachment. So i have to delete it manually from db or via cli.

root@master01:/home/hemant# cinder --os-volume-api-version 3.27 attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a │·

| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | attached | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | │·

| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | reserved | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |

cinder --os-volume-api-version 3.27 attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f

I could only gather logs from nova-compute

nova-compute logs:

cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume

This is the only choice I have if I would like to unshelve vm. But this is not a good approach for production env to delete duplicate volume attachments entries every time for every vm.

Is there any way to fix this issue ?

See original description
Tags: horizon
Hemant Sonawane (hemson95)
description: updated
Sofia Enriquez (lsofia-enriquez)
tags: added: horizon
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

Hi Hermant, hope this message finds you well.

From the output of the attachment it looks like the volume is attached to two different server IDs. Is the volume multi-attach to several instances?

Would you mind sharing the c-vol logs to see if there's any error on the cinder side?

Thanks
Regards,
Sofia

Revision history for this message
Hemant Sonawane (hemson95) wrote (last edit ):

Hello Sofia,

Thank you again. The volume is not multi-attach it is just attached to one server. We have no idea how its creating another entry for the volume-attachment after the shelving server and keep the volume in attached state. Interesting part we do not see anything in cinder logs. The logs I shared in bug report are from nova. Do you think it could be related to nova ?

I have also found something similar but the bug it seems in nova so not really sure

https://bugzilla.redhat.com/show_bug.cgi?id=1941054#c4

Thanks and Regards,
Hemant

Revision history for this message
Hemant Sonawane (hemson95) wrote :
Download full text (12.3 KiB)

Hello Sofia,

Here are some more detailed logs from nova-compute

2022-11-25 13:16:28.119 17272 ERROR nova.volume.cinder [req-aee88125-574b-4efd-85d2-cf8a6baceaf6 d23ef0fae9914621835c5f8bb2bca36e 00d3d6fa37cc4612b786e8d0253553bb - default default] Update attachment failed for attachment 683daabf-78f2-4f4c-9048-fb04ee8e0ca9. Error: Una
ble to update attachment.(Invalid volume: duplicate connectors detected on volume b942a2d5-1f93-410b-a7d2-4c975eff5fc3). (HTTP 500) (Request-ID: req-2d181126-10f9-41c1-92bb-777455cc092e) Code: 500: cinderclient.exceptions.ClientException: Unable to update attachment.(In
valid volume: duplicate connectors detected on volume b942a2d5-1f93-410b-a7d2-4c975eff5fc3). (HTTP 500) (Request-ID: req-2d181126-10f9-41c1-92bb-777455cc092e)
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [req-aee88125-574b-4efd-85d2-cf8a6baceaf6 d23ef0fae9914621835c5f8bb2bca36e 00d3d6fa37cc4612b786e8d0253553bb - default default] [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] Instance failed block device setup: c
inderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume b942a2d5-1f93-410b-a7d2-4c975eff5fc3). (HTTP 500) (Request-ID: req-2d181126-10f9-41c1-92bb-777455cc092e)
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] Traceback (most recent call last):
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 1976, in _prep_block_device
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] driver_block_device.attach_block_devices(
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 874, in attach_block_devices
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] _log_and_attach(device)
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 871, in _log_and_attach
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] bdm.attach(*attach_args, **attach_kwargs) ...

Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

I wasn't able to reproduce this problem on master branch + LVM driver. The entries on the volume_attachment table are updating correctly. Looks like the problem is fixed in master.

Steps I've tried:
1. Create a cirros server
2. Attach a LVM volume
3. Shelve the server
4. Unshelve the server

Let me know if this assumption is not correct but I think this involves Nova scope and not Cinder. Because of that I'm moving this bug to Nova for future assistance.

Thanks,
Cheers,
Sofia

affects: cinder → nova
no longer affects: cinder/wallaby
Revision history for this message
Uggla (rene-ribaud) wrote :

Hello Hement,

Can you please check if the instance is shelve or shelve_offloaded.
Note: There is the shelved_offload_time and shelved_poll_interval parameter (https://docs.openstack.org/nova/latest/configuration/config.html) that define the time an instance is kept shelve then moved to to shelve_offloaded.

Can you confirm there is a device in reserved state created as soon as you shelve the instance ?

Changed in nova:
status: New → Incomplete
Revision history for this message
Hemant Sonawane (hemson95) wrote (last edit ):

Hello Uggla,

I can confirm that instance is shelve_offloaded. But once I shelve the instance I see 2 entries in db one with reserved status and one with attached status. Take a look at following db entries of vm volume attachments.

1) db entry I delete every time of shelve_offloaded vm:

2023-03-09 11:48:28 2023-03-09 11:48:31 NULL 0 682f21cf-8dd0-4782-9c81-01dc099f32db 80ae4cb8-e93d-49fa-bfc3-6b5e5c282a54 xxxxxxx 2aca9b66-a3aa-4460-a17d-0a31c6c10656 /dev/vda 2023-03-09 11:48:30 NULL rw attached "{""name"": ""volumes/80ae4cb8-e93d-49fa-bfc3-6b5e5c282a54"", ""hosts"": [""xxxxx"", ""xxxxxx"", ""xxxx""], ""ports"": [""6789"", ""6789"", ""6789""], ""cluster_name"": ""ceph"", ""auth_enabled"": true, ""auth_username"": ""xxxxx"", ""secret_type"": ""xxx"", ""secret_uuid"": ""xxxxx"", ""volume_id"": ""xxxxxx"", ""discard"": true, ""qos_specs"": null, ""access_mode"": ""rw"", ""encrypted"": false, ""cacheable"": false, ""driver_volume_type"": ""rbd"", ""attachment_id"": ""682f21cf-8dd0-4782-9c81-01dc099f32db""}" "{""platform"": ""x86_64"", ""os_type"": ""linux"", ""ip"": ""10.44.168.88"", ""host"": ""xxxxxx"", ""multipath"": false, ""do_local_attach"": false, ""system uuid"": ""xxxxx"", ""mountpoint"": ""/dev/vda""}"

2) db entry which being used after unshelving vm:

2023-03-09 11:50:31 2023-03-09 11:50:31 NULL 0 9b998cd9-accb-4963-a79f-627552951a4e 80ae4cb8-e93d-49fa-bfc3-6b5e5c282a54 NULL 2aca9b66-a3aa-4460-a17d-0a31c6c10656 NULL NULL NULL null reserved {} NULL

cli :
root# cinder --os-volume-api-version 3.27 attachment-list --all | grep 2aca9b66-a3aa-4460-a17d-0a31c6c10656

| 682f21cf-8dd0-4782-9c81-01dc099f32db | 80ae4cb8-e93d-49fa-bfc3-6b5e5c282a54 | attached | 2aca9b66-a3aa-4460-a17d-0a31c6c10656 |
| 9b998cd9-accb-4963-a79f-627552951a4e | 80ae4cb8-e93d-49fa-bfc3-6b5e5c282a54 | reserved | 2aca9b66-a3aa-4460-a17d-0a31c6c10656 |

In this case I have to delete the entry from db or via cli which has "attach_status = attached" and then unshelving the vm works fine which uses the 2nd entry and attach volume device to instance. This happens everytime when we do shelve, unshelve, delete and resize operations.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.