Cinder NFS backend with image-volume cache creates volumes with wrong size

Bug #1990053 reported by Danila Balagansky
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Low
Pete Zaitcev

Bug Description

Cinder version (commit SHA): 83089b92467d8deac5793f4f4d739c936a758ade

backend configuration in cinder.conf:
[vm]
image_volume_cache_enabled = True
image_volume_cache_max_count = 40
image_volume_cache_max_size_gb = 30
nas_secure_file_operations = False
nas_secure_file_permissions = False
nfs_mount_options = rsize=65535,wsize=65535,timeo=1200,actimeo=120
nfs_qcow2_volumes = True
nfs_shares_config = /etc/cinder/vm_nfs_shares
nfs_snapshot_support = True
report_discard_supported = True
volume_backend_name = vm
volume_driver = cinder.volume.drivers.nfs.NfsDriver

Debian 11 image: https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-genericcloud-amd64.qcow2

Image in Glance:
+------------------+--------------------------------------+
| Field | Value |
+------------------+--------------------------------------+
| checksum | 06e481ddd23682af4326226661c13d8f |
| container_format | bare |
| disk_format | qcow2 |
| id | 7673db9f-6c20-4b21-b817-2932019d5f2b |
| name | debian11 |
| size | 254672896 |
| virtual_size | 2147483648 |
+------------------+--------------------------------------+

Creating volume without image-volume cache, like this:
root@oscp99:~# openstack volume create --image debian11 --size 11 test
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | office |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2022-09-18T07:25:54.029899 |
| description | None |
| encrypted | False |
| id | 4eb3d4e8-043d-4a2f-9fb8-98f4c91d01c0 |
| migration_status | None |
| multiattach | False |
| name | test |
| properties | |
| replication_status | None |
| size | 11 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | vm |
| updated_at | None |
| user_id | b14242e3bdef45e797a7973bc75d680d |
+---------------------+--------------------------------------+

Produces the log `cinder-volume_create_volume_without_cache.txt` with stacktrace, but in the end, volume does get created successfully:
+--------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [] |
| availability_zone | office |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2022-09-18T07:25:54.000000 |
| description | None |
| encrypted | False |
| id | 4eb3d4e8-043d-4a2f-9fb8-98f4c91d01c0 |
| migration_status | None |
| multiattach | False |
| name | test |
| os-vol-host-attr:host | oscn99.local@vm#vm |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 8e0353b219594c819af69294e15f4e23 |
| properties | |
| replication_status | None |
| size | 11 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| type | vm |
| updated_at | 2022-09-18T07:28:53.000000 |
| user_id | b14242e3bdef45e797a7973bc75d680d |
| volume_image_metadata | {'signature_verified': 'False', 'os_glance_importing_to_stores': '', 'os_glance_failed_import': '', 'image_id': '7673db9f-6c20-4b21-b817-2932019d5f2b', 'image_name': 'debian11', 'checksum': '06e481ddd23682af4326226661c13d8f', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': '0', 'min_ram': '0', 'size': '254672896'} |
+--------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

But the size on the NFS share is wrong:

qemu-img info /vm/cinder/volumes/volume-4eb3d4e8-043d-4a2f-9fb8-98f4c91d01c0
image: /vm/cinder/volumes/volume-4eb3d4e8-043d-4a2f-9fb8-98f4c91d01c0
file format: qcow2
virtual size: 2 GiB (2147483648 bytes)
disk size: 700 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Image-volume cache volume:
+--------------------------------+--------------------------------------------+
| Field | Value |
+--------------------------------+--------------------------------------------+
| attachments | [] |
| availability_zone | office |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2022-09-18T07:25:54.000000 |
| description | None |
| encrypted | False |
| id | 4a85b808-2ccf-442e-875e-1c71e65f21e1 |
| migration_status | None |
| multiattach | False |
| name | image-7673db9f-6c20-4b21-b817-2932019d5f2b |
| os-vol-host-attr:host | oscn99.local@vm#vm |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 09d81658b99d464aa3b4d747389e38e1 |
| properties | readonly='True' |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | 4eb3d4e8-043d-4a2f-9fb8-98f4c91d01c0 |
| status | available |
| type | vm |
| updated_at | 2022-09-18T07:27:58.000000 |
| user_id | b14242e3bdef45e797a7973bc75d680d |
+--------------------------------+--------------------------------------------+

Creating volume for the second time (with image-volume cache populated):

openstack volume create --image debian11 --size 11 tttt

produces `cinder-volume_create_volume_with_cache.txt` log, where `qemu-img create` is called with the original volume size (2G in this example):

qemu-img create -f qcow2 -o backing_file=/var/lib/cinder/mnt/0825b3db343399a47ca1773445a9278f/volume-4a85b808-2ccf-442e-875e-1c71e65f21e1,backing_fmt=qcow2 /var/lib/cinder/mnt/0825b3db343399a47ca1773445a9278f/volume-4a85b808-2ccf-442e-875e-1c71e65f21e1.337354d2-07eb-4b92-9ab7-6f300d8a7440 2G

and after that, volume never gets resized to the requested size.

Created volume has expected size in DB:
+--------------------------------------+------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+------+-----------+------+-------------+
| 21be2d3a-1788-4234-934c-de4a0e2e12f4 | tttt | available | 11 | |
+--------------------------------------+------+-----------+------+-------------+

But not on the NFS share:
root@oscn99:~# qemu-img info /vm/cinder/volumes/volume-21be2d3a-1788-4234-934c-de4a0e2e12f4
image: /vm/cinder/volumes/volume-21be2d3a-1788-4234-934c-de4a0e2e12f4
file format: qcow2
virtual size: 2 GiB (2147483648 bytes)
disk size: 700 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Supposed fix: https://review.opendev.org/c/openstack/cinder/+/855964
Related bug: https://bugs.launchpad.net/cinder/+bug/1988803 (without information about used backend), but it's about creating volumes with size, less that image-volume cache volume.

When using LVM backend, logical volume is created with expected size.

Revision history for this message
Danila Balagansky (dbalagansky) wrote :
Revision history for this message
Danila Balagansky (dbalagansky) wrote :
description: updated
Changed in cinder:
importance: Undecided → Low
tags: added: cache drivers generic-nfs glance image nfs
Pete Zaitcev (zaitcev)
Changed in cinder:
assignee: nobody → Pete Zaitcev (zaitcev)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.