Fail to unify allocation_capacity_gb values among multiple Active-Active Cinder-Volume services

Bug #1995204 reported by zhaoleilc
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Low
Unassigned
Victoria
New
Low
Unassigned

Bug Description

Description
===========
Three cinder-volume services was started in different hosts and
they all refer to the same rbd backend by setting same cinder backend
configuration. In order to achieve Active-Active configurations
the cluster configuration option is defined in the [DEFAULT] section
of cinder.conf for each cinder-volume service.

After these cinder-volume services was started, three records could be
seen in volume service list. However, just one record could be seen for
cinder get-pools conmmand.

When 10 volumes of 10GB of that backend was created successfully, three
different values of allocation_capacity_gb i.e. 30, 40 and 30 could be
inspected by executing cinder get-pools conmmand many times.

Therefore, each cinder-volume service has its own allocation_capacity_gb
and allocation_capacity_gb does not have Active-Active realization.

Steps to reproduce
==================
1. Spawn three cinder-volume services which all refer to the same rbd
backend and belong to one cluster whose name is clus1 e.g.

$ openstack volume service list
+------------------+-----------------------+-------------+---------+-------+----------------------------+
| Binary | Host | Zone | Status | State | Updated At |
+------------------+-----------------------+-------------+---------+-------+----------------------------+
| cinder-volume | host1@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:29:07.000000 |
| cinder-volume | host2@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:29:07.000000 |
| cinder-volume | host3@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:27:51.000000

$ cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail
+-----------------------------+---------------------------------------------------------+
| Property | Value |
+-----------------------------+---------------------------------------------------------+
| allocated_capacity_gb | 0 |
| backend_state | up |
| driver_version | 1.2.0 |
| filter_function | None |
| free_capacity_gb | 5912.85 |
| goodness_function | None |
| location_info | ceph:/etc/ceph/ceph_a4842e0c-3207-4.....:cinder.volumes |
| max_over_subscription_ratio | 20 |
| multiattach | True |
| name | clus1@rbd-clus-1#rbd-clust-1 |
| replication_enabled | False |
| reserved_percentage | 0 |
| storage_protocol | ceph |
| thin_provisioning_support | True |
| timestamp | 2022-10-30T11:35:29.157787 |
| total_capacity_gb | 6310.3 |
| vendor_name | Open Source |
| volume_backend_name | rbd-clust-1 |
+-----------------------------+---------------------------------------------------------+

2. Create 10 volumes of 10GB of that cinder backend e.g.
$ for i in {1..10};do openstack volume create --type rbd-clust-1 --size 10 clus-v$i;done
$ openstack volume list
+--------------------------------------+----------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+----------+-----------+------+-------------+
| 20ac2305-6017-4f96-b05e-126b1dbe6b75 | clus-v10 | available | 10 | |
| a902574e-8e57-4a4c-ac9c-daaa335464bc | clus-v9 | available | 10 | |
| 5a3e85f3-fe09-45e0-980e-be6ab3279b22 | clus-v8 | available | 10 | |
| 1c0d720f-d407-4089-8c4f-574c74874c86 | clus-v7 | available | 10 | |
| d7c08ba6-88cf-409b-9bcf-a3a5901bd67e | clus-v6 | available | 10 | |
| 5d01544e-1c21-45d0-964f-1532f894b8a8 | clus-v5 | available | 10 | |
| b510820d-79ba-4f64-b8b6-f443b9d8ae11 | clus-v4 | available | 10 | |
| 139c1922-cb2f-4ca5-a594-196067cf649e | clus-v3 | available | 10 | |
| 888e3fc7-d8d2-47e5-be40-ce910e27713e | clus-v2 | available | 10 | |
| 82b29c4c-805d-4491-9d03-70183359ff85 | clus-v1 | available | 10 | |
+--------------------------------------+----------+-----------+------+-------------+

3. Inspect the allocated_capacity_gb value by executing cinder get-pools conmmand many times.

Expected result
===============
The allocated_capacity_gb value returns 100 no matter how many times cinder get-pools conmmand is executed.

Actual result
=============
Different allocated_capacity_gb values are returned and in aggregate they are accurate.

[stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail
+-----------------------------+-----------------------+
| Property | Value |
+-----------------------------+-----------------------+
| allocated_capacity_gb | 30 |
...
[stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail
+-----------------------------+-----------------------+
| Property | Value |
+-----------------------------+-----------------------+
| allocated_capacity_gb | 40 |
...
[stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail
+-----------------------------+-----------------------+
| Property | Value |
+-----------------------------+-----------------------+
| allocated_capacity_gb | 30 |
...

Environment
===========
Victoria branch Of Openstack

zhaoleilc (zhaoleilc)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/863600

Changed in cinder:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/863769

Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

Hello,
Some backends like DELL EMC, Hitachi or IBM have a custom way of calculating pool stats. Do you mind sharing the backend you are using in the cluster?
Cheers,
Sofia

Changed in cinder:
importance: Undecided → Low
tags: added: capacity get-pools
tags: added: active-active
removed: get-pools
Revision history for this message
zhaoleilc (zhaoleilc) wrote (last edit ):

Hi,
All backends I used in the cluster are rbd backends and they
refer to the same ceph cluster.
In addition, Although various backends may have a custom way
of calculating pool stats, the allocation_capacity_gb value is
calculated by cinder-volume service and it will override the
allocation_capacity_gb value in backends.

Cheers,
Zhaoleilc

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.