Comment 6 for bug 1789828

Ok I think I have a more accurate picture of what is going on here. For context I should say that we have observed this issue in an environment where a very large number (several thousand) of volumes where deleted at once and this manifested as all rpc threads consumed by the tasks that ensue. The *real* problem here is simply that when deleting an RBD volume, each rpc threads servicing the request will (by default) list and query *every* volume. This has two fallouts, the first being that the get_usage_info loop [0] is racing with deletes because each time it gets a list of volumes, by the time it queries each entry in the list that volume may have been deleted. The second issue is simply that is it doing this query of all volumes for every query all the time which simply doesn't scale. The first (images not found) has been fixed as previously mentioned but the second issue is not resolved. There is a way to stop this behaviour from happening which is to enable rbd_exclusive_cinder_pool [1] such that provisioned_capacity_gb stats are not queried and updated on deletes (or in the periodic task for that matter). Apart from the caveat described in the config desc I don't see any pitfall since, so long as the pool is only used by cinder, the allocated_capacity_gb should not get out of sync with provisioned_capacity_gb although probably worth keeping an eye on this.

An improvement to the volume/manager.py might be to allow a driver to differentiate between stats gathering from the periodic task [2] vs. an operation such as a delete [3] so that it can decide whether or not to do it.

[0] https://github.com/openstack/cinder/blob/12.0.3/cinder/volume/drivers/rbd.py#L402
[1] https://github.com/openstack/cinder/blob/12.0.3/cinder/volume/drivers/rbd.py#L106
[2] https://github.com/openstack/cinder/blob/12.0.3/cinder/volume/manager.py#L2544
[3] https://github.com/openstack/cinder/blob/12.0.3/cinder/volume/manager.py#L887