Cinder backend for glance raises 500 if quota of volume exceeds

Bug #1875425 reported by Abhishek Kekane
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glance_store
New
Undecided
Abhishek Kekane

Bug Description

If you are using cinder backend for glance and quota of volumes is exceeded it raises 500 error on new image creation.

Expected error:
40* error response (mostly 401 or 403)

Actual error:
HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.

glance-api logs:

Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi Traceback (most recent call last):
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 1347, in __call__
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi request, **action_args)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 1390, in dispatch
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return method(*args, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/utils.py", line 416, in wrapped
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/v2/image_data.py", line 299, in upload
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi self._restore(image_repo, image)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi self.force_reraise()
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi raise value
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/v2/image_data.py", line 164, in upload
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/domain/proxy.py", line 198, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/notifier.py", line 501, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi self.force_reraise()
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi raise value
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/notifier.py", line 448, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi set_active=set_active)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/policy.py", line 204, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/quota/__init__.py", line 319, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi set_active=set_active)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 559, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 486, in _upload_to_store
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi verifier=verifier)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/glance_store/backend.py", line 491, in add_to_backend_with_multihash
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/glance_store/backend.py", line 468, in store_add_to_backend_with_multihash
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/glance_store/driver.py", line 279, in add_adapter
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/glance_store/capabilities.py", line 176, in op_checker
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/glance_store/_drivers/cinder/cinder.py", line 780, in add
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi volume_type=volume_type)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/cinderclient/v3/volumes.py", line 125, in create
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return self._create('/volumes', body, 'volume')
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/cinderclient/base.py", line 302, in _create
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi resp, body = self.api.client.post(url, body=body)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/cinderclient/client.py", line 476, in post
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return self._cs_request(url, 'POST', **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/cinderclient/client.py", line 429, in _cs_request
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi resp, body = self.request(url, method, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/cinderclient/client.py", line 411, in request
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi raise exceptions.from_response(resp, body)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi cinderclient.exceptions.OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes'. (HTTP 413) (Request-ID: req-44598b3d-fee7-403b-8ce4-a59e695a1953)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi
Apr 27 15:15:46 victoria-cinder glance-api[29768]: INFO eventlet.wsgi.server [None req-d2dd6858-73c9-405f-9833-1c6142f2d7ab admin admin] 192.168.100.9 - - [27/Apr/2020 15:15:46] "PUT /v2/images/e815975d-0836-40aa-b4b9-9b290e287562/file HTTP/1.1" 500 454 0.164477

Changed in glance-store:
assignee: nobody → Abhishek Kekane (abhishek-kekane)
Revision history for this message
Rajat Dhasmana (whoami-rajat) wrote :

If it's related to devstack environment, the fix has landed in devstack https://review.opendev.org/c/openstack/devstack/+/774172

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.