Glance

BackendException from glance_store causing 500 Internal Server Error

Bug #1526641 reported by Abhishek Kekane
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Glance
In Progress
Undecided
Abhishek Kekane

Bug Description

1.Metadata error:

Passing invalid metadata value to location-add or location-update command raises 500 Internal Server Error.

1). glance location-add <image-id> --url http://10.69.4.136:8111/run_tests.sh --metadata '{"a":1234}'
or
2). glance location-update <image-id> --url http://10.69.4.136:8111/run_tests.sh --metadata '{"a":1234}'

Stacktrace on g-api:

2015-12-15 03:47:49.876 ERROR glance.common.wsgi [req-3a349a22-8b69-40bf-93f1-6388eb5cb4ee 6c9c7376e75b4e37ab3c052a911731dd d1ee7fd5dcc341c3973f19f790238e63] Caught error: The image metadata key a has an invalid type of <type 'int'>. Only dict, list, and unicode are supported.
2015-12-15 03:47:49.876 TRACE glance.common.wsgi Traceback (most recent call last):2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 882, in __call__
2015-12-15 03:47:49.876 TRACE glance.common.wsgi request, **action_args)
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 911, in dispatch
2015-12-15 03:47:49.876 TRACE glance.common.wsgi return method(*args, **kwargs)
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/opt/stack/glance/glance/common/utils.py", line 423, in wrapped
2015-12-15 03:47:49.876 TRACE glance.common.wsgi return func(self, req, *args,**kwargs)
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/opt/stack/glance/glance/api/v2/images.py", line 147, in update
2015-12-15 03:47:49.876 TRACE glance.common.wsgi change_method(req, image, change)
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/opt/stack/glance/glance/api/v2/images.py", line 197, in _do_add
2015-12-15 03:47:49.876 TRACE glance.common.wsgi self._do_add_locations(image, path[1], value)
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/opt/stack/glance/glance/api/v2/images.py", line 292, in _do_add_locations
2015-12-15 03:47:49.876 TRACE glance.common.wsgi image.locations.insert(pos, value)
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/opt/stack/glance/glance/api/policy.py", line 275, in _checker
2015-12-15 03:47:49.876 TRACE glance.common.wsgi return method(*args, **kwargs)
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/opt/stack/glance/glance/quota/__init__.py", line 273, in insert
2015-12-15 03:47:49.876 TRACE glance.common.wsgi return self.locations.insert(index, object)
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 179, in insert
2015-12-15 03:47:49.876 TRACE glance.common.wsgi location)
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 91, in _check_image_location
2015-12-15 03:47:49.876 TRACE glance.common.wsgi store_api.check_location_metadata(location['metadata'])
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 309, in check_location_metadata
2015-12-15 03:47:49.876 TRACE glance.common.wsgi check_location_metadata(val[key], key=key)
2015-12-15 03:47:49.876 TRACE glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 320, in check_location_metadata
2015-12-15 03:47:49.876 TRACE glance.common.wsgi % dict(key=key, type=type(val)))
2015-12-15 03:47:49.876 TRACE glance.common.wsgi BackendException: The image metadata key a has an invalid type of <type 'int'>. Only dict, list, and unicode are supported.
2015-12-15 03:47:49.876 TRACE glance.common.wsgi

2.image-upload error:

when the storage back-end raise BackendException error,Glance doesn't catch it and raise 500 error. Such as if there is no enough space for image store, the back-end(Such as Swift) will sometimes raise a BackendException error.

Reproduce:
1.suppose use Swift back-end, and there is 1G could be use.
2.create an image with 2G.
3.Glance will return 500 error

2015-12-26 08:48:47.534 ERROR glance.common.wsgi [req-ab660c64-e832-4dd3-ac55-da40893cafdf b7290f0ac6eb40049eedd026fc351e33 5e9b38f638d04cbd92444c0d9de996fc] Caught error: Failed to add object to Swift.
Got error from Swift: put_object('glance', '73492217-f108-47b8-bfa4-1980a8e420bc-00010', ...) failure and no ability to reset contents for reupload..
2015-12-26 08:48:47.534 TRACE glance.common.wsgi Traceback (most recent call last):
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 882, in __call__
2015-12-26 08:48:47.534 TRACE glance.common.wsgi request, **action_args)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 911, in dispatch
2015-12-26 08:48:47.534 TRACE glance.common.wsgi return method(*args, **kwargs)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/common/utils.py", line 423, in wrapped
2015-12-26 08:48:47.534 TRACE glance.common.wsgi return func(self, req, *args, **kwargs)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/api/v2/image_data.py", line 204, in upload
2015-12-26 08:48:47.534 TRACE glance.common.wsgi self._restore(image_repo, image)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-12-26 08:48:47.534 TRACE glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/api/v2/image_data.py", line 89, in upload
2015-12-26 08:48:47.534 TRACE glance.common.wsgi image.set_data(data, size)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/domain/proxy.py", line 166, in set_data
2015-12-26 08:48:47.534 TRACE glance.common.wsgi self.base.set_data(data, size)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/notifier.py", line 433, in set_data
2015-12-26 08:48:47.534 TRACE glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-12-26 08:48:47.534 TRACE glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/notifier.py", line 380, in set_data
2015-12-26 08:48:47.534 TRACE glance.common.wsgi self.repo.set_data(data, size)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/api/policy.py", line 192, in set_data
2015-12-26 08:48:47.534 TRACE glance.common.wsgi return self.image.set_data(*args, **kwargs)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/quota/__init__.py", line 298, in set_data
2015-12-26 08:48:47.534 TRACE glance.common.wsgi self.image.set_data(data, size=size)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 386, in set_data
2015-12-26 08:48:47.534 TRACE glance.common.wsgi context=self.context)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance_store/glance_store/backend.py", line 367, in add_to_backend
2015-12-26 08:48:47.534 TRACE glance.common.wsgi return store_add_to_backend(image_id, data, size, store, context)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance_store/glance_store/backend.py", line 342, in store_add_to_backend
2015-12-26 08:48:47.534 TRACE glance.common.wsgi context=context)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance_store/glance_store/capabilities.py", line 226, in op_checker
2015-12-26 08:48:47.534 TRACE glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi File "/opt/stack/glance_store/glance_store/_drivers/swift/store.py", line 626, in add
2015-12-26 08:48:47.534 TRACE glance.common.wsgi raise glance_store.BackendException(msg)
2015-12-26 08:48:47.534 TRACE glance.common.wsgi BackendException: Failed to add object to Swift.
2015-12-26 08:48:47.534 TRACE glance.common.wsgi Got error from Swift: put_object('glance', '73492217-f108-47b8-bfa4-1980a8e420bc-00010', ...) failure and no ability to reset contents for reupload..

See original description
Abhishek Kekane (abhishek-kekane)
Changed in glance:
assignee: nobody → Abhishek Kekane (abhishek-kekane)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/259963

Changed in glance:
status: New → In Progress
Revision history for this message
Abhishek Kekane (abhishek-kekane) wrote : Re: Invalid metadata value to location-add causing 500 Internal Server Error

Note: This bug is not confirmed by other community member yet.

Revision history for this message
wangxiyuan (wangxiyuan) wrote :

Confirm.

500 error could be raised by any BackendExecption error.

At least, image-update could raise 500 as well.

The bug both related to glance_store's BackendExecption error. So It's better to fix them together.

summary: - Invalid metadata value to location-add causing 500 Internal Server Error
+ BackendException from glance_store causing 500 Internal Server Error
description: updated
wangxiyuan (wangxiyuan)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance (master)

Change abandoned by Nikhil Komawar (<email address hidden>) on branch: master
Review: https://review.openstack.org/259963
Reason: as per confirmation, let's abandon now and think of restoring in ocata

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.