When create server image of instance larger than 2GB (for example using openstack server image create), glance swift store driver reports
glance-wsgi-api[2154497]: 2023-08-17 05:52:04.413 2154497 DEBUG glance_store._drivers.swift.store [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Cannot determine image size because it is either not provided in the request or chunked-transfer encoding is used. Adding image as a segmented object to Swift. add /openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/_drivers/swift/store.py:965
and then
glance-wsgi-api[2154497]: 2023-08-17 05:52:40.169 2154497 ERROR glance_store._drivers.swift.store [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Error during chunked upload to backend, deleting stale chunks.: OverflowError: string longer than 2147483647 bytes
glance-wsgi-api[2154497]: 2023-08-17 05:52:40.185 2154497 ERROR glance.api.v2.image_data [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Failed to upload image data due to internal error: OverflowError: string longer than 2147483647 bytes
glance-wsgi-api[2154497]: 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Caught error: string longer than 2147483647 bytes: OverflowError: string longer than 2147483647 bytes
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi Traceback (most recent call last):
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/common/wsgi.py", line 1286, in __call__
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action,
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/common/wsgi.py", line 1329, in dispatch
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return method(*args, **kwargs)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/common/utils.py", line 414, in wrapped
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/api/v2/image_data.py", line 302, in upload
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self._restore(image_repo, image)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.force_reraise()
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi raise self.value
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/api/v2/image_data.py", line 162, in upload
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/notifier.py", line 497, in set_data
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.force_reraise()
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi raise self.value
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/notifier.py", line 443, in set_data
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend,
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/quota/__init__.py", line 322, in set_data
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend,
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/location.py", line 585, in set_data
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/location.py", line 476, in _upload_to_store
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash(
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/multi_backend.py", line 397, in add_with_multihash
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return store_add_to_backend_with_multihash(
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/multi_backend.py", line 479, in store_add_to_backend_with_multihash
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add(
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/driver.py", line 277, in add_adapter
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/capabilities.py", line 176, in op_checker
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/_drivers/swift/store.py", line 1010, in add
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self._delete_stale_chunks(
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.force_reraise()
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi raise self.value
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/_drivers/swift/store.py", line 999, in add
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi manager.get_connection().put_object(
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 1956, in put_object
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents,
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 1797, in _retry
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi rv = func(self.url, self.token, *args,
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 1411, in put_object
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi conn.putrequest(path, headers=headers, data=data)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 427, in putrequest
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self.request('PUT', full_path, data, headers, files)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 416, in request
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.resp = self._request(method, url, headers=headers, data=data,
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 400, in _request
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self.request_session.request(*arg, **kwarg)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi resp = self.send(prep, **send_kwargs)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi r = adapter.send(request, **kwargs)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/requests/adapters.py", line 526, in send
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi low_conn.send(i)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/usr/lib/python3.9/http/client.py", line 971, in send
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.sock.sendall(data)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/eventlet/green/ssl.py", line 220, in sendall
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi v = self.send(data_to_send)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/eventlet/green/ssl.py", line 194, in send
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self._call_trampolining(
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return func(*a, **kw)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/usr/lib/python3.9/ssl.py", line 1173, in send
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self._sslobj.write(data)
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi OverflowError: string longer than 2147483647 bytes
2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi
from further debug it seems, that add method from store.py in our case calls manager.get_connection().put_object( location.container,chunk_name,reader,content_length=content_length), which leads to executing ReadableToIterable from python-swiftclient over ChunkReader from store.py with 64k "sub"-chunk sizes. For some reason I cannot handle out, it eventually leads to that self.bytes_read within ChunkReader becomes larger than self.total which is swift chunk size in our case, and when it then tries to do fd.read(negative number) it returns the whole file, which then leads to overflow.
If I just put in ChunkReader read method something like left = max(0,self.total - self.bytes_read) it solves the problem, so the image is uploaded correctly, but it does not seem right, because it seems to me that problem is somewhere else, maybe in CooperativeReader of glance/utils.py...
For me bug is reproducible with any late version of glance/python-swiftclient/nova.
