Glance

  1. Bug #2031606
  2. Activity log

Activity log for bug #2031606

Date Who What changed Old value New value Message
2023-08-17 06:13:31 Alex Alexandrov bug added bug
2023-08-17 08:01:38 Alex Alexandrov description When create server image of instance larger than 2GB (for example using openstack server image create), glance swift store driver reports glance-wsgi-api[2154497]: 2023-08-17 05:52:04.413 2154497 DEBUG glance_store._drivers.swift.store [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Cannot determine image size because it is either not provided in the request or chunked-transfer encoding is used. Adding image as a segmented object to Swift. add /openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/_drivers/swift/store.py:965 and then glance-wsgi-api[2154497]: 2023-08-17 05:52:40.169 2154497 ERROR glance_store._drivers.swift.store [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Error during chunked upload to backend, deleting stale chunks.: OverflowError: string longer than 2147483647 bytes glance-wsgi-api[2154497]: 2023-08-17 05:52:40.185 2154497 ERROR glance.api.v2.image_data [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Failed to upload image data due to internal error: OverflowError: string longer than 2147483647 bytes glance-wsgi-api[2154497]: 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Caught error: string longer than 2147483647 bytes: OverflowError: string longer than 2147483647 bytes 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi Traceback (most recent call last): 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/common/wsgi.py", line 1286, in __call__ 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/common/wsgi.py", line 1329, in dispatch 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return method(*args, **kwargs) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/common/utils.py", line 414, in wrapped 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/api/v2/image_data.py", line 302, in upload 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self._restore(image_repo, image) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.force_reraise() 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi raise self.value 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/api/v2/image_data.py", line 162, in upload 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/notifier.py", line 497, in set_data 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.force_reraise() 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi raise self.value 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/notifier.py", line 443, in set_data 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/quota/__init__.py", line 322, in set_data 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/location.py", line 585, in set_data 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/location.py", line 476, in _upload_to_store 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/multi_backend.py", line 397, in add_with_multihash 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/multi_backend.py", line 479, in store_add_to_backend_with_multihash 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/driver.py", line 277, in add_adapter 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/capabilities.py", line 176, in op_checker 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/_drivers/swift/store.py", line 1010, in add 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self._delete_stale_chunks( 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.force_reraise() 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi raise self.value 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/_drivers/swift/store.py", line 999, in add 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi manager.get_connection().put_object( 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 1956, in put_object 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents, 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 1797, in _retry 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi rv = func(self.url, self.token, *args, 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 1411, in put_object 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi conn.putrequest(path, headers=headers, data=data) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 427, in putrequest 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self.request('PUT', full_path, data, headers, files) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 416, in request 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.resp = self._request(method, url, headers=headers, data=data, 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 400, in _request 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self.request_session.request(*arg, **kwarg) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/requests/sessions.py", line 587, in request 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi resp = self.send(prep, **send_kwargs) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/requests/sessions.py", line 701, in send 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi r = adapter.send(request, **kwargs) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/requests/adapters.py", line 526, in send 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi low_conn.send(i) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/usr/lib/python3.9/http/client.py", line 971, in send 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.sock.sendall(data) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/eventlet/green/ssl.py", line 220, in sendall 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi v = self.send(data_to_send) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/eventlet/green/ssl.py", line 194, in send 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self._call_trampolining( 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return func(*a, **kw) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/usr/lib/python3.9/ssl.py", line 1173, in send 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self._sslobj.write(data) 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi OverflowError: string longer than 2147483647 bytes 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi from further debug it seems, that add method from store.py in our case calls manager.get_connection().put_object( location.container,chunk_name,reader,content_length=content_length), which leads to executing ReadableToIterable from python-swiftclient over ChunkReader from store.py with 64k "sub"-chunk sizes. For some reason I cannot handle out, it eventually leads to that self.bytes_read within ChunkReader becomes larger than self.total which is swift chunk size in our case, and when it then tries to do fd.read(negative number) it returns the whole file, which then leads to overflow. If I just put in CunkReader read method something like left = max(0,self.total - self.bytes_read) it solves the problem, so the image is uploaded correctly, but it does not seem right, because it seems to me that problem is somewhere else, maybe in CooperativeReader of glance/utils.py... For me bug is reproducible with any late version of glance/python-swiftclient/nova. When create server image of instance larger than 2GB (for example using openstack server image create), glance swift store driver reports glance-wsgi-api[2154497]: 2023-08-17 05:52:04.413 2154497 DEBUG glance_store._drivers.swift.store [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Cannot determine image size because it is either not provided in the request or chunked-transfer encoding is used. Adding image as a segmented object to Swift. add /openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/_drivers/swift/store.py:965 and then glance-wsgi-api[2154497]: 2023-08-17 05:52:40.169 2154497 ERROR glance_store._drivers.swift.store [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Error during chunked upload to backend, deleting stale chunks.: OverflowError: string longer than 2147483647 bytes glance-wsgi-api[2154497]: 2023-08-17 05:52:40.185 2154497 ERROR glance.api.v2.image_data [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Failed to upload image data due to internal error: OverflowError: string longer than 2147483647 bytes glance-wsgi-api[2154497]: 2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi [None req-0a904444-9be1-405b-a419-c9f288196e5b ac20eaa011b04d14b793ff97f056f190 5567a3a1c02e4238b68493eb5d858d9a - - default default] Caught error: string longer than 2147483647 bytes: OverflowError: string longer than 2147483647 bytes                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi Traceback (most recent call last):                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/common/wsgi.py", line 1286, in __call__                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action,                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/common/wsgi.py", line 1329, in dispatch                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return method(*args, **kwargs)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/common/utils.py", line 414, in wrapped                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/api/v2/image_data.py", line 302, in upload                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self._restore(image_repo, image)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.force_reraise()                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi raise self.value                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/api/v2/image_data.py", line 162, in upload                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/notifier.py", line 497, in set_data                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.force_reraise()                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi raise self.value                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/notifier.py", line 443, in set_data                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend,                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/quota/__init__.py", line 322, in set_data                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend,                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/location.py", line 585, in set_data                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance/location.py", line 476, in _upload_to_store                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash(                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/multi_backend.py", line 397, in add_with_multihash                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return store_add_to_backend_with_multihash(                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/multi_backend.py", line 479, in store_add_to_backend_with_multihash                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add(                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/driver.py", line 277, in add_adapter                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/capabilities.py", line 176, in op_checker                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/_drivers/swift/store.py", line 1010, in add                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self._delete_stale_chunks(                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.force_reraise()                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi raise self.value                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/glance_store/_drivers/swift/store.py", line 999, in add                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi manager.get_connection().put_object(                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 1956, in put_object                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents,                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 1797, in _retry                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi rv = func(self.url, self.token, *args,                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 1411, in put_object                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi conn.putrequest(path, headers=headers, data=data)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 427, in putrequest                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self.request('PUT', full_path, data, headers, files)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 416, in request                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.resp = self._request(method, url, headers=headers, data=data,                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/swiftclient/client.py", line 400, in _request                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self.request_session.request(*arg, **kwarg)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/requests/sessions.py", line 587, in request                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi resp = self.send(prep, **send_kwargs)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/requests/sessions.py", line 701, in send                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi r = adapter.send(request, **kwargs)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/requests/adapters.py", line 526, in send                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi low_conn.send(i)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/usr/lib/python3.9/http/client.py", line 971, in send                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi self.sock.sendall(data)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/eventlet/green/ssl.py", line 220, in sendall                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi v = self.send(data_to_send)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/eventlet/green/ssl.py", line 194, in send                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self._call_trampolining(                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/openstack/venvs/glance-26.0.0/lib/python3.9/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return func(*a, **kw)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi File "/usr/lib/python3.9/ssl.py", line 1173, in send                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi return self._sslobj.write(data)                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi OverflowError: string longer than 2147483647 bytes                                                           2023-08-17 05:52:40.248 2154497 ERROR glance.common.wsgi from further debug it seems, that add method from store.py in our case calls manager.get_connection().put_object( location.container,chunk_name,reader,content_length=content_length), which leads to executing ReadableToIterable from python-swiftclient over ChunkReader from store.py with 64k "sub"-chunk sizes. For some reason I cannot handle out, it eventually leads to that self.bytes_read within ChunkReader becomes larger than self.total which is swift chunk size in our case, and when it then tries to do fd.read(negative number) it returns the whole file, which then leads to overflow. If I just put in ChunkReader read method something like left = max(0,self.total - self.bytes_read) it solves the problem, so the image is uploaded correctly, but it does not seem right, because it seems to me that problem is somewhere else, maybe in CooperativeReader of glance/utils.py... For me bug is reproducible with any late version of glance/python-swiftclient/nova.