From the run:
10:33:24 2020-09-07 10:31:11 [INFO] Downloading http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
10:33:24 2020-09-07 10:33:04 [INFO] Uploading precise to glance
10:33:24 Traceback (most recent call last):
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/requests/adapters.py", line 472, in send
10:33:24 low_conn.send(i)
10:33:24 File "/usr/lib/python3.5/http/client.py", line 920, in send
10:33:24 self.sock.sendall(data)
10:33:24 File "/usr/lib/python3.5/ssl.py", line 891, in sendall
10:33:24 v = self.send(data[count:])
10:33:24 File "/usr/lib/python3.5/ssl.py", line 861, in send
10:33:24 return self._sslobj.write(data)
10:33:24 File "/usr/lib/python3.5/ssl.py", line 586, in write
10:33:24 return self._sslobj.write(data)
10:33:24 ConnectionResetError: [Errno 104] Connection reset by peer
10:33:24
10:33:24 During handling of the above exception, another exception occurred:
10:33:24
10:33:24 Traceback (most recent call last):
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/keystoneauth1/session.py", line 1012, in _send_request
10:33:24 resp = self.session.request(method, url, **kwargs)
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
10:33:24 resp = self.send(prep, **send_kwargs)
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
10:33:24 r = adapter.send(request, **kwargs)
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/requests/adapters.py", line 498, in send
10:33:24 raise ConnectionError(err, request=request)
10:33:24 requests.exceptions.ConnectionError: [Errno 104] Connection reset by peer
10:33:24
10:33:24 During handling of the above exception, another exception occurred:
10:33:24
10:33:24 Traceback (most recent call last):
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/glanceclient/common/http.py", line 373, in request
10:33:24 **kwargs)
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 257, in request
10:33:24 return self.session.request(url, method, **kwargs)
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/keystoneauth1/session.py", line 921, in request
10:33:24 resp = send(**kwargs)
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/keystoneauth1/session.py", line 1028, in _send_request
10:33:24 raise exceptions.ConnectFailure(msg)
10:33:24 keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://172.17.104.236:9292/v2/images/ecb205ea-aab4-4770-9217-025dd171c3ce/file: [Errno 104] Connection reset by peer
10:33:24
10:33:24 During handling of the above exception, another exception occurred:
10:33:24
10:33:24 Traceback (most recent call last):
10:33:24 File "/tmp/tmp.fqLKON3Kte/mojo-openstack-specs/xenial/osci-mojo/spec/specs/full_stack/next_designate_ha/pike/image_setup.py", line 49, in <module>
10:33:24 sys.exit(main(sys.argv))
10:33:24 File "/tmp/tmp.fqLKON3Kte/mojo-openstack-specs/xenial/osci-mojo/spec/specs/full_stack/next_designate_ha/pike/image_setup.py", line 45, in main
10:33:24 image_config[image]['container_format'])
10:33:24 File "/tmp/tmp.fqLKON3Kte/mojo-openstack-specs/xenial/osci-mojo/spec/helper/setup/utils/mojo_os_utils.py", line 110, in upload_image
10:33:24 gclient.images.upload(image.id, fimage)
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/glanceclient/common/utils.py", line 598, in inner
10:33:24 return RequestIdProxy(wrapped(*args, **kwargs))
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/glanceclient/v2/images.py", line 288, in upload
10:33:24 resp, body = self.http_client.put(url, headers=hdrs, data=body)
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 404, in put
10:33:24 return self.request(url, 'PUT', **kwargs)
10:33:24 File "/var/lib/jenkins/tools/0/charm-test-infra/.tox/clients/lib/python3.5/site-packages/glanceclient/common/http.py", line 385, in request
10:33:24 raise exc.CommunicationError(message=message)
10:33:24 glanceclient.exc.CommunicationError: Error finding address for https://172.17.104.236:9292/v2/images/ecb205ea-aab4-4770-9217-025dd171c3ce/file: Unable to establish connection to https://172.17.104.236:9292/v2/images/ecb205ea-aab4-4770-9217-025dd171c3ce/file: [Errno 104] Connection reset by peer
10:33:24 Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
10:33:24 Traceback (most recent call last):
10:33:24 File "/usr/lib/python3.5/asyncio/base_events.py", line 431, in __del__
10:33:24 File "/usr/lib/python3.5/asyncio/unix_events.py", line 58, in close
10:33:24 File "/usr/lib/python3.5/asyncio/unix_events.py", line 139, in remove_signal_handler
10:33:24 File "/usr/lib/python3.5/signal.py", line 47, in signal
10:33:24 TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object
From the glance_api.log for the glance unit:
2020-09-07 10:33:06.173 9728 WARNING glance_store._drivers.rbd [req-e81d01cc-2fd6-4b07-abbc-79012cc50499 45d201916ae54e0ba6a8264a9685a5e9 404242ab4ff44fa0b4f10d81a54f855e - default default] since image size is zero we will be doing resize-before-write for each chunk which will be considerably slower than normal
2020-09-07 10:36:29.697 9728 ERROR glance_store._drivers.rbd [req-e81d01cc-2fd6-4b07-abbc-79012cc50499 45d201916ae54e0ba6a8264a9685a5e9 404242ab4ff44fa0b4f10d81a54f855e - default default] Failed to store image ecb205ea-aab4-4770-9217-025dd171c3ce Store Exception invalid literal for int() with base 16: '': ChunkReadError: invalid literal for int() with base 16: ''
2020-09-07 10:36:31.166 9728 ERROR glance.api.v2.image_data [req-e81d01cc-2fd6-4b07-abbc-79012cc50499 45d201916ae54e0ba6a8264a9685a5e9 404242ab4ff44fa0b4f10d81a54f855e - default default] Failed to upload image data due to HTTP error: HTTPBadRequest: invalid literal for int() with base 16: ''
2020-09-07 10:36:31.229 9728 ERROR eventlet.wsgi.server [req-e81d01cc-2fd6-4b07-abbc-79012cc50499 45d201916ae54e0ba6a8264a9685a5e9 404242ab4ff44fa0b4f10d81a54f855e - default default] chunked encoding error while discarding request body. ip=172.17.104.55,127.0.0.1 request="PUT /v2/images/ecb205ea-aab4-4770-9217-025dd171c3ce/file HTTP/1.1" error="invalid literal for int() with base 16: ''": ChunkReadError: invalid literal for int() with base 16: ''
2020-09-07 10:36:31.229 9728 INFO eventlet.wsgi.server [req-e81d01cc-2fd6-4b07-abbc-79012cc50499 45d201916ae54e0ba6a8264a9685a5e9 404242ab4ff44fa0b4f10d81a54f855e - default default] 172.17.104.55,127.0.0.1 - - [07/Sep/2020 10:36:31] "PUT /v2/images/ecb205ea-aab4-4770-9217-025dd171c3ce/file HTTP/1.1" 400 384 205.248547
Searching google indicates that this can happen (the ChunkReadError due to):
* Mismatched MTUs in the network (e.g. 1500 vs 9000)
* haproxy timeouts
* timeouts in swift
* timeouts in glance
This was probably due to a busy OSCI test cloud. Also attaching logs for future explorers.
