I noticed the following Traceback at oops.canonical.com (https://oops.canonical.com/oops/?oopsid=OOPS-82d71fcf875be98075d799a24dc2ed3d) today:
ClientException: Authorization Failure. Authorization Failed: HTTPConnectionPool(host='10.24.0.132', port=5000): Max retries exceeded with url: /v2.0/tokens
File "/usr/lib/python2.7/dist-packages/oops_wsgi/middleware.py", line 208, in oops_middleware
app(environ, oops_start_response))
File "/<email address hidden>/daisy/version_middleware.py", line 34, in __call__
return self.app(environ, custom_start_response)
File "/<email address hidden>/daisy/wsgi.py", line 98, in app
response = handle_core_dump(_pool, environ, fileobj, components, content_type)
File "/<email address hidden>/daisy/wsgi.py", line 47, in handle_core_dump
return submit_core.submit(_pool, environ, fileobj, uuid, arch)
File "/<email address hidden>/daisy/submit_core.py", line 297, in submit
message = write_to_storage_provider(environ, fileobj, uuid)
File "/<email address hidden>/daisy/submit_core.py", line 225, in write_to_storage_provider
written = write_to_swift(environ, fileobj, uuid, provider_data)
File "/<email address hidden>/daisy/submit_core.py", line 103, in write_to_swift
_cached_swift.put_container(bucket)
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1065, in put_container
return self._retry(None, put_container, container, headers=headers)
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1003, in _retry
self.url, self.token = self.get_auth()
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 991, in get_auth
insecure=self.insecure)
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 328, in get_auth
insecure=insecure)
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 265, in get_keystoneclient_2_0
raise ClientException('Authorization Failure. %s' % err)
The core file was not put in swift and we have a count of 100 of these for the past week. Looking into the daisy log files the core was never asked for again.
brian@asrai:~$ ~/bin/submission-result.sh 20160629 bf7b200e-3da3-11e6-afe5-fa163e192766
./e-t-daisy-app-3108a1fc-44b3-46d0-8c20-83f9687990f4/production-logs/daisy-access.log-20160629.gz:10.25.44.180 - - [29/Jun/2016:02:47:30 +0000] "POST /bf7b200e-3da3-11e6-afe5-fa163e192766/submit-core/amd64/8011a232432ab191d75c1171277293e4ca1f09cd64fa09e1125489dc75867320def0e4718db8b5c97ef6eb4ddf7b2545427129e415753088ec565a1ce33911d4 HTTP/1.1" 500 - "-" "-"
./e-t-daisy-app-d1489f6d-31e3-4000-884d-8c6a5007656a/production-logs/daisy-error.log-20160629.gz:2016-06-29 02:46:58 [5154] [INFO] (bf7b200e-3da3-11e6-afe5-fa163e192766) inserted into OOPS CF
./e-t-daisy-app-d1489f6d-31e3-4000-884d-8c6a5007656a/production-logs/daisy-error.log-20160629.gz:2016-06-29 02:46:58 [5154] [INFO] (bf7b200e-3da3-11e6-afe5-fa163e192766) asked for CORE
The server returned a 500, but I'm not certain how whoopsie would handle this error. It certainly seems like it didn't try to resubmit the core file.
