Comment 2 for bug 1886968

Happened again here: http://osci:8080/job/mojo_runner/23871/console

2020-09-30 05:02:17 DEBUG ceilometer-upgrade none
2020-09-30 05:02:17 DEBUG ceilometer-upgrade none
2020-09-30 05:02:18 DEBUG juju-log Running ceilometer-upgrade: ceilometer-upgrade --debug
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.541 11628 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.543 11628 DEBUG ceilometer.cmd.storage [-] Upgrading metering database upgrade /usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py:48
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.543 11628 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/dist-packages/ceilometer/storage/__init__.py:87
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.588 11628 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('172.17.107.28', 27017)]
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.656 11628 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py:57
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 CRITICAL ceilometer [-] Unhandled error: ClientException: {"message": "The server is currently unavailable. Please try again at a later time.<br /><br />\n\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} (HTTP 503)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer Traceback (most recent call last):
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer sys.exit(upgrade())
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py", line 59, in upgrade
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer gnocchi_client.upgrade_resource_types(conf)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/gnocchi_client.py", line 194, in upgrade_resource_types
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer gnocchi.resource_type.get(name=name)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/gnocchiclient/v1/resource_type.py", line 44, in get
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer headers={'Content-Type': "application/json"}).json()
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/gnocchiclient/v1/base.py", line 37, in _get
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer return self.client.api.get(*args, **kwargs)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 288, in get
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/gnocchiclient/client.py", line 38, in request
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer raise exceptions.from_response(resp, method)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer ClientException: {"message": "The server is currently unavailable. Please try again at a later time.<br /><br />\n\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} (HTTP 503)

So it looks like Gnocchi is giving a 503 to ceilometer although it advertises as ready.