When openstack services are restarted and Keystone is not fully up or is up but failed to respond the connection request, then Gnocchi publisher plugin fails to load.
I am not sure if the Gnocchi publisher can be refactored to allow plugin to load initially and initialize/retry for keystone connection later when it becomes available.. ?
Openstack release: ussuri.
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base [-] Unable to load publisher gnocchi://: keystoneauth1.exceptions.connection.ConnectTimeout: Request to https://<host>:5000/v3/auth/tokens timed out
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base six.raise_from(e, None)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "<string>", line 3, in raise_from
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base httplib_response = conn.getresponse()
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base response.begin()
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib64/python3.6/http/client.py", line 307, in begin
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base version, status, reason = self._read_status()
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib64/python3.6/http/client.py", line 268, in _read_status
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib64/python3.6/socket.py", line 586, in readinto
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base return self._sock.recv_into(b)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 328, in recv_into
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base return self.recv_into(*args, **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 326, in recv_into
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base raise timeout("The read operation timed out")
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base socket.timeout: The read operation timed out
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base During handling of the above exception, another exception occurred:
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base timeout=timeout
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base raise six.reraise(type(error), error, _stacktrace)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 703, in reraise
....
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/ceilometer/pipeline/base.py", line 124, in __init__
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base self.publishers.append(publisher_manager.get(p))
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/ceilometer/pipeline/base.py", line 179, in get
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base 'ceilometer.%s.publisher' % self._purpose)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/ceilometer/publisher/__init__.py", line 36, in get_publisher
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base return loaded_driver.driver(conf, parse_result)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/ceilometer/publisher/gnocchi.py", line 249, in __init__
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base conf, request_timeout=timeout)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/ceilometer/gnocchi_client.py", line 39, in get_gnocchiclient
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base region_name=region_name)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1225, in get_endpoint
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base return auth.get_endpoint(self, **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base allow_version_hack=allow_version_hack, **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base service_catalog = self.get_access(session).service_catalog
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base self.auth_ref = self.get_auth_ref(session)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base authenticated=False, log=False, **rkwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1131, in post
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base return self.request(url, 'POST', **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base resp = send(**kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in _send_request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base raise exceptions.ConnectTimeout(msg)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base keystoneauth1.exceptions.connection.ConnectTimeout: Request to https://<host>:5000/v3/auth/tokens timed out
