Comment 4 for bug 1823011

Revision history for this message
Nobuto Murata (nobuto) wrote (last edit ): Re: charm does not create metrics cache file

This is most likely another error rather than the cache file itself. The snap doesn't expose a backend error in a nice way to the /metrics endpoint.

In my case, it was an exception in keystone client with SSL verification failure:

HTTPSConnectionPool(host='192.168.151.118', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),))

$ curl localhost:9183/metrics
Traceback (most recent call last):
  File "/snap/prometheus-openstack-exporter/29/bin/prometheus-openstack-exporter", line 740, in do_GET
    collectors = [COLLECTORS[collector]() for collector in get_collectors(config.get('enabled_collectors'))]
  File "/snap/prometheus-openstack-exporter/29/bin/prometheus-openstack-exporter", line 326, in __init__
    with open(config['cache_file'], 'rb') as f:
IOError: [Errno 2] No such file or directory: '/var/snap/prometheus-openstack-exporter/common/RegionOne'

$ journalctl -u snap.prometheus-openstack-exporter.prometheus-openstack-exporter.service
...
Aug 11 01:05:19 juju-2ac58b-1-lxd-7 prometheus-openstack-exporter.prometheus-openstack-exporter[15486]: 127.0.0.1 - - [11/Aug/2021 01:05:19] "GET /metrics HTTP/1.1" 500 -
Aug 11 01:10:10 juju-2ac58b-1-lxd-7 python2[15450]: Client setup done, keystone ver 3
Aug 11 01:10:10 juju-2ac58b-1-lxd-7 python2[15450]: Error getting tenants.list, continue with projects.list
Aug 11 01:10:10 juju-2ac58b-1-lxd-7 prometheus-openstack-exporter.prometheus-openstack-exporter[15450]: Client setup done, keystone ver 3
Aug 11 01:10:10 juju-2ac58b-1-lxd-7 prometheus-openstack-exporter.prometheus-openstack-exporter[15450]: Error getting tenants.list, continue with projects.list
Aug 11 01:10:10 juju-2ac58b-1-lxd-7 python2[15450]: Error getting stats: Traceback (most recent call last):
                                                      File "/snap/prometheus-openstack-exporter/29/bin/prometheus-openstack-exporter", line 204, in run
                                                        prodstack.update(self._get_keystone_info(keystone))
                                                      File "/snap/prometheus-openstack-exporter/29/bin/prometheus-openstack-exporter", line 152, in _get_keystone_info
                                                        info['tenants'] = [x._info for x in keystone.projects.list()]
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/positional/__init__.py", line 108, in inner
                                                        return wrapped(*args, **kwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneclient/v3/projects.py", line 119, in list
                                                        **kwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneclient/base.py", line 75, in func
                                                        return f(*args, **new_kwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneclient/base.py", line 390, in list
                                                        self.collection_key)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneclient/base.py", line 125, in _list
                                                        resp, body = self.client.get(url, **kwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 386, in get
                                                        return self.request(url, 'GET', **kwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 545, in request
                                                        resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request
                                                        return self.session.request(url, method, **kwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/session.py", line 747, in request
                                                        auth_headers = self.get_auth_headers(auth)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/session.py", line 1158, in get_auth_headers
                                                        return auth.get_headers(self, **kwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
                                                        token = self.get_token(session)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
                                                        return self.get_access(session).auth_token
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
                                                        self.auth_ref = self.get_auth_ref(session)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref
                                                        authenticated=False, log=False, **rkwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/session.py", line 1106, in post
                                                        return self.request(url, 'POST', **kwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/session.py", line 888, in request
                                                        resp = send(**kwargs)
                                                      File "/snap/prometheus-openstack-exporter/29/lib/python2.7/site-packages/keystoneauth1/session.py", line 983, in _send_request
                                                        raise exceptions.SSLError(msg)
                                                    SSLError: SSL exception connecting to https://192.168.151.118:5000/v3/auth/tokens: HTTPSConnectionPool(host='192.168.151.118', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),))