Gnocchi Charm

Running Ceilometer's setup action crashes on gnocchi shortly after deployment when using TLS

Bug #1824480 reported by Chris MacNaughton
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Gnocchi Charm
Triaged
High
Unassigned
Openstack Mojo Testing
Fix Committed
Undecided
Chris MacNaughton

Bug Description

IN Gnocchi's Apache error log, I see:

[Fri Apr 12 07:53:57.054035 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] mod_wsgi (pid=1925): Exception occurred processing WSGI script '/usr/bin/gnocchi-api'.
[Fri Apr 12 07:53:57.059296 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] Traceback (most recent call last):
[Fri Apr 12 07:53:57.059604 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/webob/dec.py", line 131, in __call__
[Fri Apr 12 07:53:57.059620 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] resp = self.call_func(req, *args, **self.kwargs)
[Fri Apr 12 07:53:57.059640 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/webob/dec.py", line 196, in call_func
[Fri Apr 12 07:53:57.059650 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] return self.func(req, *args, **kwargs)
[Fri Apr 12 07:53:57.059668 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/oslo_middleware/base.py", line 131, in __call__
[Fri Apr 12 07:53:57.059677 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] response = req.get_response(self.application)
[Fri Apr 12 07:53:57.059695 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/webob/request.py", line 1316, in send
[Fri Apr 12 07:53:57.059705 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] application, catch_exc_info=False)
[Fri Apr 12 07:53:57.059723 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/webob/request.py", line 1280, in call_application
[Fri Apr 12 07:53:57.059732 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] app_iter = application(self.environ, start_response)
[Fri Apr 12 07:53:57.059750 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/paste/urlmap.py", line 216, in __call__
[Fri Apr 12 07:53:57.059759 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] return app(environ, start_response)
[Fri Apr 12 07:53:57.059776 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/webob/dec.py", line 131, in __call__
[Fri Apr 12 07:53:57.059785 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] resp = self.call_func(req, *args, **self.kwargs)
[Fri Apr 12 07:53:57.059803 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/webob/dec.py", line 196, in call_func
[Fri Apr 12 07:53:57.059813 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] return self.func(req, *args, **kwargs)
[Fri Apr 12 07:53:57.059835 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/oslo_middleware/base.py", line 131, in __call__
[Fri Apr 12 07:53:57.059847 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] response = req.get_response(self.application)
[Fri Apr 12 07:53:57.059864 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/webob/request.py", line 1316, in send
[Fri Apr 12 07:53:57.059873 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] application, catch_exc_info=False)
[Fri Apr 12 07:53:57.059891 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/webob/request.py", line 1280, in call_application
[Fri Apr 12 07:53:57.059900 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] app_iter = application(self.environ, start_response)
[Fri Apr 12 07:53:57.059918 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/webob/dec.py", line 131, in __call__
[Fri Apr 12 07:53:57.059941 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] resp = self.call_func(req, *args, **self.kwargs)
[Fri Apr 12 07:53:57.059960 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/webob/dec.py", line 196, in call_func
[Fri Apr 12 07:53:57.059970 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] return self.func(req, *args, **kwargs)
[Fri Apr 12 07:53:57.059987 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 334, in __call__
[Fri Apr 12 07:53:57.059997 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] response = self.process_request(req)
[Fri Apr 12 07:53:57.060019 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 633, in process_request
[Fri Apr 12 07:53:57.060031 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] resp = super(AuthProtocol, self).process_request(request)
[Fri Apr 12 07:53:57.060048 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 407, in process_request
[Fri Apr 12 07:53:57.060057 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] allow_expired=allow_expired)
[Fri Apr 12 07:53:57.060073 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 440, in _do_fetch_token
[Fri Apr 12 07:53:57.060081 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] data = self.fetch_token(token, **kwargs)
[Fri Apr 12 07:53:57.060097 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 772, in fetch_token
[Fri Apr 12 07:53:57.060105 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] allow_expired=allow_expired)
[Fri Apr 12 07:53:57.060120 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
[Fri Apr 12 07:53:57.060128 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] auth_ref = self._request_strategy.verify_token(
[Fri Apr 12 07:53:57.060144 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
[Fri Apr 12 07:53:57.060152 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] strategy_class = self._get_strategy_class()
[Fri Apr 12 07:53:57.060167 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
[Fri Apr 12 07:53:57.060176 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
[Fri Apr 12 07:53:57.060192 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 223, in get_endpoint
[Fri Apr 12 07:53:57.060201 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] return self.session.get_endpoint(auth or self.auth, **kwargs)
[Fri Apr 12 07:53:57.060218 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 942, in get_endpoint
[Fri Apr 12 07:53:57.060226 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] return auth.get_endpoint(self, **kwargs)
[Fri Apr 12 07:53:57.060251 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 379, in get_endpoint
[Fri Apr 12 07:53:57.060260 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] allow_version_hack=allow_version_hack, **kwargs)
[Fri Apr 12 07:53:57.060276 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 270, in get_endpoint_data
[Fri Apr 12 07:53:57.060285 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] service_catalog = self.get_access(session).service_catalog
[Fri Apr 12 07:53:57.060302 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
[Fri Apr 12 07:53:57.060312 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] self.auth_ref = self.get_auth_ref(session)
[Fri Apr 12 07:53:57.060330 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref
[Fri Apr 12 07:53:57.060340 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] self._plugin = self._do_create_plugin(session)
[Fri Apr 12 07:53:57.060358 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin
[Fri Apr 12 07:53:57.060367 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
[Fri Apr 12 07:53:57.060410 2019] [wsgi:error] [pid 1925:tid 140223984846592] [remote 10.5.0.6:46298] keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not determine a suitable URL for the plugin

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

I see this a lot on the current batch of SSL Mojo specs, targeting the -next charms

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

https://github.com/openstack-charmers/openstack-mojo-specs/issues/108

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

/var/log/gnocchi/gnocchi-api.log has the following at the time it fails setup

2019-04-12 09:56:50,576 [7775] INFO gnocchi.service: Gnocchi version 4.2.5
2019-04-12 09:56:50,587 [7775] INFO gnocchi.rest.app: WSGI config used: /etc/gnocchi/api-paste.ini
2019-04-12 09:56:50,904 [7775] WARNING keystonemiddleware._common.config: The option "configkey" in conf is not known to auth_token
2019-04-12 09:56:50,904 [7775] WARNING keystonemiddleware._common.config: The option "here" in conf is not known to auth_token
2019-04-12 09:56:50,904 [7775] WARNING keystonemiddleware._common.config: The option "__file__" in conf is not known to auth_token
2019-04-12 09:56:50,907 [7775] WARNING keystonemiddleware.auth_token: AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.
2019-04-12 09:56:50,912 [7775] WARNING oslo_config.cfg: Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken".
2019-04-12 09:56:50,934 [7775] INFO gnocchi.common.ceph: Ceph storage backend use 'rados' python library
2019-04-12 09:56:51,397 [7775] INFO gnocchi.common.ceph: Ceph storage backend use 'rados' python library
2019-04-12 09:56:53,029 [7776] INFO gnocchi.service: Gnocchi version 4.2.5
2019-04-12 09:56:53,040 [7776] INFO gnocchi.rest.app: WSGI config used: /etc/gnocchi/api-paste.ini
2019-04-12 09:56:53,370 [7776] WARNING keystonemiddleware._common.config: The option "configkey" in conf is not known to auth_token
2019-04-12 09:56:53,370 [7776] WARNING keystonemiddleware._common.config: The option "here" in conf is not known to auth_token
2019-04-12 09:56:53,370 [7776] WARNING keystonemiddleware._common.config: The option "__file__" in conf is not known to auth_token
2019-04-12 09:56:53,373 [7776] WARNING keystonemiddleware.auth_token: AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.
2019-04-12 09:56:53,380 [7776] WARNING oslo_config.cfg: Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken".
2019-04-12 09:56:53,424 [7776] WARNING keystoneauth.identity.generic.base: Failed to discover available identity versions when contacting https://10.5.0.36:35357. Attempting to parse version from URL.

Chris MacNaughton (chris.macnaughton)
summary: Running Ceilometer's setup action crashes on gnocchi shortly after
- deployment
+ deployment when using TLS
Alex Kavanagh (ajkavanagh)
Changed in charm-gnocchi:
status: New → Triaged
importance: Undecided → High
Chris MacNaughton (chris.macnaughton)
Changed in openstack-mojo-specs:
status: New → Fix Committed
assignee: nobody → Chris MacNaughton (chris.macnaughton)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-gnocchi (master)

Change abandoned by "Billy Olsen <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-gnocchi/+/667918
Reason: Change has not been updated to address feedback in > 180 days

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.