Cloudkitty horizon module gets 404 from keystone

Operating system distribution and version:
Ubuntu 16.04

Kolla-Ansible package version:
7.0.1 (7.0.1)

install_type:
source

Are you using official images from Docker Hub or self built?
Dockerhub

I enabled cloudkitty and deploy it with kolla-ansible.

When I try to use cloudkitty dashboard, Horizon gets 404 from keystone.

For example: /admin/rating_modules/ request

[Thu Mar 28 18:26:55.431156 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] Internal Server Error: /admin/rating_modules/
[Thu Mar 28 18:26:55.431647 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] Traceback (most recent call last):
[Thu Mar 28 18:26:55.431941 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
[Thu Mar 28 18:26:55.432336 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] response = get_response(request)
[Thu Mar 28 18:26:55.432724 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
[Thu Mar 28 18:26:55.433078 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] response = self.process_exception_by_middleware(e, request)
[Thu Mar 28 18:26:55.433520 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
[Thu Mar 28 18:26:55.433854 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Thu Mar 28 18:26:55.434263 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec
[Thu Mar 28 18:26:55.434639 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return view_func(request, *args, **kwargs)
[Thu Mar 28 18:26:55.435110 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/horizon/decorators.py", line 52, in dec
[Thu Mar 28 18:26:55.435783 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return view_func(request, *args, **kwargs)
[Thu Mar 28 18:26:55.436129 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec
[Thu Mar 28 18:26:55.436505 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return view_func(request, *args, **kwargs)
[Thu Mar 28 18:26:55.436786 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/horizon/decorators.py", line 113, in dec
[Thu Mar 28 18:26:55.437380 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return view_func(request, *args, **kwargs)
[Thu Mar 28 18:26:55.437711 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
[Thu Mar 28 18:26:55.438163 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return self.dispatch(request, *args, **kwargs)
[Thu Mar 28 18:26:55.438671 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
[Thu Mar 28 18:26:55.439236 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return handler(request, *args, **kwargs)
[Thu Mar 28 18:26:55.439640 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/horizon/tables/views.py", line 223, in get
[Thu Mar 28 18:26:55.440110 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] handled = self.construct_tables()
[Thu Mar 28 18:26:55.440402 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/horizon/tables/views.py", line 214, in construct_tables
[Thu Mar 28 18:26:55.440775 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] handled = self.handle_table(table)
[Thu Mar 28 18:26:55.441197 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/horizon/tables/views.py", line 123, in handle_table
[Thu Mar 28 18:26:55.441693 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] data = self._get_data_dict()
[Thu Mar 28 18:26:55.442163 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/horizon/tables/views.py", line 252, in _get_data_dict
[Thu Mar 28 18:26:55.442766 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] self._data = {self.table_class._meta.name: self.get_data()}
[Thu Mar 28 18:26:55.443301 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkittydashboard/dashboards/admin/modules/views.py", line 34, in get_data
[Thu Mar 28 18:26:55.443818 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] self.request).rating.get_module()['modules']
[Thu Mar 28 18:26:55.444332 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkittyclient/v1/rating/__init__.py", line 45, in get_module
[Thu Mar 28 18:26:55.444744 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return self.api_client.get(url).json()
[Thu Mar 28 18:26:55.445194 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 328, in get
[Thu Mar 28 18:26:55.445531 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return self.request(url, 'GET', **kwargs)
[Thu Mar 28 18:26:55.445942 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 213, in request
[Thu Mar 28 18:26:55.446283 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return self.session.request(url, method, **kwargs)
[Thu Mar 28 18:26:55.446663 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 684, in request
[Thu Mar 28 18:26:55.447008 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] auth_headers = self.get_auth_headers(auth)
[Thu Mar 28 18:26:55.447450 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers
[Thu Mar 28 18:26:55.447830 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return auth.get_headers(self, **kwargs)
[Thu Mar 28 18:26:55.448025 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
[Thu Mar 28 18:26:55.448057 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] token = self.get_token(session)
[Thu Mar 28 18:26:55.448161 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
[Thu Mar 28 18:26:55.448401 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return self.get_access(session).auth_token
[Thu Mar 28 18:26:55.448607 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
[Thu Mar 28 18:26:55.448639 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] self.auth_ref = self.get_auth_ref(session)
[Thu Mar 28 18:26:55.448875 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref
[Thu Mar 28 18:26:55.449075 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] authenticated=False, log=False, **rkwargs)
[Thu Mar 28 18:26:55.449172 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 1019, in post
[Thu Mar 28 18:26:55.449289 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] return self.request(url, 'POST', **kwargs)
[Thu Mar 28 18:26:55.449489 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 869, in request
[Thu Mar 28 18:26:55.449531 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] raise exceptions.from_response(resp, method, url)
[Thu Mar 28 18:26:55.449702 2019] [wsgi:error] [pid 22:tid 139915245643520] [remote 192.168.100.133:57244] NotFound: Not Found (HTTP 404) (Request-ID: req-fff5aa6d-540c-424e-95ce-3854a8404974)

I found that cloudkitty client send POST /auth/tokens to keystone instead of /v3/auth/tokens. Cloudkitty client gets OPENSTACK_KEYSTONE_URL from local_settings and use it as auth_url directly.

https://github.com/openstack/cloudkitty-dashboard/blob/8942b9a2a64f8db960fdc76ee55838a9f0650e3e/cloudkittydashboard/api/cloudkitty.py#L30

I talk with cloudkitty guys on IRC and they say "v3" is supposed to be part of the keystone url: https://docs.openstack.org/horizon/latest/admin/customize-configure.html#configure-the-dashboard

I change it and test it with rocky release and it works.

I think we need the change here: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/horizon/defaults/main.yml#L93

with:
horizon_keystone_url: "{{ keystone_internal_url }}/v3/"