Wallaby fails to run ceilometer-upgrade due to policy restriction

Bug #1932975 reported by Boris Lukashev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Ceilometer Charm
New
Undecided
Unassigned

Bug Description

When attempting to run ceilometer-upgrade on the leader unit, the request fails with a 503:
```
ubuntu@juju-4e82f9-2-lxd-18:~$ sudo ceilometer-upgrade --debug
2021-06-18 23:10:51.352 233004 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python3/dist-packages/ceilometer/cmd/storage.py:42
2021-06-18 23:10:58.341 233004 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException: {"message": "The server is currently unavailable. Please try again at a later time.<br /><br />\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} (HTTP 503)
2021-06-18 23:10:58.341 233004 ERROR ceilometer Traceback (most recent call last):
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2021-06-18 23:10:58.341 233004 ERROR ceilometer sys.exit(upgrade())
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3/dist-packages/ceilometer/cmd/storage.py", line 49, in upgrade
2021-06-18 23:10:58.341 233004 ERROR ceilometer tenacity.Retrying(
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 409, in call
2021-06-18 23:10:58.341 233004 ERROR ceilometer do = self.iter(retry_state=retry_state)
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 356, in iter
2021-06-18 23:10:58.341 233004 ERROR ceilometer return fut.result()
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
2021-06-18 23:10:58.341 233004 ERROR ceilometer return self.__get_result()
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
2021-06-18 23:10:58.341 233004 ERROR ceilometer raise self._exception
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 412, in call
2021-06-18 23:10:58.341 233004 ERROR ceilometer result = fn(*args, **kwargs)
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3/dist-packages/ceilometer/gnocchi_client.py", line 266, in upgrade_resource_types
2021-06-18 23:10:58.341 233004 ERROR ceilometer gnocchi.resource_type.get(name=name)
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3/dist-packages/gnocchiclient/v1/resource_type.py", line 43, in get
2021-06-18 23:10:58.341 233004 ERROR ceilometer return self._get(self.url + name,
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py", line 37, in _get
2021-06-18 23:10:58.341 233004 ERROR ceilometer return self.client.api.get(*args, **kwargs)
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
2021-06-18 23:10:58.341 233004 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2021-06-18 23:10:58.341 233004 ERROR ceilometer File "/usr/lib/python3/dist-packages/gnocchiclient/client.py", line 52, in request
2021-06-18 23:10:58.341 233004 ERROR ceilometer raise exceptions.from_response(resp, method)
2021-06-18 23:10:58.341 233004 ERROR ceilometer gnocchiclient.exceptions.ClientException: {"message": "The server is currently unavailable. Please try again at a later time.<br /><br />\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} (HTTP 503)
2021-06-18 23:10:58.341 233004 ERROR ceilometer
```
Concurrently, keystone reports:
```
    (py.warnings): 2021-06-18 23:09:55,465 WARNING /usr/lib/python3/dist-packages/oslo_policy/policy.py:1054: UserWarning: Policy identity:list_domains failed scope check. The token used to make the request was domain scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required

```
indicating that the way gnocchi is set up doesnt comply with the new policy. The action works if run with `--skip-gnocchi-resource-types`

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.