In a yoga/focal testrun (https://solutions.qa.canonical.com/v2/testruns/b20906d4-122b-4d16-870d-754b10a7d906), ceilometer ignores the ceilometer-upgrade action which causes the testrun to fail:
```
2023-01-08-23:55:20 root DEBUG [localhost]: juju status -m foundations-maas:openstack --format=yaml
2023-01-08-23:55:32 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s ceilometer/0 --query 'workload-message=="Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi" || workload-status == "active"'
2023-01-08-23:55:33 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s ceilometer/1 --query 'workload-message=="Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi" || workload-status == "active"'
2023-01-08-23:55:33 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s ceilometer/2 --query 'workload-message=="Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi" || workload-status == "active"'
2023-01-08-23:55:34 root DEBUG [localhost]: juju status -m foundations-maas:openstack --format=yaml
2023-01-08-23:55:41 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s gnocchi/0 --query 'workload-message=="Unit is ready"'
2023-01-08-23:55:41 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s gnocchi/1 --query 'workload-message=="Unit is ready"'
2023-01-08-23:55:42 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s gnocchi/2 --query 'workload-message=="Unit is ready"'
2023-01-08-23:55:42 root DEBUG [localhost]: juju run -m foundations-maas:openstack --format=yaml ceilometer/leader ceilometer-upgrade --max-wait 14400s
2023-01-08-23:55:50 root DEBUG [localhost]: juju wait -m foundations-maas:openstack -t 14400 --workload --machine-error-timeout 1800 --retry_errors 0 --verbose -x openstack-service-checks -x prometheus-ceph-exporter -x octavia -x prometheus-openstack-exporter -x landscape-client -x juju-lint
2023-01-08-23:56:00 root DEBUG DEBUG:root:ceilometer/0 workload status is blocked since 2023-01-08 23:04:20+00:00
2023-01-08-23:56:00 root DEBUG DEBUG:root:ceilometer/1 workload status is blocked since 2023-01-08 23:04:18+00:00
2023-01-08-23:56:00 root DEBUG DEBUG:root:ceilometer/2 workload status is blocked since 2023-01-08 23:04:13+00:00
2023-01-08-23:56:10 root DEBUG DEBUG:root:ceilometer/0 workload status is blocked since 2023-01-08 23:04:20+00:00
2023-01-08-23:56:10 root DEBUG DEBUG:root:ceilometer/1 workload status is blocked since 2023-01-08 23:04:18+00:00
2023-01-08-23:56:10 root DEBUG DEBUG:root:ceilometer/2 workload status is blocked since 2023-01-08 23:04:13+00:00
2023-01-08-23:56:18 root DEBUG DEBUG:root:ceilometer/0 workload status is blocked since 2023-01-08 23:04:20+00:00
2023-01-08-23:56:18 root DEBUG DEBUG:root:ceilometer/1 workload status is blocked since 2023-01-08 23:04:18+00:00
```
In the ceilometer logs, we see a lot of keystone connection errors:
```
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base During handling of the above exception, another exception occurred:
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base Traceback (most recent call last):
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 136, in _do_create_plugin
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base disc = self.get_discovery(session,
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 608, in get_discovery
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base return discover.get_discovery(session=session, url=url,
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 1452, in get_discovery
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base disc = Discover(session, url, authenticated=authenticated)
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 535, in __init__
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base self._data = get_version_data(session, url,
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 102, in get_version_data
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base resp = session.get(url, headers=headers, authenticated=authenticated)
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1141, in get
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base return self.request(url, 'GET', **kwargs)
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 931, in request
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base resp = send(**kwargs)
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1026, in _send_request
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base raise exceptions.SSLError(msg)
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://keystone.silo5.lab1.solutionsqa:5000: HTTPSConnectionPool(host='keystone.silo5.lab1.solutionsqa', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)')))
```
ceilometer-agent/4 is also blocked at the time when the ceilometer-upgrade action is run, which could be part of the issue.
Configs and crashdumps can be found here:
https://oil-jenkins.canonical.com/artifacts/b20906d4-122b-4d16-870d-754b10a7d906/index.html