Ceilometer ignores ceilometer-upgrade action

Bug #2002334 reported by Bas de Bruijne
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Ceilometer Charm
New
Undecided
Unassigned

Bug Description

In a yoga/focal testrun (https://solutions.qa.canonical.com/v2/testruns/b20906d4-122b-4d16-870d-754b10a7d906), ceilometer ignores the ceilometer-upgrade action which causes the testrun to fail:

```
2023-01-08-23:55:20 root DEBUG [localhost]: juju status -m foundations-maas:openstack --format=yaml
2023-01-08-23:55:32 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s ceilometer/0 --query 'workload-message=="Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi" || workload-status == "active"'
2023-01-08-23:55:33 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s ceilometer/1 --query 'workload-message=="Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi" || workload-status == "active"'
2023-01-08-23:55:33 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s ceilometer/2 --query 'workload-message=="Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi" || workload-status == "active"'
2023-01-08-23:55:34 root DEBUG [localhost]: juju status -m foundations-maas:openstack --format=yaml
2023-01-08-23:55:41 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s gnocchi/0 --query 'workload-message=="Unit is ready"'
2023-01-08-23:55:41 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s gnocchi/1 --query 'workload-message=="Unit is ready"'
2023-01-08-23:55:42 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s gnocchi/2 --query 'workload-message=="Unit is ready"'
2023-01-08-23:55:42 root DEBUG [localhost]: juju run -m foundations-maas:openstack --format=yaml ceilometer/leader ceilometer-upgrade --max-wait 14400s
2023-01-08-23:55:50 root DEBUG [localhost]: juju wait -m foundations-maas:openstack -t 14400 --workload --machine-error-timeout 1800 --retry_errors 0 --verbose -x openstack-service-checks -x prometheus-ceph-exporter -x octavia -x prometheus-openstack-exporter -x landscape-client -x juju-lint
2023-01-08-23:56:00 root DEBUG DEBUG:root:ceilometer/0 workload status is blocked since 2023-01-08 23:04:20+00:00
2023-01-08-23:56:00 root DEBUG DEBUG:root:ceilometer/1 workload status is blocked since 2023-01-08 23:04:18+00:00
2023-01-08-23:56:00 root DEBUG DEBUG:root:ceilometer/2 workload status is blocked since 2023-01-08 23:04:13+00:00
2023-01-08-23:56:10 root DEBUG DEBUG:root:ceilometer/0 workload status is blocked since 2023-01-08 23:04:20+00:00
2023-01-08-23:56:10 root DEBUG DEBUG:root:ceilometer/1 workload status is blocked since 2023-01-08 23:04:18+00:00
2023-01-08-23:56:10 root DEBUG DEBUG:root:ceilometer/2 workload status is blocked since 2023-01-08 23:04:13+00:00
2023-01-08-23:56:18 root DEBUG DEBUG:root:ceilometer/0 workload status is blocked since 2023-01-08 23:04:20+00:00
2023-01-08-23:56:18 root DEBUG DEBUG:root:ceilometer/1 workload status is blocked since 2023-01-08 23:04:18+00:00
```

In the ceilometer logs, we see a lot of keystone connection errors:

```
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base During handling of the above exception, another exception occurred:
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base Traceback (most recent call last):
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 136, in _do_create_plugin
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base disc = self.get_discovery(session,
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 608, in get_discovery
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base return discover.get_discovery(session=session, url=url,
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 1452, in get_discovery
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base disc = Discover(session, url, authenticated=authenticated)
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 535, in __init__
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base self._data = get_version_data(session, url,
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 102, in get_version_data
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base resp = session.get(url, headers=headers, authenticated=authenticated)
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1141, in get
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base return self.request(url, 'GET', **kwargs)
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 931, in request
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base resp = send(**kwargs)
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1026, in _send_request
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base raise exceptions.SSLError(msg)
ceilometer/ceilometer-agent-notification.log:2023-01-08 23:51:43.480 277681 ERROR ceilometer.pipeline.base keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://keystone.silo5.lab1.solutionsqa:5000: HTTPSConnectionPool(host='keystone.silo5.lab1.solutionsqa', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)')))
```

ceilometer-agent/4 is also blocked at the time when the ceilometer-upgrade action is run, which could be part of the issue.

Configs and crashdumps can be found here:
https://oil-jenkins.canonical.com/artifacts/b20906d4-122b-4d16-870d-754b10a7d906/index.html

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.