Gnocchi fails to receive metrics after loading gnocchi:// publisher fails
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ceilometer |
New
|
Undecided
|
Yadnesh Kulkarni |
Bug Description
The archive policy used in the configuration is `ceilometer-high`
~~~
# cat pipeline.yaml
---
sources:
- name: meter_source
meters:
- "*"
sinks:
- meter_sink
sinks:
- name: meter_sink
publishers:
- gnocchi:
- notifier:
# cat event_pipeline.yaml
---
sources:
- name: event_source
events:
- "*"
sinks:
- event_sink
sinks:
- name: event_sink
transformers:
triggers:
publishers:
- gnocchi:
- notifier:
~~~
No such archive policy exists in gnocchi which should've been generated during "ceilometer-
However, it doesn't complain/log anything about the incoming metrics having an undefined archive policy.
~~~
$ openstack metric archive-policy list
+------
| name | back_window | definition | aggregation_methods |
+------
| bool | 3600 | - timespan: 365 days, 0:00:00, granularity: 0:00:01, points: 31536000 | last |
| high | 0 | - timespan: 1:00:00, granularity: 0:00:01, points: 3600 | min, mean, count, max, sum, std |
| | | - timespan: 7 days, 0:00:00, granularity: 0:01:00, points: 10080 | |
| | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | |
| low | 0 | - timespan: 30 days, 0:00:00, granularity: 0:05:00, points: 8640 | min, mean, count, max, sum, std |
| medium | 0 | - timespan: 7 days, 0:00:00, granularity: 0:01:00, points: 10080 | min, mean, count, max, sum, std |
| | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | |
+------
~~~
Upon restarting notification agent on one of the ctrl nodes, the missing policies were created after which Gnocchi starts processing metrics
~~~
$ openstack metric archive-policy list
+------
| name | back_window | definition | aggregation_methods |
+------
| bool | 3600 | - timespan: 365 days, 0:00:00, granularity: 0:00:01, points: 31536000 | last |
| ceilometer-high | 0 | - timespan: 1:00:00, granularity: 0:00:01, points: 3600 | mean |
| | | - timespan: 1 day, 0:00:00, granularity: 0:01:00, points: 1440 | |
| | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | |
| ceilometer-
| | | - timespan: 1 day, 0:00:00, granularity: 0:01:00, points: 1440 | |
| | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | |
| ceilometer-low | 0 | - timespan: 30 days, 0:00:00, granularity: 0:05:00, points: 8640 | mean |
| ceilometer-low-rate | 0 | - timespan: 30 days, 0:00:00, granularity: 0:05:00, points: 8640 | mean, rate:mean |
| high | 0 | - timespan: 1:00:00, granularity: 0:00:01, points: 3600 | mean, count, max, min, sum, std |
| | | - timespan: 7 days, 0:00:00, granularity: 0:01:00, points: 10080 | |
| | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | |
| low | 0 | - timespan: 30 days, 0:00:00, granularity: 0:05:00, points: 8640 | mean, count, max, min, sum, std |
| medium | 0 | - timespan: 7 days, 0:00:00, granularity: 0:01:00, points: 10080 | mean, count, max, min, sum, std |
| | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | |
+------
~~~
Changed in ceilometer: | |
assignee: | nobody → Yadnesh Kulkarni (ykulkarn) |
It seems that during deployment, keystone didn't respond to ceilometer's request to obtain gnocchi endpoint using gnocchiclient [1] identity. generic. base [-] Failed to discover available identity versions when contacting http:// 172.17. 1.82:5000. Attempting to parse version from URL.: keystoneauth1. exceptions. connection. ConnectTimeout: Request to http:// 172.17. 1.82:5000 timed out pipeline. base [-] Unable to load publisher gnocchi: //?filter_ project= service& archive_ policy= ceilometer- high: keystoneauth1. exceptions. discovery. DiscoveryFailur e: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Request to http:// 172.17. 1.82:5000 timed out pipeline. base Traceback (most recent call last): pipeline. base File "/usr/lib/ python3. 9/site- packages/ urllib3/ connectionpool. py", line 445, in _make_request pipeline. base six.raise_from(e, None) pipeline. base File "<string>", line 3, in raise_from pipeline. base File "/usr/lib/ python3. 9/site- packages/ urllib3/ connectionpool. py", line 440, in _make_request pipeline. base httplib_response = conn.getresponse() pipeline. base File "/usr/lib64/ python3. 9/http/ client. py", line 1377, in getresponse pipeline. base response.begin() pipeline. base File "/usr/lib64/ python3. 9/http/ client. py", line 320, in begin pipeline. base version, status, reason = self._read_status() pipeline. base File "/usr/lib64/ python3. 9/http/ client. py", line 281, in _read_status pipeline. base line = str(self. fp.readline( _MAXLINE + 1), "iso-8859-1") pipeline. base File "/usr/lib64/ python3. 9/socket. py", line 704, in readinto pipeline. base return self._sock. recv_into( b) pipeline. base socket.timeout: timed out pipeline. base
~~~
2023-05-08 18:33:49.147 14 WARNING keystoneauth.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
2023-05-08 18:33:49.150 14 ERROR ceilometer.
~~~
Since ceilometer couldn't get gnocchiclient[2] with proper auth values, it couldn't create the necessary archive policies[3]
Restarting agent_notification service after deployment fixes this because by that time keystone is healthy and responding. This seems intermittent because ceilometer & gnocchi services
are spawned during step 4 & 5 till then keystone should be completely operational.
[1] https:/ /github. com/openstack/ ceilometer/ blob/stable/ wallaby/ ceilometer/ gnocchi_ client. py#L36- L39 /github. com/openstack/ ceilometer/ blob/stable/ wallaby/ ceilometer/ publisher/ gnocchi. py#L216- L217 /github. com/openstack/ ceilometer/ blob/stable...
[2] https:/
[3] https:/