I am using bionic/stein
We've seen this across multiple, fresh deployments.
For some reason, VMs are not getting all the statistics filled into Gnocchi.
I can see instance e9e24e54-7d0b-41a0-aa09-7249c606cfa9's ceilometer-agent log [0] and it is sending all the infos, including networking and disk usage.
However, this is what I can recover from gnocchi:
openstack metric resource show e9e24e54-7d0b-41a0-aa09-7249c606cfa9
+-----------------------+---------------------------------------------------------------------+
| Field | Value |
+-----------------------+---------------------------------------------------------------------+
| created_by_project_id | 35a9bdf29105436b83158cf81c96aa43 |
| created_by_user_id | 7a4549ad2ebf4db195fe947e1c8ff341 |
| creator | 7a4549ad2ebf4db195fe947e1c8ff341:35a9bdf29105436b83158cf81c96aa43 |
| ended_at | None |
| id | e9e24e54-7d0b-41a0-aa09-7249c606cfa9 |
| metrics | compute.instance.booting.time: a19cd16a-b965-44bb-939a-561f7bb3fc1a |
| | cpu: 7d4ace3b-3c81-4214-be62-c93e7f985e06 |
| | disk.ephemeral.size: b1856a97-5481-4331-9897-3dd586759ce9 |
| | disk.root.size: 85624f60-12ec-4ebd-b09c-a581fcfac0e4 |
| | memory.usage: ae5e0405-0dbf-414e-8508-05bcdebef932 |
| | memory: 215ab286-deeb-4870-b969-33521ae5da44 |
| | vcpus: b70db7cd-c689-49e2-a684-66dd033c40a1 |
| original_resource_id | e9e24e54-7d0b-41a0-aa09-7249c606cfa9 |
| project_id | 35a9bdf29105436b83158cf81c96aa43 |
| revision_end | None |
| revision_start | 2020-07-01T08:11:21.271557+00:00 |
| started_at | 2020-06-25T21:12:46.030361+00:00 |
| type | instance |
| user_id | a4df1ebe9e3e4ba5be76e86628404ad8 |
+-----------------------+---------------------------------------------------------------------+
Ceilometer log is pretty old (collected all this log today, but ceilometer-notification.log.1 last entries comes from 25/06):
2020-06-24 19:46:48.954 118459 WARNING ceilometer.pipeline.sample [-] metering data image.size for d693c40a-8bdc-4692-b4fe-67b70244005a @ 2020-06-24T19:46:48.830115+00:00 has no volume (volume: None), the sample will be dropped
2020-06-24 19:46:58.387 118462 WARNING ceilometer.publisher.gnocchi [-] filtered project not found in keystone, ignoring the filter_project option: keystoneauth1.exceptions.http.NotFound: 404 (HTTP 404)
2020-06-24 19:46:58.461 118462 ERROR ceilometer.publisher.gnocchi [-] Error creating resource d693c40a-8bdc-4692-b4fe-67b70244005a: Resource type image does not exist (HTTP 404): gnocchiclient.exceptions.ResourceTypeNotFound: Resource type image does not exist (
HTTP 404)
2020-06-24 19:48:18.528 118459 WARNING ceilometer.publisher.gnocchi [-] filtered project not found in keystone, ignoring the filter_project option: keystoneauth1.exceptions.http.NotFound: 404 (HTTP 404)
2020-06-24 19:48:18.627 118459 ERROR ceilometer.publisher.gnocchi [-] Error creating resource ffd2f57c-6b62-419b-967e-80954196ba0f: Resource type image does not exist (HTTP 404): gnocchiclient.exceptions.ResourceTypeNotFound: Resource type image does not exist (
HTTP 404)
2020-06-24 19:48:18.910 118465 WARNING ceilometer.publisher.gnocchi [-] filtered project not found in keystone, ignoring the filter_project option: keystoneauth1.exceptions.http.NotFound: 404 (HTTP 404)
2020-06-24 19:48:19.027 118465 ERROR ceilometer.publisher.gnocchi [-] Error creating resource ffd2f57c-6b62-419b-967e-80954196ba0f: Resource type image does not exist (HTTP 404): gnocchiclient.exceptions.ResourceTypeNotFound: Resource type image does not exist (
HTTP 404)
2020-06-24 19:50:44.441 118470 WARNING ceilometer.publisher.gnocchi [-] filtered project not found in keystone, ignoring the filter_project option: keystoneauth1.exceptions.http.NotFound: 404 (HTTP 404)
2020-06-24 19:50:44.906 118470 ERROR ceilometer.publisher.gnocchi [-] Error creating resource 9192f686-839a-4eb1-ae64-d61f5762b163: Resource type image does not exist (HTTP 404): gnocchiclient.exceptions.ResourceTypeNotFound: Resource type image does not exist (
HTTP 404)
2020-06-25 18:07:11.292 118462 ERROR ceilometer.publisher.gnocchi [-] Error creating resource 9192f686-839a-4eb1-ae64-d61f5762b163: Invalid input: required key not provided @ data['disk_format'] (HTTP 400): gnocchiclient.exceptions.BadRequest: Invalid input: req
uired key not provided @ data['disk_format'] (HTTP 400)
2020-06-25 18:08:38.758 118465 ERROR ceilometer.publisher.gnocchi [-] Error creating resource 9192f686-839a-4eb1-ae64-d61f5762b163: Invalid input: required key not provided @ data['disk_format'] (HTTP 400): gnocchiclient.exceptions.BadRequest: Invalid input: req
uired key not provided @ data['disk_format'] (HTTP 400)
----------------------------------------------------------------------------
[0]
2020-07-01 07:56:04.980 1015984 WARNING ceilometer.compute.pollsters [-] cpu_l3_cache statistic in not available for instance e9e24e54-7d0b-41a0-aa09-7249c606cfa9: ceilometer.compute.pollsters.NoVolumeException
2020-07-01 07:56:04.980 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.read.bytes in the context of some_pollsters
2020-07-01 07:56:04.984 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.incoming.bytes in the context of some_pollsters
2020-07-01 07:56:05.003 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.incoming.packets in the context of some_pollsters
2020-07-01 07:56:05.015 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.outgoing.packets in the context of some_pollsters
2020-07-01 07:56:05.046 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.outgoing.bytes in the context of some_pollsters
2020-07-01 07:56:05.055 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.write.requests in the context of some_pollsters
2020-07-01 07:56:05.060 1015984 INFO ceilometer.polling.manager [-] Polling pollster memory.usage in the context of some_pollsters
2020-07-01 07:56:05.063 1015984 INFO ceilometer.polling.manager [-] Polling pollster cpu in the context of some_pollsters
2020-07-01 07:56:05.076 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.write.bytes in the context of some_pollsters
2020-07-01 08:01:04.932 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.incoming.packets in the context of some_pollsters
2020-07-01 08:01:04.951 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.outgoing.packets in the context of some_pollsters
2020-07-01 08:01:04.955 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.write.bytes in the context of some_pollsters
2020-07-01 08:01:04.973 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.write.requests in the context of some_pollsters
2020-07-01 08:01:04.978 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.incoming.bytes in the context of some_pollsters
2020-07-01 08:01:04.984 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.outgoing.bytes in the context of some_pollsters
2020-07-01 08:01:04.988 1015984 INFO ceilometer.polling.manager [-] Polling pollster cpu_l3_cache in the context of some_pollsters
2020-07-01 08:01:05.007 1015984 WARNING ceilometer.compute.pollsters [-] cpu_l3_cache statistic in not available for instance e9e24e54-7d0b-41a0-aa09-7249c606cfa9: ceilometer.compute.pollsters.NoVolumeException
2020-07-01 08:01:05.007 1015984 INFO ceilometer.polling.manager [-] Polling pollster memory.usage in the context of some_pollsters
2020-07-01 08:01:05.010 1015984 INFO ceilometer.polling.manager [-] Polling pollster cpu in the context of some_pollsters
2020-07-01 08:01:05.013 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.read.requests in the context of some_pollsters
2020-07-01 08:01:05.016 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.read.bytes in the context of some_pollsters
2020-07-01 08:06:04.933 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.write.bytes in the context of some_pollsters
2020-07-01 08:06:04.960 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.read.bytes in the context of some_pollsters
2020-07-01 08:06:04.964 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.write.requests in the context of some_pollsters
2020-07-01 08:06:04.967 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.incoming.bytes in the context of some_pollsters
2020-07-01 08:06:04.972 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.read.requests in the context of some_pollsters
2020-07-01 08:06:04.980 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.outgoing.packets in the context of some_pollsters
2020-07-01 08:06:04.984 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.incoming.packets in the context of some_pollsters
2020-07-01 08:06:04.987 1015984 INFO ceilometer.polling.manager [-] Polling pollster memory.usage in the context of some_pollsters
2020-07-01 08:06:05.004 1015984 INFO ceilometer.polling.manager [-] Polling pollster cpu in the context of some_pollsters
2020-07-01 08:06:05.006 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.outgoing.bytes in the context of some_pollsters
2020-07-01 08:06:05.009 1015984 INFO ceilometer.polling.manager [-] Polling pollster cpu_l3_cache in the context of some_pollsters
2020-07-01 08:06:05.010 1015984 WARNING ceilometer.compute.pollsters [-] cpu_l3_cache statistic in not available for instance e9e24e54-7d0b-41a0-aa09-7249c606cfa9: ceilometer.compute.pollsters.NoVolumeException
2020-07-01 08:11:04.934 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.outgoing.packets in the context of some_pollsters
2020-07-01 08:11:04.944 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.outgoing.bytes in the context of some_pollsters
2020-07-01 08:11:04.964 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.read.bytes in the context of some_pollsters
2020-07-01 08:11:05.010 1015984 INFO ceilometer.polling.manager [-] Polling pollster cpu in the context of some_pollsters
2020-07-01 08:11:05.059 1015984 INFO ceilometer.polling.manager [-] Polling pollster memory.usage in the context of some_pollsters
2020-07-01 08:11:05.070 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.incoming.bytes in the context of some_pollsters
2020-07-01 08:11:05.081 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.write.requests in the context of some_pollsters
2020-07-01 08:11:05.099 1015984 INFO ceilometer.polling.manager [-] Polling pollster network.incoming.packets in the context of some_pollsters
2020-07-01 08:11:05.108 1015984 INFO ceilometer.polling.manager [-] Polling pollster cpu_l3_cache in the context of some_pollsters
2020-07-01 08:11:05.108 1015984 WARNING ceilometer.compute.pollsters [-] cpu_l3_cache statistic in not available for instance e9e24e54-7d0b-41a0-aa09-7249c606cfa9: ceilometer.compute.pollsters.NoVolumeException
2020-07-01 08:11:05.108 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.read.requests in the context of some_pollsters
2020-07-01 08:11:05.116 1015984 INFO ceilometer.polling.manager [-] Polling pollster disk.device.write.bytes in the context of some_pollsters
