Cinder not configured to provide telemetry to ceilometer

If i'm not mistaken the following simple change to cinder.conf does the trick:

[oslo_messaging_notifications]
driver = messagingv2

As for cinder-volume-usage-audit would it be ill-advised to just always run it? It seems like a fairly low-touch action since it runs every 5 minutes. If so we could just add this to the cinder charm natively and not have to worry about extra relations, config and subordinates.

Sounds about right. Good input on a more simplistic way to solve it.

I have these concerns for such an approach:
a) Notifications sent by cinder-volume-usage-audit will never be consumed and might at some point create trouble on the end users rabbitmq-cluster if they do not have ceilometer running.
b) The audit gathering process might put an extra toll on the cinder nodes and become a unnecessary resource drain for end users not using ceilometer.

For a):
If ceilometer is designed in such a way that the consuming agent creates the queue and not the producer, I guess the messages will not be sent/stored unless the central ceilometer agent is present. We will probably have to check for problematic and unnecessary error logging on the cinder nodes though?

For b):
I guess we will have to evaluate and measure what amount of resources this process hogs in a sizable cluster.

I just ried a test. The cinder charm is already configured to provide notifications ("[oslo_messaging_notifications] driver = messagingv2") but I am not seeing all the expected meters as per http://docs.openstack.org/admin-guide/telemetry-measurements.html#openstack-block-storage:

ubuntu@xenial$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| d154a1e8-c903-45d1-867c-8c82dc5f9c2d | available | - | 1 | - | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
ubuntu@xenial$ ceilometer meter-list| grep volume
| volume.size | gauge | GB | d154a1e8-c903-45d1-867c-8c82dc5f9c2d | 0a91d38c85c9437caa92b8ac10e1eebe | 71aaf5f27079464fa6f82e23b0360a4a |
ubuntu@xenial$ ceilometer sample-list| grep volume
| 09458900-e3d9-11e6-991e-fa163e150a2a | d154a1e8-c903-45d1-867c-8c82dc5f9c2d | volume.size | gauge | 1.0 | GB | 2017-01-26T15:06:39.201000 |
| 08656f00-e3d9-11e6-991e-fa163e150a2a | d154a1e8-c903-45d1-867c-8c82dc5f9c2d | volume.size | gauge | 1.0 | GB | 2017-01-26T15:06:37.723000 |
ubuntu@xenial$ nova volume-attach bb5a6774-8620-4a7d-aa85-e590c6dfa6b0 d154a1e8-c903-45d1-867c-8c82dc5f9c2d
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | d154a1e8-c903-45d1-867c-8c82dc5f9c2d |
| serverId | bb5a6774-8620-4a7d-aa85-e590c6dfa6b0 |
| volumeId | d154a1e8-c903-45d1-867c-8c82dc5f9c2d |
+----------+--------------------------------------+
ubuntu@xenial$ ceilometer meter-list| grep volume
| volume.size | gauge | GB | d154a1e8-c903-45d1-867c-8c82dc5f9c2d | 0a91d38c85c9437caa92b8ac10e1eebe |
juju ssh cinder/0 sudo cinder-volume-usage-audit
Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
2017-01-26 15:25:24.008 13018 WARNING oslo_config.cfg [req-21c4fbd0-ee80-4b19-8ea1-a978c354cb6e - - - - -] Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py:241: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
Connection to 2001:db8:0:1:f816:3eff:fef0:165c closed.
ubuntu@xenial$ ceilometer meter-list| grep volume
| volume.size | gauge | GB | d154a1e8-c903-45d1-867c-8c82dc5f9c2d | 0a91d38c85c9437caa92b8ac10e1eebe |

I enabled debug logging for the volume audit and it is apparently not detecting any volumes even though i do have one:

2017-01-26 16:23:13.212 19385 DEBUG cinder [req-e4437c8d-3d19-4bb7-aab7-7903ee61bdfd - - - - -] Found 0 volumes main /usr/lib/python2.7/dist-packages/cinder/cmd/volume_usage_audit.py:106
2017-01-26 16:23:13.246 19385 DEBUG cinder [req-e4437c8d-3d19-4bb7-aab7-7903ee61bdfd - - - - -] Found 0 snapshots main /usr/lib/python2.7/dist-packages/cinder/cmd/volume_usage_audit.py:177

My bad, the default audit window is too short. This works:

root@juju-853dc7-default-4:~# cinder-volume-usage-audit --end_time "2017-02-01 00:00:00" 2>&1| grep Found
2017-01-26 21:25:23.582 18440 DEBUG cinder [req-90f87894-ba96-48d6-be43-f1c9ff8e1c8b - - - - -] Found 1 volumes main /usr/lib/python2.7/dist-packages/cinder/cmd/volume_usage_audit.py:106
2017-01-26 21:25:23.888 18440 DEBUG cinder [req-90f87894-ba96-48d6-be43-f1c9ff8e1c8b - - - - -] Found 0 snapshots main /usr/lib/python2.7/dist-packages/cinder/cmd/volume_usage_audit.py:177

Still only have volume.size meter though so something still not quite right.

Ok something is definitely out of kilter here. While the openstack docs say that there are numerous meters defined for volume, the trunk meters.yaml file only defines one:

https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml#L33

Which presumably explains why no matter what events i trigger i only get volume.size as a meter.

ftr disk usage meters seems to be working fine - http://pastebin.ubuntu.com/23874037/

I completely missed the fact that a cron entry for running cinder-volume-usage-audit was added to the 16.10 charms (see LP: #1623144). This allows the user to adjust the size of the volume audit window via charm config but does not, however, provide any configurability for the cron frequency.

I am going to close this bug since as far as I can tell the cinder charm is adequately configuring cinder to provide metrics to ceilometer. Please re-open if you disagree.