openstack-ansible

os_panko missing. Ceilometer event.sample queue stuck with no consumers

Bug #1624789 reported by Paulo Matias
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
openstack-ansible
Fix Released
High
Unassigned

Bug Description

When setting up Ceilometer with Gnocchi as its only dispatcher, the rabbitmq /ceilometer vhost's event.sample queue is filled up with messages which are never consumed. The rabbitmq admin panel points out the queue has no consumers.

Gnocchi does not store events (it only stores metrics), however if someone doesn't deploy a db backend (nor Panko, for which we still don't have a playbook), it is dangerous that this queue might overwhelm Rabbit after getting stuck with too many messages.

If I remember well, the last week (before RC1) ceilometer-collector was consuming this queue and discarding its messages. I didn't investigate yet if the new behaviour was caused by an OSA change or by an upstream change.

Revision history for this message
Paulo Matias (paulo-matias) wrote :

This is fixed by pulling https://review.openstack.org/#/c/371244 and configuring the event pipeline as follows:

# --[cut]--

ceilometer_event_pipeline_yaml_overrides:
  sources: []
  sinks: []

# --[cut]--

https://review.openstack.org/#/c/371244 should be merged and this setting should be documented.

Jean-Philippe Evrard (jean-philippe-evrard)
Changed in openstack-ansible:
status: New → Confirmed
importance: Undecided → Medium
importance: Medium → High
Revision history for this message
Steve Lewis (steve-lewis) wrote :

We shouldn't use overrides in the role or integrated gate. I'd like to find what changed in Ceilometer and consider alternatives, even if it is to specify a /dev/null file dispatcher.

Revision history for this message
Paulo Matias (paulo-matias) wrote :

I'm not entirely sure the issue was not occurring before. Maybe I just didn't observe it until now.

In my point of view what is delicate about this issue is that the user might be using Gnocchi and MongoDB simultaneously, and in this case he might want to store events to MongoDB. We should probably account for this option.

All in all, I agree that asking the user to set an override is not the definite solution here. As you said, if the deployer specifies no place to store events, it makes much more sense to discard the events without requiring additional intervention from the deployer.

Jean-Philippe Evrard (jean-philippe-evrard)
tags: added: newton-rc-potential
Revision history for this message
Steve Lewis (steve-lewis) wrote :

Asking the user to set an override is an option, but it seems like the current state of the world is that events should be sent to the Panko service.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/426325

Changed in openstack-ansible:
assignee: nobody → Steve Lewis (steve-lewis)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_ceilometer (master)

Reviewed: https://review.openstack.org/426325
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_ceilometer/commit/?id=949df106b03e570126a882e908c80146bf768cb7
Submitter: Jenkins
Branch: master

commit 949df106b03e570126a882e908c80146bf768cb7
Author: Steve Lewis <email address hidden>
Date: Fri Jan 27 09:36:27 2017 -0800

    Update event_dispatcher when gnocchi enabled

    Per the current Ceilometer docs[1] event_dispatcher can be
    set to gnocchi.

    When gnocchi is enabled this should ensure messages are not
    accumulated endlessly in RabbitMQ.

    [1] http://docs.openstack.org/developer/ceilometer/install/dbreco.html#moving-from-ceilometer-to-gnocchi

    Change-Id: I760d135f4abd89cac3ae66f002a24a083addafb7
    Partial-Bug: 1624789

Steve Lewis (steve-lewis)
Changed in openstack-ansible:
assignee: Steve Lewis (steve-lewis) → nobody
summary: - Ceilometer event.sample queue stuck with no consumers
+ os_panko missing. Ceilometer event.sample queue stuck with no consumers
Revision history for this message
Steve Lewis (steve-lewis) wrote :

We need a role created for the panko project to resolve this isssue.

Miguel Alejandro Cantu (miguel-cantu)
Changed in openstack-ansible:
status: In Progress → Confirmed
Revision history for this message
Jesse Pretorius (jesse-pretorius) wrote :

There is now a Panko role available: https://github.com/openstack/openstack-ansible-os_panko

Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote :

I guess since we have panko role, this bug can be considered as resolved.

Changed in openstack-ansible:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.