OpenStack RabbitMQ Server Charm

Rabbitmq nrpe queue check should check for freshness

Bug #1898523 reported by Peter Sabaini
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack RabbitMQ Server Charm
Fix Released
Low
Peter Sabaini
OpenStack RabbitMQ Server Charm 21.01

Bug Description

The check_rabbitmq_queue will check a generated stats file, typically created via a cronjob /etc/cron.d/rabbitmq-stats

The check_rabbitmq_queues.py script operates on that generated file but does not check for recentness/freshness of this file. So if for any reason the rabbitmq-stats job gets stuck this check will proceed to happily proclaim rabbit queues OK

To catch this situation check_rabbitmq_queues.py should check if the cached stats file is recent, where "recent" could e.g. mean mtime not older than some multiple of the cronjob period time ago

Revision history for this message
Aurelien Lourot (aurelien-lourot) wrote :

https://review.opendev.org/#/c/757955

Changed in charm-rabbitmq-server:
assignee: nobody → Peter Sabaini (peter-sabaini)
James Page (james-page)
Changed in charm-rabbitmq-server:
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-rabbitmq-server (master)

Reviewed: https://review.opendev.org/757955
Committed: https://git.openstack.org/cgit/openstack/charm-rabbitmq-server/commit/?id=943f4f63ab256cba61ce9eceb473ed4d671f132e
Submitter: Zuul
Branch: master

commit 943f4f63ab256cba61ce9eceb473ed4d671f132e
Author: Peter Sabaini <email address hidden>
Date: Tue Oct 13 23:14:32 2020 +0200

    Fix: nrpe queue check should check for freshness

    Make the rabbitmq queue check also check if its input data file was
    recently updated. This input data is created via cronjob; if that gets
    stuck we might not actually be getting meaningful data.

    The charm supports configuring the check interval via a full cron time
    specification, so technically one could have that updated only once a
    year even if this doesn't make much sense in a monitoring scenario.

    Also fix a buglet in the nrpe update hook function: only deploy a
    queue check if the cron job hasn't been deconfigured by setting it to
    the empty string

    Change-Id: I60141397f39e3b1b0274230db8d984934c98a08d
    Closes-Bug: #1898523

Changed in charm-rabbitmq-server:
status: In Progress → Fix Committed
David Ames (thedac)
Changed in charm-rabbitmq-server:
milestone: none → 21.01
David Ames (thedac)
Changed in charm-rabbitmq-server:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.