Restart rabbit if can't list queues or found memory alert
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Confirmed
|
Medium
|
Fuel Documentation Team |
Bug Description
https:/
commit 599961e60858f18
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jun 10 13:44:53 2015 +0200
Restart rabbit if can't list queues or found memory alert
W/o this fix the dead end situation is possible
when the rabbit node have no free memory resources left
and the cluster blocks all publishing, by design.
But the app thinks "let's wait for the publish block have
lifted" and cannot recover.
The workaround is to monitor results
of crucial rabbitmqctl commands and restart the rabbit node,
if queues/
memory alarms found.
This is the similar logic as we have for the cases when
rabbitmqctl list_channels hangs. But the channels check is also
fixed to verify if the exit code>0 when the rabbit app is
running.
Additional checks added to the monitor also require extending
the timeout window for the monitor action from 60 to 180 seconds.
Besides that, this patch makes the monitor action to gather the
rabbit status and runtime stats, like consumed memory by all
queues of total Mem+Swap, total messages in all queues and
average queue consumer utilization. This info should help to
troubleshoot failures better.
DocImpact: ops guide. If any rabbitmq node exceeded its memory
threshold the publish became blocked cluster-wide, by design.
For such cases, this rabbit node would be recovered from the
raised memory alert and immediately stopped to be restarted
later by the pacemaker. Otherwise, this blocked publishing state
might never have been lifted, if the pressure persists from the
OpenStack apps side.
Closes-bug: #1463433
Change-Id: I91dec2d30d77b1
Signed-off-by: Bogdan Dobrelya <email address hidden>
(cherry picked from commit bf604f80d72f69e
Changed in fuel: | |
assignee: | nobody → Fuel Library Team (fuel-library) |
milestone: | none → 8.0 |
importance: | Undecided → Medium |
Changed in fuel: | |
assignee: | Fuel Library Team (fuel-library) → Fuel Documentation Team (fuel-docs) |
Changed in fuel: | |
status: | New → Confirmed |
tags: | added: area-docs |
This is for docs team. Also, it might not need any action since it's a backport for 6.1 (master was done in June 2015 )