Rabbitmq app fails to start due to mnesia dir for plugins never cleaned up
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
Critical
|
Bogdan Dobrelya | ||
5.1.x |
Won't Fix
|
Critical
|
Denis Meltsaykin | ||
6.0.x |
Won't Fix
|
Critical
|
Denis Meltsaykin |
Bug Description
How to reproduce:
1) Move master to node-2
2) wait for ostf ha passed
3) kill the node-2 and start count for failover time in a wait loop (wait for ostf ha passed)
4) power on node-2 and wait for it joined the rabbit cluster
5) repeat 1-4
Expected: the node-2 will always rejoin cluster after the step #4.
Actual: after few iterations, the node-2 ended up in bad state - cannot start rabbit app, even mnesia erasing doesn't help.
Here is the output of the manual check of the action start with OCF wrapper command:
ocf_handler_
http://
Here is the output of commands doing start rabbitmq-server in foreground and try to start_app:
# RABBITMQ_
(note, beam starting as always and kernel module reported as running)
# rabbitmqctl start_app
Starting node 'rabbit@node-2' ...
BOOT FAILED
===========
Error description:
{case_
Log files (may contain more information):
/<email address hidden>
/<email address hidden>
Stack trace:
[{rabbit_
{rabbit_
{rabbit_
{rabbit_
{rabbit,
{rabbit,
{rpc,
Error: {rabbit,
This happens due to incomplete mnesia cleaning method in OCF script:
# rm -rf /var/lib/
This is not enougth and may cause reported error!
Plugins dir should be erased as well:
# rm -rf /var/lib/
After that, If beam restarted, rabbit app will start w/o issues:
# rabbitmqctl start_app
Starting node 'rabbit@node-2' ...
...done.
ISO info:
build_id: 2015-05-20_08-41-33
build_number: '441'
Changed in fuel: | |
importance: | Undecided → Critical |
milestone: | none → 6.1 |
status: | New → Confirmed |
assignee: | nobody → Bogdan Dobrelya (bogdando) |
Changed in fuel: | |
status: | Confirmed → Triaged |
summary: |
- Rabbitmq app fails to start + Rabbitmq app fails to start due to mnesia dir for plugins never cleaned + up |
description: | updated |
tags: | added: on-verification |
Fix proposed to branch: master /review. openstack. org/184971
Review: https:/