Fuel for OpenStack

Rabbit ocf unable to cleanup mnesia in some conditions

Bug #1441226 reported by Sergey Yudin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Sergey Yudin
Fuel for OpenStack 6.1
5.1.x
Fix Committed
Critical
Bogdan Dobrelya
Fuel for OpenStack 5.1.1-updates
6.0.x
Fix Committed
Critical
Bogdan Dobrelya
Fuel for OpenStack 6.0-updates

Bug Description

In some rare conditions rabbit unable to cleanup mnesia because of logical bug in ocf.

Here is relevant logfile
http://paste2.org/MKHKE0Fx
as you can see it tries to check the status and rabbit is running there, but because of the bug it didn't try to kill rabbit and thats lead to "mnesia_unexpectedly_running" error.

Here is proposed solution
https://review.openstack.org/#/c/171253/

Tags: scale
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: nobody → Sergey Yudin (tsipa740)
status: New → In Progress
Bogdan Dobrelya (bogdando)
Changed in fuel:
importance: Undecided → High
milestone: none → 6.1
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Sergey Yudin (tsipa740) → Bogdan Dobrelya (bogdando)
Bogdan Dobrelya (bogdando)
Changed in fuel:
assignee: Bogdan Dobrelya (bogdando) → Sergey Yudin (tsipa740)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/171253
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=40dcf6661de26faaccee024ea287c03f96eb5f8f
Submitter: Jenkins
Branch: master

commit 40dcf6661de26faaccee024ea287c03f96eb5f8f
Author: Sergey V. Yudin <email address hidden>
Date: Tue Apr 7 15:30:29 2015 +0000

    Fix rabbit OCF status check for reset_mnesia

    In some conditions rabbit RA is unable to cleanup
    mnesia because of the "reverse logic" bug in ocf
    script, which is:

    * It tries to check the status and rabbit is running
    there
    * But because of the reverse logic bug it didn't try
    to kill rabbit
    * That leads to "mnesia_unexpectedly_running" error.

    Closes-Bug: #1441226

    Change-Id: I2c71935148ad842ee5682112af6fb4e06bcc794f

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.0)

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/171622

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/171623

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Raised to critical as this issue impacts the UX with unexpected mnesia resets while doing HA fail-over

Changed in fuel:
importance: High → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/171623
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=45e56302af69142d060428baa992876428e32506
Submitter: Jenkins
Branch: stable/5.1

commit 45e56302af69142d060428baa992876428e32506
Author: Sergey V. Yudin <email address hidden>
Date: Tue Apr 7 15:30:29 2015 +0000

    Fix rabbit OCF status check for reset_mnesia

    In some conditions rabbit RA is unable to cleanup
    mnesia because of the "reverse logic" bug in ocf
    script, which is:

    * It tries to check the status and rabbit is running
    there
    * But because of the reverse logic bug it didn't try
    to kill rabbit
    * That leads to "mnesia_unexpectedly_running" error.

    Closes-Bug: #1441226

    Change-Id: I2c71935148ad842ee5682112af6fb4e06bcc794f

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/6.0)

Reviewed: https://review.openstack.org/171622
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d970170569f99255ad483fa5b45801dded26392c
Submitter: Jenkins
Branch: stable/6.0

commit d970170569f99255ad483fa5b45801dded26392c
Author: Sergey V. Yudin <email address hidden>
Date: Tue Apr 7 15:30:29 2015 +0000

    Fix rabbit OCF status check for reset_mnesia

    In some conditions rabbit RA is unable to cleanup
    mnesia because of the "reverse logic" bug in ocf
    script, which is:

    * It tries to check the status and rabbit is running
    there
    * But because of the reverse logic bug it didn't try
    to kill rabbit
    * That leads to "mnesia_unexpectedly_running" error.

    Closes-Bug: #1441226

    Change-Id: I2c71935148ad842ee5682112af6fb4e06bcc794f

Leontiy Istomin (listomin)
tags: added: scale
Revision history for this message
Sergey Novikov (snovikov) wrote :

Verified on fuel-6.1-432-2015-05-18_03-43-53.iso.

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.