Fuel for OpenStack

fuel-rabbit-fence is not able to manage its pid file

Bug #1614963 reported by Dmitry Burmistrov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Dmitry Burmistrov
Fuel for OpenStack 10.0
8.0.x
Fix Released
High
Oleksiy Molchanov
Fuel for OpenStack 8.0-mu-6
Mitaka
Fix Released
High
Dmitry Burmistrov
Fuel for OpenStack 9.1

Bug Description

Detailed bug description:
 9.0 ISO
 Syslog on nodes with rabbit-fence is full of this errors: http://paste.openstack.org/show/561296/
 In case of incorrect pid-file rabbit-fence can't work - it is being respawned by upstart (you can check "service fuel-rabbit-fence status" under "watch" tool).
 It affects re-applying "rabbitmq/rabbitmq.pp" modular (when services are refreshed) - command "rabbitmqctl -q list_users" starts failing sometimes.
Steps to reproduce:
 Deploy cluster.
 echo "$RANDOM" > /var/run/rabbitmq/rabbit-fence.pid
 service fuel-rabbit-fence restart
 watch service fuel-rabbit-fence status
Expected results:
 watch is showing the same PID (process is not falling down)
Actual result:
 process is respawning
Workaround:
 add "rm -f /var/run/rabbitmq/rabbit-fence.pid" at the end of 'pre-start script' in /etc/init/fuel-rabbit-fence.conf
Description of the environment:
 Operation system: Ubuntu 14.04

Ivan Berezovskiy (iberezovskiy)
Changed in fuel:
importance: Undecided → High
assignee: nobody → Dmitry Burmistrov (dmburmistrov)
milestone: none → 10.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/357893

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/357895

Maksim Malchuk (mmalchuk)
no longer affects: fuel/newton
Maksim Malchuk (mmalchuk)
tags: added: area-library area-python
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/357895
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=41b1294f9a496e5031b34abcd7f4ad9bd84c027b
Submitter: Jenkins
Branch: stable/mitaka

commit 41b1294f9a496e5031b34abcd7f4ad9bd84c027b
Author: dmburmistrov <email address hidden>
Date: Fri Aug 19 16:36:26 2016 +0300

    Refactor lock-file handling

    Currently fuel-rabbit-fence is not able
    to manage its pid-file properly.
    Current implementation was refactored.
    "fcntl" is used.

    Closes-bug: #1614963

    Change-Id: Ia5f58c3eb964d25c733cc91a6a5373ddbf193e77

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/357893
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5501d2d14cd0b00f6669b017497c7e0f31780869
Submitter: Jenkins
Branch: master

commit 5501d2d14cd0b00f6669b017497c7e0f31780869
Author: dmburmistrov <email address hidden>
Date: Fri Aug 19 16:36:26 2016 +0300

    Refactor lock-file handling

    Currently fuel-rabbit-fence is not able
    to manage its pid-file properly.
    Current implementation was refactored.
    "fcntl" is used.

    Closes-bug: #1614963

    Change-Id: Ia5f58c3eb964d25c733cc91a6a5373ddbf193e77

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Sofiia Andriichenko (sandriichenko) wrote :

Verified on MOS 9.1 (snapshot #209)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0rc1

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0

This issue was fixed in the openstack/fuel-library 10.0.0 release.

Revision history for this message
Rene Soto (rsoto) wrote :

Customer also just experienced this exact issue in a MOS 8.0 environment. Identical error messages were flooding /var/log/syslog on a specific controller node. The same fix seems to have resolved the issue in this particular environment as well.

tags: added: customer-found support
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Change author: Dmitry Burmistrov <email address hidden>
Review: https://review.fuel-infra.org/37194

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/fuel-library (openstack-ci/fuel-8.0/liberty)

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Dmitry Burmistrov <email address hidden>
Review: https://review.fuel-infra.org/37436

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/fuel-library (openstack-ci/fuel-8.0/liberty)

Reviewed: https://review.fuel-infra.org/37436
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: b083396199e62017f7197b7011fcb13c721a0e9a
Author: Dmitry Burmistrov <email address hidden>
Date: Tue Dec 12 12:54:28 2017

Refactor lock-file handling

Currently fuel-rabbit-fence is not able
to manage its pid-file properly.
Current implementation was refactored.
"fcntl" is used.

Closes-bug: #1614963

Change-Id: Ia5f58c3eb964d25c733cc91a6a5373ddbf193e77

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/fuel-library (stable/8.0)

Change abandoned by Denis V. Meltsaykin <email address hidden> on branch: stable/8.0
Review: https://review.fuel-infra.org/37194
Reason: I must have recommended the wrong branch, sorry.

Revision history for this message
Vladimir Jigulin (vjigulin) wrote :

Cannot reproduce in different labs. Closing on the basis of Rene Soto (rsoto) message #9.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.