Fuel for OpenStack

Mysql ocf race introduce

  1. Series ocata
  2. Bug #1658144
Bug #1658144 reported by Nikita Koshikov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Sergii Golovatiuk
Fuel for OpenStack 11.0
Mitaka
Fix Released
High
Sergii Golovatiuk
Fuel for OpenStack 9.2-mu-1
Newton
Fix Released
High
Sergii Golovatiuk
Fuel for OpenStack 10.1
Ocata
Fix Committed
High
Sergii Golovatiuk
Fuel for OpenStack 11.0

Bug Description

Hello,

fix to this bug introduce sleep, that always executed on primary-contoller:
https://review.openstack.org/#/c/356123/

$ if [ "$GTID"="0" ]; then echo 'BINGO'; fi
BINGO
$ if [ "$GTID"="0" ]; then echo 'BINGO'; fi
BINGO
$ if [ "$GTID"=="0" ]; then echo 'BINGO'; fi
BINGO
$ if [[ "$GTID"=="0" ]]; then echo 'BINGO'; fi
BINGO

Because of that new function + sleep, mysql split brain condition became really easy thing to obtain. Run while true; update database; sleep 0.1; done and in next 5 minutes you'll have split-brain. The fix is trivial:
root@node-6:/usr/lib/ocf/resource.d/fuel# diff -u mysql-wss mysql-wss.fixed
--- mysql-wss 2017-01-20 16:53:24.764587866 +0000
+++ mysql-wss.fixed 2017-01-20 16:53:12.212299614 +0000
@@ -343,7 +343,7 @@

     GTID=$(get_node_gtid $NODE)

- if [ "$GTID"="0" ]; then
+ if [ "$GTID" = "0" ]; then
         sleep $[ ( $RANDOM % 10 ) + 1]
         GTID=$(get_node_gtid $NODE)
     fi

Evgeniy L (rustyrobot)
tags: added: area-library customer-found
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

https://review.openstack.org/#/c/423415/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/424498

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/424565

Roman Rufanov (rrufanov)
tags: added: release-notes
Sergii Golovatiuk (sgolovatiuk)
tags: removed: release-notes
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/424498
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=70239cadfef134bdfc45ada59c7c225f6cbfcb5d
Submitter: Jenkins
Branch: stable/newton

commit 70239cadfef134bdfc45ada59c7c225f6cbfcb5d
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Jan 20 20:09:13 2017 +0100

    Fix mysql OCF race

    - Fix bug
    - Fix issues found in bashate

    Closes-Bug: #1658144

    Change-Id: I93e9ab269d3ad4a285154fafcac6426ef36f6b52

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/424565
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=111ed81108c4ab68d2d5e24400a663f94335397d
Submitter: Jenkins
Branch: stable/mitaka

commit 111ed81108c4ab68d2d5e24400a663f94335397d
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Jan 20 20:09:13 2017 +0100

    Fix mysql OCF race

    - Fix bug
    - Fix issues found in bashate

    Closes-Bug: #1658144

    Change-Id: I93e9ab269d3ad4a285154fafcac6426ef36f6b52

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/423415
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=2326ef963a51379d54d402ab19b9dc98985b1424
Submitter: Jenkins
Branch: master

commit 2326ef963a51379d54d402ab19b9dc98985b1424
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Jan 20 20:09:13 2017 +0100

    Fix mysql OCF race

    - Fix bug
    - Fix issues found in bashate

    Closes-Bug: #1658144

    Change-Id: I93e9ab269d3ad4a285154fafcac6426ef36f6b52

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 11.0.0.0rc1

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on 9.2 mu1 updates.
No 'split-brain' observed in logs during writing to db during ~5min.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.