Fuel for OpenStack

[regression] MySQL OCF may choose different nodes as possible masters

Bug #1578278 reported by Bogdan Dobrelya on 2016-05-04

This bug report is a duplicate of: Bug #1583173: MySQL OCF RA action monitor must check if a seed node is running the most recent of known GTIDs. Edit Remove

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Committed	High	Bogdan Dobrelya	Fuel for OpenStack 10.0
6.1.x	Won't Fix	High	MOS Maintenance	Fuel for OpenStack 6.1-updates
7.0.x	Won't Fix	High	MOS Maintenance	Fuel for OpenStack 7.0-updates
8.0.x	Won't Fix	High	MOS Maintenance	Fuel for OpenStack 8.0-updates
Mitaka	Fix Released	High	Bogdan Dobrelya	Fuel for OpenStack 9.0
Newton	Fix Committed	High	Bogdan Dobrelya	Fuel for OpenStack 10.0

Bug Description

When nodes have the same GTID=UUID:SEQNO, different nodes may report different choosen masters.
In result, several node attept to start mysqld --wsrep-new-cluster and fail the cluster (re)assemble.

Steps to reproduce were given in the Galera reliability testing https://goo.gl/VHyIIE paper. Briefly: deploy a 5 nodes galera cluster, run the given Jepsen cases to verify its self-heal capabilities.

There is also a way to reproduce it on a Fuel env given that the node-1, node-2, node-3 deployed as controller nodes:
1) https://github.com/bogdando/jepsen/tree/fuel/noop , see "How-to run tests from the Fuel master..."
2) PURGE=true ./vagrant_script/lein_test.sh noop ssh-test
3) docker exec -it jepsen bash -c "TESTPROC=mysqld lein test :only jepsen.noop-test/factors-crashstop-test"

See original description

Tags:

Bogdan Dobrelya (bogdando) on 2016-05-04

Changed in fuel:
importance:	Undecided → High
assignee:	nobody → Bogdan Dobrelya (bogdando)
milestone:	none → 10.0
tags:	added: galera

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-04: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/312622

Changed in fuel:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-05: Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/312856

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-05: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/312622
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d9b44f11a6eb6b83eab8785a0e66e4bf85f5ca1a
Submitter: Jenkins
Branch: master

commit d9b44f11a6eb6b83eab8785a0e66e4bf85f5ca1a
Author: Bogdan Dobrelya <email address hidden>
Date: Wed May 4 18:13:34 2016 +0200

Ensure same node lists for possible masters search

    The 'crm_node --partition' returns nodes in different order
    for different nodes. Ensure the same node lists to
    reach a consensus for the choosen master across all of the nodes
    in a partition.

Closes-bug: #1578278

Change-Id: I55f487b1021e5971b60f34fa338b5fc87967495f
Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-05: Change abandoned on fuel-library (stable/mitaka)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/312856
Reason: to early to backport, let's fix all of the issues discovered in the master first

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2016-05-05: Re: MySQL OCF may choose different nodes as possible masters

The fix is not complete, here is example of search:

=====n1
May 5 11:00:46 n1 ocf-mysql-wss: INFO: p_mysql: get_master() Possible masters: n3 n1 n2 n4 n5
=====n2
May 5 11:00:46 n2 ocf-mysql-wss: INFO: p_mysql: get_master() Possible masters: n3 n2 n4 n5 n1
=====n3
May 5 11:00:46 n3 ocf-mysql-wss: INFO: p_mysql: get_master() Possible masters: n1 n5 n4 n3 n2
=====n4
May 5 11:00:46 n4 ocf-mysql-wss: INFO: p_mysql: get_master() Possible masters: n5 n3 n2 n4 n1
=====n5
May 5 11:01:12 n5 ocf-mysql-wss: INFO: p_mysql: get_master() Possible masters: n5

=====n1
May 5 11:00:46 n1 ocf-mysql-wss: INFO: p_mysql: get_master() Choosed master: n4 with GTID: 96681b82-128a-11e6-9e6e-47e1f0e1d7ed:2434
=====n2
May 5 11:00:46 n2 ocf-mysql-wss: INFO: p_mysql: get_master() Choosed master: n1 with GTID: 96681b82-128a-11e6-9e6e-47e1f0e1d7ed:2434
=====n3
May 5 11:00:46 n3 ocf-mysql-wss: INFO: p_mysql: get_master() Choosed master: n2 with GTID: 96681b82-128a-11e6-9e6e-47e1f0e1d7ed:2434
=====n4
May 5 11:00:46 n4 ocf-mysql-wss: INFO: p_mysql: get_master() Choosed master: n1 with GTID: 96681b82-128a-11e6-9e6e-47e1f0e1d7ed:2434
=====n5
May 5 11:01:12 n5 ocf-mysql-wss: INFO: p_mysql: get_master() Choosed master: n5 with GTID: 96681b82-128a-11e6-9e6e-47e1f0e1d7ed:2101

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-05-05:

This is regression issue which reproduced in 100% of cases now after feature "v2 tasks based deploy".

summary:

- MySQL OCF may choose different nodes as possible masters
+ [regression] MySQL OCF may choose different nodes as possible masters

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-05: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/312911

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-06: Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/313273

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-06: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/312911
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=03991059adcc99989431c27f38c5ffa4ae35b486
Submitter: Jenkins
Branch: master

commit 03991059adcc99989431c27f38c5ffa4ae35b486
Author: Bogdan Dobrelya <email address hidden>
Date: Thu May 5 13:43:24 2016 +0200

Rework SST check, fix possible masters search

    * Fix racing of monitoring with SST
    * Fix printf multilines sorting
      Expected: printf -- '%s\n' ${a} | sort -u (returns a sorted multiline)
      Actuacl: printf -- '%s\n' "$a" | sort -u (returns a single string)
    * Fix possible masters search, by the greatest SEQNO found for a
      magority UUID
    (Those blocks each other in CI and must be fixed at once)

    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779

Change-Id: I3d0d376e6bef3ccc3e738731b71f4dd60a59e653
Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-06: Change abandoned on fuel-library (stable/mitaka)

#10

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/312856
Reason: superseded by https://review.openstack.org/#/c/313273

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-13: Fix merged to fuel-library (stable/mitaka)

#11

Reviewed: https://review.openstack.org/313273
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0197b09fc6ef25657a81a8398b6da0ea1b0603af
Submitter: Jenkins
Branch: stable/mitaka

commit 0197b09fc6ef25657a81a8398b6da0ea1b0603af
Author: Bogdan Dobrelya <email address hidden>
Date: Thu May 5 13:43:24 2016 +0200

Rework SST check, fix possible masters search

Fuel-CI: disable

    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779

    Change-Id: I3d0d376e6bef3ccc3e738731b71f4dd60a59e653
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 03991059adcc99989431c27f38c5ffa4ae35b486)
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    Signed-off-by: Sergii Golovatiuk <email address hidden>

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-13: Fix proposed to fuel-library (stable/6.1)

#12

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/315989

Bogdan Dobrelya (bogdando) on 2016-05-13

tags:

added: tech-debt

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-16: Fix proposed to fuel-library (stable/7.0)

#13

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/316802

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-18: Fix proposed to fuel-library (stable/8.0)

#14

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/317978

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-20: Change abandoned on fuel-library (stable/8.0)

#15

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/317978

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-20: Change abandoned on fuel-library (stable/7.0)

#16

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/316802

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-20: Change abandoned on fuel-library (stable/6.1)

#17

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/315989

Bogdan Dobrelya (bogdando) on 2016-06-20

description:

updated

Bogdan Dobrelya (bogdando) on 2016-06-22

description:

updated

Maksym Strukov (unbelll) on 2016-06-22

tags:

added: on-verification

Revision history for this message

Maksym Strukov (unbelll) wrote on 2016-06-24:

#18

Verified as fixed in 9.0-mos-495

tags:

removed: on-verification

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2016-09-07:

#19

Won't Fix for 6.1-, 7.0- and 8.0-updates as this is too risky change to be accepted to stable branch

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-21: Fix proposed to fuel-library (stable/7.0)

#20

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/374219

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-26: Fix merged to fuel-library (stable/7.0)

#21

Reviewed: https://review.openstack.org/374219
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f9a2d479f3687157d2b17a927a09ce5f995522d6
Submitter: Jenkins
Branch: stable/7.0

commit f9a2d479f3687157d2b17a927a09ce5f995522d6
Author: Denis Puchkin <email address hidden>
Date: Wed Sep 21 17:38:54 2016 +0300

Backport mysql OCF from stable/mitaka

backport mysql ocf script from stable/mitaka

    Closes-bug: #1524826
    Closes-bug: #1542256
    Closes-bug: #1572239
    Closes-bug: #1572557
    Closes-bug: #1572601
    Closes-bug: #1574747
    Closes-bug: #1574497
    Closes-bug: #1576244
    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779
    Closes-bug: #1574999
    Closes-bug: #1576244
    Closes-bug: #1583173
    Closes-bug: #1585125

Change-Id: I1cc6f95884a8fbd5c3418ede89bdf9ec6864bdc8

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-27: Fix proposed to fuel-library (stable/8.0)

#22

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/377597