Fuel for OpenStack

Full reassemble of Galera cluster fails in case of epoch divergence

Bug #1388779 reported by Dennis Dmitriev on 2014-11-03

This bug affects 3 people

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Committed	High	Bogdan Dobrelya	Fuel for OpenStack 10.0
5.1.x	Invalid	High	Bogdan Dobrelya	Fuel for OpenStack 5.1.1-updates
6.0.x	Won't Fix	High	Bogdan Dobrelya	Fuel for OpenStack 6.0-updates
6.1.x	Won't Fix	High	Bogdan Dobrelya	Fuel for OpenStack 6.1-updates
7.0.x	Won't Fix	High	Bogdan Dobrelya	Fuel for OpenStack 7.0-updates
8.0.x	Won't Fix	High	Bogdan Dobrelya	Fuel for OpenStack 8.0-updates
Mitaka	Won't Fix	High	Sergii Golovatiuk	Fuel for OpenStack 9.0
Newton	Won't Fix	High	Bogdan Dobrelya	Fuel for OpenStack 10.0

Bug Description

Regularly observed on the system test 'ceph_ha_restart' , this time on: http://jenkins-product.srt.mirantis.net:8080/view/6.0_swarm/job/6.0_fuelmain.system_test.ubuntu.thread_3/15/

Steps to reproduce:
            1. Create cluster (Ubuntu, nova-network flat-dhcp, Ceph for images and volumes)
            2. Add 3 nodes with controller and ceph OSD roles
            3. Add 1 node with ceph OSD roles
            4. Add 2 nodes with compute and ceph OSD roles
            5. Deploy the cluster
            6. Reset all nodes.
            7. Check cluster status with 'crm status' and pacemaker logs on all controllers.

If mysql failed to start after nodes are reset, then pacemaker hangs on waiting for mysql status for 475 sec. That cause a long time to re-assemble cluster for other resources such as rabbitmq.

Related bug about rabbitmq: https://bugs.launchpad.net/fuel/+bug/1383247

There is more detailed information while CI test was running:

`crm status` right after the nodes was rebooted (Nov 3 09:43:15 2014) : http://paste.openstack.org/show/128612/

Pacemaker logs taken from ssh session:
- from controller-1: http://paste.openstack.org/show/128613/
- from controller-2: http://paste.openstack.org/show/128614/
- from controller-3: http://paste.openstack.org/show/128618/

`crm status` before the timeout of the test (Nov 3 09:49:12 2014) : http://paste.openstack.org/show/128619/

In the pacemaker logs from controller-2 and 3 is the following warning:
"<28>Nov 3 09:48:19 node-6 lrmd[2021]: warning: operation_finished: p_mysql_start_0:4218 - timed out after 475000ms"

Pacemaker was not doing any operations for ~8 minutes on all controllers until this timeout appeared, so the rabbitmq resource wasn't processed too.

Tags:

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-11-03:

fail_error_ceph_ha_restart-2014_11_03__09_50_20.tar.gz Edit (9.8 MiB, application/x-tar)

Bogdan Dobrelya (bogdando) on 2014-11-03

Changed in fuel:
importance:	Undecided → High
tags:	added: ha pacemaker

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2014-11-03:

There are no any "timed out after 475000ms" log records in snapshot, please make sure you created snapshot as appropriate

Changed in fuel:
status:	New → Incomplete

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-11-03:

fuel-snapshot-logs.tar.gz Edit (25.5 MiB, application/x-tar)

Looks like some logs weren't collected after "2014-11-03 09:40:" despite the fact that the diagnostic snapshot was made at 09:50.

I've reverted the env and manually collected all logs from nodes (node-3 and node-5 are not loaded).
Please look at the pacemaker.log on the controller nodes node-1, node-4 and node-6.

Changed in fuel:
status:	Incomplete → Confirmed

Matthew Mosesohn (raytrac3r) on 2014-11-14

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2014-11-17:

According to mysqld.log from node-6

Something went wrong then galera cluster had been reassambling for 8 minutes since pcs ordered p_mysql to start (http://pastebin.com/ca3hEccL, see FATAL):

<27>Nov 3 09:41:05 node-6 mysqld: 2014-11-03 09:41:05 6763 [ERROR] WSREP: Local state seqno (8961) is greater than group seqno (8955): states diverged. Aborting to avoid potential data loss. Remove '/var/lib/mysql//grastate.dat' file and restart if you wish to continue. (FATAL)

And starting from this time, it only had been reporting MySQL is not running / mysqld.pid of MySQL server not found until pcs time out at 9:48:19

Changed in fuel:
assignee:	Bogdan Dobrelya (bogdando) → Sergii Golovatiuk (sgolovatiuk)
summary:	- Pacemaker freezes while it is waiting a resource status + Full reassemble of Galera cluster failure

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2014-11-25: Re: Full reassemble of Galera cluster failure

Problem is also reproduced on 5.1.1 ISO
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1.1"
  api: "1.0"
  build_number: "37"
  build_id: "2014-11-24_21-01-00"
  astute_sha: "dade74af41d4972fe05a1c16ae1db2a2e60c6715"
  fuellib_sha: "444339cae21c369c1d95e96c1059d4099077138e"
  ostf_sha: "64cb59c681658a7a55cc2c09d079072a41beb346"
  nailgun_sha: "24b956d739e8e9d8f728701522c6fa8364526c45"
  fuelmain_sha: "3797f2f2eff42812715840293a618de73fdef26f"

pacemaker log on node-2:
<29>Nov 25 06:04:17 node-2 attrd[1818]: notice: attrd_perform_update: Sent update 21: pingd=1000
<28>Nov 25 06:11:40 node-2 lrmd[1817]: warning: child_timeout_callback: p_mysql_start_0 process (PID 3531) timed out
<28>Nov 25 06:11:40 node-2 lrmd[1817]: warning: operation_finished: p_mysql_start_0:3531 - timed out after 475000ms
[12:37:54] Dennis Dmitriev: <29>Nov 25 06:11:41 node-2 crmd[1820]: notice: process_lrm_event: LRM operation vip__public_old_start_0 (call=70, rc=0, cib-update=24, confirmed=true) ok
<28>Nov 25 06:19:35 node-2 lrmd[1817]: warning: child_timeout_callback: p_mysql_start_0 process (PID 17034) timed out
<28>Nov 25 06:19:35 node-2 lrmd[1817]: warning: operation_finished: p_mysql_start_0:17034 - timed out after 475000ms

Logs are attached

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2014-11-25:

fail_error_ceph_ha_restart-2014_11_25__06_12_48.tar.gz Edit (10.8 MiB, application/x-tar)

Revision history for this message

Sergii Golovatiuk (sgolovatiuk) wrote on 2014-11-25:

According to galeracluster.com/2013/10/order-of-business/ that case cannot be resolved automatically as nodes have completely different commits and cannot be diverged.

Revision history for this message

Mike Scherbakov (mihgen) wrote on 2014-11-28:

Sergii,
so what do we do then? Recommend anything in docs/release notes, or what?

Revision history for this message

Sergii Golovatiuk (sgolovatiuk) wrote on 2014-12-03:

I am going to produce patch today which fixes the issue.

OpenStack Infra (hudson-openstack) on 2014-12-03

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

Mike Scherbakov (mihgen) wrote on 2014-12-04:

#10

Sergii, I do not see a patch in the bug. Did you forget to link it to the bug?..

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2014-12-04:

#11

https://review.openstack.org/#/c/137105/

summary:	- Full reassemble of Galera cluster failure + Full reassemble of Galera cluster fails in case epoch divergence
summary:	- Full reassemble of Galera cluster fails in case epoch divergence + Full reassemble of Galera cluster fails in case of epoch divergence

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-05: Related fix merged to fuel-main (master)

#12

Reviewed: https://review.openstack.org/139596
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=134f70e8332d9a52f17565e23e8b14f092efb735
Submitter: Jenkins
Branch: master

commit 134f70e8332d9a52f17565e23e8b14f092efb735
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Dec 5 11:27:12 2014 +0100

Fix MySQL destructive test

    * I7f50e32ced83af8f32fd7907cb5fc723055da121 separated pid directories for
      Pacemaker managed resources moving PID files to $HA_RSCTMP/$__SCRIPT_NAME
      This patch changes test to use name rather than PID
    * Change pkill to to pkill -x to kill mysqld rathar than all processes
      that match mysqld pattern

    Closes-Bug: 1399605
    Related-Bug: 1388779
    Change-Id: I34d540200f0ce2ff17b30e20a43f4d13c86b7492
    Signed-off-by: Sergii Golovatiuk <email address hidden>

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-05: Fix merged to fuel-library (master)

#13

Reviewed: https://review.openstack.org/137105
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2447e44989e3af8dc1b4f4652b1180f25cb5322b
Submitter: Jenkins
Branch: master

commit 2447e44989e3af8dc1b4f4652b1180f25cb5322b
Author: Sergii Golovatiuk <email address hidden>
Date: Tue Nov 25 16:43:00 2014 +0100

Kill xtrabackup spawned processes

* Send KILL signals to all processed spawned by mysqld
* Refactor OCF script to make it more simple and more understandable

Closes-Bug: 1388779

Change-Id: Ie3f6149d3f97f5e8cf3f1f3329b9e335e551b3e6
Signed-off-by: Sergii Golovatiuk <email address hidden>

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-05: Fix proposed to fuel-library (master)

#14

Fix proposed to branch: master
Review: https://review.openstack.org/139674

Vladimir Kuklin (vkuklin) on 2014-12-05

Changed in fuel:
status:	Fix Committed → Confirmed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-05: Fix merged to fuel-library (master)

#15

Reviewed: https://review.openstack.org/139674
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0524973e9a4470a8269542c8327dfaec750831b3
Submitter: Jenkins
Branch: master

commit 0524973e9a4470a8269542c8327dfaec750831b3
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Dec 5 17:19:28 2014 +0100

Fix type in pkill function

In Different --pgroup behaves differently. -g behaves exactly the same

Change-Id: I3254f333b9caf3d417ec595896c6695fad617f1e
Closes-Bug: 1388779

Changed in fuel:
status:	Confirmed → Fix Committed

Revision history for this message

Sergii Golovatiuk (sgolovatiuk) wrote on 2014-12-05:

#16

This issue was affected by issue in corosync package. It doesn't affect 5.1

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2016-05-03:

#17

This issue can be easy reproduced by recurring network partitions, see the duplicated bug details. Cannot be fixed, AFAIKT, a known issue for docs perhaps. Only manual recovery is possible.

Dmitry Pyzhov (dpyzhov) on 2016-05-04

tags:

added: area-library

OpenStack Infra (hudson-openstack) on 2016-05-05

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-06: Fix proposed to fuel-library (stable/mitaka)

#18

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/313273

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-06: Fix merged to fuel-library (master)

#19

Reviewed: https://review.openstack.org/312911
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=03991059adcc99989431c27f38c5ffa4ae35b486
Submitter: Jenkins
Branch: master

commit 03991059adcc99989431c27f38c5ffa4ae35b486
Author: Bogdan Dobrelya <email address hidden>
Date: Thu May 5 13:43:24 2016 +0200

Rework SST check, fix possible masters search

    * Fix racing of monitoring with SST
    * Fix printf multilines sorting
      Expected: printf -- '%s\n' ${a} | sort -u (returns a sorted multiline)
      Actuacl: printf -- '%s\n' "$a" | sort -u (returns a single string)
    * Fix possible masters search, by the greatest SEQNO found for a
      magority UUID
    (Those blocks each other in CI and must be fixed at once)

    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779

Change-Id: I3d0d376e6bef3ccc3e738731b71f4dd60a59e653
Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-13: Fix merged to fuel-library (stable/mitaka)

#20

Reviewed: https://review.openstack.org/313273
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0197b09fc6ef25657a81a8398b6da0ea1b0603af
Submitter: Jenkins
Branch: stable/mitaka

commit 0197b09fc6ef25657a81a8398b6da0ea1b0603af
Author: Bogdan Dobrelya <email address hidden>
Date: Thu May 5 13:43:24 2016 +0200

Rework SST check, fix possible masters search

Fuel-CI: disable

    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779

    Change-Id: I3d0d376e6bef3ccc3e738731b71f4dd60a59e653
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 03991059adcc99989431c27f38c5ffa4ae35b486)
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    Signed-off-by: Sergii Golovatiuk <email address hidden>

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-13: Fix proposed to fuel-library (stable/6.1)

#21

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/315989

Bogdan Dobrelya (bogdando) on 2016-05-13

tags:

added: tech-debt

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-16: Fix proposed to fuel-library (stable/7.0)

#22

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/316802

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-18: Fix proposed to fuel-library (stable/8.0)

#23

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/317978

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2016-06-16:

#24

This is a corner case requiring manual resolving, cannot be fixed automatically w/o data loss. Such decisions shall be made by ops and recovery steps to be done manually.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-20: Change abandoned on fuel-library (stable/8.0)

#25

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/317978

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-20: Change abandoned on fuel-library (stable/7.0)

#26

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/316802

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-20: Change abandoned on fuel-library (stable/6.1)

#27

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/315989

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-21: Fix proposed to fuel-library (stable/7.0)

#28

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/374219

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-26: Fix merged to fuel-library (stable/7.0)

#29

Reviewed: https://review.openstack.org/374219
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f9a2d479f3687157d2b17a927a09ce5f995522d6
Submitter: Jenkins
Branch: stable/7.0

commit f9a2d479f3687157d2b17a927a09ce5f995522d6
Author: Denis Puchkin <email address hidden>
Date: Wed Sep 21 17:38:54 2016 +0300

Backport mysql OCF from stable/mitaka

backport mysql ocf script from stable/mitaka

    Closes-bug: #1524826
    Closes-bug: #1542256
    Closes-bug: #1572239
    Closes-bug: #1572557
    Closes-bug: #1572601
    Closes-bug: #1574747
    Closes-bug: #1574497
    Closes-bug: #1576244
    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779
    Closes-bug: #1574999
    Closes-bug: #1576244
    Closes-bug: #1583173
    Closes-bug: #1585125

Change-Id: I1cc6f95884a8fbd5c3418ede89bdf9ec6864bdc8

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-27: Fix proposed to fuel-library (stable/8.0)

#30

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/377597