Detailed bug description:
MySQL crashes after restart via ‘pcs resource restart clone_p_mysqld’ command in MOS 9.0
Almost 50% of MySQL restart attempts are finished with next MySQL error: WSREP: exception from gcomm, backend must be restarted: 281c836a aborting due to conflicting prims: older overrides (FATAL)
Steps to reproduce:
Run next command:
pcs resource restart clone_p_mysqld
Expected results:
MySQL will be restarted without errors in log files
After restart MySQL wsrep_cluster_size will be equal to controllers count (in my case 3)
Actual result:
MySQL restarted and error appeared in MySQL log file on one of controller nodes:
2016-08-23T16:59:02.214831+02:00 node-3 mysqld: 2016-08-23 16:59:02 22172 [ERROR] WSREP: exception from gcomm, backend must be restarted: 1a2cd5b7 aborting due to conflicting prims: older overrides (FATAL)
Node with this error will have wsrep_cluster_size=1, other two nodes will have wsrep_cluster_size=2
Reproducibility:
This error can be reproduced almost in 50% restart attempts
Also found on Customer environment
Workaround:
Replace MOS 9.0 native OCF script (/usr/lib/ocf/resource.d/fuel/mysql-wss) with another one from master branch: https://github.com/openstack/fuel-library/blob/master/files/fuel-ha-utils/ocf/mysql-wss
Impact:
Critical impact
Leads to MySQL split-brain
Description of the environment:
Operation system: Ubuntu 14.04
Versions of components: MOS 9.0
Additional information:
Logs from controllers:
controller-1
2016-08-26T16:59:02.221676+02:00 node-1 mysqld: 2016-08-26 16:59:02 24841 [Warning] WSREP: bae1b7a9 conflicting prims: my prim: view_id(PRIM,bae1b7a9,6) other prim: view_id(PRIM,1a2cd5b7,1)
2016-08-26T16:59:02.221682+02:00 node-1 mysqld: 2016-08-26 16:59:02 24841 [Warning] WSREP: bae1b7a9 discarding other prim view: older overrides
controller-2
2016-08-26T16:59:02.218143+02:00 node-2 mysqld: 2016-08-26 16:59:02 9624 [ERROR] WSREP: failed to open gcomm backend connection: 131: 20c98253 last prims not consistent (FATAL)
2016-08-26T16:59:02.218143+02:00 node-2 mysqld: 2016-08-26 16:59:02 9624 [ERROR] WSREP: gcs/src/gcs.cpp:long int gcs_open(gcs_conn_t*, const char*, const char*, bool)():1379: Failed to open channel 'openstack' at 'gcomm://192.168.2.7,192.168.2.8,192.168.2.9': -131 (State not recoverable)
controller-3
2016-08-26T16:59:02.214527+02:00 node-3 mysqld: 2016-08-26 16:59:02 22172 [Warning] WSREP: 1a2cd5b7 conflicting prims: my prim: view_id(PRIM,1a2cd5b7,1) other prim: view_id(PRIM,bae1b7a9,6)
2016-08-26T16:59:02.214831+02:00 node-3 mysqld: 2016-08-26 16:59:02 22172 [ERROR] WSREP: exception from gcomm, backend must be restarted: 1a2cd5b7 aborting due to conflicting prims: older overrides (FATAL)
2016-08-26T16:59:02.470825+02:00 node-3 mysqld: 2016-08-26 16:59:02 22172 [Warning] WSREP: Send action {(nil), 7222, TORDERED} returned -103 (Software caused connection abort)
Customer found on version MOS 9.0
Backport is needed