2 of 3 Nodes Remains Non-Primary after brief Split

Bug #1269236 reported by Jervin R
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Galera
Won't Fix
Undecided
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Invalid
Undecided
Unassigned

Bug Description

The seriees of events based on the logs:

1. node3 dropped out of cluster
2. node2 splits, goes non-primary
3. node1 splits, goes non-primary
4. node2 and node1 found each other, remained non-primary while waiting for node3 to come back

I would expect at #4 above, the cluster would've have regained quorum and switched to PRIMARY but it did not.

Relevant logs from node2:

140114 20:32:22 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.2.9:4567
140114 20:32:22 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 0
140114 20:32:22 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.38318S), skipping check
140114 20:32:23 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, OPERATIONAL, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:23 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:24 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:24 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 381d72fa-4b83-11e3-8d48-02e7ece87890 (tcp://192.168.2.102:4567), attempt 0
140114 20:32:24 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:25 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:25 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 381d72fa-4b83-11e3-8d48-02e7ece87890
140114 20:32:25 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:26 [Note] WSREP: view(view_id(NON_PRIM,381d72fa-4b83-11e3-8d48-02e7ece87890,309) memb {
 ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
} joined {
} left {
} partitioned {
 381d72fa-4b83-11e3-8d48-02e7ece87890,
 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
})
140114 20:32:26 [Note] WSREP: view(view_id(NON_PRIM,ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,310) memb {
 ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
} joined {
} left {
} partitioned {
 381d72fa-4b83-11e3-8d48-02e7ece87890,
 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
})
140114 20:32:26 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140114 20:32:26 [Note] WSREP: Flow-control interval: [16, 16]
140114 20:32:26 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:26 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 441177589)
140114 20:32:26 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140114 20:32:26 [Note] WSREP: Flow-control interval: [16, 16]
140114 20:32:26 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:26 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
140114 20:32:26 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:32:27 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
140114 20:32:27 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:32:35 [Note] WSREP: declaring 381d72fa-4b83-11e3-8d48-02e7ece87890 stable
140114 20:32:35 [Note] WSREP: view(view_id(NON_PRIM,381d72fa-4b83-11e3-8d48-02e7ece87890,311) memb {
 381d72fa-4b83-11e3-8d48-02e7ece87890,
 ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
} joined {
} left {
} partitioned {
 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
})
140114 20:32:35 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 2
140114 20:32:35 [Note] WSREP: Flow-control interval: [23, 23]
140114 20:32:35 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:35 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 2, my index: 1, protocol version 2
140114 20:32:35 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:34:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 30
140114 20:36:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 60
140114 20:38:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 90
140114 20:40:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 120
140114 20:42:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 150
140114 20:44:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 180
140114 20:46:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 210
140114 20:48:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 240
140114 20:50:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 270
140114 20:52:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 300
140114 20:54:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 330
140114 20:56:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 360
140114 20:58:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 390
140114 21:00:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 420
140114 21:02:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 450
140114 21:04:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 480
140114 21:06:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 510
140114 21:08:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 540
140114 21:10:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 570
140114 21:12:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 600
140114 21:14:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 630
140114 21:16:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 660
140114 21:18:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 690
140114 21:20:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 720
140114 21:22:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 750
140114 21:24:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 780
140114 21:26:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 810
140114 21:28:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 840
140114 21:30:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 870
140114 21:32:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 900
140114 21:34:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 930
140114 21:36:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 960
140114 21:38:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 990
140114 21:40:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 1020
140114 21:42:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 1050
140114 21:44:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 1080

Relevant logs from node1:

140114 2:53:48 [Warning] IP address '210.5.27.49' could not be resolved: Name or service not known
140114 20:32:33 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.2.9:4567
140114 20:32:33 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 0
140114 20:32:33 [Warning] WSREP: last inactive check more than PT1.5S ago (PT12.7579S), skipping check
140114 20:32:33 [Note] WSREP: evs::proto(381d72fa-4b83-11e3-8d48-02e7ece87890, OPERATIONAL, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) detected inactive node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:33 [Note] WSREP: evs::proto(381d72fa-4b83-11e3-8d48-02e7ece87890, OPERATIONAL, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: ade80c49-54b9-11e3-81f2-2bdc4faa4ea7
140114 20:32:34 [Note] WSREP: view(view_id(NON_PRIM,381d72fa-4b83-11e3-8d48-02e7ece87890,309) memb {
 381d72fa-4b83-11e3-8d48-02e7ece87890,
} joined {
} left {
} partitioned {
 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
 ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
})
140114 20:32:34 [Note] WSREP: view(view_id(NON_PRIM,381d72fa-4b83-11e3-8d48-02e7ece87890,310) memb {
 381d72fa-4b83-11e3-8d48-02e7ece87890,
} joined {
} left {
} partitioned {
 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
 ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
})
140114 20:32:34 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140114 20:32:34 [Note] WSREP: Flow-control interval: [16, 16]
140114 20:32:34 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:34 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 441177589)
140114 20:32:34 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140114 20:32:34 [Note] WSREP: Flow-control interval: [16, 16]
140114 20:32:34 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:34 [Warning] WSREP: Send action {0x7f49a55bea50, 394, TORDERED} returned -107 (Transport endpoint is not connected)
140114 20:32:34 [Warning] WSREP: Send action {0x7f49a4a47590, 394, TORDERED} returned -107 (Transport endpoint is not connected)
140114 20:32:34 [Warning] WSREP: Send action {0x7f49a50bf5c0, 396, TORDERED} returned -107 (Transport endpoint is not connected)
140114 20:32:34 [Warning] WSREP: Send action {0x7f4949ccc860, 735, TORDERED} returned -107 (Transport endpoint is not connected)
140114 20:32:34 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
140114 20:32:34 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:32:34 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
140114 20:32:34 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:32:35 [Note] WSREP: declaring ade80c49-54b9-11e3-81f2-2bdc4faa4ea7 stable
140114 20:32:35 [Note] WSREP: view(view_id(NON_PRIM,381d72fa-4b83-11e3-8d48-02e7ece87890,311) memb {
 381d72fa-4b83-11e3-8d48-02e7ece87890,
 ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
} joined {
} left {
} partitioned {
 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
})
140114 20:32:35 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 2
140114 20:32:35 [Note] WSREP: Flow-control interval: [23, 23]
140114 20:32:35 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:35 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 2, my index: 0, protocol version 2
140114 20:32:35 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:34:36 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 30
140114 20:36:36 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 60
140114 20:38:38 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 90
140114 20:40:39 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 120
140114 20:42:39 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 150
140114 20:44:39 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 180
140114 20:46:40 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 210
140114 20:48:41 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 240
140114 20:50:43 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 270
140114 20:52:46 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 300
140114 20:54:46 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 330
140114 20:56:51 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 360
140114 20:58:53 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 390
140114 21:00:57 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 420
140114 21:03:00 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 450
140114 21:05:01 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 480
140114 21:07:17 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 510
140114 21:09:27 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 540
140114 21:11:33 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 570
140114 21:13:47 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 600
140114 21:16:01 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 630
140114 21:18:16 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 660
140114 21:20:28 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 690
140114 21:22:39 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 720
140114 21:24:52 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 750
140114 21:27:07 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 780
140114 21:29:21 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 810
140114 21:31:35 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 840
140114 21:33:48 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 870
140114 21:36:02 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 900
140114 21:38:18 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 930
140114 21:40:33 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 960
140114 21:42:48 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 990
140114 21:45:01 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 1020

~# dpkg -l|grep percona
ii percona-nagios-plugins 1.1.1-1 Percona Monitoring Plugins for Nagios
ii percona-toolkit 2.2.5-2 Advanced MySQL and system command-line tools
ii percona-xtrabackup 2.1.5-680-1.precise Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-client-5.5 5.5.34-23.7.6-565.precise Percona Server database client binaries
ii percona-xtradb-cluster-common-5.5 5.5.34-23.7.6-565.precise Percona Server database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-2.x 162.precise Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.5 5.5.34-23.7.6-565.precise Percona Server database server binaries
ii xtrabackup 2.1.3-608.precise Transitional package for percona-xtrabackup

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

It turns out to be an expected behaviour due to safety considerations. It was speculated that in the case of a total cluster split:

{1,2,3} -> (1),(2),(3)

there is a chance of operator bootstrapping node 1 into a primary component. In that case merging and recovering primary component by simple majority

(2),(3) -> {2,3}

will lead to a second PC and state divergence.

Also (and it is not a speculation) in a busy cluster a split like that may lead to one node to be a bit more updated than the others. So complete remerge of the cluster

(1),(2),(3) -> {1,2,3}

is essential here for PC recovery due to consistency considerations.

Changed in galera:
status: New → Won't Fix
Changed in percona-xtradb-cluster:
status: New → Invalid
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1577

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.