After crash the node refuses to start in SYNC
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Galera |
Confirmed
|
Medium
|
Teemu Ollakka |
Bug Description
The master crashed ( report will be in different bug)
and I am trying to start it, but it can't start in SYNCED
/usr/sbin/mysqld
111223 1:47:27 [Note] Flashcache bypass: disabled
111223 1:47:27 [Note] Flashcache setup error is : ioctl failed
111223 1:47:27 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/
111223 1:47:27 [Note] WSREP: wsrep_load(): Galera 2.0beta(r99) by Codership Oy <email address hidden> loaded succesfully.
111223 1:47:27 [Note] WSREP: Reusing existing '/mnt/data/
111223 1:47:27 [Note] WSREP: Passing config to GCS: gcache.dir = /mnt/data/; gcache.
111223 1:47:27 [Note] WSREP: wsrep_sst_grab()
111223 1:47:27 [Note] WSREP: Start replication
111223 1:47:27 [Note] WSREP: Found saved state: a11cef9b-
111223 1:47:27 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1
111223 1:47:27 [Note] WSREP: Setting initial position to a11cef9b-
111223 1:47:27 [Note] WSREP: protonet asio version 0
111223 1:47:27 [Note] WSREP: backend: asio
111223 1:47:27 [Note] WSREP: GMCast version 0
111223 1:47:27 [Note] WSREP: (fa996d02-
111223 1:47:27 [Note] WSREP: (fa996d02-
111223 1:47:27 [Note] WSREP: EVS version 0
111223 1:47:27 [Note] WSREP: PC version 0
111223 1:47:27 [Note] WSREP: gcomm: connecting to group 'trimethylxanth
111223 1:47:27 [Note] WSREP: GMCast:
} joined {
} left {
} partitioned {
})
111223 1:47:27 [Note] WSREP: gcomm: connected
111223 1:47:27 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
111223 1:47:27 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
111223 1:47:27 [Note] WSREP: Opened channel 'trimethylxanthine'
111223 1:47:27 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 0, memb_num = 1
111223 1:47:27 [Note] WSREP: Waiting for SST to complete.
111223 1:47:27 [Note] WSREP: STATE_EXCHANGE: sent state UUID: fa9a2969-
111223 1:47:27 [Note] WSREP: STATE EXCHANGE: sent state msg: fa9a2969-
111223 1:47:27 [Note] WSREP: STATE EXCHANGE: got state msg: fa9a2969-
111223 1:47:27 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 0,
members = 1/1 (joined/total),
act_id = 0,
last_appl. = -1,
protocols = 0/2/1 (gcs/repl/appl),
group UUID = a11cef9b-
111223 1:47:27 [Note] WSREP: Flow-control interval: [8, 16]
111223 1:47:27 [Note] WSREP: Restored state OPEN -> JOINED (0)
111223 1:47:27 [Note] WSREP: Member 0 (node1) synced with group.
111223 1:47:27 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
111223 1:47:27 [Note] WSREP: New cluster view: global state: a11cef9b-
111223 1:47:27 [Note] WSREP: SST complete, seqno: 0
111223 1:47:27 [Note] Plugin 'FEDERATED' is disabled.
111223 1:47:27 InnoDB: The InnoDB memory heap is disabled
111223 1:47:27 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111223 1:47:27 InnoDB: Compressed tables use zlib 1.2.3
111223 1:47:27 InnoDB: Using Linux native AIO
111223 1:47:27 InnoDB: Initializing buffer pool, size = 128.0M
111223 1:47:27 InnoDB: Completed initialization of buffer pool
111223 1:47:27 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
111223 1:47:27 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
111223 1:47:27 InnoDB: Waiting for the background threads to start
111223 1:47:28 Percona XtraDB (http://
111223 1:47:28 [Note] Event Scheduler: Loaded 0 events
111223 1:47:28 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.17' socket: '/var/lib/
111223 1:47:28 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
111223 1:47:28 [Note] WSREP: Assign initial position for certification: 0, protocol version: 1
111223 1:47:28 [Note] WSREP: Synchronized with group, ready for connections
111223 1:47:28 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
111223 1:47:29 [Note] WSREP: evs::msg{
} 64
111223 1:47:29 [ERROR] WSREP: exception caused by message: evs::msg{
}
111223 1:47:29 [ERROR] WSREP: state after handling message: evs::proto(
current_
} joined {
} left {
} partitioned {
}),
input_map=
}
,recovery_index= (0,0),evs:
111223 1:47:29 [ERROR] WSREP: exception from gcomm, backend must be restarted:
at gcomm/src/
111223 1:47:29 [Note] WSREP: Received self-leave message.
111223 1:47:29 [Note] WSREP: Flow-control interval: [0, 0]
111223 1:47:29 [Note] WSREP: Received SELF-LEAVE. Closing connection.
111223 1:47:29 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 0)
111223 1:47:29 [Note] WSREP: RECV thread exiting 0: Success
111223 1:47:29 [Note] WSREP: New cluster view: global state: a11cef9b-
111223 1:47:29 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
111223 1:47:29 [Note] WSREP: applier thread exiting (code:0)
When I try to shutdown it, it hung in
111223 1:49:26 [Note] /usr/sbin/mysqld: Normal shutdown
111223 1:49:26 [Note] WSREP: Stop replication
111223 1:49:26 [Note] WSREP: Closing send monitor...
111223 1:49:26 [Note] WSREP: Closed send monitor.
config is:
[mysqld]
datadir=/mnt/data
user=mysql
binlog_format=ROW
wsrep_provider=
wsrep_cluster_
wsrep_slave_
wsrep_cluster_
wsrep_sst_
wsrep_node_
innodb_
innodb_
affects: | codership-mysql → galera |
Changed in galera: | |
milestone: | 23.1.2 → 23.2.1 |
Changed in galera: | |
milestone: | 23.2.1 → 24.2.5 |
milestone: | 24.2.5 → none |
This is a consequence of lp:908025 - Node B didn't crash and was trying to reconnect to Node A. However Node B was in non-primary component at the moment, whereas Node A was explicitly primary and in this particular case it would be desirable that Node B obeys Node A.