Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC

Bug #1210509
Comment #14

Comment 14 for bug 1210509

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-10-02:

#14

Hitting this with PS 5.6 - due to https://bugs.launchpad.net/percona-server/+bug/1168265

"/media/Tintin/Work/code/randgen/pxc" is not a core dump: Is a directory
backtrace.gdb:19: Error in sourced command file:
No stack.
# 2013-10-03T01:59:25 [6270] Whether to confirm potentially dangerous operations is on.
"/media/Tintin/Work/code/randgen/pxc" is not a core dump: Is a directory
# 2013-10-03T01:59:26 [6270] Whether to confirm potentially dangerous operations is on.
# 2013-10-03T01:59:26 [6270] The last 100 lines from /pxc56/mysql-test/var/node0/data//../mysql.err :
2013-10-03 01:59:04 6691 [Note] WSREP: Aborting query: START TRANSACTION
2013-10-03 01:59:04 6691 [Note] WSREP: withdraw for BF trx: 4038, state: 4
2013-10-03 01:59:04 6691 [Note] WSREP: victim 4038 in state 4
WSREP: conc slot cancel not supported
2013-10-03 01:59:04 6691 [Note] WSREP: MDL conflict
request: (13 seqno 271 wsrep (2, 1, 0) cmd 3 9 DROP TABLE `table1000_innodb_int`)
granted: (16 seqno 267 wsrep (0, 1, 1) cmd 3 6 INSERT INTO p SELECT * FROM `table1000_innodb_int`)
2013-10-03 01:59:04 6691 [Note] WSREP: MDL ticket: type: shared read space: TABLE db: test name: table1000_innodb_int
2013-10-03 01:59:04 6691 [Note] WSREP: DROP caused BF abort
2013-10-03 01:59:04 6691 [Note] WSREP: MDL ticket: type: shared read space: TABLE db: test name: table1000_innodb_int
2013-10-03 01:59:04 6691 [Note] WSREP: wsrep_abort_thd, by: 139989696304896, victim: 139989694711552
2013-10-03 01:59:04 6691 [Note] WSREP: abort transaction: BF: DROP TABLE `table1000_innodb_int` victim: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: cluster conflict due to high priority abort for threads:
2013-10-03 01:59:04 6691 [Note] WSREP: Winning thread:
   THD: 13, mode: total order, state: executing, conflict: no conflict, seqno: 271
   SQL: DROP TABLE `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: Victim thread:
   THD: 16, mode: local, state: executing, conflict: must abort, seqno: 267
   SQL: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: BF kill (1, seqno: 271), victim: (16) trx: 4210
2013-10-03 01:59:04 6691 [Note] WSREP: Aborting query: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: victim 4210 in MUST ABORT state
WSREP: conc slot cancel not supported
2013-10-03 01:59:04 6691 [Note] WSREP: client rollback due to BF abort for (14), query: DELETE IGNORE FROM `table1000_innodb_int` WHERE `col_int_key` IN ( 620298240 , 3424452608 , 214 , 144 , 7 ) ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 3
2013-10-03 01:59:04 6691 [Note] WSREP: abort in exec query state, avoiding autocommit
2013-10-03 01:59:04 6691 [Note] WSREP: BF Aborted, thd: 14 is_AC: 1, retry: 1 - 1 SQL: DELETE IGNORE FROM `table1000_innodb_int` WHERE `col_int_key` IN ( 620298240 , 3424452608 , 214 , 144 , 7 ) ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 3
2013-10-03 01:59:04 6691 [Note] WSREP: releasing retry_query: conf 0 sent 0 kill 0 errno 0 SQL DELETE IGNORE FROM `table1000_innodb_int` WHERE `col_int_key` IN ( 620298240 , 3424452608 , 214 , 144 , 7 ) ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 3
mysqld: /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/protocol.cc:526: void Protocol::end_statement(): Assertion `0' failed.
20:29:04 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=20
max_threads=153
thread_count=19
connection_count=19
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69207 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x21dedf0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f51e41abed8 thread_stack 0x40000
/pxc56/bin/mysqld(my_print_stacktrace+0x2c)[0x942498]
2013-10-03 01:59:04 6691 [Note] WSREP: MDL conflict
request: (13 seqno 271 wsrep (2, 1, 0) cmd 3 9 DROP TABLE `table1000_innodb_int`)
granted: (16 seqno 267 wsrep (3, 1, 1) cmd 3 6 INSERT INTO p SELECT * FROM `table1000_innodb_int`)
2013-10-03 01:59:04 6691 [Note] WSREP: MDL ticket: type: shared read space: TABLE db: test name: table1000_innodb_int
2013-10-03 01:59:04 6691 [Note] WSREP: DROP caused BF abort
2013-10-03 01:59:04 6691 [Note] WSREP: MDL ticket: type: shared read space: TABLE db: test name: table1000_innodb_int
2013-10-03 01:59:04 6691 [Note] WSREP: wsrep_abort_thd, by: 139989696304896, victim: 139989694711552
2013-10-03 01:59:04 6691 [Note] WSREP: abort transaction: BF: DROP TABLE `table1000_innodb_int` victim: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: cluster conflict due to high priority abort for threads:
2013-10-03 01:59:04 6691 [Note] WSREP: Winning thread:
   THD: 13, mode: total order, state: executing, conflict: no conflict, seqno: 271
   SQL: DROP TABLE `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: Victim thread:
   THD: 16, mode: local commit, state: executing, conflict: must abort, seqno: 267
   SQL: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: BF kill (1, seqno: 271), victim: (16) trx: 4210
2013-10-03 01:59:04 6691 [Note] WSREP: Aborting query: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: withdraw for BF trx: 4210, state: 1
2013-10-03 01:59:04 6691 [Note] WSREP: victim 4210 in MUST ABORT state
WSREP: conc slot cancel not supported
2013-10-03 01:59:04 6691 [Note] WSREP: replay trx: START TRANSACTION 278
/pxc56/bin/mysqld(handle_fatal_signal+0x304)[0x6d8248]
/usr/lib/libpthread.so.0(+0xf870)[0x7f51f8c36870]
/usr/lib/libc.so.6(gsignal+0x39)[0x7f51f6fc43d9]
/usr/lib/libc.so.6(abort+0x148)[0x7f51f6fc57d8]
/usr/lib/libc.so.6(+0x2e506)[0x7f51f6fbd506]
/usr/lib/libc.so.6(+0x2e5b2)[0x7f51f6fbd5b2]
2013-10-03 01:59:04 6691 [Note] WSREP: commit waiting for replaying: replayers 1, thd: (20) conflict: 0 (round: 1)
/pxc56/bin/mysqld(_ZN8Protocol13end_statementEv+0x23c)[0x6d2a44]
/pxc56/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1740)[0x75ca02]
/pxc56/bin/mysqld(_Z10do_commandP3THD+0x50c)[0x75d3c3]
/pxc56/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x129)[0x729a7d]
/pxc56/bin/mysqld(handle_one_connection+0x20)[0x729ba5]
/usr/lib/libpthread.so.0(+0x80a2)[0x7f51f8c2f0a2]
/usr/lib/libc.so.6(clone+0x6d)[0x7f51f707443d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f517c004c50): DELETE IGNORE FROM `table1000_innodb_int` WHERE `col_int_key` IN ( 620298240 , 3424452608 , 214 , 144 , 7 ) ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 3
Connection ID (thread ID): 14
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file
# 2013-10-03T01:59:26 [6270] Test completed with failure status STATUS_SERVER_CRASHED (101)
# 2013-10-03T01:59:26 [6270] GenTest exited with exit status STATUS_SERVER_CRASHED (101)
# 2013-10-03T01:59:26 [6270] Stopping server on port 19300
# 2013-10-03T01:59:26 [6270] Stale connection. Reconnecting
# 2013-10-03T01:59:26 [6270] Reconnect failed due to 2003:Can't connect to MySQL server on '127.0.0.1' (111)
# 2013-10-03T01:59:26 [6270] Server would not shut down properly. Terminate it
# 2013-10-03T01:59:26 [6270] Terminated process 6691
# 2013-10-03T01:59:26 [6270] Stopping server on port 19301
# 2013-10-03T01:59:26 [6270] Stale connection. Reconnecting
# 2013-10-03T01:59:26 [6270] Reconnect failed due to 2003:Can't connect to MySQL server on '127.0.0.1' (111)
# 2013-10-03T01:59:26 [6270] Server would not shut down properly. Terminate it
# 2013-10-03T01:59:26 [6270] Terminated process 6735
# 2013-10-03T01:59:26 [6270] [6270] runall-new.pl will exit with exit status STATUS_SERVER_CRASHED (101)
perl runall-new.pl --basedir=/pxc56 --galera=ms --threads=16 --seed=tim 9.87s user 0.91s system 20% cpu 53.861 total

Will workaround with either altered grammar or adding WSREP_WARN there, preferrably former.

Hitting this with PS 5.6 - due to https://bugs.launchpad.net/percona-server/+bug/1168265

"/media/Tintin/Work/code/randgen/pxc" is not a core dump: Is a directory
backtrace.gdb:19: Error in sourced command file:
No stack.
# 2013-10-03T01:59:25 [6270] Whether to confirm potentially dangerous operations is on.
"/media/Tintin/Work/code/randgen/pxc" is not a core dump: Is a directory
# 2013-10-03T01:59:26 [6270] Whether to confirm potentially dangerous operations is on.
# 2013-10-03T01:59:26 [6270] The last 100 lines from /pxc56/mysql-test/var/node0/data//../mysql.err :
2013-10-03 01:59:04 6691 [Note] WSREP: Aborting query: START TRANSACTION
2013-10-03 01:59:04 6691 [Note] WSREP: withdraw for BF trx: 4038, state: 4
2013-10-03 01:59:04 6691 [Note] WSREP: victim 4038 in state 4
WSREP: conc slot cancel not supported
2013-10-03 01:59:04 6691 [Note] WSREP: MDL conflict
request: (13    seqno 271       wsrep (2, 1, 0) cmd 3 9         DROP TABLE `table1000_innodb_int`)
granted: (16    seqno 267       wsrep (0, 1, 1) cmd 3 6         INSERT INTO p SELECT * FROM `table1000_innodb_int`)
2013-10-03 01:59:04 6691 [Note] WSREP: MDL ticket: type: shared read space: TABLE db: test name: table1000_innodb_int
2013-10-03 01:59:04 6691 [Note] WSREP: DROP caused BF abort
2013-10-03 01:59:04 6691 [Note] WSREP: MDL ticket: type: shared read space: TABLE db: test name: table1000_innodb_int
2013-10-03 01:59:04 6691 [Note] WSREP: wsrep_abort_thd, by: 139989696304896, victim: 139989694711552
2013-10-03 01:59:04 6691 [Note] WSREP: abort transaction: BF: DROP TABLE `table1000_innodb_int` victim: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: cluster conflict due to high priority abort for threads:
2013-10-03 01:59:04 6691 [Note] WSREP: Winning thread:
   THD: 13, mode: total order, state: executing, conflict: no conflict, seqno: 271
   SQL: DROP TABLE `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: Victim thread:
   THD: 16, mode: local, state: executing, conflict: must abort, seqno: 267
   SQL: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: BF kill (1, seqno: 271), victim: (16) trx: 4210
2013-10-03 01:59:04 6691 [Note] WSREP: Aborting query: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: victim 4210 in MUST ABORT state
WSREP: conc slot cancel not supported
2013-10-03 01:59:04 6691 [Note] WSREP: client rollback due to BF abort for (14), query: DELETE IGNORE FROM `table1000_innodb_int` WHERE `col_int_key` IN ( 620298240 , 3424452608 , 214 , 144 , 7 ) ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 3
2013-10-03 01:59:04 6691 [Note] WSREP: abort in exec query state, avoiding autocommit
2013-10-03 01:59:04 6691 [Note] WSREP: BF Aborted, thd: 14 is_AC: 1, retry: 1 - 1 SQL: DELETE IGNORE FROM `table1000_innodb_int` WHERE `col_int_key` IN ( 620298240 , 3424452608 , 214 , 144 , 7 ) ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 3
2013-10-03 01:59:04 6691 [Note] WSREP: releasing retry_query: conf 0 sent 0 kill 0  errno 0 SQL DELETE IGNORE FROM `table1000_innodb_int` WHERE `col_int_key` IN ( 620298240 , 3424452608 , 214 , 144 , 7 ) ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 3
mysqld: /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/protocol.cc:526: void Protocol::end_statement(): Assertion `0' failed.
20:29:04 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=20
max_threads=153
thread_count=19
connection_count=19
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69207 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x21dedf0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f51e41abed8 thread_stack 0x40000
/pxc56/bin/mysqld(my_print_stacktrace+0x2c)[0x942498]
2013-10-03 01:59:04 6691 [Note] WSREP: MDL conflict
request: (13    seqno 271       wsrep (2, 1, 0) cmd 3 9         DROP TABLE `table1000_innodb_int`)
granted: (16    seqno 267       wsrep (3, 1, 1) cmd 3 6         INSERT INTO p SELECT * FROM `table1000_innodb_int`)
2013-10-03 01:59:04 6691 [Note] WSREP: MDL ticket: type: shared read space: TABLE db: test name: table1000_innodb_int
2013-10-03 01:59:04 6691 [Note] WSREP: DROP caused BF abort
2013-10-03 01:59:04 6691 [Note] WSREP: MDL ticket: type: shared read space: TABLE db: test name: table1000_innodb_int
2013-10-03 01:59:04 6691 [Note] WSREP: wsrep_abort_thd, by: 139989696304896, victim: 139989694711552
2013-10-03 01:59:04 6691 [Note] WSREP: abort transaction: BF: DROP TABLE `table1000_innodb_int` victim: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: cluster conflict due to high priority abort for threads:
2013-10-03 01:59:04 6691 [Note] WSREP: Winning thread:
   THD: 13, mode: total order, state: executing, conflict: no conflict, seqno: 271
   SQL: DROP TABLE `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: Victim thread:
   THD: 16, mode: local commit, state: executing, conflict: must abort, seqno: 267
   SQL: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: BF kill (1, seqno: 271), victim: (16) trx: 4210
2013-10-03 01:59:04 6691 [Note] WSREP: Aborting query: INSERT INTO p SELECT * FROM `table1000_innodb_int`
2013-10-03 01:59:04 6691 [Note] WSREP: withdraw for BF trx: 4210, state: 1
2013-10-03 01:59:04 6691 [Note] WSREP: victim 4210 in MUST ABORT state
WSREP: conc slot cancel not supported
2013-10-03 01:59:04 6691 [Note] WSREP: replay trx: START TRANSACTION 278
/pxc56/bin/mysqld(handle_fatal_signal+0x304)[0x6d8248]
/usr/lib/libpthread.so.0(+0xf870)[0x7f51f8c36870]
/usr/lib/libc.so.6(gsignal+0x39)[0x7f51f6fc43d9]
/usr/lib/libc.so.6(abort+0x148)[0x7f51f6fc57d8]
/usr/lib/libc.so.6(+0x2e506)[0x7f51f6fbd506]
/usr/lib/libc.so.6(+0x2e5b2)[0x7f51f6fbd5b2]
2013-10-03 01:59:04 6691 [Note] WSREP: commit waiting for replaying: replayers 1, thd: (20) conflict: 0 (round: 1)
/pxc56/bin/mysqld(_ZN8Protocol13end_statementEv+0x23c)[0x6d2a44]
/pxc56/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1740)[0x75ca02]
/pxc56/bin/mysqld(_Z10do_commandP3THD+0x50c)[0x75d3c3]
/pxc56/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x129)[0x729a7d]
/pxc56/bin/mysqld(handle_one_connection+0x20)[0x729ba5]
/usr/lib/libpthread.so.0(+0x80a2)[0x7f51f8c2f0a2]
/usr/lib/libc.so.6(clone+0x6d)[0x7f51f707443d]

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file
# 2013-10-03T01:59:26 [6270] Test completed with failure status STATUS_SERVER_CRASHED (101)
# 2013-10-03T01:59:26 [6270] GenTest exited with exit status STATUS_SERVER_CRASHED (101)
# 2013-10-03T01:59:26 [6270] Stopping server on port 19300
# 2013-10-03T01:59:26 [6270] Stale connection. Reconnecting
# 2013-10-03T01:59:26 [6270] Reconnect failed due to 2003:Can't connect to MySQL server on '127.0.0.1' (111)
# 2013-10-03T01:59:26 [6270] Server would not shut down properly. Terminate it
# 2013-10-03T01:59:26 [6270] Terminated process 6691
# 2013-10-03T01:59:26 [6270] Stopping server on port 19301
# 2013-10-03T01:59:26 [6270] Stale connection. Reconnecting
# 2013-10-03T01:59:26 [6270] Reconnect failed due to 2003:Can't connect to MySQL server on '127.0.0.1' (111)
# 2013-10-03T01:59:26 [6270] Server would not shut down properly. Terminate it
# 2013-10-03T01:59:26 [6270] Terminated process 6735
# 2013-10-03T01:59:26 [6270] [6270] runall-new.pl will exit with exit status STATUS_SERVER_CRASHED (101)
perl runall-new.pl --basedir=/pxc56 --galera=ms     --threads=16   --seed=tim  9.87s user 0.91s system 20% cpu 53.861 total

Will workaround with either altered grammar or adding WSREP_WARN there, preferrably former.