Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC

Deadlock between MDL, pre-commit and TOI under high DDL concurrency.

Bug #1274260 reported by Raghavendra D Prabhu
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
New
Undecided
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Confirmed
High
Unassigned

Bug Description

Here, we see threads primarily in

a) pre-commit stage
b) Preparing for TOI
c) MDL wait.

There is a deadlock here requiring an external signal to kill it
(to produce this backtrace). Full bt and consoleText will be
attached.

Analysis:

'DROP TABLE IF EXISTS f' (Thread 41) is waiting for MDL lock on
"CREATE TABLE f AS SELECT * FROM `table1_innodb_int_autoinc"
(Thread 25) which is waiting in commit_monitor (pre-commit) with other DDL's
TOIs. The other TOIs are waiting on Thread 41 to complete, thus
completing the deadlock.

Backtrace: (truncated for brevity, full bt attached)
====================

Thread 41 (Thread 0x7f89e8271700 (LWP 14619)):
#0 0x00007f89ff01998e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000671d9e in inline_mysql_cond_timedwait (this=0x1e0f560, owner=0x1e0f420, abs_timeout=0x7f89e826ebf0, set_status_on_timeout=false, wait_state_name=0x0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/include/mysql/psi/mysql_thread.h:1199
#2 MDL_wait::timed_wait (this=0x1e0f560, owner=0x1e0f420, abs_timeout=0x7f89e826ebf0, set_status_on_timeout=false, wait_state_name=0x0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/mdl.cc:1337
#3 0x00000000006740af in MDL_context::acquire_lock (this=0x1e0f560, mdl_request=0x7f896c005050, lock_wait_timeout=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/mdl.cc:2352
#4 0x000000000067431b in MDL_context::acquire_locks (this=0x1e0f560, mdl_requests=<value optimized out>, lock_wait_timeout=31536000) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/mdl.cc:2464
#5 0x00000000006b70fe in lock_table_names (thd=0x1e0f420, tables_start=0x7f896c004cb0, tables_end=<value optimized out>, lock_wait_timeout=<value optimized out>, flags=0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_base.cc:4856
#6 0x0000000000757777 in mysql_rm_table (thd=0x1e0f420, tables=0x7f896c004cb0, if_exists=1 '\001', drop_temporary=0 '\000') at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_table.cc:2111
#7 0x0000000000701e37 in mysql_execute_command (thd=0x1e0f420) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:4310
#8 0x00000000007072f8 in mysql_parse (thd=0x1e0f420, rawbuf=<value optimized out>, length=31529528, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7188
#9 0x0000000000707421 in wsrep_mysql_parse (thd=0x1e0f420, rawbuf=0x7f896c004bd0 "DROP TABLE IF EXISTS f", length=<value optimized out>, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#10 0x00000000007096a4 in dispatch_command (command=COM_QUERY, thd=0x1e0f420, packet=0x1dc3c91 "DROP TABLE IF EXISTS f ", packet_length=23) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#11 0x000000000070aa73 in do_command (thd=0x1e0f420) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#12 0x00000000006d431f in do_handle_one_connection (thd_arg=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#13 0x00000000006d44f7 in handle_one_connection (arg=0x1e0f420) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#14 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7f89e82b2700 (LWP 14617)):
#0 0x00007f89ff0195bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f89fa8b3666 in wait (this=0x18ba620, obj=...) at galerautils/src/gu_lock.hpp:56
#2 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x18ba620, obj=...) at galera/src/monitor.hpp:121
#3 0x00007f89fa8aec64 in galera::ReplicatorSMM::to_isolation_begin (this=0x18b9ae0, trx=0x7f8968027580, meta=<value optimized out>) at galera/src/replicator_smm.cpp:1007
#4 0x00007f89fa8c0e7e in galera_to_execute_start (gh=<value optimized out>, conn_id=16, keys=<value optimized out>, keys_num=2147483450, data=0x7f89e82aff90, count=1, meta=0x1e403b8) at galera/src/wsrep_provider.cpp:711
#5 0x00000000005b4a05 in wsrep_TOI_begin (thd=0x1e3e050, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1220
#6 wsrep_to_isolation_begin (thd=0x1e3e050, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1387
#7 0x00000000007058f9 in mysql_execute_command (thd=0x1e3e050) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:4303
#8 0x00000000007072f8 in mysql_parse (thd=0x1e3e050, rawbuf=<value optimized out>, length=31721064, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7188
#9 0x0000000000707421 in wsrep_mysql_parse (thd=0x1e3e050, rawbuf=0x7f8968004bd0 "DROP TABLE IF EXISTS t", length=<value optimized out>, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#10 0x00000000007096a4 in dispatch_command (command=COM_QUERY, thd=0x1e3e050, packet=0x1d620f1 "DROP TABLE IF EXISTS t ", packet_length=23) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#11 0x000000000070aa73 in do_command (thd=0x1e3e050) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#12 0x00000000006d431f in do_handle_one_connection (thd_arg=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#13 0x00000000006d44f7 in handle_one_connection (arg=0x1e3e050) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#14 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7f89e81ae700 (LWP 14625)):
#0 0x00007f89ff0195bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f89fa8b3666 in wait (this=0x18ba620, obj=...) at galerautils/src/gu_lock.hpp:56
#2 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x18ba620, obj=...) at galera/src/monitor.hpp:121
#3 0x00007f89fa8aec64 in galera::ReplicatorSMM::to_isolation_begin (this=0x18b9ae0, trx=0x7f8958037f90, meta=<value optimized out>) at galera/src/replicator_smm.cpp:1007
#4 0x00007f89fa8c0e7e in galera_to_execute_start (gh=<value optimized out>, conn_id=20, keys=<value optimized out>, keys_num=2147483442, data=0x7f89e81ab830, count=1, meta=0x1e611f8) at galera/src/wsrep_provider.cpp:711
#5 0x00000000005b4a05 in wsrep_TOI_begin (thd=0x1e5ee90, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1220
#6 wsrep_to_isolation_begin (thd=0x1e5ee90, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1387
#7 0x00000000008462e6 in Sql_cmd_alter_table::execute (this=<value optimized out>, thd=0x1e5ee90) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_alter.cc:317
#8 0x00000000007033ff in mysql_execute_command (thd=0x1e5ee90) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:5696
#9 0x00000000007072f8 in mysql_parse (thd=0x1e5ee90, rawbuf=<value optimized out>, length=31855784, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7188
#10 0x0000000000707421 in wsrep_mysql_parse (thd=0x1e5ee90, rawbuf=0x7f8958004bd0 "ALTER TABLE x RENAME `table1_innodb_int`", length=<value optimized out>, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#11 0x00000000007096a4 in dispatch_command (command=COM_QUERY, thd=0x1e5ee90, packet=0x1e00091 " ALTER TABLE x RENAME `table1_innodb_int`", packet_length=41) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#12 0x000000000070aa73 in do_command (thd=0x1e5ee90) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#13 0x00000000006d431f in do_handle_one_connection (thd_arg=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#14 0x00000000006d44f7 in handle_one_connection (arg=0x1e5ee90) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#15 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7f89e816d700 (LWP 14627)):
#0 0x00007f89ff0195bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f89fa8b3666 in wait (this=0x18ba620, obj=...) at galerautils/src/gu_lock.hpp:56
#2 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x18ba620, obj=...) at galera/src/monitor.hpp:121
#3 0x00007f89fa8aec64 in galera::ReplicatorSMM::to_isolation_begin (this=0x18b9ae0, trx=0x7f895c035690, meta=<value optimized out>) at galera/src/replicator_smm.cpp:1007
#4 0x00007f89fa8c0e7e in galera_to_execute_start (gh=<value optimized out>, conn_id=21, keys=<value optimized out>, keys_num=2147483450, data=0x7f89e816af90, count=1, meta=0x1e183d8) at galera/src/wsrep_provider.cpp:711
#5 0x00000000005b4a05 in wsrep_TOI_begin (thd=0x1e16070, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1220
#6 wsrep_to_isolation_begin (thd=0x1e16070, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1387
#7 0x00000000007058f9 in mysql_execute_command (thd=0x1e16070) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:4303
#8 0x00000000007072f8 in mysql_parse (thd=0x1e16070, rawbuf=<value optimized out>, length=31557256, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7188
#9 0x0000000000707421 in wsrep_mysql_parse (thd=0x1e16070, rawbuf=0x7f895c004bd0 "DROP TABLE IF EXISTS p", length=<value optimized out>, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#10 0x00000000007096a4 in dispatch_command (command=COM_QUERY, thd=0x1e16070, packet=0x1d53d11 "DROP TABLE IF EXISTS p ", packet_length=23) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#11 0x000000000070aa73 in do_command (thd=0x1e16070) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#12 0x00000000006d431f in do_handle_one_connection (thd_arg=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#13 0x00000000006d44f7 in handle_one_connection (arg=0x1e16070) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#14 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f89e0cef700 (LWP 14637)):
#0 0x00007f89ff0195bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f89fa8b3666 in wait (this=0x18ba620, obj=...) at galerautils/src/gu_lock.hpp:56
#2 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x18ba620, obj=...) at galera/src/monitor.hpp:121
#3 0x00007f89fa8aec64 in galera::ReplicatorSMM::to_isolation_begin (this=0x18b9ae0, trx=0x7f89400374b0, meta=<value optimized out>) at galera/src/replicator_smm.cpp:1007
#4 0x00007f89fa8c0e7e in galera_to_execute_start (gh=<value optimized out>, conn_id=26, keys=<value optimized out>, keys_num=2147483442, data=0x7f89e0cec830, count=1, meta=0x1dcc9b8) at galera/src/wsrep_provider.cpp:711
#5 0x00000000005b4a05 in wsrep_TOI_begin (thd=0x1dca650, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1220
#6 wsrep_to_isolation_begin (thd=0x1dca650, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1387
#7 0x00000000008462e6 in Sql_cmd_alter_table::execute (this=<value optimized out>, thd=0x1dca650) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_alter.cc:317
#8 0x00000000007033ff in mysql_execute_command (thd=0x1dca650) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:5696
#9 0x00000000007072f8 in mysql_parse (thd=0x1dca650, rawbuf=<value optimized out>, length=31247464, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7188
#10 0x0000000000707421 in wsrep_mysql_parse (thd=0x1dca650, rawbuf=0x7f8940004bd0 "ALTER TABLE e RENAME `table1000_innodb_key_pk_parts_2_int_autoinc`", length=<value optimized out>, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#11 0x00000000007096a4 in dispatch_command (command=COM_QUERY, thd=0x1dca650, packet=0x1dce251 " ALTER TABLE e RENAME `table1000_innodb_key_pk_parts_2_int_autoinc`", packet_length=67) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#12 0x000000000070aa73 in do_command (thd=0x1dca650) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#13 0x00000000006d431f in do_handle_one_connection (thd_arg=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#14 0x00000000006d44f7 in handle_one_connection (arg=0x1dca650) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#15 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f89e80aa700 (LWP 14633)):
#0 0x00007f89ff0195bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f89fa8b3666 in wait (this=0x18ba620, obj=...) at galerautils/src/gu_lock.hpp:56
#2 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x18ba620, obj=...) at galera/src/monitor.hpp:121
#3 0x00007f89fa8aec64 in galera::ReplicatorSMM::to_isolation_begin (this=0x18b9ae0, trx=0x7f894800c930, meta=<value optimized out>) at galera/src/replicator_smm.cpp:1007
#4 0x00007f89fa8c0e7e in galera_to_execute_start (gh=<value optimized out>, conn_id=24, keys=<value optimized out>, keys_num=2147483450, data=0x7f89e80a6130, count=1, meta=0x1db3848) at galera/src/wsrep_provider.cpp:711
#5 0x00000000005b4a05 in wsrep_TOI_begin (thd=0x1db14e0, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1220
#6 wsrep_to_isolation_begin (thd=0x1db14e0, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1387
#7 0x0000000000754ae1 in mysql_create_like_table (thd=0x1db14e0, table=0x7f8948004ce8, src_table=0x7f8948005310, create_info=0x7f89e80a8670) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_table.cc:5285
#8 0x00000000007064de in mysql_execute_command (thd=0x1db14e0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:3681
#9 0x00000000007072f8 in mysql_parse (thd=0x1db14e0, rawbuf=<value optimized out>, length=31144696, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7188
#10 0x0000000000707421 in wsrep_mysql_parse (thd=0x1db14e0, rawbuf=0x7f8948004bd0 "CREATE TABLE g LIKE `table1000_innodb_int_autoinc`", length=<value optimized out>, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#11 0x00000000007096a4 in dispatch_command (command=COM_QUERY, thd=0x1db14e0, packet=0x1db50e1 " CREATE TABLE g LIKE `table1000_innodb_int_autoinc` ", packet_length=53) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#12 0x000000000070aa73 in do_command (thd=0x1db14e0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#13 0x00000000006d431f in do_handle_one_connection (thd_arg=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#14 0x00000000006d44f7 in handle_one_connection (arg=0x1db14e0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#15 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f89e83b6700 (LWP 14609)):
#0 0x00007f89ff0195bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f89fa8b3666 in wait (this=0x18ba620, obj=...) at galerautils/src/gu_lock.hpp:56
#2 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x18ba620, obj=...) at galera/src/monitor.hpp:121
#3 0x00007f89fa8ae98a in galera::ReplicatorSMM::pre_commit (this=0x18b9ae0, trx=0x7f89780881c0, meta=<value optimized out>) at galera/src/replicator_smm.cpp:761
#4 0x00007f89fa8bf728 in galera_pre_commit (gh=<value optimized out>, conn_id=12, trx_handle=<value optimized out>, flags=<value optimized out>, meta=0x1e4e378) at galera/src/wsrep_provider.cpp:496
#5 0x00000000007ba152 in wsrep_run_wsrep_commit (thd=0x1e4c010, hton=<value optimized out>, all=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_hton.cc:443
#6 0x00000000007baae3 in wsrep_prepare (hton=0x18e1540, thd=0x1e4c010, all=false) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_hton.cc:159
#7 0x00000000005c37ac in ha_prepare_low (thd=0x1e4c010, all=false) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/handler.cc:2181
#8 0x00000000005c50cc in ha_commit_trans (thd=0x1e4c010, all=<value optimized out>, ignore_global_read_lock=false) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/handler.cc:1519
#9 0x000000000079f765 in trans_commit_stmt (thd=0x1e4c010) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/transaction.cc:473
#10 0x00000000006eb6b0 in select_create::send_eof (this=0x7f8978005b58) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_insert.cc:4420
#11 0x00000000006e1b67 in do_select (this=0x7f8978005cd8) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_executor.cc:980
#12 JOIN::exec (this=0x7f8978005cd8) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_executor.cc:191
#13 0x0000000000727f30 in mysql_execute_select (thd=0x1e4c010, tables=0x7f8978005438, wild_num=1, fields=<value optimized out>, conds=0x0, order=<value optimized out>, group=0x1e4ee40, having=0x0, select_options=2416184064, result=0x7f8978005b58, unit=0x1e4e6f8, select_lex=0x1e4ed40) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_select.cc:1101
#14 mysql_select (thd=0x1e4c010, tables=0x7f8978005438, wild_num=1, fields=<value optimized out>, conds=0x0, order=<value optimized out>, group=0x1e4ee40, having=0x0, select_options=2416184064, result=0x7f8978005b58, unit=0x1e4e6f8, select_lex=0x1e4ed40) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_select.cc:1222
#15 0x00000000007287b7 in handle_select (thd=0x1e4c010, result=0x7f8978005b58, setup_tables_done_option=0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_select.cc:111
#16 0x0000000000706be5 in mysql_execute_command (thd=0x1e4c010) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:3667
#17 0x00000000007072f8 in mysql_parse (thd=0x1e4c010, rawbuf=<value optimized out>, length=31778344, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7188
#18 0x0000000000707421 in wsrep_mysql_parse (thd=0x1e4c010, rawbuf=0x7f8978004bd0 "CREATE TABLE f AS SELECT * FROM `table1_innodb_int_autoinc`", length=<value optimized out>, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#19 0x00000000007096a4 in dispatch_command (command=COM_QUERY, thd=0x1e4c010, packet=0x1e39af1 " CREATE TABLE f AS SELECT * FROM `table1_innodb_int_autoinc`", packet_length=61) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#20 0x000000000070aa73 in do_command (thd=0x1e4c010) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#21 0x00000000006d431f in do_handle_one_connection (thd_arg=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#22 0x00000000006d44f7 in handle_one_connection (arg=0x1e4c010) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#23 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f89e81ef700 (LWP 14623)):
#0 0x00007f89ff0195bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f89fa8b3666 in wait (this=0x18ba620, obj=...) at galerautils/src/gu_lock.hpp:56
#2 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x18ba620, obj=...) at galera/src/monitor.hpp:121
#3 0x00007f89fa8ae98a in galera::ReplicatorSMM::pre_commit (this=0x18b9ae0, trx=0x7f896403ed50, meta=<value optimized out>) at galera/src/replicator_smm.cpp:761
#4 0x00007f89fa8bf728 in galera_pre_commit (gh=<value optimized out>, conn_id=19, trx_handle=<value optimized out>, flags=<value optimized out>, meta=0x1d9e448) at galera/src/wsrep_provider.cpp:496
#5 0x00000000007ba152 in wsrep_run_wsrep_commit (thd=0x1d9c0e0, hton=<value optimized out>, all=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_hton.cc:443
#6 0x00000000007baae3 in wsrep_prepare (hton=0x18e1540, thd=0x1d9c0e0, all=false) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_hton.cc:159
#7 0x00000000005c37ac in ha_prepare_low (thd=0x1d9c0e0, all=false) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/handler.cc:2181
#8 0x00000000005c50cc in ha_commit_trans (thd=0x1d9c0e0, all=<value optimized out>, ignore_global_read_lock=false) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/handler.cc:1519
#9 0x000000000079f765 in trans_commit_stmt (thd=0x1d9c0e0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/transaction.cc:473
#10 0x00000000006eb6b0 in select_create::send_eof (this=0x7f8964005b58) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_insert.cc:4420
#11 0x00000000006e1b67 in do_select (this=0x7f8964005cd8) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_executor.cc:980
#12 JOIN::exec (this=0x7f8964005cd8) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_executor.cc:191
#13 0x0000000000727f30 in mysql_execute_select (thd=0x1d9c0e0, tables=0x7f8964005438, wild_num=1, fields=<value optimized out>, conds=0x0, order=<value optimized out>, group=0x1d9ef10, having=0x0, select_options=2416184064, result=0x7f8964005b58, unit=0x1d9e7c8, select_lex=0x1d9ee10) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_select.cc:1101
#14 mysql_select (thd=0x1d9c0e0, tables=0x7f8964005438, wild_num=1, fields=<value optimized out>, conds=0x0, order=<value optimized out>, group=0x1d9ef10, having=0x0, select_options=2416184064, result=0x7f8964005b58, unit=0x1d9e7c8, select_lex=0x1d9ee10) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_select.cc:1222
#15 0x00000000007287b7 in handle_select (thd=0x1d9c0e0, result=0x7f8964005b58, setup_tables_done_option=0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_select.cc:111
#16 0x0000000000706be5 in mysql_execute_command (thd=0x1d9c0e0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:3667
#17 0x00000000007072f8 in mysql_parse (thd=0x1d9c0e0, rawbuf=<value optimized out>, length=31057656, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7188
#18 0x0000000000707421 in wsrep_mysql_parse (thd=0x1d9c0e0, rawbuf=0x7f8964004bd0 "CREATE TABLE h AS SELECT * FROM `table1_innodb_int_autoinc`", length=<value optimized out>, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#19 0x00000000007096a4 in dispatch_command (command=COM_QUERY, thd=0x1d9c0e0, packet=0x1d7efe1 " CREATE TABLE h AS SELECT * FROM `table1_innodb_int_autoinc`", packet_length=61) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#20 0x000000000070aa73 in do_command (thd=0x1d9c0e0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#21 0x00000000006d431f in do_handle_one_connection (thd_arg=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#22 0x00000000006d44f7 in handle_one_connection (arg=0x1d9c0e0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#23 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f89e8334700 (LWP 14613)):
#0 0x00007f89ff0195bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f89fa8b3666 in wait (this=0x18ba620, obj=...) at galerautils/src/gu_lock.hpp:56
#2 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x18ba620, obj=...) at galera/src/monitor.hpp:121
#3 0x00007f89fa8ae98a in galera::ReplicatorSMM::pre_commit (this=0x18b9ae0, trx=0x7f89700420c0, meta=<value optimized out>) at galera/src/replicator_smm.cpp:761
#4 0x00007f89fa8bf728 in galera_pre_commit (gh=<value optimized out>, conn_id=14, trx_handle=<value optimized out>, flags=<value optimized out>, meta=0x1e37778) at galera/src/wsrep_provider.cpp:496
#5 0x00000000007ba152 in wsrep_run_wsrep_commit (thd=0x1e35410, hton=<value optimized out>, all=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_hton.cc:443
#6 0x00000000007baae3 in wsrep_prepare (hton=0x18e1540, thd=0x1e35410, all=true) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_hton.cc:159
#7 0x00000000005c37ac in ha_prepare_low (thd=0x1e35410, all=true) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/handler.cc:2181
#8 0x00000000005c50cc in ha_commit_trans (thd=0x1e35410, all=<value optimized out>, ignore_global_read_lock=false) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/handler.cc:1519
#9 0x000000000079f849 in trans_commit_implicit (thd=0x1e35410) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/transaction.cc:305
#10 0x00000000006eb6b9 in select_create::send_eof (this=0x7f8970005b60) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_insert.cc:4421
#11 0x00000000006e1b67 in do_select (this=0x7f8970005ce0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_executor.cc:980
#12 JOIN::exec (this=0x7f8970005ce0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_executor.cc:191
#13 0x0000000000727f30 in mysql_execute_select (thd=0x1e35410, tables=0x7f8970005440, wild_num=1, fields=<value optimized out>, conds=0x0, order=<value optimized out>, group=0x1e38240, having=0x0, select_options=2416708096, result=0x7f8970005b60, unit=0x1e37af8, select_lex=0x1e38140) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_select.cc:1101
#14 mysql_select (thd=0x1e35410, tables=0x7f8970005440, wild_num=1, fields=<value optimized out>, conds=0x0, order=<value optimized out>, group=0x1e38240, having=0x0, select_options=2416708096, result=0x7f8970005b60, unit=0x1e37af8, select_lex=0x1e38140) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_select.cc:1222
#15 0x00000000007287b7 in handle_select (thd=0x1e35410, result=0x7f8970005b60, setup_tables_done_option=0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_select.cc:111
#16 0x0000000000706be5 in mysql_execute_command (thd=0x1e35410) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:3667
#17 0x00000000007072f8 in mysql_parse (thd=0x1e35410, rawbuf=<value optimized out>, length=31685160, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7188
#18 0x0000000000707421 in wsrep_mysql_parse (thd=0x1e35410, rawbuf=0x7f8970004bd0 "CREATE TABLE v AS SELECT * FROM `table1000_innodb_int_autoinc`", length=<value optimized out>, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#19 0x00000000007096a4 in dispatch_command (command=COM_QUERY, thd=0x1e35410, packet=0x1e2c6d1 " CREATE TABLE v AS SELECT * FROM `table1000_innodb_int_autoinc`", packet_length=64) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#20 0x000000000070aa73 in do_command (thd=0x1e35410) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#21 0x00000000006d431f in do_handle_one_connection (thd_arg=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#22 0x00000000006d44f7 in handle_one_connection (arg=0x1e35410) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#23 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f89e812c700 (LWP 14629)):
#0 0x00007f89ff0195bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f89fa8b3666 in wait (this=0x18ba620, obj=...) at galerautils/src/gu_lock.hpp:56
#2 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x18ba620, obj=...) at galera/src/monitor.hpp:121
#3 0x00007f89fa8aec64 in galera::ReplicatorSMM::to_isolation_begin (this=0x18b9ae0, trx=0x7f895000b940, meta=<value optimized out>) at galera/src/replicator_smm.cpp:1007
#4 0x00007f89fa8c0e7e in galera_to_execute_start (gh=<value optimized out>, conn_id=22, keys=<value optimized out>, keys_num=2147483450, data=0x7f89e8129f90, count=1, meta=0x1dfb8a8) at galera/src/wsrep_provider.cpp:711
#5 0x00000000005b4a05 in wsrep_TOI_begin (thd=0x1df9540, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1220
#6 wsrep_to_isolation_begin (thd=0x1df9540, db_=<value optimized out>, table_=<value optimized out>, table_list=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_mysqld.cc:1387
#7 0x00000000007058f9 in mysql_execute_command (thd=0x1df9540) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:4303
#8 0x00000000007072f8 in mysql_parse (thd=0x1df9540, rawbuf=<value optimized out>, length=31439704, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7188
#9 0x0000000000707421 in wsrep_mysql_parse (thd=0x1df9540, rawbuf=0x7f8950004bd0 "DROP TABLE `table1_innodb_int_autoinc`", length=<value optimized out>, parser_state=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#10 0x00000000007096a4 in dispatch_command (command=COM_QUERY, thd=0x1df9540, packet=0x1e1c8d1 " DROP TABLE `table1_innodb_int_autoinc` ", packet_length=40) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#11 0x000000000070aa73 in do_command (thd=0x1df9540) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#12 0x00000000006d431f in do_handle_one_connection (thd_arg=<value optimized out>) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#13 0x00000000006d44f7 in handle_one_connection (arg=0x1df9540) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#14 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f89e3fbe700 (LWP 14436)):
#0 0x00007f89ff0195bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f89fa78118e in fifo_lock_get (q=0x7f89ff2ef010, err=0x7f89e3fbc85c) at galerautils/src/gu_fifo.c:233
#2 gu_fifo_get_head (q=0x7f89ff2ef010, err=0x7f89e3fbc85c) at galerautils/src/gu_fifo.c:292
#3 0x00007f89fa85f724 in gcs_recv (conn=0x18c1580, action=0x7f89e3fbc890) at gcs/src/gcs.c:1674
#4 0x00007f89fa889558 in galera::GcsActionSource::process (this=0x18ba0c0, recv_ctx=0x7f89d0000990, exit_loop=@0x7f89e3fbd04f) at galera/src/gcs_action_source.cpp:171
#5 0x00007f89fa8a5ac3 in galera::ReplicatorSMM::async_recv (this=0x18b9ae0, recv_ctx=0x7f89d0000990) at galera/src/replicator_smm.cpp:352
#6 0x00007f89fa8bfa73 in galera_recv (gh=<value optimized out>, recv_ctx=<value optimized out>) at galera/src/wsrep_provider.cpp:222
#7 0x00000000005bdc3f in wsrep_replication_process (thd=0x7f89d0000990) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/wsrep_thd.cc:309
#8 0x00000000005ae060 in start_wsrep_THD (arg=0x5bdbf0) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/release/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/mysqld.cc:5484
#9 0x00007f89ff0159d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f89fd733b6d in clone () from /lib64/libc.so.6

==================================================================================================================================

See original description
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Part (thread apply all bt) and full (thread apply all bt full) of node0 and node1

description: updated
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

The consoleText of ctas (along with other things)

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

The grammar I am seeing with is galera_stress-ctas.yy and following is how it is invoked:

        timeout -s 11 100m perl runall-new.pl --basedir=${MYSQL_BASEDIR} --vardir=$CVARDIR --galera=ms $EXTRAOPTS --reporter=$RPTR --grammar=conf/galera/galera_stress-ctas.yy --gendata=conf/galera/galera_stress-56.zz --threads=$RUNT --queries=$RUND --mysqld=--wsrep-provider=$MYSQL_BASEDIR/lib/libgalera_smm.so --mysqld=--innodb_flush_method=O_DIRECT --seed=time || failed

(rest of the details in consoleText)

summary: - Deadlock between MDL, pre-commit and TOI
+ Deadlock between MDL, pre-commit and TOI under high DDL concurrency.
Raghavendra D Prabhu (raghavendra-prabhu)
description: updated
description: updated
Raghavendra D Prabhu (raghavendra-prabhu)
tags: added: ctas ddl deadlock mdl
Krunal Bauskar (krunal-bauskar)
Changed in percona-xtradb-cluster:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Ovais Tariq (ovais-tariq) wrote :

Is there any update on this bug? We got hit by the same issue where the ALTER was blocked on MDL and in turn was blocking a transaction in pre-commit stage.

Revision history for this message
Krunal Bauskar (krunal-bauskar) wrote :

* Problem is here is CTAS is actually a DDL (cum DML) command that is executed using DML flow.
* Conflicting DDL can cause deadlock as explained above.

As galera protocol goes DDL are serialized so you can't replicate DDL concurrently.

-----------------

With that background and given the problem in PXC-5.7 we have disabled CTAS
but still left it out in 5.6 if user is operating CTAS in controlled fashion.
(to avoid immediate workload incompatibility on upgrade).

-----------------

If you are still hitting this problem then only solution I see is to avoid use of CTAS.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-991

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.