*** glibc detected *** /usr/sbin/mysqld: free(): corrupted unsorted chunks: 0x00007fc4905bf0b0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7a6e6)[0x7fd492f0c6e6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fd492f109cc]
/usr/sbin/mysqld(free_root+0x90)[0x7c4600]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x30b)[0x5ab6ab]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x5adb69]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x157)[0x648787]
/usr/sbin/mysqld(handle_one_connection+0x51)[0x648931]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc)[0x7fd49323aefc]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd492f7559d]
======= Memory map: ========
00400000-00d85000 r-xp 00000000 ca:01 15227 /usr/sbin/mysqld
00f85000-00ff9000 r--p 00985000 ca:01 15227 /usr/sbin/mysqld
00ff9000-0109e000 rw-p 009f9000 ca:01 15227 /usr/sbin/mysqld
0109e000-010d0000 rw-p 00000000 00:00 0
01c43000-643e9000 rw-p 00000000 00:00 0 [heap]
7fc43c000000-7fc43c86c000 rw-p 00000000 00:00 0
7fc43c86c000-7fc440000000 ---p 00000000 00:00 0
7fc444000000-7fc444e55000 rw-p 00000000 00:00 0
7fc444e55000-7fc448000000 ---p 00000000 00:00 0
7fc44c000000-7fc44f2d3000 rw-p 00000000 00:00 0
7fc44f2d3000-7fc450000000 ---p 00000000 00:00 0
7fc450000000-7fc453e91000 rw-p 00000000 00:00 0
7fc453e91000-7fc454000000 ---p 00000000 00:00 0
7fc454000000-7fc457fe1000 rw-p 00000000 00:00 0
7fc457fe1000-7fc458000000 ---p 00000000 00:00 0
... about 1000 lines of this ...
7fd38ebe7000-7fd390cec000 rw-p 00000000 00:00 0
7fd390cec000-7fd490ced000 rw-s 00000000 ca:f0 33947747 /db/data/galera.cache
7fd490ced000-7fd490d04000 r-xp 00000000 ca:01 133231 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fd490d04000-7fd490f03000 ---p 00017000 ca:01 133231 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fd490f03000-7fd490f04000 r--p 00016000 ca:01 133231 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fd490f04000-7fd490f05000 rw-p 00017000 ca:01 133231 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fd490f05000-7fd490fed000 r-xp 00000000 ca:01 9834 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd490fed000-7fd4911ed000 ---p 000e8000 ca:01 9834 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd4911ed000-7fd4911f5000 r--p 000e8000 ca:01 9834 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd4911f5000-7fd4911f7000 rw-p 000f0000 ca:01 9834 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd4911f7000-7fd49120c000 rw-p 00000000 00:00 0
7fd49120c000-7fd491394000 r-xp 00000000 ca:01 133191 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd491394000-7fd491594000 ---p 00188000 ca:01 133191 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd491594000-7fd4915ad000 r--p 00188000 ca:01 133191 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd4915ad000-7fd4915b7000 rw-p 001a1000 ca:01 133191 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd4915b7000-7fd4915bb000 rw-p 00000000 00:00 0
7fd4915bb000-7fd491605000 r-xp 00000000 ca:01 133224 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd491605000-7fd491804000 ---p 0004a000 ca:01 133224 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd491804000-7fd491807000 r--p 00049000 ca:01 133224 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd491807000-7fd49180c000 rw-p 0004c000 ca:01 133224 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd49180c000-7fd491a34000 r-xp 00000000 ca:01 15048 /usr/lib/libgalera_smm.so
:7fd491a34000-7fd491c34000 ---p 00228000 ca:01 15048 /usr/lib/libgalera_smm.so
7fd491c34000-7fd491c3a000 r--p 00228000 ca:01 15048 /usr/lib/libgalera_smm.so
7fd491c3a000-7fd491c3e000 rw-p 0022e000 ca:01 15048 /usr/lib/libgalera_smm.so
7fd491c3e000-7fd491c3f000 rw-p 00000000 00:00 0
7fd491c3f000-7fd491c4b000 r-xp 00000000 ca:01 133209 /lib/x86_64-linux-gnu/libnss_files-2.13.so
7fd491c4b000-7fd491e4a000 ---p 0000c000 ca:01 133209 /lib/x86_64-linux-gnu/libnss_files-2.13.so
7fd491e4a000-7fd491e4b000 r--p 0000b000 ca:01 133209 /lib/x86_64-linux-gnu/libnss_files-2.13.so
7fd491e4b000-7fd491e4c000 rw-p 0000c000 ca:01 133209 /lib/x86_64-linux-gnu/libnss_files-2.13.so
7fd491e4c000-7fd491e56000 r-xp 00000000 ca:01 133211 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7fd491e56000-7fd492056000 ---p 0000a000 ca:01 133211 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7fd492056000-7fd492057000 r--p 0000a000 ca:01 133211 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7fd492057000-7fd492058000 rw-p 0000b000 ca:01 133211 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7fd492058000-7fd49206f000 r-xp 00000000 ca:01 133206 /lib/x86_64-linux-gnu/libnsl-2.13.so
7fd49206f000-7fd49226e000 ---p 00017000 ca:01 133206 /lib/x86_64-linux-gnu/libnsl-2.13.so
7fd49226e000-7fd49226f000 r--p 00016000 ca:01 133206 /lib/x86_64-linux-gnu/libnsl-2.13.so
7fd49226f000-7fd492270000 rw-p 00017000 ca:01 133206 /lib/x86_64-linux-gnu/libnsl-2.13.so
7fd492270000-7fd492272000 rw-p 00000000 00:00 0
7fd492272000-7fd49227a000 r-xp 00000000 ca:01 133207 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7fd49227a000-7fd492479000 ---p 00008000 ca:01 133207 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7fd492479000-7fd49247a000 r--p 00007000 ca:01 133207 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7fd49247a000-7fd49247b000 rw-p 00008000 ca:01 133207 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7fd49247b000-7fd492490000 r-xp 00000000 ca:01 133198 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd492490000-7fd49268f000 ---p 00015000 ca:01 133198 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd49268f000-7fd492690000 r--p 00014000 ca:01 133198 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd492690000-7fd492691000 rw-p 00015000 ca:01 133198 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd492691000-7fd492692000 ---p 00000000 00:00 0
7fd492692000-7fd492e92000 rw-p 00000000 00:00 0
7fd492e92000-7fd493029000 r-xp 00000000 ca:01 133187 /lib/x86_64-linux-gnu/libc-2.13.so
7fd493029000-7fd493228000 ---p 00197000 ca:01 133187 /lib/x86_64-linux-gnu/libc-2.13.so
7fd493228000-7fd49322c000 r--p 00196000 ca:01 133187 /lib/x86_64-linux-gnu/libc-2.13.so
7fd49322c000-7fd49322d000 rw-p 0019a000 ca:01 133187 /lib/x86_64-linux-gnu/libc-2.13.so
7fd49322d000-7fd493233000 rw-p 00000000 00:00 0
7fd493233000-7fd49324b000 r-xp 00000000 ca:01 133219 /lib/x86_64-linux-gnu/libpthread-2.13.so
7fd49324b000-7fd49344a000 ---p 00018000 ca:01 133219 /lib/x86_64-linux-gnu/libpthread-2.13.so
7fd49344a000-7fd49344b000 r--p 00017000 ca:01 133219 /lib/x86_64-linux-gnu/libpthread-2.13.so
7fd49344b000-7fd49344c000 rw-p 00018000 ca:01 133219 /lib/x86_64-linux-gnu/libpthread-2.13.so
7fd49344c000-7fd493450000 rw-p 00000000 00:00 0
7fd493450000-7fd493452000 r-xp 00000000 ca:01 133193 /lib/x86_64-linux-gnu/libdl-2.13.so
7fd493452000-7fd493652000 ---p 00002000 ca:01 133193 /lib/x86_64-linux-gnu/libdl-2.13.so
7fd493652000-7fd493653000 r--p 00002000 ca:01 133193 /lib/x86_64-linux-gnu/libdl-2.13.so
7fd493653000-7fd493654000 rw-p 00003000 ca:01 133193 /lib/x86_64-linux-gnu/libdl-2.13.so
7fd493654000-7fd49365d000 r-xp 00000000 ca:01 133190 /lib/x86_64-linux-gnu/libcrypt-2.13.so
7fd49365d000-7fd49385d000 ---p 00009000 ca:01 133190 /lib/x86_64-linux-gnu/libcrypt-2.13.so
7fd49385d000-7fd49385e000 r--p 00009000 ca:01 133190 /lib/x86_64-linux-gnu/libcrypt-2.13.so
7fd49385e000-7fd49385f000 rw-p 0000a000 ca:01 133190 /lib/x86_64-linux-gnu/libcrypt-2.13.so
7fd49385f000-7fd49388d000 rw-p 00000000 00:00 0
7fd49388d000-7fd493894000 r-xp 00000000 ca:01 133221 /lib/x86_64-linux-gnu/librt-2.13.so
7fd493894000-7fd493a93000 ---p 00007000 ca:01 133221 /lib/x86_64-linux-gnu/librt-2.13.so
7fd493a93000-7fd493a94000 r--p 00006000 ca:01 133221 /lib/x86_64-linux-gnu/librt-2.13.so
7fd493a94000-7fd493a95000 rw-p 00007000 ca:01 133221 /lib/x86_64-linux-gnu/librt-2.13.so
7fd493a95000-7fd493b18000 r-xp 00000000 ca:01 133203 /lib/x86_64-linux-gnu/libm-2.13.so
7fd493b18000-7fd493d17000 ---p 00083000 ca:01 133203 /lib/x86_64-linux-gnu/libm-2.13.so
7fd493d17000-7fd493d18000 r--p 00082000 ca:01 133203 /lib/x86_64-linux-gnu/libm-2.13.so
7fd493d18000-7fd493d19000 rw-p 00083000 ca:01 133203 /lib/x86_64-linux-gnu/libm-2.13.so
7fd493d19000-7fd493d1a000 r-xp 00000000 ca:01 131173 /lib/libaio.so.1.0.1
7fd493d1a000-7fd493f19000 ---p 00001000 ca:01 131173 /lib/libaio.so.1.0.1
7fd493f19000-7fd493f1a000 r--p 00000000 ca:01 131173 /lib/libaio.so.1.0.1
7fd493f1a000-7fd493f1b000 rw-p 00001000 ca:01 131173 /lib/libaio.so.1.0.1
7fd493f1b000-7fd493f3c000 r-xp 00000000 ca:01 133180 /lib/x86_64-linux-gnu/ld-2.13.so
7fd493f4b000-7fd493f4c000 ---p 00000000 00:00 0
7fd493f4c000-7fd494134000 rw-p 00000000 00:00 0
7fd494138000-7fd49413b000 rw-p 00000000 00:00 0
7fd49413b000-7fd49413c000 r--p 00020000 ca:01 133180 /lib/x86_64-linux-gnu/ld-2.13.so
7fd49413c000-7fd49413e000 rw-p 00021000 ca:01 133180 /lib/x86_64-linux-gnu/ld-2.13.so
7fffdb17b000-7fffdb19c000 rw-p 00000000 00:00 0 [stack]
7fffdb1ff000-7fffdb200000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
00:11:17 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=618
max_threads=10000
thread_count=35
connection_count=35
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 21919096 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fc4906c9410
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc512e8de50 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7cec1e]
/usr/sbin/mysqld(handle_fatal_signal+0x484)[0x6a64b4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10060)[0x7fd493243060]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fd492ec83a5]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7fd492ecbb0b]
/lib/x86_64-linux-gnu/libc.so.6(+0x6fd63)[0x7fd492f01d63]
/lib/x86_64-linux-gnu/libc.so.6(+0x7a6e6)[0x7fd492f0c6e6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fd492f109cc]
/usr/sbin/mysqld(free_root+0x90)[0x7c4600]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x30b)[0x5ab6ab]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x5adb69]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x157)[0x648787]
/usr/sbin/mysqld(handle_one_connection+0x51)[0x648931]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc)[0x7fd49323aefc]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd492f7559d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1820363
Status: NOT_KILLED
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
121125 00:11:22 mysqld_safe Number of processes running now: 0
121125 00:11:22 mysqld_safe mysqld restarted
121125 0:11:23 [Note] Flashcache bypass: disabled
121125 0:11:23 [Note] Flashcache setup error is : ioctl failed
121125 0:11:23 [Note] WSREP: Read nil XID from storage engines, skipping position init
121125 0:11:23 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
121125 0:11:23 [Note] WSREP: wsrep_load(): Galera 2.2(r115) by Codership Oy <email address hidden> loaded succesfully.
121125 0:11:23 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
121125 0:11:23 [Note] WSREP: Reusing existing '/db/data//galera.cache'.
121125 0:11:23 [Note] WSREP: Passing config to GCS: base_host = 107.21.214.231; evs.inactive_check_period = PT10S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M; evs.keepalive_period = PT3S; evs.suspect_timeout = PT30S; gcache.dir = /db/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /db/data//galera.cache; gcache.page_size = 128M; gcache.size = 4G; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; ist.recv_addr = 107.21.214.231; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
121125 0:11:23 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
21125 0:11:23 [Note] WSREP: wsrep_sst_grab()
121125 0:11:23 [Note] WSREP: Start replication
121125 0:11:23 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
121125 0:11:23 [Note] WSREP: protonet asio version 0
121125 0:11:23 [Note] WSREP: backend: asio
121125 0:11:23 [Note] WSREP: GMCast version 0
121125 0:11:23 [Note] WSREP: (a57b0e34-3694-11e2-0800-c4f8ff7afc9a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
121125 0:11:23 [Note] WSREP: (a57b0e34-3694-11e2-0800-c4f8ff7afc9a, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
121125 0:11:23 [Note] WSREP: EVS version 0
121125 0:11:23 [Note] WSREP: PC version 0
121125 0:11:23 [Note] WSREP: gcomm: connecting to group 'rscluster1', peer '54.245.82.2:4567'
121125 0:11:23 [Note] WSREP: (a57b0e34-3694-11e2-0800-c4f8ff7afc9a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.137.97.21:4567
121125 0:11:24 [Note] WSREP: (a57b0e34-3694-11e2-0800-c4f8ff7afc9a, 'tcp://0.0.0.0:4567') turning message relay requesting off
121125 0:11:53 [Note] WSREP: view((empty))
121125 0:11:53 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():157
121125 0:11:53 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():195: Failed to open backend connection: -110 (Connection timed out)
121125 0:11:53 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1290: Failed to open channel 'rscluster1' at 'gcomm://54.245.82.2:4567': -110 (Connection timed out)
121125 0:11:53 [ERROR] WSREP: gcs connect failed: Connection timed out
121125 0:11:53 [ERROR] WSREP: wsrep::connect() failed: 6
121125 0:11:53 [ERROR] Aborting
121125 0:11:53 [Note] WSREP: Service disconnected.
121125 0:11:54 [Note] WSREP: Some threads may fail to exit.
121125 0:11:54 [Note] /usr/sbin/mysqld: Shutdown complete
@Forrest,
Did any particular query/command cause this?
Do you have core-dumps enabled? If not, can you enable them?
I see that you are using 5.5.27 version of PXC. Can you see if
you can reproduce it with 5.5.28 PXC recently released?