Percona Server moved to https://jira.percona.com/projects/PS

InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page)

Series 5.7
Bug #1428331

Bug #1428331 reported by Michael Coburn on 2015-03-04

This bug affects 2 people

	Status	Importance	Assigned to
Percona Server moved to https://jira.percona.com/projects/PS	Status tracked in 5.7
5.5	New	Undecided	Unassigned
5.6	New	Undecided	Vlad Lesin
5.7	New	Undecided	Vlad Lesin

Bug Description

Percona Server 5.5.41 crashes with "InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page)". This also occurred on PS 5.5.32 so upgrade to latest 5.5 was attempted to find problem still persists.

* Server is replicating traffic only, no SELECT queries
* Possibly related:
http://bugs.mysql.com/bug.php?id=65196
http://bugs.mysql.com/bug.php?id=74596
https://bugs.launchpad.net/percona-server/+bug/995258

I have provided error log, and we had gdb attached so we have output of 'bt' and 'thread apply all bt'. Finally I have included the packages installed, and the my.cnf . Please let me know what further information I can provide.

150304 18:36:27 InnoDB: Assertion failure in thread 140331370538752 in file btr0cur.c line 329
InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
20:35:48 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=4
max_threads=16386
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 35924364 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fa1717de700 (LWP 11218)]
0x00007fd4efd5f425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007fd4efd5f425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fd4efd62b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fd4f2367d85 in btr_cur_latch_leaves (page=0x7fc8872c8000 "\356\032\205\275",
    space=837, zip_size=0, page_no=28006, latch_mode=<optimized out>, mtr=0x7fa1717dd6f0,
    cursor=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/btr/btr0cur.c:328
#3 0x00007fd4f2368e2b in btr_cur_search_to_nth_level (index=0x7fd4e8009578, level=0,
    tuple=0x7fd4e8009158, mode=4, latch_mode=33, cursor=0x7fd61ae245a8, has_search_latch=0,
    file=0x7fd4f2553718 "/mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c", line=96,
    mtr=0x7fa1717dd6f0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/btr/btr0cur.c:777
#4 0x00007fd4f237312c in btr_pcur_open_with_no_init_func (mtr=0x7fa1717dd6f0, line=96,
    file=0x7fd4f2553718 "/mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c", has_search_latch=0,
    cursor=0x7fd61ae245a8, latch_mode=33, mode=<optimized out>, tuple=0x7fd4e8009158,
    index=0x7fd4e8009578)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/include/btr0pcur.ic:482
#5 btr_pcur_restore_position_func (latch_mode=33, cursor=0x7fd61ae245a8,
    file=0x7fd4f2553718 "/mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c", line=96,
    mtr=0x7fa1717dd6f0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/btr/btr0pcur.c:342
#6 0x00007fd4f2420472 in row_purge_remove_clust_if_poss_low (node=0x7fd61ae24548, mode=33)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c:142
#7 0x00007fd4f2421a90 in row_purge_remove_clust_if_poss (node=0x7fd61ae24548)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c:211
#8 row_purge_del_mark (node=0x7fd61ae24548)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c:491
#9 row_purge (thr=0x7fd61ae24390, node=0x7fd61ae24548)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c:765
#10 row_purge_step (thr=0x7fd61ae24390)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c:805
#11 0x00007fd4f2417ae5 in que_thr_step (thr=0x7fd61ae24390)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/que/que0que.c:1259
#12 que_run_threads_low (thr=0x7fd61ae24390)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/que/que0que.c:1319
#13 que_run_threads (thr=0x7fd61ae24390)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/que/que0que.c:1356
#14 0x00007fd4f234352b in trx_purge (limit=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/trx/trx0purge.c:1220
#15 0x00007fd4f2338b0c in srv_purge_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0srv.c:4178
#16 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#17 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#18 0x0000000000000000 in ?? ()

(gdb) thread apply all bt

Thread 24 (Thread 0x7fd4ef8f8700 (LWP 11132)):
#0 0x00007fd4f19a0644 in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00007fd4f24033b7 in os_aio_linux_collect (seg_size=256, segment=<optimized out>,
    array=0x7fd4f42457c0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4744
#2 os_aio_linux_handle (global_seg=0, message1=0x7fd4ef8f7e50, message2=0x7fd4ef8f7e58,
    type=0x7fd4ef8f7e60, space_id=0x7fd4ef8f7e68)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4890
#3 0x00007fd4f23be9d8 in fil_aio_wait (segment=0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/fil/fil0fil.c:5682
#4 0x00007fd4f2338db8 in io_handler_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0start.c:489
#5 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 23 (Thread 0x7fa1787ec700 (LWP 11133)):
#0 0x00007fd4f19a0644 in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00007fd4f24033b7 in os_aio_linux_collect (seg_size=256, segment=<optimized out>,
    array=0x7fd4f4253a40)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4744
#2 os_aio_linux_handle (global_seg=1, message1=0x7fa1787ebe50, message2=0x7fa1787ebe58,
    type=0x7fa1787ebe60, space_id=0x7fa1787ebe68)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4890
#3 0x00007fd4f23be9d8 in fil_aio_wait (segment=1)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/fil/fil0fil.c:5682
#4 0x00007fd4f2338db8 in io_handler_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0start.c:489
#5 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()
hread 22 (Thread 0x7fa177feb700 (LWP 11134)):
#0 0x00007fd4f19a0644 in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00007fd4f24033b7 in os_aio_linux_collect (seg_size=256, segment=<optimized out>,
    array=0x7fd4f4261cc0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4744
#2 os_aio_linux_handle (global_seg=2, message1=0x7fa177feae50, message2=0x7fa177feae58,
    type=0x7fa177feae60, space_id=0x7fa177feae68)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4890
#3 0x00007fd4f23be9d8 in fil_aio_wait (segment=2)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/fil/fil0fil.c:5682
#4 0x00007fd4f2338db8 in io_handler_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0start.c:489
#5 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7fa1777ea700 (LWP 11135)):
#0 0x00007fd4f19a0644 in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00007fd4f24033b7 in os_aio_linux_collect (seg_size=256, segment=<optimized out>,
    array=0x7fd4f4261cc0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-ser---Type <return> to continue, or q <return> to quit---
ver-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4744
#2 os_aio_linux_handle (global_seg=3, message1=0x7fa1777e9e50, message2=0x7fa1777e9e58,
    type=0x7fa1777e9e60, space_id=0x7fa1777e9e68)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4890
#3 0x00007fd4f23be9d8 in fil_aio_wait (segment=3)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/fil/fil0fil.c:5682
#4 0x00007fd4f2338db8 in io_handler_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0start.c:489
#5 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 20 (Thread 0x7fa176fe9700 (LWP 11136)):
#0 0x00007fd4f19a0644 in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00007fd4f24033b7 in os_aio_linux_collect (seg_size=256, segment=<optimized out>,
    array=0x7fd4f4261cc0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4744
#2 os_aio_linux_handle (global_seg=4, message1=0x7fa176fe8e50, message2=0x7fa176fe8e58,
    type=0x7fa176fe8e60, space_id=0x7fa176fe8e68)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4890
#3 0x00007fd4f23be9d8 in fil_aio_wait (segment=4)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/fil/fil0fil.c:5682
#4 0x00007fd4f2338db8 in io_handler_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0start.c:489
#5 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 19 (Thread 0x7fa1767e8700 (LWP 11137)):
#0 0x00007fd4f19a0644 in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00007fd4f24033b7 in os_aio_linux_collect (seg_size=256, segment=<optimized out>,
    array=0x7fd4f4261cc0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4744
#2 os_aio_linux_handle (global_seg=5, message1=0x7fa1767e7e50, message2=0x7fa1767e7e58,
    type=0x7fa1767e7e60, space_id=0x7fa1767e7e68)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-ser---Type <return> to continue, or q <return> to quit---
ver-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4890
#3 0x00007fd4f23be9d8 in fil_aio_wait (segment=5)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/fil/fil0fil.c:5682
#4 0x00007fd4f2338db8 in io_handler_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0start.c:489
#5 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7fa175fe7700 (LWP 11138)):
#0 0x00007fd4f19a0644 in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00007fd4f24033b7 in os_aio_linux_collect (seg_size=256, segment=<optimized out>,
    array=0x7fd4f4269f40)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4744
#2 os_aio_linux_handle (global_seg=6, message1=0x7fa175fe6e50, message2=0x7fa175fe6e58,
    type=0x7fa175fe6e60, space_id=0x7fa175fe6e68)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4890
#3 0x00007fd4f23be9d8 in fil_aio_wait (segment=6)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/fil/fil0fil.c:5682
#4 0x00007fd4f2338db8 in io_handler_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0start.c:489
#5 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7fa1757e6700 (LWP 11139)):
#0 0x00007fd4f19a0644 in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00007fd4f24033b7 in os_aio_linux_collect (seg_size=256, segment=<optimized out>,
    array=0x7fd4f4269f40)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4744
#2 os_aio_linux_handle (global_seg=7, message1=0x7fa1757e5e50, message2=0x7fa1757e5e58,
    type=0x7fa1757e5e60, space_id=0x7fa1757e5e68)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4890
#3 0x00007fd4f23be9d8 in fil_aio_wait (segment=7)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/fil/fil0fil.c:5682
#4 0x00007fd4f2338db8 in io_handler_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0start.c:489
#5 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7fa174fe5700 (LWP 11140)):
#0 0x00007fd4f19a0644 in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00007fd4f24033b7 in os_aio_linux_collect (seg_size=256, segment=<optimized out>,
    array=0x7fd4f4269f40)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4744
#2 os_aio_linux_handle (global_seg=8, message1=0x7fa174fe4e50, message2=0x7fa174fe4e58,
    type=0x7fa174fe4e60, space_id=0x7fa174fe4e68)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4890
#3 0x00007fd4f23be9d8 in fil_aio_wait (segment=8)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/fil/fil0fil.c:5682
#4 0x00007fd4f2338db8 in io_handler_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-ser
ver-5.5-5.5.41-37.0/storage/innobase/srv/srv0start.c:489
#5 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7fa1747e4700 (LWP 11141)):
#0 0x00007fd4f19a0644 in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00007fd4f24033b7 in os_aio_linux_collect (seg_size=256, segment=<optimized out>,
    array=0x7fd4f4269f40)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4744
#2 os_aio_linux_handle (global_seg=9, message1=0x7fa1747e3e50, message2=0x7fa1747e3e58,
    type=0x7fa1747e3e60, space_id=0x7fa1747e3e68)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0file.c:4890
#3 0x00007fd4f23be9d8 in fil_aio_wait (segment=9)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/fil/fil0fil.c:5682
#4 0x00007fd4f2338db8 in io_handler_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0start.c:489
#5 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7fa173fe3700 (LWP 11213)):
#0 0x00007fd4f13700fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fd4f240565a in os_cond_wait_timed (abstime=0x7fa173fe2dd0, mutex=0x7fd4f3c0ef20,
    cond=0x7fd4f3c0ef58)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0sync.c:171
#2 os_event_wait_time_low (event=0x7fd4f3c0ef20, time_in_usec=<optimized out>,
    reset_sig_count=1)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0sync.c:716
#3 0x00007fd4f2335a70 in srv_lock_timeout_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0srv.c:2802
#4 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7fa1737e2700 (LWP 11214)):
#0 0x00007fd4f13700fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fd4f240565a in os_cond_wait_timed (abstime=0x7fa1737e1db0, mutex=0x7fd4f3c0eea0,
    cond=0x7fd4f3c0eed8)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0sync.c:171
#2 os_event_wait_time_low (event=0x7fd4f3c0eea0, time_in_usec=<optimized out>,
    reset_sig_count=1)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0sync.c:716
#3 0x00007fd4f2336006 in srv_error_monitor_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0srv.c:3015
#4 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7fa172fe1700 (LWP 11215)):
#0 0x00007fd4f13700fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fd4f240565a in os_cond_wait_timed (abstime=0x7fa172fe0e20, mutex=0x7fd4f3c0efa0,
    cond=0x7fd4f3c0efd8)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0sync.c:171
#2 os_event_wait_time_low (event=0x7fd4f3c0efa0, time_in_usec=<optimized out>,
    reset_sig_count=1)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0sync.c:716
#3 0x00007fd4f233576b in srv_monitor_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0srv.c:2652
#4 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fa1727e0700 (LWP 11216)):
#0 0x00007fd4efe16763 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fd4f2405eab in os_thread_sleep (tm=<optimized out>)
at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0thread.c:259
#2 0x00007fd4f233636a in srv_LRU_dump_restore_thread (arg=<optimized out>)
at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0srv.c:3061
#3 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fa171fdf700 (LWP 11217)):
#0 0x00007fd4efe16763 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fd4f2405eab in os_thread_sleep (tm=<optimized out>)
at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/os/os0thread.c:259
#2 0x00007fd4f2337175 in srv_master_thread (arg=<optimized out>)
at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0srv.c:3500
#3 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fa1717de700 (LWP 11218)):
#0 0x00007fd4efd5f425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fd4efd62b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fd4f2367d85 in btr_cur_latch_leaves (page=0x7fc8872c8000 "\356\032\205\275",
    space=837, zip_size=0, page_no=28006, latch_mode=<optimized out>, mtr=0x7fa1717dd6f0,
    cursor=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-ser---Type <return> to continue, or q <return> to quit---
ver-5.5-5.5.41-37.0/storage/innobase/btr/btr0cur.c:328
#3 0x00007fd4f2368e2b in btr_cur_search_to_nth_level (index=0x7fd4e8009578, level=0,
    tuple=0x7fd4e8009158, mode=4, latch_mode=33, cursor=0x7fd61ae245a8, has_search_latch=0,
    file=0x7fd4f2553718 "/mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c", line=96,
    mtr=0x7fa1717dd6f0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/btr/btr0cur.c:777
#4 0x00007fd4f237312c in btr_pcur_open_with_no_init_func (mtr=0x7fa1717dd6f0, line=96,
    file=0x7fd4f2553718 "/mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c", has_search_latch=0,
    cursor=0x7fd61ae245a8, latch_mode=33, mode=<optimized out>, tuple=0x7fd4e8009158,
    index=0x7fd4e8009578)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/include/btr0pcur.ic:482
#5 btr_pcur_restore_position_func (latch_mode=33, cursor=0x7fd61ae245a8,
    file=0x7fd4f2553718 "/mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c", line=96,
    mtr=0x7fa1717dd6f0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/btr/btr0pcur.c:342
#6 0x00007fd4f2420472 in row_purge_remove_clust_if_poss_low (node=0x7fd61ae24548, mode=33)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c:142
#7 0x00007fd4f2421a90 in row_purge_remove_clust_if_poss (node=0x7fd61ae24548)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c:211
#8 row_purge_del_mark (node=0x7fd61ae24548)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c:491
#9 row_purge (thr=0x7fd61ae24390, node=0x7fd61ae24548)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c:765
#10 row_purge_step (thr=0x7fd61ae24390)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/row/row0purge.c:805
#11 0x00007fd4f2417ae5 in que_thr_step (thr=0x7fd61ae24390)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/que/que0que.c:1259
#12 que_run_threads_low (thr=0x7fd61ae24390)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/que/que0que.c:1319
#13 que_run_threads (thr=0x7fd61ae24390)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-ser---Type <return> to continue, or q <return> to quit---
ver-5.5-5.5.41-37.0/storage/innobase/que/que0que.c:1356
#14 0x00007fd4f234352b in trx_purge (limit=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/trx/trx0purge.c:1220
#15 0x00007fd4f2338b0c in srv_purge_thread (arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/storage/innobase/srv/srv0srv.c:4178
#16 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#17 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#18 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fa170fdd700 (LWP 11219)):
#0 0x00007fd4f13739f7 in do_sigwait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fd4f1373a79 in sigwait () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fd4f206441b in signal_hand (arg=<optimized out>)
at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:2750
#3 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fa170fac700 (LWP 11220)):
#0 0x00007fd4f136fd84 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fd4f2068c9b in inline_mysql_cond_wait (src_line=2270, mutex=0x7fd4f2b17f60,
    that=0x7fd4f2b1a3e0, src_file=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/include/mysql/psi/mysql_thread.h:980
#2 cache_thread ()
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:2270
#3 one_thread_per_connection_end (thd=<optimized out>, put_in_cache=true)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:2336
#4 0x00007fd4f21742a7 in do_handle_one_connection (thd_arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/sql_connect.cc:1423
#5 0x00007fd4f2174410 in handle_one_connection (arg=0x7fd61af403f0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/sql_connect.cc:1324
#6 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fd4edac1700 (LWP 11224)):
#0 0x00007fd4f136fd84 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fd4f2068c9b in inline_mysql_cond_wait (src_line=2270, mutex=0x7fd4f2b17f60,
    that=0x7fd4f2b1a3e0, src_file=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/include/mysql/psi/mysql_thread.h:980
#2 cache_thread ()
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:2270
#3 one_thread_per_connection_end (thd=<optimized out>, put_in_cache=true)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:2336
#4 0x00007fd4f21742a7 in do_handle_one_connection (thd_arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/sql_connect.cc:1423
#5 0x00007fd4f2174410 in handle_one_connection (arg=0x7fd61afcf6d0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/sql_connect.cc:1324
#6 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fa163fff700 (LWP 11269)):
#0 0x00007fd4f136fd84 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fd4f2068c9b in inline_mysql_cond_wait (src_line=2270, mutex=0x7fd4f2b17f60,
    that=0x7fd4f2b1a3e0, src_file=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/include/mysql/psi/mysql_thread.h:980
#2 cache_thread ()
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:2270
#3 one_thread_per_connection_end (thd=<optimized out>, put_in_cache=true)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:2336
#4 0x00007fd4f21742a7 in do_handle_one_connection (thd_arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/sql_connect.cc:1423
#5 0x00007fd4f2174410 in handle_one_connection (arg=0x7fd61afd31b0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/sql_connect.cc:1324
#6 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fd4eda90700 (LWP 11401)):
#0 0x00007fd4f136fd84 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fd4f2068c9b in inline_mysql_cond_wait (src_line=2270, mutex=0x7fd4f2b17f60,
    that=0x7fd4f2b1a3e0, src_file=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/include/mysql/psi/mysql_thread.h:980
#2 cache_thread ()
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:2270
#3 one_thread_per_connection_end (thd=<optimized out>, put_in_cache=true)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:2336
#4 0x00007fd4f21742a7 in do_handle_one_connection (thd_arg=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/sql_connect.cc:1423
#5 0x00007fd4f2174410 in handle_one_connection (arg=0x7fd61afdefe0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/sql_connect.cc:1324
#6 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fd4eda5f700 (LWP 12204)):
#0 0x00007fd4f1372d2d in read () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fd4f24a98d2 in vio_read_buff (vio=0x7fa154008900, buf=0x7fa15400cb10 "", size=4)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/vio/viosocket.c:95
#2 0x00007fd4f206c78b in my_real_read (net=0x7fa154008310, complen=0x7fd4eda5ebe8)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/net_serv.cc:840
#3 0x00007fd4f206d26a in my_net_read (net=0x7fa154008310)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/net_serv.cc:1039
#4 0x00007fd4f21ab165 in cli_safe_read (mysql=0x7fa154008310)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql-common/client.c:735
#5 0x00007fd4f207f859 in read_event (suppress_warnings=0x7fd4eda5ee2f, mysql=0x7fa154008310,
    mi=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/slave.cc:2287
#6 handle_slave_io (arg=0x7fd61af8f5a0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/slave.cc:3010
#7 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7fd4eda2e700 (LWP 12205)):
#0 0x00007fd4f136fd84 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fd4f225535b in inline_mysql_cond_wait (src_line=6181, mutex=0x7fd61af90d90,
    that=0x7fd61af91280, src_file=<optimized out>)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/include/mysql/psi/mysql_thread.h:980
#2 MYSQL_BIN_LOG::wait_for_update_relay_log (this=0x7fd61af90d88, thd=0x7fa148000990)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/log.cc:6181
#3 0x00007fd4f207e036 in next_event (rli=0x7fd61af90930)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/slave.cc:4830
#4 exec_relay_log_event (rli=0x7fd61af90930, thd=0x7fa148000990)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/slave.cc:2559
#5 handle_slave_sql (arg=0x7fd61af8f5a0)
    at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/slave.cc:3444
#6 0x00007fd4f136be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007fd4efe1d3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fd4f1db3740 (LWP 11002)):
#0 0x00007fd4efe11a43 in poll () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fd4f2069ca1 in handle_connections_sockets ()
at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:5454
#2 0x00007fd4f206b656 in mysqld_main (argc=78, argv=0x7fd4f2f925c8)
at /mnt/workspace/percona-server-5.5-debian-binary/label_exp/ubuntu-precise-64bit/percona-server-5.5-5.5.41-37.0/sql/mysqld.cc:4891
#3 0x00007fd4efd4a76d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fd4f2061459 in _start ()

$ dpkg -l | grep -i percona
ii libmysqlclient18 1:5.5.32-rel31.0-549.precise Percona Server database client library
ii libperconaserverclient18 1:5.5.41-rel37.0-727.precise Percona Server database client library
ii percona-server-5.5-dbg 1:5.5.41-rel37.0-727.precise Debugging package for Percona Server
ii percona-server-client-5.5 1:5.5.41-rel37.0-727.precise Percona Server database client binaries
ii percona-server-common-5.5 1:5.5.41-rel37.0-727.precise Percona Server database common files (e.g. /etc/mysql/my.cnf)
ii percona-server-server-5.5 1:5.5.41-rel37.0-727.precise Percona Server database server binaries
ii percona-toolkit 2.2.4 Advanced MySQL and system command-line tools
ii percona-xtrabackup 2.2.6-5042-1.precise Open source backup tool for InnoDB and XtraDB

$ egrep -v "^$|^#" /etc/mysql/my.cnf
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
err-log = /var/log/mysql/error.log
[mysqld]
core-file
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
log_error = /var/log/mysql/error.log
back_log = 1000
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
log_slow_admin_statements = 0
log_slow_slave_statements = 1
log_slow_verbosity = full
log_queries_not_using_indexes = 0
long_query_time = 1
log_output = TABLE
port = 3306
basedir = /usr
datadir = /data/mysql
tmpdir = /data/mysql_tmp
language = /usr/share/mysql/english
skip-external-locking
skip_name_resolve
server-id = 2886938917
auto_increment_increment = 2
auto_increment_offset = 1
read_only = ON
event_scheduler = OFF
innodb_rollback_on_timeout
innodb_recovery_update_relay_log
innodb_io_capacity = 10000
innodb_write_io_threads = 4
innodb_read_io_threads = 4
innodb_adaptive_flushing_method = keep_average
innodb_flush_neighbor_pages = none
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 50
table_open_cache = 2048
innodb_fast_shutdown = 0
key_buffer = 64M
myisam_sort_buffer_size = 64M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 2048
max_connections = 16384
max_user_connections = 16384
max_connect_errors = 1000000
wait_timeout = 45
interactive_timeout = 28800
thread_concurrency = 10
connect_timeout = 10
transaction-isolation = READ-COMMITTED
myisam-recover = BACKUP
query_cache_limit = 0
query_cache_size = 1M
query_cache_type = 0
binlog-format = mixed
log_bin = /data/mysql_binlog/bin.log
sync_binlog = 1
expire_logs_days = 2
log-slave-updates
relay-log = local-relay-bin
relay-log-index = local-relay-bin.index
innodb_data_home_dir = /data/mysql
innodb_additional_mem_pool_size = 20M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_file_per_table
innodb_file_format = antelope
innodb_buffer_pool_size = 201178MB
innodb_log_file_size = 1000M
innodb_log_files_in_group = 4
innodb_log_buffer_size = 8M
innodb_open_files = 1024
open_files_limit = 32768
innodb_status_file
innodb_thread_concurrency = 30
skip-federated
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 16M
!includedir /etc/mysql/conf.d/
## ^^ There's nothing in the directory /etc/mysql/conf.d

Tags:

Revision history for this message

Michael Coburn (michaelcoburn) wrote on 2015-03-04:

Percona eventum issue https://customers.percona.com/view.php?id=51533

Valerii Kravchuk (valerii-kravchuk) on 2015-03-05

tags:

added: i51533

Revision history for this message

Valerii Kravchuk (valerii-kravchuk) wrote on 2015-03-05:

Please, upload the entire error log of this MySQL server. I wonder if you had identified the problematic table and if you checked it for corruption using innochecksum.

Changed in percona-server:
status:	New → Incomplete

Revision history for this message

Michael Coburn (michaelcoburn) wrote on 2015-03-05:

Download full text (29.4 KiB)

Complete error log from the instance, based on all files I could find still on system. Note I deleted details from the actual page dump as that leaks information about contents of the page.

$ cat error.log.3 error.log.2 error.log.1 error.log
150302 20:06:14 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150302 20:06:14 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized.
150302 20:06:14 InnoDB: The InnoDB memory heap is disabled
150302 20:06:14 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150302 20:06:14 InnoDB: Compressed tables use zlib 1.2.3.4
150302 20:06:14 InnoDB: Using Linux native AIO
150302 20:06:15 InnoDB: Initializing buffer pool, size = 196.5G
150302 20:06:27 InnoDB: Completed initialization of buffer pool
150302 20:06:27 InnoDB: highest supported file format is Barracuda.
150302 20:06:36 InnoDB: Waiting for the background threads to start
150302 20:06:37 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 1597945
InnoDB: something wrong with relay-log.info. InnoDB will not overwrite it.
150302 20:06:38 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode.
150302 20:06:38 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode.
150302 20:06:38 [Note] Event Scheduler: Loaded 0 events
150302 20:06:38 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.41-37.0-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 37.0, Revision 727
150302 20:07:06 [Note] /usr/sbin/mysqld: Normal shutdown

150302 20:07:06 [Note] Event Scheduler: Purging the queue. 0 events
150302 20:07:06 InnoDB: Starting shutdown...
150302 20:07:14 InnoDB: Shutdown completed; log sequence number 1597945
150302 20:07:14 [Note] /usr/sbin/mysqld: Shutdown complete

Complete error log from the instance, based on all files I could find still on system. Note I deleted details from the actual page dump as that leaks information about contents of the page.

$ cat error.log.3 error.log.2 error.log.1 error.log
150302 20:06:14 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150302 20:06:14 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized.
150302 20:06:14 InnoDB: The InnoDB memory heap is disabled
150302 20:06:14 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150302 20:06:14 InnoDB: Compressed tables use zlib 1.2.3.4
150302 20:06:14 InnoDB: Using Linux native AIO
150302 20:06:15 InnoDB: Initializing buffer pool, size = 196.5G
150302 20:06:27 InnoDB: Completed initialization of buffer pool
150302 20:06:27 InnoDB: highest supported file format is Barracuda.
150302 20:06:36  InnoDB: Waiting for the background threads to start
150302 20:06:37 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 1597945
InnoDB: something wrong with relay-log.info. InnoDB will not overwrite it.
150302 20:06:38 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode.
150302 20:06:38 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode.
150302 20:06:38 [Note] Event Scheduler: Loaded 0 events
150302 20:06:38 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.41-37.0-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona Server (GPL), Release 37.0, Revision 727
150302 20:07:06 [Note] /usr/sbin/mysqld: Normal shutdown

150302 20:07:06 [Note] Event Scheduler: Purging the queue. 0 events
150302 20:07:06  InnoDB: Starting shutdown...
150302 20:07:14  InnoDB: Shutdown completed; log sequence number 1597945
150302 20:07:14 [Note] /usr/sbin/mysqld: Shutdown complete

150302 20:27:55 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150302 20:27:55 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized.
150302 20:27:55 InnoDB: The InnoDB memory heap is disabled
150302 20:27:55 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150302 20:27:55 InnoDB: Compressed tables use zlib 1.2.3.4
150302 20:27:55 InnoDB: Using Linux native AIO
150302 20:27:55 InnoDB: Initializing buffer pool, size = 196.5G
150302 20:28:07 InnoDB: Completed initialization of buffer pool
150302 20:28:07 InnoDB: highest supported file format is Barracuda.
150302 20:28:17  InnoDB: Waiting for the background threads to start
150302 20:28:18 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 10279806215055
InnoDB: something wrong with relay-log.info. InnoDB will not overwrite it.
150302 20:28:18 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode.
150302 20:28:18 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode.
150302 20:28:18 [Note] Event Scheduler: Loaded 0 events
150302 20:28:18 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.41-37.0-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona Server (GPL), Release 37.0, Revision 727
150302 20:39:35 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='172.19.51.61', master_port='3306', master_log_file='bin.000131', master_log_pos='934527426'.
150302 20:39:37 [Note] Slave SQL thread initialized, starting replication in log 'bin.000131' at position 934527426, relay log './local-relay-bin.000001' position: 4
150302 20:39:37 [Note] Slave I/O thread: connected to master 'repl@172.19.51.61:3306',replication started in log 'bin.000131' at position 934527426
150304 18:36:27  InnoDB: Assertion failure in thread 140331370538752 in file btr0cur.c line 329
InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
20:35:48 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=4
max_threads=16386
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 35924364 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7fd4f22e2fa9]
/usr/sbin/mysqld(handle_fatal_signal+0x3f0)[0x7fd4f21c8950]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fd4f1373cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fd4efd5f425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7fd4efd62b8b]
/usr/sbin/mysqld(+0x5a0d85)[0x7fd4f2367d85]
/usr/sbin/mysqld(+0x5a1e2b)[0x7fd4f2368e2b]
/usr/sbin/mysqld(+0x5ac12c)[0x7fd4f237312c]
/usr/sbin/mysqld(+0x659472)[0x7fd4f2420472]
/usr/sbin/mysqld(+0x65aa90)[0x7fd4f2421a90]
/usr/sbin/mysqld(+0x650ae5)[0x7fd4f2417ae5]
/usr/sbin/mysqld(+0x57c52b)[0x7fd4f234352b]
/usr/sbin/mysqld(+0x571b0c)[0x7fd4f2338b0c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fd4f136be9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd4efe1d3fd]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file
InnoDB: Warning: a long semaphore wait:
150304 21:05:53 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150304 21:05:53 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized.
InnoDB: relay-log.info is detected.
InnoDB: relay log: position 585677554, file name ./local-relay-bin.000024
InnoDB: master log: position 585677414, file name bin.000139
150304 21:05:53 InnoDB: The InnoDB memory heap is disabled
150304 21:05:53 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150304 21:05:53 InnoDB: Compressed tables use zlib 1.2.3.4
150304 21:05:53 InnoDB: Using Linux native AIO
150304 21:05:54 InnoDB: Initializing buffer pool, size = 196.5G
150304 21:06:06 InnoDB: Completed initialization of buffer pool
150304 21:06:06 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 10295229374596
150304 21:06:08  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 10295229695015
150304 21:06:10  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 585677414, file name bin.000139
InnoDB: and relay log file
InnoDB: position 585677554, file name ./local-relay-bin.000024
InnoDB: Last MySQL binlog file position 0 494719823, file name /data/mysql_binlog/bin.000009
150304 21:06:20  InnoDB: Waiting for the background threads to start
150304 21:06:21 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 10295229695015
InnoDB: InnoDB and relay-log.info are synchronized. InnoDB will not overwrite it.
150304 21:06:21 [Note] Recovering after a crash using /data/mysql_binlog/bin
150304 21:06:24 [Note] Starting crash recovery...
150304 21:06:24 [Note] Crash recovery finished.
150304 21:06:24 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode.
150304 21:06:24 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode.
150304 21:06:24 [Note] Slave SQL thread initialized, starting replication in log 'bin.000139' at position 585677414, relay log './local-relay-bin.000024' position: 585677554
150304 21:06:24 [Note] Event Scheduler: Loaded 0 events
150304 21:06:24 [Note] Slave I/O thread: connected to master 'repl@172.19.51.61:3306',replication started in log 'bin.000139' at position 585700509
150304 21:06:24 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.41-37.0-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona Server (GPL), Release 37.0, Revision 727
150304 21:13:19  InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 838:0, should be 838:111751!
InnoDB: N heap is wrong 109, 96
150305  4:16:41  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex

... data deleted here ...

InnoDB: End of page dump
150305  4:16:41  InnoDB: Page checksum 2449463521 (32bit_calc: 3477942518), prior-to-4.0.14-form checksum 3461508486
InnoDB: stored checksum 2449463521, prior-to-4.0.14-form stored checksum 3461508486
InnoDB: Page lsn 2397 3992988175, low 4 bytes of lsn at page end 3992988175
InnoDB: Page number (if stored to page already) 347275,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 838
InnoDB: Page may be an index page where index id is 2016
InnoDB: (index "GEN_CLUST_INDEX" of table "ejabberd_ody_mobboss_online"."rosterusers")
150305  4:16:41  InnoDB: Apparent corruption of an index page n:o 347275 in space 838
InnoDB: to be written to data file. We intentionally crash server
InnoDB: to prevent corrupt data from ending up in data
InnoDB: files.
150305  4:16:41  InnoDB: Assertion failure in thread 139545601242880 in file buf0flu.c line 806
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
04:16:41 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=3
max_threads=16386
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 35924364 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f1dfce06fa9]
/usr/sbin/mysqld(handle_fatal_signal+0x3f0)[0x7f1dfccec950]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f1dfbe97cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f1dfa883425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f1dfa886b8b]
/usr/sbin/mysqld(+0x5bfe48)[0x7f1dfceaae48]
/usr/sbin/mysqld(+0x5c2c15)[0x7f1dfceadc15]
/usr/sbin/mysqld(+0x5c311c)[0x7f1dfceae11c]
/usr/sbin/mysqld(+0x5c35c2)[0x7f1dfceae5c2]
/usr/sbin/mysqld(+0x571368)[0x7f1dfce5c368]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f1dfbe8fe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f1dfa9413fd]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file
150305  4:17:35 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150305  4:17:35 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized.
InnoDB: relay-log.info is detected.
InnoDB: relay log: position 370401210, file name ./local-relay-bin.000031
InnoDB: master log: position 370401070, file name bin.000141
150305  4:17:35 InnoDB: The InnoDB memory heap is disabled
150305  4:17:35 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150305  4:17:35 InnoDB: Compressed tables use zlib 1.2.3.4
150305  4:17:35 InnoDB: Using Linux native AIO
150305  4:17:35 InnoDB: Initializing buffer pool, size = 196.5G
150305  4:17:47 InnoDB: Completed initialization of buffer pool
150305  4:17:47 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 10299028323666
150305  4:17:49  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 10299029807255
150305  4:17:53  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 370401070, file name bin.000141
InnoDB: and relay log file
InnoDB: position 370401210, file name ./local-relay-bin.000031
InnoDB: Last MySQL binlog file position 0 754983375, file name /data/mysql_binlog/bin.000011
150305  4:18:18  InnoDB: Waiting for the background threads to start
150305  4:18:19 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 10299029807255
InnoDB: InnoDB and relay-log.info are synchronized. InnoDB will not overwrite it.
150305  4:18:19 [Note] Recovering after a crash using /data/mysql_binlog/bin
150305  4:18:22 [Note] Starting crash recovery...
150305  4:18:22 [Note] Crash recovery finished.
150305  4:19:02 [Warning] 'user' entry 'root@XXX ignored in --skip-name-resolve mode.
150305  4:19:02 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode.
150305  4:19:12 [Note] Slave SQL thread initialized, starting replication in log 'bin.000141' at position 370401070, relay log './local-relay-bin.000031' position: 370401210
150305  4:19:12 [Note] Event Scheduler: Loaded 0 events
150305  4:19:12 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.41-37.0-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona Server (GPL), Release 37.0, Revision 727
150305  4:19:12 [Note] Slave I/O thread: connected to master 'repl@172.19.51.61:3306',replication started in log 'bin.000141' at position 370408948
150305  4:24:51  InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 838:0, should be 838:111751!
InnoDB: Next record offset nonsensical 0 for rec 4108
150305  7:32:52  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex

... data deleted here ...

InnoDB: End of page dump
150305  7:32:52  InnoDB: Page checksum 578674375 (32bit_calc: 1594542261), prior-to-4.0.14-form checksum 1637013161
InnoDB: stored checksum 578674375, prior-to-4.0.14-form stored checksum 1637013161
InnoDB: Page lsn 2398 715216317, low 4 bytes of lsn at page end 715216317
InnoDB: Page number (if stored to page already) 166019,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 837
InnoDB: Page may be an index page where index id is 2015
InnoDB: (index "pk_rosterg_user_jid" of table "ejabberd_ody_mobboss_online"."rostergroups")
150305  7:32:52  InnoDB: Apparent corruption of an index page n:o 166019 in space 837
InnoDB: to be written to data file. We intentionally crash server
InnoDB: to prevent corrupt data from ending up in data
InnoDB: files.
150305  7:32:52  InnoDB: Assertion failure in thread 139654153164544 in file buf0flu.c line 806
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
07:32:52 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=4
max_threads=16386
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 35924364 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f3742cc8fa9]
/usr/sbin/mysqld(handle_fatal_signal+0x3f0)[0x7f3742bae950]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f3741d59cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f3740745425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f3740748b8b]
/usr/sbin/mysqld(+0x5bfe48)[0x7f3742d6ce48]
/usr/sbin/mysqld(+0x5c2c15)[0x7f3742d6fc15]
/usr/sbin/mysqld(+0x5c311c)[0x7f3742d7011c]
/usr/sbin/mysqld(+0x5c35c2)[0x7f3742d705c2]
/usr/sbin/mysqld(+0x571368)[0x7f3742d1e368]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f3741d51e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f37408033fd]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file
150305  7:33:33 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150305  7:33:33 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized.
InnoDB: relay-log.info is detected.
InnoDB: relay log: position 147640616, file name ./local-relay-bin.000036
InnoDB: master log: position 147640513, file name bin.000142
150305  7:33:33 InnoDB: The InnoDB memory heap is disabled
150305  7:33:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150305  7:33:33 InnoDB: Compressed tables use zlib 1.2.3.4
150305  7:33:33 InnoDB: Using Linux native AIO
150305  7:33:33 InnoDB: Initializing buffer pool, size = 196.5G
150305  7:33:45 InnoDB: Completed initialization of buffer pool
150305  7:33:46 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 10300046723171
150305  7:33:48  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 10300047265987
150305  7:33:59  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 147640513, file name bin.000142
InnoDB: and relay log file
InnoDB: position 147640616, file name ./local-relay-bin.000036
InnoDB: Last MySQL binlog file position 0 139653441, file name /data/mysql_binlog/bin.000013
150305  7:34:24  InnoDB: Waiting for the background threads to start
150305  7:34:25 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 10300047265987
InnoDB: InnoDB and relay-log.info are synchronized. InnoDB will not overwrite it.
150305  7:34:25 [Note] Recovering after a crash using /data/mysql_binlog/bin
150305  7:34:26 [Note] Starting crash recovery...
150305  7:34:26 [Note] Crash recovery finished.
150305  7:34:40 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode.
150305  7:34:40 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode.
150305  7:34:44 [Note] Slave SQL thread initialized, starting replication in log 'bin.000142' at position 147640513, relay log './local-relay-bin.000036' position: 147640616
150305  7:34:44 [Note] Slave I/O thread: connected to master 'repl@172.19.51.61:3306',replication started in log 'bin.000142' at position 147640513
150305  7:34:44 [Note] Event Scheduler: Loaded 0 events
150305  7:34:44 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.41-37.0-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona Server (GPL), Release 37.0, Revision 727
InnoDB: Record 13110 is above rec heap top 12973
150305 10:36:48  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex

... data truncated here ...

InnoDB: End of page dump
150305 10:36:48  InnoDB: Page checksum 2692330666 (32bit_calc: 2825342776), prior-to-4.0.14-form checksum 1830467664
InnoDB: stored checksum 2692330666, prior-to-4.0.14-form stored checksum 1830467664
InnoDB: Page lsn 2398 1288429380, low 4 bytes of lsn at page end 1288429380
InnoDB: Page number (if stored to page already) 12766,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 821
InnoDB: Page may be an index page where index id is 1983
InnoDB: (index "uniq" of table "ejabberd_ody_mobboss_online"."muc_affiliations")
150305 10:36:48  InnoDB: Apparent corruption of an index page n:o 12766 in space 821
InnoDB: to be written to data file. We intentionally crash server
InnoDB: to prevent corrupt data from ending up in data
InnoDB: files.
150305 10:36:48  InnoDB: Assertion failure in thread 140258901358336 in file buf0flu.c line 806
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:36:48 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=4
max_threads=16386
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 35924364 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7fc410bd9fa9]
/usr/sbin/mysqld(handle_fatal_signal+0x3f0)[0x7fc410abf950]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fc40fc6acb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fc40e656425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7fc40e659b8b]
/usr/sbin/mysqld(+0x5bfe48)[0x7fc410c7de48]
/usr/sbin/mysqld(+0x5c2c15)[0x7fc410c80c15]
/usr/sbin/mysqld(+0x5c311c)[0x7fc410c8111c]
/usr/sbin/mysqld(+0x5c35c2)[0x7fc410c815c2]
/usr/sbin/mysqld(+0x571368)[0x7fc410c2f368]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fc40fc62e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc40e7143fd]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file
150305 10:37:23 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150305 10:37:23 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized.
InnoDB: relay-log.info is detected.
InnoDB: relay log: position 285907453, file name ./local-relay-bin.000038
InnoDB: master log: position 433547719, file name bin.000142
150305 10:37:23 InnoDB: The InnoDB memory heap is disabled
150305 10:37:23 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150305 10:37:23 InnoDB: Compressed tables use zlib 1.2.3.4
150305 10:37:23 InnoDB: Using Linux native AIO
150305 10:37:23 InnoDB: Initializing buffer pool, size = 196.5G
150305 10:37:35 InnoDB: Completed initialization of buffer pool
150305 10:37:36 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 10300619805596
150305 10:37:38  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 10300620307881
150305 10:37:48  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 433547719, file name bin.000142
InnoDB: and relay log file
InnoDB: position 285907453, file name ./local-relay-bin.000038
InnoDB: Last MySQL binlog file position 0 270154769, file name /data/mysql_binlog/bin.000014
150305 10:38:08  InnoDB: Waiting for the background threads to start
150305 10:38:09 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 10300620307881
InnoDB: InnoDB and relay-log.info are synchronized. InnoDB will not overwrite it.
150305 10:38:09 [Note] Recovering after a crash using /data/mysql_binlog/bin
150305 10:38:10 [Note] Starting crash recovery...
150305 10:38:10 [Note] Crash recovery finished.
150305 10:38:19 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode.
150305 10:38:19 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode.
150305 10:38:19 [Note] Slave SQL thread initialized, starting replication in log 'bin.000142' at position 433547719, relay log './local-relay-bin.000038' position: 285907453
150305 10:38:19 [Note] Event Scheduler: Loaded 0 events
150305 10:38:19 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.41-37.0-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona Server (GPL), Release 37.0, Revision 727
150305 10:38:19 [Note] Slave I/O thread: connected to master 'repl@172.19.51.61:3306',replication started in log 'bin.000142' at position 433547719

Revision history for this message

Michael Coburn (michaelcoburn) wrote on 2015-03-05:

The table name appears to change, it isn't consistently the same table (rosterusers, rostergroups, muc_affiliations). Further this database was stood up by using a mysqldump from the master, so I would assume we could not have imported in any corrupted pages through this manner of loading a slave? There are no events in Linux logging (/var/log/messages, dmesg, etc) that indicate hardware. Further this crash happens on multiple servers in this customer's cluster for this dataset, on PS 5.5.32 and now PS 5.5.41.

Note that this cluster at customer site has 4 servers, two of which have never crashed and their only difference is that they have a BP < 100GB. The instances that are crashing are ~200GB . I suggested workaround of smaller BP but that isn't an acceptable long term solution for them.

Customer is going to try a 5.6 installation on a slave to see if it too continues to crash when applying replication events.

Here is innochecksum over the three tables.

$ innochecksum -v rostergroups.ibd
file rostergroups.ibd = 3649044480 bytes (222720 pages)...
checking pages in range 0 to 222719
page 11775 okay: 5.287% done
page 25279 okay: 11.351% done
page 37887 okay: 17.011% done
page 51711 okay: 23.218% done
page 64895 okay: 29.138% done
page 79423 okay: 35.661% done
page 93951 okay: 42.184% done
page 109119 okay: 48.994% done
page 123583 okay: 55.489% done
page 139135 okay: 62.471% done
page 155135 okay: 69.655% done
page 171967 okay: 77.213% done
page 190015 okay: 85.316% done
page 206975 okay: 92.931% done
page 221887 okay: 99.626% done

$ innochecksum -v rosterusers.ibd
file rosterusers.ibd = 6803161088 bytes (415232 pages)...
checking pages in range 0 to 415231
page 12607 okay: 3.036% done
page 26367 okay: 6.350% done
page 40959 okay: 9.864% done
page 56255 okay: 13.548% done
page 70847 okay: 17.062% done
page 84607 okay: 20.376% done
page 98815 okay: 23.798% done
page 112895 okay: 27.189% done
page 126079 okay: 30.364% done
page 139135 okay: 33.508% done
page 153087 okay: 36.868% done
page 167615 okay: 40.367% done
page 181887 okay: 43.804% done
page 195839 okay: 47.164% done
page 210687 okay: 50.740% done
page 225087 okay: 54.208% done
page 237503 okay: 57.198% done
page 252223 okay: 60.743% done
page 266815 okay: 64.257% done
page 281279 okay: 67.740% done
page 294143 okay: 70.838% done
page 309311 okay: 74.491% done
page 323711 okay: 77.959% done
page 337983 okay: 81.396% done
page 352703 okay: 84.941% done
page 366911 okay: 88.363% done
page 381631 okay: 91.908% done
page 396671 okay: 95.530% done
page 412799 okay: 99.414% done

$ innochecksum -v muc_affiliations.ibd
file muc_affiliations.ibd = 1149239296 bytes (70144 pages)...
checking pages in range 0 to 70143
page 4287 okay: 6.113% done
page 18175 okay: 25.912% done
page 31679 okay: 45.164% done
page 45567 okay: 64.964% done
page 56767 okay: 80.931% done

The table name appears to change, it isn't consistently the same table (rosterusers, rostergroups, muc_affiliations).  Further this database was stood up by using a mysqldump from the master, so I would assume we could not have imported in any corrupted pages through this manner of loading a slave?   There are no events in Linux logging (/var/log/messages, dmesg, etc) that indicate hardware.  Further this crash happens on multiple servers in this customer's cluster for this dataset, on PS 5.5.32 and now PS 5.5.41.

Note that this cluster at customer site has 4 servers, two of which have never crashed and their only difference is that they have a BP < 100GB.  The instances that are crashing are ~200GB .  I suggested workaround of smaller BP but that isn't an acceptable long term solution for them.

Customer is going to try a 5.6 installation on a slave to see if it too continues to crash when applying replication events.

Here is innochecksum over the three tables.

Valerii Kravchuk (valerii-kravchuk) on 2015-03-05

Changed in percona-server:
status:	Incomplete → New

Jaime Sicam (jssicam) on 2015-03-17

tags:

added: i51943

Revision history for this message

Vlad Lesin (vlad-lesin) wrote on 2015-03-21:

Possible fix is here https://github.com/mysql/mysql-server/commit/777bf2aab05fb58fad7838aed4ef307b50722786 .

Revision history for this message

Vlad Lesin (vlad-lesin) wrote on 2015-03-23:

https://github.com/percona/percona-server/pull/44
https://github.com/percona/percona-server/pull/43

Revision history for this message

Laurynas Biveinis (laurynas-biveinis) wrote on 2015-03-24:

The error log shows

150304 18:36:27 InnoDB: Assertion failure in thread 140331370538752 in file btr0cur.c line 329

(for page space=837, page_no=28006),

Then

150304 21:13:19 InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 838:0, should be 838:111751!

repeated twice

Then

InnoDB: Record 13110 is above rec heap top 12973
...
InnoDB: Page number (if stored to page already) 12766,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 821

We have three space ids: 821, 837, 838. Is it confirmed that the innochecksumed tables indeed have these ids?

Revision history for this message

Valerii Kravchuk (valerii-kravchuk) wrote on 2015-03-24:

How can we get this information (space ids for tables by name) other than by selecting from the INFORMATION_SCHEMA.INNODB_SYS_TABLES?

As far as I understand we still have MySQL server stopped with gdb attached in case of any further checks needed there, so there is no easy way to run queries. ibdconnect or something else for quick (and safe!) check? Or just dump a page from .ibd?

Revision history for this message

Laurynas Biveinis (laurynas-biveinis) wrote on 2015-03-24:

Looking at 4 bytes at 34 byte offset from the start of the page for the innochecksum'ed files should work. It could be done a few times per tablespace to confirm (offsets 0+34, 16384+34, ...)

But in any case the assertion errors have not indicated so far that any pages have corrupted checksums. Thus, if there's corruption there, it wouldn't be diagnosed by innochecksum, and it is possibly a result of an unknown server bug.

Revision history for this message

Shahriyar Rzayev (rzayev-sehriyar) wrote on 2018-01-25:

#10

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-192

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

auto-customers.percona.com #51533 Edit

Bug watches keep track of this bug in other bug trackers.