Failing assertion: !rec_get_deleted_flag(rec, rec_offs_comp(offsets)) with debug 5.5 build

Bug #1216532 reported by Alex Yurchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
New
Undecided
Unassigned

Bug Description

Standard seesaw test yielded on donor (debug build form 5.5 branch head):

130825 13:07:25 [Note] WSREP: Quorum results:
 version = 2,
 component = PRIMARY,
 conf_id = 5,
 members = 2/3 (joined/total),
 act_id = 3675,
 last_appl. = 3552,
 protocols = 0/4/2 (gcs/repl/appl),
 group UUID = bcabf528-0d6d-11e3-87ec-1e908c37032f
130825 13:07:25 [Note] WSREP: Flow-control interval: [28, 28]
130825 13:07:25 [Note] WSREP: New cluster view: global state: bcabf528-0d6d-11e3-87ec-1e908c37032f:3675, view# 6: Primary, number of nodes: 3, my index: 0, protocol version 2
130825 13:07:25 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130825 13:07:25 [Note] WSREP: Assign initial position for certification: 3675, protocol version: 2
130825 13:07:26 InnoDB: Assertion failure in thread 140337117845248 in file row0sel.c line 2724
InnoDB: Failing assertion: !rec_get_deleted_flag(rec, rec_offs_comp(offsets))
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:07:26 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=13
max_threads=1024
thread_count=13
connection_count=13
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2248592 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x20121c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa2c80ece50 thread_stack 0x40000
/run/shm/galera0/mysql/sbin/mysqld(my_print_stacktrace+0x38)[0x8e17aa]
/run/shm/galera0/mysql/sbin/mysqld(handle_fatal_signal+0x3af)[0x76cac7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x7fa2d904abd0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fa2d806a037]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fa2d806d698]
/run/shm/galera0/mysql/sbin/mysqld[0x9c110d]
/run/shm/galera0/mysql/sbin/mysqld[0x9c2b38]
/run/shm/galera0/mysql/sbin/mysqld[0x97d366]
/run/shm/galera0/mysql/sbin/mysqld(_ZN7handler14index_read_mapEPhPKhm16ha_rkey_function+0x66)[0x777568]
/run/shm/galera0/mysql/sbin/mysqld(_ZN7handler18index_read_idx_mapEPhjPKhm16ha_rkey_function+0x6f)[0x775a79]
/run/shm/galera0/mysql/sbin/mysqld[0x63ee9a]
/run/shm/galera0/mysql/sbin/mysqld[0x63ea2e]
/run/shm/galera0/mysql/sbin/mysqld[0x6281f4]
/run/shm/galera0/mysql/sbin/mysqld(_ZN4JOIN8optimizeEv+0xb17)[0x62116d]
/run/shm/galera0/mysql/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x2cd)[0x626c4f]
/run/shm/galera0/mysql/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x195)[0x61ed4c]
/run/shm/galera0/mysql/sbin/mysqld[0x5f8900]
/run/shm/galera0/mysql/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xc0e)[0x5f087a]
/run/shm/galera0/mysql/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x201)[0x5fb895]
/run/shm/galera0/mysql/sbin/mysqld[0x5fb0d9]
/run/shm/galera0/mysql/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xc90)[0x5edb8d]
/run/shm/galera0/mysql/sbin/mysqld(_Z10do_commandP3THD+0x5b2)[0x5ecacd]
/run/shm/galera0/mysql/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x180)[0x6d5e01]
/run/shm/galera0/mysql/sbin/mysqld(handle_one_connection+0x33)[0x6d591b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e)[0x7fa2d9042f8e]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fa2d812ce1d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fa270004bc0): SELECT * FROM comm02 WHERE p = 2
Connection ID (thread ID): 27
Status: NOT_KILLED

Never had this before, DONOR status is probably not essential - state request hasn't been processed yet.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Was the innodb_flush_method set to O_DIRECT when this happened? I
noticed this crash only when it was set to default. Since
O_DIRECT is used normally, I didn't decide to pursue it further.
Also, I noticed this only on jenkins with VMs, so not having
O_DIRECT indicated to some issue with VM caching.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :
Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Raghu, thanks for the pointer. I forgot to search PXC bugs. Probably this one should be marked as a duplicate.

And yes, the innodb_flush_method was unset, so I guess it is default. However the cluster was running on real hardware with data directories in /dev/shm

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

@Alex,

Thanks, I have marked the other issue as fixed, and didn't pursue
it further since innodb_flush_method is usually set to O_DIRECT
in production. However, a further investigation of this is
welcome.

Revision history for this message
Seppo Jaakola (seppo-jaakola) wrote :

tried this regression test against native mysql server 5.5.35 (native in the sense that wsrep_provider=none), and running the test load agains single mysql server.

The assert hits very soon after the test load is started, this looks strongly as native MySQL issue

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.