mysqld crashed - got signal 11 or signal 6

Bug #792454 reported by Steve Whitehouse
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
New
High
Unassigned

Bug Description

We have a script that performs parallel imports of mysqldump created database dumps. (The dumps are from a mysql 5.0 system) If we import one database at a time, all is well. If we import 2 or more databases at a time, the database server crashes, with signal 6 or 11 (See below) The database server restarts automatically, but is not usable, giving errors like:

ERROR 145 (HY000) at line 1: Table './mysql/proc' is marked as crashed and should be repaired.

Manually stopping and restarting the database, allows the database top recover.

The problem is reproducible with the following script:

#!/bin/bash

mysql -u<user> -p<pass> <db1.sql &
mysql -u<user> -p<pass> <db2.sql &

___________________________________________

Crash details:

110506 1:02:16 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=4
max_threads=1023
thread_count=2
connection_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2254604 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f03cc08f6d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f03e89dce48 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x7c3709]
/usr/sbin/mysqld(handle_segfault+0x464)[0x517af4]
/lib/libpthread.so.0(+0xf8f0)[0x7f03e86a58f0]
/usr/sbin/mysqld[0x94538c]
/usr/sbin/mysqld[0x945aea]
/usr/sbin/mysqld[0x83a276]
/usr/sbin/mysqld[0x822a61]
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x5e)[0x6963be]
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x6f)[0x57c56f]
/usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x7d4)[0x582c94]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x895)[0x592255]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x313)[0x5964a3]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x15cf)[0x597b1f]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x370)[0x633580]
/usr/sbin/mysqld(handle_one_connection+0x51)[0x633641]
/lib/libpthread.so.0(+0x69ca)[0x7f03e869c9ca]
/lib/libc.so.6(clone+0x6d)[0x7f03e792f70d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f03cc10be00): is an invalid pointer
Connection ID (thread ID): 493
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
110506 01:02:16 mysqld_safe Number of processes running now: 0
110506 01:02:16 mysqld_safe mysqld restarted
110506 1:02:16 [ERROR] An old style --language value with language specific part detected: /usr/share/mysql/english/
110506 1:02:16 [ERROR] Use --lc-messages-dir without language specific part instead.
110506 1:02:16 [Note] Flashcache bypass: disabled
110506 1:02:16 [Note] Flashcache setup error is : setmntent failed

110506 1:02:16 [Note] Plugin 'FEDERATED' is disabled.
110506 1:02:16 InnoDB: The InnoDB memory heap is disabled
110506 1:02:16 InnoDB: Mutexes and rw_locks use GCC atomic builtins
110506 1:02:16 InnoDB: Compressed tables use zlib 1.2.3
110506 1:02:16 InnoDB: Using Linux native AIO
110506 1:02:17 InnoDB: Initializing buffer pool, size = 128.0M
110506 1:02:17 InnoDB: Completed initialization of buffer pool
110506 1:02:17 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 7948107399
110506 1:02:17 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 7953350144
InnoDB: Doing recovery: scanned up to log sequence number 7958593024
InnoDB: Doing recovery: scanned up to log sequence number 7963835904
InnoDB: Doing recovery: scanned up to log sequence number 7969078784
InnoDB: Doing recovery: scanned up to log sequence number 7974321664
InnoDB: Doing recovery: scanned up to log sequence number 7979564544
InnoDB: Doing recovery: scanned up to log sequence number 7984807424
InnoDB: Doing recovery: scanned up to log sequence number 7990050304
InnoDB: Doing recovery: scanned up to log sequence number 7995293184
InnoDB: Doing recovery: scanned up to log sequence number 8000536064
InnoDB: Doing recovery: scanned up to log sequence number 8005778944
InnoDB: Doing recovery: scanned up to log sequence number 8011021824
InnoDB: Doing recovery: scanned up to log sequence number 8016264704
110506 1:02:19 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Doing recovery: scanned up to log sequence number 8021507584
InnoDB: Doing recovery: scanned up to log sequence number 8026750464
InnoDB: Doing recovery: scanned up to log sequence number 8031920312
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 6 row operations to undo
InnoDB: Trx id counter is 9C00
110506 1:02:22 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
110506 1:02:23 InnoDB: Waiting for the background threads to start
InnoDB: Starting in background the rollback of uncommitted transactions
110506 1:02:23 InnoDB: Rolling back trx with id 9A2D, 6 rows to undo

InnoDB: Rolling back of trx id 9A2D completed
110506 1:02:23 InnoDB: Rollback of non-prepared transactions completed
110506 1:02:24 Percona XtraDB (http://www.percona.com) 1.1.6-20.1 started; log sequence number 8031920312
110506 1:02:24 [Warning] 'user' entry 'root@all10-v3' ignored in --skip-name-resolve mode.
110506 1:02:24 [Warning] 'proxies_priv' entry '@ root@all10-v3' ignored in --skip-name-resolve mode.
110506 1:02:24 [Note] Event Scheduler: Loaded 0 events
110506 1:02:24 [Note] /usr/sbin/mysqld: ready for connections.

Also seen:

110506 1:35:24 InnoDB: Assertion failure in thread 140570248005392 in file /var/lib/buildbot/slaves/percona-server-51-12/DEB_Ubuntu_lucid_amd64/work/Percona-Server-5.5.11-rel20.2/storage/innobase/os/os0sync.c line 474
InnoDB: Failing assertion: event
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
110506 1:35:24 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=3
max_threads=1023
thread_count=2
connection_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2254604 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x332bd10
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fd90fb2be48 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x7c3709]
/usr/sbin/mysqld(handle_segfault+0x464)[0x517af4]
/lib/libpthread.so.0(+0xf8f0)[0x7fd90f7c38f0]
/lib/libc.so.6(gsignal+0x35)[0x7fd90e99aa75]
/lib/libc.so.6(abort+0x180)[0x7fd90e99e5c0]
/usr/sbin/mysqld[0x925573]
/usr/sbin/mysqld[0x85b0df]
/usr/sbin/mysqld[0x85c60f]
/usr/sbin/mysqld[0x8c0fda]
/usr/sbin/mysqld[0x825c6d]
/usr/sbin/mysqld(_Z15ha_create_tableP3THDPKcS2_S2_P24st_ha_create_informationb+0x15b)[0x696a1b]
/usr/sbin/mysqld(_Z16rea_create_tableP3THDPKcS2_S2_P24st_ha_create_informationR4ListI12Create_fieldEjP6st_keyP7handler+0x1f3)[0x61e733]
/usr/sbin/mysqld(_Z26mysql_create_table_no_lockP3THDPKcS2_P24st_ha_create_informationP10Alter_infobjPb+0x8d7)[0x5f2e27]
/usr/sbin/mysqld(_Z18mysql_create_tableP3THDP10TABLE_LISTP24st_ha_create_informationP10Alter_info+0xaa)[0x5f39aa]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4750)[0x596110]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x313)[0x5964a3]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x15cf)[0x597b1f]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x370)[0x633580]
/usr/sbin/mysqld(handle_one_connection+0x51)[0x633641]
/lib/libpthread.so.0(+0x69ca)[0x7fd90f7ba9ca]
/lib/libc.so.6(clone+0x6d)[0x7fd90ea4d70d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x3776b00): CREATE TABLE `person_queue_users` (
  `id` int(11) NOT NULL auto_increment,
  `created_by` int(11) default NULL,
  `updated_by` int(11) default NULL,
  `created_at` datetime default NULL,
  `updated_at` datetime default NULL,
  `is_deleted` tinyint(1) NOT NULL default '0',
  `came_from_migration` int(11) default NULL,
  `lock_version` int(11) NOT NULL default '0',
  `queue_id` int(11) NOT NULL,
  `user_id` int(11) NOT NULL,
  `notify_on_queue` tinyint(1) NOT NULL default '1',
  `notify_off_queue` tinyint(1) NOT NULL default '0',
  PRIMARY KEY (`id`),
  KEY `fk_person_queue_users_queue_id` (`queue_id`),
  CONSTRAINT `fk_person_queue_users_queue_id` FOREIGN KEY (`queue_id`) REFERENCES `person_queues` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8
Connection ID (thread ID): 60
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
110506 01:35:24 mysqld_safe Number of processes running now: 0
110506 01:35:24 mysqld_safe mysqld restarted
110506 1:35:24 [ERROR] An old style --language value with language specific part detected: /usr/share/mysql/english/
110506 1:35:24 [ERROR] Use --lc-messages-dir without language specific part instead.
110506 1:35:24 [Note] Flashcache bypass: disabled
110506 1:35:24 [Note] Flashcache setup error is : setmntent failed

110506 1:35:24 [Note] Plugin 'FEDERATED' is disabled.
110506 1:35:24 InnoDB: The InnoDB memory heap is disabled
110506 1:35:24 InnoDB: Mutexes and rw_locks use GCC atomic builtins
110506 1:35:24 InnoDB: Compressed tables use zlib 1.2.3
110506 1:35:24 InnoDB: Using Linux native AIO
110506 1:35:24 InnoDB: Initializing buffer pool, size = 128.0M
110506 1:35:24 InnoDB: Completed initialization of buffer pool
110506 1:35:24 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 8042603074
110506 1:35:24 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 8045972937
110506 1:35:24 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
110506 1:35:25 InnoDB: Waiting for the background threads to start
110506 1:35:26 Percona XtraDB (http://www.percona.com) 1.1.6-20.1 started; log sequence number 8045972937
110506 1:35:26 [Warning] 'user' entry 'root@all10-v3' ignored in --skip-name-resolve mode.
110506 1:35:26 [Warning] 'proxies_priv' entry '@ root@all10-v3' ignored in --skip-name-resolve mode.
110506 1:35:26 [Note] Event Scheduler: Loaded 0 events
110506 1:35:26 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.11-55-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 20.2

Environment:

Server: Ubuntu Linux 10.04 Lucid, running under Vmware

Packages:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-===============================-===============================================-============================================
ii libmysqlclient-dev 5.5.11-rel20.2-115.lucid Percona Server database development files
ii libmysqlclient16 5.1.55-12.6-200.lucid Percona Server database client library
ii libmysqlclient18 5.5.11-rel20.2-115.lucid Percona Server database client library
ii percona-server-client-5.5 5.5.11-rel20.2-115.lucid Percona Server database client binaries
ii percona-server-common-5.5 5.5.11-rel20.2-115.lucid Percona Server database common files (e.g. /
ii percona-server-server-5.5 5.5.11-rel20.2-115.lucid Percona Server database server binaries

affects: percona-projects-qa → percona-server
Revision history for this message
Steve Whitehouse (steve-whitehouse) wrote :

This may be related to #758788

We too have innodb_dict_size_limit set. In fact its our primary driver for considering the use of Percona. After reading #758788 and commenting out our innodb_dict_size_limit setting, the problem above does not occur.

Revision history for this message
Patrick Crews (patrick-crews) wrote :

Could you provide some information regarding how large the db's are / how many tables / rows?
Could you also provide information regarding the innodb_dict_size_limit value?

Changed in percona-server:
importance: Undecided → Low
assignee: nobody → Patrick Crews (patrick-crews)
Stewart Smith (stewart)
Changed in percona-server:
importance: Low → High
Revision history for this message
Steve Whitehouse (steve-whitehouse) wrote :

innodb_dict_size_limit = 1048576

There are thousands of databases, limited (today) by the size of the dictionary to around 3000 physical databases per server.

Approx 10% of the databases are in active use over a 24 hour period.

Each database is around 200 tables. Table size varies but maxes out at several hundred thousand rows. Ultimately, some tables will exceed 1 Million rows.

Whilst we see this bug during a parallel mysql load, (which we could avoid) we have similar execution paths in our main application and cannot tolerate possible db crashes.

Revision history for this message
Stewart Smith (stewart) wrote :

We may have a fix on the way that solves this problem. Hopefully in the next set of releases.

Changed in percona-server:
assignee: Patrick Crews (patrick-crews) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.