backup fails with history on server enabled

Bug #1509812 reported by RSN86
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
High
Sergei Glushchenko
2.3
Fix Released
High
Sergei Glushchenko

Bug Description

After a system update innobackupex start crashing on full backups

xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2e) [0x8c502e]
innobackupex(handle_fatal_signal+0x273) [0x736dc3]
/lib64/libpthread.so.0(+0xf790) [0x7f662db8d790]
/lib64/libc.so.6(+0x1336bf) [0x7f662d0566bf]
innobackupex(write_xtrabackup_info(st_mysql*)+0x4b2) [0x59e632]
innobackupex(backup_finish()+0x16e) [0x5a10ee]
innobackupex(xtrabackup_backup_func()+0xeba) [0x58e1aa]
innobackupex(main+0xaea) [0x591efa]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x7f662cf41d5d]
innobackupex() [0x585e79]

This is how backups are taken
innobackupex --history='Full_'`date +%Y%m%d` --user=${BKP_USER} --password=${BKP_PASS} --parallel=4 /mysql/backup/

my.cnf
https://gist.github.com/rsn86/883d1e09dd1519e7e712

fuull innobackupex log
https://gist.github.com/rsn86/ecd714d45d66557fff4d

general system information (uname,memory,cpu, etc.):
https://gist.github.com/rsn86/b10c9a94f4576c524588

yum update history:
https://gist.github.com/rsn86/5433ba477cc364f70595

mysql --version
mysql Ver 14.14 Distrib 5.6.26-74.0, for Linux (x86_64) using 6.0

innobackupex --version
innobackupex version 2.3.2 Linux (x86_64) (revision id: 306a2e0)

rpm -qa Percona*
Percona-Server-shared-56-5.6.26-rel74.0.el6.x86_64
Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64
Percona-Server-client-56-5.6.26-rel74.0.el6.x86_64
Percona-Server-server-56-5.6.26-rel74.0.el6.x86_64

rpm -qa percona*
percona-zabbix-templates-1.1.5-1.noarch
percona-release-0.1-3.noarch
percona-toolkit-2.2.15-2.noarch
percona-xtrabackup-2.3.2-1.el6.x86_64

Tags: i62431
Revision history for this message
RSN86 (rsn86) wrote :

Downgrading to xtrabackup 2.2.12 solved the issue

yum downgrade percona-xtrabackup-2.2.12-1.el6

xtrabackup -v
xtrabackup version 2.2.12 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 8726828)

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :

Could be related to history on server feature. Could you please check same command without "--history='Full_'`date +%Y%m%d`"?

Revision history for this message
RSN86 (rsn86) wrote :

Without --history it worked fine.

These are the privileges granted to backuper user:
GRANT SELECT, INSERT, RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'backuper'@'localhost'
GRANT CREATE ON `PERCONA_SCHEMA`.* TO 'backuper'@'localhost'

Revision history for this message
brad svee (brad-svee) wrote :

obviously this has to do with the fact that innobackupex was just completely rewritten in a different language, looks like it went from perl to C ? That's a bold move and it seems like some things like this were totally missed, I can't find the xtrabackup_history code in the new stuff

Revision history for this message
Agustín (agustin-gallego) wrote :

Output for:

# innobackupex --parallel=10 --no-timestamp /backups/full_backup

Revision history for this message
Agustín (agustin-gallego) wrote :

Output for:

# innobackupex --parallel=10 --history="test" --no-timestamp /backups/full_backup

Revision history for this message
Agustín (agustin-gallego) wrote :

I have confirmed this using xtrabackup 2.3.2 with PS 5.6.27:

mysql> show variables like '%version%';
+-------------------------+------------------------------------------------------+
| Variable_name | Value |
+-------------------------+------------------------------------------------------+
| innodb_version | 5.6.27-75.0 |
| protocol_version | 10 |
| slave_type_conversions | |
| version | 5.6.27-75.0 |
| version_comment | Percona Server (GPL), Release 75.0, Revision 8bb53b6 |
| version_compile_machine | x86_64 |
| version_compile_os | Linux |
+-------------------------+------------------------------------------------------+
7 rows in set (0.00 sec)

[root@VM-centos67 backups]# xtrabackup --version
xtrabackup version 2.3.2 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 306a2e0)

[root@VM-centos67 backups]# rpm -qa | grep -i percona
Percona-Server-client-56-5.6.27-rel75.0.el6.x86_64
Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64
percona-xtrabackup-debuginfo-2.3.2-1.el6.x86_64
percona-release-0.1-3.noarch
Percona-Server-shared-56-5.6.27-rel75.0.el6.x86_64
percona-xtrabackup-2.3.2-1.el6.x86_64
Percona-Server-server-56-5.6.27-rel75.0.el6.x86_64

See attached outputs for two runs:

1- innobackupex --parallel=10 --no-timestamp /backups/full_backup
2- innobackupex --parallel=10 --history="test" --no-timestamp /backups/full_backup

I got the same stack trace from #2:

151126 22:25:50 [00] Writing xtrabackup_info
151126 22:25:50 [00] ...done
22:25:50 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2e) [0x8c502e]
innobackupex(handle_fatal_signal+0x273) [0x736dc3]
/lib64/libpthread.so.0(+0xf790) [0x7f3e62c12790]
/lib64/libc.so.6(+0x81211) [0x7f3e6127d211]
innobackupex(write_xtrabackup_info(st_mysql*)+0x4b2) [0x59e632]
innobackupex(backup_finish()+0x16e) [0x5a10ee]
innobackupex(xtrabackup_backup_func()+0xeba) [0x58e1aa]
innobackupex(main+0xaea) [0x591efa]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x7f3e6121ad5d]
innobackupex() [0x585e79]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup

Setting to confirmed.

Agustín.

Revision history for this message
Agustín (agustin-gallego) wrote :

See comment #7

tags: added: i62431
Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :

I am able to reproduce the crash with binary log disabled. Please confirm that you have binary log disabled.

summary: - xtrabackup got signal 11 on full backup
+ backup fails with history on server enabled
Revision history for this message
Serge Victor (ser) wrote :

Серге́й, I can confirm it is exactly my situation, no binary log enabled.

Revision history for this message
Agustín (agustin-gallego) wrote :

Right, no binary log enabled.

mysql> show variables like 'log_bin';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin | OFF |
+---------------+-------+
1 row in set (0.00 sec)

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-441

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.