xtrabackup: error: log block numbers mismatch

Thank you for the bug report. This problem is a regression introduced in the 2.0 branch by the fix for bug #805593. I will commit a fix for it shortly.

It seems this bug stil repeatable in xtrabackup-2.0.
Iam using percona-xtrabackup-2.0.0-417.rhel5 and trying to remotely backup the largest database running on Percona 5.5.16.

xtrabackup: error: Log block checksum mismatch (block no 933830996 at lsn 1028145714688):
expected 997077168, calculated checksum 438936379
xtrabackup: Error: xtrabackup_copy_logfile() failed.

xtrabackup: Error: log_copying_thread failed.

Any help would be appreciated.

Thank you

That seems to be a different problem. A checksum (rather than number) mismatch means that the checksum calculated by XtraBackup for a log block does not match whatever is stored in the block itself. This could either be a corrupted log due to a hardware or FS failure, or an XtraBackup bug. We don't have any similar reports, so I'm inclined to think it's a hardware problem. Is there anything in the kernel log?

Thanks Alexey,
There is nothing suspicious reported in system and kernel logs.
No warning/errors in MySQL logs on object corruption.
Unfortunately, this errors is repeatable and backup is failing and we will not be able to recover the data.
Is there a way to troubleshoot what caused it. Appreciate your help.

I think I found the reason. When XtraBackup reads a log block that has not been completely written by the server yet, it will fail with an error like that. Which is a regression introduced in 2.0. XtraBackup 1.6 will print a warning, but will retry reading the same block later.

I reported this problem as bug #1015416.

Thanks for the details. appreciate your time.

- percona-xtrabackup-2.0.1-446.rhel5
- datadir ~ 80GB

xtrabackup: error: log block numbers mismatch:
xtrabackup: error: expected log block no. x, but got no. y from the log file.
xtrabackup: Error: xtrabackup_copy_logfile() failed.

xtrabackup: Error: log_copying_thread failed.
120719 17:28:44 innobackupex: All tables unlocked
120719 17:28:44 innobackupex: Connection to database server closed

innobackupex: Backup created in directory '/data/var/lib/mysql'
innobackupex: MySQL binlog position: filename 'mysql-bin.000994', position 316435839
innobackupex: You must use -i (--ignore-zeros) option for extraction of the tar stream.
120719 17:28:44 innobackupex: xtrabackup_55 failed! (exit code 1) The backup may not be complete.

Same here:

- InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona Inc 2009-2012. All Rights Reserved.
- xtrabackup_55 version 2.0.1 for Percona Server 5.5.16 Linux (x86_64) (revision id: 446)
- MySQL-Server version: 5.5.24-55-log Percona Server (GPL), Release rel26.0, Revision 256
- act as a slave
- Datadir ~81GB

.
.
.
xtrabackup: error: log block numbers mismatch:
xtrabackup: error: expected log block no. 537104441, but got no. 537124913 from the log file.
xtrabackup: error: it looks like InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small.
xtrabackup: Error: xtrabackup_copy_logfile() failed.

.
.
.
xtrabackup: The latest check point (for incremental): '1374609880770'
xtrabackup: Stopping log copying thread.

xtrabackup: Error: log_copying_thread failed.
120719 12:55:21 innobackupex: All tables unlocked
120719 12:55:21 innobackupex: Connection to database server closed

innobackupex: Backup created in directory '/mnt/2012-07-19_11-52-59'
innobackupex: MySQL binlog position: filename 'shop4-slavedb-bin.000004', position 107
innobackupex: MySQL slave binlog position: master host 'shop4-masterdb-int', filename 'shop4-masterdb.002048', position 420950703
120719 12:55:21 innobackupex: xtrabackup_55 failed! (exit code 1) The backup may not be complete.

I've been trying the `innobackupex` script in the source code folder (pulling with `bzr branch lp:percona-xtrabackup`) and got the same error:

xtrabackup: error: log block numbers mismatch:
xtrabackup: error: expected log block no. 744382184, but got no. 744402656 from the log file.
xtrabackup: error: it looks like InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small.
xtrabackup: Error: xtrabackup_copy_logfile() failed.

Bernd, Quan,

The problems you have reported appear to be different from the original one reported in this bug.

In both cases it's probably what the error message says: the InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small.

Please try increasing the log file size.

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-229