XtraBackup crashes in prepare stage on MariaDB 10.1

Bug #1651885 reported by Geoff Montee
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
New
Undecided
Unassigned

Bug Description

A user saw the following crash with XtraBackup 2.4.1 while attempting to prepare a backup that was taken on a MariaDB 10.1.11 server:

2016-12-20 21:00:59 0x7ff5f3ec4740 InnoDB: Assertion failure in thread 140694336063296 in file fil0fil.cc line 3681
InnoDB: Failing assertion: fsp_flags_is_valid(flags)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
04:00:59 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2c)[0xe0b60c]
innobackupex(handle_fatal_signal+0x262)[0x9c1a72]
/lib64/libpthread.so.0(+0xf370)[0x7ff5f3aa9370]
/lib64/libc.so.6(gsignal+0x37)[0x7ff5f1e011d7]
/lib64/libc.so.6(abort+0x148)[0x7ff5f1e028c8]
innobackupex[0x6e872b]
innobackupex(_Z14fil_ibd_createmPKcS0_mm+0x92a)[0x7fb8ba]
innobackupex[0x932094]
innobackupex[0x936ec9]
innobackupex[0x93778f]
innobackupex(_Z35recv_recovery_from_checkpoint_startm+0xc2d)[0x9386dd]
innobackupex(_Z34innobase_start_or_create_for_mysqlv+0x3641)[0x811131]
innobackupex[0x6e16a2]
innobackupex(main+0xeb2)[0x6f1de2]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7ff5f1dedb35]
innobackupex[0x70b355]

No other errors occurred prior to this during the backup stage or the prepare stage.

This server is not using MariaDB-specific InnoDB features, such as tablespace encryption or page compression. However, the server does have innodb_page_size set to 64k.

XtraBackup 2.4.1 could apparently back up this server until a couple days ago, and then the crashing started to happen. As far as the user knows, nothing interesting has changed on the server since then. Since the crash started happening, the crash is 100% reproducible on this server, and backups can no longer be prepared.

Here is the server's configuration file:

[mariadb]
plugin-maturity=gamma

# -- Remove the following line to enable feedback reporting to mariadb.org
feedback-url=''

[mysqld]
enable-secure-auth
# -- Auditing - pre-load Plugin
##plugin-load=server_audit

[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
datadir = /mariadb/data/
userstat = 1
key_buffer_size = 256M
max_allowed_packet = 1M
table_cache = 256
sort_buffer_size = 1M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
max_connections = 300
tmpdir = /mariadb/temp/
log_error = /mariadb/logs/error.log

innodb_buffer_pool_size = 500M
innodb_additional_mem_pool_size = 8M
innodb_file_io_threads = 4
innodb_lock_wait_timeout = 50
innodb_file_per_table = 1
innodb_page_size = 65536

log-bin=/mariadb/arch/mariadb-bin.index
server_id=1

innodb_data_home_dir = /mariadb/ibdata
innodb_log_group_home_dir = /mariadb/logs
innodb_log_files_in_group = 2
innodb_log_file_size = 256M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_log_archive = 1
innodb_log_arch_dir = /mariadb/arch

#skip-name-resolve
log_bin_trust_function_creators = 1

#max-allowed-packet = 64M
open-files-limit = 65535
max-connect-errors = 100000
max-connections = 1000
thread-cache-size = 200
back-log = 128
#net_buffer_length = 16K
#connect_timeout = 10
#net_read_timeout = 30
#net_write_timeout = 60
#interactive_timeout = 28800
#wait_timeout = 28800
#net_retry_count = 10
character-set-server = 'utf8'
collation-server = 'utf8_general_ci'
#read_rnd_buffer_size = 256K
#sort_buffer_size = 2M
#read_buffer_size = 128K
#max_tmp_tables = 32
tmp-table-size = 32M
max-heap-table-size = 32M
sql-mode = STRICT_TRANS_TABLES

optimizer_search_depth = 0
optimizer_switch = 'index_merge=on'
optimizer_switch = 'index_merge_union=on'
optimizer_switch = 'index_merge_sort_union=on'
optimizer_switch = 'index_merge_intersection=on'
optimizer_switch = 'index_merge_sort_intersection=off'
optimizer_switch = 'index_condition_pushdown=on'
optimizer_switch = 'derived_merge=on'
optimizer_switch = 'derived_with_keys=on'
optimizer_switch = 'firstmatch=on'
optimizer_switch = 'loosescan=on'
optimizer_switch = 'materialization=on'
optimizer_switch = 'in_to_exists=on'
optimizer_switch = 'semijoin=on'
optimizer_switch = 'partial_match_rowid_merge=on'
optimizer_switch = 'partial_match_table_scan=on'
optimizer_switch = 'subquery_cache=on,mrr=on'
optimizer_switch = 'mrr_cost_based=on'
optimizer_switch = 'mrr_sort_keys=off'
optimizer_switch = 'outer_join_with_cache=on'
optimizer_switch = 'semijoin_with_cache=on'
optimizer_switch = 'join_cache_incremental=on'
optimizer_switch = 'join_cache_hashed=on'
optimizer_switch = 'join_cache_bka=on'
optimizer_switch = 'optimize_join_buffer_size=on'
optimizer_switch = 'table_elimination=on'
optimizer_switch = 'extended_keys=on'

join-buffer-space-limit = 4M
join-cache-level = 6
join-buffer-size = 4M

# Security
symbolic-links = 0
local-infile = 0

# Logging
log-warnings = 2
slow-query-log = 1
long-query-time = 5
log-slow-verbosity = 'query_plan,innodb'
#log_slow_rate_limit = 1000

#server-id = 1
slave_net_timeout = 60
#max_prepared_stmt_count = 16382
binlog-annotate-row-events = ON
#log-bin = mariadb-bin
binlog-format = MIXED
expire-logs-days = 7
max-binlog-size = 1024M
sync-binlog = 1
binlog-stmt-cache-size = 128K
binlog-cache-size = 256K
#slave_compressed_protocol = ON
slave-transaction-retries = 10
#relay_log_recovery = ON
#sync_master_info = 1
#sync_relay_log = 0
#sync_relay_log_info = 1

# InnoDB
default-storage-engine = 'InnoDB'
innodb-stats-on-metadata = 0
innodb-stats-sample-pages = 32
table-definition-cache = 2048
table-open-cache = 2048
transaction-isolation = READ-COMMITTED
innodb-support-xa = ON

query-cache-size = 0
query-cache-type = 0
innodb-buffer-pool-instances = 8
innodb-buffer-pool-size = 6G
innodb-max-dirty-pages-pct = 50
innodb-file-per-table = 1
innodb-file-format = Barracuda
innodb-file-format-max = Barracuda
innodb-flush-log-at-trx-commit = 1
#Save and restore buffer pool to be transparent for user
#innodb_flush_method = O_DIRECT
innodb-log-buffer-size = 64M
innodb-log-files-in-group = 2
#innodb-log-file-size = 1024M
#innodb_purge_threads = 1
#innodb_io_capacity = 200
#innodb_flush_neighbors = 0
innodb-read-io-threads = 8
innodb-write-io-threads = 8
#innodb-thread-concurrency = 64

innodb-open-files = 2048

key-buffer-size = 64M
flush = OFF
myisam-recover-options = BACKUP,FORCE
myisam-sort-buffer-size = 64M

userstat = ON
loose-archive = OFF
loose-blackhole = OFF
#loose-federated = OFF
#innodb = FORCE

[mysql]
default-character-set = 'utf8'
auto-rehash = FALSE
local-infile = 1
max-allowed-packet = 64M
secure-auth = TRUE

[mysqldump]
max-allowed-packet = 1G
default-character-set = 'utf8'

[myisamchk]
key-buffer-size = 1G
sort-buffer-size = 1G
read-buffer-size = 8M
write-buffer-size = 8M

Revision history for this message
Geoff Montee (geoff-montee) wrote :

I've been told that current versions of Percona XtraBackup might not be able to back up a MariaDB 10.1 server when innodb_page_size != 16K, so that might be the problem here.

Revision history for this message
Richard (richard-stracke) wrote :

Patch suggested since 2.8.2016, but not confirmed from percona.

https://github.com/percona/percona-xtrabackup/pull/200/commits/1a0c4ecfae1e0217c847b868858de4fb9d5f44a2

Richard

Changed in percona-xtrabackup:
assignee: nobody → Muhammad Irfan (muhammad-irfan)
Changed in percona-xtrabackup:
status: New → Confirmed
status: Confirmed → New
Changed in percona-xtrabackup:
assignee: Muhammad Irfan (muhammad-irfan) → nobody
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1421

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.