Non-default log location causes Galera join to "fail" (xtrabackup)

Bug #1243106 reported by Michael Nielsen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
New
Undecided
Unassigned

Bug Description

Hope this is the right place to report this.

Software
percona-xtrabackup-2.1.5-680.rhel6.x86_64
galera-24.2.7-1.rhel6.x86_64
mysql-5.5.33_wsrep_24.8-linux-x86_64.tar.gz (installed in /opt/mysql-galera/default/ and liked from /usr/local/mysql)

We have a wish to move log files away from the disk that contains the database files, to spread the IO of the system across our SAN, thus we have configured the system to use.

data -> /data/mysql/data
logs -> /data/mysql/logs

With the basic configuration (I've removed tuning information, as I think it's irrelevant, and I can reproduce the problem using just the configuration listed here.

[mysqld]

# 1. Mandatory settings: these settings are REQUIRED for proper cluster operation
query_cache_size=0
binlog_format=ROW
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
innodb_doublewrite=1 - this is the default and it should stay this way

# 2. Optional mysqld settings: your regular InnoDB tuning and such
datadir=/data/mysql/data
innodb_buffer_pool_size=1G # taken from mysql/common.cnf
innodb_log_file_size=100M # taken from mysql/common.cnf
innodb_file_per_table # taken from mysql/common.cnf
innodb_flush_log_at_trx_commit=2 # taken from mysql/common.cnf

# 3. wsrep provider configuration: basic wsrep options
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_provider_options="gcache.size=1G"
wsrep_cluster_address=gcomm://10.7.30.122,10.7.30.123,10.7.30.124,10.7.30.125
wsrep_cluster_name='hael_dev_galera_cluster'
wsrep_sst_method=xtrabackup
wsrep_sst_auth=root:testpass
wsrep_node_name='node-a' # (a/b/c/d for each node)

# 4. additional "frequently used" wsrep settings
wsrep_node_incoming_address='192.168.10.2' # default - use random
wsrep_sst_donor='node-a' # default use random
wsrep_slave_threads=16

# this block seems to make no difference.
[xtrabackup]
datadir=/data/mysql/data
log-bin=/data/mysql/logs/binlog
log-bin-index=/data/mysql/logs/binlog
log-error=/data/mysql/logs/error.log
innodb-log-group-home-dir = /data/mysql/logs

If in my.cnf I setup the following (on all nodes, and bootstrap).

log-bin=/data/mysql/logs/binlog
log-bin-index=/data/mysql/logs/binlog
innodb_log_group_home_dir = /data/mysql/logs

(it only seems to be the innodb_log_group_home_dir that's the real culprit).

I keep seeing the following errors in the database logs

131022 9:56:57 InnoDB: Error: page 3 log sequence number 7750166
InnoDB: is in the future! Current system log sequence number 1596006.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: for more information.

And I notice, that though I've requested that the innodb logs (ib_logfile0, ib_logfile1) to be placed in /data/mysql/logs

Examining nodes..
node-a => the bootstrap (first node of the cluster, started with the --wsrep_new_cluster), there are only logs in /data/mysql/logs ->

# ls /data/mysql/data/ib_logfile*
ls: cannot access /data/mysql/data/ib_logfile*: No such file or directory

(as I would expect).

on all other nodes (b,c,d) all have logs placed in this diretory, as well as in /data/mysql/logs

# ls /data/mysql/data/ib_logfile*
/data/mysql/data/ib_logfile0 /data/mysql/data/ib_logfile1

# ls /data/mysql/logs/ib_logfile*
/data/mysql/logs/ib_logfile0 /data/mysql/logs/ib_logfile1

And it seems that xtrabackup is creating the log files in /data/mysql/data, though it has been instructed to place them in /data/mysql/logs

Running the system using rsync, the above problem does not exist, and the ib_logfiles are in the correct locations.

thus on all nodes

# ls /data/mysql/data/ib_logfile*
ls: cannot access /data/mysql/data/ib_logfile*: No such file or directory
# ls /data/mysql/logs/ib_logfile*
/data/mysql/logs/ib_logfile0 /data/mysql/logs/ib_logfile1

Not sure if it's an xtrabackup bug..

Revision history for this message
Seppo Jaakola (seppo-jaakola) wrote :
Revision history for this message
Michael Nielsen (hael-n) wrote :

You are correct, it's the same issue.. I just did not find it in my searches.

Just slightly different error messages in the logs.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.