SST fails with innodb_data_home_dir/innodb_log_home_dir

Bug #1098566 reported by Raghavendra D Prabhu
58
This bug affects 13 people
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Status tracked in 5.6
5.5
Fix Released
Medium
Alex Yurchenko
5.6
Fix Released
Medium
Alex Yurchenko
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Status tracked in 5.6
5.5
Fix Released
Undecided
Raghavendra D Prabhu
5.6
Fix Released
Undecided
Raghavendra D Prabhu
percona-xtrabackup (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

This bug is similar to lp:1019288 but it shows a different error message.

================
130111 19:11:05 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file /mnt/test/ibdata1 did not exist:
InnoDB: a new database to be created!
130111 19:11:05 InnoDB: Setting file /mnt/test/ibdata1 size to 100 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100
130111 19:11:05 InnoDB: Error: all log files must be created at the same time.
130111 19:11:05 InnoDB: All log files must be created also in database creation.
130111 19:11:05 InnoDB: If you want bigger or smaller log files, shut down the
130111 19:11:05 InnoDB: database and make sure there were no errors in shutdown.
130111 19:11:05 InnoDB: Then delete the existing log files. Edit the .cnf file
130111 19:11:05 InnoDB: and start the database again.
130111 19:11:05 [ERROR] Plugin 'InnoDB' init function returned error.
130111 19:11:05 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
130111 19:11:05 [ERROR] Unknown/unsupported storage engine: InnoDB
130111 19:11:05 [ERROR] Aborting

==========================

This happens because SST will try to copy the ibdata1 file
to the datadir and InnoDB of the
receiver will also create a ibdata1 in innodb_data_home_dir, and
this clashes.

Workaround: Not to have a separate innodb_data_home_dir

Fix: To make sure SST is aware of innodb_data_home_dir.

To reproduce, have something like:

datadir=/var/lib/mysql
innodb_data_home_dir = /mnt/test
innodb_data_file_path = ibdata1:100M:autoextend

in my.cnf along with others.

Where it fails:

 for (i = 0; i < srv_n_log_files; i++) {
  err = open_or_create_log_file(create_new_db, &log_file_created,
           log_opened, 0, i);
  if (err != DB_SUCCESS) {

   return((int) err);
  }

  if (log_file_created) {
   log_created = TRUE;
  } else {
   log_opened = TRUE;
  }
  if ((log_opened && create_new_db)
      || (log_opened && log_created)) {
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: Error: all log files must be"
    " created at the same time.\n");
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: All log files must be"
    " created also in database creation.\n");
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: If you want bigger or smaller"
    " log files, shut down the\n");
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: database and make sure there"
    " were no errors in shutdown.\n");
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: Then delete the existing log files."
    " Edit the .cnf file\n");
   ut_print_timestamp(stderr);
   fprintf(stderr,
    " InnoDB: and start the database again.\n");

   return(DB_ERROR);
  }
 }

in srv0start.c

Changed in percona-xtradb-cluster:
status: New → Confirmed
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

innobackupex --copy-back/--move-back checks innodb_data_home_dir,
innodb_data_file_path and innodb_log_group_home_dir correctly.
So, it is better to use it.

wsrep_sst_xtrabackup can be modified to use a temporary directory and then use innobackupex --move-back to handle correctly.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

However, in many instances, the partition is mounted directly on
/var/lib/mysql , so a temporary directory used for SST may be
bad. OTOH, innobackupex --move-back doesn't work if tar is
extracted under /var/lib/mysql/.tmp since it complains of
non-empty data directory.

Changed in percona-xtradb-cluster:
milestone: none → 5.5.30-24.8
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Filed lp:1164945 for xtrabackup for issue mentioned in #2.

Changed in percona-xtradb-cluster:
milestone: 5.5.30-24.8 → 5.5.31-25
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Waiting for XB 2.0.8 to be released to use this.

Changed in percona-xtradb-cluster:
assignee: nobody → Raghavendra D Prabhu (raghavendra-prabhu)
status: Confirmed → Triaged
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

This is not fixed in PXB 2.0.8, will be fixed in PXB 2.1.4 or later.

no longer affects: codership-mysql
Changed in percona-xtradb-cluster:
milestone: 5.5.32-24 → 5.5.32
Revision history for this message
Frederic Descamps (lefred) wrote :

This is also an issue when only referring innodb_log_group_home_dir (which can be good to move those logs to SSD)

summary: - SST fails with innodb_data_home_dir
+ SST fails with innodb_data_home_dir/innodb_log_home_dir
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

In this case, following will be done:

a) Have a separate setting - sst-log-data-home-dir which will be ON by
default for this to work. (Optional: Detect the options from my.cnf and
set them accordingly). Setting is required because if older PXB is used
then SST shouldn't break. (so it can be turned off).

b) Will go only into xtrabackup-v2 only.

c) This will depend on --move-back --force functionality of PXB
(which will be added in 2.1.6).

d) Minimum required version of PXB will also be bumped to 2.1.6
and above.

e) Here is how it goes down:

    i) On joiner, SST will be streamed into a subdirectory of
    datadir - $datadir/.sst

    ii) Backup will be prepared there.

    iii) innobackupex --move-back --force ../ will be used to
    copy back.

    iv) Based on success, .sst directory will be removed.

f) The $datadir/.sst is required but not any other directory
because data directories usually are mounted on separate mount
points and, otherwise ENOSPC issue can arise.

g) This should also handle innodb-data-home-dir,
innodb-log-home-dir, innodb-log-arch-dir and other exotic
directory variables.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

This has been fixed, and will depend on https://bugs.launchpad.net/percona-xtrabackup/+bug/1164945 for the fix (it is going to be included in PXB 2.1.6)

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Since PXB is going to be released after 25.1, moving it to 25.2 for now.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Consequent to the fix https://bugs.launchpad.net/percona-xtrabackup/+bug/1164945, this requires following in addition:

a) Cleanup of innodb-data-home-dir and innodb-log-home-dir

b) Use of the option - --force-non-empty-directories

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Also, move the log files correctly.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

For documentation,
""
This requires sst-special-dirs to be set under [sst] in cnf file to either 0 or 1. 0 being the default. Also, innodb-data-home-dir and/or innodb-log-group-home-dir need to be defined in my.cnf under [mysqld]. Percona Xtrabackup 2.1.6 or higher is required.
""

tags: added: doc-req
Revision history for this message
Kolbe (kolbe) wrote :

Why is this resolution so complicated? Why not put this new functionality into the regular "xtrabackup" SST script instead of into a new one? Why require this sst-special-dirs variable? I don't understand why it would not be desirable for the default behavior of the normal xtrabackup SST script to support innodb_log_group_home_dir, innodb_data_home_dir, et cetera.

Revision history for this message
Serge Kolodeznyh (kolodez) wrote :

I'm agree with Kolbe %)

sst-special-dirs = ! ( $innodb_log_group_home_dir eq $innodb_data_home_dir)

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

@Kolbe, @Serge,

> Why not put this new functionality into the regular "xtrabackup" SST script instead of into a new one?

The 'regular' script is just there for backward compatibility and, so,
all additions go to v2 only. The v2 script works differently and is not
compatible with old one. This is pretty clear in our
documentation and I don't know what is the ambiguity here on that.

> Why require this sst-special-dirs variable

This is because this relied on a PXB feature added in 2.1.6 which
was also newly released. I didn't want to bump minimum dependency version of PXB onto PXC since 2.1.6 was newly released, and it may have had bugs, so didn't want to restrict in case downgrade of PXB is required.

But, now that, it looks good, I will make it default: https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1276904

tags: added: sst xtrabackup
Revision history for this message
Alex Yurchenko (ayurchen) wrote :

This must be fixed with the latest xtrabackup ssp code ported from Percona in:
http://bazaar.launchpad.net/~codership/codership-mysql/wsrep-5.5/revision/3962
http://bazaar.launchpad.net/~codership/codership-mysql/5.6/revision/4058
Marking as fixed optimistically.

Revision history for this message
Bhaskar peddireddy (rpeddireddy) wrote :

hi, I am having the same problem, my MySQL ver. is
Ver 5.5.31 for Linux on x86_64 (Percona XtraDB Cluster (GPL), wsrep_23.7.5.r3880)
xtrabackup ver. is
xtrabackup version 2.0.3 for Percona Server 5.1.59 unknown-linux-gnu (x86_64) (revision id: 470)
I downloaded the srep_sst_xtrabackup-v2
But the problem is still same. i.e., during SST, in prepare stage, it is not copying the files back to its dir. paths as configured in .cnf file
we have
innodb_log_group_home_dir is different from datadir

Please advice which ver. of xtrabackup has the fix.

Thanks for your help in advance
Bhaskar

Revision history for this message
Mikhail Baykov (rusxakep) wrote :

percona-xtrabackup_2.3.2-1 affected

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in percona-xtrabackup (Ubuntu):
status: New → Confirmed
Revision history for this message
Przemek (pmalkowski) wrote :

Bumping up as it is again a problem in latest PXC 5.6.26. Looks like SST script is broken and ibdata1 and ib_logfile* are copied from .sst temp dir to datadir instead of innodb_data_home_dir and innodb_log_group_home_dir accordingly.
Test logs in attachment.

tags: added: i62766
Revision history for this message
Ramesh Sivaraman (rameshvs02) wrote :

Server is crashing even with wsrep_sst_method=rsync with different error

dir info

datadir=/usr/local/mysql/data
basedir=/usr/local/mysql
innodb_data_home_dir=/dev/shm/inno_data
innodb_log_group_home_dir=/dev/shm/inno_data

Error info

2015-12-04 03:36:08 12101 [Note] InnoDB: Setting log file /dev/shm/inno_data/ib_logfile101 size to 48 MB
2015-12-04 03:36:08 12101 [ERROR] InnoDB: Cannot create /dev/shm/inno_data/ib_logfile1
2015-12-04 03:36:08 12101 [ERROR] Plugin 'InnoDB' init function returned error.
2015-12-04 03:36:08 12101 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-12-04 03:36:08 12101 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-12-04 03:36:08 12101 [ERROR] Aborting

-rw-rw---- 1 mysql mysql 12582912 Dec 4 02:55 ibdata1
-rw-rw---- 1 mysql mysql 50331648 Dec 4 02:55 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 Dec 4 02:55 ib_logfile1
-rw-rw---- 1 mysql mysql 50331648 Dec 4 02:55 ib_logfile101

If we run mysql_install_db on joiner node, startup script starts mysqld with success message but in background mysqld process terminating with wrong SST state

[root@c-p130-u1400-240 mysql]# ./support-files/mysql.server start
 ERROR! MySQL (Percona XtraDB Cluster) is not running, but lock file (/var/lock/subsys/mysql) exists
Starting MySQL (Percona XtraDB Cluster)..State transfer in progress, setting sleep higher
. SUCCESS!
[root@c-p130-u1400-240 mysql]#

2015-12-04 03:41:30 12681 [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:5
2015-12-04 03:41:30 12681 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.6.26-74.0-25.12-log' socket: '/usr/local/mysql/data/mysql.sock' port: 3306 Percona XtraDB Cluster binary (GPL) 5.6.26-25.12, Revision 624ef81, wsrep_25.12
2015-12-04 03:41:30 12681 [ERROR] WSREP: Application received wrong state:
        Received: 00000000-0000-0000-0000-000000000000
        Required: f3e24345-9a61-11e5-8823-eabc7692ebb8
2015-12-04 03:41:30 12681 [ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required.

Revision history for this message
Krunal Bauskar (krunal-bauskar) wrote :

Case-1:

5.6.26 (latest one from trunk)
2.2.12 (Xtrabackup)

What I observed
* on node#2 with datadir and innodb_data_home_dir all files except ibdata1 is created in datadir and ibdata1 is created in innodb_data_home_dir. Server starts perfectly fine. No issue on that front.

----------------

Case-2:

5.6.26 (latest one from trunk)
2.3.2 (Xtrabackup)

What I observed

* on node#2 with datadir and innodb_data_home_dir all files are created in datadir and so when server try to consult innodb_data_home_dir for the said ibdata1 file it doesn't find it and so it trigger creation of new ibdata1 and then eventually fails.
(Creation of new ibdata1 files set the mode of create_new_db which then also try to create iblogfiles* but iblogfiles* are created in datadir where they already exist and so finally server then aborts with error).

Revision history for this message
Krunal Bauskar (krunal-bauskar) wrote :

As confirmed this is a bug in xtrabackup. Check here: https://bugs.launchpad.net/percona-xtrabackup/+bug/1512616 and fix for it is committed here https://github.com/gl-sergei/percona-xtrabackup/commit/b7429736278e92f4e6d1bc370ca2d968b4811ae0. It should be part of next xtrabackup release.

Revision history for this message
Krunal Bauskar (krunal-bauskar) wrote :

Original bug was fixed. This bug was meant to track it so will close this bug on PXC front.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1277

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.