innobackupex should not allow --no-lock --galera-info

Bug #1047886 reported by Jay Janssen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Triaged
Low
Unassigned
2.0
Won't Fix
Low
Unassigned
2.1
Triaged
Low
Unassigned
2.2
Triaged
Low
Unassigned
2.3
Triaged
Low
Unassigned

Bug Description

5.5.27.

If, on a DONOR node, I add --no_lock to /usr/bin/wsrep_sst_xtrabackup INNOBACKUPEX_ARGS like this:

INNOBACKUPEX_ARGS="--no-lock"

Then SST fails on the JOINER node with this error:

120908 17:31:44 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup 'joiner' '192.168.70.4' '' '/var/lib/mysql/' '/etc/my.cnf' '27134' 2>sst.err: 32 (Broken pipe)
120908 17:31:44 [ERROR] WSREP: Failed to parse uuid:seqno pair: 'xtrabackup process ended without creating '/var/lib/mysql//xtrabackup_galera_info''
120908 17:31:44 [ERROR] WSREP: SST failed: 22 (Invalid argument)
120908 17:31:44 [ERROR] Aborting

There is no content in the sst.err on the JOINER.

I know adding this argument worked correctly in 5.5.24, but seems broken in 5.5.27

Tags: pxc
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Confirmed.

Now, this fails because,

from innobackupex:
===========
    if (!$option_no_lock) {
        # make a prep copy before locking tables, if using rsync
        backup_files(1);

        # flush tables with read lock
        mysql_lockall();
    }
=======

and write_galera_info is called from mysql_lockall

write_galera_info mainly does:

    # get binlog position
    mysql_send "SHOW STATUS LIKE 'wsrep_local_state_uuid';";
    mysql_send "SHOW STATUS LIKE 'wsrep_last_committed';";

Now, it needs to checked if above does require FTWRL or not.

Changed in percona-xtradb-cluster:
status: New → Confirmed
Revision history for this message
Vadim Tkachenko (vadim-tk) wrote :

we need to have FTWRL to get consistent Galera position.
--no-lock can not be used for SST.

Later when Galera supports lockless snapshot we can handle it differently.

I am setting bug as "Invalid"

Changed in percona-xtradb-cluster:
status: Confirmed → Invalid
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

During a SST from the donor node, writes are disabled, so it is interesting to see if wsrep_local_state_uuid and wsrep_last_committed change during SST or not (my guess is they do not).

Also, adding Xtrabackup project here since changes, if any, are required in that.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

@Vadim, Thanks, so locking is required then.

However, if both --no-lock and --galera-info are given, innobackupex code should fail or atleast print a warning to the error log rather than silently fail with broken pipe as in case of PXC.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Currently, there is nothing in donor innobackup.backup.log about this and joiner error log only has:

120927 22:20:25 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup 'joiner' '10.0.2.154' '' '/var/lib/mysql/' '/etc/my.cnf' '3629' 2>sst.err: 32 (Broken pipe)
120927 22:20:25 [ERROR] WSREP: Failed to parse uuid:seqno pair: 'xtrabackup process ended without creating '/var/lib/mysql//xtrabackup_galera_info''

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Actually, Galera, atleast now, supports wsrep replication position in InnoDB headers, much like how the replication position is stored and recovered during prepare phase of Xtrabackup.

According to discussion I had, it stored as follows:

""
Wsrep GTID is found from InnoDB sys_header + TRX_SYS_WSREP_XID_INFO, see:
* storage/innobase/include/trx0sys.h for TRX_SYS_WSREP_XID_INFO and other definitions
* storage/innobase/trx/trx0sys.c:trx_sys_read_wsrep_checkpoint() for reading XID from sys_header
* sql/wsrep_mysqld.h and sql/wsrep_utils.cc how to read GTID from XID (wsrep_xid_{uuid|seqno})
* XID is defined in sql/handler.h and storage/innobase/include/trx0xa.h
""

May be a different bug can be opened for this if required.

Changed in percona-xtradb-cluster:
status: Invalid → New
no longer affects: percona-xtradb-cluster
summary: - Adding --no-lock in wsrep_sst_xtrabackup causes SST to fail
+ innobackupex should not allow --no-lock --galera-info
Changed in percona-xtrabackup:
status: New → Triaged
importance: Undecided → Low
tags: added: pxc
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-83

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.