Log from SST of node2: 130925 18:57:38 [Note] WSREP: State transfer required: Group state: 4823adcb-2613-11e3-85b0-d2640a4c137f:1 Local state: 00000000-0000-0000-0000-000000000000:-1 130925 18:57:38 [Note] WSREP: New cluster view: global state: 4823adcb-2613-11e3-85b0-d2640a4c137f:1, view# 4: Primary, number of nodes: 2, my index: 1, protocol version 2 130925 18:57:38 [Warning] WSREP: Gap in state sequence. Need state transfer. 130925 18:57:40 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address 'ec2-50-17-114-69.compute-1.amazonaws.com' --auth 'sst:secret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '10758'' WSREP_SST: [INFO] Streaming with xbstream (20130925 18:57:40.811) WSREP_SST: [INFO] Using socat as streamer (20130925 18:57:40.815) WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20130925 18:57:40.839) 130925 18:57:41 [Note] WSREP: Prepared SST request: xtrabackup|ec2-50-17-114-69.compute-1.amazonaws.com:4444/xtrabackup_sst 130925 18:57:41 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130925 18:57:41 [Note] WSREP: Assign initial position for certification: 1, protocol version: 2 130925 18:57:41 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (4823adcb-2613-11e3-85b0-d2640a4c137f): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():436. IST will be unavailable. 130925 18:57:41 [Note] WSREP: Node 1 (ip-10-149-26-184) requested state transfer from '*any*'. Selected 0 (ip-10-238-195-117)(SYNCED) as donor. 130925 18:57:41 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1) 130925 18:57:41 [Note] WSREP: Requesting state transfer: success, donor: 0 WSREP_SST: [INFO] Proceeding with SST (20130925 18:57:41.263) WSREP_SST: [INFO] Cleaning the existing datadir (20130925 18:57:41.268) removed `/var/lib/mysql/error.log' removed `/var/lib/mysql/mysql/db.frm' removed `/var/lib/mysql/mysql/db.MYI' removed `/var/lib/mysql/mysql/db.MYD' removed `/var/lib/mysql/mysql/host.frm' removed `/var/lib/mysql/mysql/host.MYI' removed `/var/lib/mysql/mysql/host.MYD' removed `/var/lib/mysql/mysql/user.frm' removed `/var/lib/mysql/mysql/user.MYI' removed `/var/lib/mysql/mysql/user.MYD' removed `/var/lib/mysql/mysql/func.frm' removed `/var/lib/mysql/mysql/func.MYI' removed `/var/lib/mysql/mysql/func.MYD' removed `/var/lib/mysql/mysql/plugin.frm' removed `/var/lib/mysql/mysql/plugin.MYI' removed `/var/lib/mysql/mysql/plugin.MYD' removed `/var/lib/mysql/mysql/servers.frm' removed `/var/lib/mysql/mysql/servers.MYI' removed `/var/lib/mysql/mysql/servers.MYD' removed `/var/lib/mysql/mysql/tables_priv.frm' removed `/var/lib/mysql/mysql/tables_priv.MYI' removed `/var/lib/mysql/mysql/tables_priv.MYD' removed `/var/lib/mysql/mysql/columns_priv.frm' removed `/var/lib/mysql/mysql/columns_priv.MYI' removed `/var/lib/mysql/mysql/columns_priv.MYD' removed `/var/lib/mysql/mysql/help_topic.frm' removed `/var/lib/mysql/mysql/help_topic.MYI' removed `/var/lib/mysql/mysql/help_topic.MYD' removed `/var/lib/mysql/mysql/help_category.frm' removed `/var/lib/mysql/mysql/help_category.MYI' removed `/var/lib/mysql/mysql/help_category.MYD' removed `/var/lib/mysql/mysql/help_relation.frm' removed `/var/lib/mysql/mysql/help_relation.MYI' removed `/var/lib/mysql/mysql/help_relation.MYD' removed `/var/lib/mysql/mysql/help_keyword.frm' removed `/var/lib/mysql/mysql/help_keyword.MYI' removed `/var/lib/mysql/mysql/help_keyword.MYD' removed `/var/lib/mysql/mysql/time_zone_name.frm' removed `/var/lib/mysql/mysql/time_zone_name.MYI' removed `/var/lib/mysql/mysql/time_zone_name.MYD' removed `/var/lib/mysql/mysql/time_zone.frm' removed `/var/lib/mysql/mysql/time_zone.MYI' removed `/var/lib/mysql/mysql/time_zone.MYD' removed `/var/lib/mysql/mysql/time_zone_transition.frm' removed `/var/lib/mysql/mysql/time_zone_transition.MYI' removed `/var/lib/mysql/mysql/time_zone_transition.MYD' removed `/var/lib/mysql/mysql/time_zone_transition_type.frm' removed `/var/lib/mysql/mysql/time_zone_transition_type.MYI' removed `/var/lib/mysql/mysql/time_zone_transition_type.MYD' removed `/var/lib/mysql/mysql/time_zone_leap_second.frm' removed `/var/lib/mysql/mysql/time_zone_leap_second.MYI' removed `/var/lib/mysql/mysql/time_zone_leap_second.MYD' removed `/var/lib/mysql/mysql/proc.frm' removed `/var/lib/mysql/mysql/proc.MYI' removed `/var/lib/mysql/mysql/proc.MYD' removed `/var/lib/mysql/mysql/procs_priv.frm' removed `/var/lib/mysql/mysql/procs_priv.MYI' removed `/var/lib/mysql/mysql/procs_priv.MYD' removed `/var/lib/mysql/mysql/general_log.frm' removed `/var/lib/mysql/mysql/general_log.CSM' removed `/var/lib/mysql/mysql/general_log.CSV' removed `/var/lib/mysql/mysql/slow_log.frm' removed `/var/lib/mysql/mysql/slow_log.CSM' removed `/var/lib/mysql/mysql/slow_log.CSV' removed `/var/lib/mysql/mysql/event.frm' removed `/var/lib/mysql/mysql/event.MYI' removed `/var/lib/mysql/mysql/event.MYD' removed `/var/lib/mysql/mysql/ndb_binlog_index.frm' removed `/var/lib/mysql/mysql/ndb_binlog_index.MYI' removed `/var/lib/mysql/mysql/ndb_binlog_index.MYD' removed `/var/lib/mysql/mysql/proxies_priv.frm' removed `/var/lib/mysql/mysql/proxies_priv.MYI' removed `/var/lib/mysql/mysql/proxies_priv.MYD' removed directory: `/var/lib/mysql/mysql' removed directory: `/var/lib/mysql/test' removed `/var/lib/mysql/performance_schema/db.opt' removed `/var/lib/mysql/performance_schema/cond_instances.frm' removed `/var/lib/mysql/performance_schema/events_waits_current.frm' removed `/var/lib/mysql/performance_schema/events_waits_history.frm' removed `/var/lib/mysql/performance_schema/events_waits_history_long.frm' removed `/var/lib/mysql/performance_schema/events_waits_summary_by_instance.frm' removed `/var/lib/mysql/performance_schema/events_waits_summary_by_thread_by_event_name.frm' removed `/var/lib/mysql/performance_schema/events_waits_summary_global_by_event_name.frm' removed `/var/lib/mysql/performance_schema/file_instances.frm' removed `/var/lib/mysql/performance_schema/file_summary_by_event_name.frm' removed `/var/lib/mysql/performance_schema/file_summary_by_instance.frm' removed `/var/lib/mysql/performance_schema/mutex_instances.frm' removed `/var/lib/mysql/performance_schema/performance_timers.frm' removed `/var/lib/mysql/performance_schema/rwlock_instances.frm' removed `/var/lib/mysql/performance_schema/setup_consumers.frm' removed `/var/lib/mysql/performance_schema/setup_instruments.frm' removed `/var/lib/mysql/performance_schema/setup_timers.frm' removed `/var/lib/mysql/performance_schema/threads.frm' removed directory: `/var/lib/mysql/performance_schema' removed `/var/lib/mysql/ibdata1' removed `/var/lib/mysql/ib_logfile0' removed `/var/lib/mysql/ib_logfile1' removed `/var/lib/mysql/mysql.sock' WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20130925 18:57:41.624) 130925 18:57:55 [Note] WSREP: 0 (ip-10-238-195-117): State transfer to 1 (ip-10-149-26-184) complete. 130925 18:57:55 [Note] WSREP: Member 0 (ip-10-238-195-117) synced with group. WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql/ (20130925 18:57:55.456) WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/my.cnf --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20130925 18:57:55.460) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20130925 18:58:57.144) WSREP_SST: [INFO] Removing the sst_in_progress file (20130925 18:58:57.149) 130925 18:58:57 [Note] WSREP: SST complete, seqno: 1 130925 18:58:57 [Note] Plugin 'FEDERATED' is disabled. 130925 18:58:57 InnoDB: The InnoDB memory heap is disabled 130925 18:58:57 InnoDB: Mutexes and rw_locks use GCC atomic builtins 130925 18:58:57 InnoDB: Compressed tables use zlib 1.2.3 130925 18:58:57 InnoDB: Using Linux native AIO 130925 18:58:57 InnoDB: Initializing buffer pool, size = 10.0G 130925 18:58:58 InnoDB: Completed initialization of buffer pool 130925 18:58:58 InnoDB: highest supported file format is Barracuda. 130925 18:58:59 InnoDB: Waiting for the background threads to start 130925 18:59:00 Percona XtraDB (http://www.percona.com) 5.5.33-rel31.0 started; log sequence number 1598476 130925 18:59:00 [Note] Event Scheduler: Loaded 0 events 130925 18:59:00 [Note] WSREP: Signalling provider to continue. 130925 18:59:00 [Note] WSREP: SST received: 4823adcb-2613-11e3-85b0-d2640a4c137f:1 130925 18:59:00 [Note] WSREP: 1 (ip-10-149-26-184): State transfer from 0 (ip-10-238-195-117) complete. 130925 18:59:00 [Note] WSREP: Shifting JOINER -> JOINED (TO: 1) 130925 18:59:00 [Note] WSREP: Member 1 (ip-10-149-26-184) synced with group. 130925 18:59:00 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 1) 130925 18:59:00 [Note] WSREP: Synchronized with group, ready for connections 130925 18:59:00 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130925 18:59:00 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.33' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona XtraDB Cluster (GPL), wsrep_23.7.6.r3915