Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC

SST on IPv6 fails for xtrabackup-v2

  1. Series 5.5
  2. Bug #1380747
Bug #1380747 reported by Mario Splivalo
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Status tracked in 5.6
5.5
Confirmed
Undecided
Unassigned
5.6
Fix Released
Undecided
Unassigned
percona-cluster (Juju Charms Collection)
Fix Released
Undecided
Unassigned
percona-xtradb-cluster-5.5 (Ubuntu)
Invalid
Medium
Unassigned

Bug Description

I have three nodes configured on a ipv4+ipv6 mixed network. Both of my nodes have ipv4 and ipv6 addresses, however I have configured percona to use only ipv6. I am able to connect to 3306 port from/to each of the machines using only ipv6.

Due to issues with having IPv6 addresses in the configuration file I added the hostnames of the participating nodes in /etc/hosts file, with accompanying IPv6 addresses so that I'm sure that all the traffic goes over IPv6.

However, joiners can't receive data from donor because 'socat' on joiner is only listeninig on ipv4:

mysql 20553 20394 0 18:17 pts/1 00:00:00 socat -u TCP-LISTEN:4444,reuseaddr stdio

If, using the hostname, I try to connect, from donnor node to joiner node, I can't connect:

root@machine-1:/var/lib/mysql# telnet machine-2 4444
Trying 2001:db8:1:0:f816:3eff:fea4:2dd8...
telnet: Unable to connect to remote host: Connection refused

When I use ipv4 address, all is good:

root@machine-1:/var/lib/mysql# host machine-2
machine-2.openstacklocal has address 10.5.19.139
root@machine-1:/var/lib/mysql# telnet 10.5.19.139 4444
Trying 10.5.19.139...
Connected to 10.5.19.139.
Escape character is '^]'.

Of course, when I force socat to use TCP6-LISTEN (by hacking into the /usr/bin/wsrep_sst_xtrabackup-v2) SST finishes fine.

(As a workaround, when using rsync as a wsrep_sst_method, I am able to complete the SST, regardless of the issue mentioned in bug #1315528).

See original description
Mario Splivalo (mariosplivalo)
description: updated
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

As a better workaround I created a new SST plugin - called it xtrabackup-v2-ipv6. I merely copied /usr/bin/wsrep_sst_xtrabackup-v2 to /usr/bin/wsrep_sst_xtrabackup-v2-ipv6, and altered it so that socat always uses TCP6-LISTEN. When that new plugin is deployed across all the cluster members I have SST using xtrabackup over IPv6 working.

Revision history for this message
Xiang Hui (xianghui) wrote :

The TCP-LISTEN is not used only for ipv4, it could be set with export "SOCAT_DEFAULT_LISTEN_IP={4,6}" for socat to listening on different ip protocol, so no need to create a new wsrep_sst_xtrabackup-v2-ipv6 plugin.

xianghui@Thinkpad-x240:~/workplace/github$ export SOCAT_DEFAULT_LISTEN_IP=6
xianghui@Thinkpad-x240:~/workplace/github$ socat -u TCP-LISTEN:4444,reuseaddr stdio

xianghui@Thinkpad-x240:~/workplace/github$ telnet :: 4444
Trying ::...
Connected to ::.
Escape character is '^]'.

Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

Actually, imho, the best fix for this would be for the percona-cluster init script to honor /etc/default - one could put all the required stuff inside /etc/default/percona thus eliminating the need of fiddling with init scripts or provided wsrep plugins.

Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

I opened bug #1381492, regarding /etc/default issue.

Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

And, here is even better fix - one just has to read documentation thoroughly :)

All the wsrep_* plugins should honor the [sst] section inside my.cnf configuration file (as per this: http://www.percona.com/doc/percona-xtradb-cluster/5.5/manual/xtrabackup_sst.html#sockopt)

So, this issue is easily fixed by adding this snippet to my.cnf:

[sst]
sockopt=,pf=ip6

Revision history for this message
Miguel Angel Nieto (miguelangelnieto) wrote :

You are right Mario, that's the correct way to change IP protocol for SST. This is not a bug but I still think it worth adding your example to the documentation. It will help people to understand how to use that parameter and also help the users using ipv6 (the adoption is increasing).

I'm going to confirm the bug, but only as a documentation bug in order to improve it a bit more.

tags: added: documentation
Changed in percona-xtradb-cluster-5.5 (Ubuntu):
status: New → Confirmed
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

@Mario, @Xiang,

Yes, /etc/default/mysql should be honored on debian now, so you can do like SOCAT_DEFAULT_LISTEN_IP=6 in that file.

James Page (james-page)
Changed in percona-xtradb-cluster-5.5 (Ubuntu):
importance: Undecided → Medium
Edward Hope-Morley (hopem)
Changed in percona-cluster (Juju Charms Collection):
status: New → Fix Released
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

To clear some possible confusion - this bag is, actually, invalid. When I first opened it i wasn't aware of [sst] option in the config file for percona-xtradb-cluster.

So, to make xtrabackup-v2 plugin actually use ipv6 when doing sst, this has to be added to my.cnf:

[sst]
sockopt=,pf=ip6

So, this is actually a documentation issue, as tagged by Miguel.

James Page (james-page)
Changed in percona-xtradb-cluster-5.5 (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1753

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.