MySQL patches by Codership

Issues with re-setting the provider

Bug #1260283 reported by Raghavendra D Prabhu
14
This bug affects 4 people
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Status tracked in 5.6
5.5
Fix Committed
Undecided
Unassigned
MySQL patches by Codership 5.5.38-25.11
5.6
Fix Released
Undecided
Yan Zhang
MySQL patches by Codership 5.6.21-25.9
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Status tracked in 5.6
5.5
Fix Released
High
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC 5.5.39-25.11
5.6
Fix Released
High
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC 5.6.19-25.6

Bug Description

There are few issues with setting provider from galera to none and back to galera.

Details here https://bugs.launchpad.net/codership-mysql/+bug/1208493/comments/12 and https://bugs.launchpad.net/codership-mysql/+bug/1208493/comments/13

a) Bootstrapping doesn't work.

""
SET GLOBAL wsrep_provider_options="pc.bootstrap=1";
ERROR 1210 (HY000): Incorrect arguments to SET
""

b) Incomplete restoration of provider config. evs, pc etc. are
missing from wsrep_provider_options after restoration.

It may be possible to provide wsrep_provider_options again but

i) Not all options are dynamic, they will need to be set before
provider starts.

ii) Few components fail while setting (like the pc.bootstrap
above).

c) Incorrect restoration of values. The values restored differ
from values set in my.cnf for same key/value.

=================================================================================

131212 15:51:48 [Note] WSREP: wsrep_provider_update: /pxc56/lib/libgalera_smm.so
131212 15:51:48 [Note] WSREP: Stop replication
131212 15:51:48 [Note] WSREP: Provider disconnect
131212 15:51:50 [Note] WSREP: waiting for client connections to close: 1
131212 15:51:50 [Note] WSREP: Setting wsrep_ready to 0
131212 15:51:50 [Note] WSREP: Read WSREPXid from InnoDB: 5805bbc4-3038-11e3-937d-aa946afe7370:15933
131212 15:51:50 [Note] WSREP: Initial position: 5805bbc4-3038-11e3-937d-aa946afe7370:15933
131212 15:51:50 [Note] WSREP: wsrep_load(): loading provider library '/pxc56/lib/libgalera_smm.so'
131212 15:51:50 [Note] WSREP: wsrep_load(): Galera 3.2(rXXXX) by Codership Oy <email address hidden> loaded successfully.
131212 15:51:50 [Note] WSREP: CRC-32C: using hardware acceleration.
131212 15:51:50 [Note] WSREP: Found saved state: 5805bbc4-3038-11e3-937d-aa946afe7370:15933
131212 15:51:50 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; gcache.dir = /pxc/datadir/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /pxc/datadir//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.proto_max = 5
131212 15:51:50 [Note] WSREP: Assign initial position for certification: 15933, protocol version: -1
131212 15:51:50 [Note] WSREP: wsrep_cluster_address_init: null -> null
131212 15:51:50 [Note] WSREP: refresh_provider_options: null
131212 15:53:20 [Warning] WSREP: Unknown parameter 'pc.bootstrap'
131212 15:53:20 [ERROR] WSREP: Set options returned 7
131212 15:53:20 [Note] WSREP: refresh_provider_options: pc.bootstrap=1;
131212 15:53:22 [Warning] WSREP: Unknown parameter 'pc.bootstrap'
131212 15:53:22 [ERROR] WSREP: Set options returned 7
131212 15:53:22 [Note] WSREP: refresh_provider_options: pc.bootstrap=1
131212 15:53:41 [Warning] WSREP: Unknown parameter 'pc.bootstrap'
131212 15:53:41 [ERROR] WSREP: Set options returned 7
131212 15:53:41 [Note] WSREP: refresh_provider_options: pc.bootstrap=1
131212 15:54:23 [Warning] WSREP: Unknown parameter 'pc.bootstrap'
131212 15:54:23 [ERROR] WSREP: Set options returned 7
131212 15:54:23 [Note] WSREP: refresh_provider_options: pc.bootstrap=1
=========================================================================================

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Raghu, wsrep_provider_options loss is expected (although inconvenient). I have reported a more targeted ticket about it here: https://bugs.launchpad.net/codership-mysql/5.6/+bug/1260290

Come to think of it, the fix to it could also fix this one.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

@Alex,

a)
The loss of configuration is one of the issues.

What about the other one -

""
131212 15:53:20 [Warning] WSREP: Unknown parameter 'pc.bootstrap'
131212 15:53:20 [ERROR] WSREP: Set options returned 7
131212 15:53:20 [Note] WSREP: refresh_provider_options: pc.bootstrap=1;
""

This suggests non-initialization of pc component or failure to
set it.

b) Next, when you say set it on command line, do you want the
entire wsrep_provider_options or just the ones set in my.cnf (so
others are initialised to compiled defaults) to be specified as
set global wsrep_provider_options=''. I guess the latter?

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Raghu,
a) setting pc.bootstrap is meaningful only after connecting to a non-primary component. I.e. you need to set wsrep_cluster_address first (plus necessary wsrep_provider_options).
b) yes, just what is different from defaults.

Revision history for this message
Yan Zhang (yan.zhang) wrote :

in general, the process of updating a provider is
1. reload provider (previous wsrep_provider_options value is gone. use built-in value in galera code)
2. set wsrep_provider_options
3. set wsrep_clutser_address (connect to cluster and then start replication)

Most parameters in wsrep_provder_options should be set *before* connecting to cluster, like gcache.*, repl.* etc. However some parameters could only be set *after* connecting to cluster. AFAIK, these parameters include
a. protonet.*
b. socket.*
c. gmcast.*
d. evs.*
e. pc.*

So if you are gonna to set pc.bootstrap=1, you can only do it after you set wsrep_clutser_address .

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

@Yan,

It works but I noticed this:

MySQL [(none)]> set global wsrep_provider='none';
Query OK, 0 rows affected (2.02 sec)

MySQL [(none)]> set global wsrep_provider='/usr/lib64/libgalera_smm.so';
Query OK, 0 rows affected (2.03 sec)

MySQL [(none)]> set global wsrep_provider_options="gmcast.listen_addr=tcp://127.0.0.1:4010"; -- Fails due to no cluster address, this is OK
ERROR 1210 (HY000): Incorrect arguments to SET
MySQL [(none)]>
MySQL [(none)]> set global wsrep_cluster_address='gcomm://127.0.0.1:5010,127.0.0.1:6010'; --- Fails (in log), but lets ignore that for now.
Query OK, 0 rows affected (35.02 sec)

MySQL [(none)]> set global wsrep_cluster_address='gcomm://';
Query OK, 0 rows affected (2.00 sec)

MySQL [(none)]> set global wsrep_provider_options="gmcast.listen_addr=tcp://127.0.0.1:4010"; ------ This fails.
ERROR 1210 (HY000): Incorrect arguments to SET

It fails with:
================

2014-04-30 10:04:30 13562 [Note] WSREP: Nobody is waiting for SST.
2014-04-30 10:04:34 13562 [Warning] WSREP: error setting param gmcast.listen_addr to value tcp://127.0.0.1:4010: can't change value for 'gmcast.listen_addr' during runtime: 1 (Operation not permitted)
         at gcomm/src/gmcast.cpp:set_param():1640
2014-04-30 10:04:34 13562 [Warning] WSREP: Setting parameter 'gmcast.listen_addr' to 'tcp://127.0.0.1:4010' failed: Setting 'gmcast.listen_addr' to 'tcp://127.0.0.1:4010' failed: 1 (Operation not permitted)
         at galera/src/gcs.hpp:param_set():206
2014-04-30 10:04:34 13562 [ERROR] WSREP: Set options returned 7
2014-04-30 10:04:34 13562 [Note] WSREP: refresh_provider_options: gmcast.listen_addr=tcp://127.0.0.1:4010

The problem is that, without ability to set listen_addr before it
actually listens, it ends up listening on default addr:port.

wsrep_provider_options - http://paste.wnohang.net/5368df

Revision history for this message
Yan Zhang (yan.zhang) wrote :

@Raghu

thx for reporting this problem(or bug). And I think the main reason is that bad design of "process of updating provider".

I'll give some thought about the process.

Yan Zhang (yan.zhang)
Changed in codership-mysql:
status: New → In Progress
assignee: nobody → Yan Zhang (yan.zhang)
Revision history for this message
Yan Zhang (yan.zhang) wrote :

@Raghu

galera provider has three types of options
1. only static (should be set before provider is loaded, can not be changed in runtime) like gmcast.listen_addr, gcache.dir and repl.commit_order (as you specified in https://bugs.launchpad.net/codership-mysql/+bug/1208493/comments/13)
2. static/runtime like gcs.fc_limit, pc.ignore_sb
3. only runtime like gmcast.peer_addr, pc.bootstrap

only type 1,2 options will be preserved when provider is reloaded.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-986

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.