mysqld allows client connections even when in NON_PRIMARY view

Well, currently restarting means the node will loose all the information about cluster topology and won't know which nodes to reconnect to.

So for now the trick with global status variable wsrep_ready and thd->variables.wsrep_on seems to be the only solution.

Plan is to fix this issue by introducing new 'wsrep_ready' status variable.
 wsrep_ready flag will tell, if node is ready to operate in the cluster.
Flag will be initially 0, and is set to 1, when node state has reached the
same level as other cluster nodes. In practice, we set the flag after snapshot
state transfer is complete (or we know that SST is not needed).

When wsrep_ready is 0, the session processing will allow only non-cluster connections
to process (wsrep_on==0). Also SET commands are allowed to make it possible to
declare a connection as non-cluster connection. This SET loophole is needed to let
mysqldump to be loaded from the donor node.

The wsrep_sst_mysqldump script needs to be modified to specify the connection to joiner as wsrep_on==OFF connection. This makes it possible to load the dump, even that wsrep_ready==OFF
in the joiner side.

Assumption is that the donor node is joined to cluster already and mysqldump command from
donor node works with default wsrep_on==ON connection

wsrep_sst_mysqldump already does this:

STOP_WSREP="SET @@wsrep_on=0;"
...
(echo $STOP_WSREP && $MYSQLDUMP && echo $SET_START_POSITION || exit 1) | $MYSQL

The whole wsrep_on thing was invented to prevent state dump from being replicated by joiner.

Not all mysql sessions to the joiner were declared with wsrep_on=OFF, and this caused the issues.
A fix for this has been committed, but in my testing, I have noticed a few cases, where the donor side was not wsrep_ready and SST failed for that.
However, many changes in startup/shutdown logic have happened since then, and I need to re-test with current implementation.

There has been many changes for SST manouvres. Testing with current trunk shows, that
it can happen that mysqld init phase can assume that SST is needed, but gcs layer
decides that node is in fact in sync. This will leave the node closed for external access
(wsrep_ready==OFF).

Here is the log for 2'nd node startup phase:

090812 10:05:12 [Warning] WSREP: Gap in state sequence. Need state transfer.
090812 10:05:12 [Note] /home/galera/mysql-5.1.36-2859/mysql/libexec/mysqld: read
y for connections.
Version: '5.1.36-debug' socket: '/home/galera/mysql-5.1.36-2859/mysql/var/mysql
d.sock' port: 3306 wsrep_0.6
090812 10:05:12 [Warning] WSREP: Group state: abe9d918-1dd1-11b2-0800-c025207f70
fc:0
090812 10:05:12 [Warning] WSREP: Local state: 00000000-0000-0000-0000-0000000000

Latest commit for Bug: #411814, changeset: #2862 propagates updates wsrep_last_committed and wsrep_group_uuid values from wsrep provider library to mysqld and makes sure all logic depending on these variables is now correct.

Testing with trunk version does not reveal problems with startup phase anymore.