Deployment of third controller fails in HA

Ryan, we need description of your environment.

This environment is still running if somebody wants access to it. The problem has been 100% reproducible.

on node-3, diag snapshot from BVT (attached)
root 11284 0.3 0.1 574660 3596 ? Ssl 02:17 0:04 /usr/sbin/corosync
107 11305 41.1 0.2 97864 5048 ? S 02:17 7:26 \_ /usr/lib/pacemaker/cib

41% is a lot. It might be a root cause of MCAgent connectivity loss in BVT: http://jenkins-product.srt.mirantis.net:8080/job/fuel_master.ubuntu.bvt_2/212/.

Log details of MCAgent connectivity loss could be found here: https://bugs.launchpad.net/fuel/+bug/1350016/comments/2

According to the logs, galera cluster issues were in between

2014-07-30T03:16:02.216507 node-1 ./node-1.test.domain.local/puppet-apply.log:2014-07-30T03:16:02.216507+01:00 notice: Finished catalog run in 896.86 seconds

and

2014-07-30T03:32:25.211335 node-4 ./node-4.test.domain.local/puppet-apply.log:2014-07-30T03:32:25.211335+01:00 notice: Finished catalog run in 961.96 seconds

There was a rabbitmq cluster issues also
2014-07-30T03:25:46.822889 node-4 ./node-4.test.domain.local/rabbitmq-server.log:2014-07-30T03:25:46.822889+01:00 err: ERROR: Resetting node 'rabbit
@node-4' ... Error: {corrupt_or_missing_cluster_files,{error,enoent},{error,enoent}}
2014-07-30T03:25:46.824226 node-4 ./node-4.test.domain.local/rabbitmq-server.log:2014-07-30T03:25:46.824226+01:00 err: ERROR: Forcefully resetting n
ode 'rabbit@node-4' ... Error: {corrupt_or_missing_cluster_files,{error,enoent},{error,enoent}}
2014-07-30T03:25:46.824226 node-4 ./node-4.test.domain.local/rabbitmq-server.log:2014-07-30T03:25:46.824226+01:00 err: ERROR: Mnesia couldn't cleane
d, even by force-reset command.

rca looks like a c nnectivity issues at corosync layer of logic due to its high cpu load?
Here could be a good job for fencing :)

wsrep slave threads related as well

I'm not sure this is a duplicate of #1350245. The logs provided by Mike don't contain the galera failures and the logs from 1350245 don't contain the kernel messages about the hung tasks we saw here. I will test an ISO with the wsrep changes and see if it fixes this issue as well.

guys. failing deployment of the third controller is too abstract as a bug. hung mysqld task in kernel logs says there is a really big load that blocked mysqld for execution for 120 seconds. most of the time it is related to I/O system performance. Also, let's decide whether this bug is a duplicate of 1350245 or not. If galera is being stopped by OCF script or load is spiking right at the time when galera starts full state transfer, then it is a duplicate. Otherwise it is an environmental issue which I could not reproduce.