controller node couldn't reach rabbitmq of fuel node during deployment

rabbitmq is seem ok:

[root@fuel ~]# dockerctl shell rabbitmq
[root@ee1f879b6011 ~]# rabbitmqctl status
Status of node rabbit@ee1f879b6011 ...
[{pid,1510},
 {running_applications,
     [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.3.5"},
      {rabbitmq_management,"RabbitMQ Management Console","3.3.5"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.5"},
      {rabbit,"RabbitMQ","3.3.5"},
      {os_mon,"CPO CXC 138 46","2.2.7"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.5"},
      {webmachine,"webmachine","1.10.3-rmq3.3.5-gite9359c7"},
      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.5-git680dba8"},
      {amqp_client,"RabbitMQ AMQP Client","3.3.5"},
      {xmerl,"XML parser","1.2.10"},
      {inets,"INETS CXC 138 49","5.7.1"},
      {mnesia,"MNESIA CXC 138 12","4.5"},
      {sasl,"SASL CXC 138 11","2.1.10"},
      {stdlib,"ERTS CXC 138 10","1.17.5"},
      {kernel,"ERTS CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:6:6] [rq:6] [async-threads:30] [kernel-poll:true]\n"},
 {memory,
     [{total,208516560},
      {connection_procs,8229392},
      {queue_procs,34821296},
      {plugins,10775712},
      {other_proc,17252640},
      {mnesia,5254824},
      {mgmt_db,15881128},
      {msg_index,2102624},
      {other_ets,3191376},
      {binary,20071880},
      {code,17784544},
      {atom,1625361},
      {other_system,71525783}]},
 {alarms,[]},
 {listeners,[{clustering,41055,"::"},{amqp,5672,"::"},{stomp,61613,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,3355982233},
 {disk_free_limit,50000000},
 {disk_free,9421512704},
 {file_descriptors,
     [{total_limit,102300},
      {total_used,217},
      {sockets_limit,92068},
      {sockets_used,214}]},
 {processes,[{limit,1048576},{used,6397}]},
 {run_queue,0},
 {uptime,64870}]
...done.

[root@ee1f879b6011 ~]# rabbitmqctl list_queues
Listing queues ...
amq.gen-2wXBIlw1FIzdmdxzF-sG3Q 0
amq.gen-9eKeOiM17ib56w8WpHxvZw 0
amq.gen-G_HXOSSZYDDM4NSBAsBPUg 0
amq.gen-TIXr-6FXTbBHboSv9e4frw 0
amq.gen-pONboF2R3fXeOOTTfkYsBw 0
amq.gen-yKvDYTBIbt0XbrRrtObLsQ 0
amq.gen-ym_nLWc_QnljJYUqqsAJ0A 0
nailgun 0
naily 0
...done.

What's the status of node-1?

From mcollecitve log on node-1:

2015-03-11T20:30:42.657620+00:00 debug: 20:30:42.737132 #8391] DEBUG -- : rabbitmq.rb:225:in `receive' Waiting for a message from RabbitMQ
2015-03-11T20:30:42.657878+00:00 debug: 20:30:42.737740 #8391] DEBUG -- : runnerstats.rb:49:in `received' Incrementing total stat
2015-03-11T20:30:42.657993+00:00 debug: 20:30:42.737853 #8391] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
2015-03-11T20:30:42.658165+00:00 warning: 20:30:42.738090 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: Failed to handle message: TypeError: incompatible marshal file format (can't be read)
2015-03-11T20:30:42.658258+00:00 info: format version 4.8 required; 89.111 given
2015-03-11T20:30:42.658417+00:00 warning: 20:30:42.738166 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/libexec/mcollective/mcollective/security/psk.rb:22:in `load'
2015-03-11T20:30:42.658472+00:00 warning: 20:30:42.738228 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/libexec/mcollective/mcollective/security/psk.rb:22:in `decodemsg'
2015-03-11T20:30:42.658671+00:00 warning: 20:30:42.738287 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/lib/ruby/site_ruby/1.8/mcollective/message.rb:178:in `decode!'
2015-03-11T20:30:42.658759+00:00 warning: 20:30:42.738345 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:121:in `receive'
2015-03-11T20:30:42.658861+00:00 warning: 20:30:42.738402 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:55:in `run'
2015-03-11T20:30:42.659087+00:00 warning: 20:30:42.738461 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:53:in `loop'
2015-03-11T20:30:42.659205+00:00 warning: 20:30:42.738518 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:53:in `run'
2015-03-11T20:30:42.659321+00:00 warning: 20:30:42.738586 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:30:in `daemonize_runner'
2015-03-11T20:30:42.659421+00:00 warning: 20:30:42.738650 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:13:in `daemonize'
2015-03-11T20:30:42.659586+00:00 warning: 20:30:42.738709 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:5:in `fork'
2015-03-11T20:30:42.659692+00:00 warning: 20:30:42.738767 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:5:in `daemonize'
2015-03-11T20:30:42.659785+00:00 warning: 20:30:42.738825 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:20:in `daemonize_runner'
2015-03-11T20:30:42.659969+00:00 warning: 20:30:42.738889 #8391] WARN -- : psk.rb:22:in `decodemsg' PLMC10: /usr/sbin/mcollectived:47

The node-1 in the error state

----|----------|------------------|---------|-------------|-------------------|-------------------|---------------|--------|---------
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
1 | error | Untitled (07:e2) | 2 | 10.20.1.1 | 0c:c4:7a:1f:07:e2 | controller | | True | 2

is accessible

[root@fuel ~]# ssh 10.20.1.1
Warning: Permanently added '10.20.1.1' (RSA) to the list of known hosts.
Last login: Thu Mar 12 18:03:15 2015 from 10.20.0.2
[root@bootstrap ~]#

Main problem in this env on node 1. Different credentials for mcollective.

On node-1

plugin.rabbitmq.pool.1.user = mcollective
plugin.rabbitmq.pool.1.password= FGDVcrol

On other nodes:

plugin.rabbitmq.pool.1.user = mcollective
plugin.rabbitmq.pool.1.password= lXZ1QEWf

Looks like first node was booted on other master node and after it was not rebooted at all.

I simple rebooted node-1 and this node have been got correct config.

mco ping before:
199 replies max: 256.11 min: 71.64 avg: 171.61

after:
200 replies max: 422.21 min: 126.38 avg: 316.06

Here is list of steps to reproduce the bug

1. Fuel-master was set up, lets call it fuel-master-1
2. Fuel-master-1 discovered VMs with all necessary stuff which is needed to work with fuel-master-1 (including credentials to rabbitmq)
3. For some reasons fuel-master-1 was reinstalled into a new fuel-master-2 (which have generated new credentials to rabbitmq).
4. VMs that were discovered from fuel-master-1, connected to fuel-master-2, because fuel-master-2 has same ip addresses using nailgun-agent which doesn't require any authentication.
5. Credentials required to access rabbitmq, which are required for mcollective, mismatched.
6. Deployment fails.

Bug summary is too generic, please rename to narrow down the matching symptoms.

This bug can be worked around by rebooting bootstrap nodes after master node replacement or by fixing mcollective config on already deployed nodes in case of spontaneous mater node replacement.

Related fix proposed to branch: master
Review: https://review.openstack.org/215571

Fix proposed to branch: master
Review: https://review.openstack.org/215572

we need to detect a case when bootstrap nodes connect to incompatible masternode and warn user that he/she should reboot bootstrap nodes. Typically it happens when you upgrade master node and in result you will get a combination of old bootstrap nodes and new master node.

Copy-pasting review comment from Evgeniy L:

"I'm strongly against of rebooting user's machines in the agent, agent's purpose is to only report the data, it mustn't reboot the system, it's too dangerous.

I think it will be more safe to pass master_node_uid in node's metadata, and reject creation of such nodes by Nailgun on API level."

I agree with Evgeniy, it seems to be the most appropriate solution for this case.

And another comment from Evgeniy with more detailed explanation:

"We follow the next approaches depending on the task:

1. ask user to reboot nodes explicitly, we can notify user that there are nodes from another master node and ask him/her to reboot those bootstrap nodes

2. for other cases we do reboot from master node only, only Nailgun knows everything about nodes and in what state they are, so we do reboot in case of provisioning for example. It won't work here, because MCollective is not going to work.

To fix this particular issue, it's better to reject nodes creation on API level and create notification, with explanation, that some nodes, which were booted/installed with another fuel installation, are trying to perform discovery."

Dmitry is no longer working on this. Need a new fuel-python dev to pick this up.

Change abandoned by Dmitry Mescheryakov (dmitryme) (<email address hidden>) on branch: master
Review: https://review.openstack.org/215571
Reason: Not a complete solution, abandoning it so that it does not hang here forever

Change abandoned by Dmitry Mescheryakov (dmitryme) (<email address hidden>) on branch: master
Review: https://review.openstack.org/215572
Reason: Not a complete solution, abandoning it so that it does not hang here forever

We passed SCF in 8.0. Moving the bug to 9.0.