Wrong node-discover accessibility check logic

agree with additional checks in the nailgun-agent.cron

Also, problem should be investigated for already deployed system - looks like they also affected

they can't be affected because status has changed from 'discovered' to something else

Nope, they can :
Node discovered => provisioned (not deployed) => nailgun-agent ran, but mcollective still in progress.

From my point of view this piece of logic should be moved into nailgun agent code.
btw. we already have mcollective related code into nailgun agent code.

This bug cannot be tech-debt and critical. It must be one or the other.

@Slava, can You propose a solution in the nailgun?

@Maksim, check status of mcollective service more correctly than https://github.com/openstack/fuel-nailgun-agent/blob/master/agent#L120

This bug is the root of randomly failed BVT tests, it is marked as Critical for MOS 8.0

@TImur, could you attach bvt job example which you mentioned ?

Related:
https://bugs.launchpad.net/fuel/+bug/1538055/comments/8 =>>>
https://bugs.launchpad.net/fuel/+bug/1518306/comments/12

I'm against fixing it in a week before HCF. Let's not backport it.

And I don't think that creating of one more workaround around another workaround is a good idea. Workflow needs to be redesigned. Mcollective config is updated by at least three tools. Obviously there is a space for improvement here.

@DPyznov:
its critical issue, and should be fixed in 8.0 also - otherwise we will always catch random problems with tests.

use in nailgun-agent a kind of the following check:
def get_mco_status
  Process.kill 0, %x(status mcollective).split.last.to_i
rescue
 0
end

# returns 1 if process found and active, otherwise 0

We can check if mcollective is available without any additional tools and patches just by inserting a one-line condition into nailgun-agent.cron:

# timeout 5 ruby -rmcollective -e 'include MCollective::RPC; rpcclient("version", :options => { :config => "/etc/mcollective/server.cfg" }).get_version'
# echo $?
0
(now setting invalid configuration in /etc/mcollective/server.cfg)
# timeout 5 ruby -rmcollective -e 'include MCollective::RPC; rpcclient("version", :options => { :config => "/etc/mcollective/server.cfg" }).get_version'
Could not create RPC client: Could not connect to RabbitMQ Server: SIGTERM
# echo $?
124

@Nastya, it looks like my mistake, the BVT on #484 failed because of other issue, https://bugs.launchpad.net/fuel/+bug/1533082

Guys, nailgun agent creates a record in nailgun, takes id on the node, then updates mcollective config and restarts mcollective. Your extra check will ruin the workflow. Please update tests, add a wait till 'mco ping' returns all nodes.

@Dmitry, and what is about users? Should we add this ....issue into the release notes?

@Dmitry, fix fuel-qa didn't fix deployment process
(https://bugs.launchpad.net/fuel/+bug/1538055/comments/4)
Which actually (my assume) root cause of other problems (like ntpd granular timeout)

@slava, @Aleksey, could you provide use case that affects users? The only case that I see is when a user wants to deploy a cluster automatically as soon as all nodes are available. I guess we can live with workaround for this case until we redesign our flow.

Technical debt, doesn't affect real users. Removing from Mitaka release.