Nodes stuck in "provisioning" state despite successful reboot into newly installed OS

Moved back to 6.1. If we want to move it to 7.0, we need to explain why we do not want to fix it in 6.1 and explain the user impact.

The problem has many faces.
1) default mcollective config has daemonize=1 option and our current init script assumes mcollective does not go background. So when SysV first time starts mcollective with default config, the pid becomes wrong (contains wrong process number). Then when cloud-init runs 'service mcollective start', init script thinks process is not running and starts it again but with daemonize=0. We need to fix init script. https://bugs.launchpad.net/fuel/+bug/1454741 But when we fix it, mcollective will not be restarted by cloud-init. Fortunately, nailgun-agent will restart it later.
2) When nailgun-agent starts it checks if identity setting is defined in mcollective config. If it is not defined, nailgun-agent appends it and restarts mcollective. But sometimes it starts at the same time when cloud-init does its magic. So, sometimes we end up with corrupted mcollective config. https://bugs.launchpad.net/fuel/+bug/1449186
3) If init script is correct and nailgun-agent will not restart mcollective, we will end up with mcollective not restarted after its config has been changed by cloud-init. And it won't be able to connect to Rabbit. So we need either to prevent mcollective from starting during provisioning (update-rc.d mcollective disable) or we need to restart mcollective in cloud-init runcmd. https://bugs.launchpad.net/fuel/+bug/1455489 (this bug)

if it's ubuntu related only i can propose the possible way of fixing it: https://review.openstack.org/#/c/179128/3/fuel_agent/fuel_agent/utils/build_utils.py

it looks like mcollective should be disabled like it was done for 'puppet'

> it looks like mcollective should be disabled like it was done for puppet

No, puppet agent is disabled for a different reason: we don't use a central server (instead the manifests are copied to each node with rsync).

Disabling mcollectived will surely break the deployment (unless cloud-init or nailgun agent re-enable mcollective)

https://review.openstack.org/#/c/184230/ proposed change

Reviewed: https://review.openstack.org/184230
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=2c8daa42f8a6afd65f23b0ff046389163482140a
Submitter: Jenkins
Branch: master

commit 2c8daa42f8a6afd65f23b0ff046389163482140a
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue May 19 14:53:09 2015 +0300

    Restart mcollective at the end of cloud-init magic

    We use mcollective cloud-init module to configure
    mcollective service. This module runs the command
    'service mcollective start' which does nothing
    if the service is already running (which is the actual case).
    So, we can end up with mcollective unable to see
    config file changed made by cloud-init and hence
    prevent mcollective from being able to connect to
    rabbit.

    This patch restarts mcollective at the end of cloud-init run.

    Closes-Bug: #1455489
    Change-Id: I9ea122acc168120596bf48f378e92874936bcbe1

The bug seems to be not fixed in mitaka and master:
https://github.com/openstack/fuel-agent/blob/master/cloud-init-templates/cloud_config_ubuntu.jinja2#L107-L115

Fix proposed to branch: master
Review: https://review.openstack.org/305637

Original review https://review.openstack.org/184230 is for project fuel-web while the fix is for fuel-agent. Probably this was the reason it was merged to nowhere.

Finally, I've figured the fix has been removed later during the fix of #1536924

Change abandoned by Georgy Kibardin (<email address hidden>) on branch: master
Review: https://review.openstack.org/305637

The right solution would be to fix cloudinit so that it would ensure that mcollective service is stopped before fixing it config and attempting to start it.
However, stopping mcollective before running cloudinit would be quite good walkaround.

Fix proposed to branch: master
Review: https://review.openstack.org/305711

Change abandoned by Georgy Kibardin (<email address hidden>) on branch: master
Review: https://review.openstack.org/305711
Reason: It turned out that bootcmd is executed too early, most likely before mcollective is even started

Fix proposed to branch: master
Review: https://review.openstack.org/306275

Change abandoned by Georgy Kibardin (<email address hidden>) on branch: master
Review: https://review.openstack.org/306275
Reason: Sorry, forgot to get rid of previous commit

Fix proposed to branch: master
Review: https://review.openstack.org/306279

Change abandoned by Georgy Kibardin (<email address hidden>) on branch: master
Review: https://review.openstack.org/306279

It looks like we cannot stop mcollective reliably - it is just impossible to be in time and stop it after it is started, but before cloudinit tries to start it.
Removing mcollective package completely and letting cloudinit install, configure and start it also doesn't work - there is a dependency on that package in fuel-agent.

> it is just impossible to be in time and stop it after it is started, but before cloudinit tries to start it.

I think it's possible.

It seems we can reorder execution order of cloud-init modules via cloud.cfg

so, 'bootcmd' or 'runcmd' will be executed exactly prior 'mcollective'

https://github.com/number5/cloud-init/blob/master/config/cloud.cfg

could that work? could you check that?

Yes, we can move bootcmd right before mcollective. However, this may break some other future logic which may decide to rely upon bootcmd. In this case better solution would be introduce dedicated module for that. However even in this case it is still possible that mcollective will be started after this module, but before cloud-init reconfigures this

BTW, created bug for cloud-init https://bugs.launchpad.net/cloud-init/+bug/1570776

BTW, as an option:
we may spoil mcollective config so that it would fail to start, then cloud-init would fix it :)

Setting logfile parameter to non-existing path prevent it from starting.

in fact, mcollective service is not supposed to start automatically anyway due to :
https://github.com/openstack/fuel-agent/blob/master/fuel_agent/utils/build.py#L208-L209

any reason why that didn't work for some cases?

The reason mcollective is started despite /etc/init/mcollective.override is present is that in some circumstances it may happen that cron may start nailgun-agent BEFORE cloudinit reconfigures mcollective. In this case nailgun-agent doesn't find "identity" field in mcollective config, tries to write something there and then restart mcollective.

Fix proposed to branch: master
Review: https://review.openstack.org/308946

Reviewed: https://review.openstack.org/308946
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=50bcd8448ecab85194b4559569ebc8568191e481
Submitter: Jenkins
Branch: master

commit 50bcd8448ecab85194b4559569ebc8568191e481
Author: Georgy Kibardin <email address hidden>
Date: Thu Apr 21 14:52:43 2016 +0300

    Do restart mcollective after provisioning

    nailgun agent must not change mcollective config after provisioning
    cloudinit is responsible for that.

    It may happen that nailgun agent may be started by cron earlier than
    it is reconfigured and started by cloudinit. This leads to the
    situation when cloudinit reconfigures mcollective and issues start
    which doesn't do anything since it is already started by nailgun agent.
    Finally, mcollective is left started with incorrect default
    configuration.

    Change-Id: I0c6f3720943ad21e22899368832e451bc906b098
    Closes-Bug: #1455489

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/314711

Reviewed: https://review.openstack.org/314711
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=8b0ef4e707e6afaae1863978d8b5dc94eb6315be
Submitter: Jenkins
Branch: stable/mitaka

commit 8b0ef4e707e6afaae1863978d8b5dc94eb6315be
Author: Georgy Kibardin <email address hidden>
Date: Thu Apr 21 14:52:43 2016 +0300

    Do restart mcollective after provisioning

    nailgun agent must not change mcollective config after provisioning
    cloudinit is responsible for that.

    It may happen that nailgun agent may be started by cron earlier than
    it is reconfigured and started by cloudinit. This leads to the
    situation when cloudinit reconfigures mcollective and issues start
    which doesn't do anything since it is already started by nailgun agent.
    Finally, mcollective is left started with incorrect default
    configuration.

    Change-Id: I0c6f3720943ad21e22899368832e451bc906b098
    Closes-Bug: #1455489
    (cherry picked from commit 50bcd8448ecab85194b4559569ebc8568191e481)

The fix relies on a "magic hostname" which is ugly as hell. Also leaves open quite a number of races, such as

- upstart starts cloud-init
- upstart starts cron
- cloud-init sets hostname and gets pre-emptied
- cron starts nailgun-agent
- nailgun-agent checks the hostname and does NOT update the node ID in server.cfg

Result: the node can't report itself to the master node

Please note: just because set_hostname module is started at the 'init' stage (as soon as / and /run
get mounted) doesn't mean it will complete before cron has been started (cron does not depend on
any cloud-init jobs). Also note that starts on foo' means it's OK to run this job after the `foo'
event has occured. However it doesn't mean all 'starts on foo' job will start immediately after `foo'.
Upstart will do its best to start them within the (foo, +infinity) time interval.

So the correct fix is to have exactly one entity which manages /etc/mcollective/server.cfg.
The most natural choice is cloud-init. The script installed by fuel-agent correctly fills in
the node ID (see https://github.com/openstack/fuel-agent/blob/master/cloud-init-templates/cloud_config_centos.jinja2#L69). So nailgun-agent should just stop fiddling with mcollectived
 service and its config in order to fix this bug for real

Alexei,

Magic hostname is defenitely not the best solution. To distinguish between provisioned/non-provisioned modes it is better to rely upon more explicit configuration option. The decision about single point of configuration also looks like the best way to proceed, so lets create a tech debt ticket.

As for races.

> Also https://review.openstack.org/#/c/308946/4 introduces a different bug:
>
>- upstart starts cloud-init
>- upstart starts cron
>- cloud-init sets hostname and gets pre-emptied
>- cron starts nailgun-agent
>- nailgun-agent checks the hostname and does NOT update the node ID in server.cfg
>
>Result: the node can't report itself to the master node

When we boot into bootstrap cloud-init is not used, hostname is preset and nailgun-agent writes node id into server.cfg
When we boot into bootstrapped OS cloud-config takes care about server.cfg.

At what point does the race you are talking about occur?

Here is the tech debt ticket https://bugs.launchpad.net/fuel/+bug/1585671

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/323333

Wasn't able to reproduce on 9.0-mos #450

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 450
cat /etc/fuel_build_number:
 450
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6347.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8723.noarch
 python-packetary-9.0.0-1.mos140.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-migrate-9.0.0-1.mos8435.noarch
 rubygem-astute-9.0.0-1.mos748.noarch
 fuel-mirror-9.0.0-1.mos140.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-openstack-metadata-9.0.0-1.mos8723.noarch
 fuel-notify-9.0.0-1.mos8435.noarch
 nailgun-mcagents-9.0.0-1.mos748.noarch
 python-fuelclient-9.0.0-1.mos323.noarch
 fuel-9.0.0-1.mos6347.noarch
 fuel-utils-9.0.0-1.mos8435.noarch
 fuel-setup-9.0.0-1.mos6347.noarch
 fuel-misc-9.0.0-1.mos8435.noarch
 fuel-library9.0-9.0.0-1.mos8435.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2715.noarch
 fuel-ostf-9.0.0-1.mos935.noarch
 fuelmenu-9.0.0-1.mos272.noarch
 fuel-nailgun-9.0.0-1.mos8723.noarch

Reviewed: https://review.openstack.org/323333
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=8fe27fe12f5fb758ea7cb3fecf6e69b36db65200
Submitter: Jenkins
Branch: stable/8.0

commit 8fe27fe12f5fb758ea7cb3fecf6e69b36db65200
Author: Georgy Kibardin <email address hidden>
Date: Thu Apr 21 14:52:43 2016 +0300

    Do restart mcollective after provisioning

    nailgun agent must not change mcollective config after provisioning
    cloudinit is responsible for that.

    It may happen that nailgun agent may be started by cron earlier than
    it is reconfigured and started by cloudinit. This leads to the
    situation when cloudinit reconfigures mcollective and issues start
    which doesn't do anything since it is already started by nailgun agent.
    Finally, mcollective is left started with incorrect default
    configuration.

    Change-Id: I0c6f3720943ad21e22899368832e451bc906b098
    Closes-Bug: #1455489
    (cherry picked from commit 50bcd8448ecab85194b4559569ebc8568191e481)

Verified on MOS 8.0 + mu2 updates.

Issue is not 100% reproducible.
With updates cluster was deployed successfully, no nodes stuck in "provisioning" state.

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/336501

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/337157

Change abandoned by Sergii Rizvan (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/336501

Change abandoned by Sergii Rizvan (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/337157

Changes abandoned for 6.1 and 7.0 because this patch would block mcollective identity updates on 6.1 and 7.0 just because that functionaly thoroughly relies on the 2 patches which were merged into 8.0 https://bugs.launchpad.net/fuel/+bug/1518306
Also due to rare reproducibility and huge difficulty of making the patch we're going to close the bug for 6.1 and 7.0 as Won't Fix.

Have regression in MOS 8.0 centos bootstrap. https://bugs.launchpad.net/fuel/+bug/1604365

It is possible to have configured hostname in "reset and redeploy scenario" because cobbler provides names over dhcp during pxe load. Ubuntu bootstrap is not affected because it never change hostname due to some lucky conditions (It rewrites rewrites hostname to static allways).

While it can be fixed in centos image in same fashion as it done in ubuntu (static hostname in image) I think it is better to discuss hostname based solution one more time.

As I see now we can do one the following.
1. Hostname solution. Do static hostname in centos (8.0 only).
2. Hostname solution. Fix cobbler in reset scenario. (all versions I think)
3. Drop hostname solution to something else.

By the way. Do we really need nailgun-agent after deployment?

>By the way. Do we really need nailgun-agent after deployment?

At the very least, nailgun-agent indicates to nailgun that node is online. Apart from that, I'm not aware about any other needs for nailgun-agent after deployment.

Verified on Newton.
Passed run:
https://product-ci.infra.mirantis.net/view/10.0_acceptance/job/10.0.acceptance.ubuntu.cli_acceptance_ceph_deployment_tests/16/