Orchestrator doesn't clean deleted node from the cloud upon node removal

At least "delete removed node from Corosync/Pacemaker cluster" was addressed in the 6.1 scope. I'm setting the status as incomplete for the 6.1 and later milestones as it requires QA confirmation

The following ones are already addressed in 6.1:
- delete removed node from Corosync/Pacemaker cluster (or else deployment of new nodes might fail)
- delete removed node from Ceph cluster (or else ceph -s will report HEALTH_WARN for missing/offline mon)

I'm afraid we won't be able to backport such fixes into 6.0, due to large number of architectural and other changes (like corosync/pacemaker package versions and compatibility).

So in 6.1 user indeed has to manually clean up following:
- delete Neutron agents associated with removed nodes from DB
- delete Nova services associated with removed nodes from DB (or else certain OSTF test will be failing)
But it's not critical since it does not cause denial of service and cloud should operate just fine.

Adding swift related bug as a dup

Please create a separate ticket for docs team to explain manual steps to be done when we remove the node, if we can't automate this in 7.0 timeframe.

https://bugs.launchpad.net/fuel/+bug/1482203 created. Marking this bug as won't fix for 7.0

This is a life-cycle-management feature, not a bug. Please submit a blueprint and attach this bug there

Moving to wont-fix due
https://bugs.launchpad.net/fuel/+bug/1471172

Wrong link has been paster,
correct one:
https://bugs.launchpad.net/fuel/+bug/1471172/comments/7

Fix proposed to branch: master
Review: https://review.openstack.org/309330

Reproduced at https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.plugins.fuel_plugin_example/90/console

Two cases with "Some nova services have not been started"

That is a swarm blocker due to fact that all tests which remove/reinstall controller are failing on ostf step.

It it is hard to implement on library side, it is possible to relax requirements for ostf (allow failure of couple tests which check node statues)

Jobs like https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ha_neutron_tun/90/testReport/(root)/neutron_tun_ha_addremove/neutron_tun_ha_addremove/ are affected.
It is possible to add full list of them.

@Alexandr, this is a new feature request. Tests shouldn't rely on parts of LCM that are not supposed to be implemented in 9.0. As far as I know there is a separate bug for fix in OSTF. Removing 'swarm-blocker' tag.

Thanks, Dmitry. I found review https://review.openstack.org/#/c/309082/2/fuel_health/tests/sanity/test_sanity_infrastructure.py and will force it to merge in ostf.

Reviewed: https://review.openstack.org/309330
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c02971c21a1c0149926a0d1e048d9f798cc387e0
Submitter: Jenkins
Branch: master

commit c02971c21a1c0149926a0d1e048d9f798cc387e0
Author: Matthew Mosesohn <email address hidden>
Date: Fri Apr 22 12:57:11 2016 +0300

    Purge nova services for deleted nodes

    Deleted hosts create downed services in nova, which should
    be cleaned up on the next deploy cycle.

    Change-Id: Ie644062c9befec1745adcc8ca9e76a895cf0304c
    Depends-On: I01db215e77a3532a6fa7bf46ab7e20e281e8c165
    Depends-On: I303ab218bc6a48cf2c60727feecc522040a80a68
    Partial-Bug: #1471172

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/22320

Reviewed: https://review.fuel-infra.org/22320
Submitter: Evgeny Konstantinov <email address hidden>
Branch: master

Commit: 2845bca5e1ccec1e894672bf3fc1c07493619608
Author: Evgeny Konstantinov <email address hidden>
Date: Wed Jun 22 10:30:56 2016

Add Nova resolved issues 9.0

Change-Id: Ia8c83f6dfffa2df143cf5c494e41e8676aff8028
Related-Bug: #1556819
Related-Bug: #1471172