custom roles cannot update other nodes on deletion

https://drive.google.com/file/d/0B9Qf0veURSnySWNiRHd1RG15OGc/view?usp=sharing

Is it OK the repo https://github.com/stackforge/fuel-plugin-detach-rabbitmq is empty?

Sorry, code hasn't been moved yet, I updated link

Set to incomplete as it is not clear should we track plugins' bugs in Fuel project scope or in Fuel plugins scope

Nastya, please comment about change of bug status from Incomplete to Confirmed: why is this tracked as a Fuel bug? Is detach-rabbitmq plugin needed to reproduce it? How did you confirm that it's a bug in Fuel and not in the plugin?

This is a nailgun issue. It seems that update_required role field will update other nodes on addition, but not on removal.
http://paste.openstack.org/show/z1KXnWoGrw0qSrBKIZZE/

I think we should just add node.pending_deletion to this if statement: https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/task/helpers.py#L169

It is dangerous to merge in 7.0. But 'separate services' feature cannot be fully tested without this fix. So we are going to fix it and merge into master after HCF. Keeping in 7.0 release in order to have it on the plate for the team. Adding 'non-release' tag in order to highlight that this fix does not affect HCF.

There is legit workaround for this case. After operator removed node, he should re-run all required tasks with CLI [0].
Ofcourse he need to know exactly what tasks needs to be executed, but in worst case it is possible just to re-run whole node, like

  fuel node --node <id> --deploy

0. https://review.openstack.org/#/c/161192/20/pages/reference-architecture/task-deployment/0020-api.rst

Reproduced on 7.0-301 + MU1
Steps:
1) Deploy env with detach db+keystone (3 ctrls, 4 db+keystone)
2) Delete 1 db+keystone node
3) Deploy changes
4) Run 'hiera memcache_roles' on one of controllers

Result: the value wasn't updated after node deletion. (I will add logs later)
https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.plugins.thread_keystone_separate_services/16/testReport/%28root%29/separate_keystone_service_add_delete_node/separate_keystone_service_add_delete_node/

Snapshot https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.plugins.thread_keystone_separate_services/16/artifact/logs/fail_error_separate_keystone_service_add_delete_node-fuel-snapshot-2015-11-12_05-56-54.tar.xz.filtered.log

Addition to #10:
file /etc/hiera/override/detach-keystone wasn't updated and contains
http://paste.openstack.org/show/478650/

Issue was reproduced - https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/127/
Snapshot https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/127/artifact/logs/fail_error_separate_rabbit_service_add_delete_node-fuel-snapshot-2016-02-01_05-23-44.tar.xz
This issue will block total 3 cases for detach-service feature - raise to critical and add swarm-blocker.

Well, since Fuel 8.0 we support "reexecute_on" tasks. So plugin developers could add task:

  - id: my-task
    groups: [primary-controller, controller]
    reexecute_on: [deploy_changes]

and that would mean: "Run this task on all controllers *EACH* time deploy changes is pressed".

So I believe plugin developer could handle scale-down case using this approach.

So for 8.0 we can workaround this, but it doesn't resolve the issue that update_required field only applies on new nodes, but deletion doesn't work.

I confirm that using the reexecute_on stanza works as expected since this is what we're using for the LMA toolchain since MOS 8.

And I agree with Matt: the behavior should be consistent between scale up and scale down operations. It's very frustrating right now for plugin developers to understand exactly when the tasks are executed or not.

Matt, Simon,

I agree with you folks, but this issue *is not* about plugin SDK. It's about detached rabbit plugin. And the solution is to use what we have now. We simply can't *solve* scale down trigger problem two days before HCF. So you either fix it in your plugin, or I move bug to next release (since it's not regression, and feature request).

This has to proceed in the following way:
1) SDK needs to be updated with instructions for plugin developers on how to handle "downscale" case right (in plugins like detached Keystone, separated RabbitMQ, etc)
2) Upon SDK update, Mirantis developers of detached Keystone and separated RabbitMQ plugins need to be notified on how to make changes in their plugins

With these 2 conditions we can move this bug out of 8.0 and treat it as a feature request/Pluggable Framework enhancement in one of the upcoming cycles

Dmitry,

We can and we should treat it as a feature request. It's something what we never supported. We simply have no time to implement this enhancement.

Yet, what the action items? Who is going to write this documentation?

Well, I've created a bug #1541329 to track limitations of Fuel Plugin SDK.

Doc Team, could you please contact Simon P. and document the workaround of this problem? It would be very useful for plugin developers.

reexecute_on: [deploy_changes] doesn't reflect the use case and adds even more burden to plugin developers. If I'm a plugin developer, I need to override the following base tasks if I want to update rabbitmq and memcache nodes list:
${plugin}_hiera_config
hiera
globals (actually this is already in master)
cluster
memcache

But then you will re-execute these tasks on controller as well... and again on add/remove of controllers, cinder nodes, compute nodes... etc. I want to narrow the scope of repeating puppet just for the case of adding/removing a node with my plugin's role.

The fact remains that a parameter "update_required" exists and has a very clear criteria, but does not work as expected.

Added workaround to Fuel Plugin SDK: https://wiki.openstack.org/wiki/Fuel/Plugins#Configuration_of_Fuel_Plugins_with_new_roles

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

The original bug still exists: only controller role can be redeployed on node deletion. You can repeat individual tasks, but only applying to ALL changes. Either we need a better set of conditions for "redeploy-on:" field or fix update_required node role to work for deletion as well.

This bug is NOT high for detach-rabbitmq plugin because controller is updated on node deletion and the list is accurate for listing all amqp hosts.

I think this issue was reproduced on 9.0 iso #101

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/56/console

Test: separate_keystone_service_add_delete_node

2016-03-24 02:01:19,732 - ERROR decorators.py:123 -- Traceback (most recent call last):
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/fuelweb_test/helpers/decorators.py", line 117, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/fuelweb_test/tests/tests_separate_services/test_separate_keystone.py", line 269, in separate_keystone_service_add_delete_node
    cmd='hiera memcache_roles')
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/fuelweb_test/helpers/checkers.py", line 1132, in check_hiera_hosts
    ' others'.format(node['hostname']))
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Hosts on node node-8 differ from others

LCM changes will land soon. I'm assigning to Vladimir Kuklin and he can pass it back to me when the capability is introduced for 9.0.

Now we can define 2.1.0 tasks in 9.0. An example task is here:
https://review.openstack.org/#/c/298341/6/deployment/puppet/osnailyfacter/modular/cluster/tasks.yaml

if network_scheme is updated (added/deleted nodes), we can repeat a given task.

Verification blocked by https://bugs.launchpad.net/fuel/+bug/1587415

Verified as fixed in 9.0-mos-485