Unable to add plugin into existing deployed environment

# fuel --version
9.0.0
# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+----------+------------------+---------+-------------+-------------------+---------------------+---------------+--------+---------
 9 | discover | Untitled (01:b5) | | 10.220.0.11 | 64:b3:64:95:01:b5 | | | 1 |
10 | discover | Untitled (3b:20) | | 10.220.0.12 | 64:be:95:bf:3b:20 | | | 1 |
 3 | ready | Untitled (c8:74) | 1 | 10.220.0.7 | 64:52:2d:41:c8:74 | controller | | 1 | 1
 1 | ready | Untitled (fc:78) | 1 | 10.220.0.3 | 64:ff:c0:3d:fc:78 | controller | | 1 | 1
 8 | ready | Untitled (2d:ab) | 1 | 10.220.0.10 | 64:29:e6:ad:2d:ab | standalone-rabbitmq | | 1 | 1
 5 | ready | Untitled (1d:8f) | 1 | 10.220.0.9 | 64:7c:6a:46:1d:8f | ceph-osd, compute | | 1 | 1
 6 | ready | Untitled (b5:00) | 1 | 10.220.0.5 | 64:47:d2:ca:b5:00 | standalone-rabbitmq | | 1 | 1
 2 | ready | Untitled (5b:1d) | 1 | 10.220.0.4 | 64:36:c7:d1:5b:1d | controller | | 1 | 1
 7 | ready | Untitled (e6:44) | 1 | 10.220.0.8 | 64:51:b5:0d:e6:44 | standalone-rabbitmq | | 1 | 1
 4 | ready | Untitled (48:a4) | 1 | 10.220.0.6 | 64:1a:95:ef:48:a4 | ceph-osd, compute | | 1 | 1
# fuel plugins
id | name | version | package_version | releases
---+-----------------------------+---------+-----------------+----------------------------------------------
1 | detach-rabbitmq | 1.1.1 | 3.0.0 | ubuntu (mitaka-9.0)
2 | elasticsearch_kibana | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
3 | influxdb_grafana | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
4 | lma_collector | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
5 | lma_infrastructure_alerting | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)

I re-check with clean Fuel 9.0 without any plugins
# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+----------+------------------+---------+-------------+-------------------+-------------------+---------------+--------+---------
 8 | discover | Untitled (4b:90) | | 10.220.0.10 | 64:18:82:a0:4b:90 | | | 1 |
10 | discover | Untitled (cb:68) | | 10.220.0.11 | 64:2e:56:93:cb:68 | | | 1 |
 7 | discover | Untitled (e5:15) | | 10.220.0.9 | 64:ef:a5:b7:e5:15 | | | 1 |
 3 | ready | Untitled (f7:cc) | 1 | 10.220.0.7 | 64:fd:a2:04:f7:cc | controller | | 1 | 1
 5 | ready | Untitled (f4:62) | 1 | 10.220.0.4 | 64:2d:0b:9f:f4:62 | ceph-osd, compute | | 1 | 1
 9 | discover | Untitled (84:15) | | 10.220.0.12 | 64:e3:e3:0f:84:15 | | | 1 |
 2 | ready | Untitled (a0:a5) | 1 | 10.220.0.5 | 64:c8:e5:7a:a0:a5 | controller | | 1 | 1
 1 | ready | Untitled (c3:23) | 1 | 10.220.0.3 | 64:ea:13:b3:c3:23 | controller | | 1 | 1
 4 | ready | Untitled (bd:6f) | 1 | 10.220.0.6 | 64:6f:44:7a:bd:6f | ceph-osd, compute | | 1 | 1
 6 | ready | Untitled (0d:2a) | 1 | 10.220.0.8 | 64:8f:e6:ad:0d:2a | ceph-osd, compute | | 1 | 1

# fuel plugins
id | name | version | package_version | releases
---+-----------------------------+---------+-----------------+----------------------------------------------
1 | ceilometer-redis | 1.0.3 | 3.0.0 | ubuntu (mitaka-9.0)
2 | elasticsearch_kibana | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
3 | influxdb_grafana | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
4 | lma_collector | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
5 | lma_infrastructure_alerting | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)

As error message says these plugins are not hot-pluggable and cannot be added to deployed environment.

@Dmitry for some reason the message is incorrect because the plugins are hot-pluggable [1]...

[1] https://github.com/openstack/fuel-plugin-lma-collector/blob/0254a53cbbbd3ebcb90adf2fe719b036395384df/metadata.yaml#L16

[root@nailgun ~]# fuel --version
9.0.0
[root@nailgun ~]# cat /var/www/nailgun/plugins/lma_collector-0.10/metadata.yaml
# Plugin name
name: lma_collector
# Human-readable name for your plugin
title: The StackLight Collector Plugin
# Plugin version
version: 0.10.0
# Description
description: The StackLight Collector is the advanced monitoring agent of the so called Logging, Monitoring and Alerting (LMA) Toolchain of Mirantis OpenStack.
# Required fuel version
fuel_version: ['8.0', '9.0']

authors: [Mirantis Inc.]
licenses: [Apache License Version 2.0]
homepage: https://github.com/openstack/fuel-plugin-lma-collector
groups: ['monitoring']
is_hotpluggable: true

# The plugin is compatible with releases in the list
releases:
  - os: ubuntu
    version: liberty-8.0
    mode: ['ha']
    deployment_scripts_path: deployment_scripts/
    repository_path: repositories/ubuntu
  - os: ubuntu
    version: liberty-9.0
    mode: ['ha']
    deployment_scripts_path: deployment_scripts/
    repository_path: repositories/ubuntu
  - os: ubuntu
    version: mitaka-9.0
    mode: ['ha']
    deployment_scripts_path: deployment_scripts/
    repository_path: repositories/ubuntu

# Version of plugin package
package_version: '4.0.0'

Denis, re-read the message. It's just a warning and it doesn't block the installation of not hot-pluggable plugins.

Could you please attach the plugins metadata files from /var/www/nailgun/plugins/plugin_name/metadata.yaml from your master node?

@Alexey, please see my previous comment:
https://bugs.launchpad.net/lma-toolchain/+bug/1616825/comments/6

@Denis, thank you for the bug report.
I think the issue priority should be decreased to Medium because plugins are enabled on your screenshot. Message about not hot-pluggable version is a warning only, it doesn't block enabling and deploying the plugin.
Another question is why this message appears. It is displayed on UI in operational environment and when the chosen plugin version has hot_pluggable: false flag in its metadata.

So, we need to check response of GET /api/clusters/<cluster_id>/attributes to triage the issue.
Could you please provide it?

@Julia,

>>> I think the issue priority should be decreased to Medium because plugins are enabled on your screenshot.
Plugins are enables but you can not save changes due to plugins waiting for some data, all of them have additional options

I move importance to High back

From my point of view it is a critical one, due to Customers can not use hot-pluggable architecture

This behavior can be reproduced on HA (3 controller).
To unblock LMA(Stacklight) plugins settings you can additional install NetApp Fuel Plugin, even if you do not plan to use it. After installation of the NetApp Fuel Plugin refresh Fuel UI to get LMA(Stacklight) plugins settings available.

Can not reproduce the issue with the plugins installed to HA operational environment (9.0 release ISO). Diagnostic shapshot attached.

Also, it doesn't seem to be UI bug as UI displays controls properly if correct plugin data comes from Nailgun. I would check Nailgun responses for the environment where the issue reproduced.

The latest research shows that workaround for the issue is to refresh the page in browser.

Cluster settings will be reinitialized and new plugins data will be parsed properly:
https://github.com/openstack/fuel-ui/blob/master/static/models.js#L831-L833

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/391735

Fix proposed to branch: master
Review: https://review.openstack.org/394468

Reviewed: https://review.openstack.org/391735
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=e06dbe5b5dd996c72bcaef80a78ab2550aebf250
Submitter: Jenkins
Branch: stable/mitaka

commit e06dbe5b5dd996c72bcaef80a78ab2550aebf250
Author: Julia Aranovich <email address hidden>
Date: Mon Oct 31 10:48:50 2016 +0300

    Process newly installed plugins

    If new plugins appear in settings response,
    their data should be processed and plugin settings
    should be shown immediately on UI.

    Closes-Bug: #1616825

    Change-Id: Iddedd93b11ab22f67af60efaf9f052d8e6f09815

Reviewed: https://review.openstack.org/394468
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=278fd74c5a46ced37401b42c3d504d9b2fa073b6
Submitter: Jenkins
Branch: master

commit 278fd74c5a46ced37401b42c3d504d9b2fa073b6
Author: Julia Aranovich <email address hidden>
Date: Mon Oct 31 10:48:50 2016 +0300

    Process newly installed plugins

    If new plugins appear in settings response,
    their data should be processed and plugin settings
    should be shown immediately on UI.

    Closes-Bug: #1616825

    Change-Id: Iddedd93b11ab22f67af60efaf9f052d8e6f09815

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/396172

Reviewed: https://review.openstack.org/396172
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=142d8c1abaa859adceb622e340e97a0f5f3f2d12
Submitter: Jenkins
Branch: stable/newton

commit 142d8c1abaa859adceb622e340e97a0f5f3f2d12
Author: Julia Aranovich <email address hidden>
Date: Mon Oct 31 10:48:50 2016 +0300

    Process newly installed plugins

    If new plugins appear in settings response,
    their data should be processed and plugin settings
    should be shown immediately on UI.

    Closes-Bug: #1616825

    Change-Id: Iddedd93b11ab22f67af60efaf9f052d8e6f09815
    (cherry picked from commit 278fd74c5a46ced37401b42c3d504d9b2fa073b6)

Verified on 9.2 snapshot #465 (2016-11-10)

shotgun2 report:
http://paste.openstack.org/show/589041/

Actually, the wrong message about hot-plugging appears with plugin settings. It disappears after refreshing the page. There is no any limits to set up hot pluggable plugin on deployed environment.

This bug has been described in 9.1 release notes. Adding the release-notes tag to describe the bug in 9.2 resolved issues.

A release note has been added to 9.2 resolved issues: https://review.fuel-infra.org/#/c/29581.

This issue was fixed in the openstack/fuel-ui 11.0.0.0rc1 release candidate.

We don't have any "hot pluggable" plugin for 10.0 release. According to our rules issue was moved to "Fix Released", but we cannot verify the fix on stable/newton branch !