Tempest plugins does not set the corresponding service in service_available config option

Fix proposed to branch: master
Review: https://review.openstack.org/534624

Yea, this is issue at least for congress plugin tests. let me describe from where issue started and what can be potential fix:

Congress issue:
aodh and ceilometer tempest plugins are merged into single repo now - telemetry-tempest-plugin. telemetry-tempest-plugin set both ceilomter and aodh_plugin config option in service_available group with default value as True.

Now what happened, congress tempest plugin which is user of aodh and ceilometer tempest plugin interfaces to gets both service available in service_available config group (because of default value). Even Aodh service has not been installed in congress's job setup. This is issue and causing false service availability to congress tests.

General issue:
This issue can considered as general case also where service is being installed and setup bu different repo/pckg (project) and service_available config options is being registered with default value by separate repo (tempest plugin tests).

There can be case where actual service is not installed but you install the plugin for interface usage etc. This is what happening in gate for congress case now.

As solution we can give ownership of setting the service_available options on devstack side which should check actual service availability before setting it to true/false.

Now question is whether it can be done on devstack side[1] or devstack plugin of each service. Devstack plugin can give same issue as it would not run if service is not installed.

..1 https://github.com/openstack-dev/devstack/blob/52237ee5482ff8c614dc68088ecaaf7d59a85915/lib/tempest#L529

thanks for adding discussion.

based on above discussion on IRC, we can check to expand the devstack setting the service availability or if any issue on telemetry-tempest-plugin side.

In Devstack TEMPEST_SERVICES had been made local variable (https://review.openstack.org/#/c/281788) to move all tempest supported services in devstack setting only. - I9c24705e494689f09a885eb0a640efd50db33fcf

Based on local tempest_services devstack register the available service on tempest side
- https://github.com/openstack-dev/devstack/blob/614cab33c40159f0bc10d92c9f8dc3f9783708d9/lib/tempest#L528

Almost all tempest plugin still using unsupported global variable TEMPEST_SERVICES in impressions that they have registered them as service available on tempest side which is not the case.

- http://codesearch.openstack.org/?q=TEMPEST_SERVICES&i=nope&files=&repos=

Below is plan to solve this issue which exist for almost all the plugins
1. make all tempest plugin services set by devstack with local variable
2. remove unsupported TEMPEST_SERVICES setting for all plugins.
3. any new tempest plugin want their testing service to available in tempest should register that in devstack itself.

Fix proposed to branch: master
Review: https://review.openstack.org/536723

Seems like devstack patch would not be enough as aodh tempest plugin register aodh as "aodh_plugin" in service_available which is being used same way in congress tempest plugin.

Change abandoned by Anusha Ramineni (<email address hidden>) on branch: master
Review: https://review.openstack.org/534624

Fix proposed to branch: master
Review: https://review.openstack.org/536741

Related fix proposed to branch: master
Review: https://review.openstack.org/536745

Related fix proposed to branch: master
Review: https://review.openstack.org/536753

Change abandoned by Ghanshyam Mann (<email address hidden>) on branch: master
Review: https://review.openstack.org/536745
Reason: not needed as fixed in https://review.openstack.org/#/c/536753/

Related fix proposed to branch: master
Review: https://review.openstack.org/536830

Reviewed: https://review.openstack.org/536830
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=b3539b9fd58b4dd34759bc913e50d95561e451e4
Submitter: Zuul
Branch: master

commit b3539b9fd58b4dd34759bc913e50d95561e451e4
Author: ghanshyam <email address hidden>
Date: Tue Jan 23 16:48:09 2018 +0300

    Remove use of unsupported TEMPEST_SERVICES variable

    TEMPEST_SERVICES global variable is not supported
    by devstack since long back.
    - I380dd20e5ed716a0bdf92aa02c3730359b8136e4
    - I9c24705e494689f09a885eb0a640efd50db33fcf

    Service availability of tempest known services will be
    set by devstack with local check.
    - I02be777bf93143d946ccbb8e9eff637bfd1928d4

    This commit removes the unused TEMPEST_SERVICES setting
    Related-Bug: #1743688

    Change-Id: I50bd32fc2103af0170ee8d350f31cf589d85422f

Reviewed: https://review.openstack.org/536741
Committed: https://git.openstack.org/cgit/openstack/telemetry-tempest-plugin/commit/?id=8bccc06c2bc71798f532f4ba6bc873c5e5dbb896
Submitter: Zuul
Branch: master

commit 8bccc06c2bc71798f532f4ba6bc873c5e5dbb896
Author: ghanshyam <email address hidden>
Date: Tue Jan 23 12:35:06 2018 +0300

    Fix aodh service config option name

    aodh service config option name registered in
    Tempest's service_available group is 'aodh_plugin' which
    is not consistent with all other service_available config
    option.

    After below patch Devstack will automatically set all service
    availability on tempest side. For that all services name should
    going to be registered on Tempest must be consistent with
    service name registered on devstack side.
    - I02be777bf93143d946ccbb8e9eff637bfd1928d4

    This patch rename the config option name to 'aodh'
    and keep the old name with deprecation warning.
    Closes-Bug: #1743688

    Change-Id: I7739e0b672a7211a1d643543fdc79031affa0a0b

Change abandoned by Ghanshyam Mann (<email address hidden>) on branch: master
Review: https://review.openstack.org/536723
Reason: doing it on plugin side.

As per discussion on IRC and review, below is the agreed plan

- Devstack should set only tempest own services
- each plugin to set their service as service_availability in their devstack plugin
- plugin using other plugins can set service_availability of required plugins in devstack plugin as per job definition

As per comment above, not a Tempest bug. Marking as "incomplete" on Tempest

Reviewed: https://review.openstack.org/536753
Committed: https://git.openstack.org/cgit/openstack/congress-tempest-plugin/commit/?id=e58b0eb52d6c3794820e33e60bd208328c3a1380
Submitter: Zuul
Branch: master

commit e58b0eb52d6c3794820e33e60bd208328c3a1380
Author: ghanshyam <email address hidden>
Date: Tue Jan 23 13:25:35 2018 +0300

    Use 'aodh' for aodh service availability

    below patch rename the aodh service_available
    config option from 'aodh_plugin' to 'aodh'.
    - I7739e0b672a7211a1d643543fdc79031affa0a0b

    This patch make congress tempest test to use the
    new name so that it can be fetched correctly from
    devstack setting on tempest config file.

    Depends-On: I7739e0b672a7211a1d643543fdc79031affa0a0b

    Change-Id: Iccfc40cc738994d6c7d08cbea845e0c2f888b995
    Related-Bug: #1743688

Reviewed: https://review.openstack.org/533133
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=aa779c5bd46649d9b672e1112c9e65ec857cde66
Submitter: Zuul
Branch: master

commit aa779c5bd46649d9b672e1112c9e65ec857cde66
Author: Anusha Ramineni <email address hidden>
Date: Mon Feb 5 10:03:26 2018 +0530

    Configure tempest service_availability correctly

    To be able to correctly identify services are enabled or not to
    run respective tempest tests, configure the same in devstack plugin
    in proper way, instead of deciding the service availability based on
    import of plugins.

    Closes-Bug: #1743688
    Change-Id: I9cbe3f1307b78d0e703069bc6c011b62c0d3c51f

This issue was fixed in the openstack/congress 7.0.0.0rc1 release candidate.

Patch for devstack has been abandoned, please update if there is still an issue with devstack.

marked invalid for Tempest.

For Devstack - We need to move the non tempest owned service availability setting on devtsack plugins side and remove those from devstack.

Fix proposed to branch: master
Review: https://review.openstack.org/619973

Fix proposed to branch: master
Review: https://review.openstack.org/619984

Fix proposed to branch: master
Review: https://review.openstack.org/619985

Fix proposed to branch: master
Review: https://review.openstack.org/620002

Fix proposed to branch: master
Review: https://review.openstack.org/620255

Reviewed: https://review.openstack.org/620255
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=6611df0e2da0abc96979c52da0013535abf79428
Submitter: Zuul
Branch: master

commit 6611df0e2da0abc96979c52da0013535abf79428
Author: ghanshyam <email address hidden>
Date: Tue Nov 27 08:21:35 2018 +0000

    Set Tempest's service_available setting for congress

    Tempest's service_available config option includes all the service
    availability which is further used by tests to take decision of skip
    or run the test.

    For example, [service_available].congress is true then, congress test will run
    or if [service_available].congress is false then, all congress related tests either
    in telemetry tempest plugin or any other plugins will be skipped.

    This commit add the setting of congress service[1] on congress devstack plugin.

    Closes-Bug: #1743688
    [1] http://git.openstack.org/cgit/openstack/congress-tempest-plugin/tree/congress_tempest_plugin/config.py#n24

    Change-Id: I57de34d2b185b8e9838dd288d3673149391f0024

Reviewed: https://review.openstack.org/619984
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=d5d9342beeff5c8ac655073b8986feb6b91a9f25
Submitter: Zuul
Branch: master

commit d5d9342beeff5c8ac655073b8986feb6b91a9f25
Author: ghanshyam <email address hidden>
Date: Mon Nov 26 08:25:23 2018 +0000

    Set Tempest's service_availability setting for Trove

    Tempest's service_availability config option includes all the service
    availability which is further used by tests to take decision of skip
    or run the test.

    For example, [service_availability].trove is true then, trove test will run
    or if [service_availability].trove is false then, all trove related tests either
    in trove tempest plugin or any other plugins will be skipped.

    Currently this setting for trove service[1] is in devstack lib/tempest
    which is being removed by - https://review.openstack.org/#/c/619973/

    For better maintenance, we are moving all tempest non-owned service setting
    to service devstack plugin side.

    This commit add the setting of trove service on trove devstack plugin.

    Closes-Bug: #1743688
    [1] https://git.openstack.org/cgit/openstack/trove-tempest-plugin/tree/trove_tempest_plugin/config.py#n17

    Needed-By: https://review.openstack.org/#/c/619973/
    Change-Id: Ie3c6bab6e0be9dfcfe29ec3f412e704f3ad79d61

Octavia and other tempest plugins are also facing same issue bug#1817154.

Note: we are facing these issue because service_available config option is registered by tempest plugin repo which is separate repo from service. So registration of config option and setting value is happening via separate repo.

These are the final steps to fix the service_availabilty config option.

1. Devstack should set only tempest own services - https://review.openstack.org/#/c/619973/
2. each plugin to set their service as service_availability in their devstack plugin - In progress
3. Change the default value of service_available.<service> config option default value to false.
4. Add used tempest plugins in tempest plugins's requirements.txt after adding it in g-r.

That way we will solve all the issues currently faced by cross tempest plugin:
1. Required tempest plugin's service_availability config option is registered:
- Any tempest plugin can install required tempest plugins as a dependency in requirement.txt.
2. Service_available value is set correctly:
- Each service will set its service_available config option correctly while installation itself via devstack plugin.

This issue was fixed in the openstack/congress 9.0.0.0rc1 release candidate.

This issue was fixed in the openstack/trove 11.0.0.0rc1 release candidate.

Reviewed: https://review.openstack.org/620002
Committed: https://git.openstack.org/cgit/openstack/aodh/commit/?id=0e40e57c1ab2e8468016d4d877a91ce3a06df40a
Submitter: Zuul
Branch: master

commit 0e40e57c1ab2e8468016d4d877a91ce3a06df40a
Author: ghanshyam <email address hidden>
Date: Mon Nov 26 10:19:39 2018 +0000

    Set Tempest's service_availability setting for Aodh

    Tempest's service_available config option includes all the service
    availability which is further used by tests to take decision of skip
    or run the test.

    For example, [service_available].aodh is true then, aodh test will run
    or if [service_available].aodh is false then, all aodh related tests either
    in telemetry tempest plugin or any other plugins will be skipped.

    This commit add the setting of aodh service[1] on aodh devstack plugin.

    Closes-Bug: #1743688
    [1] https://github.com/openstack/telemetry-tempest-plugin/blob/master/telemetry_tempest_plugin/config.py#L27

    Change-Id: Idac9f38aac6964ce4d40b6afccbd9dfbf6e16a41

This issue was fixed in the openstack/aodh 9.0.0.0rc1 release candidate.

Reviewed: https://review.opendev.org/619973
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=f0dd9996cc635709276f51e4d94c5ebbf5d5b49f
Submitter: Zuul
Branch: master

commit f0dd9996cc635709276f51e4d94c5ebbf5d5b49f
Author: ghanshyam <email address hidden>
Date: Mon Nov 26 07:38:54 2018 +0000

    Fix DevStack to configure tempest's service_availability

    Tempest's service_availability config option includes all the service
    availability which is further used by tests to take decision of skip
    or run the test.

    For example, [service_availability].nova is true then, compute test will run
    or if [service_availability].aodh is false then, all aodh related tests either
    in aodh tempest plugin or any other plugins will be skipped.

    Now question is what is the best way to set the each service availability for
    tempest or tempest plugins tests. We have 2 category of service here-
    1. Service tested by Tempest (nova, cinder, keystone, glance, swift, neutron)
       (let's say type1 service)
    2. Services tested by Tempest plugins (all other than above list)
       (let's say type2 service)

    We need the standard way to set both type of service so that we can maintain
    the setting of service_availability config options in consistent way.

    As discussed on bug#1743688/ and review https://review.openstack.org/#/c/536723/,
    we will use devstack lib/tempest to set the type1 service which is services test
    owned by Tempest and type2 service setting will be done by devstack plugins of
    those service.

    For example - [service_availability].ironic will be set by ironic's devstack plugin.
    because that is best place we know ironic is installed and available.

    To do that we need:
    1. Add setting of [service_availability].* in devstack plugins
    2. Remove setting of type2 service from devstack lib/tempest

    This commit does the second part and all depends-on patches handle the first part.

    Related-Bug: #1743688

    Change-Id: If3aec9fd1c61e2bb53233be437b97b811dc82414

Reviewed: https://review.opendev.org/620000
Committed: https://git.openstack.org/cgit/openstack/panko/commit/?id=a725f644120f94f03c575915c70e0f44c7114687
Submitter: Zuul
Branch: master

commit a725f644120f94f03c575915c70e0f44c7114687
Author: ghanshyam <email address hidden>
Date: Mon Nov 26 10:09:13 2018 +0000

    Set Tempest's service_availability setting for panko

    Tempest's service_availability config option includes all the service
    availability which is further used by tests to take decision of skip
    or run the test.

    For example, [service_availability].panko is true then, panko test will run
    or if [service_availability].panko is false then, all panko related tests either
    in telemetry tempest plugin or any other plugins will be skipped.

    This commit add the setting of panko service[1] on panko devstack plugin.

    Closes-Bug: #1743688
    [1] https://github.com/openstack/telemetry-tempest-plugin/blob/master/telemetry_tempest_plugin/config.py#L23

    Change-Id: I57e182850fbf7fcab5df75c05f0ef95a8c23e923

This seems to be done for devstack's side by https://review.opendev.org/c/openstack/devstack/+/619973, the rest of the work was done in devstack plugins (per the commit message of 619973) and the latest comment in: https://review.opendev.org/c/openstack/devstack/+/536723/