Broken error message for stx-os apply rejection when services are not at services-enabled state

Bug #1974221 reported by Thales Elero Cervi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
New
Low
Thales Elero Cervi

Bug Description

Brief Description
-----------------
When a stx-openstack application apply operation is launched on a node in which VIM services are not at services-enabled state, the returned error message is broken and not providing useful information for the user.

Severity
--------
Minor: System/Feature is usable with minor issue

Steps to Reproduce
------------------
- Lock/unlock the controller
- Launch a manual "application-apply stx-openstack" right after the node reboot

Expected Behavior
------------------
The operation should be rejected, informing the reason to the user: "operation is not allowed while the node controller-0 not in services-enabled state."

Actual Behavior
----------------
The operation is rejected, with a broken error message: "Application-apply rejected: Remote error: %(exc_type)s %(value)s %(traceback)s."

Reproducibility
---------------
100% reproducible (given the node is not in services-enabled state)

System Configuration
--------------------
One node system (virtual environment)

Branch/Pull Time/Commit
-----------------------
master - 20220518

Last Pass
---------
Don't know.

Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ system application-apply stx-openstack
Application-apply rejected: Remote error: %(exc_type)s %(value)s
%(traceback)s.

from /var/log/sysinv.log:
sysinv 2022-05-19 13:40:37.218 98244 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672
sysinv 2022-05-19 13:40:37.225 98013 INFO sysinv.conductor.kube_app [-] lifecycle hook for application stx-openstack (1.0-199-centos-stable-versioned) started {'lifecycle_type': u'check', 'relative_timing': u'pre', 'mode': u'manual', 'operation': u'apply', 'extra': {}}.
sysinv 2022-05-19 13:40:37.251 98013 INFO k8sapp_openstack.lifecycle.lifecycle_openstack [-] stx-openstack requires application-apply, but some VIM services are not at services-enabled state on node controller-0.
Application-apply rejected.
sysinv 2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: LifecycleSemanticCheckException: Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state.
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 436, in _process_data
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp **args)
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 13795, in app_lifecycle_actions
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp self._app.app_lifecycle_actions(context, self, rpc_app, hook_info)
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 2379, in app_lifecycle_actions
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp lifecycle_op.app_lifecycle_actions(context, conductor_obj, self, app, hook_info)
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/opt/platform/helm/22.06/stx-openstack/1.0-199-centos-stable-versioned/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py", line
90, in app_lifecycle_actions
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp return self._pre_manual_apply_check(conductor_obj, app, hook_info)
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/opt/platform/helm/22.06/stx-openstack/1.0-199-centos-stable-versioned/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py", line
317, in _pre_manual_apply_check
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp active_controller.hostname, constants.VIM_SERVICES_ENABLED))
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp LifecycleSemanticCheckException: Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state.
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp
sysinv 2022-05-19 13:40:37.260 98013 ERROR sysinv.openstack.common.rpc.common [-] Returning exception Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state. to caller: LifecycleSemanticCheckException: Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state.
sysinv 2022-05-19 13:40:37.260 98013 ERROR sysinv.openstack.common.rpc.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 436, in _process_data\n **args)\n', ' File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 13795, in app_lifecycle_actions\n self._app.app_lifecycle_actions(context, self, rpc_app, hook_info)\n', ' File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 2379, in app_lifecycle_actions\n lifecycle_op.app_lifecycle_actions(context, conductor_obj, self, app, hook_info)\n', ' File "/opt/platform/helm/22.06/stx-openstack/1.0-199-centos-stable-versioned/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py", line 90, in app_lifecycle_actions\n return self._pre_manual_apply_check(conductor_obj, app, hook_info)\n', ' File
"/opt/platform/helm/22.06/stx-openstack/1.0-199-centos-stable-versioned/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py", line 317, in _pre_manual_apply_check\n active_controller.hostname, constants.VIM_SERVICES_ENABLED))\n', 'LifecycleSemanticCheckException: Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state.\n']: LifecycleSemanticCheckException: Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state.
sysinv 2022-05-19 13:40:37.272 98244 WARNING wsme.api [-] Client-side error: Application-apply rejected: Remote error: %(exc_type)s %(value)s
%(traceback)s.: ClientSideError: Application-apply rejected: Remote error: %(exc_type)s %(value)s

Test Activity
-------------
Developer Testing

Workaround
----------
None

Revision history for this message
Ghada Khalil (gkhalil) wrote :

@Thales, Is this an stx-openstack specific issue or a generic application framework issue?

Changed in starlingx:
importance: Undecided → Low
Revision history for this message
Thales Elero Cervi (tcervi) wrote :

I just saw it when trying to apply the stx-openstack application on that system state, did not try with other apps.
Since the stx-openstack plugin appears to be generating the correct lifecyle message:

k8sapp_openstack.lifecycle.lifecycle_openstack [-] stx-openstack requires application-apply, but some VIM services are not at services-enabled state on node controller-0.
Application-apply rejected.

I would say that the wsme.api may have an issue when handling it:
wsme.api [-] Client-side error: Application-apply rejected: Remote error: %(exc_type)s %(value)s

But didn't have time to investigate it further so far.

Ghada Khalil (gkhalil)
tags: added: stx.distro.openstack
Changed in starlingx:
assignee: nobody → Thales Elero Cervi (tcervi)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.