Mistral

Error validate token when run an action with cron trigger

Bug #1843175 reported by Bo Tran on 2019-09-08

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Mistral	Fix Released	Medium	Bo Tran	Mistral train-1 "train-1"

Bug Description

When i run mistral cron trigger with my workflow use openstack client such as nova client, cinder client. I get a same error when trigger run:

"You are not authorized to perform the requested action: identity:validate_token."

Content log:
2019-09-08 20:13:35.181 28881 DEBUG mistral.actions.openstack.actions [req-6bb9e511-6c94-4441-bee2-24cc40d92dad 1bd5194dfe534ec1935c75330691b7$
e 55453907ace44200a87b591a41aaf077 - - -] Cinder action security context: <mistral_lib.actions.context.ActionContext object at 0x7f2032909590>
_create_client /root/mistral/mistral/actions/openstack/actions.py:245
2019-09-08 20:13:37.016 28881 WARNING mistral.actions.openstack.base [req-6bb9e511-6c94-4441-bee2-24cc40d92dad 1bd5194dfe534ec1935c75330691b7f$
55453907ace44200a87b591a41aaf077 - - -] Traceback (most recent call last):
  File "/root/mistral/mistral/actions/openstack/base.py", line 115, in run
    method = self._get_client_method(self._get_client(context))
  File "/root/mistral/mistral/actions/openstack/base.py", line 84, in _get_client
    return self._create_client(context)
  File "/root/mistral/mistral/actions/openstack/actions.py", line 247, in _create_client
    cinder_endpoint = self.get_service_endpoint()
  File "/root/mistral/mistral/actions/openstack/base.py", line 108, in get_service_endpoint
    region_name=self.action_region
  File "/root/mistral/mistral/utils/openstack/keystone.py", line 160, in get_endpoint_for_project
    service_catalog = obtain_service_catalog(ctx)
  File "/root/mistral/mistral/utils/openstack/keystone.py", line 229, in obtain_service_catalog
    include_catalog=True
  File "/root/env/local/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 93, in get_token_data
    resp, body = self._client.get(url, headers=headers)
  File "/root/env/local/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 386, in get
    return self.request(url, 'GET', **kwargs)
  File "/root/env/local/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 545, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/root/env/local/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request
    return self.session.request(url, method, **kwargs)
  File "/root/env/local/lib/python2.7/site-packages/keystoneauth1/session.py", line 943, in request
    raise exceptions.from_response(resp, method, url)
Forbidden: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (Request-ID: req-3525f308-02c2-4c32-abfe
-a3e3a062919b)
: Forbidden: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (Request-ID: req-3525f308-02c2-4c32-ab
fe-a3e3a062919b)
2019-09-08 20:13:37.017 28881 WARNING mistral.executors.default_executor [req-6bb9e511-6c94-4441-bee2-24cc40d92dad 1bd5194dfe534ec1935c75330691
b7fe 55453907ace44200a87b591a41aaf077 - - -] The action raised an exception [action_ex_id=2b96f67b-6078-4a90-903d-144de10bbdf0, action_cls='<cl
ass 'mistral.actions.action_factory.CinderAction'>', attributes='{u'client_method_name': u'volumes.get'}', params='{u'volume_id': u'6412ff92-06
f6-4d75-b6c4-3f513c227482'}']
CinderAction.volumes.get failed: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (Request-ID: req-3525f308-02c2-4c32-abfe-a3e3a062919b): ActionException: CinderAction.vo
lumes.get failed: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (Request-ID: req-3525f308-02c2-4c32-abfe-a3e3a062919b)
2019-09-08 20:13:37.017 28881 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-09-08 20:13:37.017 28881 ERROR mistral.executors.default_executor File "/root/mistral/mistral/executors/default_executor.py", line 114, in run_action
2019-09-08 20:13:37.017 28881 ERROR mistral.executors.default_executor result = action.run(action_ctx)
2019-09-08 20:13:37.017 28881 ERROR mistral.executors.default_executor File "/root/mistral/mistral/actions/openstack/base.py", line 130, in run
2019-09-08 20:13:37.017 28881 ERROR mistral.executors.default_executor (self.__class__.__name__, self.client_method_name, str(e))
2019-09-08 20:13:37.017 28881 ERROR mistral.executors.default_executor ActionException: CinderAction.volumes.get failed: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (
Request-ID: req-3525f308-02c2-4c32-abfe-a3e3a062919b)
2019-09-08 20:13:37.017 28881 ERROR mistral.executors.default_executor
2019-09-08 20:13:37.035 28881 WARNING mistral.actions.openstack.base [req-6bb9e511-6c94-4441-bee2-24cc40d92dad 1bd5194dfe534ec1935c75330691b7fe 55453907ace44200a87b591a41aaf077 - - -] Traceback (most recent call la
st):
  File "/root/mistral/mistral/actions/openstack/base.py", line 115, in run
    method = self._get_client_method(self._get_client(context))
  File "/root/mistral/mistral/actions/openstack/base.py", line 84, in _get_client
    return self._create_client(context)
  File "/root/mistral/mistral/actions/openstack/actions.py", line 247, in _create_client
    cinder_endpoint = self.get_service_endpoint()
  File "/root/mistral/mistral/actions/openstack/base.py", line 108, in get_service_endpoint
    region_name=self.action_region
  File "/root/mistral/mistral/utils/openstack/keystone.py", line 160, in get_endpoint_for_project
    service_catalog = obtain_service_catalog(ctx)
  File "/root/mistral/mistral/utils/openstack/keystone.py", line 229, in obtain_service_catalog
    include_catalog=True
  File "/root/env/local/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 93, in get_token_data
    resp, body = self._client.get(url, headers=headers)
  File "/root/env/local/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 386, in get
    return self.request(url, 'GET', **kwargs)
  File "/root/env/local/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 545, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/root/env/local/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request
    return self.session.request(url, method, **kwargs)
  File "/root/env/local/lib/python2.7/site-packages/keystoneauth1/session.py", line 943, in request
    raise exceptions.from_response(resp, method, url)
Forbidden: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (Request-ID: req-e1de6e60-2d88-40a3-86bd-12eb43f6a4b3)
: Forbidden: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (Request-ID: req-e1de6e60-2d88-40a3-86bd-12eb43f6a4b3)
2019-09-08 20:13:37.036 28881 WARNING mistral.executors.default_executor [req-6bb9e511-6c94-4441-bee2-24cc40d92dad 1bd5194dfe534ec1935c75330691b7fe 55453907ace44200a87b591a41aaf077 - - -] The action raised an excep
tion [action_ex_id=6286d964-7deb-43f1-8767-6d911a3e56ad, action_cls='<class 'mistral.actions.action_factory.CinderAction'>', attributes='{u'client_method_name': u'volumes.get'}', params='{u'volume_id': u'5c1b3ea4-d
873-453c-9daf-f06dfc46a000'}']
CinderAction.volumes.get failed: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (Request-ID: req-e1de6e60-2d88-40a3-86bd-12eb43f6a4b3): ActionException: CinderAction.vo
lumes.get failed: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (Request-ID: req-e1de6e60-2d88-40a3-86bd-12eb43f6a4b3)
2019-09-08 20:13:37.036 28881 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-09-08 20:13:37.036 28881 ERROR mistral.executors.default_executor File "/root/mistral/mistral/executors/default_executor.py", line 114, in run_action
2019-09-08 20:13:37.036 28881 ERROR mistral.executors.default_executor result = action.run(action_ctx)
2019-09-08 20:13:37.036 28881 ERROR mistral.executors.default_executor File "/root/mistral/mistral/actions/openstack/base.py", line 130, in run
2019-09-08 20:13:37.036 28881 ERROR mistral.executors.default_executor (self.__class__.__name__, self.client_method_name, str(e))
2019-09-08 20:13:37.036 28881 ERROR mistral.executors.default_executor ActionException: CinderAction.volumes.get failed: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (
Request-ID: req-e1de6e60-2d88-40a3-86bd-12eb43f6a4b3)
2019-09-08 20:13:37.036 28881 ERROR mistral.executors.default_executor
2019-09-08 20:13:37.171 28881 DEBUG mistral.executors.executor_server [req-6bb9e511-6c94-4441-bee2-24cc40d92dad 1bd5194dfe534ec1935c75330691b7fe 55453907ace44200a87b591a41aaf077 - - -] Sending action result to engi
ne [action_ex_id=2b96f67b-6078-4a90-903d-144de10bbdf0, action_cls=mistral.actions.openstack.actions.CinderAction] run_action /root/mistral/mistral/executors/executor_server.py:121
2019-09-08 20:13:37.204 28881 DEBUG mistral.executors.executor_server [req-6bb9e511-6c94-4441-bee2-24cc40d92dad 1bd5194dfe534ec1935c75330691b7fe 55453907ace44200a87b591a41aaf077 - - -] Sending action result to engi
ne [action_ex_id=6286d964-7deb-43f1-8767-6d911a3e56ad, action_cls=mistral.actions.openstack.actions.CinderAction] run_action /root/mistral/mistral/executors/executor_server.py:121

Tags:

Bo Tran (ministry.nd) on 2019-09-09

Changed in mistral:
status:	New → In Progress
assignee:	nobody → Bo Tran (ministry.nd)

Renat Akhmerov (rakhmerov) on 2019-09-10

Changed in mistral:
milestone:	none → train-1
importance:	Undecided → Medium

Revision history for this message

Boxiang Zhu (bxzhu-5355) wrote on 2019-09-11:

Hi, when I use the normal user(with `member` role), and I add a cron trigger to do something with cinder api. But I met the same problem. The error log is as followed:

2019-09-11 03:23:59.589 6 WARNING mistral.actions.openstack.base [req-4097987f-7527-48aa-8e2e-8261eabd8bd4 7c8adfbef47241a7b5e1d3b4e4fe5a09 efa0e71fb6974ad3a252449f8a376927 - - -] Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/mistral/actions/openstack/base.py", line 115, in run
    method = self._get_client_method(self._get_client(context))
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/mistral/actions/openstack/base.py", line 84, in _get_client
    return self._create_client(context)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/mistral/actions/openstack/actions.py", line 233, in _create_client
    cinder_endpoint = self.get_service_endpoint()
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/mistral/actions/openstack/base.py", line 108, in get_service_endpoint
    region_name=self.action_region
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/mistral/utils/openstack/keystone.py", line 160, in get_endpoint_for_project
    service_catalog = obtain_service_catalog(ctx)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/mistral/utils/openstack/keystone.py", line 229, in obtain_service_catalog
    include_catalog=True
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 85, in get_token_data
    resp, body = self._client.get(url, headers=headers)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 375, in get
    return self.request(url, 'GET', **kwargs)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 534, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 237, in request
    return self.session.request(url, method, **kwargs)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 890, in request
    raise exceptions.from_response(resp, method, url)
Forbidden: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (Request-ID: req-e7c8b915-bbc6-4703-96f1-50258a233f62)
: Forbidden: You are not authorized to perform the requested action: identity:validate_token. (HTTP 403) (Request-ID: req-e7c8b915-bbc6-4703-96f1-50258a233f62)

Hi, when I use the normal user(with `member` role), and I add a cron trigger to do something with cinder api. But I met the same problem. The error log is as followed:

Revision history for this message

Francois Scheurer (scheuref) wrote on 2019-09-12:

Dear All

Same problem here...
Running openstack rocky with mistral 7.0.1.1

- creating and executing the workflow works.

- creating the cron trigger works and we can verify that the trust get created with:
openstack trust list

- but the execution of cron trigger fails on identity:validate_token.

The last msg from keystone debug log are (with some uid replaced with text):

2019-09-05 09:38:00.902 29 DEBUG keystone.policy.backends.rules [req-1a276b9d-8276-4ec3-b516-f51f86cd1df6 fsc fscproject - testdom testdom]
            enforce identity:validate_token:
            {
               'service_project_id':None,
               'service_user_id':None,
               'service_user_domain_id':None,
               'service_project_domain_id':None,
               'trustor_id':None,
               'user_domain_id':u'testdom',
               'domain_id':None,
               'trust_id':u'mytrustid',
               'project_domain_id':u'testdom',
               'service_roles':[],
               'group_ids':[],
               'user_id':u'fsc',
               'roles':[
                  u'_member_',
                  u'creator',
                  u'reader',
                  u'heat_stack_owner',
                  u'member',
                  u'load-balancer_member'],
               'system_scope':None,
               'trustee_id':None,
               'domain_name':None,
               'is_admin_project':True,
               'token':<TokenModel (audit_id=0LAsW_0dQMWXh2cTZTLcWA, audit_chain_id=[u'0LAsW_0dQMWXh2cTZTLcWA']) at 0x7f208f4a3bd0>,
               'project_id':u'fscproject'
            } enforce /var/lib/kolla/venv/local/lib/python2.7/site-packages/keystone/policy/backends/rules.py:33
        2019-09-05 09:38:00.920 29 WARNING keystone.common.wsgi [req-1a276b9d-8276-4ec3-b516-f51f86cd1df6 fsc fscproject - testdom testdom]
            You are not authorized to perform the requested action: identity:validate_token.: ForbiddenAction: You are not authorized to perform the requested action: identity:validate_token.

The problem does not arise when the role service or admin is added to the user.

Cheers
Francois Scheurer

Dear All

Same problem here...
Running openstack rocky with mistral 7.0.1.1

- creating and executing the workflow works.

- creating the cron trigger works and we can verify that the trust get created with:
    openstack trust list

- but the execution of cron trigger fails on identity:validate_token.

The last msg from keystone debug log are (with some uid replaced with text):

2019-09-05 09:38:00.902 29 DEBUG keystone.policy.backends.rules [req-1a276b9d-8276-4ec3-b516-f51f86cd1df6 fsc fscproject - testdom testdom]
            enforce identity:validate_token: 
            {
               'service_project_id':None,
               'service_user_id':None,
               'service_user_domain_id':None,
               'service_project_domain_id':None,
               'trustor_id':None,
               'user_domain_id':u'testdom',
               'domain_id':None,
               'trust_id':u'mytrustid',
               'project_domain_id':u'testdom',
               'service_roles':[],
               'group_ids':[],
               'user_id':u'fsc',
               'roles':[
                  u'_member_',
                  u'creator',
                  u'reader',
                  u'heat_stack_owner',
                  u'member',
                  u'load-balancer_member'],
               'system_scope':None,
               'trustee_id':None,
               'domain_name':None,
               'is_admin_project':True,
               'token':<TokenModel (audit_id=0LAsW_0dQMWXh2cTZTLcWA, audit_chain_id=[u'0LAsW_0dQMWXh2cTZTLcWA']) at 0x7f208f4a3bd0>,
               'project_id':u'fscproject'
            } enforce /var/lib/kolla/venv/local/lib/python2.7/site-packages/keystone/policy/backends/rules.py:33
        2019-09-05 09:38:00.920 29 WARNING keystone.common.wsgi [req-1a276b9d-8276-4ec3-b516-f51f86cd1df6 fsc fscproject - testdom testdom] 
            You are not authorized to perform the requested action: identity:validate_token.: ForbiddenAction: You are not authorized to perform the requested action: identity:validate_token.

The problem does not arise when the role service or admin is added to the user.

Cheers
Francois Scheurer

Revision history for this message

Bo Tran (ministry.nd) wrote on 2019-09-12:

Dear all,

I processing this problem in here: https://review.opendev.org/#/c/680858/

Thanks you!

Revision history for this message

Francois Scheurer (scheuref) wrote on 2019-09-12:

Dear Bo

your new patch fixed the issue by us !
Congrats and many Thx!

Francois

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-13: Fix merged to mistral (master)

Reviewed: https://review.opendev.org/680858
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=51b7dd0c6d5c4d0de354719765968ca883b9f8fb
Submitter: Zuul
Branch: master

commit 51b7dd0c6d5c4d0de354719765968ca883b9f8fb
Author: Bo Tran <email address hidden>
Date: Sun Sep 8 20:32:25 2019 +0700

Fix error validate token when run cron trigger

A trust client can't do validate token when run cron trigger
This patch will fix that.

Change-Id: I793fbfec03032d9ff7137c11becb6d1c18ec54bc
Closes-Bug: #1843175

Changed in mistral:
status:	In Progress → Fix Released

Revision history for this message

Francois Scheurer (scheuref) wrote on 2019-09-13:

Hi Bo

Thanks for the fix!
BTW, you also wrote another fix for a similar/same issue here:

https://opendev.org/openstack/mistral/commit/6102c5251e29c1efe73c92935a051feff0f649c7?style=split

This was not fixing the problem by us.
Using https://review.opendev.org/680858 without https://opendev.org/openstack/mistral/commit/6102c5251e29c1efe73c92935a051feff0f649c7
is working for us.

Do you advise to use https://opendev.org/openstack/mistral/commit/6102c5251e29c1efe73c92935a051feff0f649c7 as well however? or better not?

Thank you in advance.

Best Regard
Francois Scheurer

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-13: Fix included in openstack/mistral 9.0.0.0b1

This issue was fixed in the openstack/mistral 9.0.0.0b1 development milestone.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-10: Fix proposed to mistral (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/687933

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-16: Fix merged to mistral (stable/rocky)

Reviewed: https://review.opendev.org/687933
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=d1b8dd1635aa45d3d644b356f883484997f31ca7
Submitter: Zuul
Branch: stable/rocky

commit d1b8dd1635aa45d3d644b356f883484997f31ca7
Author: Bo Tran <email address hidden>
Date: Sun Sep 8 20:32:25 2019 +0700

Fix error validate token when run cron trigger

A trust client can't do validate token when run cron trigger
This patch will fix that.

    Change-Id: I793fbfec03032d9ff7137c11becb6d1c18ec54bc
    Closes-Bug: #1843175
    (cherry picked from commit 51b7dd0c6d5c4d0de354719765968ca883b9f8fb)

tags:

added: in-stable-rocky

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-11-21: Fix proposed to mistral (stable/stein)

#10

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/695332

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-11-21: Fix merged to mistral (stable/stein)

#11

Reviewed: https://review.opendev.org/695332
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=9da790c7fbcb6d27748787da44a62290e5a3a528
Submitter: Zuul
Branch: stable/stein

commit 9da790c7fbcb6d27748787da44a62290e5a3a528
Author: Bo Tran <email address hidden>
Date: Sun Sep 8 20:32:25 2019 +0700

Fix error validate token when run cron trigger

A trust client can't do validate token when run cron trigger
This patch will fix that.

    Change-Id: I793fbfec03032d9ff7137c11becb6d1c18ec54bc
    Closes-Bug: #1843175
    (cherry picked from commit 51b7dd0c6d5c4d0de354719765968ca883b9f8fb)