OpenStack Keystone Charm

Tempest test fails due to keystone policy, tempest.lib.exceptions.Forbidden: Forbidden

Bug #1943850 reported by Bas de Bruijne
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Keystone Charm
Fix Released
High
Bas de Bruijne

Bug Description

This tempest test, when used from the upstream plugin fails due to keystone policy:

octavia_tempest_plugin.tests.api.v2.test_load_balancer.LoadBalancerAPITest.test_load_balancer_list
-------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/snap/fcbtest/x1/.rally/verification/verifier-191296ad-d570-4dc9-8243-cfd028acdadb/repo/tempest/test.py", line 181, in setUpClass
    raise value.with_traceback(trace)
  File "/home/ubuntu/snap/fcbtest/x1/.rally/verification/verifier-191296ad-d570-4dc9-8243-cfd028acdadb/repo/tempest/test.py", line 166, in setUpClass
    cls.setup_credentials()
  File "/snap/fcbtest/x1/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 150, in setup_credentials
    **params)['role_assignments']
  File "/home/ubuntu/snap/fcbtest/x1/.rally/verification/verifier-191296ad-d570-4dc9-8243-cfd028acdadb/repo/tempest/lib/services/identity/v3/role_assignments_client.py", line 46, in list_role_assignments
    resp, body = self.get(url)
  File "/home/ubuntu/snap/fcbtest/x1/.rally/verification/verifier-191296ad-d570-4dc9-8243-cfd028acdadb/repo/tempest/lib/common/rest_client.py", line 314, in get
    return self.request('GET', url, extra_headers, headers)
  File "/home/ubuntu/snap/fcbtest/x1/.rally/verification/verifier-191296ad-d570-4dc9-8243-cfd028acdadb/repo/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/home/ubuntu/snap/fcbtest/x1/.rally/verification/verifier-191296ad-d570-4dc9-8243-cfd028acdadb/repo/tempest/lib/common/rest_client.py", line 804, in _error_checker
    raise exceptions.Forbidden(resp_body, resp=resp)
tempest.lib.exceptions.Forbidden: Forbidden
Details: {'code': 403, 'message': 'You are not authorized to perform the requested action: identity:list_role_assignments.', 'title': 'Forbidden'}
-------------------------------------------------------------------------------------------------

Snippets of keystone policy for this action:
-------------------------------------------------------------------------------------------------
    "admin_required": "role:Admin",
    "cloud_admin": "rule:admin_required and (is_admin_project:True or domain_id:<admin_domain_id> or project_id:<project_named_services_id>)",
    "admin_on_domain_filter": "rule:admin_required and domain_id:%(scope.domain.id)s",
    "admin_on_project_filter": "rule:admin_required and project_id:%(scope.project.id)s",
    "admin_on_domain_of_project_filter": "rule:admin_required and domain_id:%(target.project.domain_id)s",
    "identity:list_role_assignments": "rule:cloud_admin or rule:admin_on_domain_filter or rule:admin_on_project_filter",
-------------------------------------------------------------------------------------------------

Tempest definitely gets the Admin role, I'm not sure why it fails the other rules. It might be a bug or wrong setting in tempest. Commands from the CLI have no problems.

Tempest might be asking for a token from one domain and attempting to list role assignments from another domain, but I can't find proof of this in the logs.

I am wondering if the keystone policy is too strict for the tempest test.

Tags: cdo-tempest
Revision history for this message
Bas de Bruijne (basdbruijne) wrote :

Possibly duplicate of https://bugs.launchpad.net/charm-keystone/+bug/1830076

1830076 has overlapping keystone policy rule:cloud_admin

Revision history for this message
Bas de Bruijne (basdbruijne) wrote :

Problem is solved by downgrading the Tempest Octavia plugin version to stein-last.

Bas de Bruijne (basdbruijne)
tags: added: cdo-tempest
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

For clarification:

Charms 21.04
Ubuntu: focal
OpenStack: ussuri

Changed in charm-keystone:
status: New → Triaged
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

This is probably due to mismatched default policies from the packages or charms. Needs to be investigated to work out where the issue is.

Changed in charm-keystone:
assignee: nobody → Alex Kavanagh (ajkavanagh)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-keystone (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-keystone/+/848145

Changed in charm-keystone:
status: Triaged → In Progress
Felipe Reyes (freyes)
Changed in charm-keystone:
importance: Undecided → High
Revision history for this message
Bas de Bruijne (basdbruijne) wrote :

With tempest 31.1.0 on Xena focal I also see:

```
tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_put_object_using_temp_url
fail 0.069
40f45f0b-ceb2-4339-865b-861ac4247924
Traceback (most recent call last):
  File "/home/ubuntu/snap/fcbtest/34/.rally/verification/verifier-e11fad09-b3d6-4b69-9e14-5ff1e67146b2/repo/tempest/common/utils/__init__.py", line 89, in wrapper
    return func(*func_args, **func_kwargs)
  File "/home/ubuntu/snap/fcbtest/34/.rally/verification/verifier-e11fad09-b3d6-4b69-9e14-5ff1e67146b2/repo/tempest/api/object_storage/test_object_temp_url.py", line 148, in test_put_object_using_temp_url
    resp, _ = self.object_client.put(url, new_data, None)
  File "/home/ubuntu/snap/fcbtest/34/.rally/verification/verifier-e11fad09-b3d6-4b69-9e14-5ff1e67146b2/repo/tempest/lib/common/rest_client.py", line 363, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "/home/ubuntu/snap/fcbtest/34/.rally/verification/verifier-e11fad09-b3d6-4b69-9e14-5ff1e67146b2/repo/tempest/lib/common/rest_client.py", line 720, in request
    self._error_checker(resp, resp_body)
  File "/home/ubuntu/snap/fcbtest/34/.rally/verification/verifier-e11fad09-b3d6-4b69-9e14-5ff1e67146b2/repo/tempest/lib/common/rest_client.py", line 821, in _error_checker
    raise exceptions.Forbidden(resp_body, resp=resp)
tempest.lib.exceptions.Forbidden: Forbidden
Details: {'Code': 'AccessDenied', 'RequestId': 'tx0000051a956df4268d577-0062d83601-26bc-default', 'HostId': '26bc-default-default'}
```

I assume that this is the same problem.

Revision history for this message
Bas de Bruijne (basdbruijne) wrote :

Is there any progress on this bug? I'm updating the tempest versions that SQA/Field use for the next FCE release, it would be nice if we can update the octavia-tempest-plugin from stein-last as well.

Revision history for this message
Bas de Bruijne (basdbruijne) wrote (last edit ):

The issue is fixed in https://review.opendev.org/#/q/I8aea2b597b9dd9bbdc5a1527fae03e86364aab4c

TLDR: The octavia-tempest-plugin records all the role assignments for debugging purposes, but this does not have an effect on the tests themselves. The user role logging can now be skipped by setting `log_user_roles=False` in the tempest load-balancer configs.

Changed in charm-keystone:
status: In Progress → Fix Committed
Bas de Bruijne (basdbruijne)
Changed in charm-keystone:
status: Fix Committed → Fix Released
assignee: Alex Kavanagh (ajkavanagh) → Bas de Bruijne (basdbruijne)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.