Deploy will fail if keystone.conf has '[oslo_policy]/enforce_scope=true'

Some more information if the URL changes in the future.

== https://docs.openstack.org/releasenotes/keystone/en_GB/train.html ==
[bug 1806762] The user policies in policy.v3cloudsample.json policy file have been removed in favor of better defaults in code. These policies weren’t tested exhaustively and were misleading to users and operators.

[bug 1806762] The grant policies in policy.v3cloudsample.json policy file have been removed in favor of better defaults in code. These policies weren’t tested exhaustively and were misleading to users and operators.

[bug 1804462] The group policies in policy.v3cloudsample.json policy file have been removed in favor of better defaults in code. These policies weren’t tested exhaustively and were misleading to users and operators.

Related fix proposed to branch: master
Review: https://review.opendev.org/692179

The proposed changes tests it in CI.

Reproduced.

So trying to get auth not scoped to a project but domain instead, I get this:
failed: [primary] (item={u'service_type': u'identity', u'name': u'keystone'}) => {
    "action": "os_keystone_service",
    "attempts": 5,
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "module_args": {
                "auth": {
                    "auth_url": "http://192.0.2.10:35357",
                    "domain_name": "default",
                    "password": "9PJVm6kJI1k00JgNzhXpRAosMAXBkIqSSmDYDwR3",
                    "user_domain_name": "default",
                    "username": "admin"
                },
                "cacert": "",
                "description": "Openstack Identity Service",
                "interface": "admin",
                "name": "keystone",
                "region_name": "RegionOne",
                "service_type": "identity"
            },
            "module_extra_vars": null,
            "module_name": "os_keystone_service",
            "timeout": 180,
            "user": null
        }
    },
    "item": {
        "description": "Openstack Identity Service",
        "endpoints": [
            {
                "interface": "admin",
                "url": "http://192.0.2.10:35357"
            },
            {
                "interface": "internal",
                "url": "http://192.0.2.10:5000"
            },
            {
                "interface": "public",
                "url": "http://192.0.2.10:5000"
            }
        ],
        "name": "keystone",
        "type": "identity"
    },
    "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible-tmp-1572531912.06-54869509402289/AnsiballZ_os_keystone_service.py\", line 114, in <module>\n _ansiballz_main()\n File \"/tmp/ansible-tmp-1572531912.06-54869509402289/AnsiballZ_os_keystone_service.py\", line 106, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/tmp/ansible-tmp-1572531912.06-54869509402289/AnsiballZ_os_keystone_service.py\", line 49, in invoke_module\n imp.load_module('__main__', mod, module, MOD_DESC)\n File \"/tmp/ansible_os_keystone_service_payload_RpqMjI/__main__.py\", line 194, in <module>\n File \"/tmp/ansible_os_keystone_service_payload_RpqMjI/__main__.py\", line 153, in main\n File \"/opt/ansible/lib/python2.7/site-packages/openstack/cloud/_identity.py\", line 510, in search_services\n services = self.list_services()\n File \"/opt/ansible/lib/python2.7/site-packages/openstack/cloud/_identity.py\", line 485, in list_services\n if self._is_client_version('identity', 2):\n File \"/opt/ansible/lib/python2.7/site-packages/openstack/cloud/openstackcloud.py\", line 459, in _is_client_version\n client = getattr(self, client_name)\n File \"/opt/ansible/lib/python2.7/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/opt/ansible/lib/python2.7/site-packages/openstack/cloud/openstackcloud.py\", line 422, in _get_versioned_client\n endpoint_override=self.config.get_endpoint(service_type))\n File \"/o...

Hey keystoners, if you could take a look at the situation we are in. I have no idea how to bend openstacksdk (as used by Ansible) to let me create service endpoints. It either ends up having Client Error with admin-project-scoped auth or 401 with supposedly defalt-domain-scoped auth.

service logs:
2019-10-31 14:25:13.159673 2019-10-31 14:25:13.158 22 DEBUG keystone.server.flask.request_processing.req_logging [req-610327c9-f08d-46b8-b9b7-6f6d18120f41 - - - - -] REQUEST_METHOD: `GET` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2019-10-31 14:25:13.160259 2019-10-31 14:25:13.159 22 DEBUG keystone.server.flask.request_processing.req_logging [req-610327c9-f08d-46b8-b9b7-6f6d18120f41 - - - - -] SCRIPT_NAME: `` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2019-10-31 14:25:13.160798 2019-10-31 14:25:13.160 22 DEBUG keystone.server.flask.request_processing.req_logging [req-610327c9-f08d-46b8-b9b7-6f6d18120f41 - - - - -] PATH_INFO: `/` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2019-10-31 14:25:13.168445 2019-10-31 14:25:13.167 22 DEBUG keystone.server.flask.request_processing.req_logging [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] REQUEST_METHOD: `POST` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2019-10-31 14:25:13.169235 2019-10-31 14:25:13.168 22 DEBUG keystone.server.flask.request_processing.req_logging [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] SCRIPT_NAME: `` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2019-10-31 14:25:13.170971 2019-10-31 14:25:13.169 22 DEBUG keystone.server.flask.request_processing.req_logging [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] PATH_INFO: `/v3/auth/tokens` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2019-10-31 14:25:13.495747 2019-10-31 14:25:13.494 22 DEBUG keystone.auth.core [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] MFA Rules not processed for user `e292abd123704368832b9383f9c8b31d`. Rule list: `[]` (Enabled: `True`). check_auth_methods_against_rules /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/auth/core.py:446\x1b[00m
2019-10-31 14:25:13.498527 2019-10-31 14:25:13.498 22 DEBUG keystone.common.fernet_utils [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/common/fernet_utils.py:289\x1b[00m
2019-10-31 14:25:13.501483 2019-10-31 14:25:13.500 22 DEBUG keystone.models.token_model [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] User e292abd123704368832b9383f9c8b31d has no access to domain default _validate_domain_scope /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/models/token_model.py:511\x1b[00m
2019-10-31 14:25:13.506320 2019-10-31 14:25:13.505 22 WARNING keystone.server.flask.application [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] Authorization failed. The requ...

> It either ends up having Client Error with admin-project-scoped auth or 401 with supposedly defalt-domain-scoped auth.

The scope_type for identity:create_endpoint is "system": https://docs.openstack.org/keystone/latest/configuration/policy.html

So neither a project- nor domain-scoped token will be usable with this policy if enforce_scope is set to true. You either need a system-scoped token or you need to set enforce_scope=false (which is the default).

Thanks, Colleen. Indeed I felt like I was missing something obvious there. Will look in that direction then. Though I am not sure whether I saw system-scope toggle there...

Settings:
[oslo_policy]
enforce_scope = True

GET /v3/services
identity:list_services

"openstack_auth": {
  "auth_url": "http://192.0.2.10:35357",
  "username": "admin",
  "password": "9PJVm6kJI1k00JgNzhXpRAosMAXBkIqSSmDYDwR3"
}

Error:
keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in user. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-ea92e74e-2cf1-4524-9c78-a02f24a180a1)

Goal:
Getting auth into scope 'system'. I can not figure out how to do this.

-------------------------

Traceback (most recent call last):
  File "/tmp/ansible-tmp-1573484557.4130447-82566091548974/AnsiballZ_os_keystone_service.py", line 114, in <module>
    _ansiballz_main()
  File "/tmp/ansible-tmp-1573484557.4130447-82566091548974/AnsiballZ_os_keystone_service.py", line 106, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/tmp/ansible-tmp-1573484557.4130447-82566091548974/AnsiballZ_os_keystone_service.py", line 49, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/opt/ansible/lib/python3.6/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/opt/ansible/lib/python3.6/imp.py", line 170, in load_source
    module = _exec(spec, sys.modules[name])
  File "<frozen importlib._bootstrap>", line 618, in _exec
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/ansible_os_keystone_service_payload_4370z2m9/__main__.py", line 194, in <module>
  File "/tmp/ansible_os_keystone_service_payload_4370z2m9/__main__.py", line 153, in main
  File "/opt/ansible/lib/python3.6/site-packages/openstack/cloud/_identity.py", line 510, in search_services
    services = self.list_services()
  File "/opt/ansible/lib/python3.6/site-packages/openstack/cloud/_identity.py", line 485, in list_services
    if self._is_client_version('identity', 2):
  File "/opt/ansible/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py", line 459, in _is_client_version
    client = getattr(self, client_name)
  File "/opt/ansible/lib/python3.6/site-packages/openstack/cloud/_identity.py", line 32, in _identity_client
    'identity', min_version=2, max_version='3.latest')
  File "/opt/ansible/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py", line 422, in _get_versioned_client
    endpoint_override=self.config.get_endpoint(service_type))
  File "/opt/ansible/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 345, in get_api_major_version
    return self.session.get_api_major_version(auth or self.auth, **kwargs)
  File "/opt/ansible/lib/python3.6/site-packages/keystoneauth1/session.py", line 1233, in get_api_major_version
    return auth.get_api_major_version(self, **kwargs)
  File "/opt/ansible/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 500, in get_api_major_version
    data = get_endpoint_data(discover_versions=discover_versions)
  File "/opt/ansible/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 2...

Yeah, had the same issue in that review I linked to above.

We had a chat with openstacksdkers and it is probably not doable from ansible side atm.

If you could raise the issue with ansible module, it would help resolving this (actually, that's the only path I can think of).

A bug ticket has been open against openstacksdk. https://storyboard.openstack.org/#!/story/2006839

openstacksdk: https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/cloud/openstackcloud.py
keystoneauth1: https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth1/identity/v3/base.py
python project openstacksdk is using python project keystoneauth1 for authentication.

Openstacksdk is using keystoneauth1. keystoneauth1 can handle the argument 'system_scope' with value 'all'. Problem is that this is lost somehow along the path to keystoneauth1 from kolla-ansible and it's 'openstack_auth' setting. Maybe kolla-ansible is calling the wrong function/class in openstacksdk. Maybe openstacksdk is calling wrong function/class in keystoneauth1.

Hi. I do think I have solved this problem. Can you please create a new patch set with openstack_auth as below? Tell Zuul to run it.

Currently I do think it is required to write "Default" and not "default".
It is important to not specify domain_id, domain_name, project_id, project_name, trust_id or unscoped.

In ansible/group_vars/all.yml edit:
openstack_auth:
  auth_url: "{{ keystone_admin_url }}"
  username: "{{ keystone_admin_user }}"
  password: "{{ keystone_admin_password }}"
  user_domain_name: "Default"
  system_scope: "all"

Done. I had "system-scope". I will rotfl if it passes because it was one character off. ;D

Passes. Rotfl. ;D

Still, we have various dependencies on this variable so it needs a little bit of more work but at least we have a fix for the very issue. Thanks for investigating. I was literally one character from the fix.

Next fix is to give nova-cell system scope permission. openstackcli seems to put into scope system if no domain, projects (and so on) is not specified.

share/kolla-ansible/ansible/roles/nova-cell/tasks/discover_computes.yml.
Remove:
--os-project-domain-name
--os-tenant-name
Change:
--os-user-domain-name {{ openstack_auth.user_domain_name }}

Later in the deploy play I have problem with missing config file "/var/lib/kolla/config_files/config.json", TASK [neutron : Running Neutron bootstrap container] found in file share/kolla-ansible/ansible/roles/nova/tasks/bootstrap_service.yml. Currently I do not know if it is related to the change above. Still troubleshooting.

I have pushed successful build into https://review.opendev.org/692179
Please review this patch.

Given the current size of the patch, we'll drop this from 9.0.0.

Hello,

this affect kolla-ansible/wallaby, too.

Hi,
I'm having the same issue deploying the Xena release.
I'm using kolla-ansible 13.0.1.
Thank you!