Bug #1658641 “Moving/disabling LDAP users break Keystone queries...” : Bugs : OpenStack Identity (keystone)

Revision history for this message

Lance Bragstad (lbragstad) wrote on 2017-01-23:

#1

I did some checking with other keystone developers that are a bit more familiar with keystone+LDAP integration [0]. It sounds like the short answer is that keystone doesn't support inspecting group assignment changes on the LDAP side.

We could clarify that in our LDAP documentation though.

[0] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-01-23.log.html#t2017-01-23T20:20:56

tags:

added: documentation

Revision history for this message

Tim S (timss) wrote on 2017-01-24:

#2

Thank you Lance, that's very helpful and pretty much what I expected.

Looking at the existing documentation of deleting users[1], it would seem that purging specific Local IDs (of users) using keystone-manage is one way to go. Alternatively, purge the entire domain and repopulate the domain right afterwards.

| While keystone will dynamically maintain the identity mapping, including
| removing entries when entities are deleted via the keystone, for those
| entities in backends that are managed outside of keystone (e.g. a read-only
| LDAP), keystone will not know if entities have been deleted and hence will
| continue to carry stale identity mappings in its table. While benign,
| keystone provides an ability for operators to purge the mapping table of such
| stale entries using the keystone-manage command, for example:
|
| $ keystone-manage mapping_purge --domain-name DOMAINA --local-id <email address hidden>

Looking at the populate function however, it does not seem to support populating only specific entries [2]. Would it be ok to run a full repopulate even after only purging specific entries? Or would they be regenerated automatically?

I think that only minor clarifications are needed in the documentation as it is already quite good. It could possibly underline that any changes to DNs for already mapped entities would break the mapping, not just deleting entities. Depending on how you suggest to repopulate purged specific entries, maybe also some words about that if you feel it's necessary.

Suppose this is a bug for python-keystoneclient, but it would also be nice if it was able to still list the users in a project even if one or more users have invalid mappings. Currently it breaks ungracefully, even if it's able to request information about most users.

[1]: http://docs.openstack.org/developer/keystone/configuration.html#domain-specific-drivers
[2]: https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/cmd/cli.py#L1239

Thank you Lance, that's very helpful and pretty much what I expected.

Looking at the existing documentation of deleting users[1], it would seem that purging specific Local IDs (of users) using keystone-manage is one way to go. Alternatively, purge the entire domain and repopulate the domain right afterwards.

| While keystone will dynamically maintain the identity mapping, including
| removing entries when entities are deleted via the keystone, for those
| entities in backends that are managed outside of keystone (e.g. a read-only
| LDAP), keystone will not know if entities have been deleted and hence will
| continue to carry stale identity mappings in its table. While benign,
| keystone provides an ability for operators to purge the mapping table of such
| stale entries using the keystone-manage command, for example:
|
| $ keystone-manage mapping_purge --domain-name DOMAINA --local-id abc@de.com

Looking at the populate function however, it does not seem to support populating only specific entries [2]. Would it be ok to run a full repopulate even after only purging specific entries? Or would they be regenerated automatically?

I think that only minor clarifications are needed in the documentation as it is already quite good. It could possibly underline that any changes to DNs for already mapped entities would break the mapping, not just deleting entities. Depending on how you suggest to repopulate purged specific entries, maybe also some words about that if you feel it's necessary.

Suppose this is a bug for python-keystoneclient, but it would also be nice if it was able to still list the users in a project even if one or more users have invalid mappings. Currently it breaks ungracefully, even if it's able to request information about most users.

[1]: http://docs.openstack.org/developer/keystone/configuration.html#domain-specific-drivers
[2]: https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/cmd/cli.py#L1239

Revision history for this message

Lance Bragstad (lbragstad) wrote on 2017-01-24:

#3

I did some digging and this is what I found. The `keystone-manage mapping_populate` command is simply a wrapper that makes a call to list all users [0], since the responsibility to map local and public ID falls within the various layers of the identity API. `keystone-manage` only passes the domain ID of the domain to populate, which in turn is passed to the identity APIs list users method [1]. This is all pretty straight forward, and before we exit that method, we have a list of references [2] from LDAP (in the multi-domain case). This is passed to _set_domain_id_and_mapping() [3], which handles parsing each entity in the list. At the same time, it looks up all current id_mappings (one should exist for every user) and compares them to the list of existing entities. If a reference, recently pulled from LDAP, matches a mapping from the id_mapping table, we pull the public id from the id_mapping and update the reference [4]. At this point, any entity that has already been mapped has its corresponding public ID. Then we make another pass through the list to populate any public ID for entities that haven't been mapped (i.e. these would be the ones that you just explicitly removed using the `keystone-manage mapping_purge`` command) [5][6]. If there isn't a public ID in the entity, we will go ahead and create one from attributes [7].

So, to answer your question. You should be able to remove explicit mappings with the ``keystone-manage mapping_purge`` command, and when you go to repopulate them, existing reference should maintain their original public IDs.

As far as the python-keystoneclient bug, do you have a trace, or are you referencing the one from the description (which would be a server issue I believe)?

Hopefully this helps.

I did some digging and this is what I found. The `keystone-manage mapping_populate` command is simply a wrapper that makes a call to list all users [0], since the responsibility to map local and public ID falls within the various layers of the identity API. `keystone-manage` only passes the domain ID of the domain to populate, which in turn is passed to the identity APIs list users method [1]. This is all pretty straight forward, and before we exit that method, we have a list of references [2] from LDAP (in the multi-domain case). This is passed to _set_domain_id_and_mapping() [3], which handles parsing each entity in the list. At the same time, it looks up all current id_mappings (one should exist for every user) and compares them to the list of existing entities. If a reference, recently pulled from LDAP, matches a mapping from the id_mapping table, we pull the public id from the id_mapping and update the reference [4]. At this point, any entity that has already been mapped has its corresponding public ID. Then we make another pass through the list to populate any public ID for entities that haven't been mapped (i.e. these would be the ones that you just explicitly removed using the `keystone-manage mapping_purge`` command) [5][6]. If there isn't a public ID in the entity, we will go ahead and create one from attributes [7].

So, to answer your question. You should be able to remove explicit mappings with the ``keystone-manage mapping_purge`` command, and when you go to repopulate them, existing reference should maintain their original public IDs.

As far as the python-keystoneclient bug, do you have a trace, or are you referencing the one from the description (which would be a server issue I believe)?

Hopefully this helps.

[0] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/cmd/cli.py#L1258
[1] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L963
[2] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L975
[3] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L522
[4] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L648
[5] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L654-L659
[6] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L580-L588
[7] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/mapping_backends/sql.py#L74-L76

Revision history for this message

Tim S (timss) wrote on 2017-01-30:

#4

Download full text (5.6 KiB)

Great explaination, and thanks for looking into this.

Unfortunately I've not been able to verify if a purge was successful yet due to residing role assignment issues. Using LDAP, should the local ID be "<email address hidden>" (typically capital letters) or just "USERNAME"?

Listing users still yield the error described below. Trying to remove the role assignment using the user's ID (`openstack role remove`) fails for the same reason. Could it be that the mapping is purged successfully, but the role assignment remains in a sort stale state? In that case, how should I proceed?

---

Regarding the possible python-keystoneclient bug I am refering to the one included in part in the bug description. I'm using python-openstackclient, but if I'm not mistaken these operations are a part of python-keystoneclient[1]? Chose to initially not include too much of the trace due to its verbosity and because it might be a seperate bug.

In short, when users are listed with `openstack user list --project PROJECT` or similar, the command seems to issue these API requests but fails at step 4 since it can't find an user with the ID referenced in the role assignments.

1. Authenticate/issue token (if applicable)
2. Get details about the project via `/v3/projects?name=PROJECT` (or by ID)
3. Using now acquired project ID, it asks for the role assignments in the scope of the project via `/v3/role_assignments?scope.project.id=ID`
4. Iterate through and get info about each user found in role assignments via `/v3/users/ID`
5. Print (list) users in the project

Step 4 and out for the user not found (`openstack user list --debug --project PROJECT --long`):

---

REQ: curl -g -i -X GET https://domain.com:35357/v3/users/<ID> -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/users/<ID> HTTP/1.1" 404 89
RESP: [404] Content-Length: 89 Vary: X-Auth-Token Keep-Alive: timeout=5, max=89 Server: Apache/2.4.18 (Ubuntu) Connection: Keep-Alive Date: Fri, 20 Jan 2017 13:54:19 GMT x-openstack-request-id: req-<ID> Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"error": {"message": "Could not find user: <username>", "code": 404, "title": "Not Found"}}

Request returned failure status: 404
REQ: curl -g -i -X GET https://domain.com:35357/v3/users?name=<ID> -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/users?name=<ID> HTTP/1.1" 401 114
RESP: [401] Content-Length: 114 Vary: X-Auth-Token Keep-Alive: timeout=5, max=88 X-Distribution: Ubuntu Connection: Keep-Alive Date: Fri, 20 Jan 2017 13:54:19 GMT WWW-Authenticate: Keystone uri="https://domain.com:35357" Server: Apache/2.4.18 (Ubuntu) Content-Type: application/json x-openstack-request-id: req-<ID>
RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}

Request returned failure status 401
REQ: curl -g -i -X GET https://domain.com:35357/v3/users -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/users HTTP/1.1" 401 114
RESP: [401] Content-Length: 114 Vary: X-Auth-Token Keep-Al...

Great explaination, and thanks for looking into this.

Unfortunately I've not been able to verify if a purge was successful yet due to residing role assignment issues. Using LDAP, should the local ID be "USERNAME@DOMAIN.COM" (typically capital letters) or just "USERNAME"?

Listing users still yield the error described below. Trying to remove the role assignment using the user's ID (`openstack role remove`) fails for the same reason. Could it be that the mapping is purged successfully, but the role assignment remains in a sort stale state? In that case, how should I proceed?

---

Regarding the possible python-keystoneclient bug I am refering to the one included in part in the bug description. I'm using python-openstackclient, but if I'm not mistaken these operations are a part of python-keystoneclient[1]? Chose to initially not include too much of the trace due to its verbosity and because it might be a seperate bug.

In short, when users are listed with `openstack user list --project PROJECT` or similar, the command seems to issue these API requests but fails at step 4 since it can't find an user with the ID referenced in the role assignments.

1. Authenticate/issue token (if applicable)
2. Get details about the project via `/v3/projects?name=PROJECT` (or by ID)
3. Using now acquired project ID, it asks for the role assignments in the scope of the project via `/v3/role_assignments?scope.project.id=ID`
4. Iterate through and get info about each user found in role assignments via `/v3/users/ID`
5. Print (list) users in the project

Step 4 and out for the user not found (`openstack user list --debug --project PROJECT --long`):

---

REQ: curl -g -i -X GET https://domain.com:35357/v3/users/<ID> -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/users/<ID> HTTP/1.1" 404 89
RESP: [404] Content-Length: 89 Vary: X-Auth-Token Keep-Alive: timeout=5, max=89 Server: Apache/2.4.18 (Ubuntu) Connection: Keep-Alive Date: Fri, 20 Jan 2017 13:54:19 GMT x-openstack-request-id: req-<ID> Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"error": {"message": "Could not find user: <username>", "code": 404, "title": "Not Found"}}

Request returned failure status: 404
REQ: curl -g -i -X GET https://domain.com:35357/v3/users?name=<ID> -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/users?name=<ID> HTTP/1.1" 401 114
RESP: [401] Content-Length: 114 Vary: X-Auth-Token Keep-Alive: timeout=5, max=88 X-Distribution: Ubuntu Connection: Keep-Alive Date: Fri, 20 Jan 2017 13:54:19 GMT WWW-Authenticate: Keystone uri="https://domain.com:35357" Server: Apache/2.4.18 (Ubuntu) Content-Type: application/json x-openstack-request-id: req-<ID>
RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}

Request returned failure status 401
REQ: curl -g -i -X GET https://domain.com:35357/v3/users -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/users HTTP/1.1" 401 114
RESP: [401] Content-Length: 114 Vary: X-Auth-Token Keep-Alive: timeout=5, max=87 X-Distribution: Ubuntu Connection: Keep-Alive Date: Fri, 20 Jan 2017 13:54:19 GMT WWW-Authenticate: Keystone uri="https://domain.com:35357" Server: Apache/2.4.18 (Ubuntu) Content-Type: application/json x-openstack-request-id: req-<ID>
RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}

Request returned failure status: 401
Could not find resource <ID>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/common/command.py", line 38, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 92, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/identity/v3/user.py", line 239, in take_action
    user = utils.find_resource(identity_client.users, user_id)
  File "/usr/lib/python2.7/dist-packages/openstackclient/common/utils.py", line 118, in find_resource
    raise exceptions.CommandError(msg)
CommandError: Could not find resource <ID>
clean_up ListUser: Could not find resource <ID>

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/openstackclient/shell.py", line 118, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 255, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/dist-packages/openstackclient/shell.py", line 153, in run_subcommand
    ret_value = super(OpenStackShell, self).run:_subcommand(argv)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/common/command.py", line 38, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 92, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/identity/v3/user.py", line 239, in take_action
    user = utils.find_resource(identity_client.users, user_id)
  File "/usr/lib/python2.7/dist-packages/openstackclient/common/utils.py", line 118, in find_resource
    raise exceptions.CommandError(msg)
CommandError: Could not find resource <ID>

---

[1]: http://docs.openstack.org/developer/python-openstackclient/authentication.html

Revision history for this message

Tim S (timss) wrote on 2017-01-30:

#5

Didn't realize how unreadable launchpad made the trace, so here's a paste[1] of the `--debug` output instead. Again, everything is normal up to this point (see steps), so I chose to leave that out for now.

[1]: http://paste.openstack.org/show/596876/

Revision history for this message

Matthew Edmonds (edmondsw) wrote on 2017-02-07:

#6

based on the config you have in the description, I believe a user's local_id would be that user's sAMAccountName.

Revision history for this message

Andreas Karis (akaris) wrote on 2017-06-30:

#7

Download full text (7.6 KiB)

Hi,

The same happens if users are deleted or their group membership is changed.

Users that are auto removed from LDAP break roles and cant easily be removed.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Followed: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/9/html/integrate_with_identity_service/sec-active-directory

Hi,

The same happens if users are deleted or their group membership is changed.

Users that are auto removed from LDAP break roles and cant easily be removed.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Followed: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/9/html/integrate_with_identity_service/sec-active-directory

User test was removed from Active Directory:
~~~
[stack@undercloud-6 ~]$ openstack user list --domain redhat+------------------------------------------------------------------+----------+
| ID                                                               | Name     |
+------------------------------------------------------------------+----------+
| 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90 | svc-ldap |
| 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | akaris   |
| 39e5b866156f05d6b3f95409a663a44718bec62eeabc9ec6f08ff78ef5fd457d | nalmond  |
[stack@undercloud-6 ~]$ openstack role assignment list | head -2
+----------------------------------+------------------------------------------------------------------+-------+----------------------------------+----------------------------------+-----------+
| Role                             | User                                                             | Group | Project                          | Domain                           | Inherited |
[stack@undercloud-6 ~]$ openstack role assignment list | grep 1c3e304811d8457a871a6c67f6f63a75
| 9fe2ff9ee4384b1894a90878d3e92bab | f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 |       | 1c3e304811d8457a871a6c67f6f63a75 |                                  | False     |
[stack@undercloud-6 ~]$ openstack role remove --project demo --user f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 9fe2ff9ee4384b1894a90878d3e92bab
No user with a name or ID of 'f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2' exists.
~~~

The role assignment cannot be removed:
~~~
[stack@undercloud-6 ~]$ openstack role remove --project demo --user 1c3e304811d8457a871a6c67f6f63a75 _member_
No user with a name or ID of '1c3e304811d8457a871a6c67f6f63a75' exists.
~~~

The user cannot be deleted:
~~~
[stack@undercloud-6 ~]$ openstack user delete f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
No user with a name or ID of 'f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2' exists.
~~~

Revision history for this message

Andreas Karis (akaris) wrote on 2017-06-30:

#8

Download full text (6.9 KiB)

Looking at the database, the id_mapping table and the assignment table seem to be the only ones holding references to this user (given the test which I ran, meaning that I assigned a role to the user)
~~~
[root@overcloud-controller-0 domains]# mysql keystone -e 'show tables;' | awk '{print $1}' | while read t;do echo "XXXXX $t XXXXX"; mysql keystone -e "select * from $t \G" | grep f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 -C5 ; done
XXXXX Tables_in_keystone XXXXX
ERROR 1146 (42S02) at line 1: Table 'keystone.Tables_in_keystone' doesn't exist
XXXXX access_token XXXXX
XXXXX assignment XXXXX
target_id: dfc01178c51b4688be78188b5e8c9581
  role_id: 9fe2ff9ee4384b1894a90878d3e92bab
inherited: 0
*************************** 22. row ***************************
     type: UserProject
actor_id: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
target_id: 1c3e304811d8457a871a6c67f6f63a75
  role_id: 9fe2ff9ee4384b1894a90878d3e92bab
inherited: 0
*************************** 23. row ***************************
     type: UserProject
XXXXX config_register XXXXX
XXXXX consumer XXXXX
XXXXX credential XXXXX
XXXXX domain XXXXX
XXXXX endpoint XXXXX
XXXXX endpoint_group XXXXX
XXXXX federated_user XXXXX
XXXXX federation_protocol XXXXX
XXXXX group XXXXX
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'group' at line 1
XXXXX id_mapping XXXXX
  public_id: 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90
  domain_id: 210bdf7974a14693843ec7f9b1956105
   local_id: svc-ldap
entity_type: user
*************************** 4. row ***************************
  public_id: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
  domain_id: 210bdf7974a14693843ec7f9b1956105
   local_id: test
entity_type: user
XXXXX identity_provider XXXXX
XXXXX idp_remote_ids XXXXX
XXXXX implied_role XXXXX
XXXXX local_user XXXXX
XXXXX mapping XXXXX
XXXXX migrate_version XXXXX
XXXXX password XXXXX
XXXXX policy XXXXX
XXXXX policy_association XXXXX
XXXXX project XXXXX
XXXXX project_endpoint XXXXX
XXXXX project_endpoint_group XXXXX
XXXXX region XXXXX
XXXXX request_token XXXXX
XXXXX revocation_event XXXXX
XXXXX role XXXXX
XXXXX sensitive_config XXXXX
XXXXX service XXXXX
XXXXX service_provider XXXXX
XXXXX token XXXXX
XXXXX trust XXXXX
XXXXX trust_role XXXXX
XXXXX user XXXXX
XXXXX user_group_membership XXXXX
XXXXX whitelisted_config XXXXX
~~~

In order to clean up the mapping:
~~~
[root@overcloud-controller-0 domains]# mysql keystone -e 'delete from assignment where actor_id="f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2"'
[root@overcloud-controller-0 domains]#
~~~

After this:
~~~
[stack@undercloud-6 ~]$ openstack role assignment list | grep f3f3e1b1c01c79299154f85f...

Looking at the database, the id_mapping table and the assignment table seem to be the only ones holding references to this user (given the test which I ran, meaning that I assigned a role to the user)
~~~
[root@overcloud-controller-0 domains]# mysql keystone -e 'show tables;' | awk '{print $1}' | while read t;do echo "XXXXX $t XXXXX";  mysql keystone -e "select * from $t \G" | grep f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 -C5 ; done
XXXXX Tables_in_keystone XXXXX
ERROR 1146 (42S02) at line 1: Table 'keystone.Tables_in_keystone' doesn't exist
XXXXX access_token XXXXX
XXXXX assignment XXXXX
target_id: dfc01178c51b4688be78188b5e8c9581
  role_id: 9fe2ff9ee4384b1894a90878d3e92bab
inherited: 0
*************************** 22. row ***************************
     type: UserProject
 actor_id: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
target_id: 1c3e304811d8457a871a6c67f6f63a75
  role_id: 9fe2ff9ee4384b1894a90878d3e92bab
inherited: 0
*************************** 23. row ***************************
     type: UserProject
XXXXX config_register XXXXX
XXXXX consumer XXXXX
XXXXX credential XXXXX
XXXXX domain XXXXX
XXXXX endpoint XXXXX
XXXXX endpoint_group XXXXX
XXXXX federated_user XXXXX
XXXXX federation_protocol XXXXX
XXXXX group XXXXX
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'group' at line 1
XXXXX id_mapping XXXXX
  public_id: 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90
  domain_id: 210bdf7974a14693843ec7f9b1956105
   local_id: svc-ldap
entity_type: user
*************************** 4. row ***************************
  public_id: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
  domain_id: 210bdf7974a14693843ec7f9b1956105
   local_id: test
entity_type: user
XXXXX identity_provider XXXXX
XXXXX idp_remote_ids XXXXX
XXXXX implied_role XXXXX
XXXXX local_user XXXXX
XXXXX mapping XXXXX
XXXXX migrate_version XXXXX
XXXXX password XXXXX
XXXXX policy XXXXX
XXXXX policy_association XXXXX
XXXXX project XXXXX
XXXXX project_endpoint XXXXX
XXXXX project_endpoint_group XXXXX
XXXXX region XXXXX
XXXXX request_token XXXXX
XXXXX revocation_event XXXXX
XXXXX role XXXXX
XXXXX sensitive_config XXXXX
XXXXX service XXXXX
XXXXX service_provider XXXXX
XXXXX token XXXXX
XXXXX trust XXXXX
XXXXX trust_role XXXXX
XXXXX user XXXXX
XXXXX user_group_membership XXXXX
XXXXX whitelisted_config XXXXX
~~~

In order to clean up the mapping:
~~~
[root@overcloud-controller-0 domains]# mysql keystone -e 'delete from assignment where actor_id="f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2"'
[root@overcloud-controller-0 domains]# 
~~~

Note that I'm not saying that you should go into the database and execute this in a production environment. I took this from a lab and this is an approach that you might use, after testing it in a lab and making sure that this won't affect your prod. In case of doubt, do not manipulate the database, and always take a backup before with mysqldump or mysqldump --all-databases.

Revision history for this message

Andreas Karis (akaris) wrote on 2017-06-30:

#9

Note that I recreated user test in AD:

Output from today:
~~~
[stack@undercloud-6 ~]$ openstack user list --domain redhat
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90 | svc-ldap |
| 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | akaris |
| 39e5b866156f05d6b3f95409a663a44718bec62eeabc9ec6f08ff78ef5fd457d | nalmond |
| f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | test |
+------------------------------------------------------------------+----------+
~~~

Output from yesterday:
~~~
[stack@undercloud-6 ~]$ openstack user list --domain redhat
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90 | svc-ldap |
| 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | akaris |
| 39e5b866156f05d6b3f95409a663a44718bec62eeabc9ec6f08ff78ef5fd457d | nalmond |
| f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | test |
+------------------------------------------------------------------+----------+
~~~

"The public ID is automatically generated by keystone when it first encounters the entity. If the local ID of the entity is from a backend that does not guarantee to generate UUIDs, a hash algorithm will generate a public ID for that entity, which is what will be exposed by keystone."

The interesting observation here is that UUIDs are the same, so by recreating a deleted user with the same user name (sAMAccountname ???) one could "fix" keystone. Just a quick observation.

Revision history for this message

Andreas Karis (akaris) wrote on 2017-06-30:

#10

Download full text (4.7 KiB)

The same issue by the way happens when a user's membership is changed and thus removes him from the domain:

The same issue by the way happens when a user's membership is changed and thus removes him from the domain:

Disabling an account in Active Directory keeps the user in the user list and hence does not have the same effect. Everything is o.k. in this case.

Revision history for this message

Andreas Karis (akaris) wrote on 2017-06-30:

#11

Download full text (5.1 KiB)

I deleted the user to test the mapping_purge ...

~~~
[stack@undercloud-6 ~]$ openstack user list --domain redhat
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90 | svc-ldap |
| 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | akaris |
| 39e5b866156f05d6b3f95409a663a44718bec62eeabc9ec6f08ff78ef5fd457d | nalmond |
+------------------------------------------------------------------+----------+
[stack@undercloud-6 ~]$ openstack role assignment list | grep 1c3e304811d8457a871a6c67f6f63a75
| 9fe2ff9ee4384b1894a90878d3e92bab | 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | | 1c3e304811d8457a871a6c67f6f63a75 | | False |
| 9fe2ff9ee4384b1894a90878d3e92bab | f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | | 1c3e304811d8457a871a6c67f6f63a75 | | False |
[stack@undercloud-6 ~]$ openstack role assignment list --names
Could not find user: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 (HTTP 404) (Request-ID: req-dacdaa34-d07c-40f7-ac7c-2864ca6d6608)
~~~

Note that this does not work:

~~~
[root@overcloud-controller-0 ~]# keystone-manage mapping_purge --domain-name redhat
[root@overcloud-controller-0 ~]#
~~~

~~~
[stack@undercloud-6 ~]$ openstack role assignment list | grep 1c3e304811d8457a871a6c67f6f63a75
| 9fe2ff9ee4384b1894a90878d3e92bab | 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | | 1c3e304811d8457a871a6c67f6f63a75 | | False |
| 9fe2ff9ee4384b1894a90878d3e92bab | f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | | 1c3e304811d8457a871a6c67f6f63a75 | | False |
[stack@undercloud-6 ~]$ openstack role assignment list --names
Could not find user: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 (HTTP 404) (Request-ID: req-e53543a3-2164-4fa1-a86b-55a38d199d57)
~~~

I also tried `keystone-manage mapping_purge --domain-name redhat --local-id f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2`

Yes,. it purges the mapping table, but it does not touch the assignment table:
~~~
[root@overcloud-controller-0 ~]# mysql keystone -e 'show tables;' | awk '{print $1}' | while read t;do echo "XXXXX $t XXXXX"; mysql keystone -e "select * from $t \G" | grep f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 -C5 ; done
XXXXX Tables_in_keystone XXXXX
ERROR 1146 (42S02) at line 1: Table 'keystone.Tables_in_keystone' doesn't exist
XXXXX access_token XXXXX
XXXXX assignment XXXXX
target_id: dfc01178c51b4688be78188b5e8c9581
  role_id: 9fe2ff9ee4384b1894a90878d3e92bab
inherited: 0
*************************** 23. row ***************************
     type: UserProject
actor_id: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
target_id: 1c3e304811d8457a871a6c67f6f63a75
  role_id: 9fe...

I deleted the user to test the mapping_purge ...

~~~
[stack@undercloud-6 ~]$ openstack user list --domain redhat
+------------------------------------------------------------------+----------+
| ID                                                               | Name     |
+------------------------------------------------------------------+----------+
| 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90 | svc-ldap |
| 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | akaris   |
| 39e5b866156f05d6b3f95409a663a44718bec62eeabc9ec6f08ff78ef5fd457d | nalmond  |
+------------------------------------------------------------------+----------+
[stack@undercloud-6 ~]$ openstack role assignment list | grep 1c3e304811d8457a871a6c67f6f63a75
| 9fe2ff9ee4384b1894a90878d3e92bab | 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 |       | 1c3e304811d8457a871a6c67f6f63a75 |                                  | False     |
| 9fe2ff9ee4384b1894a90878d3e92bab | f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 |       | 1c3e304811d8457a871a6c67f6f63a75 |                                  | False     |
[stack@undercloud-6 ~]$ openstack role assignment list --names
Could not find user: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 (HTTP 404) (Request-ID: req-dacdaa34-d07c-40f7-ac7c-2864ca6d6608)
~~~

Note that this does not work:

~~~
[root@overcloud-controller-0 ~]# keystone-manage mapping_purge --domain-name redhat
[root@overcloud-controller-0 ~]# 
~~~

~~~
[stack@undercloud-6 ~]$ openstack role assignment list | grep 1c3e304811d8457a871a6c67f6f63a75
| 9fe2ff9ee4384b1894a90878d3e92bab | 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 |       | 1c3e304811d8457a871a6c67f6f63a75 |                                  | False     |
| 9fe2ff9ee4384b1894a90878d3e92bab | f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 |       | 1c3e304811d8457a871a6c67f6f63a75 |                                  | False     |
[stack@undercloud-6 ~]$ openstack role assignment list --names
Could not find user: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 (HTTP 404) (Request-ID: req-e53543a3-2164-4fa1-a86b-55a38d199d57)
~~~

I also tried `keystone-manage mapping_purge --domain-name redhat --local-id f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2`

Yes,. it purges the mapping table, but it does not touch the assignment table:
~~~
[root@overcloud-controller-0 ~]#  mysql keystone -e 'show tables;' | awk '{print $1}' | while read t;do echo "XXXXX $t XXXXX";  mysql keystone -e "select * from $t \G" | grep f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 -C5 ; done
XXXXX Tables_in_keystone XXXXX
ERROR 1146 (42S02) at line 1: Table 'keystone.Tables_in_keystone' doesn't exist
XXXXX access_token XXXXX
XXXXX assignment XXXXX
target_id: dfc01178c51b4688be78188b5e8c9581
  role_id: 9fe2ff9ee4384b1894a90878d3e92bab
inherited: 0
*************************** 23. row ***************************
     type: UserProject
 actor_id: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
target_id: 1c3e304811d8457a871a6c67f6f63a75
  role_id: 9fe2ff9ee4384b1894a90878d3e92bab
inherited: 0
*************************** 24. row ***************************
     type: UserProject
XXXXX config_register XXXXX
XXXXX consumer XXXXX
XXXXX credential XXXXX
XXXXX domain XXXXX
XXXXX endpoint XXXXX
XXXXX endpoint_group XXXXX
XXXXX federated_user XXXXX
XXXXX federation_protocol XXXXX
XXXXX group XXXXX
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'group' at line 1
XXXXX id_mapping XXXXX
XXXXX identity_provider XXXXX
XXXXX idp_remote_ids XXXXX
XXXXX implied_role XXXXX
XXXXX local_user XXXXX
XXXXX mapping XXXXX
XXXXX migrate_version XXXXX
XXXXX password XXXXX
XXXXX policy XXXXX
XXXXX policy_association XXXXX
XXXXX project XXXXX
XXXXX project_endpoint XXXXX
XXXXX project_endpoint_group XXXXX
XXXXX region XXXXX
XXXXX request_token XXXXX
XXXXX revocation_event XXXXX
XXXXX role XXXXX
XXXXX sensitive_config XXXXX
XXXXX service XXXXX
XXXXX service_provider XXXXX
XXXXX token XXXXX
XXXXX trust XXXXX
XXXXX trust_role XXXXX
XXXXX user XXXXX
XXXXX user_group_membership XXXXX
XXXXX whitelisted_config XXXXX
[root@overcloud-controller-0 ~]# 
~~~

And role deletion still fails:
~~~
[stack@undercloud-6 ~]$ openstack role remove --project demo --user f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 9fe2ff9ee4384b1894a90878d3e92bab
No user with a name or ID of 'f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2' exists.
[stack@undercloud-6 ~]$ openstack role assignment list | grep 1c3e304811d8457a871a6c67f6f63a75
| 9fe2ff9ee4384b1894a90878d3e92bab | 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 |       | 1c3e304811d8457a871a6c67f6f63a75 |                                  | False     |
| 9fe2ff9ee4384b1894a90878d3e92bab | f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 |       | 1c3e304811d8457a871a6c67f6f63a75 |                                  | False     |
~~~

Revision history for this message

Andreas Karis (akaris) wrote on 2017-06-30:

#12

Download full text (5.0 KiB)

Same issue in Ocata.

Revision history for this message

Tim S (timss) wrote on 2017-06-30:

#13

Thanks for sharing your experiences and tests, @akaris. This is very useful!
Especially the pointers on how to delete the entries manually from the databse and its effect.

Unfortunately I haven't been able to look further into this issue in a while, although it's still a problem in the Keystone installation I'm running. I'd like to put some more time into resolving it, but not sure when as it's been too easy to put into the backlog for now.

Anyway, thanks for the input and possibly confirming the bug.

Revision history for this message

Lance Bragstad (lbragstad) wrote on 2017-07-20:

#14

It sounds like a possible solution would be to enhance the ``keystone-manage`` tool. What we could do is add a path to it that accepts a user_id, check to make sure it doesn't exist in the identity backend, and then forcibly removes any role assignments that user has.

Changed in keystone:
status:	New → Triaged
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-26: Fix proposed to keystone (master)

#15

Fix proposed to branch: master
Review: https://review.openstack.org/487579

Changed in keystone:
assignee:	nobody → Kristi Nikolla (knikolla)
status:	Triaged → In Progress

Colleen Murphy (krinkle) on 2018-01-24

Changed in keystone:
milestone:	none → queens-rc1

Revision history for this message

Lance Bragstad (lbragstad) wrote on 2018-02-08:

#16

We're probably going to have to cut queens-rc1 without this fix. If we get a patch landed we can backport and cut an rc2.

Lance Bragstad (lbragstad) on 2018-02-09

Changed in keystone:
milestone:	queens-rc1 → queens-rc2

Revision history for this message

Lance Bragstad (lbragstad) wrote on 2018-02-09:

#17

Thinking about this a bit more and it seems we have two different options when using the mapping purge.

1.) We can have keystone-manage mapping_purge accept an argument for a specific user ID. If that user id isn't in a specific domain backend, then it should be removed from the id_mapping table, similiar to what happens today. In addition to the path, keystone could remove the role assignment for that ID if they exist in the assignment tables.

2.) We can add another argument to the mapping_purge command that forces the removal of all assignments associated to purged identities (essentially option #1 but batch processing).

My concern for both approach is that id_mappings can be regenerated because keystone pulls the identity from the domain backend (LDAP/AD/etc...), which it treats as the source of truth. Since it's possible for users to have assignments, when we remove them, there is no recovery option if the users is enabled or recreated in the domain backend. In a way, this makes mapping_purge less idempotent because;

- user is create in ldap
- user is mapped in keystone and given an id based on a hash
- user is granted assignments to various projects
- user is disabled or moves groups in ldap
- the users id and mapping are purged in keystone, along with all of the user's assignments
- the user is enabled or moved back to the original group, the mapping backend maps the user to their previous ID
- user doesn't have any of the assignment they had prior to the purge operation

Is this going to be an issue?

Revision history for this message

Kristi Nikolla (knikolla) wrote on 2018-02-12:

#18

Played around for a bit.

1. Created a user in ldap
2. Listed users in the domain Users, new user correctly showed
3. Assigned Member role to that user on project demo
4. Deleted the user directly in ldap
5. Cleaned mappings and restarted keystone to clear cache

http://paste.openstack.org/show/670540/

In the above paste, only the last command unexpectedly fails. That is because the `openstack user list --project <project>` command does the following:

1. It queries the API /role_assignment?scope.project.id=<project> # Works
2. From the list returned above it does /users?name=<user_id> # Works
3. For a user which doesn't exist, the above command will return an empty list # Works
4. If the above call returned an empty list, the openstackclient will throw an exception.

Revision history for this message

Adam Young (ayoung) wrote on 2018-02-13:

#19

Sounds like it is a valid return code and message and an error on the part of the client code, then. Should we refile the bug against OSC?

Revision history for this message

Lance Bragstad (lbragstad) wrote on 2018-02-13:

#20

Removing this as a queens rc target based on comment number #18. Kristi, thanks for recreating this.

Changed in keystone:
milestone:	queens-rc2 → none

Lance Bragstad (lbragstad) on 2018-06-25

tags:

added: office-hours

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-19: Change abandoned on keystone (master)

#21

Change abandoned by Kristi Nikolla (<email address hidden>) on branch: master
Review: https://review.openstack.org/487579
Reason: Won't have time to work on this.

Revision history for this message

Lars Erik Pedersen (pedersen-larserik) wrote on 2020-01-15:

#22

Can someone have another look on this? It is still a problem in stein. While 'mapping_purge' can remove a stale user, it does not remove the role assignments. And; there is no other way to remove role assignments from a stale user, than directly in the database..

Colleen Murphy (krinkle) on 2020-02-05

Changed in keystone:
status:	In Progress → Triaged

Revision history for this message

Tim S (timss) wrote on 2021-08-06:

#23

Although possibly susceptible to the same bug, but for those facing the same/similar issue it might be worth considering assigning/granting roles in projects to whole LDAP groups rather than each user (member) in a LDAP group. This can help simplify managing membership changes, and its position in the LDAP tree (DIT) is hopefully rather stable for most organizations, compared to user objects.

For those using AD, Keystone also has good support for traversing nested group structures using the `group_ad_nesting` option, ability to map attributes and other things you might need.

Adding groups is also simple, and if creating 1:1 project to LDAP group is your primary use case, removes the need for looking up membership status before adding users, effectively skipping a step.

$ openstack role add --project-domain default --project myproject --group-domain mydomain --group mygroup member

OpenStack Identity (keystone)

Moving/disabling LDAP users break Keystone queries depending on role ID

Bug Description

Other bug subscribers

Remote bug watches