Slow listing projects for user with many role assignments

Bug #1700852 reported by Steve McLellan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Medium
Lance Bragstad
Newton
Won't Fix
Undecided
Unassigned
Ocata
Confirmed
Medium
Unassigned

Bug Description

With a large number of role assignments (e.g 500) it becomes very slow to list the projects a user has access to (via /users/<id>/projects). I'm seeing times of around 4 seconds versus 0.1 for a user with a couple of assignments.

Instrumenting list_projects_for_user (https://github.com/openstack/keystone/blob/stable/newton/keystone/assignment/core.py#L268), where each number is the time elapsed since the start of list_projects_for_user, I get:

  list_projects_for_user 3.998 role_assignments
  list_projects_for_user 3.999 project_ids
  list_projects_for_user 4.105 list_projects

The time is spent on the call made to list_role_assignments on line 269 which in turns calls _list_effective_role_assignments at https://github.com/openstack/keystone/blob/stable/newton/keystone/assignment/core.py#L986.

Listing role assignments for a user directly (with GET /role_assignments?user.id=<id>) is very fast, which is further indication that something in the effective role processing is the cause. I haven't yet timed the internal of _list_effective_role_assignments.

Running keystone mitaka (though I believe this would apply to master) with users in LDAP but roles and projects managed by keystone.

Revision history for this message
Lance Bragstad (lbragstad) wrote :

This seems like something we should be able to fix and backport to Ocata since it has an impact on performance.

tags: added: performance
Changed in keystone:
status: New → Confirmed
importance: Undecided → Medium
tags: added: ocata-backport-potential
Changed in keystone:
milestone: none → pike-3
Steve McLellan (sjmc7)
description: updated
tags: added: office-hours
Revision history for this message
Steve McLellan (sjmc7) wrote :

Looks like the slow code here is strip_domain_roles:

    def _strip_domain_roles(self, role_refs):
        """Post process assignment list for domain roles.

        Domain roles are only designed to do the job of inferring other roles
        and since that has been done before this method is called, we need to
        remove any assignments that include a domain role.

        """
        def _role_is_global(role_id):
            ref = self.role_api.get_role(role_id)
            return (ref['domain_id'] is None)

        filter_results = []
        for ref in role_refs:
            if _role_is_global(ref['role_id']):
                filter_results.append(ref)
        return filter_results

With caching the time would probably go away, but as it is, it's making a DB lookup for each role.

Changed in keystone:
milestone: pike-3 → pike-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/487143

Changed in keystone:
assignee: nobody → Lance Bragstad (lbragstad)
status: Confirmed → In Progress
Revision history for this message
Lance Bragstad (lbragstad) wrote :

Steve, I've proposed a patch that implements caching for the API in question [0]. Let me know if that helps in any way. It might end up being a work-around until we can get some patches proposed to speed up the assignment API.

[0] https://review.openstack.org/#/c/487143/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to keystone (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/491921

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to keystone (master)

Reviewed: https://review.openstack.org/491921
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=9fccd38d1b177664c64bbf987a8c31d06d7cfc60
Submitter: Jenkins
Branch: master

commit 9fccd38d1b177664c64bbf987a8c31d06d7cfc60
Author: Lance Bragstad <email address hidden>
Date: Tue Aug 8 20:37:10 2017 +0000

    Remove unused hints from assignment APIs

    The controller is responsible for listing user projects based on role
    assignments and would build a hints objects and pass it to the
    assignment manager. This is a common pattern used across keystone's
    APIs. But, the assignment API never actually passed the hints objects
    to the backend implementation.

    This commit removes the hints from being passed to the manager for
    list_projects_for_user and list_domains_for_user because those
    APIs never use the hints object. This should allow us to implement
    caching to speed up those calls later.

    Change-Id: I9b1c8c30ca6a78dd6e78add7de278e467ceea046
    Related-Bug: 1700852

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/487143
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=63124f703a81074793360c1b91711b6ee5a76196
Submitter: Jenkins
Branch: master

commit 63124f703a81074793360c1b91711b6ee5a76196
Author: Lance Bragstad <email address hidden>
Date: Tue Jul 25 17:03:55 2017 +0000

    Cache list projects and domains for user

    Listing projects and domains for a user based on their role
    assignments was noted as being really slow, especially when users
    have a lot of assignments. This commit implements caching to mitigate
    the issue while we continue to investigate ways to speed up the
    assignment API.

    Change-Id: I72e398c65f01aa4f9a37f817d184a13ed01089ce
    Closes-Bug: 1700852

Changed in keystone:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 12.0.0.0rc1

This issue was fixed in the openstack/keystone 12.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.