tripleo

keystone kvs driver causes process to grow indefinitely and spin on CPU with thousands of keys in a single python dict

Bug #1188301 reported by Clint Byrum
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Medium
Joe Gordon
OpenStack Identity (keystone) 2013.2 "havana"
tripleo
Fix Released
Critical
Clint Byrum

Bug Description

After about 10 days of being up, things got sluggish in our proof of concept rack.

The kvs middleware for tokens is basically just for testing. It is in-process and there is no persistence, and no garbage collection or memory limit at all.

We should use sql if we want to persist tokens, or memcache if we are fine with losing some.

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 6955 keystone 20 0 3255m 3.0g 5640 S 92 3.2 6532:10 keystone-all

Robert Collins (lifeless)
Changed in tripleo:
status: Triaged → Fix Released
assignee: nobody → Clint Byrum (clint-fewbar)
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-image-elements (master)

Reviewed: https://review.openstack.org/32970
Committed: http://github.com/stackforge/tripleo-image-elements/commit/8fa5669f4984503f62c87ab5f0b797dbf8e0f416
Submitter: Jenkins
Branch: master

commit 8fa5669f4984503f62c87ab5f0b797dbf8e0f416
Author: Clint Byrum <email address hidden>
Date: Thu Jun 13 15:30:30 2013 -0700

    Set keystone token backend to sql.

    Because the table could grow forever, also add a cron job to flush tokens
    every two hours.

    Due to bug #1188378 we need to use percona-toolkit's pt-archiver to
    clean out the table on mysql.

    Fixes bug #1188301

    Change-Id: I90aac347993ee741231fbfb99a32dc6e3f3a75ec

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/34650

Changed in keystone:
assignee: nobody → Joe Gordon (jogo)
status: New → In Progress
Dolph Mathews (dolph)
Changed in keystone:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/34650
Committed: http://github.com/openstack/keystone/commit/3f51d2addd38c418a55c01ca2f37e8bbe94319b5
Submitter: Jenkins
Branch: master

commit 3f51d2addd38c418a55c01ca2f37e8bbe94319b5
Author: Joe Gordon <email address hidden>
Date: Wed Jun 26 18:09:13 2013 -0700

    Deprecate kvs token backend

    This backend is not usable in any production environment. All OpenStack
    environments will already have a SQL DB, and if someone does not want to
    use the DB they can use the memcache backend.

    Fixes bug 1188301 and bug 1188370

    DocImpact This backend should not be mentioned in documentation, as it
    is not production grade and is deprecated.

    Change-Id: I41b147bcc70b79b4fc6df50b242a73cfcad33114

Changed in keystone:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in keystone:
milestone: none → havana-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in keystone:
milestone: havana-3 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.