[Problem statement]
In Havana, when using memcached as the backend of token store, we have been seeing significant performance drop by comparison with Grizzly.
[How to reproduce]
We used a Python script to boot VMs at the rate of 1 VM per second. We have seen a lot of VM creation failed and the Keystone-all process's CPU utilization was nearly 100%.
[Analysis]
When using memcached as token's backend, keystone stores two types of K-V pairs into memcached.
token_id ===> token data (associated with an TTL)
user_id ===> a list of ids for tokens that belong to the user
When creating a new token, Keystone first adds the (token_id, data) pair into memcahce, and then update the (user_id, token_id_list) pair in function _update_user_list_with_cas.
What _update_user_list_with_cas does are:
1. retrieve the old list
2. for each token_id in the old list, retrieve the token data to check whether it is expired or not.
3. discard the expired tokens, add the valid token_ids to a new list
4. append the newly created token's id to the new list too.
5. use memcached's Compare-And-Set function to replace the old list with the new list
In practice we have found it is very usual that a user have thousands of valid tokens at a given moment, so the step 2 consumes a lot of time. What's worse is that CAS tends to end up with failure and retry, which makes this function even less efficient.
[Proposed fix]
I'd like to propose a 'lazy cleanup of expired token_ids from the user list' solution.
The idea is to avoid doing the clean up EVERY TIME when a new token is created. We can set a dynamic threshold T for each user, and cleanup job will be triggered only when the number of token_ids exceeds the threshold T. After every cleanup, it will check how many token_ids have been cleaned up, if the percentage is lower than a pre-specified P, than the T needs to be increased to T*(1+P) to avoid too frequent clean-ups.
Besides, every time the list_tokens function for a given user is called, it will always trigger a clean-up action. It is necessary to ensure list_tokens always return valid tokens only.
There are a couple things to address here. The first is that list_tokens does not actually need to only return valid tokens. The only cases where list_tokens should be used is on the revoke_
In the Grizzly timeframe, this list regularly could be filled with bogus/invalid tokens and when the token list hits a memcache page size, no further tokens for that user can be issued (token data is not returned, and therefore token isn't usable). In Grizzly this list also was simply appended (using the memcache append call) rather than loaded and inspected in any cases.
Inherently, this is a flaw in the use of memcache as a token store and the need to maintain an active list of tokens for the user.
In the Icehouse development cycle we will be moving to a new token revocation event system that will help eliminate the need for this specific tracking list, and remove the need to do the CAS work in memcache to be able to properly revoke tokens on the events that need to revoke many tokens (e.g. password change).
Would it be possible to re-use tokens (especially in the case of booting 1 VM per second)? This would help to alleviate the extra load due to memcache token management.