Delayed S3 query processing due to memcached token locks

Full story from customer

"When we use the S3 interface we use it from our application which saves a lot of small images. Naturally we set up a user that our application can use to access the S3 interface with one access and corresponding secret key (keystone ec2-credentials-create). The sending of the images (objects) is multi-threaded to overcome the natural latency of the object storage. What I have found out is that when keystone validates the log-ins through the rados gateway it seems to take some lock on user level when handling the back-end store of the tokens. In MOS it seems to be memcached that is used for storing tokens.

All keystone requests using the same user that is storing a lot of images through the S3 interface are slow when the application is sending stuff to the S3 interface. keystone token-get can take between 0 and 9 seconds when we send images to the S3 interface if you do it with the same user. If you try keystone token-get with another user (OS_USERNAME=some other user) it always takes < 1s.

I have been able to verify this in a virtual env. as well. It's very easy to see that the performance of the S3 interface goes down when issuing a lot of parallel requests to the S3 interface with the same user.

I have also tried to change the storage back end for the tokens to mysql in the virtual env. Changing driver=keystone.token.backends.memcache.Token to driver=keystone.token.persistence.backends.sql.Token seems to increase the performance quite a lot. Of course the load on mysql goes up significantly and you have to purge expired tokens from the keystone database, but all in all it works better. Would it be ok to change that in the real env. as well, or do you see any pitfalls with that?

The ldap queries (keystone is configured with ldap backend) doesn't seem to have anything to do with the bottleneck in the S3 interface we see right now."

To get this fixed, I need to have [token]revoke_by_id = false. This needs to be done in fuel.

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Boris Bobrov <email address hidden>
Review: https://review.fuel-infra.org/10967

> multiple requests using same credentials going in parallel

How many requests happen per second in your case?

The bug also affects fuel because in order to fix it we need to make changes to keystone config. Please see comment #2.

Nope, I need someone from fuel folks to do it for me

I am marking the bugreport as invalid for 7.0 because it's not clear whether the bug can be reproduced in 7.0, because the locks that the reporter speaks about were heavily changed in 7.0. The original report also mentions only 6.0 or 6.1.

Reassigning to keystone team for 6.0/6.1 as maintenance team need a fix in some to start backporting.

Please follow the suggestion mentioned in #2
Not sure what team does this particular puppet, but it should set [token]revoke_by_id=False in /etc/keystone/keystone.conf upon initialization.

Setting bug's status to "Won't fix" for MOS 6.0, since at this point we only fix security issues there and this bug is not a security issue.

Plus we don't update puppet manifests in 6.0 and it takes puppet manifests to be updated to fix this issue.

Retargeted to 6.1-updates as we need more time for investigation.

I have set this bug's status to invalid for MOS 6.1 because I have tested S3 API performance of MOS 6.1 using a code from attachment. The results http://pastebin.com/9cnN6KbP are pretty consistent and doesn't differ much from simple file-by-file upload.

Change abandoned by Boris Bobrov <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/10967