Memcache token backend eventually stops working
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Identity (keystone) |
Fix Released
|
High
|
Rafael Durán Castañeda | ||
Essex |
Fix Released
|
High
|
Alan Pevec | ||
keystone (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Precise |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
Hi,
At BVOX, my company, we've got a weird issue while using the memcache token backend. Eventually, the token validation stops working and after some debugging, I've detected that the error is always triggered after a token validation request, returning the information about the token used for validation instead of the token being validating., e.g.:
GET /v2.0/tokens/123
x-auth-token: 456
Returns information about 456 instead of 123.
Working on the issue, under heavy load, we've also got this error:
RuntimeError: Second simultaneous read on fileno XX detected.
Unless you really know what you're doing, make sure that only one greenthread can read any particular socket.
Consider using a pools.Pool.
This error can be solved just monkey patching the threading module (as Nova does and I think Glance too).
I've attached a quite simple Python script using multiprocessing that is able to trigger the error quite fast, it drops some requests however token validation still works (3 of 4 processes dropped in my tests); not matching what happens on real deployment, where the token validation never works again until Memcached is restarted (I think just because of the ptython-memcache connection reset).
Related branches
- Ubuntu Server Developers: Pending requested
-
Diff: 13 lines (+6/-0)1 file modifieddebian/changelog (+6/-0)
Changed in keystone: | |
status: | New → Triaged |
importance: | Undecided → High |
tags: | added: essex |
Changed in keystone: | |
assignee: | nobody → Rafael Durán Castañeda (rafadurancastaneda) |
tags: |
added: essex-backport removed: essex |
Changed in keystone: | |
milestone: | none → folsom-2 |
status: | Fix Committed → Fix Released |
Changed in keystone (Ubuntu): | |
status: | New → Fix Released |
Changed in keystone (Ubuntu Precise): | |
status: | New → Confirmed |
Changed in keystone: | |
milestone: | folsom-2 → 2012.2 |
I can send a patch, unless someone has a good reason to Keystone not monkey patching the threading module.