At some point Keystone stops processing HTTP requests and HAProxy moves the server out of pool. Once Keystones on all controllers stuck HAProxy starts to respond with HTTP 503 error.
Symptoms:
~~~~~~~~~
HAProxy log:
2014-12-08T11:01:06.135849+00:00 info: 192.168.0.23:34984 [08/Dec/2014:11:01:06.135] keystone-2 keystone-2/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 366/0/0/0/0 0/0 "POST /v2.0/tokens HTTP/1.1"
Keystone keeps accepting TCP connection, but hangs at HTTP level (meaning that telnet works, but curl hangs)
root@node-21:/var/log/keystone# strace -p 26330
Process 26330 attached - interrupt to quit
futex(0xe50420, FUTEX_WAIT_PRIVATE, 0, NULL
The last logged message from stuck process:
2014-12-05 17:01:20.749 26330 DEBUG keystone.common.cache._memcache_pool [-] Memcached pool 39391504, thread 139781728069376: Acquiring connection _debug_logger /usr/lib/python2.7/dist-packages/keystone/common/cache/_memcache_pool.py:91
Memcache pool has asymmetric behaviour on aquire and release of a connection:
If pool is depleted a new connection is created
If pool has reached it's maximum size that connection excess still being pushed back to the pool causing thread lock.
Fix proposed to branch: master
Review: https:/