Keystone hangs trying to set a lock in Memcache
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mirantis OpenStack |
Fix Committed
|
High
|
Alexander Makarov | ||
5.0.x |
Fix Committed
|
High
|
Alexander Makarov | ||
5.1.x |
Fix Committed
|
High
|
Alexander Makarov | ||
6.0.x |
Fix Committed
|
High
|
Alexander Makarov |
Bug Description
Preconditions:
1. Keystone is configured to use Memcache
backend=
backend_
backend_
2. Memcached is deployed on each of 3 controllers.
3. 2 of 3 memcached servers are down (only the one on the primary controller is up: 10.108.12.3 is up, 10.108.12.5 and 10.108.12.6 are down)
Result: the keystone API hangs when a user do something like "keystone tenant-list". haproxy will drop the connection after 60s timeout.
strace shows that keystone tries to connect to unavailable servers in a loop, ignoring the available one.
Debug showed that keystone-all process is stuck while trying to set a lock: http://
Changed in mos: | |
status: | New → Confirmed |
milestone: | none → 6.0 |
tags: | added: keystone |
Changed in mos: | |
assignee: | nobody → Roman Podoliaka (rpodolyaka) |
tags: | added: memcached |
summary: |
- "keystone tenant-list" hangs sometimes + Keystone hangs trying to set a lock in Memcache |
description: | updated |
description: | updated |
Changed in mos: | |
assignee: | Roman Podoliaka (rpodolyaka) → nobody |
Changed in mos: | |
status: | Confirmed → Triaged |
Changed in mos: | |
status: | Triaged → In Progress |
description: | updated |
Looks like this happens for ~1.500 keys per 100.000 tries (~1.5%) when 1 of 3 memcache servers is available