Comment 0 for bug 1571626

Revision history for this message
Mikhail Chernik (mchernik) wrote :

Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188

Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .

After that keystone breaks its operation and cluster in not usable.

Keystone error:

2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):

Steps to reproduce:

run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
  "kw": {
    "runner": {
      "type": "constant",
      "concurrency": 20,
      "times": 1970
    },
    "sla": {
      "failure_rate": {
        "max": 0
      }
    },
    "context": {
      "api_versions": {
        "keystone": {
          "version": 2
        }
      }
    }
  },
  "name": "KeystoneBasic.add_and_remove_user_role",
  "pos": 0
}

diagnostic snapshot will be added later