Deployment in scale test case failed with '409 Conflict' error on flavor creation

I guess the solution is to mask the flavor creation command exit code with /bin/true

The issue is with the "unless" guard on the flavor creation statement. When it attempts to list all flavors to look for m1.micro it fails with a 401 error. That causes the flavor create command to run when it shouldn't. There is an error message in the nova logs:

 Identity response: {"error": {"message": "Could not find token: bccaec7e76974e7d8b2e54565f1df2f4", "code": 404, "title": "Not Found"}}

There is an issue with memcache_pool backend "invalidating" cached tokens after new nodes have been added into the memcached pool. The "invalidation" is nothing more than just a case when keystone tries to search existing tokens at the recently added memcached nodes and fails as there are no such ones - and it cannot yet failover due to the hash function is being rebuild for new nodes.

The solution is to fix memcache pool backend as appropriate, but this cannot be done for the 6.1. The workaround is to issue token-get right before to perform any CLI operations, and pass this token as argument.

The documentation should also be updated to point on this scale issue for memcache_pool keystone backend and recommend to always re-get new tokens after new controller nodes have been added.

> The solution is to fix memcache pool backend as appropriate

No. It is not memcache_pool's logic, it is python-memcached logic. We need to either fix it there, or use sql. Or wait for Fernet tokens.

To make this case work we need something like Cassandra as a token backend since it stores locations for keys, and it would be an overkill. It was a proposal in the Community and it was rejected.
The problem will be no more as soon as we adopt non-persistent Fernet tokens.

Folks

This is a critical issue for production deployments - if you add controller you do not expect interruption of service. We need to fix keys storing algorithm and we cannot wait for Fernet tokens.

> This is a critical issue for production deployments - if you add controller you do not expect interruption of service. We need to fix keys storing algorithm and we cannot wait for Fernet tokens.

The way you work with keystone tokens is wrong. You should not assume any time of token validity. The code should assume that the token can become invalid at any time and be ready to fetch a new one. There are many events that can lead to invalidation of token: node failure, token or user credentials compromisation, user role changes, project changes etc. Adding a controller node is just one of these events.

There is no way to "fix key storing algorithm" without rewriting python-memcache. Even if we exert ourselves and rewrite python-memcache, I don't think it's OK to include such a huge change during code freeze.

>The way you work with keystone tokens is wrong. You should not assume
>any time of token validity. The code should assume that the token can
>become invalid at any time and be ready to fetch a new one.

I agree here. That is why I suggested to modify the way Fuel works with tokens and update the documentation it the way users would know that to change as well.

> We need to fix keys storing algorithm and we cannot wait for Fernet tokens.
Disagree: this is not the root cause. If you want to fix lost tokens, you need to make client to ask tokens again if 401 response is received on a token supposed to be valid.

This problem will disappear automatically in Kilo where Fernet tokens are already implemented. Memcache dependancy is already removed in Kilo https://github.com/openstack/keystone/blob/master/requirements.txt

Sergii, it was removed by another reason, not because of tokens.

Guys, the program creating failed token request here is the python-novaclient CLI, with password based auth from

https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/openstack-controller/openstack-controller.pp#L297-309

We can reproduce this with (you may need to change the IP to match your controllers mgmt addr). It should occur within 10 times

while true; do echo 'flush_all 2' | nc 192.168.0.4 11211; nova --debug flavor-list || break ; done

Paste of the client failure http://paste.openstack.org/show/214577/
Paste of the keystone-api (in debug) http://paste.openstack.org/show/214580/
Paste of the nova-api http://paste.openstack.org/show/214579/

@Andrew, confirmed - I can reproduce it every time on the each 3rd iteration, even w/o adding new controllers:

while true; do echo 'flush_all 2' | nc 192.168.0.4 11211; nova flavor-list || break ; done

http://paste.openstack.org/

This looks like a major issue of aforementioned keystone backend

Sorry, it looks like floating, not stable :)

Here is a script you can try to check both token-based and creds-based cases http://pastebin.com/NnMW0c39
if you pass the 'creds' argument it would request nova flavor-list by nova user and password credentials, otherwise it would do it by new tokens for each request.
The script will silently exit with 1, if got "Unauthorized" error.

Note, I can reproduce the issue even if new tokens requested for each flavor-list

Alexander

I posted another bug to split the discussion https://bugs.launchpad.net/fuel/+bug/1451515 . This original bug can be worked around with puppet kludges, but should be fixed in mainline code obviously.

The backend should not return 401 unless it asked all memcached instances configured. This looks now like a programmatic error.

Given comments above, I can suggest nothing but this kludge https://bugs.launchpad.net/fuel/+bug/1449584/comments/2

Alexander, we are waiting for your suggestions/feedback.

related bug https://bugs.launchpad.net/fuel/+bug/1449219

Reviewed: https://review.openstack.org/180105
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d333d4d7fcbfe012f8621bf7515beb0f0c7d99ba
Submitter: Jenkins
Branch: master

commit d333d4d7fcbfe012f8621bf7515beb0f0c7d99ba
Author: Bogdan Dobrelya <email address hidden>
Date: Tue May 5 12:23:49 2015 +0200

    Use retries for flavor-create

    W/o this fix, keystone backend for tokens may
    sporadically fail to find a token on each nova CLI command.
    It may even fail if there would be a new token passed
    as an argument for each CLI command.

    For now, the solution is cannot be done at puppet side, but
    only the workaround, which is to retry nova CLI commands

    Related-bug: #1449584

    Change-Id: I0f4895fe01c6811b89699677a46cbcecc6f04930
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Added to known issues in release notes: https://review.openstack.org/#/c/180196/

Added to 7.0 to attempt to find better solution than hacky retries in puppet

@Vova, Bogdan, Andrew, please set one correct state for 6.1 and 7.0

It's already fixed in 6.1 with re-tries.

Since it already has a solution in 7.0 (re-tries) I'm lowering priority to medium.

Better solution here is fernet tokens, which are not going to happen in 7.0 and will be implemented as blueprint for fernet tokens.