Mirantis OpenStack

Token not found

Bug #1468839 reported by Viktoria Efimova
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Medium
MOS Keystone
Mirantis OpenStack 8.0
6.1.x
Won't Fix
Medium
MOS Maintenance
Mirantis OpenStack 6.1-updates
7.0.x
Won't Fix
Medium
MOS Keystone
Mirantis OpenStack 7.0-updates
8.0.x
Fix Released
Medium
MOS Keystone
Mirantis OpenStack 8.0

Bug Description

Two keystone rally tests failed with Could not find token: 5731bfea58c64784b9e7e83c29f83c6d (Disable debug mode to suppress these details.) (HTTP 401) and Could not find token: fbe9320a1057442b934d82e3c3357369 (Disable debug mode to suppress these details.) (HTTP 401)

grepping logs using 5731bfea58c64784b9e7e83c29f83c6d token id gives only:
/var/log/keystone-all.log:<12>Jun 24 22:41:01 node-9 keystone-all Authorization failed. Could not find token: 5731bfea58c64784b9e7e83c29f83c6d (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from 192.168.0.2
/var/log/keystone/keystone-all.log.1:2015-06-24 22:41:01.538 32676 WARNING keystone.common.wsgi [-] Authorization failed. Could not find token: 5731bfea58c64784b9e7e83c29f83c6d (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from 192.168.0.2

Topology:
CentOS+GRE+525(RC3)
Controllers: 3, Computes: 21

api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'

Log snapshot at http://mos-scale-share.mirantis.com/fuel-snapshot-2015-06-25_16-05-41.tar.xz

Revision history for this message
Boris Bobrov (bbobrov) wrote :

In the logs I see

```
2015-06-24T22:41:01.537318+00:00 debug: Memcached pool 90129936, thread 140052079146752: Marked host 192.168.0.10:11211 dead until 1435185691.53
2015-06-24T22:41:01.542767+00:00 warning: Authorization failed. Could not find token: 5731bfea58c64784b9e7e83c29f83c6d (Disable debug mode to suppress these details.) (Disable debug mode to s
uppress these details.) from 192.168.0.2

```
same for the second token:
```
2015-06-24T22:31:07.226938+00:00 debug: Memcached pool 61707152, thread 140200754067200: Marked host 192.168.0.10:11211 dead until 1435185097.09
2015-06-24T22:31:07.251874+00:00 warning: Authorization failed. Could not find token: fbe9320a1057442b934d82e3c3357369 (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from 192.168.0.2
```
which means that memcache host became dead there. Keystone in 6.1 requires some seconds to failover.

Were the tests destructive?

Revision history for this message
Dina Belova (dbelova) wrote :

Boris, these tests are not destructive, we process tests like that in manual way, this is simple rally test... May it be the case when memache host was moving to other node because of any issue?

Changed in mos:
assignee: nobody → MOS Keystone (mos-keystone)
importance: Undecided → Medium
Revision history for this message
Boris Bobrov (bbobrov) wrote :

Dina, Viktoria, was it reproduced?

Revision history for this message
Alexander Makarov (amakarov) wrote :

Yes, we have an env with the bug

Revision history for this message
Alexander Makarov (amakarov) wrote :

The problem is in memcache sharding: it's unstable under high load (somtimes fails to connect).
The solution lies in configuration rather than coding, and problem is about to go away along with memcache token storage.

Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Nominated for 8.0 and Won't Fix for 7.0-updates and 6.1-updates because of Medium importance

Revision history for this message
Alexander Makarov (amakarov) wrote :

Looks like sometimes keystone fails to connect to memcache holding the token:
https://bugs.launchpad.net/fuel/+bug/1515350

Revision history for this message
Boris Bobrov (bbobrov) wrote :

This should be fixed with Fernet tokens.

Anastasia Kuznetsova (akuznetsova)
tags: added: area-keystone
removed: keystone
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

It looks like the issue was fixed, not reproduced in my environment.
Please reopen it and add more information if the issue will be reproduced again.

Verified on MOS 8.0 RC1.

Thank you!

Revision history for this message
Ivan Lozgachev (ilozgachev) wrote :

Cannot verify because of https://bugs.launchpad.net/mos/+bug/1538251. Moving to Fix Released.

Revision history for this message
Ivan Lozgachev (ilozgachev) wrote :

Sorry, didn't saw that it's already in Fix Released state...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.