Service user's password unnecessarily updated

A bit more info, I have just tried this out and I see that as mentioned, the real password does not change but its record in the keystone database does. This in turn triggers a revocation_event that causes all active tokens for that service user to be revoked. I can only assume that when the password update call is made, keystone and/or mysql is using extra information such as timestamp when encoding the password string and this results in a new record in the database each time the call is made even if the password itself has not changed.

I've now tried the update password action on both nova-compute and glance services and I see that it is only nova-compute that is affected since when Glance gets a 401 from keystone it requests a new token whereas nova-compute simply explodes when it gets the 401.

Adding the following to my nova.conf appears to stop Nova from failing when it gets a 401:

[keystone_authtoken]
...
check_revocations_for_cached = True

So, since Nova appears to by-design not automatically check and handle revoked tokens, I think we have the following options to solve the problem:

  1. set the above config in the nova-compute charm

  2. stop the keystone charm from calling update if password has not changed

  3. always restart nova-compute service when password update occurs (i.e. whenever nova-cloud-controller identity-service relation is called)

Personally I think that option 1 is least intrusive and simplest to implement.

Sounds like option 1 is the preferred path so i will go ahead and propose a patch.

Ok on second thoughts we'll go with option 2. The service passwords are stored in the leader setting so that charm should easily be able to check whether a password update is necessary. That way we'll avoid service token revocations for any service related to keystone.

> check_revocations_for_cached = True
> ...
> 1. set the above config in the nova-compute charm

We need to be careful with this option, because this will make nova-compute daemons to ask keystone for the list of revoked tokens (GET /tokens/revoked)[0] every X seconds[1], so the extra pressure on keystone is something to consider.

On top of the previously said, the real problem with this option is that this only works for PKI token format which is being deprecated[2]

[0] https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L744
[1] https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L601
[2] https://github.com/openstack/keystonemiddleware/commit/77909fdc169e4b6f9b177212514f10913bc389e6

Agreed Felipe, although i tried with UUID tokens and it worked as well. I am now actually implementing option 2.

Fix proposed to branch: master
Review: https://review.openstack.org/417036

As dosaboy mentioned in IRC, this fix will not initially target stable charms.

Reviewed: https://review.openstack.org/417036
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=f9670295de698b44ff9e9db4653af4ea6ded82b9
Submitter: Jenkins
Branch: master

commit f9670295de698b44ff9e9db4653af4ea6ded82b9
Author: Edward Hope-Morley <email address hidden>
Date: Thu Jan 5 16:28:47 2017 +0000

    Avoid keystone password update if unchanged

    Avoid calling update_password() if the password has not
    changed since it will actually change the db value
    regardless resulting in a revocation event and all current
    tokens being invalidated.

    Change-Id: Icb901b5e87d9cd716fa1a0d146e2252339e5678b
    Closes-Bug: 1648677

Definitely seeing this on newton deploys with stable charms. Lots of 503s and Unauthorized messages from various services talking to keystone causing failures on a number of nova api calls which prevent glance simplestreams from uploading images and cloud configuration calls to configure the deployed region.

Below you can see prints of all of the superfluous password updates from almost every service.
https://pastebin.canonical.com/175833/

cs:xenial/keystone-260 for reference

Just a note that landscape has been using cs:xenial/keystone-260 since landscape trunk r10798. Let's see if we have seen this bug since then in our ci test runs.

Oh, sorry, I see now that the fix is NOT in keystone-r260.

Fix proposed to branch: stable/16.10
Review: https://review.openstack.org/423662

Reviewed: https://review.openstack.org/423662
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=7bd0832bc3730a42d103143f1331f54257b3a3de
Submitter: Jenkins
Branch: stable/16.10

commit 7bd0832bc3730a42d103143f1331f54257b3a3de
Author: Edward Hope-Morley <email address hidden>
Date: Thu Jan 5 16:28:47 2017 +0000

    Avoid keystone password update if unchanged

    Avoid calling update_password() if the password has not
    changed since it will actually change the db value
    regardless resulting in a revocation event and all current
    tokens being invalidated.

    Change-Id: Icb901b5e87d9cd716fa1a0d146e2252339e5678b
    Closes-Bug: 1648677
    (cherry picked from commit f9670295de698b44ff9e9db4653af4ea6ded82b9)

Marking as fixed released - its now available in charm revision 261