[OSSA 2016-005] PKI Token Revocation Bypass (CVE-2015-7546)

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

The tests scripts does not show how token are created/revoked. What type of token is this ?

Maybe it's a swift setting that allows direct access to the swift account.

Please also be aware that paste.openstack.org contents are not secret/private and can be easily enumerated. We should probably consider this issue publicly disclosed at this point since an exploit has been published to the paste server.

I would like to know what version of keystone is being used here and what type of token.

This is definitely not uuid tokens, and I don't believe this can be fernet as fernet is live validated and does not rely on the token_id for revocations.

Can this occur with services other than swift as well? I'll need to dig further on this unless the information can be provided.

If this is PKI/PKIZ tokens, there are no "internal" IDs as the internal ID would be part of the signed data. The signed data cannot contain the ID itself as it would change the ID.

Discussing this a bit further it looks like "revoke by ID" should have been based on fingerprint or the signed data (as per adam young). Alternatively, we can now use the audit_ids for "revoke by id"

I am still in process of confirming outside of swift.

This sounds like it must be limited to PKI/PKIZ tokens? Fernet tokens would fail their HMAC and there is no room to manipulate UUID tokens unless you're guessing a completely different one.

In the script to reproduce, mutating token_bin[1] would be manipulating the CMS length prefix (MIIE-*, etc), in which case, I wonder if there's a bug in openssl?

Jeremy: please don't open this bug just because the paste is public. There's nothing in the paste that shouts "security vulnerability," and we should certainly have the ability to delete pastes (and move them here), no?

I won't switch this to public without consensus anyway, but can assure that there are crawlers continuously spidering our paste server (I know because I see them in the logs and they make up quite possibly the majority of requests there).

sorry, my mistake, this issue is occured with PKI token.

This issue is found by my colleagues, I will dig more and provide more information, thanks for your attention.

I won't be able to spend much time on this until at least Sept 4th, but I suspect this is easily reproducible and I'm wondering if the fix must be to make PKI depend on revocation events like Fernet does? Or perhaps to ignore the mutable characters when querying the token table?

Jeremy: I did not consider crawlers / mirrors. Deleting the paste now might only draw attention to it :)

Liusheng: if you need to share any other files/logs/patches related to the bug, please add them as attachments to this bug (using the "Add attachment or patch" link at the bottom of the page).

Dolph: sure, I am sorry I didn't realized the test script shouldn't be pasted in paste.openstack.org.

I have confirmed they guys found this issue in Juno openstack, some version info used:

keystoneclient 0.11.2
keystone 2014.2

openssl-1.0.1i-2.1.4
openssl-certs-1.97-0.3.1

This should only be a problem with PKI(Z) tokens. The fix can be made here:

http://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/utils.py

Using the logic in the blog post here: http://adam.younglogic.com/2014/10/who-signed-that-token/ we can pull out the signed data from the token and take a hash of that; modifying the hashed data will mean the token will not longer pass the CMS validation.

This will pull in an additional dependency on pyasn1 to Keystone. We cannot avoid that by calculating the hash on the JSON data itself, as it will not (necessarily) be reproducible.

In addition, Horizon will need an update, as they currently Md5 Hash the tokens before sending them to the remote services.

Another possibility is to generate a UUID for the token ID, and stick that inside the JSON body before signing. revocation could be done on the UUID value instead of the HASH. That would involve changing both Keystone and keystonemiddleware.

another alternative it to compare the signer info in the PKIZ token with the Certificate. If they don't match, the token is invalid. I don't know if it is possible to be 100% certain that the teokn cannot be tampered with.

Thanks for the information. It helps us to be sure we are looking at all avenues for the problem. I can see where this is happening and have discussed with Adam and a few other folks on it and how we are approachig potential fixes.

The attached code shows the modification of the binary data that still generates a valid token; it can be run from

python-keystoneclient/examples/pki and usese the sample tokens and certificates in the associated subdirectories.

This shows the modified bytes that successfully pass are:
77 78 79 80 81 82 83 85 86 87 88 3979 3980 3981 3982 3983 3985 3986 3987 3988 3991 3992 4364

The range 77 78 79 80 81 82 83 85 86 87 88 looks like the object identitfier s

1 3 14 3 2 26
and
1 2 840 113549 1 7 1

The last byte is padding.

To see the content of the token in PEM format

cat cms/auth_v3_token_scoped.pem | awk ' ! /---/ {print $1}'| base64 -d | /usr/lib64/nss/unsupported-tools/derdump > /tmp/tokendata

The attached test manipulates UUID, PKI, PKIz, and Fernet tokens by incrementing and decrementing individual bytes, as illustrated in the bug description. This test passes against each token type, showing that the manipulated tokens cannot be validated *online* against keystone. To be clear, this patch does not contain a fix, nor does it confirm or invalidate the bug report.

Unless I missed something, that means the scope of this bug is limited to validating tokens offline (against the token revocation list in keystonemiddleware.auth_token).

Of course, it would help if I attached the patch.

Dolph; yeah, this is only for checking the revocation list; the tokens themselves need to be valid tokens. It is restricted to PKI based tokens only. The mistake was using the hash of the whole document for the token id; it should have been the same hash of the token tdata that is used for the cryptographic signature. The values being manipulated are the ASN1 specific values that indicate "This next block is signed data" which openssl really should be validating.

For example; here is a diff of a derdump of the origianl file and the modified file, with some lines of context;

$ diff -U 10 /tmp/auth_v3_token_scoped_orig.pem.txt /tmp/auth_v3_token_scoped77.pem.txt
--- /tmp/auth_v3_token_scoped_orig.pem.txt 2015-09-02 11:27:46.423313737 -0400
+++ /tmp/auth_v3_token_scoped77.pem.txt 2015-09-02 11:27:46.398313755 -0400
@@ -4,21 +4,21 @@
    C-[0] (3176)
       C-Sequence (3172)
          Integer (1)
             01
          C-Set (9)
             C-Sequence (7)
                Object Identifier (5)
                   1 3 14 3 2 26 (SHA-1)
          C-Sequence (2692)
             Object Identifier (9)
- 1 2 840 113549 1 7 1 (PKCS #7 Data)
+ 0 26 840 113549 1 7 1
             C-[0] (2677)
                Octet String (2673)
                   7b 22 74 6f 6b 65 6e 22 3a 20 7b 22 6d 65 74 68 6f 64
                   73 22 3a 20 5b 22 70 61 73 73 77 6f 72 64 22 5d 2c 20
                   22 72 6f 6c 65 73 22 3a 20 5b 7b 22 69 64 22 3a 20 22
                   66 30 33 66 64 61 38 66 38 61 33 32 34 39 62 32 61 37
                   30 66 62 31 66 31 37 36 61 37 62 36 33 31 22 2c 20 22
                   6e 61 6d 65 22 3a 20 22 72 6f 6c 65 31 22 7d 2c 20 7b
                   22 69 64 22 3a 20 22 66 30 33 66 64 61 38 66 38 61 33
                   32 34 39 62 32 61 37 30 66 62 31 66 31 37 36 61 37 62

All of the places susceptible to the modifications seem to be this kind of string, with the exception of the final padding;
$ diff -u /tmp/auth_v3_token_scoped_orig.pem /tmp/auth_v3_token_scoped4364.pem
--- /tmp/auth_v3_token_scoped_orig.pem 2015-09-02 11:14:11.414915072 -0400
+++ /tmp/auth_v3_token_scoped4364.pem 2015-09-02 11:14:28.749902282 -0400
@@ -66,4 +66,4 @@
 4RQzkShFx97ioGPE9wE1Jv9Yzxo5uTg6z246gi42e50uSIFZMmNMqkUdz0n6Qct+
 1zMf8W+e1pykKscVBz757F9n+qAhx15LxxdCKIrQ1kNT8dAyLRBQU0qZeLmLvS3W
 nmf/JlQ7LDQmq3XQEHRbQ1s5dz/mwM8IiSm8
------END CMS-----
+-----END CMS-----
\ No newline at end of file

@dolph,

Yes this would be limited to an auth_token middleware issue. We should retarget the bug.

Added keystonemiddleware as there are likely changes needed to KSM as well

The fix could impact the contents of the token revocation list itself, though (in keystone).

(read comment #22 & #23 before I saw the new bug status - it seems you already agree!)

The following patch detects modifications of the Token.

I am not certain, however, that this is exhaustive; there might be ways to modify the document such that it is still valid, but has a different hash.

We can lock down what is accepted as a token format: additional checks could confirm that the certs are stripped (adding them will still fail validation anyways) or any other cosmetic changes that could change the document.

We might be able to get away with a stop gap like this, and then migrate people to Fernet or change how PKI tokens are hashed.

A few issues that came up in discussion earlier tonight:

* What happens if someone respins a token from PKI to PKIZ or vice-versa?
* PKI the delimiter (---- START CMS ---- and corresponding end ones) could be pre/post substituted in PKI (not PKIZ) and change the resulting ID but still pass due to the token_to_cms (or cms_to_token) code doing a blind substitution between '/' and '-'.

One of the other options could be to provide an options for KeystoneMiddleware to disable offline validations (default to off) and have it hash the tokens the same way django_openstack_auth would (which is not affected by this issue).

I'm looking over the code in the Reject-modified-PKI-Tokens.patch and it is a partial solution, but I have not spent enough time poking at the code to be sure what other gaps are there in PKI(Z) validation.

This patch is probably the apparoach we need to take; it changes the logic of generating a token_id from hashing the whole body of the token to hashing the data, same way that the signing algorithm does.

The patch will not yet pass unit tests.

Django OpenStack Auth needs to be modified to call this function in keystoneclient instead of doing their own hashing.

I am going to be -1 on changing DOA and changing how we hash immediately as this will break anyone who does not use the libraries (aka any language but python).

Lets aim for a fix that doesn't break absolutely everyone.

DOA is not impacted by this as it forces live validation.

Let's not change DOA and only change how middleware interacts in a non-live validated token workflow for Liberty (preferably something back-portable).

The DOA fix is trivial. It should go in anyway.

This is the lowest impact change. It won't break anyone unless they are doing the undocumented Hash hack. The number of people that are doing that should be vnishingly small, and they should update was well, too, for the same reason.

We can post code samples in Ruby and Java for how to do the hash.

This is the DOA change, and it can and should be applied regardless. I think it is safe to post as a review publicly.

Is there any direction on this one? Online validation is going to break people doing distributed signing, which is the reason to use PKI tokens. Any fix here has the potential to fix someone. Changing the hash approach is, I suspect, going to have the least impact.

Note that always going to online validation would preclude the use of memcache for tokens.

I repeat the previous statement: Changing the hash mechanism will break everyone. DOA is absolutely not affected here.

DOA forces online validation because it hashes the PKI(z) tokens to the SHA form. If a token is already hashed, it will only rely on the online validation which requires matching the key in the database.

The solution for the time being is going to be addressing how middleware validates the token offline via SSL. Options I see are:

* Done via audit_ids with new forms of middleware if we make audit_ids available as a form of the TRL.
* Option to only online-validate PKI(Z) tokens.
* Offer a different hash-form TRL (not just audit-id) for new versions of middleware
* Deprecate PKI(Z) tokens [not a good idea]
* Issue OSSA and indicate this is not something we can/will fix with the PKI tokens (and deprecate) [not a great idea]

I would prefer to see an online-only validation option with an OSSA as the immediate (and potentially backportable) fix. Discussion on changing hashes etc can be delayed until the summit and done in the open - not under embargo. For the embargo'd bug, I really want to see this not be a breaking change.

Breaking memcache for validated tokens [optionally] where the deployer is aware of the issues (due to OSSA being issued) is fine.

Ouch, I'd love to do a security blogpost on this once it's announced, a full root cause analysis.

I'm not close-enough to Keystone to understand the full impact of all the the mitigation options however it's clear that this is a pretty severe issue.

I know the numbers are hard to quantify but "thumb in the air" - what percentage of deployments use PKI tokens?

-Rob

@Robert,

It's enough to be a serious consideration (in the <50% but more than 15% is my guess; but this is just hazarding a guess). However we are now seeing a move towards Fernet tokens from the orgs that have been using PKI.

Some of the largest deployments are PKI token based.

This also seems like a prime target for a security note (OSSN). Basically we'll say something along the lines of "PKI(Z) tokens without online validation shouldn't be considered safe for certain versions" (if I understand this correctly).

This will enable cloud deployers to better understand the documented risks they are accepting.

Root cause is the assumption that any tampering with the PKI token would invalidate the token. There are three major fields in the token that are resistant:

1. The message. Any change to the message will be detected due to it not matching the signature.
2. The signature. Any change to the signature and the message will no longer match its signature.
3. The singning info. This checked against the X509 signing cert byte-by-byte. Any change and it won't match.

What I found in testing is that there are other fields that X509 ignores but should not. These include 0 length fields that are place holders for things like the stripped out Certificates.

The other attack now possible is due to there being 2 different PKI token versions. A PKI token could be converted to a PKIZ token, and the signature is still valid. That was not the case when the hashing approach was chosen.

The most common reason for token revocations is logout from the WebUI.

The impact is probably small but consequential; it only affects revoked tokens in the CLI cases. In the webUI case, the tokens are never stored in full except inside the token database. Instead, the WebUI hashes them, and discards the CMS document. Since this is the primary cause of revocations, these tokens are not at risk.

The second most common is password changes. Since these are performed by the end users themselves, the risk here is slight, but not 0.

The most worry some are token revocations for users leaving a project or being disabled altogether. In order for this attack to be viable, the user must have had a valid token prior to the event, and then they could re-enable the token. Since token timeout is a variable, we can;t make any assumptions. We turned the default down to 1 hour, but many sites found this was too short for workflows, and set the value back to 12 hours.

Hi Adam Young,

My colleague has applied your patch with Juno version keystone, the original problem is successfully solved, however some random errors occur when issuing both v2 and v3 token, the tracebacks are as below.

Seems the errors are introduced by the patch, and we didn't found the root cause. traceback info please see the attachment could you please take a look ?

Keystone is running on apache with worker mode.

Version of pyasn1 packages are:
python-pyasn1-modules-0.0.7-33.2
python-pyasn1-0.1.2-0.5.1

OS is Suse SLE11 SP3

Since this seems related to a bad design in PKI(z) token for which the fix isn't easily backportable, I suggest a B type of bug.

Also, since this requires a fair amount of social engineering to abuse this bug, is there a good reason to keep this under embargo ?

I agree, this most likely falls into class B2.

https://security.openstack.org/vmt-process.html#incident-report-taxonomy

Let's open this bug and remove the OSSA task by the end of the week if nobody objects.

This is the first I'm seeing this bug and it looks like B2 and we need to send an OSSN. We should fix in master and liberty; backporting to kilo and beyond might be tough.

I'd like it if we could get feedback from the reporter and possibly A.Young before we open this up more widely. However if they concur regarding severity etc I'm happy to move forward with an OSSN.

Adam, regarding your patch in #28 (0001-hash-the-data-in-the-token.patch ), can't you use cms_verify to get the output instead of doing asn.1 parsing?

In any case, doing short-lived PKI tokens instead of relying on revocation is probably easier if deployer can tolerate the risk.

Liusheng and A.Young, any concerns if this bug is made public tomorrow (Wednesday October 14) ?

Hi Tristan Cacqueray, I have talked with my colleagues, they have applied the patch provided by Adam young, and this bug has been fixed. But they have found another issue after using this patch, after we issue a valid PKI token, then modify the token by appending some characters to the end of the token, the modified token can also be authenticated successfully. So it seems this fix will cause another security issue. could you please make this bug public few days later ? It is also because we have online productions that using Havana/Juno OpenStack, this issue still exist, we need a few days to release a patch before this bug made public, it will be a risk if we make this bug public now. with great thanks to you! And, Could Adam Young please to check about this new issue? very thanks for your patch!

Disables the PKI token provider and replaces it with the UUID provider

It seems like PKI[z] tokens revocation is beyond repair and Keystone is looking forward deprecating those token now (see patch in comment #48).

I've raised the OSSN tasks priority to critical since this is not going to warrant an advisory (not backportable fix, etc...), can ossn-coresec works on that asap ?

The amount of work and change it would take to fix this makes it impractical; Fernet tokens will provide a better performing solution that does not suffer from this defect; fernet tokens will be GA faster than we could fix this bug.

With K2K Using SAML, the primary rationale for PKI tokens has gone away, and thus there will be no more fixes for PKI tokens

See Keystone version of bug for explanation

Here's how I'd summarize the problem:

Users can modify a PKI(Z) token to avoid token revocation

When keystone is configured for PKI or PKIZ tokens, you get a token which is a base64-encoded signed document. In order to validate the token, auth_token middleware un-base64 encodes the document and validates it using the `openssl cms -verify` subcommand. `openssl cms -verify` doesn't validate all the fields in the document, so the document can be modified in some places and it will continue to pass validation.

When auth_token middleware validates that a token is not revoked, it compares the hash of the token to see if it's in the revocation list. Since the modified token hash isn't going to match the value in the revocation list the token will be allowed. So the user can modify their token and continue to use the token even after revocation.

Could we validate the fields that cms doesn't verify? For example, if the Object Identifier is always supposed to be `1 2 840 113549 1 7 1` then we reject any tokens that have a different value. Same for the padding.

Looks like you can get the fields out of the token using -cmsout:

  openssl cms -cmsout -inform PEM -nosmimecap -nodetach -nocerts -noattr -print < data

Using that output, we could validate that eContentType is 1.2.840.113549.1.7.1 and algorithm is 1.2.840.113549.1.1.1:

< eContentType: undefined (1.2.70.72.113549.1.7.1)
---
> eContentType: pkcs7-data (1.2.840.113549.1.7.1)

< algorithm: undefined (1.1.840.113549.1.1.1)
---
> algorithm: rsaEncryption (1.2.840.113549.1.1.1)

There might be other fields that we have to validate also, although it must be harder to manipulate those.

Then there's also the issue of the padding which I didn't look into.

So I think it's possible to fix this but is it worth it?

I won't pretend to know the intricacies of the PKI/PKIZ implementation, though presumably if you're going to call into `openssl cms` during validation against the revocation list then storing/checking a hash of just the tdata (instead of or alongside the full document hash currently used) doesn't impose much additional performance penalty? This of course wouldn't help already stored hashes, but for any new ones added the revocation could be matched regardless of optional fields since the tdata seems like it should be unique.

I'm completely in favor of ripping out PKI/PKIZ tokens in Mitaka (with appropriate communication in release notes/upgrade documentation), but we continue to claim security support for features in Kilo and Liberty for some months to come. If there's a compatible solution which can be backported to our current stable branches, it's of potential temporary benefit to deployments which may have trouble switching to UUID or Fernet quickly and are looking for some alternative mitigation in the interim. If that really just can't work, then sure we should proceed with the current plan to tell those people they're stuck with a gaping insecurity until they alter a fundamental aspect of their deployment.

It's of course also possible I'm overestimating the amount of work involved in switching the token type for a deployment, since I'm not an operator.

The keystone server doesn't send the tdata hash in the revocation list, only the full token hash, so that would be a change to keystone. Then auth_token middleware would have to change to hash the tdata and check that, too. If we were going to do that, then maybe it would be easier to switch to revocation by audit IDs instead.

I'll try implementing the extra field checks in auth_token. Shouldn't be too difficult.

And I agree that we should have a fix for stable branches since they're supported.

The issue with validating fields in the token is that a PKI token could be changed into a PKIZ token and then the hash wouldn't match. So we'd have to have some way for the auth_token middleware to know that the identity server is issuing PKI or PKIZ tokens, either a new auth_token config option or keystone rest api or change to the revocation list response to also indicate the token format.

I'll look into also having keystone also store the audit id in the revocation list. We have audit IDs in tokens going back to juno, and v2 tokens have audit_ids, too. So should be a pretty easy change.

Changing auth_token to also check the revocation list for the audit_ids should be easy, too.

WIP patch for keystone to include audit_ids in revocation list

WIP patch for auth_token to check revoked by audit_id.

I attached the changes to have keystone include audit_ids in the revocation list and to have auth_token middleware validate by audit ID. These are WIP since there need to be unit tests and I need to make sure that multiple audit IDs are handled correctly. I tried it with devstack and it worked for me.

I'd prefer to do the reviews in gerrit, and I don't think that the reviews for these changes need to be done as patches in this bug. I don't think anyone would figure out the security problem by looking at the code changes and the commit message doesn't have to mention it. So I propose that I'll push the patches to gerrit if there aren't any objections.

i'm OK with posting the patches to gerrit, your reasoning sounds fair.

Considering this bug exploitation requires administrator action (e.g., to issue token revocation), I would prefer this bug report to be public. Operator could use this information to enforce a more strict revocation process while the fix is being implemented.

What is the status of the OSSN ?

How about we open the bug at the same time we issue the security note, then we could review the proposed patch on gerrit and we'll eventually issue an OSSA if the solution is safely backportable ?

The exploit doesn't require admin action. Tokens can be revoked by the user, or they can be revoked by the user changing their password. Tokens can also be made invalid by admins taking away the user's roles.

Brant, the only scenario I considered is when a malicious user is able to keep access after an operator revoked his token or took away his roles. iiuc the exploit you described requires valid token, what can the user do with tokens he invalidated himself ?

Tristan - The only thing I can think of is the user revokes their own tokens because they're concerned that it's been exposed. So they think they're stopping the 3rd party from using their token but the 3rd party can still use the token by modifying it slightly (until the token expires).

Yes, the typical concern would be an admin thinks the user's old tokens are revoked but they're still usable until they expire.

Tristan - I have the OSSN draft written up, which has been reviewed by a few people from the Security team as well as by Adam Young. If we are also going to provide a fix, I will need to reword a few things in the "recommendations" section. Here is the current draft of the OSSN:

-----
Potential reuse of revoked Identity tokens
---

### Summary ###
An authorization token issued by the Identity service can be revoked,
which is designed to immediately make that token invalid for future use.
When the PKI or PKIZ token formats are used, it is possible for an
attacker to manipulate the token contents of a revoked token such that
the token will still be considered to be valid. This can allow
unauthorized access to cloud resources if a revoked token is intercepted
by an attacker.

### Affected Services / Software ###
Keystone, Icehouse, Juno, Kilo, Liberty

### Discussion ###
Token revocation is used in OpenStack to invalidate a token for further
use. This token revocation takes place automatically in certain
situations, such as when a user logs out of the Dashboard. If a revoked
token is obtained by another party, it should no longer be possible to
use it to perform any actions within the cloud. Unfortuantely, this is
not the case when the PKI or PKIZ token format is used.

When a PKI or PKIZ token is validated, the Identity service checks it
by searching for a revocation by the entire token. It is possible for
an attacker to manipulate portions of an intercepted PKI or PKIZ token
that are not cryptographically protected, which will cause the
revocation check to improperly consider the token to be valid.

### Recommended Actions ###
We recommend that you do not use the PKI or PKIZ token formats. Due to
the way that these formats were designed, it is not possible to fix this
issue in a way that is backwards compatible. The PKI and PKIZ token
formats do not offer any significant benefit over other token formats
such as the UUID or Fernet formats.

If you are using the PKI or PKIZ token format, it is recommended that
you switch to using another supported token format such as the UUID
format. There are no plans to fix this issue in the PKI and PKIZ token
providers in the Identity service.

To check what token provider you are using, you must look in the
'keystone.conf' file for your Identity service. An example is provided
below:

---- begin keystone.conf sample snippet ----
[token]
#provider = keystone.token.providers.pki.Provider
#provider = keystone.token.providers.pkiz.Provider
provider = keystone.token.providers.uuid.Provider
---- end keystone.conf sample snippet ----

This configuration snippet is using the UUID token format. If you are
using either of the commented out settings from this example, your cloud
is vulnerable to this issue and you should switch to a different token
provider.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0061
Original LaunchPad Bug : https://bugs.launchpad.net/keystone/+bug/1490804
OpenStack Security ML : <email address hidden>
OpenStack Security Group : https://launchpad.net/~openstack-ossg

Thanks, so we could remove the first chapter of "Recommended Actions" and send this OSSN as an advance notification. We'll then open the bug and review Brant's patch on gerrit. Thought ?

It seems reasonable to go forward with the OSSN worded to assume there is no fix forthcoming. Then once there's an actual fix merged to stable branches we can follow up with an OSSA (though worth keeping in mind the proposed backport, as far as I understand, only solves the problem for tokens revoked after the patch gets applied, not any which predate its application).

Jeremy - Agreed. Are we ready to make this issue public then? Is everyone OK with the OSSN draft I provided in my previous comment?

As soon as we make it public, I can propose the OSSN review and fast-track it's merge and get it published.

Jeremy - Once the change is applied to keystone then the revocation list will include the audit IDs. The audit IDs are already in the token table so that will be available. As such, the fix will work for old tokens.

Comments on the proposed OSSN

1) change "token formats" to "token providers"

2) In the sample config, since keystone has support for using entrypoints for the provider, the examples should also have:

 #provider = pki
 #provider = pkiz

This was introduced in liberty so earlier releases need to use keystone.token.providers.uuid.Provider whereas liberty and later should use uuid or fernet

Brant: Excellent! That's better than I had hoped (was assuming only the full document hash was being stored for earlier tokens). So this really would, in theory, be a complete fix for stable branch consumers. With that in mind, I agree it makes sense to slightly lighten the bit of the OSSN about no possibility of a backward compatible fix. Still shouldn't imply that there is definitely one coming in case something unforeseen happens in review, but getting awareness of the bug and mitigation recommendations out to deployers/operators in the near term is still a good idea.

Here is a new draft of the OSSN that contains the suggestions made by Brant as well the suggestion made by Jeremy with regards to there being a possibility of a future fix. If this looks good to everyone, I think we can move forward with publishing and opening this bug up.

----------

Potential reuse of revoked Identity tokens
---

### Summary ###
An authorization token issued by the Identity service can be revoked,
which is designed to immediately make that token invalid for future use.
When the PKI or PKIZ token providers are used, it is possible for an
attacker to manipulate the token contents of a revoked token such that
the token will still be considered to be valid. This can allow
unauthorized access to cloud resources if a revoked token is intercepted
by an attacker.

### Affected Services / Software ###
Keystone, Icehouse, Juno, Kilo, Liberty

### Discussion ###
Token revocation is used in OpenStack to invalidate a token for further
use. This token revocation takes place automatically in certain
situations, such as when a user logs out of the Dashboard. If a revoked
token is obtained by another party, it should no longer be possible to
use it to perform any actions within the cloud. Unfortunately, this is
not the case when the PKI or PKIZ token providers are used.

When a PKI or PKIZ token is validated, the Identity service checks it
by searching for a revocation by the entire token. It is possible for
an attacker to manipulate portions of an intercepted PKI or PKIZ token
that are not cryptographically protected, which will cause the
revocation check to improperly consider the token to be valid.

### Recommended Actions ###
We recommend that you do not use the PKI or PKIZ token providers. The
PKI and PKIZ token providers do not offer any significant benefit over
other token providers such as the UUID or Fernet.

If you are using the PKI or PKIZ token providers, it is recommended that
you switch to using another supported token provider such as the UUID
provider. This issue might be fixed in a future update of the PKI and
PKIZ token providers in the Identity service.

To check what token provider you are using, you must look in the
'keystone.conf' file for your Identity service. An example is provided
below:

---- begin keystone.conf sample snippet ----
[token]
#provider = keystone.token.providers.pki.Provider
#provider = keystone.token.providers.pkiz.Provider
provider = keystone.token.providers.uuid.Provider
---- end keystone.conf sample snippet ----

In the Liberty release of the Identity service, the token provider
configuration is different than previous OpenStack releases. An
example from the Libery release is provided below:

---- begin keystone.conf sample snippet ----
[token]
#provider = pki
#provider = pkiz
provider = uuid
---- end keystone.conf sample snippet ----

These configuration snippets are using the UUID token provider. If you
are using any of the commented out settings from these examples, your
cloud is vulnerable to this issue and you should switch to a different
token provider.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0061
Original LaunchPad Bug : https://bugs...

@Nate, not sure if you also want to mention that the PKI and PKIz token providers are not deprecated in Mitaka.

Once the OSSN is approved, I'd like to send it as an advance notification to stakeholders before opening this bug.
Assuming it is done by tomorrow, this could be disclosed next Tuesday:
2015-12-15, 1500UTC

the OSSN in comment 73 looks good to me.

The OSSN has been sent, let's open this bug next Tuesday (2015-12-15, 1500UTC).

Red Hat preemptively assigned CVE-2015-7546 for this vulnerability.

This bug is now public, OSSN and Keystone patches can be submitted to gerrit.

The OSSN review has been submitted:

  https://review.openstack.org/#/c/258119/

This issue has been published as OSSN-0062:

  https://wiki.openstack.org/wiki/OSSN/OSSN-0062

The fix for keystone[master] is https://review.openstack.org/#/c/258141 . Can this be proposed to stable liberty and kilo too ?

Tristan - It can be proposed, although some might consider the change to be a new feature rather than a bug fix, so not sure if it's appropriate for stable. I'd prefer for the change to be merged in master before I backport it otherwise I'll have make changes in multiple releases if there's a -1.

Brant & Tristan, I think it should be fine to backport to kilo and liberty. It's an important security related fix and I consider the proposed changes to be a bug and not a feature.

Thanks, then considering this is going to be fixed in stable release, I've confirmed the OSSA task.

Reviewed: https://review.openstack.org/258141
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=d5378f173da14a34ca010271477337879002d6d0
Submitter: Jenkins
Branch: master

commit d5378f173da14a34ca010271477337879002d6d0
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 11:09:14 2015 -0600

    Add audit IDs to revocation events

    The revoked tokens' audit ID is now included in the data returned in
    the revocation list.

    Closes-Bug: 1490804
    Change-Id: Ifcf88f1158bebddc4f927121fbf4136fb53b659f

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/265988

Reviewed: https://review.openstack.org/258143
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=96ab58e6863c92575ada57615b19652e502adfd8
Submitter: Jenkins
Branch: master

commit 96ab58e6863c92575ada57615b19652e502adfd8
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 16:08:00 2015 -0600

    auth_token verify revocation by audit_id

    If the revocation list includes audit_ids, then when doing offline
    validation also validate the token isn't revoked by audit_id.

    Closes-Bug: 1490804
    Change-Id: I483bc57bd38eb81a0905bcaf94e4ea82604919d6

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/266022

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/266045

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/266607

This issue was fixed in the openstack/keystonemiddleware 4.1.0 release.

Reviewed: https://review.openstack.org/266045
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=9c9c1331e0c004897d5f4c5847f7143b56373f10
Submitter: Jenkins
Branch: stable/kilo

commit 9c9c1331e0c004897d5f4c5847f7143b56373f10
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 11:09:14 2015 -0600

    Add audit IDs to revocation events

    The revoked tokens' audit ID is now included in the data returned in
    the revocation list.

    Closes-Bug: 1490804
    Change-Id: Ifcf88f1158bebddc4f927121fbf4136fb53b659f
    (cherry picked from commit d5378f173da14a34ca010271477337879002d6d0)
    Conflicts:
     keystone/tests/unit/test_backend.py

Reviewed: https://review.openstack.org/266022
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=bff03b5726fe5cac93d44a66715eea49b89c8cb0
Submitter: Jenkins
Branch: stable/liberty

commit bff03b5726fe5cac93d44a66715eea49b89c8cb0
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 11:09:14 2015 -0600

    Add audit IDs to revocation events

    The revoked tokens' audit ID is now included in the data returned in
    the revocation list.

    Closes-Bug: 1490804
    Change-Id: Ifcf88f1158bebddc4f927121fbf4136fb53b659f
    (cherry picked from commit d5378f173da14a34ca010271477337879002d6d0)
    Conflicts:
     keystone/tests/unit/test_backend.py

Reviewed: https://review.openstack.org/265988
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=9167afaafc216c63fdc875e737bc379400e94059
Submitter: Jenkins
Branch: stable/liberty

commit 9167afaafc216c63fdc875e737bc379400e94059
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 16:08:00 2015 -0600

    auth_token verify revocation by audit_id

    If the revocation list includes audit_ids, then when doing offline
    validation also validate the token isn't revoked by audit_id.

    Backport notes:
    - test_auth_token_middleware was refactored with commit 9cbd47b to
      check responses differently, so the test changed to use the old
      method.
    - reno was not supported so the release note is removed.

    Closes-Bug: 1490804
    Change-Id: I483bc57bd38eb81a0905bcaf94e4ea82604919d6
    (cherry picked from commit 96ab58e6863c92575ada57615b19652e502adfd8)

This issue was fixed in the openstack/keystone 9.0.0.0b2 development milestone.

I've put the stable/kilo task back to In Progress. The fix is half committed it is still missing:
  https://review.openstack.org/#/c/266607/

Reviewed: https://review.openstack.org/266607
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=70eeda3e06f5422aa953a38fe93d302079d7cc21
Submitter: Jenkins
Branch: stable/kilo

commit 70eeda3e06f5422aa953a38fe93d302079d7cc21
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 16:08:00 2015 -0600

    auth_token verify revocation by audit_id

    If the revocation list includes audit_ids, then when doing offline
    validation also validate the token isn't revoked by audit_id.

    Backport notes:
    - test_auth_token_middleware was refactored with commit 9cbd47b to
      check responses differently, so the test changed to use the old
      method.
    - reno was not supported so the release note is removed.

    Closes-Bug: 1490804
    Conflicts:
     keystonemiddleware/auth_token/__init__.py

    (cherry picked from commit 96ab58e6863c92575ada57615b19652e502adfd8)
    Change-Id: I483bc57bd38eb81a0905bcaf94e4ea82604919d6

Related fix proposed to branch: master
Review: https://review.openstack.org/273679

Reviewed: https://review.openstack.org/273679
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=1e03c88750db3cfdd01a59c6084b624b22c78c72
Submitter: Jenkins
Branch: master

commit 1e03c88750db3cfdd01a59c6084b624b22c78c72
Author: Tristan Cacqueray <email address hidden>
Date: Thu Jan 28 12:24:15 2016 -0500

    Adds OSSA-2016-005 (CVE-2015-7546)

    This change also remove issues 'type' which isn't used and can't
    be extended to support other type such as OSSN.

    Change-Id: I037c8e808466bbdceac38d6cf10a3f98703ad99f
    Related-Bug: #1490804

This issue was fixed in the openstack/keystone 8.1.0 release.

This issue was fixed in the openstack/keystonemiddleware 2.3.3 release.