[OSSA 2016-005] PKI Token Revocation Bypass (CVE-2015-7546)

Bug #1490804 reported by Liusheng
264
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
High
Brant Knudson
Kilo
Fix Released
High
Unassigned
OpenStack Security Advisory
Fix Released
Undecided
Unassigned
OpenStack Security Notes
Fix Released
Critical
Nathan Kinder
django-openstack-auth
Invalid
Undecided
Adam Young
keystonemiddleware
Fix Released
High
Brant Knudson
python-keystoneclient
Won't Fix
Undecided
Adam Young

Bug Description

A keystone token which has been revoked can still be used by manipulating particular byte fields within the token.
When a Keystone token is revoked it is added to the revoked list which stores the exact token value. Any API will look at the token to see whether or not it should accept a token. By changing a single byte within the token, the revocation can be bypassed. see the testing script [1].

It is suggested that the revocation should be changed to only check the token's inner ID.

[1] http://paste.openstack.org/show/436516/

CVE References

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Changed in ossa:
status: New → Incomplete
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

The tests scripts does not show how token are created/revoked. What type of token is this ?

Maybe it's a swift setting that allows direct access to the swift account.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Please also be aware that paste.openstack.org contents are not secret/private and can be easily enumerated. We should probably consider this issue publicly disclosed at this point since an exploit has been published to the paste server.

description: updated
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

I would like to know what version of keystone is being used here and what type of token.

This is definitely not uuid tokens, and I don't believe this can be fernet as fernet is live validated and does not rely on the token_id for revocations.

Can this occur with services other than swift as well? I'll need to dig further on this unless the information can be provided.

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

If this is PKI/PKIZ tokens, there are no "internal" IDs as the internal ID would be part of the signed data. The signed data cannot contain the ID itself as it would change the ID.

Discussing this a bit further it looks like "revoke by ID" should have been based on fingerprint or the signed data (as per adam young). Alternatively, we can now use the audit_ids for "revoke by id"

I am still in process of confirming outside of swift.

Revision history for this message
Dolph Mathews (dolph) wrote :

This sounds like it must be limited to PKI/PKIZ tokens? Fernet tokens would fail their HMAC and there is no room to manipulate UUID tokens unless you're guessing a completely different one.

In the script to reproduce, mutating token_bin[1] would be manipulating the CMS length prefix (MIIE-*, etc), in which case, I wonder if there's a bug in openssl?

Jeremy: please don't open this bug just because the paste is public. There's nothing in the paste that shouts "security vulnerability," and we should certainly have the ability to delete pastes (and move them here), no?

Changed in keystone:
importance: Undecided → High
Revision history for this message
Jeremy Stanley (fungi) wrote :

I won't switch this to public without consensus anyway, but can assure that there are crawlers continuously spidering our paste server (I know because I see them in the logs and they make up quite possibly the majority of requests there).

Revision history for this message
Liusheng (liusheng) wrote :

sorry, my mistake, this issue is occured with PKI token.

summary: - Token Revocation Bypass
+ PKI Token Revocation Bypass
Revision history for this message
Liusheng (liusheng) wrote : Re: PKI Token Revocation Bypass

This issue is found by my colleagues, I will dig more and provide more information, thanks for your attention.

Revision history for this message
Dolph Mathews (dolph) wrote :

I won't be able to spend much time on this until at least Sept 4th, but I suspect this is easily reproducible and I'm wondering if the fix must be to make PKI depend on revocation events like Fernet does? Or perhaps to ignore the mutable characters when querying the token table?

Revision history for this message
Dolph Mathews (dolph) wrote :

Jeremy: I did not consider crawlers / mirrors. Deleting the paste now might only draw attention to it :)

Revision history for this message
Dolph Mathews (dolph) wrote :

Liusheng: if you need to share any other files/logs/patches related to the bug, please add them as attachments to this bug (using the "Add attachment or patch" link at the bottom of the page).

Revision history for this message
Liusheng (liusheng) wrote :

Dolph: sure, I am sorry I didn't realized the test script shouldn't be pasted in paste.openstack.org.

Revision history for this message
Liusheng (liusheng) wrote :

I have confirmed they guys found this issue in Juno openstack, some version info used:

keystoneclient 0.11.2
keystone 2014.2

openssl-1.0.1i-2.1.4
openssl-certs-1.97-0.3.1

Adam Young (ayoung)
Changed in keystone:
assignee: nobody → Adam Young (ayoung)
Revision history for this message
Adam Young (ayoung) wrote :

This should only be a problem with PKI(Z) tokens. The fix can be made here:

http://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/utils.py

Using the logic in the blog post here: http://adam.younglogic.com/2014/10/who-signed-that-token/ we can pull out the signed data from the token and take a hash of that; modifying the hashed data will mean the token will not longer pass the CMS validation.

This will pull in an additional dependency on pyasn1 to Keystone. We cannot avoid that by calculating the hash on the JSON data itself, as it will not (necessarily) be reproducible.

In addition, Horizon will need an update, as they currently Md5 Hash the tokens before sending them to the remote services.

Another possibility is to generate a UUID for the token ID, and stick that inside the JSON body before signing. revocation could be done on the UUID value instead of the HASH. That would involve changing both Keystone and keystonemiddleware.

another alternative it to compare the signer info in the PKIZ token with the Certificate. If they don't match, the token is invalid. I don't know if it is possible to be 100% certain that the teokn cannot be tampered with.

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

Thanks for the information. It helps us to be sure we are looking at all avenues for the problem. I can see where this is happening and have discussed with Adam and a few other folks on it and how we are approachig potential fixes.

Revision history for this message
Adam Young (ayoung) wrote :

The attached code shows the modification of the binary data that still generates a valid token; it can be run from

python-keystoneclient/examples/pki and usese the sample tokens and certificates in the associated subdirectories.

This shows the modified bytes that successfully pass are:
77 78 79 80 81 82 83 85 86 87 88 3979 3980 3981 3982 3983 3985 3986 3987 3988 3991 3992 4364

The range 77 78 79 80 81 82 83 85 86 87 88 looks like the object identitfier s

1 3 14 3 2 26
and
1 2 840 113549 1 7 1

The last byte is padding.

To see the content of the token in PEM format

cat cms/auth_v3_token_scoped.pem | awk ' ! /---/ {print $1}'| base64 -d | /usr/lib64/nss/unsupported-tools/derdump > /tmp/tokendata

Revision history for this message
Adam Young (ayoung) wrote :
Revision history for this message
Dolph Mathews (dolph) wrote :

The attached test manipulates UUID, PKI, PKIz, and Fernet tokens by incrementing and decrementing individual bytes, as illustrated in the bug description. This test passes against each token type, showing that the manipulated tokens cannot be validated *online* against keystone. To be clear, this patch does not contain a fix, nor does it confirm or invalidate the bug report.

Unless I missed something, that means the scope of this bug is limited to validating tokens offline (against the token revocation list in keystonemiddleware.auth_token).

Revision history for this message
Dolph Mathews (dolph) wrote :

Of course, it would help if I attached the patch.

Revision history for this message
Adam Young (ayoung) wrote :

Dolph; yeah, this is only for checking the revocation list; the tokens themselves need to be valid tokens. It is restricted to PKI based tokens only. The mistake was using the hash of the whole document for the token id; it should have been the same hash of the token tdata that is used for the cryptographic signature. The values being manipulated are the ASN1 specific values that indicate "This next block is signed data" which openssl really should be validating.

For example; here is a diff of a derdump of the origianl file and the modified file, with some lines of context;

$ diff -U 10 /tmp/auth_v3_token_scoped_orig.pem.txt /tmp/auth_v3_token_scoped77.pem.txt
--- /tmp/auth_v3_token_scoped_orig.pem.txt 2015-09-02 11:27:46.423313737 -0400
+++ /tmp/auth_v3_token_scoped77.pem.txt 2015-09-02 11:27:46.398313755 -0400
@@ -4,21 +4,21 @@
    C-[0] (3176)
       C-Sequence (3172)
          Integer (1)
             01
          C-Set (9)
             C-Sequence (7)
                Object Identifier (5)
                   1 3 14 3 2 26 (SHA-1)
          C-Sequence (2692)
             Object Identifier (9)
- 1 2 840 113549 1 7 1 (PKCS #7 Data)
+ 0 26 840 113549 1 7 1
             C-[0] (2677)
                Octet String (2673)
                   7b 22 74 6f 6b 65 6e 22 3a 20 7b 22 6d 65 74 68 6f 64
                   73 22 3a 20 5b 22 70 61 73 73 77 6f 72 64 22 5d 2c 20
                   22 72 6f 6c 65 73 22 3a 20 5b 7b 22 69 64 22 3a 20 22
                   66 30 33 66 64 61 38 66 38 61 33 32 34 39 62 32 61 37
                   30 66 62 31 66 31 37 36 61 37 62 36 33 31 22 2c 20 22
                   6e 61 6d 65 22 3a 20 22 72 6f 6c 65 31 22 7d 2c 20 7b
                   22 69 64 22 3a 20 22 66 30 33 66 64 61 38 66 38 61 33
                   32 34 39 62 32 61 37 30 66 62 31 66 31 37 36 61 37 62

All of the places susceptible to the modifications seem to be this kind of string, with the exception of the final padding;
$ diff -u /tmp/auth_v3_token_scoped_orig.pem /tmp/auth_v3_token_scoped4364.pem
--- /tmp/auth_v3_token_scoped_orig.pem 2015-09-02 11:14:11.414915072 -0400
+++ /tmp/auth_v3_token_scoped4364.pem 2015-09-02 11:14:28.749902282 -0400
@@ -66,4 +66,4 @@
 4RQzkShFx97ioGPE9wE1Jv9Yzxo5uTg6z246gi42e50uSIFZMmNMqkUdz0n6Qct+
 1zMf8W+e1pykKscVBz757F9n+qAhx15LxxdCKIrQ1kNT8dAyLRBQU0qZeLmLvS3W
 nmf/JlQ7LDQmq3XQEHRbQ1s5dz/mwM8IiSm8
------END CMS-----
+-----END CMS-----
\ No newline at end of file

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

@dolph,

Yes this would be limited to an auth_token middleware issue. We should retarget the bug.

Changed in keystonemiddleware:
status: New → Confirmed
status: Confirmed → New
importance: Undecided → Critical
importance: Critical → High
assignee: nobody → Adam Young (ayoung)
Changed in keystone:
status: New → Invalid
assignee: Adam Young (ayoung) → nobody
status: Invalid → New
assignee: nobody → Adam Young (ayoung)
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

Added keystonemiddleware as there are likely changes needed to KSM as well

Revision history for this message
Dolph Mathews (dolph) wrote :

The fix could impact the contents of the token revocation list itself, though (in keystone).

Revision history for this message
Dolph Mathews (dolph) wrote :

(read comment #22 & #23 before I saw the new bug status - it seems you already agree!)

Revision history for this message
Adam Young (ayoung) wrote :

The following patch detects modifications of the Token.

I am not certain, however, that this is exhaustive; there might be ways to modify the document such that it is still valid, but has a different hash.

We can lock down what is accepted as a token format: additional checks could confirm that the certs are stripped (adding them will still fail validation anyways) or any other cosmetic changes that could change the document.

We might be able to get away with a stop gap like this, and then migrate people to Fernet or change how PKI tokens are hashed.

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

A few issues that came up in discussion earlier tonight:

* What happens if someone respins a token from PKI to PKIZ or vice-versa?
* PKI the delimiter (---- START CMS ---- and corresponding end ones) could be pre/post substituted in PKI (not PKIZ) and change the resulting ID but still pass due to the token_to_cms (or cms_to_token) code doing a blind substitution between '/' and '-'.

One of the other options could be to provide an options for KeystoneMiddleware to disable offline validations (default to off) and have it hash the tokens the same way django_openstack_auth would (which is not affected by this issue).

I'm looking over the code in the Reject-modified-PKI-Tokens.patch and it is a partial solution, but I have not spent enough time poking at the code to be sure what other gaps are there in PKI(Z) validation.

Revision history for this message
Adam Young (ayoung) wrote :

This patch is probably the apparoach we need to take; it changes the logic of generating a token_id from hashing the whole body of the token to hashing the data, same way that the signing algorithm does.

The patch will not yet pass unit tests.

Django OpenStack Auth needs to be modified to call this function in keystoneclient instead of doing their own hashing.

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

I am going to be -1 on changing DOA and changing how we hash immediately as this will break anyone who does not use the libraries (aka any language but python).

Lets aim for a fix that doesn't break absolutely everyone.

DOA is not impacted by this as it forces live validation.

Let's not change DOA and only change how middleware interacts in a non-live validated token workflow for Liberty (preferably something back-portable).

Revision history for this message
Adam Young (ayoung) wrote :

The DOA fix is trivial. It should go in anyway.

This is the lowest impact change. It won't break anyone unless they are doing the undocumented Hash hack. The number of people that are doing that should be vnishingly small, and they should update was well, too, for the same reason.

We can post code samples in Ruby and Java for how to do the hash.

Revision history for this message
Adam Young (ayoung) wrote :

This is the DOA change, and it can and should be applied regardless. I think it is safe to post as a review publicly.

Changed in python-keystoneclient:
assignee: nobody → Adam Young (ayoung)
Adam Young (ayoung)
Changed in django-openstack-auth:
assignee: nobody → Adam Young (ayoung)
Revision history for this message
Adam Young (ayoung) wrote :

Is there any direction on this one? Online validation is going to break people doing distributed signing, which is the reason to use PKI tokens. Any fix here has the potential to fix someone. Changing the hash approach is, I suspect, going to have the least impact.

Revision history for this message
Adam Young (ayoung) wrote :

Note that always going to online validation would preclude the use of memcache for tokens.

Changed in django-openstack-auth:
status: New → Invalid
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

I repeat the previous statement: Changing the hash mechanism will break everyone. DOA is absolutely not affected here.

DOA forces online validation because it hashes the PKI(z) tokens to the SHA form. If a token is already hashed, it will only rely on the online validation which requires matching the key in the database.

The solution for the time being is going to be addressing how middleware validates the token offline via SSL. Options I see are:

* Done via audit_ids with new forms of middleware if we make audit_ids available as a form of the TRL.
* Option to only online-validate PKI(Z) tokens.
* Offer a different hash-form TRL (not just audit-id) for new versions of middleware
* Deprecate PKI(Z) tokens [not a good idea]
* Issue OSSA and indicate this is not something we can/will fix with the PKI tokens (and deprecate) [not a great idea]

I would prefer to see an online-only validation option with an OSSA as the immediate (and potentially backportable) fix. Discussion on changing hashes etc can be delayed until the summit and done in the open - not under embargo. For the embargo'd bug, I really want to see this not be a breaking change.

Breaking memcache for validated tokens [optionally] where the deployer is aware of the issues (due to OSSA being issued) is fine.

Revision history for this message
Robert Clark (robert-clark) wrote :

Ouch, I'd love to do a security blogpost on this once it's announced, a full root cause analysis.

I'm not close-enough to Keystone to understand the full impact of all the the mitigation options however it's clear that this is a pretty severe issue.

I know the numbers are hard to quantify but "thumb in the air" - what percentage of deployments use PKI tokens?

-Rob

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

@Robert,

It's enough to be a serious consideration (in the <50% but more than 15% is my guess; but this is just hazarding a guess). However we are now seeing a move towards Fernet tokens from the orgs that have been using PKI.

Some of the largest deployments are PKI token based.

Revision history for this message
Travis McPeak (travis-mcpeak) wrote :

This also seems like a prime target for a security note (OSSN). Basically we'll say something along the lines of "PKI(Z) tokens without online validation shouldn't be considered safe for certain versions" (if I understand this correctly).

This will enable cloud deployers to better understand the documented risks they are accepting.

Adam Young (ayoung)
Changed in keystone:
status: New → Confirmed
status: Confirmed → Triaged
Changed in keystonemiddleware:
status: New → Triaged
Changed in python-keystoneclient:
status: New → Triaged
Revision history for this message
Adam Young (ayoung) wrote :

Root cause is the assumption that any tampering with the PKI token would invalidate the token. There are three major fields in the token that are resistant:

1. The message. Any change to the message will be detected due to it not matching the signature.
2. The signature. Any change to the signature and the message will no longer match its signature.
3. The singning info. This checked against the X509 signing cert byte-by-byte. Any change and it won't match.

What I found in testing is that there are other fields that X509 ignores but should not. These include 0 length fields that are place holders for things like the stripped out Certificates.

The other attack now possible is due to there being 2 different PKI token versions. A PKI token could be converted to a PKIZ token, and the signature is still valid. That was not the case when the hashing approach was chosen.

The most common reason for token revocations is logout from the WebUI.

The impact is probably small but consequential; it only affects revoked tokens in the CLI cases. In the webUI case, the tokens are never stored in full except inside the token database. Instead, the WebUI hashes them, and discards the CMS document. Since this is the primary cause of revocations, these tokens are not at risk.

The second most common is password changes. Since these are performed by the end users themselves, the risk here is slight, but not 0.

The most worry some are token revocations for users leaving a project or being disabled altogether. In order for this attack to be viable, the user must have had a valid token prior to the event, and then they could re-enable the token. Since token timeout is a variable, we can;t make any assumptions. We turned the default down to 1 hour, but many sites found this was too short for workflows, and set the value back to 12 hours.

Revision history for this message
Liusheng (liusheng) wrote :

Hi Adam Young,

My colleague has applied your patch with Juno version keystone, the original problem is successfully solved, however some random errors occur when issuing both v2 and v3 token, the tracebacks are as below.

Seems the errors are introduced by the patch, and we didn't found the root cause. traceback info please see the attachment could you please take a look ?

Keystone is running on apache with worker mode.

Version of pyasn1 packages are:
python-pyasn1-modules-0.0.7-33.2
python-pyasn1-0.1.2-0.5.1

OS is Suse SLE11 SP3

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Since this seems related to a bad design in PKI(z) token for which the fix isn't easily backportable, I suggest a B type of bug.

Also, since this requires a fair amount of social engineering to abuse this bug, is there a good reason to keep this under embargo ?

Changed in ossn:
importance: Undecided → Critical
Nathan Kinder (nkinder)
Changed in ossn:
assignee: nobody → Nathan Kinder (nkinder)
Adam Young (ayoung)
Changed in keystone:
status: Triaged → Won't Fix
Changed in keystonemiddleware:
status: Triaged → Won't Fix
Changed in python-keystoneclient:
status: Triaged → Won't Fix
22 comments hidden view all 102 comments
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Considering this bug exploitation requires administrator action (e.g., to issue token revocation), I would prefer this bug report to be public. Operator could use this information to enforce a more strict revocation process while the fix is being implemented.

What is the status of the OSSN ?

How about we open the bug at the same time we issue the security note, then we could review the proposed patch on gerrit and we'll eventually issue an OSSA if the solution is safely backportable ?

Revision history for this message
Brant Knudson (blk-u) wrote :

The exploit doesn't require admin action. Tokens can be revoked by the user, or they can be revoked by the user changing their password. Tokens can also be made invalid by admins taking away the user's roles.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Brant, the only scenario I considered is when a malicious user is able to keep access after an operator revoked his token or took away his roles. iiuc the exploit you described requires valid token, what can the user do with tokens he invalidated himself ?

Revision history for this message
Brant Knudson (blk-u) wrote :

Tristan - The only thing I can think of is the user revokes their own tokens because they're concerned that it's been exposed. So they think they're stopping the 3rd party from using their token but the 3rd party can still use the token by modifying it slightly (until the token expires).

Yes, the typical concern would be an admin thinks the user's old tokens are revoked but they're still usable until they expire.

Revision history for this message
Nathan Kinder (nkinder) wrote :

Tristan - I have the OSSN draft written up, which has been reviewed by a few people from the Security team as well as by Adam Young. If we are also going to provide a fix, I will need to reword a few things in the "recommendations" section. Here is the current draft of the OSSN:

-----
Potential reuse of revoked Identity tokens
---

### Summary ###
An authorization token issued by the Identity service can be revoked,
which is designed to immediately make that token invalid for future use.
When the PKI or PKIZ token formats are used, it is possible for an
attacker to manipulate the token contents of a revoked token such that
the token will still be considered to be valid. This can allow
unauthorized access to cloud resources if a revoked token is intercepted
by an attacker.

### Affected Services / Software ###
Keystone, Icehouse, Juno, Kilo, Liberty

### Discussion ###
Token revocation is used in OpenStack to invalidate a token for further
use. This token revocation takes place automatically in certain
situations, such as when a user logs out of the Dashboard. If a revoked
token is obtained by another party, it should no longer be possible to
use it to perform any actions within the cloud. Unfortuantely, this is
not the case when the PKI or PKIZ token format is used.

When a PKI or PKIZ token is validated, the Identity service checks it
by searching for a revocation by the entire token. It is possible for
an attacker to manipulate portions of an intercepted PKI or PKIZ token
that are not cryptographically protected, which will cause the
revocation check to improperly consider the token to be valid.

### Recommended Actions ###
We recommend that you do not use the PKI or PKIZ token formats. Due to
the way that these formats were designed, it is not possible to fix this
issue in a way that is backwards compatible. The PKI and PKIZ token
formats do not offer any significant benefit over other token formats
such as the UUID or Fernet formats.

If you are using the PKI or PKIZ token format, it is recommended that
you switch to using another supported token format such as the UUID
format. There are no plans to fix this issue in the PKI and PKIZ token
providers in the Identity service.

To check what token provider you are using, you must look in the
'keystone.conf' file for your Identity service. An example is provided
below:

---- begin keystone.conf sample snippet ----
[token]
#provider = keystone.token.providers.pki.Provider
#provider = keystone.token.providers.pkiz.Provider
provider = keystone.token.providers.uuid.Provider
---- end keystone.conf sample snippet ----

This configuration snippet is using the UUID token format. If you are
using either of the commented out settings from this example, your cloud
is vulnerable to this issue and you should switch to a different token
provider.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0061
Original LaunchPad Bug : https://bugs.launchpad.net/keystone/+bug/1490804
OpenStack Security ML : <email address hidden>
OpenStack Security Group : https://launchpad.net/~openstack-ossg

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Thanks, so we could remove the first chapter of "Recommended Actions" and send this OSSN as an advance notification. We'll then open the bug and review Brant's patch on gerrit. Thought ?

Revision history for this message
Jeremy Stanley (fungi) wrote :

It seems reasonable to go forward with the OSSN worded to assume there is no fix forthcoming. Then once there's an actual fix merged to stable branches we can follow up with an OSSA (though worth keeping in mind the proposed backport, as far as I understand, only solves the problem for tokens revoked after the patch gets applied, not any which predate its application).

Revision history for this message
Nathan Kinder (nkinder) wrote :

Jeremy - Agreed. Are we ready to make this issue public then? Is everyone OK with the OSSN draft I provided in my previous comment?

As soon as we make it public, I can propose the OSSN review and fast-track it's merge and get it published.

Revision history for this message
Brant Knudson (blk-u) wrote :

Jeremy - Once the change is applied to keystone then the revocation list will include the audit IDs. The audit IDs are already in the token table so that will be available. As such, the fix will work for old tokens.

Comments on the proposed OSSN

1) change "token formats" to "token providers"

2) In the sample config, since keystone has support for using entrypoints for the provider, the examples should also have:

 #provider = pki
 #provider = pkiz

This was introduced in liberty so earlier releases need to use keystone.token.providers.uuid.Provider whereas liberty and later should use uuid or fernet

Revision history for this message
Jeremy Stanley (fungi) wrote :

Brant: Excellent! That's better than I had hoped (was assuming only the full document hash was being stored for earlier tokens). So this really would, in theory, be a complete fix for stable branch consumers. With that in mind, I agree it makes sense to slightly lighten the bit of the OSSN about no possibility of a backward compatible fix. Still shouldn't imply that there is definitely one coming in case something unforeseen happens in review, but getting awareness of the bug and mitigation recommendations out to deployers/operators in the near term is still a good idea.

Revision history for this message
Nathan Kinder (nkinder) wrote :
Download full text (3.3 KiB)

Here is a new draft of the OSSN that contains the suggestions made by Brant as well the suggestion made by Jeremy with regards to there being a possibility of a future fix. If this looks good to everyone, I think we can move forward with publishing and opening this bug up.

----------

Potential reuse of revoked Identity tokens
---

### Summary ###
An authorization token issued by the Identity service can be revoked,
which is designed to immediately make that token invalid for future use.
When the PKI or PKIZ token providers are used, it is possible for an
attacker to manipulate the token contents of a revoked token such that
the token will still be considered to be valid. This can allow
unauthorized access to cloud resources if a revoked token is intercepted
by an attacker.

### Affected Services / Software ###
Keystone, Icehouse, Juno, Kilo, Liberty

### Discussion ###
Token revocation is used in OpenStack to invalidate a token for further
use. This token revocation takes place automatically in certain
situations, such as when a user logs out of the Dashboard. If a revoked
token is obtained by another party, it should no longer be possible to
use it to perform any actions within the cloud. Unfortunately, this is
not the case when the PKI or PKIZ token providers are used.

When a PKI or PKIZ token is validated, the Identity service checks it
by searching for a revocation by the entire token. It is possible for
an attacker to manipulate portions of an intercepted PKI or PKIZ token
that are not cryptographically protected, which will cause the
revocation check to improperly consider the token to be valid.

### Recommended Actions ###
We recommend that you do not use the PKI or PKIZ token providers. The
PKI and PKIZ token providers do not offer any significant benefit over
other token providers such as the UUID or Fernet.

If you are using the PKI or PKIZ token providers, it is recommended that
you switch to using another supported token provider such as the UUID
provider. This issue might be fixed in a future update of the PKI and
PKIZ token providers in the Identity service.

To check what token provider you are using, you must look in the
'keystone.conf' file for your Identity service. An example is provided
below:

---- begin keystone.conf sample snippet ----
[token]
#provider = keystone.token.providers.pki.Provider
#provider = keystone.token.providers.pkiz.Provider
provider = keystone.token.providers.uuid.Provider
---- end keystone.conf sample snippet ----

In the Liberty release of the Identity service, the token provider
configuration is different than previous OpenStack releases. An
example from the Libery release is provided below:

---- begin keystone.conf sample snippet ----
[token]
#provider = pki
#provider = pkiz
provider = uuid
---- end keystone.conf sample snippet ----

These configuration snippets are using the UUID token provider. If you
are using any of the commented out settings from these examples, your
cloud is vulnerable to this issue and you should switch to a different
token provider.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0061
Original LaunchPad Bug : https://bugs...

Read more...

Revision history for this message
Steve Martinelli (stevemar) wrote :

@Nate, not sure if you also want to mention that the PKI and PKIz token providers are not deprecated in Mitaka.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Once the OSSN is approved, I'd like to send it as an advance notification to stakeholders before opening this bug.
Assuming it is done by tomorrow, this could be disclosed next Tuesday:
2015-12-15, 1500UTC

Revision history for this message
Brant Knudson (blk-u) wrote :

the OSSN in comment 73 looks good to me.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

The OSSN has been sent, let's open this bug next Tuesday (2015-12-15, 1500UTC).

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote : Re: PKI Token Revocation Bypass (CVE-2015-7546)

Red Hat preemptively assigned CVE-2015-7546 for this vulnerability.

summary: - PKI Token Revocation Bypass
+ PKI Token Revocation Bypass (CVE-2015-7546)
information type: Private Security → Public Security
description: updated
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

This bug is now public, OSSN and Keystone patches can be submitted to gerrit.

Revision history for this message
Nathan Kinder (nkinder) wrote :

The OSSN review has been submitted:

  https://review.openstack.org/#/c/258119/

Changed in keystone:
assignee: Adam Young (ayoung) → Brant Knudson (blk-u)
status: Won't Fix → In Progress
Changed in keystonemiddleware:
assignee: Adam Young (ayoung) → Brant Knudson (blk-u)
status: Won't Fix → In Progress
Changed in keystone:
milestone: none → mitaka-2
Revision history for this message
Nathan Kinder (nkinder) wrote :

This issue has been published as OSSN-0062:

  https://wiki.openstack.org/wiki/OSSN/OSSN-0062

Changed in ossn:
status: New → Fix Released
Brant Knudson (blk-u)
tags: added: kilo-backport-potential liberty-backport-potential
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

The fix for keystone[master] is https://review.openstack.org/#/c/258141 . Can this be proposed to stable liberty and kilo too ?

Revision history for this message
Brant Knudson (blk-u) wrote :

Tristan - It can be proposed, although some might consider the change to be a new feature rather than a bug fix, so not sure if it's appropriate for stable. I'd prefer for the change to be merged in master before I backport it otherwise I'll have make changes in multiple releases if there's a -1.

Revision history for this message
Steve Martinelli (stevemar) wrote :

Brant & Tristan, I think it should be fine to backport to kilo and liberty. It's an important security related fix and I consider the proposed changes to be a bug and not a feature.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Thanks, then considering this is going to be fixed in stable release, I've confirmed the OSSA task.

Changed in ossa:
status: Incomplete → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/258141
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=d5378f173da14a34ca010271477337879002d6d0
Submitter: Jenkins
Branch: master

commit d5378f173da14a34ca010271477337879002d6d0
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 11:09:14 2015 -0600

    Add audit IDs to revocation events

    The revoked tokens' audit ID is now included in the data returned in
    the revocation list.

    Closes-Bug: 1490804
    Change-Id: Ifcf88f1158bebddc4f927121fbf4136fb53b659f

Changed in keystone:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystonemiddleware (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/265988

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystonemiddleware (master)

Reviewed: https://review.openstack.org/258143
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=96ab58e6863c92575ada57615b19652e502adfd8
Submitter: Jenkins
Branch: master

commit 96ab58e6863c92575ada57615b19652e502adfd8
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 16:08:00 2015 -0600

    auth_token verify revocation by audit_id

    If the revocation list includes audit_ids, then when doing offline
    validation also validate the token isn't revoked by audit_id.

    Closes-Bug: 1490804
    Change-Id: I483bc57bd38eb81a0905bcaf94e4ea82604919d6

Changed in keystonemiddleware:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/266022

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/266045

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystonemiddleware (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/266607

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/keystonemiddleware 4.1.0

This issue was fixed in the openstack/keystonemiddleware 4.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/kilo)

Reviewed: https://review.openstack.org/266045
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=9c9c1331e0c004897d5f4c5847f7143b56373f10
Submitter: Jenkins
Branch: stable/kilo

commit 9c9c1331e0c004897d5f4c5847f7143b56373f10
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 11:09:14 2015 -0600

    Add audit IDs to revocation events

    The revoked tokens' audit ID is now included in the data returned in
    the revocation list.

    Closes-Bug: 1490804
    Change-Id: Ifcf88f1158bebddc4f927121fbf4136fb53b659f
    (cherry picked from commit d5378f173da14a34ca010271477337879002d6d0)
    Conflicts:
     keystone/tests/unit/test_backend.py

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/liberty)

Reviewed: https://review.openstack.org/266022
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=bff03b5726fe5cac93d44a66715eea49b89c8cb0
Submitter: Jenkins
Branch: stable/liberty

commit bff03b5726fe5cac93d44a66715eea49b89c8cb0
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 11:09:14 2015 -0600

    Add audit IDs to revocation events

    The revoked tokens' audit ID is now included in the data returned in
    the revocation list.

    Closes-Bug: 1490804
    Change-Id: Ifcf88f1158bebddc4f927121fbf4136fb53b659f
    (cherry picked from commit d5378f173da14a34ca010271477337879002d6d0)
    Conflicts:
     keystone/tests/unit/test_backend.py

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystonemiddleware (stable/liberty)

Reviewed: https://review.openstack.org/265988
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=9167afaafc216c63fdc875e737bc379400e94059
Submitter: Jenkins
Branch: stable/liberty

commit 9167afaafc216c63fdc875e737bc379400e94059
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 16:08:00 2015 -0600

    auth_token verify revocation by audit_id

    If the revocation list includes audit_ids, then when doing offline
    validation also validate the token isn't revoked by audit_id.

    Backport notes:
    - test_auth_token_middleware was refactored with commit 9cbd47b to
      check responses differently, so the test changed to use the old
      method.
    - reno was not supported so the release note is removed.

    Closes-Bug: 1490804
    Change-Id: I483bc57bd38eb81a0905bcaf94e4ea82604919d6
    (cherry picked from commit 96ab58e6863c92575ada57615b19652e502adfd8)

Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/keystone 9.0.0.0b2

This issue was fixed in the openstack/keystone 9.0.0.0b2 development milestone.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote : Re: PKI Token Revocation Bypass (CVE-2015-7546)

I've put the stable/kilo task back to In Progress. The fix is half committed it is still missing:
  https://review.openstack.org/#/c/266607/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystonemiddleware (stable/kilo)

Reviewed: https://review.openstack.org/266607
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=70eeda3e06f5422aa953a38fe93d302079d7cc21
Submitter: Jenkins
Branch: stable/kilo

commit 70eeda3e06f5422aa953a38fe93d302079d7cc21
Author: Brant Knudson <email address hidden>
Date: Tue Dec 1 16:08:00 2015 -0600

    auth_token verify revocation by audit_id

    If the revocation list includes audit_ids, then when doing offline
    validation also validate the token isn't revoked by audit_id.

    Backport notes:
    - test_auth_token_middleware was refactored with commit 9cbd47b to
      check responses differently, so the test changed to use the old
      method.
    - reno was not supported so the release note is removed.

    Closes-Bug: 1490804
    Conflicts:
     keystonemiddleware/auth_token/__init__.py

    (cherry picked from commit 96ab58e6863c92575ada57615b19652e502adfd8)
    Change-Id: I483bc57bd38eb81a0905bcaf94e4ea82604919d6

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ossa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/273679

summary: - PKI Token Revocation Bypass (CVE-2015-7546)
+ [OSSA 2015-006] PKI Token Revocation Bypass (CVE-2015-7546)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ossa (master)

Reviewed: https://review.openstack.org/273679
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=1e03c88750db3cfdd01a59c6084b624b22c78c72
Submitter: Jenkins
Branch: master

commit 1e03c88750db3cfdd01a59c6084b624b22c78c72
Author: Tristan Cacqueray <email address hidden>
Date: Thu Jan 28 12:24:15 2016 -0500

    Adds OSSA-2016-005 (CVE-2015-7546)

    This change also remove issues 'type' which isn't used and can't
    be extended to support other type such as OSSN.

    Change-Id: I037c8e808466bbdceac38d6cf10a3f98703ad99f
    Related-Bug: #1490804

summary: - [OSSA 2015-006] PKI Token Revocation Bypass (CVE-2015-7546)
+ [OSSA 2015-005] PKI Token Revocation Bypass (CVE-2015-7546)
Changed in ossa:
status: Confirmed → Fix Released
summary: - [OSSA 2015-005] PKI Token Revocation Bypass (CVE-2015-7546)
+ [OSSA 2016-005] PKI Token Revocation Bypass (CVE-2015-7546)
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/keystone 8.1.0

This issue was fixed in the openstack/keystone 8.1.0 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/keystonemiddleware 2.3.3

This issue was fixed in the openstack/keystonemiddleware 2.3.3 release.

Displaying first 40 and last 40 comments. View all 102 comments or add a comment.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.