[OSSA 2014-021] auth token is exposed in meter http.request (CVE-2014-4615)

notifier.py grabs all environment variables. it should probably filter out HTTP_X_AUTH_TOKEN

Once a bug is public, we won't handle it privately. Setting it back to public

Fix proposed to branch: master
Review: https://review.openstack.org/94666

Reviewed: https://review.openstack.org/94666
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=09281ccf7837b70962ad2dfbaa1e84722ad987e8
Submitter: Jenkins
Branch: master

commit 09281ccf7837b70962ad2dfbaa1e84722ad987e8
Author: Gordon Chung <email address hidden>
Date: Tue May 20 12:30:41 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/94770

I think this one will need an OSSA. I suspect that meter is traditionally read by people other than tenant admins ?

Fix proposed to branch: master
Review: https://review.openstack.org/94878

Fix proposed to branch: master
Review: https://review.openstack.org/94891

Reviewed: https://review.openstack.org/94878
Committed: https://git.openstack.org/cgit/openstack/pycadf/commit/?id=966d4410a1a69e0a3af678442a1a965dae80d720
Submitter: Jenkins
Branch: master

commit 966d4410a1a69e0a3af678442a1a965dae80d720
Author: Gordon Chung <email address hidden>
Date: Thu May 22 10:11:52 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: I11d9f2f23fc3b60c945c33d4d02bd7640d88a083
    Closes-Bug: #1321080

It seems that this was introduced in Icehouse, thus we are still missing those patches:
* Ceilometer master and stable/icehouse
* Neutron stable/icehouse

Gordon, could you please propose Ceilometer fixes as well ?

Fix proposed to branch: master
Review: https://review.openstack.org/96943

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/96944

@Tristan, thanks for letting me know. i completely forgot we had middleware module in Ceilometer.

@Gordon, so I have a couple question for the impact description draft.

From the bug description, it appears that the leaked "request.HTTP_X_AUTH_TOKEN: 4724" is not the same than the one provided in the curl command "-H 'X-Auth-Token: 258ab"
So is the leak the token of the user requesting the notifier, or is it the admin_token defined in [filter:authtoken] configuration ?

The conditions for this leak to happen is when the notifier middleware is set after authtoken, which is not by default right ?

so the leaked HTTP_X_AUTH_TOKEN value is the one in provided in curl command (i assume the description is using curl command and request object that aren't related)... it is not the admin_token defined in [filter:authtoken] configuration

you are correct that the leak happens only if notifier middleware is used after auth_token middleware (which it usually is)... by default the notifier middleware is not enabled in any service.

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/96943
Reason: this code doesn't work against master due to switch to oslo.messaging. abandoning for this bug fix: https://bugs.launchpad.net/ceilometer/+bug/1327084

@Zhi Kun Liu, Havana is impacted as well ?

@All, While oslo-incubator is not supported, should we include it in this OSSA ? Is it realistic to use this middleware out of Oslo in another service or only Neutron and Ceilometer are actually impacted ?

In the meantime, here is the impact description draft #1:

Title: User token leak to message queue in the notifier middleware
Reporter: Zhi Kun Liu (IBM)
Products: Neutron, Ceilometer, Oslo
Versions: 2014.1.1

Description:
Zhi Kun Liu from IBM reported a vulnerability in the notifier middleware available in Neutron and Ceilometer or through the Oslo library. An attacker with read access to the message queue may obtain authentication tokens used in REST requests (X_AUTH_TOKEN) that goes through the notifier middleware. All services using the notifier middleware configured after the auth_token middleware pipeline are impacted.

There are 2 copies of the notifier middleware in different places in Oslo.

The copy in the incubator is used by projects that have not yet updated to oslo.messaging, such as neutron.

There is also a copy in the PyCADF library, used by projects that have updated to oslo.messaging, such as ceilometer.

Based on the history here, it looks like both copies have been fixed, so I think changing the impact description to say "the PyCADF library" instead of "the Oslo Library" will make it clear which library needs to be updated.

@Doug, Thanks for clarifying!
Though from https://wiki.openstack.org/wiki/Security_supported_projects, oslo-incubator, oslo.messaging and PyCADF are not security supported projects (at least not in OSSA territory).

However if the notifier middleware is known to be used in services other than Neutron and Ceilometer, I'm wondering how to cover that.

The incubator isn't on the OSSA list because the code in the incubator is copied into other projects that are on the list, and it's assumed that changes go into the incubator before being released into the project(s) using the modules.

oslo.messaging and PyCADF are new releases from the oslo program, and are being added to the list (probably during Juno, but that's not set for certain).

I'm not certain what uses the notifier middleware. Technically, it's middleware, so it don't have to be included in a project for a deployer to use it.

@Gordon, do you have any insight into other projects using the middleware?

the original blueprint for notifier middleware is: https://blueprints.launchpad.net/ceilometer/+spec/count-api-requests. i'm unaware of anyone using the notifier middleware on its alone. to my knowledge, the main consumer of notifier middlware is pyCADF (and its audit middlware).

regarding the audit middleware:

the audit middleware (from oslo-incubator) was synced into Neutron in icehouse as a side effect of another patch (so it may not even be used). the audit middleware was also synced into Ceilometer in havana i believe (to my knowledge it's not used either as pycadf is not a requirement in Ceilometer)

the audit middleware (from pycadf) was purposely set as a requirement in Nova in icehouse and is used (it is optionally enabled by deployer). this audit middleware (from pycadf) did not exist before icehouse.

i'm not aware of any other projects pulling in pyCADF (and it's audit middleware).

hope this brain dump helps :)

Reviewed: https://review.openstack.org/94891
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bb4f44654f6765c4e1fbcf92375c273494151099
Submitter: Jenkins
Branch: master

commit bb4f44654f6765c4e1fbcf92375c273494151099
Author: Gordon Chung <email address hidden>
Date: Thu May 22 10:51:25 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Closes-Bug: #1321080
    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d

@Tristan Cacqueray, I checked nova and neutorn codes in Havana, they don't have audit and notifier middleware. So this does not impact Havana. It's only an internal problem of us. Thanks for your reminding!

OK, this is confusing. Let me try to get an accurate picture of affected versions:

oslo-incubator contains affected code in master (patched), stable/icehouse (in review) and stable/havana
That code was copied in:

Neutron: Juno (patched), Icehouse
Ceilometer: Icehouse (in review), Havana

Then it was adopted in:
pyCADF all versions <= 0.5 (0.5.1 contains the fix)

My understanding is that oslo.messaging is not affected.

Adjusted tasks to match.
Here is how I would rewrite the advisory:

------
Title: User token leak to message queue in pyCADF notifier middleware
Reporter: Zhi Kun Liu (IBM)
Products: Neutron (2014.1 versions up to 2014.1.1)
          Ceilometer (2013.2 versions up to 2013.2.3, 2014.1 versions up to 2014.1.1)
          pyCADF library (all versions up to 0.5.0)

Description:
Zhi Kun Liu from IBM reported a vulnerability in the notifier middleware available in the PyCADF library and formerly copied into Neutron and Ceilometer code. An attacker with read access to the message queue may obtain authentication tokens used in REST requests (X_AUTH_TOKEN) that goes through the notifier middleware. All services using the notifier middleware configured after the auth_token middleware pipeline are impacted.
------

NB: that would mean from now on we support PyCADF, but I think now would be a good time to start.

Reviewed: https://review.openstack.org/94770
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=354a9f99d177fd14d86e099ee3ffa91b9d12b5bd
Submitter: Jenkins
Branch: stable/icehouse

commit 354a9f99d177fd14d86e099ee3ffa91b9d12b5bd
Author: Gordon Chung <email address hidden>
Date: Tue May 20 12:30:41 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080
    (cherry picked from commit 09281ccf7837b70962ad2dfbaa1e84722ad987e8)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/100414

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/101097

Reviewed: https://review.openstack.org/101097
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0324965a0c2987e5cad6276f011682dec184205f
Submitter: Jenkins
Branch: stable/icehouse

commit 0324965a0c2987e5cad6276f011682dec184205f
Author: Grant Murphy <email address hidden>
Date: Thu Jun 19 02:30:13 2014 +0000

    remove token from notifier middleware

    oslo-incubator sync to address the security bug
    in middleware (as below).

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

Reviewed: https://review.openstack.org/100414
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=d97bd2a564cb06c613678407fd074985be73f4d5
Submitter: Jenkins
Branch: stable/havana

commit d97bd2a564cb06c613678407fd074985be73f4d5
Author: Gordon Chung <email address hidden>
Date: Tue May 20 12:30:41 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080
    (cherry picked from commit 09281ccf7837b70962ad2dfbaa1e84722ad987e8)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/101799

Reviewed: https://review.openstack.org/101799
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=264f3b0d9640edeac743f339786e0a3b22c0f6c2
Submitter: Jenkins
Branch: stable/havana

commit 264f3b0d9640edeac743f339786e0a3b22c0f6c2
Author: Grant Murphy <email address hidden>
Date: Mon Jun 23 05:07:54 2014 +0000

    remove token from notifier middleware

    oslo-incubator sync to address the security bug
    in middleware (as below).

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

Reviewed: https://review.openstack.org/96944
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=2b6454f9f4e0585949ab68a91ed405755438d76e
Submitter: Jenkins
Branch: stable/icehouse

commit 2b6454f9f4e0585949ab68a91ed405755438d76e
Author: gordon chung <email address hidden>
Date: Fri May 30 17:11:18 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

why is the CVE for this still not public? It still just says it has been reserved... "This candidate has been reserved by an organization or individual that will use it when announcing a new security problem. When the candidate has been publicized, the details for this candidate will be provided."

I'm guessing this was just an oversight. Can someone fix it?

It takes months, sometimes years for MITRE to come back to a reserved CVE and fill the appropriate information on their website. In the mean time, the CVE number serves as a reference number for all the people that need to coordinate on an issue.

It was publicly assigned by MITRE in http://www.openwall.com/lists/oss-security/2014/06/24/6 and sometimes it takes their editorial board a while to compose and publish the official CVE description (can be on the order of several months).