Cause conflicts within glance public metadefs

Bug #1479385 reported by Niall Bunting
22
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Triaged
High
Unassigned
Declined for Juno by Alan Pevec
Nominated for Kilo by Tristan Cacqueray
OpenStack Security Advisory
Won't Fix
Undecided
Tristan Cacqueray

Bug Description

Overview:

Through creation of a new public namespace by any user of the system, you can create a clash of namespaces, that breaks all accessibility to that namespace. This therefore can be used to cause a denial of service attack or you have to disable the service completely.

How to produce:

As a regular user run the command:
curl -v -X POST http://16.49.138.140:9292/v2/metadefs/namespaces -H "Content-Type: application/json" -H "X-Auth-Token: 1a499605071a46a8b9b2a938fac5fac7" -d '{"namespace": "OS::Computer::WebServers", "visibility": "public"}'

This will create a new namespace with the same name as the existing namespace. This has now rendered the original namespace inaccessible. If a GET request is done to the namespaces name by any other user via (or viewing in horizon):
curl -v -X GET http://16.49.138.140:9292/v2/metadefs/namespaces/OS::Computer::WebServers -H "Content-Type: application/json" -H "X-Auth-Token: 1a499605071a46a8b9b2a938fac5fac7"

It will cause the following output in the api console:
2015-07-28 23:41:42.175 ERROR glance.api.v2.metadef_properties [req-e3a80995-6f37-4e5c-b7dd-a1ce978478c7 f76c222365fb490792300f9e49ec9bd0 9db14ac3320b4396b58222f99dd04e4e] Multiple rows were found for one()

Returning a 500 to the user and therefore the namespace inaccessible meaning a successful denial of service to most of the metadefs api as most require it.

Attempted preventative measures:
In the policy.json files there are only the following values:
"get_metadef_namespace": "",
 "get_metadef_namespaces":"",
 "modify_metadef_namespace":"",
"add_metadef_namespace":"",
meaning that creating namespaces has to be disabled completely(not default ) as there in no publicize option.

Revision history for this message
Grant Murphy (gmurphy) wrote :

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

description: updated
Changed in ossa:
status: New → Incomplete
Revision history for this message
Niall Bunting (niall-bunting) wrote :

The two fixes that need to be in place are:
1. The not allowing to create two namespaces with the same name.
2. Adding a policy to disallow non-admins publicizing namespaces.

Revision history for this message
Nikhil Komawar (nikhil-komawar) wrote :

Niall, are you working on solution for this?

OpenStack Vulnerability team: I would like to add Travis Tripp to subscribers as he is most familiar with metadef use cases and would like a solution proposal from him.

Changed in glance:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Niall Bunting (niall-bunting) wrote :

Nikhil, I am not currently working on this, however I would be happy to.

Revision history for this message
Lakshmi N Sampath (lakshmi-sampath) wrote :

Verified the issue. Seems like duplicate namespaces are getting created which was not the case before. Will have to verify the db code.

Revision history for this message
Wayne (wayne-okuma) wrote :

This bug has a fix that was submitted back in June: https://review.openstack.org/#/c/195820/

Revision history for this message
Nikhil Komawar (nikhil-komawar) wrote : Re: [Bug 1479385] Re: Cause conflicts within glance public metadefs

Thanks Wayne. Did not click then!

On 8/7/15 2:14 PM, Wayne wrote:
> This bug has a fix that was submitted back in June:
> https://review.openstack.org/#/c/195820/
>

--

Thanks,
Nikhil

Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :

There is a previous bug/patch for this:

https://review.openstack.org/#/c/195820

but I'm not sure if the security aspect of it was covered.

From my -- fairly limited -- understanding of metadefs it does seem like there is a vulnerability here: a DOS of the metadefs API.

Changed in ossa:
status: Incomplete → Confirmed
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Is there others side effects beside the metadata API not being available ? For example, in a Kilo devstack setup, having the metadef API unavailable (returning 500) does not prevents instances to start or the image list to be displayed in Horizon.

I'm not familiar with that feature, is there a configuration option to enable its use by the others API ?

Assuming this was introduced in Juno, here is the impact description draft #1:

Title: Glance metadata v2 API DoS through duplicate namespaces
Reporter: Niall Bunting (HP)
Products: Glance
Affects: 2014.2 versions through 2014.2.3 and 2015.1.0 versions through 2015.1.1

Description:
Niall Buntin from HP reported a vulnerability in the Glance metadata v2 API. By creating duplicated namespaces, an authenticated user may crash the Glance metadata v2 API, resulting in a denial of services. Only setups using the Glance V2 API are affected by this flaw.

Changed in ossa:
status: Confirmed → Triaged
assignee: nobody → Tristan Cacqueray (tristan-cacqueray)
Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :

Hi Tristan,

Thanks for the draft.

> Title: Glance metadata v2 API DoS through duplicate namespaces

I'd suggest changing "metadata" here to "metadefs". I think that makes it clearer that we're talking about the /v2/metadefs resource

Description: typo: Niall Buntin -> Niall Bunting

@Niall

Is the original metadef description recoverable after a user has written over it? Or has it been lost?

Revision history for this message
Niall Bunting (niall-bunting) wrote :

@Stuart

It's not lost its still in the database. However its just accessible through the API, with control over the database you could delete the other interfering entry.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Thanks for the review, here is the updated impact description draft #2:

Title: Glance metadefs v2 API DoS through duplicate namespaces
Reporter: Niall Bunting (HP)
Products: Glance
Affects: 2014.2 versions through 2014.2.3 and 2015.1.0 versions through 2015.1.1

Description:
Niall Bunting from HP reported a vulnerability in the Glance metadata v2 API. By creating duplicated namespaces, an authenticated user may crash the Glance metadata v2 API, resulting in a denial of services. Only setups using the Glance V2 API are affected by this flaw.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

The fix https://review.openstack.org/#/c/195820/ does not applies properly to stable/juno at least.

Since it's already public, backports could be pushed to gerrit (referencing bug 1468946) while we keep this bug private to finalize the advisory side.

Revision history for this message
Jeremy Stanley (fungi) wrote :

I've subscribed the OSSG core security reviewers too.

I'm dubious about keeping this bug private. At this point bug 1468946 is public and mostly explains the risk (without explicitly calling it a denial of service). The fix is public in master and seems very close to being approved. Making this bug public will also make it a lot easier to explain to the stable branch reviewers why the backports are urgent and can hopefully speed up resolution there.

Unless there are serious objections, I want to switch this bug to public security this Wednesday, August 19.

Revision history for this message
Robert Clark (robert-clark) wrote :

Given the circumstances, I agree with Jeremy's proposed path of action (OSSG CoreSec)

Revision history for this message
Travis McPeak (travis-mcpeak) wrote :

+1 - enough of the issue is already public. It's a simple jump to go from what's listed in 1468946 to DoS.

Jeremy Stanley (fungi)
information type: Private Security → Public Security
description: updated
Revision history for this message
Thierry Carrez (ttx) wrote :

The fix for master included a database schema migration, which is usually out of bounds for a stable backport (stable branches are a source of safe and automated fixes -- we need people to be able to upgrade without manual risky or lengthy steps like a database migration).

Is there any other way to fix the issue that does not involve database schema changes ?

Revision history for this message
Wayne (wayne-okuma) wrote :

I don't think there is a better way than to have the database system reject duplicates automatically. Otherwise, will have to introduce code to check before inserting rows that each record's keys don't already exist. This would be redundant to what the unique constraints do.

Revision history for this message
Wayne (wayne-okuma) wrote :

Also, not running the migration won't get the user out of the DOS against a duplicate namespace.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

If this can't be backported, then we'll have to make this a class B1 ( https://security.openstack.org/vmt-process.html#incident-report-taxonomy ).

@ossg-coresec, thought about OSSA vs OSSN here ?

Revision history for this message
Travis McPeak (travis-mcpeak) wrote :

How about back-porting the input validation solutions to the branches where we don't want to do database migration?

My concern with a note is that I'm not sure what we would advise deployers to do. If we're fairly comfortable recommending the policy change mentioned by Niall in comment #2, a note around that advice seems appropriate.

Revision history for this message
Niall Bunting (niall-bunting) wrote :

@Travis We can't really advice what I mentioned due to the fact that there is no policy that will disallow publicizing images. The only other thing we could do is advise them to implement the fix.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Until this can be safely backported, the OSSA task is switched to Won't fix.

Changed in ossa:
status: Triaged → Won't Fix
information type: Public Security → Public
Changed in glance:
assignee: nobody → Khuong Luu (organic-doge)
assignee: Khuong Luu (organic-doge) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.