use_forwarded_for exposes metadata

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

This seems like a real information leak.

@nova-coresec please confirm/triage this bug report.

@nova-coresec, any progess ?

I haven't tested this, but this definitely sounds like a real problem to me. For example, the root password for the instance would be exposed if this works.

That said, I also can't think of an immediate fix which wouldn't require fairly significant architecture work. I need to think on that more.

Two immediate work arounds, neither of them great:

 * tell people to use config drive
 * stop honouring X-Forwarded-For (i.e. no proxies)

I don't love either, but that's what I can think of.

There is a 3rd option which IMO is slightly less terrible but probably still an ops wart.

* Add a config option that is the list of trusted IPs (ie proxy servers) and only use the X-Forwarded-For header if the actual peer address is in that list. There is no sane default that will work for ops so if we go this way we'll need to call it out in the release notes / OSSA

Yes this moves the trust issue to the proxy server/load balancer squid[1] and f5 have trivial ways to do that.

[1] http://www.squid-cache.org/Doc/config/forwarded_for/
[2] https://devcentral.f5.com/questions/prevent-x-forwarded-for-spoofing

So in keystone, glance and some others I have been transitioning services over to using the http_proxy_to_wsgi middleware [1] which handles this in the paste pipeline.

If you deprecate the CONF.use_forwarded_for option in nova (as other services have) then you can insert by default the http_proxy_to_wsgi middleware in front of nova-api but not the metadata service which would mean X-Forwarded-For was ignored in metadata services.

This would obviously cause problems for anyone who is deploying the metadata service behind a load balancer (which i'm told is a real usecase) but that can be solved by adding the middleware to the metadata pipeline manually.

This doesn't help with the current problem but would give you a sensible default and a way forward.

[1] http://git.openstack.org/cgit/openstack/oslo.middleware/tree/oslo_middleware/http_proxy_to_wsgi.py

It needs work, I'm running tox/tempest over it now but this is what I was thinking.

I wonder if it's worth making trusted_forwarders understand network notation, I don't if the list will get long without it.

It wont cherry=pick cleanly to stable/* as the config options stuff has moved a lot of that code around but I think functionally the code will move pretty easily.

Jamie validated that the API auth stuff isn't affected.

We have some longer term work to do here but lets address the immediate problem.

Passes tox and tempest. I'll drum up some reviews ....

So I thought X-Forwarded-For was a known weakness, all be it a badly documented one.

The configuration option used to say: "Only enable this if you have a sanitizing proxy."
https://github.com/openstack/nova/commit/ca1017988f98a246aa82c16f471791c7ea3eceec

Now somehow that has morphed to "a HTML santizing proxy" which is totally miss-leading:
https://github.com/openstack/nova/blob/master/nova/conf/api.py#L48

We do now have this shared secret checking feature, that the neutron proxy uses:
https://github.com/openstack/nova/blob/2a0fce7a44d4050e112a6b987f24236627632d2c/nova/api/metadata/handler.py#L259
I believe the proxy uses OVS (et al) protections to ensure only the correct VMs can talk to their proxy, then shared secret sent to the load balanced nova metadata api server, or something like that.

@tonyb, I think the issue with your extra option is that with spoofing the source IP, or similar, it doesn't really give us any more security, unless I am missing something? In reality folks really should consider carefully about securing this channel, via some other means they can trust within the context of their network.

What about beefing up the documentation, issuing a security advisory? I think we could also deprecate the proxy feature, telling users to configure a proxy to use the above shared secret system? Does that sound practical?

FWIW, I know many users worry about instance traffic being routed into the control plane, so tend to only deploy with Configuration Drive.

PS
Having access does also allow you to write passwords into the database, blocking the user generating a password, if you time it correctly, if my memory servers me correctly.

John is right, the new config option is just a feature-poor version of iptables.

I think the right option here is up the documentation and work with the VMT on the OSSA to point people at best practice implementations of a correct proxy setup.

A challenge is I can really only help squid/f5 config in front of the meta data server.

So for most folks, they will be using the shared secret from a neutron proxy, which I believe is much safer, as neutron secures the connections to the neutron proxy (I guess we should double check that).

I wouldn't worry about limiting it to squid/f5 config. I thought with nova-network multi node it was more "normal" to run a metadata api node on very compute node, to avoid the need for a proxy. We can largely ignore the nova-network needs LB case, given the nova-network deprecation. Documenting a few examples of the kinds of thing you need to do seems like a good starting point. It gives folks a place to add to, if they want to share how they do it.

If the plan is to fix/improve documentation and config comments to mark this as clearly unsafe without additional mitigation, then the announcement process would be to have the OSSN editors draft and publish an OpenStack Security Note detailing the risks. I've subscribed the OSSP's core security reviewers to chime in on any concerns with that option.

I'm trying to understand the impact - what is the sensitivity of the data that goes into Nova metadata?

This is definitely an information leak, I'm just trying to understand how bad. For example, can we advise not to put anything sensitive in Nova metadata or is that one of the core purposes?

The primary purpose of metadata is to configure an instance on first boot. It therefore includes networking setup information, root password (if any), ssh public keys, and so forth. It is also commonly extended to include cryptographic material required to register the instance with things like Active Directory.

OK so it sounds like this is a pretty serious issue. Thoughts on OSSA vs OSSN? At minimum we should do an embargoed OSSN so that stakeholders have a chance to implement our recommendations before public notification IMO.

OSSA would only be if there's an actual patch we're backporting to "fix" this in existing supported releases. OSSN is more for "don't do this, or if you're going to then make sure you do it this way" sorts of issues.

It sounds like the thinking here is that this is and always was a feature whose use is discouraged unless you're actually coupling it with some other (external to Nova) filtering solution, but the documentation warning about it has gotten too weak in recent releases. If so, that's more OSSN territory (an article warning that if you're going to turn on proxy support for your metadata server then make sure you have some means of controlling its availability) in addition to shoring up the existing warnings in Nova itself.

Ahh ok, good point. My only concern is if people ARE using it this way already they might be in for an unpleasant surprise.

OSSN is new to me but I think that's the right solution here for everything that's released (including I suppose Newton) I think it'll be Ocata before we "fix" this.

I'm switching the OSSA task status to opinion until a Security Note (OSSN) is defined.

Agree this should be an OSSN.

It's been repeated plenty of times in the community that the metadata
service shouldn't be used for sensitive data but I've also seen multiple
examples of production services doing this.

I share Travis' concern that this could be a major issue for some users. It
would be useful to know if there's a way an operator can assess if
sensitive data is stored in the metadata service (I've seen SSH keys, LDAP
credentials etc before)

This seems like a good candidate for an embargoed release

-Rob

On Wed, Aug 17, 2016 at 3:31 PM, Travis McPeak <email address hidden> wrote:

> ** Changed in: ossn
> Assignee: (unassigned) => Travis McPeak (travis-mcpeak)
>
> --
> You received this bug notification because you are a member of OSSG
> CoreSec, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1563954
>
> Title:
> use_forwarded_for exposes metadata
>
> Status in OpenStack Compute (nova):
> Confirmed
> Status in OpenStack Security Advisory:
> Opinion
> Status in OpenStack Security Notes:
> New
>
> Bug description:
> This issue is being treated as a potential security risk under
> embargo. Please do not make any public mention of embargoed (private)
> security vulnerabilities before their coordinated publication by the
> OpenStack Vulnerability Management Team in the form of an official
> OpenStack Security Advisory. This includes discussion of the bug or
> associated fixes in public forums such as mailing lists, code review
> systems and bug trackers. Please also avoid private disclosure to
> other individuals not already approved for access to this information,
> and provide this same reminder to those who are made aware of the
> issue prior to publication. All discussion should remain confined to
> this private bug report, and any proposed fixes should be added to the
> bug as attachments.
>
> --
>
> The nova metadata service uses the remote address to determine which
> metadata to retrieve. In order to work behind a proxy there is an
> option use_forwarded_for which will use the X-Forwarded-For header to
> determine the remote IP.
>
> If this option is set then anyone who can access the metadata port can
> request metadata for any instance if they know the IP.
>
> The user data is also exposed.
>
> $ echo 123456 > /tmp/data
> $ openstack server create --image CentOS7 --flavor fedora --user-data
> /tmp/data test
> <wait>
> $ curl -H 'X-Forwarded-For: 10.0.0.7' http://localhost:8775/latest/
> user-data/
> 123456
>
> At a minimum this side-effect isn't documented anywhere I could find.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1563954/+subscriptions
>

Any progress on this security note ?

Hi Tristan,

The note is written and in internal review.

Recently we've started using restricted google docs to review the content for embargoed OSSNs. The Gerrit process we normally use is great but I'm understanding a little bit more of the VMT's pain when it comes to doing things out-of-process.

For expediency here's the current draft, I'm happy to take comments here:

Nova metadata service should not be used for sensitive information
---

### Summary ###
A recent security report has highlighted how users may be using the
metadata service to store inappropriate information.

The Nova metadata service should not be considered a secure repository
of confidential information required by compute instances.

### Affected Services / Software ###
Nova, <I'm guessing all releases with metadata service?>

### Discussion ###
A recent vulnerability report for Nova stated that the metadata service
will obey the `X-Forwarded-For` HTTP header. This header is often
supplied by proxies so that the end service can identify which IP the
request originated from.

The Nova metadata service typically uses the source IP address of the
incoming request to respond with the appropriate data for the compute
instance making the request. This is a sort of weak authentication,
designed to ensure that metadata for one tenant isn't accidentally
provided to another.

If the request contains a `X-Forwarded-For` HTTP header then the
metadata service will use that for the source authentication rather than
the actual TCP/IP source.

An attacker with access to a compute instance in the cloud could send a
request to the metadata service and include the `X-Forwarded-For` header
in order to effectively spoof their source and cause the metadata
service to provide information that should not have been provided to
that instance.

Consider the following:
Alice creates a compute instance. She places the root password for that
instance in the metadata service. The instance is assigned a 10.1.2.2
IP address. Alice believes that the root password for her instance is
safe within the metadata service.

Alice retrieves metadata by running a command similar to:
`curl http://169.254.169.254/latest/meta-data`
this will retrieve any metadata stored for Alice's compute instance,
which has an IP address of 10.1.2.2

Bob has a compute instance with IP address 10.1.9.9 however Bob wants
access to the metadata for Alice's compute instance. If Bob runs a
similar command to Alice, but includes a customer header as below, he
will get access to all of Alice's metadata, including the root password
she chose to store there:
`curl -H "X-Forwarded-For: 10.1.2.2" http://169.254.169.254/latest/meta-data`

The Nova metadata service is a useful utility within OpenStack but
clearly not intended as a strongly authenticated system for storing
sensitive data such as private keys or passwords.

### Recommended Actions ###
The metadata service should not be used to store sensitive information.

The IP forwarding issue is not a defect of itself, it exists to allow
the metadata service to provide IP addresses for instances that are
behind a proxy as may be the case in more complex deployments.

Cloud users who ...

@tristan-cacqueray I've made a couple of small tweaks and the OSSP cores have reviewed. We need the VMT to review and a core from Nova before we publish.

Hi VMT / Nova Cores,

Can you please provide review on Roberts note, or give it your blessing to get published?

Luke

The draft content looks reasonable. Could we see the final draft?

Hi Grant, I believe its #23

I mean with the blanks filled in e.g.

### Affected Services / Software ###
Nova, <I'm guessing all releases with metadata service?>

For the record I went back through the eol tags to diablo and they're all vulnerable.

So I think we can say Nova <All versions>

The OSSN look good to me but it's be nice to see some of the mitigations mentioned
1. squid/f5 config
2. running a metadata server on each compute

If you have a google-linked email address (i.e you can view google docs with it) reply here and I'll add you to the working draft.

Thanks Robert. tony.breeds@{gmail,bakeyournoodle}.com should do that thing

Nova metadata service should not be used for sensitive information
---

### Summary ###
A recent security report has highlighted how users may be using the
metadata service to store security sensitive information.

The Nova metadata service should not be considered a secure repository
of confidential information required by compute instances.

### Affected Services / Software ###
Nova, All Versions

### Discussion ###
A recent vulnerability report for Nova stated that the metadata service
will obey the `X-Forwarded-For` HTTP header. This header is often
supplied by proxies so that the end service can identify which IP the
request originated from.

The Nova metadata service typically uses the source IP address of the
incoming request to respond with the appropriate data for the compute
instance making the request. This is a sort of weak authentication,
designed to ensure that metadata for one tenant isn't accidentally
provided to another.

If the request contains a `X-Forwarded-For` HTTP header then the
metadata service will use that for the source authentication rather than
the actual TCP/IP source.

An attacker with access to a compute instance in the cloud could send a
request to the metadata service and include the `X-Forwarded-For` header
in order to effectively spoof their source and cause the metadata
service to provide information that should not have been provided to
that instance.

Consider the following:
Alice creates a compute instance. She places the root password for that
instance in the metadata service. The instance is assigned a 10.1.2.2
IP address. Alice believes that the root password for her instance is
safe within the metadata service.

Alice retrieves metadata by running a command similar to:
`curl http://169.254.169.254/latest/meta-data`
this will retrieve any metadata stored for Alice's compute instance,
which has an IP address of 10.1.2.2

Bob has a compute instance with IP address 10.1.9.9 however Bob wants
access to the metadata for Alice's compute instance. If Bob runs a
similar command to Alice, but includes a customer header as below, he
will get access to all of Alice's metadata, including the root password
she chose to store there:
`curl -H "X-Forwarded-For: 10.1.2.2" http://169.254.169.254/latest/meta-data`

The Nova metadata service is a useful utility within OpenStack but
clearly not intended as a strongly authenticated system for storing
sensitive data such as private keys or passwords.

### Recommended Actions ###
The metadata service should not be used to store sensitive information.

The IP forwarding issue is not a defect of itself, it exists to allow
the metadata service to provide IP addresses for instances that are
behind a proxy as may be the case in more complex deployments.

Cloud users who have a requirement to store sensitive information that
compute instances require for operation should instead look to the
Config drive to provide this service. It's operation is much more
tightly bound to individual compute instances.

Where use of config drive is not an option, operators should consider
other mitigations such as placing a proxy in front of the metadata service
which can filter ou...

@tony if you want to provide alternative language for the last paragraph (mitigations) then I'd be happy to swap it out.

Robert: Did you mean to say "custom header" instead of "customer header"? Also, in the second-to-last paragraph you used "It's" (a conjunction) where you wanted "Its" (possessive, so no apostrophe).

Those minor editorial nits aside, it reads well. Thanks for writing this up!

Thanks Jeremy, I'll update the draft on gdocs, I'll wait to hear back from Tony for a +1 on the content.

We need a +1 from a Nova core.

Very sorry. The text in Comment 32 LGTM. For the record I'm nova-coresec but not nova-core

I will put this note out to downstream steakholders this evening, unless any objections?

No objection from me, but please get up with me or someone else from the VMT first for an updated list (we've had at least one recent addition).

Note sent to downstream stakeholders. Will release to public 7 days from now (19/12).

OSSN sent to public lists

[that was me correcting the bug type/tags a moment ago, I just forgot I was still logged into a test account instead of this one]

this bug might already fixed, i tried to reproduce but failed with devstack.
[stack@devstack devstack]$ openstack server create --image centos7 --flavor ds1G --network f2a39df5-0938-4973-9810-a80d341229bf --user-data /tmp/data test1
+-------------------------------------+------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | TuHp5eHEMMfW |
| config_drive | |
| created | 2019-12-06T10:32:10Z |
| flavor | ds1G (d2) |
| hostId | |
| id | afdecb55-f3e1-4099-b947-1704112cb9ae |
| image | centos7 (2f2396d2-b32e-4275-93b3-df9fb376dc36) |
| key_name | None |
| name | test1 |
| progress | 0 |
| project_id | 9aafb875525b45b79be2d1ca5d27ffb0 |
| properties | |
| security_groups | name='default' |
| status | BUILD |
| updated | 2019-12-06T10:32:10Z |
| user_id | d994888b0c764c8288cc1162f69b8d8b |
| volumes_attached | |
+---------------------------...

[stack@devstack devstack]$ ip addr | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.199.151/24 brd 192.168.199.255 scope global noprefixroute ens33

[root@devstack nova]# git log
commit d2bf17eaf4c66fc7ffec671e0d2d7ed2b4dde87c
Merge: dd12b3b 0461921
Author: Zuul <email address hidden>
Date: Thu Dec 5 01:24:30 2019 +0000

    Merge "Cache security group driver"

commit dd12b3b407ea3d5b8cdaf404c43b7d25b2e3927a
Merge: 0a1b604 846fc0a
Author: Zuul <email address hidden>
Date: Thu Dec 5 00:30:00 2019 +0000

add my ip of my environment is above. and my current branch is at master

shall we close this bug

Any idea what would have fixed it? My understanding of this report was that it's an architectural issue in deployments where if you set up a proxy (for example a load balancer) between the guest instances and the metadata service then you need to be able to tell the metadata service to look at the X-Forwarded-For header added by the load balancer to know which instance is making the request. If you set that and your network is mis-designed to allow instances to also contact the metadata service directly without passing through the load balancer, then the request from the instance can be specially constructed so as to include a spoofed X-Forwarded-For header which allows it to obtain metadata for a different instance associated with the address included in that injected header.

Experience crystal-clear voice calls and lightning-fast data connectivity that keeps you ahead in today's fast-paced world.
https://aadishesh.com/