[FG-VD-16-011] Openstack Logging DoS Vulnerability Notification

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

I fail to see why a single connection is more difficult to detect compared to multiple connection... This seems to be a class B2 type of bug according to VMT taxonomy ( https://security.openstack.org/vmt-process.html#incident-report-taxonomy ), the bad design being the lack of rate-limiting.

In glance-api.conf:

# If False, server will return the header "Connection: close", If
# True, server will return "Connection: Keep-Alive" in its responses.
# In order to close the client socket connection explicitly after the
# response is sent and read successfully by the client, you simply
# have to set this option to False when you create a wsgi server.
# (boolean value)
#http_keepalive = true

This is passed to eventlet's wsgi server:

 http://eventlet.net/doc/modules/wsgi.html

 keepalive – If set to False, disables keepalives on the server; all connections will be closed after serving one request.

If you set this I think you need more than one connection in order to send all the GET requests. (But I'm not 100% sure why that's considered more secure?)

You can also increase the default log level, eg to WARN. Then the unauthenticated requests will not create a log entry at all.

> the lack of rate-limiting.

I think there's some consensus in the community that rate limiting should happen outside of the openstack service (you probably want 'C' not 'Python').

Thank stuart for the analysis. Then the B2 class still holds, I'm waiting for feedback from reporter before removing the OSSA task and privacy settings.

The logging exhaust DoS exist in all openstack server, such as neutron, nava, these server also have risk.

more efficent python PoC, and when install openstack, it is almost maybe bring the security risk to system, if have additional server configure parameter can protect from the vulnerability, at least, openstack should post a blog to declare the issue.

The openstack logging DoS vulnerability key cause is openstack server keep record every generated log while repeat do GET request with long url, and openstack server does not reset the long connection, untill openstack log is exhausted.

I test apache with repeat python submit GET request, apache response with 1. after about 14000 request,the server reset the connection; 2.apache only record some repeat log, it does not generate every log record in logger file.

I think only via configure server http keepalive false to protect from the attack is not enough, because the options have negative impact for openstack server performance after set the keepalive option false, so it seems is not good choice, The apache server protect from the attack is 1: too many connection reset, 2: apache access log not record all repeat log. I think apache deal with keepalive method is better.

The way I'm reading Chro's analysis makes me think the issue might not be in Glance itself. Resetting the connection properly is something that, unless I'm misinterpreting the problem, should be taken care of by eventlet.

As far as not repeating log lines, this is something that oslo.log should probably do. At least, provide an API to tell it what lines can/should be filtered when duplicated. Doing this in oslo.log would also help other projects and not just Glance.

Does this make sense?

Chro, the PoC you attached is incomplete. Could you explain why this behavior is more easily detected/prevented when keepalive is disabled ? And about repeating logs, what if the attacker use randomized http payload ?

In short, the vulnerability described here is that unauthenticated and inexpensive calls to the service can generate very large amounts of log data. This type of protection are probably better implemented before the call is even processed by OpenStack's services and this could be documented as a Security Note or as part of the Security Guide.

I've subscribed OSSG-coresec to discuss an eventual document about rate limiting.
If nobody objects, I'd like to close the OSSA task and remove the privacy setting by the end of this week.

I think send many connection with large url is have performance impact for openstack server when keepalive is disabled, so easily cause pay attention and take action such timely empty log, so no result in logging DoS.

If url large payload with randomized, I think there is not better means.

Just to understand the impact - is there an asymmetric aspect to this? I expect most deployments won't suffer an actual DoS as log rotation should be configured, but this may of course cause legitimate log information to be lost.

I also agree rate limiting shouldn't be Glance's job, this seems like the kind of thing we want a universal solution for (something in Oslo as Flavio suggested).

Is there any legitimate reason to allow URLs beyond a reasonable value (for example 4K)? I know most browsers do this already. This seems like a reasonable mitigation for this attack.

Travis' comment seems appropriate. Needs a Glance person to weigh in.

As long as the client needs to repeatedly send the request, the load seems symmetric and this report will likely not warrant an advisory. Any concern if we remove the privacy settings and close the OSSA task ?