DOS : API_RESULT_LIMIT does not work for swift objects

Is it intended that the result limit doesn't work when full_listing is passed? It's not entirely clear to me why there are 2 different parameters controlling API result quantity. Its seems like a valid bug, however.

NOTE: all active branches are affected.

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

So, just to clarify the claimed exploit scenario: an authenticated normal user can upload an excessive number of objects into a Swift container, then request a full listing of that container through Horizon and effect a denial of service against Horizon itself even if the administrator of the deployment has intentionally set API_RESULT_LIMIT to mitigate such attacks?

@Jeremy Stanley Yes, exactly this scenario.

Even if the administrator of the deployment has intentionally set API_RESULT_LIMIT to mitigate such attacks, this limit does not work and an authenticated normal user can upload an excessive number of objects into a Swift container, and then crash the horizon server while viewing this container.

More info, here is the apache LOG when crash occurs:

    [Mon Oct 16 15:41:28.695638 2017] [core:error] [pid 21470:tid 139843967366912] [client 10.12.0.39:46851] End of script output before headers: horizon.wsgi, referer: https://<Sorry_if_I_hide_the_affected_domain>/project/containers/container/<Sorry_if_I_hide_the_customer_container_name>
    [Mon Oct 16 15:41:55.935525 2017] [:error] [pid 24160:tid 139844023068416] [remote 10.12.0.39:28931] mod_wsgi (pid=24160): Exception occurred processing WSGI script '/opt/console-cw/lib/python2.7/site-packages/openstack_dashboard/wsgi/horizon.wsgi'.
    [Mon Oct 16 15:41:55.937407 2017] [:error] [pid 24160:tid 139844023068416] [remote 10.12.0.39:28931] IOError: failed to write data

The crash lead to DOS and was produced by a legit customer who uploaded a million objects in a container.

We have multiple loadballanced horizon instances, but when the crash occurred, the customer hit "F5" and crashed all loadballanced nodes... And there was no more service (DOS).

The provided patch solved the issue.

We deploy Horizon from source "almost" this way (Updated a bit since):
https://dev.cloudwatt.com/en/blog/deploy-horizon-from-source-with-apache-and-ssl.html

So it's not a distro packaging issue, however packaged versions are theoretically impacted too.

Thanks for the updated details. We're still waiting for Horizon security reviewers to confirm their bugtask, evaluate the provided patch in comment #1 or suggest an alternate fix, and determine which stable branches will require corresponding backports.

The fix is valid; it seems like some overly ambitious caching on Horizons side. I think the patch will cause a regression in behaviour without any further work, however. Since security is more important, I think the only way forward would be to fix and add a release note documenting this, unless someone volunteers their time for further investigation.

Does the issue apply to all currently supported stable branches, and is the proposed patch backportable to them (what sort of regression are you anticipating)?

Its valid for every active branch. My concern is that the Swift interface was written expecting all containers to be available for the client to page through; so without the full_listing parameter, it will only return the first 10000 and stop there.

This seems like an acceptable regression, a user having that many container shouldn't be using a web interface to manage them individually anyway right?

Yves-Gwenael, would you mind attaching a properly formatted patch as explained here: https://security.openstack.org/#how-to-propose-and-review-a-security-patch .

Hello Tristan,

Here is the patch.

Thanks Yves-Gwenael,

@horizon-coresec, so is this an acceptable regression that could be backported down to stable/ocata?

In keeping with recent OpenStack vulnerability management policy changes, no report should remain under private embargo for more than 90 days. Because this report predates the change in policy, the deadline for public disclosure is being set to 90 days from today. If the report is not resolved within the next 90 days, it will revert to our public workflow as of 2020-05-27. Please see http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012721.html for further details.

It doesn't look like this report has seen any activity since my update two months ago, so consider this a friendly reminder:

The embargo for this report is due to expire one month from today, on May 27, and will be switched public on or shortly after that day if it is not already resolved sooner.

Thanks!

The embargo for this report has expired and is now lifted, so it's acceptable to discuss further in public.

Was the patch from Comment #15 an acceptable fix? If so I can push to gerrit for reviews.

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/horizon/+/790504

Hi, I am unable to reproduce this bug in my devstack env. but I have purposed
a patch[1] as @Yves-Gwenael Bourhis suggested in comment 15.
let's discussed it in more detail while reviewing it.

[1] https://review.opendev.org/c/openstack/horizon/+/790504

The lack of priority on this over the past 6 years seems to indicate it's not a severe enough risk to warrant a widely published advisory even if a fix ever does merge. The VMT and other OpenStack Security SIG members agreed during the 2023.1 cycle that this should be considered class B2 per our report taxonomy: https://security.openstack.org/vmt-process.html#report-taxonomy