Eventlet green threads not released back to the pool leading to choking of new requests

Thanks for the report! The OSSA task is set to incomplete pending additional details from nova-coresec.

It sounds like a slowloris DoS like scenario... What are the side effect of disabling keepalive ?

Presently, we are trying to fix on one issue in Glance [1] and during running functional tests, we have found out some of the tests were failing because greenthreads are not released back to the eventlet green pool (client socket connection is not closed) due to which the glance api service doesn't shutdown gracefully. Fixing functional tests in glance without fixing this issue would be temporary solution (we will need to pass “connection: close” to the header for every request made in the functional tests).

Could you please suggest how to go about this security issue and the glance patch?

[1] (glance Patch) https://review.openstack.org/#/c/110012/

So the trick here is, I'm not sure disabling keepalive is an acceptable patch, as it breaks backward compatibility for existing users of the API. It also affects all OpenStack services. So I expect this patch to take a bit of discussion before we agree on the way forward.

So my suggestion would be to fix the test with a temporary solution, if possible silently enough so that we don't reveal this issue while doing so.

Hi Mark Washenberger:

Would you please suggest what should be done with this patch https://review.openstack.org/#/c/117191/?
Presently, glance doesn't handle SIGHUP signal properly (glance process exits) and because of it, it is not possible to reload the config changes. We really want this patch to land in Juno-3 and are ready to do whatever you suggest.

Note: Please ignore https://review.openstack.org/#/c/110012/ patch as my team member wanted to set dependency and he couldn't do it correctly.

@ttx how do you see disabling keepalive breaking backwards compat? Keepalive should just be an optimization, not impact the application semantics.

I based that comment on the "Note" in the bug description: "it might impact all existing applications built with the assumptions that OpenStack API services uses persistent connections" -- do you think it's a non-existant concern?

I think it's a non existant concern. HTTP keep alive exists below the application level. It's a pretty standard scaling technique in Apache configuration to disable HTTP keep alive because it means slow clients take proportionally more resource than fast clients (basically inverts your ability to service max throughput).

I think blanket disable of keepalive is probably the sane thing to do.

Hi,

We have done some testing using python-clients, httplib and requests api and haven't seen any issue.
With the same connection, its processing multiple api's, one after another.

OK, that sounds like a sane move then.

I'm still on the fence on the need for an advisory here, though, as this makes DoS harder but not impossible... so it sounds like a good strengthening move, but it's not really closing a vulnerability, just mitigating one we have to live with anyway ?

Realistically, all the service launch code has been refactored into Oslo code now, so there is probably a common fix there. This actually isn't familiar enough to me any more to be able to propose the fix.

Never mind, this isn't in the common service layer, here would be the Nova patch. Probably would make it configurable by the operator.

Hi Sean,

IMO, there is no need to keep this keepalive as a configurable parameter. Because if someone has configured the keepalive as True then probably he will end-up with this issue.

This is a cinder patch to fix this issue.

This is a glance patch to fix this issue.

This is a keystone patch to fix this issue.

This is a neutron patch to fix this issue.

Abhishek, for internal well defined clouds, keepalive could be a useful optimization. So config option would probably be fine.

It would be good if someone who has working tests for this sees if the eventlet change fixes things.

Adding OSSG-coresec to get a wider perspective on this.

This is one of those bugs where it's difficult to backport the fix, since it changes established behavior and is therefore not really acceptable as a stable branch patch. If there is a workaround for icehouse/havana, then we can document it in a security note. If there isn't a workaround, we might want to backport the option, and default it to current behavior, and then issue an OSSN about the issue and how you can manually switch that option on.

This is the cinder patch with configurable keepalive option.

This is the glance patch with configurable keepalive option.

This is the keystone patch with configurable keepalive option.

This is the neutron patch with configurable keepalive option.

This is the nova patch with configurable keepalive option.

So my preference would be to backport the fix as this seems like a pretty trivial DoS. If that is deemed unacceptable, then I would support the solution that Thierry proposed in msg #19. Specifically:

"we might want to backport the option, and default it to current behavior, and then issue an OSSN about the issue and how you can manually switch that option on."

Adding Ihar, Alan and Adam to get the pulse on the stable side of things.
The question is: would you consider it OK to disable keepalive as part of a stable update ?

General rule states that 'Incompatible config file changes' are forbidden. It's not clear what *incompatible* config file changes are though. If keepalive is considered to be speed optimization and does not change the way clouds behave on API level (I guess the answer to both assumptions is 'yes'?), then I think we won't introduce any incompatibilities in existing deployments by disabling it. So I'm in favour of flipping the default value to avoid shipping code that is insecure by default.

That said, the change will probably require explicit notice in release notes, and there should be a way to re-enable keepalive for those who know what they are doing.

Also, Havana was mentioned in the comments. The thing is that the last Havana release was tagged this Monday, so there is no way for us to deliver the fix to users, and since Monday we don't consider Havana as a supported branch anyway. So in terms of backports, we should be interested in Icehouse only.

Still, I think it's worth to mention Havana in CVE, so that affected distributions are explicitly notified to backport those patches to their downstream repositories.

I'm personally fine with the proposed patches going to stable/icehouse (keep it enabled by default, provide a config option and notes in release notes for operators to consider) That is, assuming we are okay with adding the new config option to stable (I am) I dont' have an example off the top of my head, but we've done similar in the past. We need to provide the toggle, but whether we keep it enabled by default is where it gets sticky. We can either keep it enabled by default and reference the bug as we normally do, or change behavior + disable it and be more vocal about it in the release notes. TBH I could go either way if people feel strongly, but the patches propose LGTM as-is.

OK, let's just backport this, and try to get it all out in time for Juno release.
We need an impact desc, and we need coresec reviews on proposed patches.

Is there a second aspect to this?

A client can currently open a connection and do nothing with it.

The socket will never be closed (even with the above keepalive change in place).

(This can be alleviated to some degree by timing out such idle client sockets https://review.openstack.org/#/c/119365/)

Hi,

Should I need to resubmit the patches by setting default value for keepalive as False?

Please advise.

@Abhishek: no, master patches look good. We'll need the same patches for the stable/icehouse branches, and get them reviewed by core reviewers here on this bug, before we can coordinate their publication.

@ttx what do you think of the second aspect to this? Namely that a client can choke the system by opening connections to the server and doing nothing with them. Unlike the keepalive case this doesn't require authentication.

@Stuart: If I understand correctly, you're describing a Slowloris attack, for which there is no great solution, only mitigation... But then, why close a keepalive-based hole if we keep a simpler one open ? If we require security proxies on open API ports anyway, why close the keepalive hole ? I'll discuss this with the Vulnerability Management Team today.

In general I think we should be striving to create a system that's secure by default. An open-source project that's only secure when you buy x,y,z to put in front of it is less than ideal. Though I accept that in cases like this Sloworis style attacks can potentially be mitigated by some open-source proxies too.

Maybe an OSSN or even a chapter on the Security Guide directly address this sort of setup might make sense ? - I'm considering only the aspects discussed by @ttx and @stuart directly above...

So just to confirm... we have a potential for a denial of service condition here. The proposed patch mitigates it for authenticated sessions, but does nothing to solve the more general issue which also can be triggered without authentication and for which we can't reasonably backport a solution.

This sounds more like something we need to document and hope it can be solved more completely in the future.

If you pass socket_timeout parameter to the wsgi server, then the telnet client connection is closed automatically by the server when the specified timeout elapses.

Stuart Mclaren has proposed this patch for the cinder project.
https://review.openstack.org/#/c/119365/

So to recap, considering both keep-alive disabling and eventlet socket timeout, it now boils down to how effective these changes are:
* Can a single user still reserve all the resource as shown in this bug description ?
* Is the default timeout proposed (900 seconds) enough to prevent trivial attacks as shown in this bug description ?

Combined together these patches might warrant an embargoed advisory if they makes this type of attack really more difficult,
else we should consider marking this public security and focus on a good documentation (OSSN and/or security guide).

Also, for what it worth, as long as the relationship between the client side resources and server side resources is a linear one, I guess we should not consider an OSSA, especially an embargoed OSSA that will make the coordination much more difficult...

Just a datapoint for ceilometer WRT the proposed approach of setting the eventlet.wsgi.server keep-alive to false in order to mitigate this issue.

Ceilometer does not currently use eventlet.wsgi.server ... instead we use wsgiref.simple_server with pecan, and AFAICS an analogous means of disabling keep-alive is not surfaced in that case.

Hi Thierry,

Should I submit the patches with socket_timeout parameter as well?

Please advise.

@Abishek: I think we need to decide first if it makes sense to fix a specific case when the general case is left open. It will probably be solved after Juno release, since this is hard to coordinate one week away from release.

Still no clear way forward on this one. Does it make sense to close the authenticated thread exhaustion through keepalive while unauthenticated users can just get the same effect as easily using slowloris to starve connections ?

@ttx
I think it makes sense to address the 'authenticated thread exhaustion' issue regardless. Even from a non-security standpoint, it could help to prevent an accidental DoS by a poorly written client.

@Nathan: Oh, we should definitely fix this, there is no question about that. The question is, do we consider it a vulnerability (and issue an OSSA for it), do we need to keep this embargoed (while opening it up would hugely facilitate the fix), and do we need to backport it to previous releases (and bend our stable rules to do so).

If we consider it a security enhancement rather than a vulnerability, then the answer to those 3 above questions is "no". Since there are more convenient ways to achieve the same effect (DoS on endpoints) and we wouldn't patch those, it feels like considering this one a specific vulnerability that this fix would "close" is at best misleading. So my vote goes to considering it a security enhancement.

@ttx
I'm in agreement with you that this should be considered a security enhancement and not treated as an embargoed vulnerability. I do think that this is a case where backporting to stable releases would be ideal.

OK, so unless someone complains, we'll switch this to public bug and start pushing patches publicly on Thursday to harden the future releases.

Fix proposed to branch: master
Review: https://review.openstack.org/130821

Fix proposed to branch: master
Review: https://review.openstack.org/130824

Fix proposed to branch: master
Review: https://review.openstack.org/130834

Fix proposed to branch: master
Review: https://review.openstack.org/130839

Fix proposed to branch: master
Review: https://review.openstack.org/130843

It is recommended Keystone is deployed under mod_wsgi, not eventlet, this still does affect Keystone's eventlet deployment model.

Reviewed: https://review.openstack.org/130821
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=fc87da7eeb3451e139ee71b31095d0b9093332ce
Submitter: Jenkins
Branch: master

commit fc87da7eeb3451e139ee71b31095d0b9093332ce
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 02:31:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    In order to maintain the backward compatibility, setting wsgi_keep_alive
    as True by default. Recommended is set it to False.

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: Ic57b2aceb136e8626388cfe4df72b2f47cb0661c

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/131987

Reviewed: https://review.openstack.org/131987
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=b57c024bfb5f542e0412f268038f38bf52e06456
Submitter: Jenkins
Branch: stable/juno

commit b57c024bfb5f542e0412f268038f38bf52e06456
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 02:31:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    In order to maintain the backward compatibility, setting wsgi_keep_alive
    as True by default. Recommended is set it to False.

    Conflicts:
            cinder/wsgi.py
            etc/cinder/cinder.conf.sample

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: Ic57b2aceb136e8626388cfe4df72b2f47cb0661c
    (cherry picked from commit fc87da7eeb3451e139ee71b31095d0b9093332ce)

Reviewed: https://review.openstack.org/130843
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=04d7a724fdf80db51e73f12c5b8c982db9310742
Submitter: Jenkins
Branch: master

commit 04d7a724fdf80db51e73f12c5b8c982db9310742
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I399b812f6d452226fd306c423de8dcea8520d2aa

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/138365

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/138368

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/138811

Reviewed: https://review.openstack.org/138365
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=4041d30611baa476d33627f5078d5bcc12ef50eb
Submitter: Jenkins
Branch: stable/icehouse

commit 4041d30611baa476d33627f5078d5bcc12ef50eb
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 02:31:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    In order to maintain the backward compatibility, setting wsgi_keep_alive
    as True by default. Recommended is set it to False.

    Conflicts:
            cinder/wsgi.py
            etc/cinder/cinder.conf.sample

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: Ic57b2aceb136e8626388cfe4df72b2f47cb0661c
    (cherry picked from commit fc87da7eeb3451e139ee71b31095d0b9093332ce)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/140304

Fix proposed to branch: master
Review: https://review.openstack.org/144074

Reviewed: https://review.openstack.org/130834
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8e7a0dbb12082f6159d98a4628fb8a6fcd05e95a
Submitter: Jenkins
Branch: master

commit 8e7a0dbb12082f6159d98a4628fb8a6fcd05e95a
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:15:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections. Hence even after
    the response is sent to the client, it doesn't close the client socket
    connection. Because of this problem, the green thread is not released
    back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Added a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I3a361d6590d1800b85791f23ac1cdfd79815341b

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/144708

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/144714

Reviewed: https://review.openstack.org/144708
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a89ae1bd93dea2dd1902c1d49572329f6646abcc
Submitter: Jenkins
Branch: stable/juno

commit a89ae1bd93dea2dd1902c1d49572329f6646abcc
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:15:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections. Hence even after
    the response is sent to the client, it doesn't close the client socket
    connection. Because of this problem, the green thread is not released
    back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Added a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I3a361d6590d1800b85791f23ac1cdfd79815341b
    (cherry picked from commit 8e7a0dbb12082f6159d98a4628fb8a6fcd05e95a)

Reviewed: https://review.openstack.org/140304
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f88dc8495f7ec5dc7f1b01c43f0237141af6b98f
Submitter: Jenkins
Branch: stable/juno

commit f88dc8495f7ec5dc7f1b01c43f0237141af6b98f
Author: Stuart McLaren <email address hidden>
Date: Fri Sep 5 12:48:04 2014 +0000

    Add client_socket_timeout option

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added client_socket_timeout option (default=0).

    Conflicts:
            cinder/wsgi.py
            etc/cinder/cinder.conf.sample

    Note: This patch is not 1:1 cherry-pick, I have changed the default value
    of client_socket_timeout to 0.

    Change-Id: If492810a2f10fa5954f8c8bb708b14be0b77fb90
    Closes-bug: #1361360
    Closes-bug: #1371022
    (cherry picked from commit 08bfa77aeccb8ca589e3fb5cf9771879818f59de)

Reviewed: https://review.openstack.org/138368
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=965fd3059eff0fdc7228754742042bbff3abf705
Submitter: Jenkins
Branch: stable/juno

commit 965fd3059eff0fdc7228754742042bbff3abf705
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=0).

    Conflicts:
            nova/tests/unit/test_wsgi.py

    Note: The required unit-tests are manually added to the below path,
    as new path for unit-tests is not present in stable/juno release.
    nova/tests/compute/test_host_api.py

    This patch is not 1:1 cherry-pick, I have changed the default value
    of client_socket_timeout to 0, as per the policy for changes to
    stable branches.
    (https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes)

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I399b812f6d452226fd306c423de8dcea8520d2aa
    (cherry picked from commit 04d7a724fdf80db51e73f12c5b8c982db9310742)

Reviewed: https://review.openstack.org/138811
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a657582c5cd8a39580c44ad401fd3e69870068b1
Submitter: Jenkins
Branch: stable/icehouse

commit a657582c5cd8a39580c44ad401fd3e69870068b1
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=0).

    Conflicts:
            nova/tests/unit/test_wsgi.py

    Note: The required unit-tests are manually added to the below path,
    as new path for unit-tests is not present in stable/icehouse release.
    nova/tests/compute/test_host_api.py

    This patch is not 1:1 cherry-pick, I have changed the default value
    of client_socket_timeout to 0, as per the policy for changes to
    stable branches.
    (https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes)

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I399b812f6d452226fd306c423de8dcea8520d2aa
    (cherry picked from commit 04d7a724fdf80db51e73f12c5b8c982db9310742)

Reviewed: https://review.openstack.org/130839
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=16a821e00d15520d2f6e940e184bd289b8782620
Submitter: Jenkins
Branch: master

commit 16a821e00d15520d2f6e940e184bd289b8782620
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:39:59 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections. Hence even after
    the response is sent to the client, it doesn't close the client socket
    connection. Because of this problem, the green thread is not released
    back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added http_keepalive option (default=True).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I93aaca24935a4f3096210233097dd6b8c5440176

This should not be committed to icehouse branches. It is incompatible with the minimum eventlet version described in icehouse's globall-requirements.txt

@Adam, we have Cinder patch applied. Does it mean we should revert it?

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/162964

Hi Adam, Ihar,

In global-requirements.txt eventlet version is mentioned as 'eventlet>=0.13.0,<0.16.0' and keepalive option is availabe in version 0.13.2 but socket_timeout is not available in the same version.

The patch which was backported to stable/icehouse for cinder [1] does not include 'socket_timeout' applied to wsgi server, so there is no need to revert this patch.

[1] https://review.openstack.org/#/c/138365/

If keepalive is enough to fix this bug, nova/neutron Icehouse backports could be amended to drop socket_timeout part.
If not, Cinder Icehouse fix is incomplete.

Hi Alan,

keepalive is enough to fix this bug.

Reviewed: https://review.openstack.org/144714
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4cdbefc202acc2499c253da625db48d6165b7c84
Submitter: Jenkins
Branch: stable/icehouse

commit 4cdbefc202acc2499c253da625db48d6165b7c84
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:15:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections. Hence even after
    the response is sent to the client, it doesn't close the client socket
    connection. Because of this problem, the green thread is not released
    back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Icehouse backport note: socket_timeout was dropped, it was introduced
    in 0.14[*] and Icehouse eventlet lower bound is 0.13

    [*] https://github.com/eventlet/eventlet/commit/7d4916f01462de09cb58853d9de2e85777c2ad5b

    DocImpact:
    Added wsgi_keep_alive option (default=True).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I3a361d6590d1800b85791f23ac1cdfd79815341b
    (cherry picked from commit 8e7a0dbb12082f6159d98a4628fb8a6fcd05e95a)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/164643

Eventlet is deprecated for keystone and not the primary deployment method. For many reasons it is recommended that keystone not be deployed under event let

Reviewed: https://review.openstack.org/130824
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=3b08644eb9cf4c5aca51a36250ae93105c17f6c4
Submitter: Jenkins
Branch: master

commit 3b08644eb9cf4c5aca51a36250ae93105c17f6c4
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:10:57 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I03b9c5c64f4bd8bca78dfc83199ef17d9a7ea5b7

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/177670

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/177686

Reviewed: https://review.openstack.org/177670
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=67cda0ccae04471340bcada099d945d90979e64d
Submitter: Jenkins
Branch: stable/kilo

commit 67cda0ccae04471340bcada099d945d90979e64d
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:10:57 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I03b9c5c64f4bd8bca78dfc83199ef17d9a7ea5b7
    (cherry picked from commit 3b08644eb9cf4c5aca51a36250ae93105c17f6c4)

Reviewed: https://review.openstack.org/164643
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=392dc228034bbd8968f4c65ddfce6343bff938ea
Submitter: Jenkins
Branch: stable/icehouse

commit 392dc228034bbd8968f4c65ddfce6343bff938ea
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Icehouse backport note: socket_timeout was dropped, it was introduced
    in 0.14[*] and Icehouse eventlet lower bound is 0.13

    [*] https://github.com/eventlet/eventlet/commit/7d4916f01462de09cb58853d9de2e85777c2ad5b

    Note: The required unit-tests are manually added to the below path,
    as new path for unit-tests is not present in stable/icehouse release.
    nova/tests/test_wsgi.py

    DocImpact:
    Added wsgi_keep_alive option (default=True).

    SecurityImpact

    Conflicts:
            nova/tests/unit/test_wsgi.py

    Closes-Bug: #1361360
    (cherry picked from commit 04d7a724fdf80db51e73f12c5b8c982db9310742)

    Change-Id: I3b14a37edbe4bdc5db31ff4f08f78e78b60077ff

Reviewed: https://review.openstack.org/144074
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=2be72f42f9debf9c2f186212dd27cb4f860d3399
Submitter: Jenkins
Branch: master

commit 2be72f42f9debf9c2f186212dd27cb4f860d3399
Author: yangxurong <email address hidden>
Date: Mon Mar 30 14:39:17 2015 +1000

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s
    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).
    SecurityImpact

    Change-Id: I303d87addeed8b103eeb26dbcc48e3acce06ee6a
    Closes-Bug: #1361360

Reviewed: https://review.openstack.org/162964
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=d569ed9db9dc1941ef74d38f85f8f67a85ff10b0
Submitter: Jenkins
Branch: stable/juno

commit d569ed9db9dc1941ef74d38f85f8f67a85ff10b0
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:39:59 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections. Hence even after
    the response is sent to the client, it doesn't close the client socket
    connection. Because of this problem, the green thread is not released
    back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added http_keepalive option (default=True).

    Conflicts:
            doc/source/configuring.rst
            etc/glance-api.conf
            glance/common/wsgi.py
            glance/tests/unit/test_opts.py

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I93aaca24935a4f3096210233097dd6b8c5440176
    (cherry picked from commit 16a821e00d15520d2f6e940e184bd289b8782620)

Fix proposed to branch: master
Review: https://review.openstack.org/204705

Since there is unlikely to ever be a patch for the slowloris side of this, I'm opening an OSSN task.

Reviewed: https://review.openstack.org/204705
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=accb273c7aa369f624e8062137b356dc3e5589cf
Submitter: Jenkins
Branch: master

commit accb273c7aa369f624e8062137b356dc3e5589cf
Author: Valeriy Ponomaryov <email address hidden>
Date: Wed Jul 22 20:30:42 2015 +0300

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after the
    response is sent to the client, it doesn't close the client socket connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    In order to maintain the backward compatibility, setting wsgi_keep_alive
    as True by default. Recommended is set it to False.

    This is port of Cinder change - [1]

    [1] Ic57b2aceb136e8626388cfe4df72b2f47cb0661c

    SecurityImpact
    Closes-Bug: #1361360

    Change-Id: If9241e6f6ba10592a64ca312cb479e8cea929913

Reviewed: https://review.openstack.org/177686
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=d2d6aba069ea3101dfbc3363689eb6142ffb6d1f
Submitter: Jenkins
Branch: stable/juno

commit d2d6aba069ea3101dfbc3363689eb6142ffb6d1f
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:10:57 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Conflicts:
            keystone/common/config.py
            keystone/common/environment/eventlet_server.py

    NOTE: This is not 1:1 cherry-pick because 'eventlet_server' config
    group is not present in juno.

    Closes-Bug: #1361360
    Change-Id: I03b9c5c64f4bd8bca78dfc83199ef17d9a7ea5b7
    (cherry picked from commit 3b08644eb9cf4c5aca51a36250ae93105c17f6c4)
    (cherry picked from commit 67cda0ccae04471340bcada099d945d90979e64d)

There isn't anything specific to OpenStack about this. Since we don't really have any good advice other than limiting connections, I'm closing the OSSN for this. Anybody interested should look into "slowloris attacks".

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/224941

Reviewed: https://review.openstack.org/224941
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=5f86d05c50e02a895093d4cb6e4285dce4ba93a2
Submitter: Jenkins
Branch: stable/kilo

commit 5f86d05c50e02a895093d4cb6e4285dce4ba93a2
Author: yangxurong <email address hidden>
Date: Mon Mar 30 14:39:17 2015 +1000

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s
    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).
    SecurityImpact

    Change-Id: I303d87addeed8b103eeb26dbcc48e3acce06ee6a
    Closes-Bug: #1361360
    (cherry picked from commit 2be72f42f9debf9c2f186212dd27cb4f860d3399)

Reviewed: https://review.openstack.org/231989
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=64583e4b839d96e2cbe7dcba7d0f08852b40bcca
Submitter: Jenkins
Branch: master

commit 64583e4b839d96e2cbe7dcba7d0f08852b40bcca
Author: Sergey Reshetnyak <email address hidden>
Date: Wed Oct 7 16:24:35 2015 +0300

    Use api-paste.ini for loading middleware

    Changes:
    * use api-paste.ini config for loading middleware
    * refactoring middlewares to support loading via pastedeploy
    * use debug middleware from oslo_middleware library instead log_exchange
      middleware

    Closes-bug: #1503983
    Closes-bug: #1361360

    Change-Id: I444c1799ef53dbb19a601e51dd95cd8509fb1c0c