| 2016-04-20 20:38:49 |
Chris Auston |
bug |
|
|
added bug |
| 2016-04-20 21:04:55 |
Jeremy Stanley |
description |
You can reproduce this by issuing a GET request for a few hundred MB file and never consuming the response, but keep the client socket open. Swift will log a 499 but the socket does not always close.
ChunkWriteTimeout is meant to protect the proxy from a slow reading client:
https://github.com/openstack/swift/blob/master/swift/proxy/controllers/base.py#L889-L905
Sometimes when this exception is thrown there is still data in the process socket buffer, so when eventlet tries to close the socket it first flushes it:
https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L631
https://hg.python.org/cpython/file/v2.7.11/Lib/SocketServer.py#l711
The problem is that if the client is not consuming the socket buffer then that flush will wait forever; it's trying to write on a socket that just threw a timeout trying to write! The flush write is not protected by any timeout.
As far as I can tell, this WRITE_TIMEOUT does nothing:
https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L407
wsgi.server() takes a socket_timeout that might be what we're after?
https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L437-L440
Even with socket_timeout, eventlet needs to be patched. This should be in a finally block:
https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L636-L637
All of this is probably mitigated by most operators setting an idle timeout in their load balancers, but I wanted to report it. Going directly to a proxy I was able to hold sockets open for long periods of time.
I did the initial research on version 2.2.2 but I was able to reproduce on 2.7.0. I'm trying to translate links to master branch on github. I apologize in advance if it's not quite right. |
This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments.
You can reproduce this by issuing a GET request for a few hundred MB file and never consuming the response, but keep the client socket open. Swift will log a 499 but the socket does not always close.
ChunkWriteTimeout is meant to protect the proxy from a slow reading client:
https://github.com/openstack/swift/blob/master/swift/proxy/controllers/base.py#L889-L905
Sometimes when this exception is thrown there is still data in the process socket buffer, so when eventlet tries to close the socket it first flushes it:
https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L631
https://hg.python.org/cpython/file/v2.7.11/Lib/SocketServer.py#l711
The problem is that if the client is not consuming the socket buffer then that flush will wait forever; it's trying to write on a socket that just threw a timeout trying to write! The flush write is not protected by any timeout.
As far as I can tell, this WRITE_TIMEOUT does nothing:
https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L407
wsgi.server() takes a socket_timeout that might be what we're after?
https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L437-L440
Even with socket_timeout, eventlet needs to be patched. This should be in a finally block:
https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L636-L637
All of this is probably mitigated by most operators setting an idle timeout in their load balancers, but I wanted to report it. Going directly to a proxy I was able to hold sockets open for long periods of time.
I did the initial research on version 2.2.2 but I was able to reproduce on 2.7.0. I'm trying to translate links to master branch on github. I apologize in advance if it's not quite right. |
|
| 2016-04-20 21:05:03 |
Jeremy Stanley |
bug task added |
|
ossa |
|
| 2016-04-20 21:05:16 |
Jeremy Stanley |
ossa: status |
New |
Incomplete |
|
| 2016-04-20 21:05:40 |
Jeremy Stanley |
bug |
|
|
added subscriber Swift Core security contacts |
| 2016-04-30 22:11:01 |
Tristan Cacqueray |
bug |
|
|
added subscriber Victor Stinner |
| 2016-05-09 15:30:45 |
Tristan Cacqueray |
bug |
|
|
added subscriber Morgan Fainberg |
| 2016-05-30 15:43:33 |
Tristan Cacqueray |
bug |
|
|
added subscriber Sergey Shepelev |
| 2016-05-30 15:44:45 |
Tristan Cacqueray |
bug |
|
|
added subscriber Jakub Stasiak |
| 2016-06-02 00:04:04 |
John Dickinson |
bug |
|
|
added subscriber Brian Cline |
| 2016-06-27 15:58:32 |
Tristan Cacqueray |
description |
This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments.
You can reproduce this by issuing a GET request for a few hundred MB file and never consuming the response, but keep the client socket open. Swift will log a 499 but the socket does not always close.
ChunkWriteTimeout is meant to protect the proxy from a slow reading client:
https://github.com/openstack/swift/blob/master/swift/proxy/controllers/base.py#L889-L905
Sometimes when this exception is thrown there is still data in the process socket buffer, so when eventlet tries to close the socket it first flushes it:
https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L631
https://hg.python.org/cpython/file/v2.7.11/Lib/SocketServer.py#l711
The problem is that if the client is not consuming the socket buffer then that flush will wait forever; it's trying to write on a socket that just threw a timeout trying to write! The flush write is not protected by any timeout.
As far as I can tell, this WRITE_TIMEOUT does nothing:
https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L407
wsgi.server() takes a socket_timeout that might be what we're after?
https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L437-L440
Even with socket_timeout, eventlet needs to be patched. This should be in a finally block:
https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L636-L637
All of this is probably mitigated by most operators setting an idle timeout in their load balancers, but I wanted to report it. Going directly to a proxy I was able to hold sockets open for long periods of time.
I did the initial research on version 2.2.2 but I was able to reproduce on 2.7.0. I'm trying to translate links to master branch on github. I apologize in advance if it's not quite right. |
You can reproduce this by issuing a GET request for a few hundred MB file and never consuming the response, but keep the client socket open. Swift will log a 499 but the socket does not always close.
ChunkWriteTimeout is meant to protect the proxy from a slow reading client:
https://github.com/openstack/swift/blob/master/swift/proxy/controllers/base.py#L889-L905
Sometimes when this exception is thrown there is still data in the process socket buffer, so when eventlet tries to close the socket it first flushes it:
https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L631
https://hg.python.org/cpython/file/v2.7.11/Lib/SocketServer.py#l711
The problem is that if the client is not consuming the socket buffer then that flush will wait forever; it's trying to write on a socket that just threw a timeout trying to write! The flush write is not protected by any timeout.
As far as I can tell, this WRITE_TIMEOUT does nothing:
https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L407
wsgi.server() takes a socket_timeout that might be what we're after?
https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L437-L440
Even with socket_timeout, eventlet needs to be patched. This should be in a finally block:
https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L636-L637
All of this is probably mitigated by most operators setting an idle timeout in their load balancers, but I wanted to report it. Going directly to a proxy I was able to hold sockets open for long periods of time.
I did the initial research on version 2.2.2 but I was able to reproduce on 2.7.0. I'm trying to translate links to master branch on github. I apologize in advance if it's not quite right. |
|
| 2016-06-27 15:59:05 |
Tristan Cacqueray |
tags |
|
security |
|
| 2016-06-27 15:59:21 |
Tristan Cacqueray |
information type |
Private Security |
Public |
|
| 2016-11-30 23:49:20 |
clayg |
marked as duplicate |
|
1568650 |
|
| 2016-11-30 23:52:17 |
clayg |
removed duplicate marker |
1568650 |
|
|
| 2016-12-01 00:02:17 |
clayg |
swift: status |
New |
Confirmed |
|
| 2016-12-01 00:02:20 |
clayg |
swift: importance |
Undecided |
High |
|
| 2017-01-18 16:14:41 |
Jeremy Stanley |
ossa: status |
Incomplete |
Won't Fix |
|
| 2022-06-24 14:29:22 |
Albert Braden |
bug |
|
|
added subscriber Albert Braden |
| 2022-07-12 16:52:17 |
Brett Milford |
bug |
|
|
added subscriber Brett Milford |
