Cinder

String "..%c0%af" causes 500 errors in multiple locations

Bug #1613901 reported by Charles Neill on 2016-08-16

This bug affects 3 people

	Status	Importance	Assigned to
Cinder	Incomplete	Undecided	Unassigned
Glance	New	Undecided	Unassigned
OpenStack Identity (keystone)	Won't Fix	Low	Unassigned
OpenStack Security Advisory	Won't Fix	Undecided	Unassigned
neutron	Won't Fix	Undecided	Unassigned

Bug Description

While doing some testing on Keystone using Syntribos (https://github.com/openstack/syntribos), our team (myself, Michael Dong, Rahul U Nair, Vinay Potluri, Aastha Dixit, and Khanak Nangia) noticed that we got 500 status codes when the string "..%c0%af" was inserted in various places in the URL for different types of requests.

Here are some examples:

=========

DELETE /v3/policies/..%c0%af HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
X-Auth-Token: [REDACTED]
Content-Length: 0

HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:04:27 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-238fd5a9-be45-41f2-893a-97b513b27af3
Content-Length: 143
Connection: close
Content-Type: application/json

{"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

=========

PATCH /v3/policies/..%c0%af HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
Content-type: application/json
X-Auth-Token: [REDACTED]
Content-Length: 70

{"type": "--serialization-mime-type--", "blob": "--serialized-blob--"}

HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:05:36 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-57a41600-02b4-4d2a-b3e9-40f7724d65f2
Content-Length: 143
Connection: close
Content-Type: application/json

{"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

=========

GET /v3/domains/0426ac1e48f642ef9544c2251e07e261/groups/..%c0%af/roles HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
X-Auth-Token: [REDACTED]

HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:07:09 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-02313f77-63c6-4aa8-a87e-e3d2a13ad6b7
Content-Length: 143
Connection: close
Content-Type: application/json

{"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

=========

I've marked this as a security issue as a precaution in case it turns out that there is a more serious vulnerability underlying these errors. We have no reason to suspect that there is a greater vulnerability at this time, but given the many endpoints this seems to affect, I figured caution was worthwhile since this may be a framework-wide issue. Feel free to make this public if it is determined not to be security-impacting.

Here is a (possibly incomplete) list of affected endpoints. Inserting the string "..%c0%af" in any or all of the spots labeled "HERE" should yield a 500 error. As you can see, virtually all v3 endpoints exhibit this behavior.

=========

[GET|PATCH|DELETE] /v3/endpoints/[HERE]

[GET|PATCH] /v3/domains/[HERE]
GET /v3/domains/[HERE]/groups/[HERE]/roles
[HEAD|PUT|DELETE] /v3/domains/[HERE]/groups/[HERE]/roles/[HERE]
GET /v3/domains/[HERE]/users/[HERE]/roles
[HEAD|DELETE] /v3/domains/[HERE]/users/[HERE]/roles/[HERE]

[GET|PATCH|DELETE] /v3/groups/[HERE]
[HEAD|PUT|DELETE] /v3/groups[HERE]/users/[HERE]

[POST|DELETE] /v3/keys/[HERE]

[GET|PATCH|DELETE] /v3/regions/[HERE]

[PATCH|DELETE] /v3/roles/[HERE]

[GET|PATCH|DELETE] /v3/services/[HERE]

[GET|PATCH|DELETE] /v3/users/[HERE]
GET /v3/users/[HERE]/groups
POST /v3/users/[HERE]/password
GET /v3/users/[HERE]/projects

GET /v3/OS-OAUTH1/users/[HERE]/access_tokens/[HERE]/roles/[HERE]
[GET|PATCH|DELETE] /v3/OS-OAUTH1/consumers/[HERE]
[GET|DELETE] /v3/OS-OAUTH1/users/[HERE]/access_tokens/[HERE]

See original description

Tags:

Revision history for this message

David Stanek (dstanek) wrote on 2016-08-16:

Download full text (3.1 KiB)

This happens because webob tries to decode the path as UTF-8, but that is actually malformed UTF-8. This is probably something that should be caught in keystone.common.wsgi and returned as a 400, but I don't think these is any security issues. Is it possible to use this to exploit keystone is some way.

Log from a bad request:

2016-08-16 23:45:25.192 14345 ERROR keystone.common.wsgi [req-dd271afd-aa0c-4fd2-90f2-6366327e9bba - - - - -] 'utf8' codec can't decode byte 0xc0 in position 12: invalid start byte
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi Traceback (most recent call last):
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/common/wsgi.py", line 372, in _inner
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi return method(self, request)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/common/wsgi.py", line 432, in __call__
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi response = request.get_response(self.application)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi application, catch_exc_info=False)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi app_iter = application(self.environ, start_response)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi resp = self.call_func(req, *args, **self.kwargs)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi return self.func(req, *args, **kwargs)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/common/wsgi.py", line 687, in __call__
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi if request.path_info != '/':
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi return req.encget(key, encattr=encattr)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi return val.decode(encoding)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi return codecs.utf_8_decode(input, errors, True)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 12: invalid start byte
2016-08-16 23:45:25.192 14345 TRACE keystone.common....

Log from a bad request:

2016-08-16 23:45:25.192 14345 ERROR keystone.common.wsgi [req-dd271afd-aa0c-4fd2-90f2-6366327e9bba - - - - -] 'utf8' codec can't decode byte 0xc0 in position 12: invalid start byte
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi Traceback (most recent call last):
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/common/wsgi.py", line 372, in _inner
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi     return method(self, request)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/common/wsgi.py", line 432, in __call__
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi     response = request.get_response(self.application)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi     application, catch_exc_info=False)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi     app_iter = application(self.environ, start_response)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi     resp = self.call_func(req, *args, **self.kwargs)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi     return self.func(req, *args, **kwargs)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi   File "/opt/stack/keystone/keystone/common/wsgi.py", line 687, in __call__
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi     if request.path_info != '/':
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi     return req.encget(key, encattr=encattr)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi     return val.decode(encoding)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi     return codecs.utf_8_decode(input, errors, True)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 12: invalid start byte
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi

David Stanek (dstanek) on 2016-08-16

Changed in keystone:
status:	New → Confirmed

Revision history for this message

Morgan Fainberg (mdrnstm) wrote on 2016-08-17:

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

description:	updated
Changed in ossa:
status:	New → Incomplete

Revision history for this message

Morgan Fainberg (mdrnstm) wrote on 2016-08-17:

For what it is worth, this doesn't look like a security flaw that can be exploited directly as far as I can see from the trace david stanek provided. However, I will defer to the keystone-coresec team to determine if this actually does result in an exploit.

Revision history for this message

Morgan Fainberg (mdrnstm) wrote on 2016-08-17:

@David Stanek/Keystone Core-Sec,

If you would like to classify this as not a security bug, please let us know so that it can be moved to a public bug.

Revision history for this message

Dolph Mathews (dolph) wrote on 2016-08-17:

+1 for fixing the issue in public.

Revision history for this message

Steve Martinelli (stevemar) wrote on 2016-08-17:

I don't see a reason to keep this private.

Revision history for this message

Morgan Fainberg (mdrnstm) wrote on 2016-08-17:

I have marked this bug as public and since this is not a security related bug, the OSSA task has been marked as "Wont Fix".

Thanks Steve and Dolph for the quick response.

description:	updated
Changed in ossa:
status:	Incomplete → Won't Fix
information type:	Private Security → Public

Steve Martinelli (stevemar) on 2016-08-17

Changed in keystone:
importance:	Undecided → Low

Revision history for this message

Robert Clark (robert-clark) wrote on 2016-08-17:

Seems reasonable to me. Well done on finding this Syntribos team - this tool is showing real promise. I'm expecting you to find some interesting higher-severity issues in the future!

Revision history for this message

Charles Neill (charles-neill) wrote on 2016-09-28:

Download full text (14.7 KiB)

This affects a number of other OpenStack projects in a similar way, including:

- Neutron
- Cinder
- Glance

More projects may be affected that we are unaware of.

=====================================================
Example traceback from Neutron
=====================================================

2016-09-28 00:16:43.342 1218 DEBUG neutron.wsgi [-] (1218) accepted ('10.0.2.2', 50029) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/local/lib/python2.7/dist-packages/oslo_log/handlers.py", line 76, in format
    return logging.StreamHandler.format(self, record)
  File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/local/lib/python2.7/dist-packages/oslo_log/formatters.py", line 297, in format
    return logging.Formatter.format(self, record)
  File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1164, in as_text
    bytes = self.as_bytes()
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1135, in as_bytes
    url = self.url
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
    url = self.path_url
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
    bpath_info = bytes_(self.path_info, self.url_encoding)
  File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
    return req.encget(key, encattr=encattr)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
    return val.decode(encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 11: invalid start byte
Logged from file catch_errors.py, line 41
2016-09-28 00:16:43.489 1218 INFO neutron.wsgi [req-37cfe540-c134-4ec0-91d2-56687f38ffd5 admin -] 10.0.2.2 - - [28/Sep/2016 00:16:43] "PUT /v2.0/flavors/..%c0%af HTTP/1.1" 500 414 0.140984

=====================================================
Example traceback from Cinder
=====================================================

This affects a number of other OpenStack projects in a similar way, including:

- Neutron
- Cinder
- Glance

More projects may be affected that we are unaware of.

=====================================================
Example traceback from Neutron
=====================================================

=====================================================
Example traceback from Cinder
=====================================================

2016-09-27 23:50:00.142 5986 DEBUG eventlet.wsgi.server [-] (5986) accepted ('10.0.2.2', 49862) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
2016-09-27 23:50:00.198 ERROR cinder.api.middleware.fault [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Caught error: <type 'exceptions.UnicodeDecodeError'> 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault Traceback (most recent call last):
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 79, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return req.get_response(self.application)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     response = req.get_response(self._app)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     response = self.app(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/opt/stack/cinder/cinder/api/openstack/wsgi.py", line 817, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     "url": request.url})
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     url = self.path_url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     bpath_info = bytes_(self.path_info, self.url_encoding)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return req.encget(key, encattr=encattr)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return val.decode(encoding)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return codecs.utf_8_decode(input, errors, True)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault
2016-09-27 23:50:00.224 INFO eventlet.wsgi.server [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 216, in __call__
    return app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/request_id.py", line 37, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 81, in __call__
    return self._error(ex, req)
  File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 56, in _error
    msg_dict = dict(url=req.url, status=status)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
    url = self.path_url
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
    bpath_info = bytes_(self.path_info, self.url_encoding)
  File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
    return req.encget(key, encattr=encattr)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
    return val.decode(encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte

2016-09-27 23:50:00.228 INFO eventlet.wsgi.server [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] 10.0.2.2 "POST /v2/..%c0%af/backups HTTP/1.1" status: 500  len: 139 time: 0.0827849

=====================================================
Example traceback from Glance
=====================================================

2016-09-28 00:11:55.844 32495 INFO eventlet.wsgi.server [-] Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 123, in __call__
    response = self.process_request(req)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 148, in __call__
    return self.func(req, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/healthcheck/__init__.py", line 361, in process_request
    if req.path != self._path:
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 485, in path
    bpath = bytes_(self.path_info, self.url_encoding)
  File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
    return req.encget(key, encattr=encattr)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
    return val.decode(encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 13: invalid start byte

2016-09-28 00:11:55.854 32495 INFO eventlet.wsgi.server [-] 10.0.2.2 - - [28/Sep/2016 00:11:55] "GET /v2/images/..%c0%af/tags/..%c0%af HTTP/1.1" 500 139 0.043949

I'm not sure whether right approach here is to file a bug with webob (looks like there are several that haven't been resolved yet [1] [2]), or to fix this in each project's respective wsgi error-handling code.

[1] https://github.com/Pylons/webob/issues/115
[2] https://github.com/Pylons/webob/issues/161

summary:

- String "..%c0%af" causes 500 errors in multiple locations in Keystone v3
+ String "..%c0%af" causes 500 errors in multiple locations

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2016-09-28:

#10

I am unclear to where the issue lies, but should this be fixed centrally rather than being delegated to the individual projects?

Changed in neutron:
status:	New → Incomplete

Revision history for this message

Sean McGinnis (sean-mcginnis) wrote on 2016-09-28:

#11

This does seem like something that should be fixed centrally. Otherwise this will pop up repeatedly as projects come and go.

Changed in cinder:
status:	New → Incomplete

Revision history for this message

John Perkins (john-d-perkins-deactivatedaccount) wrote on 2016-09-28:

#12

Webob maintainer 'bertjwregeer' has been pinged regarding this issue and there the fix is in progress: https://github.com/Pylons/webob/issues/161

Revision history for this message

Bert JW Regeer (bertjwregeer) wrote on 2016-09-28:

#13

The "fix" in WebOb is going to be checking for UnicodeDecodeError and turning that into an URLDecodeError. There would still be a requirement for something upstream of the Request object to catch the exception and properly send it out.

URLDecodeError will be a sub-class of UnicodeDecodeError and HTTPBadRequest, so that it is a valid WebOb exc.WSGIHTTPException and thus can be used as a response object...

There is no direct fix that will suddenly handle this case directly, code that touches PATH and other such variables will simply raise a better exception than a generic UnicodeDecodeError.

Steve Martinelli (stevemar) on 2016-11-15