This affects a number of other OpenStack projects in a similar way, including: - Neutron - Cinder - Glance More projects may be affected that we are unaware of. ===================================================== Example traceback from Neutron ===================================================== 2016-09-28 00:16:43.342 1218 DEBUG neutron.wsgi [-] (1218) accepted ('10.0.2.2', 50029) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868 Traceback (most recent call last): File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit msg = self.format(record) File "/usr/local/lib/python2.7/dist-packages/oslo_log/handlers.py", line 76, in format return logging.StreamHandler.format(self, record) File "/usr/lib/python2.7/logging/__init__.py", line 724, in format return fmt.format(record) File "/usr/local/lib/python2.7/dist-packages/oslo_log/formatters.py", line 297, in format return logging.Formatter.format(self, record) File "/usr/lib/python2.7/logging/__init__.py", line 464, in format record.message = record.getMessage() File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage msg = msg % self.args File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1164, in as_text bytes = self.as_bytes() File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1135, in as_bytes url = self.url File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url url = self.path_url File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url bpath_info = bytes_(self.path_info, self.url_encoding) File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget return req.encget(key, encattr=encattr) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget return val.decode(encoding) File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 11: invalid start byte Logged from file catch_errors.py, line 41 2016-09-28 00:16:43.489 1218 INFO neutron.wsgi [req-37cfe540-c134-4ec0-91d2-56687f38ffd5 admin -] 10.0.2.2 - - [28/Sep/2016 00:16:43] "PUT /v2.0/flavors/..%c0%af HTTP/1.1" 500 414 0.140984 ===================================================== Example traceback from Cinder ===================================================== 2016-09-27 23:50:00.142 5986 DEBUG eventlet.wsgi.server [-] (5986) accepted ('10.0.2.2', 49862) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868 2016-09-27 23:50:00.198 ERROR cinder.api.middleware.fault [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Caught error: 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault Traceback (most recent call last): 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 79, in __call__ 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return req.get_response(self.application) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__ 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault response = req.get_response(self._app) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__ 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault response = self.app(environ, start_response) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/openstack/wsgi.py", line 817, in __call__ 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault "url": request.url}) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault url = self.path_url 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault bpath_info = bytes_(self.path_info, self.url_encoding) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return req.encget(key, encattr=encattr) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return val.decode(encoding) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return codecs.utf_8_decode(input, errors, True) 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte 2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault 2016-09-27 23:50:00.224 INFO eventlet.wsgi.server [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response result = self.application(self.environ, start_response) File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 216, in __call__ return app(environ, start_response) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__ response = req.get_response(self.application) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send application, catch_exc_info=False) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application app_iter = application(self.environ, start_response) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__ response = req.get_response(self.application) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send application, catch_exc_info=False) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application app_iter = application(self.environ, start_response) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/request_id.py", line 37, in __call__ response = req.get_response(self.application) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send application, catch_exc_info=False) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application app_iter = application(self.environ, start_response) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 81, in __call__ return self._error(ex, req) File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 56, in _error msg_dict = dict(url=req.url, status=status) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url url = self.path_url File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url bpath_info = bytes_(self.path_info, self.url_encoding) File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget return req.encget(key, encattr=encattr) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget return val.decode(encoding) File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte 2016-09-27 23:50:00.228 INFO eventlet.wsgi.server [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] 10.0.2.2 "POST /v2/..%c0%af/backups HTTP/1.1" status: 500 len: 139 time: 0.0827849 ===================================================== Example traceback from Glance ===================================================== 2016-09-28 00:11:55.844 32495 INFO eventlet.wsgi.server [-] Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response result = self.application(self.environ, start_response) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__ response = req.get_response(self.application) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send application, catch_exc_info=False) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application app_iter = application(self.environ, start_response) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 123, in __call__ response = self.process_request(req) File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 148, in __call__ return self.func(req, *args, **kw) File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/healthcheck/__init__.py", line 361, in process_request if req.path != self._path: File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 485, in path bpath = bytes_(self.path_info, self.url_encoding) File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget return req.encget(key, encattr=encattr) File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget return val.decode(encoding) File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 13: invalid start byte 2016-09-28 00:11:55.854 32495 INFO eventlet.wsgi.server [-] 10.0.2.2 - - [28/Sep/2016 00:11:55] "GET /v2/images/..%c0%af/tags/..%c0%af HTTP/1.1" 500 139 0.043949 I'm not sure whether right approach here is to file a bug with webob (looks like there are several that haven't been resolved yet [1] [2]), or to fix this in each project's respective wsgi error-handling code. [1] https://github.com/Pylons/webob/issues/115 [2] https://github.com/Pylons/webob/issues/161