python3 + Fedora + SSL + wsgi mistral deployment, mistral api returns: maximum recursion depth exceeded while calling a Python object

Bug #1808953 reported by yatin
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Mistral
Confirmed
High
Unassigned

Bug Description

Description:-

So while testing python3 with Fedora in [1], Found an issue while running mistral behind wsgi. It fails with below Traceback:-

[Tue Dec 18 09:29:44.021589 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] mod_wsgi (pid=12268): Exception occurred processing WSGI script '/var/www/cgi-bin/mistral/app'.
[Tue Dec 18 09:29:44.025935 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] Traceback (most recent call last):
[Tue Dec 18 09:29:44.026189 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
[Tue Dec 18 09:29:44.026196 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] resp = self.call_func(req, *args, **kw)
[Tue Dec 18 09:29:44.026202 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
[Tue Dec 18 09:29:44.026205 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] return self.func(req, *args, **kwargs)
[Tue Dec 18 09:29:44.026211 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/oslo_middleware/base.py", line 131, in __call__
[Tue Dec 18 09:29:44.026214 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] response = req.get_response(self.application)
[Tue Dec 18 09:29:44.026219 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send
[Tue Dec 18 09:29:44.026222 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] application, catch_exc_info=False)
[Tue Dec 18 09:29:44.026227 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application
[Tue Dec 18 09:29:44.026230 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] app_iter = application(self.environ, start_response)
[Tue Dec 18 09:29:44.026234 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
[Tue Dec 18 09:29:44.026237 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] resp = self.call_func(req, *args, **kw)
[Tue Dec 18 09:29:44.026242 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
[Tue Dec 18 09:29:44.026245 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] return self.func(req, *args, **kwargs)
[Tue Dec 18 09:29:44.026250 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__
[Tue Dec 18 09:29:44.026253 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] response = self.process_request(req)
[Tue Dec 18 09:29:44.026258 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request
[Tue Dec 18 09:29:44.026260 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] resp = super(AuthProtocol, self).process_request(request)
[Tue Dec 18 09:29:44.026265 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request
[Tue Dec 18 09:29:44.026268 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] allow_expired=allow_expired)
[Tue Dec 18 09:29:44.026282 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token
[Tue Dec 18 09:29:44.026285 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] data = self.fetch_token(token, **kwargs)
[Tue Dec 18 09:29:44.026290 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token
[Tue Dec 18 09:29:44.026293 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] allow_expired=allow_expired)
[Tue Dec 18 09:29:44.026298 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
[Tue Dec 18 09:29:44.026300 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] auth_ref = self._request_strategy.verify_token(
[Tue Dec 18 09:29:44.026305 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy
[Tue Dec 18 09:29:44.026308 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] strategy_class = self._get_strategy_class()
[Tue Dec 18 09:29:44.026313 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class
[Tue Dec 18 09:29:44.026316 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
[Tue Dec 18 09:29:44.026320 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
[Tue Dec 18 09:29:44.026323 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] return self.session.get_endpoint(auth or self.auth, **kwargs)
[Tue Dec 18 09:29:44.026328 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
[Tue Dec 18 09:29:44.026331 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] return auth.get_endpoint(self, **kwargs)
[Tue Dec 18 09:29:44.026335 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
[Tue Dec 18 09:29:44.026338 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] allow_version_hack=allow_version_hack, **kwargs)
[Tue Dec 18 09:29:44.026343 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
[Tue Dec 18 09:29:44.026346 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] service_catalog = self.get_access(session).service_catalog
[Tue Dec 18 09:29:44.026351 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
[Tue Dec 18 09:29:44.026354 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] self.auth_ref = self.get_auth_ref(session)
[Tue Dec 18 09:29:44.026358 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
[Tue Dec 18 09:29:44.026361 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] self._plugin = self._do_create_plugin(session)
[Tue Dec 18 09:29:44.026366 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
[Tue Dec 18 09:29:44.026369 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] authenticated=False)
[Tue Dec 18 09:29:44.026373 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
[Tue Dec 18 09:29:44.026379 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] authenticated=authenticated)
[Tue Dec 18 09:29:44.026384 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
[Tue Dec 18 09:29:44.026387 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] disc = Discover(session, url, authenticated=authenticated)
[Tue Dec 18 09:29:44.026392 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
[Tue Dec 18 09:29:44.026395 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] authenticated=authenticated)
[Tue Dec 18 09:29:44.026399 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
[Tue Dec 18 09:29:44.026402 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] resp = session.get(url, headers=headers, authenticated=authenticated)
[Tue Dec 18 09:29:44.026407 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
[Tue Dec 18 09:29:44.026410 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] return self.request(url, 'GET', **kwargs)
[Tue Dec 18 09:29:44.026415 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request
[Tue Dec 18 09:29:44.026417 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] resp = send(**kwargs)
[Tue Dec 18 09:29:44.026422 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request
[Tue Dec 18 09:29:44.026425 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] resp = self.session.request(method, url, **kwargs)
[Tue Dec 18 09:29:44.026430 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
[Tue Dec 18 09:29:44.026432 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] resp = self.send(prep, **send_kwargs)
[Tue Dec 18 09:29:44.026437 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
[Tue Dec 18 09:29:44.026440 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] r = adapter.send(request, **kwargs)
[Tue Dec 18 09:29:44.026445 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
[Tue Dec 18 09:29:44.026447 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] timeout=timeout
[Tue Dec 18 09:29:44.026452 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
[Tue Dec 18 09:29:44.026455 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] chunked=chunked)
[Tue Dec 18 09:29:44.026460 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
[Tue Dec 18 09:29:44.026463 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] self._validate_conn(conn)
[Tue Dec 18 09:29:44.026467 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
[Tue Dec 18 09:29:44.026470 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] conn.connect()
[Tue Dec 18 09:29:44.026475 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
[Tue Dec 18 09:29:44.026478 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] cert_reqs=resolve_cert_reqs(self.cert_reqs),
[Tue Dec 18 09:29:44.026482 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
[Tue Dec 18 09:29:44.026488 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] context.options |= options
[Tue Dec 18 09:29:44.026493 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib64/python3.6/ssl.py", line 423, in options
[Tue Dec 18 09:29:44.026495 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] super(SSLContext, SSLContext).options.__set__(self, value)
[Tue Dec 18 09:29:44.026500 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib64/python3.6/ssl.py", line 423, in options
[Tue Dec 18 09:29:44.026503 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] super(SSLContext, SSLContext).options.__set__(self, value)
[Tue Dec 18 09:29:44.026508 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] File "/usr/lib64/python3.6/ssl.py", line 423, in options
[Tue Dec 18 09:29:44.026510 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] super(SSLContext, SSLContext).options.__set__(self, value)
[Tue Dec 18 09:29:44.026515 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] [Previous line repeated 305 more times]
[Tue Dec 18 09:29:44.026533 2018] [wsgi:error] [pid 12268] [remote 127.0.0.1:39652] RecursionError: maximum recursion depth exceeded while calling a Python object

Environment Details:-
OS: Fedora 28
python: 3.6.7
eventlet: 0.24.1

The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with mistral it's not the case, In mistral even behind wsgi monkey patch is running, which is causing this issue.

Expected result
===============
Mistral api should response successfully.

Actual result
=============
Mistral api returns Recursion Error.

[1] https://review.openstack.org/#/c/620561/
[2] https://github.com/openstack/mistral/blob/211e3964111816ef52588f914d4b7821e05ab625/mistral/cmd/launch.py#L22-L27 called via https://github.com/openstack/mistral/blob/211e3964111816ef52588f914d4b7821e05ab625/mistral/api/controllers/v2/service.py#L24-L26
[3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54

Dougal Matthews (d0ugal)
Changed in mistral:
status: New → Confirmed
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.