commit d0f540742efc1004b4d2c64814dcc6f7b9f0ccf6
Author: Matthew Booth <email address hidden>
Date: Wed Jan 30 15:10:25 2019 +0000
Eventlet monkey patching should be as early as possible
We were seeing infinite recursion opening an ssl socket when running
various combinations of python3, eventlet, and urllib3. It is not
clear exactly what combination of versions are affected, but for
background there is an example of this issue documented here:
The immediate cause in nova's case was that we were calling
eventlet.monkey_patch() after importing urllib3. Specifically, change
Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the
nova.utils.monkey_patch() method to make monkey patching common
between WSGI and non-WSGI services. Unfortunately, before executing
this method you must first import nova.utils, which imports a large
number of modules itself. Anything imported (transitively) by
nova.utils would therefore be imported before monkey patching, which
included urllib3. This triggers the infinite recursion problem
described above if you have an affected combination of library
versions.
While this specific issue may eventually be worked around or fixed in
eventlet or urllib3, it remains true that eventlet best practises are
to monkey patch as early as possible, which we were not doing. To
avoid this and hopefully future similar issues, this change ensures
that monkey patching happens as early as possible, and only a minimum
number of modules are imported first.
This change fixes monkey patching for both non-wsgi and wsgi callers:
* Non-WSGI services (nova/cmd)
This is fixed by using the new monkey_patch module, which has minimal
dependencies.
* WSGI services (nova/api/openstack)
This is fixed both by using the new monkey_patch module, and by moving
the patching point up one level so that it is done before importing
anything in nova/api/openstack/__init__.py.
This move causes issues for some external tools which load this path
from nova and now monkey patch where they previously did not. However,
it is unfortunately unavoidable to enable monkey patching for the wsgi
entry point without major restructuring. This change includes a
workaround for sphinx to avoid this issue.
This change has been through several iterations. I started with what
seemed like the simplest and most obvious change, and moved on as I
discovered more interactions which broke. It is clear that eventlet
monkey patching is extremely fragile, especially when done implicitly at
module load time as we do. I would advocate a code restructure to
improve this situation, but I think the time would be better spent
removing the eventlet dependency entirely.
Co-authored-by: Lee Yarwood <email address hidden>
Reviewed: https:/ /review. opendev. org/647310 /git.openstack. org/cgit/ openstack/ nova/commit/ ?id=d0f540742ef c1004b4d2c64814 dcc6f7b9f0ccf6
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit d0f540742efc100 4b4d2c64814dcc6 f7b9f0ccf6
Author: Matthew Booth <email address hidden>
Date: Wed Jan 30 15:10:25 2019 +0000
Eventlet monkey patching should be as early as possible
We were seeing infinite recursion opening an ssl socket when running
various combinations of python3, eventlet, and urllib3. It is not
clear exactly what combination of versions are affected, but for
background there is an example of this issue documented here:
https:/ /github. com/eventlet/ eventlet/ issues/ 371
The immediate cause in nova's case was that we were calling monkey_ patch() after importing urllib3. Specifically, change 2ccbcd63c262dda f739782afcdaf56 introduced the utils.monkey_ patch() method to make monkey patching common
eventlet.
Ie7bf5d012e
nova.
between WSGI and non-WSGI services. Unfortunately, before executing
this method you must first import nova.utils, which imports a large
number of modules itself. Anything imported (transitively) by
nova.utils would therefore be imported before monkey patching, which
included urllib3. This triggers the infinite recursion problem
described above if you have an affected combination of library
versions.
While this specific issue may eventually be worked around or fixed in
eventlet or urllib3, it remains true that eventlet best practises are
to monkey patch as early as possible, which we were not doing. To
avoid this and hopefully future similar issues, this change ensures
that monkey patching happens as early as possible, and only a minimum
number of modules are imported first.
This change fixes monkey patching for both non-wsgi and wsgi callers:
* Non-WSGI services (nova/cmd)
This is fixed by using the new monkey_patch module, which has minimal
dependencies.
* WSGI services (nova/api/ openstack)
This is fixed both by using the new monkey_patch module, and by moving openstack/ __init_ _.py.
the patching point up one level so that it is done before importing
anything in nova/api/
This move causes issues for some external tools which load this path
from nova and now monkey patch where they previously did not. However,
it is unfortunately unavoidable to enable monkey patching for the wsgi
entry point without major restructuring. This change includes a
workaround for sphinx to avoid this issue.
This change has been through several iterations. I started with what
seemed like the simplest and most obvious change, and moved on as I
discovered more interactions which broke. It is clear that eventlet
monkey patching is extremely fragile, especially when done implicitly at
module load time as we do. I would advocate a code restructure to
improve this situation, but I think the time would be better spent
removing the eventlet dependency entirely.
Co-authored-by: Lee Yarwood <email address hidden>
Closes-Bug: #1808975 10ec4654d4418a9 884975b5b95 294a949852bb8c8 3d4ed77e04)
Closes-Bug: #1808951
Change-Id: Id46e76666b553a
(cherry picked from commit 3c5e2b0e9fac985