processutils.execute is 10 times slower than clean subprocess.run

Bug #2039548 reported by Denis Kolvakh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.concurrency
New
Undecided
Unassigned

Bug Description

There is a bug discovered by me in eventlet with incorrect PIPE_BUF size assignment: https://github.com/eventlet/eventlet/issues/813. In short, fallback to default 512 bytes due to absence of PIPE_BUF constant in patched 'select' module decrease I/O operations speed up to 10 times in comparison with Linux-style 4096 bytes.

Due to wide usage of eventlet around Openstack, this issue affects many Openstack components and flows, especially when significant amount of data should be sent/received via Popen.

While issue in eventlet is still orphan, I suggest the following w/a to speed up I/O processes up to 10 times (also attached).

Index: oslo_concurrency/processutils.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/oslo_concurrency/processutils.py b/oslo_concurrency/processutils.py
--- a/oslo_concurrency/processutils.py (revision 774f604c16b47ad4ce47e6390ec30f9fc8f30c67)
+++ b/oslo_concurrency/processutils.py (date 1697534335526)
@@ -53,6 +53,16 @@
         subprocess.threading = eventlet.patcher.original('threading')
     else:
         from eventlet.green import subprocess
+ # NOTE(dvkolvakh): during monkey_patch 'eventlet' doesn't care about
+ # constants in built-in 'select' module, that's why 'subprocess'
+ # falls back to default 512 bytes for IO buffer trying to lookup
+ # for non-existing constant in 'eventlet.green.select' module,
+ # see https://github.com/eventlet/eventlet/issues/813.
+ # By default, linux systems has PIPE_BUF = 4096 in 'select' module,
+ # so rely on constant definition in original 'subprocess' module
+ # to speed-up IO operations.
+ subprocess_orig = eventlet.patcher.original('subprocess')
+ subprocess._PIPE_BUF = subprocess_orig._PIPE_BUF

     from eventlet import tpool

Revision history for this message
Denis Kolvakh (dendvk) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.