Mellanox backend integration with Neutron (networking-mlnx)

[SRU] eventlet monkey patch results in assert len(_active) == 1 AssertionError

Bug #1863021 reported by Corey Bryant
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
BaGPipe
Fix Released
Undecided
Unassigned
Cinder
Fix Released
Undecided
Corey Bryant
Designate
Fix Released
Undecided
Corey Bryant
Glance
Fix Released
Undecided
Unassigned
Mellanox backend integration with Neutron (networking-mlnx)
Fix Released
Undecided
Unassigned
Mistral
Fix Released
Undecided
Unassigned
Murano
Fix Released
Undecided
Corey Bryant
OpenStack Compute (nova)
Fix Released
Undecided
Corey Bryant
OpenStack DBaaS (Trove)
Fix Released
Undecided
Lingxian Kong
OpenStack Object Storage (swift)
In Progress
Undecided
Chris MacNaughton
OpenStack Shared File Systems Service (Manila)
Fix Released
Undecided
suzhengwei
masakari
Fix Released
Undecided
Corey Bryant
networking-hyperv
Fix Released
Undecided
Corey Bryant
networking-l2gw
Fix Released
Undecided
Unassigned
networking-sfc
Fix Released
Undecided
Unassigned
neutron
Fix Released
High
Dr. Jens Harbott
oslo.service
Fix Released
Undecided
Corey Bryant
senlin
Fix Released
Undecided
Unassigned
watcher
Fix Released
Undecided
Chris MacNaughton
barbican (Ubuntu)
Fix Released
High
Arthur Dayne
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Arthur Dayne
cinder (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
designate (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
glance (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
Undecided
Unassigned
Groovy
Fix Released
High
Unassigned
heat (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
ironic (Ubuntu)
Fix Released
High
Arthur Dayne
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Arthur Dayne
ironic-inspector (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
magnum (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
manila (Ubuntu)
Fix Released
High
Arthur Dayne
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Arthur Dayne
masakari (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
mistral (Ubuntu)
Fix Released
High
Arthur Dayne
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Arthur Dayne
murano (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
murano-agent (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
networking-bagpipe (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
networking-hyperv (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
networking-l2gw (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
networking-mlnx (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
networking-sfc (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
neutron (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
neutron-dynamic-routing (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
nova (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
Undecided
Unassigned
Groovy
Fix Released
High
Unassigned
openstack-trove (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
python-os-ken (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
python-oslo.service (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
sahara (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
senlin (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
swift (Ubuntu)
Triaged
Low
Unassigned
Focal
Triaged
Low
Unassigned
Groovy
Triaged
Low
Unassigned
watcher (Ubuntu)
Fix Released
High
Arthur Dayne
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Arthur Dayne

Bug Description

[Impact]
This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592

However I seem to only hit this with python3.8. Basically nova, glance, neutron, and cinder services fail with:

 Exception ignored in: <function _after_fork at 0x7f1100d075e0>
 Traceback (most recent call last):
   File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
     assert len(_active) == 1
 AssertionError:
 Exception ignored in: <function _after_fork at 0x7f1100d075e0>
 Traceback (most recent call last):
   File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
     assert len(_active) == 1
 AssertionError:

Patching nova/monkey_patch.py with the following appears to fix this:

diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py
index a07ff91dac..bb7252c643 100644
--- a/nova/monkey_patch.py
+++ b/nova/monkey_patch.py
@@ -59,6 +59,9 @@ def _monkey_patch():
     else:
         eventlet.monkey_patch()

+ import __original_module_threading
+ import threading
+ __original_module_threading.current_thread.__globals__['_active'] = threading._active
     # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet
     # hub use a monotonic clock to avoid issues with drifts of system time (see

Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and glance/cmd/registry.py appears to fix it for glance.

[Test Case]
Tempest regression on a deployed openstack should pass successfully.

Individual services can be tested as follows but with the scope of this issue tempest regression testing will be the goal for verifying this SRU.
'systemctl status <service>' should not show the failure reported in

[Impact].

[Regression Potential]
Should be low. The fix provided is a well-known solution to the problem. Openstack deployment + tempest regression testing will be performed to help reduce regression potential.

See original description
Revision history for this message
Corey Bryant (corey.bryant) wrote :

See attached for much more detailed recreation/patching/fixing.

tags: added: py38
Changed in nova (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/707474

Changed in nova:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote : Re: eventlet monkey patch results in assert len(_active) == 1 AssertionError

This bug was fixed in the package nova - 2:21.0.0~b2~git2020021008.1fcd74730d-0ubuntu2

---------------
nova (2:21.0.0~b2~git2020021008.1fcd74730d-0ubuntu2) focal; urgency=medium

  * d/p/monkey-patch-original-current-thread-active.patch: Cherry-picked from
    https://review.opendev.org/#/c/707474/. This fixes nova service failures
    that autopkgtests are hitting with Python 3.8 (LP: #1863021).

 -- Corey Bryant <email address hidden> Thu, 20 Feb 2020 09:35:53 -0500

Changed in nova (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/707474
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2078ef850c93b47ec1bcd044450b589d7a8fc0b5
Submitter: Zuul
Branch: master

commit 2078ef850c93b47ec1bcd044450b589d7a8fc0b5
Author: Corey Bryant <email address hidden>
Date: Wed Feb 12 16:32:40 2020 -0500

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I4872169413f27aeaff8d8fdfa5cdaf6ee32f4680
    Closes-Bug: #1863021

Changed in nova:
status: In Progress → Fix Released
Corey Bryant (corey.bryant)
Changed in glance (Ubuntu):
status: New → Triaged
importance: Undecided → High
description: updated
Corey Bryant (corey.bryant)
description: updated
Corey Bryant (corey.bryant)
description: updated
Revision history for this message
Corey Bryant (corey.bryant) wrote : Re: eventlet monkey patch results in assert len(_active) == 1 AssertionError

Glance review: https://review.opendev.org/#/c/716058/

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glance - 2:20.0.0~b3~git2020032414.30ece7aa-0ubuntu2

---------------
glance (2:20.0.0~b3~git2020032414.30ece7aa-0ubuntu2) focal; urgency=medium

  * d/tests/test-shebang.py: Dropped. No longer used.
  * d/p/monkey-patch-original-current-thread-active.patch: Cherry-picked from
    https://review.opendev.org/#/c/716058/. This fixes glance service failures
    that autopkgtests are hitting with Python 3.8 (LP: #1863021).

 -- Corey Bryant <email address hidden> Mon, 30 Mar 2020 14:41:49 -0400

Changed in glance (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.opendev.org/716058
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=90146b62c765ad1b8be1ffec1799cba9f3994c2d
Submitter: Zuul
Branch: master

commit 90146b62c765ad1b8be1ffec1799cba9f3994c2d
Author: Corey Bryant <email address hidden>
Date: Mon Mar 30 15:14:15 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I95a8d8cf02a0cb923418c0b5655442b8d7bc6b08
    Closes-Bug: #1863021

Changed in glance:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 20.0.0.0b3

This issue was fixed in the openstack/glance 20.0.0.0b3 development milestone.

Corey Bryant (corey.bryant)
description: updated
Changed in cinder (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/724753

Changed in neutron:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Changed in cinder:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/724754

Corey Bryant (corey.bryant)
Changed in neutron (Ubuntu):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant)
summary: - eventlet monkey patch results in assert len(_active) == 1 AssertionError
+ [SRU] eventlet monkey patch results in assert len(_active) == 1
+ AssertionError
description: updated
Corey Bryant (corey.bryant)
Changed in neutron (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in cinder (Ubuntu Focal):
status: New → Fix Released
importance: Undecided → High
status: Fix Released → Triaged
Changed in glance (Ubuntu Focal):
status: New → Fix Released
Changed in nova (Ubuntu Focal):
status: New → Fix Released
Revision history for this message
Corey Bryant (corey.bryant) wrote :

heat is affected as well: https://storyboard.openstack.org/#!/story/2007614

Revision history for this message
Corey Bryant (corey.bryant) wrote :

It looks like at least the following are also affected:

barbican
designate
heat
ironic
ironic-inspector
magnum
manila
masakari
masakari-monitors
mistral
murano
murano-agent
networking-bagpipe
networking-hyperv,
networking-l2gw
networking-mlnx
networking-ovn
networking-sfc
neutron-dynamic-routing
neutron-fwaas
openstack-trove
python-os-ken
python-oslo.concurrency
python-oslo.messaging
python-oslo.service
ryu
sahara
senlin
swift
vmware-nsx
watcher

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:16.0.0~b3~git2020041516.5f42488a9a-0ubuntu3

---------------
neutron (2:16.0.0~b3~git2020041516.5f42488a9a-0ubuntu3) groovy; urgency=medium

  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/724753. This fixes neutron service failures
    with Python 3.8 (LP: #1863021).

 -- Corey Bryant <email address hidden> Thu, 30 Apr 2020 16:42:56 -0400

Changed in neutron (Ubuntu Groovy):
status: Triaged → Fix Released
Akihiro Motoki (amotoki)
Changed in neutron:
importance: Undecided → High
Corey Bryant (corey.bryant)
Changed in heat (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in heat (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in swift (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in swift (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cinder - 2:16.0.0~b3~git2020041012.eb915e2db-0ubuntu2

---------------
cinder (2:16.0.0~b3~git2020041012.eb915e2db-0ubuntu2) groovy; urgency=medium

  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/724754. This fixes neutron service failures
    with Python 3.8 (LP: #1863021).

 -- Corey Bryant <email address hidden> Thu, 30 Apr 2020 16:44:03 -0400

Changed in cinder (Ubuntu Groovy):
status: Triaged → Fix Released
Corey Bryant (corey.bryant)
Changed in designate (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in designate (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to designate (master)

Fix proposed to branch: master
Review: https://review.opendev.org/725346

Changed in designate:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Corey Bryant (corey.bryant)
Changed in barbican (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in barbican (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant)
Changed in ironic (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in ironic (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant)
Changed in ironic-inspector (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in ironic-inspector (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant)
Changed in python-oslo.service (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in python-oslo.service (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (master)

Fix proposed to branch: master
Review: https://review.opendev.org/725359

Changed in oslo.service:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Corey Bryant (corey.bryant)
Changed in networking-bagpipe (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in networking-bagpipe (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-hyperv (master)

Fix proposed to branch: master
Review: https://review.opendev.org/725362

Changed in networking-hyperv:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Corey Bryant (corey.bryant)
Changed in networking-l2gw (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in networking-l2gw (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in networking-hyperv (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in networking-hyperv (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in networking-mlnx (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant)
Changed in networking-mlnx (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in networking-sfc (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant)
Changed in magnum (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in magnum (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant)
Changed in manila (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/725393

Changed in manila:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Corey Bryant (corey.bryant)
Changed in manila (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in masakari (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to masakari (master)

Fix proposed to branch: master
Review: https://review.opendev.org/725397

Changed in masakari:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Corey Bryant (corey.bryant)
Changed in masakari (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant)
Changed in mistral (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in mistral (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant)
Changed in neutron-dynamic-routing (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in neutron-dynamic-routing (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-dynamic-routing (master)

Fix proposed to branch: master
Review: https://review.opendev.org/725403

OpenStack Infra (hudson-openstack)
Changed in neutron:
assignee: Corey Bryant (corey.bryant) → Dr. Jens Harbott (j-harbott)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.service (master)

Reviewed: https://review.opendev.org/725359
Committed: https://git.openstack.org/cgit/openstack/oslo.service/commit/?id=b83ce338312bfeabda6b6755f49530acde61141e
Submitter: Zuul
Branch: master

commit b83ce338312bfeabda6b6755f49530acde61141e
Author: Corey Bryant <email address hidden>
Date: Mon May 4 14:06:38 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: Icc2277b72f6f8f7812be22c43bbc281334aa2373
    Closes-Bug: #1863021

Changed in oslo.service:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/725613

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-dynamic-routing (master)

Reviewed: https://review.opendev.org/725403
Committed: https://git.openstack.org/cgit/openstack/neutron-dynamic-routing/commit/?id=44e77ea9a975e9222f38591151e2c32640ef7379
Submitter: Zuul
Branch: master

commit 44e77ea9a975e9222f38591151e2c32640ef7379
Author: Corey Bryant <email address hidden>
Date: Mon May 4 17:31:09 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I8e8a9ce5085b7915994317af5ea903cf8a39e809
    Closes-Bug: #1863021

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/724753
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ec7a5aa5161f438736c5a1dcf10f5d0e48f329cd
Submitter: Zuul
Branch: master

commit ec7a5aa5161f438736c5a1dcf10f5d0e48f329cd
Author: Corey Bryant <email address hidden>
Date: Thu Apr 30 15:44:10 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I49bfd9673abc7602b27dc48b8b490daaded2882c
    Closes-Bug: #1863021

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/725639

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-dynamic-routing (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/725660

Corey Bryant (corey.bryant)
Changed in murano (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in murano (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in murano-agent (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in murano-agent (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in openstack-trove (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in openstack-trove (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in sahara (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in sahara (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in senlin (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in senlin (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in networking-sfc (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to murano (master)

Fix proposed to branch: master
Review: https://review.opendev.org/725670

Changed in murano:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to murano-agent (master)

Fix proposed to branch: master
Review: https://review.opendev.org/725671

Changed in trove:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.opendev.org/725672

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/724754
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9f0aae5b5876b6a85fc7a30a4acb1f33f76fdb03
Submitter: Zuul
Branch: master

commit 9f0aae5b5876b6a85fc7a30a4acb1f33f76fdb03
Author: Corey Bryant <email address hidden>
Date: Thu Apr 30 15:31:18 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: Ida548b4bec00530418fd3d7ab254e971af77d3fe
    Closes-Bug: #1863021

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to murano-agent (master)

Reviewed: https://review.opendev.org/725671
Committed: https://git.openstack.org/cgit/openstack/murano-agent/commit/?id=f2e41f2f659fc4490e55d3372cfbcebb295c8ad4
Submitter: Zuul
Branch: master

commit f2e41f2f659fc4490e55d3372cfbcebb295c8ad4
Author: Corey Bryant <email address hidden>
Date: Tue May 5 15:41:02 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I0e8f80bb2e7acfb00e5ff47e0aa983afb7ee6e38
    Closes-Bug: #1863021

Changed in murano:
status: In Progress → Fix Released
Chris MacNaughton (chris.macnaughton)
Changed in swift:
assignee: nobody → Chris MacNaughton (chris.macnaughton)
Changed in watcher:
assignee: nobody → Chris MacNaughton (chris.macnaughton)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to murano (master)

Reviewed: https://review.opendev.org/725670
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=fe543a67c70e212967bda0e3e45dc9363eaf8fcf
Submitter: Zuul
Branch: master

commit fe543a67c70e212967bda0e3e45dc9363eaf8fcf
Author: Corey Bryant <email address hidden>
Date: Tue May 5 15:39:52 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Also disable E402 to allow the change in.

    Change-Id: I508fcd0707ecdd2bf720303f6cbb4087a38aaadd
    Closes-Bug: #1863021

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/725774

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-dynamic-routing (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/725790

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to murano (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/725795

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to murano-agent (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/725796

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/725797

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/725639
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1e81440b63dcafee9ad693dc0198c1598631086d
Submitter: Zuul
Branch: stable/ussuri

commit 1e81440b63dcafee9ad693dc0198c1598631086d
Author: Corey Bryant <email address hidden>
Date: Thu Apr 30 15:44:10 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I49bfd9673abc7602b27dc48b8b490daaded2882c
    Closes-Bug: #1863021
    (cherry picked from commit ec7a5aa5161f438736c5a1dcf10f5d0e48f329cd)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/725774
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9cae1e4660647dc06e1554b4fb74ea75abdad19a
Submitter: Zuul
Branch: stable/train

commit 9cae1e4660647dc06e1554b4fb74ea75abdad19a
Author: Corey Bryant <email address hidden>
Date: Thu Apr 30 15:44:10 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I49bfd9673abc7602b27dc48b8b490daaded2882c
    Closes-Bug: #1863021
    (cherry picked from commit ec7a5aa5161f438736c5a1dcf10f5d0e48f329cd)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to murano-agent (stable/ussuri)

Reviewed: https://review.opendev.org/725796
Committed: https://git.openstack.org/cgit/openstack/murano-agent/commit/?id=97f53bb54a6209bc917e1a5a727ea8f6333bafc1
Submitter: Zuul
Branch: stable/ussuri

commit 97f53bb54a6209bc917e1a5a727ea8f6333bafc1
Author: Corey Bryant <email address hidden>
Date: Tue May 5 15:41:02 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I0e8f80bb2e7acfb00e5ff47e0aa983afb7ee6e38
    Closes-Bug: #1863021
    (cherry picked from commit f2e41f2f659fc4490e55d3372cfbcebb295c8ad4)

OpenStack Infra (hudson-openstack)
Changed in networking-l2gw:
status: In Progress → Fix Released
OpenStack Infra (hudson-openstack)
Changed in networking-bagpipe:
status: In Progress → Fix Released
OpenStack Infra (hudson-openstack)
Changed in networking-sfc:
status: In Progress → Fix Released
OpenStack Infra (hudson-openstack)
Changed in networking-hyperv:
status: In Progress → Fix Released
Corey Bryant (corey.bryant)
Changed in python-os-ken (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in python-os-ken (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in watcher (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in watcher (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Arthur Dayne (palagend)
Changed in barbican (Ubuntu Groovy):
assignee: nobody → Arthur Dayne (palagend)
Changed in mistral (Ubuntu Groovy):
assignee: nobody → Arthur Dayne (palagend)
Changed in ironic (Ubuntu Groovy):
assignee: nobody → Arthur Dayne (palagend)
Arthur Dayne (palagend)
Changed in manila (Ubuntu Groovy):
assignee: nobody → Arthur Dayne (palagend)
Changed in watcher (Ubuntu Groovy):
assignee: nobody → Arthur Dayne (palagend)
OpenStack Infra (hudson-openstack)
Changed in manila:
assignee: Corey Bryant (corey.bryant) → suzhengwei (sue.sam)
OpenStack Infra (hudson-openstack)
Changed in senlin:
status: In Progress → Fix Released
Corey Bryant (corey.bryant)
Changed in swift (Ubuntu Focal):
importance: High → Low
Changed in swift (Ubuntu Groovy):
importance: High → Low
Launchpad Janitor (janitor)
Changed in ironic (Ubuntu Groovy):
status: Triaged → Fix Released
OpenStack Infra (hudson-openstack)
Changed in mistral:
status: In Progress → Fix Released
OpenStack Infra (hudson-openstack)
Changed in watcher:
status: In Progress → Fix Released
Launchpad Janitor (janitor)
Changed in designate (Ubuntu Groovy):
status: Triaged → Fix Released
Corey Bryant (corey.bryant)
description: updated
Launchpad Janitor (janitor)
Changed in barbican (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in networking-hyperv (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in networking-bagpipe (Ubuntu Groovy):
status: Triaged → Fix Released
Launchpad Janitor (janitor)
Changed in ironic-inspector (Ubuntu Groovy):
status: Triaged → Fix Released
Launchpad Janitor (janitor)
Changed in manila (Ubuntu Groovy):
status: Triaged → Fix Released
Launchpad Janitor (janitor)
Changed in python-os-ken (Ubuntu Groovy):
status: Triaged → Fix Released
Launchpad Janitor (janitor)
Changed in networking-sfc (Ubuntu Groovy):
status: Triaged → Fix Released
Launchpad Janitor (janitor)
Changed in python-oslo.service (Ubuntu Groovy):
status: Triaged → Fix Released
Launchpad Janitor (janitor)
Changed in networking-mlnx (Ubuntu Groovy):
status: Triaged → Fix Released
Launchpad Janitor (janitor)
Changed in mistral (Ubuntu Groovy):
status: Triaged → Fix Released
Launchpad Janitor (janitor)
Changed in senlin (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in watcher (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in openstack-trove (Ubuntu Groovy):
status: Triaged → Fix Released
Corey Bryant (corey.bryant)
Changed in murano (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in murano-agent (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in networking-l2gw (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in neutron-dynamic-routing (Ubuntu Groovy):
status: Triaged → Fix Released
Steve Langasek (vorlon)
Changed in ironic (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-focal
Launchpad Janitor (janitor)
Changed in magnum (Ubuntu Groovy):
status: Triaged → Fix Released
OpenStack Infra (hudson-openstack)
Changed in manila:
status: In Progress → Fix Released
Steve Langasek (vorlon)
Changed in heat (Ubuntu Focal):
status: Triaged → Fix Committed
Steve Langasek (vorlon)
Changed in designate (Ubuntu Focal):
status: Triaged → Fix Committed
Steve Langasek (vorlon)
Changed in barbican (Ubuntu Focal):
status: Triaged → Fix Committed
Steve Langasek (vorlon)
Changed in ironic-inspector (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in magnum (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in openstack-trove (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in masakari (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in networking-mlnx (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in networking-bagpipe (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in python-os-ken (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in python-oslo.service (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in networking-hyperv (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in networking-sfc (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in manila (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in mistral (Ubuntu Focal):
status: Triaged → Fix Committed
Brian Murray (brian-murray)
Changed in senlin (Ubuntu Focal):
status: Triaged → Fix Committed
Corey Bryant (corey.bryant)
Changed in masakari (Ubuntu Focal):
status: Fix Committed → Triaged
Brian Murray (brian-murray)
Changed in watcher (Ubuntu Focal):
status: Triaged → Fix Committed
Launchpad Janitor (janitor)
Changed in masakari (Ubuntu Groovy):
status: Triaged → Fix Released
Brian Murray (brian-murray)
Changed in masakari (Ubuntu Focal):
status: Triaged → Fix Committed
OpenStack Infra (hudson-openstack)
Changed in trove:
assignee: Corey Bryant (corey.bryant) → Lingxian Kong (kong)
Corey Bryant (corey.bryant)
tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Corey Bryant (corey.bryant)
Changed in cinder (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in murano (Ubuntu Focal):
status: Triaged → Fix Committed
Corey Bryant (corey.bryant)
Changed in murano-agent (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in networking-l2gw (Ubuntu Focal):
status: Triaged → Fix Committed
Corey Bryant (corey.bryant)
Changed in neutron (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in neutron-dynamic-routing (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in sahara (Ubuntu Focal):
status: Triaged → Fix Committed
Corey Bryant (corey.bryant)
Changed in sahara (Ubuntu Groovy):
status: Triaged → Fix Released
Launchpad Janitor (janitor)
Changed in heat (Ubuntu Groovy):
status: Triaged → Fix Released
OpenStack Infra (hudson-openstack)
Changed in designate:
status: In Progress → Fix Released
70 comments hidden view all 150 comments
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ironic - 1:15.0.0-0ubuntu0.20.04.1

---------------
ironic (1:15.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/control: Align (Build-)Depends with upstream.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/725857. This fixes neutron service failures
    with Python 3.8 (LP: #1863021).

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Wed, 13 May 2020 13:06:46 -0400

Changed in ironic (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for ironic has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package heat - 1:14.0.0-0ubuntu0.20.04.1

---------------
heat (1:14.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  * d/watch: Scope to 14.x series and get tarballs from opendev.org.
  * d/control: Align (Build-)Depends with upstream.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/727181/)
    to fix Python 3.8 monkey patching (LP: #1863021).
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Wed, 13 May 2020 16:51:44 -0400

Changed in heat (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package magnum - 10.0.0-0ubuntu0.20.04.1

---------------
magnum (10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/728010. This fixes service failures
    with Python 3.8 (LP: #1863021).

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 12:31:17 -0400

Changed in magnum (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package designate - 1:10.0.0-0ubuntu0.20.04.1

---------------
designate (1:10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  * d/watch: Get tarballs from opendev.org and scope to 10.x series.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/725346/)
    to fix Python 3.8 monkey patching (LP: #1863021).
  * d/p/disable-edgegrid.patch: Rebased.
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Wed, 13 May 2020 16:50:55 -0400

Changed in designate (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ironic-inspector - 1:10.1.0-0ubuntu0.20.04.1

---------------
ironic-inspector (1:10.1.0-0ubuntu0.20.04.1) focal; urgency=medium

  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/727184. This fixes service failures
    with Python 3.8 (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Chris MacNaughton <email address hidden> Thu, 14 May 2020 09:57:10 +0000

Changed in ironic-inspector (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package barbican - 1:10.0.0-0ubuntu0.20.04.1

---------------
barbican (1:10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * d/watch: Update to get release tarballs from opendev.org.
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/watch: update to the correct version regex.
  * d/gbp.conf: Create stable/ussuri branch.

  [ Corey Bryant ]
  * d/watch: Scope to 10.x series.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/725353/)
    to fix Python 3.8 monkey patching (LP: #1863021).

 -- Chris MacNaughton <email address hidden> Thu, 14 May 2020 08:13:29 +0000

Changed in barbican (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openstack-trove - 1:13.0.0-0ubuntu0.20.04.1

---------------
openstack-trove (1:13.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/watch: Scope to 13.x series.
  * d/watch: Update to point at tarballs.opendev.org.
  * d/control: Align (Build-)Depends with upstream.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/725672/)
    to fix Python 3.8 monkey patching (LP: #1863021).

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Fri, 15 May 2020 09:50:59 -0400

Changed in openstack-trove (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networking-mlnx - 1:15.0.2-0ubuntu0.20.04.1

---------------
networking-mlnx (1:15.0.2-0ubuntu0.20.04.1) focal; urgency=medium

  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/control: Align (Build-)Depends with upstream.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/725365)
    to fix Python 3.8 monkey patching (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 15:15:09 -0400

Changed in networking-mlnx (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networking-bagpipe - 12.0.0-0ubuntu0.20.04.1

---------------
networking-bagpipe (12.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  * d/watch: Scope to 12.x series and get tarballs from opendev.org.
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream to fix Python 3.8 monkey patching (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 13:52:15 -0400

Changed in networking-bagpipe (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-oslo.service - 2.1.1-0ubuntu1.1

---------------
python-oslo.service (2.1.1-0ubuntu1.1) focal; urgency=medium

  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/725853/)
    to fix Python 3.8 monkey patching (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 12:56:30 -0400

Changed in python-oslo.service (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networking-hyperv - 1:8.0.0-0ubuntu0.20.04.1

---------------
networking-hyperv (1:8.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream to fix Python 3.8 monkey patching (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 13:48:05 -0400

Changed in networking-hyperv (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networking-sfc - 10.0.0-0ubuntu0.20.04.1

---------------
networking-sfc (10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  * d/watch: Scope to 10.x series and get tarballs from opendev.org.
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream to fix Python 3.8 monkey patching (LP: #1863021).
  * d/control: Align (Build-)Depends with upstream.
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 14:41:04 -0400

Changed in networking-sfc (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mistral - 10.0.0-0ubuntu0.20.04.1

---------------
mistral (10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/728369/. This fixes service failures
    with Python 3.8 (LP: #1863021)
  * d/watch: Scope to 10.x series.
  * d/watch: Update to point at tarballs.opendev.org.
  * d/control: Align (Build-)Depends with upstream.

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Fri, 15 May 2020 09:37:52 -0400

Changed in mistral (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package senlin - 9.0.0-0ubuntu0.20.04.1

---------------
senlin (9.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/727186/. This fixes service failures
    with Python 3.8 (LP: #1863021)
  * d/watch: Scope to 9.x series.
  * d/watch: Update to point at tarballs.opendev.org.

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Fri, 15 May 2020 10:02:46 -0400

Changed in senlin (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package masakari - 9.0.0-0ubuntu0.20.04.2

---------------
masakari (9.0.0-0ubuntu0.20.04.2) focal; urgency=medium

  * d/control, d/py3dist-overrides: Drop (Build-)Depends on
    universe package python3-sqlalchemy-utils. This was added in
    9.0.0 but only needed for upstream devstack gate job.

masakari (9.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/728104/. This fixes service failures
    with Python 3.8 (LP: #1863021)
  * d/watch: Scope to 9.x series.
  * d/watch: Update to point at tarballs.opendev.org.
  * d/control: Align (Build-)Depends with upstream.

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Tue, 19 May 2020 13:04:38 -0400

Changed in masakari (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-os-ken - 1.0.0-0ubuntu1.1

---------------
python-os-ken (1.0.0-0ubuntu1.1) focal; urgency=medium

  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream to fix Python 3.8 monkey patching (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 13:02:34 -0400

Changed in python-os-ken (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package manila - 1:10.0.0-0ubuntu0.20.04.1

---------------
manila (1:10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/728079/. This fixes service failures
    with Python 3.8 (LP: #1863021)
  * d/watch: Scope to 10.x series.
  * d/watch: Update to point at tarballs.opendev.org.

  [ Corey Bryant ]
  * d/control: Align (Build-)Depends with upstream.
  * d/gbp.conf: Create stable/ussuri branch.

 -- Chris MacNaughton <email address hidden> Thu, 14 May 2020 13:36:55 +0000

Changed in manila (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package watcher - 1:4.0.0-0ubuntu0.20.04.1

---------------
watcher (1:4.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/728397/. This fixes service failures
    with Python 3.8 (LP: #1863021)
  * d/watch: Scope to 4.x series.
  * d/watch: Update to point at tarballs.opendev.org.

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Fri, 15 May 2020 10:03:39 -0400

Changed in watcher (Ubuntu Focal):
status: Fix Committed → Fix Released
Corey Bryant (corey.bryant)
Changed in neutron (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in neutron-dynamic-routing (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in sahara (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in networking-l2gw (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in murano-agent (Ubuntu Focal):
status: Fix Committed → Fix Released
Corey Bryant (corey.bryant)
Changed in murano (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in cinder (Ubuntu Focal):
status: Fix Committed → Fix Released
Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Even with all these patches in, I'm still seeing messages like this with devstack running on focal, see https://review.opendev.org/704831:

Jun 01 16:23:06.645371 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[76054]: DEBUG neutron.agent.linux.utils [-] Running command: ['env', 'LC_ALL=C', 'PATH=/sbin:/us
r/sbin', 'dnsmasq', '--test', '--dhcp-host=tag:foo'] {{(pid=76054) create_process /opt/stack/neutron/neutron/agent/linux/utils.py:87}}
Jun 01 16:23:06.651565 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[78057]: Exception ignored in: <function _after_fork at 0x7fc960125d30>
Jun 01 16:23:06.651565 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[78057]: Traceback (most recent call last):
Jun 01 16:23:06.651565 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[78057]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 01 16:23:06.651565 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[78057]: assert len(_active) == 1
Jun 01 16:23:06.651565 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[78057]: AssertionError:

This happens for neutron-dhcp-agent, neutron-ovs-agent, nova-compute and cinder-volume, seemingly always on a call of subprocess.Popen.

Revision history for this message
sean mooney (sean-k-mooney) wrote :

im not sure that https://review.opendev.org/#/c/707474/ acttully works
or at the very least is a complete fix.

as noted in https://bugs.launchpad.net/nova/+bug/1882521/comments/1
we still see the same threading error.

i think we likely need to patch oslo_concurrency too or look into another fix.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on senlin (master)

Change abandoned by Qiu Fossen (<email address hidden>) on branch: master
Review: https://review.opendev.org/726333

Revision history for this message
Dr. Clemens Hardewig (bringha1) wrote :

Hmmm .. not sure whether this patch fixes all scenarios with nova-compute:

We just created a new compute server from scratch with clean install ubuntu focal, python 3.8 and Ussuri nova-compute (nova-compute/focal-updates,now 2:21.0.0-0ubuntu0.20.04.1).
I double checked that this patch has been applied (which is in /usr/lib/python3/dist-upgrade/nova/monkey_patch.py).
However, my syslog is spammed with
Jun 14 09:53:20 compute3 nova-compute[2759819]: AssertionError:
Jun 14 09:53:20 compute3 nova-compute[2759824]: Exception ignored in: <function _after_fork at 0x7f91837ae820>
Jun 14 09:53:20 compute3 nova-compute[2759824]: Traceback (most recent call last):
Jun 14 09:53:20 compute3 nova-compute[2759824]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 14 09:53:20 compute3 nova-compute[2759824]: assert len(_active) == 1
Jun 14 09:53:20 compute3 nova-compute[2759824]: AssertionError:
Jun 14 09:53:20 compute3 nova-compute[2759827]: Exception ignored in: <function _after_fork at 0x7f91837ae820>
Jun 14 09:53:20 compute3 nova-compute[2759827]: Traceback (most recent call last):
Jun 14 09:53:20 compute3 nova-compute[2759827]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 14 09:53:20 compute3 nova-compute[2759827]: assert len(_active) == 1
Jun 14 09:53:20 compute3 nova-compute[2759827]: AssertionError:
Jun 14 09:53:20 compute3 nova-compute[2759830]: Exception ignored in: <function _after_fork at 0x7f91837ae820>
Jun 14 09:53:20 compute3 nova-compute[2759830]: Traceback (most recent call last):
Jun 14 09:53:20 compute3 nova-compute[2759830]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 14 09:53:20 compute3 nova-compute[2759830]: assert len(_active) == 1
Jun 14 09:53:20 compute3 nova-compute[2759830]: AssertionError:
Jun 14 09:53:20 compute3 nova-compute[2759833]: Exception ignored in: <function _after_fork at 0x7f91837ae820>
Jun 14 09:53:20 compute3 nova-compute[2759833]: Traceback (most recent call last):
Jun 14 09:53:20 compute3 nova-compute[2759833]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 14 09:53:20 compute3 nova-compute[2759833]: assert len(_active) == 1
nova-compute ist running and looked to be functional.

Did I overlook something?
Thanks for your reply

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

You may be right.

I have this in monkey_patch.py:

    # Monkey patch the original current_thread to use the up-to-date _active
    # global variable. See https://bugs.launchpad.net/bugs/1863021 and
    # https://github.com/eventlet/eventlet/issues/592
    import __original_module_threading as orig_threading
    import threading
    orig_threading.current_thread.__globals__['_active'] = threading._active

However, nova-compute logs still get spammed with:

Jun 16 14:06:30 node nova-compute[12592]: Traceback (most recent call last):
Jun 16 14:06:30 node nova-compute[12592]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 16 14:06:30 node nova-compute[12592]: assert len(_active) == 1
Jun 16 14:06:30 node nova-compute[12592]: AssertionError:
Jun 16 14:06:30 node nova-compute[12592]: Exception ignored in: <function _after_fork at 0x7efdc80b7b80>

Revision history for this message
Tim Burke (1-tim-z) wrote :

I wonder if an approach like in my https://github.com/eventlet/eventlet/pull/611 might work better.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.opendev.org/725672
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=56bb6f39b6c808160d2e4c96fcc2104ba1daf817
Submitter: Zuul
Branch: master

commit 56bb6f39b6c808160d2e4c96fcc2104ba1daf817
Author: Corey Bryant <email address hidden>
Date: Tue May 5 15:42:31 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I61e5e270bf66b0355da3282c19cbc9fd42c4090b
    Closes-Bug: #1863021

Changed in trove:
status: In Progress → Fix Released
Bernard Cafarelli (bcafarel)
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to senlin (stable/ussuri)

Reviewed: https://review.opendev.org/727186
Committed: https://git.openstack.org/cgit/openstack/senlin/commit/?id=fb738fcd276e3d0aeee975964aaa576c51c0842e
Submitter: Zuul
Branch: stable/ussuri

commit fb738fcd276e3d0aeee975964aaa576c51c0842e
Author: Corey Bryant <email address hidden>
Date: Tue May 5 15:46:38 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I5c2fbe1827195d670a67dd62cdadce5ee3513ace
    Closes-Bug: #1863021
    (cherry picked from commit 666fd70b6100e329e7eaec82e808ad0722d67048)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/743431

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/train)

Reviewed: https://review.opendev.org/743431
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=779ae5f1f1cd8328d1423f13550c21fe53b0c803
Submitter: Zuul
Branch: stable/train

commit 779ae5f1f1cd8328d1423f13550c21fe53b0c803
Author: Corey Bryant <email address hidden>
Date: Mon May 4 17:04:40 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: Ifc6420d927c0ce9e04ff3b3253e81a474591e9bb
    Closes-Bug: #1863021
    (cherry picked from commit 5e9f694a5a8f90c72680acb1181318930f55aa30)
    (cherry picked from commit 7196cfcb091d66856a1ca0db53816198249196db)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-bagpipe (stable/ussuri)

Reviewed: https://review.opendev.org/726073
Committed: https://git.openstack.org/cgit/openstack/networking-bagpipe/commit/?id=0e866eed9f6c3e545a2fd6c0b680a718996661a5
Submitter: Zuul
Branch: stable/ussuri

commit 0e866eed9f6c3e545a2fd6c0b680a718996661a5
Author: Corey Bryant <email address hidden>
Date: Mon May 4 14:18:49 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I5ea4bb27361df3c489deedb51d7ca8ea64bb923b
    Closes-Bug: #1863021
    (cherry picked from commit f0db421843f1161f4daa1e08b83eb2b11e78d9dd)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-dynamic-routing (stable/train)

Reviewed: https://review.opendev.org/725790
Committed: https://git.openstack.org/cgit/openstack/neutron-dynamic-routing/commit/?id=56de1c4330de76256937ecd0b1d8cde8f16d9f0a
Submitter: Zuul
Branch: stable/train

commit 56de1c4330de76256937ecd0b1d8cde8f16d9f0a
Author: Corey Bryant <email address hidden>
Date: Mon May 4 17:31:09 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I8e8a9ce5085b7915994317af5ea903cf8a39e809
    Closes-Bug: #1863021
    (cherry picked from commit 44e77ea9a975e9222f38591151e2c32640ef7379)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to masakari (master)

Reviewed: https://review.opendev.org/725397
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=97e658417b691f6f737599c59726cc49131b05ef
Submitter: Zuul
Branch: master

commit 97e658417b691f6f737599c59726cc49131b05ef
Author: Corey Bryant <email address hidden>
Date: Mon May 4 17:11:13 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I4929bfd4dc60f97b27459a4c6e7ed649c5e7f645
    Closes-Bug: #1863021

Changed in masakari:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-sfc (stable/train)

Reviewed: https://review.opendev.org/726078
Committed: https://git.openstack.org/cgit/openstack/networking-sfc/commit/?id=31b28ea8355af8c64cfddc48d4045de4613ba199
Submitter: Zuul
Branch: stable/train

commit 31b28ea8355af8c64cfddc48d4045de4613ba199
Author: Corey Bryant <email address hidden>
Date: Mon May 4 14:30:35 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I5b00ee328f83cec8375ad1538be3a16059af08a3
    Closes-Bug: #1863021
    (cherry picked from commit 0b78dad62e72caa4d348cb3087f97f138f26a0cf)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-bagpipe 11.0.2

This issue was fixed in the openstack/networking-bagpipe 11.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-sfc 9.0.1

This issue was fixed in the openstack/networking-sfc 9.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-mlnx (master)

Reviewed: https://review.opendev.org/c/x/networking-mlnx/+/725365
Committed: https://opendev.org/x/networking-mlnx/commit/2deee96bdf718bde85d843b1983e65a9638ff593
Submitter: "Zuul (22348)"
Branch: master

commit 2deee96bdf718bde85d843b1983e65a9638ff593
Author: Corey Bryant <email address hidden>
Date: Mon May 4 14:29:36 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I113aaefc33b4b10dd037fc2ef00fc5f507be8b22
    Closes-Bug: #1863021

Changed in networking-mlnx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-sfc 10.0.1

This issue was fixed in the openstack/networking-sfc 10.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-hyperv 8.0.1

This issue was fixed in the openstack/networking-hyperv 8.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/watcher 4.0.1

This issue was fixed in the openstack/watcher 4.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/senlin 9.0.1

This issue was fixed in the openstack/senlin 9.0.1 release.

Displaying first 40 and last 40 comments. View all 150 comments or add a comment.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.