Frequent threading lockup when using rbd

Bug #2020122 reported by Dan Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glance_store
Fix Released
High
Unassigned

Bug Description

Recently glance_store can lock up while copying an image to an RBD store. This manifests in CI with the following tempest failure:

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/api/image/v2/test_images.py", line 102, in test_image_glance_direct_import
    waiters.wait_for_image_imported_to_stores(self.client, image_id)
  File "/opt/stack/tempest/tempest/common/waiters.py", line 272, in wait_for_image_imported_to_stores
    raise exc_cls(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (ImportImagesTest:test_image_glance_direct_import) Image 4a36ddf9-5f8c-42f8-821e-7e812d59f6dd failed to import on stores:

The following traceback can be found in the glance API logs when this happens:

May 18 00:03:54.307045 np0034059457 <email address hidden>[109619]: Traceback (most recent call last):
May 18 00:03:54.307045 np0034059457 <email address hidden>[109619]: File "/usr/local/lib/python3.10/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers
May 18 00:03:54.307045 np0034059457 <email address hidden>[109619]: timer()
May 18 00:03:54.307045 np0034059457 <email address hidden>[109619]: File "/usr/local/lib/python3.10/dist-packages/eventlet/hubs/timer.py", line 59, in __call__
May 18 00:03:54.307045 np0034059457 <email address hidden>[109619]: cb(*args, **kw)
May 18 00:03:54.307045 np0034059457 <email address hidden>[109619]: File "/usr/local/lib/python3.10/dist-packages/eventlet/event.py", line 175, in _do_send
May 18 00:03:54.307045 np0034059457 <email address hidden>[109619]: waiter.switch(result)
May 18 00:03:54.307750 np0034059457 <email address hidden>[109619]: greenlet.error: cannot switch to a different thread

Dan Smith (danms)
Changed in glance-store:
importance: Undecided → High
Changed in glance-store:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (master)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/883532
Committed: https://opendev.org/openstack/glance_store/commit/99f4014493fa6178961a588ca6e1c52b4cc8d549
Submitter: "Zuul (22348)"
Branch: master

commit 99f4014493fa6178961a588ca6e1c52b4cc8d549
Author: Dan Smith <email address hidden>
Date: Thu May 18 14:12:25 2023 +0000

    Revert "RBD: Wrap RBD calls in native threads"

    This reverts commit 27ab8a6aeb7a59bc56ed0ad26ffb5a83d426b447.

    Reason for revert: The change introduced a blocking library call for
    the thread running import jobs.

    Closes-Bug: #2020122
    Change-Id: I83e3a0da9fdf7167cf4adf7257d5960338901158

Changed in glance-store:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 4.5.0

This issue was fixed in the openstack/glance_store 4.5.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.