Observed in a CI job: octavia-v2-dsvm-scenario-traffic-ops-jobboard on master
The creation of a LB failed (LB stuck in PENDING_CREATE), then the worker was stuck
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7a5/898232/1/check/octavia-v2-dsvm-scenario-traffic-ops-jobboard/7a5e18f/controller/logs/screen-o-cw.txt
Oct 13 14:42:12.058434 np0035492616 octavia-worker[100079]: DEBUG cotyledon._service [-] Run service ConsumerService(1) [100079] {{(pid=100079) wait_forever /opt/stack/data/venv/lib/python3.10/site-packages/cotyledon/_service.py:241}}
Oct 13 14:42:12.059230 np0035492616 octavia-worker[100079]: INFO octavia.controller.queue.v2.consumer [-] Starting V2 consumer...
Oct 13 14:45:31.303244 np0035492616 octavia-worker[100077]: INFO octavia.controller.queue.v2.endpoints [-] Creating load balancer 'f327ad7e-d41f-4271-9aa2-b8d3c2ce645a'...
Oct 13 14:45:31.309590 np0035492616 octavia-worker[100077]: Traceback (most recent call last):
Oct 13 14:45:31.309590 np0035492616 octavia-worker[100077]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
Oct 13 14:45:31.309590 np0035492616 octavia-worker[100077]: timer()
Oct 13 14:45:31.309590 np0035492616 octavia-worker[100077]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Oct 13 14:45:31.309590 np0035492616 octavia-worker[100077]: cb(*args, **kw)
Oct 13 14:45:31.309590 np0035492616 octavia-worker[100077]: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
Oct 13 14:45:31.309590 np0035492616 octavia-worker[100077]: waiter.switch()
Oct 13 14:45:31.310105 np0035492616 octavia-worker[100077]: greenlet.error: cannot switch to a different thread
based on opensearch, it's happened twice in the last 30 days:
(note that the run reported in the launchpad doesn't appear in opensearch)
octavia- v2-dsvm- cinder- amphora master v2-dsvm- tls-barbican master
octavia-