Took a look on this and below are the findings:- - The issue is seen across multiple projects and jobs and is a random one, found a related bug in nova created long back[1] but that too don't have a RCA. - Both tls/non-tls jobs impacted across different jobs/project/branches[2]. - seeing the issue mostly in stable/zed+ i suspected dbcounter is related and to rule it out tested disabling it in https://review.opendev.org/c/openstack/neutron/+/883282 but the issue reproduced[5] - I checked couple of job logs and seen 2 categories(stuck[3] vs taking long time[4] than 60seconds or 180seconds(in rally job)), the stuck ones seen only in nova while non-stuck ones across projects so these can be considered different issue and investigated seperately, this bug can focus on the stuck case. - In some cases seen oslo_messaging disconnections, so not sure if that's the issue and if heartbeat_in_pthread=True will help in this or if it's an issue with eventlet/uwsgi threads monkey patching, i think someone from nova may have some idea over this. - Next i would like to collect gmr if that can give some hint for the issue. [1] https://bugs.launchpad.net/tempest/+bug/1999893 [2] HTTPSConnectionPool master 93.4% stable/zed 2.5% stable/2023.1 1.6% stable/wallaby 0.8% stable/victoria 0.8% nova-ceph-multistore 49.6% glance-multistore-cinder-import 13.1% tempest-ipv6-only 4.9% cinder-tempest-plugin-lvm-tgt-barbican 4.1% nova-next 2.5% openstack/nova 49.2% openstack/glance 15.6% openstack/cinder-tempest-plugin 15.2% openstack/tempest 6.6% openstack/neutron 5.7% HTTPConnectionPool nova-grenade-multinode 48.0% grenade 12.0% neutron-ovs-grenade-dvr-multinode 12.0% grenade-skip-level-always 8.0% neutron-ovs-grenade-multinode 8.0% master 72.0% stable/2023.1 24.0% stable/zed 4.0% openstack/nova 56.0% openstack/neutron 24.0% openstack/cinder 8.0% openstack/devstack 8.0% openstack/tempest 4.0% [3] http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_77c/877533/7/check/neutron-ovs-grenade-multinode/77c03cd/testr_results.html May 09 11:41:36.941261 np0033988595