Thanks Aboubacar for the well written bug report.
I agree that we have a race between the disconnect and the swap operation. Both uses a lock but they use different locks so they can overlap.
Failed case:
Sep 30 02:51:47 Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.156s {{(pid=2571444) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} Sep 30 02:51:55 Lock "0d711b7b-4693-4a7e-9a94-ca4186b4a670" "released" by "nova.compute.manager.ComputeManager.swap_volume.<locals>._do_locked_swap_volume" :: held 153.400s {{(pid=2571444) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}}
Successful case:
Sep 29 17:12:21 Lock "4aeaeb5d-295f-4149-9330-a016d9da1730" "released" by "nova.compute.manager.ComputeManager.swap_volume.<locals>._do_locked_swap_volume" :: held 632.783s {{(pid=2571444) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} Sep 29 17:12:25 Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.142s {{(pid=2571444) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}}
Thanks Aboubacar for the well written bug report.
I agree that we have a race between the disconnect and the swap operation. Both uses a lock but they use different locks so they can overlap.
Failed case:
Sep 30 02:51:47 Lock "connect_volume" "released" by "os_brick. initiator. connectors. iscsi.ISCSIConn ector.disconnec t_volume" :: held 0.156s {{(pid=2571444) inner /usr/local/ lib/python3. 8/dist- packages/ oslo_concurrenc y/lockutils. py:400} } 4693-4a7e- 9a94-ca4186b4a6 70" "released" by "nova.compute. manager. ComputeManager. swap_volume. <locals> ._do_locked_ swap_volume" :: held 153.400s {{(pid=2571444) inner /usr/local/ lib/python3. 8/dist- packages/ oslo_concurrenc y/lockutils. py:400} }
Sep 30 02:51:55 Lock "0d711b7b-
Successful case:
Sep 29 17:12:21 Lock "4aeaeb5d- 295f-4149- 9330-a016d9da17 30" "released" by "nova.compute. manager. ComputeManager. swap_volume. <locals> ._do_locked_ swap_volume" :: held 632.783s {{(pid=2571444) inner /usr/local/ lib/python3. 8/dist- packages/ oslo_concurrenc y/lockutils. py:400} } initiator. connectors. iscsi.ISCSIConn ector.disconnec t_volume" :: held 0.142s {{(pid=2571444) inner /usr/local/ lib/python3. 8/dist- packages/ oslo_concurrenc y/lockutils. py:400} }
Sep 29 17:12:25 Lock "connect_volume" "released" by "os_brick.