do we have any update regarding this issue. We are also observing the similar issue where libvirt give up after certain number of retries and we saw following logs in libvirt.
nce: 91add701-b719-496e-9d37-1332568b4ea0] Detach interface failed, port_id=0dd877b0-5431-4f52-bb63-94727c29048c, reason: Device detach failed for tap0dd877b0-54: Run out of retry while detaching device tap0dd877b0-54 with device alias hostdev6 from instance 91add701-b719-496e-9d37-1332568b4ea0 from the live domain config. Device is still attached to the guest.: nova.exception.DeviceDetachFailed: Device detach failed for tap0dd877b0-54: Run out of retry while detaching device tap0dd877b0-54 with device alias hostdev6 from instance 91add701-b719-496e-9d37-1332568b4ea0 from the live domain config. Device is still attached to the guest.
2023-09-08 03:56:11.353 3559248 DEBUG oslo_concurrency.lockutils [req-572b95b9-e051-4de1-a36d-b6ba2128141a f7de05eb5ec71d4dd8ccb3b765fb4c9eb2206acc00018d852f9ff4086fc92cdc f569a10821d544a7877d3130d9da33b8 - 8a596ebd3f9745f7a2948fd057d43920 07bbc0238c5241dd8c90d27af90a5ef7] Lock "interface-91add701-b719-496e-9d37-1332568b4ea0-0dd877b0-5431-4f52-bb63-94727c29048c" "released" by "nova.compute.manager.ComputeManager.detach_interface.<locals>.do_detach_interface" :: held 200.147s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:405
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server [req-572b95b9-e051-4de1-a36d-b6ba2128141a f7de05eb5ec71d4dd8ccb3b765fb4c9eb2206acc00018d852f9ff4086fc92cdc f569a10821d544a7877d3130d9da33b8 - 8a596ebd3f9745f7a2948fd057d43920 07bbc0238c5241dd8c90d27af90a5ef7] Exception during message handling: nova.exception.InterfaceDetachFailed: Failed to detach network adapter device from 91add701-b719-496e-9d37-1332568b4ea0
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8049, in _detach_interface
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server self.driver.detach_interface(context, instance, condemned)
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2890, in detach_interface
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server device_name=self.vif_driver.get_vif_devname(vif),
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2429, in _detach_with_retry
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server device_name)
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2508, in _detach_from_live_with_retry
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server reason=msg % (device_name, live_dev.alias, instance_uuid))
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for tap0dd877b0-54: Run out of retry while detaching device tap0dd877b0-54 with device alias hostdev6 from instance 91add701-b719-496e-9d37-1332568b4ea0 from the live domain config. Device is still attached to the guest.
do we have any update regarding this issue. We are also observing the similar issue where libvirt give up after certain number of retries and we saw following logs in libvirt.
nce: 91add701- b719-496e- 9d37-1332568b4e a0] Detach interface failed, port_id= 0dd877b0- 5431-4f52- bb63-94727c2904 8c, reason: Device detach failed for tap0dd877b0-54: Run out of retry while detaching device tap0dd877b0-54 with device alias hostdev6 from instance 91add701- b719-496e- 9d37-1332568b4e a0 from the live domain config. Device is still attached to the guest.: nova.exception. DeviceDetachFai led: Device detach failed for tap0dd877b0-54: Run out of retry while detaching device tap0dd877b0-54 with device alias hostdev6 from instance 91add701- b719-496e- 9d37-1332568b4e a0 from the live domain config. Device is still attached to the guest. y.lockutils [req-572b95b9- e051-4de1- a36d-b6ba212814 1a f7de05eb5ec71d4 dd8ccb3b765fb4c 9eb2206acc00018 d852f9ff4086fc9 2cdc f569a10821d544a 7877d3130d9da33 b8 - 8a596ebd3f9745f 7a2948fd057d439 20 07bbc0238c5241d d8c90d27af90a5e f7] Lock "interface- 91add701- b719-496e- 9d37-1332568b4e a0-0dd877b0- 5431-4f52- bb63-94727c2904 8c" "released" by "nova.compute. manager. ComputeManager. detach_ interface. <locals> .do_detach_ interface" :: held 200.147s inner /usr/lib/ python3. 6/site- packages/ oslo_concurrenc y/lockutils. py:405 rpc.server [req-572b95b9- e051-4de1- a36d-b6ba212814 1a f7de05eb5ec71d4 dd8ccb3b765fb4c 9eb2206acc00018 d852f9ff4086fc9 2cdc f569a10821d544a 7877d3130d9da33 b8 - 8a596ebd3f9745f 7a2948fd057d439 20 07bbc0238c5241d d8c90d27af90a5e f7] Exception during message handling: nova.exception. InterfaceDetach Failed: Failed to detach network adapter device from 91add701- b719-496e- 9d37-1332568b4e a0 rpc.server Traceback (most recent call last): rpc.server File "/usr/lib/ python3. 6/site- packages/ nova/compute/ manager. py", line 8049, in _detach_interface rpc.server self.driver. detach_ interface( context, instance, condemned) rpc.server File "/usr/lib/ python3. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 2890, in detach_interface rpc.server device_ name=self. vif_driver. get_vif_ devname( vif), rpc.server File "/usr/lib/ python3. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 2429, in _detach_with_retry rpc.server device_name) rpc.server File "/usr/lib/ python3. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 2508, in _detach_ from_live_ with_retry rpc.server reason=msg % (device_name, live_dev.alias, instance_uuid)) rpc.server nova.exception. DeviceDetachFai led: Device detach failed for tap0dd877b0-54: Run out of retry while detaching device tap0dd877b0-54 with device alias hostdev6 from instance 91add701- b719-496e- 9d37-1332568b4e a0 from the live domain config. Device is still attached to the guest.
2023-09-08 03:56:11.353 3559248 DEBUG oslo_concurrenc
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.
2023-09-08 03:56:11.449 3559248 ERROR oslo_messaging.