This error happens when I live-migrate or resize an instance to another compute host. I also checked the SAN configuration to ensure the corresponding LUN was completely changed to a new host's initiator and LUN-ID. My Openstack version is Victoria and Netapp Ontap driver for SAN.
After this error, the instance is marked ERROR and needs to reset state then manually do the command "dmsetup remove -f <mapper-id>" before starting VM.
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server [req-166755c4-f6b9-48e8-a607-25c4b4787f45 183ef8f0f8244784b6805cb0ca8613e8 c881b9aed5964cc2973bd6a68a4191c6 - default default] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: multipath -f 3600a09803831486e695d536269665144
Exit code: 1
Stdout: ''
Stderr: 'Oct 04 10:49:32 | 3600a09803831486e695d536269665144: map in use\n'
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server _emit_exception_notification(
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server raise value
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 216, in decorated_function
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server raise value
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 205, in decorated_function
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 9155, in rollback_live_migration_at_destination
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.driver.rollback_live_migration_at_destination(
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 9677, in rollback_live_migration_at_destination
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.destroy(context, instance, network_info, block_device_info,
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1342, in destroy
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.cleanup(context, instance, network_info, block_device_info,
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1410, in cleanup
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return self._cleanup(
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1471, in _cleanup
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server LOG.warning(
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server raise value
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1463, in _cleanup
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self._disconnect_volume(context, connection_info, instance)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1735, in _disconnect_volume
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server vol_driver.disconnect_volume(connection_info, instance)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.connector.disconnect_volume(connection_info['data'], None)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/utils.py", line 144, in trace_logging_wrapper
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/connectors/iscsi.py", line 877, in disconnect_volume
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return self._cleanup_connection(connection_properties, force=force,
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/connectors/iscsi.py", line 927, in _cleanup_connection
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server multipath_name = self._linuxscsi.remove_connection(
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/linuxscsi.py", line 313, in remove_connection
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.flush_multipath_device(multipath_name)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/linuxscsi.py", line 369, in flush_multipath_device
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self._execute('multipath', '-f', device_map_name, run_as_root=True,
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/executor.py", line 52, in _execute
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return execute_root(*cmd, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 224, in remote_call
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server raise exc_type(*result[2])
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server Command: multipath -f 3600a09803831486e695d536269665144
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server Exit code: 1
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server Stdout: ''
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server Stderr: 'Oct 04 10:49:32 | 3600a09803831486e695d536269665144: map in use\n'
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server
