When attempting to do a live-migration on an instance with one or more attached volumes, Nova expects that the IQN will be exactly the same as it's attaching the volume(s) to the new host. This conflicts with the Cinder settings such as "hp3par_iscsi_ips" which allows for multiple IPs for the purpose of load balancing.
Example:
An instance on Host A has a volume attached at "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
An attempt is made to migrate the instance to Host B.
Cinder sends the request to attach the volume to the new host.
Cinder gives the new host "/dev/disk/by-path/ip-10.10.120.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
Nova looks for the volume on the new host at the old location "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
The following error appears in n-cpu in this case:
2015-02-19 17:09:05.574 ERROR nova.virt.libvirt.driver [-] [instance: b6fa616f-4e78-42b1-a747-9d081a4701df] Live Migration failure: Failed to open file '/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2': No such file or directory
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 115, in wait
listener.cb(fileno)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
result = function(*args, **kwargs)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5426, in _live_migration
recover_method(context, instance, dest, block_migration)
File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5393, in _live_migration
CONF.libvirt.live_migration_bandwidth)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
six.reraise(c, e, tb)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
rv = meth(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1582, in migrateToURI2
if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: Failed to open file '/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2': No such file or directory
Removing descriptor: 3
When looking at the nova DB, this is the state of block_device_mapping prior to the migration attempt:
mysql> select * from block_device_mapping where instance_uuid='b6fa616f-4e78-42b1-a747-9d081a4701df' and deleted=0;
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| created_at | updated_at | deleted_at | id | device_name | delete_on_termination | snapshot_id | volume_id | volume_size | no_device | connection_info | instance_uuid | deleted | source_type | destination_type | guest_format | device_type | disk_bus | boot_index | image_id |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| 2015-02-20 00:57:01 | 2015-02-20 01:03:06 | NULL | 3 | /dev/vda | 0 | NULL | e031b804-f824-45f1-a9fa-b9330ba061e0 | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "e031b804-f824-45f1-a9fa-b9330ba061e0", "data": {"host_device": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-1", "target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 1, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | disk | virtio | 0 | NULL |
| 2015-02-20 01:07:11 | 2015-02-20 01:07:19 | NULL | 5 | /dev/vdb | 0 | NULL | c3009a3d-549d-4ee5-b7a6-b0eac6382beb | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "c3009a3d-549d-4ee5-b7a6-b0eac6382beb", "data": {"device_path": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2", "host_device": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2", "target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 2, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | NULL | NULL | NULL | NULL |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
2 rows in set (0.00 sec)
Then after the error message comes up in n-cpu:
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| created_at | updated_at | deleted_at | id | device_name | delete_on_termination | snapshot_id | volume_id | volume_size | no_device | connection_info | instance_uuid | deleted | source_type | destination_type | guest_format | device_type | disk_bus | boot_index | image_id |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| 2015-02-20 00:57:01 | 2015-02-20 01:08:55 | NULL | 3 | /dev/vda | 0 | NULL | e031b804-f824-45f1-a9fa-b9330ba061e0 | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "e031b804-f824-45f1-a9fa-b9330ba061e0", "data": {"target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 0, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | disk | virtio | 0 | NULL |
| 2015-02-20 01:07:11 | 2015-02-20 01:08:57 | NULL | 5 | /dev/vdb | 0 | NULL | c3009a3d-549d-4ee5-b7a6-b0eac6382beb | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "c3009a3d-549d-4ee5-b7a6-b0eac6382beb", "data": {"target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 1, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | NULL | NULL | NULL | NULL |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
2 rows in set (0.00 sec)
I was able to reproduce this in Kilo (the original discovery was in Juno), with the same error:
2015-02-20 16:12:52.802 ERROR nova.virt. libvirt. driver [-] [instance: 93ba0373- 15cd-4e83- 845d-4cfaf7c114 16] Live Migration failure: Failed to open file '/dev/disk/ by-path/ ip-10.10. 220.244: 3260-iscsi- iqn.2000- 05.com. 3pardata: 22210002ac002a1 3-lun-1' : No such file or directory virt/libvirt/ driver. py", line 5249, in _live_migration\n CONF.libvirt. live_migration_ bandwidth) \n', ' File "/usr/local/lib /python2. 7/dist- packages/ eventlet/ tpool.py" , line 183, in doit\n result = proxy_call( self._autowrap, f, *args, **kwargs)\n', ' File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 141, in proxy_call\n rv = execute(f, *args, **kwargs)\n', ' File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 122, in execute\n six.reraise(c, e, tb)\n', ' File "/usr/local/lib /python2. 7/dist- packages/ eventlet/ tpool.py" , line 80, in tworker\n rv = meth(*args, **kwargs)\n', ' File "/usr/lib/ python2. 7/dist- p ackages/ libvirt. py", line 1582, in migrateToURI2\n if ret == -1: raise libvirtError (\'virDomainMig rateToURI2( ) failed\', dom=self)\ n', "libvirtError: Failed to open file '/dev/disk/ by-path/ ip-10.10. 220.244: 3260-iscsi- iqn.2000- 05.com. 3pardata: 22210002ac002a1 3-lun-1' : No such file or directory\n"] lib/python2. 7/dist- packages/ eventlet/ hubs/poll. py", line 115, in wait cb(fileno) lib/python2. 7/dist- packages/ eventlet/ greenthread. py", line 214, in main nova/nova/ virt/libvirt/ driver. py", line 5282, in _live_migration method( context, instance, dest, block_migration) nova/nova/ exception. py", line 88, in wrapped lib/python2. 7/dist- packages/ oslo_utils/ excutils. py", line 82, in __exit__ reraise( self.type_ , self.value, self.tb) nova/nova/ exception. py", line 71, in wrapped nova/nova/ compute/ manager. py", line 324, in decorated_function 'instance' ], e, sys.exc_info()) lib/python2. 7/dist- packages/ oslo_utils/ excutils. py", line 82, in __exit__ reraise( self.type_ , self.value, self.tb) nova/nova/ compute/ manager. py", line 312, in decorated_function nova/nova/ compute/ manager. py", line 5297, in _rollback_ live_migration nova/nova/ compute/ rpcapi. py", line 677, in remove_ volume_ connection instance, volume_ id=volume_ id) lib/python2. 7/dist- packages/ oslo_messaging/ rpc/client. py", line 156, in call self.retry) lib/python2. 7/dist- packages/ oslo_messaging/ transport. py", line 90, in _send
2015-02-20 16:12:54.381 ERROR root [-] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/opt/stack/n ova/nova/
Traceback (most recent call last):
File "/usr/local/
listener.
File "/usr/local/
result = function(*args, **kwargs)
File "/opt/stack/
recover_
File "/opt/stack/
payload)
File "/usr/local/
six.
File "/opt/stack/
return f(self, context, *args, **kw)
File "/opt/stack/
kwargs[
File "/usr/local/
six.
File "/opt/stack/
return function(self, context, *args, **kwargs)
File "/opt/stack/
context, instance, bdm.volume_id, dest)
File "/opt/stack/
instance=
File "/usr/local/
retry=
File "/usr/local/
timeout=...