OpenStack Compute (nova)

Changed SAN iscsi IP addresses in connection_info can prevent VM startup

Bug #1581367 reported by Yafei Yu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Low
Lee Yarwood

Bug Description

The iscsi IP addresses of SAN are stored in nova's block-device-mapping table after connecting volume and are never re-validated down the line. Changing the iscsi IP adresses of SAN will prevent the instance from booting as the stale connection info will enter the instance's XML. We should check weather the iscsi IP addresses of SAN are changed before startup VM, or we need to a way to restore those VMs whose iscsi IP addresses of SAN are changed.

See original description
Tags: volumes
Yafei Yu (yu-yafei)
description: updated
Revision history for this message
Matt Riedemann (mriedem) wrote :

There is a similar bug for when the ceph cluster IP changes, and a patch proposed here:

https://review.openstack.org/#/c/289813/

The patch has some issues though and looks more or less abandoned.

We don't really have a good solution for this kind of issue right now for when things change on the backing storage underneath nova.

What version of Nova are you hitting this on?

Do you have a stacktrace of the failure?

tags: added: volumes
Changed in nova:
importance: Undecided → Low
status: New → Confirmed
Revision history for this message
Yafei Yu (yu-yafei) wrote :
Download full text (8.9 KiB)

The nova version is kilo, and the bug is still existed in liberty and mitaka.

Steps:
1. Create a volume from image and storage is SAN.
2. Boot a instance from volume.
3. Modify the iscsi IP addresses of SAN.
4. Hard reboot the instance.

The stacktrace of the failure:
2016-05-16 17:42:09.328 26645 ERROR oslo_messaging.rpc.dispatcher [req-46ecfc98-97d4-40f9-9e40-e01e14fe4790 5650304261d74dbe8a4f7848661f95a6 55bbe36a87af48c0af04c9204a49a854 - - -] Exception during message handling: iSCSI device not found at [u'/dev/disk/by-path/ip-172.10.150.52:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:51:85-lun-9', u'/dev/disk/by-path/ip-172.10.151.52:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:51:85-lun-9', u'/dev/disk/by-path/ip-172.10.152.52:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:51:85-lun-9', u'/dev/disk/by-path/ip-172.10.153.52:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:51:85-lun-9', u'/dev/disk/by-path/ip-172.10.150.53:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:52:03-lun-9', u'/dev/disk/by-path/ip-172.10.151.53:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:52:03-lun-9', u'/dev/disk/by-path/ip-172.10.152.53:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:52:03-lun-9', u'/dev/disk/by-path/ip-172.10.153.53:3260-iscsi-iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:52:03-lun-9']
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8761, in reboot_instance
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher reboot_type)
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher payload)
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2016-05-16 17:42:09.328 26645 TRACE oslo_messaging.rpc...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/441269

Changed in nova:
assignee: nobody → Lee Yarwood (lyarwood)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/441269
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Discussed this at the queens ptg, notes are here: https://etherpad.openstack.org/p/cinder-ptg-queens

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.