Cannot boot instances from volumes in iSCSI multiple paths case

Bug #1423412 reported by Seiji Aguchi
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Low
Unassigned

Bug Description

* Precondition:
- Nova: Add "iscsi_use_multipath = True" in [libvirt] section of nova.conf.
- Cinder: use iSCSI storage backend that supports multi-path.
  Nova-compute nodes and iSCSI ports of the storage array are connected to 2 LAN segments for mutli-pathing.
- One of the iSCSI paths is not active (for example, the cable is bad).

* How to reproduce:
Users create volume boot instance.
  nova boot --flavor m1.heat --boot-volume 24abb70b-c0ef-46c8-b734-6bc230fff1a7 test

* Actual results:
The instance fails to boot.
  nova list
  | 8df39f1b-d09d-42bf-8bf9-667c91a3ab34 | test | ERROR | - | NOSTATE | private=10.0.0.14 |

Error log in nova-compute is attached below.
It tries to issue "iscsiadm -m discovery -t sendtargets" command only on 1 path and gives up with ProcessExecutionError.

* Expected results:
The instance successfully boots using another iSCSI path.
It is desirable that users can use the cloud system normally, even while the administrators is resolving the path failure.

* Nova-compute Log:

2015-02-12 15:43:57.439 DEBUG oslo_concurrency.processutils [-] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p 192.168.1.2:3260" returned: 4 in 95 pid=9253) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:216
2015-02-12 15:43:57.439 DEBUG oslo_concurrency.processutils [-] u'sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p 192.168.1.2:3260' failed. Not Retrying) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:249
2015-02-12 15:43:57.439 DEBUG oslo_concurrency.lockutils [-] Lock "connect_volume" released by "connect_volume" :: held 95.187s from (pid=9253) inner /usr/lib/python2.7/site-packages/oslo_lockutils.py:442
2015-02-12 15:43:57.439 ERROR nova.compute.manager [-] [instance: 98796b31-8750-469d-87b1-537489b6749e] Instance failed to spawn
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Traceback (most recent call last):
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/compute/manager.py", line 2318, in _build_resources
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] yield resources
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/compute/manager.py", line 2188, in _build_and_run_instance
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] flavor=flavor)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2353, in spawn
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] flavor=flavor)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4165, in _get_guest_xml
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] context, flavor=flavor)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4028, in _get_guest_config
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] flavor, guest.os_type)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3213, in _get_guest_storage_c
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] self._connect_volume(connection_info, info)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 977, in _connect_volume
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] driver.connect_volume(connection_info, disk_info)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 431, in in
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] return f(*args, **kwargs)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/virt/libvirt/volume.py", line 362, in connect_volume
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] out = self._run_iscsiadm_discover(iscsi_properties)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/virt/libvirt/volume.py", line 498, in _run_iscsiadm_discove
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] check_exit_code=[0, 255])[0] or ""
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/virt/libvirt/volume.py", line 768, in _run_iscsiadm_bare
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] check_exit_code=check_exit_code)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/opt/stack/nova/nova/utils.py", line 197, in execute
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] return processutils.execute(*cmd, **kwargs)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] cmd=sanitized_cmd)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] ProcessExecutionError: Unexpected error while running command.
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p 192.168.1.2:3260
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Exit code: 4
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Stdout: u''
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Stderr: u'iscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm login retries (reopen_max) 5 exceeded\niscsiadm: Could not perform SendTargets discovery: encountered connection failure\n'
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]

Changed in nova:
status: New → Confirmed
importance: Undecided → Low
tags: added: libvirt
tags: added: volumes
ugvddm (271025598-9)
Changed in nova:
assignee: nobody → ugvddm (271025598-9)
Revision history for this message
ugvddm (271025598-9) wrote :

Hi Seiji, cloud you tell me which version openstack you use?

Revision history for this message
Sujitha (sujitha-neti) wrote :

There are no open reviews for this bug report since a long time.
To signal that to other contributors which might provide patches for
this bug, I'm removing the assignee.

Feel free to add yourself as assignee and push a review for it.

Changed in nova:
assignee: ugvddm (271025598-9) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.