Insufficient error handling when parsing iscsiadm -m node output with blank iscsi target

Bug #1886855 reported by Pushkar Acharya
30
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Committed
Medium
Unassigned
os-brick
Fix Committed
Medium
Unassigned

Bug Description

We encountered the following error when attempting to reboot a VM with multiple attached volumes -

2020-07-02 05:46:05.960 ERROR oslo_messaging.rpc.server [req-0c171deb-bc82-4687-91d6-76e8f95b8e19 **** service] Exception during message handling: IndexError: list index out of range
...
2020-07-02 05:46:05.960 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 157, in _get_iscsi_nodes
2020-07-02 05:46:05.960 TRACE oslo_messaging.rpc.server lines.append((info[0].split(',')[0], info[1]))
2020-07-02 05:46:05.960 TRACE oslo_messaging.rpc.server IndexError: list index out of range
This is observed on os-brick version - 1.15.9

The same code in current master branch -
https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L136

iscsiadm -m node output -
172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-f1ff35f1-9716-4929-831f-32e7b207c742
172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-5393e371-337f-4332-b39f-4926e4a1f9f7
172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-1520a7d6-4351-416a-a703-c82f1bc9839d
[]:3260,-1
172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-fd632af2-45d9-4266-be67-b84e61fb3cbb
172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-c1b325b9-7bd2-4d91-a3ef-295736e52eca
172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-6a1a112e-1140-482d-9064-fe1b03391f2b

The blank target causes an unhandled exception. A simple python code snippet to show the same -
>>> out = "172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-1520a7d6-4351-416a-a703-c82f1bc9839d\n[]:3260,-1\n172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-fd632af2-45d9-4266-be67-b84e61fb3cbb"
>>> lines = []
>>> out.splitlines()
['172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-1520a7d6-4351-416a-a703-c82f1bc9839d', '[]:3260,-1', '172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-fd632af2-45d9-4266-be67-b84e61fb3cbb']
>>> for line in out.splitlines():
... if line:
... info = line.split()
... lines.append((info[0].split(',')[0], info[1]))
...
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>

The blank iscsi target was most probably due to corruption of the iscsi data during discovery.

Using strace we could trace that blank target belongs to a OpenStack volume -
open("/var/lib/iscsi/nodes/iqn.2010-10.org.openstack:volume-d88869e6-d27b-4121-9bd1-d8c86ce9d7e1/172.30.0.193,3260", O_RDONLY) = 5

Expected:
Management of that single volume associated with blank iscsi target to be affected.

Observed:
None of the volumes and volume backed VMs can be managed on the affected host.

Tags: iscsiadm
Revision history for this message
Pooja Ghumre (pooja-9) wrote :

Is there a way to handle blank iscsi target files gracefully?

Revision history for this message
Luigi Toscano (ltoscano) wrote :

I suspect the gerrit integration didn't work, but this patch should address the issue:
https://review.opendev.org/c/openstack/os-brick/+/763599

Revision history for this message
Pooja Ghumre (pooja-9) wrote :

Thanks for the update, Luigi!

Will this fix be backported to the os-brick version used in Openstack Rocky release?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-brick 4.2.0

This issue was fixed in the openstack/os-brick 4.2.0 release.

Revision history for this message
Pooja Ghumre (pooja-9) wrote :

Cinder needs to consume the fixed version of os-brick in upper constraints for older releases.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-brick 4.0.2

This issue was fixed in the openstack/os-brick 4.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-brick 3.0.5

This issue was fixed in the openstack/os-brick 3.0.5 release.

Lee Yarwood (lyarwood)
no longer affects: nova
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-brick 2.10.6

This issue was fixed in the openstack/os-brick 2.10.6 release.

Changed in os-brick:
status: New → Fix Committed
importance: Undecided → Medium
Changed in cinder:
importance: Undecided → Medium
status: New → Fix Committed
tags: added: iscsiadm
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.