"IndexError: list index out of range" in ExceptionPayload.from_exception during resize failure

Bug #1777540 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

I noticed this from the nova-compute logs attached to bug 1777157:

2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [req-69050681-be73-48c5-be25-993f05b0b47e e511323a68a740f597880a1b056e3918 28f4e0797d544d758700e5be42aa82c6 - default default] [instance: daa5b10b-0655-4178-9413-66697004d7f8] Setting instance vm_state to ERROR: IndexError: list index out of range
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] Traceback (most recent call last):
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 7445, in _error_out_instance_on_exception
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] yield
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 4162, in prep_resize
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] filter_properties, host_list)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 4221, in _reschedule_resize_or_reraise
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] exception=exc_info[1])
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] return f(*args, **kwargs)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 367, in notify_about_instance_action
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] fault, priority = _get_fault_and_priority_from_exc(exception)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 348, in _get_fault_and_priority_from_exc
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] exception)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/notifications/objects/exception.py", line 42, in from_exception
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] trace = inspect.trace()[-1]
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] IndexError: list index out of range
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]

This is based on queens, and this is the problem:

https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/compute/manager.py#L4216

When we call that, we're not in an except block, so this fails later:

https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/notifications/objects/exception.py#L42

Although, looking higher up the stack, we are in an except block here:

https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/compute/manager.py#L4155

So this is a bit weird...

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

I agree, it feels strange. Let me try to reproduce it with a functional test.

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

I failed to reproduce so far. The failed trial is up in https://review.openstack.org/#/c/576593

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

The bug reported provided more information in the original bug report and from it seems [1] that the nova code is heavily diverged from upstream. Based on [1], the fact that I failed to reproduce the problem and that the reporter set the original bug to Invalid state I set this to invalid as well.

[1] https://bugs.launchpad.net/nova/+bug/1777157/comments/16

Changed in nova:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.