OpenStack Compute (nova)

live_migration_force_complete surprised error message on compute node in some situation

Bug #1556717 reported by Eli Qiao
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Low
Unassigned

Bug Description

On master branch

trigger live-migration-force-complete after invoke live-migration test3

ubuntu@lm-1:~/devstack$ nova live-migration-force-complete test3 64

on the source compute node, we can see follow error trace, but it doesn't effect running instance.

expect:

No error message come out.

actual:

2016-03-14 01:18:13.770 ERROR oslo_messaging.rpc.dispatcher [req-6d6ceb08-561b-4638-b813-0c8f8c6c4669 admin admin] Exception during message handling: Instance instance-0000002b could not be found.
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher incoming.message))
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher payload)
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 375, in decorated_function
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 5281, in live_migration_force_complete
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher self.driver.live_migration_force_complete(instance)
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6544, in live_migration_force_complete
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher self.pause(instance)
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2519, in pause
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher self._host.get_guest(instance).pause()
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/host.py", line 617, in get_guest
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher self.get_domain(instance))
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/host.py", line 607, in get_domain
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher return self._get_domain_by_name(instance.name)
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/host.py", line 654, in _get_domain_by_name
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher raise exception.InstanceNotFound(instance_id=instance_name)
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher InstanceNotFound: Instance instance-0000002b could not be found.
2016-03-14 01:18:13.770 TRACE oslo_messaging.rpc.dispatcher

Eli Qiao (taget-9)
Changed in nova:
assignee: nobody → Eli Qiao (taget-9)
tags: added: live-migration
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/292185

Matt Riedemann (mriedem)
tags: added: libvirt
Changed in nova:
status: New → In Progress
importance: Undecided → Low
Revision history for this message
Andrea Rosa (andrea-rosa-m) wrote :

I am not sure if this is a valid bug, if the instance has already migrated the failure of the action is a valid one and I think we should raise the exception reporting that the instance is not anymore on the source node.
That exception is useful even to catch any mistake made by an operator, for example putting a wrong instance id.
I'd suggest to discuss this bug to see if it is a valid one or if the behaviour is "by design"

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Eli Qiao (<email address hidden>) on branch: master
Review: https://review.openstack.org/292185

Revision history for this message
Anusha Unnam (anusha-unnam) wrote :

@Andrea Rosa @Eli Qiao
Can we mark this bug as invalid?

Revision history for this message
Eli Qiao (taget-9) wrote :

Change to New for further discussion.

Eli.

Changed in nova:
status: In Progress → New
Revision history for this message
Pushkar Umaranikar (pushkar-umaranikar) wrote :

Solving inconsistency: changing bug status from "New" to "Confirmed" as it has assignee.

Changed in nova:
status: New → Confirmed
Revision history for this message
Takashi Natsume (natsume-takashi) wrote :

A bug report with an assignee should be "In Progress" status.

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

Changed in nova:
status: In Progress → Confirmed
assignee: Eli Qiao (taget-9) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.