OpenStack Compute (nova)

Nova waits for events from neutron on resize-revert that aren't coming

Bug #1450624 reported by Dan Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Dan Smith
OpenStack Compute (nova) 12.0.0 "liberty"
Juno
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2014.2.4
Kilo
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2015.1.1

Bug Description

On resize-revert, the original host was waiting for plug events from neutron before restarting the instance. These aren't sent since we don't ever unplug the vifs. Thus, we'll always fail like this:

2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher payload)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 298, in decorated_function
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher pass
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 284, in decorated_function
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 348, in decorated_function
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 326, in decorated_function
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 314, in decorated_function
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 3573, in finish_revert_resize
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher block_device_info, power_on)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6095, in finish_revert_migration
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher block_device_info, power_on)
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher File "/opt/bbc/openstack-10.0-bbc40/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4446, in _create_domain_and_network
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher raise exception.VirtualInterfaceCreateException()
2015-04-30 19:45:42.602 23513 TRACE oslo.messaging.rpc.dispatcher VirtualInterfaceCreateException: Virtual Interface creation failed

Revision history for this message
Matt Riedemann (mriedem) wrote :

https://review.openstack.org/#/c/179228/

tags: added: juno-backport-potential kilo-backport-potential libvirt neutron
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/179232

Revision history for this message
Matt Riedemann (mriedem) wrote :

Bug 1380934 looks related.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/179234

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/179228
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4814e9126ec19a1edfa9c696138c4b3fec71aa0d
Submitter: Jenkins
Branch: master

commit 4814e9126ec19a1edfa9c696138c4b3fec71aa0d
Author: Dan Smith <email address hidden>
Date: Thu Apr 30 13:23:07 2015 -0700

    Don't wait for an event on a resize-revert

    Since we never unplugged, the event ain't comin'.

    Sadly, the test that covers this was pretty wrong on multiple fronts,
    which this patch fixes up.

    Closes-Bug: #1450624
    Change-Id: Id515137747a4b76e9b7057c95f80c8ae74017519

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/juno)

Reviewed: https://review.openstack.org/179234
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0f7fd924fdfbdcb8cd3155d385a675d1e19b1c95
Submitter: Jenkins
Branch: stable/juno

commit 0f7fd924fdfbdcb8cd3155d385a675d1e19b1c95
Author: Dan Smith <email address hidden>
Date: Thu Apr 30 13:23:07 2015 -0700

    Don't wait for an event on a resize-revert

    Since we never unplugged, the event ain't comin'.

    Sadly, the test that covers this was pretty wrong on multiple fronts,
    which this patch fixes up.

    Conflicts:
     nova/tests/unit/virt/libvirt/test_driver.py
     nova/virt/libvirt/driver.py

    Closes-Bug: #1450624
    Change-Id: Id515137747a4b76e9b7057c95f80c8ae74017519
    (cherry picked from commit 4814e9126ec19a1edfa9c696138c4b3fec71aa0d)

tags: added: in-stable-juno
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/kilo)

Reviewed: https://review.openstack.org/179232
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6463b38c0b1d330000a17b6abb074570b5061b8e
Submitter: Jenkins
Branch: stable/kilo

commit 6463b38c0b1d330000a17b6abb074570b5061b8e
Author: Dan Smith <email address hidden>
Date: Thu Apr 30 13:23:07 2015 -0700

    Don't wait for an event on a resize-revert

    Since we never unplugged, the event ain't comin'.

    Sadly, the test that covers this was pretty wrong on multiple fronts,
    which this patch fixes up.

    Closes-Bug: #1450624
    Change-Id: Id515137747a4b76e9b7057c95f80c8ae74017519
    (cherry picked from commit 4814e9126ec19a1edfa9c696138c4b3fec71aa0d)

tags: added: in-stable-kilo
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-1 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.