OpenStack Compute (nova)

Virtual Machine could not be evacuated because virtual interface creation failed

Bug #1430042 reported by Matt Rabe
36
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Chris Friesen
OpenStack Compute (nova) 12.0.0 "liberty"
Kilo
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2015.1.1
Ubuntu Cloud Archive
Invalid
Undecided
Unassigned
Kilo
Fix Released
Undecided
Unassigned
Liberty
Fix Released
Undecided
Unassigned
Mitaka
Fix Released
Undecided
Unassigned
Ubuntu
Invalid
Undecided
Unassigned
Trusty
New
Medium
Unassigned

Bug Description

I believe this issue is related to Question 257358 (https://answers.launchpad.net/ubuntu/+source/nova/+question/257358).

On the source host we see the successful vif plug:

2015-03-09 01:22:12.363 629 DEBUG neutron.plugins.ml2.rpc [req-5de70341-d64b-4a3a-bc05-54eb2802f25d None] Device 14ac5edd-269f-4808-9a34-c4cc93e9ab70 up at agent ovs-agent-ipxxxxxxxxx update_device_up /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:156
2015-03-09 01:22:12.392 629 DEBUG oslo_concurrency.lockutils [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] Acquired semaphore "db-access" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:377
2015-03-09 01:22:12.436 629 DEBUG oslo_concurrency.lockutils [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] Releasing semaphore "db-access" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:390
2015-03-09 01:22:12.437 629 DEBUG oslo_messaging._drivers.amqp [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] UNIQUE_ID is 740634ca8c7a49418a39c429669f2f27. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:224
2015-03-09 01:22:12.439 629 DEBUG oslo_messaging._drivers.amqp [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] UNIQUE_ID is 3264e8d7dd7c492d9aa17d3e9892b1fc. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:224
2015-03-09 01:22:14.436 629 DEBUG neutron.notifiers.nova [-] Sending events: [{'status': 'completed', 'tag': u'14ac5edd-269f-4808-9a34-c4cc93e9ab70', 'name': 'network-vif-plugged', 'server_uuid': u'2790be4a-5285-46aa-8ee2-c68f5b936c1d'}] send_events /usr/lib/python2.7/site-packages/neutron/notifiers/nova.py:237

Later, the destination host of the evacuation attempts to plug the vif but can't:

2015-03-09 02:15:41.441 629 DEBUG neutron.plugins.ml2.rpc [req-5ea6625c-a60c-48fb-9264-e2a5a3ed0d26 None] Device 14ac5edd-269f-4808-9a34-c4cc93e9ab70 up at agent ovs-agent-ipxxxxxxxxxx update_device_up /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:156
2015-03-09 02:15:41.485 629 DEBUG neutron.plugins.ml2.rpc [req-5ea6625c-a60c-48fb-9264-e2a5a3ed0d26 None] Device 14ac5edd-269f-4808-9a34-c4cc93e9ab70 not bound to the agent host ipxxxxxxxxx update_device_up /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:163

The cause of the problem seems to be that the neutron port does not have is binding:host_id properly updated on evacuation, the answer to question 257358 looks like the fix.

Revision history for this message
Sean Dague (sdague) wrote :

What version of OpenStack? what do your nova.conf and neutron.conf look like?

It would be good to know if this was already addressed in kilo, or remains a real issue in master.

Changed in nova:
status: New → Incomplete
Revision history for this message
Matt Rabe (mdrabe) wrote :

This is in kilo using the neutronv2 API (nova/network/neutronv2/api.py)

Sean Dague (sdague)
tags: added: neutron
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/169827

Changed in nova:
assignee: nobody → Sean Dague (sdague)
status: Incomplete → In Progress
Sean Dague (sdague)
Changed in nova:
importance: Undecided → Medium
Chris Friesen (cbf123)
tags: added: kilo-backport-potential
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Sean Dague (sdague) → Chris Friesen (cbf123)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/169827
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7c81be3c2952f26e30bb8394c8e2803d31fc3b26
Submitter: Jenkins
Branch: master

commit 7c81be3c2952f26e30bb8394c8e2803d31fc3b26
Author: Chris Friesen <email address hidden>
Date: Wed Apr 1 12:07:41 2015 -0400

    fix network setup on evacuate

    As reported in bug #1430042 when using evacuate + neutron virtual
    machines fail to evacuate because of a failure to update port
    bindings. A similar issue applies to FloatingIP setup for
    nova-network.

    A related fix was proposed in
    https://answers.launchpad.net/ubuntu/+source/nova/+question/257358
    and seemed to be fixing users.

    This handles things a bit differently in order to more cleanly
    use existing code.

    Change-Id: Iafa6d2e9eb76ea0389067c939f21ac2181df6f95
    Closes-Bug: #1430042

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/184282

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/kilo)

Change abandoned by Matt Rabe (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/184282
Reason: Wrong patch set

Revision history for this message
Matt Rabe (mdrabe) wrote :

I patched the fix into my environment and verified. The neutron port's binding:host_id was updated with successful evacuation. Thank you Chris

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/kilo)

Reviewed: https://review.openstack.org/184282
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1a6053b155ff7d76de271096df0f5660ed09f4d8
Submitter: Jenkins
Branch: stable/kilo

commit 1a6053b155ff7d76de271096df0f5660ed09f4d8
Author: Chris Friesen <email address hidden>
Date: Wed Apr 1 12:07:41 2015 -0400

    fix network setup on evacuate

    As reported in bug #1430042 when using evacuate + neutron virtual
    machines fail to evacuate because of a failure to update port
    bindings. A similar issue applies to FloatingIP setup for
    nova-network.

    A related fix was proposed in
    https://answers.launchpad.net/ubuntu/+source/nova/+question/257358
    and seemed to be fixing users.

    This handles things a bit differently in order to more cleanly
    use existing code.

    Change-Id: Iafa6d2e9eb76ea0389067c939f21ac2181df6f95
    Closes-Bug: #1430042
    (cherry picked from commit 7c81be3c2952f26e30bb8394c8e2803d31fc3b26)

tags: added: in-stable-kilo
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-1
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/213195

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/juno)

Change abandoned by Edward Hope-Morley (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/213195

Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-1 → 12.0.0
James Page (james-page)
Changed in cloud-archive:
status: New → Invalid
Revision history for this message
Corey Bryant (corey.bryant) wrote :

This appears to also exist in Trusty so marking that as triaged.

Changed in ubuntu:
status: New → Triaged
status: Triaged → Incomplete
status: Incomplete → Fix Released
status: Fix Released → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.