ComputeManager._allocate_network should not call _update_resource_tracker

Bug #1732316 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Matt Riedemann
Ocata
Fix Committed
Medium
Matt Riedemann
Pike
Fix Committed
Medium
Matt Riedemann

Bug Description

The _allocate_network method in the ComputeManager is calling _update_resource_tracker but there is no reason to, nothing in that method has changed the instance in a way that will affect how the ResourceTracker manages that instance. This can be costly when you're building several instances at the same time on the same compute because the call to the RT eventually holds a lock which the other instances being built have to contend for.

This is already fixed for Queens with change: https://review.openstack.org/#/c/513473/

This bug is for tracking it for backports.

With some internal testing, creating 70 instances on the same compute at the same time, we see about a 22% increase in throughput server create times after this change.

Tags: compute
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/519942

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/519943

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/519942
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ed11484b696e4af6f226029e88afe39277ea8e5e
Submitter: Zuul
Branch: stable/pike

commit ed11484b696e4af6f226029e88afe39277ea8e5e
Author: Matt Riedemann <email address hidden>
Date: Thu Oct 19 14:36:29 2017 -0400

    Don't update RT in _allocate_network

    The call to _update_resource_tracker from
    _allocate_network is a carryover from old
    times when _allocate_network would call
    _instance_update which would update the
    resource tracker generically for any state
    change in an instance.

    _update_resource_tracker calls rt.update_usage
    which grabs the COMPUTE_RESOURCE_SEMAPHORE lock
    which is the same lock used during claims and when
    the update_available_resource periodic task runs.

    The update_usage method can be a bit heavy weight
    if you're not actually changing anything the
    resource tracker cares about, because it's re-calculating
    stats for the capabilities scheduler filter and it's
    collecting new inventory from the virt driver and
    potentially updating that inventory for the compute
    node in placement.

    So given all _allocate_network is doing is changing
    the state on the instance, and the state it's changing
    to has nothing to do with what the resource tracker
    cares about (like deleting or shelve offloading an instance),
    we shouldn't call rt.update_usage and hold that big
    lock unnecessarily since it could hold up other
    operations happening at the same time, like creating
    instances.

    Closes-Bug: #1732316

    Change-Id: Ib588c31a4d2075f8730409d50c99dfb04180a9cd
    (cherry picked from commit 7b0d5f7f347cb1c040595000d9d6987369dce2ab)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.1.0

This issue was fixed in the openstack/nova 16.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/519943
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b9c26ad1717de62b3dc7b5211145091eb21e9b17
Submitter: Zuul
Branch: stable/ocata

commit b9c26ad1717de62b3dc7b5211145091eb21e9b17
Author: Matt Riedemann <email address hidden>
Date: Thu Oct 19 14:36:29 2017 -0400

    Don't update RT in _allocate_network

    The call to _update_resource_tracker from
    _allocate_network is a carryover from old
    times when _allocate_network would call
    _instance_update which would update the
    resource tracker generically for any state
    change in an instance.

    _update_resource_tracker calls rt.update_usage
    which grabs the COMPUTE_RESOURCE_SEMAPHORE lock
    which is the same lock used during claims and when
    the update_available_resource periodic task runs.

    The update_usage method can be a bit heavy weight
    if you're not actually changing anything the
    resource tracker cares about, because it's re-calculating
    stats for the capabilities scheduler filter and it's
    collecting new inventory from the virt driver and
    potentially updating that inventory for the compute
    node in placement.

    So given all _allocate_network is doing is changing
    the state on the instance, and the state it's changing
    to has nothing to do with what the resource tracker
    cares about (like deleting or shelve offloading an instance),
    we shouldn't call rt.update_usage and hold that big
    lock unnecessarily since it could hold up other
    operations happening at the same time, like creating
    instances.

    Closes-Bug: #1732316

    Change-Id: Ib588c31a4d2075f8730409d50c99dfb04180a9cd
    (cherry picked from commit 7b0d5f7f347cb1c040595000d9d6987369dce2ab)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.1.1

This issue was fixed in the openstack/nova 15.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.