OpenStack Compute (nova)

Random NUMA cell selection can leave NUMA cells unused

Bug #1461777 reported by Belmiro Moreira
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Nikola Đipanov
OpenStack Compute (nova) 12.0.0 "liberty"
Juno
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2014.2.4
Kilo
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2015.1.1

Bug Description

NUMA cell overcommit can leave NUMA cells unused

When no NUMA configuration is defined for the guest (no flavor extra specs),
nova identifies the NUMA topology of the host and tries to match the cpu
placement to a NUMA cell ("cpuset").

The cpuset is selected randomly.
pin_cpuset = random.choice(viable_cells_cpus) #nova/virt/libvirt/driver.py

However, this can lead to NUMA cells not being used.
This is particular noticeable when the flavor as the same number of vcpus
as the host NUMA cells and in the host CPUs are not overcommit (cpu_allocation_ratio = 1)

###
Particular use case:

Compute nodes with the NUMA topology:
<VirtNUMAHostTopology: {'cells': [{'mem': {'total': 12279, 'used': 0}, 'cpu_usage': 0, 'cpus': '0,1,2,3,8,9,10,11', 'id': 0}, {'mem': {'total': 12288, 'used': 0}, 'cpu_usage': 0, 'cpus': '4,5,6,7,12,13,14,15', 'id': 1}]}>

No CPU overcommit: cpu_allocation_ratio = 1
Boot instances using a flavor with 8 vcpus.
(No NUMA topology defined for the guest in the flavor)

In this particular case the host can have 2 instances. (no cpu overcommit)
Both instances can be allocated (random) with the same cpuset from the 2 options:
<vcpu placement='static' cpuset='4-7,12-15'>8</vcpu>
<vcpu placement='static' cpuset='0-3,8-11'>8</vcpu>

As consequence half of the host CPUs are not used.

###
How to reproduce:

Using: nova 2014.2.2
(not tested in trunk however the code path looks similar)

1. set cpu_allocation_ratio = 1
2. Identify the NUMA topology of the compute node
3. Using a flavor with a number of vcpus that matches a NUMA cell in the compute node,
boot instances until fill the compute node.
4. Check the cpu placement "cpuset" used by the each instance.

Notes:
- at this point instances can use the same "cpuset" leaving NUMA cells unused.
- the selection of the cpuset is random. Different tries may be needed.

Belmiro Moreira (moreira-belmiro-email-lists)
summary: - NUMA cell overcommit can leave NUMA cells unused
+ Random NUMA cell selection can leave NUMA cells unused
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188468

Changed in nova:
assignee: nobody → Nikola Đipanov (ndipanov)
status: New → In Progress
Nikola Đipanov (ndipanov)
Changed in nova:
importance: Undecided → Medium
milestone: none → liberty-1
Tony Breeds (o-tony)
tags: added: juno-backport-potential kilo-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/188468
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=eb253255d74cbe6e630225080c463f1eb16b9932
Submitter: Jenkins
Branch: master

commit eb253255d74cbe6e630225080c463f1eb16b9932
Author: Nikola Dipanov <email address hidden>
Date: Thu Jun 4 17:09:13 2015 +0100

    libvirt: Don't try to confine a non-NUMA instance

    Even though it seemed like a good idea at the time, it turns out that
    choosing a NUMA node at random can lead to placements that are
    suboptimal.

    Since the benefit of the current approach is unclear, and the downside
    is actually hurting users - it is best to remove it (we would be no
    worse off than before Nova was even NUMA aware), and think about
    optimizitaions going forward, if any.

    Change-Id: Idd4d5c236067fc40ed3a5049f3c1a58b75f9b5e5
    Closes-bug: #1461777

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/191594

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/194486

Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/kilo)

Reviewed: https://review.openstack.org/191594
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=41ba203ca0b526e91aea335bc3f3c17845ac6c91
Submitter: Jenkins
Branch: stable/kilo

commit 41ba203ca0b526e91aea335bc3f3c17845ac6c91
Author: Nikola Dipanov <email address hidden>
Date: Thu Jun 4 17:09:13 2015 +0100

    libvirt: Don't try to confine a non-NUMA instance

    Even though it seemed like a good idea at the time, it turns out that
    choosing a NUMA node at random can lead to placements that are
    suboptimal.

    Since the benefit of the current approach is unclear, and the downside
    is actually hurting users - it is best to remove it (we would be no
    worse off than before Nova was even NUMA aware), and think about
    optimizitaions going forward, if any.

    Change-Id: Idd4d5c236067fc40ed3a5049f3c1a58b75f9b5e5
    Closes-bug: #1461777
    (cherry picked from commit eb253255d74cbe6e630225080c463f1eb16b9932)

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/juno)

Reviewed: https://review.openstack.org/194486
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=624345b4962ca96dc0a009397bcee14b0e4fbf6b
Submitter: Jenkins
Branch: stable/juno

commit 624345b4962ca96dc0a009397bcee14b0e4fbf6b
Author: Nikola Dipanov <email address hidden>
Date: Thu Jun 4 17:09:13 2015 +0100

    libvirt: Don't try to confine a non-NUMA instance

    Even though it seemed like a good idea at the time, it turns out that
    choosing a NUMA node at random can lead to placements that are
    suboptimal.

    Since the benefit of the current approach is unclear, and the downside
    is actually hurting users - it is best to remove it (we would be no
    worse off than before Nova was even NUMA aware), and think about
    optimizitaions going forward, if any.

    Change-Id: Idd4d5c236067fc40ed3a5049f3c1a58b75f9b5e5
    Closes-bug: #1461777
    (cherry picked from commit eb253255d74cbe6e630225080c463f1eb16b9932)
    Conflicts:
     nova/tests/unit/virt/libvirt/test_driver.py
     nova/virt/libvirt/driver.py

Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-1 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.