OpenStack Compute (nova)

NUMATopologyFilter filters out a compute from candidates for booting a VM with 'hw:cpu_thread_policy=require' unless cores from all NUMA nodes are enabled in vcpu_pin_set on the compute

Series newton
Bug #1596692

Bug #1596692 reported by Mikhail Chernik on 2016-06-27

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Medium	Mikhail Chernik
	Newton	Fix Committed	Medium	Vladik Romanovsky

Bug Description

The 'require' CPU thread policy allows booting instances on hardware threads siblings only. However, if a compute host has several NUMA nodes and at least one NUMA node does not contribute to a list of pinned CPU cores, NUMATopologyFilter excludes such node from possible targets to an instance.

Steps to reproduce:

1) On a compute with several NUMA nodes vcpu_pin_set key of nova.conf does not contain sibling thread cores from one NUMA node (e.g. they are allocated for OVS with DPDK support)

ex: vcpu_pin_set=0,1,20,21

List of cores on a NUMA node can be obtained with lstopo or numactl -H command:
# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 20 21 22 23 24
node 0 size: 64328 MB
node 0 free: 21952 MB
node 1 cpus: 5 6 7 8 9 25 26 27 28 29
node 1 size: 64510 MB
node 1 free: 30091 MB

Sibling cores can be found with # cat /sys/bus/cpu/devices/cpu*/topology/thread_siblings_list command

# cat /sys/bus/cpu/devices/cpu{0,1}/topology/thread_siblings_list
0,20
1,21

2) Create corresponding flavor

# openstack flavor create pinned.threads_require --ram 2048 --disk 0 --vcpus 2
# openstack flavor set pinned.threads_require --property "hw:cpu_thread_policy=require" --property "hw:cpu_policy=dedicated"

3) Boot an instance with created flavor
# openstack server create --flavor pinned.threads_require --image cirros vm1

4) List instances:
# openstack server list

Expected result:
Instance vm1 is booted

Actiual result:
Instance vm1 is in ERROR state with message 'No valid host was found. There are not enough hosts available.' (assuming there are no other computes with sibling thread cores available)

Tags:

Mikhail Chernik (mchernik) on 2016-06-27

Changed in nova:
status:	New → In Progress
assignee:	nobody → Mikhail Chernik (mchernik)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-27: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/334659

Roman Podoliaka (rpodolyaka) on 2016-08-03

tags:	added: mitaka-backport-potential
Changed in nova:
importance:	Undecided → Medium
tags:	added: numa

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-22: Fix merged to nova (master)

Reviewed: https://review.openstack.org/334659
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1c04e18ec922b6a6f6de6734fbceb0eed336f2ee
Submitter: Jenkins
Branch: master

commit 1c04e18ec922b6a6f6de6734fbceb0eed336f2ee
Author: Mikhail Chernik <email address hidden>
Date: Mon Jun 27 22:04:46 2016 +0200

Fix require thread policy for multi-NUMA computes

    CPU thread policy "require" demands that instances are created on hosts
    with hardware threads, e.g HyperThreading. It is possible that a compute
    has several NUMA nodes, but only some of them contribute to the list of
    pinned CPUs. In this case NUMATopologyFiler will exclude the compute
    from candidates even if hardware threads are enabled and cores from
    other NUMA nodes are available for CPU pinning.

    This commit allows a compute with enabled HyperThreading to pass
    NUMATopologyFilter if at least one NUMA node contributes to the list of
    cores allowed for CPU pinning.

Change-Id: I3235cc6bf1a2a9e29a17acc43472e2a6bd4c26ec
Closes-Bug: 1596692

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-23: Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/375196

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-10-10: Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/375196
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a4a834f3bf08710d7cc6e5e256bb8963e9725a03
Submitter: Jenkins
Branch: stable/newton

commit a4a834f3bf08710d7cc6e5e256bb8963e9725a03
Author: Mikhail Chernik <email address hidden>
Date: Mon Jun 27 22:04:46 2016 +0200

Fix require thread policy for multi-NUMA computes

    This commit allows a compute with enabled HyperThreading to pass
    NUMATopologyFilter if at least one NUMA node contributes to the list of
    cores allowed for CPU pinning.

    Change-Id: I3235cc6bf1a2a9e29a17acc43472e2a6bd4c26ec
    Closes-Bug: 1596692
    (cherry picked from commit 1c04e18ec922b6a6f6de6734fbceb0eed336f2ee)