Nova-compute fit instance to numa nodes not optimal resulting instance creation failure
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
In Progress
|
Undecided
|
Ilya Popov |
Bug Description
Description
===========
Reproduced in ussuri, master has the same code.
When nova compute start to fit instance's NUMA topology on host's NUMA topology it uses host cells list. This list contains cell objects from cell 0 up to cell N always sorted from cell id 0 up to cell id N (N number depends on host numa node number). The only case when sort order of this list is changed is the case with instance without pci device requirement. If instance doesn't need pci specific to NUMA node, host cells list is reordered to place cells with PCI capabilities to the end of list. If all NUMA cells have PCI capabilities, list order won't changed.
This behaviour leads to attempt to place instance's first NUMA node to host NUMA node id 0 at the beginning.
If we will use huge pages and place several instances with number of NUMA nodes less when Host NUMA node number, we exhaust completely NUMA node id 0. Which will lead to instances with larger number of NUMA nodes failed to fit on this host (for example instance with NUMA nodes number equal to host NUMA node number).
To mitigate this issue, it will be better to take into account NUMA node memory usage.
May be related also to:
https:/
https:/
https:/
Steps to reproduce
==================
1. Configure OpenStack to use 2MB huge pages, allocate huge pages on compute host(let say compute 1) during boot
For ussuri it is described here: https:/
2. Prepare two flavors to test issue: one flavor with hw:mem_
second flavor with hw:mem_
Flavor's RAM should be large enough to exhaust compute NUMA node 0 RAM with some small number of instances. Lets say with 6 instances of flavor 1 we exhaust compute NUMA node 0 RAM. Flavor 2 RAM should be equal Flavor 1 RAM multiplied by N (number of numa nodes on compute 1.)
3. Start 6 instances with first flavor (with 1 NUMA node defined) on compute 1 (with availability zone hint pointed to compute 1). RAM of NUMA node 0 on host compute 1 will be exhausted
4. Try to start instance with second flavor. Instance will not been started with error "...was re-scheduled: Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology"
How it should work:
===================
We should take into account memory usage of NUMA nodes to reduce number of this kind of error. So it is needed to use first NUMA nodes with more free RAM available.
description: | updated |
description: | updated |
description: | updated |
Changed in nova: | |
assignee: | nobody → Ilya Popov (ilya-p) |
Fix proposed to branch: master /review. opendev. org/c/openstack /nova/+ /805649
Review: https:/