Instance affinity filters do not work in a heterogeneous cloud with Ironic computes

Bug #1606503 reported by Roman Podoliaka
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
Fix Committed
Medium
Roman Podoliaka
9.x
Fix Released
High
Roman Podoliaka

Bug Description

Upstream bug: https://bugs.launchpad.net/nova/+bug/1606496

Description
===========

In a heterogeneous cloud with both libvirt and ironic compute nodes instance affinity filters like DifferentHostFilter or SameHostFilter do not filter hosts out when scheduling a subsequent instance.

Steps to reproduce
==================

Make sure you have at least two libvirt compute nodes and one ironic node.

Make sure DifferentHostFilter and SameHostFilter are configured as nova-scheduler filters in nova.conf, filters scheduler is used.

1. Boot a libvirt instance A.
2. Check the host name of the compute node instance A is running on (nova show from an admin user).
3. Boot a libvirt instance B passing a different_host=$A.uuid hint for nova-scheduler.
4. Check the host name of the compute node instance B is running on (nova show from an admin user).

Expected result
===============

Instances A and B are running on two different compute nodes.

Actual result
=============

Instances A and B are running on the same compute node.

nova-scheduler logs shows that DifferentHost filter was run, but did not filter out one of the hosts: Filter DifferentHostFilter returned 2 host(s) get_filtered_objects

Environment
===========

MOS 9.1

2 libvirt compute nodes
1 ironic compute node
FiltersScheduler is used
DifferentHostFilter and SameHostFilter filters are enabled in nova.conf

Root cause analysis
===================

Debugging shown that IronicHostManager is configured to be used by nova-scheduler instead of the default host manager, when Ironic compute are deployed in the same cloud together with libvirt compute nodes.

IronicHostManager overrides the _get_instance_info() method and unconditionally returns an empty instance dict, even if this method is called for non-ironic computes of the same cloud. DifferentHostFilter and similar filters later use this info to find an intersection of a set of instances running on a libvirt compute node (currently, always {}) and a set of instances uuids passed as a hint for nova-scheduler, thus compute nodes are never filtered out and the hint is effectively ignored.

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

This should really be Medium, but this currently breaks a few Tempest tests which we run regularly and treat failures as High issues.

tags: added: area-nova
tags: added: tempest
Revision history for this message
Oleksiy Butenko (obutenko) wrote :
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :
Anna Babich (ababich)
tags: added: on-verification
Revision history for this message
Anna Babich (ababich) wrote :

Verified on:
ababich@srv148-bud:/$ grep SNAPSHOT /home/jenkins/env_inject.properties
SNAPSHOT_ID=#249

The bug was caught by running api.compute.admin.test_servers_on_multinodes Tempest tests on environment with Ironic nodes. At the moment, these tests passed against similar env: http://paste.openstack.org/show/572273/

tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.