compute resource providers not equal to compute nodes in deployment

Bug #1757207 reported by Hanif Kukkalli
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
iain MacDonnell
Pike
Fix Committed
Medium
s10
Queens
Fix Committed
Medium
s10
Rocky
Fix Committed
Medium
s10

Bug Description

Description
===========
When I execute the command `nova-status upgrade check`,
output:

nova-status upgrade check
+----------------------------------------------------------------------+
| Upgrade Check Results |
+----------------------------------------------------------------------+
| Check: Cells v2 |
| Result: Success |
| Details: None |
+----------------------------------------------------------------------+
| Check: Placement API |
| Result: Success |
| Details: None |
+----------------------------------------------------------------------+
| Check: Resource Providers |
| Result: Warning |
| Details: There are 4 compute resource providers and 15 compute nodes |
| in the deployment. Ideally the number of compute resource |
| providers should equal the number of enabled compute nodes |
| otherwise the cloud may be underutilized. See |
| http://docs.openstack.org/developer/nova/placement.html |
| for more details. |
+----------------------------------------------------------------------+

Steps to reproduce
==================
How to replicate this?
Remove the hosts from the openstack controller:
nova hypervisor-list
nova service-delete {id}

Then run:
su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova
The deleted compute node will be added again as a new node.
run:
nova-status upgrade check

Expected result
===============
No warning when you run:
nova-status upgrade check

Actual result
=============
You can find the warning.
This causes issue with placement of new VM's.
The compute host which was deleted and added again will not be considered during VM scheduling and placement.

Environment
===========
OpenStack Pike release
Neutron Networking which is default.

Logs and Configs
================
Config as the Openstack documentation.

Tags: placement
tags: added: neutron
tags: added: nova-manage
tags: removed: neutron
description: updated
description: updated
Revision history for this message
Matt Riedemann (mriedem) wrote :

> su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova
The deleted compute node will be added again as a new node.

That doesn't actually have anything to do with the compute node records getting re-created after you delete the service records.

Are you actually stopping the nova-compute service before you delete the service records via the API? Because if not, there is a periodic task in the nova-compute service that will automatically re-create the compute_nodes records in the database.

See the note in the API reference:

https://developer.openstack.org/api-ref/compute/#delete-compute-service

Changed in nova:
status: New → Incomplete
Revision history for this message
Matt Riedemann (mriedem) wrote :
tags: removed: nova-manage
Revision history for this message
Hanif Kukkalli (kukkalli) wrote :

Hi Matt,

I had stopped the nova-compute service.
The issue is with the Placement API which created duplicates of the records.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

Changed in nova:
status: Incomplete → Expired
Revision history for this message
iain MacDonnell (imacdonn) wrote :

Shouldn't the upgrade check simply discount deleted compute_nodes ?

+-----------------------------------------------------------------------+
| Check: Resource Providers |
| Result: Warning |
| Details: There are 34 compute resource providers and 40 compute nodes |
| in the deployment. Ideally the number of compute resource |
| providers should equal the number of enabled compute nodes |
| otherwise the cloud may be underutilized. See |
| https://docs.openstack.org/nova/latest/user/placement.html |
| for more details. |
+-----------------------------------------------------------------------+

mysql> select count(*) from compute_nodes;
+----------+
| count(*) |
+----------+
| 40 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from compute_nodes where deleted=0;
+----------+
| count(*) |
+----------+
| 34 |
+----------+
1 row in set (0.00 sec)

Changed in nova:
status: Expired → Incomplete
Revision history for this message
Chris Dent (cdent) wrote :

It appears that not accounting for deleted is discussed here: https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/cmd/status.py#L114-L128

It does seem a bit weird.

Revision history for this message
iain MacDonnell (imacdonn) wrote :

"disabled" is one thing - you can come back from that - but "deleted" is a one-way ticket ... no?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/604495

Changed in nova:
assignee: nobody → iain MacDonnell (imacdonn)
status: Incomplete → In Progress
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/604495
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3e902313e9c3117e76d6adf88a2f104bc229374c
Submitter: Zuul
Branch: master

commit 3e902313e9c3117e76d6adf88a2f104bc229374c
Author: imacdonn <email address hidden>
Date: Sat Sep 22 00:55:36 2018 +0000

    nova-status - don't count deleted compute_nodes

    When counting rows in the compute_nodes table to compare against
    resource records, those which have been marked as deleted should
    not be counted, otherwise the result is artificially high.

    Change-Id: Ia3fbb9c46c15826fc363286ce48984f59cbe1e62
    Closes-Bug: #1757207

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/604785

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/604786

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/604788

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/604785
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c559c5b357f47ae2bed9397bbfb8b2e08dc5b8a5
Submitter: Zuul
Branch: stable/rocky

commit c559c5b357f47ae2bed9397bbfb8b2e08dc5b8a5
Author: imacdonn <email address hidden>
Date: Sat Sep 22 00:55:36 2018 +0000

    nova-status - don't count deleted compute_nodes

    When counting rows in the compute_nodes table to compare against
    resource records, those which have been marked as deleted should
    not be counted, otherwise the result is artificially high.

    Change-Id: Ia3fbb9c46c15826fc363286ce48984f59cbe1e62
    Closes-Bug: #1757207
    (cherry picked from commit 3e902313e9c3117e76d6adf88a2f104bc229374c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/604786
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=72d2e82e1275035c70cb6862ff15dc41b3d9b621
Submitter: Zuul
Branch: stable/queens

commit 72d2e82e1275035c70cb6862ff15dc41b3d9b621
Author: imacdonn <email address hidden>
Date: Sat Sep 22 00:55:36 2018 +0000

    nova-status - don't count deleted compute_nodes

    When counting rows in the compute_nodes table to compare against
    resource records, those which have been marked as deleted should
    not be counted, otherwise the result is artificially high.

    Change-Id: Ia3fbb9c46c15826fc363286ce48984f59cbe1e62
    Closes-Bug: #1757207
    (cherry picked from commit 3e902313e9c3117e76d6adf88a2f104bc229374c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/604788
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ef5c02209a28510eee66edc8495acc6cd4e0993a
Submitter: Zuul
Branch: stable/pike

commit ef5c02209a28510eee66edc8495acc6cd4e0993a
Author: imacdonn <email address hidden>
Date: Sat Sep 22 00:55:36 2018 +0000

    nova-status - don't count deleted compute_nodes

    When counting rows in the compute_nodes table to compare against
    resource records, those which have been marked as deleted should
    not be counted, otherwise the result is artificially high.

    Change-Id: Ia3fbb9c46c15826fc363286ce48984f59cbe1e62
    Closes-Bug: #1757207
    (cherry picked from commit 3e902313e9c3117e76d6adf88a2f104bc229374c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.2

This issue was fixed in the openstack/nova 18.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.1.6

This issue was fixed in the openstack/nova 16.1.6 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.7

This issue was fixed in the openstack/nova 17.0.7 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.0.0rc1

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.