ResourceTracker doesn't clean up compute_nodes or stats entries

Bug #1784874 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Dan Smith
Ocata
Fix Committed
Low
Matt Riedemann
Pike
Fix Released
Low
Matt Riedemann
Queens
Fix Committed
Low
Matt Riedemann

Bug Description

This was noted in review:

https://review.openstack.org/#/c/587636/4/nova/compute/resource_tracker.py@141

That the ResourceTracker.compute_nodes and ResourceTracker.stats (and old_resources) entries only grow and are never cleaned up as we rebalance nodes or nodes are deleted, which means it just takes up memory over time.

When we cleanup compute nodes here:

https://github.com/openstack/nova/blob/47ef500f4492c731ebfa33a12822ef6b5db4e7e2/nova/compute/manager.py#L7759

We should probably call a cleanup hook into the ResourceTracker to cleanup those entries as well.

Matt Riedemann (mriedem)
Changed in nova:
assignee: nobody → Dan Smith (danms)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/587922

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/587922
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=99db9faae5c1e84e5aa8586c32ffcca7a61ae276
Submitter: Zuul
Branch: master

commit 99db9faae5c1e84e5aa8586c32ffcca7a61ae276
Author: Dan Smith <email address hidden>
Date: Wed Aug 1 10:16:41 2018 -0700

    Hook resource_tracker to remove stale node information

    When we remove a node from a host due to rebalance or decommission,
    we need to tell the resource tracker so it stops keeping track of
    the details for that node. This adds a hook for doing that and
    calls it from the place in compute manager where we purge the
    record from the database.

    Change-Id: Ie6b6bb2a9e8a4ad33675fccb3827e8197fa16398
    Closes-Bug: #1784874

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.0.0rc1

This issue was fixed in the openstack/nova 18.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/676282

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/676461

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.opendev.org/676467

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.opendev.org/676282
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7302a625ee9af2c9dd88f8630f1e95fc5977a70a
Submitter: Zuul
Branch: stable/queens

commit 7302a625ee9af2c9dd88f8630f1e95fc5977a70a
Author: Dan Smith <email address hidden>
Date: Wed Aug 1 10:16:41 2018 -0700

    Hook resource_tracker to remove stale node information

    When we remove a node from a host due to rebalance or decommission,
    we need to tell the resource tracker so it stops keeping track of
    the details for that node. This adds a hook for doing that and
    calls it from the place in compute manager where we purge the
    record from the database.

    Conflicts:
          nova/compute/manager.py

    NOTE(mriedem): The conflict is due to not having change
    I7e0658e681ead30c241ad39baa0fd4334e5a86a1 in Queens.

    Change-Id: Ie6b6bb2a9e8a4ad33675fccb3827e8197fa16398
    Closes-Bug: #1784874
    (cherry picked from commit 99db9faae5c1e84e5aa8586c32ffcca7a61ae276)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.13

This issue was fixed in the openstack/nova 17.0.13 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.opendev.org/676461
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4f112195b0ad99f07784a9fe8dbe32a25329151d
Submitter: Zuul
Branch: stable/pike

commit 4f112195b0ad99f07784a9fe8dbe32a25329151d
Author: Dan Smith <email address hidden>
Date: Wed Aug 1 10:16:41 2018 -0700

    Hook resource_tracker to remove stale node information

    When we remove a node from a host due to rebalance or decommission,
    we need to tell the resource tracker so it stops keeping track of
    the details for that node. This adds a hook for doing that and
    calls it from the place in compute manager where we purge the
    record from the database.

    Change-Id: Ie6b6bb2a9e8a4ad33675fccb3827e8197fa16398
    Closes-Bug: #1784874
    (cherry picked from commit 99db9faae5c1e84e5aa8586c32ffcca7a61ae276)
    (cherry picked from commit 7302a625ee9af2c9dd88f8630f1e95fc5977a70a)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.opendev.org/676467
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9f8ff3960cd8915af4017da64dbeaf3a74383074
Submitter: Zuul
Branch: stable/ocata

commit 9f8ff3960cd8915af4017da64dbeaf3a74383074
Author: Dan Smith <email address hidden>
Date: Wed Aug 1 10:16:41 2018 -0700

    Hook resource_tracker to remove stale node information

    When we remove a node from a host due to rebalance or decommission,
    we need to tell the resource tracker so it stops keeping track of
    the details for that node. This adds a hook for doing that and
    calls it from the place in compute manager where we purge the
    record from the database.

    Change-Id: Ie6b6bb2a9e8a4ad33675fccb3827e8197fa16398
    Closes-Bug: #1784874
    (cherry picked from commit 99db9faae5c1e84e5aa8586c32ffcca7a61ae276)
    (cherry picked from commit 7302a625ee9af2c9dd88f8630f1e95fc5977a70a)
    (cherry picked from commit 4f112195b0ad99f07784a9fe8dbe32a25329151d)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova ocata-eol

This issue was fixed in the openstack/nova ocata-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova pike-eol

This issue was fixed in the openstack/nova pike-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.