OpenStack Compute (nova)

The quota_usage_refresh counts wrong when there exists soft-deleted instances

Series queens
Bug #1773542

Bug #1773542 reported by huanhongda on 2018-05-26

This bug affects 1 person

	Status	Importance	Assigned to
OpenStack Compute (nova)	Invalid	Undecided	huanhongda
Ocata	Fix Committed	Medium	huanhongda
Pike	Fix Committed	Medium	huanhongda
Queens	Fix Committed	Medium	huanhongda

Bug Description

Description
===========
When there exists soft-deleted instances in my openstack, executing "nova-manage project quota_usage_refresh" will get wrong values in quota_usages.

Steps to reproduce
==================
1) Set reclaim_instance_interval>0 to enable soft-delete;
2) Boot a new instance, then check quota_usages tables. Assume instances' in_use is 1, ram's in_use is 512, cores' in_use is 4.
3) Delete that instance, and it's status will migrate to SOFT_DELETED.
4) Check quota_usages tables. Both instances, ram and cores in_use is 0.
5) Execute "nova-manage project quota_usage_refresh --project <project-id> --user <user-id>".
6) Check quota_usages tables. The instances' in_use is 1, ram's in_use is 512, cores' in_use is 4.

Expected result
===============
After the execution of quota_usage_refresh, both instances, ram and cores in_use should be 0.

Actual result
=============
The instances' in_use is 1, ram's in_use is 512, cores' in_use is 4. That's because quota_usage_refresh will count all instances whose "deleted" field is 0 in "instances" table. And soft-deleted instance's "deleted" field is 0, so it will be counted as in used.

Environment
===========
This bug was found in Ocata.
Pike, Queens and master don't have this bug.
The quota_usage_refresh command has been removed in 17.0.0 (Queens). Since quota usage is counted from resources instead of being tracked separately.
https://docs.openstack.org/nova/pike/cli/nova-manage.html

And they have filtered out soft-deleted instances while counting from resources:
https://github.com/openstack/nova/blob/5bf9aa7fd455e93b73dc13aba66ccb7ec00f23a5/nova/objects/instance.py#L1475-L1489

See original description

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-26: Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/570653

Revision history for this message

jichenjc (jichenjc) wrote on 2018-05-28:

assign to hongda as already a fix there

Changed in nova:
assignee:	nobody → huanhongda (hongda)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-28: Change abandoned on nova (stable/ocata)

Change abandoned by huanhongda (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/570653
Reason: Submit to master first then cherry pick.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-28: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/570736

Changed in nova:
status:	New → In Progress

huanhongda (hongda) on 2018-05-28

description:

updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-30: Change abandoned on nova (master)

Change abandoned by huanhongda (<email address hidden>) on branch: master
Review: https://review.openstack.org/570736
Reason: Submit this patch to stable/ocata directly.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-06-29: Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/579093

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-06-29: Change abandoned on nova (stable/ocata)

Change abandoned by huanhongda (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/570653
Reason: Fix the bug in stable/queens first then backport to stable/ocata.

Revision history for this message

melanie witt (melwitt) wrote on 2018-08-09:

This is invalid for Rocky, so marking the Rocky series version of the bug as Invalid.

Changed in nova:
status:	In Progress → Invalid
tags:	added: quotas

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-13: Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/602459

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-13: Change abandoned on nova (stable/pike)

#10

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/602459
Reason: Missed the stable/ocata change was already closed.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-14: Fix merged to nova (stable/queens)

#11

Reviewed: https://review.openstack.org/579093
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=58538ace357d97bd9a06ce042aadc2928b1ff5bc
Submitter: Zuul
Branch: stable/queens

commit 58538ace357d97bd9a06ce042aadc2928b1ff5bc
Author: huanhongda <email address hidden>
Date: Sat May 26 22:40:27 2018 +0800

[Stable Only] Remove soft-deleted instances from quota_usages

    The resources in quota_usages table will minus 1 while
    soft deleting an instance. But the soft-deleted instances
    are counted when synchronizing quota_usages using command
    "nova-manage project quota_usage_refresh". So we filter
    out soft-deleted instances when synchronizing quota_usage.

    We didn't need to do this on master as the quota_usage_refresh
    command has been removed in 17.0.0 (Queens). And the
    _instance_data_get_for_user function is already removed
    in commit c881e77819f1b8ca48c0dfd0ebfb75daaabf1033.
    https://review.openstack.org/#/c/570865/

Change-Id: I8dcbe1e5f3f70ddd7067fed1125ef3379044ab0f
Closes-bug: #1773542

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-15: Fix merged to nova (stable/pike)

#12

Reviewed: https://review.openstack.org/602459
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ab03100bcbd5afe5702b760057fc19e25300cd89
Submitter: Zuul
Branch: stable/pike

commit ab03100bcbd5afe5702b760057fc19e25300cd89
Author: huanhongda <email address hidden>
Date: Sat May 26 22:40:27 2018 +0800

[Stable Only] Remove soft-deleted instances from quota_usages

    Change-Id: I8dcbe1e5f3f70ddd7067fed1125ef3379044ab0f
    Closes-bug: #1773542
    (cherry picked from commit 58538ace357d97bd9a06ce042aadc2928b1ff5bc)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-24: Fix included in openstack/nova 16.1.5

#13

This issue was fixed in the openstack/nova 16.1.5 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-24: Fix included in openstack/nova 17.0.6

#14

This issue was fixed in the openstack/nova 17.0.6 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-02: Fix merged to nova (stable/ocata)

#15

Reviewed: https://review.openstack.org/570653
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b4659017936d415c4395a57c52fad6cf3ffa5302
Submitter: Zuul
Branch: stable/ocata

commit b4659017936d415c4395a57c52fad6cf3ffa5302
Author: huanhongda <email address hidden>
Date: Sat May 26 22:40:27 2018 +0800

[Stable Only] Remove soft-deleted instances from quota_usages

    Change-Id: I8dcbe1e5f3f70ddd7067fed1125ef3379044ab0f
    Closes-bug: #1773542
    (cherry picked from commit 58538ace357d97bd9a06ce042aadc2928b1ff5bc)
    (cherry picked from commit ab03100bcbd5afe5702b760057fc19e25300cd89)