Resource tracker double counts vcpus_used

Bug #1519001 reported by Paul Murray
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Paul Murray

Bug Description

The resource tracker is double counting the number of vcpu used.

It is believe this behavior was introduce in: https://review.openstack.org/#/c/240159/

That patch moved the vcpu accounting from the extensible resource tracker plugin into the main resource tracker code.

This behavior is at present openstack/nova master commit 4fb0da175d62084900e68c710f3cc0f916db64e0

To reproduce:

* Set up devstack (e.g. simple single machine setup)
* boot an instance
* initially 'nova hypervisor-show devstack' will show the correct vcpu used count
* wait for compute manager periodic task to run, updating the available resources
* now 'nova hypervisor-show devstack' will show double the number of used vcpu count

Revision history for this message
Paul Murray (pmurray) wrote :

The hypervisor is returning its own view of the number of vcpus used. This is copied to the compute_node record and is not reset before adding the resource tracker view. Code to reset the memory and disk is included in _update_usage_from_instances() as below. The extensible resource plugins are called here as well. Now the vcpu resource has gone and the accounting is directly in the compute_node object the vcpus_used should be reset to 0 here.

    def _update_usage_from_instances(self, context, instances):
        """Calculate resource usage based on instance utilization. This is
        different than the hypervisor's view as it will account for all
        instances assigned to the local compute host, even if they are not
        currently powered on.
        """
        self.tracked_instances.clear()

        # set some initial values, reserve room for host/hypervisor:
        self.compute_node.local_gb_used = CONF.reserved_host_disk_mb / 1024
        self.compute_node.memory_mb_used = CONF.reserved_host_memory_mb
        self.compute_node.free_ram_mb = (self.compute_node.memory_mb -
                                         self.compute_node.memory_mb_used)
        self.compute_node.free_disk_gb = (self.compute_node.local_gb -
                                          self.compute_node.local_gb_used)
        self.compute_node.current_workload = 0
        self.compute_node.running_vms = 0

        # Reset values for extended resources
        self.ext_resources_handler.reset_resources(self.compute_node,
                                                   self.driver)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/248742

Changed in nova:
assignee: nobody → Paul Murray (pmurray)
status: New → In Progress
Jay Pipes (jaypipes)
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/248742
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=69eca4c60d762b673fafb13dd12f285d7f1f1239
Submitter: Jenkins
Branch: master

commit 69eca4c60d762b673fafb13dd12f285d7f1f1239
Author: Paul Murray <email address hidden>
Date: Mon Nov 23 14:50:14 2015 +0000

    Fix resource tracker VCPU counting

    The resource track is double counting the number of
    vcpus used because it counts the virt driver's view
    and the resource tracker's view. It should reset
    the count to 0 before counting the resourcet racker's
    view. this is done for other resources (memory, disk)
    in _update_usage_from_instances().

    This was not caught in unit tests because they always
    provide a virt driver view with no resource used for
    tests that assume instances are already running.

    This patch adds the reset to 0 and modifies the
    tests to provide correct counts of used resources
    from the virt driver.

    Change-Id: I21acfd7d5b5ed442cd1e9049c986a06bd2b59b81
    Closes-Bug: #1519001

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/nova 13.0.0.0b1

This issue was fixed in the openstack/nova 13.0.0.0b1 development milestone.

Changed in nova:
status: Fix Committed → Fix Released
Changed in nova:
milestone: none → mitaka-1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.