instances being migrated are not accounting for disk_available_least

Bug #1517442 reported by Nikola Đipanov
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
High
Unassigned

Bug Description

Looking briefly at the code of other drivers that try to report this (xenapi and ironic) - it is also likely broken for at least xenapi.

The crux of the issue is that resource tracker works by looking at the instances Nova knows about, and also the ongoing migration, so anything that is reported by any of the virt drivers as part of the dictionary returned from get_available_resource should only be based on the available resources and should never try to factor in any resource usage. Only the resource tracker holding the global resource lock (COMPUTE_RESOURCE_SEMAPHORE) knows the current usage of resources since it can take into account migrations that are in flight etc.

Unfortunately, both libvirt and xenapi (I think) try to look at the instance currently know by the hypervisor, which is not all instances we should be taking into account, and deduce the final disk_available_least number.

To fix this we would have to rework how disk_available least is calculated - we'd have to make sure the drivers only report the total available space, and then make sure we update the usage _for each instance and migration_ to come up with the final number.

Revision history for this message
Bob Ball (bob-ball) wrote :

I believe this should work already for XenAPI? we get the physical utilisation from the SR, not counting up from instances. VDIs will be created for disks that are being live migrated at the start of the live migration.

https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/xenapi/driver.py#n448
https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/xenapi/host.py#n243

Was the issue identifed by inspection or is there a failure case that has been seen?

Revision history for this message
Bob Ball (bob-ball) wrote :

Awesome - so re-reading I can see it was identified by inspection... Sorry for being blind. I don't think XenAPI is affected by this.

Changed in nova:
status: New → Confirmed
Changed in nova:
assignee: nobody → Mohammed Ashraf (mohammed-asharaf)
status: Confirmed → In Progress
Revision history for this message
Mohammed Ashraf (mohammed-asharaf) wrote :

Hi,

I am new to Openstack .

I just started taking a look at this defect. I understand from the comment above that it's was raised upon code inspection
and there's a clarification that it's not applicable to xenapi.

So is this now valid only for libvirt driver and is this defect reproducible for libvirt driver? Any one able to reproduce it?

Please let me know.

Thanks
Ashraf

Changed in nova:
assignee: Mohammed Ashraf (mohammed-asharaf) → nobody
Changed in nova:
status: In Progress → Confirmed
Wenzhi Yu (yuywz)
Changed in nova:
assignee: nobody → Wenzhi Yu (yuywz)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova-specs (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/293272

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova-specs (master)

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/293272
Reason: Per discussion at the summit (https://etherpad.openstack.org/p/ocata-nova-summit-meetup), we are abandoning Newton specs that haven't merged. You may restore this patch and move it to Ocata if you'd like.

Revision history for this message
Jay Pipes (jaypipes) wrote : Re: libvirt/xenapi: disk_available_least reported by the driver does not take into account instances being migrated to/from the host

Unassigning. This is a real bug, yes. The issue Chris Friesen raised on the related patch (since abandoned) around accounting for the QCOW2 backing file should be tracked as a separate bug.

This bug is relatively straight-forward, as is the fix: ensure we track migrating instances for disk_available_least calculations.

The actual patch may be complicated, however, due to various side-effects and virt-driver-specific implementation details.

Changed in nova:
assignee: Wenzhi Yu (yuywz) → nobody
summary: - libvirt/xenapi: disk_available_least reported by the driver does not
- take into account instances being migrated to/from the host
+ instances being migrated are not accounting for disk_available_least
Jay Pipes (jaypipes)
tags: removed: libvirt xen
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.