scheduler report client sends allocations with value of zero, violating min_unit

Bug #1647316 reported by Chris Dent
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Ed Leafe
Newton
Fix Committed
Medium
Matt Riedemann

Bug Description

When a VM boots using non-local disk, it tries to send an allocation of 'DISK_GB': 0. This violates the default min_unit of 1 and causes an error that looks like this:

[req-858cbed4-c113-45e8-94e3-1d8ee64f9de0 488c2b05a66b441199f4c1dca7accd5b 3fa5b55ecc154427b636119f0920d252 - default default] Bad inventory
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handlers/allocation.py", line 253, in set_allocations
    allocations.create_all()
  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
    return fn(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/objects/resource_provider.py", line 1050, in create_all
    self._set_allocations(self._context, self.objects)
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 894, in wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/objects/resource_provider.py", line 1011, in _set_allocations
    before_gens = _check_capacity_exceeded(conn, allocs)
  File "/usr/lib/python2.7/site-packages/nova/objects/resource_provider.py", line 921, in _check_capacity_exceeded
    resource_provider=rp_uuid)
InvalidAllocationConstraintsViolated: Unable to create allocation for 'DISK_GB' on resource provider 'f9398126-d0e8-4cf8-ae45-9103a88aa13d'. The requested amount would violate inventory constraints.

The causing code is at https://github.com/openstack/nova/blob/474c2ef28234dacc658e9a78762cac66ef7fe334/nova/scheduler/client/report.py#L105

The correct fix is probably that whenever the value of any resource class is zero, don't send that resource class in the dict.

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Yeah I think that would be the best approach for fixing that bug.

Ed Leafe (ed-leafe)
Changed in nova:
assignee: Chris Dent (cdent) → Ed Leafe (ed-leafe)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/407180

Changed in nova:
status: Triaged → In Progress
Changed in nova:
assignee: Ed Leafe (ed-leafe) → Chris Dent (cdent)
Changed in nova:
assignee: Chris Dent (cdent) → Ed Leafe (ed-leafe)
Revision history for this message
Matt Riedemann (mriedem) wrote :

Marking this high as it's causing issues in the ceph jobs:

http://logs.openstack.org/14/414214/12/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/511c30e/logs/screen-n-cpu.txt.gz#_2017-01-03_23_17_44_657

2017-01-03 23:17:44.657 4254 DEBUG nova.scheduler.client.report [req-eced32b1-1b58-446a-bf57-02280b9f2b4d tempest-TenantUsagesTestJSON-1895317078 tempest-TenantUsagesTestJSON-1895317078] [instance: f83871cb-c77e-4c99-833f-186db97b38b1] Sending allocation for instance {'allocations': [{'resource_provider': {'uuid': 'f8ce7899-0b48-4294-8df2-057ac325f4d2'}, 'resources': {'MEMORY_MB': 64, 'VCPU': 1, 'DISK_GB': 0}}]} _allocate_for_instance /opt/stack/new/nova/nova/scheduler/client/report.py:505
2017-01-03 23:17:44.750 4254 WARNING nova.scheduler.client.report [req-eced32b1-1b58-446a-bf57-02280b9f2b4d tempest-TenantUsagesTestJSON-1895317078 tempest-TenantUsagesTestJSON-1895317078] Unable to submit allocation for instance f83871cb-c77e-4c99-833f-186db97b38b1 (409 409 Conflict

There was a conflict when trying to complete your request.

 Unable to allocate inventory: Unable to create allocation for 'DISK_GB' on resource provider 'f8ce7899-0b48-4294-8df2-057ac325f4d2'. The requested amount would violate inventory constraints. )

Changed in nova:
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/416764

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/407180
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=200666ad19e44fb76aab29ed664767885b259822
Submitter: Jenkins
Branch: master

commit 200666ad19e44fb76aab29ed664767885b259822
Author: EdLeafe <email address hidden>
Date: Mon Dec 5 16:15:53 2016 +0000

    Do not post allocations that are zero

    When posting allocations for VMs with a non-local disk, the code was
    sending an allocation of DISK_GB = 0, which violated the min_unit
    resource constraint. Since it is pointless to enter an allocation of
    zero units, this patch removes any items in the generated allocations
    dict if the value is zero.

    Closes-Bug: #1647316

    Change-Id: I59eff4310e67ceb74086f5ade1a637f8cccec7ed

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.0.0b3

This issue was fixed in the openstack/nova 15.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/416764
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5a184889916360194fdbbdfeaf1ba1c3d75fc8f5
Submitter: Jenkins
Branch: stable/newton

commit 5a184889916360194fdbbdfeaf1ba1c3d75fc8f5
Author: EdLeafe <email address hidden>
Date: Mon Dec 5 16:15:53 2016 +0000

    Do not post allocations that are zero

    When posting allocations for VMs with a non-local disk, the code was
    sending an allocation of DISK_GB = 0, which violated the min_unit
    resource constraint. Since it is pointless to enter an allocation of
    zero units, this patch removes any items in the generated allocations
    dict if the value is zero.

    Closes-Bug: #1647316

    Change-Id: I59eff4310e67ceb74086f5ade1a637f8cccec7ed
    (cherry picked from commit 200666ad19e44fb76aab29ed664767885b259822)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.5

This issue was fixed in the openstack/nova 14.0.5 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.