Unable to transition to Ironic Node Resource Classes in Pike

From IRC, my way of describing the problem:

(9:24:05 AM) mriedem: do you mean the custom resource class allocatoin isn't showing up for the instance?
(9:24:37 AM) johnthetubaguy: I mean old instances don't have allocations for the new resource, which breaks the scheduling of new flavors that use the resource class
(9:24:50 AM) mriedem: so instance A had vcpu/ram/disk allocations before the node.resource_class was set, then you set node.resource_class = baremetal and now you expect to see a 'baremetal' allocation for instance A in placement
(9:25:17 AM) johnthetubaguy: that is what I expected, yes
(9:25:24 AM) johnthetubaguy: (clearly incorrectly)
(9:25:31 AM) mriedem: that's probably because the RT isn't reporting allocations anymore once all of your computes are pike

(9:27:48 AM) mriedem: johnthetubaguy: ok so the scheduling issue is a new instance request can try to claim the 'baremetal' resource on node A even though instance A is already using it
(9:27:52 AM) mriedem: which causes a scheduler failure yes?
(9:28:00 AM) johnthetubaguy: mriedem: yes

We made a similar sounding fix in Pike https://review.openstack.org/#/c/492964/ but that was for reporting inventory with the new node.resource_class, not for updating instance allocations for a migrated flavor/instance.

One thought is that the ironic driver itself could update the allocations once a flavor is migrated and has a new custom resource class. That gets tricky for anyone that has already upgraded to pike and some of their instances / nodes have migrated - the nova code would have to check the existing allocations and see if they contain the custom resource class, and we don't want to do that check on every flavor migration run (which happens when nodes are refreshed).

Best way forward seems to be fixing up the allocations in here:
https://github.com/openstack/nova/blob/e11a8687aef527eee9f7c733db6cccd5b902afbb/nova/virt/ironic/driver.py#L523

To deal with the issue in comment #2 above, before we get here:

https://github.com/openstack/nova/blob/e11a8687aef527eee9f7c733db6cccd5b902afbb/nova/virt/ironic/driver.py#L523

We'd have to check if the instance has an allocation in placement for the normalized_rc and if not, add it. That is at most redundant once per instance per compute service restart because of this:

https://github.com/openstack/nova/blob/e11a8687aef527eee9f7c733db6cccd5b902afbb/nova/virt/ironic/driver.py#L559

So we would only process that once.

(10:00:03 AM) mriedem: johnthetubaguy: one issue might be a chicken and egg with the custom RC inventory being available when we PUT /allocations/{consumer_uuid}
(10:00:20 AM) mriedem: because the inventory gets updated via the RT periodic
(10:01:01 AM) mriedem: so if i just set a custom rc on my ironic node, the periodic runs, it detects the new rc and will update inventory, but while in the driver we're migrating the instance because it's node has an rc now, if we try updating the allocation for that rc before the node RP inventory has that rc, it will fail with a 409
(10:01:07 AM) johnthetubaguy: mriedem: dang, yeah
(10:01:08 AM) mriedem: "409 Conflict if there is no available inventory in any of the resource providers for any specified resource classes or inventories are updated by another thread while attempting the operation."

Fix proposed to branch: master
Review: https://review.openstack.org/513085

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/513198

Reviewed: https://review.openstack.org/513085
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5c2b8675e3e13e32b23681153f226de93bb99628
Submitter: Zuul
Branch: master

commit 5c2b8675e3e13e32b23681153f226de93bb99628
Author: John Garbutt <email address hidden>
Date: Wed Oct 18 17:05:43 2017 +0100

    Keep updating allocations for Ironic

    When ironic updates the instance.flavor to require the new custom
    resource class, we really need the allocations to get updated. Easiest
    way to do that is to make the resource tracker keep updating allocations
    for the ironic virt driver. This can be dropped once the transition to
    custom resource classes is complete.

    If we were not to claim the extra resources, placement will pick nodes
    that already have instances running on them when you boot an instance
    with a flavor that only requests the custom resource class. This should
    be what all ironic flavors do, before the upgrade to queens is
    performed.

    Closes-Bug: #1724589

    Change-Id: Ibbf65a8d817d359786abcdffa6358089ed1107f6

Reviewed: https://review.openstack.org/513198
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=842641c9b0b8f60d0e19b38ad1180078e9c4330c
Submitter: Zuul
Branch: stable/pike

commit 842641c9b0b8f60d0e19b38ad1180078e9c4330c
Author: John Garbutt <email address hidden>
Date: Wed Oct 18 17:05:43 2017 +0100

    Keep updating allocations for Ironic

    When ironic updates the instance.flavor to require the new custom
    resource class, we really need the allocations to get updated. Easiest
    way to do that is to make the resource tracker keep updating allocations
    for the ironic virt driver. This can be dropped once the transition to
    custom resource classes is complete.

    If we were not to claim the extra resources, placement will pick nodes
    that already have instances running on them when you boot an instance
    with a flavor that only requests the custom resource class. This should
    be what all ironic flavors do, before the upgrade to queens is
    performed.

    Closes-Bug: #1724589

    Change-Id: Ibbf65a8d817d359786abcdffa6358089ed1107f6
    (cherry picked from commit 5c2b8675e3e13e32b23681153f226de93bb99628)

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

This issue was fixed in the openstack/nova 16.0.2 release.