volume-backed instance rebuild with no image change is still going through scheduler

Bug #1732947 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Matt Riedemann
Newton
Fix Committed
High
Matt Riedemann
Ocata
Fix Committed
High
Matt Riedemann
Pike
Fix Committed
High
Matt Riedemann

Bug Description

Due to this recent change:

https://github.com/openstack/nova/commit/984dd8ad6add4523d93c7ce5a666a32233e02e34

And the fact that volume-backed instances don't store the instance.image_ref value to indicate they are volume-backed, this condition will always evaluate to True:

https://github.com/openstack/nova/blob/984dd8ad6add4523d93c7ce5a666a32233e02e34/nova/compute/api.py#L2959

And we'll unnecessarily call through the scheduler again to validate the instance on the original host during the rebuild even if the image isn't changing.

For a volume-backed instance, we have to get the original image_id from the volume's volume_image_metadata field:

https://review.openstack.org/#/c/520686/

Matt Riedemann (mriedem)
Changed in nova:
status: New → Triaged
Changed in nova:
assignee: nobody → Chris Friesen (cbf123)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/521200

Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/521391

Changed in nova:
assignee: Chris Friesen (cbf123) → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/521200
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a4eebd5de7ef4b63082536718ba7b993a66d47e7
Submitter: Zuul
Branch: master

commit a4eebd5de7ef4b63082536718ba7b993a66d47e7
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 17 16:53:39 2017 -0500

    Add regression test for rebuilding a volume-backed server

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 makes rebuild
    check to see if the user is rebuilding an instance with a new
    image and if so, to run the scheduler filters again since the
    new image might not work with the current host for the instance,
    and we rebuild to the same host that the instance is already
    running on.

    The problem is the instance.image_ref attribute is not set for
    a volume-backed (boot-from-volume) instance, so the conditional
    in the rebuild() method is always True, which means we always run
    through the scheduler for volume-backed instances during rebuild,
    even if the image in the root disk isn't changing.

    This adds a functional regression test to recreate the bug.

    Change-Id: If79c554b46c44a7f70f8367426e7da362d7234c8
    Related-Bug: #1732947

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/521391
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=54407afef3b446e34c44ba5392ccbf121e266de2
Submitter: Zuul
Branch: master

commit 54407afef3b446e34c44ba5392ccbf121e266de2
Author: Matt Riedemann <email address hidden>
Date: Sun Nov 19 17:25:56 2017 -0500

    Get original image_id from volume for volume-backed instance rebuild

    A volume-backed instance will not have the instance.image_ref
    attribute set, to indicate to the API user that it is a volume-backed
    instance.

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 missed this subtle
    difference with how instance.image_ref is used, which means a rebuild
    of any volume-backed instance will now run through the scheduler, even
    if the image_href passed to rebuild is the same image ID as for the
    root volume.

    This fixes that case in rebuild by getting the image metadata off the
    root volume for a volume-backed instance and compares that to the
    image_href passed to rebuild.

    Change-Id: I48cda813b9effa37f6c3e0cd2e8a22bb78c79d72
    Closes-Bug: #1732947

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/525632

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/525634

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/525664

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/525666

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/newton)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/526542

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/526544

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b2

This issue was fixed in the openstack/nova 17.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/525632
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c7991653a4e7cbb86656e6ba7579a86236801567
Submitter: Zuul
Branch: stable/pike

commit c7991653a4e7cbb86656e6ba7579a86236801567
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 17 16:53:39 2017 -0500

    Add regression test for rebuilding a volume-backed server

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 makes rebuild
    check to see if the user is rebuilding an instance with a new
    image and if so, to run the scheduler filters again since the
    new image might not work with the current host for the instance,
    and we rebuild to the same host that the instance is already
    running on.

    The problem is the instance.image_ref attribute is not set for
    a volume-backed (boot-from-volume) instance, so the conditional
    in the rebuild() method is always True, which means we always run
    through the scheduler for volume-backed instances during rebuild,
    even if the image in the root disk isn't changing.

    This adds a functional regression test to recreate the bug.

    Change-Id: If79c554b46c44a7f70f8367426e7da362d7234c8
    Related-Bug: #1732947
    (cherry picked from commit a4eebd5de7ef4b63082536718ba7b993a66d47e7)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/525634
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4e36c4bd0eab2be619991ae00454906e171720b6
Submitter: Zuul
Branch: stable/pike

commit 4e36c4bd0eab2be619991ae00454906e171720b6
Author: Matt Riedemann <email address hidden>
Date: Sun Nov 19 17:25:56 2017 -0500

    Get original image_id from volume for volume-backed instance rebuild

    A volume-backed instance will not have the instance.image_ref
    attribute set, to indicate to the API user that it is a volume-backed
    instance.

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 missed this subtle
    difference with how instance.image_ref is used, which means a rebuild
    of any volume-backed instance will now run through the scheduler, even
    if the image_href passed to rebuild is the same image ID as for the
    root volume.

    This fixes that case in rebuild by getting the image metadata off the
    root volume for a volume-backed instance and compares that to the
    image_href passed to rebuild.

    Change-Id: I48cda813b9effa37f6c3e0cd2e8a22bb78c79d72
    Closes-Bug: #1732947
    (cherry picked from commit 54407afef3b446e34c44ba5392ccbf121e266de2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/525664
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fe5e3e7e61d879f7a28eb177e4bdbc7ead3ae4ac
Submitter: Zuul
Branch: stable/ocata

commit fe5e3e7e61d879f7a28eb177e4bdbc7ead3ae4ac
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 17 16:53:39 2017 -0500

    Add regression test for rebuilding a volume-backed server

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 makes rebuild
    check to see if the user is rebuilding an instance with a new
    image and if so, to run the scheduler filters again since the
    new image might not work with the current host for the instance,
    and we rebuild to the same host that the instance is already
    running on.

    The problem is the instance.image_ref attribute is not set for
    a volume-backed (boot-from-volume) instance, so the conditional
    in the rebuild() method is always True, which means we always run
    through the scheduler for volume-backed instances during rebuild,
    even if the image in the root disk isn't changing.

    This adds a functional regression test to recreate the bug.

    Conflicts:
          nova/tests/fixtures.py

    NOTE(mriedem): The CinderFixture conflict is due to not having
    1928ec5606491de95abba8791b1d7b888690dd3c in Ocata. However, this
    change needs the part of that patch to the CinderFixture to work,
    so it's added here.

    NOTE(mriedem): The functional test needed to change slightly to
    use the PlacementFixture and disable the DiskFilter.

    Change-Id: If79c554b46c44a7f70f8367426e7da362d7234c8
    Related-Bug: #1732947
    (cherry picked from commit a4eebd5de7ef4b63082536718ba7b993a66d47e7)
    (cherry picked from commit c7991653a4e7cbb86656e6ba7579a86236801567)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/525666
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=10bc4fdbc568670dfd8bec109e8b854ba8ad7949
Submitter: Zuul
Branch: stable/ocata

commit 10bc4fdbc568670dfd8bec109e8b854ba8ad7949
Author: Matt Riedemann <email address hidden>
Date: Sun Nov 19 17:25:56 2017 -0500

    Get original image_id from volume for volume-backed instance rebuild

    A volume-backed instance will not have the instance.image_ref
    attribute set, to indicate to the API user that it is a volume-backed
    instance.

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 missed this subtle
    difference with how instance.image_ref is used, which means a rebuild
    of any volume-backed instance will now run through the scheduler, even
    if the image_href passed to rebuild is the same image ID as for the
    root volume.

    This fixes that case in rebuild by getting the image metadata off the
    root volume for a volume-backed instance and compares that to the
    image_href passed to rebuild.

    Change-Id: I48cda813b9effa37f6c3e0cd2e8a22bb78c79d72
    Closes-Bug: #1732947
    (cherry picked from commit 54407afef3b446e34c44ba5392ccbf121e266de2)
    (cherry picked from commit 4e36c4bd0eab2be619991ae00454906e171720b6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/526542
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5ec5886fe74514c65908ff3adc2ca46610b595ad
Submitter: Zuul
Branch: stable/newton

commit 5ec5886fe74514c65908ff3adc2ca46610b595ad
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 17 16:53:39 2017 -0500

    Add regression test for rebuilding a volume-backed server

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 makes rebuild
    check to see if the user is rebuilding an instance with a new
    image and if so, to run the scheduler filters again since the
    new image might not work with the current host for the instance,
    and we rebuild to the same host that the instance is already
    running on.

    The problem is the instance.image_ref attribute is not set for
    a volume-backed (boot-from-volume) instance, so the conditional
    in the rebuild() method is always True, which means we always run
    through the scheduler for volume-backed instances during rebuild,
    even if the image in the root disk isn't changing.

    This adds a functional regression test to recreate the bug.

    Conflicts:
          nova/tests/fixtures.py

    NOTE(mriedem): The fixture conflict is due to not having the
    CinderFixture in Newton. That was added in Ocata via commit
    47fb8b757994688a995ec3bdfa5dccc72f870f34. It's added here in
    it's entirety since that is simpler than trying to basically
    re-write it from scratch in a backport.

    NOTE(mriedem): The functional test is slightly modified to
    not rely on Placement and because of old config option names.

    Change-Id: If79c554b46c44a7f70f8367426e7da362d7234c8
    Related-Bug: #1732947
    (cherry picked from commit a4eebd5de7ef4b63082536718ba7b993a66d47e7)
    (cherry picked from commit c7991653a4e7cbb86656e6ba7579a86236801567)
    (cherry picked from commit fe5e3e7e61d879f7a28eb177e4bdbc7ead3ae4ac)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/526544
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=24953a95e5989f44acaf488c253230e476c2fa59
Submitter: Zuul
Branch: stable/newton

commit 24953a95e5989f44acaf488c253230e476c2fa59
Author: Matt Riedemann <email address hidden>
Date: Sun Nov 19 17:25:56 2017 -0500

    Get original image_id from volume for volume-backed instance rebuild

    A volume-backed instance will not have the instance.image_ref
    attribute set, to indicate to the API user that it is a volume-backed
    instance.

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 missed this subtle
    difference with how instance.image_ref is used, which means a rebuild
    of any volume-backed instance will now run through the scheduler, even
    if the image_href passed to rebuild is the same image ID as for the
    root volume.

    This fixes that case in rebuild by getting the image metadata off the
    root volume for a volume-backed instance and compares that to the
    image_href passed to rebuild.

    Change-Id: I48cda813b9effa37f6c3e0cd2e8a22bb78c79d72
    Closes-Bug: #1732947
    (cherry picked from commit 54407afef3b446e34c44ba5392ccbf121e266de2)
    (cherry picked from commit 4e36c4bd0eab2be619991ae00454906e171720b6)
    (cherry picked from commit 10bc4fdbc568670dfd8bec109e8b854ba8ad7949)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.4

This issue was fixed in the openstack/nova 16.0.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.1.0

This issue was fixed in the openstack/nova 15.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.1.0

This issue was fixed in the openstack/nova 14.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.