Ubuntu
nova package

Bug: resize instance after edit flavor with horizon

  1. Xenial (16.04)
  2. Bug #1570748
Bug #1570748 reported by Hyun Ha
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Dan Smith
Kilo
Fix Released
Medium
Matt Riedemann
OpenStack Compute (nova) 2015.1.4
Liberty
Fix Released
Medium
Matt Riedemann
Mitaka
Fix Released
Medium
Matt Riedemann
Ubuntu Cloud Archive
Fix Released
Medium
Unassigned
Liberty
Triaged
Medium
Unassigned
Mitaka
Triaged
Medium
Unassigned
nova-powervm
Fix Released
Medium
Drew Thorstensen
tempest
Fix Released
Wishlist
Matt Riedemann
nova (Ubuntu)
Fix Released
Undecided
Unassigned
Wily
Triaged
Medium
Unassigned
Xenial
Triaged
Medium
Unassigned
Yakkety
Fix Released
Undecided
Unassigned

Bug Description

Error occured when resize instance after edit flavor with horizon (and also delete flavor used by instance)

Reproduce step :

1. create flavor A
2. boot instance using flavor A
3. edit flavor with horizon (or delete flavor A)
    -> the result is same to edit or to delelet flavor because edit flavor means delete/recreate flavor)
4. resize or migrate instance
5. Error occured

Log :
nova-compute.log
   File "/opt/openstack/src/nova/nova/conductor/manager.py", line 422, in _object_dispatch
     return getattr(target, method)(*args, **kwargs)

   File "/opt/openstack/src/nova/nova/objects/base.py", line 163, in wrapper
     result = fn(cls, context, *args, **kwargs)

   File "/opt/openstack/src/nova/nova/objects/flavor.py", line 132, in get_by_id
     db_flavor = db.flavor_get(context, id)

   File "/opt/openstack/src/nova/nova/db/api.py", line 1479, in flavor_get
     return IMPL.flavor_get(context, id)

   File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 233, in wrapper
     return f(*args, **kwargs)

   File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 4732, in flavor_get
     raise exception.FlavorNotFound(flavor_id=id)

 FlavorNotFound: Flavor 7 could not be found.

This Error is occured because of below code:
/opt/openstack/src/nova/nova/compute/manager.py

    def resize_instance(self, context, instance, image,
                        reservations, migration, instance_type,
                        clean_shutdown=True):
....
            if (not instance_type or
                not isinstance(instance_type, objects.Flavor)):
                instance_type = objects.Flavor.get_by_id(
                    context, migration['new_instance_type_id'])
....

I think that deleted flavor should be taken when resize instance.
I tested this in stable/kilo, but I think stable/liberty and stable/mitaka has same bug because source code is not changed.

thanks.

Hyun Ha (raymon-ha)
summary: - Bug: Error resize instance after edit flavor with horizon
+ Bug: resize instance after edit flavor with horizon
tags: added: nova
tags: added: migrate resize
Revision history for this message
gundarapu kalyan reddy (gundarapu-reddy) wrote :

Its working fine in stable/mitaka . I am able to resize the instance after deleting the flavor with which the instance is launched.

Revision history for this message
Qijing Li (qijing-lee) wrote :

I can reproduce this bug in master.

Sean Dague (sdague)
Changed in nova:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Dan Smith (danms) wrote :

If this works in mitaka but not master, I'd want to know: Have you run online migrations to get the flavors moved over to the api database yet?

Also, I need to see more of the logs around the failure please.

Revision history for this message
Matt Riedemann (mriedem) wrote :

I was able to confirm this with a newish (created last week) newton devstack. The failure wasn't on the resize, it was on the revert. Steps:

1. create a duplicate of flavor 42, call it flavor 43 m1.nano2
2. boot instance using flavor 42
3. delete flavor 42
4. resize instance to flavor 43
5. revert the resize, at this point it fails:

ERROR (BadRequest): Flavor used by the instance could not be found. (HTTP 400) (Request-ID: req-98916fd8-846e-4564-ac45-a09933883bb0)

This is my version of nova:

stack@swapvol:/opt/stack/nova$ git log -1
commit 6ae2b4534a8a0d5394317533a41ad1ee3fa13964
Merge: 177a2c4 9e305c2
Author: Jenkins <email address hidden>
Date: Thu Apr 14 14:17:51 2016 +0000

    Merge "cleanup some issues in parameters.yaml"
stack@swapvol:/opt/stack/nova$

This is all I see in the n-api logs:

2016-04-18 18:24:59.406 DEBUG nova.compute.api [req-98916fd8-846e-4564-ac45-a09933883bb0 admin admin] [instance: 630f2bbe-b104-4497-b1a8-936b75fe9356] Fetching instance by UUID from (pid=27352) get /opt/stack/nova/nova/compute/api.py:2041
2016-04-18 18:24:59.497 INFO nova.osapi_compute.wsgi.server [req-98916fd8-846e-4564-ac45-a09933883bb0 admin admin] 9.5.125.128 "POST /v2.1/servers/630f2bbe-b104-4497-b1a8-936b75fe9356/action HTTP/1.1" status: 400 len: 383 time: 0.2037680

Revision history for this message
Matt Riedemann (mriedem) wrote :

It looks like the problem is here:

https://github.com/openstack/nova/blob/9989a75384aeb6c6b398e7caf8dee140c6bf0423/nova/compute/utils.py#L462

When doing a revert_resize from the compute API, it calls nova.compute.utils.reverse_upsize_quota_delta which tries to lookup the old flavor using the flavor ID from the migration record rather than the instance, and since the flavor was deleted we get a 404.

Revision history for this message
Matt Riedemann (mriedem) wrote :

The fix is that reverse_upsize_quota_delta should be using instance.old_flavor rather than relying on the old_instance_type in the migration object.

Revision history for this message
Matt Riedemann (mriedem) wrote :

We should also have a tempest test for this scenario since we missed it. We have tempest testing for reverting a resize, but not one where the flavor was deleted in between the resize and the revert.

Changed in nova:
status: Confirmed → Triaged
tags: added: mitaka-backport-potential
Matt Riedemann (mriedem)
Changed in tempest:
status: New → Confirmed
importance: Undecided → Wishlist
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/307438

Changed in nova:
assignee: nobody → Dan Smith (danms)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tempest (master)

Fix proposed to branch: master
Review: https://review.openstack.org/307518

Changed in tempest:
assignee: nobody → Matt Riedemann (mriedem)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/307438
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0
Submitter: Jenkins
Branch: master

commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/309040

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/309110

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/309168

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/kilo)

Reviewed: https://review.openstack.org/309168
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7e3d3abf50732a8dd9350a1ab65376780e7ee0bd
Submitter: Jenkins
Branch: stable/kilo

commit 7e3d3abf50732a8dd9350a1ab65376780e7ee0bd
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Conflicts:
            nova/compute/api.py
            nova/compute/utils.py
            nova/tests/unit/compute/test_compute_api.py
            nova/tests/unit/compute/test_compute_utils.py

    NOTE(mriedem): The conflicts are due to _reverse_upsize_quota_delta
    being moved from nova.compute.api to nova.compute.utils in liberty
    in commit 60411ddf1bada3629f0588de8726c63b5ab00357. So the
    nova.compute.utils changes are omitted here. Also, there was no direct
    testing of _reverse_upsize_quota_delta until that change in liberty
    so as part of this cherry pick we pull back the test_reverse_quota_delta
    test. It's modified slightly since it's in a NoDBTestCase.

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748
    (cherry picked from commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0)
    (cherry picked from commit 30aad1e796778a94da7719b1d4586bd3cdecb821)
    (cherry picked from commit 5374568fadebf639886febe6073baf352c2a9d83)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/mitaka)

Reviewed: https://review.openstack.org/309040
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=30aad1e796778a94da7719b1d4586bd3cdecb821
Submitter: Jenkins
Branch: stable/mitaka

commit 30aad1e796778a94da7719b1d4586bd3cdecb821
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748
    (cherry picked from commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/liberty)

Reviewed: https://review.openstack.org/309110
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5374568fadebf639886febe6073baf352c2a9d83
Submitter: Jenkins
Branch: stable/liberty

commit 5374568fadebf639886febe6073baf352c2a9d83
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Conflicts:
            nova/tests/unit/compute/test_compute_utils.py

    NOTE(mriedem): The conflict is due to uuidsentinel not being imported
    in liberty (and is not needed here) and test_migration still being
    used in liberty (but not mitaka).

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748
    (cherry picked from commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0)
    (cherry picked from commit 30aad1e796778a94da7719b1d4586bd3cdecb821)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (master)

Reviewed: https://review.openstack.org/307518
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=98c32b7860f03bedbe2d4d8e35ec53cfa0c0c5cb
Submitter: Jenkins
Branch: master

commit 98c32b7860f03bedbe2d4d8e35ec53cfa0c0c5cb
Author: Matt Riedemann <email address hidden>
Date: Mon Apr 18 19:34:47 2016 -0400

    Add a test for reverting a resize with a deleted flavor

    A user should be able to boot a server, resize it and
    then reject the resize even though the original flavor
    used to boot the instance is deleted. This is because
    the old flavor information is stored with the instance
    in the nova database, so the original flavor doesn't
    actually need to exist anymore.

    Depends-On: I5f95021410a309ac07fe9f474cbcd0214d1af208

    Change-Id: I356411f96a601f1443d75ac90e42567bef1f8228
    Closes-Bug: #1570748

Changed in tempest:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova-powervm (master)

Reviewed: https://review.openstack.org/311145
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=e72a505860b2f7ce2dd4a492f65d426f8b091e1f
Submitter: Jenkins
Branch: master

commit e72a505860b2f7ce2dd4a492f65d426f8b091e1f
Author: Drew Thorstensen <email address hidden>
Date: Fri Apr 29 10:12:33 2016 -0500

    Fix revert migration if flavor deleted

    After an instance is created, the flavor could be deleted. But
    operations still need to be able to occur against the VM when that
    occurs. A new test in Tempest showed that we had been using the
    incorrect (and potentially deleted) flavor when a resize/migration was
    reverting.

    This change updates the code to use the flavor that is stored on the
    instance itself.

    Change-Id: I16254d8eb74a724e3e0a538f47259e9627c4233a
    Closes-Bug: 1570748

Changed in nova-powervm:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova-powervm (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/311521

Eric Fried (efried)
Changed in nova-powervm:
status: Fix Released → Fix Committed
assignee: nobody → Drew Thorstensen (thorst)
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova-powervm (stable/mitaka)

Reviewed: https://review.openstack.org/311521
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=f98749a2d3154452d8edb8c0e95b54f5cde579d1
Submitter: Jenkins
Branch: stable/mitaka

commit f98749a2d3154452d8edb8c0e95b54f5cde579d1
Author: Drew Thorstensen <email address hidden>
Date: Fri Apr 29 10:12:33 2016 -0500

    Fix revert migration if flavor deleted

    After an instance is created, the flavor could be deleted. But
    operations still need to be able to occur against the VM when that
    occurs. A new test in Tempest showed that we had been using the
    incorrect (and potentially deleted) flavor when a resize/migration was
    reverting.

    This change updates the code to use the flavor that is stored on the
    instance itself.

    Change-Id: I16254d8eb74a724e3e0a538f47259e9627c4233a
    Closes-Bug: 1570748
    (cherry picked from commit e72a505860b2f7ce2dd4a492f65d426f8b091e1f)
    Conflicts:
     nova_powervm/tests/virt/powervm/test_driver.py

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 2015.1.4

This issue was fixed in the openstack/nova 2015.1.4 release.

Louis Bouchard (louis)
Changed in nova (Ubuntu Yakkety):
status: New → Fix Released
Edward Hope-Morley (hopem)
tags: added: sts-sru
Corey Bryant (corey.bryant)
no longer affects: cloud-archive/kilo
Drew Thorstensen (thorst)
Changed in nova-powervm:
status: Fix Committed → Fix Released
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/nova 14.0.0.0b1

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 12.0.4

This issue was fixed in the openstack/nova 12.0.4 release.

Corey Bryant (corey.bryant)
Changed in nova (Ubuntu Wily):
importance: Undecided → Medium
status: New → Triaged
Changed in nova (Ubuntu Xenial):
importance: Undecided → Medium
status: New → Triaged
Changed in cloud-archive:
importance: Undecided → Medium
status: New → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 13.1.0

This issue was fixed in the openstack/nova 13.1.0 release.

Louis Bouchard (louis)
tags: removed: sts-sru
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 2015.1.4

This issue was fixed in the openstack/nova 2015.1.4 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.