OpenStack Compute (nova)

resize of instance, that was booted from volume, trigger image size check then fails resize

  1. Series stein
  2. Bug #1825020
Bug #1825020 reported by Wendy Mitchell
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Matt Riedemann
Stein
Fix Committed
High
Matt Riedemann

Bug Description

Brief Description
-----------------
resize of instance, that was booted from volume, trigger image size check then fails resize

Severity
--------
standard

Steps to Reproduce
------------------

1. Create two flavors based on originating flavor and destination flavor

| 15566866-06d7-487a-b087-9fe928ec8c00 | flavor | 1024 | 4 | 0 | | 1 | 1.0 | True
 flavor:extra_specs | {"stx_storage": "remote", "hw:mem_page_size": "2048"}
flavor:ram | 1024
|
| flavor:swap | 0
|
| flavor:vcpus | 1
| flavor:disk | 4
|
| flavor:ephemeral | 0

| 20ebe3fe-c1a3-4e62-8c34-10c435f27112 | flavor-2 | 1024 | 1 | 1 | 512 | 1 | 1.0 | True
| extra_specs | {"stx_storage": "remote", "hw:mem_page_size": "2048"}
name | flavor-2
| ram | 1024
| swap | 512
| vcpus | 1
| disk | 1

2. Boot instance from volume with an originating flavor
OS-EXT-SRV-ATTR:hostname | tenant2-vm-13
id | 04f43c9e-d3fa-46c4-9605-c0d1cf8dda71
os-extended-volumes:volumes_attached | [{"id": "672aaa60-1086-45ce-8650-8195aa8dd0da", "delete_on_termination": false}
OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0
OS-EXT-SRV-ATTR:instance_name | instance-000000a4

3. Resize instance to destination flavor that has smaller disk size. Confirm resize.

[2019-04-14 10:05:04,590]
nova resize --poll 04f43c9e-d3fa-46c4-9605-c0d1cf8dda71 20ebe3fe-c1a3-4e62-8c34-10c435f27112'

Expected Behavior
------------------
Did not expect resize of instance, that was booted from volume, to trigger image size check

Actual Behavior
----------------
[2019-04-14 10:05:06,427]
Resize of instance booted from volume inadvertently triggered the image size check and results in a ClientExpection related to FlavorDiskSmalerThanMinDisk
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.FlavorDiskSmallerThanMinDisk'> (HTTP 500

nfv-vim.log (controller-0)

2019-04-14T10:05:06.228 controller-0 VIM_Thread[244649] INFO _instance_director.py.151 Instance action allowed for tenant2-vm-13, action_type=resize

2019-04-14T10:05:06.228 controller-0 VIM_Thread[244649] DEBUG _instance.py.1968 Resize instance tenant2-vm-13.

2019-04-14T10:05:06.228 controller-0 VIM_Thread[244649] INFO _instance_state_initial.py.36 Exiting state (initial) for tenant2-vm-13.

2019-04-14T10:05:06.229 controller-0 VIM_Thread[244649] INFO _instance_state_resize.py.27 Entering state (resize) for tenant2-vm-13.
2019-04-14T10:05:06.229 controller-0
VIM_Thread[244649] DEBUG nfvi_compute_api.py.3418 Instance action rest-api post path: /v2.1/6b4999fd6e414ae1b3eace662eb2ed84/servers/04f43c9e-d3fa-46c4-9605-c0d1cf8dda71/action.
2019-04-14T10:05:06.270 controller-0
VIM_Event-Log_Thread[244975] INFO fm.py.379 Generated customer log, fm_uuid=621d5dc6-79a8-40a7-814f-2809cbadb859.

2019-04-14T10:05:06.375 controller-0 VIM_Thread[244649] ERROR Caught exception while trying to resize an instance, error=[OpenStack Rest-API Exception: method=POST, url=http://nova-api.openstack.svc.cluster.local:8774/v2.1/6b4999fd6e414ae1b3eace662eb2ed84/servers/04f43c9e-d3fa-46c4-9605-c0d1cf8dda71/action, headers={'wrs-header': 'true', 'Content-Type': 'application/json', 'X-OpenStack-Nova-API-Version': '2.72'}, body={"resize": {"flavorRef": "20ebe3fe-c1a3-4e62-8c34-10c435f27112"}}, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Content-Length', '217'), ('X-Compute-Request-Id', 'req-aca6c954-1171-48ce-9286-3e9b952481d6'), ('Vary', 'OpenStack-API-Version, X-OpenStack-Nova-API-Version'), ('Openstack-Api-Version', 'compute 2.72'), ('X-Openstack-Nova-Api-Version', '2.72'), ('Date', 'Sun, 14 Apr 2019 10:05:06 GMT'), ('Content-Type', 'application/json; charset=UTF-8'), ('X-Openstack-Request-Id', 'req-aca6c954-1171-48ce-9286-3e9b952481d6')], response_body={"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.FlavorDiskSmallerThanMinDisk'>", "code": 500}}].

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_compute_api.py", line 2127, in resize_instance
    future.result = (yield)
OpenStackRestAPIException: [OpenStack Rest-API Exception: method=POST, url=http://nova-api.openstack.svc.cluster.local:8774/v2.1/6b4999fd6e414ae1b3eace662eb2ed84/servers/04f43c9e-d3fa-46c4-9605-c0d1cf8dda71/action, headers={'wrs-header': 'true', 'Content-Type': 'application/json', 'X-OpenStack-Nova-API-Version': '2.72'}, body={"resize": {"flavorRef": "20ebe3fe-c1a3-4e62-8c34-10c435f27112"}}, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Content-Length', '217'), ('X-Compute-Request-Id', 'req-aca6c954-1171-48ce-9286-3e9b952481d6'), ('Vary', 'OpenStack-API-Version, X-OpenStack-Nova-API-Version'), ('Openstack-Api-Version', 'compute 2.72'), ('X-Openstack-Nova-Api-Version', '2.72'),
('Date', 'Sun, 14 Apr 2019 10:05:06 GMT'), ('Content-Type', 'application/json; charset=UTF-8'), ('X-Openstack-Request-Id', 'req-aca6c954-1171-48ce-9286-3e9b952481d6')], response_body={"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.FlavorDiskSmallerThanMinDisk'>", "code": 500}}]

2019-04-14T10:05:06.377 controller-0 VIM_Thread[244649] INFO nfvi_compute_api.py.918 Sent response for request 8b44ee79-9371-479b-83a4-50534ddef1c7.
2019-04-14T10:05:06.377 controller-0
VIM_Thread[244649]
DEBUG _instance_task_work.py.426 Resize-Instance callback for tenant2-vm-13, response={'completed': False, 'reason': "unexpected api error. please report this at http://bugs.launchpad.net/nova/ and attach the nova api log if possible.\n<class 'nova.exception.flavordisksmallerthanmindisk'>"}.
2019-04-14T10:05:06.378 controller-0 VIM_Thread[244649] DEBUG _instance_tasks.py.367 Task (resize-instance_tenant2-vm-13) complete.

2019-04-14T10:05:06.378 controller-0 VIM_Thread[244649] INFO _instance_state_resize.py.68 Resize failed for tenant2-vm-13.
2019-04-14T10:05:06.378 controller-0
 VIM_Thread[244649]
DEBUG _instance.py.2179 Resize failed for instance tenant2-vm-13, reason=unexpected api error. please report this at http://bugs.launchpad.net/nova/ and attach the nova api log if possible.
<class 'nova.exception.flavordisksmallerthanmindisk'>.

Reproducibility
---------------
yes

System Configuration
--------------------
storage
(Lab: IP_20_27 20190414 10:02:43 test_resize_vm_positive[remote-4_0_0-1_1_512-volume])

Branch/Pull Time/Commit
--------------------
Load: 20190410T013000Z
Job: STX_build_master_master

Revision history for this message
Matt Riedemann (mriedem) wrote :

Are you sure this is reproducible in pure nova code? Because this looks like forked nova code for starlingx:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_compute_api.py", line 2127, in resize_instance
    future.result = (yield)

Looking at this code in the compute API in pure nova, it looks like that error only gets raised here:

https://github.com/openstack/nova/blob/e25d59078e61fe9f925dbef53dfe88e575d34dab/nova/compute/api.py#L655

Which is being called from resize now:

https://github.com/openstack/nova/blob/e25d59078e61fe9f925dbef53dfe88e575d34dab/nova/compute/api.py#L3652

As a result of this change in stein:

https://review.openstack.org/#/c/620706/

Note that root_bdm=None there so that's how we get into the else block of _validate_flavor_image_nostatus because it's not considering the instance as volume-backed.

Changed in nova:
status: New → Triaged
tags: added: regression starlingx stein-backport-potential
Changed in nova:
importance: Undecided → High
Matt Riedemann (mriedem)
Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/653268

Revision history for this message
Matt Riedemann (mriedem) wrote :

Hmm, even by passing the root_bdm to _validate_flavor_image_nostatus it can fail with VolumeSmallerThanMinDisk if the volume_size is smaller than the min_disk size of the backing image.

I think the original intent behind passing root_bdm=None to _validate_flavor_image_nostatus during resize was to just side-step these disk size checks for volume-backed servers but that doesn't seem to be easy without passing some parameter to _validate_flavor_image_nostatus to just tell it to ignore size checks.

Revision history for this message
Matt Riedemann (mriedem) wrote :

My fix is hitting a side effect of bug 1646740 where the min_disk stored in the image metadata for the server is based on the flavor root_gb rather than the volume size...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/653273

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.opendev.org/653268
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a40c7f3e8d06f0bc8918a25b60f73ec2c18add05
Submitter: Zuul
Branch: master

commit a40c7f3e8d06f0bc8918a25b60f73ec2c18add05
Author: Matt Riedemann <email address hidden>
Date: Tue Apr 16 23:00:15 2019 -0400

    Add functional regression recreate test for bug 1825020

    Change I06fad233006c7bab14749a51ffa226c3801f951b in Stein
    started validating extra specs on the new flavor during a
    resize but didn't take into account some validation that
    happens with disk size with the new flavor if the validation
    code isn't told that the server is volume-backed. That results
    in a 500 FlavorDiskSmallerThanMinDisk response from the API
    if the new flavor has a smaller disk size than the current
    flavor that the server is using, which shouldn't actually
    matter for a volume-backed server.

    This adds a functional regression test to recreate the bug.

    Change-Id: I9e7e44727946705506d740896349ca1833cfddcd
    Related-Bug: #1825020

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/656511

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/653273
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8aae3e39e58a05ed50b891aeae261641f544ad1f
Submitter: Zuul
Branch: master

commit 8aae3e39e58a05ed50b891aeae261641f544ad1f
Author: Matt Riedemann <email address hidden>
Date: Tue Apr 16 23:53:03 2019 -0400

    Fix volume-backed resize with a smaller disk flavor

    Change I06fad233006c7bab14749a51ffa226c3801f951b in Stein started
    calling _validate_flavor_image_nostatus during resize but did not
    pass the root_bdm since the underlying volume does not change. This
    can, however, lead to FlavorDiskSmallerThanMinDisk being raised
    if the new flavor has a disk size smaller than the current flavor.
    This is wrong in the case of a volume-backed server because we don't
    care about the root disk size in the flavor in that case.

    This fixes the bug by splitting the pci/numa validation logic out
    of _validate_flavor_image_nostatus into a separate method so that
    the resize method can call that directly for a volume-backed server
    rather than deal with the complicated disk size logic in
    _validate_flavor_image_nostatus (see bug 1646740 for details, but
    tl;dr if the image min_disk is less than the flavor root_gb, the
    min_disk stored in the instance image system metadata is the
    flavor['root_gb'] which could be larger than the root volume size).
    To see why trying to use _validate_flavor_image_nostatus during
    resize for a volume-backed server is a mess, look at PS1 of this
    change.

    Change-Id: I91c9c1e88afa035c757f692c78ad72d69cc3c431
    Closes-Bug: #1825020

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/656666

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/stein)

Reviewed: https://review.opendev.org/656511
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a1db4507ac832913a911afbee90f76f576d8ea20
Submitter: Zuul
Branch: stable/stein

commit a1db4507ac832913a911afbee90f76f576d8ea20
Author: Matt Riedemann <email address hidden>
Date: Tue Apr 16 23:00:15 2019 -0400

    Add functional regression recreate test for bug 1825020

    Change I06fad233006c7bab14749a51ffa226c3801f951b in Stein
    started validating extra specs on the new flavor during a
    resize but didn't take into account some validation that
    happens with disk size with the new flavor if the validation
    code isn't told that the server is volume-backed. That results
    in a 500 FlavorDiskSmallerThanMinDisk response from the API
    if the new flavor has a smaller disk size than the current
    flavor that the server is using, which shouldn't actually
    matter for a volume-backed server.

    This adds a functional regression test to recreate the bug.

    Change-Id: I9e7e44727946705506d740896349ca1833cfddcd
    Related-Bug: #1825020
    (cherry picked from commit a40c7f3e8d06f0bc8918a25b60f73ec2c18add05)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/stein)

Reviewed: https://review.opendev.org/656666
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3285210add2758ad0182358317fabca626cdb0ee
Submitter: Zuul
Branch: stable/stein

commit 3285210add2758ad0182358317fabca626cdb0ee
Author: Matt Riedemann <email address hidden>
Date: Tue Apr 16 23:53:03 2019 -0400

    Fix volume-backed resize with a smaller disk flavor

    Change I06fad233006c7bab14749a51ffa226c3801f951b in Stein started
    calling _validate_flavor_image_nostatus during resize but did not
    pass the root_bdm since the underlying volume does not change. This
    can, however, lead to FlavorDiskSmallerThanMinDisk being raised
    if the new flavor has a disk size smaller than the current flavor.
    This is wrong in the case of a volume-backed server because we don't
    care about the root disk size in the flavor in that case.

    This fixes the bug by splitting the pci/numa validation logic out
    of _validate_flavor_image_nostatus into a separate method so that
    the resize method can call that directly for a volume-backed server
    rather than deal with the complicated disk size logic in
    _validate_flavor_image_nostatus (see bug 1646740 for details, but
    tl;dr if the image min_disk is less than the flavor root_gb, the
    min_disk stored in the instance image system metadata is the
    flavor['root_gb'] which could be larger than the root volume size).
    To see why trying to use _validate_flavor_image_nostatus during
    resize for a volume-backed server is a mess, look at PS1 of this
    change.

    Change-Id: I91c9c1e88afa035c757f692c78ad72d69cc3c431
    Closes-Bug: #1825020
    (cherry picked from commit 8aae3e39e58a05ed50b891aeae261641f544ad1f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.1

This issue was fixed in the openstack/nova 19.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 20.0.0.0rc1

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.