OpenStack Compute (nova)

Preallocated disks are deducted twice from disk_available_least when using preallocated_images = space

  1. Series queens
  2. Bug #1764489
Bug #1764489 reported by Lee Yarwood
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Lee Yarwood
Ocata
Fix Committed
Medium
Lee Yarwood
Pike
Fix Committed
Medium
Lee Yarwood
Queens
Fix Committed
Medium
Lee Yarwood

Bug Description

Description
===========
When using preallocated file based disks (preallocate_images = space) with the Libvirt virt driver the reported allocation for each disk appears doubled, leaving disk_available_least under reporting the amount of available resources on a compute node.

Originally reported and debugged by mschuppert and mdbooth: https://bugzilla.redhat.com/show_bug.cgi?id=1554265

Steps to reproduce
==================

$ crudini --get /etc/nova/nova-cpu.conf DEFAULT preallocate_images
space

$ nova hypervisor-show eb515aa2-b79e-4f38-a6ea-d493a6e0657f
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
[..]
| disk_available_least | 43 |
| free_disk_gb | 48 |
[..]
| local_gb | 48 |
| local_gb_used | 0 |
[..]
+---------------------------+------------------------------------------+

$ nova flavor-show 2
+----------------------------+----------+
| Property | Value |
+----------------------------+----------+
[..]
| disk | 20 |
[..]
+----------------------------+----------+

$ nova boot --image cirros-0.3.5-x86_64-disk --flavor 2 test
[..]
$ nova hypervisor-show eb515aa2-b79e-4f38-a6ea-d493a6e0657f
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
[..]
| disk_available_least | 3 |
| free_disk_gb | 28 |
[..]
| local_gb | 48 |
| local_gb_used | 20 |
[..]
+---------------------------+------------------------------------------+

Expected result
===============
The allocation for each preallocated disk is reported correctly and removed from disk_available_least.

Actual result
=============
The allocation for each preallocated disk appears doubled and removed from disk_available_least.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   fb0b785169e5e422b06e82f2eb58e68f6d2008b3

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   file / qcow2

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============
See above.

Tags: libvirt
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/561704

Changed in nova:
assignee: nobody → Lee Yarwood (lyarwood)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/561704
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=23bd8f62634707fc9896a38ff4dae606c89c6c4b
Submitter: Zuul
Branch: master

commit 23bd8f62634707fc9896a38ff4dae606c89c6c4b
Author: Lee Yarwood <email address hidden>
Date: Mon Apr 16 18:08:56 2018 +0100

    libvirt: Report the allocated size of preallocated file based disks

    At present the Libvirt driver can preallocate file based disks using the
    fallocate command and importantly the `-n` option. This option allocates
    blocks on the filesystem past the initial EOF of a given file:

    ```
    $ touch test.img ; fallocate -n -l $(( 1024 * 1024 )) test.img
    $ ll -lah test.img
    -rw-rw-r--. 1 stack stack 0 Apr 16 13:28 test.img
    $ du -h test.img
    1.0M test.img
    ```

    This results in a miscalculation of the total disk overcommit for file
    based (excluding ploop) disks as os.path.getsize is currently used to
    determine the allocated size of these disks:

    ```
    >>> import os
    >>> os.path.getsize('test.img')
    0
    ```

    Using the above example the disk overcommit would be reported as 1.0M as
    the disk appears empty yet will report a potential (virtual) size of 1.0M.

    However as the required blocks have already been allocated on the
    filesystem the host will report disk_available_least as missing an
    additional 1.0M, essentially doubling the allocation for each disk.

    To correct this the allocated size of file based (excluding ploop) disks
    is reported using `disk_size` from the `qemu-img info` command. This
    should ensure blocks allocated past the EOF of the file are taken into
    account and correctly reported as allocated.

    A future change should ultimately remove the use of the `-n` option with
    fallocate, however as this would not help disks that have already been
    allocated this has not been included in this change to simplify backports.

    Change-Id: If642e51a4e186833349a8e30b04224a3687f5594
    Closes-bug: #1764489

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/562074

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/562077

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/562078

Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/562074
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2d50f6e7854543849c4cb7641ae6b88fe04cb6f6
Submitter: Zuul
Branch: stable/queens

commit 2d50f6e7854543849c4cb7641ae6b88fe04cb6f6
Author: Lee Yarwood <email address hidden>
Date: Mon Apr 16 18:08:56 2018 +0100

    libvirt: Report the allocated size of preallocated file based disks

    At present the Libvirt driver can preallocate file based disks using the
    fallocate command and importantly the `-n` option. This option allocates
    blocks on the filesystem past the initial EOF of a given file:

    ```
    $ touch test.img ; fallocate -n -l $(( 1024 * 1024 )) test.img
    $ ll -lah test.img
    -rw-rw-r--. 1 stack stack 0 Apr 16 13:28 test.img
    $ du -h test.img
    1.0M test.img
    ```

    This results in a miscalculation of the total disk overcommit for file
    based (excluding ploop) disks as os.path.getsize is currently used to
    determine the allocated size of these disks:

    ```
    >>> import os
    >>> os.path.getsize('test.img')
    0
    ```

    Using the above example the disk overcommit would be reported as 1.0M as
    the disk appears empty yet will report a potential (virtual) size of 1.0M.

    However as the required blocks have already been allocated on the
    filesystem the host will report disk_available_least as missing an
    additional 1.0M, essentially doubling the allocation for each disk.

    To correct this the allocated size of file based (excluding ploop) disks
    is reported using `disk_size` from the `qemu-img info` command. This
    should ensure blocks allocated past the EOF of the file are taken into
    account and correctly reported as allocated.

    A future change should ultimately remove the use of the `-n` option with
    fallocate, however as this would not help disks that have already been
    allocated this has not been included in this change to simplify backports.

    Change-Id: If642e51a4e186833349a8e30b04224a3687f5594
    Closes-bug: #1764489
    (cherry picked from commit 23bd8f62634707fc9896a38ff4dae606c89c6c4b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/562077
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d88b75e81eabfbd463007f6a4f27e6966a466530
Submitter: Zuul
Branch: stable/pike

commit d88b75e81eabfbd463007f6a4f27e6966a466530
Author: Lee Yarwood <email address hidden>
Date: Mon Apr 16 18:08:56 2018 +0100

    libvirt: Report the allocated size of preallocated file based disks

    At present the Libvirt driver can preallocate file based disks using the
    fallocate command and importantly the `-n` option. This option allocates
    blocks on the filesystem past the initial EOF of a given file:

    ```
    $ touch test.img ; fallocate -n -l $(( 1024 * 1024 )) test.img
    $ ll -lah test.img
    -rw-rw-r--. 1 stack stack 0 Apr 16 13:28 test.img
    $ du -h test.img
    1.0M test.img
    ```

    This results in a miscalculation of the total disk overcommit for file
    based (excluding ploop) disks as os.path.getsize is currently used to
    determine the allocated size of these disks:

    ```
    >>> import os
    >>> os.path.getsize('test.img')
    0
    ```

    Using the above example the disk overcommit would be reported as 1.0M as
    the disk appears empty yet will report a potential (virtual) size of 1.0M.

    However as the required blocks have already been allocated on the
    filesystem the host will report disk_available_least as missing an
    additional 1.0M, essentially doubling the allocation for each disk.

    To correct this the allocated size of file based (excluding ploop) disks
    is reported using `disk_size` from the `qemu-img info` command. This
    should ensure blocks allocated past the EOF of the file are taken into
    account and correctly reported as allocated.

    A future change should ultimately remove the use of the `-n` option with
    fallocate, however as this would not help disks that have already been
    allocated this has not been included in this change to simplify backports.

    Conflicts:
            nova/tests/unit/virt/libvirt/test_driver.py

    NOTE(lyarwood): I11e329ac5f5fe4b9819fefbcc32ff1ee504fc58b made
    get_domain private in Queens.

    Change-Id: If642e51a4e186833349a8e30b04224a3687f5594
    Closes-bug: #1764489
    (cherry picked from commit 23bd8f62634707fc9896a38ff4dae606c89c6c4b)
    (cherry picked from commit 2d50f6e7854543849c4cb7641ae6b88fe04cb6f6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.0.0b1

This issue was fixed in the openstack/nova 18.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/562078
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=caca71376fbc2c1195cda71608c8d2a70ef9a678
Submitter: Zuul
Branch: stable/ocata

commit caca71376fbc2c1195cda71608c8d2a70ef9a678
Author: Lee Yarwood <email address hidden>
Date: Mon Apr 16 18:08:56 2018 +0100

    libvirt: Report the allocated size of preallocated file based disks

    At present the Libvirt driver can preallocate file based disks using the
    fallocate command and importantly the `-n` option. This option allocates
    blocks on the filesystem past the initial EOF of a given file:

    ```
    $ touch test.img ; fallocate -n -l $(( 1024 * 1024 )) test.img
    $ ll -lah test.img
    -rw-rw-r--. 1 stack stack 0 Apr 16 13:28 test.img
    $ du -h test.img
    1.0M test.img
    ```

    This results in a miscalculation of the total disk overcommit for file
    based (excluding ploop) disks as os.path.getsize is currently used to
    determine the allocated size of these disks:

    ```
    >>> import os
    >>> os.path.getsize('test.img')
    0
    ```

    Using the above example the disk overcommit would be reported as 1.0M as
    the disk appears empty yet will report a potential (virtual) size of 1.0M.

    However as the required blocks have already been allocated on the
    filesystem the host will report disk_available_least as missing an
    additional 1.0M, essentially doubling the allocation for each disk.

    To correct this the allocated size of file based (excluding ploop) disks
    is reported using `disk_size` from the `qemu-img info` command. This
    should ensure blocks allocated past the EOF of the file are taken into
    account and correctly reported as allocated.

    A future change should ultimately remove the use of the `-n` option with
    fallocate, however as this would not help disks that have already been
    allocated this has not been included in this change to simplify backports.

    Conflicts:
            nova/tests/unit/virt/libvirt/test_driver.py

    NOTE(lyarwood): I11e329ac5f5fe4b9819fefbcc32ff1ee504fc58b made
    get_domain private in Queens.

    Change-Id: If642e51a4e186833349a8e30b04224a3687f5594
    Closes-bug: #1764489
    (cherry picked from commit 23bd8f62634707fc9896a38ff4dae606c89c6c4b)
    (cherry picked from commit 2d50f6e7854543849c4cb7641ae6b88fe04cb6f6)
    (cherry picked from commit d88b75e81eabfbd463007f6a4f27e6966a466530)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.1.2

This issue was fixed in the openstack/nova 16.1.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.3

This issue was fixed in the openstack/nova 17.0.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.1.1

This issue was fixed in the openstack/nova 15.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.