OpenStack Compute (nova)

backing store missing for ephemeral disk on cold migration

  1. Series newton
  2. Bug #1605720
Bug #1605720 reported by Chris Friesen
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Chris Friesen
Newton
Fix Committed
Medium
Lee Yarwood

Bug Description

I'm on stable/mitaka, but the master code looks similar.

I have compute nodes configured to use qcow2 and libvirt. The flavor has an ephemeral disk and a swap disk, with root disk size of "0" for auto-size. I boot an instance with this flavor.

When I try to cold-migrate the instance, I get an error:
2016-07-21 23:33:48.561 46340 ERROR nova.compute.manager [instance: 4e52bfd8-0c71-48dc-89fb-6f6b31dc06bb] libvirtError: Cannot access backing file '/etc/nova/instances/_base/ephemeral_1_0706d66' of storage file '/etc/nova/instances/4e52bfd8-0c71-48dc-89fb-6f6b31dc06bb/disk.eph0' (as uid:0, gid:0): No such file or directory

The problem seems to be that in nova.virt.libvirt.driver.LibvirtDriver.finish_migration() we call self._create_image(...block_device_info=None...)

Down in _create_image() we handle the case of a "disk.local" ephemeral device, but that doesn't help because the device is actually named "disk.eph0". It looks like we then try to loop over any ephemerals in block_device_info, but that's None so we don't handle any of those (which is too bad since it looks like they would be named correctly).

The end result is that we have a qcow2 "disk.eph0" image, but with potentially no backing store in <instances_path>/_base. When we tell libvirt to start the instance, this results in the above error.

See original description
Chris Friesen (cbf123)
summary: - backing store missing for ephemeral disk on migration with boot-from-vol
+ backing store missing for ephemeral disk on cold migration
description: updated
Revision history for this message
Chris Friesen (cbf123) wrote :

The proposed fix is at https://review.openstack.org/#/c/346263/

Changed in nova:
assignee: nobody → Chris Friesen (cbf123)
Maciej Szankin (mszankin)
Changed in nova:
status: New → In Progress
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Chris Friesen (cbf123) → Feodor Tersin (ftersin)
Revision history for this message
Thiago Martins (martinx) wrote :

Please, include this fix on Mitaka.

Revision history for this message
Thiago Martins (martinx) wrote :

I'm trying to include the patch:

https://review.openstack.org/changes/346263/revisions/d7b831e38cbc029b7975d1059ed212762a5f6a5e/patch?zip

...on top of Ubuntu Xenial packages but, it fails here:

---
apt source nova
cd nova-13.1.1

nova-13.1.1$ patch -p1 < ../d7b831e3.diff
patching file nova/tests/unit/virt/libvirt/fake_imagebackend.py
Hunk #1 FAILED at 138.
1 out of 1 hunk FAILED -- saving rejects to file nova/tests/unit/virt/libvirt/fake_imagebackend.py.rej
patching file nova/tests/unit/virt/libvirt/test_driver.py
Hunk #1 succeeded at 17 (offset -2 lines).
Hunk #2 succeeded at 9549 (offset -975 lines).
Hunk #3 succeeded at 9574 with fuzz 2 (offset -980 lines).
Hunk #4 succeeded at 9674 (offset -969 lines).
Hunk #5 succeeded at 14565 (offset -1092 lines).
Hunk #6 succeeded at 14579 (offset -1092 lines).
Hunk #7 succeeded at 14647 with fuzz 2 (offset -1095 lines).
patching file nova/virt/libvirt/driver.py
Hunk #1 succeeded at 3112 (offset 152 lines).
Hunk #2 succeeded at 3238 (offset 185 lines).
Hunk #3 succeeded at 7397 with fuzz 1 (offset 191 lines).
---

 We need a proper fix for this on Ubuntu! :-P

 I'll try to rebuild the package with this patch but, don't know yet if tests will pass... If not, is there any way to disable the tests during package creation?

Cheers!
Thiago

Revision history for this message
Chris Friesen (cbf123) wrote :

Thiago: It looks like the patch will need reworking for Mitaka-on-Xenial to make it apply cleanly.

Revision history for this message
Feodor Tersin (ftersin) wrote :

@Thiago, see tests on PS4. I guess such implementation may work in Mitaka (i mostly mean usage of @mock.patch.object(nova.virt.libvirt.imagebackend.Backend, 'image') instead ImageBackendFixture). So you might rewrite tests to use it, but i have no idea if the result will be accepted as the backport to Mitaka.

I know nothing about package rules. I thought that a package must exactly contain a tagged version of an OS project source code, but recently i found an exception of this rule...

melanie witt (melwitt)
Changed in nova:
importance: Undecided → Medium
tags: added: libvirt
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/346263
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d7b831e38cbc029b7975d1059ed212762a5f6a5e
Submitter: Jenkins
Branch: master

commit d7b831e38cbc029b7975d1059ed212762a5f6a5e
Author: Chris Friesen <email address hidden>
Date: Fri Jul 22 16:33:43 2016 -0600

    Fix cold migration with qcow2 ephemeral disks

    If we have qcow2 ephemeral disks we need to ensure that the
    backing file gets created on a cold migration.

    This requires passing in the block_device_info when calling
    _create_image() so that we can loop over the ephemeral disks.

    Closes-Bug: #1605720
    Co-Authored-By: Feodor Tersin <email address hidden>
    Change-Id: Ie278bb10e1675ba1d903aaa3c0249be0d1cf147b

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/398812

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.0.0b2

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

Matt Riedemann (mriedem)
Changed in nova:
assignee: Feodor Tersin (ftersin) → Chris Friesen (cbf123)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/398812
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=842f1b6651f6104f8cb0e5ef32b80e6ffb10b067
Submitter: Jenkins
Branch: stable/newton

commit 842f1b6651f6104f8cb0e5ef32b80e6ffb10b067
Author: Chris Friesen <email address hidden>
Date: Fri Jul 22 16:33:43 2016 -0600

    Fix cold migration with qcow2 ephemeral disks

    If we have qcow2 ephemeral disks we need to ensure that the
    backing file gets created on a cold migration.

    This requires passing in the block_device_info when calling
    _create_image() so that we can loop over the ephemeral disks.

    Closes-Bug: #1605720
    Co-Authored-By: Feodor Tersin <email address hidden>
    Change-Id: Ie278bb10e1675ba1d903aaa3c0249be0d1cf147b
    (cherry picked from commit d7b831e38cbc029b7975d1059ed212762a5f6a5e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.4

This issue was fixed in the openstack/nova 14.0.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/372721
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.