OpenStack Compute (nova)

Soft-reboot fails after volume-update

Series ocata
Bug #1713857

Bug #1713857 reported by melanie witt on 2017-08-29

This bug affects 1 person

	Status	Importance	Assigned to
OpenStack Compute (nova)	Fix Released	Medium	melanie witt
Ocata	Fix Committed	Undecided	melanie witt
Pike	Fix Committed	Undecided	Matt Riedemann

Bug Description

During a volume-update, we don't save the instance XML with the new volume attachment, so soft-reboot fails with:

DEBUG nova.virt.libvirt.driver [req-149741f9-ad36-4787-ac6c-1ed7a273ab6a 75fbc7a6db34480091d2a53e2e20b695 62e53e5e804e49a9890928a5a4846f60 - - -] [instance: 5f06f6f8-f8b8-4b46-afa4-cefff24d7167] Instance soft reboot failed: Cannot access storage file '/dev/disk/by-path/ip-192.168.11.121:3260-iscsi-iqn.2010-10.org.openstack:volume-d558d988-b17c-4940-8c05-9222ad750955-lun-0' (as uid:107, gid:107): No such file or directory reboot /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:2395
WARNING nova.virt.libvirt.driver [req-149741f9-ad36-4787-ac6c-1ed7a273ab6a 75fbc7a6db34480091d2a53e2e20b695 62e53e5e804e49a9890928a5a4846f60 - - -] [instance: 5f06f6f8-f8b8-4b46-afa4-cefff24d7167] Failed to soft reboot instance. Trying hard reboot.

as it can't access the old volume attachment defined in the XML. It falls back on a hard reboot which is like pulling the power cord and can result in file system inconsistencies.

We should save the updated XML to the persistent configuration after updating the volume.

Steps to reproduce:

1. Boot an instance (nova boot --image <image> --flavor <flavor> test0)
2. Create two volumes (cinder create --name A 1, cinder create --name B 1)
3. Attach the first volume to the instance (nova volume-attach test0 <uuid of volume A>)
4. Swap to the second volume (nova volume-update test0 <uuid of volume A> <uuid of volume B>)
5. Try to reboot the instance (nova reboot test0)

Actual result:

Instance fails to soft-reboot and hard reboots instead.

Expected result:

Instance soft-reboots successfully.

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-29: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/498983

Changed in nova:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-11-29: Fix merged to nova (master)

Reviewed: https://review.openstack.org/498983
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5b008c6540f948d28dc50a0ef84095ebd96d198d
Submitter: Zuul
Branch: master

commit 5b008c6540f948d28dc50a0ef84095ebd96d198d
Author: melanie witt <email address hidden>
Date: Tue Aug 29 22:53:44 2017 +0000

Save updated libvirt domain XML after swapping volume

    When a user calls the volume-update API, we swap_volume in the libvirt
    driver from the old volume attachment to the new volume attachment.
    Currently, we're saving the domain XML with the old configuration prior
    to updating the volume and upon a soft-reboot request, it results in an
    error:

Instance soft reboot failed: Cannot access storage file <old path>

and falls back to a hard reboot, which is like pulling the power cord,
possibly resulting in file system inconsistencies.

This changes to saving the new, updated domain XML after the volume
swap.

Closes-Bug: #1713857

Change-Id: I166cde5ad8b00699e4ec02609f0d7b69236d855d

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-11-29: Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/523910

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-11-29: Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/523911

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-12-07: Fix included in openstack/nova 17.0.0.0b2

This issue was fixed in the openstack/nova 17.0.0.0b2 development milestone.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-13: Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/523910
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3dbcf9d5be0f631f4d963117a242006df3649b8d
Submitter: Zuul
Branch: stable/pike

commit 3dbcf9d5be0f631f4d963117a242006df3649b8d
Author: melanie witt <email address hidden>
Date: Tue Aug 29 22:53:44 2017 +0000

Save updated libvirt domain XML after swapping volume

Instance soft reboot failed: Cannot access storage file <old path>

and falls back to a hard reboot, which is like pulling the power cord,
possibly resulting in file system inconsistencies.

This changes to saving the new, updated domain XML after the volume
swap.

Closes-Bug: #1713857

Change-Id: I166cde5ad8b00699e4ec02609f0d7b69236d855d
(cherry picked from commit 5b008c6540f948d28dc50a0ef84095ebd96d198d)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-13: Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/523911
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=aa0e4ebd2c02c042bbe2c0d58e690e3e43b04a17
Submitter: Zuul
Branch: stable/ocata

commit aa0e4ebd2c02c042bbe2c0d58e690e3e43b04a17
Author: melanie witt <email address hidden>
Date: Tue Aug 29 22:53:44 2017 +0000

Save updated libvirt domain XML after swapping volume

Instance soft reboot failed: Cannot access storage file <old path>

and falls back to a hard reboot, which is like pulling the power cord,
possibly resulting in file system inconsistencies.

This changes to saving the new, updated domain XML after the volume
swap.

Closes-Bug: #1713857

    NOTE(melwitt): The assert in test_swap_volume_block_old_libvirt had to
    be modified to account for the additional XMLDesc call from the bug
    fix. The test_swap_volume_block_old_libvirt doesn't exist in Pike, so
    that's why we didn't hit this in the master patch.

Change-Id: I166cde5ad8b00699e4ec02609f0d7b69236d855d
(cherry picked from commit 5b008c6540f948d28dc50a0ef84095ebd96d198d)