NFS volume snapshot does not update volume attachment format to qcow2

Bug #1989514 reported by melanie witt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Medium
melanie witt
OpenStack Compute (nova)
In Progress
Undecided
melanie witt

Bug Description

This was reported downstream when a customer hit the issue [1].

They found that after performing a volume snapshot on a stopped instance, that instance could no longer boot.

The instance XML had the correct disk format "qcow2" after the snapshot while it was still stopped. However, when someone then tries to start the instance, the regenerated XML contains the wrong disk format "raw" (nova regenerates fresh XML on hard reboot, boot from shutoff) and that prevents the instance from booting.

While tracing the issue I noticed that after the snapshot, the volume attachment still showed the format "raw". It should have been "qcow2" after the snapshot, I think.

The customer found that rebuilding the instance resulted in a booting instance despite the underlying volume (snapshot) being the same. Rebuild generates a new volume attachment and that volume attachment reflected the correct format "qcow2".

Based on this, it appeared to me that volume attachment format was not being updated from "raw" to "qcow2" after the volume snapshot was created as qcow2 and was made the active volume for the instance.

AFAICT, there is not a way for nova to update the volume attachment format.

Repro steps:

  $ openstack volume create --image f8e5b23e-b1d3-44e8-bc71-ac7aca4f6f8e --size 1 --bootable my_volume

  $ openstack server create --flavor c1 --volume my_volume --network private --wait my_server

  $ openstack server stop d97a6178-112d-487b-ba21-1af2a4a78456

  (Instance has disk format "raw" and volume attachment also shows format "raw")
  (To check this:

    $ virsh dumpxml d97a6178-112d-487b-ba21-1af2a4a78456

    $ openstack --os-volume-api-version 3.27 volume attachment show 2f43581f-ba30-423e-ba5f-7b0ac7768fa5 -f json
  )

  $ openstack volume snapshot create --volume d5c71e24-46ac-49be-bf33-667598b6058b --force my_volume_snapshot

  (Instance has correct disk format "qcow2" at this point, volume attachment shows format "raw")

  $ openstack server start d97a6178-112d-487b-ba21-1af2a4a78456

  (Instance now has incorrect disk format "raw" and volume attachment still shows format "raw")

To see the effect of a rebuild:

  $ openstack server rebuild d97a6178-112d-487b-ba21-1af2a4a78456 --image f8e5b23e-b1d3-44e8-bc71-ac7aca4f6f8e --wait

  (Instance has disk format "qcow2" and volume attachment also shows format "qcow2")
  (Volume attachment is newly created with new ID etc)

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2118150

Tags: nfs drivers
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/857528

Changed in cinder:
status: New → In Progress
Eric Harney (eharney)
Changed in cinder:
importance: Undecided → Medium
Eric Harney (eharney)
tags: added: drivers nfs
Changed in cinder:
assignee: nobody → Sofia Enriquez (lsofia-enriquez)
Revision history for this message
melanie witt (melwitt) wrote (last edit ):

Hi, just FYI I am already working on this, I spent a LOT of time figuring out what's going on here this week and last :)

And I've proposed https://review.opendev.org/c/openstack/cinder/+/857528 yesterday (shown in comment #1).

I am going to assign this bug to myself to try to prevent confusion.

If my approach in the above proposed patch is a completely wrong direction, please let me know and I would abandon it.

melanie witt (melwitt)
Changed in cinder:
assignee: Sofia Enriquez (lsofia-enriquez) → melanie witt (melwitt)
Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

Sure! Sorry about assigning me the bug. I'll review your patch. Thank you so much for working on this.

Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

I just want to confirm that I've reproduced this bug. I'll review the patch proposed.

Revision history for this message
melanie witt (melwitt) wrote :

I've added the nova project to this bug in case it is more appropriate to address it there.

I'm going to try a nova patch as well to see if that might be the more proper way to fix this.

Changed in nova:
assignee: nobody → melanie witt (melwitt)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/858836

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by "melanie witt <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/858836
Reason: We talked about this problem with the cinder team at the PTG and the fix is going to be in cinder: https://review.opendev.org/c/openstack/cinder/+/857528

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.