After upgrade: "libvirt.libvirtError: Requested operation is not valid: format of backing image %s of image %s was not specified"

I had thought this failure case had been relaxed in later versions of libvirt 6.

Which version of libvirt are you using?

Ah apologies it's still there in master:

https://github.com/libvirt/libvirt/blob/84dc367e2a6febfcc4aeea4c52e44c52aea298fc/src/util/virstoragefile.c#L5265-L5270

So we will need to workaround this within the libvirt driver using the following:

https://libvirt.org/kbase/backing_chains.html#vm-refuses-to-start-due-to-misconfigured-backing-store-format

This should be easy enough to wire up in the qcow2 imagebackend during cache() assuming it doesn't impact already running domains/instances using the backing file.

Ignore c#2, stephenfin correctly points out that the following landed in 6.1.0 relaxing this:

https://github.com/libvirt/libvirt/commit/ae9e6c2a2b75d958995c661f7bb64ed4353a6404

We shouldn't need to rebase existing disks without a backing file format as a result.

Given the above, the solution here seems to be to update your version of libvirt to >= 6.1.0. I'm going to mark this as WONTFIX. If this does not resolve the issue, please reset to new and provide information on the version of libvirt you've tested with and detailed logs from nova-compute showing the error.

It appears ubuntu bionic/focal ship with libvirt < 6.1.0, marking nova as affected in those distribs

Adding a task for libvirt at Ubuntu/focal since this patch[0] might need to be backported.

[0] https://github.com/libvirt/libvirt/commit/ae9e6c2a2b75d958995c661f7bb64ed4353a6404

Thanks everyone, I agree that we'd want to consider this for Focals libvirt.
But there is a quite excessive series of changes in v6.0.0 -> v6.1.0 before the referenced change took place. So it will either be a massive backport (picking much more related changes to cherry-pick) or have a rather big set of backporting changes.

I wanted to ask if out of all your testing above there is already a simpler testcase that one can use without nova and all such. Just a) how to create a broken image + b) how to make libvirt stumble over it.
We will need this for the SRU process anyway if we want to bring it to Focal.

Also for severity - since didn't have this show up in any other place yet I wonder how common this case would be? Any comment on that?

For the time being (while discussing) I have tried to rather "bluntly" backport the change.
If it builds would you be able to try this PPA [1] in your existing test environment?
If it works then MP [2] could be a simpler change than what I was afraid it could be.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4388/+packages
[2]: https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/395778

Hi Christian,

I've tested the patch in your [2] ([2] below) and it worked perfectly. I got a colleague to rebuild it for me against the current focal libvirt [1] as I couldn't find your PPA mentioned. I'm upgrading an OpenStack system from bionic-ussuri -> focal-ussuri, and confirmed that the instance wouldn't start with 6.0.0-0ubuntu8.9. I then upgraded to 6.0.0-0ubuntu8.10.1~ubuntu20.04.1~ppa202106240958 from the PPA [1] and the instanced started.

Is this sufficient to get it included in the focal archives?

Many thanks for your work on this.

[1] https://launchpad.net/~chris.macnaughton/+archive/ubuntu/focal-ussuri/
[2] https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/395778

Thanks Alex,
I'll rebase and polish it and hand it over to a team member for review then.

But what I'd still need is some written out test steps for the SRU process.
Since you have just verified it would you mind adding a statement here that I could copy into the SRU template then?

I have rebased and finalized the MP for this:
https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/395778
This is now open for review by the Team.

Furthermore I have opened a new PPA to have up-to-date test builds.
new PPA:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4591/+packages

@Alex - we will need a proper SRU Template and I could only talk about the patch, but not to much about the testing. I'd appreaciate if you could add an SRU template (https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template) to the bug description.

FYI it seems bug 1864020 talks about the same case.

Ran the test plan on focal which produced the following error:

ERROR Requested operation is not valid: format of backing image '/var/lib/libvirt/images/base/focal-server-cloudimg-amd64' of image '/var/lib/libvirt/images/testinst/root.img' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start testinst
otherwise, please restart your installation.

Ran the test plan on focal with libvirt packages from ppa:ci-train-ppa-service/4591.
The domain booted successfully.

Thanks Brett, that confirms they are the same root cause and fixed by the same solution.
Thanks.
And it also confirms that the test steps I've got from you will apply here.

MP approved for upload and subsequent SRU accept.

Given this relaxes the previous commit https://libvirt.org/git/?p=libvirt.git;a=commit;h=3615e8b39bad, would there be value in additionally testing during SRU verification that libvirt correctly still rejects what it should, as well as that this bug is fixed as you've already documented?

FYI: I uploaded the fix to Focal-unapproved for verification by the SRU Team.

"additionally testing during SRU verification that libvirt correctly still rejects what it should"

I can try then, but I'm not 100% sure I can easily recreate that case.
If I can then sure, if not we should not block on it IMHO as the backported code matches the >=Groovy behavior anyway.

Hello Peter, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hi @racb,

I have tested:

libvirt-daemon-system:
  Installed: 6.0.0-0ubuntu8.11
  Candidate: 6.0.0-0ubuntu8.11
  Version table:
 *** 6.0.0-0ubuntu8.11 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

As per the test plan.
This package fixed the bug for me.

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package libvirt - 6.0.0-0ubuntu8.11

---------------
libvirt (6.0.0-0ubuntu8.11) focal; urgency=medium

  * d/p/ubuntu/lp-1906266-virStorageFileGetMetadataRecurse-Allow-format-probin:
    relax restrictions on format probing for compat with older images
    (LP: #1906266)

 -- Christian Ehrhardt <email address hidden> Tue, 05 Jan 2021 13:48:48 +0100

Hello Peter, or anyone else affected,

Accepted libvirt into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release

Hi @corey.bryant

I have tested:

libvirt-daemon-system:
  Installed: 6.0.0-0ubuntu8.11~cloud0
  Candidate: 6.0.0-0ubuntu8.11~cloud0
  Version table:
 *** 6.0.0-0ubuntu8.11~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status

As per the test plan.
This package fixed the bug for me.

I can confirm that installing a newer version of libvirt fixes the issue.

We just did this for an nova-libvirt container in the Openstack Victoria release, wich we had upgraded from Rocky.

We used the latest available libvirt version for Ubuntu 20.04: 6.0.0-0ubuntu8.12