libvirt 2.5.0-3ubuntu5.6~cloud0 appears to be compiled without gnutls

Hi,
first of all thanks Lee for the update and your help to spot and fix another issue.

So this is 2.5 which was based on the version in Zesty IIRC.
There are no build logs anymore as zesty is out of support already.

So I checked the others first.
Xenial (1.3.1):
checking for GNUTLS... yes
checking for gnutls/crypto.h... yes
configure: gnutls: -I/usr/include/p11-kit-1 -lgnutls

Artful (3.6):
checking for GNUTLS... yes
checking for gnutls/crypto.h... yes
checking for gnutls_rnd... yes
checking for gnutls_cipher_encrypt... yes
configure: gnutls: yes (CFLAGS='-I/usr/include/p11-kit-1' LIBS='-lgnutls')

Ok, good on those two.
So I really need to dig towards the 2.5 version
IIRC there was a bug in gnutls which broke libvirt completely
2.1.0-1ubuntu12 fixed the related FTBFS
2.1.0-1ubuntu13 then depended on the newer fixed gnutls
2.5.0-3ubuntu5.6 is the last of these zesty builds in the main archive
Zesty (2.5):
checking for GNUTLS... yes
checking for gnutls/crypto.h... yes
checking for gnutls_rnd... yes
checking for gnutls_cipher_encrypt... yes
configure: gnutls: yes (CFLAGS='-I/usr/include/p11-kit-1' LIBS='-lgnutls')

But knowing that there was an issue in gnutls causing a FTBFS in libvirt, and that it relied on a newer gnutls might imply that on the cloud archive build this was reverted/dropped - as it most likely has to build against the Xenial version of the lib.

So I checked the cluod-archive build of 2.5.0-3ubuntu5.6~cloud0 at:
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/ocata-staging/+packages
But that has it as well:
checking for GNUTLS... yes
checking for gnutls/crypto.h... yes
checking for gnutls_rnd... yes
checking for gnutls_cipher_encrypt... yes
configure: gnutls: yes (CFLAGS='-I/usr/include/p11-kit-1' LIBS='-lgnutls')

hmm ...?

I subscribe the cloud-archive as well so that our OpenStack folks are aware of this as well.

@Lee - could you describe a sample configuration that lead to you this error?

@OpenStack Team - do you see that as well? Do we have such a (?LUKS?) setup that would trigger it?

So the issue only appears with the Ocata UCA repo that provides the 2.5.0-3ubuntu5.6~cloud0 build. Why we are installing this in the OpenStack CI upgrade jobs between Pike and Queens is beyond me but something we should be able to correct outside of this bug.

Anyway, here are some notes on reproducing this using the Ocata UCA repo :

- Update libvirt using the Ocata UCA repo

# add-apt-repository cloud-archive:ocata
# apt update
# apt upgrade
# dpkg -l | grep libvirt
ii libvirt-bin 2.5.0-3ubuntu5.6~cloud0 amd64 programs for the libvirt library
ii libvirt-clients 2.5.0-3ubuntu5.6~cloud0 amd64 Programs for the libvirt library
ii libvirt-daemon 2.5.0-3ubuntu5.6~cloud0 amd64 Virtualization daemon
ii libvirt-daemon-system 2.5.0-3ubuntu5.6~cloud0 amd64 Libvirt daemon configuration files
ii libvirt0:amd64 2.5.0-3ubuntu5.6~cloud0 amd64 library for interfacing with different virtualization systems
ii python-libvirt 3.0.0-2~cloud0 amd64 libvirt Python bindings

- Create a LUKS encrypted disk and associated secret containing the required passphrase

# qemu-img create -f luks --object secret,id=luks_sec,format=raw,data=1234 -o key-secret=luks_sec /var/lib/libvirt/images/test.img 1G

# cat sec.xml
<secret ephemeral='no' private='yes'>
   <uuid>8b0c5785-9e28-4c39-8b5d-21b42446ce35</uuid>
   <usage type='volume'>
      <volume>/var/lib/libvirt/images/test.img</volume>
   </usage>
</secret>
# virsh secret-define sec.xml
# virsh secret-set-value 8b0c5785-9e28-4c39-8b5d-21b42446ce35 $(printf %s "1234" | base64)

- Create a test domain to attach this disk to

wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img -O /var/lib/libvirt/images/cirros-0.4.0-x86_64-disk.img
virt-install --import --name test --vcpus 1 --ram 512 --disk path=/var/lib/libvirt/images/cirros-0.4.0-x86_64-disk.img --nographic

- Create the associated XML for the disk and attempt to attach it to the domain

# cat disk.xml
<disk type="file" device="disk">
 <driver name="qemu" type="qcow2" cache="none" io="native"/>
 <source file="/var/lib/libvirt/images/test.img"/>
 <target dev='vdb' bus='virtio'/>
 <encryption format='luks'>
  <secret type='passphrase' uuid='8b0c5785-9e28-4c39-8b5d-21b42446ce35'/>
 </encryption>
</disk>

# virsh attach-device test disk.xml
error: Failed to attach device from disk.xml
error: unsupported configuration: luks encryption requires encrypted secrets to be supported

- Upgrade to the Pike UCA and Libvirt 3.6.0:

# add-apt-repository cloud-archive:pike
# apt update
# apt upgrade

# dpkg -l | grep libvirt
ii libvirt-bin 3.6.0-1ubuntu6~cloud0 amd64 programs for the libvirt library
ii libvirt-clients 3.6.0-1ubuntu6~cloud0 amd64 Programs for the libvirt library
ii libvirt-daemon 3.6.0-1ubuntu6~cloud0 amd64 Virtual...

I just checked and we don't patch libvirt for the Ocata cloud-archive backport. Do we know if this worked on Zesty? Any chance we still can test Zesty now that it's EOL?

Since archives of Zesty are gone this is hard, you can't install packages anymore you don't have yet. The closest are Artful and Xenial.
But Lee already tested 3.6 which is more or less what is in Artful.

There is quite a chance that this is related to the gnutls FTBFS we had in zesty, as I outlined in comment #1 - but the build itself looks good.

Maybe it is time to let the CI move up to a supported repo.
@Corey - does UCA-O still have support far longer than Zesty that is EOL now?
@Corey - you might need to drill down the Ocata build or use ddebs from UCA to step through the function if the ecnryption support is really missing.

A couple of things to note about the failing code path.

The error message "luks encryption requires encrypted secrets to be supported" is coming from qemuDomainSecretDiskPrepare() in src/qemu/qemu_domain.c: https://paste.ubuntu.com/26444342/

The call to qemuDomainSecretSetup() appears to be returning 0 (zero). Whether that means virCryptoHaveCipher() is returning True or False is hard to tell but based on gnutls being included in the build log I'd have to assume HAVE_GNUTLS_CIPHER_ENCRYPT is true.

What about the other checks in the first if statement in qemuDomainSecretSetup()? https://paste.ubuntu.com/26444411/ Pasting here as well. It seems as if one of these checks in the first if is failing and we don't get to the qemuDomainSecretAESSetup() call, but instead take the else path.

static int
qemuDomainSecretSetup(virConnectPtr conn,
                      qemuDomainObjPrivatePtr priv,
                      qemuDomainSecretInfoPtr secinfo,
                      const char *srcalias,
                      virSecretUsageType secretUsageType,
                      const char *username,
                      virSecretLookupTypeDefPtr seclookupdef,
                      bool isLuks)
{
    if (virCryptoHaveCipher(VIR_CRYPTO_CIPHER_AES256CBC) &&
        virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_OBJECT_SECRET) &&
        (secretUsageType == VIR_SECRET_USAGE_TYPE_CEPH ||
         secretUsageType == VIR_SECRET_USAGE_TYPE_VOLUME ||
         secretUsageType == VIR_SECRET_USAGE_TYPE_TLS)) {
        if (qemuDomainSecretAESSetup(conn, priv, secinfo, srcalias,
                                     secretUsageType, username,
                                     seclookupdef, isLuks) < 0)
            return -1;
    } else {
        if (qemuDomainSecretPlainSetup(conn, secinfo, secretUsageType,
                                       username, seclookupdef) < 0)
            return -1;
    }
    return 0;
}

Comparing 'pull-lp-source libvirt artful' vs 'pull-lp-source libvirt zesty', I'm seeing artful has the following in the configure script and zesty doesn't:

#define HAVE_GNUTLS_CIPHER_ENCRYPT 1

This was created by:
https://libvirt.org/git/?p=libvirt.git;a=commit;h=680d2f49dad425395de627a31006cb84848cfa65

Which just is in 2.5 (that you have).
But there is an issue in that.
Later on fixed by several changes.
I had to parse through them for a while, but I think the following will fix your issue:

https://libvirt.org/git/?p=libvirt.git;a=commit;h=ba5d9dd4768f3d3d67d9eace7b6de27a2799f458
https://libvirt.org/git/?p=libvirt.git;a=commit;h=aac34eca90f37c4510b49ad85029c07c57bedb8d

That would make the detection work again fine, and due to that the feature be correctly handled again.
They apply to the last version in zesty with an offset of -4 and no complains otherwise.

This changes the .ac files, but d/rules calls --with autoreconf so I'd hope it rewrites that.
Can you give that a try in a ppa Corey?

Thanks Christian. I'll prep a PPA for Ocata and report back here when it's ready.

Lee, thanks very much for the recreate details. I was able to recreate the error you were seeing, and also tested successfully with libvirt installed from ppa:corey.bryant/xenial-ocata-bug1744758.

$ apt policy libvirt-bin
libvirt-bin:
  Installed: 2.5.0-3ubuntu5.6~cloud1~ubuntu16.04.1~ppa201801231058
  Candidate: 2.5.0-3ubuntu5.6~cloud1~ubuntu16.04.1~ppa201801231058
  Version table:
 *** 2.5.0-3ubuntu5.6~cloud1~ubuntu16.04.1~ppa201801231058 500
        500 http://ppa.launchpad.net/corey.bryant/xenial-ocata-bug1744758/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status

$ cat /tmp/disk.xml
<disk type="file" device="disk">
 <driver name="qemu" type="qcow2" cache="none" io="native"/>
 <source file="/var/lib/libvirt/images/test.img"/>
 <target dev='vdb' bus='virtio'/>
 <encryption format='luks'>
  <secret type='passphrase' uuid='8b0c5785-9e28-4c39-8b5d-21b42446ce45'/>
 </encryption>
</disk>

$ sudo virsh attach-device test /tmp/disk.xml
Device attached successfully

Since this seems to be working I'll go ahead and get this uploaded to ocata-proposed for more testing.

Hello lyarwood, or anyone else affected,

Accepted libvirt into ocata-proposed. The package will build now and
be available in the Ubuntu Cloud Archive in a few hours, and then in the
-proposed repository.

Please help us by testing this new package. To enable the -proposed
repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-ocata-needed to verification-ocata-done. If it does
not fix the bug for you, please add a comment stating that, and change
the tag to verification-ocata-failed. In either case, details of your
testing will help us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
advance!

Thanks Corey, Christian, the new build WORKSFORME!

# virsh attach-device test disk.xml
error: Failed to attach device from disk.xml
error: unsupported configuration: luks encryption requires encrypted secrets to be supported
# add-apt-repository cloud-archive:ocata-proposed
# apt-get update
# apt-get upgrade
# systemctl restart libvirtd
# virsh attach-device test disk.xml
Device attached successfully
# dpkg -l | grep libvirt
ii libvirt-bin 2.5.0-3ubuntu5.6~cloud1 amd64 programs for the libvirt library
ii libvirt-clients 2.5.0-3ubuntu5.6~cloud1 amd64 Programs for the libvirt library
ii libvirt-daemon 2.5.0-3ubuntu5.6~cloud1 amd64 Virtualization daemon
ii libvirt-daemon-system 2.5.0-3ubuntu5.6~cloud1 amd64 Libvirt daemon configuration files
ii libvirt0:amd64 2.5.0-3ubuntu5.6~cloud1 amd64 library for interfacing with different virtualization systems
ii python-libvirt 3.0.0-2~cloud0 amd64 libvirt Python bindings

\o/ Great, thanks for verifying Lee. This needs to spend a little bit more time in proposed and get some regression testing and then we'll promote to ocata-updates.

Regression testing on xenial-ocata-proposed was successful:

Tempest smoke with development charms:

======
Totals
======
Ran: 102 tests in 1735.5668 sec.
 - Passed: 94
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 1017.3351 sec.

Tempest smoke with stable charms:

======
Totals
======
Ran: 102 tests in 1735.5668 sec.
 - Passed: 94
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 1017.3351 sec.

Wrong paste for development charms above.

Tempest smoke with development charms:

======
Totals
======
Ran: 102 tests in 1631.0851 sec.
 - Passed: 94
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 955.7162 sec.

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package libvirt - 2.5.0-3ubuntu5.6~cloud1
---------------

 libvirt (2.5.0-3ubuntu5.6~cloud1) xenial-ocata; urgency=medium
 .
   * d/p/ubuntu/0001-gnutls-use-AC_CHECK_FUNCS.patch,
     d/p/ubuntu/0002-gnutls-properly-backup-CFLAGS-and-LIBS.patch:
     Enable HAVE_GNUTLS_CIPHER_ENCRYPT (LP: #1744758).