uvt-kvm: error: libvirt: Requested operation is not valid: network 'default' is not active

Thanks for reporting this bug. Could you please provide a link to the test code which is failing?

I tried to launch a VM using uvtool in Kinetic and I was able to reproduce the issue:

root@uvt-test:~# cat /etc/os-release | grep CODENAME
VERSION_CODENAME=kinetic
UBUNTU_CODENAME=kinetic
root@uvt-test:~# dpkg -l uvtool
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================-============-===============================================
ii uvtool 0~git178-0ubuntu1 all Library and tools for using Ubuntu Cloud images
root@uvt-test:~# dpkg -l uvtool-libvirt
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================-============-======================================================>
ii uvtool-libvirt 0~git178-0ubuntu1 all Library and tools for using Ubuntu Cloud Images with l>
root@uvt-test:~# uvt-simplestreams-libvirt sync release=jammy arch=amd64
root@uvt-test:~# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:r+vhHiNR5Q8NsyGUF10dqTi9jNV1kDD/bPvKoWHTcVk root@uvt-test
The key's randomart image is:
+---[RSA 3072]----+
| .o.*oooo=o|
| .+.*.o..+|
| ..+ + + E|
| . = + +o|
| . S * o.=|
| . . . + +.|
| . + . + o. |
| o = . = ..|
| o*. . o..|
+----[SHA256]-----+
root@uvt-test:~# uvt-kvm create test
uvt-kvm: error: libvirt: Requested operation is not valid: network 'default' is not active

I am adding a task for uvtool and marking it as Triaged.

Hi
here is the code we use for this test:
https://git.launchpad.net/~canonical-kernel-team/+git/autotest-client-tests/tree/ubuntu_kvm_smoke_test/the-test

Which will be executed with something like the following on an amd64 system:
sudo -u ubuntu /PATH_TO_FILE/the-test amd64

That would be all inside libvirt, uvtool is just a consumer that fails by not being able to use it.
But it worked a while ago just fine - working theory: something else has landed breaking it.

I'll set up a VM and need to debug the postinst of libvirt which is where this happens.

This will break more people and more use cases if it persists.

The relevant section in postinst output is:

Default libvirt network on 192.168.122.1/24 already taken
Changing to free 192.168.123.1/24
Enabling libvirt default network

1. It realized it would have a conflict (since it runs in another VM with default net)
2. It switched to a new network pattern that is free.
3. it enables the default network

That enabling is done using libvirts autostart feature:

set_autostart()
{
    echo "Enabling libvirt default network"
    if [ ! -e /etc/libvirt/qemu/networks/autostart/default.xml ]; then
        ln -s /etc/libvirt/qemu/networks/default.xml \
              /etc/libvirt/qemu/networks/autostart/
    fi
}

So there is issue #1.
The call to "maybe_set_autostart" needs to happen before all the service (re)starts.
That allows libvirt to pick up that the config was changed.
This was modified recently in debian and on merging did place it after the service handling.

Note: Upgrades are not affected, nor are any places that start the default network. We just miss the comfort that it is started right after installation of libvirt.

But there seems to be something else, even when libvirtd is restarted.

It recognizes that it is configured as autostart
 Name State Autostart Persistent
----------------------------------------------
 default inactive no yes

after restart then

 Name State Autostart Persistent
----------------------------------------------
 default inactive yes yes

So is it not activating it despite autostart ... hmm.

To be clear - a start works fine, no issue with it:

$ virsh net-start default
Network default started

But why not automatically ...?

On a system boot the network is correctly auto-started.

That behavior might be related to:
- https://gitlab.com/libvirt/libvirt/-/commit/bab464f8ea54d177e163f32c7c3476220694665c
- https://gitlab.com/libvirt/libvirt/-/commit/ee16a195d97315ba3610b3640d347d3f4a358b55
- https://bugzilla.redhat.com/show_bug.cgi?id=1755303

So only the "very first" start after boot counts for auto-start.

This can be verified to be the reason:
ubuntu@k:~$ sudo systemctl stop libvirtd
Warning: Stopping libvirtd.service, but it can still be activated by:
  libvirtd.socket
  libvirtd-admin.socket
  libvirtd-ro.socket
ubuntu@k:~$ sudo rm /var/run/libvirt/network/autostarted
ubuntu@k:~$ sudo systemctl start libvirtd
ubuntu@k:~$ virsh net-list --all
 Name State Autostart Persistent
--------------------------------------------
 default active yes yes

So it does start correctly on the first invocation (after boot).
Conceptually that is ok for what we need from the postinst, we only do the network-magic+set-autostart on initial install. On upgrades we keep whatever was there anyway.

So the move of the code in the postinst should be enough to fix this.

proposed fix created:
- PPA: https://launchpad.net/~paelzer/+archive/ubuntu/lp-1990853-autostart-ordering
- MP: https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/430609

Verified, the PPA fixes the issue as expected.
After install we are again back having an active default network.

Also the MP got reviewed.

Since this might break some beta-testers I'm uploading this to kinetic-proposed despite the freeze.
I'm happy if it will be stuck in -unapproved, but chances are the release team wants to accept it to avoid this breakage affecting the beta-tests.

The release team is going to let this upload wait until after the Beta.

This bug was fixed in the package libvirt - 8.6.0-0ubuntu2

---------------
libvirt (8.6.0-0ubuntu2) kinetic; urgency=medium

  * d/p/libvirt-daemon-system.postinst: default network autostart
    handling needs to happen before services start (LP: #1990853)

 -- Christian Ehrhardt <email address hidden> Wed, 28 Sep 2022 08:36:15 +0200