MAAS can no longer commission machines with OpenVSwitch Bridges

I'm also able to reproduce this behavior.

- MAAS 3.3.2
- Commissioning Image: Ubuntu Focal

Console error shown during commissioning image boot:
---
[FAILED] Failed to start Open vSwitch Forwarding Unit.
See 'systemctl status ovs-vswitchd.service' for details.
[DEPEND] Dependency failed for Open vSwitch Record Hostname.
[ 37.201868] cloud-init[2033]: Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.
[ OK ] Stopped Open vSwitch Forwarding Unit.
         Starting Open vSwitch Forwarding Unit...
[FAILED] Failed to start Open vSwitch Forwarding Unit.
See 'systemctl status ovs-vswitchd.service' for details.
[DEPEND] Dependency failed for Open vSwitch Record Hostname.
[DEPEND] Dependency failed for Open vSwitch.
[ 37.458764] cloud-init[2033]: A dependency job for openvswitch-switch.service failed. See 'journalctl -xe' for details.
[ 37.459807] cloud-init[2033]: invoke-rc.d: initscript openvswitch-switch, action "start" failed.
[ 37.462900] cloud-init[2033]: ○ openvswitch-switch.service - Open vSwitch
[ 37.463269] cloud-init[2033]: Loaded: loaded (/lib/systemd/system/openvswitch-switch.service; enabled; vendor preset: enabled)
[ 37.463402] cloud-init[2033]: Active: inactive (dead)
[ 37.463711] cloud-init[2033]:
[ 37.465523] cloud-init[2033]: Aug 29 22:35:17 maas-large systemd[1]: Dependency failed for Open vSwitch.
[ 37.465911] cloud-init[2033]: Aug 29 22:35:17 maas-large systemd[1]: openvswitch-switch.service: Job openvswitch-switch.service/start failed with result 'dependency'.
[ 37.466121] cloud-init[2033]: dpkg: error processing package openvswitch-switch (--configure):
[ 37.466371] cloud-init[2033]: installed openvswitch-switch package post-installation script subprocess returned error exit status 1
[ 37.480777] cloud-init[2033]: Processing triggers for man-db (2.10.2-1) ...

---

I retested Focal with 5.4 and Bionic with 4.15 kernels and they also failed :-(

I found the following workaround:

- Commission without an OpenVSwitch bridge
- After the commissioning is done, remove the Linux Bridge and add the OpenVSwitch bridge
- Deploy

The linux-modules package has the kernel module file:

$ dpkg-deb -c linux-modules-5.15.0-69-generic_5.15.0-69.76~20.04.1_amd64.deb | grep openvswitch.ko
-rw-r--r-- root/root 870793 2023-03-20 12:32 ./lib/modules/5.15.0-69-generic/kernel/net/openvswitch/openvswitch.ko

Not present in the squashfs filesystem (expected):

$ wget http://images.maas.io/ephemeral-v3/stable/jammy/amd64/20230328/squashfs
$ unsquashfs squashfs
$ ls -l squashfs-root/lib/modules
total 0
$ grep 'Package: linux-modules' squashfs-root/var/lib/dpkg/status
$

Not in the initrd's shipped modules:

$ wget http://images.maas.io/ephemeral-v3/stable/jammy/amd64/20230328/ga-22.04/generic/boot-initrd
$ unmkinitramfs boot-initrd initrd-ga
$ find initrd-ga/ -name '*.ko' | wc -l
1622
$ find initrd-ga/ -name '*.ko' | grep openvswitch
$

Checked multiple GA images in stable stream, can confirm that there is no `openvswitch` module in them.

Candidate stream images for focal (20230907) and jammy (20230519) both contain the modules.

Test and promote recent candidate images to fix the issue in initramfs.

I pulled the candidate images into my local repository last night and just completed a round of commissioning & testing with a machine using an OpenVswitch bridge and everything worked.

Thank you so much for resolving this. It should be good to get pushed to production I believe.

Hi Alan,
Please, can you clarify if just one machine with vlan Open vSwitch is enough? Can I deploy the other machines with normal/linux bridge? I have six physical machines, and I'm trying to deploy them all with Open vSwitch but only one of them is ok, in the others the open vSwitch vlan has an error and I can't create VMs.

Can you confirm if that's right?

Thank you.