cloud-init

jammy-server-cloudimg-amd64-disk-kvm.img NoCloud data source not detected with qemu SATA cdrom

Bug #1961832 reported by Cole Robinson
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cloud-init
New
Undecided
Unassigned
cloud-init (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

cloud-init running in ubuntu jammy-server-cloudimg-amd64-disk-kvm.img does not detect NoCloud cidata ISO in a qemu VM that uses emulated SATA cdrom drive.

SATA is the default used by qemu's -M q35 machine type, which is the preferred modern config over the default `-M pc` machine type, which uses IDE. Using -M pc and therefore IDE _does_ work.

Using the the non-KVM image also _does_ work (jammy-server-cloudimg-amd64.img)

Debugging this a bit showed that cloud-init `ds-identify --force` correctly finds the ISO once I've logged in, but not at boot up, I think because the sata cdrom is discovered by udev too late. This also affects ubuntu20.04 cloud images similarly: -disk-kvm is busted, other image is fine.

Maybe it's an initrd driver issue, or issue with bits being modularized when they shouldn't, not sure. Here's the loaded driver difference for -M pc and -M q35 with -kvm-disk.img

root@ubuntu:~# diff -rup pc q35
--- pc 2022-02-22 19:25:11.731067208 +0000
+++ q35 2022-02-22 19:26:55.749459099 +0000
@@ -7,7 +7,8 @@ loop 28672 6
 dm_multipath 24576 0
 dm_mod 86016 3 dm_multipath
 kvm_intel 200704 0
-pata_acpi 12288 0
+ahci 36864 0
+libahci 24576 1 ahci
 fuse 106496 1
 configfs 32768 1
 ip_tables 24576 0

Some background: I'm upstream virt-manager/virt-install dev. virt-install provides a `--cloud-init` option where we attach a NoCloud formatted ISO. A user reported this doesn't work out of the box with ubuntu20.04 -kvm-disk.img: https://github.com/virt-manager/virt-manager/issues/359

This isn't virt-install --cloud-init specific, see this report which pointed me towards the -kvm-disk difference: https://stackoverflow.com/questions/68512432/cannot-login-to-vm-using-qemu-nocloud-cloud-init-on-macos-with-ubuntu-cloud-imag

Though virt-install users are kinda more likely to hit this since we increasingly default to q35. If users are just testing with qemu-kvm directly without specifying -machine q35, they won't hit this.

Reproducer with virt-install:
git clone https://github.com/virt-manager/virt-manager
cd virt-manager
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-disk-kvm.img
./virt-install --osinfo ubuntu20.04 --disk jammy-server-cloudimg-amd64-disk-kvm.img --cloud-init

Revision history for this message
Cole Robinson (crobinso) wrote :

The reporter of the virt-install bug also found this expired focal cloud-images bug which sounds like the same root issue: https://bugs.launchpad.net/cloud-images/+bug/1940791

If someone with more launchpad knowledge can route this report to the correct place, please do. Would be nice to get this straightened out before 22.04 is released.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi Cole, FYI I have yesterday pinged the cloud-init develpers about both to check these cases.
I also added the extra bug task here.
And for bug 1940791 I have mentioned that I think it might have expired without a real conclusion.

Revision history for this message
James Falcon (falcojr) wrote :

Thanks for the detailed bug report.

I used your reproducer steps, but I got a slightly different result. I'm still not getting the NoCloud identification, but in my case, I can't force ds-identify to identify it. In order for ds-identify to identify NoCloud, it needs to see a "cidata" filesystem label somewhere (or see the relevant kernel args), but "cidata" is not showing up on the sr0 device:

root@ubuntu:/run/cloud-init# lsblk -o name,mountpoint,label
NAME MOUNTPOINT LABEL
loop0 /snap/core20/1328
loop1 /snap/snapd/14549
loop2 /snap/lxd/22340
sr0
vda
├─vda1 / cloudimg-rootfs
├─vda14
└─vda15 /boot/efi UEFI

Have you seen this as well, or have I missed a step in trying to reproduce this?

Revision history for this message
Cole Robinson (crobinso) wrote :

Thanks for testing James. But that's different from what I'm seeing.

True full reproducer for me:

Host: Fedora 35

git clone https://github.com/virt-manager/virt-manager
cd virt-manager
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-disk-kvm.img
# change image root password to 'root'
virt-customize -a jammy-server-cloudimg-amd64-disk-kvm.img --root-password password:root
./virt-install --osinfo ubuntu20.04 --disk jammy-server-cloudimg-amd64-disk-kvm.img --cloud-init

<console spew>

root@ubuntu:~# cat /run/cloud-init/ds-identify.log | grep FS_LABELS
FS_LABELS=UEFI,UEFI,cloudimg-rootfs
root@ubuntu:~# DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force | grep FS_LABELS
FS_LABELS=cidata,UEFI,UEFI,cloudimg-rootfs

ds-identify.log is from cloud-init boot up, FS_LABELS shows that cidata wasn't detected.
But when re-running with --force, once logged in, the label is showing up.

If that's not reproducing for you I can come probably wrangle up a straight qemu reproducer too

Revision history for this message
James Falcon (falcojr) wrote :

Chad Smith was able to reproduce this while investigating #1940791, and this appears to be the same root cause as #1940791, so I'm going to mark this one as a duplicate.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cloud-init (Ubuntu):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.