sr0 not available at generator timeframe causes cloud-init.target not run

Could you provide exact reproduction steps with virt-install and libvirt. I am attempting to reproduce locally with setups we normally use for testing, and am unable to:

1. downloaded image from https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img
2. created a simple cloud-init yaml file:

#cloud-config
password: <INSERT PASSWORD HERE>
chpasswd: { expire: False }
ssh_pwauth: True
ssh_import_id: jchittum
sudo: ALL=(ALL) NOPASSWD:ALL

3. using `cloud-localds` from `cloud-image-utils`, made an ISO of the cloud-config
cloud-localds cloud_init_with_pass.iso cloud-init.yaml

3. used qemu to test the image:

qemu-system-x86_64 \
  -cpu host -machine type=q35,accel=kvm -m 2048 \
  -nographic \
  -snapshot \
  -netdev id=net00,type=user,hostfwd=tcp::2222-:22 \
  -device virtio-net-pci,netdev=net00 \
  -drive if=virtio,format=qcow2,file=focal-server-cloudimg-amd64-disk-kvm.img \
  -drive if=virtio,format=raw,file=cloud_init_with_pass.iso

This qemu command sets the accel to kvm, and i had no issues. I'm guessing that the drive setup is very different though.

From my working knowledge of libvirt and cloud-init, you do need to mount the cloud-init image in a specific place, and I don't think there would be an issue, generally, with the kvm image not getting sr0 up fast enough. `qemu` is mounting to the same place in that command.

Could you provide the libvirt XML definition and exact reproduction steps for us to dig a little deeper?

I also haven't been able to reproduce on focal. It makes me think that there is a potential systemd unit ordering cycle on the image/config that represented this issue?

on focal I see the reverse deps on latest daily images:

root@dev-ff:~# systemctl list-dependencies --reverse cloud-init.target
cloud-init.target
● └─multi-user.target
● └─graphical.target
root@dev-ff:~# lsb_release -sc
focal

A guess in the dark would be to check is `journalctl -b 0` and look for "ordering cycle" related messages too.

able to reproduce with image created with

```
virt-install --connect qemu:///session \
--name cloudinit-test \
--memory 2048 \
--disk /home/achasen/tmp/focal.img,device=disk,bus=virtio \
--os-type linux \
--os-variant ubuntu20.04 \
--virt-type kvm \
--graphics none \
--network bridge=virbr0,model=virtio \
--import \
--disk /tmp/test.iso,device=cdrom,bus=sata
```

/run/cloud-init/cloud-init-generator.log indicated run around 0.69s:

```
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 0.69s] returning 1
```

jornalctl shows things like "Starting Network Service" before sr0 is in the log (which makes me think the sr0 is delayed). I didn't find anything in journalctl output related to the generator.

```
[ 1.857890] ubuntu systemd[1]: Starting Network Service...
...
[ 2.364539] ubuntu kernel: ata3: SATA link down (SStatus 0 SControl 300)
[ 2.364609] ubuntu kernel: ata5: SATA link down (SStatus 0 SControl 300)
[ 2.364642] ubuntu kernel: ata1.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[ 2.364643] ubuntu kernel: ata1.00: applying bridge limits
[ 2.364884] ubuntu kernel: ata1.00: configured for UDMA/100
[ 2.364350] ubuntu kernel: ata4: SATA link down (SStatus 0 SControl 300)
[ 2.364426] ubuntu kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl>
[ 2.364539] ubuntu kernel: ata3: SATA link down (SStatus 0 SControl 300)
[ 2.364609] ubuntu kernel: ata5: SATA link down (SStatus 0 SControl 300)
[ 2.364642] ubuntu kernel: ata1.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[ 2.364643] ubuntu kernel: ata1.00: applying bridge limits
[ 2.364884] ubuntu kernel: ata1.00: configured for UDMA/100
[ 2.365032] ubuntu kernel: scsi 0:0:0:0: CD-ROM QEMU QEMU DVD>
[ 2.365242] ubuntu kernel: sr 0:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa>
[ 2.365250] ubuntu kernel: cdrom: Uniform CD-ROM driver Revision: 3.20
[ 2.379293] ubuntu kernel: sr 0:0:0:0: Attached scsi CD-ROM sr0
[ 2.416795] ubuntu systemd[1]: Finished udev Wait for Complete Device Initia>
[ 2.417385] ubuntu systemd[1]: Starting Device-Mapper Multipath Device Contr>
```

I have the same issue with the Azure/Hyper-V Image. Running on local Windows desktop, using Hyper-V as the hypervisor.

Steps to reproduce:

1. Download and extract https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-azure.vhd.zip. Save disk image as 20.04-cloud.vhd.

2. Create my-seed.iso file almost exactly as described in cloud-init documentation. Only small tweak is saving as ISO instead of IMG. https://cloudinit.readthedocs.io/en/latest/topics/debugging.html

$ cat > user-data <<EOF
  #cloud-config
  password: passw0rd
  chpasswd: { expire: False }
  EOF
$ cloud-localds my-seed.iso user-data

3. Create new VM using Hyper-V GUI
- Virtual Hard Disk Image = 20.04-cloud.vhd
- Virtual DVD Drive Image = my-seed.iso

=> After starting the VM, I cannot log in.

Possibly helpful note: When using the standard (non-cloud) installer, this file seems to prevent the VM from using an ISO attached to the system: /etc/cloud/cloud.cfg.d/99-installer.cfg

It saves the user details that I manually entered during the install process, and critically, explicitly sets the data source to none.

$ cat /run/cloud-init/ds-identify.log
/etc/cloud/cloud.cfg.d/99-installer.cfg set datasource_list: [None]

After deleting the file, the ISO was recognized (and PW of "passw0rd" for ubuntu user worked)

$ cat /run/cloud-init/ds-identify.log
/etc/cloud/cloud.cfg.d/90_dpkg.cfg set datasource_list: [ NoCloud, ConfigDrive, OpenNebula, DigitalOcean, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, SmartOS, Bigstep, Scaleway, AliYun, Ec2, CloudStack, Hetzner, IBMCloud, Oracle, Exoscale, RbxCloud, UpCloud, Vultr, None ]

I do not know how to get debug output from the cloud image, because I cannot login as any user! If someone can explain how to do that, I would be happy to provide more output from the cloud image VM.

Hi Vincent,

Thank you for your comment. What you are seeing with the Azure cloud-images is not related with the current issue.

Azure VHDs you can find on c-i.u.c are the same images we publish on Azure Cloud. Those are configured with a single Cloud-Init datasource (Azure) to make the image boot faster. While it is possible to boot those images locally on hyper-v, you will end up with a VM that is not fully functional.

If you look carefully at the bug description, you will see that @achasen uses KVM images (not Azure images) that should work out of the box on KVM.

[Expired for cloud-init because there has been no activity for 60 days.]

[Expired for cloud-images because there has been no activity for 60 days.]

I apologize for the expiry on this bug it slipped through the cracks as it was set to incomplete status which eventually expires if not set back to New.

The reason we don't have cloud-init included in your boot target is due to the ds-identify generator not seeing the /dev/sr0 yet with a cidata label due to what appears to be a later module load.

Cloud-init can tell you on focal that it's disabled due to the generator-time failure to find a matching datasource.

root@focal:~# cloud-init status --long
status: disabled
detail:
Cloud-init disabled by cloud-init-generator

I am able to reproduce the original error with the following steps and as Adam suggested:
$ sudo virt-install --connect qemu:///session --name cloudinit-test --memory 2048 --disk /home/csmith/src/cloud-init/focal-server-cloudimg-amd64-disk-kvm.img,device=disk,bus=virtio --os-type linux --os-variant ubuntu20.04 --virt-type kvm --graphics none --network bridge=virbr0,model=virtio --import --disk "/tmp/test.iso,device=cdrom,bus=sata"

 On Focal, we can see /run/cloud-init/ds-identify.log which is emitted when cloud-init's generator runs beats the journalctl -b 0 timing of when the /dev/sr0 is seen due to later kernel module load.

from journalctl:

   Feb 24 21:56:28 ubuntu kernel: sr 0:0:0:0: Attached scsi CD-ROM sr0

root@focal:~# ls -ltr --full-time /dev/disk/by-label/ /run/cloud-init/ds-identify.log
# Generator time 21:56:27
-rw-r--r-- 1 root root 1504 2022-02-24 21:56:27.241872017 +0000 /run/cloud-init/ds-identify.log

# /dev/sr0 availability no until 1 second later
/dev/disk/by-label/:
total 0
lrwxrwxrwx 1 root root 10 2022-02-24 21:56:28.173872017 +0000 cloudimg-rootfs -> ../../vda1
lrwxrwxrwx 1 root root 9 2022-02-24 21:56:28.441872017 +0000 cidata -> ../../sr0
lrwxrwxrwx 1 root root 11 2022-02-24 21:56:28.581872017 +0000 UEFI -> ../../vda15

This needs a bit more investigation and probably can be worked around with add the virt-install argument `--sysinfo system.serial='ds=nocloud'` which will force ds-identify to detect NoCloud regardless of the presence of /dev/sr0. Since the device will be up before NoCloud.get_data is run, this will avoid the race.

A duplicate bug, https://bugs.launchpad.net/bugs/1961832 , provides some additional context and consistent reproduction steps.

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3897