Thanks Shubhakar Gowda P S (shubhakara) for getting back to us.
From journal.txt in your attached logs in comment #7 is looks like the 10 minute timeout is not strictly related to cloud-init, but cloud-init waiting on snaps being seeded.
I can see a snapd log in your journalctl -b 0 output that says snapd was timed out after an inordinate amount of time waiting for NTP:
Aug 25 08:55:42.976933 ubuntu-server snapd[1642]: devicemgr.go:1927: no NTP sync after 10m0s,
I also see a whole bunch of ntpd and network.timed
The systemd sevice which performs cloud configuration cloud-config.service has a clause which makes it wait until snapd.seeded.service completes in the booting environment so cloud-init use of any snap tools/services installed:
After=snapd.seeded.service
So this 10 minute wait is due to something in either networking setup in the ephemeral/early install environment not being configured appropriately for snapd to come up. Once snapd unblocks, cloud-init can complete.
You can confirm that the problem is snap seeding on your system with the commands we listed above in comments #2 and #4:
That should point to snap seeding being the problem in this early/ephemeral install environment.
Also in your journalctl -b 0 output (in cloud-init collect-logs /var/log/journal.txt)
A red flag I'm also seeing is repeated logs in your journal that look like thrashing of time-related services:
Aug 25 08:52:55.229686 ubuntu-server dbus-daemon[1630]: [system] Successfully activated service 'org.freedesktop.timedate1'
Aug 25 08:52:55.229931 ubuntu-server systemd[1]: Started Time & Date Service.
Aug 25 08:53:55.291021 ubuntu-server systemd[1]: systemd-timedated.service: Deactivated successfully.
Something is amiss with ntp or network configuration on this installation and it looks like it is causing issues for snapd which cause issue for cloud-init to make progress on the install of this system.
In checking /var/log/cloud-init-output.log I see that the two NICs are listed as link down:
ci-info: | eno1 | False | . | . | . | f4:ee:08:19:37:49 |
ci-info: | eno2 | False | . | . | . | f4:ee:08:19:37:4a |
But journalctl -b 0 showed them as "UP":
Aug 25 08:45:04.600250 ubuntu-server systemd-networkd[1594]: eno2: Link UP
Aug 25 08:45:04.635710 ubuntu-server systemd-networkd[1594]: eno1: Link UP
Something seems to be turning off the NICs on this device maybe?
Can you please provide:
1. Was the a customized install Ubuntu 22.04 installer ISO?
2. if public ISO can you provide the URL to the ISO used for install?
3. Run the separate commands and attach that output
- systemd-analyze blame
- systemd-analyze show
- systemd-analyze critical-chain
- systemctl --failed
- ls /etc/netplan
- cat /etc/netplan/*
- ip addr
- ip route
Thanks Shubhakar Gowda P S (shubhakara) for getting back to us.
From journal.txt in your attached logs in comment #7 is looks like the 10 minute timeout is not strictly related to cloud-init, but cloud-init waiting on snaps being seeded.
I can see a snapd log in your journalctl -b 0 output that says snapd was timed out after an inordinate amount of time waiting for NTP:
Aug 25 08:55:42.976933 ubuntu-server snapd[1642]: devicemgr.go:1927: no NTP sync after 10m0s,
I also see a whole bunch of ntpd and network.timed
The systemd sevice which performs cloud configuration cloud-config. service has a clause which makes it wait until snapd.seeded. service completes in the booting environment so cloud-init use of any snap tools/services installed:
After=snapd. seeded. service
So this 10 minute wait is due to something in either networking setup in the ephemeral/early install environment not being configured appropriately for snapd to come up. Once snapd unblocks, cloud-init can complete.
You can confirm that the problem is snap seeding on your system with the commands we listed above in comments #2 and #4:
- systemd-analyze blame
- systemd-analyze show
- systemd-analyze critical-chain
- systemctl --failed
That should point to snap seeding being the problem in this early/ephemeral install environment.
Also in your journalctl -b 0 output (in cloud-init collect-logs /var/log/ journal. txt)
A red flag I'm also seeing is repeated logs in your journal that look like thrashing of time-related services: p.timedate1' timedated. service: Deactivated successfully.
Aug 25 08:52:55.229686 ubuntu-server dbus-daemon[1630]: [system] Successfully activated service 'org.freedeskto
Aug 25 08:52:55.229931 ubuntu-server systemd[1]: Started Time & Date Service.
Aug 25 08:53:55.291021 ubuntu-server systemd[1]: systemd-
Something is amiss with ntp or network configuration on this installation and it looks like it is causing issues for snapd which cause issue for cloud-init to make progress on the install of this system.
In checking /var/log/ cloud-init- output. log I see that the two NICs are listed as link down:
ci-info: | eno1 | False | . | . | . | f4:ee:08:19:37:49 |
ci-info: | eno2 | False | . | . | . | f4:ee:08:19:37:4a |
But journalctl -b 0 showed them as "UP": networkd[ 1594]: eno2: Link UP networkd[ 1594]: eno1: Link UP
Aug 25 08:45:04.600250 ubuntu-server systemd-
Aug 25 08:45:04.635710 ubuntu-server systemd-
Something seems to be turning off the NICs on this device maybe?
Can you please provide:
1. Was the a customized install Ubuntu 22.04 installer ISO?
2. if public ISO can you provide the URL to the ISO used for install?
3. Run the separate commands and attach that output
- systemd-analyze blame
- systemd-analyze show
- systemd-analyze critical-chain
- systemctl --failed
- ls /etc/netplan
- cat /etc/netplan/*
- ip addr
- ip route