Ubuntu
firewalld package

cloud-init.service/start deleted to break ordering cycle

Bug #1956629 reported by Rossen
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Confirmed
Undecided
Unassigned
firewalld (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

I'm encountering this systemd service dependecy cycle, which prevent cloud-init from running, and making my image build unusable since user-data settings do not get applied.

Issues present on bionic and focal

Steps to reproduce:

1. apt-get purge -y ufw; apt-get install -y firewalld
2. cloud-init clean
3. reboot
4. grep -B 5 cycle /var/log/syslog

I've seen these two error messages:

```
[ 3.609277] systemd[1]: Set hostname to <localhost>.
[ 3.620361] systemd[1]: Initializing machine ID from KVM UUID.
[ 3.621151] systemd[1]: Installed transient /etc/machine-id file.
[ 4.007739] systemd[1]: basic.target: Found ordering cycle on paths.target/start
[ 4.017890] systemd[1]: basic.target: Found dependency on apport-autoreport.path/start
[ 4.018837] systemd[1]: basic.target: Found dependency on sysinit.target/start
[ 4.019713] systemd[1]: basic.target: Found dependency on cloud-init.service/start
[ 4.020613] systemd[1]: basic.target: Found dependency on systemd-networkd-wait-online.service/start
[ 4.021700] systemd[1]: basic.target: Found dependency on systemd-networkd.service/start
[ 4.022677] systemd[1]: basic.target: Found dependency on network-pre.target/start
[ 4.023574] systemd[1]: basic.target: Found dependency on firewalld.service/start
[ 4.024463] systemd[1]: basic.target: Found dependency on basic.target/start
[ 4.025298] systemd[1]: basic.target: Job paths.target/start deleted to break ordering cycle starting with basic.target/start
[[0;1;31m SKIP [0m] Ordering cycle found, skipping [0;1;39mPaths[0m
[ 4.027404] systemd[1]: firewalld.service: Found ordering cycle on sysinit.target/start
[ 4.028367] systemd[1]: firewalld.service: Found dependency on cloud-init.service/start
[ 4.029359] systemd[1]: firewalld.service: Found dependency on systemd-networkd-wait-online.service/start
[ 4.034786] systemd[1]: firewalld.service: Found dependency on systemd-networkd.service/start
[ 4.035843] systemd[1]: firewalld.service: Found dependency on network-pre.target/start
[ 4.036853] systemd[1]: firewalld.service: Found dependency on firewalld.service/start
[ 4.037878] systemd[1]: firewalld.service: Job cloud-init.service/start deleted to break ordering cycle starting with firewalld.service/start
[[0;1;31m SKIP [0m] Ordering cycle found, skipping [0;���job (metadata service crawler)[0m
[ 4.041631] systemd[1]: Created slice system-modprobe.slice.
```

```
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.824604] systemd[1]: sysinit.target: Found ordering cycle on cloud-init.service/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.825678] systemd[1]: sysinit.target: Found dependency on systemd-networkd-wait-online.service/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.826718] systemd[1]: sysinit.target: Found dependency on systemd-networkd.service/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.827613] systemd[1]: sysinit.target: Found dependency on network-pre.target/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.828450] systemd[1]: sysinit.target: Found dependency on firewalld.service/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.836078] systemd[1]: sysinit.target: Found dependency on dbus.service/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.836928] systemd[1]: sysinit.target: Found dependency on basic.target/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.837814] systemd[1]: sysinit.target: Found dependency on paths.target/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.838762] systemd[1]: sysinit.target: Found dependency on ua-license-check.path/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.839750] systemd[1]: sysinit.target: Found dependency on sysinit.target/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.840612] systemd[1]: sysinit.target: Job cloud-init.service/start deleted to break ordering cycle starting with sysinit.target/start
```

Rossen (rgpio)
affects: cloud-init → cloud-init (Ubuntu)
Revision history for this message
Chad Smith (chad.smith) wrote :

Thank you for filing a bug and making Ubuntu better.

I can confirm that your procedure to reproduce on Ubuntu jammy. This does in fact represent a bug in how the firewalld systemd unit is being setup. My first guess here is the DefaultDependencies being injected by firewalld's systemd service is what is causing the ordering cycle w/ cloud-init.
It will need a bit more time to triage. But I can at least confirm the bug report.

Changed in firewalld (Ubuntu):
status: New → Confirmed
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cloud-init (Ubuntu):
status: New → Confirmed
Revision history for this message
Chad Smith (chad.smith) wrote :

This looks to also have an existing issue over in firewalld too https://github.com/firewalld/firewalld/issues/414

Revision history for this message
Ross Vandegrift (ross-kallisti) wrote :

The Debian cloud team received #[1025618](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1025618), which is a report of this bug.

I tried to follow the suggestion at https://github.com/firewalld/firewalld/issues/414#issuecomment-1061607756 but it's not quite right. It suggests dropping `Before=network-pre.target` from `cloud-init-local.service`. But:
- `cloud-init-local.service` has `Before=network-pre.target`, but is not involved in the cycles. The cycle details posted above only involve `cloud-init.service`.
   - the debian bug has a nice python script to find cycles in `systemd-analyze dot --order` output.
- `cloud-init.service` has no explicit order with `network-pre.target`, but it does have `Before=sysinit.target`.

The closest I could get to Michael's suggestion is dropping `Before=sysinit.target` from `cloud-init.service` - but this will break cloud-init's disk setup features (see cloud-init rev `80f5ec4b` which introduces `Before=basic.target`).

Could the disk and network setup pieces of cloud init be separated? That'd allow keeping disk setup before `sysinit.target`, but permitting network setup to move later (though I suspect that brings risks too, LP#1636912 in particular?)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.