OVB CI - Virtual baremetals boot fail to bring up network - Ordering cycle found, skipping Network Manager

I suspect I am also hitting this issue. In my case it manifests itself with the provisioning bits not being able to ssh to one node (out of 9) because on that one node the heat-admin user is not there.

The reason the user is not there is because cloud-init.service did not even start.

Broken node:
[root@ctrl-2-0 mnt]# journalctl |grep -i ordering
Jun 07 08:02:09 localhost.localdomain systemd[1]: network-online.target: Found ordering cycle on network.target/start

Working node:
Jun 07 08:02:03 localhost.localdomain systemd[1]: sysinit.target: Found ordering cycle on nis-domainname.service/start
Jun 07 08:02:03 localhost.localdomain systemd[1]: sysinit.target: Found ordering cycle on nis-domainname.service/start
Jun 07 08:02:03 localhost.localdomain systemd[1]: sysinit.target: Job nis-domainname.service/start deleted to break ordering cycle starting with sysinit.target/start

Working:
root@ctrl-1-0 ~]# systemd-analyze verify multi-user.target
sysinit.target: Found ordering cycle on nis-domainname.service/start
sysinit.target: Found dependency on network-online.target/start
sysinit.target: Found dependency on network.target/start
sysinit.target: Found dependency on openvswitch.service/start
sysinit.target: Found dependency on ovs-vswitchd.service/start
sysinit.target: Found dependency on sysinit.target/start
sysinit.target: Job nis-domainname.service/start deleted to break ordering cycle starting with sysinit.target/start

Broken:
[root@ctrl-2-0 mnt]# systemd-analyze verify multi-user.target
network-online.target: Found ordering cycle on network.target/start
network-online.target: Found dependency on openvswitch.service/start
network-online.target: Found dependency on ovs-vswitchd.service/start
network-online.target: Found dependency on ovs-delete-transient-ports.service/start
network-online.target: Found dependency on sysinit.target/start
network-online.target: Found dependency on nis-domainname.service/start
network-online.target: Found dependency on network-online.target/start
network-online.target: Job network.target/start deleted to break ordering cycle starting with network-online.target/start
basic.target: Found ordering cycle on sockets.target/start
basic.target: Found dependency on sssd-kcm.socket/start
basic.target: Found dependency on sysinit.target/start
basic.target: Found dependency on nis-domainname.service/start
basic.target: Found dependency on network-online.target/start
basic.target: Found dependency on NetworkManager-wait-online.service/start
basic.target: Found dependency on NetworkManager.service/start
basic.target: Found dependency on basic.target/start
basic.target: Job sockets.target/start deleted to break ordering cycle starting with basic.target/start
NetworkManager-wait-online.service: Found ordering cycle on sysinit.target/start
NetworkManager-wait-online.service: Found dependency on nis-domainname.service/start
NetworkManager-wait-online.service: Found dependency on network-online.target/start
NetworkManager-wait-online.service: Found dependency on NetworkManager-wait-online.service/start
NetworkManager-wait-online.service: Job nis-domainname.service/start deleted to break ordering cycle starting with NetworkManager-wait-online.service/start

[root@ctrl-2-0 mnt]# rpm -q --changelog hostname |head -n2
* Thu May 06 2021 Pavel Zhukov <email address hidden> - 3.20-7
- Nisdomainname service depends on network

I'll try on my envs to downgrade hostnaem and see if that improves things. The bug against hostname is already filed at https://bugzilla.redhat.com/show_bug.cgi?id=1959720 and is potentially the culprit here.

There is a new build on the stream koji https://koji.mbox.centos.org/koji/buildinfo?buildID=17875
hostname-3.20-7.el8.0.1, which seems to fix it for me. No idea when/if that will reach the stream repos

@yatin and me were debugging this, We got a local reproducer, from testing looks like new hostname-3.20-7.el8.x86_64 is our culprit.

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1959720

From train: Difference between rpm working(24th) and affected(27th)

http://pastebin.test.redhat.com/969401

hostname were one of the updated packages
~~~
hostname-3.20-6.el8.x86_64 | hostname-3.20-7.el8.x86_64
~~~

Fix is available: https://git.centos.org/rpms/hostname/c/e097d2aac3e76eebbaac3ee4c2b95f575f3798fa?branch=c8s

from local testing looks like .. hostname downgrade solve the issue...

Trying with https://review.opendev.org/c/openstack/tripleo-quickstart/+/794636 to confirm in CI. Awaiting results

Upstream C8 stream mirror have already moved back from hostname-3.20-7.el8.x86_64 to hostname-3.20-6.el8.x86_64, waiting on mirror sync.

Woot.. thanks Harold and everyone else!

http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/BaseOS/x86_64/os/Packages/
hostname-3.20-6.el8...> 2019-07-02 13:22 32K

https://review.rdoproject.org/zuul/build/cc41082bfcd345349e09de7a5bcfd8d1

worked!!!

Change abandoned by "Sandeep Yadav <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/794636
Reason: This is not needed anymore.. infra rolled out previous version of hostname package and removed the new affected version.