fs064 tests are failing on master and wallaby - FATAL | try modifying forward dns record | undercloud | error={"changed": false, "msg": "`arecord` not found."

Comment from security team:

<dwilde> rlandy: thanks, it looks like https://opendev.org/x/tripleo-ipa/src/branch/master/tripleo_ipa/roles/tripleo_ipa_dns/tasks/dns.yaml#L57 is failing because the task isn’t parsing the zone item properly

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master&skip=0

This test has been failing with 08/30

On most (all?) jobs this error happens inconsistently though. For example, after periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master [2] had failed two times on this arecord issue, it passed [0].

Besides fs64 this issue is also affecting other jobs:
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master [1]
* periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master [3][4]
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master [5][6]

[0] https://review.rdoproject.org/zuul/build/63a7bd6281634855a5260c2a9c56e197

[1] https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6ea2096/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[2] https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/0c48b0f/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz

[3] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/a0594a2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[4] https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/8226ab0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[5] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/545f2f1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[6] https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/40d3797/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

I don't see how this error could be a cause for the failure of this job.

The code that is being executed is here:

https://opendev.org/x/tripleo-ipa/src/branch/master/tripleo_ipa/roles/tripleo_ipa_dns/tasks/dns.yaml#L50-L78

Essentially, the code attempts to modify the existing DNS entries, and if that fails, then in the rescue section adds the required entry.

We see this in the logs.

2022-08-31 18:42:18 | 2022-08-31 18:42:18.080724 | fa163edd-246c-3638-fe65-000000005895 | TIMING | tripleo_ipa_dns : set record type | undercloud | 0:14:39.445888 | 0.04s
2022-08-31 18:42:18 | 2022-08-31 18:42:18.089570 | fa163edd-246c-3638-fe65-000000005896 | TASK | add dns zone
2022-08-31 18:42:18 | 2022-08-31 18:42:18.929152 | fa163edd-246c-3638-fe65-000000005896 | OK | add dns zone | undercloud
2022-08-31 18:42:18 | 2022-08-31 18:42:18.931318 | fa163edd-246c-3638-fe65-000000005896 | TIMING | tripleo_ipa_dns : add dns zone | undercloud | 0:14:40.296473 | 0.84s
2022-08-31 18:42:18 | 2022-08-31 18:42:18.943993 | fa163edd-246c-3638-fe65-000000005898 | TASK | try modifying forward dns record
2022-08-31 18:42:19 | 2022-08-31 18:42:19.889906 | fa163edd-246c-3638-fe65-000000005898 | FATAL | try modifying forward dns record | undercloud | error={"changed": false, "msg": "`arecord` not found."}
2022-08-31 18:42:19 | 2022-08-31 18:42:19.891074 | fa163edd-246c-3638-fe65-000000005898 | TIMING | tripleo_ipa_dns : try modifying forward dns record | undercloud | 0:14:41.256232 | 0.95s
2022-08-31 18:42:19 | 2022-08-31 18:42:19.903305 | fa163edd-246c-3638-fe65-00000000589a | TASK | add forward dns record
2022-08-31 18:42:20 | 2022-08-31 18:42:20.806253 | fa163edd-246c-3638-fe65-00000000589a | CHANGED | add forward dns record | undercloud
2022-08-31 18:42:20 | 2022-08-31 18:42:20.807581 | fa163edd-246c-3638-fe65-00000000589a | TIMING | tripleo_ipa_dns : add forward dns record | undercloud | 0:14:42.172740 | 0.90s

Note that the execution does not halt. Rather it fails later on trying to set up mysql - which of course, causes a bunch of other stuff to fail.

I think this error is a red herring -- unless it points to some underlying change in DNS that may be causing the real failure.

Agree with @alee-3. Digging deeper into this shows that the jobs in comment #3 fail on different errors actually. For example..

* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master failed with mysql issue 'WSREP has not yet prepared node for application use' during keystone_bootstrap [1]

* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master failed on mysql issues during keystone_bootstrap [2]

* periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master failed on https://bugs.launchpad.net/tripleo/+bug/1988053 [3]

* periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master failed on https://bugs.launchpad.net/tripleo/+bug/1988053 [4]

* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master failed on https://bugs.launchpad.net/tripleo/+bug/1987092 [5]

* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master failed on mysql related issues [6]

So whenever one sees

  FATAL | try modifying forward dns record | undercloud | error={"changed": false, "msg": "`arecord` not found."}

the error is most likely further down that log file.

[1] https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6ea2096/logs/overcloud-controller-0/var/log/containers/keystone/keystone-manage.log.txt.gz

[2] https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/0c48b0f/logs/overcloud1-controller-0/var/log/containers/keystone/keystone-manage.log.txt.gz

[3] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/a0594a2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[4] https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/8226ab0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[5] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/545f2f1/logs/overcloud-controller-1/var/log/containers/stdouts/galera-bundle.log.txt.gz

[6] https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/40d3797/logs/overcloud-controller-0/var/log/containers/stdouts/nova_db_sync.log.txt.gz

Closing this out - tracking real issues elsewhere