Possible DNS issue in the CI

This happens from time to time and we need to fix the unbound.log so we can see why

We are not seeing any logs in unbound because the log file is configured with a selinux context that does not allow unbound to write to it:

https://github.com/openstack-infra/project-config/blob/544fe001f477be137725705d92086e3bb0445b29/nodepool/elements/nodepool-base/finalise.d/89-unbound#L64-L65

Oct 06 16:57:44 upstream-centos-7-rdo-cloud-tripleo-14922 unbound[1191]: Oct 06 16:57:44 unbound[1191:0] error: Could not open logfile /var/log/unbound.log: Permission denied

type=AVC msg=audit(1507309060.078:44): avc: denied { open } for pid=1058 comm="unbound" path="/var/log/unbound.log" dev="vda1" ino=2682530 scontext=system_u:system_r:named_t:s0 tcontext=system_u:object_r:var_log_t:s0 tclass=file

bash-4.2# ls -lZ /var/log/unbound.log
-rw-r--r--. unbound root system_u:object_r:var_log_t:s0 /var/log/unbound.log

bash-4.2# chcon -t named_log_t /var/log/unbound.log

bash-4.2# systemctl restart unbound

bash-4.2# cat /var/log/unbound.log
Oct 06 17:54:52 unbound[2483:0] info: start of service (unbound 1.4.20).

It turns out that /var/log/unbound.log is not supposed to be used in the upstream nodepool images, rather /var/lib/unbound/unbound.log: https://review.openstack.org/#/c/510202/

/var/log/unbound.log is likely used on review.rdoproject.org images due to the delta between upstream and downstream images since we have stopped synchronizing project-config.

Some more information, github.com only has TTL of 30secods, which doesn't make it good for caching results in unbound. we've discussed maybe setting cache-min-ttl: 300 in unbound.conf, but that could be an issue if github.com changes the DNS (maybe the CDN network) during that cache period.

val-bogus-ttl: 60 might be able to help with this, but needs to be tested. Which dmsimard signed up for :)

A parallel effort should be to move rdoinfo off github.com.

rdoinfo is actually at https://review.rdoproject.org/r/rdoinfo
github.com is only a mirror but it was argued that it is more reliable than rdoproject.org hence it is used.
Alternative would be to mirror it into upstream infra but I've heard that proxy-cache for git over http will not work, any other suggestions?

cache-bogus-ttl is not what we expected it to be.
It sounds like what we'd like to do doesn't exist (yet), it would be the opposite of "cache-max-negative-ttl" see: https://bugzilla.redhat.com/show_bug.cgi?id=1360222

In the meantime, we're improving the configure-unbound role to allow for configuration of a minimum TTL. Jobs impacted by frequent DNS issues will be encouraged to tweak the minimum TTL value to see if this helps.

See:
- https://review.openstack.org/#/c/523228/
- https://review.openstack.org/#/c/523178/

Min. TTL in tripleo-ci is enforced after https://review.openstack.org/524018
merged Dec 1.
Are there anymore DNS issues since then?

Reopening this, because it happens again:

http://logs.openstack.org/31/518331/16/experimental-tripleo/tripleo-ci-centos-7-ovb-fakeha-caserver/96557c2/job-output.txt.gz#_2018-01-04_17_53_15_331819

Is this still an issue?

I haven't seen this for a long time, so I think we could close it.