DNS Forward failures

It looks like this could be a duplicate of LP:2012801

From what I see the error comes from the magpie charm because the check for forward resolution expects a single IP is returned in forward_dns() here: https://opendev.org/openstack/charm-magpie/src/branch/master/src/lib/charms/layer/magpie_tools.py

https://review.opendev.org/c/openstack/charm-magpie/+/888716

In what circumstances MAAS returns multiple IPs to one DNS forward resolution? Why now do we have to update a logic in Magpie if it hasn't been changed for a while?

Magpie and MAAS both need to conform to DNS RFCs and norms, among which it is possible for there to be more than one IP for a given hostname lookup.

We are investigating why SolQA runs seem to only be hitting this 20% of the time, but in the meantime have landed fixes to Magpie for correctness.

Looking at the logs from one of the recently failed runs after the magpie charm fix, it seems sometimes the DNS is not responding:

unit-magpie-ceph-replica-space-0: 2023-07-26 08:28:48 DEBUG unit.magpie-ceph-replica-space/0.juju-log magpie:4: DNS Forward command: /usr/bin/dig 69.64-26.35.168.192.in-addr.arpa. +short +tries=1 +time=3
...
unit-magpie-ceph-replica-space-0: 2023-07-26 08:28:48 INFO unit.magpie-ceph-replica-space/0.juju-log magpie:4: Forward result for unit_id: 2, ip: No forward response, exitcode: 1

Extracted logs from magpie-ceph-access-space-0.log in the juju crashdump

https://pastebin.canonical.com/p/6q2hPxPfJv/

Note that magpie does a reverse lookup first - and gets a odd looking .in-addr.arpa

When prefix length for the subnet is less than a /24, MAAS generates glue records (see https://github.com/maas/maas/blob/master/src/maasserver/dns/zonegenerator.py#L482-L509), where it returns a PTR record pointing to another PTR for the corresponding /24.

Clients usually automatically resolve PTRs until they get a meaningful record (A, AAAA, CNAME).

The magpie charm should probably do the same, checking what kind of result is returned by dig, and resolving again if it's a PTR.

Actually, the reverse lookup returns a CNAME pointing to the subnet section, but that record has no address associated to it.

As an example:

$ dig -x 192.168.36.71 +nocomments

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> -x 192.168.36.71 +nocomments
;; global options: +cmd
;71.36.168.192.in-addr.arpa. IN PTR
71.36.168.192.in-addr.arpa. 17 IN CNAME 71.64-26.36.168.192.in-addr.arpa.
64-26.36.168.192.in-addr.arpa. 17 IN SOA silo2.lab0.solutionsqa. nobody.example.com. 180 600 1800 604800 30
64-26.36.168.192.in-addr.arpa. 30 IN SOA silo2.lab0.solutionsqa. nobody.example.com. 180 600 1800 604800 30
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Aug 01 07:02:31 UTC 2023
;; MSG SIZE rcvd: 190

$ dig 71.64-26.36.168.192.in-addr.arpa. +nocomments

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> 71.64-26.36.168.192.in-addr.arpa. +nocomments
;; global options: +cmd
;71.64-26.36.168.192.in-addr.arpa. IN A
64-26.36.168.192.in-addr.arpa. 30 IN SOA silo2.lab0.solutionsqa. nobody.example.com. 180 600 1800 604800 30
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Aug 01 07:02:49 UTC 2023
;; MSG SIZE rcvd: 137

So this does seem to be a MAAS issue. When a forward answer is added to BIND from MAAS dynamically (i.e a host deploys) a reverse record is generated from the same update. Currently, MAAS will generate the reverse update using the forward update's zone, this makes BIND figure out which reverse zone it should be under. In cases where we don't have the directive to generate CNAMEs for glue records, this works. However, when we have the glue record directive for a given reverse zone, the reverse update is ignored, assuming it should use answers from the smaller subnet's reverse zone.

We will need to add logic to lookup the reverse zone rather than just generating the reverse update, and add it to the reverse update instead.