Systemd-resolved gets confused and replies to SOA queries for local breaking mDNS resolution

As discussed on https://gitlab.nic.cz/knot/knot-resolver/-/issues/686, DNS servers are expected to immediately answer any queries within local by NXDOMAIN. See https://datatracker.ietf.org/doc/html/rfc6762#section-22.1 in this respect. However, while they do so they seem to be also expected to deliver an AUTHORITY section with a SOA. As a matter of fact, I am being told the relevant RFCs do not explicitly mention that SOA should be there, but that NXDOMAIN without a SOA is generally considered incorrect, e.g. it necessarily causes caching issues due to missing TTL.

The problem is that such SOA confuses systemd-resolved (at least the version distributed with Ubuntu Focal) that ends up converting the NXDOMAIN reply into a positive reply with such a SOA.

The final consequence is that local name resolution via mDNS is broken with the default ubuntu focal setup.

A detailed discussion of the matter and of the scenario triggering the issue follows.

Suppose that on your network you have a DNS server following the .local blocking indicated in https://datatracker.ietf.org/doc/html/rfc6762#section-22.1 (bullet 4). An example of a modern DNS that has already implemented this feature is knot resolver and the feature is also present in some routers notably the Turris Omnia series. Such a DNS server would reply to queries to local in this way:

  $ dig local
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 56352
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:

;local. IN A

;; AUTHORITY SECTION:

local. 10800 IN SOA local. nobody.invalid. 1 3600 1200 604800 10800

;; ADDITIONAL SECTION:

explanation.invalid. 10800 IN TXT “Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml”

Now let's see what happens to Ubuntu Focal hosts when such a DNS server is deployed in the network and you try to reach a local host:

1. You do something like ping foo.local or ssh foo.local

2. Ubuntu focal has by default the host field in nsswitch.conf set to:

    hosts: files mdns4_minimal [NOTFOUND=return] dns so it tries the /etc/hosts/ file and then mdns via the nss mdns4_minimal client

3. The mdns4_minimal client before doing anything else tries unicast DNS looking for a SOA for
    local. This mechanism is present in the mdns4_minimal client to avoid issues when local is
    under DNS control and is documented at

    https://github.com/lathiat/nss-mdns/blob/master/README.md

4. Ubuntu focal uses by default systemd-resolved as a caching DNS, so the query from
    mdns4_minimal gets to systemd-resolved

5. systemd-resolved passes the query to the DNS it is configured to use. If this is a modern DNS
   server blocking local it gets NXDOMAIN together withthat special SOA in the authority section
   and because it is buggy, it turns it into a regular SOA reply (no NXDOMAIN)

6. mdns4_minimal receives a SOA reply for local and gives up thinking that in the particular
   environment local is under unicast DNS control

7. At this point, according to the nsswitch configuration, DNS is tried. So we are back to
   systemd-resolved now trying to get the A field for foo.local.

8. By default systemd-resolved on ubuntu is configured not to do mDNS itself (even if it has
   this capability). Hence the query fails.

9. Rather than reaching foo.local you get an error.

This is a nasty but, although it is possibly not frequently hit yet because the number of DNS servers that have already implemented local blocking is probably modest. You can only expect it to be on the rise, though.

It may be the case that more recent versions of systemd than the one shipped in Ubuntu Focal have already corrected the issue. Preliminary tests on arch seem to support this hypothesis. If this is the case, it may be possible to backport the fix.

In any case, I think that an update to the Ubuntu Focal systemd package should be provided, because Focal deployments are expected to exist for many years ahead.