Incorrect DNS (resolv.conf) setup inside docker images

This looks likely to be the result of https://github.com/moby/moby/pull/37485

On the face of it, it looks like your VPN software is stomping over the systemd-resolved config. I admin to finding DNS config fairly confusing but it seems unlikely that this is going to lead to happiness. How are you configuring your VPN?

I'm connecting to the VPN through Palo Alto Networks's globalprotect (https://www.paloaltonetworks.com/products/globalprotect/overview). I just run "globalprotect connect --portal some.company.address.com -u my_vpn_user" (so I assume most of the network config was bundled in the installer I got)

Is there any way to diagnose if the VPN software is the offender? and if that's the case, any workaround you think I could use (because I have no choice to choose a different VPN software)

Have you tried restarting the Docker daemon? (IIRC it caches the DNS settings during startup)

Yes, I tried that while connected to the vpn with no effect

On Mon, 18 Mar 2019 at 13:05, Tianon Gravi <email address hidden> wrote:

> Have you tried restarting the Docker daemon? (IIRC it caches the DNS
> settings during startup)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1820278
>
> Title:
> Incorrect DNS (resolv.conf) setup inside docker images
>
> Status in docker.io package in Ubuntu:
> New
>
> Bug description:
> I've encountered a problem after upgrading docker.io to
> 18.09.2-0ubuntu1~18.04.1 . I currently workarounded it by downgrading
> back to 18.06.1-0ubuntu1.2~18.04.1 , so I can verify that it's related
> to it.
>
> In my current setup, I'm using ubuntu 18.04 and connecting from home
> to a local ISP, and then using a VPN to access some private resources
> from the company I work for (I use globalprotect as VPN software). I
> have no connectivity issues whatsoever from my OS. Before connecting
> to the VPN, my /etc/resolv.conf looks like
>
> # This file is managed by man:systemd-resolved(8). Do not edit.
> # A lot of comments cut-out
> nameserver 217.169.20.20
> nameserver 217.169.20.21
>
> After connecting to the VPN I get this:
>
> search ml.com adminml.com mercadolibre.com
> nameserver 10.x.x.x # I edited out the actual IPs, I'm not allowed to
> share them
> nameserver 10.x.x.x
>
> After upgrading to docker.io 18.09.2, when I start a docker container
> (using docker run, and an ubuntu 16.04 image inside the container),
> the container has my ISP config (ie the 217.x.x.x nameservers) in its
> /etc/resolv.conf, while the older version (docker.io 18.06.1) uses the
> correct VPN configuration (with the search domains and the 10.x.x.x
> address). This new behaviour makes my containers unusable (given that
> I expect to access some VPN-only resources from the software running
> inside them)
>
> Some extra information I found while trying to diagnose this:
> - my host system /etc/resolv.conf is a symlink to
> ../run/systemd/resolve/stub-resolv.conf
> - the "old" (217.x.x.x) resolver configuration seems to be kept at
> /run/systemd/resolve/resolv.conf
> - docker 18.09.2 seems to be copying this old configuration; if I 1)
> edit the /run/systemd/resolve/resolv.conf adding a comment 2) docker -ti
> run myimage 3) cat /etc/resolv.conf , then I see the comment I added. This
> doesn't happen in docker.io 18.06.1
>
> I expect the old behaviour to be kept, can you confirm it is a bug?
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1820278/+subscriptions
>

--
--
Daniel F. Moisset - Consultor

Anything useful in Docker's daemon logs?

As a workaround you should be able to set --dns on your container or adjust /etc/docker/daemon.json to adjust that setting more permanently.

I assume there is "systemd-resolved is running, so using resolvconf: /run/systemd/resolve/resolv.conf" in the deamon logs, or I don't understand what is going on at all.

I think there is a docker bug upstream here: if it sees a systemd-resolved process and /run/systemd/resolve/resolv.conf exists, it assumes systemd-resolved and copies /run/systemd/resolve/resolv.conf into the container. But it doesn't check that /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf, which is probably should.

Although... reading your report again, I wanted to check, when you activate the VPN does the /etc/resolv.conf symlink get replaced with a regular file? Or does the VPN software mutate /run/systemd/resolve/stub-resolv.conf? Because if it's the latter I'm not sure what docker can really do.

Status changed to 'Confirmed' because the bug affects multiple users.

I ran into this on upgrade today.

From reading above, I think the root of the problem is that I am using a system upgraded from Xenial to Bionic, so I am still using ifupdown and resolvconf (and not netplan)

ifupdown doesn't update systemd-resoved, resolvconf/ifupdown updated /etc/resolv.conf to add my IP address, but /run/systemd/resolve/resolv.conf has no servers listed

lrwxrwxrwx 1 root root 29 Oct 12 2017 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

I stopped systemd-resolved, disabled it, restarted docker, no change.

I then removed /etc/resolv.conf's symlink and placed my own file there, restarted docker, still no change.