| 2020-08-11 17:56:40 |
Simon Déziel |
bug |
|
|
added bug |
| 2020-08-11 18:10:06 |
Kai Kasurinen |
bug |
|
|
added subscriber Kai Kasurinen |
| 2020-08-18 15:36:44 |
Balint Reczey |
systemd (Ubuntu): importance |
Undecided |
Low |
|
| 2020-08-27 19:20:23 |
Dan Streetman |
bug |
|
|
added subscriber Dan Streetman |
| 2021-06-29 12:51:17 |
Simon Déziel |
bug |
|
|
added subscriber Xelerance |
| 2021-06-29 12:52:46 |
Simon Déziel |
systemd (Ubuntu): status |
New |
Incomplete |
|
| 2021-06-29 14:38:32 |
Samy Mahmoudi |
bug watch added |
|
https://github.com/systemd/systemd/issues/17577 |
|
| 2021-06-29 17:21:35 |
Dan Streetman |
bug task added |
|
systemd |
|
| 2021-06-29 17:25:31 |
Dan Streetman |
description |
# Issue description:
On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?.
The log noise is the observable problem that would be nice to see addressed:
root@lxd02:~# uptime
17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05
root@lxd02:~# journalctl -b0 | grep -cF dnsmasq
158609
Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received.
1) One can observe that by setting systemd-resolved's service in debug mode:
$ sudo systemctl edit systemd-resolved
and in the editor that is opened, add and save this content:
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
then restart systemd-resolved and watch the logs scroll by with:
$ journalctl -fu systemd-resolved
3) In another terminal, watch the files be recreated with:
watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf
3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by:
sudo tcpdump -ni eth0 icmp6
You will see that every time a RA packet comes in, resolved's journal will log this:
Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file.
# Additional information:
root@lxd02:~# lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04
root@lxd02:~# apt-cache policy systemd
systemd:
Installed: 245.4-4ubuntu3.2
Candidate: 245.4-4ubuntu3.2
Version table:
*** 245.4-4ubuntu3.2 500
500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
100 /var/lib/dpkg/status
245.4-4ubuntu3 500
500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@lxd02:~# uname -a
Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
[impact]
networking changes, like RA events, can cause systemd-resolved to re-write the resolv.conf file, even if the contents didn't change, resulting in unnecessary increased amount of inotify events
[test case]
see original description
[regression potential]
regressions would result in incorrect or missing data in the resolv.conf file, possibly resulting in dns failures or errors
[scope]
TBD
[original description]
# Issue description:
On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?.
The log noise is the observable problem that would be nice to see addressed:
root@lxd02:~# uptime
17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05
root@lxd02:~# journalctl -b0 | grep -cF dnsmasq
158609
Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received.
1) One can observe that by setting systemd-resolved's service in debug mode:
$ sudo systemctl edit systemd-resolved
and in the editor that is opened, add and save this content:
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
then restart systemd-resolved and watch the logs scroll by with:
$ journalctl -fu systemd-resolved
3) In another terminal, watch the files be recreated with:
watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf
3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by:
sudo tcpdump -ni eth0 icmp6
You will see that every time a RA packet comes in, resolved's journal will log this:
Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file.
# Additional information:
root@lxd02:~# lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04
root@lxd02:~# apt-cache policy systemd
systemd:
Installed: 245.4-4ubuntu3.2
Candidate: 245.4-4ubuntu3.2
Version table:
*** 245.4-4ubuntu3.2 500
500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
100 /var/lib/dpkg/status
245.4-4ubuntu3 500
500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@lxd02:~# uname -a
Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
|
| 2021-06-29 17:25:34 |
Dan Streetman |
systemd (Ubuntu): status |
Incomplete |
In Progress |
|
| 2021-06-29 17:25:52 |
Dan Streetman |
nominated for series |
|
Ubuntu Hirsute |
|
| 2021-06-29 17:25:52 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Hirsute) |
|
| 2021-06-29 17:25:52 |
Dan Streetman |
nominated for series |
|
Ubuntu Focal |
|
| 2021-06-29 17:25:52 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Focal) |
|
| 2021-06-29 17:25:52 |
Dan Streetman |
nominated for series |
|
Ubuntu Groovy |
|
| 2021-06-29 17:25:52 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Groovy) |
|
| 2021-06-29 17:29:11 |
Dan Streetman |
description |
[impact]
networking changes, like RA events, can cause systemd-resolved to re-write the resolv.conf file, even if the contents didn't change, resulting in unnecessary increased amount of inotify events
[test case]
see original description
[regression potential]
regressions would result in incorrect or missing data in the resolv.conf file, possibly resulting in dns failures or errors
[scope]
TBD
[original description]
# Issue description:
On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?.
The log noise is the observable problem that would be nice to see addressed:
root@lxd02:~# uptime
17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05
root@lxd02:~# journalctl -b0 | grep -cF dnsmasq
158609
Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received.
1) One can observe that by setting systemd-resolved's service in debug mode:
$ sudo systemctl edit systemd-resolved
and in the editor that is opened, add and save this content:
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
then restart systemd-resolved and watch the logs scroll by with:
$ journalctl -fu systemd-resolved
3) In another terminal, watch the files be recreated with:
watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf
3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by:
sudo tcpdump -ni eth0 icmp6
You will see that every time a RA packet comes in, resolved's journal will log this:
Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file.
# Additional information:
root@lxd02:~# lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04
root@lxd02:~# apt-cache policy systemd
systemd:
Installed: 245.4-4ubuntu3.2
Candidate: 245.4-4ubuntu3.2
Version table:
*** 245.4-4ubuntu3.2 500
500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
100 /var/lib/dpkg/status
245.4-4ubuntu3 500
500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@lxd02:~# uname -a
Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
[impact]
networking changes, like RA events, can cause systemd-resolved to re-write the resolv.conf file, even if the contents didn't change, resulting in unnecessary increased amount of inotify events
[test case]
see original description
[regression potential]
regressions would result in incorrect or missing data in the resolv.conf file, possibly resulting in dns failures or errors
[scope]
this is needed for h and eralier
this is (potentially) fixed upstream by f3e1f00d03445911ee73729219cea88c8a70c612 which in first included in v248, so this is needed in hirsute and earlier
[original description]
# Issue description:
On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?.
The log noise is the observable problem that would be nice to see addressed:
root@lxd02:~# uptime
17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05
root@lxd02:~# journalctl -b0 | grep -cF dnsmasq
158609
Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received.
1) One can observe that by setting systemd-resolved's service in debug mode:
$ sudo systemctl edit systemd-resolved
and in the editor that is opened, add and save this content:
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
then restart systemd-resolved and watch the logs scroll by with:
$ journalctl -fu systemd-resolved
3) In another terminal, watch the files be recreated with:
watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf
3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by:
sudo tcpdump -ni eth0 icmp6
You will see that every time a RA packet comes in, resolved's journal will log this:
Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file.
# Additional information:
root@lxd02:~# lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04
root@lxd02:~# apt-cache policy systemd
systemd:
Installed: 245.4-4ubuntu3.2
Candidate: 245.4-4ubuntu3.2
Version table:
*** 245.4-4ubuntu3.2 500
500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
100 /var/lib/dpkg/status
245.4-4ubuntu3 500
500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@lxd02:~# uname -a
Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
|
| 2021-06-29 17:29:15 |
Dan Streetman |
systemd (Ubuntu): status |
In Progress |
Fix Released |
|
| 2021-06-29 17:29:18 |
Dan Streetman |
systemd (Ubuntu Focal): status |
New |
In Progress |
|
| 2021-06-29 17:29:20 |
Dan Streetman |
systemd (Ubuntu Groovy): status |
New |
In Progress |
|
| 2021-06-29 17:29:21 |
Dan Streetman |
systemd (Ubuntu Hirsute): status |
New |
In Progress |
|
| 2021-06-29 17:29:25 |
Dan Streetman |
systemd (Ubuntu Hirsute): importance |
Undecided |
Wishlist |
|
| 2021-06-29 17:29:27 |
Dan Streetman |
systemd (Ubuntu Groovy): importance |
Undecided |
Wishlist |
|
| 2021-06-29 17:29:29 |
Dan Streetman |
systemd (Ubuntu Focal): importance |
Undecided |
Wishlist |
|
| 2021-06-29 17:29:32 |
Dan Streetman |
systemd (Ubuntu Hirsute): assignee |
|
Dan Streetman (ddstreet) |
|
| 2021-06-29 17:29:33 |
Dan Streetman |
systemd (Ubuntu Groovy): assignee |
|
Dan Streetman (ddstreet) |
|
| 2021-06-29 17:29:35 |
Dan Streetman |
systemd (Ubuntu Focal): assignee |
|
Dan Streetman (ddstreet) |
|
| 2021-07-02 14:29:18 |
Dan Streetman |
nominated for series |
|
Ubuntu Bionic |
|
| 2021-07-02 14:29:18 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Bionic) |
|
| 2021-07-02 14:30:07 |
Dan Streetman |
systemd (Ubuntu Bionic): importance |
Undecided |
Wishlist |
|
| 2021-07-02 15:06:10 |
Dan Streetman |
description |
[impact]
networking changes, like RA events, can cause systemd-resolved to re-write the resolv.conf file, even if the contents didn't change, resulting in unnecessary increased amount of inotify events
[test case]
see original description
[regression potential]
regressions would result in incorrect or missing data in the resolv.conf file, possibly resulting in dns failures or errors
[scope]
this is needed for h and eralier
this is (potentially) fixed upstream by f3e1f00d03445911ee73729219cea88c8a70c612 which in first included in v248, so this is needed in hirsute and earlier
[original description]
# Issue description:
On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?.
The log noise is the observable problem that would be nice to see addressed:
root@lxd02:~# uptime
17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05
root@lxd02:~# journalctl -b0 | grep -cF dnsmasq
158609
Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received.
1) One can observe that by setting systemd-resolved's service in debug mode:
$ sudo systemctl edit systemd-resolved
and in the editor that is opened, add and save this content:
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
then restart systemd-resolved and watch the logs scroll by with:
$ journalctl -fu systemd-resolved
3) In another terminal, watch the files be recreated with:
watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf
3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by:
sudo tcpdump -ni eth0 icmp6
You will see that every time a RA packet comes in, resolved's journal will log this:
Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file.
# Additional information:
root@lxd02:~# lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04
root@lxd02:~# apt-cache policy systemd
systemd:
Installed: 245.4-4ubuntu3.2
Candidate: 245.4-4ubuntu3.2
Version table:
*** 245.4-4ubuntu3.2 500
500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
100 /var/lib/dpkg/status
245.4-4ubuntu3 500
500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@lxd02:~# uname -a
Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
[impact]
networking changes, like RA events, can cause systemd-resolved to re-write the resolv.conf file, even if the contents didn't change, resulting in unnecessary increased amount of inotify events
[test case]
see original description for ipv6ra-related reproducer, or simple reproducer here:
root@lp1891215-h:~# ip l set down dev eth0
root@lp1891215-h:~# md5sum /etc/resolv.conf
db23e80078515192c312e5f321ff0340 /etc/resolv.conf
root@lp1891215-h:~# stat -t -L /etc/resolv.conf
/etc/resolv.conf 740 8 81a4 101 103 fc 188 1 0 0 1625238218 1625238216 1625238216 0 4096
root@lp1891215-h:~# ip l set up dev eth0
root@lp1891215-h:~# md5sum /etc/resolv.conf
db23e80078515192c312e5f321ff0340 /etc/resolv.conf
root@lp1891215-h:~# stat -t -L /etc/resolv.conf
/etc/resolv.conf 740 8 81a4 101 103 fc 188 1 0 0 1625238227 1625238226 1625238226 0 4096
[regression potential]
regressions would result in incorrect or missing data in the resolv.conf file, possibly resulting in dns failures or errors
[scope]
this is needed for h and eralier
this is (potentially) fixed upstream by f3e1f00d03445911ee73729219cea88c8a70c612 which in first included in v248, so this is needed in hirsute and earlier
[original description]
# Issue description:
On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?.
The log noise is the observable problem that would be nice to see addressed:
root@lxd02:~# uptime
17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05
root@lxd02:~# journalctl -b0 | grep -cF dnsmasq
158609
Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received.
1) One can observe that by setting systemd-resolved's service in debug mode:
$ sudo systemctl edit systemd-resolved
and in the editor that is opened, add and save this content:
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
then restart systemd-resolved and watch the logs scroll by with:
$ journalctl -fu systemd-resolved
3) In another terminal, watch the files be recreated with:
watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf
3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by:
sudo tcpdump -ni eth0 icmp6
You will see that every time a RA packet comes in, resolved's journal will log this:
Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file.
# Additional information:
root@lxd02:~# lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04
root@lxd02:~# apt-cache policy systemd
systemd:
Installed: 245.4-4ubuntu3.2
Candidate: 245.4-4ubuntu3.2
Version table:
*** 245.4-4ubuntu3.2 500
500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
100 /var/lib/dpkg/status
245.4-4ubuntu3 500
500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@lxd02:~# uname -a
Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
|
| 2021-07-06 17:47:52 |
Dan Streetman |
description |
[impact]
networking changes, like RA events, can cause systemd-resolved to re-write the resolv.conf file, even if the contents didn't change, resulting in unnecessary increased amount of inotify events
[test case]
see original description for ipv6ra-related reproducer, or simple reproducer here:
root@lp1891215-h:~# ip l set down dev eth0
root@lp1891215-h:~# md5sum /etc/resolv.conf
db23e80078515192c312e5f321ff0340 /etc/resolv.conf
root@lp1891215-h:~# stat -t -L /etc/resolv.conf
/etc/resolv.conf 740 8 81a4 101 103 fc 188 1 0 0 1625238218 1625238216 1625238216 0 4096
root@lp1891215-h:~# ip l set up dev eth0
root@lp1891215-h:~# md5sum /etc/resolv.conf
db23e80078515192c312e5f321ff0340 /etc/resolv.conf
root@lp1891215-h:~# stat -t -L /etc/resolv.conf
/etc/resolv.conf 740 8 81a4 101 103 fc 188 1 0 0 1625238227 1625238226 1625238226 0 4096
[regression potential]
regressions would result in incorrect or missing data in the resolv.conf file, possibly resulting in dns failures or errors
[scope]
this is needed for h and eralier
this is (potentially) fixed upstream by f3e1f00d03445911ee73729219cea88c8a70c612 which in first included in v248, so this is needed in hirsute and earlier
[original description]
# Issue description:
On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?.
The log noise is the observable problem that would be nice to see addressed:
root@lxd02:~# uptime
17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05
root@lxd02:~# journalctl -b0 | grep -cF dnsmasq
158609
Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received.
1) One can observe that by setting systemd-resolved's service in debug mode:
$ sudo systemctl edit systemd-resolved
and in the editor that is opened, add and save this content:
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
then restart systemd-resolved and watch the logs scroll by with:
$ journalctl -fu systemd-resolved
3) In another terminal, watch the files be recreated with:
watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf
3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by:
sudo tcpdump -ni eth0 icmp6
You will see that every time a RA packet comes in, resolved's journal will log this:
Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file.
# Additional information:
root@lxd02:~# lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04
root@lxd02:~# apt-cache policy systemd
systemd:
Installed: 245.4-4ubuntu3.2
Candidate: 245.4-4ubuntu3.2
Version table:
*** 245.4-4ubuntu3.2 500
500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
100 /var/lib/dpkg/status
245.4-4ubuntu3 500
500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@lxd02:~# uname -a
Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
[impact]
networking changes, like RA events, can cause systemd-resolved to re-write the resolv.conf file, even if the contents didn't change, resulting in unnecessary increased amount of inotify events
[test case]
see original description for ipv6ra-related reproducer, or simple reproducer here:
configure networkd with some config for (e.g.) eth0, but not a config that would result in /etc/resolv.conf changing when the interface goes up/down - for example, use static config with no DNS search domains. Then bring eth0 up/down while observing the md5sum (file content) does not change but the mtime does change.
root@lp1891215-h:~# ip l set down dev eth0
root@lp1891215-h:~# md5sum /etc/resolv.conf
db23e80078515192c312e5f321ff0340 /etc/resolv.conf
root@lp1891215-h:~# stat -t -L /etc/resolv.conf
/etc/resolv.conf 740 8 81a4 101 103 fc 188 1 0 0 1625238218 1625238216 1625238216 0 4096
root@lp1891215-h:~# ip l set up dev eth0
root@lp1891215-h:~# md5sum /etc/resolv.conf
db23e80078515192c312e5f321ff0340 /etc/resolv.conf
root@lp1891215-h:~# stat -t -L /etc/resolv.conf
/etc/resolv.conf 740 8 81a4 101 103 fc 188 1 0 0 1625238227 1625238226 1625238226 0 4096
[regression potential]
regressions would result in incorrect or missing data in the resolv.conf file, possibly resulting in dns failures or errors
[scope]
this is needed for h and eralier
this is (potentially) fixed upstream by f3e1f00d03445911ee73729219cea88c8a70c612 which in first included in v248, so this is needed in hirsute and earlier
[original description]
# Issue description:
On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?.
The log noise is the observable problem that would be nice to see addressed:
root@lxd02:~# uptime
17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05
root@lxd02:~# journalctl -b0 | grep -cF dnsmasq
158609
Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received.
1) One can observe that by setting systemd-resolved's service in debug mode:
$ sudo systemctl edit systemd-resolved
and in the editor that is opened, add and save this content:
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
then restart systemd-resolved and watch the logs scroll by with:
$ journalctl -fu systemd-resolved
3) In another terminal, watch the files be recreated with:
watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf
3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by:
sudo tcpdump -ni eth0 icmp6
You will see that every time a RA packet comes in, resolved's journal will log this:
Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file.
# Additional information:
root@lxd02:~# lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04
root@lxd02:~# apt-cache policy systemd
systemd:
Installed: 245.4-4ubuntu3.2
Candidate: 245.4-4ubuntu3.2
Version table:
*** 245.4-4ubuntu3.2 500
500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
100 /var/lib/dpkg/status
245.4-4ubuntu3 500
500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@lxd02:~# uname -a
Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
|
| 2021-07-08 22:58:18 |
Brian Murray |
systemd (Ubuntu Hirsute): status |
In Progress |
Fix Committed |
|
| 2021-07-08 22:58:20 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-07-08 22:58:22 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2021-07-08 22:58:27 |
Brian Murray |
tags |
|
verification-needed verification-needed-hirsute |
|
| 2021-07-08 23:14:15 |
Brian Murray |
systemd (Ubuntu Groovy): status |
In Progress |
Fix Committed |
|
| 2021-07-08 23:14:21 |
Brian Murray |
tags |
verification-needed verification-needed-hirsute |
verification-needed verification-needed-groovy verification-needed-hirsute |
|
| 2021-07-08 23:35:13 |
Brian Murray |
systemd (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2021-07-08 23:35:20 |
Brian Murray |
tags |
verification-needed verification-needed-groovy verification-needed-hirsute |
verification-needed verification-needed-focal verification-needed-groovy verification-needed-hirsute |
|
| 2021-07-09 15:04:18 |
Simon Déziel |
tags |
verification-needed verification-needed-focal verification-needed-groovy verification-needed-hirsute |
verification-failed-focal verification-needed verification-needed-groovy verification-needed-hirsute |
|
| 2021-07-09 15:21:34 |
Simon Déziel |
tags |
verification-failed-focal verification-needed verification-needed-groovy verification-needed-hirsute |
verification-needed verification-needed-focal verification-needed-groovy verification-needed-hirsute |
|
| 2021-07-09 15:27:55 |
Simon Déziel |
tags |
verification-needed verification-needed-focal verification-needed-groovy verification-needed-hirsute |
verification-done-focal verification-needed verification-needed-groovy verification-needed-hirsute |
|
| 2021-07-09 15:36:46 |
Simon Déziel |
tags |
verification-done-focal verification-needed verification-needed-groovy verification-needed-hirsute |
verification-done verification-done-focal verification-done-groovy verification-done-hirsute |
|
| 2021-07-09 16:47:45 |
Launchpad Janitor |
systemd (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2021-07-19 20:18:15 |
Launchpad Janitor |
systemd (Ubuntu Groovy): status |
Fix Committed |
Fix Released |
|
| 2021-07-19 20:18:36 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2021-07-20 16:58:37 |
Launchpad Janitor |
systemd (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-07-20 16:58:37 |
Launchpad Janitor |
cve linked |
|
2020-13529 |
|
| 2021-07-20 16:58:37 |
Launchpad Janitor |
cve linked |
|
2021-33910 |
|
| 2021-07-20 16:58:41 |
Launchpad Janitor |
systemd (Ubuntu Hirsute): status |
Fix Committed |
Fix Released |
|
| 2021-08-25 16:43:05 |
Dan Streetman |
systemd (Ubuntu Bionic): status |
Confirmed |
Won't Fix |
|
