cloud-images

Ubuntu 23.10 cloud images unexpected UDP listening port 5353

Bug #2038894 reported by Philip Roche
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
cloud-images
New
Undecided
Unassigned
systemd (Ubuntu)
Status tracked in Mantic
Mantic
New
High
Nick Rosbrook

Bug Description

[Impact]

In the latest Ubuntu 23.10 cloud images we are seeing unexpected UDP listening port 5353.

By default and by policy, aside from port 22 there should be no other open ports on Ubuntu cloud images. Listening port 5353 is a regression.

[Test Plan]

Check that port 5353 is not open, and in particular that systemd-resolved is not listening on 5353. This is what it looks like when systemd-resolved *is* listening on 5353:

```
$ ss --listening --no-header --tcp --udp --numeric
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:*
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:*
udp UNCONN 0 0 10.154.0.17%ens4:68 0.0.0.0:*
udp UNCONN 0 0 127.0.0.1:323 0.0.0.0:*
udp UNCONN 0 0 0.0.0.0:5353 0.0.0.0:*
udp UNCONN 0 0 [::1]:323 [::]:*
udp UNCONN 0 0 [::]:5353 [::]:*
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:*
tcp LISTEN 0 4096 *:22 *:*
```

```
$ sudo lsof -i -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 153u IPv6 17848 0t0 TCP *:22 (LISTEN)
systemd-r 321 systemd-resolve 11u IPv4 16159 0t0 UDP *:5353
systemd-r 321 systemd-resolve 12u IPv6 16161 0t0 UDP *:5353
systemd-r 321 systemd-resolve 15u IPv4 16164 0t0 UDP 127.0.0.53:53
systemd-r 321 systemd-resolve 16u IPv4 16165 0t0 TCP 127.0.0.53:53 (LISTEN)
systemd-r 321 systemd-resolve 17u IPv4 16166 0t0 UDP 127.0.0.54:53
systemd-r 321 systemd-resolve 18u IPv4 16167 0t0 TCP 127.0.0.54:53 (LISTEN)
systemd-n 431 systemd-network 18u IPv4 17227 0t0 UDP 10.154.0.17:68
google_os 566 root 3u IPv4 18555 0t0 TCP 10.154.0.17:60818->169.254.169.254:80 (ESTABLISHED)
google_gu 739 root 13u IPv4 19822 0t0 TCP 10.154.0.17:35516->169.254.169.254:80 (ESTABLISHED)
sshd 747 root 3u IPv6 17848 0t0 TCP *:22 (LISTEN)
chronyd 1720 _chrony 5u IPv4 21448 0t0 UDP 127.0.0.1:323
chronyd 1720 _chrony 6u IPv6 21449 0t0 UDP [::1]:323
sshd 1761 root 4u IPv6 22688 0t0 TCP 10.154.0.17:22->185.202.17.195:45142 (ESTABLISHED)
sshd 1882 ubuntu 4u IPv6 22688 0t0 TCP 10.154.0.17:22->185.202.17.195:45142 (ESTABLISHED)

```

[Where problems could occur]

This patch reverts a change that enables MulticastDNS=resolve by default in systemd. Mantic is the first release where this is done, so it should not break existing users. If a user does want this behavior back, all they need to do is override the default /etc/systemd/resolved.conf.

[Original Description]

In the latest Ubuntu 23.10 cloud images we are seeing unexpected UDP listening port 5353.

By default and by policy, aside from port 22 there should be no other open ports on Ubuntu cloud images. Listening port 5353 is a regression.

Ubuntu 23.10 debug

```
$ ss --listening --no-header --tcp --udp --numeric
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:*
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:*
udp UNCONN 0 0 10.154.0.17%ens4:68 0.0.0.0:*
udp UNCONN 0 0 127.0.0.1:323 0.0.0.0:*
udp UNCONN 0 0 0.0.0.0:5353 0.0.0.0:*
udp UNCONN 0 0 [::1]:323 [::]:*
udp UNCONN 0 0 [::]:5353 [::]:*
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:*
tcp LISTEN 0 4096 *:22 *:*
```

This shows port 5353 open.

To find out what is listening on this port:

```
$ sudo lsof -i -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 153u IPv6 17848 0t0 TCP *:22 (LISTEN)
systemd-r 321 systemd-resolve 11u IPv4 16159 0t0 UDP *:5353
systemd-r 321 systemd-resolve 12u IPv6 16161 0t0 UDP *:5353
systemd-r 321 systemd-resolve 15u IPv4 16164 0t0 UDP 127.0.0.53:53
systemd-r 321 systemd-resolve 16u IPv4 16165 0t0 TCP 127.0.0.53:53 (LISTEN)
systemd-r 321 systemd-resolve 17u IPv4 16166 0t0 UDP 127.0.0.54:53
systemd-r 321 systemd-resolve 18u IPv4 16167 0t0 TCP 127.0.0.54:53 (LISTEN)
systemd-n 431 systemd-network 18u IPv4 17227 0t0 UDP 10.154.0.17:68
google_os 566 root 3u IPv4 18555 0t0 TCP 10.154.0.17:60818->169.254.169.254:80 (ESTABLISHED)
google_gu 739 root 13u IPv4 19822 0t0 TCP 10.154.0.17:35516->169.254.169.254:80 (ESTABLISHED)
sshd 747 root 3u IPv6 17848 0t0 TCP *:22 (LISTEN)
chronyd 1720 _chrony 5u IPv4 21448 0t0 UDP 127.0.0.1:323
chronyd 1720 _chrony 6u IPv6 21449 0t0 UDP [::1]:323
sshd 1761 root 4u IPv6 22688 0t0 TCP 10.154.0.17:22->185.202.17.195:45142 (ESTABLISHED)
sshd 1882 ubuntu 4u IPv6 22688 0t0 TCP 10.154.0.17:22->185.202.17.195:45142 (ESTABLISHED)

```

Shows that it is systemd-resolved that is listening and from https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html

> The systemd-resolved service listens on the following IP ports:

> Port 5353 on all local addresses, both IPv4 and IPv6 (0.0.0.0 and ::0), for MulticastDNS on UDP. Note that even though the socket is bound to all local interfaces via the selected "wildcard" IP addresses, the incoming datagrams are filtered by the network interface they are coming in on, and separate MulticastDNS link-local scopes are maintained for each, taking into consideration whether MulticastDNS is enabled for the interface or not.

So listening on port 5353 is expected for systemd-resolved and MulticastDNS but we do not expect this to be enabled by default on cloud images.

```
$ dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==============-============-=================================
ii systemd 253.5-1ubuntu6 amd64 system and service manager
```

Comparing the open ports on an Ubuntu 22.04 multipass VM

```
$ ss --listening --no-header --tcp --udp --numeric
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:*
udp UNCONN 0 0 10.212.201.146%ens3:68 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
tcp LISTEN 0 128 [::]:22 [::]:*
```

```
$ dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==================-============-=================================
ii systemd 249.11-0ubuntu3.10 amd64 system and service manager
```

See original description
Revision history for this message
Philip Roche (philroche) wrote :

This is related to https://bugs.launchpad.net/cloud-images/+bug/2032933 too

Revision history for this message
Philip Roche (philroche) wrote :

I do not believe this is a 23.10 release blocker

Revision history for this message
Philip Roche (philroche) wrote :

This change was introduced in 253.5-1ubuntu1 of systemd

https://launchpad.net/ubuntu/+source/systemd/253.5-1ubuntu1

Revision history for this message
Philip Roche (philroche) wrote :

This can be overridden for cloud images

```
mkdir --parents --verbose /etc/systemd/resolved.conf.d
cat > "/etc/systemd/resolved.conf.d/50-cloudimg.conf" <<EOF
[Resolve]
MulticastDNS=false
EOF
```

results in no longer listening on port 5353

```
$ sudo ss --listening --no-header --tcp --udp --numeric
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:*
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:*
udp UNCONN 0 0 10.0.2.15%enp0s2:68 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:*
tcp LISTEN 0 128 127.0.0.1:6010 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
tcp LISTEN 0 4096 *:22 *:*
tcp LISTEN 0 128 [::1]:6010 [::]:*
```

Nick Rosbrook (enr0n)
Changed in systemd (Ubuntu):
importance: Undecided → High
tags: added: foundations-todo
Changed in systemd (Ubuntu):
assignee: nobody → Nick Rosbrook (enr0n)
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
Nick Rosbrook (enr0n)
description: updated
Nick Rosbrook (enr0n)
Changed in systemd (Ubuntu Mantic):
status: Confirmed → In Progress
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hi Philip,

I have some questions here:

a) You state that some policy says that no ports other than 22 should be open, which policy is that? Does it apply only to cloud images, or is it an Ubuntu policy in general?

b) This is in mantic release at the moment, and switching that option back to "no" could regress users that were relying on this default. What exactly are we losing when we disable this service in this SRU? I checked the original commit[1] but it does not have a bug number linked to it with more details about what was the reasoning to enable this option in the first place.

c) If this is only about cloud images, is the workaround in comment #4 something that could be added to the cloud image build process, or we really want to avoid that?

d) Are there specific security concerns with keeping this service enabled? I presume these were considered when the option was set to "resolve" in that commit[1].

1. https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b308303f34484b293920473e5c4e0395142e4bcc

Changed in systemd (Ubuntu Mantic):
status: In Progress → Incomplete
Revision history for this message
Steve Langasek (vorlon) wrote :

"no open ports" is the long standing policy for all Ubuntu and is not just for cloud images. (Even port 22 is not supposed to be open on bare metal server by default, only as opt in.)

I don't have a link handy at the moment to documentation of this policy but it IS the policy.

Changed in systemd (Ubuntu Mantic):
status: Incomplete → New
Revision history for this message
Steve Langasek (vorlon) wrote :

The effect of opening a port was a not-understood consequence of changing the default.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.