systemd fails to set unit as inactive when using socket activation and the main process has exited

systemd 253.5 on Mantic is affected by a bug which makes it fail to mark a unit as inactive even when its main process exited (when using socket activation). This is affecting libvirt and possibly other services.

Upstream has a bug: https://github.com/systemd/systemd/issues/27953

which has been fixed by: https://github.com/systemd/systemd/pull/28000

To reproduce the problem:

$ lxc launch ubuntu-daily:mantic libvirt-hang --vm
$ lxc shell libvirt-hang
# apt update && apt upgrade -y
# apt install -y libvirt-daemon-system
# systemctl status libvirtd.service

You'll notice that there is a libvirt process running:

...
     CGroup: /system.slice/libvirtd.service
             ├─ 870 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
             ├─ 871 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
             └─1020 /usr/sbin/libvirtd --timeout 120
...

Wait for two minutes (or edit /etc/default/libvirtd and reduce the timeout), then check the status again. You'll notice that the libvirtd process has exited, but the unit is still marked as active:

root@libvirt-hang:~# systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; preset: enabled)
     Active: active (running) since Mon 2023-08-28 23:06:23 UTC; 57s ago
TriggeredBy: ● libvirtd-admin.socket
             ● libvirtd.socket
             ● libvirtd-ro.socket
       Docs: man:libvirtd(8)
             https://libvirt.org
    Process: 1020 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
   Main PID: 1020 (code=exited, status=0/SUCCESS)
      Tasks: 2 (limit: 32768)
     Memory: 22.4M
        CPU: 161ms
     CGroup: /system.slice/libvirtd.service
             ├─870 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
             └─871 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
...

libvirtd.service is socket-activated, but the fact that it is still considered to be active after the main process exited means that the socket won't be actively listening, and you end up seeing libvirt-related commands hang indefinitely, effectively rendering libvirt useless until you manually restart the service.