systemd leaks abandoned session scopes
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| snapd |
Invalid
|
Undecided
|
Unassigned | ||
| systemd |
New
|
Unknown
|
|||
| systemd (Ubuntu) |
Fix Released
|
Medium
|
Dan Streetman | ||
| Bionic |
Fix Released
|
Medium
|
Dan Streetman | ||
| Focal |
Fix Released
|
Medium
|
Dan Streetman | ||
| Groovy |
Won't Fix
|
Medium
|
Unassigned | ||
| Hirsute |
Fix Released
|
Medium
|
Dan Streetman | ||
| Impish |
Fix Released
|
Medium
|
Dan Streetman | ||
Bug Description
[impact]
systemd may leak sessions, leaving empty cgroups around as well as abandoned session scopes.
[test case]
on a system where the user has a ssh key that allows noninteractive login to localhost, and also has noninteractive sudo, run:
$ for i in {1..100}; do sudo -b -i -u ubuntu ssh localhost -- sleep 1; done; for i in {1..20}; do echo 'Reloading...'; sudo systemctl daemon-reload; done
check the sessions to see there have been leaked sessions:
$ loginctl list-sessions
SESSION UID USER SEAT TTY
1 1000 ubuntu ttyS0
350 1000 ubuntu
351 1000 ubuntu
360 1000 ubuntu
...
to verify the sessions were leaked, clear them out with:
$ echo '' | sudo tee /sys/fs/
that should result in all the leaked sessions being cleaned up.
[regression potential]
issues during systemd pid1 reexec/reload, or issues while cleaning up sessions, including leaking sessions/cgroups
[scope]
this is needed for all releases
upstream bug linked above, and upstream PR:
https:/
[original description]
On a system that is monitored via telegraf I found many abandoned systemd session which I believe are created by a potential race where systemd is reloading unit files and at the same time a user is connecting to the system via ssh or is executing the su command.
The simple reproducer
$ for i in {1..100}; do sleep 0.2; ssh localhost sudo systemctl daemon-reload & ssh localhost sleep 1 & done
Wait > 1 second
$ jobs -p | xargs --verbose --no-run-if-empty kill -KILL
To clean out STOPPED jobs and
$ systemctl status --all 2> /dev/null | grep --before-context 3 abandoned
will produce something similar to
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
--
● session-273.scope - Session 273 of user ubuntu
Loaded: loaded (/run/systemd/
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-274.scope - Session 274 of user ubuntu
Loaded: loaded (/run/systemd/
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
--
● session-30.scope - Session 30 of user ubuntu
Loaded: loaded (/run/systemd/
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 10:05:56 UTC; 3h 30min ago
--
● session-302.scope - Session 302 of user ubuntu
Loaded: loaded (/run/systemd/
Transient: yes
Active: active (abandoned) since Wed 2021-06-30 13:32:04 UTC; 4min 6s ago
--
│ ├─ 175 su - ubuntu
│ ├─ 178 -su
│ ├─62375 systemctl status --all
│ └─62376 grep --color=auto --before-context 3 abandoned
The system in question is running Bionic, systemd-
| Changed in systemd (Ubuntu): | |
| assignee: | nobody → Dan Streetman (ddstreet) |
| description: | updated |
| description: | updated |
| Launchpad Janitor (janitor) wrote | #1 |
| Changed in systemd (Ubuntu): | |
| status: | New → Confirmed |
| Changed in systemd (Ubuntu Hirsute): | |
| assignee: | nobody → Dan Streetman (ddstreet) |
| Changed in systemd (Ubuntu Groovy): | |
| assignee: | nobody → Dan Streetman (ddstreet) |
| Changed in systemd (Ubuntu Focal): | |
| assignee: | nobody → Dan Streetman (ddstreet) |
| Changed in systemd (Ubuntu Bionic): | |
| assignee: | nobody → Dan Streetman (ddstreet) |
| Changed in systemd (Ubuntu Impish): | |
| importance: | Undecided → Medium |
| Changed in systemd (Ubuntu Hirsute): | |
| importance: | Undecided → Medium |
| Changed in systemd (Ubuntu Groovy): | |
| importance: | Undecided → Medium |
| Changed in systemd (Ubuntu Focal): | |
| importance: | Undecided → Medium |
| Changed in systemd (Ubuntu Bionic): | |
| importance: | Undecided → Medium |
| description: | updated |
| Changed in systemd (Ubuntu Impish): | |
| status: | Confirmed → In Progress |
| Changed in systemd (Ubuntu Hirsute): | |
| status: | New → In Progress |
| Changed in systemd (Ubuntu Groovy): | |
| status: | New → In Progress |
| Changed in systemd (Ubuntu Focal): | |
| status: | New → In Progress |
| Changed in systemd: | |
| status: | Unknown → New |
| Changed in systemd (Ubuntu Bionic): | |
| status: | New → Confirmed |
| Ian Johnson (anonymouse67) wrote | #3 |
| Chris Johnston (cjohnston) wrote | #4 |
| Changed in systemd (Ubuntu Impish): | |
| assignee: | Dan Streetman (ddstreet) → Heather Lemon (hypothetical-lemon) |
| Changed in systemd (Ubuntu Hirsute): | |
| assignee: | Dan Streetman (ddstreet) → Heather Lemon (hypothetical-lemon) |
| Changed in systemd (Ubuntu Groovy): | |
| assignee: | Dan Streetman (ddstreet) → nobody |
| Changed in systemd (Ubuntu Focal): | |
| assignee: | Dan Streetman (ddstreet) → Heather Lemon (hypothetical-lemon) |
| Changed in systemd (Ubuntu Bionic): | |
| assignee: | Dan Streetman (ddstreet) → Heather Lemon (hypothetical-lemon) |
| Changed in systemd (Ubuntu Groovy): | |
| status: | In Progress → Won't Fix |
| Changed in systemd (Ubuntu Bionic): | |
| assignee: | Heather Lemon (hypothetical-lemon) → Dan Streetman (ddstreet) |
| Changed in systemd (Ubuntu Focal): | |
| assignee: | Heather Lemon (hypothetical-lemon) → Dan Streetman (ddstreet) |
| Changed in systemd (Ubuntu Hirsute): | |
| assignee: | Heather Lemon (hypothetical-lemon) → Dan Streetman (ddstreet) |
| Changed in systemd (Ubuntu Impish): | |
| assignee: | Heather Lemon (hypothetical-lemon) → Dan Streetman (ddstreet) |
| Dan Streetman (ddstreet) wrote | #5 |
| Changed in systemd (Ubuntu Impish): | |
| status: | In Progress → Fix Committed |
| Changed in systemd (Ubuntu Bionic): | |
| status: | Confirmed → In Progress |
| Chris Halse Rogers (raof) wrote Please test proposed package | #6 |
| Changed in systemd (Ubuntu Bionic): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed verification-needed-bionic |
| Chris Halse Rogers (raof) wrote | #7 |
| Changed in systemd (Ubuntu Hirsute): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed-hirsute |
| Ubuntu SRU Bot (ubuntu-sru-bot) wrote Autopkgtest regression report (systemd/237-3ubuntu10.52) | #8 |
| Ubuntu SRU Bot (ubuntu-sru-bot) wrote Autopkgtest regression report (systemd/247.3-3ubuntu3.5) | #9 |
| Łukasz Zemczak (sil2100) wrote Please test proposed package | #10 |
| Changed in systemd (Ubuntu Focal): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed-focal |
| Ubuntu SRU Bot (ubuntu-sru-bot) wrote Autopkgtest regression report (systemd/245.4-4ubuntu3.12) | #11 |
| Dan Streetman (ddstreet) wrote | #12 |
| Dan Streetman (ddstreet) wrote | #13 |
| Dan Streetman (ddstreet) wrote | #14 |
| tags: |
added: verification-done verification-done-bionic verification-done-focal verification-done-hirsute removed: verification-needed verification-needed-bionic verification-needed-focal verification-needed-hirsute |
| Brian Murray (brian-murray) wrote Please test proposed package | #15 |
| tags: |
added: verification-needed verification-needed-hirsute removed: verification-done verification-done-hirsute |
| Brian Murray (brian-murray) wrote | #16 |
| tags: |
added: verification-needed-focal removed: verification-done-focal |
| Ubuntu SRU Bot (ubuntu-sru-bot) wrote Autopkgtest regression report (systemd/247.3-3ubuntu3.6) | #17 |
| Ubuntu SRU Bot (ubuntu-sru-bot) wrote Autopkgtest regression report (systemd/245.4-4ubuntu3.13) | #18 |
| Dan Streetman (ddstreet) wrote | #19 |
| Dan Streetman (ddstreet) wrote | #20 |
| tags: |
added: verification-done verification-done-focal verification-done-hirsute removed: verification-needed verification-needed-focal verification-needed-hirsute |
| Chris Halse Rogers (raof) wrote Update Released | #21 |
| Launchpad Janitor (janitor) wrote | #22 |
| Changed in systemd (Ubuntu Bionic): | |
| status: | Fix Committed → Fix Released |
| Launchpad Janitor (janitor) wrote | #23 |
| Changed in systemd (Ubuntu Focal): | |
| status: | Fix Committed → Fix Released |
| Launchpad Janitor (janitor) wrote | #24 |
| Changed in systemd (Ubuntu Hirsute): | |
| status: | Fix Committed → Fix Released |
| Changed in snapd: | |
| status: | New → Invalid |
| Dan Streetman (ddstreet) wrote | #25 |
| Changed in systemd (Ubuntu Impish): | |
| status: | Fix Committed → Fix Released |
| Changed in systemd (Ubuntu): | |
| status: | Fix Committed → Fix Released |
Status changed to 'Confirmed' because the bug affects multiple users.