cookie overruns can cause org.freedesktop.systemd1 dbus to hang
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
systemd (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Xenial |
Fix Released
|
High
|
Heitor Alves de Siqueira | ||
Bionic |
Fix Released
|
High
|
Heitor Alves de Siqueira |
Bug Description
[Impact]
Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop
[Description]
Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop
This issue has been fixed upstream by the commit below:
- sd-bus: deal with cookie overruns (1f82f5bb4237)
$ git describe --contains 1f82f5bb4237
v242-rc1~228
$ rmadison systemd
systemd | 229-4ubuntu4 | xenial | source, ...
systemd | 229-4ubuntu21.27 | xenial-security | source, ...
systemd | 229-4ubuntu21.27 | xenial-updates | source, ...
systemd | 229-4ubuntu21.28 | xenial-proposed | source, ...
systemd | 237-3ubuntu10 | bionic | source, ...
systemd | 237-3ubuntu10.38 | bionic-security | source, ...
systemd | 237-3ubuntu10.39 | bionic-updates | source, ...
systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... <----
systemd | 242-7ubuntu3 | eoan | source, ...
Releases starting with Eoan already have this fix.
[Test Case]
There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months).
Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps:
1. Create a new system bus object through sd_bus_
2. Allocate and append a new method_call message to the bus
3. Send the message through sd_bus_call()
4. Handle the response message and free up the message objects
It's essentially the example code from the sd_bus_
ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g
ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie
Breakpoint 1 at 0xe61: file test.c, line 38.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_
(16s) cookie: 0x00010000 reply-cookie: 0x00010000
Breakpoint 1, print_unit_path (bus=0x55555575
38 r = sd_bus_
$1 = 0x10000
$2 = 0xffffff00
Call failed: Operation not supported
Sleeping and retrying...
Call failed: Invalid argument
Assertion 'm->n_ref > 0' failed at ../src/
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=0x6) at ../sysdeps/
51 ../sysdeps/
To compile and debug the test case above, libsystemd-dev and libsystemd0-dbgsym are required.
Both test.c and test.gdb source code are attached to this LP bug.
[Regression Potential]
This fix introduces some changes in the way cookie incrementation is handled. We now have a reduced number of available values, since the patch makes use of a high order bit to indicate whether we have overflowed or not. Potential issues could arise from two distinct messages repeating the cookie value, or from us not handling the cookie reuse properly. In practice, this shouldn't cause serious problems as most dbus messages should not stall long enough for a possible overlap in the 2^31 space. The patch has been present in other stable Ubuntu Series and upstream, and has been validated and tested through the systemd test suite and autopkgtests.
Related branches
- git-ubuntu developers: Pending requested
-
Diff: 123 lines (+101/-0)3 files modifieddebian/changelog (+7/-0)
debian/patches/lp1876600-sd-bus-deal-with-cookie-overruns.patch (+93/-0)
debian/patches/series (+1/-0)
- git-ubuntu developers: Pending requested
-
Diff: 119 lines (+97/-0)3 files modifieddebian/changelog (+7/-0)
debian/patches/lp1876600-sd-bus-deal-with-cookie-overruns.patch (+89/-0)
debian/patches/series (+1/-0)
Changed in systemd (Ubuntu): | |
status: | New → Fix Released |
Changed in systemd (Ubuntu Xenial): | |
status: | New → Confirmed |
Changed in systemd (Ubuntu Bionic): | |
status: | New → Confirmed |
Changed in systemd (Ubuntu Xenial): | |
importance: | Undecided → High |
Changed in systemd (Ubuntu Bionic): | |
importance: | Undecided → High |
assignee: | nobody → Heitor Alves de Siqueira (halves) |
Changed in systemd (Ubuntu Xenial): | |
assignee: | nobody → Heitor Alves de Siqueira (halves) |
description: | updated |
Changed in systemd (Ubuntu Xenial): | |
status: | Confirmed → In Progress |
Changed in systemd (Ubuntu Bionic): | |
status: | Confirmed → In Progress |
tags: |
added: sts-sponsor-ddstreet removed: sts-sponsor |
Test builds for the proposed merge can be found at the lp1876600 PPA [0].
[0] https:/ /launchpad. net/~halves/ +archive/ ubuntu/ lp1876600