Regular D-state processes impacting LXD containers
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| linux (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
| Xenial |
Fix Released
|
Medium
|
Unassigned | ||
| Bionic |
Fix Released
|
Medium
|
Unassigned | ||
Bug Description
[Impact]
* Systems running under memory pressure may hit stalls in the
order of seconds to minutes in systemd-logind and lxd mount
operations (e.g., ZFS backend), which get stuck in D state.
* The processes stuck in D state have a common stack trace,
(cat /proc/PID/stack) all blocked in register_
* The fix checks in shrink_slab() (shrinkers are called under
memory pressure) for contention/usage of the semaphore used
by register_shrinker() and returns early in that case.
This allows the register_shrinker() callers to unblock,
and not stall until the shrink operation releases that lock.
[Test Case]
* In a system under memory pressure, specifically having the
memory shrinkers being called often and taking time to run,
perform mount operations (or other operations that acquire
the shrinker_rwsem semaphore).
* The user who reported the problem has verified the fix in
systems that exhibted the problem often (sometimes daily),
and tells it resolves the problem.
[Regression Potential]
* Low. The fix just returns early from slab memory shrinker
if there's usage/contention for 'shrinker_rwsem'.
* In some scenarios, this may cause the slab memory shrinker
to require more invocations to actually finish and potentially
release memory, but this seems minor since other shrinkers can
release memory as well, and compared to the fact that this fix
allows other applications to make progress / continue to run,
which would otherwise be stalled.
[Other Info]
* This patch is already applied in Cosmic and later (v4.16+).
It is needed only in Xenial and Bionic at this time.
| Changed in linux (Ubuntu): | |
| status: | New → Invalid |
| Changed in linux (Ubuntu Xenial): | |
| status: | New → Confirmed |
| Changed in linux (Ubuntu Bionic): | |
| status: | New → Confirmed |
| Mauricio Faria de Oliveira (mfo) wrote | #1 |
| Mauricio Faria de Oliveira (mfo) wrote | #2 |
| Changed in linux (Ubuntu Xenial): | |
| importance: | Undecided → Medium |
| Changed in linux (Ubuntu Bionic): | |
| importance: | Undecided → Medium |
| Changed in linux (Ubuntu Xenial): | |
| status: | Confirmed → Fix Committed |
| Changed in linux (Ubuntu Bionic): | |
| status: | Confirmed → Fix Committed |
| Brad Figg (brad-figg) wrote | #3 |
| tags: | added: verification-needed-xenial |
| Brad Figg (brad-figg) wrote | #4 |
| tags: | added: verification-needed-bionic |
| Mauricio Faria de Oliveira (mfo) wrote | #5 |
| tags: |
added: verification-done-bionic verification-done-xenial removed: verification-needed-bionic verification-needed-xenial |
| Launchpad Janitor (janitor) wrote | #6 |
| Changed in linux (Ubuntu Xenial): | |
| status: | Fix Committed → Fix Released |
| Launchpad Janitor (janitor) wrote | #7 |
| Changed in linux (Ubuntu Bionic): | |
| status: | Fix Committed → Fix Released |
Also verified this with stress-ng's memory stressor class.
The system gets under memory pressure to the point of
calling shrink_slab() 10s to 100s thousand times a second
(verified with ftrace; function tracer; shrink_slab filter).
$ sudo stress-ng --class memory --all -1 &
# echo shrink_slab > /sys/kernel/
# echo function > /sys/kernel/
# cat /sys/kernel/
24384 998
38792 999
1 EVENTS]
22883 999
1 EVENTS]
1450 999
113849 1000
117025 1001
67285 1002
75768 1003
1 EVENTS]
14496 1003
170817 1004
98136 1005