Kernel network namespace performance regression during rcu development on kernels above 3.8
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| linux (Ubuntu) |
Fix Released
|
Undecided
|
Rafael David Tinoco | ||
| Trusty |
Fix Released
|
Undecided
|
Rafael David Tinoco | ||
| Utopic |
Fix Released
|
Undecided
|
Rafael David Tinoco | ||
Bug Description
SRU Justification:
Impact: network namespace creation has performance regression since v3.5.
Fix: my analysis, lklm discussion, upstream patch
Testcase:
http://
http://
http://
http://
Running make_fake_
"fake routers" are being created in a good rate /sec (and you can
compare with all generated charts).
-------
Original Description:
Please, follow this in: http://
--
It was brought to my attention that network namespace creation scalability was affected during kernel development.
The following script was used for all the tests and charts generation:
http://
http://
I measured how many "fake routers" (above script) could be added per second from 0 to 4000 created routers mark. Using this script and a git bisect on kernel tree I was led to one specific commit causing regression: #911af50 "rcu: Provide compile-time control for no-CBs CPUs". Even Though this change was experimental at that point, it introduced a performance scalability regression (explained below) that still last and seems to be the default option for distributions nowadays.
RCU related code looked like to be responsible for the problem. With that, every commit from tag v3.8..master that changed any of this files: "kernel/rcutree.c kernel/rcutree.h kernel/
All text below this refer to 2 groups of charts, generated during the study:
1) Kernel git tags from 3.8 to 3.14.
http://
2) Kernel git commits for rcu development (111 commits).
http://
Since there was difference in results depending on how many cpus or how the no-cb cpus were configured, 3 kernel config options were used on every measure:
- CONFIG_RCU_NOCB_CPU (disabled): nocbno
- CONFIG_
- CONFIG_
Obs: For 1 cpu cases: nocbno, nocbnone, nocball behaves the same since w/ only 1 cpu there is no no-cb cpu
After charts being generated it was clear that NOCB_CPU_ALL (4 cpus) affected the "fake routers" creation process performance and this regression continues up to upstream version. It was also clear that, after commit #911af50, having more than 1 cpu does not improve performance/
#911af50
...
+#ifdef CONFIG_
+ pr_info(
+ cpumask_
+#endif /* #ifdef CONFIG_
...
Comparing standing out points (see charts):
#81e5949 - good
#911af50 - bad
I was able to see that, from the script above, the following lines causes major impact on netns scalability/
1) ip netns add -> huge performance regression:
1 cpu: no regression
4 cpu: regression for NOCB_CPU_ALL
obs: regression from 250 netns/sec to 50 netns/sec
on 500 netns already created mark
2) ip netns exec -> some performance regression
1 cpu: no regression
4 cpu: regression for NOCB_CPU_ALL
obs: regression from 40 netns (+1 exec per netns
creation) to 20 netns/sec on 500 netns created
mark
# Assumption (to be confirmed)
rcu callbacks being offloaded to other cpus caused regression in copy_net_
| summary: |
- Kernel network namespace performance regression during kernel - development + Kernel network namespace performance regression during rcu development + on kernels above 3.8 |
| description: | updated |
| Changed in linux (Ubuntu): | |
| assignee: | nobody → Rafael David Tinoco (inaddy) |
| status: | New → In Progress |
| affects: | linux (Ubuntu) → linux |
| description: | updated |
| description: | updated |
| tags: | added: bisect-done |
| Chris J Arges (arges) wrote | #2 |
| Rafael David Tinoco (rafaeldtinoco) wrote | #3 |
| description: | updated |
| Changed in linux (Ubuntu): | |
| status: | New → Fix Committed |
| Changed in linux (Ubuntu Trusty): | |
| assignee: | nobody → Rafael David Tinoco (inaddy) |
| status: | New → Fix Committed |
| Launchpad Janitor (janitor) wrote | #4 |
| Changed in linux (Ubuntu Utopic): | |
| status: | Fix Committed → Fix Released |
| Brad Figg (brad-figg) wrote | #5 |
| tags: | added: verification-needed-trusty |
| tags: |
added: verification-done removed: verification-needed-trusty |
| tags: |
added: verification-done-trusty removed: verification-done |
| Launchpad Janitor (janitor) wrote | #6 |
| Changed in linux (Ubuntu Trusty): | |
| status: | Fix Committed → Fix Released |
| tags: | added: cts |
| Changed in linux: | |
| assignee: | Rafael David Tinoco (inaddy) → nobody |
| Changed in linux (Ubuntu Trusty): | |
| assignee: | Rafael David Tinoco (inaddy) → nobody |
| no longer affects: | linux |
| Changed in linux (Ubuntu): | |
| assignee: | nobody → Rafael David Tinoco (inaddy) |
| Changed in linux (Ubuntu Trusty): | |
| assignee: | nobody → Rafael David Tinoco (inaddy) |
| Changed in linux (Ubuntu Utopic): | |
| assignee: | nobody → Rafael David Tinoco (inaddy) |
| Gnanasekar Velu (gnanasekarkas) wrote | #7 |
| Dave Chiluk (chiluk) wrote | #8 |
| Don Bowman (donbowman) wrote | #9 |
Related upstream discussion:
https:/