Comment 71 for bug 1626436

So I did a little bisecting today on mainline x86_64 kernels with CONFIG_SLUB:

v4.4 Startup finished in 2.718s (kernel) + 11.831s (userspace) = 14.549s
v4.5 Startup finished in 2.707s (kernel) + 12.777s (userspace) = 15.484s
v4.6-rc4-1018-g5a5f0792e98b Startup finished in 2.970s (kernel) + 13.641s (userspace) = 16.611s
v4.6-3615-g1eccc6e1529e Startup finished in 3.005s (kernel) + 14.063s (userspace) = 17.069s
v4.6-5396-g07b75260ebc2 Startup finished in 3.044s (kernel) + 14.932s (userspace) = 17.977s
v4.6-5931-gf4c80d5a16eb Startup finished in 3.057s (kernel) + 13.429s (userspace) = 16.487s
v4.6-6084-g78975f23cba0 Startup finished in 3.050s (kernel) + 13.903s (userspace) = 16.954s
v4.6-5050-g18726ca8b34b Startup finished in 3.015s (kernel) + 13.510s (userspace) = 16.525s
v4.6-5055-g6052b7880a95 Startup finished in 3.029s (kernel) + 13.519s (userspace) = 16.549s
v4.6-5058-g76b342bdc71b Startup finished in 2.998s (kernel) + 14.179s (userspace) = 17.177s
v4.6-5060-g801faf0db894 Startup finished in 3.064s (kernel) + 14.667s (userspace) = 17.732s
v4.6-5061-g81ae6d03952c Startup finished in 3.057s (kernel) + 18.357s (userspace) = 21.414s
v4.6-5073-g9cc3387fa29f Startup finished in 3.073s (kernel) + 17.976s (userspace) = 21.049s
v4.6-5119-g682a3385e773 Startup finished in 3.089s (kernel) + 17.987s (userspace) = 21.077s
v4.6-6266-ga05a70db34ba Startup finished in 3.037s (kernel) + 18.550s (userspace) = 21.588s
v4.6-7135-g0eff4589c36e Startup finished in 3.008s (kernel) + 18.074s (userspace) = 21.082s
v4.7-rc5-1395-g877fa9a42ddc Startup finished in 3.179s (kernel) + 19.147s (userspace) = 22.327s
v4.8 Startup finished in 3.197s (kernel) + 20.836s (userspace) = 24.034s
v4.9.7 Startup finished in 3.674s (kernel) + 20.034s (userspace) = 23.708s
v4.9.7 + the two patches Startup finished in 3.518s (kernel) + 13.027s (userspace) = 16.546s

What we can observe from this:

The big bump came in https://github.com/torvalds/linux/commit/81ae6d03952c1bfb96e1a716809bd65e7cd14360 (or alternatively in https://github.com/torvalds/linux/commit/801faf0db8947e01877920e848a4d338dd7a99e7 for CONFIG_SLAB, if bz.k.o is trustworthy). This is likely fixed by https://github.com/torvalds/linux/commit/86d9f48534e800e4d62cdc1b5aaf539f4c1d47d6, https://github.com/torvalds/linux/commit/13583c3d3224508582ec03d881d0b68dd3ee8e10 and https://github.com/torvalds/linux/commit/89e364db71fb5e7fc8d93228152abfa67daf35fa. The last two are not in stable, though.

Speaking of https://github.com/torvalds/linux/commit/86d9f48534e800e4d62cdc1b5aaf539f4c1d47d6, this went into v4.8.7 as https://github.com/torvalds/linux/commit/68adb469dd3b28513c5b548df3437f2bc1a83e43, so that may be the fix you were looking for in comment #65.

There are also lots of small performance regressions here and there that may be difficult to bisect. A second in userspace between v4.4 and v4.5 and another second between v4.5 and v4.6. Some 300 ms in kernel between v4.5 and v4.6, 200 ms between v4.6 and v4.7 and another 300 ms between v4.8 and v4.9. That's here, on my .config and my hardware. Dimitrij has 10 seconds in kernel between v4.7 and v4.8-rc1, but I can't see anything like that here. (In this case you might want to look into dmesg if it's a general slowdown or there's one thing that blocks for 10 secs. I suspect there might be.)

I think that these other slowdowns that I and Dimitrij and possibly others are experiencing are likely unrelated to the main issue here (https://bugzilla.kernel.org/show_bug.cgi?id=172981, https://bugzilla.kernel.org/show_bug.cgi?id=172991) and we'll need to bisect further ourselves. :-)