KSM causing performance and instability issues

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1435363

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

I'll be setting up a new server in the next few days, I'll attempt to use -48 and see if that issue is present or not.

Can you also test with /sys/kernel/mm/ksm/merge_across_nodes set to 0 (and KSM enabled normally) and confirm this is a KSM+NUMA issue?

Another way to potentially cause this issue to occur faster would be to set:
/sys/kernel/mm/ksm/sleep_millisecs to a value lower than 200
or
/sys/kernel/mm/ksm/pages_to_scan to a much larger number than 100

Thanks,
--chris j arges

In addition could you explain more in depth about the performance an instability issues you've observed? How do you detect them or test for them? Thanks

I've attempted to reproduce bug 1346917 again on a NUMA machine and was unable to do so with the latest 3.13 kernel. Perhaps I could have more details on how your reproducing this issue to assist with debugging? Thanks

Hi Chris,

Thanks for the help so far. I'm deploying a new machine right now and I'll be trying to replicate it on -48.

The way I detected it was that i'd see messaging in "dmesg" on guest similar to this:

hrtimer: interrupt took 4352551231 ns

In addition, when pinging the machine, you'd have a few seconds of stable pings, then unresponsive for 2-3s, and it starts responding again (with a huge delay, latency of 3s to 4s because of the delay).

I will be running this machine and monitoring it closely and report on the output, however, I'd like to note that these machines have heavy KSM usage, before turning it off, one had almost ~45-50GB of deduplicated memory on a 256GB node, so I'm not sure if that plays in as a factor..

I'll report back on -48 and see what I can check

Thank you,
Mohammed

Hi,

I installed a new machine from the 14.04.2 media which gave me the HWE stack with kernel 3.16.0-33-generic and it's running with no problems. The machine is now loaded up to 125GB worth of VMs and with the following memory stats:

# free -m
             total used free shared buffers cached
Mem: 257599 253046 4553 3 218 161499
-/+ buffers/cache: 91328 166271
Swap: 0 0 0

So, the server is quite loaded and I haven't seen any hiccups. We'll see from there on.

Thanks,
Mohammed