Running processor intensive processes slows down disk buffer cache long after processes are completed

This issue seems similar to https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/131094, however I created a new bug because the processes I am running are more CPU intensive than IO intensive.

Due my lack of understanding of the disk buffer cache, I will probably need some handholding.

I'm running a pretty heavy-weight process (Rails tests) that involves several worker processes, in an effort to parallelize the runs.

To measure the performance impacts, I run hdparm -T /dev/sda to give me the cached read performance. Note that the disk IO is not being measured, but the disk cache IO is.

It works very well on my work machine (8-core Mac Pro running Ubuntu with 8GB of RAM).

The baseline is:
honk4:~ $ sudo hdparm -T /dev/sda
/dev/sda:
Timing cached reads: 13224 MB in 2.00 seconds = 6616.86 MB/sec

In the middle of the test run:
honk4:~ $ sudo hdparm -T /dev/sda
/dev/sda:
Timing cached reads: 4598 MB in 2.00 seconds = 2299.26 MB/sec

After the test run is finished, hdparm returns back to the original result. The test suite ran 8 workers.

---

On my laptop, it is a different story.
I have a quad core machine with 3.4GB of RAM.
The test suite runs 4 workers.

Now the baseline is around 6000 MB/sec.
When I run the suite, it ranges between 2000-5000 MB/sec.
After the suite is finished, the performance keeps on dropping. It finally goes down to ~ 350 MB/sec.

What is strange is around 30 minutes later, hdparm -T goes back to 6000 MB/sec.

I tried sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches', but it does not seem to help.

---

Having less worker processes decreases the severity and duration of the slowness.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: linux-image-2.6.32-21-generic 2.6.32-21.32
Regression: No
Reproducible: Yes
ProcVersionSignature: Ubuntu 2.6.32-21.32-generic 2.6.32.11+drm33.2
Uname: Linux 2.6.32-21-generic i686
NonfreeKernelModules: nvidia wl
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.21.
AplayDevices:
 **** List of PLAYBACK Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: STAC92xx Analog [STAC92xx Analog]
   Subdevices: 0/1
   Subdevice #0: subdevice #0
Architecture: i386
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: STAC92xx Analog [STAC92xx Analog]
   Subdevices: 2/2
   Subdevice #0: subdevice #0
   Subdevice #1: subdevice #1
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: btakita 2154 F.... pulseaudio
 /dev/snd/pcmC0D0p: btakita 2154 F...m pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xf6fdc000 irq 21'
   Mixer name : 'IDT 92HD71B7X'
   Components : 'HDA:111d76b2,10280250,00100302'
   Controls : 21
   Simple ctrls : 15
CheckboxSubmission: 0284e32f1bd1ad9229c88cd83df0f446
CheckboxSystem: d00f84de8a555815fa1c4660280da308
Date: Mon Apr 19 11:42:04 2010
HibernationDevice: RESUME=UUID=43e19dcb-222a-41b1-b5ed-d0dcb1373c00
MachineType: Dell Inc. Precision M4400
PccardctlIdent:
 Socket 0:
   no product info available
PccardctlStatus:
 Socket 0:
   no card
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.32-21-generic root=UUID=f6ca8c66-fac4-4b0e-8fe5-545b866ec315 ro elevator=noop quiet splash
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
RelatedPackageVersions: linux-firmware 1.34
SourcePackage: linux
WifiSyslog:
 Apr 19 10:35:56 saturn kernel: [ 2573.965068] dell-wmi: Unknown key ffd0 pressed
 Apr 19 11:22:24 saturn kernel: [ 5361.495008] dell-wmi: Unknown key ffd1 pressed
WpaSupplicantLog:

dmi.bios.date: 02/13/2009
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A12
dmi.board.name: 0NY980
dmi.board.vendor: Dell Inc.
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA12:bd02/13/2009:svnDellInc.:pnPrecisionM4400:pvr:rvnDellInc.:rn0NY980:rvr:cvnDellInc.:ct8:cvr:
dmi.product.name: Precision M4400
dmi.sys.vendor: Dell Inc.