kcryptd process doesn't utilize multiple cpus

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

Hi Xepra,

This bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? Can you try with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux-image-`uname -r` 246413

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

[This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

This bug seems to be have been hanging for some time. There are few dups out there of it.

I think the priority of the this bug should be put higher, encryption on enterprise machines is almost a must for most companies.

Linux kernel 2.6.30 added support for per-CPU thread for the crypto daemon.
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=254eff771441f4ee7aa9cf770a6e4820492c9dab

That must improve things.

Renice the process kcryptd between 0 and 5 improves the user experience drastically.
I am doing a heavy io load right now , burning an iso image to the harddisk. It still gets a 50%-60% usage , but it doesn't block. I can type and use the the os, what an idea right?

Even though it is taking 50% cpu , the desktop is still usable , before it would completely block.

sudo renice 0 (what ever kcryptd process id is)

I'm running 9.04 with encrypted HDD (it's a corporate laptop so needs to be). I need to run VM quite often and this issue is causing me lots of pain. Especially when copying / moving large files (2+ GB)

Renice tip above seems to cure some of the performance issue (i.e. CPU usage goes back to normal levels) but it's far from ideal.

Looks like this bug has been around for a while and as encryption is a must - especially in the corporate user base - I would suggest the priority of this bug be upped (in agreement with bongey above)

I am also experiencing this on a brand new 9.04 64-bit server installation on a very capable machine (Xeon 5520, 8GB RAM, HP SA RAID controller).

With one thread writing to the encrypted volume the system slows down with 2 kcryptd threads at 50-75% and ~6.00 load.
With two threads writing to the encrypted volume the system becomes nearly completely unresponsive with a load average around 15-16.

Re-nicing the kcryptd threads to 0 (changed from -5) makes the system much more responsive but does not improve read/write speeds at all.

I'm getting read speeds around 20MB/s under load, 40MB/s when idle. I'm seeing reads at over 200MB/s on the unencrypted portions of the volume.

Has anyone had a chance to try the new kernel mentioned above? From a design standpoint, I wonder if this is really a bug at all, but a design issue and perhaps even a necessary constrain given that there is only one pathway from disk to bus.

I can reproduce this issue with 9.10-rc (kernel version 2.6.31-14-server). I have also seen it in previous versions and on different hardware.

The slower speeds when dm-crypt is in use are obvious when testing with bonnie++ as seen below. The /data partition is an ext4-formatted 32G LVM logical volume on a multi-terabyte software RAID 10. The same volume is used in both tests - right after the first test I unmounted it, did a luksFormat, luksOpen, formatted it (with the same options) and mounted it.

The server is a quad-core Xeon and there is no reason it should be bottlenecked on this.

Without dm-crypt:

owen@fsv:~$ bonnie++ -f -n 0 -d /data
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
fsv 8G 193134 25 57920 11 171750 15 605.2 1

With dm-crypt:

owen@fsv:~$ bonnie++ -f -n 0 -d /data
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
fsv 8G 132252 8 31394 5 63636 6 426.7 1

Is everyone using ext3 or ext4 partitions?

Raid or no raid?

I didn't see the same issues with an ext3 partition but I am not for sure.

Running 9.10 on my wife's Lenovo netbook she complained of very slow performance. I investigated and found the CPU and hard drive thrashing pretty badly. Running top showed kcryptd using 80% of the CPU. She's running ext3.

While running 9.04 and 9.10 (I have tried both) the kcryptd process uses just one core on an atom 330.
I'm seeing about 18-19MB/s throughput.

I doubt this is a bug, and instead I suspect that it needs to be submitted as a feature request. Clearly, encryption will be used more and more often, and there will continue to be more and more cpus with multiple cores.

Here are some others noticing the same issue:
http://<email address hidden>/msg1643037.html
https://lists.linux-foundation.org/pipermail/bugme-new/2008-May/018825.html
http://<email address hidden>/msg03648.html

Clearly, the maximum throughput is not the only issue since others above are complaining that they have issues with system responsiveness.

I realize the atom is not very efficient at this type of calculations, but a Q6600 still being limited to 80MB/s?!! Maybe a recompile would help, but I think a multithreaded implementation is needed. Where would I go to plead for/request this?

Here a small hack is discribet to use multiple cores. http://atom330.iuselinux.net/maximise-dm-crypt

mainly use a raid0 on one disk.

Are there any news about true multithredding for dmcrypt?

AplayDevices: Error: [Errno 2] No such file or directory
Architecture: i386
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D1p', '/dev/snd/hwC0D2', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info: Error: [Errno 2] No such file or directory
Card0.Amixer.values: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 9.10
HibernationDevice: RESUME=/dev/mapper/ironbox-swap_1
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.

 virbr0 no wireless extensions.
Lsusb:
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Package: linux-image-2.6.31-20-generic-pae 2.6.31-20.57
PackageArchitecture: i386
ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.31-20-generic-pae root=/dev/mapper/ironbox-root ro quiet splash
ProcEnviron:
 SHELL=/bin/bash
 LANG=it_IT.UTF-8
ProcVersionSignature: Ubuntu 2.6.31-20.57-generic-pae
RelatedPackageVersions:
 linux-backports-modules-2.6.31-20-generic-pae N/A
 linux-firmware 1.26
RfKill:

Uname: Linux 2.6.31-20-generic-pae i686
UserGroups: adm admin cdrom dialout libvirtd lpadmin plugdev sambashare
dmi.bios.date: 05/28/2009
dmi.bios.vendor: Intel Corp.
dmi.bios.version: LF94510J.86A.0182.2009.0528.2014
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: D945GCLF2D
dmi.board.vendor: Intel Corporation
dmi.board.version: AAE59323-104
dmi.chassis.type: 3
dmi.modalias: dmi:bvnIntelCorp.:bvrLF94510J.86A.0182.2009.0528.2014:bd05/28/2009:svn:pn:pvr:rvnIntelCorporation:rnD945GCLF2D:rvrAAE59323-104:cvn:ct3:cvr:

Hello,
I have just added my system details, b/c I am having the same problem, I have an Atom330 CPU on a D945GCLF2 Intel Board with 3 HDDs connected.

1) One small hard disk contain the operating system, Ubuntu server 9.10 x32
2) Two hard disks (2Tb) in Raid 1 (mirror) with XFS filesystem on an encrypted md0 device

I know that Atom is a slow cpu for cryptographic processes, but using "top" I can see that kcryptd uses only 93-94% (of 200% max) of CPU time, and I have usually 45-60% of CPU idle time, while tranferring a BIG file. I have noticed about 18-19MB/s throughput.

Is there a way to use either CPU core for cryptd daemon??
Is it connected with the type of crypto algorithm used?

Please try the work around by derPeter for a raid device. I am pretty sure you will get better results.

Hi bongey, I checked the hack from derPeter, but it applies on RAID-0 (striping) raid type.

I have RAID-1 (Mirroring), and I am not so smart to "convert" the hack from raid-0 to raid-1, if this is possible.

I probably could make a "false" raid0 by splitting the available space in two big chuncks, and then applying the derPeter hack, but I think it's a very dirty solution.

Is this bug solved in the new LTS 4.11 ???

Thank you!!

Sorry, I meant 4.10 (Warty Warthog), not 4.11 of course! My mistake!

Ok, Double error. I was TRYING to ask (sorry for my double error) if the bug would be fixed in the new LTS version, the 10.4 Lucid Lynx, and not the 4.10...

Where do we ask to get the "new functionality" of multithreaded kcryptd?

When copying/moving a large file from an encrypted volume -while logged in on an encrypted home- on my I7 with SATA-300 disk, my 9.10 64-bit desktop slows down considerably.
When I move/copy the file while logged in on an unencrypted home the desktop is still usable.

I noticed I have 4 crypto processes running.

It looks like Gnome waits for requests somehow, but is unable to complete until the copy/move ends.
I also have the feeling the kcryptd process is handling one request at a time, and is not multithreaded.

ubuntu 9.10 amd64
kernel 2.6.31-20-generic #58-Ubuntu SMP
cryptsetup 2:1.0.6+20090405.svn49-1ubuntu7.2
I7 + ICH10 + AHCI + SATA300 disk

What's the status of this?
Any progress?

Canonical?

I don't think Canonical will fix this - They might care about showstopper bugs like the Lucid Xorg memory leak when they're in a good mood, but encryption is only used by a minority of the Ubuntu community and it works reliably already. This is just a feature request, so don't expect anything to happen. I'd say if Debian gets a fix some day Ubuntu will grab it. But threading isn't easy and concurrency bugs are hard to track down, so I don't think it will happen soon and it will probably even take a while to get it upstream into the kernel. Don't get me wrong, I'd love to see such a patch - I'm just being realistic!

I'm running Lucid now and this bug has been driving me nuts since Jaunty. My machine often hard locks for a few seconds when under heavy disk I/O as it probably waits for a buffer to sync.

This post might help to shed some light on the problem and a potential solution:
http://osdir.com/ml/linux.kernel.device-mapper.dm-crypt/2006-11/msg00018.html

@Dekar

I understand.

But, if Canonical adds the possibility of encrypted home, then they also introduce this problem.
I wouldn't call it a bug, as it works as intended, but anyway.

Canonical also chose to implement plymouth in 10.04 (Debian put it back to testing) and solved most of it's problems, so why not for this one?

Can I get any official statement from Canonical here?

Sorry, should have given you the link for plymouth progress:

https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/571707

Bummer, I thought I'd check in to see if there was some progress made.

NO PROGRESS ?

Doesn't look like any progress so far. It's really strange that this wasn't factored into the design when this feature was added to the kernel. You'd think that surely somebody must have noticed early on?

So sad it makes me mad ;(

My CPU is capable of 130MB/sec AES-256. My drives, 119MB/sec write. Combine this into a dm-crypt volume: 11MB/sec (eleven megabytes per second [!]). Uh, this thing seems broken to me.

I'm using xfs on dm-crypt.

Anyone know if iostat can provide some useful information? I'm wondering about I/O thrashing like we have been seeing with grub and apt lately. Something is/was seriously wrong with those too.

This bug is driving me nuts. My machine frequently hard locks for 5 to 15 seconds or more, countless times throughout the day.

a simplier workarround not using LVM or Raid0 is just to create multiple encrypted partitions, each one will run a single kcryptd, but you can isolate the i/o demanding volumes, like the swap or a partition to hold virtual machine disk images

the downside is you need to enter your password multiple times at boot...
somebody said that if you create them at install with the alternate disk you can just use one password, but i lost the reference

at the linux irc somebody suggested the recommended number of partitions is your number of cpu cores + 1

i'm running two partitions on a netbook now to isolate my virtual machine and it seems to work nice, sometimes the computer become slow, but it is not freezing, my VM is not saturating the kcryptd bottleneck anymore, it has its own bottleneck to deal with :P

simplier and more flexible than the raid0 hack, well...

Debian squeeze seems to be on kernel 2.6.32
The advertised patch for dm-crypt multi-threading is from kernel 2.6.30
However as of today, a fresh Debian installation with dm-crypt only uses 1 cpu core.
Is it possible to enable it with Debian or did they just not include the patch in their kernel?

Xepra, thank you for reporting this and helping make Ubuntu better. Hardy desktop reached EOL on May 12, 2011.
Please see this document for currently supported Ubuntu releases:
https://wiki.ubuntu.com/Releases

We were wondering if this is still an issue on a supported release? If so, can you try with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you run the following command in a supported release from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux <replace-with-bug-number>

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.