Support non-strict iommu mode on arm64
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Undecided
|
dann frazier | ||
Bionic |
Fix Released
|
Undecided
|
dann frazier | ||
Cosmic |
Fix Released
|
Undecided
|
dann frazier | ||
Disco |
Fix Released
|
Undecided
|
dann frazier | ||
Jammy |
Invalid
|
Undecided
|
Unassigned | ||
Kinetic |
Invalid
|
Undecided
|
Unassigned | ||
Lunar |
Fix Released
|
Undecided
|
dann frazier | ||
linux-aws (Ubuntu) |
Fix Released
|
Medium
|
Tim Gardner | ||
Bionic |
Invalid
|
Undecided
|
Unassigned | ||
Cosmic |
Invalid
|
Undecided
|
Unassigned | ||
Disco |
Invalid
|
Undecided
|
Unassigned | ||
Jammy |
Fix Released
|
Medium
|
Tim Gardner | ||
Kinetic |
Fix Released
|
Medium
|
Tim Gardner | ||
Lunar |
Fix Released
|
Medium
|
Tim Gardner |
Bug Description
[Impact]
The Intel IOMMU driver provides an option for strict mode. When disabled, batching of IOTLB flush operations is permitted, allowing the user to trade-off isolation for improved performance. Ubuntu's kernel currently lacks a parity for this feature for ARM.
There's a significant performance gain to be had by removing the need to flush the IOMMU TLB on every unmap on arm64. I'm seeing a 25% performance gain w/ fio reads on a single NVMe device.
This mode of operation is available for x86 via the "intel_
[Test Case]
Run fio with the following config before and after applying the patches and collection IOPS count. Run again after applying the patches. Finally, run a 3rd time after adding iommu.strict=0 to the kernel commandline.
Performance should not regress after the update. Performance should further improve after adding iommu.strict=0 - but if it doesn't for some reason, that is not a regression.
$ cat fio.rc
[global]
rw=read
direct=1
ioengine=libaio
iodepth=2048
numjobs=10
bs=4k
group_reporting=1
group_reporting=1
cpumask=0xff
runtime=100
loops = 10000
[job1]
filename=
[Fix]
44f6876a00e83 iommu/arm-smmu: Support non-strict mode
b2dfeba654cb0 iommu/io-
9662b99a19abc iommu/arm-smmu-v3: Add support for non-strict mode
b6b65ca20bc93 iommu/io-
68a6efe86f6a1 iommu: Add "iommu.strict" command line option
2da274cdf998a iommu/dma: Add support for non-strict mode
07fdef34d2be6 iommu/arm-smmu-v3: Implement flush_iotlb_all hook
85c7a0f1ef624 iommu/io-
[Regression Risk]
Most of these patches are specific to ARM, and have been regression tested on both arm64 (HiSilicon D06) and armhf (QEMU virt) using "stress-ng --vm $(nproc)"
2 patches do touch arch-indep code however:
> 68a6efe86f6a1 iommu: Add "iommu.strict" command line option
Adds a new command line option and sets an attribute that iommu drivers can optionally react to. Doesn't change default behavior.
> 2da274cdf998a iommu/dma: Add support for non-strict mode
This driver is only built for arm64 and ppc64el (determined by looking at the build logs). Most of this patch only changes behavior in the non-default (and new) iommu.strict=0 case. The exception, which is called out in the commit message, is this hunk:
- WARN_ON(
+ WARN_ON(
+ if (!cookie-
+ iommu_tlb_
In the default case, where fq_domain will be NULl, we are now factoring iommu_unmap() into:
iommu_
iommu_tlb_sync()
Looking at the source to iommu_unmap() confirms that this is functionally equivalent.
Changed in linux (Ubuntu Cosmic): | |
assignee: | nobody → dann frazier (dannf) |
Changed in linux (Ubuntu Bionic): | |
assignee: | nobody → dann frazier (dannf) |
description: | updated |
Changed in linux (Ubuntu Cosmic): | |
status: | New → In Progress |
Changed in linux (Ubuntu Bionic): | |
status: | New → In Progress |
Changed in linux (Ubuntu Disco): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Cosmic): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Bionic): | |
status: | In Progress → Fix Committed |
description: | updated |
tags: | added: cscc |
Changed in linux-aws (Ubuntu Bionic): | |
status: | New → Invalid |
Changed in linux-aws (Ubuntu Cosmic): | |
status: | New → Invalid |
Changed in linux-aws (Ubuntu Disco): | |
status: | New → Invalid |
Changed in linux-aws (Ubuntu Jammy): | |
status: | New → In Progress |
Changed in linux-aws (Ubuntu Kinetic): | |
status: | New → In Progress |
Changed in linux (Ubuntu Jammy): | |
status: | New → Invalid |
Changed in linux (Ubuntu Kinetic): | |
status: | New → Invalid |
Changed in linux-aws (Ubuntu): | |
status: | New → Invalid |
Changed in linux-aws (Ubuntu Jammy): | |
status: | In Progress → Fix Committed |
Changed in linux-aws (Ubuntu Kinetic): | |
status: | In Progress → Fix Committed |
Changed in linux-aws (Ubuntu Jammy): | |
status: | Fix Committed → In Progress |
assignee: | nobody → Tim Gardner (timg-tpi) |
importance: | Undecided → Medium |
Changed in linux-aws (Ubuntu Kinetic): | |
assignee: | nobody → Tim Gardner (timg-tpi) |
importance: | Undecided → Medium |
status: | Fix Committed → In Progress |
Changed in linux-aws (Ubuntu Lunar): | |
assignee: | nobody → Tim Gardner (timg-tpi) |
importance: | Undecided → Medium |
status: | Invalid → In Progress |
Changed in linux-aws (Ubuntu Jammy): | |
status: | In Progress → Fix Committed |
Changed in linux-aws (Ubuntu Kinetic): | |
status: | In Progress → Fix Committed |
Changed in linux-aws (Ubuntu Lunar): | |
status: | In Progress → Fix Committed |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification- needed- cosmic' to 'verification- done-cosmic' . If the problem still exists, change the tag 'verification- needed- cosmic' to 'verification- failed- cosmic' .
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/ /wiki.ubuntu. com/Testing/ EnableProposed for documentation how to enable and use -proposed. Thank you!