amd_iommu possible data corruption

related branch
https://code.launchpad.net/~bladernr/ubuntu/+source/linux/+git/disco/+ref/amd-iommu-cherrypick

Status changed to 'Confirmed' because the bug affects multiple users.

Test kernels are up here:
https://people.canonical.com/~jlane/testkernels/

Please test each and report back

An additional patch has been identified that will be needed to solve this issue.

https://github.com/torvalds/linux/commit/3c677d206210f53a4be972211066c0f1cd47fe12#diff-083bf3d2f128b616e730b7f9f8fc65c4

Confirming that I was unable to reproduce the issue on linux-image-4.15.0-46-generic (4.15.0-46.49) from the test kernels repository with my test setup.

Previously I was able to cause filesystem corruption within 1 hour of testing under a test load (verified by btrfs checksums) across all disks on 2 different 32-core systems.

So I'm a bit confused because I've been told that A: additional patches are necessary to solve the issue, and B: that the issue is solved without those additional patches.

When you did the test, were you using the older, failing firmware levels, or have you updated to the patched firmware that is also part of this fix?

Jeff, The second patch improves the fix against the amount of memory in the system. We still need both patches.

Hi Jerry,

The test kernels are located here and have "1" appended to them.

https://people.canonical.com/~jlane/testkernels/

All three test kernels have passed 24+ hour testing.

SRU Request submitted

This bug was fixed in the package linux - 4.15.0-55.60

---------------
linux (4.15.0-55.60) bionic; urgency=medium

  * linux: 4.15.0-55.60 -proposed tracker (LP: #1834954)

  * Request backport of ceph commits into bionic (LP: #1834235)
    - ceph: use atomic_t for ceph_inode_info::i_shared_gen
    - ceph: define argument structure for handle_cap_grant
    - ceph: flush pending works before shutdown super
    - ceph: send cap releases more aggressively
    - ceph: single workqueue for inode related works
    - ceph: avoid dereferencing invalid pointer during cached readdir
    - ceph: quota: add initial infrastructure to support cephfs quotas
    - ceph: quota: support for ceph.quota.max_files
    - ceph: quota: don't allow cross-quota renames
    - ceph: fix root quota realm check
    - ceph: quota: support for ceph.quota.max_bytes
    - ceph: quota: update MDS when max_bytes is approaching
    - ceph: quota: add counter for snaprealms with quota
    - ceph: avoid iput_final() while holding mutex or in dispatch thread

  * QCA9377 isn't being recognized sometimes (LP: #1757218)
    - SAUCE: USB: Disable USB2 LPM at shutdown

  * hns: fix ICMP6 neighbor solicitation messages discard problem (LP: #1833140)
    - net: hns: fix ICMP6 neighbor solicitation messages discard problem
    - net: hns: fix unsigned comparison to less than zero

  * Fix occasional boot time crash in hns driver (LP: #1833138)
    - net: hns: Fix probabilistic memory overwrite when HNS driver initialized

  * use-after-free in hns_nic_net_xmit_hw (LP: #1833136)
    - net: hns: fix KASAN: use-after-free in hns_nic_net_xmit_hw()

  * hns: attempt to restart autoneg when disabled should report error
    (LP: #1833147)
    - net: hns: Restart autoneg need return failed when autoneg off

  * systemd 237-3ubuntu10.14 ADT test failure on Bionic ppc64el (test-seccomp)
    (LP: #1821625)
    - powerpc: sys_pkey_alloc() and sys_pkey_free() system calls
    - powerpc: sys_pkey_mprotect() system call

  * [UBUNTU] pkey: Indicate old mkvp only if old and curr. mkvp are different
    (LP: #1832625)
    - pkey: Indicate old mkvp only if old and current mkvp are different

  * [UBUNTU] kernel: Fix gcm-aes-s390 wrong scatter-gather list processing
    (LP: #1832623)
    - s390/crypto: fix gcm-aes-s390 selftest failures

  * System crashes on hot adding a core with drmgr command (4.15.0-48-generic)
    (LP: #1833716)
    - powerpc/numa: improve control of topology updates
    - powerpc/numa: document topology_updates_enabled, disable by default

  * Kernel modules generated incorrectly when system is localized to a non-
    English language (LP: #1828084)
    - scripts: override locale from environment when running recordmcount.pl

  * [UBUNTU] kernel: Fix wrong dispatching for control domain CPRBs
    (LP: #1832624)
    - s390/zcrypt: Fix wrong dispatching for control domain CPRBs

  * CVE-2019-11815
    - net: rds: force to destroy connection if t_sock is NULL in
      rds_tcp_kill_sock().

  * Sound device not detected after resume from hibernate (LP: #1826868)
    - drm/i915: Force 2*96 MHz cdclk on glk/cnl when audio power is enabled
    - drm/i915: Save the old CDCLK atomic state
...

This bug was fixed in the package linux - 5.0.0-21.22

---------------
linux (5.0.0-21.22) disco; urgency=medium

  * linux: 5.0.0-21.22 -proposed tracker (LP: #1834902)

  * Disco update: 5.0.15 upstream stable release (LP: #1834529)
    - net: stmmac: Use bfsize1 in ndesc_init_rx_desc
    - Drivers: hv: vmbus: Remove the undesired put_cpu_ptr() in hv_synic_cleanup()
    - ubsan: Fix nasty -Wbuiltin-declaration-mismatch GCC-9 warnings
    - staging: greybus: power_supply: fix prop-descriptor request size
    - staging: wilc1000: Avoid GFP_KERNEL allocation from atomic context.
    - staging: most: cdev: fix chrdev_region leak in mod_exit
    - staging: most: sound: pass correct device when creating a sound card
    - ASoC: tlv320aic3x: fix reset gpio reference counting
    - ASoC: hdmi-codec: fix S/PDIF DAI
    - ASoC: stm32: sai: fix iec958 controls indexation
    - ASoC: stm32: sai: fix exposed capabilities in spdif mode
    - ASoC: stm32: sai: fix race condition in irq handler
    - ASoC:soc-pcm:fix a codec fixup issue in TDM case
    - ASoC:hdac_hda:use correct format to setup hda codec
    - ASoC:intel:skl:fix a simultaneous playback & capture issue on hda platform
    - ASoC: dpcm: prevent snd_soc_dpcm use after free
    - ASoC: nau8824: fix the issue of the widget with prefix name
    - ASoC: nau8810: fix the issue of widget with prefixed name
    - ASoC: samsung: odroid: Fix clock configuration for 44100 sample rate
    - ASoC: rt5682: Check JD status when system resume
    - ASoC: rt5682: fix jack type detection issue
    - ASoC: rt5682: recording has no sound after booting
    - ASoC: wm_adsp: Add locking to wm_adsp2_bus_error
    - clk: meson-gxbb: round the vdec dividers to closest
    - ASoC: stm32: dfsdm: manage multiple prepare
    - ASoC: stm32: dfsdm: fix debugfs warnings on entry creation
    - ASoC: cs4270: Set auto-increment bit for register writes
    - ASoC: dapm: Fix NULL pointer dereference in snd_soc_dapm_free_kcontrol
    - drm/omap: hdmi4_cec: Fix CEC clock handling for PM
    - IB/hfi1: Clear the IOWAIT pending bits when QP is put into error state
    - IB/hfi1: Eliminate opcode tests on mr deref
    - IB/hfi1: Fix the allocation of RSM table
    - MIPS: KGDB: fix kgdb support for SMP platforms.
    - ASoC: tlv320aic32x4: Fix Common Pins
    - drm/mediatek: Fix an error code in mtk_hdmi_dt_parse_pdata()
    - perf/x86/intel: Fix handling of wakeup_events for multi-entry PEBS
    - perf/x86/intel: Initialize TFA MSR
    - linux/kernel.h: Use parentheses around argument in u64_to_user_ptr()
    - iov_iter: Fix build error without CONFIG_CRYPTO
    - xtensa: fix initialization of pt_regs::syscall in start_thread
    - ASoC: rockchip: pdm: fix regmap_ops hang issue
    - drm/amdkfd: Add picasso pci id
    - drm/amdgpu: Adjust IB test timeout for XGMI configuration
    - drm/amdgpu: amdgpu_device_recover_vram always failed if only one node in
      shadow_list
    - drm/amd/display: fix cursor black issue
    - ASoC: cs35l35: Disable regulators on driver removal
    - objtool: Add rewind_stack_do_exit() to the noreturn list
    - slab: fix a crash by reading /proc/slab_allocators
    - drm/sun4i: tcon top: Fix NULL/inv...

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Running Ubuntu 19.10 5.4.3-050403-generic

Ryzen 3900x GA x570 UD mainboard DDR 3200 ,

I have been getting data corruption on my 4x SSD on Sata ports ( btrfs/ext4)
disks running from my LSI / Avago SAS Raid 9302-16e 12Gb/s PCIe 3.0 controller (btrfs/EXT4)
and all my disks running from my HBA (zfs)

I started off with PCI-e gen 4 dropped it to gen 3 as that is what my raid controller is as not using a NVMe yet

all 64 gig Memory test's good and HDD's test good if tested individually anymore than 8 being accessed at a time they fail

being 3 separate controllers and multiple disks I am leading towards a software issue and I remember having data issues before when I had my a10-5800k and that was solved by disabling iommu , I had no issues with this setup while running my intel 4790k

seems I finally found the real issue

in the bios of my Gigabyte x570 UD , when I have 4G encode option enabled I have silent data corruption
( or at least if its enabled along side with IOMMU )

I need to make another array that I can kill to keep testing with , but I can literally go into the bios enable 4G reboot then start a zfs scrub and instantly have crc errors pop up everywhere , reboot disable do a scrub and it's all good , seems to only kick in when multiple disks are in use

In the past ~2012 with my Gigabyte F2A85x-UP4 AMD A10 5800k I also had data issues with IOMMU enabled ... never had issues with my intel i7 4790k build , so it could very well just be AMD IOMMU that is causing it for me

@MasterCATZ, please check this with the 20.04 5.4 kernel. If it's still an issue could you please file a separate kernel bug for your corruption issue?