[thunderx] Synchronous External Abort: synchronous parity or ECC error

Also reproducible w/ the 5.0.0-37.40 kernel. I'll try a mainline 5.5-rc6 build next.

[ 602.796765] Internal error: synchronous parity or ECC error: 96000018 [#1] SMP
[ 602.803994] Modules linked in: nls_iso8859_1 cavium_rng_vf ipmi_ssif ipmi_devintf input_leds joydev ipmi_msghandler thunderx_edac cavium_rng sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear aes_ce_blk aes_ce_cipher nicvf cavium_ptp ast i2c_algo_bit ttm drm_kms_helper crct10dif_ce ghash_ce syscopyarea sysfillrect sha2_ce sysimgblt uas hid_generic nicpf fb_sys_fops sha256_arm64 drm sha1_ce usbhid usb_storage hid thunder_bgx ahci thunder_xcv i2c_thunderx mdio_thunder thunderx_mmc mdio_cavium aes_neon_bs aes_neon_blk crypto_simd cryptd aes_arm64
[ 602.872414] Process cc1 (pid: 40126, stack limit = 0x0000000090887c2f)
[ 602.878949] CPU: 10 PID: 40126 Comm: cc1 Not tainted 5.0.0-37-generic #40~18.04.1-Ubuntu
[ 602.887040] Hardware name: GIGABYTE R120-T33/MT30-GS1, BIOS T49 02/02/2018
[ 602.893921] pstate: 80000005 (Nzcv daif -PAN -UAO)
[ 602.898724] pc : __arch_copy_to_user+0x13c/0x248
[ 602.903353] lr : cp_new_stat+0x140/0x178
[ 602.907277] sp : ffff00002599bcc0
[ 602.910594] x29: ffff00002599bcc0 x28: ffff800ed0538ec0
[ 602.915912] x27: 0000000000000000 x26: 0000000000000000
[ 602.921229] x25: 0000000056000000 x24: 0000000000000015
[ 602.926547] x23: ffff000010c716d8 x22: 000000002599bd08
[ 602.931865] x21: ffff800ed0538ec0 x20: ffff00001170c000
[ 602.937181] x19: ffff00002599bdb0 x18: 0000000000000000
[ 602.942498] x17: 0000000000000000 x16: 0000000000000000
[ 602.947818] x15: 0000000000000000 x14: 0000000000000000
[ 602.953134] x13: 0000000000000000 x12: 0000000000000000
[ 602.958452] x11: 0000000000000000 x10: 000000000000152f
[ 602.963769] x9 : 0000000000001000 x8 : 00000001000081a4
[ 602.969087] x7 : 0000000000a60da3 x6 : 000000002599bd20
[ 602.974405] x5 : 000000002599bd88 x4 : 0000000000000008
[ 602.979721] x3 : 0000000000000802 x2 : fffffffffffffff8
[ 602.985038] x1 : ffff00002599bd10 x0 : 000000002599bd08
[ 602.990356] Call trace:
[ 602.992821] __arch_copy_to_user+0x13c/0x248
[ 602.997107] __se_sys_newfstat+0x58/0x88
[ 603.001045] __arm64_sys_newfstat+0x20/0x30
[ 603.005243] el0_svc_common+0x88/0x180
[ 603.009005] el0_svc_handler+0x38/0x78
[ 603.012770] el0_svc+0x8/0xc
[ 603.015664] Code: a8c12027 a88120c7 d503201f d503201f (a8c12829)
[ 603.021765] ---[ end trace 08068f2978fb8211 ]---

All 3 of my machines survived overnight testing on the 5.5-rc6 mainline build[*].
Next step is to try 5.3. 5.3 mainline doesn't boot on these systems, so I'll use Ubuntu's 5.3.0-24.

[*] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.5-rc6/

I attempted to bisect this, using the following process:
  - Run the kernel-build-reboot-loop test on 3 machines in parallel
    I used 2 CRB1S systems (anuchin, bestovius) and 1 R120-T33 (seidel)
  - If any machine crashes w/ the parity error message, consider it failed
  - If all machines survive over night, consider it "OK".

Unfortunately, the commit it landed on looks bogus:

# first bad commit: [852643165aea0999bb862b36511c5b9f6b11449f] fs//binfmt_elf.c: move variables initialization closer to their usage
(Reverse bisect - this would in theory be the commit that *fixed* it)

Just in case, I tried reverting that commit from 5.5-rc6. As noted in comment #2, 5.5-rc6 seems immune to this problem. Reverting the commit didn't change that - 5.5-rc6 still survived over night.

Note: Of the 3 systems, anuchin was usually the one that failed during the bisect. It could be that this is a generic hw issue, and anuchin is just more severely impacted than the others. It could also be that this symptom can be caused by both a sw and a hw issue, and anuchin is impacted by the hw part, making it a bad choice for a bisect. Either way, bisection seems like a poor strategy for identifying the issue.

Looking at the git log - I wonder if this could be related?

commit 94bb804e1e6f0a9a77acf20d7c70ea141c6c821e
Author: Pavel Tatashin <email address hidden>
Date: Tue Nov 19 17:10:06 2019 -0500

    arm64: uaccess: Ensure PAN is re-enabled after unhandled uaccess fault

It's interesting because ThunderX is somewhat unique in our test cluster as not having HW PAN.
We also only recently merged this into our 4.15 tree - Ubuntu-4.15.0-73.82 was the first tree to have it. I'll restart testing on our latest 4.15 (w/ this patch) to see if the issue persists.

The patch I highlighted in Comment #9 appears to be unrelated - 4.15.0-76 still fails even though it has the patch. A test build of 4.15.0-76 w/ the patch reverted also fails.

Is this related to the boot regression that we fixed or a different problem? I.e., can you still reproduce this with latest Bionic?

@Juery: I have no reason to believe this is related to the boot regression we fixed (bug 1857074). I haven't re-tested lately, but as of 4.15.0-76 it was still reproducible.

sorry s/Juery/Juerg/!