net:udpgro_frglist.sh in ubuntu_kernel_selftests cause soft lockup on K-5.19 P9

Bug #2015843 reported by Po-Hsu Lin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
New
Undecided
Unassigned
linux (Ubuntu)
Invalid
Undecided
Unassigned
Kinetic
New
Undecided
Unassigned

Bug Description

Issue found while debugging bug 2015745

This net:udpgro_frglist.sh test will cause soft lockup on K-5.19 Power9

The test itself can pass without any issue:
$ sudo ./udpgro_frglist.sh
ipv6
tcp - over veth touching data
-l 4 -6 -D 2001:db8::1 -t rx -4 -t
-4 -t
tcp tx: 525 MB/s 8908 calls/s 8908 msg/s
tcp rx: 525 MB/s 13895 calls/s
tcp tx: 543 MB/s 9223 calls/s 9223 msg/s
tcp rx: 544 MB/s 19994 calls/s
tcp tx: 544 MB/s 9230 calls/s 9230 msg/s
tcp rx: 544 MB/s 26196 calls/s
tcp tx: 566 MB/s 9601 calls/s 9601 msg/s
udp gso - over veth touching data
-l 4 -6 -D 2001:db8::1 -u -S 0 rx -4 -v
-4 -v
udp rx: 0 MB/s 7 calls/s
udp tx: 0 MB/s 1 calls/s 1 msg/s
udp gso and gro - over veth touching data
-l 4 -6 -D 2001:db8::1 -S 0 rx -4 -G
-4 -G
udp rx: 0 MB/s 8 calls/s
udp tx: 0 MB/s 1 calls/s 1 msg/s
$ echo $?
0

But it will cause soft lockup and thus tainting the kernel.

dmesg output:
[ 444.086079] watchdog: BUG: soft lockup - CPU#49 stuck for 26s! [udpgso_bench_tx:7963]
[ 444.086148] Modules linked in: cls_bpf sch_ingress algif_hash af_alg veth cfg80211 joydev input_leds mac_hid binfmt_misc ofpart cmdlinepart powernv_flash opal_prd ipmi_powernv ipmi_devintf ipmi_msghandler ibmpowernv mtd vmx_crypto at24 uio_pdrv_genirq uio dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua pstore_blk ramoops reed_solomon pstore_zone ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid ses enclosure scsi_transport_sas ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm crct10dif_vpmsum crc32c_vpmsum i40e aacraid xhci_pci drm_panel_orientation_quirks xhci_pci_renesas
[ 444.086281] CPU: 49 PID: 7963 Comm: udpgso_bench_tx Not tainted 5.19.0-38-generic #39-Ubuntu
[ 444.086287] NIP: c0000000012e5d74 LR: c00000000135c628 CTR: 000000001355bc49
[ 444.086291] REGS: c000000fffdfb6d0 TRAP: 0900 Not tainted (5.19.0-38-generic)
[ 444.086294] MSR: 9000000002009033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 4822288e XER: 20040153
[ 444.086313] CFAR: 0000000000000000 IRQMASK: 0
               GPR00: c00000000135c628 c000000fffdfb970 c000000002afde00 c00000061443010e
               GPR04: 0000000000000000 0000000000000000 0000000000000000 0016ce21ccbbf672
               GPR08: c0000008dee73c8a 0039766f02a34334 0000000000000000 0000000000000000
               GPR12: 0000000000000000 c000000ffffaa680 0000000000000000 0000000000000000
               GPR16: 0000000000000001 0000000000000000 ffffffffffffffff 0000000000000000
               GPR20: c000000fffdfbdd8 c000000002b33a80 0000000000000000 0000000000000000
               GPR24: 0000000020000000 000000000000000a c0000000059d4c00 0000000000000000
               GPR28: 000000000000000e 0000000000000014 c00000061443010e c0000000059db400
[ 444.086376] NIP [c0000000012e5d74] ip_send_check+0x94/0xf0
[ 444.086385] LR [c00000000135c628] inet_gso_segment+0x278/0x5b0
[ 444.086390] Call Trace:
[ 444.086392] [c000000fffdfb970] [c00000000135c570] inet_gso_segment+0x1c0/0x5b0 (unreliable)
[ 444.086399] [c000000fffdfb9e0] [c00000000121dde8] skb_mac_gso_segment+0xe8/0x180
[ 444.086406] [c000000fffdfba20] [c0000000011bd650] __skb_gso_segment+0xc0/0x1d0
[ 444.086413] [c000000fffdfba60] [c0000000013454bc] udp_queue_rcv_skb.part.0+0x9c/0x280
[ 444.086419] [c000000fffdfbaa0] [c000000001345c68] udp_unicast_rcv_skb+0x38/0x100
[ 444.086425] [c000000fffdfbac0] [c000000001347838] __udp4_lib_rcv+0x388/0x7b0
[ 444.086431] [c000000fffdfbb90] [c0000000012e10a0] ip_protocol_deliver_rcu+0x70/0x410
[ 444.086438] [c000000fffdfbbe0] [c0000000012e14d4] ip_local_deliver_finish+0x94/0x110
[ 444.086445] [c000000fffdfbc20] [c0000000012e0df0] ip_rcv_finish+0x110/0x160
[ 444.086451] [c000000fffdfbc60] [c0000000011cb914] __netif_receive_skb_one_core+0x74/0xd0
[ 444.086458] [c000000fffdfbcb0] [c0000000011cbd1c] process_backlog+0x11c/0x260
[ 444.086464] [c000000fffdfbd00] [c0000000011ccda8] __napi_poll+0x68/0x3a0
[ 444.086471] [c000000fffdfbd90] [c0000000011cd5ac] net_rx_action+0x23c/0x490
[ 444.086477] [c000000fffdfbe80] [c0000000014e922c] __do_softirq+0x16c/0x47c
[ 444.086484] [c000000fffdfbf90] [c0000000000180ec] do_softirq_own_stack+0x4c/0xb0
[ 444.086490] [c00000002499b5f0] [c0000000011dcb54] __neigh_event_send+0x344/0x530
[ 444.086496] [c00000002499b630] [c000000000196624] do_softirq+0xd4/0xe0
[ 444.086503] [c00000002499b660] [c0000000001966f8] __local_bh_enable_ip+0xc8/0x100
[ 444.086509] [c00000002499b680] [c0000000013deddc] ip6_finish_output2+0x24c/0x780
[ 444.086516] [c00000002499b730] [c0000000013e2db8] ip6_finish_output+0x1f8/0x4e0
[ 444.086522] [c00000002499b790] [c000000001469c4c] ip6_local_out+0x6c/0xb0
[ 444.086528] [c00000002499b7d0] [c0000000013e3af8] ip6_send_skb+0x48/0x110
[ 444.086534] [c00000002499b810] [c0000000014218dc] udp_v6_send_skb+0x1dc/0x620
[ 444.086541] [c00000002499b870] [c000000001422a44] udpv6_sendmsg+0xbc4/0x1030
[ 444.086546] [c00000002499ba80] [c0000000013d8df8] inet6_sendmsg+0x68/0xe0
[ 444.086553] [c00000002499bac0] [c00000000117ed10] sock_sendmsg+0x90/0xe0
[ 444.086559] [c00000002499baf0] [c00000000117f21c] ____sys_sendmsg+0x2ec/0x390
[ 444.086566] [c00000002499bb90] [c0000000011840e0] ___sys_sendmsg+0xd0/0x150
[ 444.086571] [c00000002499bd00] [c0000000011841f4] sys_sendmsg+0x94/0x130
[ 444.086577] [c00000002499bdb0] [c000000000031b3c] system_call_exception+0x11c/0x2f0
[ 444.086583] [c00000002499be10] [c00000000000c0e8] system_call_vectored_common+0xe8/0x278
[ 444.086590] --- interrupt: 3000 at 0x717d0396cf98
[ 444.086595] NIP: 0000717d0396cf98 LR: 0000000000000000 CTR: 0000000000000000
[ 444.086598] REGS: c00000002499be80 TRAP: 3000 Not tainted (5.19.0-38-generic)
[ 444.086601] MSR: 900000000000f033 <SF,HV,EE,PR,FP,ME,IR,DR,RI,LE> CR: 44002478 XER: 00000000
[ 444.086619] IRQMASK: 0
               GPR00: 0000000000000155 00007fffd548b480 0000717d03a76e00 0000000000000003
               GPR04: 00007fffd548b590 0000000000000000 0000000000000000 0000006700000011
               GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
               GPR12: 0000000000000000 0000717d03bba7a0 0000000000000000 000000000000f180
               GPR16: 0000000000000003 000000000000000a 00000c39af050040 000001876f2e3cbf
               GPR20: 00007fffd548b648 00000c39aea10060 0000000000000000 0000000000000000
               GPR24: 0000000000000000 000001876f2e38d7 0000000000000003 00000c39aea10010
               GPR28: 00000c39aea10100 00000c39aea10100 00000c39aea10040 00007fffd548b4c0
[ 444.086680] NIP [0000717d0396cf98] 0x717d0396cf98
[ 444.086683] LR [0000000000000000] 0x0
[ 444.086686] --- interrupt: 3000
[ 444.086688] Instruction dump:
[ 444.086691] 394affff 7d292a14 2c2a0001 85480004 7d295214 41820030 78caf082 7d4903a6
[ 444.086703] 80880004 80a80008 80c8000c 85480010 <7d292214> 7ce72a14 7d293214 7ce75214

Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu):
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.