rtw89_pci failure on Lenovo P14s with Ubuntu 22.04

Could you try upstream kernel?
5.18-rc kernel got a lot of patches for rtw89.

upstream bug:
https://github.com/lwfinger/rtw89/issues/144

Thanks for your reply Aaron. Is there an official Ubuntu package for kernel 5.18-rc which I could try?

https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.18-rc5/

Please try AMD64 build.

Is this a regression? I.e. did this work before you upgraded to Jammy?

Status changed to 'Confirmed' because the bug affects multiple users.

I have been running into the same on my P14s, and I did upgrade to Jammy early march, but it only started to happen very very much only around the 11th of april (I have been updating mostly daily). Since that date, any intensive use of the wifi started to trigger the same behavior, whatever the version of rtw89 driver I was using (bundled in the 5.15 kernel or the one on github).

This made my life mostly impossible to work, I switched my laptop to an Intel AC9260 WiFi chip and no more problem since.

Over the course of my debugging, I tested older versions of rtw89 repo (that were known to be fine from my earlier use) as well as different rtw89 firmwares versions, to no help.

As maintainer of an rtw89 repo at GitHub, I have seen lots of complaints about this kind of hangups. Nearly all of these happen with Lenovo laptops. The remainder are a few HP machines.

For some reason, the BIOS or the PCI interface in the Lenovo laptops are incompatible with the PCI code used in RTW89. My laptop is a Toshiba from 2014, and it never has any problems, thus I cannot diagnose the difficulty. The same is true for the software developers in Taiwan, with whom I have discussed the problem.

This is the first I have heard that the problem started/got worse around April 11. That sounds like a candidate for bisection, which would have to be done with the mainline kernel, or even better, the one at wireless-next (git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next.git). Due to special patches in the GitHub version needed to build on older kernels, I upgrade the repo in batches, and lose the details of each individual patch from wireless-testing.

@lissyx is that an Intel or Ryzen P14s you have?

Does kernel parameter "iommu=off" help?

This also affects my Lenovo P14s with a Ryzen CPU using a mainline kernel.

$ uname -a
Linux LNT480s 5.17.7-051707-generic #202205121146 SMP PREEMPT Thu May 12 13:20:51 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Setting kernel parameter "iommu=off" results in the network interface being unavailable.

Does the issue only happen when the system came out from suspend?

And please try kernel parameter "rtw89_pci.disable_clkreq=1 rtw89_pci.disable_aspm_l1=1 rtw89.disable_aspm_l1ss=1".

Hi!
I have the same problem on my (Ryzen) Lenovo T14s running ubuntu 22.04

uname -a
Linux gengdahl-ThinkPad-T14s-Gen-1 5.15.0-33-generic #34-Ubuntu SMP Wed May 18 13:34:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

This is extract from log /var/log/syslog when issue happens

May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884560] rtw89_pci 0000:03:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0x36b41c90 flags=0x0000]
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884566] rtw89_pci 0000:03:00.0: FW status = 0xcd001100
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884576] rtw89_pci 0000:03:00.0: FW BADADDR = 0x0
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884577] rtw89_pci 0000:03:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0x36b42000 flags=0x0000]
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884581] rtw89_pci 0000:03:00.0: FW EPC/RA = 0x0
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884585] rtw89_pci 0000:03:00.0: FW MISC = 0x0
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884587] rtw89_pci 0000:03:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0x36b42200 flags=0x0000]
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884589] rtw89_pci 0000:03:00.0: R_AX_HALT_C2H = 0x1000
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884594] rtw89_pci 0000:03:00.0: R_AX_SER_DBG_INFO = 0x1000000
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884596] rtw89_pci 0000:03:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0x36b42300 flags=0x0000]
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884602] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fea3
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884622] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe8f
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884638] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fea9
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884653] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe91
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884669] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898ff7d
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884684] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898febb
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884699] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898feaf
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884714] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe91
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884730] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fea3
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884745] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe8b
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884760] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe81
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884775] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898ff97
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen-1 kernel: [ 5688.884790] rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe3f
May 30 21:42:33 gengdahl-ThinkPad-T14s-Gen...

Using mainline kernel 5.18.0 and the following module options, it seems more stable. At least it has not crashed anymore like that.

$ uname -a
Linux LNT480s 5.18.0-051800-generic #202205222030 SMP PREEMPT_DYNAMIC Sun May 22 20:33:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/modprobe.d/50-rtw89.conf
options rtw89_pci disable_clkreq=1 disable_aspm_l1=1 disable_aspm_l1ss=1

Without these options it crashed about once per day and indeed after suspends.

Today I have discovered the same problem:

июл 07 11:31:38 lsh-ubu kernel: rtw89_pci 0000:06:00.0: failed to init addr cam
июл 07 11:31:38 lsh-ubu kernel: rtw89_pci 0000:06:00.0: failed to get available addr cam
июл 07 11:31:38 lsh-ubu kernel: rtw89_pci 0000:06:00.0: failed to init addr cam
июл 07 11:31:38 lsh-ubu kernel: rtw89_pci 0000:06:00.0: failed to get available addr cam
... repeated many many times ...

lsh@lsh-ubu:~$ uname -a
Linux lsh-ubu 5.17.15-051715-generic #202206141358 SMP PREEMPT Tue Jun 14 18:53:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

lsh@lsh-ubu:~$ uname -a
Linux lsh-ubu 5.17.15-051715-generic #202206141358 SMP PREEMPT Tue Jun 14 18:53:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I Have this kernel for two weeks, and were no problems before today.

My laptop is:
Manufacturer: LENOVO
Version: Legion 5 15ACH6A

P.S.: What does `disable_clkreq=1 disable_aspm_l1=1 disable_aspm_l1ss=1` do?

Hi, I experience the same problem with my Lenovo ThinkPad T14 Gen 2a (AMD Ryzen 7 PRO 5850U). Also, about 1 hour after waking up from suspend mode.

$ uname -a
Linux ThinkPad-T14-Gen-2a 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

The following is a condensed excerpt from the /var/log/syslog:

rtw89_pci 0000:03:00.0: FW status = 0x77001100
rtw89_pci 0000:03:00.0: FW BADADDR = 0x0
rtw89_pci 0000:03:00.0: FW EPC/RA = 0x0
rtw89_pci 0000:03:00.0: FW MISC = 0x0
rtw89_pci 0000:03:00.0: R_AX_HALT_C2H = 0x1000
rtw89_pci 0000:03:00.0: R_AX_SER_DBG_INFO = 0x1000000
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fea5
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898ffad
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe8b
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe93
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898ff9b
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898ffad
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898ff97
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898ff9b
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe87
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe8f
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fe95
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898feb3
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898fea3
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898febf
rtw89_pci 0000:03:00.0: [ERR]fw PC = 0xb898ff95
rtw89_pci 0000:03:00.0: --->
err=0x1000
rtw89_pci 0000:03:00.0: R_AX_SER_DBG_INFO =0x01000000
rtw89_pci 0000:03:00.0: R_AX_CMAC_ERR_ISR =0x00000000
rtw89_pci 0000:03:00.0: R_AX_DMAC_ERR_ISR =0x00000100
rtw89_pci 0000:03:00.0: R_AX_WDE_ERR_FLAG_CFG =0x00000000
rtw89_pci 0000:03:00.0: R_AX_PLE_ERR_FLAG_CFG =0x00000000
rtw89_pci 0000:03:00.0: R_AX_HOST_DISPATCHER_ALWAYS_IMR=0x0c000040
rtw89_pci 0000:03:00.0: R_AX_HOST_DISPATCHER_ALWAYS_ISR=0x00000040
rtw89_pci 0000:03:00.0: R_AX_CPU_DISPATCHER_ALWAYS_IMR=0x04000062
rtw89_pci 0000:03:00.0: R_AX_CPU_DISPATCHER_ALWAYS_ISR=0x00000000
rtw89_pci 0000:03:00.0: R_AX_OTHER_DISPATCHER_ALWAYS_IMR=0x00000000
rtw89_pci 0000:03:00.0: R_AX_OTHER_DISPATCHER_ALWAYS_ISR=0x00000000
rtw89_pci 0000:03:00.0: R_AX_RPQ_RXBD_IDX =0x003d003d
rtw89_pci 0000:03:00.0: R_AX_DBG_ERR_FLAG=0x00000000
rtw89_pci 0000:03:00.0: R_AX_LBC_WATCHDOG=0x00000041
rtw89_pci 0000:03:00.0: <---
rtw89_pci 0000:03:00.0: ser event = 0x1000
rtw89_pci 0000:03:00.0: firmware failed to ack for leaving ps mode
rtw89_pci 0000:03:00.0: pci dmach busy1 0x280000
rtw89_pci 0000:03:00.0: pci dmach busy1 0x280000
rtw89_pci 0000:03:00.0: lv1 rcvy pci stop dma fail
rtw89_pci 0000:03:00.0: sec cam entry is empty
rtw89_pci 0000:03:00.0: sec cam entry is empty
rtw89_pci 0000:03:00.0: sec cam entry is empty
ieee80211 phy0: Hardware restart was requested
rtw89_pci 0000:03:00.0: MAC has already powered on
rtw89_pci 0000:03:00.0: failed to get available bssid cam
rtw89_pci 0000:03:00.0: failed to init bssid cam
rtw89_pci 0000:03:00.0: failed to get available bssid cam
rtw89_pci 0000:03:00.0: failed to init bssid cam

The last two lines are repeated until the machine is rebooted.

For now, I will try the options suggested in https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1971656/comments/13

Confirm this bug affects my HP Pavilion, AMD Ryzen 7, 437D1EA#ABH

$ lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7
02:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8852AE 802.11ax PCIe Wireless Network Adapter
03:00.0 Non-Volatile memory controller: Intel Corporation SSD 660P Series (rev 03)
04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lucienne (rev c1)
04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller
04:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
04:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1
04:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1
04:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor (rev 01)
04:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller

I am also facing the same issue with my laptop. It happens sometimes and things start behaving differently and the only way is to force shutdown to get rid of issue at the moment.

Laptop Details:
HP 255 G8 Notebook PC (AMD Ryzen 5 5500U with Radeon Graphics)
OS: Ubuntu 22.04.1 LTS

$ sudo lshw -C network
*-network
 description: Wireless interface
 product: RTL8852AE 802.11ax PCIe Wireless Network Adapter
 vendor: Realtek Semiconductor Co., Ltd.

$ uname -r
5.15.0-47-generic

Same problem here. After a amount of time (can be a day, can be 10 minutes) it starts spamming the `failed to init addr cam`.

its the same:
product: RTL8852AE 802.11ax PCIe Wireless Network Adapter

5.15.0-43-generic

if more info is needed, please do ask!

Is there a workaround available for this?
Eg, would connecting to network with cable and disable wifi remedy this?
There is no rj45 on my machine so i have not been able to test this, but if it works I'll try to get the needed accessory to connect a cable.

@Göran: I tried https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1971656/comments/13
, so far so good, but sometimes I get days without this bug triggering, so I'm not yet sure it works perfectly

I've been hitting this issue for a couple of months now and just recently found this bug. For my use-case my laptop sits idling (not suspended) overnight most days. I run into this issue a couple times a month -- the key takeaway there being that it seems to not be related to suspend (since I don't suspend my laptop).

When this happens again, is there any particular data to capture that would help debug this further? When I was on 21.10 with the rtw89 (https://github.com/lwfinger/rtw89) driver I didn't see this issue -- so it appears to be something in the 22.04 upgrade.

My current kernel version is 5.15.0-48-generic -- where I had an occurrence of this issue last week

Any signs of improvement after the 20220329.git681281e4-0ubuntu3.6 update?
After updating I had no more problems.

I just installed that firmware update, in my experience it takes a week or so to hit the issue; so we'll see if it reoccurs.

Unfortunately I just hit the same issue this morning:

$ dpkg -l | grep linux-firmwar
ii linux-firmware 20220329.git681281e4-0ubuntu3.6 all Firmware for Linux kernel drivers

So seems that the issue can still occur.

I just hit this issue the first time:
dpkg -l | grep linux-firmwar
ii linux-firmware 20220329.git681281e4-0ubuntu3.6 all Firmware for Linux kernel drivers

Same issue here. I have a Lenovo legion s7 slim. Have been having this problem for the past two weeks I think. Ubuntu 22.

Also getting this same issue here on my Lenovo T14 running 22.04, happens maybe once or twice a week

- Firmware version: 20220329.git681281e4-0ubuntu3.7
- Kernel version: 5.15.0-56-generic

Hoping to see a workaround or a fix if anyone knows

same problem here, plz help

Can you please give latest mainline kernel a try:
https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.2-rc4/

Please attach full dmesg if the issue remains.