| 2022-08-22 02:00:24 |
Krister Johansen |
bug |
|
|
added bug |
| 2022-08-22 02:30:05 |
Ubuntu Kernel Bot |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2022-08-22 17:39:33 |
Krister Johansen |
tags |
|
apport-collected focal uec-images |
|
| 2022-08-22 17:39:34 |
Krister Johansen |
description |
I have systems that are regularly hitting a WARN in trace_event_dyn_put_ref.
The exact message is:
WARNING: CPU: 1 PID: 30309 at kernel/trace/trace_dynevent.c:46
+trace_event_dyn_put_ref+0x15/0x20
With the following stacktrace:
perf_trace_init+0x8f/0xd0
perf_tp_event_init+0x1f/0x40
perf_try_init_event+0x4a/0x130
perf_event_alloc+0x497/0xf40
__do_sys_perf_event_open+0x1d4/0xf70
__x64_sys_perf_event_open+0x20/0x30
do_syscall_64+0x5c/0xc0
entry_SYSCALL_64_after_hwframe+0x44/0xae
I've debugged this and worked with upstream to get a fix into Linux. It was recently merged in 6.0-rc2. See here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.0-rc2&id=7249921d94ff64f67b733eca0b68853a62032b3d
The problem started appearing as soon as our systems picked up the linux-aws-5.15 branch for Focal. (That was 5.15.0-1015-aws, if memory serves). Could you please cherry pick this fix and pull it back to the the linux and linux-aws kernels for Focal? There's test here: https://lore.kernel.org/all/cover.1660347763.git.kjlx@templeofstupid.com/ that reproduces the problem very reliably for me. With the patch applied, I no longer get the WARNs. |
I have systems that are regularly hitting a WARN in trace_event_dyn_put_ref.
The exact message is:
WARNING: CPU: 1 PID: 30309 at kernel/trace/trace_dynevent.c:46
+trace_event_dyn_put_ref+0x15/0x20
With the following stacktrace:
perf_trace_init+0x8f/0xd0
perf_tp_event_init+0x1f/0x40
perf_try_init_event+0x4a/0x130
perf_event_alloc+0x497/0xf40
__do_sys_perf_event_open+0x1d4/0xf70
__x64_sys_perf_event_open+0x20/0x30
do_syscall_64+0x5c/0xc0
entry_SYSCALL_64_after_hwframe+0x44/0xae
I've debugged this and worked with upstream to get a fix into Linux. It was recently merged in 6.0-rc2. See here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.0-rc2&id=7249921d94ff64f67b733eca0b68853a62032b3d
The problem started appearing as soon as our systems picked up the linux-aws-5.15 branch for Focal. (That was 5.15.0-1015-aws, if memory serves). Could you please cherry pick this fix and pull it back to the the linux and linux-aws kernels for Focal? There's test here: https://lore.kernel.org/all/cover.1660347763.git.kjlx@templeofstupid.com/ that reproduces the problem very reliably for me. With the patch applied, I no longer get the WARNs.
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Aug 22 17:32 seq
crw-rw---- 1 root audio 116, 33 Aug 22 17:32 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu27.24
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:
Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: Amazon EC2 c5d.12xlarge
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-1015-aws root=PARTUUID=4986e35b-1bd5-45d3-b528-fa2edb861a38 ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1
ProcVersionSignature: Ubuntu 5.15.0-1015.19~20.04.1-aws 5.15.39
RelatedPackageVersions:
linux-restricted-modules-5.15.0-1015-aws N/A
linux-backports-modules-5.15.0-1015-aws N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: focal uec-images
Uname: Linux 5.15.0-1015-aws x86_64
UnreportableReason: This report is about a package that is not installed.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: False
dmi.bios.date: 10/16/2017
dmi.bios.release: 1.0
dmi.bios.vendor: Amazon EC2
dmi.bios.version: 1.0
dmi.board.asset.tag: i-03f5d8581c7ad94aa
dmi.board.vendor: Amazon EC2
dmi.chassis.asset.tag: Amazon EC2
dmi.chassis.type: 1
dmi.chassis.vendor: Amazon EC2
dmi.modalias: dmi:bvnAmazonEC2:bvr1.0:bd10/16/2017:br1.0:svnAmazonEC2:pnc5d.12xlarge:pvr:rvnAmazonEC2:rn:rvr:cvnAmazonEC2:ct1:cvr:sku:
dmi.product.name: c5d.12xlarge
dmi.sys.vendor: Amazon EC2 |
|
| 2022-08-22 17:39:36 |
Krister Johansen |
attachment added |
|
CurrentDmesg.txt https://bugs.launchpad.net/bugs/1987232/+attachment/5610807/+files/CurrentDmesg.txt |
|
| 2022-08-22 17:39:39 |
Krister Johansen |
attachment added |
|
Lspci.txt https://bugs.launchpad.net/bugs/1987232/+attachment/5610808/+files/Lspci.txt |
|
| 2022-08-22 17:39:41 |
Krister Johansen |
attachment added |
|
Lspci-vt.txt https://bugs.launchpad.net/bugs/1987232/+attachment/5610809/+files/Lspci-vt.txt |
|
| 2022-08-22 17:39:43 |
Krister Johansen |
attachment added |
|
ProcCpuinfoMinimal.txt https://bugs.launchpad.net/bugs/1987232/+attachment/5610810/+files/ProcCpuinfoMinimal.txt |
|
| 2022-08-22 17:39:44 |
Krister Johansen |
attachment added |
|
ProcInterrupts.txt https://bugs.launchpad.net/bugs/1987232/+attachment/5610811/+files/ProcInterrupts.txt |
|
| 2022-08-22 17:39:46 |
Krister Johansen |
attachment added |
|
ProcModules.txt https://bugs.launchpad.net/bugs/1987232/+attachment/5610812/+files/ProcModules.txt |
|
| 2022-08-22 17:39:48 |
Krister Johansen |
attachment added |
|
UdevDb.txt https://bugs.launchpad.net/bugs/1987232/+attachment/5610813/+files/UdevDb.txt |
|
| 2022-08-22 17:39:50 |
Krister Johansen |
attachment added |
|
WifiSyslog.txt https://bugs.launchpad.net/bugs/1987232/+attachment/5610814/+files/WifiSyslog.txt |
|
| 2022-08-22 17:39:53 |
Krister Johansen |
attachment added |
|
acpidump.txt https://bugs.launchpad.net/bugs/1987232/+attachment/5610815/+files/acpidump.txt |
|
| 2022-08-22 17:44:36 |
Krister Johansen |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2022-08-29 07:19:13 |
Stefan Bader |
nominated for series |
|
Ubuntu Jammy |
|
| 2022-08-29 07:19:13 |
Stefan Bader |
bug task added |
|
linux (Ubuntu Jammy) |
|
| 2022-08-29 07:19:58 |
Stefan Bader |
nominated for series |
|
Ubuntu Kinetic |
|
| 2022-08-29 07:19:58 |
Stefan Bader |
bug task added |
|
linux (Ubuntu Kinetic) |
|
| 2022-08-29 07:20:17 |
Stefan Bader |
linux (Ubuntu Jammy): importance |
Undecided |
Medium |
|
| 2022-08-29 07:20:17 |
Stefan Bader |
linux (Ubuntu Jammy): status |
New |
In Progress |
|
| 2022-09-14 13:13:42 |
Stefan Bader |
description |
I have systems that are regularly hitting a WARN in trace_event_dyn_put_ref.
The exact message is:
WARNING: CPU: 1 PID: 30309 at kernel/trace/trace_dynevent.c:46
+trace_event_dyn_put_ref+0x15/0x20
With the following stacktrace:
perf_trace_init+0x8f/0xd0
perf_tp_event_init+0x1f/0x40
perf_try_init_event+0x4a/0x130
perf_event_alloc+0x497/0xf40
__do_sys_perf_event_open+0x1d4/0xf70
__x64_sys_perf_event_open+0x20/0x30
do_syscall_64+0x5c/0xc0
entry_SYSCALL_64_after_hwframe+0x44/0xae
I've debugged this and worked with upstream to get a fix into Linux. It was recently merged in 6.0-rc2. See here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.0-rc2&id=7249921d94ff64f67b733eca0b68853a62032b3d
The problem started appearing as soon as our systems picked up the linux-aws-5.15 branch for Focal. (That was 5.15.0-1015-aws, if memory serves). Could you please cherry pick this fix and pull it back to the the linux and linux-aws kernels for Focal? There's test here: https://lore.kernel.org/all/cover.1660347763.git.kjlx@templeofstupid.com/ that reproduces the problem very reliably for me. With the patch applied, I no longer get the WARNs.
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Aug 22 17:32 seq
crw-rw---- 1 root audio 116, 33 Aug 22 17:32 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu27.24
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:
Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: Amazon EC2 c5d.12xlarge
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-1015-aws root=PARTUUID=4986e35b-1bd5-45d3-b528-fa2edb861a38 ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1
ProcVersionSignature: Ubuntu 5.15.0-1015.19~20.04.1-aws 5.15.39
RelatedPackageVersions:
linux-restricted-modules-5.15.0-1015-aws N/A
linux-backports-modules-5.15.0-1015-aws N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: focal uec-images
Uname: Linux 5.15.0-1015-aws x86_64
UnreportableReason: This report is about a package that is not installed.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: False
dmi.bios.date: 10/16/2017
dmi.bios.release: 1.0
dmi.bios.vendor: Amazon EC2
dmi.bios.version: 1.0
dmi.board.asset.tag: i-03f5d8581c7ad94aa
dmi.board.vendor: Amazon EC2
dmi.chassis.asset.tag: Amazon EC2
dmi.chassis.type: 1
dmi.chassis.vendor: Amazon EC2
dmi.modalias: dmi:bvnAmazonEC2:bvr1.0:bd10/16/2017:br1.0:svnAmazonEC2:pnc5d.12xlarge:pvr:rvnAmazonEC2:rn:rvr:cvnAmazonEC2:ct1:cvr:sku:
dmi.product.name: c5d.12xlarge
dmi.sys.vendor: Amazon EC2 |
[SRU Justification]
Impact: Some imbalanced ref-counting produces kernel warnings regularly. Since it is a warning level, this triggers system monitoring on servers which in turn causes unnecessary work for inspecting the logs.
Fix: There is a fix upstream and also backported to the upstream stable branch. However we are still a bit behind catching up with the latest versions. Since this is having quite an impact and the fix is rather straight forward, we pull this in from upstream stable ahead of time.
Test case: tbd
Regression potential: Regressions would manifest as different errors related to ref-counting.
---
I have systems that are regularly hitting a WARN in trace_event_dyn_put_ref.
The exact message is:
WARNING: CPU: 1 PID: 30309 at kernel/trace/trace_dynevent.c:46
+trace_event_dyn_put_ref+0x15/0x20
With the following stacktrace:
perf_trace_init+0x8f/0xd0
perf_tp_event_init+0x1f/0x40
perf_try_init_event+0x4a/0x130
perf_event_alloc+0x497/0xf40
__do_sys_perf_event_open+0x1d4/0xf70
__x64_sys_perf_event_open+0x20/0x30
do_syscall_64+0x5c/0xc0
entry_SYSCALL_64_after_hwframe+0x44/0xae
I've debugged this and worked with upstream to get a fix into Linux. It was recently merged in 6.0-rc2. See here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.0-rc2&id=7249921d94ff64f67b733eca0b68853a62032b3d
The problem started appearing as soon as our systems picked up the linux-aws-5.15 branch for Focal. (That was 5.15.0-1015-aws, if memory serves). Could you please cherry pick this fix and pull it back to the the linux and linux-aws kernels for Focal? There's test here: https://lore.kernel.org/all/cover.1660347763.git.kjlx@templeofstupid.com/ that reproduces the problem very reliably for me. With the patch applied, I no longer get the WARNs.
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Aug 22 17:32 seq
crw-rw---- 1 root audio 116, 33 Aug 22 17:32 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu27.24
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:
Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: Amazon EC2 c5d.12xlarge
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-1015-aws root=PARTUUID=4986e35b-1bd5-45d3-b528-fa2edb861a38 ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1
ProcVersionSignature: Ubuntu 5.15.0-1015.19~20.04.1-aws 5.15.39
RelatedPackageVersions:
linux-restricted-modules-5.15.0-1015-aws N/A
linux-backports-modules-5.15.0-1015-aws N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: focal uec-images
Uname: Linux 5.15.0-1015-aws x86_64
UnreportableReason: This report is about a package that is not installed.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: False
dmi.bios.date: 10/16/2017
dmi.bios.release: 1.0
dmi.bios.vendor: Amazon EC2
dmi.bios.version: 1.0
dmi.board.asset.tag: i-03f5d8581c7ad94aa
dmi.board.vendor: Amazon EC2
dmi.chassis.asset.tag: Amazon EC2
dmi.chassis.type: 1
dmi.chassis.vendor: Amazon EC2
dmi.modalias: dmi:bvnAmazonEC2:bvr1.0:bd10/16/2017:br1.0:svnAmazonEC2:pnc5d.12xlarge:pvr:rvnAmazonEC2:rn:rvr:cvnAmazonEC2:ct1:cvr:sku:
dmi.product.name: c5d.12xlarge
dmi.sys.vendor: Amazon EC2 |
|
| 2022-09-14 13:17:25 |
Stefan Bader |
linux (Ubuntu Jammy): status |
In Progress |
Fix Committed |
|
| 2022-09-22 18:48:55 |
Krister Johansen |
tags |
apport-collected focal uec-images |
apport-collected focal uec-images verification-done-jammy |
|
| 2022-10-10 08:44:51 |
Launchpad Janitor |
linux (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2022-10-10 08:44:51 |
Launchpad Janitor |
cve linked |
|
2022-3176 |
|
| 2022-12-07 15:49:18 |
Ubuntu Kernel Bot |
tags |
apport-collected focal uec-images verification-done-jammy |
apport-collected focal kernel-spammed-jammy-linux-bluefield uec-images verification-needed-jammy |
|
| 2022-12-12 12:47:15 |
Ubuntu Kernel Bot |
tags |
apport-collected focal kernel-spammed-jammy-linux-bluefield uec-images verification-needed-jammy |
apport-collected focal kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia uec-images verification-needed-jammy |
|
| 2023-08-10 17:59:12 |
Utkarsh Gupta |
linux (Ubuntu Kinetic): status |
Confirmed |
Won't Fix |
|
