Ubuntu
linux package

[mpt3sas, UBSAN] ]linux 6.5-rc give error messages at boot

Bug #2028830 reported by Fjodor on 2023-07-27

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	linux (Ubuntu)	Incomplete	Undecided	koba

Bug Description

Ubuntu release:

sune@jekaterina:~/src/deb$ lsb_release -rd
No LSB modules are available.
Description: Ubuntu 23.04
Release: 23.04
sune@jekaterina:~/src/deb$

Package:

linux-image-unsigned-6.5.0-060500rc1-generic_6.5.0-060500rc1.202307232333_amd64.deb from https://kernel.ubuntu.com/~kernel-ppa/mainline (rc2 and rc3 also affected)

Expectation: System boots normally

Actual results: See below

On the request of Koba Ko on the <email address hidden> mailing list, I hereby submit the following:

Do note that two "types" are mentioned, and that I am willing the assist in testing, as, obviously, I have a system with a controller covered by the mpt3sas driver.

This is output from attempted boot of 6.5-rc3 from kernel-ppa, but it is similar to rcs 1 and 2:

UBSAN: array-index-out-of-bounds in /home/kerne1/COD/linux/drivers/scsi/npt3sas/mpt3sas_scsih.c:4667:12
index 1 is out of range for type ’MPI12_EVENT_SAS_TOPO_PHY_ENTRY [1]'

UBSAN: array-index—out-of-bounds in /home/kerne1/COD/linux/drivers/scsi/mpt3sas/mpt3sas_scsih-c:4023:12
index 1 is out of range for type 'MPI2_EVENT_SAS_TOPO_PHY_ENTRY [1]'

UBSAN: array-index—out-of-bounds in /home/kernel/COD/linux/drivers/scsi/mpt3sas/mpt3sas_scsih_c:6810:36
index 1 is out of range for type 'MPI2_SAS_IO_UNIT0_PHY_DATA [1]'

UBSAN: array-index—out—of-bounds in /home/kerne1/COD/linux/drivers/scsi/mpt3sas/mpt3sas_scsih_c:6598:38
index 1 is out of range for type 'MPI2_SAS_IO_UNIT0_PHY_DATA [1]'

UBSAN: array-index—out—of-bounds in /home/kerne1/COD/linux/drivers/scsi/mpt3sas/mpt3sas_scsih_c:6602:36
index 1 is out of range for type 'MPI2_SAS_IO_UNIT0_PHY_DATA [1]'

UBSAN: array-index—out—of—bounds in /home/kernel/COD/linux-drivers/scsi/mpt3sas/mpt3sas_scsih.c:6619:7
index 1 is out of range for type 'MPI2_SAS_IO_UNIT0_PHY_DATA [1]'

UBSAN: array-index—out—of—bounds in /home/kernel/COD/linux/drivers/scsi/mpt3sas/mpt3sas_scsih.c:6666:21
index 1 is out of range for type ’MPI2_SAS_IO_UNIT0_PHY_DATA [1]'

UBSAN: array-index—out—of—bounds in /home/kernel/COD/linux/drivers/scsi/mpt3sas/mpt3sas_scsih.c:7649:32
index 1 is out of range for type ’MPI2_EVENT_SAS_TOPO_PHY_ENTRY [1]'

UBSAN: array-index-out-of—bounds in /home/kernel/COD/linux/drivers/scsi/mpt3sas/mpt3sas_scsih.c:7651:23
index 1 is out of range for type ’MPI2_EVENT_SAS_TOPO_PHY_ENTRY [1]'

UBSAN: array-index-out-of-bounds in /home/kernel/COD/linux/drivers/scsi/mpt3sas/mpt3sas_scsih.c:7655:12
index 1 is out of range for type ’MPI2_EVENT_SAS_TOPO_PHY_ENTRY [1]'

[EDIT] the next error, repeating for ~1½ hours before finishing boot, was unrelated to mpt2sas and UBSAN.

That error was "Timed out for waiting the udey queue being empty."

Bug subject altered to reflect this.

See original description

Tags:

Revision history for this message

Fjodor (sune-molgaard) wrote on 2023-07-27:

PXL_20230724_221218763.jpg Edit (3.0 MiB, image/jpeg)

koba (kobako) on 2023-07-27

Changed in linux (Ubuntu):
status:	New → In Progress
assignee:	nobody → koba (kobako)

Revision history for this message

koba (kobako) wrote on 2023-08-11:

+202308111054
@Fjodor,
Would you please have a try this test kernel(applied 6.4-rc3 config but it's latest vanilla kernel)?
https://drive.google.com/drive/folders/1cGAWoaXAEWsWKjz92xhSVelDwS9moihK?usp=sharing

Applied,
https://<email address hidden>/T/

Revision history for this message

Fjodor (sune-molgaard) wrote on 2023-08-12:

The second error message is the one of interest. Edit (1.7 MiB, image/jpeg)

@Koba,

I took the liberty of applying the patches myself unto the cod/mainline/v6.5-rc5 branch of git://git.launchpad.net/~ubuntu-kernel-test/ubuntu/+source/linux/+git/mainline-crack, but I can happily report that the UBSAN error messages are all gone!

With that said though, at my very first attempt to boot an earlier 6.5 RC, I let some time pass, and after some minutes, another error appeared, which I assumed was related to the UBSAN warnings.

With those gone, the system still doesn't boot, and that one error still appears, and is visible in the attached photo (the first line always appears, it is the second one that is of interest).

The message states: "Timed out for waiting the udev queue being empty.[sic!]"

Whether or not that is related, though, I do not know, but I do know that it seems to happen consistently and that it doesn't happen on this machine with 6.4.x or earlier - my laptops boot 6.5 RCs just fine, by the way.

Revision history for this message

Fjodor (sune-molgaard) wrote on 2023-09-11:

I am increasingly convinced that the "for waiting the udev" thing is unrelated the things that the mentioned patch set addresses, so please disregard that part, and draw the conclusion that the patches work as advertised.

Revision history for this message

Fjodor (sune-molgaard) wrote on 2023-09-19:

@Koba,

As just reported on <email address hidden>, the "for waiting the udev"[sic!] matter was completely unrelated.

With the patch set, the UBSAN messages are completely gone, and with the workaround reported to the list, the system boots normally.

Having learned a bit more about UBSAN, it would seem that the system would still have booted despite the messages, so I shall try to see if I can alter the heading of this report.

summary:	- [mpt3sas, UBSAN] ]linux 6.5-rc won't boot + [mpt3sas, UBSAN] ]linux 6.5-rc give error messages at boot
description:	updated

koba (kobako) on 2023-09-27

Changed in linux (Ubuntu):
status:	In Progress → Incomplete

Revision history for this message

Keeley Hoek (khoek) wrote on 2023-10-11:

I have just upgraded to 23.10 beta and have a large number of these spammed to the console. Does that patch fix the problem? Is there any more information you need?/why didn't that patch go into Linux 6.5?

Revision history for this message

koba (kobako) wrote on 2023-10-11:

@Keeley, please check the comments from here
https://lore.kernel.org/lkml/202310101748.5E39C3A@keescook/