blk_update_request: I/O error, dev sdb, sector 976772992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1856434

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

/dev/sdb1 is my external usb HDD. In disks, I set it to standby after 10 minutes. What do those errors mean?

Good morning,

i have about 20 new hardware servers from Dell with HBA330 Controller and NL-SAS Disk for a storage server setup.
OS is Ubuntu 18.04.4 LTS with kernel Linux server01 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3

I am with this error message not able to format any of the 12 x 8TB Drives.

Error-Logs from syslog:

Mar 18 09:30:30 server01 kernel: [669635.089535] mpt3sas_cm0: log_info(0x3112043b): originator(PL), code(0x12), sub_code(0x043b)
Mar 18 09:30:30 server01 kernel: [669635.089563] sd 2:0:3:0: [sde] tag#2508 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_SENSE
Mar 18 09:30:30 server01 kernel: [669635.089570] sd 2:0:3:0: [sde] tag#2508 Sense Key : Illegal Request [current]
Mar 18 09:30:30 server01 kernel: [669635.089576] sd 2:0:3:0: [sde] tag#2508 Add. Sense: Logical block reference tag check failed
Mar 18 09:30:30 server01 kernel: [669635.089582] sd 2:0:3:0: [sde] tag#2508 CDB: Read(32)
Mar 18 09:30:30 server01 kernel: [669635.089587] sd 2:0:3:0: [sde] tag#2508 CDB[00]: 7f 00 00 00 00 00 00 18 00 09 20 00 00 00 00 03
Mar 18 09:30:30 server01 kernel: [669635.089592] sd 2:0:3:0: [sde] tag#2508 CDB[10]: a3 81 2a 00 a3 81 2a 00 00 00 00 00 00 00 00 08
Mar 18 09:30:30 server01 kernel: [669635.089604] blk_update_request: protection error, dev sde, sector 15628052992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar 18 09:30:30 server01 kernel: [669635.091790] Buffer I/O error on dev sde, logical block 1953506624, async page read
Mar 18 09:30:30 server01 kernel: [669635.097186] mpt3sas_cm0: log_info(0x3112043b): originator(PL), code(0x12), sub_code(0x043b)
Mar 18 09:30:30 server01 kernel: [669635.097208] sd 2:0:4:0: [sdf] tag#604 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_SENSE
Mar 18 09:30:30 server01 kernel: [669635.097211] sd 2:0:4:0: [sdf] tag#604 Sense Key : Illegal Request [current]
Mar 18 09:30:30 server01 kernel: [669635.097214] sd 2:0:4:0: [sdf] tag#604 Add. Sense: Logical block reference tag check failed
Mar 18 09:30:30 server01 kernel: [669635.097216] sd 2:0:4:0: [sdf] tag#604 CDB: Read(32)
Mar 18 09:30:30 server01 kernel: [669635.097218] sd 2:0:4:0: [sdf] tag#604 CDB[00]: 7f 00 00 00 00 00 00 18 00 09 20 00 00 00 00 03
Mar 18 09:30:30 server01 kernel: [669635.097220] sd 2:0:4:0: [sdf] tag#604 CDB[10]: a3 81 2a 00 a3 81 2a 00 00 00 00 00 00 00 00 08
Mar 18 09:30:30 server01 kernel: [669635.097223] blk_update_request: protection error, dev sdf, sector 15628052992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0

Every task to format the drive or wipe the drives ends up with tons of messages like this in syslog.

Thx for any hint or solution to that problem.

Best,

Timo

Hello Timo,

Did you find a solution to this problem?

We are seeing the exact same scsi errors on our dell servers with H330 controller as well, on 18.04, 14T drives.

We have installed RHEL7 on other exact same spec servers, but the issue is not present so it might be particular to Ubuntu and/or later kernels.

Regards,
Rafael

Hi Timo and Rafael,

I'm not a sysadmin, but just a hobbyist who ordered a used SAS Hitachi hus723030als641. I'm experiencing the same thing (`blk_update_request: protection error` and `blk_update_request: i/o error` on Archlinux and Ubuntu). Also, in Windows 10, I am able to view the drive, but have trouble formatting it.

Does this have anything to do with the read protection on the SAS drive? I was searching around and found the `sg_format` command. Running `sg_format /dev/sdX` reports that the drive has read protection on. I'm not entirely sure what that means, but I'm currently trying to format the drives with `sg_format /dev/sdX --format` (which, if I understood the manpages correctly, I believe will unset the protection bit). Will let you know in a few hours when it completes. In the meantime, do you know anything about whether this SCSI protection level business is related to the error at hand? I don't understand it very well and I can't seem to find much useful information on it on the web, so any input would be appreciated.

Thanks,
Jonathan

Update: performing the format worked for me (removing the read protection removed the blk_update_request errors on Ubuntu). I'm not particularly concerned about encryption, so I'll probably leave it off, but maybe in your case you need some additional packages on Ubuntu to support the self-encryption.

Jonathan

"I'm not particularly concerned about encryption, so I'll probably leave it off, but maybe in your case you need some additional packages on Ubuntu to support the self-encryption."

My drive is not encrypted.

Is any kind of power saving feature enabled?

Similar issue is found in VM ubuntu-20.04-live-server in ESXi 7.0GA
   blk_update_request: I/O error, dev sda, sector 3270728 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0

Attach the dmesg info.
sda means the system disk.

Steps:
1) install guestOS with ubuntu-20.04-live-server-amd64.iso in ESXi 7.0GA
2) Follow the wizard to finish installation
3) I take no any operations on the VM. Sometimes the console output message:
   blk_update_request: I/O error, dev sda, sector 3270728 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0

   output from command "dmesg"
   [ 313.011817] blk_update_request: I/O error, dev sda, sector 3270672 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0
   [ 357.810551] sd 3:0:0:0: [sda] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
   [ 357.810557] sd 3:0:0:0: [sda] tag#19 Sense Key : Vendor Specific(9) [current]
   [ 357.810560] sd 3:0:0:0: [sda] tag#19 <<vendor>>ASC=0x80 <<vendor>>ASCQ=0x85
   [ 357.810562] sd 3:0:0:0: [sda] tag#19 CDB: Write same(16) 93 08 00 00 00 00 00 31 e8 48 00

Same error message when sda is NVMe disk, PVSCSI, SAS disk. So this issue should be not related to virtual disk type.

Please test latest mainline kernel:
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.7-rc6/

vmware deve's comment for the issue "blk_update_request: I/O error, dev sda, sector 3270728 op 0x9 VM ubuntu-20.04-live-server in ESXi 7.0GA" :

I noticed that when it's happening, abnormal kernel log:
[ 0.000000] ? lockdown_lsm_init+0x21/0x25
[ 0.000000] Kernel is locked down from EFI Secure Boot mode; see man kernel_lockdown.7
.....
[ 66.261508] Lockdown: fwupd: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
[ 233.169823] ata3: SATA link down (SStatus 0 SControl 300)
[ 238.577700] ata3: SATA link down (SStatus 0 SControl 300)
[ 243.953448] ata3: SATA link down (SStatus 0 SControl 300)
[ 243.953455] ata3.00: disabled
[ 243.953481] sr 2:0:0:0: rejecting I/O to offline device
[ 243.958034] ata3.00: detaching (SCSI 2:0:0:0)
[ 304.517077] sd 3:0:0:0: [sda] tag#55 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 304.517081] sd 3:0:0:0: [sda] tag#55 Sense Key : Vendor Specific(9) [current]
[ 304.517086] sd 3:0:0:0: [sda] tag#55 <<vendor>>ASC=0x80 <<vendor>>ASCQ=0x85
[ 304.517090] sd 3:0:0:0: [sda] tag#55 CDB: Write same(16) 93 08 00 00 00 00 00 31 e7 e8 00 00 00 18 00 00
[ 304.517100] blk_update_request: I/O error, dev sda, sector 3270632 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0
[ 314.319273] sd 3:0:0:0: [sda] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 314.319275] sd 3:0:0:0: [sda] tag#11 Sense Key : Vendor Specific(9) [current]
[ 314.319278] sd 3:0:0:0: [sda] tag#11 <<vendor>>ASC=0x80 <<vendor>>ASCQ=0x85
[ 314.319295] sd 3:0:0:0: [sda] tag#11 CDB: Write same(16) 93 08 00 00 00 00 00 31 e8 10 00 00 00 18 00 00
[ 314.319299] blk_update_request: I/O error, dev sda, sector 3270672 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0

And no error log in hostside.

Noticed you enabled secure boot in uefi and ubuntu are configured with lockdown

vmware@ubuntu:~$ cat /boot/config-5.4.0-26-generic | grep LOCK_DOWN
CONFIG_LOCK_DOWN_IN_SECURE_BOOT=y
CONFIG_LOCK_DOWN_KERNEL_FORCE_NONE=y

So the error should caused by uefi lockdown feature.

Please refer to https://mjg59.dreamwidth.org/50577.html for more details.

To workaroud this,
echo 1 > /proc/sys/kernel/sysrq
echo x > /proc/sysrq-trigger

or you can use sudo mokutil --disable-validation to disable it.

I tried "echo to /proc" way, the error msg disappered. You might try the second way. Anyway, it's an upstream bug.

Had the exact same problem on Dell hardware with HBA330.
Possible to solve with a "sg_format --format" on every single drive or by upgrading fw on the HBA controller.

https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=nknvc&oscode=wst14&productcode=poweredge-r740xd

-Fixes a firmware initialization problem that causes T10 Protection Information (PI) Logical Block and Guard Tag errors under Linux operating systems. This can result in an inability to read from T10 PI capable drives.

More info about T10 Protection Information (T10-PI)
https://www.seagate.com/files/staticfiles/docs/pdf/whitepaper/safeguarding-data-from-corruption-technology-paper-tp621us.pdf

Check status with
sg_readcap -l /dev/sda
"Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]"
or
smartctl --all /dev/sda
"Formatted with type 2 protection"

Thanks Joel, the firmware you posted worked for us.

I will note we have the HBA330 mini, which has it's own distinct upgrade file as opposed to the regular HBA330.

HBA330 mini:
https://www.dell.com/support/home/en-au/drivers/driversdetails?driverid=124x2

i'm still a bit puzzled why a firmware update fixed this, since our rhel7 servers did not experience the issue, only Ubuntu (hence i was expecting a kernel driver issue/regression).

> vmware-gos-Yuhua (yhzou) wrote on 2020-06-01:#22

   It's because VMs on vmfs5 datastore doesn't support trim/unmap command.
it won't hit this issue with vmfs6