igb driver crashes kernel when receiving jumbo frame UDP packets with all zeros

Bug #1997764 reported by Felix Ruess
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux-signed-hwe-5.15 (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

With a high MTU (9000) set on the interface, sending packets containing over ~3000 zero bytes to a network card using the igb driver results in broken packets or in the worst case for some packet sizes even leads to kernel panics or full freeze.

So far I observed this for Intel I350 and I210 network cards which use the igb driver.

This is easy to reproduce:
* set MTU to 9000
* send ping of death: ping -M do -p 00 -s 3016 <ip>
-> kernel panic or complete freeze of system

If the data is not zero, it works fine:
ping -M do -p ff -s 3016 <ip>

Depending on packet length:
* < 3000: fine
* 3016: panic/freeze
* > 3030: packet not complete

I also tested this with the following kernel versions:
* 5.4.0-131-generic -> OK
* 5.8.0-63-generic -> OK
* 5.11.0-46-generic -> OK
* 5.13.0-52-generic -> broken
* 5.15.0-41-generic -> broken
* 5.19.5-051905-generic mainline -> broken

We noticed that since we have GigE Vision cameras which send the data via UDP with packet sizes up to 9000. When we turned off the light (and hence many bytes in the payload were zero), the packets would not be complete anymore (checked with tcpdump/wireshark) or even lead to kernel panics.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-5.15.0-53-generic 5.15.0-53.59~20.04.1
ProcVersionSignature: Ubuntu 5.15.0-53.59~20.04.1-generic 5.15.64
Uname: Linux 5.15.0-53-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.25
Architecture: amd64
CasperMD5CheckResult: skip
Date: Thu Nov 24 13:16:00 2022
SourcePackage: linux-signed-hwe-5.15
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Felix Ruess (flixr) wrote :
Revision history for this message
Felix Ruess (flixr) wrote :
Revision history for this message
Felix Ruess (flixr) wrote :

here a dmesg output before the system freezes

Felix Ruess (flixr)
description: updated
description: updated
Felix Ruess (flixr)
description: updated
Felix Ruess (flixr)
description: updated
Revision history for this message
Stefan Battmer (stefanbat) wrote (last edit ):

Since we observed a similar issue I would like to add, that it actually seems like the actual content of the network packet towards the last bytes seems to cause this issue. As stated by Felix it seems to matter that the last 32 or so bytes of the packet contain zeros. One crashdump we observed actually showed one of these zeros on the instruction stack thus it looked to me as the processor tried to execute one of these zeros as a cpu instruction. This can also be seen on the dmesg output of Felix's kernel_bug.txt:

invalid opcode: 0000 [#1] SMP NOPTI

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-signed-hwe-5.15 (Ubuntu):
status: New → Confirmed
Revision history for this message
Felix Ruess (flixr) wrote :
Revision history for this message
Felix Ruess (flixr) wrote :
Revision history for this message
Felix Ruess (flixr) wrote :

So the kernel panic only seems to happen with some firmware versions for these network cards:
* I210 firmware=3.16
* I350 firmware=1.63

I could not reproduce this with a 4port I210 card with firmware=3.20.

Nevertheless this is a regression in the kernel.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.