Ubuntu
linux package

vrf_route_leaking.sh in net from ubuntu_kernel_selftests linux ADT test failure with linux/5.11.0-18.19 (Ping received ICMP Packet too big)

Bug #1928890 reported by Kelsey Steele
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
New
Undecided
Unassigned
linux (Ubuntu)
Incomplete
Undecided
Unassigned
Hirsute
Won't Fix
Undecided
Unassigned
Impish
Won't Fix
Undecided
Unassigned
Jammy
Confirmed
Undecided
Unassigned
Kinetic
Confirmed
Undecided
Unassigned
linux-oem-5.10 (Ubuntu)
Won't Fix
Undecided
Unassigned
Hirsute
Won't Fix
Undecided
Unassigned
Impish
Invalid
Undecided
Unassigned
Jammy
New
Undecided
Unassigned
Kinetic
New
Undecided
Unassigned

Bug Description

This is a scripted bug report about ADT failures while running linux tests for linux/5.11.0-18.19 on hirsute. Whether this is caused by the dep8 tests of the tested source or the kernel has yet to be determined.

Not a regression. Found to occur previously on hirsute/linux 5.11.0-14.15

Testing failed on:
    amd64: https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/amd64/l/linux/20210515_005957_75e5a@/log.gz
    arm64: https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/arm64/l/linux/20210513_203508_96fd3@/log.gz
    ppc64el: https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/ppc64el/l/linux/20210513_163708_c0203@/log.gz
    s390x: https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/s390x/l/linux/20210513_144454_54b04@/log.gz

00:09:30 DEBUG| [stdout] # selftests: net: vrf_route_leaking.sh
00:09:30 DEBUG| [stdout] #
00:09:30 DEBUG| [stdout] # ###########################################################################
00:09:30 DEBUG| [stdout] # IPv4 (sym route): VRF ICMP ttl error route lookup ping
00:09:30 DEBUG| [stdout] # ###########################################################################
00:09:30 DEBUG| [stdout] #
00:09:32 DEBUG| [stdout] # TEST: Basic IPv4 connectivity [ OK ]
00:09:32 DEBUG| [stdout] # TEST: Ping received ICMP ttl exceeded [ OK ]
00:09:32 DEBUG| [stdout] #
00:09:32 DEBUG| [stdout] # ###########################################################################
00:09:32 DEBUG| [stdout] # IPv4 (sym route): VRF ICMP error route lookup traceroute
00:09:32 DEBUG| [stdout] # ###########################################################################
00:09:32 DEBUG| [stdout] #
00:09:32 DEBUG| [stdout] # SKIP: Could not run IPV4 test without traceroute
00:09:32 DEBUG| [stdout] #
00:09:32 DEBUG| [stdout] # ###########################################################################
00:09:32 DEBUG| [stdout] # IPv4 (sym route): VRF ICMP fragmentation error route lookup ping
00:09:32 DEBUG| [stdout] # ###########################################################################
00:09:32 DEBUG| [stdout] #
00:09:34 DEBUG| [stdout] # TEST: Basic IPv4 connectivity [ OK ]
00:09:34 DEBUG| [stdout] # TEST: Ping received ICMP Frag needed [ OK ]
00:09:34 DEBUG| [stdout] #
00:09:34 DEBUG| [stdout] # ###########################################################################
00:09:34 DEBUG| [stdout] # IPv4 (asym route): VRF ICMP ttl error route lookup ping
00:09:34 DEBUG| [stdout] # ###########################################################################
00:09:34 DEBUG| [stdout] #
00:09:36 DEBUG| [stdout] # TEST: Basic IPv4 connectivity [ OK ]
00:09:36 DEBUG| [stdout] # TEST: Ping received ICMP ttl exceeded [ OK ]
00:09:36 DEBUG| [stdout] #
00:09:36 DEBUG| [stdout] # ###########################################################################
00:09:36 DEBUG| [stdout] # IPv4 (asym route): VRF ICMP error route lookup traceroute
00:09:36 DEBUG| [stdout] # ###########################################################################
00:09:36 DEBUG| [stdout] #
00:09:36 DEBUG| [stdout] # SKIP: Could not run IPV4 test without traceroute
00:09:36 DEBUG| [stdout] #
00:09:36 DEBUG| [stdout] # ###########################################################################
00:09:36 DEBUG| [stdout] # IPv6 (sym route): VRF ICMP ttl error route lookup ping
00:09:36 DEBUG| [stdout] # ###########################################################################
00:09:36 DEBUG| [stdout] #
00:09:40 DEBUG| [stdout] # TEST: Basic IPv6 connectivity [FAIL]
00:09:40 DEBUG| [stdout] # TEST: Ping received ICMP Hop limit [ OK ]
00:09:40 DEBUG| [stdout] #
00:09:40 DEBUG| [stdout] # ###########################################################################
00:09:40 DEBUG| [stdout] # IPv6 (sym route): VRF ICMP error route lookup traceroute
00:09:40 DEBUG| [stdout] # ###########################################################################
00:09:40 DEBUG| [stdout] #
00:09:42 DEBUG| [stdout] # TEST: Basic IPv6 connectivity [ OK ]
00:09:42 DEBUG| [stdout] # TEST: Traceroute6 reports a hop on r1 [ OK ]
00:09:42 DEBUG| [stdout] #
00:09:42 DEBUG| [stdout] # ###########################################################################
00:09:42 DEBUG| [stdout] # IPv6 (sym route): VRF ICMP fragmentation error route lookup ping
00:09:42 DEBUG| [stdout] # ###########################################################################
00:09:42 DEBUG| [stdout] #
00:09:45 DEBUG| [stdout] # TEST: Basic IPv6 connectivity [FAIL]
00:09:47 DEBUG| [stdout] # TEST: Ping received ICMP Packet too big [FAIL]
00:09:47 DEBUG| [stdout] #
00:09:47 DEBUG| [stdout] # ###########################################################################
00:09:47 DEBUG| [stdout] # IPv6 (asym route): VRF ICMP ttl error route lookup ping
00:09:47 DEBUG| [stdout] # ###########################################################################
00:09:47 DEBUG| [stdout] #
00:09:49 DEBUG| [stdout] # TEST: Basic IPv6 connectivity [ OK ]
00:09:49 DEBUG| [stdout] # TEST: Ping received ICMP Hop limit [ OK ]
00:09:49 DEBUG| [stdout] #
00:09:49 DEBUG| [stdout] # ###########################################################################
00:09:49 DEBUG| [stdout] # IPv6 (asym route): VRF ICMP error route lookup traceroute
00:09:49 DEBUG| [stdout] # ###########################################################################
00:09:49 DEBUG| [stdout] #
00:09:52 DEBUG| [stdout] # TEST: Basic IPv6 connectivity [ OK ]
00:09:52 DEBUG| [stdout] # TEST: Traceroute6 reports a hop on r1 [ OK ]
00:09:52 DEBUG| [stdout] #
00:09:52 DEBUG| [stdout] # Tests passed: 13
00:09:52 DEBUG| [stdout] # Tests failed: 3
00:09:52 DEBUG| [stdout] not ok 45 selftests: net: vrf_route_leaking.sh # exit=1

See original description
Kelsey Steele (kelsey-steele)
tags: added: kernel-adt-failure
summary: - linux ADT test failure with linux/5.11.0-18.19
+ vrf_route_leaking linux ADT test failure with linux/5.11.0-18.19 (Ping
+ received ICMP Packet too big)
description: updated
tags: added: 5.11 hirsute
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1928890

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Po-Hsu Lin (cypressyew)
summary: - vrf_route_leaking linux ADT test failure with linux/5.11.0-18.19 (Ping
- received ICMP Packet too big)
+ vrf_route_leaking in net from ubuntu_kernel_selftests linux ADT test
+ failure with linux/5.11.0-18.19 (Ping received ICMP Packet too big)
tags: added: sru-20210510 ubuntu-kernel-selftests
Kelsey Steele (kelsey-steele)
tags: added: sru-20210531
Revision history for this message
Po-Hsu Lin (cypressyew) wrote : Re: vrf_route_leaking in net from ubuntu_kernel_selftests linux ADT test failure with linux/5.11.0-18.19 (Ping received ICMP Packet too big)

This issue can be found on OEM-5.10 as well.

 # IPv6 (sym route): VRF ICMP fragmentation error route lookup ping
 # ###########################################################################
 #
 # TEST: Basic IPv6 connectivity [ OK ]
 # TEST: Ping received ICMP Packet too big [FAIL]

5.10.0-1031.32-oem

Kelsey Steele (kelsey-steele)
tags: added: impish
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

5.10.0-1035.36 OEM node vought

tags: added: sru-20210621
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Found with hirsute/linux 5.11.0-24.25.

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Found on 5.11.0-1013.13-aws

tags: added: aws
Revision history for this message
Stefan Bader (smb) wrote :

Found with focal:linux-hwe-5.11 (5.11.0-24.25~20.04.1) on SRU-20210621.

Kelsey Steele (kelsey-steele)
tags: added: sru-20210719
Kleber Sacilotto de Souza (kleber-souza)
summary: - vrf_route_leaking in net from ubuntu_kernel_selftests linux ADT test
+ vrf_route_leaking.sh in net from ubuntu_kernel_selftests linux ADT test
failure with linux/5.11.0-18.19 (Ping received ICMP Packet too big)
Tim Gardner (timg-tpi)
tags: added: sru-20211018
Revision history for this message
Krzysztof Kozlowski (krzk) wrote :

Found also on d2021.10.26/jammy/linux-realtime/5.15.0-1003.3

Revision history for this message
Krzysztof Kozlowski (krzk) wrote :

Found also on 2021.11.29/impish/linux-oracle/5.13.0-1011.13

tags: added: 5.13 hinted oracle sru-20211129
Revision history for this message
Andrea Righi (arighi) wrote :

This issue is still present also on jammy 5.15. Some details that I found:

  - the purpose of this test is to verify that an ICMP6 packet is not forwarded by a virtual router if it needs to do fragmentation
  - the test creates 3 network namespaces representing host1, host2 and router
  - host1 and host2 have mtu 1500, router has mtu 1400
  - the test consists of a ping6 from host1 to host2 (via router) using a payload of 1450, so host1 and host2 don't need to fragment the packet, but router does
  - therefore the ping is expected to fail (and this is happennig), but we should also receive a "Packet too big" message from the router and we don't

So at the end the behavior is correct, but we're missing the "Packet too big" notification from ping. I'm not sure if it's a user-space issue (in iputils-ping) or a kernel issue.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux (Ubuntu Hirsute):
status: New → Confirmed
Changed in linux-oem-5.10 (Ubuntu Hirsute):
status: New → Confirmed
Changed in linux-oem-5.10 (Ubuntu):
status: New → Confirmed
Revision history for this message
Zachary Tahenakos (ztahenakos) wrote :

Running this test on jammy 5.15, it passes if the vrf module is missing, the ping will come back with the expected response of Packet too big. With the vrf module loaded, we won't get the message back. Looking at the strace of the command working vs failing, this is a difference spotted:

vrf loaded:
recvmsg(3, {msg_namelen=128}, 0) = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(3, {msg_namelen=128}, 0) = -1 EINTR (Interrupted system call)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---

vrf missing:
recvmsg(3, {msg_namelen=128}, 0) = -1 EMSGSIZE (Message too long)

Revision history for this message
Zachary Tahenakos (ztahenakos) wrote :

Doing some more research and testing, looking at the upstream commit that added the test (1a01727676a8), this is in the bottom of the message:

    [ The IPv6 MTU test still fails with the symmetric routing setup. It
      appears to be caused by source address selection picking ::1. Fixing
      this is beyond the scope of this series. ]

Running the test in the failing state, we are indeed getting a Packet too big message but we never seem to get it.

Now, within the create_vrf function, if we change the address added from ::1 to ::3
(ip -netns "${ns}" -6 addr add ::3 dev "${vrf}" nodad), it will start to work. I wonder if its getting confused since in the failing case, the src is ::1 and the dest is 2001:db8:16:1:1 so maybe the packet is going around in a loop or something.

Revision history for this message
Brian Murray (brian-murray) wrote :

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

Changed in linux-oem-5.10 (Ubuntu Hirsute):
status: Confirmed → Won't Fix
Kleber Sacilotto de Souza (kleber-souza)
Changed in linux-oem-5.10 (Ubuntu):
status: Confirmed → Won't Fix
Changed in linux-oem-5.10 (Ubuntu Impish):
status: New → Invalid
Changed in linux (Ubuntu Impish):
status: New → Confirmed
Changed in linux (Ubuntu Hirsute):
status: Confirmed → Won't Fix
Revision history for this message
Brian Murray (brian-murray) wrote :

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

Changed in linux (Ubuntu Impish):
status: Confirmed → Won't Fix
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

This bug is still failing on recent kernels.

Jammy 5.15:
https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/l/linux/20230327_235241_c3754@/log.gz

Kinetic 5.19:
https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/amd64/l/linux/20230328_203949_367cf@/log.gz

Changed in linux (Ubuntu Jammy):
status: New → Confirmed
Changed in linux (Ubuntu Kinetic):
status: New → Confirmed
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Found on J-oem-6.5.0-1002

tags: added: 6.5 jammy oem
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.