ip_defrag.sh in net from ubuntu_kernel_selftests failed with 5.0 / 5.3 / 5.4 kernel

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1826848

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

I can't reproduce this in a local VM on the disco/master-next branch (tip of my branch is currently at: ebf9271500ba "UBUNTU: upstream stable to v4.19.55, v5.1.14")

Here's what I'm doing:

sudo make -C tools/testing/selftests TARGETS=net run_tests

Here's the relevant test:

selftests: net: ip_defrag.sh
========================================
ipv4 defrag
PASS
seed = 1566927625
ipv4 defrag with overlaps
PASS
seed = 1566927626
ipv6 defrag
PASS
seed = 1566927656
ipv6 defrag with overlaps
PASS
seed = 1566927656
all tests done
ok 1..17 selftests: net: ip_defrag.sh [PASS]

Hi Connor,
I found this was caused by "ubuntu_cts_kernel" that run before this kernel_selftest.

If you just run the selftest it's OK, but if you run the ubuntu_cts_kernel first, it will make this one fail.

You can try it with:
sudo apt-get install git python-minimal python-yaml gdb -y

git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest-client-tests
git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest
rm -fr autotest/client/tests
ln -sf ~/autotest-client-tests autotest/client/tests

AUTOTEST_PATH=/home/ubuntu/autotest sudo -E autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_cts_kernel/control

And run the net selftest.

I've confirmed that if "ubuntu_cts_kernel" runs before this test, it will cause the test to fail. It will continue to fail between reboots of the virtual machine, too, so I suspect there's something persistent being added during the "ubuntu_cts_kernel" test that just isn't being cleaned up when it's done.

I disabled all of the tests that "ubuntu_cts_kernel" runs and started activating them one-by-one until it causes the ipv6 defrag test case to fail. It started failing when I activated the first one named: 'lp1262692'

Test result on 5.3 Bionic-hwe:
 # selftests: net: ip_defrag.sh
 # ipv4 defrag
 # PASS
 # seed = 1569129774
 # ipv4 defrag with overlaps
 # PASS
 # seed = 1569129774
 # ipv6 defrag
 # PASS
 # seed = 1569129780
 # ipv6 defrag with overlaps
 # seed = 1569129780
 # ./ip_defrag: sendto overlap: 2120: Operation not permitted

Spotted on AWS 5.3 ARM64 (5.3.0-1009.10) with instance a1.large:

 # selftests: net: ip_defrag.sh
 not ok 18 selftests: net: ip_defrag.sh # exit=255

I am working on this test case. ip_defrag.sh fails with non-overlapping as well, but only sometimes. I managed to find out the cause, and sent a fix upstream, but this is under discussion now on how to fix it.

The EPERM failure with overlap happens more often, you just need to run the test multiple times to see it fail once in a while. I just remove the other tests from ip_defrag.sh and run ip_defrag.sh in a loop until it fails. That still needs some investigation, but I believe this is no regression on the kernels at all.

Didn't see this issue with B-5.3 (5.3.0-41.33~18.04.1), node rizzo

Didn't see this issue with AWS-B-5.0 (5.0.0-1026.29-aws)
and 5.3.0-41.33-generic on node rizzo.

Found this failing with 5.3.0-1012.13~18.04.1-aws
test failed with:
# selftests: net: ip_defrag.sh
 not ok 18 selftests: net: ip_defrag.sh # exit=255

Didn't see this issue with Eoan 5.3 on node rizzo

Found this with GCP 5.3.0-1014.15
 # selftests: net: ip_defrag.sh
 # ipv4 defrag
 # PASS
 # seed = 1583552965
 # ipv4 defrag with overlaps
 # PASS
 # seed = 1583552965
 # ipv6 defrag
 # PASS
 # seed = 1583552971
 # ipv6 defrag with overlaps
 # PASS
 # seed = 1583552971
 # ipv6 nf_conntrack defrag
 # PASS
 # seed = 1583552976
 # ipv6 nf_conntrack defrag with overlaps
 # seed = 1583552976
 # ./ip_defrag: sendto overlap: 2800: Operation not permitted
 not ok 18 selftests: net: ip_defrag.sh # exit=1

To give more information on this issue:

ip_defrag will send a randomly-sized packet, and the result depends on that size, so we will see intermittent failures because of this reason.

But not this reason alone. The test that fails require a conntrack rule that will cause all fragments to be queued, and, then, sent all at once through the loopback interface. If the loop that sends all those packets is preempted, the problem may not show up, so there is one more cause for the failures to be intermittent. If it's not preempted, it might end up pushing more than 1000 packets, causing some of them to be dropped because of the backlog limit, which will cause the test to fail.

Raising the backlog limit or adjusting the maximum packet size on the test would fix it. The concern about raising the backlog limit is that it might affect other tests, even if it's rollbacked after the test, because some tests might run in parallel.

We could adjust the maximum size, but this would be not be upstreamable, as Eric Dumazet argues this is a real bug that should be fixed. In fact, I managed to reproduce it using veth.

On the other hand, I couldn't think of an easy way to fix this. Changing those drivers to skip the backlog doesn't seem a good idea to me.

Cascardo.

Found this with 5.3 Eoan on node rizzo

This issue can be found on Focal 5.4.0-34.38 (node rizzo) as well.
 # selftests: net: ip_defrag.sh
 # ipv4 defrag
 # PASS
 # seed = 1591378573
 # ipv4 defrag with overlaps
 # PASS
 # seed = 1591378573
 # ipv6 defrag
 # PASS
 # seed = 1591378577
 # ipv6 defrag with overlaps
 # PASS
 # seed = 1591378577
 # ipv6 nf_conntrack defrag
 # PASS
 # seed = 1591378582
 # ipv6 nf_conntrack defrag with overlaps
 # seed = 1591378582
 # ./ip_defrag: sendto overlap: 2984: Operation not permitted
 not ok 18 selftests: net: ip_defrag.sh # exit=1

However this test has passed with 5.4.0-33.37:
 # selftests: net: ip_defrag.sh
 # ipv4 defrag
 # PASS
 # seed = 1590175171
 # ipv4 defrag with overlaps
 # PASS
 # seed = 1590175171
 # ipv6 defrag
 # PASS
 # seed = 1590175175
 # ipv6 defrag with overlaps
 # PASS
 # seed = 1590175176
 # ipv6 nf_conntrack defrag
 # PASS
 # seed = 1590175180
 # ipv6 nf_conntrack defrag with overlaps
 # PASS
 # seed = 1590175180
 # all tests done
 ok 18 selftests: net: ip_defrag.sh
http://10.246.72.4/test-results/5.4.0-33.37-generic/rizzo__5.4.0-33.37__2020-05-22_18-35-00/ubuntu_kernel_selftests/results/ubuntu_kernel_selftests.net/debug/ubuntu_kernel_selftests.net.DEBUG.html

Need to verify this manually.

This is still failing with 5.4.0-33-generic on rizzo. Not sure why it passed last cycle.

BTW it has passed with the same node "rizzo" on 5.4.0-34.38 lowlatency kernel.

The test failure is intermittent, I have sent a suggested fix upstream, will send it as SAUCE to the mailing list soon.

https://lore<email address hidden>/

https://git.kernel.org/torvalds/c/065fcfd49763ec71ae345bb5c5a74f961031e70e

Applied to Torvalds' tree.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-eoan' to 'verification-done-eoan'. If the problem still exists, change the tag 'verification-needed-eoan' to 'verification-failed-eoan'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

All autopkgtests for the newly accepted linux-gcp-5.4 (5.4.0-1021.21~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

zfs-linux/0.7.5-1ubuntu16.9 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-gcp-5.4

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted linux-azure-5.3 (5.3.0-1034.35~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

zfs-linux/0.7.5-1ubuntu16.9 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-azure-5.3

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted linux-aws-5.3 (5.3.0-1032.34~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

zfs-linux/0.7.5-1ubuntu16.9 (arm64, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-aws-5.3

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted linux-aws-5.4 (5.4.0-1020.20~18.04.2) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

zfs-linux/0.7.5-1ubuntu16.9 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-aws-5.4

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted linux-gcp-5.3 (5.3.0-1032.34~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

zfs-linux/0.7.5-1ubuntu16.9 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-gcp-5.3

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

The test has passed with 5.4.0-42.46 on all different arches.

The test has passed with 5.3.0-64.58 on all different arches.

This bug was fixed in the package linux - 5.4.0-42.46

---------------
linux (5.4.0-42.46) focal; urgency=medium

  * focal/linux: 5.4.0-42.46 -proposed tracker (LP: #1887069)

  * linux 4.15.0-109-generic network DoS regression vs -108 (LP: #1886668)
    - SAUCE: Revert "netprio_cgroup: Fix unlimited memory leak of v2 cgroups"

linux (5.4.0-41.45) focal; urgency=medium

  * focal/linux: 5.4.0-41.45 -proposed tracker (LP: #1885855)

  * Packaging resync (LP: #1786013)
    - update dkms package versions

  * CVE-2019-19642
    - kernel/relay.c: handle alloc_percpu returning NULL in relay_open

  * CVE-2019-16089
    - SAUCE: nbd_genl_status: null check for nla_nest_start

  * CVE-2020-11935
    - aufs: do not call i_readcount_inc()

  * ip_defrag.sh in net from ubuntu_kernel_selftests failed with 5.0 / 5.3 / 5.4
    kernel (LP: #1826848)
    - selftests: net: ip_defrag: ignore EPERM

  * Update lockdown patches (LP: #1884159)
    - SAUCE: acpi: disallow loading configfs acpi tables when locked down

  * seccomp_bpf fails on powerpc (LP: #1885757)
    - SAUCE: selftests/seccomp: fix ptrace tests on powerpc

  * Introduce the new NVIDIA 418-server and 440-server series, and update the
    current NVIDIA drivers (LP: #1881137)
    - [packaging] add signed modules for the 418-server and the 440-server
      flavours

 -- Khalid Elmously <email address hidden> Thu, 09 Jul 2020 19:50:26 -0400

This bug was fixed in the package linux - 5.3.0-64.58

---------------
linux (5.3.0-64.58) eoan; urgency=medium

  * eoan/linux: 5.3.0-64.58 -proposed tracker (LP: #1887088)

  * linux 4.15.0-109-generic network DoS regression vs -108 (LP: #1886668)
    - SAUCE: Revert "netprio_cgroup: Fix unlimited memory leak of v2 cgroups"

linux (5.3.0-63.57) eoan; urgency=medium

  * eoan/linux: 5.3.0-63.57 -proposed tracker (LP: #1885495)

  * seccomp_bpf fails on powerpc (LP: #1885757)
    - SAUCE: selftests/seccomp: fix ptrace tests on powerpc

  * The thread level parallelism would be a bottleneck when searching for the
    shared pmd by using hugetlbfs (LP: #1882039)
    - hugetlbfs: take read_lock on i_mmap for PMD sharing

  * Eoan update: upstream stable patchset 2020-06-30 (LP: #1885775)
    - ipv6: fix IPV6_ADDRFORM operation logic
    - net_failover: fixed rollback in net_failover_open()
    - bridge: Avoid infinite loop when suppressing NS messages with invalid
      options
    - vxlan: Avoid infinite loop when suppressing NS messages with invalid options
    - tun: correct header offsets in napi frags mode
    - Input: mms114 - fix handling of mms345l
    - ARM: 8977/1: ptrace: Fix mask for thumb breakpoint hook
    - sched/fair: Don't NUMA balance for kthreads
    - Input: synaptics - add a second working PNP_ID for Lenovo T470s
    - drivers/net/ibmvnic: Update VNIC protocol version reporting
    - powerpc/xive: Clear the page tables for the ESB IO mapping
    - ath9k_htc: Silence undersized packet warnings
    - RDMA/uverbs: Make the event_queue fds return POLLERR when disassociated
    - x86/cpu/amd: Make erratum #1054 a legacy erratum
    - perf probe: Accept the instance number of kretprobe event
    - mm: add kvfree_sensitive() for freeing sensitive data objects
    - aio: fix async fsync creds
    - x86_64: Fix jiffies ODR violation
    - x86/PCI: Mark Intel C620 MROMs as having non-compliant BARs
    - x86/speculation: Prevent rogue cross-process SSBD shutdown
    - x86/reboot/quirks: Add MacBook6,1 reboot quirk
    - efi/efivars: Add missing kobject_put() in sysfs entry creation error path
    - ALSA: es1688: Add the missed snd_card_free()
    - ALSA: hda/realtek - add a pintbl quirk for several Lenovo machines
    - ALSA: usb-audio: Fix inconsistent card PM state after resume
    - ALSA: usb-audio: Add vendor, product and profile name for HP Thunderbolt
      Dock
    - ACPI: sysfs: Fix reference count leak in acpi_sysfs_add_hotplug_profile()
    - ACPI: CPPC: Fix reference count leak in acpi_cppc_processor_probe()
    - ACPI: GED: add support for _Exx / _Lxx handler methods
    - ACPI: PM: Avoid using power resources if there are none for D0
    - nilfs2: fix null pointer dereference at nilfs_segctor_do_construct()
    - spi: dw: Fix controller unregister order
    - spi: bcm2835aux: Fix controller unregister order
    - spi: bcm-qspi: when tx/rx buffer is NULL set to 0
    - PM: runtime: clk: Fix clk_pm_runtime_get() error path
    - crypto: cavium/nitrox - Fix 'nitrox_get_first_device()' when ndevlist is
      fully iterated
    - ALSA: pcm: disallow linking stream to itself
    - x86/{mce,mm}: Unmap the entire page if the who...

This bug was fixed in the package linux - 5.4.0-42.46

---------------
linux (5.4.0-42.46) focal; urgency=medium

  * focal/linux: 5.4.0-42.46 -proposed tracker (LP: #1887069)

  * linux 4.15.0-109-generic network DoS regression vs -108 (LP: #1886668)
    - SAUCE: Revert "netprio_cgroup: Fix unlimited memory leak of v2 cgroups"

linux (5.4.0-41.45) focal; urgency=medium

  * focal/linux: 5.4.0-41.45 -proposed tracker (LP: #1885855)

  * Packaging resync (LP: #1786013)
    - update dkms package versions

  * CVE-2019-19642
    - kernel/relay.c: handle alloc_percpu returning NULL in relay_open

  * CVE-2019-16089
    - SAUCE: nbd_genl_status: null check for nla_nest_start

  * CVE-2020-11935
    - aufs: do not call i_readcount_inc()

  * ip_defrag.sh in net from ubuntu_kernel_selftests failed with 5.0 / 5.3 / 5.4
    kernel (LP: #1826848)
    - selftests: net: ip_defrag: ignore EPERM

  * Update lockdown patches (LP: #1884159)
    - SAUCE: acpi: disallow loading configfs acpi tables when locked down

  * seccomp_bpf fails on powerpc (LP: #1885757)
    - SAUCE: selftests/seccomp: fix ptrace tests on powerpc

  * Introduce the new NVIDIA 418-server and 440-server series, and update the
    current NVIDIA drivers (LP: #1881137)
    - [packaging] add signed modules for the 418-server and the 440-server
      flavours

 -- Khalid Elmously <email address hidden> Thu, 09 Jul 2020 19:50:26 -0400