kernel warning: skb_warn_bad_offload

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1715609

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

I don't have apport installed on my servers and was hoping the linked bug (1705447) is providing enough context. If not, please let me know and I'll try to provide what I can.

I built a test kernel with the commit to fix bug 1705447 reverted. The reverted commit is:

commit b2504a5dbef3305ef41988ad270b0e8ec289331c
Author: Eric Dumazet <email address hidden>
Date: Tue Jan 31 10:20:32 2017 -0800
    net: reduce skb_warn_bad_offload() noise

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1715609/

Can you test this kernel and see if it resolves the regression?

Joseph, thanks for your help! Will need a bit of time to set up a testing environment for this.

If it helps anyone else with testing/debugging, it looks like the warning only happens on LXD container hosts that run containers which in turn connect to an IPSec VPN (using StrongSwan). We get them on LXD hosts with either "4.4.0-93-generic #116~14.04.1-Ubuntu" or "4.10.0-33-generic #37~16.04.1-Ubuntu".

I'm also experiencing this issue with 4.4.0-93-generic in combination with docker,kvm and bonding on host. Not sure if docker or kvm is causing the warnings.

Can confirm the same issue with "4.10.0-33-generic #37~16.04.1-Ubuntu". I also have LXD containers, strongswan ipsec VPN server running on that host. Also, there are 6 e1000e/igb network cards installed and being used, if that matters.

Can confirm the same issue with the 4.4.0-93-generic with LXC container running OpenVPN client.

Can those affected by this bug test the kernel posted in comment #3?

Also reproduced on:
Linux node-dock-b 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Running docker in bridge configuration

Joseph, I've been running your test kernel since yesterday. Looking good, no skb_warn_bad_offload warnings so far. With the original kernel I was getting hundreds of these warnings every day.

Can you also build an update for the 4.4.0-93 kernel ?

@jo Vanvoorden,

A Xenial 4.4 based test kernel has been built with a revert of commit b2504a5dbe.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1715609/xenial

It might also be good to test the latest mainline kernel. This will tell us if this commit needs to be reverted, or if there are other commits in mainline that might be needed in combination with b2504a5dbef.

The latest mainline kernel can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14-rc1/

Booted the test kernel and so far no more warnings

Can you also give the mainline kernel a test? A link to it is posted in comment #13.

This bug also affects the low-latency kernel Linux iglulik 4.10.0-37-lowlatency #41-Ubuntu SMP PREEMPT Fri Oct 6 21:30:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux and you can make that four people that are affected.

Seeing this in

Linux andor 4.10.0-37-generic #41-Ubuntu SMP Fri Oct 6 20:20:37 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

as well

I'm still seeing this in the latest 4.10, 4.11, 4.13 and 4.15 (from mainline).

Dec 10 23:09:05 infra1 kernel: [ 571.226805] RIP: 0010:skb_warn_bad_offload+0xb7/0xf0
Dec 10 23:09:05 infra1 kernel: [ 571.226810] FS: 0000000000000000(0000) GS:ffff97c25eb80000(0000) knlGS:0000000000000000
Dec 10 23:09:05 infra1 kernel: [ 571.226818] ipt_do_table+0x2eb/0x5f0 [ip_tables]
Dec 10 23:09:05 infra1 kernel: [ 571.226829] nf_hook_slow+0x42/0xb0
Dec 10 23:09:05 infra1 kernel: [ 571.226849] ? br_dev_queue_push_xmit+0x140/0x140 [bridge]
Dec 10 23:09:05 infra1 kernel: [ 571.226866] br_handle_frame_finish+0x2a4/0x540 [bridge]
Dec 10 23:09:05 infra1 kernel: [ 571.226881] ? iptable_nat_ipv4_in+0x15/0x20 [iptable_nat]
Dec 10 23:09:05 infra1 kernel: [ 571.226895] __netif_receive_skb+0x18/0x60
Dec 10 23:09:05 infra1 kernel: [ 571.226906] common_interrupt+0x93/0x93
Dec 10 23:09:05 infra1 kernel: [ 571.226912] R10: ffffbc4f00cf3e40 R11: 000000000000012b R12: ffff97c25eba4e30
Dec 10 23:09:05 infra1 kernel: [ 571.226921] secondary_startup_64+0xa5/0xa5

Dec 10 23:09:15 infra1 kernel: [ 582.507590] CPU: 0 PID: 20250 Comm: apache2 Tainted: G W 4.14.5-041405-generic #201712101332
Dec 10 23:09:15 infra1 kernel: [ 582.507595] RBP: ffff97c25ea036e0 R08: 0000000000000001 R09: 0000000000000c10
Dec 10 23:09:15 infra1 kernel: [ 582.507599] Call Trace:
Dec 10 23:09:15 infra1 kernel: [ 582.507610] iptable_mangle_hook+0x37/0x110 [iptable_mangle]
Dec 10 23:09:15 infra1 kernel: [ 582.507622] br_forward_finish+0x9d/0xb0 [bridge]
Dec 10 23:09:15 infra1 kernel: [ 582.507631] br_nf_forward_ip+0x30c/0x430 [br_netfilter]
Dec 10 23:09:15 infra1 kernel: [ 582.507641] br_forward+0x97/0xa0 [bridge]
Dec 10 23:09:15 infra1 kernel: [ 582.507650] ? br_pass_frame_up+0x150/0x150 [bridge]
Dec 10 23:09:15 infra1 kernel: [ 582.507659] br_handle_frame+0x1ec/0x300 [bridge] ...

Hi Eric,

A kernel bug report was opened against Ubuntu [0].  It was found that
reverting the following commit resolved this bug:

commit b2504a5dbef3305ef41988ad270b0e8ec289331c
Author: Eric Dumazet <email address hidden>
Date:   Tue Jan 31 10:20:32 2017 -0800

    net: reduce skb_warn_bad_offload() noise

The regression was introduced as of v4.11-rc1 and still exists in
current mainline.

I was hoping to get your feedback, since you are the patch author.  Do
you think gathering any additional data will help diagnose this issue,
or would it be best to submit a revert request?

This commit did in fact resolve another bug[1], but in the process
introduced this regression.

Thanks,

Joe

[0] http://pad.lv/1715609
[1] http://pad.lv/1705447

I requested feedback from upstream regarding this bug. I will update the bug as soon as there is anything new.

Comment #20 is the message sent upstream.

From: Joseph Salisbury <email address hidden>
Date: Mon, 11 Dec 2017 15:35:34 -0500

> A kernel bug report was opened against Ubuntu [0].  It was found that
> reverting the following commit resolved this bug:
>
> commit b2504a5dbef3305ef41988ad270b0e8ec289331c
> Author: Eric Dumazet <email address hidden>
> Date:   Tue Jan 31 10:20:32 2017 -0800
>
>     net: reduce skb_warn_bad_offload() noise
>
>
> The regression was introduced as of v4.11-rc1 and still exists in
> current mainline.
>
> I was hoping to get your feedback, since you are the patch author.  Do
> you think gathering any additional data will help diagnose this issue,
> or would it be best to submit a revert request?
>
> This commit did in fact resolve another bug[1], but in the process
> introduced this regression.

It helps if you can consolidate the information obtained in your bug
tracking here in the email so that people on this list can get an idea
of what the problem scope might be without having to go to your
special bug tracking site.

This is really not about us being snobs about this mailing list, it's
about you wanting to get a result. And you'll get a better result
faster if you post the details here on the lsit because most
developers are not going to go to your bug tracking site to read the
bug comments.

Also, this isn't a functional regression, it is just that we are
generating warnings that we didn't before. It doesn't mean that
Eric's patch is wrong, it could just be that his new check is
triggering for a bug that has always been there.

Scanning the bug myself it seems that the critical required component
is IPSEC, and IPSEC has it's own way of doing segmentation offload.

Thanks.

From: Willem de Bruijn <email address hidden>
Date: Mon, 11 Dec 2017 16:56:56 -0500

> On Mon, Dec 11, 2017 at 4:44 PM, Greg Kroah-Hartman
> <email address hidden> wrote:
>> On Mon, Dec 11, 2017 at 04:25:26PM -0500, Willem de Bruijn wrote:
>>> Note that UFO was removed in 4.14 and that skb_warn_bad_offload
>>> can happen for various types of packets, so there may be multiple
>>> independent bug reports. I'm investigating two other non-UFO reports
>>> just now.
>>
>> Meta-comment, now that UFO is gone from mainline, I'm wondering if I
>> should just delete it from 4.4 and 4.9 as well. Any objections for
>> that? I'd like to make it easy to maintain these kernels for a while,
>> and having them diverge like this, with all of the issues around UFO,
>> seems like it will just make life harder for myself if I leave it in.
>>
>> Any opinions?
>
> Some of that removal had to be reverted with commit 0c19f846d582
> ("net: accept UFO datagrams from tuntap and packet") for VM live
> migration between kernels.
>
> Any backports probably should squash that in at the least. Just today
> another thread discussed that that patch may not address all open
> issues still, so it may be premature to backport at this point.
> http://lkml.kernel.org/r/<d71df64e-e65f-4db4-6f2e-c002c15fcbe4@01019freenet.de>

I would probably discourage backporting the UFO removal, at least for
now.

Can those affected by this bug give the following kernel a try:
http://kernel.ubuntu.com/~jsalisbury/lp1715609/

Just a note that I'm seeing far less skb_warn_bad_offload warnings in Xenial with the last 3 or 4 official 4.10 kernels (4.10.0-42, 4.10.0-40, 4.10.0-38, etc). Sometimes I don't see any warnings for a week or so, sometimes a few pop out. So it looks much better now, I'm just not sure it is fixed completely.

I can confirm the bug is still present in 4.10.0-42. I can experience it with LinPhone working on a VLAN defined on an igb managed card.

Hi,

I got this on 4.4.0-116-generic :

[266447.178906] ------------[ cut here ]------------
[266447.178919] WARNING: CPU: 0 PID: 3343 at /build/linux-fQ94TU/linux-4.4.0/net/core/dev.c:2448 skb_warn_bad_offload+0xd1/0x120()
[266447.178924] qr-8a4bd2f3-04: caps=(0x00000184075b59e9, 0x00000184075b59e9) len=2058 data_len=582 gso_size=1406 gso_type=1 ip_summed=3
[266447.178926] Modules linked in: kpatch_livepatch_Ubuntu_4_4_0_116_140_generic_36(OE) xt_REDIRECT nf_nat_redirect ip6table_mangle xt_nat xt_mark xt_connmark ip6table_raw iptable_raw ip_gre ip_tunnel gre xt_recent ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 xt_hl ip6t_rt nf_conntrack_ipv6 ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_addrtype xt_conntrack dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag veth netlink_diag nbd ip6table_filter ip6_tables xt_CHECKSUM iptable_mangle xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_filter ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack binfmt_misc ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel bridge stp kvm llc
[266447.178988] irqbypass sb_edac edac_core ipmi_si hpilo ipmi_msghandler serio_raw 8250_fintek lpc_ich shpchp acpi_power_meter ioatdma mac_hid dca ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse tg3 ptp pps_core hpsa scsi_transport_sas wmi fjes
[266447.179045] CPU: 0 PID: 3343 Comm: haproxy Tainted: G W OE K 4.4.0-116-generic #140-Ubuntu
[266447.179048] Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 01/22/2018
[266447.179050] 0000000000000286 6f5737dc927d16dd ffff880a3646b818 ffffffff813ffc13
[266447.179055] ffff880a3646b860 ffffffff81d715e8 ffff880a3646b850 ffffffff810830f2
[266447.179059] ffff880064507ce8 ffff880b4ff12000 0000000000000001 ffff8808723268fc
[266447.179064] Call Trace:
[266447.179072] [<ffffffff813ffc13>] dump_stack+0x63/0x90
[266447.179079] [<ffffffff810830f2>] warn_slowpath_common+0x82/0xc0
[266447.179084] [<ffffffff8108318c>] warn_slowpath_fmt+0x5c/0x80
[266447.179090] [<ffffffff81405d92>] ? ___ratelimit+0xa2/0xe0
[266447.179094] [<ffffffff8173ff81>] skb_warn_bad_offload+0xd1/0x120
[266447.179100] [<ffffffff81740155>] skb_checksum_help+0x185/0x1a0
[266447.179106] [<ffffffffc0524072>] checksum_tg+0x22/0x29 [xt_CHECKSUM]
[266447.179112] [<ffffffffc03a4e21>] ipt_do_table+0x301/0x720 [ip_tables]
[266447.179117] [<ffffffffc03a4e69>] ? ipt_do_table+0x349/0x720 [ip_tables]
[266447.179122] [<ffffffff810bae37>] ? enqueue_entity+0x3a7/0xd20
[266447.179129] [<ffffffffc04980c9>] iptable_mangle_hook+0x39/0x107 [iptable_mangle]
[266447.179134] [<ffffffff81778e48>] nf_iterate+0x68/0x80
[266447.179137] [<ffffffff81778ed3>] nf_hook_slow+0x73/0xd0
[266447.179143] [<ffffffff8178642f>] ip_output+0xcf/0xe0
[266447.179148] [...

I built a test kernel with commit b2504a5dbef3305ef41988ad270b0e8ec289331c reverted.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1715609

Note, to test this kernel, you need to install both the linux-image and linux-image-extra .deb packages.

It might also be worth testing the latest mainline kernel, to see if this bug was fixed another way. That kernel can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16/

Thanks in advance!

Same output on 4.10.0-42：

Jul 20 16:04:19 ubuntu kernel: [832874.880924] nf_hook_slow+0x2a/0xb0
Jul 20 16:04:19 ubuntu kernel: [832874.880926] ? __dev_queue_xmit+0x276/0x690
Jul 20 16:04:19 ubuntu kernel: [832874.880928] br_forward_finish+0x83/0xb0 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880931] ? br_fdb_external_learn_del+0x120/0x120 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880932] br_nf_hook_thresh+0xac/0xc0 [br_netfilter]
Jul 20 16:04:19 ubuntu kernel: [832874.880934] ? skb_free_head+0x21/0x30
Jul 20 16:04:19 ubuntu kernel: [832874.880935] br_nf_forward_finish+0xe0/0x1b0 [br_netfilter]
Jul 20 16:04:19 ubuntu kernel: [832874.880938] ? br_dev_queue_push_xmit+0x150/0x150 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880939] br_nf_forward_ip+0x378/0x450 [br_netfilter]
Jul 20 16:04:19 ubuntu kernel: [832874.880946] ? tcp_in_window+0x109/0xa00 [nf_conntrack]
Jul 20 16:04:19 ubuntu kernel: [832874.880947] nf_hook_slow+0x2a/0xb0
Jul 20 16:04:19 ubuntu kernel: [832874.880950] __br_forward+0xc6/0x1e0 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880953] ? br_dev_queue_push_xmit+0x150/0x150 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880955] br_forward+0xa7/0xb0 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880958] br_handle_frame_finish+0x263/0x540 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880961] ? udp_error+0xe2/0x1e0 [nf_conntrack]
Jul 20 16:04:19 ubuntu kernel: [832874.880964] ? br_pass_frame_up+0x150/0x150 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880965] br_nf_hook_thresh+0xac/0xc0 [br_netfilter]
Jul 20 16:04:19 ubuntu kernel: [832874.880967] br_nf_pre_routing_finish+0x196/0x330 [br_netfilter]
Jul 20 16:04:19 ubuntu kernel: [832874.880970] ? br_pass_frame_up+0x150/0x150 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880972] ? ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4]
Jul 20 16:04:19 ubuntu kernel: [832874.880973] br_nf_pre_routing+0x2d6/0x430 [br_netfilter]
Jul 20 16:04:19 ubuntu kernel: [832874.880974] ? br_nf_forward_arp+0x260/0x260 [br_netfilter]
Jul 20 16:04:19 ubuntu kernel: [832874.880976] nf_hook_slow+0x2a/0xb0
Jul 20 16:04:19 ubuntu kernel: [832874.880978] br_handle_frame+0x1bc/0x2d0 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880981] ? br_pass_frame_up+0x150/0x150 [bridge]
Jul 20 16:04:19 ubuntu kernel: [832874.880982] __netif_receive_skb_core+0x323/0xa70
Jul 20 16:04:19 ubuntu kernel: [832874.880984] __netif_receive_skb+0x18/0x60
Jul 20 16:04:19 ubuntu kernel: [832874.880985] process_backlog+0x9f/0x140
Jul 20 16:04:19 ubuntu kernel: [832874.880987] net_rx_action+0x23e/0x390
Jul 20 16:04:19 ubuntu kernel: [832874.880989] __do_softirq+0xed/0x297
Jul 20 16:04:19 ubuntu kernel: [832874.880991] do_softirq_own_stack+0x1c/0x30
Jul 20 16:04:19 ubuntu kernel: [832874.880991] </IRQ>
Jul 20 16:04:19 ubuntu kernel: [832874.880993] do_softirq.part.18+0x41/0x50
Jul 20 16:04:19 ubuntu kernel: [832874.880994] __local_bh_enable_ip+0x7d/0x80
Jul 20 16:04:19 ubuntu kernel: [832874.880995] ip_finish_output2+0x193/0x390
Jul 20 16:04:19 ubuntu kernel: [832874.880997] ip_finish_output+0x176/0x270
Jul 20 16:04:19 ubuntu kernel: [832874.880998] ip_output+0x76/0xe0
Jul 20 1...