Ubuntu
linux package

Kernel Bug at skb_segment + 0x95a

Bug #1331219 reported by Dave Chiluk
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Medium
Dave Chiluk
Trusty
Fix Released
Undecided
Dave Chiluk
Utopic
Fix Released
Medium
Dave Chiluk

Bug Description

SRU Justification:
Impact: A crash occurs with VXLAN packets when using GSO capable hardware such as the case with openstack neutron.

Fix: Upstream fix 5882a07c7 fixes a locking error that previously allowed
certain timings to hit this bug.

Testcase: Create and run a Neutron gateway with VXLANs, and stress the network for an extended period of time.

Kernel Bug affects 3.11-3.15 on machines using gso offload.

Stack trace
[ 871.025524] ------------[ cut here ]------------
[ 871.030742] kernel BUG at /build/buildd/linux-3.13.0/net/core/skbuff.c:2903!
[ 871.038669] invalid opcode: 0000 [#1] SMP
[ 871.043318] Modules linked in: xt_nat xt_REDIRECT dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conn
track nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables veth openvswitch gre vxlan ip_tunnel
 dm_crypt 8021q garp stp mrp llc bonding nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache sb_edac gpio_ich joydev edac_core x86_pkg_temp_thermal intel_powerclamp coretemp lpc_ich acpi_power_m
eter kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ipmi_si mac_hid lp parport btrfs xor raid6_pq libcrc32c
hid_generic qla2xxx fnic libfcoe megaraid_sas
[ 871.124025] igb usbhid libfc hid dca scsi_transport_fc ptp enic scsi_tgt pps_core i2c_algo_bit wmi
[ 871.132967] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.13.0-29-generic #53-Ubuntu
[ 871.141500] Hardware name: Cisco Systems Inc UCSC-C220-M3S/UCSC-C220-M3S, BIOS C220M3.1.5.4f.0.111320130449 11/13/2013
[ 871.153541] task: ffff881fd2d117f0 ti: ffff881fd2d18000 task.ti: ffff881fd2d18000
[ 871.161975] RIP: 0010:[<ffffffff81612a6a>] [<ffffffff81612a6a>] skb_segment+0x95a/0x980
[ 871.171124] RSP: 0018:ffff881fffce3498 EFLAGS: 00010206
[ 871.177099] RAX: 0000000000000000 RBX: ffff881fb8929700 RCX: ffff881fcfba32f0
[ 871.185120] RDX: 0000000000000050 RSI: ffff881fcfba3200 RDI: ffff881fcfba2200
[ 871.193140] RBP: ffff881fffce3560 R08: 0000000000000042 R09: 0000000000000000
[ 871.201160] R10: ffff881fb8929e00 R11: 00000000000005ea R12: ffff881fcfba22f0
[ 871.209171] R13: 0000000000000000 R14: ffff881fd09a8400 R15: 0000000000000050
[ 871.217191] FS: 0000000000000000(0000) GS:ffff881fffce0000(0000) knlGS:0000000000000000
[ 871.226307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 871.232772] CR2: 00007fbd1ccb4000 CR3: 0000000001c0e000 CR4: 00000000001407e0
[ 871.240793] Stack:
[ 871.243079] 0000000000000000 0000000000000000 0000000000000042 ffffffffffffffbe
[ 871.251470] 00000001000005ea 0000000000000040 ffff881fb8929e00 0000000100000000
[ 871.259864] ffffffffffffffce 0000000000000074 00000032000005a8 ffff881fb8929700
[ 871.268257] Call Trace:
[ 871.271032] <IRQ>
[ 871.273193]
[ 871.274913] [<ffffffff8167cfbd>] tcp_gso_segment+0x10d/0x3f0
[ 871.279926] [<ffffffffa03740f0>] ? vxlan_set_owner+0x60/0x60 [vxlan]
[ 871.287178] [<ffffffff8168d422>] inet_gso_segment+0x132/0x360
[ 871.293749] [<ffffffffa045c701>] ? bond_mode_name+0x21/0x30 [bonding]
[ 871.301097] [<ffffffff8161fdac>] skb_mac_gso_segment+0x9c/0x180
[ 871.307858] [<ffffffff816840d7>] skb_udp_tunnel_segment+0xd7/0x390
[ 871.314906] [<ffffffff816847f0>] udp4_ufo_fragment+0x120/0x130
[ 871.321566] [<ffffffff8168d422>] inet_gso_segment+0x132/0x360
[ 871.328132] [<ffffffffa0563f67>] ? ipt_do_table+0x317/0x6aa [ip_tables]
[ 871.335669] [<ffffffffa05a3500>] ? nf_conntrack_hash_check_insert+0x280/0x2e0 [nf_conntrack]
[ 871.345268] [<ffffffff8161fdac>] skb_mac_gso_segment+0x9c/0x180
[ 871.352026] [<ffffffff8161feed>] __skb_gso_segment+0x5d/0xb0
[ 871.358494] [<ffffffff816201fa>] dev_hard_start_xmit+0x18a/0x560
[ 871.365351] [<ffffffff816208e8>] __dev_queue_xmit+0x318/0x500
[ 871.371910] [<ffffffffa05c0238>] ? ipv4_confirm+0x78/0x100 [nf_conntrack_ipv4]
[ 871.380156] [<ffffffff8163e03b>] ? eth_header+0x2b/0xd0
[ 871.386135] [<ffffffff81620ae0>] dev_queue_xmit+0x10/0x20
[ 871.392315] [<ffffffff81629367>] neigh_connected_output+0xb7/0x100
[ 871.399366] [<ffffffff81658eb0>] ip_finish_output+0x1b0/0x3b0
[ 871.405922] [<ffffffff8165a418>] ip_output+0x58/0x90
[ 871.411609] [<ffffffff81659b75>] ip_local_out+0x25/0x30
[ 871.417596] [<ffffffff8169c43e>] iptunnel_xmit+0xee/0x110
[ 871.423772] [<ffffffffa0376581>] vxlan_xmit_skb+0x1d1/0x350 [vxlan]
[ 871.430928] [<ffffffffa055956b>] vxlan_tnl_send+0x11b/0x190 [openvswitch]
[ 871.438666] [<ffffffffa055895d>] ovs_vport_send+0x1d/0x80 [openvswitch]
[ 871.446203] [<ffffffffa054f19a>] do_output+0x2a/0x50 [openvswitch]
[ 871.453255] [<ffffffffa054f643>] do_execute_actions+0x2e3/0xa90 [openvswitch]
[ 871.461394] [<ffffffff8163f1ff>] ? sch_direct_xmit+0x5f/0x1c0
[ 871.467960] [<ffffffffa054fe1b>] ovs_execute_actions+0x2b/0x30 [openvswitch]
[ 871.475979] [<ffffffffa05526e2>] ovs_dp_process_received_packet+0x92/0x120 [openvswitch]
[ 871.485198] [<ffffffff8136ced1>] ? csum_partial+0x11/0x20
[ 871.491376] [<ffffffffa055889a>] ovs_vport_receive+0x2a/0x30 [openvswitch]
[ 871.499206] [<ffffffffa0558ced>] internal_dev_xmit+0x1d/0x30 [openvswitch]
[ 871.507035] [<ffffffff81620388>] dev_hard_start_xmit+0x318/0x560
[ 871.513890] [<ffffffff816208e8>] __dev_queue_xmit+0x318/0x500
[ 871.520454] [<ffffffff81620ae0>] dev_queue_xmit+0x10/0x20
[ 871.526630] [<ffffffff81658fc9>] ip_finish_output+0x2c9/0x3b0
[ 871.533193] [<ffffffff8165a418>] ip_output+0x58/0x90
[ 871.538884] [<ffffffff8165639b>] ip_forward_finish+0x8b/0x170
[ 871.545446] [<ffffffff816567d5>] ip_forward+0x355/0x410
[ 871.551427] [<ffffffff816544ed>] ip_rcv_finish+0x7d/0x350
[ 871.557600] [<ffffffff81654e38>] ip_rcv+0x298/0x3d0
[ 871.563191] [<ffffffff8161e9b6>] __netif_receive_skb_core+0x666/0x840
[ 871.570532] [<ffffffff8161eba8>] __netif_receive_skb+0x18/0x60
[ 871.586559] [<ffffffff8161f6ce>] process_backlog+0xae/0x1a0
[ 871.592928] [<ffffffff8161ef92>] net_rx_action+0x152/0x250
[ 871.599211] [<ffffffff8106caec>] __do_softirq+0xec/0x2c0
[ 871.605288] [<ffffffff8106d035>] irq_exit+0x105/0x110
[ 871.611081] [<ffffffff8172cf16>] do_IRQ+0x56/0xc0
[ 871.616485] [<ffffffff817226ad>] common_interrupt+0x6d/0x6d
[ 871.622850] <EOI>
[ 871.625006]
[ 871.626727] [<ffffffff815cd0f2>] ? cpuidle_enter_state+0x52/0xc0
[ 871.632122] [<ffffffff815cd219>] cpuidle_idle_call+0xb9/0x1f0
[ 871.638694] [<ffffffff8101ce9e>] arch_cpu_idle+0xe/0x30
[ 871.644681] [<ffffffff810beb95>] cpu_startup_entry+0xc5/0x290
[ 871.651257] [<ffffffff81040fb8>] start_secondary+0x218/0x2c0
[ 871.657722] Code: 0f 84 0c ff ff ff 8b 54 24 50 48 c7 c7 18 ae ae 81 44 89 ce 31 c0 e8 75 14 10 00 48 8b 7c 24 58 e9 ff fe ff ff 0f 1f 40 00 0f 0b <0f> 0b 0f 0b e8 d0 a5 10 00 48 c7 c0 ea f
f ff ff e9 c7 fd ff ff
[ 871.679616] RIP [<ffffffff81612a6a>] skb_segment+0x95a/0x980
[ 871.686097] RSP <ffff881fffce3498>

Fix has been committed in the upstream linux tree at 5882a07c72093dc3a18e2d2b129fb200686bb6ee

Netdev conversation can be viewed here. http://patchwork.ozlabs.org/patch/357291/

See original description
Dave Chiluk (chiluk)
description: updated
description: updated
Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1331219

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Dave Chiluk (chiluk) wrote :

I have sent an e-mail to the linux-stable mailing list requesting that this patch be included in the stable trees.

Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu):
importance: Undecided → Medium
status: Incomplete → Triaged
Dave Chiluk (chiluk)
description: updated
Changed in linux (Ubuntu):
assignee: nobody → Dave Chiluk (chiluk)
status: Triaged → In Progress
Tim Gardner (timg-tpi)
Changed in linux (Ubuntu Trusty):
assignee: nobody → Dave Chiluk (chiluk)
status: New → Fix Committed
Changed in linux (Ubuntu Utopic):
status: In Progress → Fix Released
Revision history for this message
Adam Conrad (adconrad) wrote : Update Released

The verification of the Stable Release Update for linux-lts-trusty has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 3.13.0-33.58

---------------
linux (3.13.0-33.58) trusty; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1349897

  [ Upstream Kernel Changes ]

  * mm: numa: do not automatically migrate KSM pages
    - LP: #1346917
  * net: fix UDP tunnel GSO of frag_list GRO packets
    - LP: #1331219
  * auditsc: audit_krule mask accesses need bounds checking
    - LP: #1347088
  * n_tty: Fix buffer overruns with larger-than-4k pastes
    - LP: #1208740
 -- Tim Gardner <email address hidden> Fri, 18 Jul 2014 14:57:50 +0000

Changed in linux (Ubuntu Trusty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.