Juniper Openstack

kernel crash when port mirroring enabled

Bug #1577473 reported by amit surana
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.22.x
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r2.22.2
R3.0
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.0.2.0
R3.1
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.1.0.0-fcs
R3.2
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.2.9.0
Trunk
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"

Bug Description

3.0.2 b 32.

[33994.105976] ------------[ cut here ]------------
[33994.106019] kernel BUG at /build/buildd/linux-3.13.0/net/core/skbuff.c:2904!
[33994.106069] invalid opcode: 0000 [#1] SMP
[33994.106098] Modules linked in: vhost_net vhost macvtap macvlan ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables nbd ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vrouter(OX) ast ttm drm_kms_helper drm bonding x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm gpio_ich syscopyarea sysfillrect dm_multipath scsi_dh sb_edac crct10dif_pclmul sysimgblt crc32_pclmul ghash_clmulni_intel mei_me nfsd mei aesni_intel ipmi_si edac_core auth_rpcgss aes_x86_64 lrw gf128mul lpc_ich shpchp glue_helper ablk_helper cryptd wmi mac_hid nfs_acl nfs
[33994.111734] lockd sunrpc fscache lp parport igb ixgbe isci i2c_algo_bit mdio dca libsas ahci ptp libahci megaraid_sas scsi_transport_sas pps_core
[33994.113883] CPU: 32 PID: 0 Comm: swapper/32 Tainted: G OX 3.13.0-40-generic #69-Ubuntu
[33994.115037] Hardware name: Quanta S810-X52L/S2L, BIOS S2L_4A14 01/29/2015
[33994.116148] task: ffff883fd2ad8000 ti: ffff881fd2ff0000 task.ti: ffff881fd2ff0000
[33994.117288] RIP: 0010:[<ffffffff8161764a>] [<ffffffff8161764a>] skb_segment+0x95a/0x980
[33994.118696] RSP: 0018:ffff883fff303890 EFLAGS: 00010202
[33994.120043] RAX: 0000000000000000 RBX: ffff883ece7c9b00 RCX: ffff88198cd08900
[33994.121275] RDX: 0000000000005e0a RSI: 0000000000000580 RDI: ffffea00fee30d80
[33994.122482] RBP: ffff883fff303958 R08: 0000000000005dbe R09: 0000000000000011
[33994.123705] R10: ffff883ece7c9700 R11: 0000000000005e8a R12: ffff883a12adcc00
[33994.124970] R13: 0000000000000011 R14: ffff88189d601d00 R15: 0000000000005e0a
[33994.126214] FS: 0000000000000000(0000) GS:ffff883fff300000(0000) knlGS:0000000000000000
[33994.127484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[33994.128736] CR2: 00007fbc915b5ec0 CR3: 0000003df3748000 CR4: 00000000001427e0
[33994.129992] Stack:
[33994.131232] ffffffff81d19f30 0000000000000080 000000000000003e ffffffffffffffc2
[33994.132535] 0000000000005e8a 0000000000000014 ffff883ece7c9700 0000000101001d00
[33994.133838] 0000000000000000 000000000000003e 0000000000000580 ffff883ece7c8500
[33994.135112] Call Trace:
[33994.136477] <IRQ>
[33994.136497]
[33994.137937] [<ffffffff816896b0>] udp4_ufo_fragment+0xc0/0x130
[33994.139361] [<ffffffff81692332>] inet_gso_segment+0x132/0x360
[33994.140730] [<ffffffff81624a8c>] skb_mac_gso_segment+0x9c/0x180
[33994.142070] [<ffffffff81624bcd>] __skb_gso_segment+0x5d/0xb0
[33994.143407] [<ffffffffa049976b>] linux_if_tx+0x2fb/0x3b0 [vrouter]
[33994.144765] [<ffffffffa04a3e6d>] eth_tx+0x8d/0x1a0 [vrouter]
[33994.146156] [<ffffffffa04a4fb0>] ? vif_cmn_rewrite.part.10+0x40/0x90 [vrouter]
[33994.147547] [<ffffffffa049f6cb>] nh_gre_tunnel+0x21b/0x3d0 [vrouter]
[33994.148956] [<ffffffffa049fbca>] nh_output+0x4a/0x150 [vrouter]
[33994.150371] [<ffffffffa04a791e>] vr_forward+0xee/0x200 [vrouter]
[33994.151721] [<ffffffffa049e350>] ? nh_udp_tunnel_helper+0x110/0x150 [vrouter]
[33994.153099] [<ffffffffa049ec54>] nh_udp_tunnel+0xf4/0x430 [vrouter]
[33994.154502] [<ffffffff81614f83>] ? pskb_expand_head+0x113/0x250
[33994.155884] [<ffffffffa049fbca>] nh_output+0x4a/0x150 [vrouter]
[33994.157139] [<ffffffffa04aee9d>] vr_mirror+0x1cd/0x300 [vrouter]
[33994.158422] [<ffffffffa04a3f0f>] eth_tx+0x12f/0x1a0 [vrouter]
[33994.159688] [<ffffffffa049eae6>] nh_encap_l3+0x1b6/0x230 [vrouter]
[33994.161140] [<ffffffffa049fbca>] nh_output+0x4a/0x150 [vrouter]
[33994.162488] [<ffffffffa0498cbc>] pkt_gro_dev_rx_handler+0x23c/0x2b0 [vrouter]
[33994.163742] [<ffffffff81623222>] __netif_receive_skb_core+0x262/0x840
[33994.165093] [<ffffffff81623818>] __netif_receive_skb+0x18/0x60
[33994.166375] [<ffffffff816243ae>] process_backlog+0xae/0x1a0
[33994.167738] [<ffffffff81623c02>] net_rx_action+0x152/0x250
[33994.169022] [<ffffffff8106cc1c>] __do_softirq+0xec/0x2c0
[33994.170226] [<ffffffff8106d165>] irq_exit+0x105/0x110
[33994.171420] [<ffffffff81040a25>] smp_call_function_single_interrupt+0x35/0x40
[33994.172536] [<ffffffff81730f1d>] call_function_single_interrupt+0x6d/0x80
[33994.173579] <EOI>
[33994.173591]
[33994.174648] [<ffffffff815d1d82>] ? cpuidle_enter_state+0x52/0xc0
[33994.175726] [<ffffffff815d1ea9>] cpuidle_idle_call+0xb9/0x1f0
[33994.176779] [<ffffffff8101d2be>] arch_cpu_idle+0xe/0x30
[33994.177771] [<ffffffff810bee55>] cpu_startup_entry+0xc5/0x290
[33994.178776] [<ffffffff8104134d>] start_secondary+0x21d/0x2d0
[33994.179766] Code: 0f 84 0c ff ff ff 8b 54 24 50 48 c7 c7 d0 d1 ae 81 44 89 ce 31 c0 e8 e7 17 10 00 48 8b 7c 24 58 e9 ff fe ff ff 0f 1f 40 00 0f 0b <0f> 0b 0f 0b e8 fc a9 10 00 48 c7 c0 ea ff ff ff e9 c7 fd ff ff
[33994.181949] RIP [<ffffffff8161764a>] skb_segment+0x95a/0x980
[33994.182990] RSP <ffff883fff303890>

Tags: vrouter soln
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/20197
Submitter: Divakar Dharanalakota (<email address hidden>)

Jeba Paulaiyan (jebap)
tags: added: releasenote
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20197
Committed: http://github.org/Juniper/contrail-vrouter/commit/5dfa88c2a7136ab4ec1583789552e011bb1a785a
Submitter: Zuul
Branch: R2.22.x

commit 5dfa88c2a7136ab4ec1583789552e011bb1a785a
Author: Divakar <email address hidden>
Date: Fri May 13 20:02:58 2016 +0530

Disabling GRO when mirroring is enabled

When the packets are received on Fabric interface, these are subjected
to GRO before transmitting them to Tap interface. If Tx port mirroring
is enabled on Tap interface, as of now, mirroring is applied after GRO
of the packets. If the mirroring server is on another compute node, these
mirrored packets have GSO enabled. Vrouter makes use of Linux GSO routines
and these GSO routines expect that skb in skb_list of head_skb does not
contain any linear data (skb_headlen should be zero). Due to GRO the
skb_headle of some skb's in skb_list contains linear data resulting in
GSO routiness hitting a BUG_ON.

To over come this, the GRO needs to be applied post mirroring. To
enabled GSO on the mirrored packets the skb's gso_len also need to be
supplied. Instead of this fix, GRO is disabled if Tx port mirroring is
enabled on Tap interface as stop gap fix till the complete fix is in
place.

Change-Id: I7dd86e1bd90fef60efa4c9dbac78e853952c4fdc
closes-bug: #1577473

Revision history for this message
Nagabhushana R (bhushana) wrote :

removing the blocker tag, a preventive fix is in place.

tags: removed: blocker
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/20566
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20566
Committed: http://github.org/Juniper/contrail-vrouter/commit/0919fd40795d6510aaa1bf6266174bbbaf00a793
Submitter: Zuul
Branch: R3.0

commit 0919fd40795d6510aaa1bf6266174bbbaf00a793
Author: Divakar <email address hidden>
Date: Fri May 13 20:02:58 2016 +0530

Disabling GRO when mirroring is enabled

When the packets are received on Fabric interface, these are subjected
to GRO before transmitting them to Tap interface. If Tx port mirroring
is enabled on Tap interface, as of now, mirroring is applied after GRO
of the packets. If the mirroring server is on another compute node, these
mirrored packets have GSO enabled. Vrouter makes use of Linux GSO routines
and these GSO routines expect that skb in skb_list of head_skb does not
contain any linear data (skb_headlen should be zero). Due to GRO the
skb_headle of some skb's in skb_list contains linear data resulting in
GSO routiness hitting a BUG_ON.

To over come this, the GRO needs to be applied post mirroring. To
enabled GSO on the mirrored packets the skb's gso_len also need to be
supplied. Instead of this fix, GRO is disabled if Tx port mirroring is
enabled on Tap interface as stop gap fix till the complete fix is in
place.

Change-Id: I7dd86e1bd90fef60efa4c9dbac78e853952c4fdc
closes-bug: #1577473

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/22620
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/22620
Committed: http://github.org/Juniper/contrail-vrouter/commit/47d62dcd4a499941235550447ec15faed5e33abb
Submitter: Zuul
Branch: R3.1

commit 47d62dcd4a499941235550447ec15faed5e33abb
Author: Divakar <email address hidden>
Date: Fri May 13 20:02:58 2016 +0530

Disabling GRO when mirroring is enabled

When the packets are received on Fabric interface, these are subjected
to GRO before transmitting them to Tap interface. If Tx port mirroring
is enabled on Tap interface, as of now, mirroring is applied after GRO
of the packets. If the mirroring server is on another compute node, these
mirrored packets have GSO enabled. Vrouter makes use of Linux GSO routines
and these GSO routines expect that skb in skb_list of head_skb does not
contain any linear data (skb_headlen should be zero). Due to GRO the
skb_headle of some skb's in skb_list contains linear data resulting in
GSO routiness hitting a BUG_ON.

To over come this, the GRO needs to be applied post mirroring. To
enabled GSO on the mirrored packets the skb's gso_len also need to be
supplied. Instead of this fix, GRO is disabled if Tx port mirroring is
enabled on Tap interface as stop gap fix till the complete fix is in
place.

Change-Id: Iaeeeb4332f2166c2bf23a8770c15e736e47120bb
closes-bug: #1577473

Nagabhushana R (bhushana)
tags: removed: releasenote
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/23923
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/23923
Committed: http://github.org/Juniper/contrail-vrouter/commit/cd7804d55705d664d8bdfcaecc0f99a4601f5e35
Submitter: Zuul
Branch: master

commit cd7804d55705d664d8bdfcaecc0f99a4601f5e35
Author: Divakar <email address hidden>
Date: Fri May 13 20:02:58 2016 +0530

Disabling GRO when mirroring is enabled

When the packets are received on Fabric interface, these are subjected
to GRO before transmitting them to Tap interface. If Tx port mirroring
is enabled on Tap interface, as of now, mirroring is applied after GRO
of the packets. If the mirroring server is on another compute node, these
mirrored packets have GSO enabled. Vrouter makes use of Linux GSO routines
and these GSO routines expect that skb in skb_list of head_skb does not
contain any linear data (skb_headlen should be zero). Due to GRO the
skb_headle of some skb's in skb_list contains linear data resulting in
GSO routiness hitting a BUG_ON.

To over come this, the GRO needs to be applied post mirroring. To
enabled GSO on the mirrored packets the skb's gso_len also need to be
supplied. Instead of this fix, GRO is disabled if Tx port mirroring is
enabled on Tap interface as stop gap fix till the complete fix is in
place.

Change-Id: I02b585bf919323b5dd30e36f1f2edade7705a60f
closes-bug: #1577473

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/31230
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/31230
Committed: http://github.com/Juniper/contrail-vrouter/commit/7d7b8967514e199caafae966011f9477b756bf96
Submitter: Zuul (<email address hidden>)
Branch: master

commit 7d7b8967514e199caafae966011f9477b756bf96
Author: Divakar D <email address hidden>
Date: Tue May 9 16:20:33 2017 +0530

Unset GRO flag, if GRO not invoked

Due to bug 1577473, we do not invoke GRO on a packet if Tx port
mirroring is enabled on the interface. If the packet is already marked
for GRO and if we do not subject the packet for GRO, the VP_FLAG_GRO
will not be unset. While transmitting this packet on interface, to ensure
that the tx counters are incremented only once, packet with GRO flag
disables the statistics. Due to this, tx counters are not getting
incremented when Tx port mirroring is enabled.

As a fix, GRO flag is unset unconditionally after GRO invocation checks

Change-Id: I3e5c1c2583d82e9eabfdc6dcda0eb022c9f4f422
closes-bug: #1672534

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/34031
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/34031
Committed: http://github.com/Juniper/contrail-vrouter/commit/554d2c924f7120f2e882c7f5bce849d5409e2eeb
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 554d2c924f7120f2e882c7f5bce849d5409e2eeb
Author: Divakar D <email address hidden>
Date: Tue May 9 16:20:33 2017 +0530

Unset GRO flag, if GRO not invoked

Due to bug 1577473, we do not invoke GRO on a packet if Tx port
mirroring is enabled on the interface. If the packet is already marked
for GRO and if we do not subject the packet for GRO, the VP_FLAG_GRO
will not be unset. While transmitting this packet on interface, to ensure
that the tx counters are incremented only once, packet with GRO flag
disables the statistics. Due to this, tx counters are not getting
incremented when Tx port mirroring is enabled.

As a fix, GRO flag is unset unconditionally after GRO invocation checks

closes-bug: #1672534

Conflicts:
 dp-core/vr_nexthop.c

Change-Id: Ibcd0677a300ee7b3774b5f0cb1ee3593d3924361

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.