Juniper Openstack

kernel crash at lh_free in a small scale setup

Bug #1572397 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.0.2.0
R3.1
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.1.0.0-fcs
R3.2
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.2.9.0
R4.0
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r4.0.3.0
R4.1
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r4.1.1.0
Trunk
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r5.0.0

Bug Description

R3.0.2.0 Build 26 Ubuntu 14.04 Kilo

Below kernel crash was seen on a compute node when 100 vms were sending traffic to each other in a VN with mirroring configured.

Crash details will be in http://10.204.216.50/Docs/bugs/#

[419003.127632] ------------[ cut here ]------------
[419003.167029] kernel BUG at /build/buildd/linux-3.13.0/mm/slub.c:3365!
[419003.207462] invalid opcode: 0000 [#1] SMP
[419003.247432] Modules linked in: vhost_net vhost macvtap macvlan vrouter(OX) xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nbd ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache gpio_ich lp dm_multipath parport scsi_dh ttm x86_pkg_temp_thermal intel_powerclamp coretemp drm_kms_helper kvm_intel kvm drm crct10dif_pclmul crc32_pclmul syscopyarea ghash_clmulni_intel sysfillrect cryptd sysimgblt joydev ioatdma lpc_ich shpchp wmi ipmi_si acpi_power_meter acpi_pad mac_hid hid_generic usbhid hid
[419003.680981] igb ixgbe i2c_algo_bit dca ahci ptp libahci pps_core mdio [last unloaded: vrouter]
[419003.771578] CPU: 28 PID: 26 Comm: rcuos/17 Tainted: G OX 3.13.0-40-generic #69-Ubuntu
[419003.862639] Hardware name: Supermicro X10DRW/X10DRW-i, BIOS 1.0c 01/07/2015
[419003.910415] task: ffff881fd3324800 ti: ffff881fd332c000 task.ti: ffff881fd332c000
[419004.005351] RIP: 0010:[<ffffffff811a1e73>] [<ffffffff811a1e73>] kfree+0x133/0x140
[419004.103024] RSP: 0018:ffff881fd332dc18 EFLAGS: 00010246
[419004.152751] RAX: 06ffff0000000000 RBX: ffff883e6110e000 RCX: ffff883e6110e100
[419004.250816] RDX: 06ffff0000000000 RSI: 0000000000000020 RDI: ffff883e6110e000
[419004.350325] RBP: ffff881fd332dc30 R08: ffff881fd332dd58 R09: 0000000000000000
[419004.453778] R10: ffffea00f9844380 R11: ffffffffa052100f R12: ffff883e6110e000
[419004.559118] R13: ffffffffa0520fa1 R14: 0000000000000002 R15: ffff881b8e0f46d0
[419004.665832] FS: 0000000000000000(0000) GS:ffff88407fd80000(0000) knlGS:0000000000000000
[419004.772906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[419004.826248] CR2: 00007fff72306b58 CR3: 0000000001c0e000 CR4: 00000000001427e0
[419004.930895] Stack:
[419004.981377] ffff883e6110e000 ffff883e6110e000 0000000000000003 ffff881fd332dc48
[419005.083090] ffffffffa0520fa1 ffff881edb4a62a0 ffff881fd332dc88 ffffffffa0527a62
[419005.186616] 0000000000000282 ffff881edb4a62a0 ffff881b8e0f4000 0000000000000002
[419005.292577] Call Trace:
[419005.344454] [<ffffffffa0520fa1>] lh_free+0x21/0x30 [vrouter]
[419005.400316] [<ffffffffa0527a62>] __vr_itable_del+0xf2/0x190 [vrouter]
[419005.455918] [<ffffffffa05279d0>] __vr_itable_del+0x60/0x190 [vrouter]
[419005.509695] [<ffffffffa05279d0>] __vr_itable_del+0x60/0x190 [vrouter]
[419005.561167] [<ffffffff811a24be>] ? __kmalloc+0x2e/0x230
[419005.611545] [<ffffffffa05279d0>] __vr_itable_del+0x60/0x190 [vrouter]
[419005.662208] [<ffffffffa0527ef9>] vr_itable_del+0x29/0x40 [vrouter]
[419005.712201] [<ffffffffa053a935>] vr_mirror_meta_entry_del+0x15/0x30 [vrouter]
[419005.808194] [<ffffffffa0537898>] vr_flow_reset_mirror+0x48/0x70 [vrouter]
[419005.857779] [<ffffffffa0537911>] __vr_flow_reset_entry+0x51/0x80 [vrouter]
[419005.907276] [<ffffffffa05379e5>] vr_flow_evict_flow+0x55/0x70 [vrouter]
[419005.955217] [<ffffffffa0538daa>] vr_flow_defer_cb+0x7a/0xd0 [vrouter]
[419006.002343] [<ffffffffa0520ff8>] rcu_cb+0x18/0x40 [vrouter]
[419006.047462] [<ffffffff810c8141>] rcu_nocb_kthread+0x1d1/0x2e0
[419006.092029] [<ffffffff810ab0a0>] ? prepare_to_wait_event+0x100/0x100
[419006.136510] [<ffffffff810c7f70>] ? __call_rcu_nocb_enqueue+0xc0/0xc0
[419006.180634] [<ffffffff8108b562>] kthread+0xd2/0xf0
[419006.223159] [<ffffffff8108b490>] ? kthread_create_on_node+0x1c0/0x1c0
[419006.266079] [<ffffffff8172fc7c>] ret_from_fork+0x7c/0xb0
[419006.307899] [<ffffffff8108b490>] ? kthread_create_on_node+0x1c0/0x1c0
[419006.349031] Code: 49 8b 02 31 f6 f6 c4 40 74 04 41 8b 72 68 4c 89 d7 e8 82 58 fb ff eb 8f 4c 8b 50 30 48 8b 10 80 e6 80 4c 0f 44 d0 e9 32 ff ff ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 89 f1
[419006.478521] RIP [<ffffffff811a1e73>] kfree+0x133/0x140
[419006.518250] RSP <ffff881fd332dc18>

amit surana (asurana-t)
tags: added: blocker
Revision history for this message
Ashish Ranjan (aranjan-n) wrote :

Seems same as 1577473. Need to observe. Not targeted for 3.0.2 release.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/20712
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20712
Committed: http://github.org/Juniper/contrail-vrouter/commit/063ab3dfea324e526894bdf41908ea0a13b37ab4
Submitter: Zuul
Branch: R3.0

commit 063ab3dfea324e526894bdf41908ea0a13b37ab4
Author: Divakar <email address hidden>
Date: Fri May 27 15:58:31 2016 +0530

Keeping the mirror metadata entry in Flow entry

Currently mirror metadata is stored in index table using flow index
as key. Before the Flow eviction, the addition and deletion of this meta
data to Index table is always from Agent and hence no parallel
manipulation of Index table. After eviction, the metadata deletion from
index table can happen in parallel and hence leading to memory
corruption/leak in index table manipulation.

The right fix is making the index table SMP ready. As a stop gap fix,
the mirror metada data pointer is stored in the flow entry itself
directly.

Change-Id: I4c98af36646c3389996575af8eac3e98b3ffdca2
closes-bug: #1572397

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/22376
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/22376
Committed: http://github.org/Juniper/contrail-vrouter/commit/e9067998b9dde14a937fa2c96106b8b439659430
Submitter: Zuul
Branch: R3.1

commit e9067998b9dde14a937fa2c96106b8b439659430
Author: Divakar <email address hidden>
Date: Fri May 27 15:58:31 2016 +0530

Keeping the mirror metadata entry in Flow entry

Currently mirror metadata is stored in index table using flow index
as key. Before the Flow eviction, the addition and deletion of this meta
data to Index table is always from Agent and hence no parallel
manipulation of Index table. After eviction, the metadata deletion from
index table can happen in parallel and hence leading to memory
corruption/leak in index table manipulation.

The right fix is making the index table SMP ready. As a stop gap fix,
the mirror metada data pointer is stored in the flow entry itself
directly.

closes-bug: #1572397

Change-Id: I216d8b6c55025a6053b808fd2807338265b307e0

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/39801
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/39801
Committed: http://github.com/Juniper/contrail-vrouter/commit/a437faa4bb2b2c52ffb06c5e3635766d89e4ef60
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit a437faa4bb2b2c52ffb06c5e3635766d89e4ef60
Author: Divakar <email address hidden>
Date: Fri May 27 15:58:31 2016 +0530

Keeping the mirror metadata entry in Flow entry

Currently mirror metadata is stored in index table using flow index
as key. Before the Flow eviction, the addition and deletion of this meta
data to Index table is always from Agent and hence no parallel
manipulation of Index table. After eviction, the metadata deletion from
index table can happen in parallel and hence leading to memory
corruption/leak in index table manipulation.

The right fix is making the index table SMP ready. As a stop gap fix,
the mirror metada data pointer is stored in the flow entry itself
directly.

Conflicts:
 include/vr_flow.h

closes-bug: #1572397, #1744369
Change-Id: I216d8b6c55025a6053b808fd2807338265b307e0

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/40943
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/40944
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/40945
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/40943
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/40945
Committed: http://github.com/Juniper/contrail-vrouter/commit/9cb7dbdb52e74b7ac6212e5c0f9bf82b087d9f86
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 9cb7dbdb52e74b7ac6212e5c0f9bf82b087d9f86
Author: Divakar <email address hidden>
Date: Fri May 27 15:58:31 2016 +0530

Keeping the mirror metadata entry in Flow entry

Currently mirror metadata is stored in index table using flow index
as key. Before the Flow eviction, the addition and deletion of this meta
data to Index table is always from Agent and hence no parallel
manipulation of Index table. After eviction, the metadata deletion from
index table can happen in parallel and hence leading to memory
corruption/leak in index table manipulation.

The right fix is making the index table SMP ready. As a stop gap fix,
the mirror metada data pointer is stored in the flow entry itself
directly.

Conflicts:
 include/vr_flow.h

closes-bug: #1572397, #1744369
Change-Id: I216d8b6c55025a6053b808fd2807338265b307e0
(cherry picked from commit a437faa4bb2b2c52ffb06c5e3635766d89e4ef60)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/40944
Committed: http://github.com/Juniper/contrail-vrouter/commit/505bafc1148844539bb961132286437da92c2a3d
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 505bafc1148844539bb961132286437da92c2a3d
Author: Divakar <email address hidden>
Date: Fri May 27 15:58:31 2016 +0530

Keeping the mirror metadata entry in Flow entry

Currently mirror metadata is stored in index table using flow index
as key. Before the Flow eviction, the addition and deletion of this meta
data to Index table is always from Agent and hence no parallel
manipulation of Index table. After eviction, the metadata deletion from
index table can happen in parallel and hence leading to memory
corruption/leak in index table manipulation.

The right fix is making the index table SMP ready. As a stop gap fix,
the mirror metada data pointer is stored in the flow entry itself
directly.

Conflicts:
 include/vr_flow.h

closes-bug: #1572397, #1744369
Change-Id: I216d8b6c55025a6053b808fd2807338265b307e0
(cherry picked from commit a437faa4bb2b2c52ffb06c5e3635766d89e4ef60)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/40943
Committed: http://github.com/Juniper/contrail-vrouter/commit/b04b55134f70ee19448989c70e9ca66831176ebb
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit b04b55134f70ee19448989c70e9ca66831176ebb
Author: Divakar <email address hidden>
Date: Fri May 27 15:58:31 2016 +0530

Keeping the mirror metadata entry in Flow entry

Currently mirror metadata is stored in index table using flow index
as key. Before the Flow eviction, the addition and deletion of this meta
data to Index table is always from Agent and hence no parallel
manipulation of Index table. After eviction, the metadata deletion from
index table can happen in parallel and hence leading to memory
corruption/leak in index table manipulation.

The right fix is making the index table SMP ready. As a stop gap fix,
the mirror metada data pointer is stored in the flow entry itself
directly.

Conflicts:
 include/vr_flow.h

closes-bug: #1572397, #1744369
Change-Id: I216d8b6c55025a6053b808fd2807338265b307e0

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.