controller cores during scale tests

Bug #1426175 reported by Nagendra Prasath
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R1.1
Fix Released
High
Prakash Bailkeri
R2.0
Fix Released
High
Prakash Bailkeri
R2.1
Fix Committed
High
Prakash Bailkeri
Trunk
Fix Committed
High
Prakash Bailkeri

Bug Description

As a part of scale testing, script creates 4 VNs per tenant and 16 VMs per VN. When script was creating #155th tenant, below error is seen.
Seems API is down and noticed control cores.

cores @ /cs-shared/npchandran/scale-logs-1/cores/control for reference.
logs @ /cs-shared/npchandran/scale-logs-1

Image Used:
Ubuntu-14-04, Icehouse, R2.10/34

Parent Bug:
https://bugs.launchpad.net/juniperopenstack/+bug/1424802

description: updated
Changed in juniperopenstack:
assignee: nobody → Prakash Bailkeri (prakashmb)
Revision history for this message
Nagendra Prasath (npchandran) wrote :
Download full text (4.2 KiB)

Trace from core:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-control'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 IFMapDependencyTracker::PropagateEdge (this=this@entry=0x2ee51b0, node=node@entry=0x667265746e695f65, metadata=..., in_edges=in_edges@entry=0x7feb6effaaa0)
    at controller/src/ifmap/ifmap_dependency_tracker.cc:174
174 controller/src/ifmap/ifmap_dependency_tracker.cc: No such file or directory.
(gdb)
(gdb) bt
#0 IFMapDependencyTracker::PropagateEdge (this=this@entry=0x2ee51b0, node=node@entry=0x667265746e695f65, metadata=..., in_edges=in_edges@entry=0x7feb6effaaa0)
    at controller/src/ifmap/ifmap_dependency_tracker.cc:174
#1 0x000000000086f851 in IFMapDependencyTracker::PropagateNode (this=this@entry=0x2ee51b0, node=<optimized out>, in_edges=in_edges@entry=0x7feb6effaaa0)
    at controller/src/ifmap/ifmap_dependency_tracker.cc:165
#2 0x000000000086f9e7 in IFMapDependencyTracker::PropagateChanges (this=0x2ee51b0) at controller/src/ifmap/ifmap_dependency_tracker.cc:82
#3 0x000000000045515a in IFMapConfigListener::GetChangeList (this=0x2ecbdb0, change_list=0x7feb6effab30) at controller/src/ifmap/ifmap_config_listener.cc:100
#4 0x00000000005b2ce7 in BgpConfigManager::ConfigHandler (this=0x2ecbc80) at controller/src/bgp/bgp_config.cc:1238
#5 0x0000000000a6101f in operator() (this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#6 TaskTrigger::WorkerTask::Run (this=0x7feb3093be10) at controller/src/base/task_trigger.cc:19
#7 0x0000000000a5b820 in TaskImpl::execute (this=0x7feb80394a40) at controller/src/base/task.cc:232
#8 0x00007feb8ad45b3a in ?? () from /usr/lib/libtbb.so.2
#9 0x00007feb8ad41816 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007feb8ad40f4b in ?? () from /usr/lib/libtbb.so.2
#11 0x00007feb8ad3d0ff in ?? () from /usr/lib/libtbb.so.2
#12 0x00007feb8ad3d2f9 in ?? () from /usr/lib/libtbb.so.2
#13 0x00007feb8af61182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007feb8a031fbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-control'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 IFMapDependencyTracker::PropagateEdge (this=this@entry=0x148e180, node=node@entry=0x11, metadata=..., in_edges=in_edges@entry=0x7f3db3bfdaa0) at controller/src/ifmap/ifmap_dependency_tracker.cc:174
174 controller/src/ifmap/ifmap_dependency_tracker.cc: No such file or directory.
(gdb)
(gdb) bt
#0 IFMapDependencyTracker::PropagateEdge (this=this@entry=0x148e180, node=node@entry=0x11, metadata=..., in_edges=in_edges@entry=0x7f3db3bfdaa0) at controller/src/ifmap/ifmap_dependency_tracker.cc:174
#1 0x000000000086f851 in IFMapDependencyTracker::PropagateNode (this=this@entry=0x148e180, node=<optimized out>, in_edges=in_edges@entry=0x7f3db3bfdaa0)
    at controller/src/ifmap/ifmap_dependency_tracker.cc:165
#2 0x000000000086f9e7 in IFMapDependencyTracker::PropagateCha...

Read more...

information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/7952
Committed: http://github.org/Juniper/contrail-controller/commit/d23ad0a87e972156e2c27516609f1789cb1564ea
Submitter: Zuul
Branch: master

commit d23ad0a87e972156e2c27516609f1789cb1564ea
Author: Prakash Bailkeri <email address hidden>
Date: Mon Mar 2 02:10:06 2015 -0800

Don'tuse edge_id as IFMAPLink table key

"edge_id" is valid only when the edge is present in the graph.
In the current code, even after deleting the edge from the graph,
the edge_id(stored in DBGraphEdge) is used for compare and lookp.
This is a problem as we could be referring to freed/reused memory.

In the problem case, the edge_id gets reused for next IFMapLink/Edge that gets added to the graph.

Problem solution:
1. we don't use "edge_id" for tree compare function (i.e. edge_id is not a key for IFMapLink Table)
2. Use string (meta data + left node + right node) as key to IFMapLink table
3. Assert if "edge_id()" is invoked on deleted DBGraphEdge()

Closes-Bug: 1426175

Change-Id: I3f8d5041cf80900a43c365adc164e44acd5b5bc3

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/7951
Committed: http://github.org/Juniper/contrail-controller/commit/8faa77e791f22b43667702d7104cb9383c049dc7
Submitter: Zuul
Branch: R1.10

commit 8faa77e791f22b43667702d7104cb9383c049dc7
Author: Prakash Bailkeri <email address hidden>
Date: Mon Mar 2 02:10:06 2015 -0800

Don'tuse edge_id as IFMAPLink table key

"edge_id" is valid only when the edge is present in the graph.
In the current code, even after deleting the edge from the graph,
the edge_id(stored in DBGraphEdge) is used for compare and lookp.
This is a problem as we could be referring to freed/reused memory.

In the problem case, the edge_id gets reused for next IFMapLink/Edge that gets added to the graph.

Problem solution:
1. we don't use "edge_id" for tree compare function (i.e. edge_id is not a key for IFMapLink Table)
2. Use string (meta data + left node + right node) as key to IFMapLink table
3. Assert if "edge_id()" is invoked on deleted DBGraphEdge()

Closes-Bug: 1426175

Change-Id: I3f8d5041cf80900a43c365adc164e44acd5b5bc3

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/7949
Committed: http://github.org/Juniper/contrail-controller/commit/8999c53ca02762b999aa69b8b49b1722d69a6b95
Submitter: Zuul
Branch: R2.1

commit 8999c53ca02762b999aa69b8b49b1722d69a6b95
Author: Prakash Bailkeri <email address hidden>
Date: Mon Mar 2 02:10:06 2015 -0800

Don'tuse edge_id as IFMAPLink table key

"edge_id" is valid only when the edge is present in the graph.
In the current code, even after deleting the edge from the graph,
the edge_id(stored in DBGraphEdge) is used for compare and lookp.
This is a problem as we could be referring to freed/reused memory.

In the problem case, the edge_id gets reused for next IFMapLink/Edge that gets added to the graph.

Problem solution:
1. we don't use "edge_id" for tree compare function (i.e. edge_id is not a key for IFMapLink Table)
2. Use string (meta data + left node + right node) as key to IFMapLink table
3. Assert if "edge_id()" is invoked on deleted DBGraphEdge()

Closes-Bug: 1426175

Change-Id: I3f8d5041cf80900a43c365adc164e44acd5b5bc3

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/7950
Committed: http://github.org/Juniper/contrail-controller/commit/a9d5fb3a9aa0170a5277b20313a5287fa7d4591b
Submitter: Zuul
Branch: R2.0

commit a9d5fb3a9aa0170a5277b20313a5287fa7d4591b
Author: Prakash Bailkeri <email address hidden>
Date: Mon Mar 2 02:10:06 2015 -0800

Don'tuse edge_id as IFMAPLink table key

"edge_id" is valid only when the edge is present in the graph.
In the current code, even after deleting the edge from the graph,
the edge_id(stored in DBGraphEdge) is used for compare and lookp.
This is a problem as we could be referring to freed/reused memory.

In the problem case, the edge_id gets reused for next IFMapLink/Edge that gets added to the graph.

Problem solution:
1. we don't use "edge_id" for tree compare function (i.e. edge_id is not a key for IFMapLink Table)
2. Use string (meta data + left node + right node) as key to IFMapLink table
3. Assert if "edge_id()" is invoked on deleted DBGraphEdge()

Closes-Bug: 1426175

Change-Id: I3f8d5041cf80900a43c365adc164e44acd5b5bc3

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.