Juniper Openstack

contrail-control crash in BgpServer::RemoveStaticRouteMgr

Bug #1547181 reported by vageesan
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
Critical
Prakash Bailkeri
Juniper Openstack r3.0-fcs
Trunk
Fix Committed
Critical
Prakash Bailkeri
Juniper Openstack r3.0-fcs

Bug Description

contrail-control crash with following backtrace in solution test run.

mainline 3.0 , 2713 kilo.

core is in 10.84.5.112:/cs-shared/bugs/<bug-id>/

#1 0x000000000084d971 in _M_erase_aux (__position=..., this=0x1a47238)
    at /usr/include/c++/4.8/bits/stl_tree.h:1745
#2 erase (__position=..., this=0x1a47238) at /usr/include/c++/4.8/bits/stl_tree.h:830
#3 _M_erase_aux (__last=..., __first=..., this=0x1a47238)
    at /usr/include/c++/4.8/bits/stl_tree.h:1760
#4 erase (__last=..., __first=..., this=0x1a47238) at /usr/include/c++/4.8/bits/stl_tree.h:848
#5 erase (__x=<synthetic pointer>, this=0x1a47238) at /usr/include/c++/4.8/bits/stl_tree.h:1771
#6 erase (__x=<synthetic pointer>, this=0x1a47238) at /usr/include/c++/4.8/bits/stl_set.h:582
#7 BgpServer::RemoveStaticRouteMgr (this=0x1a47160, srt_manager=srt_manager@entry=0x7f8aa40357c0)
    at controller/src/bgp/bgp_server.cc:643
#8 0x0000000000636c93 in StaticRouteMgr<StaticRouteInet>::StaticRouteEventCallback (
    this=0x7f8aa40357c0, req=0x7f8b082e56f0)
    at controller/src/bgp/routing-instance/static_route.cc:633
#9 0x000000000063a7af in operator() (a0=0x7f8b082e56f0, this=0x7f8b295e4b30)
    at /usr/include/boost/function/function_template.hpp:767
#10 RunQueue (this=0x7f8b0837d780) at controller/src/base/queue_task.h:87
#11 QueueTaskRunner<StaticRouteRequest*, WorkQueue<StaticRouteRequest*> >::Run (this=0x7f8b0837d780)
    at controller/src/base/queue_task.h:66
#12 0x0000000000c3b68c in TaskImpl::execute (this=0x7f8b2acf3940) at controller/src/base/task.cc:253
#13 0x00007f8b32346b3a in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f8b32342816 in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f8b32341f4b in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f8b3233e0ff in ?? () from /usr/lib/libtbb.so.2
#17 0x00007f8b3233e2f9 in ?? () from /usr/lib/libtbb.so.2
---Type <return> to continue, or q <return> to quit---q

Revision history for this message
Prakash Bailkeri (prakashmb) wrote :

Fix is committed to master and R3.0
Reviewed: https://review.opencontrail.org/17750
Committed: http://github.org/Juniper/contrail-controller/commit/29a78345b9e40ddbe480a528fbbb16c5f1f29ea2
Submitter: Zuul
Branch: master

commit 29a78345b9e40ddbe480a528fbbb16c5f1f29ea2
Author: Prakash Bailkeri <email address hidden>
Date: Wed Feb 24 22:47:27 2016 +0530

Concurrency issue in unregister of static route

There are two issues in delete of static route config from routing instance
1. bgp:StaticRoute task run with instance id of routing instance index. So it
is possible to have multiple bgp:StaticRoute task concurrently. Due to this
RemoveStaticRouteMgr called on BgpServer object corrupts the srt_manager_list_

2. UnregisterMatchCondition is called from bgp:StaticRoute task. Due to multiple
bgp:StaticRoute task running in parallel, BgpConditionListener::UnregisterMatchCondition
corrupts the condition listener datastructure.(TableMap map_).

Fix is to handle unregister of static route from bgp:Config task using task trigger

Added UT to verify static route config with multiple routing instance

Change-Id: I44446dc558b3d7a8440c7e9802d97a8ea61b8caf
Closes-bug: #1547184,1547181

Reviewed: https://review.opencontrail.org/17758
Committed: http://github.org/Juniper/contrail-controller/commit/3a748abd2fd7eb25470ccd9915a30422c2a18368
Submitter: Zuul
Branch: R3.0

commit 3a748abd2fd7eb25470ccd9915a30422c2a18368
Author: Prakash Bailkeri <email address hidden>
Date: Wed Feb 24 22:47:27 2016 +0530

Concurrency issue in unregister of static route

There are two issues in delete of static route config from routing instance
1. bgp:StaticRoute task run with instance id of routing instance index. So it
is possible to have multiple bgp:StaticRoute task concurrently. Due to this
RemoveStaticRouteMgr called on BgpServer object corrupts the srt_manager_list_

2. UnregisterMatchCondition is called from bgp:StaticRoute task. Due to multiple
bgp:StaticRoute task running in parallel, BgpConditionListener::UnregisterMatchCondition
corrupts the condition listener datastructure.(TableMap map_).

Fix is to handle unregister of static route from bgp:Config task using task trigger

Added UT to verify static route config with multiple routing instance

Change-Id: I44446dc558b3d7a8440c7e9802d97a8ea61b8caf
Closes-bug: #1547184,1547181
(cherry picked from commit 29a78345b9e40ddbe480a528fbbb16c5f1f29ea2)

Prakash Bailkeri (prakashmb)
information type: Proprietary → Public
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.