Juniper Openstack

agent core at EvpnDerivedPathData::EvpnDerivedPathData(EvpnRouteEntry const*) ()

Bug #1548265 reported by Daisuke Nakajima on 2016-02-22

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R2.20	Fix Committed	High	Prabhjot Singh Sethi	Juniper Openstack r2.23 "r2.23"
R2.21.x	Fix Committed	High	Prabhjot Singh Sethi	Juniper Openstack r2.21.2
R2.22.x	Fix Committed	High	Prabhjot Singh Sethi	Juniper Openstack r2.22.2
R3.0	Fix Committed	High	Prabhjot Singh Sethi	Juniper Openstack r3.0.2.0
Trunk	Fix Committed	High	Prabhjot Singh Sethi	Juniper Openstack r3.1.0.0-fcs

Bug Description

tor-agent core while doing POST/GET/DELETE.

#0 0x0000000000847e45 in EvpnDerivedPathData::EvpnDerivedPathData(EvpnRouteEntry const*) ()
(gdb) t
[Current thread is 1 (Thread 0x7f74ffbfe700 (LWP 914))]
(gdb) bt
#0 0x0000000000847e45 in EvpnDerivedPathData::EvpnDerivedPathData(EvpnRouteEntry const*) ()
#1 0x000000000085aedf in BridgeAgentRouteTable::AddBridgeRoute(AgentRoute const*) ()
#2 0x000000000084f366 in AgentRouteTable::DeletePathFromPeer(DBTablePartBase*, AgentRoute*, AgentPath*)
()
#3 0x000000000084f5ed in AgentRouteTable::DeleteAllBgpPath(DBTablePartBase*, DBEntryBase*) ()
#4 0x0000000000d1d445 in DBTableWalker::Worker::Run() ()
#5 0x0000000000e15df0 in TaskImpl::execute() ()
#6 0x00007f75158ddb3a in ?? () from /usr/lib/libtbb.so.2
#7 0x00007f75158d9816 in ?? () from /usr/lib/libtbb.so.2
#8 0x00007f75158d8f4b in ?? () from /usr/lib/libtbb.so.2
#9 0x00007f75158d50ff in ?? () from /usr/lib/libtbb.so.2
#10 0x00007f75158d52f9 in ?? () from /usr/lib/libtbb.so.2
#11 0x00007f7515af9182 in start_thread (arg=0x7f74ffbfe700) at pthread_create.c:312
#12 0x00007f7514dd247d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

root@openc-15:/var/crashes# Package Version ---------------------15:/var/crashes# Package Version ---------------------fabric-utils contrail-install-packages contrail-lib 2.21.1-22 contrail-nodemgr 2.21.1-22 contrail-nova-vif 2.21.1-22 contrail-setup 2.21.1-22 contrail-utils 2.21.1-22 contrail-vrouter-3.13.0-40-generic contrail-vrouter-agent contrail-vrouter-common contrail-vrouter-init contrail-vrouter-utils python-contrail 2.21.1-22 python-contrail-vrouter-api python-neutronclient python-nova python-opencontrail-vrouter-netns contrail-version -d
Build-ID | Repo | Package Name
/>----------------- ------------------------------ ----------------------------------
contrail-version
Build-ID | Repo | Package Name
/>----------------- ------------------------------ ----------------------------------
2.21.1-22 22
2.21.1-22~juno 22
22
22
22
22
22
2.21.1-22 22
2.21.1-22 22
2.21.1-22 22
2.21.1-22 22
2.21.1-22 22
22
2.21.1-22 22
1:2.3.8-0ubuntu1~cloud0.2contrail22
1:2014.2.3-0ubuntu1~cloud0.3contrail22
2.21.1-22 22

Tags:

Revision history for this message

Daisuke Nakajima (dnakajima) wrote on 2016-02-22:

369_20160222_tor-core.zip Edit (4.2 KiB, application/zip)

Revision history for this message

Daisuke Nakajima (dnakajima) wrote on 2016-02-22:

core.contrail-tor-ag.2870.openc-15.1456134499.gz Edit (158.6 MiB, application/octet-stream)

Nagabhushana R (bhushana) on 2016-02-22

tags:	added: customer vrouter
tags:	added: blocker

Hari Prasad Killi (haripk) on 2016-03-14

Changed in juniperopenstack:
assignee:	nobody → Manish Singh (manishs)
importance:	Undecided → High

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-03: [Review update] R3.0

Review in progress for https://review.opencontrail.org/19849
Submitter: Manish Singh (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-04: A change has been merged

Reviewed: https://review.opencontrail.org/19849
Committed: http://github.org/Juniper/contrail-controller/commit/629349ba4ac335bf1bef35184051ee051da21fa7
Submitter: Zuul
Branch: R3.0

commit 629349ba4ac335bf1bef35184051ee051da21fa7
Author: Manish <email address hidden>
Date: Tue May 3 16:42:37 2016 +0530

Validation checks and diagnostics.

For following:
- If MPLS label is changing NH, then make sure non MCAST label does not point to
Composite NH of type L2 Comp.
- Temporarily store peer name in path.

Change-Id: I456ef6875e34f074a105f67e8e2866993e0ea30d
Partial-bug: #1548265

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-05: [Review update] master

Review in progress for https://review.opencontrail.org/19923
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-05: [Review update] R3.0

Review in progress for https://review.opencontrail.org/19924
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-05: [Review update] R2.20

#10

Review in progress for https://review.opencontrail.org/19927
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-05: [Review update] R2.21.x

#12

Review in progress for https://review.opencontrail.org/19928
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-05: [Review update] R2.22.x

#14

Review in progress for https://review.opencontrail.org/19929
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-06: [Review update] master

#16

Review in progress for https://review.opencontrail.org/19923
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-06: [Review update] R3.0

#18

Review in progress for https://review.opencontrail.org/19924
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-06: [Review update] R2.20

#20

Review in progress for https://review.opencontrail.org/19927
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-06: [Review update] R2.21.x

#22

Review in progress for https://review.opencontrail.org/19928
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-06: [Review update] R2.22.x

#24

Review in progress for https://review.opencontrail.org/19929
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-06: A change has been merged

#26

Reviewed: https://review.opencontrail.org/19927
Committed: http://github.org/Juniper/contrail-controller/commit/0dd26da84e571392c37fd7fc7b0f05a8533c2d7d
Submitter: Zuul
Branch: R2.20

commit 0dd26da84e571392c37fd7fc7b0f05a8533c2d7d
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu May 5 14:18:18 2016 +0530

Assure Paths deleted before Peer

Issue:
------
In a scenario, where VRF delete is triggered followed by
BGP peer down, we see that some of the routes have Paths
pending from a deleted BGP Peer. which results in
segmentation fault and some times VRF delete timeouts

Issues happens because of two parallel vrf/route table
walks, with an assumptions that a walk in deleted VRF has
been started to clean up all the BGP paths will go fine

However if the BGP peer goes down in between it starts
a parallel walk for DELPEER and in this walk it skips
deleted VRF entries assuming the other walk will delete
paths from routes in deleted VRF

Now if DELPEER walk completes before the walk on deleted
VRF PEER pointer is free'd, further refering this free'd
memory doesnot assure the deletes of associated paths from
route, which will stay there till VRF delete timeout

Fix:
----
Don't skip deleted VRF in DELPEER walk to assure the
paths added for the current BGP PEER are deleted before
the PEER pointer is free'd

Conflicts:
src/vnsw/agent/test/test_l2route.cc

Closes-Bug: 1548265
Change-Id: I5a7ec0a62fd1de6101a117648d970ecc8949ea37
(cherry picked from commit 5a931126d165d1468aca781868a8da50d5b2e5d6)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-07:

#27

Reviewed: https://review.opencontrail.org/19924
Committed: http://github.org/Juniper/contrail-controller/commit/5b0b979412bc5da9e1434cdb5ea950113e7c5b4a
Submitter: Zuul
Branch: R3.0

commit 5b0b979412bc5da9e1434cdb5ea950113e7c5b4a
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu May 5 14:18:18 2016 +0530

Assure Paths deleted before Peer

Issues happens because of two parallel vrf/route table
walks, with an assumptions that a walk in deleted VRF has
been started to clean up all the BGP paths will go fine

However if the BGP peer goes down in between it starts
a parallel walk for DELPEER and in this walk it skips
deleted VRF entries assuming the other walk will delete
paths from routes in deleted VRF

Fix:
----
Don't skip deleted VRF in DELPEER walk to assure the
paths added for the current BGP PEER are deleted before
the PEER pointer is free'd

Closes-Bug: 1548265
Change-Id: I5a7ec0a62fd1de6101a117648d970ecc8949ea37
(cherry picked from commit 5a931126d165d1468aca781868a8da50d5b2e5d6)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-07:

#28

Reviewed: https://review.opencontrail.org/19923
Committed: http://github.org/Juniper/contrail-controller/commit/5a931126d165d1468aca781868a8da50d5b2e5d6
Submitter: Zuul
Branch: master

commit 5a931126d165d1468aca781868a8da50d5b2e5d6
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu May 5 14:18:18 2016 +0530

Assure Paths deleted before Peer

Issues happens because of two parallel vrf/route table
walks, with an assumptions that a walk in deleted VRF has
been started to clean up all the BGP paths will go fine

However if the BGP peer goes down in between it starts
a parallel walk for DELPEER and in this walk it skips
deleted VRF entries assuming the other walk will delete
paths from routes in deleted VRF

Fix:
----
Don't skip deleted VRF in DELPEER walk to assure the
paths added for the current BGP PEER are deleted before
the PEER pointer is free'd

Closes-Bug: 1548265
Change-Id: I5a7ec0a62fd1de6101a117648d970ecc8949ea37

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-09:

#29

Reviewed: https://review.opencontrail.org/19928
Committed: http://github.org/Juniper/contrail-controller/commit/3f63d9d6c8040b024f433edf64e34d0a3f08aec4
Submitter: Zuul
Branch: R2.21.x

commit 3f63d9d6c8040b024f433edf64e34d0a3f08aec4
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu May 5 14:18:18 2016 +0530

Assure Paths deleted before Peer

Issues happens because of two parallel vrf/route table
walks, with an assumptions that a walk in deleted VRF has
been started to clean up all the BGP paths will go fine

However if the BGP peer goes down in between it starts
a parallel walk for DELPEER and in this walk it skips
deleted VRF entries assuming the other walk will delete
paths from routes in deleted VRF

Fix:
----
Don't skip deleted VRF in DELPEER walk to assure the
paths added for the current BGP PEER are deleted before
the PEER pointer is free'd

Conflicts:
src/vnsw/agent/test/test_l2route.cc

Closes-Bug: 1548265
Change-Id: I5a7ec0a62fd1de6101a117648d970ecc8949ea37
(cherry picked from commit 5a931126d165d1468aca781868a8da50d5b2e5d6)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-09:

#30

Reviewed: https://review.opencontrail.org/19929
Committed: http://github.org/Juniper/contrail-controller/commit/fde165a726e2e2a615cb2331ea33a86e665f1ad2
Submitter: Zuul
Branch: R2.22.x

commit fde165a726e2e2a615cb2331ea33a86e665f1ad2
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu May 5 14:18:18 2016 +0530

Assure Paths deleted before Peer

Issues happens because of two parallel vrf/route table
walks, with an assumptions that a walk in deleted VRF has
been started to clean up all the BGP paths will go fine

However if the BGP peer goes down in between it starts
a parallel walk for DELPEER and in this walk it skips
deleted VRF entries assuming the other walk will delete
paths from routes in deleted VRF

Fix:
----
Don't skip deleted VRF in DELPEER walk to assure the
paths added for the current BGP PEER are deleted before
the PEER pointer is free'd

Conflicts:
src/vnsw/agent/test/test_l2route.cc

Closes-Bug: 1548265
Change-Id: I5a7ec0a62fd1de6101a117648d970ecc8949ea37
(cherry picked from commit 5a931126d165d1468aca781868a8da50d5b2e5d6)

Ashish Ranjan (aranjan-n) on 2016-05-16

information type:

Proprietary → Public

OpenContrail Admin (ci-admin-f) on 2016-05-16

information type:

Public → Private

Nagabhushana R (bhushana) on 2016-05-18

information type:

Private → Public

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.