Juniper Openstack

agent asserts @ vn->GetName()) == 0

Bug #1596792 reported by Senthilnathan Murugappan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
High
Ashok Singh
Juniper Openstack r2.23 "r2.23"
R2.21.x
Fix Committed
High
Ashok Singh
Juniper Openstack r2.21.3
R2.22.x
Fix Committed
High
Ashok Singh
Juniper Openstack r2.22.3
R3.0
Fix Committed
High
Ashok Singh
Juniper Openstack r3.0.3.0
R3.0.2.x
Fix Committed
High
Ashok Singh
Juniper Openstack r3.0.2.1
Trunk
Fix Committed
High
Ashok Singh
Juniper Openstack r3.1.0.0-fcs

Bug Description

We have 40K VMIs spread over 100 agents and each VMI has two IPs, each from different VNs.
In total, 80K VNs, 40K VMIs across 100 agents.

At this scale when the control process was restarted (both the xmpp peers are brought down) we observed agent cores on many of the agent and after a while things settles down and the cluster becomes stable.

Below is the bt for the same and the core will be copied to /auto/cores/<bugid>

2016-06-27 21:38:32:453061: [root@10.84.25.20] out: [Thread debugging using libthread_db enabled]
2016-06-27 21:38:32:754443: [root@10.84.25.20] out: Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
2016-06-27 21:38:32:754579: [root@10.84.25.20] out: Core was generated by `/usr/bin/contrail-vrouter-agent'.
2016-06-27 21:38:33:118962: [root@10.84.25.20] out: Program terminated with signal SIGABRT, Aborted.
2016-06-27 21:38:33:119100: [root@10.84.25.20] out: #0 0x00007f3011a39cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
2016-06-27 21:38:33:150948: [root@10.84.25.20] out: 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
2016-06-27 21:38:33:151086: [root@10.84.25.20] out: #0 0x00007f3011a39cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
2016-06-27 21:38:33:151195: [root@10.84.25.20] out: #1 0x00007f3011a3d0d8 in __GI_abort () at abort.c:89
2016-06-27 21:38:33:151283: [root@10.84.25.20] out: #2 0x00007f3011a32b86 in __assert_fail_base (fmt=0x7f3011b83830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n
%n", assertion=assertion@entry=0x127eeb0 "state->vn_name_.compare(vn->GetName()) == 0", file=file@entry=0x127ee78 "controller/src/vnsw/agent/uve/vn_uve_table
_base.cc", line=line@entry=289, function=function@entry=0x127eee0 <VnUveTableBase::InterfaceNotify(DBTablePartBase*, DBEntryBase*)::__PRETTY_FUNCTION__> "void VnUveTableBase::InterfaceNotify(DBTablePartBase*, DBEntryBase*)") at assert.c:92
2016-06-27 21:38:33:167045: [root@10.84.25.20] out: #3 0x00007f3011a32c32 in __GI___assert_fail (assertion=0x127eeb0 "state->vn_name_.compare(vn->GetName())
 == 0", file=0x127ee78 "controller/src/vnsw/agent/uve/vn_uve_table_base.cc", line=289, function=0x127eee0 <VnUveTableBase::InterfaceNotify(DBTablePartBase*, DBEntryBase*)::__PRETTY_FUNCTION__> "void VnUveTableBase::InterfaceNotify(DBTablePartBase*, DBEntryBase*)") at assert.c:101
2016-06-27 21:38:33:599229: [root@10.84.25.20] out: #4 0x0000000000e43c37 in VnUveTableBase::InterfaceNotify (this=0x7f2ffc003640, partition=0x7f2ffc524c20, e=0x7f2fd4a96928) at controller/src/vnsw/agent/uve/vn_uve_table_base.cc:289
2016-06-27 21:38:33:799814: [root@10.84.25.20] out: #5 0x00000000010a767a in operator() (a1=0x7f2fd4a96928, a0=0x7f2ffc524c20, this=0x7f30094b8aa0) at /usr/include/boost/function/function_template.hpp:767
2016-06-27 21:38:33:799954: [root@10.84.25.20] out: #6 RunNotify (entry=0x7f2fd4a96928, tpart=0x7f2ffc524c20, this=0x7f2ffc51ad40) at controller/src/db/db_table.cc:114
2016-06-27 21:38:33:800063: [root@10.84.25.20] out: #7 DBTableBase::RunNotify (this=<optimized out>, tpart=tpart@entry=0x7f2ffc524c20, entry=entry@entry=0x7f2fd4a96928) at controller/src/db/db_table.cc:204
2016-06-27 21:38:33:864033: [root@10.84.25.20] out: #8 0x00000000010a9d68 in DBTablePartBase::RunNotify (this=this@entry=0x7f2ffc524c20) at controller/src/db/db_table_partition.cc:47
2016-06-27 21:38:33:895896: [root@10.84.25.20] out: #9 0x00000000010a5e70 in DBPartition::QueueRunner::Run (this=0x7f2ff92de490) at controller/src/db/db_partition.cc:209
2016-06-27 21:38:34:060009: [root@10.84.25.20] out: #10 0x00000000011b886f in TaskImpl::execute (this=0x7f300b279f40) at controller/src/base/task.cc:262
2016-06-27 21:38:34:060146: [root@10.84.25.20] out: #11 0x00007f3012608b3a in ?? () from /usr/lib/libtbb.so.2
2016-06-27 21:38:34:060251: [root@10.84.25.20] out: #12 0x00007f3012604816 in ?? () from /usr/lib/libtbb.so.2
2016-06-27 21:38:34:060337: [root@10.84.25.20] out: #13 0x00007f3012603f4b in ?? () from /usr/lib/libtbb.so.2
2016-06-27 21:38:34:061749: [root@10.84.25.20] out: #14 0x00007f30126000ff in ?? () from /usr/lib/libtbb.so.2
2016-06-27 21:38:34:065267: [root@10.84.25.20] out: #15 0x00007f30126002f9 in ?? () from /usr/lib/libtbb.so.2
2016-06-27 21:38:34:065604: [root@10.84.25.20] out: #16 0x00007f3012824182 in start_thread (arg=0x7f30094b9700) at pthread_create.c:312
2016-06-27 21:38:34:072979: [root@10.84.25.20] out: #17 0x00007f3011afd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
2016-06-27 21:38:34:073131: [root@10.84.25.20] out:

Senthilnathan Murugappan (msenthil)
tags: added: quench2
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/21749
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/21752
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0.2.x

Review in progress for https://review.opencontrail.org/21753
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/21749
Committed: http://github.org/Juniper/contrail-controller/commit/79986fdf34e61f0e0fd82c5a19810450f98e985f
Submitter: Zuul
Branch: master

commit 79986fdf34e61f0e0fd82c5a19810450f98e985f
Author: ashoksingh <email address hidden>
Date: Fri Jul 8 11:57:56 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Change-Id: I7679612fd1d663ffce452a8f9c21bd4aede66d03
Closes-Bug: #1596792

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/21752
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/21794
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/21795
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/21796
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/21753
Committed: http://github.org/Juniper/contrail-controller/commit/a6f03e5250c75f21fa2e44000d40b122efffbd14
Submitter: Zuul
Branch: R3.0.2.x

commit a6f03e5250c75f21fa2e44000d40b122efffbd14
Author: ashoksingh <email address hidden>
Date: Fri Jul 8 11:57:56 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Closes-Bug: #1596792
(cherry picked from commit 79986fdf34e61f0e0fd82c5a19810450f98e985f)

Change-Id: I2d8b28d166a53da9c166392eafb1ee8407e8ecc4

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/21752
Committed: http://github.org/Juniper/contrail-controller/commit/e3f93bc9e69f9db6f6a258cd4b6e8b3e3d2bfbee
Submitter: Zuul
Branch: R3.0

commit e3f93bc9e69f9db6f6a258cd4b6e8b3e3d2bfbee
Author: ashoksingh <email address hidden>
Date: Fri Jul 8 11:57:56 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Closes-Bug: #1596792
(cherry picked from commit 79986fdf34e61f0e0fd82c5a19810450f98e985f)

Change-Id: I56d564b851f930ce6bb61bfd8639d8061b045ef2

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/21796
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/21794
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/21794
Committed: http://github.org/Juniper/contrail-controller/commit/f9eca51ca02ac7bbea1e592aa195f265e8fe7937
Submitter: Zuul
Branch: R2.22.x

commit f9eca51ca02ac7bbea1e592aa195f265e8fe7937
Author: ashoksingh <email address hidden>
Date: Mon Jul 11 10:54:12 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Change-Id: Ia3c11ad71558a6e9e215bbb6817ccd9f65253a5d
Closes-Bug: #1596792

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/21796
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/21795
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/21796
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/21796
Committed: http://github.org/Juniper/contrail-controller/commit/2656b8ca109dc468545a5a74300714affaaed435
Submitter: Zuul
Branch: R2.20

commit 2656b8ca109dc468545a5a74300714affaaed435
Author: ashoksingh <email address hidden>
Date: Mon Jul 11 10:54:12 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Closes-Bug: #1596792
(cherry picked from commit 64ef02488ed11d637a46f5b8c51664838cf2b95c)

Change-Id: I037631d09af36e7d191cc0f45a129a99d3c800ea

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/23434
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/21795
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/21795
Committed: http://github.org/Juniper/contrail-controller/commit/303fed3880d38a987ab11117fba35eb97ae148e4
Submitter: Zuul
Branch: R2.21.x

commit 303fed3880d38a987ab11117fba35eb97ae148e4
Author: ashoksingh <email address hidden>
Date: Mon Jul 11 10:54:12 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Closes-Bug: #1596792
(cherry picked from commit 64ef02488ed11d637a46f5b8c51664838cf2b95c)

Change-Id: I68cbaaf067523abb62ce8da49e6d2418080b9a1f

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23434
Committed: http://github.org/Juniper/contrail-controller/commit/0f05b970259c948d4081fc020b38571606e6f8f0
Submitter: Zuul
Branch: R2.22.x

commit 0f05b970259c948d4081fc020b38571606e6f8f0
Author: ashoksingh <email address hidden>
Date: Fri Aug 19 09:06:01 2016 +0530

Fix compilation failure in test_vn_uve

Change-Id: I03dacaa89ee70a76e63731be8c354267bfda63cd
Closes-Bug: #1596792

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.