few overflow entries are not cleared on fast tcp setup/teardown

Bug #1614382 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
High
Divakar Dharanalakota
R2.22.x
Fix Committed
High
Divakar Dharanalakota
R3.0
Fix Committed
High
Divakar Dharanalakota
R3.0.2.x
Fix Committed
High
Divakar Dharanalakota
R3.1
Fix Committed
High
Divakar Dharanalakota
Trunk
Fix Committed
High
Divakar Dharanalakota

Bug Description

R3.1 Kilo

On a cluster with a lot of fast tcp setup/teardown flows, few overflow entries were stuck.
Divakar is aware of the issue

  Other settings
    NIC cksum offload for outer UDP hdr 0
    Flow hold limit: 8192
    MPLS over UDP globally 0
    Used Flow entries 20
    Used Over Flow entries 6
---------
4302404
     Oflow entries:
         4280093 4454942

4408058
     Oflow entries:
         4279839

Tags: vrouter
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/23407
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/23407
Committed: http://github.org/Juniper/contrail-vrouter/commit/d65e4b98489da0cec9f7a109a9c1ff8341ab81fb
Submitter: Zuul
Branch: master

commit d65e4b98489da0cec9f7a109a9c1ff8341ab81fb
Author: Divakar <email address hidden>
Date: Thu Aug 18 11:17:21 2016 +0530

Hash table fixes

1) Currently the number of delete marked hash entries in Hash bucket is used
for triggering the work item for the deletion. Work item is triggered
when the first entry is delete marked. But due to contention in handling
"number of delete marked entries", the next work item is never
triggered.
As a fix, a new variable is used per hash bucket to trigger the work
item.

2) When Agent gives soft reset, Vrouter marks vr_not_ready. Once
vr_not_ready is set, Work items and RCU callbacks should not be
registered any more. In hash table deletion process, once this is set,
hash bucket is directly manipulated without work item. This results in
issues, if any pending work item is currently being executed. If work
item and hash entry deletion (vr_htable_release_hentry()) access the
same hash bucket, the hash bucket list goes bad.

As a fix, a new API vr_htable_reset() is introduced.
vr_htable_release_hentry() does not manipulate the hash entry if
vr_not_ready is setup. Modules are expected to call vr_htable_reset() to
reset the hash table. Call backs are invoked to act on the reset hash
entries.

Change-Id: I1b153d5e5579d933481b7574cd9f2e8b2410b335
closes-bug: #1614382

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/23851
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/23853
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0.2.x

Review in progress for https://review.opencontrail.org/23854
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/23855
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/23857
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/23855
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/23853
Committed: http://github.org/Juniper/contrail-vrouter/commit/93ec45929ead2a74cd24bcc97ec14f15afb06dda
Submitter: Zuul
Branch: R3.0

commit 93ec45929ead2a74cd24bcc97ec14f15afb06dda
Author: Divakar <email address hidden>
Date: Thu Aug 18 11:17:21 2016 +0530

Hash table fixes

1) Currently the number of delete marked hash entries in Hash bucket is used
for triggering the work item for the deletion. Work item is triggered
when the first entry is delete marked. But due to contention in handling
"number of delete marked entries", the next work item is never
triggered.
As a fix, a new variable is used per hash bucket to trigger the work
item.

2) When Agent gives soft reset, Vrouter marks vr_not_ready. Once
vr_not_ready is set, Work items and RCU callbacks should not be
registered any more. In hash table deletion process, once this is set,
hash bucket is directly manipulated without work item. This results in
issues, if any pending work item is currently being executed. If work
item and hash entry deletion (vr_htable_release_hentry()) access the
same hash bucket, the hash bucket list goes bad.

As a fix, a new API vr_htable_reset() is introduced.
vr_htable_release_hentry() does not manipulate the hash entry if
vr_not_ready is setup. Modules are expected to call vr_htable_reset() to
reset the hash table. Call backs are invoked to act on the reset hash
entries.

Change-Id: I9edc2c9e7718e3e0eae74a29fd4faaab8c3055b5
closes-bug: #1614382

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23851
Committed: http://github.org/Juniper/contrail-vrouter/commit/da3c6846a905619bfe08c44521dfade1e1c958d3
Submitter: Zuul
Branch: R3.1

commit da3c6846a905619bfe08c44521dfade1e1c958d3
Author: Divakar <email address hidden>
Date: Thu Aug 18 11:17:21 2016 +0530

Hash table fixes

1) Currently the number of delete marked hash entries in Hash bucket is used
for triggering the work item for the deletion. Work item is triggered
when the first entry is delete marked. But due to contention in handling
"number of delete marked entries", the next work item is never
triggered.
As a fix, a new variable is used per hash bucket to trigger the work
item.

2) When Agent gives soft reset, Vrouter marks vr_not_ready. Once
vr_not_ready is set, Work items and RCU callbacks should not be
registered any more. In hash table deletion process, once this is set,
hash bucket is directly manipulated without work item. This results in
issues, if any pending work item is currently being executed. If work
item and hash entry deletion (vr_htable_release_hentry()) access the
same hash bucket, the hash bucket list goes bad.

As a fix, a new API vr_htable_reset() is introduced.
vr_htable_release_hentry() does not manipulate the hash entry if
vr_not_ready is setup. Modules are expected to call vr_htable_reset() to
reset the hash table. Call backs are invoked to act on the reset hash
entries.

Change-Id: I1b153d5e5579d933481b7574cd9f2e8b2410b335
closes-bug: #1614382

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23854
Committed: http://github.org/Juniper/contrail-vrouter/commit/29cf0d18b2a55f6a663bc064dd4d5b92f6e6e1c0
Submitter: Zuul
Branch: R3.0.2.x

commit 29cf0d18b2a55f6a663bc064dd4d5b92f6e6e1c0
Author: Divakar <email address hidden>
Date: Thu Aug 18 11:17:21 2016 +0530

Hash table fixes

1) Currently the number of delete marked hash entries in Hash bucket is used
for triggering the work item for the deletion. Work item is triggered
when the first entry is delete marked. But due to contention in handling
"number of delete marked entries", the next work item is never
triggered.
As a fix, a new variable is used per hash bucket to trigger the work
item.

2) When Agent gives soft reset, Vrouter marks vr_not_ready. Once
vr_not_ready is set, Work items and RCU callbacks should not be
registered any more. In hash table deletion process, once this is set,
hash bucket is directly manipulated without work item. This results in
issues, if any pending work item is currently being executed. If work
item and hash entry deletion (vr_htable_release_hentry()) access the
same hash bucket, the hash bucket list goes bad.

As a fix, a new API vr_htable_reset() is introduced.
vr_htable_release_hentry() does not manipulate the hash entry if
vr_not_ready is setup. Modules are expected to call vr_htable_reset() to
reset the hash table. Call backs are invoked to act on the reset hash
entries.

Change-Id: I1b0865ad3cc9091cd682e1eb857e51b5538483ab
closes-bug: #1614382

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23857
Committed: http://github.org/Juniper/contrail-vrouter/commit/aac3f8a16519af9d6050cbea2938c8acbd757836
Submitter: Zuul
Branch: R2.20

commit aac3f8a16519af9d6050cbea2938c8acbd757836
Author: Divakar <email address hidden>
Date: Thu Aug 18 11:17:21 2016 +0530

Hash table fixes

1) Currently the number of delete marked hash entries in Hash bucket is used
for triggering the work item for the deletion. Work item is triggered
when the first entry is delete marked. But due to contention in handling
"number of delete marked entries", the next work item is never
triggered.
As a fix, a new variable is used per hash bucket to trigger the work
item.

2) When Agent gives soft reset, Vrouter marks vr_not_ready. Once
vr_not_ready is set, Work items and RCU callbacks should not be
registered any more. In hash table deletion process, once this is set,
hash bucket is directly manipulated without work item. This results in
issues, if any pending work item is currently being executed. If work
item and hash entry deletion (vr_htable_release_hentry()) access the
same hash bucket, the hash bucket list goes bad.

As a fix, a new API vr_htable_reset() is introduced.
vr_htable_release_hentry() does not manipulate the hash entry if
vr_not_ready is setup. Modules are expected to call vr_htable_reset() to
reset the hash table. Call backs are invoked to act on the reset hash
entries.

Change-Id: Ie70f452260f7e6e44404c1ee74fe69e5652f9000
closes-bug: #1614382

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23855
Committed: http://github.org/Juniper/contrail-vrouter/commit/7e4a8ccd3887658bb377342a645d3c9b9ce843ef
Submitter: Zuul
Branch: R2.22.x

commit 7e4a8ccd3887658bb377342a645d3c9b9ce843ef
Author: Divakar <email address hidden>
Date: Thu Aug 18 11:17:21 2016 +0530

Hash table fixes

1) Currently the number of delete marked hash entries in Hash bucket is used
for triggering the work item for the deletion. Work item is triggered
when the first entry is delete marked. But due to contention in handling
"number of delete marked entries", the next work item is never
triggered.
As a fix, a new variable is used per hash bucket to trigger the work
item.

2) When Agent gives soft reset, Vrouter marks vr_not_ready. Once
vr_not_ready is set, Work items and RCU callbacks should not be
registered any more. In hash table deletion process, once this is set,
hash bucket is directly manipulated without work item. This results in
issues, if any pending work item is currently being executed. If work
item and hash entry deletion (vr_htable_release_hentry()) access the
same hash bucket, the hash bucket list goes bad.

As a fix, a new API vr_htable_reset() is introduced.
vr_htable_release_hentry() does not manipulate the hash entry if
vr_not_ready is setup. Modules are expected to call vr_htable_reset() to
reset the hash table. Call backs are invoked to act on the reset hash
entries.

closes-bug: #1614382

Change-Id: I576fc4de08242aebacf714ffb5ae1d37b5f93b9f

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.