Juniper Openstack

All TCP connections are dropped after 10 mins of sustained 25k flows/sec

Bug #1584224 reported by amit surana on 2016-05-20

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R3.0	Fix Committed	High	RAVI KIRAN	Juniper Openstack r3.0.2.0
Trunk	Fix Committed	High	RAVI KIRAN

Bug Description

After few minutes (~10) of sustained 25-30k flows/sec, it is seen that vRouter drops most incoming TCP SYNs and flow setup rate starts to go down. After few more minutes, flow table is fully allocated, packets are dropped (flow unusable error, flow table full errors).

If the max flow setup rate is say 20k flows/sec, then vRouter should be able to maintain a steady plateau at 20k, irrespective of the incoming rate. There needs to be some form of rate limiting that can keep the system operating gracefully. What happens currently is that performance starts to degrade and eventually all traffic can be starved out (by filling the flow table)- as such leaving the system vulnerable to a DoS attack.

Tags:

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-27: [Review update] R3.0

Review in progress for https://review.opencontrail.org/20705
Submitter: Praveen K V (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-27:

Review in progress for https://review.opencontrail.org/20706
Submitter: Praveen K V (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-27:

Review in progress for https://review.opencontrail.org/20707
Submitter: Praveen K V (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-28: A change has been merged

Reviewed: https://review.opencontrail.org/20705
Committed: http://github.org/Juniper/contrail-controller/commit/97069a614b11b39ebc84b60637ebe68a9739bc38
Submitter: Zuul
Branch: R3.0

commit 97069a614b11b39ebc84b60637ebe68a9739bc38
Author: Praveen K V <email address hidden>
Date: Tue May 24 22:00:29 2016 +0530

Introspect changes to debug flow queue performances

- Added knobs to show "busy-time" for a queue
- Added flow-misc-event queue. This queue is used in subsequent comits

Change-Id: I879ef9e0afe0d13c59fef9b42895fcb0b88de022
Partial-Bug: #1584210
Partial-Bug: #1584224

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-28:

Reviewed: https://review.opencontrail.org/20706
Committed: http://github.org/Juniper/contrail-controller/commit/43c1c764ef32cbe40aae059026d12ab315fedc9f
Submitter: Zuul
Branch: R3.0

commit 43c1c764ef32cbe40aae059026d12ab315fedc9f
Author: Praveen K V <email address hidden>
Date: Fri May 27 11:00:00 2016 +0530

Flow mgmt optimization

- State compress flow-mgmt-request generated from flow-table.
Flow-entry stores flow-mgmt-request pointer as a state to specify that
a request is enqueued. Subsequent enqueues are compressed.
- Store flow-mgmt state per-flow in flow-entry
Per flow-information was stored in a tree earlier. In case of scaled
setups, the depth of tree can increase significantly. Store the
per-flow information into flow-entry itself to avoid the lookups

Change-Id: Ifbfe539682b560061683d2269b4d4efe4451d383
Partial-Bug: #1584210
Partial-Bug: #1584224

amit surana (asurana-t) on 2016-05-31

tags:

added: releasenote
removed: blocker

Nagabhushana R (bhushana) on 2016-06-03

information type:

Proprietary → Public

Ashish Ranjan (aranjan-n) on 2016-06-03

tags:

removed: releasenote

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-06: [Review update] master

Review in progress for https://review.opencontrail.org/20956
Submitter: RAVI KIRAN (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-07:

Review in progress for https://review.opencontrail.org/20977
Submitter: Praveen K V (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-07:

Review in progress for https://review.opencontrail.org/20978
Submitter: Praveen K V (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-07: A change has been merged

#10

Reviewed: https://review.opencontrail.org/20977
Committed: http://github.org/Juniper/contrail-controller/commit/6c43e4b393526724ce44368bbcc1df0862bdc01e
Submitter: Zuul
Branch: master

commit 6c43e4b393526724ce44368bbcc1df0862bdc01e
Author: Praveen K V <email address hidden>
Date: Tue May 24 22:00:29 2016 +0530

Introspect changes to debug flow queue performances

- Added knobs to show "busy-time" for a queue
- Added flow-misc-event queue. This queue is used in subsequent comits

Change-Id: I879ef9e0afe0d13c59fef9b42895fcb0b88de022
Partial-Bug: #1584210
Partial-Bug: #1584224

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-07:

#11

Reviewed: https://review.opencontrail.org/20978
Committed: http://github.org/Juniper/contrail-controller/commit/9e9cd00288d0a35f7b00a5fbc847b8c02e6985cc
Submitter: Zuul
Branch: master

commit 9e9cd00288d0a35f7b00a5fbc847b8c02e6985cc
Author: Praveen K V <email address hidden>
Date: Fri May 27 11:00:00 2016 +0530

Flow mgmt optimization

Conflicts:
src/vnsw/agent/pkt/flow_entry.cc
src/vnsw/agent/pkt/flow_entry.h

Change-Id: Ifbfe539682b560061683d2269b4d4efe4451d383
Partial-Bug: #1584210
Partial-Bug: #1584224

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-10: [Review update] master

#12

Review in progress for https://review.opencontrail.org/21078
Submitter: Manish Singh (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-13:

#14

Review in progress for https://review.opencontrail.org/21100
Submitter: RAVI KIRAN (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-14:

#15

Review in progress for https://review.opencontrail.org/21078
Submitter: Manish Singh (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-16: A change has been merged

#17

Reviewed: https://review.opencontrail.org/21078
Committed: http://github.org/Juniper/contrail-controller/commit/4c17144fa37e21934c0ce7e890a0a8ac660d2809
Submitter: Zuul
Branch: master

commit 4c17144fa37e21934c0ce7e890a0a8ac660d2809
Author: Manish Singh <email address hidden>
Date: Sat Jun 11 14:37:57 2016 +0530

In BGP flow mgmt key add CN index.

Done to ensure different flow keys are created for session established to
primary or secondary bgp peer from same VM using same source port.

Partial-bug: #1584224

Conflicts:
src/vnsw/agent/pkt/test/SConscript
Change-Id: Iaf5ef824087b290d9be801cdd2392631e9ecaa41

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-16: [Review update] master

#18

Review in progress for https://review.opencontrail.org/21212
Submitter: RAVI KIRAN (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-16: [Review update] R3.0

#20

Review in progress for https://review.opencontrail.org/21224
Submitter: RAVI KIRAN (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-17: [Review update] master

#21

Review in progress for https://review.opencontrail.org/21253
Submitter: RAVI KIRAN (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-18: [Review update] R3.0

#24

Review in progress for https://review.opencontrail.org/21281
Submitter: RAVI KIRAN (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-18:

#25

Review in progress for https://review.opencontrail.org/21224
Submitter: RAVI KIRAN (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-21: A change has been merged

#26

Reviewed: https://review.opencontrail.org/21212
Committed: http://github.org/Juniper/contrail-controller/commit/fff0b34eb0672d38977f6b952a570cc5c2c94aeb
Submitter: Zuul
Branch: master

commit fff0b34eb0672d38977f6b952a570cc5c2c94aeb
Author: Ravi BK <email address hidden>
Date: Thu Jun 16 12:28:35 2016 +0530

Issue: Optimise flow setup rate

Currently Flow entry pointers are stored as a map
under FlowMgmtEntry. Map is unnecessary here since
the elements need not be present in sorted-order.

Fix: Use Boost Intrusive list to store the Node objects
(which contains FlowEntry pointers) to enhance
performance.

Change-Id: I048028f452aca64032c1ea30c4eefbddbeda614c
Partial-Bug: #1584224

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-21:

#27

Reviewed: https://review.opencontrail.org/21281
Committed: http://github.org/Juniper/contrail-controller/commit/711a98f79130267113d4c38cd2565ebf753525c5
Submitter: Zuul
Branch: R3.0

commit 711a98f79130267113d4c38cd2565ebf753525c5
Author: Manish Singh <email address hidden>
Date: Sat Jun 11 14:37:57 2016 +0530

In BGP flow mgmt key add CN index.

Done to ensure different flow keys are created for session established to
primary or secondary bgp peer from same VM using same source port.

Change-Id: I9924127e52659576d7ed287f76a1ba42cdad4106
Partial-bug: #1584224

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-21:

#28

Reviewed: https://review.opencontrail.org/21224
Committed: http://github.org/Juniper/contrail-controller/commit/de1273e9f929055692e82efb54e43bee921726fe
Submitter: Zuul
Branch: R3.0

commit de1273e9f929055692e82efb54e43bee921726fe
Author: Ravi BK <email address hidden>
Date: Thu Jun 16 19:09:11 2016 +0530

Issue: Optimise flow setup rate

Currently Flow entry pointers are stored as a map
under FlowMgmtEntry. Map is unnecessary here since
the elements need not be present in sorted-order.

Fix: Use Boost Intrusive list to store the Node objects
(which contains FlowEntry pointers) to enhance
performance.

Change-Id: Ib1280b49ef75c3d05f533f9c07cb41a91a13a605
Partial-Bug: #1584224

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-24: [Review update] master

#29

Review in progress for https://review.opencontrail.org/21253
Submitter: RAVI KIRAN (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-25: A change has been merged

#30

Reviewed: https://review.opencontrail.org/21253
Committed: http://github.org/Juniper/contrail-controller/commit/8008f7cca891a17e9f5a86ef1c803c089a6e5ff7
Submitter: Zuul
Branch: master

commit 8008f7cca891a17e9f5a86ef1c803c089a6e5ff7
Author: Ravi BK <email address hidden>
Date: Fri Jun 17 10:00:56 2016 +0530

Run Multiple instances of flow_mgmt_manager

Issue: Currently only one instance of flow_mgmt_manager is created
where as flowTable has multiple instances(where each instance
manages its logical partition based on flow hash parameters). Any
Flow updates(Add/Delete) sent to flow_mgmt_manager are not processed
faster since there is only one instance of it. Need to run
multiple instances of flow_mgmt_manager for faster processing of events.

Fix: Create Multiple instances of flow_mgmt_manager and process events
in respective instance based on table_index of FlowTable.
Added UT case to make sure DB Entry is not freed while FlowMgr
Instance hold reference.

Change-Id: I33526e04ced30636abd455edfa5fa02e79d12bac
Partial-Bug: #1584224

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-27: [Review update] R3.0

#31

Review in progress for https://review.opencontrail.org/21459
Submitter: RAVI KIRAN (<email address hidden>)

amit surana (asurana-t) on 2016-08-08

tags:

added: releasenote

amit surana (asurana-t) on 2016-08-15

tags:

added: performance
removed: perf

Revision history for this message

Rudra Rugge (rrugge) wrote on 2017-09-19:

#32

Please open a new bug for further tracking.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.