One flow stuck in hold forever on a setup

Bug #1578949 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
High
Hari Prasad Killi
Trunk
Fix Committed
High
Hari Prasad Killi

Bug Description

R3.0.20 Build 33 with latest contrail-vrouter-agent from praveen

The traffic generator was sending 1K pps of udp traffic from MX via a VM

It was seen that 4k flows were created fine.
But one of them is stuck in HOLD state

980152 10.11.1.2:68 17 (5)
                         10.11.2.2:63
(Gen: 72, K(nh):63, Action:H, Flags:, S(nh):0, Stats:185/20350, SPort 0)

In agent flow trace :

2016-05-06 00:29:58.763 FlowTrace: operation = ADD info= [ gen_id = 72 flow_index = 980152 nh_id = 63 source_ip = 10.11.1.2 source_port = 68 destination_ip = 10.11.2.2 destination_port = 63 protocol = 17 vrf = 4294967295 mirror_l= [ [ ] ] mirror_vrf = 65535 implicit_deny = 0 short_flow = 1 source_vn_list= [ [ (*_iter95) = __UNKNOWN__, ] ] dest_vn_list= [ [ (*_iter96) = __UNKNOWN__, ] ] source_vn_match = dest_vn_match = source_sg_id_l= [ [ ] ] dest_sg_id_l= [ [ ] ] vrf_assign = l3_flow = 1 smac = 00:00:00:00:00:00 dmac = 00:00:00:00:00:00 drop_reason = UNKNOWN table_id = 3 ] file = controller/src/vnsw/agent/pkt/flow_mgmt.cc line = 484
2016-05-06 00:29:58.861 FlowTrace: operation = DEL info= [ gen_id = 72 flow_index = 980152 nh_id = 63 source_ip = 10.11.1.2 source_port = 68 destination_ip = 10.11.2.2 destination_port = 63 protocol = 17 vrf = 4294967295 mirror_l= [ [ ] ] mirror_vrf = 65535 implicit_deny = 0 short_flow = 1 source_vn_list= [ [ (*_iter95) = __UNKNOWN__, ] ] dest_vn_list= [ [ (*_iter96) = __UNKNOWN__, ] ] source_vn_match = dest_vn_match = source_sg_id_l= [ [ ] ] dest_sg_id_l= [ [ ] ] vrf_assign = l3_flow = 1 smac = 00:00:00:00:00:00 dmac = 00:00:00:00:00:00 drop_reason = UNKNOWN table_id = 3 ] file = controller/src/vnsw/agent/pkt/flow_mgmt.cc line = 484

---

2016-05-06 00:38:18.187 FlowTrace: operation = ADD info= [ gen_id = 73 flow_index = 980152 nh_id = 63 source_ip = 10.11.1.2 source_port = 68 destination_ip = 10.11.2.2 destination_port = 63 protocol = 17 vrf = 4294967295 mirror_l= [ [ ] ] mirror_vrf = 65535 implicit_deny = 0 short_flow = 1 source_vn_list= [ [ (*_iter95) = __UNKNOWN__, ] ] dest_vn_list= [ [ (*_iter96) = __UNKNOWN__, ] ] source_vn_match = dest_vn_match = source_sg_id_l= [ [ ] ] dest_sg_id_l= [ [ ] ] vrf_assign = l3_flow = 1 smac = 00:00:00:00:00:00 dmac = 00:00:00:00:00:00 drop_reason = UNKNOWN table_id = 3 ] file = controller/src/vnsw/agent/pkt/flow_mgmt.cc line = 484
2016-05-06 00:38:18.224 FlowTrace: operation = DEL info= [ gen_id = 73 flow_index = 980152 nh_id = 63 source_ip = 10.11.1.2 source_port = 68 destination_ip = 10.11.2.2 destination_port = 63 protocol = 17 vrf = 4294967295 mirror_l= [ [ ] ] mirror_vrf = 65535 implicit_deny = 0 short_flow = 1 source_vn_list= [ [ (*_iter95) = __UNKNOWN__, ] ] dest_vn_list= [ [ (*_iter96) = __UNKNOWN__, ] ] source_vn_match = dest_vn_match = source_sg_id_l= [ [ ] ] dest_sg_id_l= [ [ ] ] vrf_assign = l3_flow = 1 smac = 00:00:00:00:00:00 dmac = 00:00:00:00:00:00 drop_reason = UNKNOWN table_id = 3 ] file = controller/src/vnsw/agent/pkt/flow_mgmt.cc line = 484

vrouter --info :

vRouter module version 3.0.2.0 (Built by contrail-builder@contrail-ec-build16 on 2016-04-29 16:28:31.068957)

Startup parameters
    Interfaces limit 4352
    VRF tables limit 4096
    NextHops limit 65536
    MPLS Labels limit 5120
    Bridge Table limit 262144
    Bridge Table Overflow limit 53248
    Flow Table limit 4194304
    Flow Table overflow limit 839680
    Mirror entries limit 255

Runtime parameters
  Performance tweaks
    GRO 1
    Segmentation in software 1
  TCP MSS adjust settings
    TCP MSS on packets from VM 1
    TCP MSS on packet sent to VM 1
  RPS settings
    RPS after pulling inner hdr (perfr1) 0
    RPS after GRO on pkt1 (perfr2) 1
    RPS from phys rx handler (perfr3) 1
    Pull inner header (faster version) 1
    CPU to send pkts to, if perfr1 set 0
    CPU to send pkts to, if perfr2 set 0
    CPU to send pkts to, if perfr3 set 0
  Other settings
    NIC cksum offload for outer UDP hdr 0
    Flow hold limit: 4096
    MPLS over UDP globally 0
    Used Flow entries 3999
    Used Over Flow entries 0
    Used Bridge entries 30
    Used Over Flow bridge entries 0

root@nodek1:/var/log/nova#

Jeba Paulaiyan (jebap)
tags: added: blocker
Revision history for this message
Hari Prasad Killi (haripk) wrote :

As the packet has DHCP client source port (68), it is processed as DHCP packet. Since the data is not DHCP, the packet is later discarded. The flow setup in vrouter is removed later as part of audit.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/20182
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/20258
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20182
Committed: http://github.org/Juniper/contrail-controller/commit/549644d72247ba408a3eb8039ef5c10405ef2f4c
Submitter: Zuul
Branch: R3.0

commit 549644d72247ba408a3eb8039ef5c10405ef2f4c
Author: Hari <email address hidden>
Date: Fri May 13 11:10:55 2016 +0530

Do not consider flow trap packets for DHCP handling.

In case a packet is trapped for flow setup, do not send it to DHCP
even when the packet is DHCP.

Change-Id: I37d274d56b25d0e85dad39b98a0ab5a1e555234f
closes-bug: #1578949

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/20258
Committed: http://github.org/Juniper/contrail-controller/commit/08c50ee52d251151d5f3d097094088f0c021700b
Submitter: Zuul
Branch: master

commit 08c50ee52d251151d5f3d097094088f0c021700b
Author: Hari <email address hidden>
Date: Fri May 13 11:10:55 2016 +0530

Do not consider flow trap packets for DHCP handling.

In case a packet is trapped for flow setup, do not send it to DHCP
even when the packet is DHCP.

Change-Id: I2c89f2e8f6fae879f6cab22fc0e1c91a4616a093
closes-bug: #1578949

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.