Juniper Openstack

[Mainline/Mitaka] xmpp connection getting reset while running testcases

  1. Series trunk
  2. Bug #1716356
Bug #1716356 reported by Suresh
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
Trunk
Fix Committed
Critical
Naveen N
Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

   In the latest mainline build 44, xmpp connection getting reset while running testcases. Because of this issue, many testcases hitting intermittent issues such as ping failure, VN verification failure etc.

   Able to see the below failure messages in contrail-contril.log and contrail-vrouter-agent.log

contrail-contril.log:
=====================
2017-09-11 Mon 12:36:46:966.125 IST nodem7 [Thread 47126734178048, Pid 21478]: BGP [SYS_CRIT]: XmppPeerInstanceLog: XMPP Peer nodem7:10.10.10.10 RECV in instance default-domain:ctest-TestIntfMirror-32613908:ctest-vn3-01828050:ctest-vn3-01828050 : Received route without subscribe controller/src/bgp/bgp_xmpp_channel.cc 747
2017-09-11 Mon 12:36:46:966.300 IST nodem7 [Thread 47126734178048, Pid 21478]: BGP [SYS_CRIT]: XmppPeerInstanceLog: XMPP Peer nodem7:10.10.10.10 RECV in instance default-domain:ctest-TestIntfMirror-32613908:ctest-vn3-01828050:ctest-vn3-01828050 : Received route without subscribe controller/src/bgp/bgp_xmpp_channel.cc 747
2017-09-11 Mon 12:36:47:017.698 IST nodem7 [Thread 47125970855680, Pid 21478]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem7:10.10.10.10 RECV Mode Server: PassiveOpen in state: Idle peer ip: 10.10.10.10 ( ) controller/src/xmpp/xmpp_state_machine.cc 1327
2017-09-11 Mon 12:37:05:085.025 IST nodem7 [Thread 47126801286912, Pid 21478]: BGP [SYS_CRIT]: XmppPeerInstanceLog: XMPP Peer nodem7:10.10.10.8 RECV in instance default-domain:ctest-TestIntfMirror-32613908:ctest-src_parent_port_vn-04114043:ctest-src_parent_port_vn-04114043 : Received route without subscribe controller/src/bgp/bgp_xmpp_channel.cc 747
2017-09-11 Mon 12:37:05:134.930 IST nodem7 [Thread 47125958260480, Pid 21478]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem7:10.10.10.8 RECV Mode Server: PassiveOpen in state: Idle peer ip: 10.10.10.8 ( ) controller/src/xmpp/xmpp_state_machine.cc 1327
2017-09-11 Mon 12:37:33:710.380 IST nodem7 [Thread 47125949863680, Pid 21478]: IFMap [SYS_WARN]: IFMapIdentifierNotFound: Cant find identifier target:64510:8000016 controller/src/ifmap/ifmap_server_table.cc 255

contrail-vrouter-agent.log:
===========================
root@nodem8:/var/log/contrail# tail -f contrail-vrouter-agent.log
2017-09-11 Mon 10:37:29:747.289 IST nodem8 [Thread 139782277740288, Pid 34275]: IFMapAgent [SYS_ERR]: InterfaceConfiguration: Virtual-network UUID mismatch for interface: 40ed75e8-7513-4fdf-ac77-3df132a8aade configuration VN uuid 00000000-0000-0000-0000-000000000000 compute VN uuid c74b365b-3c43-4a57-bf56-1dbf675c1f24 controller/src/vnsw/agent/oper/vm_interface_config.cc 814
2017-09-11 Mon 12:02:04:410.651 IST nodem8 [Thread 139782436730816, Pid 34275]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.10.10.8:39298::10.10.10.14:5269 < Read failed due to error 104 : Connection reset by peer controller/src/io/tcp_session.cc 480
2017-09-11 Mon 12:02:04:410.808 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.14 RECV Mode Client: Event: Tcp Connection Closed peer ip: 10.10.10.14 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1312
2017-09-11 Mon 12:02:04:410.964 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connection Closed peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1312
2017-09-11 Mon 12:02:04:468.495 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.14 RECV Mode Client: Event: Tcp Connected peer ip: 10.10.10.14 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1297
2017-09-11 Mon 12:02:04:468.671 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connected peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1297
2017-09-11 Mon 12:07:34:488.403 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.14 RECV Mode Client: Event: Tcp Connection Closed peer ip: 10.10.10.14 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1312
2017-09-11 Mon 12:07:34:488.580 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connection Closed peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1312
2017-09-11 Mon 12:07:34:540.885 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connected peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1297
2017-09-11 Mon 12:07:34:541.072 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.14 RECV Mode Client: Event: Tcp Connected peer ip: 10.10.10.14 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1297
2017-09-11 Mon 12:33:03:690.704 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connection Closed peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1312
2017-09-11 Mon 12:33:03:709.333 IST nodem8 [Thread 139782273541888, Pid 34275]: VRouter operation failed. Error <19:Object not present>. Object <NextHop Index: 35 Type: Tunnel to 10.10.10.8>. Operation <Addition>. Message number :10889
2017-09-11 Mon 12:33:03:741.519 IST nodem8 [Thread 139782265145088, Pid 34275]: VRouter operation failed. Error <22:Invalid object parameters>. Object <NextHop Index: 35 Type: Tunnel to 10.10.10.8>. Operation <Deletion>. Message number :10961
2017-09-11 Mon 12:33:03:744.254 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connected peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1297

Suresh (suresha)
Changed in juniperopenstack:
milestone: none → r4.1.0.0-fcs
Hari Prasad Killi (haripk)
tags: added: vrouter
Sudheendra Rao (sudheendra-k)
tags: added: sanityblocker
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/35507
Submitter: Naveen N (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/35512
Submitter: Naveen N (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/35512
Committed: http://github.com/Juniper/contrail-controller/commit/8e388d259ddb856564dcd8bf137d3e265807a41b
Submitter: Zuul (<email address hidden>)
Branch: master

commit 8e388d259ddb856564dcd8bf137d3e265807a41b
Author: Naveen N <email address hidden>
Date: Tue Sep 12 09:59:45 2017 +0530

* Dont export multicast route upon VRF delete

Ideally upon VRF delete control-node implictly deletes all the
paths added by agent in particular VRF, in problematic scenario
after VRF unsubscribe was done agent used to send a route delete
for multicast route, resulting in control-node closing the channel.

Change-Id: I440fa9bc561bf50229a7134122093d487b93bfb7
Closes-bug: #1716356

Revision history for this message
Sergey Kreys (skreys) wrote :

Are there plans to backport this fix to other branches?
We see similar issue on R3.1.1.x

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.