OpenStack-Ansible Queens/Stable OVS-router HA issue

Hi I have Deployed OpenStack Queens/Stable version using OSA, and I am using Openvswitch for the deployment. Everything works fine except for the HA-Router, neutron keeps on trying to create the namespace router and deletes it. Please find below the log file from the network node

 tail -f /var/log/neutron/neutron-l3-agent.log
2018-12-19 13:56:49.577 11895 ERROR neutron.agent.l3.agent pm.enable()
2018-12-19 13:56:49.577 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 88, in enable
2018-12-19 13:56:49.577 11895 ERROR neutron.agent.l3.agent run_as_root=self.run_as_root)
2018-12-19 13:56:49.577 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 903, in execute
2018-12-19 13:56:49.577 11895 ERROR neutron.agent.l3.agent log_fail_as_error=log_fail_as_error, **kwargs)
2018-12-19 13:56:49.577 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 151, in execute
2018-12-19 13:56:49.577 11895 ERROR neutron.agent.l3.agent raise ProcessExecutionError(msg, returncode=returncode)
2018-12-19 13:56:49.577 11895 ERROR neutron.agent.l3.agent ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory
2018-12-19 13:56:49.577 11895 ERROR neutron.agent.l3.agent
2018-12-19 13:56:49.577 11895 ERROR neutron.agent.l3.agent
2018-12-19 13:56:57.122 11895 INFO neutron.agent.l3.ha [-] Router c1c17287-4c65-4fa3-bdb7-4183b960040c transitioned to backup
2018-12-19 13:56:57.463 11895 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory
: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent [-] Error while initializing router c1c17287-4c65-4fa3-bdb7-4183b960040c: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 364, in _router_added
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent ri.initialize(self.process_monitor)
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 129, in initialize
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent self.spawn_state_change_monitor(process_monitor)
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 375, in spawn_state_change_monitor
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent pm.enable()
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 88, in enable
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent run_as_root=self.run_as_root)
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 903, in execute
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent log_fail_as_error=log_fail_as_error, **kwargs)
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 151, in execute
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent raise ProcessExecutionError(msg, returncode=returncode)
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent
2018-12-19 13:56:57.465 11895 ERROR neutron.agent.l3.agent
2018-12-19 13:56:57.858 11895 WARNING neutron.agent.l3.router_info [-] Can't gracefully delete the router c1c17287-4c65-4fa3-bdb7-4183b960040c: no router namespace found: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory
2018-12-19 13:56:57.860 11895 WARNING neutron.agent.l3.namespaces [-] Namespace qrouter-c1c17287-4c65-4fa3-bdb7-4183b960040c does not exist. Skipping delete: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: c1c17287-4c65-4fa3-bdb7-4183b960040c: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 560, in _process_router_update
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 479, in _process_router_if_compatible
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent self._process_added_router(router)
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 484, in _process_added_router
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent self._router_added(router['id'], router)
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 375, in _router_added
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent router_id)
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent self.force_reraise()
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 364, in _router_added
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent ri.initialize(self.process_monitor)
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 129, in initialize
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent self.spawn_state_change_monitor(process_monitor)
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 375, in spawn_state_change_monitor
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent pm.enable()
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 88, in enable
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent run_as_root=self.run_as_root)
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 903, in execute
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent log_fail_as_error=log_fail_as_error, **kwargs)
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent File "/openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 151, in execute
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent raise ProcessExecutionError(msg, returncode=returncode)
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent
2018-12-19 13:56:57.861 11895 ERROR neutron.agent.l3.agent

========================================================================================================

 tail -f /var/log/neutron/neutron-openvswitch-agent.log
2018-12-19 13:58:05.377 11892 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbListCommand(if_exists=True, records=[u'ha-bc3c1c27-26'], table=Port, columns=['name', 'tag'], row=False) do_commit /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2018-12-19 13:58:05.378 11892 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:121
2018-12-19 13:58:05.379 11892 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-53f56c67-7c1a-427d-bd43-64ec5324cfa5 - - - - -] Skipping ARP spoofing rules for port 'ha-bc3c1c27-26' because it has port security disabled
2018-12-19 13:58:05.382 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-53f56c67-7c1a-427d-bd43-64ec5324cfa5 - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x48,xid=0xa658b89a,OFPFlowMod(buffer_id=4294967295,command=3,cookie=9348531806942299602L,cookie_mask=18446744073709551615L,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'eth_type': 2054, 'in_port': 336}),out_group=4294967295,out_port=4294967295,priority=0,table_id=0) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:05.384 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-53f56c67-7c1a-427d-bd43-64ec5324cfa5 - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x50,xid=0xa658b89c,OFPFlowMod(buffer_id=4294967295,command=3,cookie=9348531806942299602L,cookie_mask=18446744073709551615L,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'ip_proto': 58, 'icmpv6_type': 136, 'eth_type': 34525, 'in_port': 336}),out_group=4294967295,out_port=4294967295,priority=0,table_id=0) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:05.386 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-53f56c67-7c1a-427d-bd43-64ec5324cfa5 - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x40,xid=0xa658b89e,OFPFlowMod(buffer_id=4294967295,command=3,cookie=9348531806942299602L,cookie_mask=18446744073709551615L,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'in_port': 336}),out_group=4294967295,out_port=4294967295,priority=0,table_id=24) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:05.388 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-53f56c67-7c1a-427d-bd43-64ec5324cfa5 - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x40,xid=0xa658b8a0,OFPFlowMod(buffer_id=4294967295,command=3,cookie=9348531806942299602L,cookie_mask=18446744073709551615L,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'in_port': 336}),out_group=4294967295,out_port=4294967295,priority=0,table_id=0) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:05.390 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-53f56c67-7c1a-427d-bd43-64ec5324cfa5 - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x40,xid=0xa658b8a2,OFPFlowMod(buffer_id=4294967295,command=3,cookie=9348531806942299602L,cookie_mask=18446744073709551615L,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'in_port': 336}),out_group=4294967295,out_port=4294967295,priority=0,table_id=25) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:05.391 11892 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbSetCommand(table=Port, col_values=(('tag', 230),), record=ha-bc3c1c27-26) do_commit /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2018-12-19 13:58:05.398 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-53f56c67-7c1a-427d-bd43-64ec5324cfa5 - - - - -] Setting status for bc3c1c27-267e-4761-a4ec-9a1fe2cdbc9c to UP _bind_devices /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:867
2018-12-19 13:58:06.101 11892 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method add_fdb_entries called with arguments (<neutron_lib.context.Context object at 0x7f43a2e2d710>,) {u'fdb_entries': {u'7bbad5b4-4532-447e-b281-a23077a3829f': {u'segment_id': 62, u'ports': {u'10.168.194.13': [[u'00:00:00:00:00:00', u'0.0.0.0'], [u'fa:16:3e:79:db:63', u'169.254.192.5']], u'10.168.194.14': [[u'00:00:00:00:00:00', u'0.0.0.0'], [u'fa:16:3e:cd:a0:49', u'169.254.192.8']]}, u'network_type': u'vxlan'}}} wrapper /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-12-19 13:58:06.102 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] fdb_add received fdb_add /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:493
2018-12-19 13:58:06.102 11892 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method fdb_add_tun called with arguments (<neutron_lib.context.Context object at 0x7f43a2e2d710>, <neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.br_tun.OVSTunnelBridge object at 0x7f43a2dc8250>, <neutron.plugins.ml2.drivers.openvswitch.agent.vlanmanager.LocalVLANMapping object at 0x7f43a2dc3c90>, {u'10.168.194.13': [PortInfo(mac_address=u'00:00:00:00:00:00', ip_address=u'0.0.0.0'), PortInfo(mac_address=u'fa:16:3e:79:db:63', ip_address=u'169.254.192.5')], u'10.168.194.14': [PortInfo(mac_address=u'00:00:00:00:00:00', ip_address=u'0.0.0.0'), PortInfo(mac_address=u'fa:16:3e:cd:a0:49', ip_address=u'169.254.192.8')]}, <bound method OVSNeutronAgent._tunnel_port_lookup of <neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent object at 0x7f43a324df90>>) {} wrapper /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-12-19 13:58:06.105 11892 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method add_fdb_entries called with arguments (<neutron_lib.context.Context object at 0x7f43a2b145d0>,) {u'fdb_entries': {u'7bbad5b4-4532-447e-b281-a23077a3829f': {u'segment_id': 62, u'ports': {u'10.168.194.12': [[u'00:00:00:00:00:00', u'0.0.0.0'], [u'fa:16:3e:3a:f6:61', u'169.254.192.9']]}, u'network_type': u'vxlan'}}} wrapper /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-12-19 13:58:06.106 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] fdb_add received fdb_add /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:493
2018-12-19 13:58:06.109 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x80,xid=0x1b82c66f,OFPFlowMod(buffer_id=4294967295,command=0,cookie=10474921376408558733L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[OFPInstructionActions(actions=[OFPActionPopVlan(len=8,type=18), OFPActionSetField(tunnel_id=62), OFPActionOutput(len=16,max_len=0,port=2,type=0), OFPActionOutput(len=16,max_len=0,port=3,type=0)],len=64,type=4)],match=OFPMatch(oxm_fields={'vlan_vid': 4326}),out_group=0,out_port=0,priority=1,table_id=22) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:06.114 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x78,xid=0x1b82c671,OFPFlowMod(buffer_id=4294967295,command=0,cookie=10474921376408558733L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[OFPInstructionActions(actions=[OFPActionPopVlan(len=8,type=18), OFPActionSetField(tunnel_id=62), OFPActionOutput(len=16,max_len=0,port=3,type=0)],len=48,type=4)],match=OFPMatch(oxm_fields={'eth_dst': 'fa:16:3e:79:db:63', 'vlan_vid': 4326}),out_group=0,out_port=0,priority=2,table_id=20) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:06.118 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x80,xid=0x1b82c673,OFPFlowMod(buffer_id=4294967295,command=0,cookie=10474921376408558733L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[OFPInstructionActions(actions=[OFPActionPopVlan(len=8,type=18), OFPActionSetField(tunnel_id=62), OFPActionOutput(len=16,max_len=0,port=2,type=0), OFPActionOutput(len=16,max_len=0,port=3,type=0)],len=64,type=4)],match=OFPMatch(oxm_fields={'vlan_vid': 4326}),out_group=0,out_port=0,priority=1,table_id=22) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:06.122 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x78,xid=0x1b82c675,OFPFlowMod(buffer_id=4294967295,command=0,cookie=10474921376408558733L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[OFPInstructionActions(actions=[OFPActionPopVlan(len=8,type=18), OFPActionSetField(tunnel_id=62), OFPActionOutput(len=16,max_len=0,port=2,type=0)],len=48,type=4)],match=OFPMatch(oxm_fields={'eth_dst': 'fa:16:3e:cd:a0:49', 'vlan_vid': 4326}),out_group=0,out_port=0,priority=2,table_id=20) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:06.281 11892 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-a6ca0650-42ac-45c8-810b-e7a2458d530c - - - - -] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method remove_fdb_entries called with arguments (<neutron_lib.context.Context object at 0x7f43a2b3bd50>,) {u'fdb_entries': {u'7bbad5b4-4532-447e-b281-a23077a3829f': {u'segment_id': 62, u'ports': {u'10.168.194.13': [[u'00:00:00:00:00:00', u'0.0.0.0'], [u'fa:16:3e:79:db:63', u'169.254.192.5']]}, u'network_type': u'vxlan'}}} wrapper /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-12-19 13:58:06.283 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a6ca0650-42ac-45c8-810b-e7a2458d530c - - - - -] fdb_remove received fdb_remove /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:506
2018-12-19 13:58:06.283 11892 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-a6ca0650-42ac-45c8-810b-e7a2458d530c - - - - -] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method fdb_remove_tun called with arguments (<neutron_lib.context.Context object at 0x7f43a2b3bd50>, <neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.br_tun.OVSTunnelBridge object at 0x7f43a2dc8250>, <neutron.plugins.ml2.drivers.openvswitch.agent.vlanmanager.LocalVLANMapping object at 0x7f43a2dc3c90>, {u'10.168.194.13': [PortInfo(mac_address=u'00:00:00:00:00:00', ip_address=u'0.0.0.0'), PortInfo(mac_address=u'fa:16:3e:79:db:63', ip_address=u'169.254.192.5')]}, <bound method OVSNeutronAgent._tunnel_port_lookup of <neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent object at 0x7f43a324df90>>) {} wrapper /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-12-19 13:58:06.295 11892 DEBUG neutron.agent.resource_cache [req-b05031ba-9f54-4848-862e-e6fd523084b3 - - - - -] Resource Port bc3c1c27-267e-4761-a4ec-9a1fe2cdbc9c updated (revision_number 322->323). Old fields: {'status': u'DOWN', 'binding_levels': [PortBindingLevel(driver='openvswitch',host='altbthdloscon01a',level=0,port_id=bc3c1c27-267e-4761-a4ec-9a1fe2cdbc9c,segment=NetworkSegment(f2cabc0f-404f-42fb-9937-bb28b798bab6))]} New fields: {'status': u'ACTIVE', 'binding_levels': [PortBindingLevel(driver='openvswitch',host='altbthdloscon01a',level=0,port_id=bc3c1c27-267e-4761-a4ec-9a1fe2cdbc9c,segment=NetworkSegment(f2cabc0f-404f-42fb-9937-bb28b798bab6))]} record_resource_update /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/resource_cache.py:185
2018-12-19 13:58:06.296 11892 DEBUG neutron_lib.callbacks.manager [req-b05031ba-9f54-4848-862e-e6fd523084b3 - - - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update-2041076', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-1845346'] for Port, after_update _notify_loop /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-12-19 13:58:06.300 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-a6ca0650-42ac-45c8-810b-e7a2458d530c - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x70,xid=0x1b82c677,OFPFlowMod(buffer_id=4294967295,command=0,cookie=10474921376408558733L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[OFPInstructionActions(actions=[OFPActionPopVlan(len=8,type=18), OFPActionSetField(tunnel_id=62), OFPActionOutput(len=16,max_len=0,port=2,type=0)],len=48,type=4)],match=OFPMatch(oxm_fields={'vlan_vid': 4326}),out_group=0,out_port=0,priority=1,table_id=22) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:06.305 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-a6ca0650-42ac-45c8-810b-e7a2458d530c - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x48,xid=0x1b82c679,OFPFlowMod(buffer_id=4294967295,command=3,cookie=10474921376408558733L,cookie_mask=18446744073709551615L,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'eth_dst': 'fa:16:3e:79:db:63', 'vlan_vid': 4326}),out_group=4294967295,out_port=4294967295,priority=0,table_id=20) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:06.470 11892 DEBUG neutron.agent.resource_cache [req-85766f54-7abc-4c36-8246-f165f8b29390 - - - - -] Resource Port b74b4fdb-70f4-4d45-bd47-93bb15d54d15 updated (revision_number 321->322). Old fields: {'status': u'ACTIVE'} New fields: {'status': u'DOWN'} record_resource_update /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/resource_cache.py:185
2018-12-19 13:58:06.471 11892 DEBUG neutron_lib.callbacks.manager [req-85766f54-7abc-4c36-8246-f165f8b29390 - - - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update-2041076', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-1845346'] for Port, after_update _notify_loop /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-12-19 13:58:06.516 11892 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method add_fdb_entries called with arguments (<neutron_lib.context.Context object at 0x7f43a29af550>,) {u'fdb_entries': {u'7bbad5b4-4532-447e-b281-a23077a3829f': {u'segment_id': 62, u'ports': {u'10.168.194.14': [[u'00:00:00:00:00:00', u'0.0.0.0'], [u'fa:16:3e:cd:a0:49', u'169.254.192.8']]}, u'network_type': u'vxlan'}}} wrapper /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-12-19 13:58:06.516 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] fdb_add received fdb_add /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:493
2018-12-19 13:58:06.517 11892 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method fdb_add_tun called with arguments (<neutron_lib.context.Context object at 0x7f43a29af550>, <neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.br_tun.OVSTunnelBridge object at 0x7f43a2dc8250>, <neutron.plugins.ml2.drivers.openvswitch.agent.vlanmanager.LocalVLANMapping object at 0x7f43a2dc3c90>, {u'10.168.194.14': [PortInfo(mac_address=u'00:00:00:00:00:00', ip_address=u'0.0.0.0'), PortInfo(mac_address=u'fa:16:3e:cd:a0:49', ip_address=u'169.254.192.8')]}, <bound method OVSNeutronAgent._tunnel_port_lookup of <neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent object at 0x7f43a324df90>>) {} wrapper /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-12-19 13:58:06.521 11892 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method add_fdb_entries called with arguments (<neutron_lib.context.Context object at 0x7f43a2adc210>,) {u'fdb_entries': {u'7bbad5b4-4532-447e-b281-a23077a3829f': {u'segment_id': 62, u'ports': {u'10.168.194.12': [[u'00:00:00:00:00:00', u'0.0.0.0'], [u'fa:16:3e:3a:f6:61', u'169.254.192.9']]}, u'network_type': u'vxlan'}}} wrapper /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-12-19 13:58:06.522 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] fdb_add received fdb_add /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:493
2018-12-19 13:58:06.523 11892 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-53f56c67-7c1a-427d-bd43-64ec5324cfa5 - - - - -] Configuration for devices up [u'bc3c1c27-267e-4761-a4ec-9a1fe2cdbc9c'] and devices down [] completed.
2018-12-19 13:58:06.523 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-53f56c67-7c1a-427d-bd43-64ec5324cfa5 - - - - -] Agent rpc_loop - iteration:905 - ports processed. Elapsed:2.516 rpc_loop /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2099
2018-12-19 13:58:06.524 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-53f56c67-7c1a-427d-bd43-64ec5324cfa5 - - - - -] Agent rpc_loop - iteration:905 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 1, 'removed': 0}}. Elapsed:2.517 loop_count_and_wait /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1817
2018-12-19 13:58:06.528 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x70,xid=0x1b82c67b,OFPFlowMod(buffer_id=4294967295,command=0,cookie=10474921376408558733L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[OFPInstructionActions(actions=[OFPActionPopVlan(len=8,type=18), OFPActionSetField(tunnel_id=62), OFPActionOutput(len=16,max_len=0,port=2,type=0)],len=48,type=4)],match=OFPMatch(oxm_fields={'vlan_vid': 4326}),out_group=0,out_port=0,priority=1,table_id=22) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:06.535 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-a1d38472-08a8-4da2-80f0-a893b4a4b20c - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x78,xid=0x1b82c67d,OFPFlowMod(buffer_id=4294967295,command=0,cookie=10474921376408558733L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[OFPInstructionActions(actions=[OFPActionPopVlan(len=8,type=18), OFPActionSetField(tunnel_id=62), OFPActionOutput(len=16,max_len=0,port=2,type=0)],len=48,type=4)],match=OFPMatch(oxm_fields={'eth_dst': 'fa:16:3e:cd:a0:49', 'vlan_vid': 4326}),out_group=0,out_port=0,priority=2,table_id=20) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:06.610 11892 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-a6ca0650-42ac-45c8-810b-e7a2458d530c - - - - -] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method remove_fdb_entries called with arguments (<neutron_lib.context.Context object at 0x7f43a2d265d0>,) {u'fdb_entries': {u'7bbad5b4-4532-447e-b281-a23077a3829f': {u'segment_id': 62, u'ports': {u'10.168.194.13': [[u'fa:16:3e:79:db:63', u'169.254.192.5']]}, u'network_type': u'vxlan'}}} wrapper /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-12-19 13:58:06.613 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a6ca0650-42ac-45c8-810b-e7a2458d530c - - - - -] fdb_remove received fdb_remove /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:506
2018-12-19 13:58:06.614 11892 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-a6ca0650-42ac-45c8-810b-e7a2458d530c - - - - -] neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent method fdb_remove_tun called with arguments (<neutron_lib.context.Context object at 0x7f43a2d265d0>, <neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.br_tun.OVSTunnelBridge object at 0x7f43a2dc8250>, <neutron.plugins.ml2.drivers.openvswitch.agent.vlanmanager.LocalVLANMapping object at 0x7f43a2dc3c90>, {u'10.168.194.13': [PortInfo(mac_address=u'fa:16:3e:79:db:63', ip_address=u'169.254.192.5')]}, <bound method OVSNeutronAgent._tunnel_port_lookup of <neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent object at 0x7f43a324df90>>) {} wrapper /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-12-19 13:58:06.620 11892 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-a6ca0650-42ac-45c8-810b-e7a2458d530c - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x48,xid=0x1b82c67f,OFPFlowMod(buffer_id=4294967295,command=3,cookie=10474921376408558733L,cookie_mask=18446744073709551615L,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'eth_dst': 'fa:16:3e:79:db:63', 'vlan_vid': 4326}),out_group=4294967295,out_port=4294967295,priority=0,table_id=20) result None _send_msg /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2018-12-19 13:58:06.742 11892 DEBUG neutron.agent.linux.async_process [-] Output received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: {"data":[["c2b6166f-a1e3-4a04-bc87-ff3b92fec8e7","delete","ha-bc3c1c27-26",336,["map",[["attached-mac","fa:16:3e:3a:f6:61"],["iface-id","bc3c1c27-267e-4761-a4ec-9a1fe2cdbc9c"],["iface-status","active"]]]]],"headings":["row","action","name","ofport","external_ids"]} _read_stdout /openstack/venvs/neutron-17.1.4/lib/python2.7/site-packages/neutron/agent/linux/async_process.py:239

=========================================================================================================
tail -f /var/log/openvswitch/ovs-vswitchd.log
2018-12-19T18:58:27.927Z|01683|bridge|INFO|bridge br-int: added interface ha-bc3c1c27-26 on port 339
2018-12-19T18:58:28.591Z|01684|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on ha-bc3c1c27-26 device failed: No such device
2018-12-19T18:58:30.553Z|01685|bridge|WARN|Dropped 54 log messages in last 58 seconds (most recently, 1 seconds ago) due to excessive rate
2018-12-19T18:58:30.553Z|01686|bridge|WARN|port bond1: Using the default bond_mode active-backup. Note that in previous versions, the default bond_mode was balance-slb
2018-12-19T18:58:31.586Z|01687|bridge|INFO|bridge br-int: deleted interface ha-bc3c1c27-26 on port 339
2018-12-19T18:58:35.926Z|01688|bridge|INFO|bridge br-int: added interface ha-bc3c1c27-26 on port 340
2018-12-19T18:58:36.641Z|01689|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on ha-bc3c1c27-26 device failed: No such device
2018-12-19T18:58:36.647Z|01690|netdev_linux|INFO|Dropped 1 log messages in last 8 seconds (most recently, 8 seconds ago) due to excessive rate
2018-12-19T18:58:36.647Z|01691|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on ha-bc3c1c27-26 device failed: No such device
2018-12-19T18:58:39.948Z|01692|bridge|INFO|bridge br-int: deleted interface ha-bc3c1c27-26 on port 340
2018-12-19T18:58:44.694Z|01693|bridge|INFO|bridge br-int: added interface ha-bc3c1c27-26 on port 341
2018-12-19T18:58:45.486Z|01694|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on ha-bc3c1c27-26 device failed: No such device
2018-12-19T18:58:45.489Z|01695|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on ha-bc3c1c27-26 device failed: No such device
2018-12-19T18:58:45.996Z|01696|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 76 flow_mods in the last 59 s (43 adds, 33 deletes)
2018-12-19T18:58:46.017Z|01697|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 35 flow_mods in the 55 s starting 56 s ago (35 deletes)
2018-12-19T18:58:48.486Z|01698|bridge|INFO|bridge br-int: deleted interface ha-bc3c1c27-26 on port 341
2018-12-19T18:58:52.865Z|01699|bridge|INFO|bridge br-int: added interface ha-bc3c1c27-26 on port 342
2018-12-19T18:58:53.591Z|01700|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on ha-bc3c1c27-26 device failed: No such device
2018-12-19T18:58:56.902Z|01701|bridge|INFO|bridge br-int: deleted interface ha-bc3c1c27-26 on port 342
2018-12-19T18:59:01.036Z|01702|bridge|INFO|bridge br-int: added interface ha-bc3c1c27-26 on port 343
2018-12-19T18:59:01.835Z|01703|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on ha-bc3c1c27-26 device failed: No such device
2018-12-19T18:59:01.838Z|01704|netdev_linux|INFO|Dropped 1 log messages in last 8 seconds (most recently, 8 seconds ago) due to excessive rate
2018-12-19T18:59:01.838Z|01705|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on ha-bc3c1c27-26 device failed: No such device
2018-12-19T18:59:04.680Z|01706|bridge|INFO|bridge br-int: deleted interface ha-bc3c1c27-26 on port 343