control-node ifmap continued to have snat si even after router is deleted

Bug #1377467 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R1.1
Fix Committed
Medium
Hampapur Ajay
R2.0
Fix Committed
Undecided
Vedamurthy Joshi
Trunk
Fix Committed
Medium
Vedamurthy Joshi

Bug Description

R1.10 44 (Ubuntu icehouse multinode setup)

In the test test_basic_snat_behavior_with_subnet_attach_detach in contrail-test, it creates snat setup, removes the VN interface from the router, adds the VN back again and finally cleans up everything.

At the end of it, the script failed because the VN object in control-node config db continued to exist even after the VN and router was deleted.
It was seen that the snat VM's instance IP continued to refer to the VN in control-node config.

The VN is virtual-network:default-domain:TestRouters-82931130:ext_vn-51078609

In control-node introspect : ifmap table entry :
virtual-network:default-domain:TestRouters-82931130:ext_vn-51078609
- - -
obj_info
neighbors
instance-ip:si_808bbb6b-dc7b-4020-8cce-4816611b85ce-right
00:38:46.051751

Logs will be in http://10.204.216.50/Docs/bugs/#

root@nodec22:/var/log/contrail# neutron net-list
+--------------------------------------+-------------------------+---------------------------------------------------+
| id | name | subnets |
+--------------------------------------+-------------------------+---------------------------------------------------+
| dd32ac7e-527c-486f-addb-4118a65ed43a | default-virtual-network | |
| d9efaf6b-d60c-4b48-b5e7-89ce4b29647c | ip-fabric | |
| eae8c123-451b-43cb-948d-d37ec2c827d5 | vn1-31619381 | 0bc7104c-1a59-47ec-ad19-6fbc6dbab0b5 51.6.68.0/24 |
| 2993df96-842a-4f5f-b7f8-93fa36e37a04 | __link_local__ | |
+--------------------------------------+-------------------------+---------------------------------------------------+
root@nodec22:/var/log/contrail#

test logs :
=========
2014-10-04 05:42:38,453 - INFO - Deleting interface with subnet_id aca8ed2d-e966-4304-a6a3-c66d7aaf410f, port_id None from router 808bb
b6b-dc7b-4020-8cce-4816611b85ce
2014-10-04 05:42:39,062 - INFO - Deleting the VM vm_public-53323424
2014-10-04 05:42:44,221 - INFO - Deleting the VM vm_private-26268979
2014-10-04 05:42:49,326 - INFO - Deleting the VN vn_private-41715389
2014-10-04 05:42:49,488 - INFO - VN vn_private-41715389 is not found in API Server
None
2014-10-04 05:42:49,494 - INFO - VN vn_private-41715389 is not present in Agent 10.204.217.69
None
2014-10-04 05:42:49,501 - INFO - VN vn_private-41715389 is not present in Agent 10.204.217.108
None
2014-10-04 05:42:49,507 - INFO - VN vn_private-41715389 is not present in Agent 10.204.217.109
2014-10-04 05:42:49,510 - WARNING - Routing instance for VN vn_private-41715389 is still found in Control-node 10.204.217.7
2014-10-04 05:42:49,513 - WARNING - Routing instance for VN vn_private-41715389 is still found in Control-node 10.204.217.108
2014-10-04 05:42:49,550 - WARNING - Control-node config DB still has VN vn_private-41715389
2014-10-04 05:42:54,601 - INFO - Routing instances and Config db in Control-nodes does not have VN vn_private-41715389 info
2014-10-04 05:42:54,601 - INFO - Deleting RT for VN ext_vn-51078609
2014-10-04 05:42:54,624 - INFO - Deleting the VN ext_vn-51078609
2014-10-04 05:42:55,016 - INFO - VN ext_vn-51078609 is not found in API Server
None
2014-10-04 05:42:55,022 - INFO - VN ext_vn-51078609 is not present in Agent 10.204.217.69
None
2014-10-04 05:42:55,030 - INFO - VN ext_vn-51078609 is not present in Agent 10.204.217.109
2014-10-04 05:42:55,032 - WARNING - Routing instance for VN ext_vn-51078609 is still found in Control-node 10.204.217.7
2014-10-04 05:42:55,036 - WARNING - Routing instance for VN ext_vn-51078609 is still found in Control-node 10.204.217.108
2014-10-04 05:42:55,066 - WARNING - Control-node config DB still has VN ext_vn-51078609
2014-10-04 05:43:00,110 - WARNING - Control-node config DB still has VN ext_vn-51078609
2014-10-04 05:43:05,159 - WARNING - Control-node config DB still has VN ext_vn-51078609
2014-10-04 05:43:10,197 - WARNING - Control-node config DB still has VN ext_vn-51078609

Tags: blocker snat
Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :

Prakash debugged on a similar issue :
Have these errors in contrail-api-0.log :

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1168, in _dbe_oper_subscribe
    self._dbe_delete_notification(oper_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1270, in _dbe_delete_notification
    (ok, ifmap_result) = method(obj_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py", line 7949, in _ifmap_virtual_machine_delete
    self._delete_id_pair_meta(r_elem[0].attrib['name'], to_imid, meta_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 343, in _delete_id_pair_meta
    _id_to_metas_delete(id_x, id_y, meta_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 324, in _id_to_metas_delete
    if meta_name not in self._id_to_metas[id1]:
KeyError: 'contrail:service-instance:default-domain:TestRouters-01229666:si_21072655-bd1c-433b-b818-fb3c838bd734'

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1168, in _dbe_oper_subscribe
    self._dbe_delete_notification(oper_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1270, in _dbe_delete_notification
    (ok, ifmap_result) = method(obj_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py", line 1216, in _ifmap_instance_ip_delete
    self._delete_id_pair_meta(r_elem[0].attrib['name'], to_imid, meta_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 343, in _delete_id_pair_meta
    _id_to_metas_delete(id_x, id_y, meta_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 324, in _id_to_metas_delete
    if meta_name not in self._id_to_metas[id1]:
KeyError: 'contrail:virtual-machine-interface:default-domain:TestRouters-01229666:default-domain__TestRouters-01229666__c2a0c10f-f6f2-4b51-bdb7-9f3d062f760e__1-right'

tags: added: blocker
Revision history for this message
Sachin Bansal (sbansal) wrote :

Already fixed in mainline with commit 4196290833e41fc9488d2ec119ddb77597cb5a94

Revision history for this message
Rudra Rugge (rrugge) wrote :

Sachins fixes are for https://bugs.launchpad.net/juniperopenstack/+bug/1365322. Please ignore his comments or commit id above this message.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/3619
Committed: http://github.org/Juniper/contrail-controller/commit/f2dc24032032b4a6dbe952200464704bee77787d
Submitter: Zuul
Branch: R1.10

commit f2dc24032032b4a6dbe952200464704bee77787d
Author: Sachin Bansal <email address hidden>
Date: Thu Oct 9 12:28:45 2014 -0700

set pending_field_update after updating static routes so that it is reflected in api server

Change-Id: If42a04863d598286ddb18727340897d0d8302ed0
Closes-Bug: 1377467

Revision history for this message
Rudra Rugge (rrugge) wrote :

ifmap still has residual instance-ip entries. Ajay believes that it is related to rabbitmq restart and ha mode not being set.

Revision history for this message
Ashish Ranjan (aranjan-n) wrote :

Review pending

Revision history for this message
Hampapur Ajay (hajay) wrote :

Can this be retried after https://review.opencontrail.org/#/c/5234/ is in?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.