Juniper Openstack

Unable to delete port - instance-ip deletion throws error

Bug #1521853 reported by Harish Kumar on 2015-12-02

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Juniper Openstack	New	Undecided	Jeba Paulaiyan

Bug Description

During running a rally scenario, I have seen contrail-api got stuck (not responding on time), so restarted contrail-api - then on, one of the port is not getting deleted - its instance-ip deletion is causing exception because its virtual_network_refs are recorded as "to: [ERROR]" rather than id (see below output).

It is three node setup with three control nodes as VMs (three VMs). But I only enabled harishku-vm3 for config api for easy troubleshooting. contrail version - 2.21-105

curl -u admin:secret http://localhost:8095/instance-ip/04c016bd-b9c1-412e-9eed-6ee09504f3b4 ; echo
{"instance-ip": {"display_name": "default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left", "uuid": "04c016bd-b9c1-412e-9eed-6ee09504f3b4", "instance_ip_address": "100.64.0.4", "virtual_machine_interface_refs": [{"to": ["default-domain", "c_rally_d68a4cf1_ZGNDTqOJ", "default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9__2__left__2"], "href": "http://localhost:8095/virtual-machine-interface/a4de7e38-872c-4aab-9a04-b24dbe6f3ae3", "attr": null, "uuid": "a4de7e38-872c-4aab-9a04-b24dbe6f3ae3"}], "instance_ip_mode": "active-standby", "href": "http://localhost:8095/instance-ip/04c016bd-b9c1-412e-9eed-6ee09504f3b4", "id_perms": {"enable": true, "description": null, "created": "2015-12-01T09:36:35.060840", "creator": null, "uuid": {"uuid_mslong": 342298575801237806, "uuid_lslong": 11451931338324898740}, "user_visible": true, "last_modified": "2015-12-01T10:23:31.651271", "permissions": {"owner": "cloud-admin", "owner_access": 7, "other_access": 7, "group": "cloud-admin-group", "group_access": 7}}, "fq_name": ["default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left"], "virtual_network_refs": [{"to": ["ERROR"], "href": "http://localhost:8095/virtual-network/9f0f9931-1f21-4152-ba4f-51a38c28f703", "attr": null, "uuid": "9f0f9931-1f21-4152-ba4f-51a38c28f703"}, {"to": ["default-domain", "c_rally_d68a4cf1_ZGNDTqOJ", "snat-si-left_si_6840042f-351b-4726-93d4-8bd4f41848d9"], "href": "http://localhost:8095/virtual-network/faf035ec-9bc6-4abd-9644-ee53a9339dbe", "attr": null, "uuid": "faf035ec-9bc6-4abd-9644-ee53a9339dbe"}], "name": "default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left"}}

See original description

Revision history for this message

Harish Kumar (hkumarmk) wrote on 2015-12-02:

Logs available at http://10.204.216.50/Docs/bugs/1521853/

Harish Kumar (hkumarmk) on 2015-12-02

description:

updated

Vedamurthy Joshi (vedujoshi) on 2015-12-02

information type:

Proprietary → Public

Revision history for this message

Sachin Bansal (sbansal) wrote on 2017-05-12:

Download full text (11.5 KiB)

ERROR:harishku-vm1:contrail-api:Config:0:__default__ [SYS_ERR]: VncApiError: <class 'cfgm_common.exceptions.NoIdError'>
Python 2.7.6: /usr/bin/python
Tue Dec 1 15:10:23 2015

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in handler_trap_exception(*args=(), **kwargs={'id': '04c016bd-b9c1-412e-9eed-6ee09504f3b4'})
  534
  535 trace = self._generate_rest_api_request_trace()
  536 response = handler(*args, **kwargs)
  537 self._generate_rest_api_response_trace(trace, response)
  538
response undefined
handler = <bound method VncApiServer.instance_ip_http_dele...i_server.vnc_cfg_api_server.VncApiServer object>>
args = ()
kwargs = {'id': '04c016bd-b9c1-412e-9eed-6ee09504f3b4'}

/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py in instance_ip_http_delete(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>, id='04c016bd-b9c1-412e-9eed-6ee09504f3b4')
2379 self._instance_ip_delete_default_children(read_result)
2380
2381 (ok, del_result) = r_class.http_delete(id, read_result, db_conn)
2382 if not ok:
2383 (code, msg) = del_result
ok = True
del_result = ''
r_class = <class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>
r_class.http_delete = <bound method type.http_delete of <class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>>
id = '04c016bd-b9c1-412e-9eed-6ee09504f3b4'
read_result = {'display_name': u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left', 'fq_name': [u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left'], 'id_perms': {u'created': u'2015-12-01T09:36:35.060840', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2015-12-01T09:36:39.131312', u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 11451931338324898740L, u'uuid_mslong': 342298575801237806}}, 'instance_ip_address': u'100.64.0.4', 'instance_ip_mode': u'active-standby', 'uuid': '04c016bd-b9c1-412e-9eed-6ee09504f3b4', 'virtual_machine_interface_refs': [{'attr': None, 'href': 'http://10.204.217.89:8082/virtual-machine-interface/a4de7e38-872c-4aab-9a04-b24dbe6f3ae3', 'to': [u'default-domain', u'c_rally_d68a4cf1_ZGNDTqOJ', u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9__2__left__2'], 'uuid': 'a4de7e38-872c-4aab-9a04-b24dbe6f3ae3'}, {'attr': None, 'href': 'http://10.204.217.89:8082/virtual-machine-interface/caf74340-7ce3-4f8d-bbdd-9292416aeb4b', 'to': [u'default-domain', u'c_rally_d68a4cf1_ZGNDTqOJ', u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9__1__left__2'], 'uuid': 'caf74340-7ce3-4f8d-bbdd-9292416aeb4b'}], 'virtual_network_refs': [{'attr': None, 'href': 'http://10.204.217.89:8082/virtual-network/9f0f9931-1f21-4152-ba4f-51a38c28f703', 'to': ['ERRO...

ERROR:harishku-vm1:contrail-api:Config:0:__default__ [SYS_ERR]: VncApiError: <class 'cfgm_common.exceptions.NoIdError'>
Python 2.7.6: /usr/bin/python
Tue Dec  1 15:10:23 2015

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in handler_trap_exception(*args=(), **kwargs={'id': '04c016bd-b9c1-412e-9eed-6ee09504f3b4'})
  534 
  535                 trace = self._generate_rest_api_request_trace()
  536                 response = handler(*args, **kwargs)
  537                 self._generate_rest_api_response_trace(trace, response)
  538 
response undefined
handler = <bound method VncApiServer.instance_ip_http_dele...i_server.vnc_cfg_api_server.VncApiServer object>>
args = ()
kwargs = {'id': '04c016bd-b9c1-412e-9eed-6ee09504f3b4'}

/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py in instance_ip_http_delete(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>, id='04c016bd-b9c1-412e-9eed-6ee09504f3b4')
 2379                 self._instance_ip_delete_default_children(read_result)
 2380 
 2381                 (ok, del_result) = r_class.http_delete(id, read_result, db_conn)
 2382                 if not ok:
 2383                     (code, msg) = del_result
ok = True
del_result = ''
r_class = <class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>
r_class.http_delete = <bound method type.http_delete of <class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>>
id = '04c016bd-b9c1-412e-9eed-6ee09504f3b4'
read_result = {'display_name': u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left', 'fq_name': [u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left'], 'id_perms': {u'created': u'2015-12-01T09:36:35.060840', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2015-12-01T09:36:39.131312', u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 11451931338324898740L, u'uuid_mslong': 342298575801237806}}, 'instance_ip_address': u'100.64.0.4', 'instance_ip_mode': u'active-standby', 'uuid': '04c016bd-b9c1-412e-9eed-6ee09504f3b4', 'virtual_machine_interface_refs': [{'attr': None, 'href': 'http://10.204.217.89:8082/virtual-machine-interface/a4de7e38-872c-4aab-9a04-b24dbe6f3ae3', 'to': [u'default-domain', u'c_rally_d68a4cf1_ZGNDTqOJ', u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9__2__left__2'], 'uuid': 'a4de7e38-872c-4aab-9a04-b24dbe6f3ae3'}, {'attr': None, 'href': 'http://10.204.217.89:8082/virtual-machine-interface/caf74340-7ce3-4f8d-bbdd-9292416aeb4b', 'to': [u'default-domain', u'c_rally_d68a4cf1_ZGNDTqOJ', u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9__1__left__2'], 'uuid': 'caf74340-7ce3-4f8d-bbdd-9292416aeb4b'}], 'virtual_network_refs': [{'attr': None, 'href': 'http://10.204.217.89:8082/virtual-network/9f0f9931-1f21-4152-ba4f-51a38c28f703', 'to': ['ERROR'], 'uuid': '9f0f9931-1f21-4152-ba4f-51a38c28f703'}, {'attr': None, 'href': 'http://10.204.217.89:8082/virtual-network/faf035ec-9bc6-4abd-9644-ee53a9339dbe', 'to': [u'default-domain', u'c_rally_d68a4cf1_ZGNDTqOJ', u'snat-si-left_si_6840042f-351b-4726-93d4-8bd4f41848d9'], 'uuid': 'faf035ec-9bc6-4abd-9644-ee53a9339dbe'}]}
db_conn = <vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>

/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_types.py in http_delete(cls=<class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>, id='04c016bd-b9c1-412e-9eed-6ee09504f3b4', obj_dict={'display_name': u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left', 'fq_name': [u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left'], 'id_perms': {u'created': u'2015-12-01T09:36:35.060840', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2015-12-01T09:36:39.131312', u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 11451931338324898740L, u'uuid_mslong': 342298575801237806}}, 'instance_ip_address': u'100.64.0.4', 'instance_ip_mode': u'active-standby', 'uuid': '04c016bd-b9c1-412e-9eed-6ee09504f3b4', 'virtual_machine_interface_refs': [{'attr': None, 'href': 'http://10.204.217.89:8082/virtual-machine-interface/a4de7e38-872c-4aab-9a04-b24dbe6f3ae3', 'to': [u'default-domain', u'c_rally_d68a4cf1_ZGNDTqOJ', u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9__2__left__2'], 'uuid': 'a4de7e38-872c-4aab-9a04-b24dbe6f3ae3'}, {'attr': None, 'href': 'http://10.204.217.89:8082/virtual-machine-interface/caf74340-7ce3-4f8d-bbdd-9292416aeb4b', 'to': [u'default-domain', u'c_rally_d68a4cf1_ZGNDTqOJ', u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9__1__left__2'], 'uuid': 'caf74340-7ce3-4f8d-bbdd-9292416aeb4b'}], 'virtual_network_refs': [{'attr': None, 'href': 'http://10.204.217.89:8082/virtual-network/9f0f9931-1f21-4152-ba4f-51a38c28f703', 'to': ['ERROR'], 'uuid': '9f0f9931-1f21-4152-ba4f-51a38c28f703'}, {'attr': None, 'href': 'http://10.204.217.89:8082/virtual-network/faf035ec-9bc6-4abd-9644-ee53a9339dbe', 'to': [u'default-domain', u'c_rally_d68a4cf1_ZGNDTqOJ', u'snat-si-left_si_6840042f-351b-4726-93d4-8bd4f41848d9'], 'uuid': 'faf035ec-9bc6-4abd-9644-ee53a9339dbe'}]}, db_conn=<vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>)
  277                            % (ip_addr, vn_fq_name),
  278                            level=SandeshLevel.SYS_DEBUG)
  279         cls.addr_mgmt.ip_free_req(ip_addr, vn_fq_name)
  280         return True, ""
  281     # end http_delete
cls = <class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>
cls.addr_mgmt = <vnc_cfg_api_server.vnc_addr_mgmt.AddrMgmt object>
cls.addr_mgmt.ip_free_req = <bound method AddrMgmt.ip_free_req of <vnc_cfg_api_server.vnc_addr_mgmt.AddrMgmt object>>
ip_addr = u'100.64.0.4'
vn_fq_name = ['ERROR']

/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_addr_mgmt.py in ip_free_req(self=<vnc_cfg_api_server.vnc_addr_mgmt.AddrMgmt object>, ip_addr=u'100.64.0.4', vn_fq_name=['ERROR'], sub=None)
  867     def ip_free_req(self, ip_addr, vn_fq_name, sub=None):
  868         vn_fq_name_str = ':'.join(vn_fq_name)
  869         subnet_dicts = self._get_subnet_dicts(vn_fq_name)
  870         for subnet_name in subnet_dicts:
  871             if sub and sub != subnet_name:
subnet_dicts undefined
self = <vnc_cfg_api_server.vnc_addr_mgmt.AddrMgmt object>
self._get_subnet_dicts = <bound method AddrMgmt._get_subnet_dicts of <vnc_cfg_api_server.vnc_addr_mgmt.AddrMgmt object>>
vn_fq_name = ['ERROR']

/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_addr_mgmt.py in _get_subnet_dicts(self=<vnc_cfg_api_server.vnc_addr_mgmt.AddrMgmt object>, vn_fq_name=['ERROR'], vn_dict=None)
  354         # Read in the VN details if not passed in
  355         if not vn_dict:
  356             vn_uuid = db_conn.fq_name_to_uuid('virtual-network', vn_fq_name)
  357             (ok, result) = self._db_conn.dbe_read(
  358                                 obj_type='virtual-network',
vn_uuid undefined
db_conn = <vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>
db_conn.fq_name_to_uuid = <bound method VncDbClient.fq_name_to_uuid of <vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>>
vn_fq_name = ['ERROR']

/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in fq_name_to_uuid(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>, obj_type='virtual-network', fq_name=['ERROR'])
 1503 
 1504     def fq_name_to_uuid(self, obj_type, fq_name):
 1505         obj_uuid = self._cassandra_db.fq_name_to_uuid(obj_type, fq_name)
 1506         return obj_uuid
 1507     # end fq_name_to_uuid
obj_uuid undefined
self = <vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>
self._cassandra_db = <vnc_cfg_api_server.vnc_cfg_ifmap.VncServerCassandraClient object>
self._cassandra_db.fq_name_to_uuid = <bound method VncServerCassandraClient.fq_name_t...r.vnc_cfg_ifmap.VncServerCassandraClient object>>
obj_type = 'virtual-network'
fq_name = ['ERROR']

/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py in fq_name_to_uuid(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncServerCassandraClient object>, obj_type='virtual-network', fq_name=['ERROR'])
  275 
  276         if len(col_infos) == 0:
  277             raise NoIdError('%s %s' % (obj_type, fq_name))
  278 
  279         for (col_name, col_val) in col_infos:
global NoIdError = <class 'cfgm_common.exceptions.NoIdError'>
obj_type = 'virtual-network'
fq_name = ['ERROR']
<class 'cfgm_common.exceptions.NoIdError'>: Unknown id: virtual-network ['ERROR']
    __class__ = <class 'cfgm_common.exceptions.NoIdError'>
    __delattr__ = <method-wrapper '__delattr__' of NoIdError object>
    __dict__ = {'_unknown_id': "virtual-network ['ERROR']"}
    __doc__ = None
    __format__ = <built-in method __format__ of NoIdError object>
    __getattribute__ = <method-wrapper '__getattribute__' of NoIdError object>
    __getitem__ = <method-wrapper '__getitem__' of NoIdError object>
    __getslice__ = <method-wrapper '__getslice__' of NoIdError object>
    __hash__ = <method-wrapper '__hash__' of NoIdError object>
    __init__ = <bound method NoIdError.__init__ of NoIdError()>
    __module__ = 'cfgm_common.exceptions'
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of NoIdError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of NoIdError object>
    __repr__ = <method-wrapper '__repr__' of NoIdError object>
    __setattr__ = <method-wrapper '__setattr__' of NoIdError object>
    __setstate__ = <built-in method __setstate__ of NoIdError object>
    __sizeof__ = <built-in method __sizeof__ of NoIdError object>
    __str__ = <bound method NoIdError.__str__ of NoIdError()>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of NoIdError object>
    __weakref__ = None
    _unknown_id = "virtual-network ['ERROR']"
    args = ()
    message = ''

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 536, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py", line 2381, in instance_ip_http_delete
    (ok, del_result) = r_class.http_delete(id, read_result, db_conn)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_types.py", line 279, in http_delete
    cls.addr_mgmt.ip_free_req(ip_addr, vn_fq_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_addr_mgmt.py", line 869, in ip_free_req
    subnet_dicts = self._get_subnet_dicts(vn_fq_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_addr_mgmt.py", line 356, in _get_subnet_dicts
    vn_uuid = db_conn.fq_name_to_uuid('virtual-network', vn_fq_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1505, in fq_name_to_uuid
    obj_uuid = self._cassandra_db.fq_name_to_uuid(obj_type, fq_name)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 277, in fq_name_to_uuid
    raise NoIdError('%s %s' % (obj_type, fq_name))
NoIdError: Unknown id: virtual-network ['ERROR']

Changed in juniperopenstack:
assignee:	Hampapur Ajay (hajay) → Atul Moghe (moghea)

Revision history for this message

Atul Moghe (moghea) wrote on 2017-08-03:

Assigning to Jeba to reproduce with some later branches like 3.2 or 4.0/mainline.. 2.21 logs may not give much clue..

Changed in juniperopenstack:
assignee:	Atul Moghe (moghea) → Jeba Paulaiyan (jebap)

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.