Juniper Openstack

VN Deletion fails due to stale open ports.

Series r3.1
Bug #1445899

Bug #1445899 reported by Ritam Gangopadhyay on 2015-04-19

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R2.20	Won't Fix	Medium	Hampapur Ajay	Juniper Openstack r2.20-fcs
R3.0	Incomplete	Medium	Ritam Gangopadhyay
R3.1	Incomplete	Medium	Ritam Gangopadhyay
Trunk	Incomplete	Medium	Ritam Gangopadhyay

Bug Description

2015-04-18 22:55:30,630 - ERROR - Quantum exception while deleting a VN 1f1e5bf7-d157-428c-96b8-7c5475c61c91
Traceback (most recent call last):
  File "/root/contrail-test/fixtures/quantum_test.py", line 282, in delete_vn
    net_rsp = self.obj.delete_network(vn_id)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 111, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 361, in delete_network
    return self.delete(self.network_path % (network))
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1379, in delete
    headers=headers, params=params)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1368, in retry_request
    headers=headers, params=params)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1311, in do_request
    self._handle_fault_response(status_code, replybody)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1281, in _handle_fault_response
    exception_handler_v20(status_code, des_error_body)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 81, in exception_handler_v20
    raise ex
NetworkInUseClient: Unable to complete operation on network 1f1e5bf7-d157-428c-96b8-7c5475c61c91. There are one or more ports still in use on the network.

Nova logs:-

2015-04-18 22:54:54.369 28931 INFO nova.osapi_compute.wsgi.server [req-66d16937-82a0-4c02-bccf-c59e84d0451a 33cbb0c44e404a989af69773086e7943 d85cba1f25b1422c876125329a2a9502] 22.22.22.35 "GET /v1.1/d85cba1f25b1422c876125329a2a9502/servers/ac99cf9a-de7e-4c78-99e5-f27ff1b8c1c9 HTTP/1.1" status: 200 len: 1931 time: 0.1411300
2015-04-18 22:54:54.523 28931 ERROR nova.api.openstack [req-3c7d65d9-3219-46d5-9907-870bbdc3761f 33cbb0c44e404a989af69773086e7943 d85cba1f25b1422c876125329a2a9502] Caught error: 500-{u'NeutronError': {u'message': u'An unknown exception occurred.', u'type': u'NeutronException', u'detail': u''}}
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack Traceback (most recent call last):
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack return req.get_response(self.application)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack application, catch_exc_info=False)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack return resp(environ, start_response)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack return self.app(env, start_response)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack return resp(environ, start_response)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack return resp(environ, start_response)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack response = self.app(environ, start_response)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack return resp(environ, start_response)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 917, in __call__
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack content_type, body, accept)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1009, in _process_stack
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack request, action_args)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 887, in post_process_extensions
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack **action_args)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 590, in show
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack return self._show(req, resp_obj)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 586, in _show
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack self._extend_servers(req, [resp_obj.obj['server']])
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 550, in _extend_servers
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack servers))
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 345, in get_instances_security_groups_bindings
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack ports = self._get_ports_from_server_list(servers, neutron)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 304, in _get_ports_from_server_list
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack ports.extend(neutron.list_ports(**search_opts).get('ports'))
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 111, in with_params
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack ret = self.function(instance, *args, **kwargs)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 314, in list_ports
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack **_params)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1397, in list
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack for r in self._pagination(collection, path, **params):
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1410, in _pagination
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack res = self.get(path, params=params)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1383, in get
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack headers=headers, params=params)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1368, in retry_request
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack headers=headers, params=params)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1311, in do_request
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack self._handle_fault_response(status_code, replybody)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1281, in _handle_fault_response
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack exception_handler_v20(status_code, des_error_body)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 96, in exception_handler_v20
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack message=msg)
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack NeutronClientException: 500-{u'NeutronError': {u'message': u'An unknown exception occurred.', u'type': u'NeutronException', u'detail': u''}}
2015-04-18 22:54:54.523 28931 TRACE nova.api.openstack
2015-04-18 22:54:54.526 28931 INFO nova.api.openstack [req-3c7d65d9-3219-46d5-9907-870bbdc3761f 33cbb0c44e404a989af69773086e7943 d85cba1f25b1422c876125329a2a9502] http://22.22.22.34:8774/v1.1/d85cba1f25b1422c876125329a2a9502/servers/ac99cf9a-de7e-4c78-99e5-f27ff1b8c1c9 returned with HTTP 500
2015-04-18 22:54:54.527 28931 INFO nova.osapi_compute.wsgi.server [req-3c7d65d9-3219-46d5-9907-870bbdc3761f 33cbb0c44e404a989af69773086e7943 d85cba1f25b1422c876125329a2a9502] 22.22.22.35 "GET /v1.1/d85cba1f25b1422c876125329a2a9502/servers/ac99cf9a-de7e-4c78-99e5-f27ff1b8c1c9 HTTP/1.1" status: 500 len: 335 time: 0.1552291
2015-04-18 22:54:54.651 28931 INFO nova.osapi_compute.wsgi.server [req-7f444a0e-611a-4197-bc35-989b5cd55f01 33cbb0c44e404a989af69773086e7943 d85cba1f25b1422c876125329a2a9502] 22.22.22.35 "DELETE /v1.1/d85cba1f25b1422c876125329a2a9502/servers/fe6d7227-b2a7-4cee-923f-2e5ae61f8c68 HTTP/1.1" status: 204 len: 179 time: 0.1101139

Tags:

Nagabhushana R (bhushana) on 2015-04-20

tags:

added: config

OpenContrail Admin (ci-admin-f) on 2015-04-23

Changed in juniperopenstack:
assignee:	nobody → Rahul (rahuls)
importance:	High → Medium

Revision history for this message

Hampapur Ajay (hajay) wrote on 2015-11-24:

I would need the log files for /var/log/contrail/contrail-api* /var/log/contrail/api-0-zk* /var/log/neutron/server.log* from all controllers to analyze further.

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-19: [Bug update]

bug update...

Ashish Ranjan (aranjan-n) on 2016-08-04

information type:

Proprietary → Public

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.