Nova cannot handle BadRequest or a Vendor Specific Exceptions

Bug #1481977 reported by Prinika
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Low
Unassigned

Bug Description

When a different process say neutron sends an exception to nova that Exception is not handled by nova and the error message is not relayed correctly to the cli.

For example;
I tried the following command failed due to a restriction:

"nova interface-attach vm --net <net-id>

Expected Result:
What was expected when this command failed was an error message as follows " VM with ports belonging to subnets across enter prises is not allowed in VSP"

Actual Result:
However, instead this message was relayed back to the cli "ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-fd69f004-591c-4ea8-b7b6-088a74cbe518)"

Nova-api logs:

The ERROR seen on nova-api is as follows:
2015-08-05 23:30:42.116 INFO oslo_messaging._drivers.impl_rabbit [req-fd69f004-591c-4ea8-b7b6-088a74cbe518 admin admin] Connected to AMQP server on 10.31.101.215:5672
2015-08-05 23:30:43.035 ERROR nova.api.openstack [req-fd69f004-591c-4ea8-b7b6-088a74cbe518 admin admin] Caught error: Remote error: BadRequest Bad request: VM with ports belonging to subnets across enter prises is not allowed in VSP
[u'Traceback (most recent call last):\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', u' File "/usr/
local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
 line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 6602, in attach_interface\n port_id, requested_ip)\n', u' File "/opt/stack
/nova/nova/compute/manager.py", line 443, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/opt/stack/nova/nova/exception.py", line 88, in wrapped\n payload)\n'
, u' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n ctxt.reraise = False\n', u' File "/opt/stack/nova/nova/exception.py", line 71, in wrapped\n retur
n f(self, context, *args, **kw)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 327, in decorated_function\n LOG.warning(msg, e, instance_uuid=instance_uuid)\n', u' File "/usr/local/lib/
python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n ctxt.reraise = False\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 298, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 355, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n ctxt.reraise = False\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 343, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 5024, in attach_interface\n context, instance, port_id, network_id, requested_ip)\n', u' File "/opt/stack/nova/nova/network/neutronv2/api.py", line 705, in allocate_port_for_instance\n requested_networks=requested_networks)\n', u' File "/opt/stack/nova/nova/network/neutronv2/api.py", line 577, in allocate_for_instance\n self._delete_ports(neutron, instance, created_port_ids)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n ctxt.reraise = False\n', u' File "/opt/stack/nova/nova/network/neutronv2/api.py", line 569, in allocate_for_instance\n security_group_ids, available_macs, dhcp_opts)\n', u' File "/opt/stack/nova/nova/network/neutronv2/api.py", line 331, in _create_port\n network_id, instance=instance)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n ctxt.reraise = False\n', u' File "/opt/stack/nova/nova/network/neutronv2/api.py", line 304, in _create_port\n port_id = port_client.create_port(port_req_body)[\'port\'][\'id\']\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102, in with_params\n ret = self.function(instance, *args, **kwargs)\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 544, in create_port\n return self.post(self.ports_path, body=body)\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in post\n headers=headers, params=params)\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 211, in do_request\n self._handle_fault_response(status_code, replybody)\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response\n exception_handler_v20(status_code, des_error_body)\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 70, in exception_handler_v20\n status_code=status_code)\n', u'BadRequest: Bad request: VM with ports belonging to subnets across enterprises is not allowed in VSP\n'].
2015-08-05 23:30:43.035 TRACE nova.api.openstack Traceback (most recent call last):
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
2015-08-05 23:30:43.035 TRACE nova.api.openstack return req.get_response(self.application)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-08-05 23:30:43.035 TRACE nova.api.openstack application, catch_exc_info=False)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-08-05 23:30:43.035 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-08-05 23:30:43.035 TRACE nova.api.openstack return resp(environ, start_response)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
2015-08-05 23:30:43.035 TRACE nova.api.openstack return self._call_app(env, start_response)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
2015-08-05 23:30:43.035 TRACE nova.api.openstack return self._app(env, _fake_start_response)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-08-05 23:30:43.035 TRACE nova.api.openstack return resp(environ, start_response)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-08-05 23:30:43.035 TRACE nova.api.openstack return resp(environ, start_response)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2015-08-05 23:30:43.035 TRACE nova.api.openstack response = self.app(environ, start_response)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-08-05 23:30:43.035 TRACE nova.api.openstack return resp(environ, start_response)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-08-05 23:30:43.035 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-08-05 23:30:43.035 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 756, in __call__
2015-08-05 23:30:43.035 TRACE nova.api.openstack content_type, body, accept)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack
2015-08-05 23:30:43.035 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 911, in dispatch
2015-08-05 23:30:43.035 TRACE nova.api.openstack return method(req=request, **action_args)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/contrib/attach_interfaces.py", line 122, in create
2015-08-05 23:30:43.035 TRACE nova.api.openstack instance, network_id, port_id, req_ip)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 230, in wrapped
2015-08-05 23:30:43.035 TRACE nova.api.openstack return func(self, context, target, *args, **kwargs)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 219, in inner
2015-08-05 23:30:43.035 TRACE nova.api.openstack return function(self, context, instance, *args, **kwargs)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 200, in inner
2015-08-05 23:30:43.035 TRACE nova.api.openstack return f(self, context, instance, *args, **kw)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 3108, in attach_interface
2015-08-05 23:30:43.035 TRACE nova.api.openstack requested_ip=requested_ip)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/rpcapi.py", line 355, in attach_interface
2015-08-05 23:30:43.035 TRACE nova.api.openstack port_id=port_id, requested_ip=requested_ip)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-08-05 23:30:43.035 TRACE nova.api.openstack retry=self.retry)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-08-05 23:30:43.035 TRACE nova.api.openstack timeout=timeout, retry=retry)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-08-05 23:30:43.035 TRACE nova.api.openstack retry=retry)
2015-08-05 23:30:43.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-08-05 23:30:43.035 TRACE nova.api.openstack raise result
2015-08-05 23:30:43.035 TRACE nova.api.openstack RemoteError: Remote error: BadRequest Bad request: VM with ports belonging to subnets across enterprises is not allowed in VSP
2015-08-05 23:30:43.035 TRACE nova.api.openstack [u'Traceback (most recent call last):\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 6602, in attach_interface\n port_id, requested_ip)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 443, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/opt/stack/nova/nova/exception.py", line 88, in wrapped\n payload)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n ctxt.reraise = False\n', u' File "/opt/stack/nova/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 327, in decorated_function\n LOG.warning(msg, e, instance_uuid=instance_uuid)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n ctxt.reraise = False\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 298, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 355, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n ctxt.reraise = False\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 343, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 5024, in attach_interface\n context, instance, port_id, network_id, requested_ip)\n', u' File "/opt/stack/nova/nova/network/neutronv2/api.py", line 705, in allocate_port_for_instance\n requested_networks=requested_networks)\n', u' File "/opt/stack/nova/nova/network/neutronv2/api.py", line 577, in allocate_for_instance\n self._delete_ports(neutron, instance, created_port_ids)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n ctxt.reraise = False\n', u' File "/opt/stack/nova/nova/network/neutronv2/api.py", line 569, in allocate_for_instance\n security_group_ids, available_macs, dhcp_opts)\n', u' File "/opt/stack/nova/nova/network/neutronv2/api.py", line 331, in _create_port\n network_id, instance=instance)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n ctxt.reraise = False\n', u' File "/opt/stack/nova/nova/network/neutronv2/api.py", line 304, in _create_port\n port_id = port_client.create_port(port_req_body)[\'port\'][\'id\']\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102, in with_params\n ret = self.function(instance, *args, **kwargs)\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 544, in create_port\n return self.post(self.ports_path, body=body)\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in post\n headers=headers, params=params)\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 211, in do_request\n self._handle_fault_response(status_code, replybody)\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response\n exception_handler_v20(status_code, des_error_body)\n', u' File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 70, in exception_handler_v20\n status_code=status_code)\n', u'BadRequest: Bad request: VM with ports belonging to subnets across enterprises is not allowed in VSP\n'].

Tags: api neutron
Changed in nova:
assignee: nobody → Chung Chih, Hung (lyanchih)
tags: added: api
Revision history for this message
jichenjc (jichenjc) wrote :

we had a lot of code in network layer to handle neutron client error and translate it
this one should be another one need to do so

Changed in nova:
status: New → Confirmed
importance: Undecided → Low
tags: added: network
Revision history for this message
Chung Chih, Hung (lyanchih) wrote :

I can't find which kind of exception was thrown.
Can you offer neutron api error message
Your can add debug option in cli, just like following
> nova interface-attach vm --net <net-id>
Response should contain type and message
Thanks.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/242861

Changed in nova:
assignee: Chung Chih, Hung (lyanchih) → Davanum Srinivas (DIMS) (dims-v)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Davanum Srinivas (dims) (<email address hidden>) on branch: master
Review: https://review.openstack.org/242861

Changed in nova:
assignee: Davanum Srinivas (DIMS) (dims-v) → Chirag Shahani (chirag-shahani)
Revision history for this message
Chirag Shahani (chirag-shahani) wrote :

Tried to attach the interface accoss enterprise using the Nuage Plugin.

Before the patch:

stack@stack-ubuntu:/opt/stack/nova$ nova interface-attach vm --net 535d9379-0306-4852-b8f9-05ae0b3d262e
(ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-fd69f004-591c-4ea8-b7b6-088a74cbe518)

After the patch:

stack@stack-ubuntu:/opt/stack/nova$ nova interface-attach vm --net 535d9379-0306-4852-b8f9-05ae0b3d262e
ERROR (BadRequest): Neutron error creating port on network 535d9379-0306-4852-b8f9-05ae0b3d262e: Bad request: VM with ports belonging to subnets across enterprises is not allowed in VSP (HTTP 400) (Request-ID: req-fbf83ccf-04f7-48d2-8ce6-98a44ddeb71f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/278788

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/278788
Reason: This code hasn't been updated in a long time, and is in merge conflict. I am going to abandon this review, but feel free to restore it if you're still working on this.

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

Cleanup
=======

There are no open reviews for this bug report since more than 2 weeks.
To signal that to other contributors which might provide patches for
this bug, I switch the status from "In Progress" to "Confirmed" and
remove the assignee.
Feel free to add yourself as assignee and to push a review for it.

Changed in nova:
assignee: Chirag Shahani (chirag-shahani) → nobody
status: In Progress → Confirmed
tags: added: low-hanging-fruit
Changed in nova:
assignee: nobody → omkar_telee (omkar-telee)
Revision history for this message
omkar_telee (omkar-telee) wrote :

Started working on it

Changed in nova:
status: Confirmed → In Progress
Changed in nova:
assignee: omkar_telee (omkar-telee) → nobody
status: In Progress → Confirmed
Changed in nova:
assignee: nobody → Hesam Chobanlou (hesamchobanlou)
Changed in nova:
assignee: Hesam Chobanlou (hesamchobanlou) → nobody
Peter Kendall (halprin)
Changed in nova:
assignee: nobody → Peter Kendall (halprin)
status: Confirmed → In Progress
Peter Kendall (halprin)
Changed in nova:
assignee: Peter Kendall (halprin) → nobody
status: In Progress → Confirmed
Changed in nova:
assignee: nobody → Sean Dague (sdague)
status: Confirmed → In Progress
Matt Riedemann (mriedem)
Changed in nova:
assignee: Sean Dague (sdague) → nobody
status: In Progress → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/278788
Reason: The bug looks legit but the fix does not, and is old and in merge conflict so I'm going to abandon this.

Matt Riedemann (mriedem)
tags: added: neutron
removed: low-hanging-fruit network
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.