OpenStack Compute (nova)

PortLimitExceeded results in 500 error from POST /servers/{server_id}/os-interface

  1. Series rocky
  2. Bug #1823203
Bug #1823203 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Matt Riedemann
Pike
Confirmed
Medium
Unassigned
Queens
Confirmed
Medium
Unassigned
Rocky
Confirmed
Medium
Unassigned
Stein
Fix Committed
Medium
Matt Riedemann

Bug Description

This is based on code inspection at the moment, but attaching an interface (port) to a server by providing a network and having nova-compute create the port on that network can result in an over-quota failure from neutron:

https://github.com/openstack/nova/blob/6ebb2c4cae65cb437e17a8c02fe5174a9825d8e0/nova/network/neutronv2/api.py#L565

Which is not handled in the API controller:

https://github.com/openstack/nova/blob/6ebb2c4cae65cb437e17a8c02fe5174a9825d8e0/nova/api/openstack/compute/attach_interfaces.py#L168

So it will result in a 500 response from the API when it should really be something like 403.

Tags: api neutron
Revision history for this message
Matt Riedemann (mriedem) wrote :

It should be relatively easy to recreate this in devstack by setting [quotas]/quota_port=1 in neutron.conf, create a server and then try to attach a new port to the server via the same network.

Revision history for this message
Matt Riedemann (mriedem) wrote :
Download full text (4.7 KiB)

Yup I was able to easily recreate this in devstack:

g [None req-124eeb28-2f0f-4b47-b2c0-3889d65dd931 admin admin] 199.204.45.68 "POST /compute/v2.1/servers/8e90f1f0-e8dd-4783-8bb3-ec8d594e60f1/os-interface" status: 500 len: 1
9.204.45.68 () {64 vars in 1385 bytes} [Thu Apr 4 18:23:37 2019] POST /compute/v2.1/servers/8e90f1f0-e8dd-4783-8bb3-ec8d594e60f1/os-interface => generated 198 bytes in 2046
 ESCOD
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/manager.py", line 201, in decorated_function
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi return function(self, context, *args, **kwargs)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/manager.py", line 6081, in attach_interface
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi bind_host_id=bind_host_id, tag=tag)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1697, in allocate_port_for_in
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi bind_host_id=bind_host_id, attach=True)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1106, in allocate_for_instanc
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi security_group_ids)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1013, in _create_ports_for_in
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi neutron, instance, created_port_ids)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi self.force_reraise()
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi six.reraise(self.type_, self.value, self.tb)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.a...

Read more...

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/650097

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/650097
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8ff0fae42bce9f98fd5897888c3c4c739b4b5f45
Submitter: Zuul
Branch: master

commit 8ff0fae42bce9f98fd5897888c3c4c739b4b5f45
Author: Matt Riedemann <email address hidden>
Date: Thu Apr 4 14:44:02 2019 -0400

    Handle PortLimitExceeded in POST /servers/{server_id}/os-interface

    When attaching an interface to a server, if an existing port is not
    specified, nova-compute will attempt to create a port on either the
    user-specified network or the network that is available to the tenant.

    If the tenant exceeds their port quota in neutron, a PortLimitExceeded
    exception is raised up from nova-compute [1] which is not being handled
    in the API controller code - which is fixed in this change.

    Note that this is one of the few synchronous RPC call operations [2]
    in the compute API so exceptions from the compute service will leak
    back to the API like in this case and need to be handled to avoid a
    500 response to the user.

    The 403 response used here matches how PortLimitExceeded is handled in
    the server create API [3].

    [1] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/network/neutronv2/api.py#L565
    [2] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/compute/rpcapi.py#L489
    [3] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/api/openstack/compute/servers.py#L688

    Change-Id: I5687480a22542eea31d299442837bd64bf731285
    Closes-Bug: #1823203

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/665513

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/stein)

Reviewed: https://review.opendev.org/665513
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=43926ebca0f9a7f59be96e965e5be70b84501fa3
Submitter: Zuul
Branch: stable/stein

commit 43926ebca0f9a7f59be96e965e5be70b84501fa3
Author: Matt Riedemann <email address hidden>
Date: Thu Apr 4 14:44:02 2019 -0400

    Handle PortLimitExceeded in POST /servers/{server_id}/os-interface

    When attaching an interface to a server, if an existing port is not
    specified, nova-compute will attempt to create a port on either the
    user-specified network or the network that is available to the tenant.

    If the tenant exceeds their port quota in neutron, a PortLimitExceeded
    exception is raised up from nova-compute [1] which is not being handled
    in the API controller code - which is fixed in this change.

    Note that this is one of the few synchronous RPC call operations [2]
    in the compute API so exceptions from the compute service will leak
    back to the API like in this case and need to be handled to avoid a
    500 response to the user.

    The 403 response used here matches how PortLimitExceeded is handled in
    the server create API [3].

    [1] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/network/neutronv2/api.py#L565
    [2] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/compute/rpcapi.py#L489
    [3] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/api/openstack/compute/servers.py#L688

    Change-Id: I5687480a22542eea31d299442837bd64bf731285
    Closes-Bug: #1823203
    (cherry picked from commit 8ff0fae42bce9f98fd5897888c3c4c739b4b5f45)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.2

This issue was fixed in the openstack/nova 19.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 20.0.0.0rc1

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.