OpenStack Compute (nova)

ExternalNetworkAttachForbidden should result in BuildAbortException, not reschedule

  1. Series stein
  2. Bug #1852465
Bug #1852465 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Matt Riedemann
Rocky
Confirmed
Low
Unassigned
Stein
Confirmed
Low
Unassigned
Train
Confirmed
Low
Unassigned

Bug Description

I saw this in a CI run where creating a server on an external network as a non-admin user failed and was rescheduled. Here are the failures from the compute logs:

https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n-cpu.txt.gz#28774

https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/compute1/logs/screen-n-cpu.txt.gz#37415

Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [None req-41117798-8a4e-469f-bfbb-8bdfdea1a83f demo demo] [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Instance failed to spawn: nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Traceback (most recent call last):
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 2659, in _build_resources
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] yield resources
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 2433, in _build_and_run_instance
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] block_device_info=block_device_info)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3467, in spawn
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] mdevs=mdevs)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6019, in _get_guest_xml
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] network_info_str = str(network_info)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 601, in __str__
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] return self._sync_wrapper(fn, *args, **kwargs)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 584, in _sync_wrapper
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] self.wait()
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 616, in wait
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] self[:] = self._gt.wait()
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 181, in wait
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] return self._exit_event.wait()
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 132, in wait
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] current.throw(*self._exc)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 221, in main
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] result = function(*args, **kwargs)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/utils.py", line 674, in context_wrapper
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] return func(*args, **kwargs)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 1704, in _allocate_network_async
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] six.reraise(*exc_info)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] raise value
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 1687, in _allocate_network_async
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] resource_provider_mapping=resource_provider_mapping)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 990, in allocate_for_instance
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] context, instance, neutron, requested_networks, ordered_networks)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 868, in _validate_requested_network_ids
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] self._check_external_network_attach(context, nets)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 542, in _check_external_network_attach
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] network_uuid=net['id'])
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]

And eventually that results in MaxRetriesExceeded in conductor:

https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n-cond-cell1.txt.gz#878

Nov 13 15:35:47.968308 ubuntu-bionic-rax-ord-0012778423 nova-conductor[25967]: WARNING nova.scheduler.utils [None req-41117798-8a4e-469f-bfbb-8bdfdea1a83f demo demo] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance be6eb09a-f0d9-4c04-9c55-29230a253fbd.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance be6eb09a-f0d9-4c04-9c55-29230a253fbd.

ExternalNetworkAttachForbidden is not something that a reschedule is going to fix, it's essentially a bad request from the user and needs to just abort.

Matt Riedemann (mriedem)
Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/694179

Changed in nova:
status: Triaged → In Progress
Revision history for this message
Matt Riedemann (mriedem) wrote :

This goes back a long time but I'll just target to the non-extended-maintenance branches.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/694179
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=36b110dfc5f8f04234b189a50e96de8dde5f3471
Submitter: Zuul
Branch: master

commit 36b110dfc5f8f04234b189a50e96de8dde5f3471
Author: Matt Riedemann <email address hidden>
Date: Wed Nov 13 18:27:36 2019 -0500

    Do not reschedule on ExternalNetworkAttachForbidden

    When creating a server, if network setup fails with
    ExternalNetworkAttachForbidden then we should abort
    the build rather than reschedule because it's not
    going to work on another host either.

    A functional test is added for this since it's hard
    to reason about whether or not it works properly
    due to the async nature of network setup during
    server create and the _build_resources context
    manager. It's probably not a bad idea to have a
    functional test for a scenario that uses external
    networks either (and we can build on this).

    Change-Id: I6dd5b3f75e2746f53f9758f1a186794a98dba9bf
    Closes-Bug: #1852465

Changed in nova:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.