OpenStack Nova Compute Charm

Tempest test "test_reassign_port_between_servers " fails with server fault

Bug #1994158 reported by Konstantinos Kaskavelis
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
New
Undecided
Unassigned

Bug Description

Solutions QA team has a failed run where a tempest test failed with the message: "tempest.lib.exceptions.ServerFault: Got server fault"

The failure happened after we deployed Yoga on Focal.

From the tempest report:

Traceback (most recent call last):
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/api/compute/servers/test_attach_interfaces.py", line 337, in test_reassign_port_between_servers
    server['id'], port_id=port_id)['interfaceAttachment']
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/lib/services/compute/interfaces_client.py", line 48, in create_interface
    body=post_body)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request
    method, url, extra_headers, headers, body, chunked)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/lib/common/rest_client.py", line 880, in _error_checker
    message=message)
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception_Remote.PortBindingFailed_Remote'>

From the logs:

2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi [req-d7f47a9a-bcbc-4852-b16d-21676f2d252d 5b37d935cf704131a7e0ffc4ca909598 6383a5bda46c49e68522db7da47854df - 541f14c98a464950bf423d15c973b6de 541f14c98a464950bf423d15c973b6de] Unexpected exception in API method: nova.exception_Remote.PortBindingFailed_Remote: Binding failed for port ab1a4b97-b704-462f-a767-f4319ab1f278, please check neutron logs for more information.
Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 71, in wrapped
    _emit_versioned_exception_notification(

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
    return f(self, context, *args, **kw)

  File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 212, in decorated_function
    compute_utils.add_instance_fault_from_exc(context,

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 201, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7846, in attach_interface
    return do_attach_interface(context, instance, network_id, port_id,

  File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 391, in inner
    return f(*args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 71, in wrapped
    _emit_versioned_exception_notification(

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
    return f(self, context, *args, **kw)

  File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 212, in decorated_function
    compute_utils.add_instance_fault_from_exc(context,

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 201, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7846, in attach_interface
    return do_attach_interface(context, instance, network_id, port_id,

  File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 391, in inner
    return f(*args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7843, in do_attach_interface
    return self._attach_interface(context, instance, network_id,

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7914, in _attach_interface
    network_info = self.network_api.allocate_for_instance(

  File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1213, in allocate_for_instance
    created_port_ids = self._update_ports_for_instance(

  File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1351, in _update_ports_for_instance
    vif.destroy()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1320, in _update_ports_for_instance
    updated_port = self._update_port(

  File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 584, in _update_port
    _ensure_no_port_binding_failure(port)

  File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 293, in _ensure_no_port_binding_failure
    raise exception.PortBindingFailed(port_id=port['id'])

nova.exception.PortBindingFailed: Binding failed for port ab1a4b97-b704-462f-a767-f4319ab1f278, please check neutron logs for more information.
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/api/openstack/compute/attach_interfaces.py", line 168, in create
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi vif = self.compute_api.attach_interface(context,
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 238, in inner
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 164, in inner
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 5258, in attach_interface
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return self.compute_rpcapi.attach_interface(context,
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/compute/rpcapi.py", line 572, in attach_interface
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return cctxt.call(ctxt, 'attach_interface', **kw)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi result = self.transport._send(
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return self._driver.send(target, ctxt, message,
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return self._send(target, ctxt, message, wait_for_reply, timeout,
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi raise result
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi nova.exception_Remote.PortBindingFailed_Remote: Binding failed for port ab1a4b97-b704-462f-a767-f4319ab1f278, please check neutron logs for more information.

Test run:

https://solutions.qa.canonical.com/testruns/testRun/0c561619-ee15-46c1-8aa6-1cf9c86883dc

Rally report:

https://oil-jenkins.canonical.com/artifacts/0c561619-ee15-46c1-8aa6-1cf9c86883dc/generated/generated/openstack/tempest_result.html

Logs:

https://oil-jenkins.canonical.com/artifacts/0c561619-ee15-46c1-8aa6-1cf9c86883dc/index.html

Tags: cdo-qa
Jeffrey Chang (modern911)
tags: added: cdo-qa
Revision history for this message
Billy Olsen (billy-olsen) wrote :

This looks like the result of a change in the 20.2.0 version for Yoga. Specifically, bug #1958501 was fixed to cause Neutron to not bind the port to a dead agent in the OVN case. It was fixed in Zed release and backported to yoga/stable in https://review.opendev.org/c/openstack/neutron/+/844494.

In this case, the agent is not determined to be alive for solqa-lab1-server-35.nosilo.lab1.solutionsqa

neutron-api-plugin-ovn_1/var/log/neutron/neutron-server.log
--------------------
2022-10-25 12:12:36.952 192922 ERROR neutron.plugins.ml2.managers [req-ef882b1c-f715-41fa-89d9-fd029f38576f a12a96846bce4d16af50a3389e87a7e9 a0c0be4aa6a3407d9b695dae6ee1861f - 58f5e7f1a82c452e8affaa6b14c7b
d00 58f5e7f1a82c452e8affaa6b14c7bd00] Failed to bind port ab1a4b97-b704-462f-a767-f4319ab1f278 on host solqa-lab1-server-35.nosilo.lab1.solutionsqa for vnic_type normal using segments [{'id': '898f8692-fdb
0-46b3-8388-8a007efbac46', 'network_type': 'geneve', 'physical_network': None, 'segmentation_id': 1376, 'network_id': 'de98a060-d23a-4520-bba1-3b6092008b6f'}]
2022-10-25 12:12:36.953 192922 INFO neutron.plugins.ml2.plugin [req-ef882b1c-f715-41fa-89d9-fd029f38576f a12a96846bce4d16af50a3389e87a7e9 a0c0be4aa6a3407d9b695dae6ee1861f - 58f5e7f1a82c452e8affaa6b14c7bd00
 58f5e7f1a82c452e8affaa6b14c7bd00] Attempt 6 to bind port ab1a4b97-b704-462f-a767-f4319ab1f278
2022-10-25 12:12:36.970 192922 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-ef882b1c-f715-41fa-89d9-fd029f38576f a12a96846bce4d16af50a3389e87a7e9 a0c0be4aa6a3407d9b695dae6ee1861f - 58f5e7f1a82c452e8affaa6b14c7bd00 58f5e7f1a82c452e8affaa6b14c7bd00] Refusing to bind port ab1a4b97-b704-462f-a767-f4319ab1f278 to dead agent: <neutron.plugins.ml2.drivers.ovn.agent.neutron_agent.ControllerAgent object at 0x7fdff23a26d0>

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.