oslo_messaging.exceptions.MessagingTimeout

We have been struggling to land changes to Octavia due to various bugs and the length of time for a full test run [0].

In attempting to run these tests manually, I have occasionally hit an error in the bionic-train-ha with an oslo_messaging.exceptions.MessagingTimeout [1]. We have been experiencing ServerStack (CI cloud) load problems which are the likely culprit, but I wanted to document this here in case we see it again.

The symptom is loadblancers or loadbalancer primitives (pools, members, etc) go into ERROR state.

The octavia-worker.log has [1] the following:

<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-b4fd45a8-8dc1-4aa6-8aa7-5d20771136e8)

2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs Traceback (most recent call last):
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3/dist-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 520, in plug_network
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs ip_address=ip_address)
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3/dist-packages/octavia/compute/drivers/nova_driver.py", line 351, in attach_network_or_port
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs port_id=port_id)
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3/dist-packages/novaclient/api_versions.py", line 393, in substitution
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs return methods[-1].func(obj, *args, **kwargs)
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3/dist-packages/novaclient/v2/servers.py", line 2067, in interface_attach
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs obj_class=NetworkInterface)
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3/dist-packages/novaclient/base.py", line 364, in _create
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs resp, body = self.api.client.post(url, body=body)
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs return self.request(url, 'POST', **kwargs)
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3/dist-packages/novaclient/client.py", line 78, in request
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs raise exceptions.from_response(resp, body, url, method)
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs novaclient.exceptions.ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs <class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-b4fd45a8-8dc1-4aa6-8aa7-5d20771136e8)
2020-07-17 20:17:39.211 22973 ERROR octavia.network.drivers.neutron.allowed_address_pairs

[0] https://review.opendev.org/#/c/737648/
[1] http://paste.ubuntu.com/p/XJYCdJ7cjn/