After that there are failures connecting to the amphorae, which probably makes sense since the list of allowed address pairs couldn't be updated
$ grep taskflow.exceptions.WrappedFailure octavia-worker.log
2023-06-05 22:53:43.184 199015 ERROR oslo_messaging.rpc.server [-] Exception during message handling: taskflow.exceptions.WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67', port=9443): Max retries exceeded with url: // (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fb07d081b40>, 'Connection to fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67 timed out. (connect timeout=10.0)')), Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe34:d41e', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb07d03c5b0>: Failed to establish a new connection: [Errno 111] Connection refused')), Failure: octavia.network.base.PlugVIPException: Error plugging amphora (compute_id: c8acf9f4-15f2-4a5e-ac7b-92c957de620a) into vip network e94cf707-e23c-405b-a582-6e1df99ca271., Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb07c6b88b0>: Failed to establish a new connection: [Errno 111] Connection refused'))]
2023-06-05 22:53:43.184 199015 ERROR oslo_messaging.rpc.server taskflow.exceptions.WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67', port=9443): Max retries exceeded with url: // (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fb07d081b40>, 'Connection to fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67 timed out. (connect timeout=10.0)')), Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe34:d41e', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb07d03c5b0>: Failed to establish a new connection: [Errno 111] Connection refused')), Failure: octavia.network.base.PlugVIPException: Error plugging amphora (compute_id: c8acf9f4-15f2-4a5e-ac7b-92c957de620a) into vip network e94cf707-e23c-405b-a582-6e1df99ca271., Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb07c6b88b0>: Failed to establish a new connection: [Errno 111] Connection refused'))]
I'm seeing this other exception which seems to be more interesting from a troubleshooting perspective:
$ grep nova.exception_ Remote. PortBindingFail ed_Remote octavia-worker.log _Remote. PortBindingFail ed_Remote' > (HTTP 500) (Request-ID: req-cb067766- 8e5c-4c61- 9582-d08f1766d9 f4). network. drivers. neutron. allowed_ address_ pairs <class 'nova.exception _Remote. PortBindingFail ed_Remote' > (HTTP 500) (Request-ID: req-cb067766- 8e5c-4c61- 9582-d08f1766d9 f4) network. drivers. neutron. allowed_ address_ pairs <class 'nova.exception _Remote. PortBindingFail ed_Remote' > (HTTP 500) (Request-ID: req-cb067766- 8e5c-4c61- 9582-d08f1766d9 f4). network. drivers. neutron. allowed_ address_ pairs <class 'nova.exception _Remote. PortBindingFail ed_Remote' > (HTTP 500) (Request-ID: req-cb067766- 8e5c-4c61- 9582-d08f1766d9 f4) network. drivers. neutron. allowed_ address_ pairs <class 'nova.exception _Remote. PortBindingFail ed_Remote' > (HTTP 500) (Request-ID: req-cb067766- 8e5c-4c61- 9582-d08f1766d9 f4). controller. worker. v2.controller_ worker <class 'nova.exception _Remote. PortBindingFail ed_Remote' > (HTTP 500) (Request-ID: req-cb067766- 8e5c-4c61- 9582-d08f1766d9 f4) controller. worker. v2.controller_ worker <class 'nova.exception _Remote. PortBindingFail ed_Remote' > (HTTP 500) (Request-ID: req-cb067766- 8e5c-4c61- 9582-d08f1766d9 f4).
<class 'nova.exception
2023-06-05 22:53:41.089 199015 ERROR octavia.
2023-06-05 22:53:41.089 199015 ERROR octavia.
2023-06-05 22:53:41.091 199015 ERROR octavia.
2023-06-05 22:53:41.091 199015 ERROR octavia.
2023-06-05 22:53:41.682 199015 ERROR octavia.
2023-06-05 22:53:41.682 199015 ERROR octavia.
After that there are failures connecting to the amphorae, which probably makes sense since the list of allowed address pairs couldn't be updated
$ grep taskflow. exceptions. WrappedFailure octavia-worker.log rpc.server [-] Exception during message handling: taskflow. exceptions. WrappedFailure: WrappedFailure: [Failure: octavia. amphorae. driver_ exceptions. exceptions. AmpConnectionRe try: Could not connect to amphora, exception caught: HTTPSConnection Pool(host= 'fc00:5a76: ae64:bd5a: f816:3eff: fe8f:7a67' , port=9443): Max retries exceeded with url: // (Caused by ConnectTimeoutE rror(<urllib3. connection. HTTPSConnection object at 0x7fb07d081b40>, 'Connection to fc00:5a76: ae64:bd5a: f816:3eff: fe8f:7a67 timed out. (connect timeout=10.0)')), Failure: octavia. amphorae. driver_ exceptions. exceptions. AmpConnectionRe try: Could not connect to amphora, exception caught: HTTPSConnection Pool(host= 'fc00:5a76: ae64:bd5a: f816:3eff: fe34:d41e' , port=9443): Max retries exceeded with url: // (Caused by NewConnectionEr ror('<urllib3. connection. HTTPSConnection object at 0x7fb07d03c5b0>: Failed to establish a new connection: [Errno 111] Connection refused')), Failure: octavia. network. base.PlugVIPExc eption: Error plugging amphora (compute_id: c8acf9f4- 15f2-4a5e- ac7b-92c957de62 0a) into vip network e94cf707- e23c-405b- a582-6e1df99ca2 71., Failure: octavia. amphorae. driver_ exceptions. exceptions. AmpConnectionRe try: Could not connect to amphora, exception caught: HTTPSConnection Pool(host= 'fc00:5a76: ae64:bd5a: f816:3eff: fe8f:7a67' , port=9443): Max retries exceeded with url: // (Caused by NewConnectionEr ror('<urllib3. connection. HTTPSConnection object at 0x7fb07c6b88b0>: Failed to establish a new connection: [Errno 111] Connection refused'))] rpc.server taskflow. exceptions. WrappedFailure: WrappedFailure: [Failure: octavia. amphorae. driver_ exceptions. exceptions. AmpConnectionRe try: Could not connect to amphora, exception caught: HTTPSConnection Pool(host= 'fc00:5a76: ae64:bd5a: f816:3eff: fe8f:7a67' , port=9443): Max retries exceeded with url: // (Caused by ConnectTimeoutE rror(<urllib3. connection. HTTPSConnection object at 0x7fb07d081b40>, 'Connection to fc00:5a76: ae64:bd5a: f816:3eff: fe8f:7a67 timed out. (connect timeout=10.0)')), Failure: octavia. amphorae. driver_ exceptions. exceptions. AmpConnectionRe try: Could not connect to amphora, exception caught: HTTPSConnection Pool(host= 'fc00:5a76: ae64:bd5a: f816:3eff: fe34:d41e' , port=9443): Max retries exceeded with url: // (Caused by NewConnectionEr ror('<urllib3. connection. HTTPSConnection object at 0x7fb07d03c5b0>: Failed to establish a new connection: [Errno 111] Connection refused')), Failure: octavia. network. base.PlugVIPExc eption: Error plugging amphora (compute_id: c8acf9f4- 15f2-4a5e- ac7b-92c957de62 0a) into vip network e94cf707- e23c-405b- a582-6e1df99ca2 71., Failure: octavia. amphorae. driver_ exceptions. exceptions. AmpConnectionRe try: Could not connect to amphora, exception caught: HTTPSConnection Pool(host= 'fc00:5a76: ae64:bd5a: f816:3eff: fe8f:7a67' , port=9443): Max retries exceeded with url: // (Caused by NewConnectionEr ror('<urllib3. connection. HTTPSConnection object at 0x7fb07c6b88b0>: Failed to establish a new connection: [Errno 111] Connection refused'))]
2023-06-05 22:53:43.184 199015 ERROR oslo_messaging.
2023-06-05 22:53:43.184 199015 ERROR oslo_messaging.