I'm seeing this other exception which seems to be more interesting from a troubleshooting perspective:

$ grep nova.exception_Remote.PortBindingFailed_Remote octavia-worker.log 
<class 'nova.exception_Remote.PortBindingFailed_Remote'> (HTTP 500) (Request-ID: req-cb067766-8e5c-4c61-9582-d08f1766d9f4).
2023-06-05 22:53:41.089 199015 ERROR octavia.network.drivers.neutron.allowed_address_pairs <class 'nova.exception_Remote.PortBindingFailed_Remote'> (HTTP 500) (Request-ID: req-cb067766-8e5c-4c61-9582-d08f1766d9f4)
2023-06-05 22:53:41.089 199015 ERROR octavia.network.drivers.neutron.allowed_address_pairs <class 'nova.exception_Remote.PortBindingFailed_Remote'> (HTTP 500) (Request-ID: req-cb067766-8e5c-4c61-9582-d08f1766d9f4).
2023-06-05 22:53:41.091 199015 ERROR octavia.network.drivers.neutron.allowed_address_pairs <class 'nova.exception_Remote.PortBindingFailed_Remote'> (HTTP 500) (Request-ID: req-cb067766-8e5c-4c61-9582-d08f1766d9f4)
2023-06-05 22:53:41.091 199015 ERROR octavia.network.drivers.neutron.allowed_address_pairs <class 'nova.exception_Remote.PortBindingFailed_Remote'> (HTTP 500) (Request-ID: req-cb067766-8e5c-4c61-9582-d08f1766d9f4).
2023-06-05 22:53:41.682 199015 ERROR octavia.controller.worker.v2.controller_worker <class 'nova.exception_Remote.PortBindingFailed_Remote'> (HTTP 500) (Request-ID: req-cb067766-8e5c-4c61-9582-d08f1766d9f4)
2023-06-05 22:53:41.682 199015 ERROR octavia.controller.worker.v2.controller_worker <class 'nova.exception_Remote.PortBindingFailed_Remote'> (HTTP 500) (Request-ID: req-cb067766-8e5c-4c61-9582-d08f1766d9f4).

After that there are failures connecting to the amphorae, which probably makes sense since the list of allowed address pairs couldn't be updated

$ grep taskflow.exceptions.WrappedFailure octavia-worker.log 
2023-06-05 22:53:43.184 199015 ERROR oslo_messaging.rpc.server [-] Exception during message handling: taskflow.exceptions.WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67', port=9443): Max retries exceeded with url: // (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fb07d081b40>, 'Connection to fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67 timed out. (connect timeout=10.0)')), Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe34:d41e', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb07d03c5b0>: Failed to establish a new connection: [Errno 111] Connection refused')), Failure: octavia.network.base.PlugVIPException: Error plugging amphora (compute_id: c8acf9f4-15f2-4a5e-ac7b-92c957de620a) into vip network e94cf707-e23c-405b-a582-6e1df99ca271., Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb07c6b88b0>: Failed to establish a new connection: [Errno 111] Connection refused'))]
2023-06-05 22:53:43.184 199015 ERROR oslo_messaging.rpc.server taskflow.exceptions.WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67', port=9443): Max retries exceeded with url: // (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fb07d081b40>, 'Connection to fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67 timed out. (connect timeout=10.0)')), Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe34:d41e', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb07d03c5b0>: Failed to establish a new connection: [Errno 111] Connection refused')), Failure: octavia.network.base.PlugVIPException: Error plugging amphora (compute_id: c8acf9f4-15f2-4a5e-ac7b-92c957de620a) into vip network e94cf707-e23c-405b-a582-6e1df99ca271., Failure: octavia.amphorae.driver_exceptions.exceptions.AmpConnectionRetry: Could not connect to amphora, exception caught: HTTPSConnectionPool(host='fc00:5a76:ae64:bd5a:f816:3eff:fe8f:7a67', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb07c6b88b0>: Failed to establish a new connection: [Errno 111] Connection refused'))]