Test run https://solutions.qa.canonical.com/testruns/1cce9709-db0c-4bb1-80cb-a3fc9652a12c (microstack on jammy, single-node), fails with the following status: https://oil-jenkins.canonical.com/artifacts/1cce9709-db0c-4bb1-80cb-a3fc9652a12c/generated/generated/sunbeam/juju_status_openstack.txt
In the debug-log I see a lot of these messages:
=====
machine-0: 07:11:16 INFO juju.kubernetes.klog Waited for 5.723907577s due to client-side throttling, not priority and fairness, request: GET:https://10.245.130.51:16443/api/v1/namespaces/openstack/pods?labelSelector=app.kubernetes.io%2Fname%3Dcinder-mysql-router
machine-0: 07:11:16 ERROR juju.apiserver.uniter resolving "": lookup : no such host
=====
Also, the nova pod logs show:
=====
(...)
2023-08-31T08:07:25.729Z [nova-scheduler] 2023-08-31 08:07:25.715 96 ERROR nova self.dbapi_connection = connection = pool._invoke_creator(self)
2023-08-31T08:07:25.729Z [nova-scheduler] 2023-08-31 08:07:25.715 96 ERROR nova File "/usr/lib/python3/dist-packages/sqlalchemy/engine/create.py", line 590, in connect
2023-08-31T08:07:25.729Z [nova-scheduler] 2023-08-31 08:07:25.715 96 ERROR nova return dialect.connect(*cargs, **cparams)
2023-08-31T08:07:25.729Z [nova-scheduler] 2023-08-31 08:07:25.715 96 ERROR nova File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 597, in connect
2023-08-31T08:07:25.729Z [nova-scheduler] 2023-08-31 08:07:25.715 96 ERROR nova return self.dbapi.connect(*cargs, **cparams)
2023-08-31T08:07:25.729Z [nova-scheduler] 2023-08-31 08:07:25.715 96 ERROR nova File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 353, in __init__
2023-08-31T08:07:25.729Z [nova-scheduler] 2023-08-31 08:07:25.715 96 ERROR nova self.connect()
2023-08-31T08:07:25.729Z [nova-scheduler] 2023-08-31 08:07:25.715 96 ERROR nova File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 664, in connect
2023-08-31T08:07:25.729Z [nova-scheduler] 2023-08-31 08:07:25.715 96 ERROR nova raise exc
2023-08-31T08:07:25.729Z [nova-scheduler] 2023-08-31 08:07:25.715 96 ERROR nova oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'nova-api-mysql-router.openstack.svc.cluster.local' ([Errno -2] Name or service not known)")
=====
I'm not sure what the cause of the name resolutions errors is.
Logs and configs can be found here: https://oil-jenkins.canonical.com/artifacts/1cce9709-db0c-4bb1-80cb-a3fc9652a12c/index.html
The failure to resolve hostnames within K8S is most likely in the coredns service within Kubernetes - either its not working or its just not up-to-date.
FWIW I have seen this issue once before but I never got to the bottom of it.