nova-compute in error state due to hook failed: "update-status"

Solutions QA has a failed yoga-jammy deployment where in the Openstack layer, nova-compute-0 stayed in error state due to a hook failing in "update-status".

From the logs, we see:

2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 /var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py:176: DeprecationWarning: Call to deprecated function '_post'. This method will be removed in version '0.8.0' Please use the 'post' method on the 'hvac.adapters' class moving forward.
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 response = client._post('/v1/sys/wrapping/unwrap')
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 Traceback (most recent call last):
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 conn = connection.create_connection(
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in create_connection
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 socket.gaierror: [Errno -2] Name or service not known
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 During handling of the above exception, another exception occurred:
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 Traceback (most recent call last):
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 httplib_response = self._make_request(
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 conn.request(method, url, **httplib_request_kw)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 234, in request
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 super(HTTPConnection, self).request(method, url, body=body, headers=headers)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3.10/http/client.py", line 1283, in request
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 self._send_request(method, url, body, headers, encode_chunked)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 self.endheaders(body, encode_chunked=encode_chunked)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 self._send_output(message_body, encode_chunked=encode_chunked)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 self.send(msg)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3.10/http/client.py", line 976, in send
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 self.connect()
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 200, in connect
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 conn = self._new_conn()
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 181, in _new_conn
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 raise NewConnectionError(
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fd93613c2b0>: Failed to establish a new connection: [Errno -2] Name or service not known
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 During handling of the above exception, another exception occurred:
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 Traceback (most recent call last):
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 resp = conn.urlopen(
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 retries = retries.increment(
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 raise MaxRetryError(_pool, url, error or ResponseError(cause))
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='vault-internal.silo3.lab1.solutionsqa', port=8200): Max retries exceeded with url: /v1/sys/wrapping/unwrap (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd93613c2b0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 During handling of the above exception, another exception occurred:
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 Traceback (most recent call last):
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/update-status", line 967, in <module>
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 main()
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/update-status", line 963, in main
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 assess_status(CONFIGS)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/nova_compute_utils.py", line 1010, in assess_status
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 assess_status_func(configs, services_to_check)()
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1822, in _assess_status_func
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 state, message = _determine_os_workload_status(*args, **kwargs)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1031, in _determine_os_workload_status
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 state, message = _ows_check_generic_interfaces(
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1120, in _ows_check_generic_interfaces
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 incomplete_rel_data = incomplete_relation_data(configs,
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1409, in incomplete_relation_data
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 complete_ctxts = configs.complete_contexts()
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 342, in complete_contexts
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 interfaces.extend(i.complete_contexts())
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 122, in complete_contexts
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 self.context()
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 107, in context
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 _ctxt = context()
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 75, in __call__
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 secret_id = retrieve_secret_id(
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 176, in retrieve_secret_id
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 response = client._post('/v1/sys/wrapping/unwrap')
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/hvac/utils.py", line 201, in new_func
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 return method(*args, **kwargs)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 3034, in _post
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 return self._adapter.post(*args, **kwargs)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/hvac/adapters.py", line 126, in post
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 return self.request("post", url, **kwargs)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/hvac/adapters.py", line 313, in request
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 response = self.session.request(
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 resp = self.send(prep, **send_kwargs)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 r = adapter.send(request, **kwargs)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 raise ConnectionError(e, request=request)
2023-10-15 07:45:30 WARNING unit.nova-compute/0.update-status logger.go:60 requests.exceptions.ConnectionError: HTTPConnectionPool(host='vault-internal.silo3.lab1.solutionsqa', port=8200): Max retries exceeded with url: /v1/sys/wrapping/unwrap (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd93613c2b0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
2023-10-15 07:45:30 ERROR juju.worker.uniter.operation runhook.go:180 hook "update-status" (via explicit, bespoke hook script) failed: exit status 1

Failed run: https://solutions.qa.canonical.com/testruns/442a476d-8033-4029-84ff-d79d4073aebd

Logs (with two weeks retention) can be found here: https://oil-jenkins.canonical.com/artifacts/442a476d-8033-4029-84ff-d79d4073aebd/index.html