Solutions QA has a failed run due to kubeapi-load-balancer charm being in an error state with the message: hook failed: "certificates-relation-joined"
The error happened while deploying charmed kubernetes 1.28 on top of MAAS 3.2.9 on jammy, with cilium as the CNI.
From the logs:
machine-0-lxd-1: 2023-09-24 07:13:00 DEBUG jujuc running hook tool "juju-log" for kubeapi-load-balancer/0-certificates-relation-joined-5032928647907253954
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 ERROR unit.kubeapi-load-balancer/0.juju-log certificates:72: Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/reactive/load_balancer.py", line 81, in request_server_certificates
common_name = hookenv.unit_public_ip()
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 877, in unit_public_ip
return unit_get('public-address')
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 85, in wrapper
res = func(*args, **kwargs)
File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 870, in unit_get
return json.loads(subprocess.check_output(_args).decode('UTF-8'))
File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['unit-get', '--format=json', 'public-address']' returned non-zero exit status 1.
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined Traceback (most recent call last):
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/hooks/certificates-relation-joined", line 22, in <module>
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined main()
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined bus.dispatch(restricted=restricted_mode)
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined _invoke(other_handlers)
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined handler.invoke()
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined self._action(*args)
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/reactive/load_balancer.py", line 81, in request_server_certificates
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined common_name = hookenv.unit_public_ip()
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 877, in unit_public_ip
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined return unit_get('public-address')
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 85, in wrapper
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined res = func(*args, **kwargs)
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 870, in unit_get
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined return json.loads(subprocess.check_output(_args).decode('UTF-8'))
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined File "/usr/lib/python3.10/subprocess.py", line 526, in run
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined raise CalledProcessError(retcode, process.args,
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 WARNING unit.kubeapi-load-balancer/0.certificates-relation-joined subprocess.CalledProcessError: Command '['unit-get', '--format=json', 'public-address']' returned non-zero exit status 1.
unit-telegraf-4: 2023-09-24 07:13:00 DEBUG unit.telegraf/4.install Building wheel for setuptools-scm (setup.py): finished with status 'done'
unit-telegraf-4: 2023-09-24 07:13:00 DEBUG unit.telegraf/4.install Created wheel for setuptools-scm: filename=setuptools_scm-1.17.0-py2.py3-none-any.whl size=17711 sha256=20a7240fada68df962385456eae2bcdeb3cfb91db711725031620994be9a25d3
unit-telegraf-4: 2023-09-24 07:13:00 DEBUG unit.telegraf/4.install Stored in directory: /root/.cache/pip/wheels/82/68/c0/1dba89691fc134d0ae26144cf7d9ad1d98f30399ff0a9d8a00
unit-telegraf-4: 2023-09-24 07:13:00 DEBUG unit.telegraf/4.install Successfully built setuptools-scm
unit-telegraf-4: 2023-09-24 07:13:00 DEBUG unit.telegraf/4.install Installing collected packages: setuptools-scm
unit-telegraf-4: 2023-09-24 07:13:00 DEBUG unit.telegraf/4.install Successfully installed setuptools-scm-1.17.0
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 ERROR juju.worker.uniter.operation hook "certificates-relation-joined" (via explicit, bespoke hook script) failed: exit status 1
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 DEBUG juju.machinelock created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 DEBUG juju.machinelock machine lock "machine-lock" released for kubeapi-load-balancer/0 uniter (run relation-joined (72; unit: vault/0) hook)
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 DEBUG juju.worker.uniter.operation lock released for kubeapi-load-balancer/0
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 INFO juju.worker.uniter awaiting error resolution for "relation-joined" hook
unit-kubeapi-load-balancer-0: 2023-09-24 07:13:00 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "certificates-relation-joined"
Failed test run: https://solutions.qa.canonical.com/testruns/83ffcb1d-78ad-44da-8e0e-5b0c80f6fd43
Logs: https://oil-jenkins.canonical.com/artifacts/83ffcb1d-78ad-44da-8e0e-5b0c80f6fd43/index.html
To add more context:
In `unit-kubeapi-
2023-09-24 07:13:00 WARNING unit.kubeapi-