```
kubernetes-control-plane/0* error idle 7 54.237.76.42 hook failed: "aws-relation-changed"
calico/1 waiting idle 54.237.76.42 Waiting to retry Calico node configuration
canonical-livepatch/7 active idle 54.237.76.42 Running kernel 5.13.0-1031.35~20.04.1-aws, patchState: nothing-to-apply (source version/commit dad6199)
containerd/1 active idle 54.237.76.42 Container runtime available
filebeat/7 active idle 54.237.76.42 Filebeat ready.
ntp/7 active idle 54.237.76.42 123/udp chrony: Ready
telegraf/8 active idle 54.237.76.42 9103/tcp Monitoring kubernetes-control-plane/0
kubernetes-control-plane/1 maintenance executing 8 54.227.155.67 6443/tcp (leader-settings-changed) Restarting snap.kube-scheduler.daemon service
calico/4 waiting idle 54.227.155.67 Waiting to retry Calico node configuration
canonical-livepatch/12 active idle 54.227.155.67 Running kernel 5.13.0-1031.35~20.04.1-aws, patchState: nothing-to-apply (source version/commit dad6199)
containerd/4 active idle 54.227.155.67 Container runtime available
filebeat/12 active idle 54.227.155.67 Filebeat ready.
ntp/12 active idle 54.227.155.67 123/udp chrony: Ready
telegraf/12 active idle 54.227.155.67 9103/tcp Monitoring kubernetes-control-plane/1
```
In the logs we see:
```
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 73, in main
hookenv._run_atstart()
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charmhelpers/core/hookenv.py", line 1348, in _run_atstart
callback(*args, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/reactive/vault_kv.py", line 46, in manage_app_kv_flags
app_kv = vault_kv.VaultAppKV()
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 33, in __call__
cls._singleton_instance = super().__call__(*args, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 131, in __init__
super().__init__()
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 41, in __init__
response = self._client.read(self._path)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 60, in _client
client.auth_approle(self._config["role_id"], self._config["secret_id"])
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/utils.py", line 201, in new_func
return method(*args, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1805, in auth_approle
return self.login(
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1495, in login
return self._adapter.login(url=url, use_token=use_token, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 197, in login
response = self.post(url, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 126, in post
return self.request("post", url, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 364, in request
response = super(JSONAdapter, self).request(*args, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 330, in request
utils.raise_for_error(
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/utils.py", line 49, in raise_for_error
raise exceptions.InternalServerError(
hvac.exceptions.InternalServerError: internal error, on post http://172.31.45.105:8200/v1/auth/approle/login
```
172.31.45.105 is the IP of machine 15, which is the vault unit. It would just be a network interruption, but maybe there is something else behind it.
In testrun https:/ /solutions. qa.canonical. com/testruns/ testRun/ 89e74981- 4d6c-4efc- b5d4-45868278f4 5d, with FCE logs: https:/ /oil-jenkins. canonical. com/job/ fce_build/ 4480//console we see a k8s control-plane unit going into an error state:
``` control- plane/0* error idle 7 54.237.76.42 hook failed: "aws-relation- changed" livepatch/ 7 active idle 54.237.76.42 Running kernel 5.13.0- 1031.35~ 20.04.1- aws, patchState: nothing-to-apply (source version/commit dad6199) control- plane/0 control- plane/1 maintenance executing 8 54.227.155.67 6443/tcp (leader- settings- changed) Restarting snap.kube- scheduler. daemon service livepatch/ 12 active idle 54.227.155.67 Running kernel 5.13.0- 1031.35~ 20.04.1- aws, patchState: nothing-to-apply (source version/commit dad6199) control- plane/1
kubernetes-
calico/1 waiting idle 54.237.76.42 Waiting to retry Calico node configuration
canonical-
containerd/1 active idle 54.237.76.42 Container runtime available
filebeat/7 active idle 54.237.76.42 Filebeat ready.
ntp/7 active idle 54.237.76.42 123/udp chrony: Ready
telegraf/8 active idle 54.237.76.42 9103/tcp Monitoring kubernetes-
kubernetes-
calico/4 waiting idle 54.227.155.67 Waiting to retry Calico node configuration
canonical-
containerd/4 active idle 54.227.155.67 Container runtime available
filebeat/12 active idle 54.227.155.67 Filebeat ready.
ntp/12 active idle 54.227.155.67 123/udp chrony: Ready
telegraf/12 active idle 54.227.155.67 9103/tcp Monitoring kubernetes-
```
In the logs we see: juju/agents/ unit-kubernetes -control- plane-0/ .venv/lib/ python3. 8/site- packages/ charms/ reactive/ __init_ _.py", line 73, in main _run_atstart( ) juju/agents/ unit-kubernetes -control- plane-0/ .venv/lib/ python3. 8/site- packages/ charmhelpers/ core/hookenv. py", line 1348, in _run_atstart juju/agents/ unit-kubernetes -control- plane-0/ charm/reactive/ vault_kv. py", line 46, in manage_app_kv_flags VaultAppKV( ) juju/agents/ unit-kubernetes -control- plane-0/ charm/lib/ charms/ layer/vault_ kv.py", line 33, in __call__ _singleton_ instance = super() .__call_ _(*args, **kwargs) juju/agents/ unit-kubernetes -control- plane-0/ charm/lib/ charms/ layer/vault_ kv.py", line 131, in __init__ ).__init_ _() juju/agents/ unit-kubernetes -control- plane-0/ charm/lib/ charms/ layer/vault_ kv.py", line 41, in __init__ read(self. _path) juju/agents/ unit-kubernetes -control- plane-0/ charm/lib/ charms/ layer/vault_ kv.py", line 60, in _client auth_approle( self._config[ "role_id" ], self._config[ "secret_ id"]) juju/agents/ unit-kubernetes -control- plane-0/ .venv/lib/ python3. 8/site- packages/ hvac/utils. py", line 201, in new_func juju/agents/ unit-kubernetes -control- plane-0/ .venv/lib/ python3. 8/site- packages/ hvac/v1/ __init_ _.py", line 1805, in auth_approle juju/agents/ unit-kubernetes -control- plane-0/ .venv/lib/ python3. 8/site- packages/ hvac/v1/ __init_ _.py", line 1495, in login login(url= url, use_token= use_token, **kwargs) juju/agents/ unit-kubernetes -control- plane-0/ .venv/lib/ python3. 8/site- packages/ hvac/adapters. py", line 197, in login juju/agents/ unit-kubernetes -control- plane-0/ .venv/lib/ python3. 8/site- packages/ hvac/adapters. py", line 126, in post "post", url, **kwargs) juju/agents/ unit-kubernetes -control- plane-0/ .venv/lib/ python3. 8/site- packages/ hvac/adapters. py", line 364, in request *args, **kwargs) juju/agents/ unit-kubernetes -control- plane-0/ .venv/lib/ python3. 8/site- packages/ hvac/adapters. py", line 330, in request raise_for_ error( juju/agents/ unit-kubernetes -control- plane-0/ .venv/lib/ python3. 8/site- packages/ hvac/utils. py", line 49, in raise_for_error InternalServerE rror( .InternalServer Error: internal error, on post http:// 172.31. 45.105: 8200/v1/ auth/approle/ login
```
Traceback (most recent call last):
File "/var/lib/
hookenv.
File "/var/lib/
callback(*args, **kwargs)
File "/var/lib/
app_kv = vault_kv.
File "/var/lib/
cls.
File "/var/lib/
super(
File "/var/lib/
response = self._client.
File "/var/lib/
client.
File "/var/lib/
return method(*args, **kwargs)
File "/var/lib/
return self.login(
File "/var/lib/
return self._adapter.
File "/var/lib/
response = self.post(url, **kwargs)
File "/var/lib/
return self.request(
File "/var/lib/
response = super(JSONAdapter, self).request(
File "/var/lib/
utils.
File "/var/lib/
raise exceptions.
hvac.exceptions
```
172.31.45.105 is the IP of machine 15, which is the vault unit. It would just be a network interruption, but maybe there is something else behind it.
Crashdumps etc: /oil-jenkins. canonical. com/artifacts/ 89e74981- 4d6c-4efc- b5d4-45868278f4 5d/index. html
https:/