Kubernetes Control Plane unit goes into error state: hook failed: "aws-relation-changed" or "leader-settings-changed"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Kubernetes Control Plane Charm |
New
|
Undecided
|
Unassigned |
Bug Description
In testrun https:/
```
kubernetes-
calico/1 waiting idle 54.237.76.42 Waiting to retry Calico node configuration
canonical-
containerd/1 active idle 54.237.76.42 Container runtime available
filebeat/7 active idle 54.237.76.42 Filebeat ready.
ntp/7 active idle 54.237.76.42 123/udp chrony: Ready
telegraf/8 active idle 54.237.76.42 9103/tcp Monitoring kubernetes-
kubernetes-
calico/4 waiting idle 54.227.155.67 Waiting to retry Calico node configuration
canonical-
containerd/4 active idle 54.227.155.67 Container runtime available
filebeat/12 active idle 54.227.155.67 Filebeat ready.
ntp/12 active idle 54.227.155.67 123/udp chrony: Ready
telegraf/12 active idle 54.227.155.67 9103/tcp Monitoring kubernetes-
```
In the logs we see:
```
Traceback (most recent call last):
File "/var/lib/
hookenv.
File "/var/lib/
callback(*args, **kwargs)
File "/var/lib/
app_kv = vault_kv.
File "/var/lib/
cls.
File "/var/lib/
super(
File "/var/lib/
response = self._client.
File "/var/lib/
client.
File "/var/lib/
return method(*args, **kwargs)
File "/var/lib/
return self.login(
File "/var/lib/
return self._adapter.
File "/var/lib/
response = self.post(url, **kwargs)
File "/var/lib/
return self.request(
File "/var/lib/
response = super(JSONAdapter, self).request(
File "/var/lib/
utils.
File "/var/lib/
raise exceptions.
hvac.exceptions
```
172.31.45.105 is the IP of machine 15, which is the vault unit. It could just be a network interruption, but maybe there is something else behind it.
Crashdumps etc:
https:/
description: | updated |
I'm seeing this again in https:/ /solutions. qa.canonical. com/testruns/ testRun/ 2e420b50- 01b5-496f- bb01-0a91dcbcd6 44, except that the message is `hook failed: "leader- settings- changed" `. In the logs:
``` juju/agents/ unit-kubernetes -control- plane-1/ .venv/lib/ python3. 10/site- packages/ charms/ reactive/ __init_ _.py", line 73, in main _run_atstart( ) juju/agents/ unit-kubernetes -control- plane-1/ .venv/lib/ python3. 10/site- packages/ charmhelpers/ core/hookenv. py", line 1348, in _run_atstart juju/agents/ unit-kubernetes -control- plane-1/ charm/reactive/ vault_kv. py", line 46, in manage_app_kv_flags VaultAppKV( ) juju/agents/ unit-kubernetes -control- plane-1/ charm/lib/ charms/ layer/vault_ kv.py", line 33, in __call__ _singleton_ instance = super() .__call_ _(*args, **kwargs) juju/agents/ unit-kubernetes -control- plane-1/ charm/lib/ charms/ layer/vault_ kv.py", line 131, in __init__ ).__init_ _() juju/agents/ unit-kubernetes -control- plane-1/ charm/lib/ charms/ layer/vault_ kv.py", line 41, in __init__ read(self. _path) juju/agents/ unit-kubernetes -control- plane-1/ charm/lib/ charms/ layer/vault_ kv.py", line 60, in _client auth_approle( self._config[ "role_id" ], self._config[ "secret_ id"]) juju/agents/ unit-kubernetes -control- plane-1/ .venv/lib/ python3. 10/site- packages/ hvac/utils. py", line 201, in new_func juju/agents/ unit-kubernetes -control- plane-1/ .venv/lib/ python3. 10/site- packages/ hvac/v1/ __init_ _.py", line 1805, in auth_approle juju/agents/ unit-kubernetes -control- plane-1/ .venv/lib/ python3. 10/site- packages/ hvac/v1/ __init_ _.py", line 1495, in login login(url= url, use_token= use_token, **kwargs) juju/agents/ unit-kubernetes -control- plane-1/ .venv/lib/ python3. 10/site- packages/ hvac/adapters. py", line 197, in login juju/agents/ unit-kubernetes -control- plane-1/ .venv/lib/ python3. 10/site- packages/ hvac/adapters. py", line 126, in post "post", url, **kwargs) juju/agents/ unit-kubernetes -control- plane-1/ .venv/lib/ python3. 10/site- packages/ hvac/adapters. py", line 364, in request *args, **kwargs) juju/agents/ unit-kubernetes -control- plane-1/ .venv/lib/ python3. 10/site- packages/ hvac/adapters. py", line 330, in request raise_for_ error( juju/agents/ unit-kubernetes -control- plane-1/ .venv/lib/ python3. 10/site- packages/ hvac/utils. py", line 49, in raise_for_error InternalServerE rror( .InternalServer Error: internal error, on post http:// 172.31. 35.161: 8200/v1/ auth/approle/ login
Traceback (most recent call last):
File "/var/lib/
hookenv.
File "/var/lib/
callback(*args, **kwargs)
File "/var/lib/
app_kv = vault_kv.
File "/var/lib/
cls.
File "/var/lib/
super(
File "/var/lib/
response = self._client.
File "/var/lib/
client.
File "/var/lib/
return method(*args, **kwargs)
File "/var/lib/
return self.login(
File "/var/lib/
return self._adapter.
File "/var/lib/
response = self.post(url, **kwargs)
File "/var/lib/
return self.request(
File "/var/lib/
response = super(JSONAdapter, self).request(
File "/var/lib/
utils.
File "/var/lib/
raise exceptions.
hvac.exceptions
```