Don't allow "" value for kubelet-extra-config

Bug #2008054 reported by Aliaksandr Vasiuk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Kubernetes Control Plane Charm
Triaged
Medium
Unassigned
Kubernetes Worker Charm
Triaged
Medium
Unassigned

Bug Description

Hello,

When set "" value to kubelet-extra-config it is accepted by the charm, but then agent fails to restart kubelet, which leads to cluster pods going down in 5 minutes.

Environment: kubernetes-worker charm revision 44 from channel 1.24/stable deployed on Bionic machine.

Steps to reproduce:
1. Deploy juju k8s bundle
2. juju config kubernetes-worker kubelet-extra-config=""
3. seeing this in kubernetes-worker logs
```
unit-kubernetes-worker-15: 13:53:21 ERROR unit.kubernetes-worker/15.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kubernetes-worker-15/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-kubernetes-worker-15/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-kubernetes-worker-15/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-kubernetes-worker-15/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-kubernetes-worker-15/charm/reactive/kubernetes_worker.py", line 668, in start_worker
    configure_kubelet(dns_domain, dns_ip, registry, has_xcp=kube_control.has_xcp)
  File "lib/charms/layer/kubernetes_common.py", line 1094, in configure_kubelet
    merge_kubelet_extra_config(kubelet_config, kubelet_extra_config)
  File "lib/charms/layer/kubernetes_common.py", line 965, in merge_kubelet_extra_config
    for k, extra_config_value in extra_config.items():
AttributeError: 'NoneType' object has no attribute 'items'
```

The expected behavior would be:
1. Deploy juju k8s bundle
2. juju config kubernetes-worker kubelet-extra-config=""
3. Get an error from `juju config` saying that "" is not allowed value.

The issue's impact was pretty bad in production as we used `juju config kubernetes-worker kubelet-extra-config=""` to reset the experimental kubelet configuration and thought that reset didn't work. So needed to fix kubelet configuration manually, while part of the cluster went down in 5 minutes timeout. Of course we will use only `juju config --reset` from now on. But still would be great to fix configuration validation.
Many thanks in advance!

Best Regards,
Alex.

Revision history for this message
George Kraft (cynerva) wrote :

Thanks for the report. We don't have a way to prevent Juju from accepting the config, but there are two things we can do in the charm to improve the experience here:

1. Accept "" and treat it the same as "{}"
2. Validate the config in the charm, and if it's invalid, enter Blocked status with a clear message

Changed in charm-kubernetes-worker:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-kubernetes-master:
importance: Undecided → Medium
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.