kubelet and kube-proxy hang due to systemd start limit
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Kubernetes Common Layer |
Fix Released
|
High
|
George Kraft | ||
Kubernetes Control Plane Charm |
Fix Released
|
High
|
George Kraft | ||
Kubernetes Worker Charm |
Fix Released
|
High
|
George Kraft |
Bug Description
In a CI test run, we saw kube-proxy hang due to systemd start rate limiting.
The charm had just upgraded to k8s 1.26 snaps, which caused kube-proxy to crash since it was still using a deprecated --logtostderr argument. These crashes caused kube-proxy to hit the systemd start limit, where no further restarts will be attempted.
The charm then reconfigured the service and tried to restart it, but failed:
2022-12-09 23:34:40 INFO unit.kubernetes
2022-12-09 23:34:40 WARNING unit.kubernetes
2022-12-09 23:34:40 WARNING unit.kubernetes
Which failed due to:
Dec 9 23:34:40 juju-86e872-8 systemd[1]: snap.kube-
Dec 9 23:34:40 juju-86e872-8 systemd[1]: snap.kube-
Dec 9 23:34:40 juju-86e872-8 systemd[1]: Failed to start Service for snap application kube-proxy.daemon.
No further restart attempts occurred.
Changed in charm-kubernetes-worker: | |
assignee: | nobody → George Kraft (cynerva) |
importance: | Undecided → High |
status: | New → In Progress |
Changed in charm-kubernetes-master: | |
milestone: | none → 1.26 |
Changed in charm-kubernetes-worker: | |
milestone: | none → 1.26 |
Changed in charm-kubernetes-master: | |
status: | New → In Progress |
Changed in layer-kubernetes-common: | |
status: | New → In Progress |
Changed in charm-kubernetes-master: | |
importance: | Undecided → High |
Changed in layer-kubernetes-common: | |
importance: | Undecided → High |
milestone: | none → 1.26 |
tags: | added: backport-needed |
Changed in layer-kubernetes-common: | |
status: | Fix Committed → Fix Released |
Changed in charm-kubernetes-master: | |
status: | Fix Committed → Fix Released |
Changed in charm-kubernetes-worker: | |
status: | Fix Committed → Fix Released |
PRs: /github. com/charmed- kubernetes/ layer-kubernete s-common/ pull/38 /github. com/charmed- kubernetes/ charm-kubernete s-control- plane/pull/ 263 /github. com/charmed- kubernetes/ charm-kubernete s-worker/ pull/134
https:/
https:/
https:/