Kubernetes Control Plane Charm

k8s 1.16: kube-proxy fails to get NodeIP on masters

Bug #1841114 reported by George Kraft
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Kubernetes Control Plane Charm
Fix Released
Critical
George Kraft
Kubernetes Control Plane Charm 1.16

Bug Description

I deployed cs:charmed-kubernetes-219 with k8s 1.16.0-beta.0.

kube-proxy is failing to run on both units of kubernetes-master:

$ journalctl -o cat -u snap.kube-proxy.daemon
...
W0822 21:37:56.732643 11389 server_others.go:250] Flag proxy-mode="" unknown, assuming iptables proxy
E0822 21:37:56.735616 11389 node.go:124] Failed to retrieve node info: nodes "ip-172-31-19-168" not found
E0822 21:37:57.781831 11389 node.go:124] Failed to retrieve node info: nodes "ip-172-31-19-168" not found
E0822 21:37:59.901682 11389 node.go:124] Failed to retrieve node info: nodes "ip-172-31-19-168" not found
E0822 21:38:03.914607 11389 node.go:124] Failed to retrieve node info: nodes "ip-172-31-19-168" not found
E0822 21:38:12.402178 11389 node.go:124] Failed to retrieve node info: nodes "ip-172-31-19-168" not found
F0822 21:38:12.402201 11389 server.go:438] unable to get node IP for hostname ip-172-31-19-168
snap.kube-proxy.daemon.service: Main process exited, code=exited, status=255/n/a

Revision history for this message
George Kraft (cynerva) wrote :

Introduced by upstream PR: https://github.com/kubernetes/kubernetes/pull/77167

This is happening because we run kube-proxy without kubelet on the masters. The obvious fix is to add kubelet.

A possible workaround is to configure kube-proxy with --bind-address=<node-ip>. It sounds like the node IP is only used by the ipvs proxier, which we don't use, so we don't need to be too careful there. We do need to make sure that changing --bind-address won't prevent any of kube-proxy's clients from reaching it. But, to be honest, I'm not sure if kube-proxy even has any clients in CK. Needs investigation.

Revision history for this message
George Kraft (cynerva) wrote :

I would suggest that we just go ahead and configure kube-proxy's bind-address to match the advertise-address of kube-apiserver, merge it, and look out for related issues as we're testing CK 1.16.

Revision history for this message
George Kraft (cynerva) wrote :

PRs:
https://github.com/charmed-kubernetes/layer-kubernetes-common/pull/7
https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/48

George Kraft (cynerva)
Changed in charm-kubernetes-master:
assignee: nobody → George Kraft (cynerva)
status: New → Fix Committed
Tim Van Steenburgh (tvansteenburgh)
Changed in charm-kubernetes-master:
milestone: none → 1.16
importance: Undecided → Critical
Revision history for this message
George Kraft (cynerva) wrote :

Fix is available in edge with cs:~containers/kubernetes-master-735

Tim Van Steenburgh (tvansteenburgh)
Changed in charm-kubernetes-master:
status: Fix Committed → Fix Released
Revision history for this message
Cory Johns (johnsca) wrote :

For future reference, https://github.com/kubernetes/kubernetes/pull/83822 fixed this by changing the behavior back to defaulting nodeIP to localhost if bind-address is unspecified, and was backported to 1.16, so setting it is no longer required.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.