Kubernetes-control-plane status stuck waiting

Bug #1986704 reported by David Gomez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
AWS IAM Subordinate Charm
Invalid
Undecided
Unassigned

Bug Description

When deploying the charmed kubernetes bundle, kubernetes-control-plane app 1.24.3 stuck in waiting status for over 30+ minutes.

Deployed with "juju deploy charmed-kubernetes --overlay ./openstack-overlay.yaml --trust"

openstack-overlay.yaml:

description: Charmed Kubernetes overlay to add native OpenStack support.
applications:
  openstack-integrator:
    annotations:
      gui-x: "600"
      gui-y: "300"
    charm: openstack-integrator
    num_units: 1
    trust: true
relations:
  - ['openstack-integrator', 'kubernetes-control-plane:openstack']
  - ['openstack-integrator', 'kubernetes-worker:openstack']

See juju-crashdump tarball attached

Workaround:
Switching from calico to flannel in the deployment seems to get around this issue

Revision history for this message
David Gomez (dvgomez) wrote :
Revision history for this message
George Kraft (cynerva) wrote :

From the crashdump, Calico CNI is getting a Forbidden response trying to talk to the Kubernetes API. From kubelet:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e61f7be828eb2f62bfef34c9388cb
592311becf8a8eb84fe2fd6594ab1217228": Get "https://10.5.0.157:6443/api/v1/namespaces/ingress-nginx-kubernetes-worker": Forbidden

It looks like your containerd charm is configured with:

http_proxy: http://squid.internal:3128
https_proxy: http://squid.internal:3128
no_proxy: <blank>

The Calico CNI plugin inherits HTTP proxy variables from containerd. When it tries to reach the Kubernetes API at 10.5.0.157, the request gets sent to the HTTP proxy, which refuses to proxy the request and replies with 403 Forbidden.

I recommend setting the containerd charm's no_proxy config to:

127.0.0.1,localhost,::1,10.5.0.0/16

to ensure that traffic to the 10.5.0.0/16 network does not go through the HTTP proxy. Can you give that a try and let us know if it works?

Changed in charm-aws-iam:
status: New → Incomplete
Revision history for this message
David Gomez (dvgomez) wrote :

Hey George, that seemed to do the trick. Juju status now shows calico and kubernetes-control-plane are all green now(active).

George Kraft (cynerva)
Changed in charm-aws-iam:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.