Replacing vault with easyrsa results in "error: You must be logged in to the server (Unauthorized)"

Bug #1956482 reported by Przemyslaw Lal
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Kubernetes Control Plane Charm
Fix Released
Medium
George Kraft

Bug Description

I tried to switch from vault to easyrsa as a certificate provider. This resulted in "error: You must be logged in to the server (Unauthorized)" being returned for each kubectl command and kubernetes-master units stuck in waiting/blocked loop.

### Steps to reproduce:
1) Deploy CDK cluster with vault acting as the certificates provider (fully initialized, "green" model).
2) Remove "vault:ceritificates" relations with kubernetes-master, kubernetes-worker and kubeapi-load-balancer apps.
3) Add relations between easyrsa and kubernetes-master, kubernetes-worker and kubeapi-load-balancer apps.
4) Model doesn't settle, all kubectl commands result in "error: You must be logged in to the server (Unauthorized)"

kube-apiserver logs get filled with error messages:

2022-01-05T13:26:12Z kube-apiserver.daemon[153375]: E0105 13:26:12.882813 153375 webhook.go:155] Failed to make webhook authenticator request: Post "https://10.5.1.148:5000/v1beta1?timeout=30s": x509: certificate signed by unknown authority

kubernetes-master juju agents get stuck on:
unit-kubernetes-master-1: 14:56:43 DEBUG unit.kubernetes-master/1.certificates-relation-changed active
unit-kubernetes-master-1: 14:56:43 INFO unit.kubernetes-master/1.juju-log certificates:34: Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'get', 'po', '-n', 'kube-system', '-o', 'json', '--request-timeout', '10s']
unit-kubernetes-master-1: 14:56:44 WARNING unit.kubernetes-master/1.certificates-relation-changed error: You must be logged in to the server (Unauthorized)

### Workaround:

Run against each kubernetes-master unit:
$ juju ssh kubernetes-master/0 sudo systemctl restart cdk.master.auth-webhook

After that Juju model turned all green and kubectl started working again with all functionalities being fully restored.

### Versions:

App Version Store Channel Rev
containerd go1.13.8 charmstore stable 200
easyrsa 3.0.1 charmstore stable 441
etcd 3.4.5 charmstore stable 655
flannel 0.11.0 charmstore stable 619
keystone 17.0.1 charmhub stable 539
keystone-mysql-router 8.0.27 charmhub stable 15
kubeapi-load-balancer 1.18.0 charmstore stable 866
kubernetes-master 1.23.1 charmstore stable 1106
kubernetes-worker 1.23.1 charmstore stable 838
mysql-innodb-cluster 8.0.27 charmstore stable 15
openstack-integrator xena charmstore stable 204
vault 1.5.9 charmhub stable 54
vault-mysql-router 8.0.27 charmhub stable 15

Juju: 2.9.15
OS: Ubuntu 20.04.3 LTS (Focal Fossa)
Kernel: 5.4.0-92-generic

Revision history for this message
George Kraft (cynerva) wrote :

Thanks for the detailed report.

The certs_changed handler[1] needs to be updated to also restart the cdk.master.auth-webhook service.

[1]: https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/2d7eda74e22b1fe67e2ed0ae17556c95da935077/reactive/kubernetes_master.py#L1227-L1230

Changed in charm-kubernetes-master:
importance: Undecided → Medium
status: New → Triaged
George Kraft (cynerva)
Changed in charm-kubernetes-master:
assignee: nobody → George Kraft (cynerva)
status: Triaged → In Progress
milestone: none → 1.24+ck1
Revision history for this message
George Kraft (cynerva) wrote :
George Kraft (cynerva)
Changed in charm-kubernetes-master:
status: In Progress → Fix Committed
tags: added: backport-needed
Adam Dyess (addyess)
tags: removed: backport-needed
Adam Dyess (addyess)
Changed in charm-kubernetes-master:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.