Controller-0 Not Ready after force rebooting active controller (Controller-1)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Chris Friesen |
Bug Description
Brief Description
-----------------
After force rebooting controller-1, controller-0 did not reach 'Ready' status.
Severity
--------
Major
Steps to Reproduce
------------------
Force reboot active controller
Expected Behavior
------------------
Upon rebooting active controller, standby controller takes over in ready state, the system pods, applications, any test pods are up and running.
Actual Behavior
----------------
After force rebooting controller-1, controller-0 did not reach 'Ready' status.
controller-0:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION compute-0 Ready <none> 8h v1.18.1 compute-1 Ready <none> 8h v1.18.1 compute-2 Ready <none> 8h v1.18.1 controller-0 NotReady master 9h v1.18.1 controller-1 Ready master 8h v1.18.1
Following pods never reached healthy status:
cert-manager cm-cert-
cert-manager cm-cert-
cert-manager cm-cert-
kube-system coredns-
kube-system ic-nginx-
kube-system rbd-provisioner
Reproducibility
---------------
Reproduced on same lab with 2 diff builds.
System Configuration
-------
Standard System
2 Controllers and 3 Computes
LAB: WCP_71_75
Branch/Pull Time/Commit
-------
first failure
BUILD_ID=
BUILD_DATE=
second failure
BUILD_ID=
BUILD_DATE=
Last Pass
---------
Build BUILD_ID=
Timestamp/Logs
--------------
Collect all logs/ describe unhealthy pods: https:/
Test Activity
-------------
System Test Automation Development
Workaround
----------
Possible workaround:
From https:/
"... after all nodes were running again [...] restarting kubelet on the "NotReady" node was enough to make it go "Ready" again."
summary: |
Controller-0 Not Ready after force rebooting active controller - (Controller-1)) + (Controller-1) |
description: | updated |
description: | updated |
description: | updated |
Changed in starlingx: | |
assignee: | Frank Miller (sensfan22) → Chris Friesen (cbf123) |
Can you provide the output of kubectl get nodes?