unexpected pod restart during high memory pressure
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Snap |
New
|
Undecided
|
Unassigned |
Bug Description
Configured machine only has 10GB of RAM; after cluster bootstrap memory utilisation was at 85%; configure command executed during which (after image was uploaded) all of the pods in the microk8s deployment restarted.
A few issues
a) why did the pods get restarted
b) mysql did restart and recovered but no mysql-router instances where able to reconnect to the backend database:
2023-06-06 10:23:45 metadata_cache INFO [7feee411b640] Starting metadata cache refresh thread
2023-06-06 10:23:45 metadata_cache WARNING [7feee411b640] Failed connecting with Metadata Server mysql-0.
2023-06-06 10:23:45 metadata_cache ERROR [7feee411b640] Failed fetching metadata from any of the 1 metadata servers.
2023-06-06 10:23:45 routing INFO [7feeb1ffb640] [routing:
Just before the mass restart I also see this in the syslog:
Jun 6 09:01:52 microstack-minimum microk8s. daemon- containerd[ 24327]: time="2023- 06-06T10: 01:51.359460913 +01:00" level=info msg="shim disconnected" id=75949a72c7ef 2b38d129ab13bbf 644c4a0d57221d9 ec0e69a4bd9260e 3e1404d daemon- containerd[ 24327]: time="2023- 06-06T10: 01:52.204910617 +01:00" level=warning msg="cleaning up after shim disconnected" id=75949a72c7ef 2b38d129ab13bbf 644c4a0d57221d9 ec0e69a4bd9260e 3e1404d namespace=k8s.io daemon- containerd[ 24327]: time="2023- 06-06T10: 01:52.204959778 +01:00" level=info msg="cleaning up dead shim" daemon- containerd[ 24327]: time="2023- 06-06T10: 01:52.921974888 +01:00" level=warning msg="cleanup warnings time=\" 2023-06- 06T10:01: 52+01:00\ " level=info msg=\"starting signal loop\" namespace=k8s.io pid=471036 runtime= io.containerd. runc.v2\ n"
Jun 6 09:01:52 microstack-minimum microk8s.
Jun 6 09:01:52 microstack-minimum microk8s.
Jun 6 09:01:52 microstack-minimum microk8s.