etcdctl snap created thousands of systemd scope processes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Etcd Charm |
Incomplete
|
Undecided
|
Unassigned |
Bug Description
During investigation of strange CPU spikes, we've noticed that there were over 24k long running processes of etcdctl.
root@juju-
● snap.etcd.
Loaded: loaded
Transient: yes
Active: active (running) since Wed 2021-04-14 17:03:36 UTC; 1 weeks 4 days ago
Tasks: 0
Memory: 408.0K
CPU: 17ms
root@juju-
24178
When running below command it timed out
systemctl list-units --type=scope
Failed to list units: Connection timed out
To improve this issue, we had to manually stop them by:
for i in `ls | grep etcdctl`; do echo $i; systemctl stop $i; done
ubuntu@
Name Version Rev Tracking Publisher Notes
core 16-2.49.1 10908 latest/stable canonical✓ core
etcd 3.1.10 81 3.1/stable canonical✓ -
ubuntu@
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
etcd:
charm: cs:etcd-544
series: xenial
os: ubuntu
charm-origin: jujucharms
charm-name: etcd
charm-rev: 544
can-upgrade-to: cs:etcd-569
description: | updated |
Changed in charm-etcd: | |
status: | Won't Fix → New |
I can reproduce this, but it appears to be specific to LXD containers running xenial. I could not reproduce this in bionic or focal.
Given that xenial support ends in just a few days, I do not think we will be fixing this. I recommend upgrading your deployment to bionic or focal. See instructions: https:/ /ubuntu. com/kubernetes/ docs/upgrading# upgrading- the-machines- series