Bind backend rndc commands aren't limited
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Designate |
Fix Committed
|
High
|
Michael Chapman |
Bug Description
In a local test environment I misconfigured the pools.yaml to point at bind servers that don't exist, then ran tempest. The result was about 1500 rndc processes that don't appear to ever disappear, causing significant load on the system.
The environment is deployed using tripleo, so the processes are all in the designate-worker container:
[root@controller-1 ~]# podman stats designate_worker
ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS
18103f368b4e designate_worker 1.73% 19.13GB / 33.56GB 57.02% -- / -- 4.882GB / 8.536MB 2076
Steps to reproduce:
1. Deploy devstack with the bind backend.
2. Edit /etc/designate/
3. Update the pools: designate-manage pool update --file /etc/designate/
4. Run tempest: tox -e all-plugin -- designate
This is despite designate worker only having 2 configured workers:
/etc/designate.
[service:worker]
workers = 2
It might make sense to maintain a task queue for the rndc commands so that only a limited number can be active at any given time.
rndc doesn't have a timeout option that I can see in the man page, so it might makes sense to add one via oslo processutils. I think the built in timeout is about 30 seconds.
Changed in designate: | |
assignee: | nobody → Michael Chapman (michaeltchapman) |
summary: |
- Bind backend rndc commands have no timeout + Bind backend rndc commands aren't limited |
description: | updated |
Changed in designate: | |
status: | New → Fix Committed |
importance: | Undecided → High |
Related fix proposed to branch: master /review. opendev. org/761274
Review: https:/