Agents DoS Keystone when tokens don't validate
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Monasca |
Fix Committed
|
High
|
Ryan | ||
Bug Description
The forwarder tries to reuse tokens, and gets a new one whenever the api reports auth failure for the one it had, which sounds great until the tokens never validate for some reason. Assuming the forwarder runs every 5 seconds, on 50 compute nodes and assuming ~4 vms with different tenants per node the forwarder will make 5 (4 + default tenant) token refresh requests per run. So that's 50*5/5=50 tokens per second or almost 3000/min.
We encountered this recently during a deploy which changed a role name for the monasca-agent. The deploy updated the roles in keystone before it got to the monasca-api. This resulted in the agents getting tokens that the monasca-api didn't recognize as valid because it didn't know about the new role name.
The agent should have some mechanism preventing it from refreshing tokens this often.
| summary: |
- Agents cause DOS attack on Keystone when tokens don't validate + Agents DoS Keystone when tokens don't validate |
| Changed in monasca: | |
| assignee: | nobody → Ryan (ryan-brandt) |
| importance: | Undecided → High |
| status: | New → Triaged |
| Changed in monasca: | |
| status: | Triaged → In Progress |
| OpenStack Infra (hudson-openstack) wrote Fix merged to monasca-agent (master) | #1 |
| Changed in monasca: | |
| status: | In Progress → Fix Committed |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit bed88841c58401e
Author: Ryan Brandt <email address hidden>
Date: Fri May 15 12:08:11 2015 -0600
Add wait time to refresh tokens
Add a random wait time between requests for a new token and
continue attempting to send metrics while waiting.
The Monasca client will handle retrying once, which should
catch expired tokens.
Closes-Bug: #1454432
Change-Id: Ifd2e4891c24dbb