Newton - gnocchi-metricd is hammering redis
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
Note that this is different than https:/
I can see that metricd is hammering redis quite a bit and using a lot of CPU.
So we have:
# grep -irE "Connect from.*redis" /var/log/
25966
# grep -irE "Connect from.*redis" /var/log/
Nov 7 14:52:19 localhost haproxy[18671]: Connect from 172.16.2.5:48742 to 172.16.2.5:6379 (redis/TCP)
# grep -irE "Connect from.*redis" /var/log/
Nov 7 15:42:38 localhost haproxy[24643]: Connect from 172.16.2.5:49824 to 172.16.2.5:6379 (redis/TCP)
So in the timespan of ~50 minutes we have ~26000 requests. Which makes it ~520 requests per minute.
Which turns into ~ *8.5* requests per second.
(Not sure yet why haproxy ends up in swift.log, have not investigated that yet)
Changed in tripleo: | |
milestone: | ocata-1 → ocata-2 |
Changed in tripleo: | |
status: | New → Fix Released |
So this might have been due to https:/ /review. openstack. org/#/c/ 394387/ not being merged yet and redis was temporarily down. Still investigating