mysql process consuming 400% CPU
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| tripleo |
Triaged
|
High
|
Nagasai Vinaykumar Kapalavai | ||
Bug Description
Description of problem:
On all three controllers, the mysql process is consuming around 4000% CPU.
Version-Release number of selected component (if applicable):
How reproducible:
Hard to reproduce outside of customer environment
Observed in ths OSP13,14,15
Steps to Reproduce:
1. Deploy overcloud with or without composable roles
2. Check CPU utilization of mysql process
3.
Actual results:
Very high CPU utilisation. Even after restarting galera from pcs we're seeing 400% CPU utilization. No iowait observed
Expected results:
Much lower CPU utilization
This occurs when the panko events grow indefinitely and as a result mysql process consumes 400% cpu to generate query results. The current solution to this is running the panko_api_cron in a seperate container.
| Changed in tripleo: | |
| assignee: | nobody → Nagasai Vinaykumar Kapalavai (vinaykns2) |
| Changed in tripleo: | |
| importance: | Undecided → High |
| status: | New → Triaged |
| milestone: | none → train-2 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to tripleo-heat-templates (stable/queens) | #1 |
| tags: | added: in-stable-queens |
| OpenStack Infra (hudson-openstack) wrote Fix merged to tripleo-heat-templates (stable/rocky) | #2 |
| tags: | added: in-stable-rocky |
| OpenStack Infra (hudson-openstack) wrote Fix merged to tripleo-heat-templates (stable/stein) | #3 |
| tags: | added: in-stable-stein |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/tripleo-heat-templates 9.4.0 | #4 |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/tripleo-heat-templates 8.4.0 | #5 |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/tripleo-heat-templates 10.6.0 | #6 |
| Changed in tripleo: | |
| milestone: | train-2 → train-3 |
| Changed in tripleo: | |
| milestone: | train-3 → ussuri-1 |
| Changed in tripleo: | |
| milestone: | ussuri-1 → ussuri-2 |
| Changed in tripleo: | |
| milestone: | ussuri-2 → ussuri-3 |
| Changed in tripleo: | |
| milestone: | ussuri-3 → ussuri-rc3 |
| Changed in tripleo: | |
| milestone: | ussuri-rc3 → victoria-1 |
| Changed in tripleo: | |
| milestone: | victoria-1 → victoria-3 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/queens
commit 7b1e923ede7a2c5
Author: Vinay Kapalavai <email address hidden>
Date: Fri Jun 7 15:11:43 2019 -0400
Add panko_api_cron container
Currently panko expirer isn't running in a separate container,
which results in events piling up in the configured database.
This change deploys panko_api_cron as a container and
periodically deletes events from db
Closes-Bug: #1832274
Change-Id: I4c941048a3561a
(cherry picked from commit 66a638a8df8fa5b
(cherry picked from commit 9e876dfe0d743d5