Ceilometer

The processing time of ceilometer-expirer is too long when using MySQL backend

Bug #1289166 reported by Mitsuru Kanabuchi
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Medium
gordon chung
Ceilometer 2014.2 "juno"

Bug Description

We built an environment of i2 with MySQL backend and we were evaluating it.
And we tried to execute ceilometer-expirer with following datas:

  * 5,000 meter objects
  * 1,250 resource objects

The elapsed time was 178 secs.

This performance would be a problem in case of a large-scale (or middle-scale) environment.
In my trial calculation, the deletion can't keep up when 20,000 samples registered per pollster's one cycle.
20,000 samples will generated by only 1,000 VMs. It's too small.

I think the ceilometer-expirer needs performance tuning especially in case of using MySQL backend.

gordon chung (chungg)
Changed in ceilometer:
status: New → Triaged
Revision history for this message
gordon chung (chungg) wrote :

this is related to the way we delete expired data. we aren't using batch deletes (individually doing it currently)... i believe this will fall inline with our model cleanup... we can't use batch deletes because of a sourceassoc table (which we don't use). we need to clean that table from our model before we can run batch delete.

Changed in ceilometer:
importance: Undecided → Medium
Revision history for this message
Mitsuru Kanabuchi (kanabuchi) wrote :

Hi Gordon, thank you for your comment.

I agree with your opinion that this relates to the model / schema of the database.
We have been investigating the sourceassoc table, since it seems to be one of the cause of slow speed, as you have pointed out.

However, we don't understand the purpose of sourceassoc table yet.
It doesn't seem to have important roll.
We're thinking, this table might be able to remove. What do you think?

Changed in ceilometer:
assignee: nobody → Mitsuru Kanabuchi (kanabuchi)
Revision history for this message
gordon chung (chungg) wrote :

Mitsuru Kanabuchi, i hope sourceassoc, user, project, source, and resource tables can be removed as well. something to discuss at summit to redesign schema... from my view, none of these tables are useful and are just purely overhead.

Revision history for this message
Mitsuru Kanabuchi (kanabuchi) wrote :

Hi Gordon, thank you for your answer.

I think it will have great effect in improving the performance, if we can remove sourceassoc, user, project, source, and resource tables.
We hope we can join the discussion to redesign schema at summit.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/96489

Changed in ceilometer:
assignee: Mitsuru Kanabuchi (kanabuchi) → gordon chung (chungg)
status: Triaged → In Progress
Eoghan Glynn (eglynn)
Changed in ceilometer:
milestone: none → juno-2
OpenStack Infra (hudson-openstack)
Changed in ceilometer:
assignee: gordon chung (chungg) → Eoghan Glynn (eglynn)
Eoghan Glynn (eglynn)
Changed in ceilometer:
assignee: Eoghan Glynn (eglynn) → nobody
assignee: nobody → gordon chung (chungg)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/96489
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=f5a900c6225a7e6886f4cc860d7e11782e3e84ae
Submitter: Jenkins
Branch: master

commit f5a900c6225a7e6886f4cc860d7e11782e3e84ae
Author: gordon chung <email address hidden>
Date: Thu May 29 10:59:11 2014 -0400

    improve expirer performance for sql backend

    the model has been simplified which allows us to do bulk deletes
    rather than looping through and deleting records individually.
    this should improve performance significantly.

    Change-Id: Ie99880438d7415d805a94d78df648082d5161417
    Closes-Bug: #1289166

Changed in ceilometer:
status: In Progress → Fix Committed
Russell Bryant (russellb)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: juno-2 → 2014.2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ceilometer (stable/icehouse)

Change abandoned by Alberto Planas (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/175289
Reason: Icehouse is reaching the end of the live cycle.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.