[fuel-library] Ceilometer's mongoDB is rapidly eating up partition space

Bug #1399164 reported by Michael Polenchuk
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Ivan Berezovskiy
5.1.x
Won't Fix
High
Ivan Berezovskiy
6.0.x
Fix Committed
High
Ivan Berezovskiy
6.1.x
Fix Released
High
Ivan Berezovskiy

Bug Description

Description of the environment:
Ceilometer with mongodb as backend

Steps to reproduce:
run in production environment

Expected result:
* time_to_live is configured ( N seconds > 0 )
* ceilometer-expirer is enabled to wipe out outdated metrics from database

Actual result:
ceilometer-expirer is disabled

Workaround:
Set time_to_live option in ceilometer.conf
and activate cron periodic expirer task

Impact:
Ceilometer collects quite huge amounts of metrics which keep in the database forever

In fuel 7.0 this should be done using native mongodb feature (as it's support was introduced in early kilo upstream code).

Changed in fuel:
status: New → Triaged
importance: Undecided → High
milestone: none → 6.1
assignee: nobody → Fuel Library Team (fuel-library)
tags: added: low-hanging-fruit
tags: added: release-notes
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

There is currently no best practice solution and we are too close to release of 6.0 right now. So we are putting this into release notes.

In order to apply a fix, user needs to setup cron job for ceilomter-expirer script and change time_to_live parameter to desired value in ceilometer.conf. After that he will need to restart collector service on all the controllers:

On CentOS:

/etc/init.d/openstack-ceilometer-collector restart

On Ubuntu:

/etc/init.d/ceilometer-collector restart

But user should be aware that all the old data will still be created with old ttl value, so if he needs he will need to wipe the data himself.

description: updated
Changed in fuel:
status: Triaged → In Progress
Revision history for this message
Ivan Berezovskiy (iberezovskiy) wrote :

It's impossible to choose right value ttl for ceilometer data, because it depends on ceilometer usage. For Fuel 6.1 we chose one week for storing data, for Fuel 7.0 we want to provide ability to set ttl from Web UI . Default value will be the same (one week), but it could be possible for users to change it.

Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

The bug is rather important, hence moving to Triaged for 6.0.1

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.0)

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/155368

no longer affects: fuel/7.0.x
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/148501
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4a05aa97b74097f8aae64389556175bdaeb52641
Submitter: Jenkins
Branch: master

commit 4a05aa97b74097f8aae64389556175bdaeb52641
Author: iberezovskiy <email address hidden>
Date: Tue Jan 20 14:13:51 2015 +0300

    Setup ttl for ceilometer data and cron daemon to cleanup

    * ttl is set to 1 week
    * cron job runs one a week

    Closes-bug: #1399164

    Change-Id: I5dc1ccc252602866d77ee44a3580842570401684

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/6.0)

Reviewed: https://review.openstack.org/155368
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=34b163f2ab3ca8abf53367b53fd0589a1160da0a
Submitter: Jenkins
Branch: stable/6.0

commit 34b163f2ab3ca8abf53367b53fd0589a1160da0a
Author: iberezovskiy <email address hidden>
Date: Tue Jan 20 14:13:51 2015 +0300

    Setup ttl for ceilometer data and cron daemon to cleanup

    * ttl is set to 1 week
    * cron job runs one a week

    Closes-bug: #1399164

    Change-Id: I5dc1ccc252602866d77ee44a3580842570401684

tags: added: release-notes-done
removed: release-notes
Revision history for this message
Vitaly Gusev (vgusev) wrote :

Verified on ISO #521

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.