ceilometer.hardware.discovery polls nova api without bound

Bug #1474369 reported by Chris Dent
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
High
jasonamyers
Kilo
Fix Released
Undecided
Unassigned

Bug Description

https://github.com/openstack/ceilometer/blob/617ae2b38498f40de08c995dab3a1a435d3be3ae/ceilometer/hardware/discovery.py#L52

The hardware (snmp) pollster discovery mechanism, in the default config, polls the Nova API for a list of all servers, every single polling cycle.

This is akin the problem described in the compute-agent resource metadata caching spec: http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/resource-metadata-caching.html

If this is happening every in a default ceilometer install, with a default pipeline, it would be good to make the call have less impact.

Several things to consider:

* If changes-since is used to make subsequent calls have less impact that still means the first call will have large impact both in terms of cpu and memory usage on the nova-api side, and the memory impact _will_remain_ because of the way Python does memory allocation.
* At the moment there are three ways to turn off or change the hardware pollsters behavior:
** remove it from entry points
** set resources explicitly in pipeline.yaml
** disable hardware.* metrics in pipeline.yaml

I'm not sure this counts specifically as a bug but I wanted to get it registered somehow, lest I forget.

Changed in ceilometer:
assignee: nobody → jasonamyers (jason-jasonamyers)
gordon chung (chungg)
Changed in ceilometer:
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/206665

Changed in ceilometer:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/206665
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=bf62510102c02bb6127f1200e960c8219040656a
Submitter: Jenkins
Branch: master

commit bf62510102c02bb6127f1200e960c8219040656a
Author: Jason Myers <email address hidden>
Date: Tue Jul 28 14:29:43 2015 -0500

    Instance Cache in Node Discovery Pollster

    Leverages the same technique as the instance cache for the compute agent.

    Closes-Bug: #1474369

    Change-Id: I2ceff882a33dea6e1dfd0835c8c3ea928b81794d
    Signed-off-by: Jason Myers <email address hidden>

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: none → liberty-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: liberty-2 → 5.0.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/263495

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/kilo)

Reviewed: https://review.openstack.org/263495
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=14b897910bedde46edae1aa87118a42f162e080b
Submitter: Jenkins
Branch: stable/kilo

commit 14b897910bedde46edae1aa87118a42f162e080b
Author: Jason Myers <email address hidden>
Date: Tue Jul 28 14:29:43 2015 -0500

    Instance Cache in Node Discovery Pollster

    Leverages the same technique as the instance cache for the compute agent.

    Closes-Bug: #1474369

    Change-Id: I2ceff882a33dea6e1dfd0835c8c3ea928b81794d
    Signed-off-by: Jason Myers <email address hidden>

tags: added: in-stable-kilo
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.