Trigger nova client may cause large number of requests at same time

Bug #1412613 reported by Zeng Ming
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
High
Zeng Ming

Bug Description

For each polling inetervals, Ceilometer compute agent will trigger nova
client directly to acquire the list of VMs.

However, there's a potential risk that the workload for nova client will
increases proportional to the number of Ceilometer compute agents. In
other word, there's a possibility that thousands of Ceilometer compute
agents will call nova client in a very short time period or even same
time which may lead large number of requests to nova client in short time.
As a result, the nova client can not handle so many requests in short or
same time, and Ceilometer compute agent can't fetch metrics then.

So we think it would be better to shuffle agents to send request to Nova
to avoid large number of requests to nova client in short or same time.
This would be more performance effective and reliable.

Zeng Ming (mizeng)
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/148404

Changed in ceilometer:
assignee: nobody → Zeng Ming (mizeng)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/148404
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=3b38a1e27c75a788446aa2c6b9f1418110e99c71
Submitter: Jenkins
Branch: master

commit 3b38a1e27c75a788446aa2c6b9f1418110e99c71
Author: mizeng <email address hidden>
Date: Tue Jan 20 10:02:36 2015 +0800

    Shuffle agents to send request

    For each polling interval, Ceilometer compute agent will trigger
    nova client directly to acquire the list of VMs.

    However, there's a potential risk that the workload for nova client
    will increases proportional to the number of Ceilometer compute
    agents. In other word, there's a possibility that thousands of
    ceilometer compute agents will call nova client in a very short time
    period which may lead large number of requests to nova client in
    short time. As a result, the nova client can not handle so many
    requests in short time, and ceilometer compute agent can't fetch
    metrics.

    So we here shuffle agents to start polling task in order to fluff up
    the time of sending request to nova or other components to avoid
    large number of requests in short time. This would be more
    performance effective and reliable.

    DocImpact

    Change-Id: I12e3f104fc92fe15adc05e2b981627f31ee5bfaa
    Closes-bug: #1412613

Changed in ceilometer:
status: In Progress → Fix Committed
Eoghan Glynn (eglynn)
Changed in ceilometer:
milestone: none → kilo-3
importance: Undecided → High
Thierry Carrez (ttx)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: kilo-3 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.