Ceilometer

Timeout should be added to http request in pollsters

Bug #1388778 reported by ZhiQiang Fan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
High
ZhiQiang Fan
Ceilometer 2015.1.0 "kilo"
Juno
Fix Released
High
Eoghan Glynn
Ceilometer 2014.2.1

Bug Description

Currently, we lots of generate samples by polling data from other services, but theses rest request have no timeout limitation. We have observed that some requests (for example, keystone due to openssl problem) may stuck for over several days (maybe forever if we don't restart the service). Other pollsters in same thread will not be able to work. The worst thing is that, when outside (keystone) service becomes normal, Ceilometer cannot recover itself automatically, cloud operator needs to restart it manually.

So I strongly suggest that we should add timeout limit to **every** rest api call, this is quite important to Ceilometer's robust and reliability.

See original description
ZhiQiang Fan (aji-zqfan)
Changed in ceilometer:
assignee: nobody → ZhiQiang Fan (aji-zqfan)
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/132974

Changed in ceilometer:
status: New → In Progress
Eoghan Glynn (eglynn)
Changed in ceilometer:
milestone: none → kilo-1
Eoghan Glynn (eglynn)
Changed in ceilometer:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/132974
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=bd0244ffe63b752649ae74f65a46563e986dcb00
Submitter: Jenkins
Branch: master

commit bd0244ffe63b752649ae74f65a46563e986dcb00
Author: ZhiQiang Fan <email address hidden>
Date: Mon Nov 3 20:57:04 2014 +0800

    Add timeout to all http requests

    Currently, we generate lots of samples by polling data from other services,
    but theses rest requests have no timeout limitation. We have observed that
    some requests (for example, keystone due to openssl problem) may stuck for
    over several days (maybe forever if we don't restart the service). Other
    pollsters in same thread will not be able to work too. The worst thing is
    that, when outside (keystone) service becomes normal, Ceilometer cannot
    recover itself automatically, cloud operator needs to restart it manually.

    So I strongly suggest that we should add timeout limit to **every** rest api
    call, this is quite important to improve Ceilometer's robust and reliability.

    This patch adds a new option named http_timeout, and applies it to almost
    all http requests in Ceilometer project.

    Change-Id: I76df2c0a9ffacb252e15edbb125e37ccb2aac4aa
    Closes-Bug: #1388778

Changed in ceilometer:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/137682

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/juno)

Reviewed: https://review.openstack.org/137682
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=638c66c889caef65873a9c4df5760583ad29ce3a
Submitter: Jenkins
Branch: stable/juno

commit 638c66c889caef65873a9c4df5760583ad29ce3a
Author: ZhiQiang Fan <email address hidden>
Date: Mon Nov 3 20:57:04 2014 +0800

    Add timeout to all http requests

    Currently, we generate lots of samples by polling data from other services,
    but theses rest requests have no timeout limitation. We have observed that
    some requests (for example, keystone due to openssl problem) may stuck for
    over several days (maybe forever if we don't restart the service). Other
    pollsters in same thread will not be able to work too. The worst thing is
    that, when outside (keystone) service becomes normal, Ceilometer cannot
    recover itself automatically, cloud operator needs to restart it manually.

    So I strongly suggest that we should add timeout limit to **every** rest api
    call, this is quite important to improve Ceilometer's robust and reliability.

    This patch adds a new option named http_timeout, and applies it to almost
    all http requests in Ceilometer project.

    Change-Id: I76df2c0a9ffacb252e15edbb125e37ccb2aac4aa
    Closes-Bug: #1388778
    (cherry picked from commit bd0244ffe63b752649ae74f65a46563e986dcb00)

Thierry Carrez (ttx)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: kilo-1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.