Gnocchi freezes if coordination URL is unresponsive

Bug #1584083 reported by Nicolas Vila
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gnocchi
Fix Released
Medium
Julien Danjou
2.1
Fix Released
Medium
Julien Danjou

Bug Description

Using gnocchi 2.1.1.dev36, when configuring a coordination URL that is unresponsive, gnocchi-metricd freezes upon start. When debug logging is enabled, metricd outputs all configuration parameters and then just stays there:

(...)
2016-05-20 10:05:07.831 12240 DEBUG gnocchi.service [-] api.workers = 2 log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2525
2016-05-20 10:05:07.831 12240 DEBUG gnocchi.service [-] archive_policy.default_aggregation_methods = ['mean', 'min', 'max', 'sum', 'std', 'median', 'count', '95pct'] log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2525
2016-05-20 10:05:07.831 12240 DEBUG gnocchi.service [-] oslo_policy.policy_default_rule = default log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2525
2016-05-20 10:05:07.832 12240 DEBUG gnocchi.service [-] oslo_policy.policy_dirs = ['policy.d'] log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2525
2016-05-20 10:05:07.832 12240 DEBUG gnocchi.service [-] oslo_policy.policy_file = policy.json log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2525
2016-05-20 10:05:07.832 12240 DEBUG gnocchi.service [-] ******************************************************************************** log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2527

It would be useful for metricd to log a message saying that the coordination URL is not working, it'd help a lot with troubleshooting.

Thanks, regards.

Julien Danjou (jdanjou)
Changed in gnocchi:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Julien Danjou (jdanjou)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to gnocchi (master)

Fix proposed to branch: master
Review: https://review.openstack.org/322834

Changed in gnocchi:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to gnocchi (stable/2.1)

Fix proposed to branch: stable/2.1
Review: https://review.openstack.org/323407

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to gnocchi (master)

Reviewed: https://review.openstack.org/322834
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=8932aad1c3fa95c9ee901aab2b5c2f089a020312
Submitter: Jenkins
Branch: master

commit 8932aad1c3fa95c9ee901aab2b5c2f089a020312
Author: Julien Danjou <email address hidden>
Date: Mon May 30 15:53:18 2016 +0200

    metricd: only retry on attended errors and print error when coordinator fails

    Currently, `retrying' retry on every error raised by either the storage or
    indexer. Explicitely retry on what is expected to be failing, and not on random
    errors. Also print a log indicating something is wrong.

    Change-Id: I7d3e94cf94a22dcc08fb529537f10758db7d58ca
    Closes-Bug: #1584083

Changed in gnocchi:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to gnocchi (stable/2.1)

Reviewed: https://review.openstack.org/323407
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=6c49492db68e921b515c819463c58e4800ea0dc8
Submitter: Jenkins
Branch: stable/2.1

commit 6c49492db68e921b515c819463c58e4800ea0dc8
Author: Julien Danjou <email address hidden>
Date: Mon May 30 15:53:18 2016 +0200

    metricd: only retry on attended errors and print error when coordinator fails

    Currently, `retrying' retry on every error raised by either the storage or
    indexer. Explicitely retry on what is expected to be failing, and not on random
    errors. Also print a log indicating something is wrong.

    Change-Id: I7d3e94cf94a22dcc08fb529537f10758db7d58ca
    Closes-Bug: #1584083
    (cherry picked from commit 8932aad1c3fa95c9ee901aab2b5c2f089a020312)

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/gnocchi 2.2.0

This issue was fixed in the openstack/gnocchi 2.2.0 release.

Julien Danjou (jdanjou)
Changed in gnocchi:
status: Fix Committed → Fix Released
milestone: none → 2.2.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.