Ceilometer

compute agent gives up discovery if it receives a HTTP 503 from nova-api

Bug #1644297 reported by Frode Nordahl
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Triaged
Low
Unassigned

Bug Description

If the ceilometer-agent on the compute node is started before the Nova API is quite ready to receive requests and subsequently receives a HTTP 503 error, it gives up discovery and goes into a indefinite loop doing nothing.

The end user will have to identify this and manually restart the ceilometer-agent on the compute nodes to have meters updating.

ceilometer-agent should keep retrying discovery until it succeeds.

2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client [req-70ea1c70-356b-4518-97d8-edc1652264f5 admin - - - -] Unknown Error (HTTP 503) (Request-ID: req-298939b5-601c-4592-acf3-5b01e4876ec2)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client return func(*args, **kwargs)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client search_opts=search_opts))
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client "servers")
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client resp, body = self.api.client.get(url)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client return self.request(url, 'GET', **kwargs)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 94, in request
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client raise exceptions.from_response(resp, body, url, method)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client ClientException: Unknown Error (HTTP 503) (Request-ID: req-298939b5-601c-4592-acf3-5b01e4876ec2)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client

Revision history for this message
gordon chung (chungg) wrote :

i don't know if we should retry until it succeeds but i understand the bug. maybe it's better to just let the service die after retrying for set time.

can you confirm what code branch you see this error on?

Revision history for this message
gordon chung (chungg) wrote :

don't use nova-api anymore for polling by default but people are welcome to fix it.

Changed in ceilometer:
importance: Undecided → Low
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.