Persister shutdowns continuously

Bug #1619335 reported by Yu Aoki
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Monasca
Incomplete
Undecided
Unassigned

Bug Description

Persister shutdowns continuously with the following logs. Please investigate.

2016-08-29 14:50:02.750 17673 ERROR repositories.persister InfluxDBClientError: 400: partial write:
2016-08-29 14:50:02.750 17673 ERROR repositories.persister unable to parse 'http_status,_region=useast,_tenant_id=1e01b2fcb628400183ba20ac61908aa4,hostname=server6560,service=orchestration,url=http://localhost:8004 value=1.0,value_meta="{\"error\": \"Pattern match failed! \\".*200 OK.*\\" not in \\"{\\"versions\\": [{\\"status\\": \\"CURRENT\\", \\"id\\": \\"v1.0\\", \\"links\\": [{\\"href\\": \\"http://localhost:8004/v1/\\", \\"rel\\": \\"self\\"}]}]}\\"\"}" 1472482232000': bad timestamp
2016-08-29 14:50:02.750 17673 ERROR repositories.persister
2016-08-29 14:50:02.750 17673 ERROR repositories.persister
2016-08-29 14:50:02.763 17673 ERROR repositories.persister [-] Persister encountered fatal exception processing messages. Shutting down all threads and exiting
2016-08-29 14:50:02.763 17673 ERROR repositories.persister Traceback (most recent call last):
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/lib/python2.7/site-packages/monasca_persister/repositories/persister.py", line 75, in run
2016-08-29 14:50:02.763 17673 ERROR repositories.persister self._flush()
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/lib/python2.7/site-packages/monasca_persister/repositories/persister.py", line 51, in _flush
2016-08-29 14:50:02.763 17673 ERROR repositories.persister self.repository.write_batch(self._data_points)
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/lib/python2.7/site-packages/monasca_persister/repositories/influxdb/abstract_repository.py", line 36, in write_batch
2016-08-29 14:50:02.763 17673 ERROR repositories.persister self._influxdb_client.write_points(data_points, 'ms')
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/local/lib/python2.7/site-packages/influxdb/client.py", line 383, in write_points
2016-08-29 14:50:02.763 17673 ERROR repositories.persister tags=tags)
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/local/lib/python2.7/site-packages/influxdb/client.py", line 428, in _write_points
2016-08-29 14:50:02.763 17673 ERROR repositories.persister expected_response_code=204
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/local/lib/python2.7/site-packages/influxdb/client.py", line 273, in write
2016-08-29 14:50:02.763 17673 ERROR repositories.persister headers=headers
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/local/lib/python2.7/site-packages/influxdb/client.py", line 243, in request
2016-08-29 14:50:02.763 17673 ERROR repositories.persister raise InfluxDBClientError(response.content, response.status_code)
2016-08-29 14:50:02.763 17673 ERROR repositories.persister InfluxDBClientError: 400: partial write:
2016-08-29 14:50:02.763 17673 ERROR repositories.persister unable to parse 'http_status,_region=useast,_tenant_id=1e01b2fcb628400183ba20ac61908aa4,hostname=server6560,service=orchestration,url=http://localhost:8004 value=1.0,value_meta="{\"error\": \"Pattern match failed! \\".*200 OK.*\\" not in \\"{\\"versions\\": [{\\"status\\": \\"CURRENT\\", \\"id\\": \\"v1.0\\", \\"links\\": [{\\"href\\": \\"http://localhost:8004/v1/\\", \\"rel\\": \\"self\\"}]}]}\\"\"}" 1472482232000': bad timestamp
2016-08-29 14:50:02.763 17673 ERROR repositories.persister
2016-08-29 14:50:02.763 17673 ERROR repositories.persister
2016-08-29 14:50:02.768 17614 INFO __main__ [-] Received signal 17, beginning graceful shutdown.

Revision history for this message
Yu Aoki (yu-aoki) wrote :
Revision history for this message
Ekasit Kijsipongse (ekasit-kijsipongse) wrote :

Here is my workaround.

The Heat's match_pattern in /etc/monasca/agent/conf.d/http_check.yaml is not correct. Try changing it to
match_pattern: .*v1.0.* (my installation uses v1.0). Then, restart supervisord.

You may need to delete old metrics still being retained in Kafka too.
1) Stop Kafka.
2) Remove everything from /tmp/kafka-logs/*
3) Start Kafka

Revision history for this message
Ekasit Kijsipongse (ekasit-kijsipongse) wrote :

For the root cause, I think monasca-collector does not correctly escape comma "," in the error message when the pattern is not found.

Revision history for this message
Ryan (ryan-brandt) wrote :

Can you please provide the version of the persister you encounter the error with?

Changed in monasca:
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.