Persister shutdowns continuously with the following logs. Please investigate.
2016-08-29 14:50:02.750 17673 ERROR repositories.persister InfluxDBClientError: 400: partial write:
2016-08-29 14:50:02.750 17673 ERROR repositories.persister unable to parse 'http_status,_region=useast,_tenant_id=1e01b2fcb628400183ba20ac61908aa4,hostname=server6560,service=orchestration,url=http://localhost:8004 value=1.0,value_meta="{\"error\": \"Pattern match failed! \\".*200 OK.*\\" not in \\"{\\"versions\\": [{\\"status\\": \\"CURRENT\\", \\"id\\": \\"v1.0\\", \\"links\\": [{\\"href\\": \\"http://localhost:8004/v1/\\", \\"rel\\": \\"self\\"}]}]}\\"\"}" 1472482232000': bad timestamp
2016-08-29 14:50:02.750 17673 ERROR repositories.persister
2016-08-29 14:50:02.750 17673 ERROR repositories.persister
2016-08-29 14:50:02.763 17673 ERROR repositories.persister [-] Persister encountered fatal exception processing messages. Shutting down all threads and exiting
2016-08-29 14:50:02.763 17673 ERROR repositories.persister Traceback (most recent call last):
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/lib/python2.7/site-packages/monasca_persister/repositories/persister.py", line 75, in run
2016-08-29 14:50:02.763 17673 ERROR repositories.persister self._flush()
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/lib/python2.7/site-packages/monasca_persister/repositories/persister.py", line 51, in _flush
2016-08-29 14:50:02.763 17673 ERROR repositories.persister self.repository.write_batch(self._data_points)
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/lib/python2.7/site-packages/monasca_persister/repositories/influxdb/abstract_repository.py", line 36, in write_batch
2016-08-29 14:50:02.763 17673 ERROR repositories.persister self._influxdb_client.write_points(data_points, 'ms')
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/local/lib/python2.7/site-packages/influxdb/client.py", line 383, in write_points
2016-08-29 14:50:02.763 17673 ERROR repositories.persister tags=tags)
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/local/lib/python2.7/site-packages/influxdb/client.py", line 428, in _write_points
2016-08-29 14:50:02.763 17673 ERROR repositories.persister expected_response_code=204
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/local/lib/python2.7/site-packages/influxdb/client.py", line 273, in write
2016-08-29 14:50:02.763 17673 ERROR repositories.persister headers=headers
2016-08-29 14:50:02.763 17673 ERROR repositories.persister File "/opt/monasca-persister/local/lib/python2.7/site-packages/influxdb/client.py", line 243, in request
2016-08-29 14:50:02.763 17673 ERROR repositories.persister raise InfluxDBClientError(response.content, response.status_code)
2016-08-29 14:50:02.763 17673 ERROR repositories.persister InfluxDBClientError: 400: partial write:
2016-08-29 14:50:02.763 17673 ERROR repositories.persister unable to parse 'http_status,_region=useast,_tenant_id=1e01b2fcb628400183ba20ac61908aa4,hostname=server6560,service=orchestration,url=http://localhost:8004 value=1.0,value_meta="{\"error\": \"Pattern match failed! \\".*200 OK.*\\" not in \\"{\\"versions\\": [{\\"status\\": \\"CURRENT\\", \\"id\\": \\"v1.0\\", \\"links\\": [{\\"href\\": \\"http://localhost:8004/v1/\\", \\"rel\\": \\"self\\"}]}]}\\"\"}" 1472482232000': bad timestamp
2016-08-29 14:50:02.763 17673 ERROR repositories.persister
2016-08-29 14:50:02.763 17673 ERROR repositories.persister
2016-08-29 14:50:02.768 17614 INFO __main__ [-] Received signal 17, beginning graceful shutdown.
Here is my workaround.
The Heat's match_pattern in /etc/monasca/ agent/conf. d/http_ check.yaml is not correct. Try changing it to
match_pattern: .*v1.0.* (my installation uses v1.0). Then, restart supervisord.
You may need to delete old metrics still being retained in Kafka too.
1) Stop Kafka.
2) Remove everything from /tmp/kafka-logs/*
3) Start Kafka