| 2018-06-11 02:33:47 |
xiaoxu790 |
description |
gnocchi:queens
# gnocchi metric list
Unable to establish connection to http://localhost:8041/v1/metric?: HTTPConnectionPool(host='localhost', port=8041): Max retries exceeded with url: /v1/metric (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbb45ae2ad0>: Failed to establish a new connection: [Errno 111] Connection refused',))
# cat /var/lib/docker/volumes/kolla_logs/_data/gnocchi/gnocchi-api-error.log
[Sun Jun 10 06:20:11.478074 2018] [:error] [pid 19] [remote 172.17.51.52:140] mod_wsgi (pid=19): Exception occurred processing WSGI script '/var/lib/kolla/venv/bin/gnocchi-api'.
[Sun Jun 10 06:20:11.478199 2018] [:error] [pid 19] [remote 172.17.51.52:140] Traceback (most recent call last):
[Sun Jun 10 06:20:11.478275 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
[Sun Jun 10 06:20:11.478351 2018] [:error] [pid 19] [remote 172.17.51.52:140] resp = self.call_func(req, *args, **self.kwargs)
[Sun Jun 10 06:20:11.478381 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
[Sun Jun 10 06:20:11.478428 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.func(req, *args, **kwargs)
[Sun Jun 10 06:20:11.478452 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
[Sun Jun 10 06:20:11.478501 2018] [:error] [pid 19] [remote 172.17.51.52:140] response = req.get_response(self.application)
[Sun Jun 10 06:20:11.478524 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1327, in send
[Sun Jun 10 06:20:11.478573 2018] [:error] [pid 19] [remote 172.17.51.52:140] application, catch_exc_info=False)
[Sun Jun 10 06:20:11.478598 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1291, in call_application
[Sun Jun 10 06:20:11.478646 2018] [:error] [pid 19] [remote 172.17.51.52:140] app_iter = application(self.environ, start_response)
[Sun Jun 10 06:20:11.478671 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/paste/urlmap.py", line 216, in __call__
[Sun Jun 10 06:20:11.478716 2018] [:error] [pid 19] [remote 172.17.51.52:140] return app(environ, start_response)
[Sun Jun 10 06:20:11.478739 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
[Sun Jun 10 06:20:11.478784 2018] [:error] [pid 19] [remote 172.17.51.52:140] resp = self.call_func(req, *args, **self.kwargs)
[Sun Jun 10 06:20:11.478809 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
[Sun Jun 10 06:20:11.478851 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.func(req, *args, **kwargs)
[Sun Jun 10 06:20:11.478874 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
[Sun Jun 10 06:20:11.478917 2018] [:error] [pid 19] [remote 172.17.51.52:140] response = req.get_response(self.application)
[Sun Jun 10 06:20:11.478941 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1327, in send
[Sun Jun 10 06:20:11.478984 2018] [:error] [pid 19] [remote 172.17.51.52:140] application, catch_exc_info=False)
[Sun Jun 10 06:20:11.479007 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1291, in call_application
[Sun Jun 10 06:20:11.479048 2018] [:error] [pid 19] [remote 172.17.51.52:140] app_iter = application(self.environ, start_response)
[Sun Jun 10 06:20:11.479072 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
[Sun Jun 10 06:20:11.479113 2018] [:error] [pid 19] [remote 172.17.51.52:140] resp = self.call_func(req, *args, **self.kwargs)
[Sun Jun 10 06:20:11.479136 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
[Sun Jun 10 06:20:11.479199 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.func(req, *args, **kwargs)
[Sun Jun 10 06:20:11.479237 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 337, in __call__
[Sun Jun 10 06:20:11.479296 2018] [:error] [pid 19] [remote 172.17.51.52:140] response = req.get_response(self._app)
[Sun Jun 10 06:20:11.479320 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1327, in send
[Sun Jun 10 06:20:11.479363 2018] [:error] [pid 19] [remote 172.17.51.52:140] application, catch_exc_info=False)
[Sun Jun 10 06:20:11.479386 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1291, in call_application
[Sun Jun 10 06:20:11.479427 2018] [:error] [pid 19] [remote 172.17.51.52:140] app_iter = application(self.environ, start_response)
[Sun Jun 10 06:20:11.479450 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/exc.py", line 1162, in __call__
[Sun Jun 10 06:20:11.479496 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.application(environ, start_response)
[Sun Jun 10 06:20:11.479520 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/rest/app.py", line 124, in __call__
[Sun Jun 10 06:20:11.479566 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.app(environ, start_response)
[Sun Jun 10 06:20:11.479590 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
[Sun Jun 10 06:20:11.479641 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.application(environ, start_response)
[Sun Jun 10 06:20:11.479665 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/core.py", line 840, in __call__
[Sun Jun 10 06:20:11.479710 2018] [:error] [pid 19] [remote 172.17.51.52:140] return super(Pecan, self).__call__(environ, start_response)
[Sun Jun 10 06:20:11.479731 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__
[Sun Jun 10 06:20:11.479776 2018] [:error] [pid 19] [remote 172.17.51.52:140] self.invoke_controller(controller, args, kwargs, state)
[Sun Jun 10 06:20:11.479799 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
[Sun Jun 10 06:20:11.479837 2018] [:error] [pid 19] [remote 172.17.51.52:140] result = controller(*args, **kwargs)
[Sun Jun 10 06:20:11.479874 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/rest/api.py", line 1667, in post
[Sun Jun 10 06:20:11.479938 2018] [:error] [pid 19] [remote 172.17.51.52:140] for metric in known_metrics))
[Sun Jun 10 06:20:11.479961 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/incoming/ceph.py", line 89, in add_measures_batch
[Sun Jun 10 06:20:11.480010 2018] [:error] [pid 19] [remote 172.17.51.52:140] sack = self.get_sack_name(self.sack_for_metric(metric_id))
[Sun Jun 10 06:20:11.480035 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 170, in sack_for_metric
[Sun Jun 10 06:20:11.480081 2018] [:error] [pid 19] [remote 172.17.51.52:140] return metric_id.int % self.NUM_SACKS
[Sun Jun 10 06:20:11.480106 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 53, in NUM_SACKS
[Sun Jun 10 06:20:11.480154 2018] [:error] [pid 19] [remote 172.17.51.52:140] raise SackDetectionError(e)
[Sun Jun 10 06:20:11.480230 2018] [:error] [pid 19] [remote 172.17.51.52:140] SackDetectionError: [errno 2] Ioctx.read(gnocchi): failed to read gnocchi-config
# cat /etc/kolla/gnocchi-api/gnocchi.conf
[storage]
#driver = file
#file_basepath = /var/lib/gnocchi
driver = ceph
ceph_username = gnocchi
ceph_keyring = /etc/ceph/ceph.client.gnocchi.keyring
ceph_conffile = /etc/ceph/ceph.conf
# ceph -s
cluster:
id: 6af4cb86-21b9-4eba-be44-77827a7906bf
health: HEALTH_OK
services:
mon: 3 daemons, quorum computer1,computer2,computer3
mgr: computer1(active), standbys: computer2, computer3
osd: 9 osds: 9 up, 9 in
data:
pools: 7 pools, 512 pgs
objects: 77307 objects, 45907 MB
usage: 126 GB used, 3891 GB / 4018 GB avail
pgs: 512 active+clean |
gnocchi:queens
# gnocchi measures show fc375953-0ca0-4ada-b702-08a07fa768ce
（not measures data）
# cat /var/lib/docker/volumes/kolla_logs/_data/gnocchi/gnocchi-api-error.log
[Sun Jun 10 06:20:11.478074 2018] [:error] [pid 19] [remote 172.17.51.52:140] mod_wsgi (pid=19): Exception occurred processing WSGI script '/var/lib/kolla/venv/bin/gnocchi-api'.
[Sun Jun 10 06:20:11.478199 2018] [:error] [pid 19] [remote 172.17.51.52:140] Traceback (most recent call last):
[Sun Jun 10 06:20:11.478275 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
[Sun Jun 10 06:20:11.478351 2018] [:error] [pid 19] [remote 172.17.51.52:140] resp = self.call_func(req, *args, **self.kwargs)
[Sun Jun 10 06:20:11.478381 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
[Sun Jun 10 06:20:11.478428 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.func(req, *args, **kwargs)
[Sun Jun 10 06:20:11.478452 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
[Sun Jun 10 06:20:11.478501 2018] [:error] [pid 19] [remote 172.17.51.52:140] response = req.get_response(self.application)
[Sun Jun 10 06:20:11.478524 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1327, in send
[Sun Jun 10 06:20:11.478573 2018] [:error] [pid 19] [remote 172.17.51.52:140] application, catch_exc_info=False)
[Sun Jun 10 06:20:11.478598 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1291, in call_application
[Sun Jun 10 06:20:11.478646 2018] [:error] [pid 19] [remote 172.17.51.52:140] app_iter = application(self.environ, start_response)
[Sun Jun 10 06:20:11.478671 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/paste/urlmap.py", line 216, in __call__
[Sun Jun 10 06:20:11.478716 2018] [:error] [pid 19] [remote 172.17.51.52:140] return app(environ, start_response)
[Sun Jun 10 06:20:11.478739 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
[Sun Jun 10 06:20:11.478784 2018] [:error] [pid 19] [remote 172.17.51.52:140] resp = self.call_func(req, *args, **self.kwargs)
[Sun Jun 10 06:20:11.478809 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
[Sun Jun 10 06:20:11.478851 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.func(req, *args, **kwargs)
[Sun Jun 10 06:20:11.478874 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
[Sun Jun 10 06:20:11.478917 2018] [:error] [pid 19] [remote 172.17.51.52:140] response = req.get_response(self.application)
[Sun Jun 10 06:20:11.478941 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1327, in send
[Sun Jun 10 06:20:11.478984 2018] [:error] [pid 19] [remote 172.17.51.52:140] application, catch_exc_info=False)
[Sun Jun 10 06:20:11.479007 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1291, in call_application
[Sun Jun 10 06:20:11.479048 2018] [:error] [pid 19] [remote 172.17.51.52:140] app_iter = application(self.environ, start_response)
[Sun Jun 10 06:20:11.479072 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
[Sun Jun 10 06:20:11.479113 2018] [:error] [pid 19] [remote 172.17.51.52:140] resp = self.call_func(req, *args, **self.kwargs)
[Sun Jun 10 06:20:11.479136 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
[Sun Jun 10 06:20:11.479199 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.func(req, *args, **kwargs)
[Sun Jun 10 06:20:11.479237 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 337, in __call__
[Sun Jun 10 06:20:11.479296 2018] [:error] [pid 19] [remote 172.17.51.52:140] response = req.get_response(self._app)
[Sun Jun 10 06:20:11.479320 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1327, in send
[Sun Jun 10 06:20:11.479363 2018] [:error] [pid 19] [remote 172.17.51.52:140] application, catch_exc_info=False)
[Sun Jun 10 06:20:11.479386 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1291, in call_application
[Sun Jun 10 06:20:11.479427 2018] [:error] [pid 19] [remote 172.17.51.52:140] app_iter = application(self.environ, start_response)
[Sun Jun 10 06:20:11.479450 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/exc.py", line 1162, in __call__
[Sun Jun 10 06:20:11.479496 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.application(environ, start_response)
[Sun Jun 10 06:20:11.479520 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/rest/app.py", line 124, in __call__
[Sun Jun 10 06:20:11.479566 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.app(environ, start_response)
[Sun Jun 10 06:20:11.479590 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
[Sun Jun 10 06:20:11.479641 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.application(environ, start_response)
[Sun Jun 10 06:20:11.479665 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/core.py", line 840, in __call__
[Sun Jun 10 06:20:11.479710 2018] [:error] [pid 19] [remote 172.17.51.52:140] return super(Pecan, self).__call__(environ, start_response)
[Sun Jun 10 06:20:11.479731 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__
[Sun Jun 10 06:20:11.479776 2018] [:error] [pid 19] [remote 172.17.51.52:140] self.invoke_controller(controller, args, kwargs, state)
[Sun Jun 10 06:20:11.479799 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
[Sun Jun 10 06:20:11.479837 2018] [:error] [pid 19] [remote 172.17.51.52:140] result = controller(*args, **kwargs)
[Sun Jun 10 06:20:11.479874 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/rest/api.py", line 1667, in post
[Sun Jun 10 06:20:11.479938 2018] [:error] [pid 19] [remote 172.17.51.52:140] for metric in known_metrics))
[Sun Jun 10 06:20:11.479961 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/incoming/ceph.py", line 89, in add_measures_batch
[Sun Jun 10 06:20:11.480010 2018] [:error] [pid 19] [remote 172.17.51.52:140] sack = self.get_sack_name(self.sack_for_metric(metric_id))
[Sun Jun 10 06:20:11.480035 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 170, in sack_for_metric
[Sun Jun 10 06:20:11.480081 2018] [:error] [pid 19] [remote 172.17.51.52:140] return metric_id.int % self.NUM_SACKS
[Sun Jun 10 06:20:11.480106 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 53, in NUM_SACKS
[Sun Jun 10 06:20:11.480154 2018] [:error] [pid 19] [remote 172.17.51.52:140] raise SackDetectionError(e)
[Sun Jun 10 06:20:11.480230 2018] [:error] [pid 19] [remote 172.17.51.52:140] SackDetectionError: [errno 2] Ioctx.read(gnocchi): failed to read gnocchi-config
# cat /etc/kolla/gnocchi-api/gnocchi.conf
[storage]
#driver = file
#file_basepath = /var/lib/gnocchi
driver = ceph
ceph_username = gnocchi
ceph_keyring = /etc/ceph/ceph.client.gnocchi.keyring
ceph_conffile = /etc/ceph/ceph.conf
# ceph -s
cluster:
id: 6af4cb86-21b9-4eba-be44-77827a7906bf
health: HEALTH_OK
services:
mon: 3 daemons, quorum computer1,computer2,computer3
mgr: computer1(active), standbys: computer2, computer3
osd: 9 osds: 9 up, 9 in
data:
pools: 7 pools, 512 pgs
objects: 77307 objects, 45907 MB
usage: 126 GB used, 3891 GB / 4018 GB avail
pgs: 512 active+clean
# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR
gnocchi 0 0 0 0 0 0 0 0 0 0 0 |
|
