Ubuntu
gnocchi package

SackDetectionError: [errno 2] Ioctx.read(gnocchi): failed to read gnocchi-config

Bug #1776061 reported by xiaoxu790 on 2018-06-10

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	gnocchi (Ubuntu)	New	Undecided	Unassigned

Bug Description

gnocchi:queens

# gnocchi measures show fc375953-0ca0-4ada-b702-08a07fa768ce
（not measures data）

# cat /var/lib/docker/volumes/kolla_logs/_data/gnocchi/gnocchi-api-error.log
[Sun Jun 10 06:20:11.478074 2018] [:error] [pid 19] [remote 172.17.51.52:140] mod_wsgi (pid=19): Exception occurred processing WSGI script '/var/lib/kolla/venv/bin/gnocchi-api'.
[Sun Jun 10 06:20:11.478199 2018] [:error] [pid 19] [remote 172.17.51.52:140] Traceback (most recent call last):
[Sun Jun 10 06:20:11.478275 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
[Sun Jun 10 06:20:11.478351 2018] [:error] [pid 19] [remote 172.17.51.52:140] resp = self.call_func(req, *args, **self.kwargs)
[Sun Jun 10 06:20:11.478381 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
[Sun Jun 10 06:20:11.478428 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.func(req, *args, **kwargs)
[Sun Jun 10 06:20:11.478452 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
[Sun Jun 10 06:20:11.478501 2018] [:error] [pid 19] [remote 172.17.51.52:140] response = req.get_response(self.application)
[Sun Jun 10 06:20:11.478524 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1327, in send
[Sun Jun 10 06:20:11.478573 2018] [:error] [pid 19] [remote 172.17.51.52:140] application, catch_exc_info=False)
[Sun Jun 10 06:20:11.478598 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1291, in call_application
[Sun Jun 10 06:20:11.478646 2018] [:error] [pid 19] [remote 172.17.51.52:140] app_iter = application(self.environ, start_response)
[Sun Jun 10 06:20:11.478671 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/paste/urlmap.py", line 216, in __call__
[Sun Jun 10 06:20:11.478716 2018] [:error] [pid 19] [remote 172.17.51.52:140] return app(environ, start_response)
[Sun Jun 10 06:20:11.478739 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
[Sun Jun 10 06:20:11.478784 2018] [:error] [pid 19] [remote 172.17.51.52:140] resp = self.call_func(req, *args, **self.kwargs)
[Sun Jun 10 06:20:11.478809 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
[Sun Jun 10 06:20:11.478851 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.func(req, *args, **kwargs)
[Sun Jun 10 06:20:11.478874 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
[Sun Jun 10 06:20:11.478917 2018] [:error] [pid 19] [remote 172.17.51.52:140] response = req.get_response(self.application)
[Sun Jun 10 06:20:11.478941 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1327, in send
[Sun Jun 10 06:20:11.478984 2018] [:error] [pid 19] [remote 172.17.51.52:140] application, catch_exc_info=False)
[Sun Jun 10 06:20:11.479007 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1291, in call_application
[Sun Jun 10 06:20:11.479048 2018] [:error] [pid 19] [remote 172.17.51.52:140] app_iter = application(self.environ, start_response)
[Sun Jun 10 06:20:11.479072 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
[Sun Jun 10 06:20:11.479113 2018] [:error] [pid 19] [remote 172.17.51.52:140] resp = self.call_func(req, *args, **self.kwargs)
[Sun Jun 10 06:20:11.479136 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
[Sun Jun 10 06:20:11.479199 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.func(req, *args, **kwargs)
[Sun Jun 10 06:20:11.479237 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 337, in __call__
[Sun Jun 10 06:20:11.479296 2018] [:error] [pid 19] [remote 172.17.51.52:140] response = req.get_response(self._app)
[Sun Jun 10 06:20:11.479320 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1327, in send
[Sun Jun 10 06:20:11.479363 2018] [:error] [pid 19] [remote 172.17.51.52:140] application, catch_exc_info=False)
[Sun Jun 10 06:20:11.479386 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1291, in call_application
[Sun Jun 10 06:20:11.479427 2018] [:error] [pid 19] [remote 172.17.51.52:140] app_iter = application(self.environ, start_response)
[Sun Jun 10 06:20:11.479450 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/exc.py", line 1162, in __call__
[Sun Jun 10 06:20:11.479496 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.application(environ, start_response)
[Sun Jun 10 06:20:11.479520 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/rest/app.py", line 124, in __call__
[Sun Jun 10 06:20:11.479566 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.app(environ, start_response)
[Sun Jun 10 06:20:11.479590 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
[Sun Jun 10 06:20:11.479641 2018] [:error] [pid 19] [remote 172.17.51.52:140] return self.application(environ, start_response)
[Sun Jun 10 06:20:11.479665 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/core.py", line 840, in __call__
[Sun Jun 10 06:20:11.479710 2018] [:error] [pid 19] [remote 172.17.51.52:140] return super(Pecan, self).__call__(environ, start_response)
[Sun Jun 10 06:20:11.479731 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__
[Sun Jun 10 06:20:11.479776 2018] [:error] [pid 19] [remote 172.17.51.52:140] self.invoke_controller(controller, args, kwargs, state)
[Sun Jun 10 06:20:11.479799 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
[Sun Jun 10 06:20:11.479837 2018] [:error] [pid 19] [remote 172.17.51.52:140] result = controller(*args, **kwargs)
[Sun Jun 10 06:20:11.479874 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/rest/api.py", line 1667, in post
[Sun Jun 10 06:20:11.479938 2018] [:error] [pid 19] [remote 172.17.51.52:140] for metric in known_metrics))
[Sun Jun 10 06:20:11.479961 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/incoming/ceph.py", line 89, in add_measures_batch
[Sun Jun 10 06:20:11.480010 2018] [:error] [pid 19] [remote 172.17.51.52:140] sack = self.get_sack_name(self.sack_for_metric(metric_id))
[Sun Jun 10 06:20:11.480035 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 170, in sack_for_metric
[Sun Jun 10 06:20:11.480081 2018] [:error] [pid 19] [remote 172.17.51.52:140] return metric_id.int % self.NUM_SACKS
[Sun Jun 10 06:20:11.480106 2018] [:error] [pid 19] [remote 172.17.51.52:140] File "/var/lib/kolla/venv/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 53, in NUM_SACKS
[Sun Jun 10 06:20:11.480154 2018] [:error] [pid 19] [remote 172.17.51.52:140] raise SackDetectionError(e)
[Sun Jun 10 06:20:11.480230 2018] [:error] [pid 19] [remote 172.17.51.52:140] SackDetectionError: [errno 2] Ioctx.read(gnocchi): failed to read gnocchi-config

# cat /etc/kolla/gnocchi-api/gnocchi.conf
[storage]
#driver = file
#file_basepath = /var/lib/gnocchi
driver = ceph
ceph_username = gnocchi
ceph_keyring = /etc/ceph/ceph.client.gnocchi.keyring
ceph_conffile = /etc/ceph/ceph.conf

# ceph -s
  cluster:
    id: 6af4cb86-21b9-4eba-be44-77827a7906bf
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum computer1,computer2,computer3
    mgr: computer1(active), standbys: computer2, computer3
    osd: 9 osds: 9 up, 9 in

  data:
    pools: 7 pools, 512 pgs
    objects: 77307 objects, 45907 MB
    usage: 126 GB used, 3891 GB / 4018 GB avail
    pgs: 512 active+clean

# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR
gnocchi 0 0 0 0 0 0 0 0 0 0 0

See original description

xiaoxu790 (xiaoxu790) on 2018-06-11

description:

updated

Revision history for this message

Yoo KyoungDuk (ttog) wrote on 2018-10-19:

I think U need more information in gnocchi.conf.
I got a same error when deploying openstack(queens) by kolla-ansible and i solved this problem adding sacks_number to gnocchi.conf.

Here is my default contents of gnocchi.conf file.
# cat gnocchi.conf
[DEFAULT]
debug = True
log_dir = /var/log/kolla/gnocchi
log_file = /var/log/kolla/gnocchi/gnocchi-api.log
sacks_number = 10

When i deployed first time by kolla-ansible, there was no sacks_number variable in gnocchi.conf. If you add it and restart gnocchi, you'll may not find more error message.

PS. Actually i'm not sure '10' is proper number.
If you want right sacks_number on your environment, please check this out.
sacks value = number of **active** metrics / 300

https://gnocchi.xyz/operating.html?highlight=sacks