gnocchi-metricd container missconfigured on second and third nodes

Bug #1780245 reported by Gabriel Batir
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Triaged
Medium
Unassigned
Queens
Triaged
Medium
Unassigned
Rocky
Triaged
Medium
Unassigned
Stein
Triaged
Medium
Unassigned
Train
Triaged
Medium
Unassigned

Bug Description

Hi,

I am testing kolla-ansible on three nodes.
I have an issue with the gnocchi_metricsd container on the second and third containers.

This is the error message in /var/lib/docker/volumes/kolla_logs/_data/gnocchi/gnocchi-metricd.log:

2018-07-05 08:12:36,365 [9755] ERROR cotyledon._utils: Unhandled exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cotyledon/_utils.py", line 84, in exit_on_exception
    yield
  File "/usr/lib/python2.7/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 76, in run
    self._configure()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 214, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 295, in call
    start_time=start_time)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 252, in iter
    return fut.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 161, in _configure
    six.moves.range(self.incoming.NUM_SACKS))
  File "/usr/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 53, in NUM_SACKS
    raise SackDetectionError(e)
SackDetectionError: [Errno 2] No such file or directory: '/var/lib/gnocchi/tmp/gnocchi-config'

Indeed, on the second and third nodes /var/lib/gnocchi/tmp/gnocchi-config doesn't exist while on the first node the file exists with the following content:

{"sacks": 128}

Also, ont he second and third nodes the gnocchi volumes are empty.

From what I could figure out the gnocchi volume is populated in the "Running gnocchi bootstrap container" ansible task that runs only on the first node.

Running gnocchi status returns a 500 Internal Server Error when the request hits the nodes with this problem.

kolla version is 6.0.0 running on Centos7 with centos binary containers.

Revision history for this message
Björn Hagemeier (b-hagemeier) wrote :

Hi there,

has there been any progress on this? I am having the exact same problem. What can I do to get around this problem?

Best regards,
Björn

Revision history for this message
Björn Hagemeier (b-hagemeier) wrote :

Hi all,

I just gave it a little try and did a 'gnocchi-upgrade' inside the container. That worked a deal and the missing gnocchi-conf has been created.

I found the idea for this in the comment of another bug: https://bugs.launchpad.net/kolla/+bug/1803522/comments/3

Best regards,
Björn

Revision history for this message
Maciej Kucia (maciejkucia) wrote :

gnocchi-upgrade is executed during bootstrapping:
 extended_start.sh:
 if [[ "${!KOLLA_BOOTSTRAP[@]}" ]]; then
     sudo -H -u gnocchi gnocchi-upgrade --log-file /var/log/kolla/gnocchi/gnocchi-upgrade.log
     exit 0
 fi

In kolla-ansible the bootstraping is done on single node.
 bootstrap_service.yml:
 - name: Running gnocchi bootstrap container
   kolla_docker:
     action: "start_container"
     environment:
       KOLLA_BOOTSTRAP:
       KOLLA_CONFIG_STRATEGY: "{{ config_strategy }}"
   [...]
   delegate_to: "{{ groups[gnocchi_api.group][0] }}"

Revision history for this message
Mark Goddard (mgoddard) wrote :

Looks like you need to remove run_once and delegate_to from that task.

Changed in kolla-ansible:
status: New → Triaged
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.