check grafana containers task fails when adding new dashboard

Bug #1997984 reported by Dawud
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla
Status tracked in Bobcat
Antelope
Fix Committed
Undecided
Unassigned
Bobcat
Fix Committed
Undecided
Unassigned
Xena
Won't Fix
Undecided
Unassigned
Yoga
Fix Committed
Undecided
Unassigned
Zed
Fix Committed
Undecided
Unassigned

Bug Description

When adding a new dashboard to etc/kolla/config/grafana/dashboards and running Kolla Ansible the task `TASK [grafana : Check grafana containers]` fails with a file not found error.

Steps to reproduce:
 - Add a new dashboard to etc/kolla/config/grafana/dashboards
 - Reconfigure Grafana
 - You will then hit the issue below.

Workaround:
 The current work around at the moment is to manually restart the Grafana container

System Details:
 - OS: Stable/Yoga

Error Log:

TASK [grafana : Remove templated Grafana dashboards] ***********************************************************************************************************
changed: [host1]
changed: [host2]
changed: [host3]

TASK [grafana : Copying over custom dashboards] ****************************************************************************************************************
changed: [host1]
changed: [host2]
changed: [host3]

TASK [grafana : Check grafana containers] **********************************************************************************************************************
failed: [host2] (item={'key': 'grafana', 'value': {'container_name': 'grafana', 'group': 'grafana', 'enabled': True, 'image': '10.10.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200', 'volumes': ['/etc/kolla/grafana/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '', 'grafana:/var/lib/grafana/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'haproxy': {'grafana_server': {'enabled': True, 'mode': 'http', 'external': False, 'port': '3000'}, 'grafana_server_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '3000'}}}}) => changed=true
  ansible_loop_var: item
  item:
    key: grafana
    value:
      container_name: grafana
      dimensions: {}
      enabled: true
      group: grafana
      haproxy:
        grafana_server:
          enabled: true
          external: false
          mode: http
          port: '3000'
        grafana_server_external:
          enabled: true
          external: true
          mode: http
          port: '3000'
      image: 10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200
      volumes:
      - /etc/kolla/grafana/:/var/lib/kolla/config_files/:ro
      - /etc/localtime:/etc/localtime:ro
      - ''
      - grafana:/var/lib/grafana/
      - kolla_logs:/var/log/kolla/
  msg: '''Traceback (most recent call last):\n File "/tmp/ansible_kolla_docker_payload_ya9jvcwp/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py", line 381, in main\n File "/tmp/ansible_kolla_docker_payload_ya9jvcwp/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 115, in compare_container\n self.compare_config()):\n File "/tmp/ansible_kolla_docker_payload_ya9jvcwp/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 416, in compare_config\n (exec_inspect[\''ExitCode\''], output))\nException: Failed to compare container configuration: ExitCode: 2 Message: b\''INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nERROR:__main__:Unexpected error:\\nTraceback (most recent call last):\\n File "/usr/local/bin/kolla_set_configs", line 420, in main\\n execute_config_check(config)\\n File "/usr/local/bin/kolla_set_configs", line 406, in execute_config_check\\n config_file.check()\\n File
    "/usr/local/bin/kolla_set_configs", line 231, in check\\n if not self._cmp_dir(source, dest):\\n File "/usr/local/bin/kolla_set_configs", line 211, in _cmp_dir\\n if not self._cmp_file(full_path, dest_full_path):\\n File "/usr/local/bin/kolla_set_configs", line 169, in _cmp_file\\n with open(source, \\\''rb\\\'') as f1, open(dest, \\\''rb\\\'') as f2:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\''/var/lib/grafana/dashboards/openstack/hardware_overview.json\\\''\\n\''\n'''
failed: [host3] (item={'key': 'grafana', 'value': {'container_name': 'grafana', 'group': 'grafana', 'enabled': True, 'image': '10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200', 'volumes': ['/etc/kolla/grafana/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '', 'grafana:/var/lib/grafana/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'haproxy': {'grafana_server': {'enabled': True, 'mode': 'http', 'external': False, 'port': '3000'}, 'grafana_server_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '3000'}}}}) => changed=true
  ansible_loop_var: item
  item:
    key: grafana
    value:
      container_name: grafana
      dimensions: {}
      enabled: true
      group: grafana
      haproxy:
        grafana_server:
          enabled: true
          external: false
          mode: http
          port: '3000'
        grafana_server_external:
          enabled: true
          external: true
          mode: http
          port: '3000'
      image: 10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200
      volumes:
      - /etc/kolla/grafana/:/var/lib/kolla/config_files/:ro
      - /etc/localtime:/etc/localtime:ro
      - ''
      - grafana:/var/lib/grafana/
      - kolla_logs:/var/log/kolla/
  msg: '''Traceback (most recent call last):\n File "/tmp/ansible_kolla_docker_payload__46t34ml/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py", line 381, in main\n File "/tmp/ansible_kolla_docker_payload__46t34ml/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 115, in compare_container\n self.compare_config()):\n File "/tmp/ansible_kolla_docker_payload__46t34ml/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 416, in compare_config\n (exec_inspect[\''ExitCode\''], output))\nException: Failed to compare container configuration: ExitCode: 2 Message: b\''INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nERROR:__main__:Unexpected error:\\nTraceback (most recent call last):\\n File "/usr/local/bin/kolla_set_configs", line 420, in main\\n execute_config_check(config)\\n File "/usr/local/bin/kolla_set_configs", line 406, in execute_config_check\\n config_file.check()\\n File
    "/usr/local/bin/kolla_set_configs", line 231, in check\\n if not self._cmp_dir(source, dest):\\n File "/usr/local/bin/kolla_set_configs", line 211, in _cmp_dir\\n if not self._cmp_file(full_path, dest_full_path):\\n File "/usr/local/bin/kolla_set_configs", line 169, in _cmp_file\\n with open(source, \\\''rb\\\'') as f1, open(dest, \\\''rb\\\'') as f2:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\''/var/lib/grafana/dashboards/openstack/hardware_overview.json\\\''\\n\''\n'''
failed: [host1] (item={'key': 'grafana', 'value': {'container_name': 'grafana', 'group': 'grafana', 'enabled': True, 'image': '10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200', 'volumes': ['/etc/kolla/grafana/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '', 'grafana:/var/lib/grafana/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'haproxy': {'grafana_server': {'enabled': True, 'mode': 'http', 'external': False, 'port': '3000'}, 'grafana_server_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '3000'}}}}) => changed=true
  ansible_loop_var: item
  item:
    key: grafana
    value:
      container_name: grafana
      dimensions: {}
      enabled: true
      group: grafana
      haproxy:
        grafana_server:
          enabled: true
          external: false
          mode: http
          port: '3000'
        grafana_server_external:
          enabled: true
          external: true
          mode: http
          port: '3000'
      image: 10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200
      volumes:
      - /etc/kolla/grafana/:/var/lib/kolla/config_files/:ro
      - /etc/localtime:/etc/localtime:ro
      - ''
      - grafana:/var/lib/grafana/
      - kolla_logs:/var/log/kolla/
  msg: '''Traceback (most recent call last):\n File "/tmp/ansible_kolla_docker_payload_0uk3g3pb/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py", line 381, in main\n File "/tmp/ansible_kolla_docker_payload_0uk3g3pb/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 115, in compare_container\n self.compare_config()):\n File "/tmp/ansible_kolla_docker_payload_0uk3g3pb/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 416, in compare_config\n (exec_inspect[\''ExitCode\''], output))\nException: Failed to compare container configuration: ExitCode: 2 Message: b\''INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nERROR:__main__:Unexpected error:\\nTraceback (most recent call last):\\n File "/usr/local/bin/kolla_set_configs", line 420, in main\\n execute_config_check(config)\\n File "/usr/local/bin/kolla_set_configs", line 406, in execute_config_check\\n config_file.check()\\n File
    "/usr/local/bin/kolla_set_configs", line 231, in check\\n if not self._cmp_dir(source, dest):\\n File "/usr/local/bin/kolla_set_configs", line 211, in _cmp_dir\\n if not self._cmp_file(full_path, dest_full_path):\\n File "/usr/local/bin/kolla_set_configs", line 169, in _cmp_file\\n with open(source, \\\''rb\\\'') as f1, open(dest, \\\''rb\\\'') as f2:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\''/var/lib/grafana/dashboards/openstack/hardware_overview.json\\\''\\n\''\n'''

RUNNING HANDLER [grafana : Restart first grafana container] ****************************************************************************************************

RUNNING HANDLER [grafana : Waiting for grafana to start on first node] *****************************************************************************************

RUNNING HANDLER [grafana : Restart remaining grafana containers] ***********************************************************************************************

Dawud (dawudm)
Changed in kolla:
assignee: nobody → Dawud (dawudm)
status: New → In Progress
no longer affects: kolla-ansible
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla/+/873972

affects: kolla → kolla-ansible
no longer affects: kolla-ansible
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla/+/893766

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/892876
Committed: https://opendev.org/openstack/kolla/commit/288a400dbf30797b70e64af64f6deadcca582851
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 288a400dbf30797b70e64af64f6deadcca582851
Author: Dawud M <email address hidden>
Date: Wed Feb 15 18:02:05 2023 +0000

    Add comprehensive checks for container restarts

    When adding a dashboard to grafana the containers aren't restarted when
    they should be. This is due to a bug in Kolla where the logic to
    determine whether or the container needs to be restarted fails in the
    case where the file does not exist in the container. This patch adds
    more comprehensive checks for container restarts in the set_configs.py
    file. This patch also adds a test to ensure that the functions work as
    expected.

    Closes-Bug: #1997984
    Co-Authored-By: Will Szumski <email address hidden>
    Change-Id: I67f5f12700d7b55f26bff81e9b54559303da6d83
    (cherry picked from commit d9a6c5f3901e078a30d341397ac57d867d8191cc)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla (stable/xena)

Change abandoned by "Michal Nasiadka <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla/+/893766
Reason: xena going EOL

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/892878
Committed: https://opendev.org/openstack/kolla/commit/0353c51c68c30632ecb29fd0ae4372a375086865
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 0353c51c68c30632ecb29fd0ae4372a375086865
Author: Dawud M <email address hidden>
Date: Wed Feb 15 18:02:05 2023 +0000

    Add comprehensive checks for container restarts

    When adding a dashboard to grafana the containers aren't restarted when
    they should be. This is due to a bug in Kolla where the logic to
    determine whether or the container needs to be restarted fails in the
    case where the file does not exist in the container. This patch adds
    more comprehensive checks for container restarts in the set_configs.py
    file. This patch also adds a test to ensure that the functions work as
    expected.

    Closes-Bug: #1997984
    Co-Authored-By: Will Szumski <email address hidden>
    Change-Id: I67f5f12700d7b55f26bff81e9b54559303da6d83
    (cherry picked from commit d9a6c5f3901e078a30d341397ac57d867d8191cc)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.