When adding a new dashboard to etc/kolla/config/grafana/dashboards and running Kolla Ansible the task `TASK [grafana : Check grafana containers]` fails with a file not found error.
Steps to reproduce:
- Add a new dashboard to etc/kolla/config/grafana/dashboards
- Reconfigure Grafana
- You will then hit the issue below.
Workaround:
The current work around at the moment is to manually restart the Grafana container
System Details:
- OS: Stable/Yoga
Error Log:
TASK [grafana : Remove templated Grafana dashboards] ***********************************************************************************************************
changed: [host1]
changed: [host2]
changed: [host3]
TASK [grafana : Copying over custom dashboards] ****************************************************************************************************************
changed: [host1]
changed: [host2]
changed: [host3]
TASK [grafana : Check grafana containers] **********************************************************************************************************************
failed: [host2] (item={'key': 'grafana', 'value': {'container_name': 'grafana', 'group': 'grafana', 'enabled': True, 'image': '10.10.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200', 'volumes': ['/etc/kolla/grafana/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '', 'grafana:/var/lib/grafana/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'haproxy': {'grafana_server': {'enabled': True, 'mode': 'http', 'external': False, 'port': '3000'}, 'grafana_server_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '3000'}}}}) => changed=true
ansible_loop_var: item
item:
key: grafana
value:
container_name: grafana
dimensions: {}
enabled: true
group: grafana
haproxy:
grafana_server:
enabled: true
external: false
mode: http
port: '3000'
grafana_server_external:
enabled: true
external: true
mode: http
port: '3000'
image: 10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200
volumes:
- /etc/kolla/grafana/:/var/lib/kolla/config_files/:ro
- /etc/localtime:/etc/localtime:ro
- ''
- grafana:/var/lib/grafana/
- kolla_logs:/var/log/kolla/
msg: '''Traceback (most recent call last):\n File "/tmp/ansible_kolla_docker_payload_ya9jvcwp/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py", line 381, in main\n File "/tmp/ansible_kolla_docker_payload_ya9jvcwp/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 115, in compare_container\n self.compare_config()):\n File "/tmp/ansible_kolla_docker_payload_ya9jvcwp/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 416, in compare_config\n (exec_inspect[\''ExitCode\''], output))\nException: Failed to compare container configuration: ExitCode: 2 Message: b\''INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nERROR:__main__:Unexpected error:\\nTraceback (most recent call last):\\n File "/usr/local/bin/kolla_set_configs", line 420, in main\\n execute_config_check(config)\\n File "/usr/local/bin/kolla_set_configs", line 406, in execute_config_check\\n config_file.check()\\n File
"/usr/local/bin/kolla_set_configs", line 231, in check\\n if not self._cmp_dir(source, dest):\\n File "/usr/local/bin/kolla_set_configs", line 211, in _cmp_dir\\n if not self._cmp_file(full_path, dest_full_path):\\n File "/usr/local/bin/kolla_set_configs", line 169, in _cmp_file\\n with open(source, \\\''rb\\\'') as f1, open(dest, \\\''rb\\\'') as f2:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\''/var/lib/grafana/dashboards/openstack/hardware_overview.json\\\''\\n\''\n'''
failed: [host3] (item={'key': 'grafana', 'value': {'container_name': 'grafana', 'group': 'grafana', 'enabled': True, 'image': '10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200', 'volumes': ['/etc/kolla/grafana/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '', 'grafana:/var/lib/grafana/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'haproxy': {'grafana_server': {'enabled': True, 'mode': 'http', 'external': False, 'port': '3000'}, 'grafana_server_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '3000'}}}}) => changed=true
ansible_loop_var: item
item:
key: grafana
value:
container_name: grafana
dimensions: {}
enabled: true
group: grafana
haproxy:
grafana_server:
enabled: true
external: false
mode: http
port: '3000'
grafana_server_external:
enabled: true
external: true
mode: http
port: '3000'
image: 10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200
volumes:
- /etc/kolla/grafana/:/var/lib/kolla/config_files/:ro
- /etc/localtime:/etc/localtime:ro
- ''
- grafana:/var/lib/grafana/
- kolla_logs:/var/log/kolla/
msg: '''Traceback (most recent call last):\n File "/tmp/ansible_kolla_docker_payload__46t34ml/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py", line 381, in main\n File "/tmp/ansible_kolla_docker_payload__46t34ml/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 115, in compare_container\n self.compare_config()):\n File "/tmp/ansible_kolla_docker_payload__46t34ml/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 416, in compare_config\n (exec_inspect[\''ExitCode\''], output))\nException: Failed to compare container configuration: ExitCode: 2 Message: b\''INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nERROR:__main__:Unexpected error:\\nTraceback (most recent call last):\\n File "/usr/local/bin/kolla_set_configs", line 420, in main\\n execute_config_check(config)\\n File "/usr/local/bin/kolla_set_configs", line 406, in execute_config_check\\n config_file.check()\\n File
"/usr/local/bin/kolla_set_configs", line 231, in check\\n if not self._cmp_dir(source, dest):\\n File "/usr/local/bin/kolla_set_configs", line 211, in _cmp_dir\\n if not self._cmp_file(full_path, dest_full_path):\\n File "/usr/local/bin/kolla_set_configs", line 169, in _cmp_file\\n with open(source, \\\''rb\\\'') as f1, open(dest, \\\''rb\\\'') as f2:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\''/var/lib/grafana/dashboards/openstack/hardware_overview.json\\\''\\n\''\n'''
failed: [host1] (item={'key': 'grafana', 'value': {'container_name': 'grafana', 'group': 'grafana', 'enabled': True, 'image': '10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200', 'volumes': ['/etc/kolla/grafana/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '', 'grafana:/var/lib/grafana/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'haproxy': {'grafana_server': {'enabled': True, 'mode': 'http', 'external': False, 'port': '3000'}, 'grafana_server_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '3000'}}}}) => changed=true
ansible_loop_var: item
item:
key: grafana
value:
container_name: grafana
dimensions: {}
enabled: true
group: grafana
haproxy:
grafana_server:
enabled: true
external: false
mode: http
port: '3000'
grafana_server_external:
enabled: true
external: true
mode: http
port: '3000'
image: 10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200
volumes:
- /etc/kolla/grafana/:/var/lib/kolla/config_files/:ro
- /etc/localtime:/etc/localtime:ro
- ''
- grafana:/var/lib/grafana/
- kolla_logs:/var/log/kolla/
msg: '''Traceback (most recent call last):\n File "/tmp/ansible_kolla_docker_payload_0uk3g3pb/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py", line 381, in main\n File "/tmp/ansible_kolla_docker_payload_0uk3g3pb/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 115, in compare_container\n self.compare_config()):\n File "/tmp/ansible_kolla_docker_payload_0uk3g3pb/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 416, in compare_config\n (exec_inspect[\''ExitCode\''], output))\nException: Failed to compare container configuration: ExitCode: 2 Message: b\''INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nERROR:__main__:Unexpected error:\\nTraceback (most recent call last):\\n File "/usr/local/bin/kolla_set_configs", line 420, in main\\n execute_config_check(config)\\n File "/usr/local/bin/kolla_set_configs", line 406, in execute_config_check\\n config_file.check()\\n File
"/usr/local/bin/kolla_set_configs", line 231, in check\\n if not self._cmp_dir(source, dest):\\n File "/usr/local/bin/kolla_set_configs", line 211, in _cmp_dir\\n if not self._cmp_file(full_path, dest_full_path):\\n File "/usr/local/bin/kolla_set_configs", line 169, in _cmp_file\\n with open(source, \\\''rb\\\'') as f1, open(dest, \\\''rb\\\'') as f2:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\''/var/lib/grafana/dashboards/openstack/hardware_overview.json\\\''\\n\''\n'''
RUNNING HANDLER [grafana : Restart first grafana container] ****************************************************************************************************
RUNNING HANDLER [grafana : Waiting for grafana to start on first node] *****************************************************************************************
RUNNING HANDLER [grafana : Restart remaining grafana containers] ***********************************************************************************************
Fix proposed to branch: master /review. opendev. org/c/openstack /kolla/ +/873972
Review: https:/