Comment 12 for bug 1672240

I try to destroy with clean image then deploy the environment but still reproduced the issue.

this time :
TASK [mariadb : Waiting for MariaDB service to be ready] ***********************
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
fatal: [kode1]: FAILED! => {"attempts": 10, "changed": false, "failed": true, "module_stderr": "Shared connection to kode1 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_Q9j2N1/ansible_module_wait_for.py\", line 540, in <module>\r\n main()\r\n File \"/tmp/ansible_Q9j2N1/ansible_module_wait_for.py\", line 481, in main\r\n response = s.recv(1024)\r\nsocket.error: [Errno 104] Connection reset by peer\r\n", "msg": "MODULE FAILURE"}
fatal: [kode2]: FAILED! => {"attempts": 10, "changed": false, "failed": true, "module_stderr": "Shared connection to kode2 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_7uFJw5/ansible_module_wait_for.py\", line 540, in <module>\r\n main()\r\n File \"/tmp/ansible_7uFJw5/ansible_module_wait_for.py\", line 481, in main\r\n response = s.recv(1024)\r\nsocket.error: [Errno 104] Connection reset by peer\r\n", "msg": "MODULE FAILURE"}
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
fatal: [kode0]: FAILED! => {"attempts": 10, "changed": false, "elapsed": 60, "failed": true, "msg": "Timeout when waiting for search string MariaDB in 192.168.102.20:3306"}
        to retry, use: --limit @/opt/bgi-kolla/kolla-ansible/ansible/site.retry

PLAY RECAP *********************************************************************
kode0 : ok=62 changed=31 unreachable=0 failed=1
kode1 : ok=58 changed=32 unreachable=0 failed=1
kode2 : ok=58 changed=32 unreachable=0 failed=1
kode3 : ok=36 changed=6 unreachable=0 failed=0
kode4 : ok=5 changed=0 unreachable=0 failed=0
kode5 : ok=5 changed=0 unreachable=0 failed=0

Command failed ansible-playbook -i ../ansible/inventory/bgi-openstack -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e horizon_tag=4.0.0.2 -e action=deploy /opt/bgi-kolla/kolla-ansible/ansible/site.yml

there are the following output on kode2 node
170318 7:50:27 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
170318 7:50:27 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'openstack' at 'gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567': -110 (Connection timed out)
170318 7:50:27 [ERROR] WSREP: gcs connect failed: Connection timed out
170318 7:50:27 [ERROR] WSREP: wsrep::connect(gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567) failed: 7
170318 7:50:27 [ERROR] Aborting

170318 7:50:27 [Note] WSREP: Service disconnected.
170318 7:50:28 [Note] WSREP: Some threads may fail to exit.
170318 7:50:28 [Note] /usr/sbin/mysqld: Shutdown complete

so I check the iptable rules and found the clue, the rule on kode1 blocked port 4567.

I collected the output of "iptables-save > iptable.${HOSTNAME}", you can check the attachement.