Error at "Check Default IPv4 Gateway availability" deploying the overcloud

Error checking the Default ipv4 gateway availability.
the 3 controllers environment try to test the 10.0.0.1 gateway but the 10.0.0.1 is not available.

ERROR
======
2021-02-10 14:19:56.057753 | 0099cddf-658b-2b90-7e73-000000002125 | TASK | Collect Network Facts
2021-02-10 14:19:56.160941 | 0099cddf-658b-2b90-7e73-000000002125 | OK | Collect Network Facts | overcloud-controller-2
2021-02-10 14:19:56.173824 | 0099cddf-658b-2b90-7e73-000000002125 | TIMING | tripleo_nodes_validation : Collect Network Facts | overcloud-controller-2 | 0:02:09.305014 | 0.44s
2021-02-10 14:19:56.226160 | 0099cddf-658b-2b90-7e73-000000002126 | TASK | Check Default IPv4 Gateway availability
2021-02-10 14:19:56.409609 | 0099cddf-658b-2b90-7e73-000000002125 | OK | Collect Network Facts | overcloud-controller-1
2021-02-10 14:19:56.417541 | 0099cddf-658b-2b90-7e73-000000002125 | TIMING | tripleo_nodes_validation : Collect Network Facts | overcloud-controller-1 | 0:02:09.548726 | 0.48s
2021-02-10 14:19:56.467037 | 0099cddf-658b-2b90-7e73-000000002126 | TASK | Check Default IPv4 Gateway availability
2021-02-10 14:19:56.488956 | 0099cddf-658b-2b90-7e73-000000002125 | OK | Collect Network Facts | overcloud-controller-0
2021-02-10 14:19:56.495701 | 0099cddf-658b-2b90-7e73-000000002125 | TIMING | tripleo_nodes_validation : Collect Network Facts | overcloud-controller-0 | 0:02:09.626905 | 0.44s
2021-02-10 14:19:56.530248 | 0099cddf-658b-2b90-7e73-000000002126 | TASK | Check Default IPv4 Gateway availability
2021-02-10 14:19:59.505414 | 0099cddf-658b-2b90-7e73-000000002126 | FATAL | Check Default IPv4 Gateway availability | overcloud-controller-2 | error={"changed": false, "cmd": ["ping", "-w", "10", "-c", "1", "10.0.0.1"], "delta": "0:00:03.061315", "end": "2021-02-10 14:19:59.489959", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 14:19:56.428644", "stderr": "", "stderr_lines": [], "stdout": "PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.\nFrom 10.0.0.225 icmp_seq=1 Destination Host Unreachable\nFrom 10.0.0.225 icmp_seq=2 Destination Host Unreachable\nFrom 10.0.0.225 icmp_seq=3 Destination Host Unreachable\n\n--- 10.0.0.1 ping statistics ---\n3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 35ms\npipe 3", "stdout_lines": ["PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.", "From 10.0.0.225 icmp_seq=1 Destination Host Unreachable", "From 10.0.0.225 icmp_seq=2 Destination Host Unreachable", "From 10.0.0.225 icmp_seq=3 Destination Host Unreachable", "", "--- 10.0.0.1 ping statistics ---", "3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 35ms", "pipe 3"]}
2021-02-10 14:19:59.506851 | 0099cddf-658b-2b90-7e73-000000002126 | TIMING | tripleo_nodes_validation : Check Default IPv4 Gateway availability | overcloud-controller-2 | 0:02:12.638061 | 3.28s
2021-02-10 14:19:59.766603 | 0099cddf-658b-2b90-7e73-000000002126 | FATAL | Check Default IPv4 Gateway availability | overcloud-controller-1 | error={"changed": false, "cmd": ["ping", "-w", "10", "-c", "1", "10.0.0.1"], "delta": "0:00:03.104717", "end": "2021-02-10 14:19:59.752598", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 14:19:56.647881", "stderr": "", "stderr_lines": [], "stdout": "PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.\nFrom 10.0.0.235 icmp_seq=1 Destination Host Unreachable\nFrom 10.0.0.235 icmp_seq=2 Destination Host Unreachable\nFrom 10.0.0.235 icmp_seq=3 Destination Host Unreachable\n\n--- 10.0.0.1 ping statistics ---\n3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 78ms\npipe 3", "stdout_lines": ["PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.", "From 10.0.0.235 icmp_seq=1 Destination Host Unreachable", "From 10.0.0.235 icmp_seq=2 Destination Host Unreachable", "From 10.0.0.235 icmp_seq=3 Destination Host Unreachable", "", "--- 10.0.0.1 ping statistics ---", "3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 78ms", "pipe 3"]}
2021-02-10 14:19:59.767482 | 0099cddf-658b-2b90-7e73-000000002126 | TIMING | tripleo_nodes_validation : Check Default IPv4 Gateway availability | overcloud-controller-1 | 0:02:12.898698 | 3.30s
2021-02-10 14:19:59.832866 | 0099cddf-658b-2b90-7e73-000000002126 | FATAL | Check Default IPv4 Gateway availability | overcloud-controller-0 | error={"changed": false, "cmd": ["ping", "-w", "10", "-c", "1", "10.0.0.1"], "delta": "0:00:03.112967", "end": "2021-02-10 14:19:59.814618", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 14:19:56.701651", "stderr": "", "stderr_lines": [], "stdout": "PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.\nFrom 10.0.0.24 icmp_seq=1 Destination Host Unreachable\nFrom 10.0.0.24 icmp_seq=2 Destination Host Unreachable\nFrom 10.0.0.24 icmp_seq=3 Destination Host Unreachable\n\n--- 10.0.0.1 ping statistics ---\n3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 81ms\npipe 3", "stdout_lines": ["PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.", "From 10.0.0.24 icmp_seq=1 Destination Host Unreachable", "From 10.0.0.24 icmp_seq=2 Destination Host Unreachable", "From 10.0.0.24 icmp_seq=3 Destination Host Unreachable", "", "--- 10.0.0.1 ping statistics ---", "3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 81ms", "pipe 3"]}
2021-02-10 14:19:59.833739 | 0099cddf-658b-2b90-7e73-000000002126 | TIMING | tripleo_nodes_validation : Check Default IPv4 Gateway availability | overcloud-controller-0 | 0:02:12.964956 | 3.30s

PLAY RECAP *********************************************************************
overcloud-cephstorage-0 : ok=67 changed=31 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0
overcloud-cephstorage-1 : ok=64 changed=31 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0
overcloud-cephstorage-2 : ok=64 changed=31 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0
overcloud-controller-0 : ok=65 changed=31 unreachable=0 failed=1 skipped=34 rescued=0 ignored=0
overcloud-controller-1 : ok=62 changed=31 unreachable=0 failed=1 skipped=34 rescued=0 ignored=0
overcloud-controller-2 : ok=62 changed=31 unreachable=0 failed=1 skipped=34 rescued=0 ignored=0
overcloud-novacompute-0 : ok=64 changed=31 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0
overcloud-novacompute-1 : ok=64 changed=31 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0
undercloud : ok=9 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
2021-02-10 14:19:59.870858 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-02-10 14:19:59.871215 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 208 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-02-10 14:19:59.871569 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:02:13.002788 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-02-10 14:19:59.871849 | UUID | Info | Host | Task Name | Run Time
2021-02-10 14:19:59.872210 | 0099cddf-658b-2b90-7e73-00000000202c | SUMMARY | overcloud-controller-1 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 33.90s
2021-02-10 14:19:59.872569 | 0099cddf-658b-2b90-7e73-00000000202c | SUMMARY | overcloud-controller-2 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 33.72s
2021-02-10 14:19:59.872901 | 0099cddf-658b-2b90-7e73-00000000207e | SUMMARY | overcloud-controller-0 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 33.53s
2021-02-10 14:19:59.873263 | 0099cddf-658b-2b90-7e73-0000000020a7 | SUMMARY | overcloud-novacompute-0 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 24.23s
2021-02-10 14:19:59.873616 | 0099cddf-658b-2b90-7e73-0000000001f4 | SUMMARY | overcloud-controller-0 | Gathering Facts | 23.85s
2021-02-10 14:19:59.873938 | 0099cddf-658b-2b90-7e73-0000000020bf | SUMMARY | overcloud-novacompute-1 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 23.73s
2021-02-10 14:19:59.874302 | 0099cddf-658b-2b90-7e73-0000000001f4 | SUMMARY | overcloud-cephstorage-0 | Gathering Facts | 23.12s
2021-02-10 14:19:59.874604 | 0099cddf-658b-2b90-7e73-0000000001f4 | SUMMARY | overcloud-cephstorage-2 | Gathering Facts | 21.77s
2021-02-10 14:19:59.874950 | 0099cddf-658b-2b90-7e73-00000000207e | SUMMARY | overcloud-cephstorage-2 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 20.96s
2021-02-10 14:19:59.875332 | 0099cddf-658b-2b90-7e73-000000002060 | SUMMARY | overcloud-cephstorage-1 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 20.86s
2021-02-10 14:19:59.875662 | 0099cddf-658b-2b90-7e73-00000000202c | SUMMARY | overcloud-cephstorage-0 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 20.73s
2021-02-10 14:19:59.875971 | 0099cddf-658b-2b90-7e73-0000000001f4 | SUMMARY | overcloud-cephstorage-1 | Gathering Facts | 20.51s
2021-02-10 14:19:59.876369 | 0099cddf-658b-2b90-7e73-0000000001f4 | SUMMARY | overcloud-novacompute-0 | Gathering Facts | 19.32s
2021-02-10 14:19:59.876676 | 0099cddf-658b-2b90-7e73-0000000001f4 | SUMMARY | overcloud-novacompute-1 | Gathering Facts | 19.16s
2021-02-10 14:19:59.877017 | 0099cddf-658b-2b90-7e73-0000000001f4 | SUMMARY | overcloud-controller-2 | Gathering Facts | 18.76s
2021-02-10 14:19:59.877355 | 0099cddf-658b-2b90-7e73-0000000001f4 | SUMMARY | overcloud-controller-1 | Gathering Facts | 18.19s
2021-02-10 14:19:59.877860 | 0099cddf-658b-2b90-7e73-000000001aa5 | SUMMARY | overcloud-controller-2 | tripleo_hieradata : Render hieradata from template | 11.12s
2021-02-10 14:19:59.878221 | 0099cddf-658b-2b90-7e73-000000001aa5 | SUMMARY | overcloud-controller-1 | tripleo_hieradata : Render hieradata from template | 10.86s
2021-02-10 14:19:59.878548 | 0099cddf-658b-2b90-7e73-000000001aa5 | SUMMARY | overcloud-controller-0 | tripleo_hieradata : Render hieradata from template | 10.83s
2021-02-10 14:19:59.878899 | 0099cddf-658b-2b90-7e73-000000001aa5 | SUMMARY | overcloud-cephstorage-2 | tripleo_hieradata : Render hieradata from template | 10.78s
2021-02-10 14:19:59.879285 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-02-10 14:19:59.879619 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-02-10 14:19:59.880022 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 3 ~~~~~~~~~~~~~~~~~
2021-02-10 14:19:59.880376 | The following node(s) had failures: overcloud-controller-0, overcloud-controller-1, overcloud-controller-2
2021-02-10 14:19:59.880743 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-02-10 14:20:00.258 235852 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleodm04ifix ] cleaned up
2021-02-10 14:20:00.258 235852 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/config-download/overcloud/ansible-playbook-command.sh
2021-02-10 14:20:00.258 235852 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleovr2gikfz ] cleaned up
Host 10.0.0.5 not found in /home/stack/.ssh/known_hosts
Overcloud Endpoint: http://10.0.0.5:5000
Overcloud Horizon Dashboard URL: http://10.0.0.5:80/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed with error
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/config-download/overcloud/ansible-playbook-command.sh
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args)
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args)
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1207, in take_action
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise(deploy_trace)
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1173, in take_action
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud forks=parsed_args.ansible_forks
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 463, in config_download
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud forks=forks
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 711, in run_ansible_playbook
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg)
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/config-download/overcloud/ansible-playbook-command.sh
2021-02-10 14:20:00.422 235852 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2021-02-10 14:20:00.426 235852 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/config-download/overcloud/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/config-download/overcloud/ansible-playbook-command.sh
2021-02-10 14:20:00.427 235852 INFO osc_lib.shell [-] END return value: 1
+ status_code=1
+ openstack stack list
+ grep -q overcloud
+ openstack stack list
+ grep -Eq '(CREATE|UPDATE)_COMPLETE'
+ openstack overcloud status --plan overcloud
+ grep -Eq DEPLOY_SUCCESS
+ openstack overcloud failures --plan overcloud
+ true
+ exit 1

Steps to reproduce:
=======
Deploy a tripleo-quickstart environment

1.- Provisioning
./quickstart.sh -n -X -R master --tags all -T none --extra-vars undercloud_disk=70 --nodes config/nodes/3ctlr_2comp_3ceph.yml -p quickstart.yml 127.0.0.2

2.- Installing the undercloud (fixing the tripleo_validations missing vars)
./quickstart.sh -R master --no-clone --tags all -I -T none --extra-vars undercloud_enable_swift=true --nodes config/nodes/3ctlr_2comp_3ceph.yml -p quickstart-extras-undercloud.yml 127.0.0.2

3.- Prepare the overcloud deployment ( fixing the introspection )
./quickstart.sh -R master --no-clone --tags all -I -T none --nodes config/nodes/3ctlr_2comp_3ceph.yml -p quickstart-extras-overcloud-prep.yml 127.0.0.2

4.- Create the overcloud scripts
./quickstart.sh -R master --no-clone --tags overcloud-scripts -I -T none --nodes config/nodes/3ctlr_2comp_3ceph.yml -p quickstart-extras-overcloud.yml 127.0.0.2

5.- Before executing the overcloud-deploy
I had to add some steps regarding ceph.
Got them from https://gist.github.com/jbadiapa/8feb58c1a4833fe46f7b543856cb9c9e (CEPH PART)

UNDERCLOUD
===========
(undercloud) [stack@undercloud ~]$ cat network-environment.yaml
{
    "parameter_defaults": {
        "ControlPlaneDefaultRoute": "192.168.24.1",
        "ControlPlaneSubnetCidr": "24",
        "DnsServers": [
            "192.168.23.1"
        ],
        "EC2MetadataIp": "192.168.24.1",
        "ExternalAllocationPools": [
            {
                "end": "10.0.0.250",
                "start": "10.0.0.4"
            }
        ],
        "ExternalInterfaceDefaultRoute": "10.0.0.1",
        "ExternalNetCidr": "10.0.0.1/24",
        "NeutronExternalNetworkBridge": "",
        "PublicVirtualFixedIPs": [
            {
                "ip_address": "10.0.0.5"
            }
        ]
    }
}

MORE from the controller-1
==========================

Before deploying the overcloud
[root@overcloud-controller-1 ~]# ip r s
default via 192.168.24.1 dev ens3 proto dhcp metric 100
192.168.24.0/24 dev ens3 proto kernel scope link src 192.168.24.6 metric 100

After the error
[heat-admin@overcloud-controller-1 ~]$ ip r s
default via 10.0.0.1 dev vlan10
10.0.0.0/24 dev vlan10 proto kernel scope link src 10.0.0.235
172.16.0.0/24 dev vlan50 proto kernel scope link src 172.16.0.88
172.16.1.0/24 dev vlan30 proto kernel scope link src 172.16.1.19
172.16.2.0/24 dev vlan20 proto kernel scope link src 172.16.2.125
172.16.3.0/24 dev vlan40 proto kernel scope link src 172.16.3.145
192.168.24.0/24 dev br-ex proto kernel scope link src 192.168.24.6