centos 9 master/wallaby OVB check jobs are failing the node provision step (in Overcloud Node Grow Volumes) with a 'no route to host' error:
PLAY [Overcloud Node Grow Volumes] *********************************************
2022-02-10 10:01:37.969659 | fa163e53-3c0b-2b3b-2b30-00000000000c | TASK | Wait for provisioned nodes to boot
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 113] No route to host
2022-02-10 10:11:14.654638 | fa163e53-3c0b-2b3b-2b30-00000000000c | FATAL | Wait for provisioned nodes to boot | overcloud-controller-0 | error={"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 107, in <module>\n File \"<stdin>\", line 99, in _ansiballz_main\n File \"<stdin>\", line 47, in invoke_module\n File \"/usr/lib64/python3.9/runpy.py\", line 210, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_wait_for_payload_k1o8x7ny/ansible_wait_for_payload.zip/ansible/modules/wait_for.py\", line 682, in <module>\n File \"/tmp/ansible_wait_for_payload_k1o8x7ny/ansible_wait_for_payload.zip/ansible/modules/wait_for.py\", line 612, in main\nOSError: [Errno 113] No route to host\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
However on master:
PLAY [Overcloud Node Grow Volumes] *********************************************
2022-02-10 10:43:22.455318 | fa163e1a-46ca-7598-5c3f-00000000000c | TASK | Wait for provisioned nodes to boot
2022-02-10 10:53:24.590799 | fa163e1a-46ca-7598-5c3f-00000000000c | FATAL | Wait for provisioned nodes to boot | overcloud-controller-0 | error={"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "elapsed": 601, "msg": "Timeout waiting for provisioned nodes to become available"}
Example logs:
https://logserver.rdoproject.org/71/828671/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/eda3778/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz
https://logserver.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/51a3274/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz
Job history:
https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby
The failures looks to have started on 02/09
https:/ /logserver. rdoproject. org/71/ 828671/ 1/openstack- check/tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t001-wallaby/ eda3778/ logs/baremetal_ 1_25010_ 0-console. log /logserver. rdoproject. org/71/ 828671/ 1/openstack- check/tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t001-wallaby/ eda3778/ logs/baremetal_ 1_25010_ 1-console. log /logserver. rdoproject. org/71/ 828671/ 1/openstack- check/tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t001-wallaby/ eda3778/ logs/baremetal_ 1_25010_ 2-console. log /logserver. rdoproject. org/71/ 828671/ 1/openstack- check/tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t001-wallaby/ eda3778/ logs/baremetal_ 1_25010_ 3-console. log
https:/
https:/
https:/
Baremetal nodes does not boot grup issue.
[H[J[1;1Herror: ../../grub- core/disk/ diskfilter. c:916:diskfilte r writes are not H[J[1;1H[ 20;7H[mUse the ^ and v keys to change the selection. 52.el9. x86_64) 9 [m[4;79H[m[7m[5;1H CentOS Stream (5.14.0- 55.el9. x86_64) 9 [m[5;79H[m[m[6;1H [m[6;79H[m[m[7;1H [m[7;79H[m[m[8;1H [m[8;79H[m[m[9;1H [m[9;79H[m[m[10;1H [m[10;79H[m[m[11;1H [m[11;79H[m[m[12;1H [m[12;79H[m[m[13;1H [m[13;79H[m[m[14;1H [m[14;79H[m[m[15;1H [m[15;79H[m[m[16;1H [m[16;79H[m[m[17;1H [m[17;79H[m[m[18;1H [m[18;79H[m[18;80H [5;79H[22;1H The selected entry will be started automatically in 5s. [5;79H[22;1H The selected entry will be started automatically in 4s. [5;79H[22;1H The selected entry will be started automatically in 3s. [5;79H[22;1H The selected entry will be started automatically in 2s. [5;79H[22;1H The selected entry will be started automatically in 1s. [5;79H[22;1H The selected entry will be started automatically in 0s. [5;79H[ ?25h[H[ J[1;1H[ H[J[1;1H
supported.
[?25l[m[
Press 'e' to edit the selected item, or 'c' for a command prompt. [4;80H [m[4;1H CentOS Stream (5.14.0-