tripleo

cs8 train: os_tempest ping router ip address task failing with Destination Host Unreachable

Bug #2033644 reported by Bhagyashri Shewale
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Critical
Unassigned
tripleo antelope-1

Bug Description

periodic-tripleo-ci-centos-8-scenario007-standalone-train (non-voting) is failing with blow error:

Error logs:

2023-08-30 10:21:25.008998 | primary | TASK [os_tempest : Ping router ip address] *************************************
2023-08-30 10:21:25.009006 | primary | Wednesday 30 August 2023 10:21:25 +0000 (0:00:00.090) 0:23:31.740 ******
2023-08-30 10:21:29.631453 | primary | FAILED - RETRYING: Ping router ip address (5 retries left).
2023-08-30 10:21:43.997530 | primary | FAILED - RETRYING: Ping router ip address (4 retries left).
2023-08-30 10:21:58.528763 | primary | FAILED - RETRYING: Ping router ip address (3 retries left).
2023-08-30 10:22:13.247635 | primary | FAILED - RETRYING: Ping router ip address (2 retries left).
2023-08-30 10:22:27.770777 | primary | FAILED - RETRYING: Ping router ip address (1 retries left).
2023-08-30 10:22:42.141586 | primary | fatal: [undercloud]: FAILED! => {"attempts": 5, "changed": true, "cmd": "set -e\nping -c2 \"192.168.24.173\"\n", "delta": "0:00:03.060747", "end": "2023-08-30 10:22:41.951456", "msg": "non-zero return code", "rc": 1, "start": "2023-08-30 10:22:38.890709", "stderr": "", "stderr_lines": [], "stdout": "PING 192.168.24.173 (192.168.24.173) 56(84) bytes of data.\nFrom 192.168.24.1 icmp_seq=1 Destination Host Unreachable\nFrom 192.168.24.1 icmp_seq=2 Destination Host Unreachable\n\n--- 192.168.24.173 ping statistics ---\n2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1006ms\npipe 2", "stdout_lines": ["PING 192.168.24.173 (192.168.24.173) 56(84) bytes of data.", "From 192.168.24.1 icmp_seq=1 Destination Host Unreachable", "From 192.168.24.1 icmp_seq=2 Destination Host Unreachable", "", "--- 192.168.24.173 ping statistics ---", "2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1006ms", "pipe 2"]}
2023-08-30 10:22:42.141707 | primary |
2023-08-30 10:22:42.141715 | primary | NO MORE HOSTS LEFT *************************************************************
2023-08-30 10:22:42.145585 | primary |
2023-08-30 10:22:42.145607 | primary | PLAY RECAP *********************************************************************
2023-08-30 10:22:42.145615 | primary | undercloud : ok=130 changed=48 unreachable=0 failed=1 skipped=168 rescued=0 ignored=0

[1]: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario007-standalone-train&skip=0
[2]: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-standalone-train/af697cc/job-output.txt

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/893512

Revision history for this message
Douglas Viroel (dviroel) wrote :

The nftable major update might be causing this:

-nftables-0.9.3-26.el8.x86_64
+nftables-1.0.4-2.el8.x86_64

Error from iptables-manager:

2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info [-] Failed to process floating IPs.: neutron_lib.exceptions.l3.IpTablesApplyException: Failure applying iptables rules
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/iptables_manager.py", line 433, in defer_apply
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info self.defer_apply_off()
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/iptables_manager.py", line 447, in defer_apply_off
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info self._apply()
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/iptables_manager.py", line 464, in _apply
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info first = self._apply_synchronized()
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/iptables_manager.py", line 624, in _apply_synchronized
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info raise err
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/iptables_manager.py", line 516, in _do_run_restore
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info run_as_root=True, **kwargs)
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 148, in execute
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info returncode=returncode)
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Stdin: # Generated by iptables_manager

https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-standalone-train/d98ae11/logs/undercloud/var/log/extra/big-errors.txt

Revision history for this message
Douglas Viroel (dviroel) wrote :

These errors are actually related to iptables 1.8.5-8, according with:

https://bugzilla.redhat.com/show_bug.cgi?id=2236501

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/893512
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/e08e44b88f387fd23471dca5c05283029374ae79
Submitter: "Zuul (22348)"
Branch: master

commit e08e44b88f387fd23471dca5c05283029374ae79
Author: Douglas Viroel <email address hidden>
Date: Fri Sep 1 10:13:40 2023 -0300

    Block iptables updates on Centos-8 train

    This patch excludes newer versions of iptables that might me breaking
    some CS-8 stable/train jobs. It also pins the rpm to the latest
    knwon good version.

    Related-Bug: #2033644

    Change-Id: I7d22932ab32bea52b302a1d0274e11b4d86525c3

Revision history for this message
Alan Pevec (apevec) wrote :

IMO this workaround is fine for the remaining (short) life of TripleO stable/train.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.