test_traffic_ops octavia tempest tests are failing with Connection to the 192.168.24.149 via SSH timed out.

Bug #1883687 reported by chandan kumar
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Unassigned

Bug Description

For testing octavia, we have three jobs to run octavia tempest tests:
* sc10
* sc10-kvm
* sc10-ovn

While running octavia scenario tests, this tests is continuously failing:
{0} setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest) [0.000000s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 112, in _get_ssh_connection
        sock=proxy_chan)
      File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 349, in connect
        retry_on_signal(lambda: sock.connect(addr))
      File "/usr/lib/python3.6/site-packages/paramiko/util.py", line 283, in retry_on_signal
        return function()
      File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 349, in <lambda>
        retry_on_signal(lambda: sock.connect(addr))
    socket.timeout: timed out

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/test.py", line 188, in setUpClass
        six.reraise(etype, value, trace)
      File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
        raise value
      File "/usr/lib/python3.6/site-packages/tempest/test.py", line 181, in setUpClass
        cls.resource_setup()
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 38, in resource_setup
        super(TrafficOperationsScenarioTest, cls).resource_setup()
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 693, in resource_setup
        cls.webserver1_response)
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 861, in _install_start_webserver
        linux_client.validate_authentication()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 60, in wrapper
        six.reraise(*original_exception)
      File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
        raise value
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 33, in wrapper
        return function(self, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 116, in validate_authentication
        self.ssh_client.test_connection_auth()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 216, in test_connection_auth
        connection = self._get_ssh_connection()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 128, in _get_ssh_connection
        password=self.password)
    tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.149 via SSH timed out.
    User: cirros, Password: None

https://logserver.rdoproject.org/12/27912/5/check/periodic-tripleo-ci-centos-8-scenario010-standalone-master/6029d95/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

It is one of the important test for testing octavia FIP tests. It is failing on train, ussuri and master.
Here is the tempest.conf
load_balancer]
member_role = "Member"
test_with_ipv6 = False
enable_security_groups = True
admin_role = admin
RBAC_test_type = owner_or_admin

In octavia workers logs, we found nothing. There might be some issue on network side.
It is a tracker bug, till we fix it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/735854

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.opendev.org/735854
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=c5f857606cd0311511838c5b4cb8ffb270ec0aac
Submitter: Zuul
Branch: master

commit c5f857606cd0311511838c5b4cb8ffb270ec0aac
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Tue Jun 16 15:51:17 2020 +0530

    Move TrafficOperationsScenarioTest to skip list

    With octavia tempest tests, TrafficOperationsScenarioTest is failing
    continously with ssh timeout. We are moving it to skip list for
    train, ussuri and master release to enable octavia scenario
    tests.

    Related-Bug: #1883687

    Change-Id: Ibe0aea3d2a45c66f62a32a940c4f52562085ca14
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

Changed in tripleo:
milestone: victoria-1 → victoria-3
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :
Changed in tripleo:
milestone: wallaby-3 → wallaby-rc1
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Revision history for this message
Pooja Jadhav (poojajadhav22) wrote :

It is also failing for wallaby too, "periodic-tripleo-ci-centos-8-scenario010-standalone-wallaby" is also consistently failing because of the above tempest test :

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-wallaby/fe93297/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

tags: added: promotion-blocker
wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
Slawek Kaplonski (slaweq) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.