tripleo

periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master failing intermitently with paramiko.ssh_exception.AuthenticationException: Authentication failed

Bug #1906769 reported by Pooja Jadhav
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Unassigned
tripleo xena-3

Bug Description

periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master failing intermitently with the below error trace back :

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 112, in _get_ssh_connection
        sock=proxy_chan)
      File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 446, in connect
        passphrase,
      File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 764, in _auth
        raise saved_exception
      File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 664, in _auth
        self._transport.auth_publickey(username, pkey)
      File "/usr/lib/python3.6/site-packages/paramiko/transport.py", line 1580, in auth_publickey
        return self.auth_handler.wait_for_response(my_event)
      File "/usr/lib/python3.6/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response
        raise e
    paramiko.ssh_exception.AuthenticationException: Authentication failed.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_qos.py", line 340, in test_attach_previously_used_port_to_new_instance
        self._create_server_by_port(port=vm_port)
      File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_qos.py", line 181, in _create_server_by_port
        keypair['private_key'])
      File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/base.py", line 274, in check_connectivity
        ssh_client.test_connection_auth()
      File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/ssh.py", line 137, in connect
        return super(Client, self)._get_ssh_connection(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 128, in _get_ssh_connection
        password=self.password)
    tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.122 via SSH timed out.
    User: cirros, Password: None

Reference Links :

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/fe8f969/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/679270a/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/fe8f969/job-output.txt

Revision history for this message
wes hayutin (weshayutin) wrote :

 ERROR ovsdbapp.backend.ovs_idl.transaction [req-555039b5-dadb-4bea-8d1a-b0fd3eaa52a8 16fa5ca836144979a4de2594c5300be2 ef7ad5f37f24492ea97784b08b808e39 - default default] Traceback (most recent call last):
2020-12-03 10:52:01.376 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR ovsdbapp.backend.ovs_idl.transaction [req-3f929960-b6ec-4754-b935-6d62a985008f e3b9957d9ca948dfafa162f2142fbb88 62c62e2216e647a1a5a50025e799a47c - default default] Traceback (most recent call last):
2020-12-03 10:53:13.835 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR ovsdbapp.backend.ovs_idl.transaction [req-cfd5efe2-89c1-4f82-9096-9dbf26dbff87 - - - - -] Traceback (most recent call last):
2020-12-03 10:53:25.405 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR ovsdbapp.backend.ovs_idl.transaction [req-783daeb5-d1d2-409f-872a-c9c0f748498d 0ee34f5cb94c48d58d786cc3ec91d96d 1cd910b24e9f4de1978a9a3c24eb12f3 - default default] Traceback (most recent call last):
2020-12-03 10:54:33.850 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR ovsdbapp.backend.ovs_idl.transaction [req-7176d257-c806-462f-bb22-e7a576407319 0ee34f5cb94c48d58d786cc3ec91d96d

3}}, resource_type=ports, if_exists=True) do_commit /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:88
2020-12-03 11:41:00.240 28 ERROR ovsdbapp.backend.ovs_idl.transaction [req-c66218df-9497-4110-8941-658aa2233755 a3f40a76f1d946d5bc7d7ae25c33053a 147435c3276343f6ad8256efe6d6e043 - default default] Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 129, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 90, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/commands.py", line 763, in run_idl
    resource_id=self.name, resource_type=self.resource_type)
neutron.common.ovn.exceptions.RevisionConflict: OVN revision number for bb01e64c-10b6-45ff-a2fd-9d993b972b37 (type: ports) is equal or higher than the given resource. Skipping update

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/fe8f969/logs/undercloud/var/log/extra/errors.txt.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/fe8f969/logs/undercloud/var/log/containers/neutron/

Not sure I see anything clear here.

Revision history for this message
wes hayutin (weshayutin) wrote :

perhaps related?
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/fe8f969/logs/undercloud/var/log/containers/openvswitch/ovsdb-server-nb.log.txt.gz

2020-12-03T10:31:29.760Z|00004|jsonrpc|WARN|unix#6: receive error: Connection reset by peer
2020-12-03T10:31:29.760Z|00005|reconnect|WARN|unix#6: connection dropped (Connection reset by peer)
2020-12-03T10:31:37.662Z|00006|memory|INFO|7952 kB peak resident set size after 10.0 seconds
2020-12-03T10:31:37.663Z|00007|memory|INFO|cells:43 monitors:2 sessions:1
2020-12-03T10:47:02.878Z|00008|reconnect|ERR|tcp:192.168.24.3:57128: no response to inactivity probe after 180 seconds, disconnecting
2020-12-03T10:47:02.884Z|00009|reconnect|ERR|tcp:192.168.24.3:57130: no response to inactivity probe after 180 seconds, disconnecting
2020-12-03T10:47:02.935Z|00010|reconnect|ERR|tcp:192.168.24.3:57140: no response to inactivity probe after 180 seconds, disconnecting
2020-12-03T10:47:03.001Z|00011|reconnect|ERR|tcp:192.168.24.3:57144: no response to inactivity probe after 180 seconds, disconnecting
2020-12-03T10:47:03.007Z|00012|reconnect|ERR|tcp:192.168.24.3:57148: no response to inactivity probe after 180 seconds, disconnecting
2020-12-03T10:47:03.023Z|00013|reconnect|ERR|tcp:192.168.24.3:57150: no response to inactivity probe after 180 seconds, disconnecting

Revision history for this message
Marios Andreou (marios-b) wrote :

latest run from today fails with the same authentication error but for a different test than in the bug description

        * https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/a833907/logs/undercloud/var/log/tempest/stestr_results.html.gz
        * paramiko.ssh_exception.AuthenticationException: Authentication failed..
        * neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest * test_two_sec_groups[id-3d73ec1a-2ec6-45a9-b0f8-04a283d9d964]

Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Revision history for this message
wes hayutin (weshayutin) wrote :

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/69a0cc5/logs/undercloud/var/log/tempest/stestr_results.html.gz

I think neutron_tempest_plugin.scenario.test_qos.QoSTest.test_attach_previously_used_port_to_new_instance is hitting this.

Changed in tripleo:
importance: Undecided → High
Revision history for this message
wes hayutin (weshayutin) wrote :

two more skipped tests on this

https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/766987

Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-3 → wallaby-rc1
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Marios Andreou (marios-b)
Changed in tripleo:
milestone: xena-1 → xena-2
Marios Andreou (marios-b)
Changed in tripleo:
milestone: xena-2 → xena-3
Revision history for this message
Ronelle Landy (rlandy) wrote :

https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/768059 revert merged - closing out

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.