| 2022-04-12 13:37:11 |
chandan kumar |
bug |
|
|
added bug |
| 2022-04-12 13:45:06 |
chandan kumar |
description |
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-containers-multinode-wallaby&skip=0
and similar version of RHEL job running on RHOS-17 RHEL-9 integration pipeline is failing with the same error at tempest
[1] and [2]
```
0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state [355.391581s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 131, in _get_ssh_connection
ssh.connect(self.host, port=self.port, username=self.username,
File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 192.168.24.165
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
return f(*func_args, **func_kwargs)
File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 557, in test_update_router_admin_state
self._check_public_network_connectivity(
File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 212, in _check_public_network_connectivity
self.check_vm_connectivity(
File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 952, in check_vm_connectivity
self.get_remote_client(ip_address, username, private_key,
File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 723, in get_remote_client
linux_client.validate_authentication()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 31, in wrapper
return function(self, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 117, in validate_authentication
self.ssh_client.test_connection_auth()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 240, in test_connection_auth
connection = self._get_ssh_connection()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 150, in _get_ssh_connection
raise exceptions.SSHTimeout(host=self.host,
tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.165 via SSH timed out.
User: cirros, Password: None
```
By comparing with passing job : https://6696badaf160a21814ac-f8803ca7587e43f6b66c9edae98e760b.ssl.cf1.rackcdn.com/837455/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/cd5e873/logs/subnode-1/var/log/extra/errors.txt
and failing job: https://057b70f24ff595a5f3b8-448e63f3241620894d174670888496e6.ssl.cf5.rackcdn.com/837512/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/a2518f7/logs/subnode-1/var/log/extra/errors.txt
```
2022-04-12 12:28:43.660 ERROR /var/log/containers/nova/nova-api.log: 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [f125b2b9-d1f6-4535-b795-a3e41539b09e] AMQP server on centos-9-stream-rax-iad-0029292021.ctlplane.localdomain:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
```
I am not sure what is causing the above error. Moving this test to skiplist till we investigate.
Logs:
[1]. https://057b70f24ff595a5f3b8-448e63f3241620894d174670888496e6.ssl.cf5.rackcdn.com/837512/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/a2518f7/logs/undercloud/var/log/tempest/tempest_run.log
[2]. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f65/837516/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/f65ef99/logs/undercloud/var/log/tempest/tempest_run.log |
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-containers-multinode-wallaby&skip=0
and similar version of RHEL job running on RHOS-17 RHEL-9 integration pipeline(periodic-tripleo-ci-rhel-9-containers-multinode-rhos-17) is failing with the same error at tempest
[1] and [2]
```
0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state [355.391581s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 131, in _get_ssh_connection
ssh.connect(self.host, port=self.port, username=self.username,
File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 192.168.24.165
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
return f(*func_args, **func_kwargs)
File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 557, in test_update_router_admin_state
self._check_public_network_connectivity(
File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 212, in _check_public_network_connectivity
self.check_vm_connectivity(
File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 952, in check_vm_connectivity
self.get_remote_client(ip_address, username, private_key,
File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 723, in get_remote_client
linux_client.validate_authentication()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 31, in wrapper
return function(self, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 117, in validate_authentication
self.ssh_client.test_connection_auth()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 240, in test_connection_auth
connection = self._get_ssh_connection()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 150, in _get_ssh_connection
raise exceptions.SSHTimeout(host=self.host,
tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.165 via SSH timed out.
User: cirros, Password: None
```
By comparing with passing job : https://6696badaf160a21814ac-f8803ca7587e43f6b66c9edae98e760b.ssl.cf1.rackcdn.com/837455/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/cd5e873/logs/subnode-1/var/log/extra/errors.txt
and failing job: https://057b70f24ff595a5f3b8-448e63f3241620894d174670888496e6.ssl.cf5.rackcdn.com/837512/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/a2518f7/logs/subnode-1/var/log/extra/errors.txt
```
2022-04-12 12:28:43.660 ERROR /var/log/containers/nova/nova-api.log: 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [f125b2b9-d1f6-4535-b795-a3e41539b09e] AMQP server on centos-9-stream-rax-iad-0029292021.ctlplane.localdomain:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
```
I am not sure what is causing the above error. Moving this test to skiplist till we investigate.
Logs:
[1]. https://057b70f24ff595a5f3b8-448e63f3241620894d174670888496e6.ssl.cf5.rackcdn.com/837512/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/a2518f7/logs/undercloud/var/log/tempest/tempest_run.log
[2]. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f65/837516/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/f65ef99/logs/undercloud/var/log/tempest/tempest_run.log |
|
| 2022-04-12 14:19:52 |
chandan kumar |
description |
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-containers-multinode-wallaby&skip=0
and similar version of RHEL job running on RHOS-17 RHEL-9 integration pipeline(periodic-tripleo-ci-rhel-9-containers-multinode-rhos-17) is failing with the same error at tempest
[1] and [2]
```
0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state [355.391581s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 131, in _get_ssh_connection
ssh.connect(self.host, port=self.port, username=self.username,
File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 192.168.24.165
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
return f(*func_args, **func_kwargs)
File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 557, in test_update_router_admin_state
self._check_public_network_connectivity(
File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 212, in _check_public_network_connectivity
self.check_vm_connectivity(
File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 952, in check_vm_connectivity
self.get_remote_client(ip_address, username, private_key,
File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 723, in get_remote_client
linux_client.validate_authentication()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 31, in wrapper
return function(self, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 117, in validate_authentication
self.ssh_client.test_connection_auth()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 240, in test_connection_auth
connection = self._get_ssh_connection()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 150, in _get_ssh_connection
raise exceptions.SSHTimeout(host=self.host,
tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.165 via SSH timed out.
User: cirros, Password: None
```
By comparing with passing job : https://6696badaf160a21814ac-f8803ca7587e43f6b66c9edae98e760b.ssl.cf1.rackcdn.com/837455/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/cd5e873/logs/subnode-1/var/log/extra/errors.txt
and failing job: https://057b70f24ff595a5f3b8-448e63f3241620894d174670888496e6.ssl.cf5.rackcdn.com/837512/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/a2518f7/logs/subnode-1/var/log/extra/errors.txt
```
2022-04-12 12:28:43.660 ERROR /var/log/containers/nova/nova-api.log: 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [f125b2b9-d1f6-4535-b795-a3e41539b09e] AMQP server on centos-9-stream-rax-iad-0029292021.ctlplane.localdomain:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
```
I am not sure what is causing the above error. Moving this test to skiplist till we investigate.
Logs:
[1]. https://057b70f24ff595a5f3b8-448e63f3241620894d174670888496e6.ssl.cf5.rackcdn.com/837512/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/a2518f7/logs/undercloud/var/log/tempest/tempest_run.log
[2]. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f65/837516/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/f65ef99/logs/undercloud/var/log/tempest/tempest_run.log |
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-containers-multinode-wallaby&skip=0
and similar version of RHEL job running on RHOS-17 RHEL-9 integration pipeline(periodic-tripleo-ci-rhel-9-containers-multinode-rhos-17) is failing with the same error at tempest
[1] and [2]
```
0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state [355.391581s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 131, in _get_ssh_connection
ssh.connect(self.host, port=self.port, username=self.username,
File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 192.168.24.165
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
return f(*func_args, **func_kwargs)
File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 557, in test_update_router_admin_state
self._check_public_network_connectivity(
File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 212, in _check_public_network_connectivity
self.check_vm_connectivity(
File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 952, in check_vm_connectivity
self.get_remote_client(ip_address, username, private_key,
File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 723, in get_remote_client
linux_client.validate_authentication()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 31, in wrapper
return function(self, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 117, in validate_authentication
self.ssh_client.test_connection_auth()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 240, in test_connection_auth
connection = self._get_ssh_connection()
File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 150, in _get_ssh_connection
raise exceptions.SSHTimeout(host=self.host,
tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.165 via SSH timed out.
User: cirros, Password: None
```
By comparing with passing job : https://6696badaf160a21814ac-f8803ca7587e43f6b66c9edae98e760b.ssl.cf1.rackcdn.com/837455/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/cd5e873/logs/subnode-1/var/log/extra/errors.txt
and failing job: https://057b70f24ff595a5f3b8-448e63f3241620894d174670888496e6.ssl.cf5.rackcdn.com/837512/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/a2518f7/logs/subnode-1/var/log/extra/errors.txt
```
2022-04-12 12:28:43.660 ERROR /var/log/containers/nova/nova-api.log: 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [f125b2b9-d1f6-4535-b795-a3e41539b09e] AMQP server on centos-9-stream-rax-iad-0029292021.ctlplane.localdomain:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
```
I am not sure what is causing the above error. Moving this test to skiplist till we investigate.
Logs:
[1]. https://057b70f24ff595a5f3b8-448e63f3241620894d174670888496e6.ssl.cf5.rackcdn.com/837512/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/a2518f7/logs/undercloud/var/log/tempest/tempest_run.log
[2]. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f65/837516/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/f65ef99/logs/undercloud/var/log/tempest/tempest_run.log
[3]. https://2a0929bb28433d9a9dd5-585b06f1274d5f0e5ed27601745f1d4b.ssl.cf2.rackcdn.com/837055/1/gate/tripleo-ci-centos-9-containers-multinode/e946bbd/logs/undercloud/var/log/tempest/stestr_results.html |
|
| 2022-04-18 07:47:29 |
yatin |
bug |
|
|
added subscriber yatin |
| 2022-04-19 14:56:52 |
yatin |
bug watch added |
|
https://bugzilla.redhat.com/show_bug.cgi?id=2076604 |
|
| 2022-07-28 06:41:18 |
yatin |
tripleo: assignee |
|
yatin (yatinkarel) |
|
| 2022-10-12 05:28:00 |
Rabi Mishra |
tripleo: status |
Triaged |
Fix Released |
|
