Description
===========
Job periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master is quite unstable. This job is mostly failing on the tempest issues, but for each run we can see different tempest test failures.
Actual Error
```
[1] test_get_object_at_expiry_time[id-e592f18d-679c-48fe-9e36-4be5f47102c5]
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/test_object_expiry.py", line 98, in test_get_object_at_expiry_time
self._test_object_expiry(metadata)
File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/test_object_expiry.py", line 54, in _test_object_expiry
self.assertIn('x-delete-at', resp)
File "/usr/lib/python3.9/site-packages/testtools/testcase.py", line 399, in assertIn
self.assertThat(haystack, Contains(needle), message)
File "/usr/lib/python3.9/site-packages/testtools/testcase.py", line 480, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 'x-delete-at' not in {'content-type': 'application/json', 'etag': '40deff5a10faee36168dd65eb0bf65d4', 'last-modified': 'Thu, 15 Dec 2022 22:20:46 GMT', 'x-timestamp': '1671142845.38153', 'accept-ranges': 'bytes', 'content-length': '1024', 'x-trans-id': 'txfd52fbb8adea4c9495915-00639b9dc1', 'x-openstack-request-id': 'txfd52fbb8adea4c9495915-00639b9dc1', 'date': 'Thu, 15 Dec 2022 22:20:49 GMT', 'connection': 'close', 'status': '200', 'content-location': 'https://10.0.0.5:13808/v1/AUTH_ec6bb6464020444e80ff9c2901ef1c0b/tempest-TestContainer-431573027/tempest-TestObject-1954279089'}
[2] test_list_large_object_metadata[id-e69ad766-e1aa-44a2-bdd2-bf62c09c1456]
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/test_object_slo.py", line 36, in tearDown
self.delete_containers()
File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/base.py", line 144, in delete_containers
delete_containers(cls.containers, container_client, object_client)
File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/base.py", line 51, in delete_containers
container_client.delete_container(cont)
File "/usr/lib/python3.9/site-packages/tempest/lib/services/object_storage/container_client.py", line 57, in delete_container
resp, body = self.delete(url)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 330, in delete
return self.request('DELETE', url, extra_headers, headers, body)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 720, in request
self._error_checker(resp, resp_body)
File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 841, in _error_checker
raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: b'<html><h1>Conflict</h1><p>There was a conflict when trying to complete your request.</p></html>'
```
[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master&skip=0
[2] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/15fb786/logs/undercloud/var/log/tempest/stestr_results.html.gz
[3] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/d6ffb33/logs/undercloud/var/log/tempest/stestr_results.html.gz
[4] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/bcaf911/logs/undercloud/var/log/tempest/stestr_results.html.gz
[5] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/77c68d3/logs/undercloud/var/log/tempest/stestr_results.html.gz
I also see that job history is not total green. Can we mark this job as non-voting or remove it ?
We can't remove this from voting history until we remove OVB in general.
Please try this on IBM to debug. I would advocate for removing some tempest tests before killing the entire job