Tripleo-CI Master and Wallaby jobs failing with keystone lost mysql connection
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Unassigned |
Bug Description
There are some jobs failing on tempest test, with two different errors however related, the first one, all tempest tests that relies on keystone endpoint are failing with the following error:
ft1.1: setUpClass (cinder_
File "/usr/lib/
raise value.with_
File "/usr/lib/
cls.
File "/usr/lib/
manager = cls.get_
File "/usr/lib/
cred_provider = cls._get_
File "/usr/lib/
cls.
File "/usr/lib/
**get_
File "/usr/lib/
admin_creds = admin_creds or get_configured_
File "/usr/lib/
credentials = get_credentials
File "/usr/lib/
return auth.get_
File "/usr/lib/
creds = auth_provider.
File "/usr/lib/
auth_data = self.get_auth()
File "/usr/lib/
self.set_auth()
File "/usr/lib/
self.cache = self._get_auth()
File "/usr/lib/
token, auth_data = auth_func(
File "/usr/lib/
body = self.auth(**kwargs)
File "/usr/lib/
resp, body = self.post(
File "/usr/lib/
return self.request(
File "/usr/lib/
raise exceptions.
tempest.
Details: Unexpected status code 500
summary: |
- Some tripleo-ci jobs are failing on tempest tests with ipv6 error - related + Tripleo-CI Master and Wallaby jobs failing with keystone lost mysql + connection |
Changed in tripleo: | |
milestone: | xena-3 → yoga-3 |
Checking the keystone logs I found this:
https:/ /logserver. rdoproject. org/46/ 38646/16/ check/periodic- tripleo- ci-centos- 9-ovb-1ctlr_ 2comp-featurese t020-master/ b9ff5d0/ logs/overcloud- controller- 0/var/log/ containers/ keystone/ keystone. log.txt. gz
2022-03-17 23:24:08.820 110 ERROR keystone. server. flask.request_ processing. middleware. auth_context Traceback (most recent call last): server. flask.request_ processing. middleware. auth_context File "/usr/lib/ python3. 9/site- packages/ keystone/ server/ flask/request_ processing/ middleware/ auth_context. py", line 103, in _inner server. flask.request_ processing. middleware. auth_context return method(self, request) server. flask.request_ processing. middleware. auth_context File "/usr/lib/ python3. 9/site- packages/ keystone/ server/ flask/request_ processing/ middleware/ auth_context. py", line 353, in process_request server. flask.request_ processing. middleware. auth_context resp = super(AuthConte xtMiddleware, self).process_ request( request) server. flask.request_ processing. middleware. auth_context File "/usr/lib/ python3. 9/site- packages/ keystonemiddlew are/auth_ token/_ _init__ .py", line 409, in process_request server. flask.request_ processing. middleware. auth_context data, user_auth_ref = self._do_ fetch_token( server. flask.request_ processing. middleware. auth_context File "/usr/lib/ python3. 9/site- packages/ keystonemiddlew are/auth_ token/_ _init__ .py", line 445, in _do_fetch_token server. flask.request_ processing. middleware. auth_context data = self.fetch_ token(token, **kwargs) server. flask.request_ processing. middleware. auth_context File "/usr/lib/ python3. 9/site- packages/ keystone/ server/ flask/request_ processing/ middleware/ auth_context. py", line 247, in fetch_token server. flask.request_ processing. middleware. auth_context self.token = self.token_ provider_ api.validate_ token( server. flask.request_ processing. middleware. auth_context File "/usr/lib/ python3. 9/site- packages/ keystone/ common/ manager. py", line 115, in wrapped server. flask.request_ processing. middleware. auth_context __ret_val = __f(*args, **kwargs) server. flask.request_ processing. middleware. auth_context File "/usr/lib/ python3. 9/site- packages/ keystone/ token/provider. py", line 145, in validate_token server. flask.request_ processing. middleware. auth_context token = self._validate_ token(token_ id) server. flask.request_ processing. middleware. auth_context File "<decorator- gen-26> ", line 2, in _validate_token server. flask.request_ processing. middleware. auth_context File "/usr/lib/ python3. ..
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.
2022-03-17 23:24:08.820 110 ERROR keystone.