Running tempest tests on a mistral deployment from master (pike cycle) with mistral api in WSGI we get intermitent failures in different jobs with similar trace:
{1} mistral_tempest_tests.tests.api.v2.test_actions.ActionTestsV2.test_get_list_actions_equal_to_filter [500.009070s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mistral_tempest_tests/tests/api/v2/test_actions.py", line 134, in test_get_list_actions_equal_to_filter
resp, body = self.client.create_action('action_v2.yaml')
File "/usr/lib/python2.7/site-packages/mistral_tempest_tests/services/v2/mistral_client.py", line 123, in create_action
resp, body = self.post_request('actions', yaml_file)
File "/usr/lib/python2.7/site-packages/mistral_tempest_tests/services/v2/mistral_client.py", line 30, in post_request
return self.post(url, base.get_resource(file_name), headers=headers)
File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 276, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 650, in request
body=body, chunked=chunked)
File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 546, in _request
chunked=chunked
File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 579, in raw_request
body=body, chunked=chunked)
File "/usr/lib/python2.7/site-packages/tempest/lib/common/http.py", line 60, in request
*args, **new_kwargs)
File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 73, in request
**urlopen_kw)
File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 151, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 248, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 595, in urlopen
chunked=chunked)
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request
httplib_response = conn.getresponse(buffering=True)
File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib64/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib64/python2.7/ssl.py", line 759, in recv
return self.read(buflen)
File "/usr/lib64/python2.7/ssl.py", line 653, in read
v = self._sslobj.read(len or 1024)
File "/usr/lib/python2.7/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException
In all failures we find following error message in mistral apache wsgi log:
[Wed Mar 08 14:07:41.424793 2017] [:error] [pid 29897] Traceback (most recent call last):
[Wed Mar 08 14:07:41.424844 2017] [:error] [pid 29897] File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
[Wed Mar 08 14:07:41.425250 2017] [:error] [pid 29897] timer()
[Wed Mar 08 14:07:41.425277 2017] [:error] [pid 29897] File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
[Wed Mar 08 14:07:41.425381 2017] [:error] [pid 29897] cb(*args, **kw)
[Wed Mar 08 14:07:41.425401 2017] [:error] [pid 29897] File "/usr/lib/python2.7/site-packages/eventlet/semaphore.py", line 145, in _do_acquire
[Wed Mar 08 14:07:41.425584 2017] [:error] [pid 29897] waiter.switch()
[Wed Mar 08 14:07:41.425612 2017] [:error] [pid 29897] error: cannot switch to a different thread
Which i think it's related to the issue.
We are hitting this in RDO-CI jobs. You can find full logs for a failed job in:
https://ci.centos.org/artifacts/rdo/weirdo-generic-puppet-openstack-scenario003/1168/weirdo-project/
We haven't hit this error after 17-march so i think it has been fixed by some of the commits merged between 14-march (last hit) and 17-march (firs success).