Recently I found that this intermittent failure still occurs. This time I reproduced the network delay by means of script (run as root):
delay.sh
<BEGIN OF FILE>
delay=$1
ip=$2
tc qdisc del dev eth0 root
if [ -n $delay ]; then
tc qdisc add dev eth0 root handle 1: prio
tc filter add dev eth0 parent 1:0 protocol ip pref 55 handle ::55 u32 match ip dst $ip flowid 2:1
tc qdisc add dev eth0 parent 1:1 handle 2: netem delay ${delay}ms
fi
<END OF FILE>
I realized that first the window is opened, then it hangs for the page to become accessible, and if I set timeout large enough (say, 30 seconds), I get the following traceback:
Error
Traceback (most recent call last):
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run
self.runTest(result)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest
test(result)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/unittest2/case.py", line 673, in __call__
return self.run(*args, **kwds)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 606, in run
return run_test.run(result)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 80, in run
return self._run_one(actual_result)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 94, in _run_one
return self._run_prepared_result(ExtendedToOriginalDecorator(result))
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 108, in _run_prepared_result
self._run_core()
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 149, in _run_core
self.case._run_teardown, self.result):
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 193, in _run_user
return self._got_user_exception(sys.exc_info())
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 213, in _got_user_exception
self.case.onException(exc_info, tb_label=tb_label)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 558, in onException
handler(exc_info)
File "/home/timur/develop/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 132, in _save_screenshot
self.driver.get_screenshot_as_file(filename)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 758, in get_screenshot_as_file
png = self.get_screenshot_as_png()
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 777, in get_screenshot_as_png
return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 787, in get_screenshot_as_base64
return self.execute(Command.SCREENSHOT)['value']
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 199, in execute
response = self.command_executor.execute(driver_command, params)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
return self._request(command_info[0], url, body=data)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 425, in _request
self._conn.request(method, parsed_url.path, body, headers)
File "/usr/lib/python2.7/httplib.py", line 979, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 797, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 778, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 111] Connection refused
It's different from the traceback that follows intermittent failure on gate:
2015-11-24 19:48:29.619 | 2015-11-24 19:48:29.606 | Traceback (most recent call last):
2015-11-24 19:48:29.620 | 2015-11-24 19:48:29.607 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/tests/test_dashboard_help_redirection.py", line 28, in test_dashboard_help_redirection
2015-11-24 19:48:29.621 | 2015-11-24 19:48:29.608 | "help link did not redirect to the right URL")
2015-11-24 19:48:29.623 | 2015-11-24 19:48:29.609 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
2015-11-24 19:48:29.624 | 2015-11-24 19:48:29.610 | self.assertThat(observed, matcher, message)
2015-11-24 19:48:29.625 | 2015-11-24 19:48:29.611 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
2015-11-24 19:48:29.626 | 2015-11-24 19:48:29.613 | raise mismatch_error
2015-11-24 19:48:29.627 | 2015-11-24 19:48:29.614 | testtools.matchers._impl.MismatchError: 'http://docs.openstack.org/' != u'http://localhost/dashboard/project/': help link did not redirect to the right URL
The second one looks more like the command to switch the window was issued before window was actually opened. If that's truly the root cause, fix should be trivial.
A watchful reviewer may notice that sometimes test_dashboard_ help_redirectio n fails. Initially I thought that it was caught by an increased network timeout from Jenkins worker node to docs.openstack.org site. So I made a patch https:/ /github. com/openstack/ horizon/ commit/ 6474215840d4cdd e3751c335f46f32 cb317238c0
Recently I found that this intermittent failure still occurs. This time I reproduced the network delay by means of script (run as root):
delay.sh
<BEGIN OF FILE>
delay=$1
ip=$2
tc qdisc del dev eth0 root
if [ -n $delay ]; then
tc qdisc add dev eth0 root handle 1: prio
tc filter add dev eth0 parent 1:0 protocol ip pref 55 handle ::55 u32 match ip dst $ip flowid 2:1
tc qdisc add dev eth0 parent 1:1 handle 2: netem delay ${delay}ms
fi
<END OF FILE>
I realized that first the window is opened, then it hangs for the page to become accessible, and if I set timeout large enough (say, 30 seconds), I get the following traceback:
Error timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ nose/case. py", line 133, in run runTest( result) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ nose/case. py", line 151, in runTest timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ unittest2/ case.py" , line 673, in __call__ timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ testtools/ testcase. py", line 606, in run run(result) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ testtools/ runtest. py", line 80, in run one(actual_ result) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ testtools/ runtest. py", line 94, in _run_one prepared_ result( ExtendedToOrigi nalDecorator( result) ) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ testtools/ runtest. py", line 108, in _run_prepared_ result _run_core( ) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ testtools/ runtest. py", line 149, in _run_core case._run_ teardown, self.result): timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ testtools/ runtest. py", line 193, in _run_user user_exception( sys.exc_ info()) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ testtools/ runtest. py", line 213, in _got_user_exception case.onExceptio n(exc_info, tb_label=tb_label) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ testtools/ testcase. py", line 558, in onException exc_info) timur/develop/ horizon/ openstack_ dashboard/ test/integratio n_tests/ helpers. py", line 132, in _save_screenshot driver. get_screenshot_ as_file( filename) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ selenium/ webdriver/ remote/ webdriver. py", line 758, in get_screenshot_ as_file screenshot_ as_png( ) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ selenium/ webdriver/ remote/ webdriver. py", line 777, in get_screenshot_ as_png b64decode( self.get_ screenshot_ as_base64( ).encode( 'ascii' )) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ selenium/ webdriver/ remote/ webdriver. py", line 787, in get_screenshot_ as_base64 Command. SCREENSHOT) ['value' ] timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ selenium/ webdriver/ remote/ webdriver. py", line 199, in execute executor. execute( driver_ command, params) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ selenium/ webdriver/ remote/ remote_ connection. py", line 395, in execute command_ info[0] , url, body=data) timur/develop/ horizon/ .venv/local/ lib/python2. 7/site- packages/ selenium/ webdriver/ remote/ remote_ connection. py", line 425, in _request _conn.request( method, parsed_url.path, body, headers) python2. 7/httplib. py", line 979, in request _send_request( method, url, body, headers) python2. 7/httplib. py", line 1013, in _send_request endheaders( body) python2. 7/httplib. py", line 975, in endheaders _send_output( message_ body) python2. 7/httplib. py", line 835, in _send_output python2. 7/httplib. py", line 797, in send python2. 7/httplib. py", line 778, in connect address) python2. 7/socket. py", line 571, in create_connection
Traceback (most recent call last):
File "/home/
self.
File "/home/
test(result)
File "/home/
return self.run(*args, **kwds)
File "/home/
return run_test.
File "/home/
return self._run_
File "/home/
return self._run_
File "/home/
self.
File "/home/
self.
File "/home/
return self._got_
File "/home/
self.
File "/home/
handler(
File "/home/
self.
File "/home/
png = self.get_
File "/home/
return base64.
File "/home/
return self.execute(
File "/home/
response = self.command_
File "/home/
return self._request(
File "/home/
self.
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
self.send(msg)
File "/usr/lib/
self.connect()
File "/usr/lib/
self.timeout, self.source_
File "/usr/lib/
raise err
error: [Errno 111] Connection refused
It's different from the traceback that follows intermittent failure on gate:
2015-11-24 19:48:29.619 | 2015-11-24 19:48:29.606 | Traceback (most recent call last): new/horizon/ openstack_ dashboard/ test/integratio n_tests/ tests/test_ dashboard_ help_redirectio n.py", line 28, in test_dashboard_ help_redirectio n new/horizon/ .tox/py27integr ation/local/ lib/python2. 7/site- packages/ testtools/ testcase. py", line 350, in assertEqual (observed, matcher, message) new/horizon/ .tox/py27integr ation/local/ lib/python2. 7/site- packages/ testtools/ testcase. py", line 435, in assertThat matchers. _impl.MismatchE rror: 'http:// docs.openstack. org/' != u'http:// localhost/ dashboard/ project/ ': help link did not redirect to the right URL
2015-11-24 19:48:29.620 | 2015-11-24 19:48:29.607 | File "/opt/stack/
2015-11-24 19:48:29.621 | 2015-11-24 19:48:29.608 | "help link did not redirect to the right URL")
2015-11-24 19:48:29.623 | 2015-11-24 19:48:29.609 | File "/opt/stack/
2015-11-24 19:48:29.624 | 2015-11-24 19:48:29.610 | self.assertThat
2015-11-24 19:48:29.625 | 2015-11-24 19:48:29.611 | File "/opt/stack/
2015-11-24 19:48:29.626 | 2015-11-24 19:48:29.613 | raise mismatch_error
2015-11-24 19:48:29.627 | 2015-11-24 19:48:29.614 | testtools.
The second one looks more like the command to switch the window was issued before window was actually opened. If that's truly the root cause, fix should be trivial.