A watchful reviewer may notice that sometimes test_dashboard_help_redirection fails. Initially I thought that it was caught by an increased network timeout from Jenkins worker node to docs.openstack.org site. So I made a patch https://github.com/openstack/horizon/commit/6474215840d4cdde3751c335f46f32cb317238c0
Recently I found that this intermittent failure still occurs. This time I reproduced the network delay by means of script (run as root):
delay.sh
<BEGIN OF FILE>
delay=$1
ip=$2
tc qdisc del dev eth0 root
if [ -n $delay ]; then
tc qdisc add dev eth0 root handle 1: prio
tc filter add dev eth0 parent 1:0 protocol ip pref 55 handle ::55 u32 match ip dst $ip flowid 2:1
tc qdisc add dev eth0 parent 1:1 handle 2: netem delay ${delay}ms
fi
<END OF FILE>
I realized that first the window is opened, then it waits for the page to become accessible, and if I set timeout large enough (say, 10 seconds), I get the following traceback:
Error
Traceback (most recent call last):
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run
self.runTest(result)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest
test(result)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/unittest2/case.py", line 673, in __call__
return self.run(*args, **kwds)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 606, in run
return run_test.run(result)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 80, in run
return self._run_one(actual_result)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 94, in _run_one
return self._run_prepared_result(ExtendedToOriginalDecorator(result))
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 108, in _run_prepared_result
self._run_core()
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 149, in _run_core
self.case._run_teardown, self.result):
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 193, in _run_user
return self._got_user_exception(sys.exc_info())
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 213, in _got_user_exception
self.case.onException(exc_info, tb_label=tb_label)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 558, in onException
handler(exc_info)
File "/home/timur/develop/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 132, in _save_screenshot
self.driver.get_screenshot_as_file(filename)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 758, in get_screenshot_as_file
png = self.get_screenshot_as_png()
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 777, in get_screenshot_as_png
return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 787, in get_screenshot_as_base64
return self.execute(Command.SCREENSHOT)['value']
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 199, in execute
response = self.command_executor.execute(driver_command, params)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
return self._request(command_info[0], url, body=data)
File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 425, in _request
self._conn.request(method, parsed_url.path, body, headers)
File "/usr/lib/python2.7/httplib.py", line 979, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 797, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 778, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 111] Connection refused
It's different from the traceback that follows intermittent failure on gate:
2015-11-24 19:48:29.619 | 2015-11-24 19:48:29.606 | Traceback (most recent call last):
2015-11-24 19:48:29.620 | 2015-11-24 19:48:29.607 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/tests/test_dashboard_help_redirection.py", line 28, in test_dashboard_help_redirection
2015-11-24 19:48:29.621 | 2015-11-24 19:48:29.608 | "help link did not redirect to the right URL")
2015-11-24 19:48:29.623 | 2015-11-24 19:48:29.609 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
2015-11-24 19:48:29.624 | 2015-11-24 19:48:29.610 | self.assertThat(observed, matcher, message)
2015-11-24 19:48:29.625 | 2015-11-24 19:48:29.611 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
2015-11-24 19:48:29.626 | 2015-11-24 19:48:29.613 | raise mismatch_error
2015-11-24 19:48:29.627 | 2015-11-24 19:48:29.614 | testtools.matchers._impl.MismatchError: 'http://docs.openstack.org/' != u'http://localhost/dashboard/project/': help link did not redirect to the right URL
The second one looks more like the command to switch the window was issued before window was actually opened. If that's truly the root cause, fix should be trivial.
Update the above traceback with "Connection refused" was reproduced with normally run Firefox, while for SELENIUM_HEADLESS env var (running in a virtual framebuffer) I got the same "help link did not redirect to the right URL" traceback. Since in gate integration tests are run headless this still might be a timeout/ connectivity issue. I'm still going to give it it a try, if it still doesn't help, the most straightforward way will be to just remove this test.