Recent fix to integration tests didn't fix the intermittent test_dashboard_help_redirection failure

Bug #1520582 reported by Timur Sufiev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Dashboard (Horizon)
Fix Released
Undecided
Unassigned

Bug Description

A watchful reviewer may notice that sometimes test_dashboard_help_redirection fails. Initially I thought that it was caught by an increased network timeout from Jenkins worker node to docs.openstack.org site. So I made a patch https://github.com/openstack/horizon/commit/6474215840d4cdde3751c335f46f32cb317238c0

Recently I found that this intermittent failure still occurs. This time I reproduced the network delay by means of script (run as root):

delay.sh
<BEGIN OF FILE>
delay=$1
ip=$2

tc qdisc del dev eth0 root
if [ -n $delay ]; then
    tc qdisc add dev eth0 root handle 1: prio
    tc filter add dev eth0 parent 1:0 protocol ip pref 55 handle ::55 u32 match ip dst $ip flowid 2:1
    tc qdisc add dev eth0 parent 1:1 handle 2: netem delay ${delay}ms
fi
<END OF FILE>

I realized that first the window is opened, then it waits for the page to become accessible, and if I set timeout large enough (say, 10 seconds), I get the following traceback:

Error
Traceback (most recent call last):
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run
    self.runTest(result)
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest
    test(result)
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/unittest2/case.py", line 673, in __call__
    return self.run(*args, **kwds)
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 606, in run
    return run_test.run(result)
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 80, in run
    return self._run_one(actual_result)
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 94, in _run_one
    return self._run_prepared_result(ExtendedToOriginalDecorator(result))
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 108, in _run_prepared_result
    self._run_core()
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 149, in _run_core
    self.case._run_teardown, self.result):
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 193, in _run_user
    return self._got_user_exception(sys.exc_info())
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 213, in _got_user_exception
    self.case.onException(exc_info, tb_label=tb_label)
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 558, in onException
    handler(exc_info)
  File "/home/timur/develop/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 132, in _save_screenshot
    self.driver.get_screenshot_as_file(filename)
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 758, in get_screenshot_as_file
    png = self.get_screenshot_as_png()
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 777, in get_screenshot_as_png
    return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 787, in get_screenshot_as_base64
    return self.execute(Command.SCREENSHOT)['value']
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 199, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
    return self._request(command_info[0], url, body=data)
  File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 425, in _request
    self._conn.request(method, parsed_url.path, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 979, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 797, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 778, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err
error: [Errno 111] Connection refused

It's different from the traceback that follows intermittent failure on gate:

2015-11-24 19:48:29.619 | 2015-11-24 19:48:29.606 | Traceback (most recent call last):
2015-11-24 19:48:29.620 | 2015-11-24 19:48:29.607 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/tests/test_dashboard_help_redirection.py", line 28, in test_dashboard_help_redirection
2015-11-24 19:48:29.621 | 2015-11-24 19:48:29.608 | "help link did not redirect to the right URL")
2015-11-24 19:48:29.623 | 2015-11-24 19:48:29.609 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
2015-11-24 19:48:29.624 | 2015-11-24 19:48:29.610 | self.assertThat(observed, matcher, message)
2015-11-24 19:48:29.625 | 2015-11-24 19:48:29.611 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
2015-11-24 19:48:29.626 | 2015-11-24 19:48:29.613 | raise mismatch_error
2015-11-24 19:48:29.627 | 2015-11-24 19:48:29.614 | testtools.matchers._impl.MismatchError: 'http://docs.openstack.org/' != u'http://localhost/dashboard/project/': help link did not redirect to the right URL

The second one looks more like the command to switch the window was issued before window was actually opened. If that's truly the root cause, fix should be trivial.

Timur Sufiev (tsufiev-x)
description: updated
tags: added: integration-tests
Revision history for this message
Timur Sufiev (tsufiev-x) wrote :

Update the above traceback with "Connection refused" was reproduced with normally run Firefox, while for SELENIUM_HEADLESS env var (running in a virtual framebuffer) I got the same "help link did not redirect to the right URL" traceback. Since in gate integration tests are run headless this still might be a timeout/connectivity issue. I'm still going to give it it a try, if it still doesn't help, the most straightforward way will be to just remove this test.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to horizon (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/250839

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to horizon (master)

Reviewed: https://review.openstack.org/250839
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=f523af9bed5ea93e00a6d42d143814401b5cd62a
Submitter: Jenkins
Branch: master

commit f523af9bed5ea93e00a6d42d143814401b5cd62a
Author: Timur Sufiev <email address hidden>
Date: Fri Nov 27 17:24:12 2015 +0300

    Try to avoid intermittent test_dashboard_help_redirection failure

    Assuming that the additional browser window is actually opened after
    we command test to switch to it, swap these 2 commands. If it doesn't
    help, remove the failing test, because ensuring good network
    connectivity to external URLs is out of integration tests scope.

    Change-Id: I2c972a9ac813f30bb2ea7e6e7d366f729fd90bfb
    Related-Bug: #1520582

Revision history for this message
Gary W. Smith (gary-w-smith) wrote :

Marking as closed because the fix was merged.

Changed in horizon:
status: New → Fix Released
Akihiro Motoki (amotoki)
Changed in horizon:
milestone: none → queens-1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.