Aodh

aodh.tests.storage.test_get_connection.ConnectionRetryTest.test_retries is mocking time module so sometimes fails

Bug #1482722 reported by Chris Dent
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Aodh
Fix Released
Undecided
Chris Dent
Aodh 1.0.0
Ceilometer
Fix Released
Undecided
Chris Dent
Ceilometer 5.0.0 "liberty"

Bug Description

Sometimes ConnectionRetryTest.test_retries can fail as follows. This is because the assertions are checking for the number of calls on the sleep method in retrying. Turns out that's the same sleep used in the time module which is apparently sometimes getting called elsewhere in this concurrent environment.

2015-08-07 15:52:18.189 | ======================================================================
2015-08-07 15:52:18.190 | FAIL: aodh.tests.storage.test_get_connection.ConnectionRetryTest.test_retries
2015-08-07 15:52:18.190 | tags: worker-1
2015-08-07 15:52:18.190 | ----------------------------------------------------------------------
2015-08-07 15:52:18.191 | Empty attachments:
2015-08-07 15:52:18.191 | stderr
2015-08-07 15:52:18.191 | stdout
2015-08-07 15:52:18.192 |
2015-08-07 15:52:18.192 | Traceback (most recent call last):
2015-08-07 15:52:18.193 | File "aodh/tests/storage/test_get_connection.py", line 74, in test_retries
2015-08-07 15:52:18.193 | self.assertEqual(9, retry_sleep.call_count)
2015-08-07 15:52:18.193 | File "/home/jenkins/workspace/gate-aodh-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
2015-08-07 15:52:18.193 | self.assertThat(observed, matcher, message)
2015-08-07 15:52:18.193 | File "/home/jenkins/workspace/gate-aodh-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
2015-08-07 15:52:18.193 | raise mismatch_error
2015-08-07 15:52:18.193 | testtools.matchers._impl.MismatchError: 9 != 10

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to aodh (master)

Fix proposed to branch: master
Review: https://review.openstack.org/210538

Changed in aodh:
status: New → In Progress
Chris Dent (cdent)
summary: aodh.tests.storage.test_get_connection.ConnectionRetryTest.test_retries
- is time dependent and thus subject to failure in slow gate
+ is mocking time module so sometimes fails
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/210542

Changed in ceilometer:
assignee: nobody → Chris Dent (chdent)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to aodh (master)

Reviewed: https://review.openstack.org/210538
Committed: https://git.openstack.org/cgit/openstack/aodh/commit/?id=d4f24e1996215d754944487b18035f1eacc0fc62
Submitter: Jenkins
Branch: master

commit d4f24e1996215d754944487b18035f1eacc0fc62
Author: Chris Dent <email address hidden>
Date: Fri Aug 7 17:03:52 2015 +0000

    Make ConnectionRetryTest more reliable

    The previous test relied on mocks of retrying.time.sleep. This is
    the same sleep as time.sleep so there's a chance, if eventlet is in
    context, that something else can call that.

    So instead the test now checks to see how many times should_reject
    is called. This is the same as the max retries that will be
    attempted.

    Also the retry wait time has been set to something very small so the
    test doesn't hang around sleeping uselessly.

    Change-Id: Ib85fff55588f7183d4e27a8a4769f647aeaa0ce3
    Closes-Bug: #1482722

Changed in aodh:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/210542
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=00aa6cec3ec0c9725649ba3eaed9545a4076a28a
Submitter: Jenkins
Branch: master

commit 00aa6cec3ec0c9725649ba3eaed9545a4076a28a
Author: Chris Dent <email address hidden>
Date: Fri Aug 7 17:20:02 2015 +0000

    Make ConnectionRetryTest more reliable

    The previous test relied on mocks of retrying.time.sleep. This is
    the same sleep as time.sleep so there's a chance, if eventlet is in
    context, that something else can call that.

    So instead the test now checks to see how many times should_reject
    is called. This is the same as the max retries that will be
    attempted.

    Also the retry wait time has been set to something very small so the
    test doesn't hang around sleeping uselessly, making test runs
    faster.

    Change-Id: Ib15d4d6f8dd23ba534d58005da5dc64aad9e0e90
    Closes-Bug: #1482722

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in aodh:
milestone: none → 1.0.0
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: liberty-3 → 5.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.