Support for retrying intermittently failing tests

Bug #1515933 reported by Jonathan Lange
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
testtools
Triaged
Wishlist
Unassigned

Bug Description

testtools should provide a way to retry intermittently failing tests, ala the https://pypi.python.org/pypi/flaky decorator that works with py.test and nose.

Ideally, we would support the @flaky decorator provided by the flaky package.

Essentially, users could specify a maximum number of attempts, N, and a minimum number of successes, M. The test would be retried up N times until there were M successes. If there are M successes, the test counts as passing. If there are not, then the test counts as failed.

Either way, we would attach the errors to the test as details, and also provide some other sort of annotation to make it clear that the test was flaky.

I think the right way to implement this is a RunTest object that wraps another arbitrary RunTest object, so that the flaky retrying logic would work with both Twisted & regular tests. This might require clarifying the interfaces provided by RunTest implementations.

Revision history for this message
Jonathan Lange (jml) wrote :

Would help to have a suite of interface tests for RunTest as well.

Revision history for this message
Jonathan Lange (jml) wrote :

Would help to have an interface for RunTest.

Would help for RunTest to have a better interface.

https://github.com/ClusterHQ/flocker/blob/master/flocker/testtools/_flaky.py implements something along these lines.

Changed in testtools:
status: New → Triaged
importance: Undecided → Wishlist
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.