Support for retrying intermittently failing tests
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
testtools |
Triaged
|
Wishlist
|
Unassigned |
Bug Description
testtools should provide a way to retry intermittently failing tests, ala the https:/
Ideally, we would support the @flaky decorator provided by the flaky package.
Essentially, users could specify a maximum number of attempts, N, and a minimum number of successes, M. The test would be retried up N times until there were M successes. If there are M successes, the test counts as passing. If there are not, then the test counts as failed.
Either way, we would attach the errors to the test as details, and also provide some other sort of annotation to make it clear that the test was flaky.
I think the right way to implement this is a RunTest object that wraps another arbitrary RunTest object, so that the flaky retrying logic would work with both Twisted & regular tests. This might require clarifying the interfaces provided by RunTest implementations.
Would help to have a suite of interface tests for RunTest as well.