test regressions should be calculated per-test, not per-package
Bug #1767237 reported by
Steve Langasek
This bug affects 1 person
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Auto Package Testing |
New
|
Wishlist
|
Unassigned | ||
| britney |
New
|
Wishlist
|
Unassigned | ||
Bug Description
currently the britney autopkgtest policy calculates whether a test failure is a regression on a per-package basis: if there has previously been a successful test run of the tests for this package on this architecture where all the tests pass, any test failure is a regression. If there has previously a run of the tests for this package where *some* of the tests pass, no test failure is a regression even if there is an increase in the set of tests that are failing.
It would be nice to be able to surface individual test results to britney so that we could gate more cleanly - and even to be able to force-badtest individual tests.
| Changed in auto-package-testing: | |
| importance: | Undecided → Wishlist |
| Changed in britney: | |
| importance: | Undecided → Wishlist |
Revision history for this message
| Iain Lane (laney) wrote | #1 |
To post a comment you must log in.
The individual test results are in the artifacts "summary" file, so this can be done.
From my end "wishlist" is about right, not least since it is quite common for packages to bundle many tests into one autopkgtest test so this is probably not going to end up being as useful as you might hope.