Currently, if an autopkgtest has ever passed for a particular package on a given architecture, any future failures of that autopkgtest are treated as a regression.
In some cases, an autopkgtest will have succeeded in the past, but now fails even in the release pocket. This can be for a variety of reasons: changes to the autopkgtest infrastructure, newer versions of undeclared test dependencies making it into the release, newer versions of indirect dependencies making it into the release.
When I see a regression blocking a package in proposed-migration, I will often retry the test against the release pocket alone to confirm that this is actually a regression in the release. If it fails, I will add a force-badtest hint for the confirmed-bad test in release.
It occurs to me that the second step may be redundant, if we instead reset the baseline pass/fail of the test to the most recent run of the current release version of the test. This would reduce the need for the release team to spend so much time managing hints for failing autopkgtests.
- if the package is confirmed to fail in release, it will not block other packages from migrating.
- if a new version of the package is uploaded, and the test still fails, the new version will also be allowed to migrate without the need for bumping hints.
- if a package test is flaky, we don't have to hammer the retry button repeatedly for each reverse-dependency (though we can, if we care about confirming that the test suite *can* pass, or if this is easier because the testsuite usually passes); we can retry for the release pocket and reset the baseline.
- but if a flaky test has passed in the most recent version, the default is that it will still gate, so we don't have to worry about this causing us to blindly accept regressions.
- if a test has regressed due to a regression on the infrastructure which will later be fixed, no special handling is required to re-enable gating when the test starts passing again in the same package version (i.e. no inaccurate 'badtest' hint to be removed)
- release team members no longer have to choose between badtesting a single version (which requires active management when a new version lands in -proposed) and hinting all versions (which leaves us blind to actual regressions).
- indeed, the release team can be hands-off as a whole, since anyone with autopkgtest retry privs can do this.
As a concrete example of this, the latest perl upload triggered test failures on 34 reverse-
Having a mechanism to reset the baseline would make better use of release team time than on this manual, highly error-prone, and incomplete analysis of build failures.