Currently UTAH proposes the following options:
1. Run the tests on a fresh installation: it means between 20 and 30 minutes of provisioning before the test really starts.
2. Run the tests on an already provisioned system: this means the test environment is not a fresh installation any more after the first run if the test altered the state of the system.
In the context of autolanding, there are 14 stacks (a group of projects) that run autopilot tests against packages from PPAs on physical systems. None of the current options proposed by UTAH is satisfying.
In case 1, it spends 7hours a day to provision the same systems again and again which clearly doesn't scale.
In case 2, only the 1rst run uses a fresh installation, next runs will have packages from a PPA installed on the system.
We'd need the best of 1 and 2: reuse an existing installation reverted to the original state immediately after installation.
We've discussed a couple possible ways of handling this. Ensuring testsuites are idempotent using setup and cleanup routines is one method. LVM snapshots could also work, but this would most likely need to be done at the provisioning level.