Add autopilot suite and code coverage gating to autopilot MPs

Bug #1259334 reported by Francis Ginther
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu CI Services
New
Undecided
Francis Ginther

Bug Description

1) Add the autopilot test suite as a gating test for lp:autopilot MPs.
 - Begin with amd64 (and ideally i386) and add touch devices when the suite is ready. The QA team will notify CI when touch is ready to enable.
 - Note: we currently only have amd64 systems configured for this type of testing. Enabling tests on i386 is a stretch goal here.

2) Add code coverage as a gating test such that MPs which lower the code coverage are rejected.

1) is more urgent then 2).

Revision history for this message
Thomi Richards (thomir-deactivatedaccount) wrote :

The commands to generate coverage reports for the autopilot unit tests are as follows:

python -m coverage erase
python -m coverage run --branch --include "autopilot/*" -m autopilot.run run autopilot.tests.unit
python -m coverage xml

The first command erases any existing coverage data. If you're running in a clean environment you don't need this (but it can't hurt).
The second command runs the tests, and generates coverage data in an internal data format.
The third and final command writes a file named 'coverage.xml' in the current directory that jenkins should be able to parse.

Let me know if you need anything else.

Revision history for this message
Thomi Richards (thomir-deactivatedaccount) wrote :

Actually, those commands should instead be:

python -m coverage erase
python -m coverage run --branch --include "autopilot/*" -m autopilot.run run autopilot.tests.unit
python3 -m coverage run --append --branch --include "autopilot/*" -m autopilot.run run autopilot.tests.unit
python -m coverage xml --omit "autopilot/tests/*"

This will run the python3 unit test suite as well, and will omit test lines from the final coverage report.

Cheers,

Revision history for this message
Francis Ginther (fginther) wrote :

Attempts to add the autopilot suite to CI have run into:
https://bugs.launchpad.net/autopilot/+bug/1259721

The bug causes the test runner to hang. So we either need a fix or a special case workaround. A couple of fixes have been proposed, which look promising.

Revision history for this message
Francis Ginther (fginther) wrote :

QA team has requested adding the touch testing now (back around March 11). I attempted this here (this is the job chain):
http://s-jenkins.ubuntu-ci:8080/job/generic-mediumtests-touch/3265/
http://s-jenkins.ubuntu-ci:8080/job/generic-mediumtests-builder-saucy-armhf/3267/

This ultimately led to the autopilot armhf build that is attached, but I now see the problem. I linked in the wrong job (note the saucy above :-( )... I should have used generic-mediumtests-trusty-touch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.