Zope layer setup and teardown 'tests' cannot be filtered by testr
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Launchpad itself |
Triaged
|
High
|
Unassigned | ||
| Testrepository |
Fix Released
|
Wishlist
|
Robert Collins | ||
| zope.testing |
Invalid
|
Undecided
|
Unassigned | ||
Bug Description
What happens:
If you run the Launchpad tests with --subunit, it will report the layer setup and teardown notices as tests, which they aren't. This leads to test count inflation, which gets very confusing for developers (especially when you're running parallel tests, which means that you add about 20 such notices per lxc instance).
Proposed solution (from lifeless):
- Update the zope.testing subunit support so that layer setup and teardown messages are tagged with "layer".
- already the case
- Subunit needs to know about tags.
- already the case
- Testr needs to not count or record layer-tagged things as tests.
- needs to be configurable, and is pending
- Testr should still dump the layer-tagged messages to the raw stream on disk.
- same as above
Sub-bugs:
bug 914281 - noop - case of 518016
bug 914279 - fixed
bug 914278 - fixed
bug 518016 - fixed
Related branches
- testrepository committers: Pending requested
-
Diff: 918 lines (+359/-155)19 files modifiedNEWS (+8/-3)
doc/MANUAL.txt (+20/-2)
testrepository/commands/failing.py (+15/-4)
testrepository/commands/last.py (+7/-3)
testrepository/commands/load.py (+17/-4)
testrepository/results.py (+40/-36)
testrepository/testcommand.py (+59/-2)
testrepository/tests/__init__.py (+17/-0)
testrepository/tests/commands/test_failing.py (+15/-1)
testrepository/tests/commands/test_last.py (+19/-2)
testrepository/tests/commands/test_load.py (+17/-1)
testrepository/tests/test_results.py (+16/-7)
testrepository/tests/test_testcommand.py (+53/-1)
testrepository/tests/test_ui.py (+4/-65)
testrepository/tests/ui/test_cli.py (+35/-9)
testrepository/ui/__init__.py (+3/-1)
testrepository/ui/cli.py (+8/-10)
testrepository/ui/decorator.py (+3/-2)
testrepository/ui/model.py (+3/-2)
| Jonathan Lange (jml) wrote Re: [Bug 914166] [NEW] Testr thinks that Zope layer setup and teardown messages are tests | #1 |
| Graham Binns (gmb) wrote | #2 |
| Changed in zope.testing: | |
| status: | New → Invalid |
| Changed in launchpad: | |
| status: | Triaged → In Progress |
| Changed in launchpad: | |
| importance: | Undecided → High |
| description: | updated |
| description: | updated |
| summary: |
- Testr thinks that Zope layer setup and teardown messages are tests + Zope layer setup and teardown 'tests' cannot be filtered by testr |
| description: | updated |
| description: | updated |
| Changed in launchpad: | |
| status: | In Progress → Triaged |
| Changed in testrepository: | |
| status: | New → Triaged |
| importance: | Undecided → Wishlist |
| Changed in testrepository: | |
| status: | Triaged → Fix Committed |
| assignee: | nobody → Robert Collins (lifeless) |
| milestone: | none → next |
| Changed in testrepository: | |
| status: | Fix Committed → Fix Released |
On Tue, Jan 10, 2012 at 9:06 AM, Graham Binns <email address hidden> wrote:
> Public bug reported:
>
> What happens:
> If you run the Launchpad tests with --subunit, it will report the layer setup and teardown notices as tests, which they aren't. This leads to test count inflation, which gets very confusing for developers (especially when you're running parallel tests, which means that you add about 20 such notices per lxc instance).
>
> Proposed solution (from lifeless):
> - Update the zope.testing subunit support so that layer setup and teardown messages are tagged with "layer".
> - Subunit needs to know about tags.
> - Testr needs to not count or record layer-tagged things as tests.
> - Testr should still dump the layer-tagged messages to the raw stream on disk.
Some thoughts:
* zope.testing already outputs a 'zope:layer' tag after emitting the
start of the test that represents the layer
* It treats layers as tests to provide a place to put layer
setUp/tearDown errors
* It would be nicer for testr to not know about layers, but rather to
achieve the same result with (not yet existing) subunit filtering
tools
jml