QA verification of fixes in Launchpad is non-obvious and error-prone

Why is using tags a suboptimal workaround?

When the fixes to bugs turn out not to work, bugs often get re-opened (that is, their status changes from Fixed Something to one that indicates that the bug needs work), so capturing the QA results using the status can be problematic for some projects.

I also think that being able to capture QA results using the bug tracker is very important.

<https://bugs.launchpad.net/malone/+bugs?field.tag=bugtag> probably offers some clues. If those bugs were fixed, projects could much more easily use tags to track their project-specific QA processes.

I've changed this bug to private so I can include some additional information from an offline conversation:

That is what is exactly what is being done now, but it causes unnecessary stress and confusion. Can't you have a critical bug that also has a workaround? The workaround isn't sufficient because (in no particular order)

a.) the lack of a verification status in our standard bug flow alarms customers. QA is an important stage, and a fix that has been verified or failed does actually put the bug in a different state. Having it represented by a state is just the right thing to do (imho). I suspect this isn't just an OEM team request.

b.) the tags are too loose, and too subject to error. We end up with bugs that are in a FixCommitted state with both verify and verifyfailed tags (and of course there is nothing in the system to avoid tags that should be logically exclusive). Or even at the end of the process with a FixReleased bug and a verify-failed tag. Yes, more education and discipline could solve some of this, but that's back to the unnecessary stress and confusion, when you consider explaining to a customer and their far-flung QA teams why this important part of the workflow (QA) is represented in our system by what is clearly a workaround.

c.) tags don't work well in part because of some of the other bugs I filed (e.g, the fixed columns in a bug list). You can search for a tag, but for example you can't see a list of FixCommitted bugs and easily identify (or even sort by) which ones have what verification tags. Yes, you can do this in separate searches (but again there are bugs about not being able to see what search criteria have been applied or not being able to save searches), so you end up with different tabs that both have FixCommitted bugs and you have to remember what search you did in what tab. The entire workflow is just a bunch of workarounds that feel very unpleasant and clunky.

I think the bottom line is that tags are an awkward workaround, and then they are made even more awkward by the limitations in searching/reporting.

I don't think having a "Fix Verified" status is a bad idea per se, but every extra bug status makes Launchpad harder to understand (q.v. bug 29495), even for projects that never use it. (Launchpad Bugs currently has nine bug statuses, three of which are useless for most projects.) So if fixing those various tag-related bugs instead could make tags convenient enough for everyday use, and upstanding enough to reassure customers (e.g. prominent checkboxes for a project's common tags), I think that's worth considering. Especially since that would help every project that uses tags for anything, as opposed to only those projects that have verification processes.

What would "Fix Failed" be used for as distinct from "Confirmed"/"Triaged"? (I know what it would mean, I'm asking what it would be used for.)

> ... if fixing those various tag-related bugs instead could make tags convenient enough for everyday use, and upstanding enough to reassure customers (e.g. prominent checkboxes for a project's common tags), I think that's worth considering

+1

Another problem with adding statuses is the order. Where will they come, after Fix Committed? After Fix Released? every project will have a different workflow for when bugs are being QAed.

2008/11/14 Tom Berger <email address hidden>:
> Another problem with adding statuses is the order. Where will they come,
> after Fix Committed? After Fix Released? every project will have a
> different workflow for when bugs are being QAed.

AIUI the desired order is Fix Committed -> Fix Released -> Fix
{Verified,Failed}.

Matthew Paul Thomas wrote:
> I don't think having a "Fix Verified" status is a bad idea per se, but
> every extra bug status makes Launchpad harder to understand (q.v. bug
> 29495), even for projects that never use it. (Launchpad Bugs currently
> has nine bug statuses, three of which are useless for most projects.)

I'm not sure I see that as a very good argument - i.e., "we already
implemented useless states, so don't want to add any more states whether
they are useful or useless"! ;)

> What would "Fix Failed" be used for as distinct from
> "Confirmed"/"Triaged"? (I know what it would mean, I'm asking what it
> would be used for.)
>
>

It would be used to know a bug's actual status in the workflow - i.e.,
that a fix has been tried and didn't work. When you consider that at
any given time there are lots of bugs that are confirmed but don't have
a fix attempted for them yet, it's useful to have a different state for
those that have had a fix attempted. When this is done with tags, you
then have 2 data elements to manage for the rest of the workfloe. As it
is re-fixed and iterates through the workflow again, you have to manage
bug status and edit/remove tags in order to represent bug status - i.e.,
we're representing the same thing (status) in two different and unlinked
ways (status and tags). In my view, FixVerified and FixFailed (or
similar) are discrete steps in most project's workflow, and should be
represented as discrete status values.

But I also suggest that a wider view should be taken on this. I know
Launchpad serves many customers, but the primary ones (at least at this
stage) are Ubuntu and Canonical (i.e,. our private projects). Perhaps a
survey of the Ubuntu, Landscape, Online Services, OEM Services, etc team
would help shed light on what LP users actual requirements are on this
particular topic? (I.e,. I don't presume to speak for all of them, just
as I'm sure LP devs don't.)

> AIUI the desired order is Fix Committed -> Fix Released -> Fix {Verified,Failed}.

This wouldn't work for Launchpad, for example, since we QA after Fix Committed but before Fix Released.

> > What would "Fix Failed" be used for as distinct from
> > "Confirmed"/"Triaged"? (I know what it would mean, I'm asking what it
> > would be used for.)
>
> It would be used to know a bug's actual status in the workflow - i.e.,
> that a fix has been tried and didn't work. When you consider that at
> any given time there are lots of bugs that are confirmed but don't have
> a fix attempted for them yet, it's useful to have a different state for
> those that have had a fix attempted.

This is something that should be easy to get from looking at the bug activity. Again, to the extent that this isn't easy to do now we should fix that (by displaying the status transition on the main bug page, for example). We could also display it in a unique way, but without requiring the user the record any additional info. The Ubuntu QA team, for example, inspects the time when a bug left a 'Close' status and went back to an 'Open' one. This is done using the API, but if it proves very useful we should display this information in the web interface.

> But I also suggest that a wider view should be taken on this. I know
> Launchpad serves many customers, but the primary ones (at least at this
> stage) are Ubuntu and Canonical (i.e,. our private projects). Perhaps a
> survey of the Ubuntu, Landscape, Online Services, OEM Services, etc team
> would help shed light on what LP users actual requirements are on this
> particular topic? (I.e,. I don't presume to speak for all of them, just
> as I'm sure LP devs don't.)

Couldn't agree more. To be clear - I'm not saying that adding these statuses is not the best solution, just that we should first understand what the optimal workflow is, then design the user interface to facilitate it.

2008/11/14 Tom Berger <email address hidden>:
>> AIUI the desired order is Fix Committed -> Fix Released -> Fix
> {Verified,Failed}.
>
> This wouldn't work for Launchpad, for example, since we QA after Fix
> Committed but before Fix Released.

We don't necessarily have to use Fix Verified in LP, though. Also, we
could always add these and enable them on a project-by-project basis
so that it doesn't affect those projects that don't need it.

I do think the set of existing statuses bears on the introduction of new ones, because for example if someone considering adopting Launchpad browsed a project milestone page and saw a jumble of "Fix Committed" vs. "Fix Released" vs. "Fix Verified" vs. "Fix Failed", they might reasonably conclude it was too complicated, and flee. The same applies to potential QA volunteers for Ubuntu and other large projects. (Fixing bug 163694 would help in this case.)

Meanwhile on "Fix Failed": Who needs to know that a fix has been tried and didn't work, and for how long do they need to know it? A developer working on a second attempt at fixing a bug will presumably still need to use the "In Progress" and "Fix Committed" statuses to track that second fix, thereby causing the bug to lose its "Fix Failed" status. That suggests to me that whether a previous fix failed is orthogonal to the bug's current status, and therefore needs to be tracked somewhere other than status, even if that other place isn't a tag. Perhaps it could be auto-generated ... as Tom already suggested while I was writing this. :-)

On Fri, Nov 14, 2008 at 02:11:59PM -0000, Graham Binns wrote:
> 2008/11/14 Tom Berger <email address hidden>:
> > Another problem with adding statuses is the order. Where will they come,
> > after Fix Committed? After Fix Released? every project will have a
> > different workflow for when bugs are being QAed.
>
> AIUI the desired order is Fix Committed -> Fix Released -> Fix
> {Verified,Failed}.

I think you just proved Tom's point :) For example, in Launchpad we do
the QA mainly between Fix Committed and Fix Released, so the order you
suggested wouldn't work for us.