Draft email to maintainers regarding Priority / Severity in Bugzilla

[James Laska]

On Wed, 2009-04-15 at 10:56 -0700, Adam Williamson wrote:
> 
> So, really, we just want your feedback: do you think this system might
> prove useful to you as a maintainer? Can you see any problems with it,
> or potential refinements or improvements? Bugzappers' mission is to
> ease
> the lives of maintainers, so we don't want to put this in place unless
> it's seen as beneficial by at least some maintainers. Thanks!

Thanks for the proposal Adam.  I read this as a two-part request.
Please correct if I've misread.

     1. First, is a draft definition of bugzilla severity/priority
        (http://fedoraproject.org/wiki/User:Beland/Bugzilla_Legend#Severity_and_Priority) up for review
     2. Second, is a proposal for how best to incorporate those
        definitions into QA/BugZapper bug procedures (this thread)

Where QA feels the pinch here is there is currently no objective measure
for providing guidance on the impact of bugs.  Why does it matter?
Currently I see 4079 OPEN rawhide bugs (2152 filed since 2009-01-01 and
1120 filed since F-11-Beta).  When asked the common question, "how does
the release look?"  The supporting data is too broad to provide anything
other than a gut-level response, "feels [good,bad]."

There are ways we can narrow the scope of the current data set by
improving our view:
      * view by component
      * view by feature (component collections)
      * view by DUP count

Improving our view of bug data will be instrumental (and is something
BugZapper Brennan Ashton has been investigating).  But views are only as
good as the data.  Gaining confidence in severity/priority is a good
step towards improving the data.  In fact we are implicitly doing this
now by creating tracker bugs.  We all apply some rules based on our
experiences to assess whether something should be considered as a
blocker issue (whether it's F11AnacondaBlocker, F11Beta, F11Preview,
F11Blocker, F11VirtBlocker etc...).

> We have a draft convention for how these fields should be set here:
> http://fedoraproject.org/wiki/User:Beland/Bugzilla_Legend#Severity_and_Priority 

What is most interesting to me is the 'severity' and keeping it an
objective measurement of impact.  The system panics or it doesn't.
There is a workaround, or there isn't.  

As currently written: 

> Severity is used to describe how bad a bug is for the reporter

I agree, but I fear the definition leaves room for the perception of
abuse.  How about a slight rewording, "Severity is used to provide an
objective assessment on defect impact".

> Priority is used to indicate what order bugs should be fixed, in the
> context of the entire release:

I agree in spirit, but would recommend wording focused around the
'subjective' nature of the defect.  This is where <someone> (maintainer,
skilled triager, skilled reporter) get's to apply their subject matter
expertise to provide guidance on the impact.  It's a typo on the gdm
login screen (low severity, but as gdm maintainer I might consider this
high priority given it's user visibility).

As for the scale provided: 

> >       * Urgent: Software is completely unusable, loses data, or the
> >         RPM won't update properly. Frequent or commonly encountered
> >         crashes.

Maybe nit-picky, but "unusable" and "frequent" are subjective terms.
Once weekly can be just as bad as once/per hour depending on component,
workload, reporter etc...

Not pretty, but perhaps ... "Urgent: The system is down or the software
is not functioning, loss or corruption of data exists.  No known
workarounds."

> >       * High: Loss of important functionality, or usability problem
> >         producing major frustration. Infrequent and un-reproducible
> >         crashes. 

I might suggest removing "major frustration" ... that's subjective, not
objective.  Perhaps add "running at a severely reduced capacity"

> >       * Medium: Highly visible cosmetic defect, moderate usability
> >         problem, loss of minor functionality, moderate loss of
> >         functionality with workaround, or high priority request for
> >         enhancement. [default choice] 

Recommend dropping "or high priority request for enhancement".  

> >       * Low: Minor cosmetic defect, minor usability problem, or low
> >         priority request for enhancement. 

Another thought, how about we include common examples with each
severity?

Thanks,
James
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/fedora-test-list/attachments/20090417/241ea002/attachment.sig>