First step at useful metrics

[John Poelstra]

We've been searching for long time for a way to accurately reflect the 
efforts of bug triagers.  Now I think we are one step closer though 
there are many more to go.

As a result of an earlier post here, James Laska showed me some db 
queries he uses which after some more back and forth inspired a new idea 
and thus the data attached here.  Big thanks to James for crafting the 
query and script to help me generate these results!  If you are 
interested in helping with the SQL queries and data representation 
please let me know--off list is fine.

There are many different ways we can look at this data and refine the 
different views of it.

The purpose of this email is to ask: Are we on the right track?  Does 
this seem like a reasonable way to measure bug triager participation and 
recognize the top performers?

The purpose is NOT to get bogged down in whether the data is 100% 
accurate or why invalid states were used or 500 different ways this 
report should have been created.  The focus of this first step is 
"triage".  Naturally the aggregated data could also be used to rank top 
bug reporters and package maintainers, but that is not the focus of this 
email.

As I thought about how to best capture bug triager activity it occurred 
to me that a triager has the potential to touch all of the bug states 
so... why not aggregate all of that activity?  Yes there are obvious 
ways to "game" the data and other potential deficiencies, but this is 
better than what we have today (which is nothing).

All this to say... the attached spreadsheet in ods and pdf was generated 
  in the following way building on the previous step:
1) All bugs with product = Fedora
2) all state changes for those bugs during the period of 2007-11-07 to 
2008-05-12 (Fedora 9 development)
3) aggregate state changes by user
4) add all state changes together
5) report all users with >= 100 state changes (just for this exercise)

So the thought is that "total state changes" represents the level of 
activity a certain person had in bugzilla during the specified time.  By 
further filtering the results using
https://fedoraproject.org/wiki/BugZappers/ActiveTriagers we arrive at an 
activity level per bug triager.  For example, if the NEEDINFO column for 
Bug Zapper says 500, this means Bug Zapper placed 500 bugs in the 
NEEDINFO state.

DISCLAIMER#1: the spreadsheet contains a column for "FILED".  This 
represents new bugs filed by that user vs. bugs that were transitioned 
to NEW from another state by that user.  Naturally FILING bugs is not 
really a "triage" activity so it is just here for example and to 
generate ideas for other groups.

DISCLAIMER#2: While every attempt has been made to present correct data 
we could have made an error in the way we ran the query. While we 
definitely want to know what might be wrong we are more interested in 
asking if this is this is a (ONE OF MANY) worthy means of measuring 
community participation.

Thanks for reading,
John
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fedora9-campaign-100.pdf
Type: application/pdf
Size: 32526 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-test-list/attachments/20080603/d1fb4c09/attachment.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fedora9-campaign-100.ods
Type: application/vnd.oasis.opendocument.spreadsheet
Size: 17405 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-test-list/attachments/20080603/d1fb4c09/attachment.ods>