First step at useful metrics

[John Poelstra]

John Summerfield said the following on 06/04/2008 09:04 PM Pacific Time:
> Jon Stanley wrote:
>> On Wed, Jun 4, 2008 at 9:27 PM, John Summerfield
>> <debian at herakles.homelinux.org> wrote:
>>
>>> I'd rather John spoke for himself.
>>
>> Well with Will and I being John's cohorts in crime, I think we can
>> speak to what each of us wants.  First, we want to measure the impact
>> that we're having in the project. This is just good sense, how do you
>> know that you're doing something that's worthwhile if you have no
>> metrics to back up your position? The second is, as Will said,
>> recognition and reward for top triagers.  Triage is a thankless job,
>> so we need some way to recognize the rock stars of triage, and reward
>> them appropriately.
> 
> 
> I've been in IT since it was called ADP/EDP, and remember punched cards, 
>  paper tape and typewriter keyboards on large computers.
> 
> On occasion I've been hired to write programs, and one of the questions 
> much in the mind of managers is how to measure productivity.
> 
> A measure in one place I worked was "lines of code." Programmers writing 
> more lines of code was held to be more productive than those writing fewer.
> 
> This without any regard for complexity, or code quality (another thing 
> that's hard to quantify), or consumption of resources (such as _then_ 
> very expensive computer time).
> 
> A number of LOC I've seen quoted is 100 per day, for a basically capable 
> programmer on not too-difficult code, and including documentation (to 
> standards I don't recall, but undoubtedly better than that which one 
> ordinarily sees in OS code).
> 
> At number of LOC was found to be fairly constant over a variety of 
> programming languages (but I suspect C wasn't included in the survey), 
> so writing in higher-level languages such as COBOL and PL/1, where 
> suitable, had obvious benefits in business applications than assembler, 
> and interpretive languages such as REXX (which probably didn't exist 
> then) produced even greater productivity benefits.
> 
> Now, writing computer programs is just one example of problem solving; 
> writing (books, stories etc) and triaging are other examples.
> 
> I don't think metrics that are easily measured are very useful in 
> measuring productivity.
> 
> If Will triaged 100 bugs in May, that's (I expect) quite good. However, 
> if 20 of them are misassigned or otherwise badly handled, I think I 
> would rather he handled 50 bugs and got 48 of them right.
> 
> A little while ago I filed a bug report against kernels 2.6.25, my 
> DC7700 wouldn't boot because the kernel couldn't find its disks.
> 
> The mkinitrd team said, "That's ours," without (I think, but I might be 
> defaming them unfairly) looking very closely at it. Had I left it at 
> that, it might still be on their list, but I went to some effort to 
> prove it wasn't mkinitrd and then changed the assignment.
> 
> Misassignment can be quite expensive, in terms of getting problems fixed.
> 
> I think that the idea of acknowledging good performance is excellent, 
> but that its implementation needs to be done carefully.
> 
> To achieve an award, I suggest these need to be considered. Change the 
> numbers to suit, I have no idea what absolute numbers are realistic.
> 1. Activity. People need to be more than casual workers. Maybe disposing 
> of 20 bugs in a month. If you don't do at least this amount of work, you 
> won't be recognised[1].
> 2. Accuracy. Allow maybe 5% error rate: one mistake in 20. If you don't 
> measure up on this, you are not eligible for recognition (but you might 
> attract some attention form a  mentor).
> 3. Relative performance. Not the absolute best performance, this isn't 
> the Olympics, but compared with the worker's previous efforts. If Jon's 
> doing more, or making fewer mistakes than before, acknowledge that.
> 4. Knowing when to ask for advice. Jo says, "I've done this and that, 
> and I think .., what do you say?" This is way better than mishandling 
> the problem, and makes it into a learning experience. OTOH if Jo's only 
> had a quick look and done no analysis of the problem, then asking for 
> help is a distraction to others. Better to pass on that one, and go for 
> the next.
> 5. Mentoring. Jeff's been around for a while, maybe doesn't do much 
> triaging himself these days, but he's always ready to help people like 
> Jo. And he keeps an eye on newcomers, and those prone to making mistakes.
> 
> [1] Unless you're new. New workers should be publicly welcomed.
> 
> 
> I might be using "mentor" a little loosely; those more experienced 
> should be expected to help others (and package maintainers probably 
> should be mentoring too). I don't mean a formal mentoring role such as 
> Debian has (but it might be a good idea).
> 
> I can see that some simple metrics might help, but be careful they arent 
> used too mechanistically.
> 
> 
> 

John Speaks :)

Big picture right now is that we have no way to provide positive 
feedback to those doing bug triage.  On its face many would consider bug 
triage to be a "miserable job"... I'm looking for ways to mitigate that: 
http://poelcat.wordpress.com/2008/02/23/miserable/

I hope that providing information to the individuals and the wider 
community can help alleviate Anonymity, Irrelevance, and Immeasurement.

Each person is motivated by different things.  For me the biggest 
detractor for getting involved in bug triage was not how my performance 
would be measured, but that what I did wouldn't matter--thus for me I 
want to charts of overall performance and how the efforts of the team 
are paying off.  Others might be competitive and want to see how they 
rank against others.  There are many motivators too.

My approach to these metrics is positive motivation, not issuing 
accurate performance reviews to be considered for year-end compensation 
  increases :)

Thanks for sharing your ideas cautions about using metrics. The big 
difference between your example of misguided metrics and our project 
here is that there is NO "management".  Management is all of us and what 
WE decide we want things to be.

Here is the hard part, but the part that will help all of this 
most--what can we tangibly start doing today to recognize the efforts of 
bug triagers and chart our overall progress?  I'm not talking about 
t-shirts, candy bars and USB keys.  Yes, we may have those from time to 
time, but that is like trying to motivate employees soley by 
money--which most studies have shown isn't a long term sustainable 
motivator.

I believe to really scale we need to be able to collect motivational 
information about our individual and group progress in an automated way 
that can be reported in an automated way.  So while I agree with your 
ideas, a helpful and much needed next step would be to outline exactly 
how we can practically start doing them in Fedora.

Thanks,
John