First step at useful metrics

John Summerfield debian at herakles.homelinux.org
Thu Jun 5 04:04:20 UTC 2008


Jon Stanley wrote:
> On Wed, Jun 4, 2008 at 9:27 PM, John Summerfield
> <debian at herakles.homelinux.org> wrote:
> 
>> I'd rather John spoke for himself.
> 
> Well with Will and I being John's cohorts in crime, I think we can
> speak to what each of us wants.  First, we want to measure the impact
> that we're having in the project. This is just good sense, how do you
> know that you're doing something that's worthwhile if you have no
> metrics to back up your position? The second is, as Will said,
> recognition and reward for top triagers.  Triage is a thankless job,
> so we need some way to recognize the rock stars of triage, and reward
> them appropriately.


I've been in IT since it was called ADP/EDP, and remember punched cards, 
  paper tape and typewriter keyboards on large computers.

On occasion I've been hired to write programs, and one of the questions 
much in the mind of managers is how to measure productivity.

A measure in one place I worked was "lines of code." Programmers writing 
more lines of code was held to be more productive than those writing fewer.

This without any regard for complexity, or code quality (another thing 
that's hard to quantify), or consumption of resources (such as _then_ 
very expensive computer time).

A number of LOC I've seen quoted is 100 per day, for a basically capable 
programmer on not too-difficult code, and including documentation (to 
standards I don't recall, but undoubtedly better than that which one 
ordinarily sees in OS code).

At number of LOC was found to be fairly constant over a variety of 
programming languages (but I suspect C wasn't included in the survey), 
so writing in higher-level languages such as COBOL and PL/1, where 
suitable, had obvious benefits in business applications than assembler, 
and interpretive languages such as REXX (which probably didn't exist 
then) produced even greater productivity benefits.

Now, writing computer programs is just one example of problem solving; 
writing (books, stories etc) and triaging are other examples.

I don't think metrics that are easily measured are very useful in 
measuring productivity.

If Will triaged 100 bugs in May, that's (I expect) quite good. However, 
if 20 of them are misassigned or otherwise badly handled, I think I 
would rather he handled 50 bugs and got 48 of them right.

A little while ago I filed a bug report against kernels 2.6.25, my 
DC7700 wouldn't boot because the kernel couldn't find its disks.

The mkinitrd team said, "That's ours," without (I think, but I might be 
defaming them unfairly) looking very closely at it. Had I left it at 
that, it might still be on their list, but I went to some effort to 
prove it wasn't mkinitrd and then changed the assignment.

Misassignment can be quite expensive, in terms of getting problems fixed.

I think that the idea of acknowledging good performance is excellent, 
but that its implementation needs to be done carefully.

To achieve an award, I suggest these need to be considered. Change the 
numbers to suit, I have no idea what absolute numbers are realistic.
1. Activity. People need to be more than casual workers. Maybe disposing 
of 20 bugs in a month. If you don't do at least this amount of work, you 
won't be recognised[1].
2. Accuracy. Allow maybe 5% error rate: one mistake in 20. If you don't 
measure up on this, you are not eligible for recognition (but you might 
attract some attention form a  mentor).
3. Relative performance. Not the absolute best performance, this isn't 
the Olympics, but compared with the worker's previous efforts. If Jon's 
doing more, or making fewer mistakes than before, acknowledge that.
4. Knowing when to ask for advice. Jo says, "I've done this and that, 
and I think .., what do you say?" This is way better than mishandling 
the problem, and makes it into a learning experience. OTOH if Jo's only 
had a quick look and done no analysis of the problem, then asking for 
help is a distraction to others. Better to pass on that one, and go for 
the next.
5. Mentoring. Jeff's been around for a while, maybe doesn't do much 
triaging himself these days, but he's always ready to help people like 
Jo. And he keeps an eye on newcomers, and those prone to making mistakes.

[1] Unless you're new. New workers should be publicly welcomed.


I might be using "mentor" a little loosely; those more experienced 
should be expected to help others (and package maintainers probably 
should be mentoring too). I don't mean a formal mentoring role such as 
Debian has (but it might be a good idea).

I can see that some simple metrics might help, but be careful they arent 
used too mechanistically.



-- 

Cheers
John

-- spambait
1aaaaaaa at coco.merseine.nu  Z1aaaaaaa at coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)




More information about the fedora-test-list mailing list