Metrics: RFC

[Christopher Blizzard]

Jason L Tibbitts III wrote:
> I and I'm sure a good portion of Fedora users would happily install
> a package that periodically sent various bits of info related to what
> hardware is in the system, what packages are installed, what kernel is
> currently running, etc.  Such information could be immensely useful to
> community packagers as a validation that their packages are actually
> being used somewhere.

That would be useful.  But I want us to stay focused one one thing to 
start and one of the areas we're really hurting in is good hardware 
support.  That can really have a positive impact on our users, so I 
think it's worth it to try and start there.

> 
> There are all sorts of useful bits of information we could collect,
> most of which require that some client be installed.  I suggest that
> this isn't a problem as long as:
> 
> There's only one thing to install, not a collection of random
> info-gathering clients.

Keeping in mind what we want to measure, I would guess that we have a 
few things here:

1. Something that's run every time you change the kernel that can ask 
users to run through a set of simple tests?  Annoying, but maybe we can 
only run it for 1/20 people and since we're just interested in 
statistical information, that's enough.  Think high level (external 
monitor support, sound, suspend/resume, etc.)

2. A small program that's run during each suspend and resume.  Leaves a 
file behind reporting success/failure and what was running right before 
and after each suspend.  This gets picked up by a program, maybe from 
cron and is submitted.

> 
> The client runs from cron and doesn't consume any memory when it's not
> running.
> 
> It is not installed by default.  The installer can ask, sure.  "Please
> help us improve Fedora!  Do you want to install the blahblah client?".
> "Yes" should probably not be the default choice.  (It could be
> installed by default but not enabled by default, but even that is
> probably pushing it.  It's about the appearance of impropriety.)

Yeah, we'll need to be careful.  But I also want to make sure that we 
talk about really what we're trying to do here (really improve the 
system) as opposed to just find out how many users we have.  The second 
is just a side effect of the first.

> The list of information gathered is presented to the user somehow.  An
> option should be there to provide someone (root?) with a report of
> everything sent in when it's sent in if someone really wants to know.
> Updates to the package which provide new information-gathering plugins
> should not cause those new plugins to be enabled without user
> intervention.
> 
> That's pretty much all the evil-minimization you can reasonably do.
> The question is whether the response rate will be high enough and
> whether the responses you get give you a statistically valid sample.

A sample of even a few thousand systems can give us a sense of what 
hardware people are using.  If we can connect hardware -> drivers -> 
kernel version -> suspend success/fail and higher level success/failure 
metrics (like, does my backlight control work?  How about sound?) we can 
really start to make dave jones' life much easier.  And that's one of my 
real goals.

--Chris