Expectation Management for Test Releases

[Dave Jones]

On Tue, 2004-04-20 at 18:05, Keith Lofstrom wrote:

Hi Keith,

> He said he had big problems running Fedora Core 2 Test 1;  he was not
> mollified when I tried to point out that one can *expect* huge problems
> trying to run test releases.   He insisted that it was entirely
> unprofessional to release software that wouldn't even install, and that
> using the FC2T1 upgrade process on top of a working FC1 system made a
> real hash of it.  He had performed alpha and beta testing on Windows
> and never had such problems. 

I'd put money on there being alpha/beta versions of Windows that he
didn't see that wouldn't have booted on some systems. The point is that
with Linux, you see *every* release, no matter how many rough edges
there may be left. Sure we do everything we can to make it as painless
as possible, but there's always going to be a number of systems that
are problematic.

I personally got quite a few complaints after we screwed up test1 so
that it wouldn't boot on 586's. After telling people 'ok we screwed up'
in bugzilla reports, a majority of personal mail I got as follow ups
were "This is just typical of Red Hat blah blah", "I cant believe you
tried to deprecate 586s" etc. Even though we did nothing of the sort,
and it was a honest mistake, a lot of people went on the offensive and
assumed we did it on purpose.  We do everything we can including testing
rawhide daily to make sure it's at least installable, but we're only
human, and mistakes happen.

> And it is hard to argue about the nature of good code development with
> a person whose career is based on successfully producing some of the
> world's most reliable and mission-critical code.

I find it a little disturbing if they're considering using Fedora
for anything mission critical, but I'll give the benefit of doubt and
assume that he was looking at Fedora for personal reasons rather than
professional reasons.

> Many people, including professional software people heavily oriented
> towards testing, may download this code without understanding what
> they are in for.

It's arguable how 'professional' they are if they don't do some basic
research first to find out what it is they're testing. Can you suggest
better wording that's needed on any specific part of the fedora website?

>   You can read this mail group for months without
> seeing a FAQ;  there are apparently none of the warnings that used to
> come with rawhide.

As well as the warning in the installer, there are numerous reports
here of problems each day. What more could we do?

> We should let potential testers know the huge difference between a
> test release and a general availability release

To me, this is obvious from reading the schedule at 
http://fedora.redhat.com/participate/schedule/

How could we make it any more clear?

> before they start downloads.  This would save
> this list from a lot of carping from people like the Boeing manager,
> and like me.  State the goals.  Be honest about the downside.

http://fedora.redhat.com/about/objectives.html
point 4 is "Provide a robust system..." maybe that needs to be extended
to mention that test releases may eat your hamster.

> That will cause a lot of potential testers to go away, sure.  It might
> also prevent disappointments that result in Linux (and Linux programmers)
> being rejected by major software-using organizations.

Nothing ever works first time, for any distributor, of any software,
on any platform.  Peoples expectations of a piece of software is
something that we have no direct control over, so other than amendments
to documentation, I don't think there's a great deal we can do, other
than continue to do our best to ship _usable_ test releases.

If you've feedback on how to improve wording on the fedora website,
I'm sure the folks who deal with that content will be glad to hear
about it.

> A decade from now, I don't want to ride on an airplane controlled by
> Windows CE-RT, because in 2004 we pissed off the wrong person at Boeing. 

Whilst I'll advocate the usage of Fedora far and wide, I'm not so sure
I'd feel comfortable flying in a plane powered by it 8-)

Thanks,

	Dave