Testing test releases: do not update (was: yum.conf that works)

[Robert P. J. Day]

On Sun, 22 Feb 2004, Axel Thimm wrote:

> On Sat, Feb 21, 2004 at 11:24:16AM -0800, Rick Johnson wrote:
> > You want development for updates. The core/test/1.90/i386/os is for the 
> > initial install. It *seems* that just about every package has been 
> > updated since then. The stock yum.conf that ships uses the devel tree - 
> > and the Red Hat folks here have suggested that you pull updates from there.
> > 
> > I imagine come FC2 Test 2 or 3, they may split the updates to 
> > updates/test/1.91 or something like that in order to provide us with a 
> > more stable testing ground.
> 
> I would think the reason to have a test release is to have a well
> defined point in development time to test against. If you point your
> package resolvers to development/rawhide, then that's what you will be
> testing instead.

exactly.  see below.

> E.g. if you want to have FC2test1 installed and tested, file bug
> reports etc., then do not update. Or if you do, probably you should
> not file bugs against FC2test1 anymore (but FC devel), as the set of
> bugs will already be different.

against my better judgment, i'm going to take one last stab at explaining
why i think the current testing procedure is badly defined, because axel's
post matches nicely some of my concerns.

let's start simple.  as an example of the testing process, red hat
releases a "test" version -- at the moment, that's FC2-t1.  so far, so
good.  and red hat obviously wants folks to download, install and beat on 
that test release and report bugs/errors/issues/whatever.  it all sounds 
so simple, but here's where it starts to break down.

guaranteed, within minutes of release, testers will start to find 
problems.  and i don't mean things to go on a wish list, or request for 
enhancements.  i mean things that are really and truly borked.  so they
wander over to bugzilla and file a report against ... what?  well, 
obviously, against FC2-t1, right?  again, so far, so good.  but what to do 
about features that are truly broken?

the first question is, if testers are claiming to test FC2-t1, should they 
be allowed to change/update their systems in any way?  well, sure, you 
say.  some of those bugs could be serious enough that they get in the way 
of further testing, so of course, red hat will quickly release some 
"updated" RPMs to fix the more serious problems so that testers can get on 
with the business of testing and find even more problems.

but the instant someone applies an updated RPM to their FC2-t1 system, is 
it technically a FC2-t1 system any more?  sure, i'm being pedantic, but 
this is going to become an issue a bit later.  so the question remains, 
once i start applying red hat updates to my system, at what point have i 
changed it enough so that it's no longer really an FC2-t1 system?  would i 
have to start reporting bugs against some other release?  (don't answer 
that yet.)

so, *knowing* that there will be bugs and, consequently, updated RPMs,
where can one quickly and conveniently get those RPMs?  on the one hand,
red hat could make everyone painfully and laboriously update packages
individually, which means 1,000 testers will almost certainly have 1,000
slightly different test systems -- an obvious nightmare.  or red hat could
make everyone's life easy (including their own) and give everyone a single
channel (in yum.conf) against which they could do a simple "yum update".  
the beauty of that is that 1) it's easy for testers to keep up to date,
and 2) it guarantees that everyone who uses that channel is running the
same updated system, so that there's some consistency in bug reports.  
and what single channel would that be?

as it stands, FC2-t1 was shipped with yum.conf pointing at rawhide.  and 
what a bad idea that was.  as more than one person has pointed out, 
rawhide represents the latest, greatest, bleeding edge, 
not-even-guaranteed-to-work software.  why on earth would anyone ship a 
test system which is set up to update against rawhide?  (rick johnson 
above claims that red hat actually *recommends* people update against 
rawhide.  say what?  given that at least one other poster has emphasized 
that stuff in rawhide isn't even guaranteed to work?  i think someone 
needs to get their story straight.  but, onward.)

in order to make the testing process as useful as possible, one would 
think that red hat wants to at least partially control how far someone can 
deviate from the initial FC2-t1 install.  testers should be able to update 
to fix identified and known bugs, that makes sense.  but what's the point 
of updating against rawhide?  it would make far more sense to have just an 
"update" repo for test releases, representing just those packages that 
were found to be broken.  if, instead, you update against rawhide, it 
would seem you've so contaminated your original system, can you even call 
it FC2-t1 anymore?

and for those who think it's too much work for red hat to do it this way,
i submit that it's exactly the opposite.  when there's a test release out 
in the wild, red hat should be spending their time dealing with bug 
reports that represent things that are just flat out *broken*.  not stuff 
from rawhide, not RFEs, not wish lists.  they should be focused on fixing 
stuff that's broken, and that means restricting their attention to just 
those things so that the testers' time is well spent.  and that means not 
having to deal with rawhide.  there should be a *separate* update channel 
purely for fixing broken things.  and, no, i don't care if it takes red 
hat a few more minutes to set this up.  it's not about *them*, it's about 
making the testers' time as productive as possible.  but wait, there's 
more.

on a final note, if you start with FC2-t1, it makes sense that you'll 
report bugs against the release at bugzilla.  but once you start updating 
your test system, do you still report against FC2-t1?  i would think so, 
but if you go to bugzilla, you'll notice that, in addition to the standard
FC1, FC2, test1 and so on, there a fedora core "devel" version.  what the 
heck is "devel" in terms of a version?  when would you ever file or query 
against "fedora core devel"?

anyway, what would be nice is if there was a real and comprehensive 
document for testers, explaining how releases work, how to query bugs, how 
to report bugs and so on.  and by "comprehensive", i mean something a 
little more meaty than "file bugs at bugzilla against your current 
release."  

i'm convinced that the testing procedure deserves an update repo, and i'm 
just as convinced that that repo shouldn't be rawhide.

thoughts?

rday

p.s.  i'm just a bit amused by rick johnson's statement that,

> > I imagine come FC2 Test 2 or 3, they may split the updates to
> > updates/test/1.91 or something like that in order to provide us with a
> > more stable testing ground.

i could have *sworn* that this is the very suggestion i made recently, and 
it was thoroughly shot down by someone who claimed that it was just 
waaaaaay too much work for red hat to implement.  apparently not.

once again, some people should get their stories straight.