Testing test releases: do not update

[Ernest L. Williams Jr.]

On Thu, 2004-02-26 at 16:59, Mike A. Harris wrote:
> On Tue, 24 Feb 2004, shmuel siegel wrote:
> 
> >Test Release - is it merely a convenient snapshot for installation but
> >serving no useful purpose after that (other than PR)? This is what I
> >would gather from those that advocate that testers stay in sync with
> >Rawhide.
> 
> Test release of an OS snapshot, ie: "Fedora Core 2 test 1", is
> essentially a prebeta snapshot of the OS, which is very likely to
> be unstable.
> 
> In the past, there were private beta testers and approximately 3
> "alpha" releases that were generally not ready for public
> consumption, but needed wider testing than could be done
> internally.
> 
> With the opening up of OS development, and creation of the Fedora 
> Project, it was decided to make _all_ of the beta/alpha/whatever 
> you want to call it releases public and open.  The upside is that 
> there would be more testers this way.  The downside, is that 
> people who voluntarily test the first few initial test releases 
> without realizing that they are playing with fire, are likely to 
> end up with very broken systems, as "test1" is a very first test, 
> and is nowhere near "stable".  The warnings in the installer 
> should not be taken lightly.
> 
> 
> >Rawhide - is it the staging ground for release candidates or is it a
> >communication point between a developer and those that are in contact
> >with him? If it is the latter, then there needs to be another repository
> >that indicates that a package is ready for global testing. If it is the
> >former, than I wonder why an intermediate stage exists in the Core 1
> >tree.
> 
> Rawhide is the current set of packages that were built 
> internally.  The rough process, is this:
> 
> - Developer updates a package to a newer version, or fixes some 
>   bugs, adds patches, whatever.
> 
> - Developer builds package locally on his workstation and does 
>   whatever testing he deems necessary.
> 
> - Developer submits package into the buildsystem, telling it to 
>   build the package into the Fedora Core 2 development tree.
> 
> - Once the package has been successfully built on all 7 
>   architectures, the buildsystem accepts the package into the 
>   internal Fedora Core 2 development tree.
> 
> The above process is how we update all of our packages during 
> development essentially. So what is rawhide then?  Simple.  Once 
> every day or so, a script is ran either automated or manually, 
> which takes all of the latest src.rpm and binary rpms in the 
> current internal Fedora Core development tree, and mirrors them 
> to our ftp staging server.  The staging server then pushes the 
> rpms to the public ftp servers.  This is called "rawhide".
> 
> There is zero QA testing done on any of the rawhide packages, 
> because they are not "production ready", they are "work in 
> progress, fresh off the press, caveat emptor, beware of large 
> dog" or as Jef Spaleta puts it "rawhide might kill babies".
> 
> Do not use rawhide if you can not accept the possibility of total 
> system meltdown and data destruction.  While it does not occur 
> very often, it _CAN_ occur, and it does from time to time.  ;o)
> 
> 
> How does rawhide differ from "test1" or other test releases?  
> Again, that's a simple question with a simple answer.  A 'test' 
> release, or beta, or whatever you want to call it, is a snapshot 
> in time of rawhide.  Essentially, rawhide is snapshotted, and 
> then ISO's are built, and some testing occurs with them.  Any 
> major problems that pop up, we try to fix in hopes that it will 
> be as installable as possible for as many people as possible, 
> without delaying the test release unnecessarily.  In other words, 
> a "test" release is unstable rawhide which might burn your house 
> down, only very slightly better.
> 
> Packages continue to build into our internal development tree 
> after that, which will continue to be mirrored to rawhide daily 
> or so, and will go on eventually to become "test2", etc.  
> Eventually after all test/beta/release candidates, etc. are done, 
> we work up towards the final "gold" OS release which would be 
> "Fedora Core 2".
> 
> That's it in a nutshell, excluding the finer details.
> 
> 
> >I think that Robert Day's initial point is correct. If there is no
> >stable baseline then testers are constantly finding superficial bugs;
> >deep bugs that take hours of testing will never get reached. Alan Cox is
> >also right when he states that a tester should check against the current
> >state of rawhide before he reports a bug.
> 
> By definition, "in development" *means* "there is no stable 
> baseline".
> 
> 
> >I think that a lot of the confusion comes form a lack of a
> >public test plan and the lack of guidelines for testers. Also,
> >for some reason, the difference between internal testing (which
> >in the framework of open source I would consider them to be
> >dedicated testers) and beta testers (those trying to use the
> >features in a real environment) has been totally blurred. Most
> >of the arguments against Robert Day were from the perspective of
> >internal testers. They are right for their function. But most of
> >them didn't need Test 1 except to test Anaconda; they were in
> >sync with rawhide anyway. For beta testers, a stable platform is
> >needed. If they are not at least pretending to do useful work
> >then real life considerations will never be actualized.
> 
> If people require a stable platform, they definitely should not 
> be using rawhide at all, and should not be using any test 
> release.  The entire purpose of the test releases, is that they 
> *are not* stable, and we need people willing to test them anyway, 
> and to report the instabilities to us, so that those 
> instabilities _can_ be fixed.  If nobody ever wants to test 
> anything except "stable" packages, then unstable packages wont 
> get tested, wont get fixed, and the final OS wont be stable 
> either.
> 
> >In summary, I also advocate an intermediate repository whose
> >sole purpose is to keep the baseline usable.
> 
> I disagree, and I oppose the idea of having an additional 
> intermediate tree.  It would do nothing really beneficial, would 
> make the latest software that now goes into rawhide, remain 
> untested for LONGER, and thus remain buggy longer.  It also would 
> put an additional burden upon developers to have to track another 
> tree, and to manually move packages from the "unstable" to the 
> "not quite as unstable" tree from time to time.
Should this not be just another CVS branch for the test release.  Then
bugs should be filed against it.  Updates should be placed in an updates
repository so that testers get updates.  The testers should not have to
go to rawhide for a "released test", right.  You have the technology to
push updates to the rawhide repository as well as a Fedora 2 Test 1
Repository.   Where is all of this difficulty coming from? Isn't this
basic software development practices.

You do indeed describe the Release process.  You have provided great
information.





> 
> 
> >If Redhat is unwilling to do this until development branches to
> >Core 3, then I must assume that Redhat regards final release of
> >Fedora as the real beta.
> 
> Feel free to have that opinion if you wish, just note that your
> opinion is not shared by everyone, and is just that - one
> opinion.  Others will both agree with you, and disagree with you.
> 
> I disagree.
> 
> 
> -- 
> Mike A. Harris     ftp://people.redhat.com/mharris
> OS Systems Engineer - XFree86 maintainer - Red Hat
-- 
Ernest L. Williams Jr. <ernesto at ornl.gov>