What is rawhide for?

[Will Woods]

On Mon, 2008-06-16 at 10:16 -0400, John Poelstra wrote:

> 1) An "officially" available version for reproducing bugs or test 
> results only exists for one day--contrast that with our build 
> environment that was created specifically with the intention of always 
> being able to recreate binaries in their original environment.

The difference between rawhide on any two days is, on average, somewhere
between 0.1% and 0.5%. If this *blazing* development pace is too much
for you, you're welcomed and encouraged to keep multiple copies of the
rawhide repos. 

It'd definitely be nice if we had some good scripts for maintaining
multiple local copies of rawhide, with hardlinks to save space. 

> 2) For most people, the economical way to get rawhide is a mirror--there 
> is no simple definitive way to determine that you have "the originally 
> composed version" of rawhide for that day.  There are a few methods that 
> can give you "reasonable certainty", but nothing like a single check sum 
> or a way to confirm that the tree you've downloaded is exactly the same 
> as what was composed.

This is just plain wrong. The timestamp in .treeinfo is unique
per-tree. 

Compare the master mirror:
http://download.fedora.redhat.com/pub/fedora/linux/development/i386/os/.treeinfo
With your local mirror, say:
http://limestone.uoregon.edu/ftp/fedora/development/i386/os/.treeinfo
If the timestamps match, you have the most current rawhide. 

You can also use 'snake' to do this:
[wwoods at metroid ~]$ snake-tree info MIRROR_URL
Name         : Fedora development i386 (20080617)
Arch         : i386
Id           : 1213693723.22
Version      : development
Family       : Fedora
Variant      : 
Time         : 2008-06-17 05:08 EDT
URI's        : 
Images       : i386 (kernel, family, timestamp, variant, boot.iso,
initrd, version, arch), xen (kernel, family, timestamp, variant, initrd,
version, arch)

> 3) There is no "last known good version" when things completely get 
> messed up and you need to reinstall from scratch except the Alpha, Beta, 
> and Preview Release.

This is definitely a problem, and a tricky one at that. It deserves
further discussion in its own mail thread and/or at FUDCon.

> 4) You never know when it will install or not.  It isn't smoke 
> tested--we leave that for *everyone* to do themselves based on their own 
> install attempt.  Even if Fedora hosted an automated smoke test to 
> determine if it installs you still have the problem of mirrors being out 
> of sync and not knowing if what you have is the exact same group of 
> packages that passed the smoke test.

This is the problem that the rawhide dashboard page is supposed to
solve. We're automating SNAKE to do smoke tests on rawhide every night,
and post results on the dashboard. So you *will* know whether it
installs or not.

We've got a prototype rawhide dashboard page that shows whether boot
images are present, at least: 
http://wwoods.fedorapeople.org/rawhide.html

As for the mirrors being out of sync - see above.

> 5) It is the community's only access point for obtaining the (what is 
> close to) the Release Candidate for testing.  I know several people will 
> disagree with me immediately here--we've had the argument several times 
> on IRC.  I still think the underlying assumptions that it is "close 
> enough" and "most likely the same thing" are not good enough.

I think you're wrong. I also think you underestimate the time it
requires to put out an RC versus the turnaround time on testing it. 

Given the amount of time it adds to the schedule and lengthens the
development freeze, plus the amount of work it takes from every part of
the release team, what would we actually *gain* from doing all that
work? 

> 6) We often place a higher value on daily rawhide than the Alpha and 
> Beta releases by proclaiming that they "don't really matter that much 
> because they are 'simply snapshots' of rawhide."  The community at large 
> seems to focus more on the Alpha, Beta, and Preview releases as 
> evidenced by spikes in traffic on fedora-test-list after these releases.

I think you're mixing up cause and effect here. 

People don't focus on the milestones because they just inherently *love*
milestones. They focus on the milestones because we go to the effort of
making them easy to consume, and we publicize them and test them to
ensure they'll be installable and put up ISOs on torrents and the
mirrors. So *obviously* more people use them. 

> 7) We consider rawhide our primary testing target yet there is no 
> practical way to create a test matrix around it because it changes every 
> day.  Instead we create test matrices for the Alpha, Beta, and Preview 
> releases which..... see the previous point.  How do we know when we have 
> completed a full test run?  How can you thoroughly test a moving target?

I completely disagree with the assertion that "there is no practical way
to create a test matrix for Rawhide". 

First of all, the only reason we don't create test matrices for every
day's rawhide is that it's time-consuming to create the matrix itself.
The old wiki's pretty slow, remember?

Better test run tracking (with Testopia, for instance) will make it
trivial to create a matrix for every day's rawhide. Further, automating
our test cases will let us fill in most of that matrix automatically.

The test plan for rawhide is, obviously going to be somewhat less
exhaustive than the test plans for milestone releases or the final
release. So we can complete a "full test run" by looking at the
(mostly-full, auto-created) matrix for that day's rawhide and.. filling
in the blanks.

> that doesn't mean one of Fedora's goals has to be 
> emphasizing a testing process that is flawed and could be better if we 
> all put our collective brains together to come up with something better 
> :-)  We innovate in so many other areas... why not innovate here?

> I would like to advocate that we reconsider the value we place on 
> rawhide and the emphasis we place around the Alpha, Beta, and Preview 
> Release.  

Innovate.. by emphasizing the traditional milestones? I think you'll
find that "innovation" and "tradition" are actually *opposites*. 

Rawhide might be a strange testing target from the point of view of
someone coming from RHEL or proprietary systems or other traditional
milestone-focused development, but I don't think Rawhide is that strange
in the Open Source world. 

You can apply nearly all of your arguments to, say, the way the kernel
is developed - it changes too quickly! Not enough freeze time! We need
more milestones! You can't possibly test it!

You know what would *really* be innovative? Engineering our test efforts
to match the pace and reality of typical Open Source development, rather
than working the other way around.

> I think a good place to start would be document in our test 
> plan where using rawhide for test results makes sense and where it does 
> not.  I believe rawhide does have its place, but I think we are trying 
> to use it to cover too many bases and could do more effective testing 
> with a more refined approach which in the end makes Fedora better!

Okay. So start documenting where you think it does and doesn't make
sense and we'll discuss *that*. But so far all you've done is rehashed a
bunch of complaints and offered no solutions.

> What do other people think?  Is there something here worth throwing 
> around here on this list with a following up discussing at FUDCon later 
> in the week?

Sure, if we go to the effort of actually *defining* problems and
discussing *solutions*, it's totally worth it.

-w
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3153 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-test-list/attachments/20080617/4619d02d/attachment.bin>