automated regression testing, was Re: Desktop SIG meeting today

[Douglas McClendon]

Harald Hoyer wrote:
> Jeremy Katz wrote:
>> On Wed, 2007-09-05 at 10:39 -0400, Matthias Clasen wrote:
>>> desktop spin status (from booting a spin built with yesterdays rawhide)
>>> • udev slow
>>
>> I've noticed this too, but haven't gotten around to investigating what
>> in our rules is slow.
> 
> 
> Hmm, I do not experience that... maybe some rules from another package?

I noticed that F7-livecd took about 3 minutes to fully boot on my system 
under qemu (with kqemu).  And that ubuntu 7.04 took maybe 20 seconds 
longer than that.  But that f8t1 took *15* minutes.  Specifically, udev 
took 20 seconds (reliably +/-3s) on f7-livecd, but f8t1 udev took 
~65seconds.

I also recall that f8t1 seemed equally bad on native hardware, though I 
didn't do as detailed a test.

This seems like a great time to harp on an idea I've been advocating on 
fedora-livecd-list for years now-  Fully automated regression testing 
using qemu.

Think about it- every night, a rawhide livecd is built.  One that has a 
simple (or complex) automated regression test added to it.  I'm thinking 
a great start would be - gdm-autologin, firefox opening local release 
notes, and then a shell script which runs vmstat waiting for all io to 
settle, followed by a screenshot and a shutdown.

Basically what you would have would be this headless dedicated 
regression build server, posting the timing results of the test, and the 
screenshot every night.

Then you could immediately see when something checked into rawhide 
starts seriously impacting system performance (in a good, or more likely 
bad way).

The ways to add more complexity to the tests, and the automated 
verification of the tests, are endless and endlessly valuable IMHO.

<steps off soap box>

-dmc

P.S.- yes, I'll perhaps get around to it eventually, but at this rate 
with my todo list and other priorities, not for quite some time.