future f12 test days

[Callum Lerwick]

On Fri, 2009-04-03 at 10:51 -0700, Adam Williamson wrote:
> On Fri, 2009-04-03 at 03:35 -0500, Callum Lerwick wrote:
> 
> > The tests should be automated and run daily, hourly, or even before
> > every SCM commit. Ideally, *nothing* would be allowed to be
> > committed/pushed into the repo that broke the tests. That would also
> > provide motivation to keep the test scripts working...
> > 
> > All reported bugs should be turned into automated test cases if possible
> > and added to the test suite. This ensures once something is fixed it
> > never breaks, ever again. Regressions are not an option.
> 
> If you actually look at the test cases we've been doing, this is not
> practical. How do you automate "you should see a smooth graphical boot
> process that fades into GDM"?

You do low level unit tests of all render paths. Readback from the frame
buffer.

> Or looking at the smoothness of video
> playback?

Yes, some things can't be fully automated, but you can still automate
them to the point where a test suite tells a user what to look for,
displays a test case, and asks the user "Hey did that look right?". Two
clicks and ~10sec per test.

Out of all the tests on radeon test day, only the XV test, DPMS and
multihead tests are the only one's that couldn't be easily automated.

Though with enough money, you could automate those too, with the use of
some cameras and video capture hardware, or for more $$$ high speed
digital sampling hardware... Get a hardware hacker to instrument the
laptop's backlight...

How much is stability worth to Red Hat Inc?

> I like the idea of automated testing, but a lot of stuff - especially
> the really critical stuff to do with hardware - is not amenable to
> automation. Or at least it would be extremely difficult.
> 
> > The lack of automated testing in the project is saddening. However a lot
> > of the problem is hardware. We really need a diverse hardware testing
> > lab. As it is, testing seems to get done on the latest shinyest hardware
> > of the month that the (paid) developers just bought, leaving those of us
> > who don't buy new hardware every month/year in a dustpile of
> > regressions.
> 
> Will Woods and Jesse Keating are working very hard on automated QA
> topics, but as I said, I just don't think automation can ever be the
> answer for some areas.

... Really what I have in mind here is stuff like "Video playback locks
up the machine", "OpenGL locks up the machine", "Second Life hangs the X
server", "World of Warcraft crashes the X server", "This web site
crashes the X server", "Rosegarden crashes the X server". All of which
are entirely automateable, though the "hard locks the machine" cases
will require some sort of hardware watchdog arrangement...

https://bugzilla.redhat.com/show_bug.cgi?id=441665
https://bugzilla.redhat.com/show_bug.cgi?id=474973
https://bugzilla.redhat.com/show_bug.cgi?id=474977
https://bugzilla.redhat.com/show_bug.cgi?id=487432

At any rate, *some* testing is way better than none at all.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/fedora-devel-list/attachments/20090403/41006c17/attachment.sig>