future f12 test days

[Callum Lerwick]

On Thu, 2009-04-02 at 11:45 -0700, Adam Williamson wrote:
> On Thu, 2009-04-02 at 11:11 -0700, Adam Williamson wrote:
> 
> > > c) make the test cases as automated as possible. Offer sending smolt
> > > profile (smoltGUI) and run test cases directly from the desktop. Also We
> > > should add a link (or the document itself) about how to test what and
> > > why to the desktop. If it's possible for the test generated results
> > > should also be posted automagically to the results page, or, if tests
> > > fail, bug reports could be created by bugbuddy.
> > 
> > These would all be nice, but we *are* running on a one-week cycle here,
> > it may be hard to get it all done. Assistance welcome. =) it's pretty
> > hard to automate some elements of some test cases, but some could
> > certainly be done with scripts etc.
> 
> Oh, I forgot to mention another consideration here: I'm trying to write
> these test cases to be pretty future-proof, so we could use them
> virtually unmodified for F12, F13, F14...after all, the URLs don't have
> expire-by dates and the test cases will still be hanging around in 10
> years most likely. I suspect scripts are liable to go stale faster than
> instructions.

The tests should be automated and run daily, hourly, or even before
every SCM commit. Ideally, *nothing* would be allowed to be
committed/pushed into the repo that broke the tests. That would also
provide motivation to keep the test scripts working...

All reported bugs should be turned into automated test cases if possible
and added to the test suite. This ensures once something is fixed it
never breaks, ever again. Regressions are not an option.

The lack of automated testing in the project is saddening. However a lot
of the problem is hardware. We really need a diverse hardware testing
lab. As it is, testing seems to get done on the latest shinyest hardware
of the month that the (paid) developers just bought, leaving those of us
who don't buy new hardware every month/year in a dustpile of
regressions.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/fedora-devel-list/attachments/20090403/1c00e6bd/attachment.sig>