F11 Xorg/Mesa/DRI testing proposal

FD Cami fcami at fedoraproject.org
Fri Feb 13 00:38:50 UTC 2009


Hi,

We would like to organize efficient Xorg/Mesa/DRI testing before F11 GA.

Our goal is to record what works as expected on mainstream, supported
hardware with free drivers (nv, i810, radeon, soon nouveau), what does
not, and track regressions from Alpha to GA.

We can't rely on beta-testers alone to find the critical bugs in Xorg
before the GA. However, the Fedora project has a large number of
contributors, probably covering most available mainstream hardware ;
there is currently no way to tap that existing resource, because we
don't know who owns which hardware.

Each of us having limited ressources both in time and hardware at our
disposal, we'd like to :

0. be able to use a opt-in searchable database of hardware belonging to
   FAS account holders ; linking FAS accounts and smolt profiles, at the
   account holder's discretion has been suggested and would work well.
   We may also need to use that database to contact contributors that
   would also have elected to participate in the testing program.

1. be able to write and use test plans efficiently. The wiki is suitable
   for that task.

2. record the results of test runs ; those should be at least searchable
   per application, per driver and per actual hardware ID. Any bugs
   encountered will of course be recorded in Bugzilla, but we also need
   to know which tests were successfully completed. A few solutions have
   already been suggested, including the laptop.org wiki scripts solution
   (see http://wiki.laptop.org/go/Test_cases_8.2.1 for instance), or a
   dedicated application like TestLink or Nitrate (which may not be ready
   for F11).

3. regularly schedule test sessions, using the opt-in system mentioned at
   step #0.

We will need some help from -infra for at least #0 and #2.

Running test cases only requires having the missing hardware, some free
time, and being able to use bugzilla.
Possible test cases are basic 2D tests, for instance x11perf and full
screen movies ; 3D tests, like glxgears and mesa-demos, then GL
regression testing with using piglit.
Desktop effects using Compiz/KDE would be next, then to 3D applications
like Blender, ending with 3D games like GL-117 and FlightGear for
instance.

Obviously, this methodology could be expanded to other components of
Fedora as well. We are specifically thinking of testing for regressions
in the audio subsystem and validating updates in Bodhi.

Feedback, suggestions, flames welcome.

Thanks to adamw, airlied, fenris02, jlaska, lmacken, mcepl and others.

Signed-off-by: François Cami <fcami at fedoraproject.org>




More information about the fedora-test-list mailing list