[redhat-lspp] LSPP/RBACPP requirements v.002

[Stephen Smalley]

On Thu, 2005-09-29 at 16:45 -0500, Joy Latten wrote:
> To run or debug individual testcases, the ltp/runtest directory contains
> the file, "selinux" which contains the testcases to run. You can save
> this file to another name and then modify it to contain only the test
> you need to run or debug. Also, add "set -x" to shell script  you want
> to debug. If I recall correctly, the debug info will be printed to one
> of the results/selinux* files.

Just to note:  I don't mean to criticize the port of the selinux
testsuite to the LTP; we are glad to see it as part of the LTP, but I'm
just concerned that in its current form, it might not be conducive to
getting people to run it and extend it.

Having to modify files in order to re-run just a single testcase or to
enable more verbose error reporting is painful.  That seems like a
general problem of the LTP itself, not just the selinux tests in it.

Other things that I found painful with the LTP-based testsuite included:
a) for building, you had to first build the rest of LTP, which included
a lot of unnecessary baggage, and you had to separately build the
selinux tests, and you had to "install" the tests (versus just a simple
cd policy && make load; cd ../tests && make all in the original
testsuite).  As above, this seems like a general problem of the LTP
itself.

b) the running of the tests in the LTP included the test policy load and
unload every time (versus the original testsuite, where you could load
the test policy as part of the setup once, and then could run the tests
repeatedly by themselves via make test without reloading policy at all,
and then could revert the policy when you were all done or optionally
just leave it in place for future testing).

c) the reporting of the tests in the LTP was done to three log files
that had to be separately inspected to determine the final result, and
if you wanted to track down which tests actually failed, you had to
check one log file to get the SELinuxNN name and then go look up which
test directory that means in another file (versus the original
testsuite, where the basic success/fail per test with meaningful names
and summary was written to stdout, which one could always redirect to a
file if desired).
 
d) given the above issues, developing new tests seemed much more
cumbersome than previously.

> There is the alternative of keeping the tests in LTP, but removing the
> test harness and perhaps porting them back to perl. I think this may
> make them easier to run, but I do not know if this will 
> get folks enthusiastic about writing testcases. :-)  

We could retain the ltp-based harness for running as part of the ltp,
but also provide support within the selinux-testsuite directory to build
and run it standalone without the rest of the ltp, as suggested by
Serge.  I don't know how hard it would be to do that while re-using the
existing scripts you've created in those subdirectories so that we don't
have to maintain separate scripts.  If we can address the issues above,
I don't particularly care about using perl Test as the harness.

-- 
Stephen Smalley
National Security Agency