[redhat-lspp] LSPP/RBACPP requirements v.002

[Joy Latten]

On Fri, 2005-09-30 at 10:01 -0400, Stephen Smalley wrote:
> On Thu, 2005-09-29 at 16:45 -0500, Joy Latten wrote:
> > To run or debug individual testcases, the ltp/runtest directory contains
> > the file, "selinux" which contains the testcases to run. You can save
> > this file to another name and then modify it to contain only the test
> > you need to run or debug. Also, add "set -x" to shell script  you want
> > to debug. If I recall correctly, the debug info will be printed to one
> > of the results/selinux* files.
> 
> Just to note:  I don't mean to criticize the port of the selinux
> testsuite to the LTP; we are glad to see it as part of the LTP, but I'm
> just concerned that in its current form, it might not be conducive to
> getting people to run it and extend it.
> 
> Having to modify files in order to re-run just a single testcase or to
> enable more verbose error reporting is painful.  That seems like a
> general problem of the LTP itself, not just the selinux tests in it.
> 
> Other things that I found painful with the LTP-based testsuite included:
> a) for building, you had to first build the rest of LTP, which included
> a lot of unnecessary baggage, and you had to separately build the
> selinux tests, and you had to "install" the tests (versus just a simple
> cd policy && make load; cd ../tests && make all in the original
> testsuite).  As above, this seems like a general problem of the LTP
> itself.
> 
> b) the running of the tests in the LTP included the test policy load and
> unload every time (versus the original testsuite, where you could load
> the test policy as part of the setup once, and then could run the tests
> repeatedly by themselves via make test without reloading policy at all,
> and then could revert the policy when you were all done or optionally
> just leave it in place for future testing).
> 
> c) the reporting of the tests in the LTP was done to three log files
> that had to be separately inspected to determine the final result, and
> if you wanted to track down which tests actually failed, you had to
> check one log file to get the SELinuxNN name and then go look up which
> test directory that means in another file (versus the original
> testsuite, where the basic success/fail per test with meaningful names
> and summary was written to stdout, which one could always redirect to a
> file if desired).
>  
> d) given the above issues, developing new tests seemed much more
> cumbersome than previously.
> 
> > There is the alternative of keeping the tests in LTP, but removing the
> > test harness and perhaps porting them back to perl. I think this may
> > make them easier to run, but I do not know if this will 
> > get folks enthusiastic about writing testcases. :-)  
> 
> We could retain the ltp-based harness for running as part of the ltp,
> but also provide support within the selinux-testsuite directory to build
> and run it standalone without the rest of the ltp, as suggested by
> Serge.  I don't know how hard it would be to do that while re-using the
> existing scripts you've created in those subdirectories so that we don't
> have to maintain separate scripts.  If we can address the issues above,
> I don't particularly care about using perl Test as the harness.
> 
Yes, that would be good. I will look into how we could do this when I
get some time.

Regards,
Joy