[Avocado-devel] import junit output from external test tool

[Lukáš Doktor]

Dne 12. 10. 18 v 17:58 Brian J. Murrell napsal(a):
> If I run an external test tool with process.system(), say, and that
> test tool produces it's own junit output, is there any way to have
> avocado consume that and make it available in it's own job-
> results/*/results.xml file rather than just the simplified "pass/fail"
> result that it puts in there for a simple test?
> 
> Cheers,
> b.
> 

Hello Brian,

we've had these many times and unfortunately there is no solution so far. We've had suggestions for allowing sub-test results (one test reporting many stages) that could be used for that but it was rejected for simplicity reasons (one job, many tests, no subtests)

We've had import-tap discussion as well, the problem is test should not grow. Once you define a job, it's given, that is the current definition. Reporting multiple results imported from tap/xml breaks this definition as test can't report multiple test results. It'd be a level violation.

All of those discussions always finished with job-api discussion, which is the way we intend to take, but was postponed due to avocado-qemu work. But we intend to pursue this once we have more resources (external contributions are also welcome, see the job-api RFCs on mailing list). The way it would work is one would define a job, triggered it's suite that reports xml/tap and inject those into the job results. That is not a level violation, because one creates the job.

Anyway there is a "workaround" I'm actually using in my CI for reporting failed libguestfs results. I run the `make check` prior Avocado execution, then process the results and create fake tests executing "/bin/false ...":

            # Now let's run the installcheck
            make installcheck LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 && LIBGUESTFS_TESTS+=("/bin/true libguestfs_testsuite_status") || LIBGUESTFS_TESTS+=("/bin/false libguestfs_testsuite_status")
            # Populate LIBGUESTFS_TESTS with status received from logs
            for match in $(find -name test-suite.log | xargs grep "^FAIL:" | tr ' ' '_'); do
                LIBGUESTFS_TESTS+=("'/bin/false libguestfs_testsuite_$match'")
            done
            cd ..
            mkdir -p "libguestfs_testsuite_logs"
            for log in $(find libguestfs -iname test-suite.log); do
                mkdir -p $(dirname "libguestfs_testsuite_logs/$log")
                cp "$log" "libguestfs_testsuite_logs/$log"
            done
            # Run the tests
            avocado run --xunit "$WORKSPACE/results.xml" --job-results-dir "$WORKSPACE" --loaders file external:"/bin/sh -c" --mux-inject /run:timeout:7200 -- "${{LIBGUESTFS_TESTS[@]}}" "$(which libguestfs-test-tool)"  ...

Is it nice? No. Does it work? Yes. And in a way it's similar to job-api approach only it's not part of the job execution.

Regards,
Lukáš

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20181030/d5aea613/attachment.sig>