[libvirt] libvirt TCK wrapper for autotest review

[Guannan Ren]

     Hi Lucas,

          Thanks for your these good modifications.
          There is one place I noticed where you output each testcase of 
"*.t" into a separate file with .tap extension.
          hence, it has a corresponding log file with little content for 
each testcase. it seem a little harder to check compared
          to just one log file.
          The rest of them is perfect for me.

     Guannan Ren


On 02/23/2012 04:45 AM, Lucas Meneghel Rodrigues wrote:
> Hi guys,
>
> I was here looking at the autotest wrapper for libvirt TCK and then 
> decided to work on it, as I had the review fresh on my mind. Things 
> that I've worked on:
>
> * Fixed some download links, that were already sent to upstream tck 
> and applied (thanks Dan Berrange)
> * Instead of making all tests output to the same DEBUG log, make them 
> output to separate .tap files on the results directory
> * Run all tests available for a given item, rather than stopping the 
> test on the first failure
> * removed capitalization on the wrapper name, since it's project policy
> * Use os.environ, and some features of the subcommand execution API to 
> execute the tests
> * Remove usages of error.JobError, as the problems there are more 
> error.TestError, since they are restricted to the libvirt_tck test, 
> not the entire job (in autotest, a job can do more stuff than just a 
> sequence of job.runtest() calls).
> * Made the error messages more descriptive, with info of all failed tests
>
> So, the current output of the tests is like this:
>
> $ sudo client/bin/autotest run libvirt_tck
> 18:29:27 INFO | Writing results to 
> /home/lmr/Code/autotest.lmr/client/results/default
> 18:29:27 INFO | START    ----    ----    timestamp=1329942567    
> localtime=Feb 22 18:29:27
> 18:29:27 INFO |     START    libvirt_tck.domain    libvirt_tck.domain 
> timestamp=1329942567    localtime=Feb 22 18:29:27
> 18:30:19 ERROR| child process failed
> 18:30:19 INFO |         FAIL    libvirt_tck.domain    
> libvirt_tck.domain timestamp=1329942619    localtime=Feb 22 
> 18:30:19    FAIL: ['120-disks-stats.t', '205-disk-hotplug-ordering.t']
> 18:30:19 INFO |     END FAIL    libvirt_tck.domain    
> libvirt_tck.domain timestamp=1329942619    localtime=Feb 22 18:30:19
> 18:30:19 INFO |     START    libvirt_tck.hooks    libvirt_tck.hooks 
> timestamp=1329942619    localtime=Feb 22 18:30:19
> 18:30:19 ERROR| child process failed
> 18:30:19 INFO |         FAIL    libvirt_tck.hooks    libvirt_tck.hooks 
> timestamp=1329942619    localtime=Feb 22 18:30:19    FAIL: 
> ['051-daemon-hook.t', '052-domain-hook.t']
> 18:30:19 INFO |     END FAIL    libvirt_tck.hooks    libvirt_tck.hooks 
> timestamp=1329942619    localtime=Feb 22 18:30:19
> 18:30:19 INFO |     START    libvirt_tck.networks    
> libvirt_tck.networks timestamp=1329942619    localtime=Feb 22 18:30:19
> 18:30:28 INFO |         GOOD    libvirt_tck.networks    
> libvirt_tck.networks timestamp=1329942628    localtime=Feb 22 
> 18:30:28    completed successfully
> 18:30:28 INFO |     END GOOD    libvirt_tck.networks    
> libvirt_tck.networks timestamp=1329942628    localtime=Feb 22 18:30:28
> 18:30:28 INFO |     START    libvirt_tck.nwfilter    
> libvirt_tck.nwfilter timestamp=1329942628    localtime=Feb 22 18:30:28
> 18:30:32 ERROR| child process failed
> 18:30:32 INFO |         FAIL    libvirt_tck.nwfilter    
> libvirt_tck.nwfilter timestamp=1329942632    localtime=Feb 22 
> 18:30:32    FAIL: ['090-install-image.t', '100-ping-still-working.t', 
> '210-no-mac-spoofing.t', '220-no-ip-spoofing.t', 
> '230-no-mac-broadcast.t', '240-no-arp-spoofing.t', '300-vsitype.t']
> 18:30:32 INFO |     END FAIL    libvirt_tck.nwfilter    
> libvirt_tck.nwfilter timestamp=1329942632    localtime=Feb 22 18:30:32
> 18:30:32 INFO |     START    libvirt_tck.qemu    libvirt_tck.qemu 
> timestamp=1329942632    localtime=Feb 22 18:30:32
> 18:30:40 ERROR| child process failed
> 18:30:40 INFO |         FAIL    libvirt_tck.qemu    libvirt_tck.qemu 
> timestamp=1329942640    localtime=Feb 22 18:30:40    FAIL: 
> ['205-qcow2-double-backing-file.t']
> 18:30:40 INFO |     END FAIL    libvirt_tck.qemu    libvirt_tck.qemu 
> timestamp=1329942640    localtime=Feb 22 18:30:40
> 18:30:40 INFO |     START    libvirt_tck.selinux    
> libvirt_tck.selinux timestamp=1329942640    localtime=Feb 22 18:30:40
> 18:30:49 ERROR| child process failed
> 18:30:49 INFO |         FAIL    libvirt_tck.selinux    
> libvirt_tck.selinux timestamp=1329942649    localtime=Feb 22 
> 18:30:49    FAIL: ['055-dynamic-base-label.t', '100-static-relabel-no.t']
> 18:30:49 INFO |     END FAIL    libvirt_tck.selinux    
> libvirt_tck.selinux timestamp=1329942649    localtime=Feb 22 18:30:49
> 18:30:49 INFO |     START    libvirt_tck.storage    
> libvirt_tck.storage timestamp=1329942649    localtime=Feb 22 18:30:49
> 18:31:24 INFO |         GOOD    libvirt_tck.storage    
> libvirt_tck.storage timestamp=1329942684    localtime=Feb 22 
> 18:31:24    completed successfully
> 18:31:24 INFO |     END GOOD    libvirt_tck.storage    
> libvirt_tck.storage timestamp=1329942684    localtime=Feb 22 18:31:24
> 18:31:24 INFO | END GOOD    ----    ----    timestamp=1329942684    
> localtime=Feb 22 18:31:24
>
>