[Libvirt-cim] KVM test report on Fedora 9

[Zhengang Li]

Kaitlin Rupert wrote:

>> AllocationCapabilities - 02_alloccap_gi_errs.py: FAIL
>> "Requested Object could not be found." vs. "Instance not found." error 
>> string issue. The return code is correct.
>> [Known Issue]
> 
> Dan has submitted a patch that sets the changeset number and revision 
> number of the providers as attributes in the VSMS class.
> 
> Could you add a function that gets these values, and then sets them as 
> globals for the suite?  This will allow the individual test cases to use 
> the changeset/revision number as a way to check for different behavior 
> depending on when a particular patch went in.
Ok, I'll add that today.

> 
> Also, there was some discussion on the mailing list about modifying the 
> negative test cases so that they only check the provider return codes. I 
> think it'll be awhile before we can add implementation specific return 
> codes to the providers.  Since the CIM return codes aren't specific 
> enough to indicate exactly what kind of error occurred, I'm inclined to 
> continue checking the return messages in the test cases for now.
> 
> Thoughts?
I agree with you on checking both the return codes and the messages. But 
I thought branching the test cases for different changeset of providers 
is a little risky. I am in the mood that we're going to maintain massive 
if-else branches on this if the provider message strings change too fast.

An optimistic view would be that even though we need to maintain a 
little bit too many of branches at first. But as the providers get more 
stable, these frequent changes are less likely to happen.

Ok, my third view is a little unrealistic. We can develop a fifth test 
case return code, named 'conditional pass', specifically for the rc 
matches, string doesn't match issue. :=)


-- 
- Zhengang