[libvirt] [PATCH v3 01/12] tests: Add some tests for PCI controller options

[Andrea Bolognani]

On Fri, 2018-02-23 at 14:18 -0500, Laine Stump wrote:
> On 02/21/2018 09:14 AM, Andrea Bolognani wrote:
> > The input configurations set all existing options for all PCI
> > controllers, even those that are not valid for the controller.
> > As we implement validation for PCI controller options, we expect
> > these test to start failing.
> 
> A noble cause, but since multiple options are being tested for multiple
> controllers in the same file, once you have all the proper checks in
> place the tests won't actually be verifying all of the negative tests -
> only the first failure will be noticed - if one of the others is missed,
> it won't "fail extra hard" or anything.

I'm well aware of that.

> Although I hate to explode the number of tests, I think if you want to
> have proper negative testing for every options that doesn't belong on a
> particular controller, then you'll need a separate test case for each
> combination of option and controller model. And since that would make
> for a *lot* of test cases if we tried to extrapolate to all other
> options for all other elements, I don't know that it's worth going down
> that rabbit hole.

So should I just drop this one, or is it still somewhat valuable
to have any sort of test suite coverage for PCI controller options?

-- 
Andrea Bolognani / Red Hat / Virtualization