[libvirt] [PATCH v2 2/8] tests: qemuxml2xml: Always use different output file

[Cole Robinson]

On 01/25/2016 02:05 AM, Martin Kletzander wrote:
> On Fri, Jan 22, 2016 at 02:09:18PM -0500, Cole Robinson wrote:
>> Most qemuxml2xml tests expect that the input XML is unchanged after
>> parsing. This is unlike 99% of new qemu configs in the wild, which after
>> initial parsing end up with stable PCI device addresses. The xml2xml bit
>> doesn't currently hit that code path though, so most XML testing indeed
>> does not change.
>>
>> Future patches will add that PCI address bits, which means most test cases
>> will have different output. So let's do away with the hardcoded same vs
>> different test split, and always track a separate output file. Tests can
>> still have same input and output, it just necessitates 2 separate XML files.
> 
> I don't know whether there was a discussion behind this, if there was, I
> regret missing it. 

There wasn't you, you didn't miss anything :)


I don't like this very much as it's not visible from
> the test if that file should change or not.  It is visible in the files,
> but you have to diff them.  If we really need this, I suggest using
> symlinks to save some space and also time as it will be clearer which
> XMLs are the same and which are not.

I agree that this method is not space efficient, and that it loses some
visibility into testing that XML should not change. And the patch isn't
necessary for this patch series so I can drop it

However the point I made above still stands: nearly all these test files are
going to need to change, because we should be testing qemu device address
assignment unconditionally.

In that future case, we _could_ change all the input to match the output and
maintain the current state... however I don't think that's a good idea, since
that means we will have very little xml2xml testing for the device address My
opinion is that the way we do xml2xml testing right now is not ideal and it
deserves a larger rework, of which this is a step in the right direction.

xml2xml tests should take a few forms:

- testing generic XML is parsed and formatted correctly
- testing generic XML validation failures
- testing driver specific XML handling

Right now the vast majority of the xml2xml testing is handled by qemuxml2xml,
when the tests are not driver specific. I think we should aim to move the
generic XML tests to (the new) genericxml2xml. If we want to keep the explicit
sameness tracking I think that's the place to do it.

That would also be a good opportunity to consolidate a lot of the xml2xml
tests... I figure we could have genericxml2xml-disks.xml and
genericxml2xml-interfaces.xml and rather than drop in a whole new file we just
extend those for new XML testing specific to the device type.

Personally though I don't think we need to enforce the 'sameness' checking too
hard. Any regressions should stick out clearly whether they share the XML or
not: make changes, do VIR_TEST_REGENERATE_OUTPUT=1 make check, then carefully
review the diff is the same pattern either way

Thanks,
Cole