Add option to skip cpu feature and model verification in libvirt and rely on qemu 'enforce' option

[manish.mishra]

Hi Jiri, Thank you so much for response.

On 18/11/22 3:22 pm, Jiri Denemark wrote:
> Hi.
>
> On Wed, Nov 16, 2022 at 22:53:18 +0530, manish.mishra wrote:
>> We had a requirement to skip all CPU feature validations from libvirt
>> and totally rely on Qemu for following reasons.
> You are mixing two different things here. The "enforce" CPU checking for
> making sure a guest gets exactly what the user asked QEMU for and
> missing CPU models or features support in libvirt.


I actually meant, if we do checks in libvirt then if some cpu features or models are missing in libvirt we can not use those. 'enforce' was just fallback for checks in that case as atleast someone need to verify if what we passed is taken so that live migrations does not have issues.

>>    * As libvirt always lies behind in term of cpu feature management
>>    compare to qemu, so we may not be able to use all of the latest
>>    support provided by qemu till it is updated in libvirt too. For
>>    example like if new cpu features are added in qemu but not yet
>>    updated in libvirt.
> So you effectively ask for a "passthrough" configuration, that is, CPU
> model and features would be passed directly to QEMU without libvirt even
> looking at them, right?


Yes but we want live migrations too.

>
> Anyway, the current goal is to make libvirt ask for supported CPU models
> including their definition so that we don't have to keep our own
> definitions. That is, libvirt would support all models as they are added
> to QEMU without any additional work. Not sure if CPU features would
> follow, but adding them is fairly easy and what's more important they
> don't change in time (unlike CPU models).
>

Jiri, you mean libvirt will ask qemu for all supported models and features instead of maintaining its own list? Is it there in libvirt currently? We also had similar idea in one of our KVM forum talk but thought it is little long task so may take some time, till then we thought we can have this workaround.

>>    * Libvirt checks if a features is supported or not based on cpuid, a
>>    feature supported by host may not be supported by qemu or there can
>>    be some features which are emulated by Qemu so libvirt level
>>    checking does not make sense in those cases.
> This has not been the case for several years already. Libvirt asks QEMU
> what CPU features it can provide on the host (either natively or
> emulated) and checks against them. So no issue here. Well, expect that
> some features are strange and QEMU does not report them as enabled for
> -cpu host even though it would enable them for some specific CPU models.
> But that's more in a "bug" area which needs to be solved between QEMU
> and libvirt rather than being it a principal issue.


You mean with check =='full', i never tried that but wondered how it works for cpu feature strings which are not defined in libvirt, because if some feature is not defined in libvirt we fail in basic checks like virCPUValidateFeatures. Currently also i agree libvirt asks Qemu for CPU features but i thought that is just for domcapability output or qemuCapability cache file but it was not used in checks.

>>    * Qemu already provide an option 'enforce' to validate if features
>>    with which vm is started is exactly same as one provided and nothing
>>    is silently dropped.
> Right, but it's not enough. In addition to removed features libvirt also
> checks for unexpectedly added features. And you really need to do both.
> Because if you ask for -cpu Model,feat1=on,feat2=on,enforce and QEMU
> says everything is fine, the guest might see more than what you asked.
> For example, if a feature is enabled only if a host supports it you may
> or may not get it without any complains from QEMU. But if you get it you
> really need to explicitly ask for it during migration, otherwise the
> feature can just silently disappear. Of course, this would be a really
> bad behavior from QEMU, but that does not mean it can't happen (I think
> SVM is a bit problematic in this way) and the whole point of libvirt's
> checks is to prevent this kind of issues.
>
> So once QEMU starts we checks all features enabled in the vCPU and note
> those that were (unexpectedly) added so that we can explicitly ask for
> them during migration and also check that no other features were added
> on top. And for this whole process to work, we need to understand, i.e.,
> have explicit support, for all CPU features (QEMU report a lot of CPU
> properties and we need to know which one needs to be stored as enabled
> features) and CPU models to make sure both sides of migration understand
> the CPU definition in domain XML in the same way.


Our assumption was that qemu-enforce will make sure live migrations are safe as nothing can be silently dropped with enforce and it will be user's resposnsibilty to always pass suported features. But i was not aware of this, that Qemu can drop some features silently even with enforce, Jiri can you please point us to some of the example for this. How libvirt handles these cases as of now with check == 'full' ? Then again i had same doubt does it work fine for features which are defined in qemu but not in libvirt?

>> Basically we want another check option like 'qemu-enforce' which will
>> skip all feature and cpu model verfications in libvirt and pass
>> 'enforce' option to qemu. It will work similar to check 'none' i mean
>> no check to verify if provide features are supported by host, also on
>> top of that skip things like virCPUValidateFeatures where ever
>> required.
> In short, I don't think this is a good idea and we should or could
> support it and still maintain migration safety. If you don't care about
> migration safety, you don't need to care about "enforce" QEMU check
> either and can just use host-passthrough and get whatever QEMU supports
> on the host.
>
> Jirka


Thanks

Manish Mishra