[libvirt-users] Nested KVM: L0 guest produces kernel BUG on wakeup from managed save (while a nested VM is running)

[Kashyap Chamarthy]

On Thu, Feb 08, 2018 at 01:07:33PM +0100, David Hildenbrand wrote:

[...]

> > So to clarify things, could you enumerate the currently known
> > limitations when enabling nesting? I'd be happy to summarize those and
> > add them to the linux-kvm.org FAQ so others are less likely to hit
> > their head on this issue. In particular:
> 
[...] # Snip description of what works in context of migration

> > - Is https://fedoraproject.org/wiki/How_to_enable_nested_virtualization_in_KVM
> > still accurate in that -cpu host (libvirt "host-passthrough") is the
> > strongly recommended configuration for the L2 guest?

That wiki is a bit outdated.  And it is not accurate  — if we can just
expose the Intel 'vmx' (or AMD 'svm') CPU feature flag to the L2 guest,
that should be sufficient.  No need for a full passthrough.

That above document should definitely be modified to add more verbiage
comparing 'host-passthrough' vs. 'host-model' vs. custom CPU.

> > - If so, are there any recommendations for how to configure the L1
> > guest with regard to CPU model?
> 
> You have to indicate the VMX feature to your L1 ("nested hypervisor"),
> that is usually automatically done by using the "host-passthrough" or
> "host-model" value. If you're using a custom CPU model, you have to
> enable it explicitly.
> 
> > 
> > - Is live migration with nested guests _always_ expected to break on
> > all architectures, and if not, which are safe?
> 
> x86 VMX: running nested guests works, migrating nested hypervisors does
> not work
> 
> x86 SVM: running nested guests works, migrating nested hypervisor does
> not work (somebody correct me if I'm wrong)
> 
> s390x: running nested guests works, migrating nested hypervisors works
> 
> power: running nested guests works only via KVM-PR ("trap and emulate").
> migrating nested hypervisors therefore works. But we are not using
> hardware virtualization for L1->L2. (my latest status)
> 
> arm: running nested guests is in the works (my latest status), migration
> is therefore also not possible.

That's a great summary.

> > 
> > - Idem, for savevm/loadvm?
> > 
> 
> savevm/loadvm is not expected to work correctly on an L1 if it is
> running L2 guests. It should work on L2 however.

Yes, that works as intended.

> > - With regard to the problem that Kashyap and I (and Dennis, the
> > kernel.org bugzilla reporter) are describing, is this expected to work
> > any better on AMD CPUs?  (All reports are on Intel)
> 
> No, remeber that they are also still missing migration support of the
> nested SVM state.

Right.  I partly mixed up migration of L1-running-L2 (which doesn't fly
for reasons David already explained) vs. migrating L2 (which works).

> > - Do you expect nested virtualization functionality to be adversely
> > affected by KPTI and/or other Meltdown/Spectre mitigation patches?
> 
> Not an expert on this. I think it should be affected in a similar way as
> ordinary guests :)
> 
> > 
> > Kashyap, can you think of any other limitations that would benefit
> > from improved documentation?
> 
> We should certainly document what I have summaries here properly at a
> central palce!

Yeah, agreed.  Also, when documentation in context of nested, it'd be
useful to explicitly spell out what works or doesn't work at each level
— e.g. L2 can be migrated to a destination L1 just fine; mirating an
L1-running-L2 to a destination L0 will be in dodgy waters for reasons X,
etc.

[...]

-- 
/kashyap