libvirt-6.5.0 breaks host passthrough migration

[Mark Mielke]

Hi all:

Any thoughts on this, or not time enough to look?

I'm trying to decide what the right solution is on my end to deal with this
breaking change, before more widely deploying it. If I know what the fix
will be, I would like to align my strategy against it. Or, if we discuss
and the libvirt team disagrees with my analysis, then I will have to decide
whether to enable some sort of conversion mechanism to safely migrate
machines from libvirt-6.4.0 and before, to libvirt-6.5.0 without losing the
migration capability.


On Sun, Jul 5, 2020 at 12:45 PM Mark Mielke <mark.mielke at gmail.com> wrote:

> Hi all:
>
> With 6.4.0, live migration was working fine with Qemu 5.0. After trying
> out 6.5.0, migration broke with the following error:
>
> libvirt.libvirtError: internal error: unable to execute QEMU command
> 'migrate': State blocked by non-migratable CPU device (invtsc flag)
>
>
> I believe I traced the error back to this commit:
>
> commit 201bd5db639c063862b0c1b1abfab9a9a7c92591
> Author: Jiri Denemark <jdenemar at redhat.com>
> Date:   Tue Jun 2 15:34:07 2020 +0200
>
>     qemu: Fill default value in //cpu/@migratable attribute
>
>     Before QEMU introduced migratable CPU property, "-cpu host" included
> all
>     features that could be enabled on the host, even those which would
> block
>     migration. In other words, the default was equivalent to
> migratable=off.
>     When the migratable property was introduced, the default changed to
>     migratable=on. Let's record the default in domain XML.
>
>     Signed-off-by: Jiri Denemark <jdenemar at redhat.com>
>     Reviewed-by: Michal Privoznik <mprivozn at redhat.com>
>
>
> Before this change, qemu was still being launched with "-cpu host", which
> for any somewhat modern version of qemu, defaults to migratable=on. The
> above comment acknowledges this, however, the implementation chooses the
> pessimistic and ancient (and no longer applicable!) value of migratable=off:
>
> +    if (qemuCaps &&
> +        def->cpu->mode == VIR_CPU_MODE_HOST_PASSTHROUGH &&
> +        !def->cpu->migratable) {
> +        if (virQEMUCapsGet(qemuCaps, QEMU_CAPS_CPU_MIGRATABLE))
> +            def->cpu->migratable = VIR_TRISTATE_SWITCH_ON;
>
> *+        else if (ARCH_IS_X86(def->os.arch))+
>  def->cpu->migratable = VIR_TRISTATE_SWITCH_OFF;*
> +    }
>
> Consequently, after live migration qemu is started with "-cpu
> host,migratable=off" and this enables invtsc, which then prevents migration
> as when invtsc is enabled it activates a migration blocker.
>
> I think this patch should be reverted. If the migratable state cannot be
> determined, then it should not be guessed. It should be left as "absent",
> just as it has been working fine prior to libvirt-6.5.0. Qemu still knows
> what flags are enabled, so Qemu remains the authority on what can be safely
> migrated, and what cannot be. I reverted this patch and re-built, and this
> seems to clear the migration problems. If the user chooses to explicitly
> specify migratable=yes or migratable=no, then this value should be
> preserved and passed through to the Qemu "-cpu host,XXX" option.
>
> I think it is not a requirement for "migratable=XXX" to be explicit in
> libvirt. However, if there is some reason I am unaware of, and it is
> important for libvirt to know, then I think it is important for libvirt to
> find out the authoritative state rather than guessing.
>
> Please start by reverting this patch, so that other people do not get
> broken in the same way, and I don't need to carry my revert patch.
> Personally, I think this is important enough to build a 6.5.1. However,
> since I have a local revert patch in place, I am not waiting for this.
>
> Thanks!
>
> --
> Mark Mielke <mark.mielke at gmail.com>
>
>

-- 
Mark Mielke <mark.mielke at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20200710/8b4d03b5/attachment-0001.htm>