libvirt-6.5.0 breaks host passthrough migration

[Mark Mielke]

Hi all:

With 6.4.0, live migration was working fine with Qemu 5.0. After trying out
6.5.0, migration broke with the following error:

libvirt.libvirtError: internal error: unable to execute QEMU command
'migrate': State blocked by non-migratable CPU device (invtsc flag)


I believe I traced the error back to this commit:

commit 201bd5db639c063862b0c1b1abfab9a9a7c92591
Author: Jiri Denemark <jdenemar at redhat.com>
Date:   Tue Jun 2 15:34:07 2020 +0200

    qemu: Fill default value in //cpu/@migratable attribute

    Before QEMU introduced migratable CPU property, "-cpu host" included all
    features that could be enabled on the host, even those which would block
    migration. In other words, the default was equivalent to migratable=off.
    When the migratable property was introduced, the default changed to
    migratable=on. Let's record the default in domain XML.

    Signed-off-by: Jiri Denemark <jdenemar at redhat.com>
    Reviewed-by: Michal Privoznik <mprivozn at redhat.com>


Before this change, qemu was still being launched with "-cpu host", which
for any somewhat modern version of qemu, defaults to migratable=on. The
above comment acknowledges this, however, the implementation chooses the
pessimistic and ancient (and no longer applicable!) value of migratable=off:

+    if (qemuCaps &&
+        def->cpu->mode == VIR_CPU_MODE_HOST_PASSTHROUGH &&
+        !def->cpu->migratable) {
+        if (virQEMUCapsGet(qemuCaps, QEMU_CAPS_CPU_MIGRATABLE))
+            def->cpu->migratable = VIR_TRISTATE_SWITCH_ON;

*+        else if (ARCH_IS_X86(def->os.arch))+
 def->cpu->migratable = VIR_TRISTATE_SWITCH_OFF;*
+    }

Consequently, after live migration qemu is started with "-cpu
host,migratable=off" and this enables invtsc, which then prevents migration
as when invtsc is enabled it activates a migration blocker.

I think this patch should be reverted. If the migratable state cannot be
determined, then it should not be guessed. It should be left as "absent",
just as it has been working fine prior to libvirt-6.5.0. Qemu still knows
what flags are enabled, so Qemu remains the authority on what can be safely
migrated, and what cannot be. I reverted this patch and re-built, and this
seems to clear the migration problems. If the user chooses to explicitly
specify migratable=yes or migratable=no, then this value should be
preserved and passed through to the Qemu "-cpu host,XXX" option.

I think it is not a requirement for "migratable=XXX" to be explicit in
libvirt. However, if there is some reason I am unaware of, and it is
important for libvirt to know, then I think it is important for libvirt to
find out the authoritative state rather than guessing.

Please start by reverting this patch, so that other people do not get
broken in the same way, and I don't need to carry my revert patch.
Personally, I think this is important enough to build a 6.5.1. However,
since I have a local revert patch in place, I am not waiting for this.

Thanks!

-- 
Mark Mielke <mark.mielke at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20200705/a63c8f42/attachment-0001.htm>