[libvirt] How to best handle the reoccurring of rom changes breaking cross version migrations?

[Philipp Hahn]

Hello

Am 03.11.2017 um 08:30 schrieb Christian Ehrhardt:
> On Thu, Nov 2, 2017 at 4:34 PM, Daniel P. Berrange <berrange at redhat.com> wrote:
>>
>> On Thu, Nov 02, 2017 at 04:14:06PM +0100, Christian Ehrhardt wrote:
>>> Ping - since there wasn't any reply so far - any best practices one could
>>> share?
>>>
>>> Let me add a TL;DR:
>>> - bump of ipxe rom versions change the size of virtio-net-pci.rom
>>> - that breaks on migration "Length mismatch"
>>>
>>> I'd guess the size of that rom has to be fixed up on the fly, but if that
>>> is really ok and how/where is the question.
>>
>> The actual ROM contents will be transferred in the migration stream, so
>> the fact that the target host has ROMs with different content is not
>> important. The key thing that matters is that QEMU the target host loads
>> the ROMs at the same location, so that when the ROM contents is overwritten
>> with data from the incoming migration scheme, it all ends up at the same
>> place as it was on the source.
> 
> 
> Thanks Daniel for your answer, although you try to kill my remaining
> hopes of a better solution :-)
> But if the actual ROM content is migrated over anyway "all I'd have to
> do" is to make clear that the newer system (qemu) knows that the
> incoming rom has a different size than it thinks.
> I thought that was what the machine types and their mechanisms were
> for, but so far I only scratched like 10% of those - maybe they don't
> cover these rom sizes?

The problem is that libvirt launches a qemu process without knowing
those details.
Later when you try to restore an old snapshot or try to do a live
migration, the newly spawned qemu process must have the same layout to
be able to load the old VMState.

Qemu has gone through several such changes and is worsened by the fact,
that distributions like Debian use their own ROM files, which might be
different from the sizes shipped with Qemu.
For that reason we (Univention) still ship the old ROM images next to
the new ones and carry the attached patch to switch between them based
on the model. Maybe that works for you, too.

One more warning (we experiences): Even if the sizes are the same, live
migration can still fail afterwards: We have observed several cases,
where a migrated VM carried an old version of SeaBIOS, which fails when
you reboot the VM. We traced that down to Qemu implementing a new
feature (SMM), which was not supported by the old SeaBIOS and then got
it wrong after reboot only.

Hope that helps.

Philipp
-------------- next part --------------
Bug #24702: Fix PXE ROM size mismatch

For pc-0.14 and older use the PXE ROMs from the deprecated etherboot-qemu
package, which have different sizes. Without the patch the virtual PC gets
build with the current ROM sizes. Loading then fails because the stored PCI ROM
BAR size mismatches the configured size.
--- a/hw/pc_piix.c
+++ b/hw/pc_piix.c
@@ -433,6 +433,30 @@ static QEMUMachine pc_machine_v0_15 = {
             .driver   = "virtio-balloon-pci",\
             .property = "event_idx",\
             .value    = "off",\
+        },{\
+            .driver   = "ne2000",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/rtl8029.rom",\
+        },{\
+            .driver   = "rtl8139",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/rtl8139.rom",\
+        },{\
+            .driver   = "e1000",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/e1000-82540em.rom",\
+        },{\
+            .driver   = "pcnet",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/pcnet32.rom",\
+        },{\
+            .driver   = "ne2k-isa",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/ne.rom",\
+        },{\
+            .driver   = "virtio-net-pci",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/virtio-net.rom",\
         }
 
 static QEMUMachine pc_machine_v0_14 = {