[libvirt] QEMU migration with non-shared storage

[Michael Chapman]

Hello,

I am trying to understand libvirt's logic for checking whether migration 
of a VM is safe, and how it determines which disks should be mirrored by 
QEMU. My particular use case involves VMs that may have disks backed onto 
LVM or onto Ceph RBD, or both.

As far as I can tell, the qemuMigrationIsSafe check is there to ensure 
that all disks are readonly, or have cache=none, or their backends can 
guarantee cache coherence. As far as I can tell, however, QEMU flushes 
*all* block devices when it pauses a VM's CPUs (just before the final part 
of migration, for instance), so I'm wondering why this check is needed. Is 
there any possible situation for the source VM to be paused, for its block 
devices to be flushed, and yet the destination VM can't see all completed 
writes?

Why is RBD is handled specially in this function? The current logic is 
that an RBD-backed disk is safe to be migrated even if it's got caching 
enabled, but I'm not sure how RBD is different from other backends in this 
regard.

If VIR_MIGRATE_NON_SHARED_DISK or _INC is specified, should these safety 
checks be relaxed? It seems to me that if any non-shared disk is going to 
be *explicitly* copied from the source to the destination VM, then cache 
coherence in the backend is irrelevant.

At the moment, the set of non-shared block devices copied by 
VIR_MIGRATE_NON_SHARED_* differs depending on whether NBD is being used in 
the migration:

- If NBD can't be used (e.g. with a tunnelled migration), then QEMU will 
copy *all* non-readonly block devices;
- If NBD is being used, then QEMU will only mirror "shareable", "readonly" 
or "sourceless" disks.

A problem arises with RBD disks that have caching enabled. According to 
qemuMigrationIsSafe, these disks are "safe" to be migrated. However in 
either the NBD or the non-NBD case, the RBD disk will be copied. This is 
clearly not desirable. If RBD is a special case in qemuMigrationIsSafe, 
does it also need to be a special case when configuring the NBD server? 
Or, if an NBD server is not going to be used, should the migration be 
considered "unsafe" if an RBD disk is present?

I'd very much appreciate some help in understanding all of this. At the 
moment, I think my only option is to run RBD without caching at all. 
However, not only does that result in very poor performance, it also 
doesn't seem to match the qemuMigrationIsSafe check.

Regards,
Michael