[libvirt-users] Multi-threaded virsh migrate

[Daniel P. Berrangé]

On Mon, May 07, 2018 at 11:55:14AM -0400, Shawn Q wrote:
> Hi folk, we are using 10gb NICs with multithreaded compression.
> 
> We're finding that the standard `virsh migrate` gets at most ~1.2gbps,
> similar to a single scp session.

Hmm, I didn't actively measure the throughput when I tested, but looking
at the results and trying to infer bandwidth, I'm fairly sure I saw in
excess of 1.2gbps. With scp I would expect transfer rate to be limited
by the speed your CPUS can perform AES encryption.

Your migration command is just transferring plaintext so should not be
limited in that way and should thus get better results. I wonder if
the compression code is harming you because that burns massive amounts
of CPU time, often for minimal compression benefit.

> When we do a multipart upload with multiple scp connections we can squeeze
> as high as 9.6gbps.
> 
> Is there was a way to get `virsh migrate` to perform multiple connections
> as well when transferring?

There is work underway in QEMU to add this feature, but it will be a
while before it is ready.

> Would be useful to be able to migrate big guests using the full capacity of
> the 10gb nics.
> 
> Our example command to migrate:
> 
> # virsh migrate --compressed --comp-methods mt --comp-mt-level 9
> --comp-mt-threads 16 --comp-mt-dthreads 16 --verbose --live
> --copy-storage-all --undefinesource --unsafe --persistent
> dev-testvm790.mydomain.net qemu+tcp://myhost-10g.mydomain.net/system

I would caution that the compression code should not be assumed to be
beneficial. When I tested compression against extreme guest workloads,
it was actually harmful[1] - it burns a lot of CPU time, which takes
significant time away from the guest OS vCPUs, while not getting very
good compression.

If you have trouble with migration not completing, the best advice
at this time is to use the post-copy migration method. This will
guarantee that migration can complete in finite time, while having
the lower impact on guest vCPUs, assuming you let it run pre-copy
to copy bulk of RAM, before flipping to post-copy.

Regards,
Daniel

[1] https://www.berrange.com/posts/2016/05/12/analysis-of-techniques-for-ensuring-migration-completion-with-kvm/
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|