[libvirt] Question about parallel migration connections

[Daniel P. Berrangé]

On Fri, Jan 10, 2020 at 02:22:00PM +0000, Jim Fehlig wrote:
> On 1/10/20 2:26 AM, Daniel P. Berrangé wrote:
> > On Thu, Jan 09, 2020 at 11:04:19PM +0000, Jim Fehlig wrote:
> >> Are they supported with tunneled migration? The feature seems limited to native
> >> migration, in which case I can send a patch prohibiting parallel migration
> >> connections with the tunnel.
> > 
> > Native migration should be preferred over tunneled migration these days.
> > The tunneled migration feature was primarily a workaround for the lack
> > of TLS support in QEMU, in order to leverage libvirtd's TLS connection.
> 
> Tunneled serves the same purpose in the xen driver.
> 
> > QEMU has support for TLS directly in its native migration protocol these
> > days. That should be preserved as it provides a better performing data
> > channel than tunnelling. This will especially be seen with parallel
> > migration. Even if libvirt enabled parallel migration with tunnelling,
> > libvirtd does all I/O in a single thread, so you wouldn't see any
> > performance benefit from it, especially when TLS is used.  This is
> > actually true whether you've got a single QEMU with multiple TCP
> > connections for migration, or multiple QEMU's migrating concurrently.
> > Both situations will be limited by libvirt's single thread for I/O.
> 
> Nod.
> 
> > With QEMU's native TLS support and parallell migration you'll be able
> > to max out performance of many CPUs to maximise data throughput.
> 
> The docs on parallel migration are slim. What guidance should we provide wrt 
> selecting a reasonable number of connections for parallel migration? Should 
> users experiment to find a number that saturates the network link used for 
> migration? AFAICT there are currently no bounds checks on the number. E.g. there 
> is nothing preventing 'virsh migrate --parallel --parallel-connections 1000 ...'.

This is a topical subject right now:

  https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg00914.html


If you are using TLS, then the upper limit on the number of connections
you want is equal to the number of logical host CPUs. This is because
when using TLS, you are probably going to max out CPU time on AES
encryption, even when using AESNI acceleration.

Of course if you have other guests running on the host, you likely dont
want all CPUs being used for migration AES work. You also may not want
to max out the network link, if you have other guests using that same
link.

So a real world upper limit on connections is going to be some subset of
host CPUs TBD.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|