[libvirt PATCH 1/2] remote: make ssh-helper massively faster

[Daniel P. Berrangé]

On Wed, Nov 25, 2020 at 08:50:07PM +0100, Michal Prívozník wrote:
> On 11/25/20 7:04 PM, Daniel P. Berrangé wrote:
> > It was reported that the performance of tunnelled migration and
> > volume upload/download regressed in 6.9.0, when the virt-ssh-helper
> > is used for remote SSH tunnelling instead of netcat.
> > 
> > When seeing data available to read from stdin, or the socket,
> > the current code will allocate at most 1k of extra space in
> > the buffer it has.
> > 
> > After writing data to the socket, or stdout, if more than 1k
> > of extra space is in the buffer, it will reallocate to free
> > up that space.
> > 
> > This results in a huge number of mallocs when doing I/O, as
> > well as a huge number of syscalls since at most 1k of data
> > will be read/written at a time.
> > 
> > Also if writing blocks for some reason, it will continue to
> > read data with no memory bound which is bad.
> > 
> > This changes the code to use a 1 MB fixed size buffer in each
> > direction. If that buffer becomes full, it will update the
> > watches to stop reading more data. It will never reallocate
> > the buffer at runtime.
> > 
> > This increases the performance by orders of magnitude.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange at redhat.com>
> > ---
> >   src/remote/remote_ssh_helper.c | 113 ++++++++++++++++++++-------------
> >   1 file changed, 68 insertions(+), 45 deletions(-)
> > 
> > diff --git a/src/remote/remote_ssh_helper.c b/src/remote/remote_ssh_helper.c
> > index 0da55c1d1f..8ed7e64507 100644
> > --- a/src/remote/remote_ssh_helper.c
> > +++ b/src/remote/remote_ssh_helper.c
> > @@ -32,6 +32,8 @@
> >   #define VIR_FROM_THIS VIR_FROM_REMOTE
> > +#define SSH_BUF_SIZE (1024 * 1024)
> 
> 
> In theory, our RPC messages can be up to 32MB long (since v3.4.0-rc1~26). Do
> you see any improvements with bigger buffer? But since this buffer is
> allocated at all times it's a trade off I guess.

I've not tested such a large buffer. The performance benefits of larger
buffers drop off fairly quickly though - every doubling of buffer has
50% less benefit than the previous doubling. Since tests show that this
1 MB buffer is enough to beat existing netcat perf, I figure we're fine
as is. If performance is absolutely critical, then honestly you should
not use tunnelled migration at all. QEMU's native TLS migration is
better, especially when combined with multi-FD which easily beats libvirtd.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|