[Libguestfs] nbdcpy: from scratch nbdcopy using io_uring

[Richard W.M. Jones]

On Thu, Jun 24, 2021 at 07:16:10PM +0100, Richard W.M. Jones wrote:
> On Thu, Jun 24, 2021 at 09:30:00PM +0530, Abhay Raj Singh wrote:
> > 
> > In a previous email, Mr. Jones suggested, writing a test implementation of
> > nbdcopy from scratch. First of all, thank you, Mr. Jones, as I got a better
> > perspective of the problem and the solution.
> > 
> > I have almost written the bare-bones implementation for handling NBD-related
> > stuff on which I will base the io_uring core piece by piece. It is written in
> > C++, as I am more comfortable with it. But, made it easily translatable to C.
> > 
> > I have been working on a solution and have documented the current approach, it
> > would be great if you have a look at it and let me know what you think.
> > 
> > Approach document:
> > https://gitlab.com/rathod-sahaab/nbdcpy/-/blob/dev/docs/SOLUTION.md
> 
> So the document is fine.  Since this is throw-away code I wouldn't
> worry about handling errors except to abort the program.
> 
> You can issue the read requests out of order, although perhaps it
> isn't helpful.  And you're correct that you should try to limit the
> number of requests in flight (otherwise you'll run out of memory when
> you try to copy very large disks).  However 6 is probably too low - in
> nbdcopy we use 64.
> 
> NBD supports multi-conn (well, technically speaking you should check a
> flag on the server but that doesn't matter for testing).  So you can
> open multiple TCP connections on each side, and issue multiple
> commands in flight on each of those connections.  We found this
> improves performance a lot.
> 
> The code looks very minimal at the moment.  I'm not very familiar with
> the fmt:: class.  Does it successfully make a handshake to ‘nbdkit -o’
> yet?

Another thing is that real nbdcopy handshakes with the servers
synchronously, and only does the data transfer part asynchronously.
This simplifies the current code quite a bit.
(https://gitlab.com/nbdkit/libnbd/-/blob/master/copy/nbd-ops.c)

> I notice that although you create an io_uring you're not using it.
> But I guess that's still to come.
> 
> Rich.
> 
> -- 
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> Fedora Windows cross-compiler. Compile Windows programs, test, and
> build Windows installers. Over 100 libraries supported.
> http://fedoraproject.org/wiki/MinGW
> 
> _______________________________________________
> Libguestfs mailing list
> Libguestfs at redhat.com
> https://listman.redhat.com/mailman/listinfo/libguestfs

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v