[Libguestfs] parallelization

[Richard W.M. Jones]

On Tue, Feb 26, 2013 at 03:00:42PM +0000, Daniel P. Berrange wrote:
> On Tue, Feb 26, 2013 at 02:56:53PM +0000, Richard W.M. Jones wrote:
> > On Tue, Feb 26, 2013 at 09:36:30AM -0500, Colin Walters wrote:
> > > So Richard recently blogged about running many libguestfs mounts in
> > > parallel.
> > > 
> > > I actually want parallelization in the other way; I use guestmount to
> > > access a disk image, and my program is multithreaded.  It does
> > > read/write and most especially calls to link() and fdatasync() from
> > > threads.
> > > 
> > > However, this all ends up being serialized, because the guestfsd
> > > protocol is synchronous, and the daemon side is single-threaded.
> > > Related to this, the qemu instance is always booted with only one
> > > vCPU.
> > > 
> > > Have there been any thoughts on fixing this?
> > 
> > Yes.  However it requires changes to the virtio-serial protocol or
> > perhaps its entire replacement with VMware's vsock protocol (currently
> > being discussed in upstream qemu).  Without this, it's pretty clunky
> > to open more than one connection from the library to the appliance.
> 
> That's if you need to open multiple connections though. Sounds like
> in Colin's case he just has one multi-threaded app, in which case
> all apps could just share one connection & have the RPC messages
> multi-plexed as we do with libvirt. Of course still needs the daemon
> to be multithreaded to get any benefit from that.

I'm really extremely cautious about making any change to the protocol.

I found and fixed yet another bug (a crasher) in it only a few days
ago (RHBZ#914931).  You'd have thought we would have worked out all
the bugs in it after 5 years, but apparently not ...

Having parallel connections avoids making any protocol changes (except
if you consider parallel connections to be a protocol change, but I
guess you know what I mean ...)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org