[Libvir] RFC: PATCH 0/5: asynchronous background jobs

[Daniel Veillard]

On Sat, Jan 05, 2008 at 01:06:52AM +0000, Daniel P. Berrange wrote:
> 
> We currently have a restriction that virConnectPtr objects
> must only be used by a single thread. At the same time we
> have a number of operations that can take a very long time.
> This means that applications are pretty much forced to use
> multi-threading if they want to continue to respond to user
> input, and thus are also forced to use multiple virConnectPtr
> objects.
> 
> This is sub-optimal because opening multiple virConnectPtr
> objects may well result in the user being prompted for auth
> credentials multiple times. It also doesn't provide any way
> of providing progess information on the long running jobs,
> nor a way to cancel them.
> 
> Thus the following series of patches introduces the concept of
> an 'asynchronous background job', represented by a virJobPtr
> object in the libvirt public API.
> 
> A job has an elapsed time, and may optionally include an
> estimated time of completion. If it is a job with a bounded
> amount of work, it will also have a percentage completion
> counter. There is the ability to request cancellation of an
> in progress job, and fetch any error associated with a
> completed job.
> 
> The primary reason I implemented this support, is that the
> storage APIs will make the current situation untennable.
> Allocating storage can take a seriously long time (many
> many minutes if cloning multi-GB files). This absolutely
> requires progress information and an ability to cancel
> an operation.

  Okay, I understand the need for asynch operations. And I 
also agree that an API which forces to use threads in practice
is a really annoying thing and should be avoided. I didn't 
really though about solutions for this until now, the 'job'
approach is interesting, it sounds close to what OSes provides
for some asynchronous I/O. I like the idea, the question is 
if we will provide any synchronization primitive or if the
time estimate will be sufficient to handle to an application
select/poll main loop.

> There are a number of existing APIs which can benefit from
> this, hence I decided to work on this separately from the
> main storage APIs. The APIs which can make use of this are
> virDomainCreateLinux, virDomainCreate, virNetworkCreate,
> virNetworkCreateXML, virDomainSave, virDomainRestore, and
> virDomainDumpCore. For all of these we add a second variant
> postfixed with 'Job' in the name, returning a virJobPtr object
> 
> This work isn't finished, but I wanted to put these patches
> out for comment before going further. In particular I need
> to figure out how to hook this up to the remote driver and
> daemon, and deal with a few lifecycle issues. eg what todo
> if a virConnectPtr object is released while a background job
> is active.

  Sounds to me that the creation of a Job should increate a reference
counter for conn, which will be decreased when the Job object is destroyed.
This shouldmap well with Python bindings too.

> 
> With all 5 patches applied you should be able to run the
> following test case:
> 
>   virsh --connect test:///default save --verbose test foo
> 
> 
> No other drivers / commands are hooked up aside from 'save'
> in the 'test' driver.

I will try to provide a finer grained review of your patches
for Tuesday, not really urgent as this is a work in progress, right ?

  thanks a lot for looking at it, sounds a good idea, 

Daniel

-- 
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillard at redhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine  http://rpmfind.net/