[libvirt] Applying for GSoC project 'Introducing job control to the storage driver' in libvirt

Michal Privoznik mprivozn at redhat.com
Tue Mar 10 15:58:42 UTC 2015


On 09.03.2015 07:29, Taowei Luo wrote:
> Hello, everyone.
> 

Hey!

> I'm Taowei Luo. A participator in Google Summer of Code last year for
> the project rewriting vbox driver in libvirt. I am so glad that my
> project isn't found in this year's GSoC project list :)
> 
> Having a nice experience last year, libvirt is my top option to join
> the GSoC 2015. And my interesting project is “Introducing job control
> to the storage driver”.

It's nice to see people returning.

> 
> I have read some materials about this project. Including I noticed
> that Tucker had attempted it last year. It seems he didn't make it
> through the final evaluation. During his working, some discussion were
> made in mail-list. It is really inspiring to me. So I have some basic
> ideas for now.
> 
> The project requires job control functions (at least, reporting the
> progress) in storage driver. Obviously, it contains two part. First
> find codes that really do the storage work which may take a long
> period and can be asynchronized. Then, extract it to the job control
> part so make it under the control. It would not be hard to find codes
> that really need asynchronization. Maybe by dumping the debug message
> and tracing the function calls. So the big part is design the APIs for
> job control. I think the goal for API design is not only make it
> workable but also make it reasonable and extendable.

The idea is to turn qemuDomainObj*Job* a separate module, that would
work over virObject. Subsequently, we can add BeginJob and EndJob APIs
into storage driver too. Moreover, places where libvirt monitors
progress of a storage operation, we can fill in jobInfo strucut
(although, that one in qemu is not generic enough I guess, so maybe we
need to invent a new one). Then, CancelJob API can be implemented in
storage driver too.

> 
> To achieve this, a lot of details were discussed in last year's
> proceeding. I summered it and add my own opinion.
> 
> Parallel jobs: The idea result is that several jobs work in parallel
> and libvirt monitors and controls it. There are two ways for parallel:
> thread and process. I prefer process. In process, we can easier
> implement the idea of *control* by signal. Process has better
> independence than thread. What's more, it is a low coupling design.

Libvirt already uses that. virCommandExec* is to be find all over the
storage driver.

> 
> Synchronization: process can use system level lock to make sure it
> obtain the resources. If the process can't obtain it could exit with
> failure (or wait). In process, we can leave most resource competition
> handling by OS. If thread is used instead, we need to think about
> resource competition between libvirt and other process, and at the
> same time, those competitions in libvirt thread.
> 
> Management: We execute those asynchronous codes in a new process. In
> libvirt, it invokes those processes with parsed arguments. Libvirtd
> would have a process pool to store the pids and some attached
> information for each process. Signal would be used to communicate
> between process and libvirt.
> 
> Expandability: Some other jobs like domain migration could be in
> implemented under this design. It's all about creating new jobs with
> parsed arguments, which tells the child process what to do and what
> resources they need.
> 
> Privacy: If new jobs are created in process, user may access the
> process directly and not noticing libvirt. Function sigaction(), which
> provides the pid of sending process, could be used to register
> responding functions. Meanwhile, atexit() register functions that
> execute when the process is going to exit. It is helpful on notifying
> libvirtd the end of the job.

This is already handled by virCommand subsystem.

> 
> What already have: Besides the discussion, some other resources would
> be helpful for this project. qemu has a prototype for job control in
> migration. We already have gnulib and tools in util/virprocess.h and
> util/virthread.h to achieve parallel.
> 
> When undertaking, I would firstly implement the job control part. It
> would have some basic functions. Make one storage API work in it.
> Then, adding job control support for the rest APIs.
> 
> This is all about what I came up with. Maybe those ideas are old and
> repeating. But I think it is a workable plan. Waiting for feedback.

Majority looks reasonable to me.

Michal




More information about the libvir-list mailing list