[libvirt] Applying for GSoC project 'Introducing job control to the storage driver' in libvirt

[Taowei Luo]

Thank you, I will take a look on those APIs you suggested.

BTW, about the project virsh, I found
libvirt-client(http://www.rpmfind.net/linux/rpm2html/search.php?query=libvirt-client)
already support automatic completion on virsh commands. It is a big
progress on the project. So, why it hasn't merged into git source
repository?

2015-03-10 23:58 GMT+08:00 Michal Privoznik <mprivozn at redhat.com>:
> On 09.03.2015 07:29, Taowei Luo wrote:
>> Hello, everyone.
>>
>
> Hey!
>
>> I'm Taowei Luo. A participator in Google Summer of Code last year for
>> the project rewriting vbox driver in libvirt. I am so glad that my
>> project isn't found in this year's GSoC project list :)
>>
>> Having a nice experience last year, libvirt is my top option to join
>> the GSoC 2015. And my interesting project is “Introducing job control
>> to the storage driver”.
>
> It's nice to see people returning.
>
>>
>> I have read some materials about this project. Including I noticed
>> that Tucker had attempted it last year. It seems he didn't make it
>> through the final evaluation. During his working, some discussion were
>> made in mail-list. It is really inspiring to me. So I have some basic
>> ideas for now.
>>
>> The project requires job control functions (at least, reporting the
>> progress) in storage driver. Obviously, it contains two part. First
>> find codes that really do the storage work which may take a long
>> period and can be asynchronized. Then, extract it to the job control
>> part so make it under the control. It would not be hard to find codes
>> that really need asynchronization. Maybe by dumping the debug message
>> and tracing the function calls. So the big part is design the APIs for
>> job control. I think the goal for API design is not only make it
>> workable but also make it reasonable and extendable.
>
> The idea is to turn qemuDomainObj*Job* a separate module, that would
> work over virObject. Subsequently, we can add BeginJob and EndJob APIs
> into storage driver too. Moreover, places where libvirt monitors
> progress of a storage operation, we can fill in jobInfo strucut
> (although, that one in qemu is not generic enough I guess, so maybe we
> need to invent a new one). Then, CancelJob API can be implemented in
> storage driver too.
>
>>
>> To achieve this, a lot of details were discussed in last year's
>> proceeding. I summered it and add my own opinion.
>>
>> Parallel jobs: The idea result is that several jobs work in parallel
>> and libvirt monitors and controls it. There are two ways for parallel:
>> thread and process. I prefer process. In process, we can easier
>> implement the idea of *control* by signal. Process has better
>> independence than thread. What's more, it is a low coupling design.
>
> Libvirt already uses that. virCommandExec* is to be find all over the
> storage driver.
>
>>
>> Synchronization: process can use system level lock to make sure it
>> obtain the resources. If the process can't obtain it could exit with
>> failure (or wait). In process, we can leave most resource competition
>> handling by OS. If thread is used instead, we need to think about
>> resource competition between libvirt and other process, and at the
>> same time, those competitions in libvirt thread.
>>
>> Management: We execute those asynchronous codes in a new process. In
>> libvirt, it invokes those processes with parsed arguments. Libvirtd
>> would have a process pool to store the pids and some attached
>> information for each process. Signal would be used to communicate
>> between process and libvirt.
>>
>> Expandability: Some other jobs like domain migration could be in
>> implemented under this design. It's all about creating new jobs with
>> parsed arguments, which tells the child process what to do and what
>> resources they need.
>>
>> Privacy: If new jobs are created in process, user may access the
>> process directly and not noticing libvirt. Function sigaction(), which
>> provides the pid of sending process, could be used to register
>> responding functions. Meanwhile, atexit() register functions that
>> execute when the process is going to exit. It is helpful on notifying
>> libvirtd the end of the job.
>
> This is already handled by virCommand subsystem.
>
>>
>> What already have: Besides the discussion, some other resources would
>> be helpful for this project. qemu has a prototype for job control in
>> migration. We already have gnulib and tools in util/virprocess.h and
>> util/virthread.h to achieve parallel.
>>
>> When undertaking, I would firstly implement the job control part. It
>> would have some basic functions. Make one storage API work in it.
>> Then, adding job control support for the rest APIs.
>>
>> This is all about what I came up with. Maybe those ideas are old and
>> repeating. But I think it is a workable plan. Waiting for feedback.
>
> Majority looks reasonable to me.
>
> Michal