[libvirt] Using virtEvents

Thu Jan 5 18:04:54 UTC 2012

On 12/29/2011 04:03 AM, D. Herrendoerfer wrote:
> Sorry to come back on this.

[Caveat - I didn't write the threading model used in libvirt, so my
answers are my interpretation of the code as I have come to understand
it, but might not be 100% accurate]

> 
> How are the different threads of the libvirt daemon separated ?
> I understand that there are 2 separate threads running that are
> forked. There are also 4 more that appear to be threads.
> Is that correct ?

I'm not quite sure what you were asking, or which processes vs. threads
you are viewing.  Remember, in Linux, threads and processes share the
same namespace of thread ids (that is, a process id in Linux is the
thread-id of its primary thread), and the difference between a
single-threaded app and a multi-threaded app is whether the process owns
more than one thread id.  Calling fork() creates a new process (and thus
a new thread), and the process of starting libvirtd ends up calling
fork() twice to get the daemonization right, so that may explain the "2
separate threads running that are forked".  Once the daemon is up and
running, it becomes multi-threaded, by spawning helper threads all
within the same process, which would explain the "4 more that appear to
be threads".  Basically, libvirtd is set up in an event loop, where the
main thread exists solely to monitor shutdown requests, and all other
threads do round-robin servicing of RPC events that come in over the
sockets.

> 
> Now, the virEvents main polling loop is running in a forked thread,
> it has no access to worker threads that where created through
> virsh for example
> Is that correct ?

Threads aren't forked, only processes are (but forking a new process
creates a new thread, mainly the primary thread of the new process).
Also, processes do not share memory, but threads within a process do, so
a polling loop in a main thread _does_ have access to worker threads in
the same process.  It appears that you have some confusion in the
terminology, which is in turn making it hard for me to know if I'm
answering your questions.

This question added in virsh, which adds another wrinkle into the
picture - virsh is often capable of doing its job as a single-threaded
application, which connects to the libvirtd socket to request libvirtd
to do something with the qemu hypervisor.  However, virsh is also
designed for multi-threaded tasks - in particular, the ability to start
a long-running task such as a migration, and cancel it midway, requires
that virsh have a couple of threads, the primary thread that processes
user commands delegates to a helper thread that interacts with libvirtd
as well as listening for Ctrl-C events from the user or for migration
completion events from libvirtd.

This means that both libvirtd _and_ virsh are using an event loop -
libvirtd to listen for incoming activity on the sockets as well as
completion of those activities by the hypervisor, and virsh to listen
for incoming activity from libvirtd completion.  So in true code reuse
fashion, we reuse the event loop in both processes, and you thus have to
be careful which process you are asking about when discussing threads
that are handling various events.

> 
> Last, if a VM needs something to watch over the state of its
> network devices, a separate thread needs to be created to do
> that because the VMs libvirt daemon thread can't make use
> of the virEvent functions without creating a new poll-loop thread ?

The event loop driver supports both file-based events, such as
recognizing when traffic arrives on the public libvirtd socket which
needs handling by one of the RPC worker threads, as well as time-based
events.  But it is not limited to the public libvirtd socket; the qemu
hypervisor driver can register additional fd's to be watched, such as
the monitor socket tied to the qemu running each domain.  As long as you
can come up with a scenario where an fd can track a change, then you can
register that fd with the existing event loop.  And don't forget that
you can use pipes as a way to set up fds (after all, our SIGINT handler
for Ctrl-C in virsh is nothing more than a pipe-to-self so that we have
an fd to wake up the event loop).

> 
> Best regards,
> 
> D.Herrendoerfer <herrend at de dot ibm dot com > <d.herrendoerfer at
> herrendoerfer dot name>
> 
> On Dec 22, 2011, at 6:04 PM, Dirk Herrendoerfer wrote:
> 
>> Hi all,
>>
>> I'm trying to get libvirt to re-associate lost connections when a vepa
>> connection
>> is lost due to a switch error, or lldpad restart.
>>
>> My take was to use the virtEvent infrastructure to poll for messages
>> on a netlink
>> socket and then restart the association if the message indicates that
>> a link came
>> back up.
>>
>> I ran into a problem, that if I start the polling netlink event from
>> the daemon thread
>> I would get the file events an the netlink messages, but I cannot
>> configure the
>> event handler because the rpc client threads and the daemon do not
>> share the
>> same address space.

So it sounds like you are asking how to make a client, such as virsh,
aware that an event was recognized in libvirtd.  At this point, it boils
down to an RPC event - whenever something in libvirtd fires that a
client might usefully want to learn about, we can then map that to the
RPC event mechanisms.  Look at the virConnectDomainEventRegisterAny API,
which is a means for setting up a callback function in the client that
will be called when the event happens in libvirtd, by means of an RPC
message from libvirtd to the client containing the details appropriate
for the callback.

Also, docs/internals/rpc.html describes the details behind RPC event
calls; normally, things are driven by the client (client issues a call,
server issues a response), but in the case of events, it is an
asynchronous call where the server issues the event, and the client
reacts to the event by calling any registered callback functions.

I'm not sure if I'm on the right track for answering what you are really
trying to do, but hopefully it helped, or someone else can chime in to
correct anything I got wrong.

>>
>> Is there a way to get file events in the VMs setup code so I can
>> register a callback
>> at VM initialization time to receive netlink messages and restart
>> association if needed ?
>>
>> Best regards,
>>
>> D.Herrendoerfer <herrend at de dot ibm dot com > <d.herrendoerfer at
>> herrendoerfer dot name>
>>
>> -- 
>> libvir-list mailing list
>> libvir-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/libvir-list
> 
> -- 
> libvir-list mailing list
> libvir-list at redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list
> 

-- 
Eric Blake   eblake at redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 620 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20120105/4942cdae/attachment-0001.sig>