[Libvir] Remote & the problem with qemu deadlock

[Richard W.M. Jones]

As I wrote here: 
https://www.redhat.com/archives/libvir-list/2007-April/msg00114.html 
remote access to qemu:/// URLs currently deadlocks.  A bit of 
explanation follows as to why this happens in the current architecture 
of the remote patch.

If you apply the remote patch right now, you'll get a modified 
libvirt_qemud server which can handle both qemu and remote requests over 
the same socket.  (Basically both qemu_internal.c and remote_internal.c 
connect to the same Unix domain socket, then depending on the "program 
number" encoded within the RPC messages, they get dispatched accordingly 
inside libvirt_qemud).  The server is written to handle multiple 
connections at once using non-blocking poll[1], but once the server has 
assembled a whole incoming message, it then blocks while dealing with 
that message.

The problem occurs in the qemu case when the remote server issues a call 
into qemu_internal (now linked into the server), and qemu_internal then 
tries to connect out to the qemu daemon.  Unfortunately the qemu daemon 
/is/ the remote server, which is blocked handling the current call. 
Thus it is unable to accept the new connection (from itself) and 
completely deadlocks.

One solution suggested was to have qemu_internal recognise when it is 
linked into the server and make local calls back to its counterpart in 
the server.  However I think this is pretty ugly and unnecessarily 
complicates qemu_internal which shouldn't care how it is linked.

My thoughts on the long term solution
-------------------------------------

I'd really like to see the qemu case not require a daemon.  At the 
moment (AFAICS) the qemu daemon serves two purposes: (a) it keeps track 
of the monitor file descriptors of the existing qemu processes, and (b) 
it handles the networking stuff, basically starting and stopping 
dnsmasq[2].  (a) is easily handled, I think, by having qemu processes 
put their monitor sockets into a well-known directory.  This has the 
other advantage that qemu processes can survive and continue after qemu 
programs restart.  For the (b) case, I don't think networking which is a 
general service should be mixed up with qemu code.

I think (although probably I've missed some things) that this means that 
qemu_internal could be implemented entirely as a local service, 
requiring no daemon.  (In the case where qemu_internal needs to control 
a root-owned system-wide set of qemu processes then we'd still have to 
have a daemon, but it can be the remote daemon, same way that Xen will 
work).

My suggestion for a short term solution
---------------------------------------

Since doing the above is way more than I'm going to do just to get 
remote working, my suggestion is that we have two daemons running, qemud 
and remote.  Both would be built from the exact same codebase: the mode 
would be selected by having a 'libvirtd --remote' flag on the command 
line (or however, but the important thing is that they'd keep the same 
codebase).

This immediately fixes the deadlock problem[3] because qemu_internal 
linked to the server is now talking to a different daemon.

Rich.

[1] As an aside, I don't particularly like this architecture.  Linus has 
handed down to us an efficient operating system which is already capable 
of multiplexing different clients using a marvellous system called 
"multitasking".  Hand-coding polling code inside a single process is 
really only necessary in very limited circumstances, such as highly 
tuned uniprocessor machines with enormous loads, and we are not likely 
to get into such a situation with libvirtd.  If we ever did we'd need to 
benchmark a range of possible solutions.  Processes on the other hand 
have real advantages, such as security through isolation and controlled 
sharing, and easy scalability onto SMP machines.

[2] Although I don't really understand the networking stuff too well, so 
perhaps it does other things.

[3] Except in the "remote-on-remote" case, but we should disallow this 
explicitly because it seems like it would be nothing but trouble.

-- 
Emerging Technologies, Red Hat  http://et.redhat.com/~rjones/
64 Baker Street, London, W1U 7DF     Mobile: +44 7866 314 421

Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SL4 1TE, United Kingdom.
Registered in England and Wales under Company Registration No. 3798903
Directors: Michael Cunningham (USA), Charlie Peters (USA) and David
Owens (Ireland)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20070418/f67f7440/attachment-0001.bin>