[Libvir] libvirt and accessing remote systems

[Daniel P. Berrange]

On Wed, Jan 24, 2007 at 02:17:31PM +0000, Richard W.M. Jones wrote:
> This is a follow on to this thread:
> https://www.redhat.com/archives/libvir-list/2007-January/thread.html#00064
> but I think it deserves a thread of its own for discussion.
> 
> Background:
> 
> Dan drew this diagram proposing a way to include remote access to
> systems from within libvirt:
> 
>   http://people.redhat.com/berrange/libvirt/libvirt-arch-remote-2.png
> 
> libvirt would continue as now to provide direct hypervisor calls,
> direct access to xend and so on.  But in addition, a new backend
> would be written ("remote") which could talk to a remote daemon
> ("libvirtd") using some sort of RPC mechanism.
> 
> Position:
> 
> I gave this architecture some thought over the weekend, and I
> like it for the following reasons (some not very technical):
> 
>  * Authentication and encryption is handled entirely within the
>    libvirt / libvirtd library, allowing us to use whatever RPC
>    mechanism we like on top of a selection of transports of our
>    choosing (eg. GnuTLS, ssh, unencrypted TCP sockets, ...)

Yes, having a single consistent wire encyption & user auth system across
all virt backends makes a very nice end user / admin story

>  * We don't need to modify xend at all, and additionally we won't
>    need to modify future flavour-of-the-month virtual machine monitors.
> 
>    I have a particular issue with xend (written in Python) because
>    in my own tests I've seen my Python XMLRPC/SSL server
>    actually segfault.  It doesn't inspire me that this Python
>    solution is adding anything more than apparent security.

Did I mention XenD is slow. If we can get remote management of Xen
bypassing XenD just like we do for the local case, we'll be much
better off.

>  * The architecture is very flexible: It allows virt-manager to
>    run as root or as non-root, according to customer wishes.
>    virt-manager can make direct HV calls, or everything can be
>    remoted, and it's easy to explain to the user about the
>    performance vs management trade-offs.
> 
>  * It's relatively easy to implement.  Note that libvirtd is just
>    a thin server layer linked to its own copy of libvirt.
> 
>  * Another proposal was to make all libvirt calls remote
>    (http://people.redhat.com/berrange/libvirt/libvirt-arch-remote-3.png)
>    but I don't think this is a going concern because (1) it requires
>    a daemon always be run, which is another installation problem and
>    another chance for sysadmins to give up, and (2) the perception will
>    be that this is slow, whether or not that is actually true.

I'd never compared performance of direct hypercalls vs libvirt_proxy
before, so I did a little test. The most commonly called method is
virt-manager is virDomainGetInfo  for fetching current status of a
running domain - we call that once a second per guest.

So I wrote a simple program in C which calls virDomainGetInfo 100,000
times for 3 active guest VMs. I ran the test under a couple of different
libvirt backends. The results were:

 1. As root, direct hypercalls                ->  1.4 seconds
 2. As non-root, hypercalls via libvirt_proxy -> 9 seconds
 3. As non-root, via XenD                     -> 45 minutes [1]

So although it is x10 slower than hypercalls, the libvirt_proxy is 
actaully pretty damn fast - 9 seconds for 300,000 calls.

There are many reasons the XenD path is slow. Each operation makes
a new HTTP request. It spawns a new thread per request. It talks to
XenStore for every request which has very high I/O overhead. It uses
the old SEXPR protocol which requests far more info than we actually
need. It is written in Python.  Now I'm sure we can improve performance
somewhat by switching to the new XML-RPC api, and getting persistent
connections running, but I doubt it'll ever be as fast as libvirt_proxy
let alone hypercalls. So as mentioned above, I'd like to take XenD
out of the loop for remote management just like we do for the local
case with libvirt_proxy, but with full authenticated read+write access.

> Now some concerns:
> 
>  * libvirtd will likely need to be run as root, so another root
>    daemon written in C listening on a public port.  (On the other
>    hand, xend listening on a public port also isn't too desirable,
>    even with authentication).

For Xen we have no choice but to have something running as root
since hypercalls needs to mlock() memory and access the Xen device 
node. So both options are unpleasent, but we have to choose one,
and I can't say XenD is the obvious winner - particularly given the
tendancy of python SSL code to segfault. We can, however, make sure
that libvirtd is written to allow the full suite of modern protection
mechanisms to be applied - SELinux, execshield, TLS, fortify source
etc.

>  * If Xen upstream in the meantime come up with a secure remote access
>    method then potentially this means clients could have to choose
>    between the two, or run two services (libvirtd + Xen/remote).
> 
>  * There are issues with versioning the remote API.  Do we allow
>    different versions of libvirt/libvirtd to talk to each other?
>    Do we provide backwards compatibility when we move to a new API?

We can simply apply the same rules as we do for public API. No changes
to existing calls, only additions are allowed. A simple protocol 
version number can allow the client & server to negotiate the mutually
supported feature set.

>  * Do we allow more than one client to talk to a libvirtd daemon
>    (no | multiple readers one writer | multiple readers & writers).

The latter - since we have things modifiying domains via XenD, or the
HV indirectly updating domain state, we defacto have multiple writers
already. From my work in the qemu daemon, I didn't encounter any major
problems with allowing multiple writers - by using a poll() based 
single-thread event loop approach, I avoided any nasty multi-thread
problems associated with multiple connections. Provided each request
can be completed in a short amount of time, there should be no need to
go fully threaded.

>  * What's the right level to make a remote API?  Should we batch
>    calls up together?

We may be already constrained by the client side API - all calls in
the libvirt public API are synchronous so from a single client thread
there is nothing available to batch. 

> I've been investigating RPC mechanisms and there seem to be two
> reasonable possibilities,  SunRPC and XMLRPC.  (Both would need to
> run over some sort of secure connection, so there is a layer below
> both).  My analysis of those is here:
> 
>   http://et.redhat.com/~rjones/secure_rpc/

SunRPC would handle our current APIs fine. We've talked every now & then
about providing asynchronous callbacks into the API - eg, so the client
can be notified of VM state changes without having to poll the 
virDomainGetInfo api every second. The RPC wire protocol certainly 
supports that, but its not clear the C apis do.

The XDR wire formating rules are very nicely defined - another option is
to use XDR as the wire encoding for our existing prototype impl in the
qemud.

For XML-RPC I'd like to do a proof of concept of the virDomainGetInfo
method impl to see how much overhead it adds. Hopefully it would be
acceptable, although I'm sure its a fair bit more than XDR / SunRPC.
We would need persistent connections for XML-RPC to be viable, particularly
with TLS enabled. Since XML-RPC doesn't really impose any firm C API
I imagine we could get a-synchronous notifications from the server
working without much trouble.

Dan.

[1] It didn't actually finish after 45 seconds. I just got bored of waiting.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|