[libvirt] Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Mon, Mar 22, 2010 at 02:25:00PM -0500, Anthony Liguori wrote:
> Hi,
> 
> I've mentioned this to a few folks already but I wanted to start a 
> proper thread.
> 
> We're struggling in qemu with usability and one area that concerns me is 
> the disparity in features that are supported by qemu vs what's 
> implemented in libvirt.
> 
> This isn't necessarily libvirt's problem if it's mission is to provide a 
> common hypervisor API that covers the most commonly used features.

That is more or less our current mission. If this mission leads to QEMU 
creating a non-libvirt based API & telling people to use that instead, 
then I'd say libvirt's mission needs to change to avoid that scenario !
I strongly believe that libvirt's strategy is good for application 
developers over the medium to long term. We need to figure out how to
get rid of the short term pain from the feature timelag, rather than
inventing a new library API for them to use.
 
> However, for qemu, we need an API that covers all of our features that 
> people can develop against.  The ultimate question we need to figure out 
> is, should we encourage our users to always use libvirt or should we 
> build our own API for people (and libvirt) to consume.
> 
> I don't think it's necessarily a big technical challenge for libvirt to 
> support qemu more completely.  I think it amounts to introducing a 
> series of virQemuXXXX APIs that implement qemu specific functions.  Over 
> time, qemu specific APIs can be deprecated in favour of more generic 
> virDomain APIs.

Stepping back a bit first, there are the two core areas in which people can
be limited by libvirt currently.

 1. Monitor commands
 2. Command line flags

Ultimately, IIUC, you are suggesting we need to allow arbitrary passthrough
for both of these in libvirt.

At the libvirt level, we have 3 core requirements

 1. The XML format is extend only (new elements allowed, or add attributes
    or children to existing elements)
 2. The C library API is append only (new symbols only)
 3. The RPC wire protocol is append only (maps 1-1 to the C API generally)

The core question for us as libvirt developers is how we could support
QEMU specific features that may change arbitrarily, without it impacting
on our ability to maintain these 3 requirements for the non-hypervisor
specific APIs.

We don't ever want to be in a situation where a QEMU specific API will 
require us to change the soname of the main libvirt library, or introduce
incompatible wire protocol changes. If we were to introduce QEMU specific
APIs, we also need a way to easily remove those over time, as & when we 
have them available as generic APIs.

At the C API level, this to me suggests that we'd want to introduce a
separate libvirt-qemu.so  library for the QEMU specific APIs. This 
library would not have the same requirements of fixed long term ABI
that the main libvirt.so did. We'd add QEMU APIs to libvirt-qemu.so 
any time needed, but remove them when the equivalent functionality
were in libvirt.so, and increment the soname of libvirt-qemu.so at
that point.

At the wire protocol level, the protocol allows us to support multiple
versioned protocols in parallel over the same data stream. So again
there, we could define a sub-protocol for QEMU specific features for
which we don't provide the indefinite ABI compatability.

Finally the XML format is "easy" - just have a versioned XML namespace
for extra pieces, that's distinct from the default namespace, again
without the permanent long term compatability guarentees.

There are, however, some bits that are unlikely to work when QEMU
is under libvirt. Specifically any of the device backends that use
stdio (eg, -serial stdio, or the ncurses graphics), simply because
all libvirt spawned VMs are fully daemonized & so stdio is /dev/null

Other items are hard, but not entirely impossible to solve. eg, any 
use of the 'script=' arg for -net devices doesn't work, because libvirt
clears all capabilities from the QEMU process so it'll be lacking
CAP_NET_ADMIN which most TAP device setup scripts in fact need.

Some parts of the C library/wire protocol here are related to another 
feature I'd like to introduce for libvirt, namely a administrative 
library. eg a API to configure and manage the libvirtd daemon itself 
on the fly. This could easily hook into the wire protocol, but live 
as a separate libvirt-daemon.so library API in similar way to what I 
suggest for QEMU specific API

> What's the feeling about this from the libvirt side of things?  Is there 
> interest in support hypervisor specific interfaces should we be looking 
> to provide our own management interface for libvirt to consume?

Adding yet another library in the stack isn't really going to solve the 
problem from the POV of libvirt users, but rather fork the community
effort which I imagine we'd all rather avoid. Having to tell people to
switch to a different library API to get access to a specific feature 
is a short term win, but with a significant long term cost/burden. This 
means we really do need to figure out how to better/fully support QEMU's 
features in libvirt, removing the feature timelag pain.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|