[Libvir] Making QEMU use the real internal driver API

[Daniel P. Berrange]

So currently, as we all know, the QEMU driver is just a shim through to the
QEMU daemon. Looking at the code for Rich's remote driver I quite like the
way the remote driver / server works like, and don't like the ugliness of
having the special case QEMU code in there. We always had a long term plan
to try & make the QEMU driver fit into the regular driver API more cleanly
but not really tackled it yet. Similarly we've periodically brought up the 
idea of having asynchronous notifications of 'interesting' state changes 
in the drivers, to eliminate the need for polling.

After a little thinking I have an idea that may let us address both issues 
in one go. Or to be more precise, put infrastructure in place to let us
move closer to allowing async notiifcations.


First let me consider the files under qemud/ ....

Self-contained helper routines, ignore for rest of discussion

  bridge.c
  bridge.h
  iptables.c
  iptables.h
  uuid.c
  uuid.h
  buf.c
  buf.h

The on the wire protcol definition, this is the bit we want to kill in favour
of the generic remote wire protocol:

  protocol.h

The code to convert from wire -> function calls. Again this ought to die in
favour of the generic remote dispatcher.

  dispatch.c
  dispatch.h

The QEMU config handling routines.

  conf.c
  conf.h

The impl of core APIs:

  driver.c
  driver.h

Finally the 'server':

  internal.h
  qemud.c


The end-goal is that 'qemud/driver.c'  would become 'src/qemu_internal.c'
implementing the official internal driver API.  The 'qemud/conf.c' would
likely become 'src/qemu_config.c'. 

The key difference between the internal driver API, and the QEMU specific
driver 'API' inside the daemon is the prescense of some global state via
the 'qemud_server' struct. If we can figure out a way to store this state
inside the regular driver API we can refactor that without much problem.

Then there is the need for an event loop for monitoring the file handles 
of the QEMU processes, which at the moment is fairly tied into the event 
loop used to monitor the client/server socket handles. This existing code
is rather ugly & has bad structure IMHO. The result is that code which 
ought to be isolated in driver.c, is instead part of qemud.c because of
event loop integration.

Finally, there is some code which runs on startup, and shutdown of the
daemon, to do stuff like auto-start of VMs / networks, and clean shutdown
of the same.

Oh, and we need to make sure the QEMU driver is never activated locally,
only when libvirt.so is being used as part of 'libvirtd'.

So at a high level what I think we'd need to add to the internal driver
API to make this possible is

In driver.h:

 typedef int (*virDrvStartup)(void);
 typedef int (*virDrvShutdown)(void);

Adding these to the 'virDriver' struct. Regular apps using libvirt.so
would never trigger these APIs - they're only intended for the daemon,
so I figure we'd have to private APIs in the libivrt.so  (but not in
the libvirt.h)

  __virStartup(void);
  __virShutdown(void);

Since these are only called by the daemon, we now conveniently also have
a way to stop the QEMU driver being locally activated.


Next up, event loops. First, we don't want to mandate a particular impl
of an event loop. If you get into the argument about QT vs GLib vs libevent
and libvirt picks one, then you make it hard for libvirt to integrate with
apps using the other. Second, we don't want to assume that 1 connection 
== 1 file handle. We also don't want to assume that the number of file
handles used by a driver stays the same over time. Basically we need to
expect that any single driver connection will have zero or more file
handles that it is interested in.

The DBus library deals with this exact same issue & the solution is very
simple. The app using the libvirt registers a callback which the internal
driver than then use to (un)register file handles. 

So in src/internal.h we'd add methods for internal drivers to call to
add / remove file handles to/from the event loop:

  /* 
   * @fd: the file handle on which the event occurred
   * @event: bitset of one or more of POLLxxx constants from poll.h
   * @opaque: data associated with 'fd'
   *
   * Return 0 to continue monitoring, -1 if an error occurred
   * and the handle should no longer be monitored
   */
  typedef int (*virEventCallback)(int fd, int events, void *opaque);

  /*
   * @fd: the file handle to start monitoring
   * @events: bitset of POLLxxx contants to monitor
   * @cb: callback to invoke when an event ocurrs
   * @opaque: data to pass into callback's opaque parameter
   */
  int virEventAddHandle(int fd, int events, virEventCallback cb, void *opaque);

  /*
   * @fd: the file handle to stop monitoring
   */
  int virEventRemoveHandle(int fd);


And we'd also add methods for a user of libvirt to register an event loop
implementation:

   typedef int (*virEventAddHandleImpl)(int fd, int events, opaque);
   typedef int (*virEventRemoveHandleImpl)(int fd);

   void virEventInitialize(virEventAddHandleImpl add, virEventRemoveHandleImpl remove);


If we want to add support for async events into the public API, then the
virEventInitialize method could instead be in libvirt.h, and exported in
the  libvirt.so. Otherwise we could mark it private __virEventInitialize
in libvirt.so.  

That I think should be enough to let us move QEMU into a regular style libvirt
driver.


NB, what follows now is a quick brain-dump - I'm not suggesting implementing 
this yet - its really a rough idea of general architecture. We'd definitely
want to prototype it in a test app before committing to something like this.
There's also many many scenarios to be thought about  beyond just watching
create/destroy events

Thinking about a public API we could have:

    enum {
       VIR_CONN_DOMAIN_CREATED,
       VIR_CONN_DOMAIN_DESTROYED;
    } virConnectEvent;

    typedef int (*virConnectNotifyCallback)(virConnectPtr dom, unsigned char uuid[16], int event);

    virConnectNotify(virConnectPtr conn, virConnectNofifyCallback cb);


An application using this would thus be able to do something like

    /* Assume there's a convenient libvirtglib.so addon providing these for apps */
    extern virEventAddHandleImpl virGLibEventAddHandle;
    extern virEventRemoveHandleImpl virGLibEventRemoveHandle;

    int domainWatcher(virConnectPtr dom, unsigned char uuid[16], int event) {
        ... do something ...
    }

    int main () {
     virConnectPtr con;

     con = virConnectOpen(NULL);

     /* Setup an event loop via glib */
     virEventInitilize(virGLibEventAddHandle, virGLibEventRemoveHandle);

     virConnectNotify(conn, domainWatcher);


Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|