[libvirt] [RFC][scale] new API for querying domains stats

[Daniel P. Berrange]

On Wed, Jul 02, 2014 at 11:56:23AM -0400, Francesco Romani wrote:
> > > This is made only worse by the fact that VDSM is a python 2.7 application,
> > > and notoriously
> > > python 2.x behaves very badly with threads. We are already working to
> > > improve our code,
> > > but I'd like to bring the discussion here and see if and when the querying
> > > API can be improved.
> > >
> > > We currently use these APIs for our sempling:
> > >    virDomainBlockInfo
> > >    virDomainGetInfo
> > >    virDomainGetCPUStats
> > >    virDomainBlockStats
> > >    virDomainBlockStatsFlags
> > >    virDomainInterfaceStats
> > >    virDomainGetVcpusFlags
> > >    virDomainGetMetadata
> > >
> > > What we'd like to have is
> > >
> > > * asynchronous APIs for querying domain stats
> > > (https://bugzilla.redhat.com/show_bug.cgi?id=1113106)
> > >    This would be just awesome. Either a single callback or a different one
> > >    per call is fine
> > >    (let's discuss this!).
> > >    please note that we are much more concerned about thread reduction then
> > >    about performance
> > >    numbers. We had report of thread number becoming a real harm, while
> > >    performance so far
> > >    is not yet a concern
> > >    (https://bugzilla.redhat.com/show_bug.cgi?id=1102147#c54)
> > 
> > I'm not a big fan of this approach. I mean, IIRC python has this Big
> > Python Lock, which effectively prevents two threads run concurrently.
> 
> It has the GIL, yes. Only one thread can run python code at any given time.
> This however it is not true for extensions modules written in C which if carefully
> designed (read: coded to properly release the GIL) can run concurrently.
> This is one of the reasons while threading in python it is tolerated for I/O,
> evne though never recommended.
> 
> AFAIK/IIRC the code the libvirt module for python allows this, so we should
> be good to go.

For the sake of completeness I'll point out that there's another theoretical
option. The libvirt-gobject binding to libvirt provides async APIs to libvirt
APIs. It does this by using threads internally. Since these are C level threads
though, if VDSM were to use libvirt-gobject it could get async APIs and the
benefits of real threads, while remaining single threaded at the python layer.

That all said, I'm not sure whether libvirt-gobject has sufficient API
coverage for all the APIs VDSM needs. It primarily just has bindings for
the APIs used by GNOME Boxes & libvirt-sandbox so far. Also not sure if
it is a widely deployed enough dep for VDSM to mandate.


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|