[libvirt] [RFC] Introduce API for retrieving bulk domain stats

[Eric Blake]

On 08/19/2014 08:43 AM, Daniel P. Berrange wrote:
> On Tue, Aug 19, 2014 at 03:14:19PM +0200, Peter Krempa wrote:
>> I'd like to propose a (hopefully) fairly future-proof API to retrieve
>> various statistics for domains.
>>
>> The motivation is that management layers that use libvirt usually poll
>> libvirt for statistics using various split up APIs we currently provide.
>> To get all the necessary stuff, the mgmt app need to issue Ndomains *
>> Napis calls and cope with the various returned formats. The APIs I'm
>> wanting to introduce here will:
>>
>> 1) Return data in a format that we can expand in the future and is
>> hierarchical. For starters I'll use XML, with possible expansion to
>> something like JSON if it will be favourable for a consumer (switchable
>> by a flag)
> 
> I'm not particularly a fan of using XML for this. As a guiding principal
> we've used structures when we needed APIs for which efficiency is
> important, and XML for APIs where efficiency is irrelevant. Even with
> the ability to bulk list data from many VMs at once, I'd think that
> efficiency is still a very important property of the API. Consider if
> we have 1000 virtual machines on a host - the XML document is going to
> get very large and frankly most XML parsers are terribly slow. Would
> not surprise me if it too several seconds or more to parse an XML doc
> that this proposed API would return, which I don't think is really
> viable. JSON might be slightly better in this respect, but not by
> as much as we might imagine.

I'm also not a fan of XML or JSON for the all-domains-in-one-API call in
this respect, because for a monolithic string, you can't return it
unless you have the data for ALL domains, and you may be hitting RPC
limits far sooner than you wish (it doesn't scale well to machines that
can run thousands of VMs).  I'm also not a fan of JSON in a public API
since none of our existing APIs return JSON (well, libvirt-qemu.so
returns JSON for arbitrary monitor commands, but that's an unsupported
API and is merely exposing what qemu does) - adding JSON to a public API
doubles the parsing support that a client has to use.  Using XML for a
single domain might be doable, though, but that's one XML document per
domain, not one XML document for the overall API.

I still think we're going to have to come up with some sort of iterative
callback approach, where the callback is visited once per domain, rather
than trying to return all results in a single call.

> 
> So I'd rather think we need to figure out a way to map this into some
> kind of struct, where reading any single statistic could be done in
> approx constant time, or at least time that is independant of the
> number of VMs.
> 
> Much as I dislike the virTypedParameter struct, it might actually be
> a reasonable fit here. Perhaps a struct
> 
>   struct virDomainRecord {
>      virDomainPtr dom;
>      size_t nparams;
>      virTypedParameter params;
>   };
> 
> and the API returns an array of virDomainRecord elements.
> For the keys in the parameters we could use dot separated
> components. eg
> 
>     state=runing
>     cpu.online=8
>     cpu.0.state=running
>     cpu.0.time=10001231231
>     cpu.1.state=running
>     cpu.1.time=10001231231
>     ...
>     cpu.7.state=running
>     cpu.7.time=10001231231

I'm not sure whether this is a net gain on memory usage.  It seems like
this approach would require that libvirt malloc's each parameter, and
that the user must free them all at the end.  At least with an XML
document, there's only a single malloc'd chunk of memory, rather than
lots of pieces, on the API side.  On the other hand, if a user is given
an XML document, but ends up parsing it into an in-memory
representation, they may end up malloc'ing even more memory for that
in-memory representation, so our parameters might be a net win after
all, where it is just a tradeoff of libvirt instead of the xml parser
doing all the mallocing of small chunks.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 539 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20140819/5b2adf41/attachment-0001.sig>