[libvirt] [RFC][scale] new API for querying domains stats

[Francesco Romani]

----- Original Message -----
> From: "Richard W.M. Jones" <rjones at redhat.com>
> To: "Francesco Romani" <fromani at redhat.com>
> Cc: libvir-list at redhat.com
> Sent: Friday, July 4, 2014 1:11:54 PM
> Subject: Re: [libvirt] [RFC][scale] new API for querying domains stats

> > Right now we aim for a number of VM per node in the (few) hundreds, but we
> > have big plans
> > to scale much more, and to possibly reach thousands in a not so distant
> > future.
> > At the moment, we use one thread per VM to gather the VM stats (CPU,
> > network, disk),
> > and of course this obviously scales poorly.
> 
> I'll just note here that a bug has been opened for virt-top, which
> is similar to this.
> 
> If a domain has a large number of disks (256 virtio-scsi disks in the
> customer's case), then virt-top spends so long fetching the data for
> each separate disk, it can take 30-40 seconds between updates.
> 
> The same thing would happen if you had lots of domains, each with a
> few disks, but with the total adding up to hundreds of disks.
> 
> The same thing would happen if you substitute network interfaces for disks.
> 
> What would help for us:
> 
>  - A way to get information for multiple objects in a single domain
> 
>  - A way to get information for multiple objects across multiple domains
> 
> in as few API round trips as possible.

I concur. Actually you also expressed our (VDSM) need better than I did.
I think we are on the same boat.

Bests,

-- 
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani