[libvirt] [PATCH v2] lib: Add note that bulk stats API queries may overrun RPC buffers

[Michal Privoznik]

On 05/24/2017 12:32 PM, Martin Kletzander wrote:
> On Tue, May 23, 2017 at 05:29:52PM +0200, Peter Krempa wrote:
>> On Tue, May 23, 2017 at 17:19:53 +0200, Martin Kletzander wrote:
>>> On Tue, May 23, 2017 at 05:07:40PM +0200, Michal Privoznik wrote:
>>> > On 05/23/2017 04:35 PM, Martin Kletzander wrote:
>>> > > On Tue, May 23, 2017 at 04:23:30PM +0200, Peter Krempa wrote:
>>
>> [...]
>>
>>> > > > + * Note that this API is prone to exceeding maximum RPC if
>>> querying
>>> > > > too many VMs
>>> > > > + * with lots of statistics. It's suggested to query in batches of
>>> > > > 10VMs, which
>>> > > > + * should be good enough for VMs with 3000 disks + networks.
>>> > > > + *
>>> > >
>>> > > Coming to think about it... Why don't we just batch this
>>> ourselves under
>>> > > the hood and just return the merged result?
>>> >
>>> > Because:
>>> >
>>> > https://www.redhat.com/archives/libvir-list/2017-May/msg00088.html
>>> >
>>>
>>> Not on the RPC level, the API would just be syntax sugar to
>>> virDomainListGetStats() if a flag was passed
>>> (e.g.
>>> VIR_DOMAIN_GET_ALL_STATS_I_DONT_REALLY_CARE_IF_THIS_IS_DONE_IN_ONE_LIBVIRT_CALL)
>>>
>>
>> Also compared to a full fragmentation of the returned data, this would
>> result into a worst-case-scenario memory usage of MAX_SIZE *
>> NVMS_QUERIED_IN_ORIGINAL_CALL, when compared to an unbounded memory use
>> of the full fragmentation approach.
> 
> If I get what you are saying, then the same would happen if the mgmt app
> (or client) implemented it themselves.  We would basically just provide
> the guessing logic.

That's quite exact. I mean the word 'guessing'. We can't really provide
reliable way of dealing with what you're suggesting (unless we cut the
limit really small) nor we can guarantee atomicity. Therefore I think it
would be a waste of time to work on this. Yes, it can be done, but the
benefits are pretty small IMO.

Michal