[augeas-devel] improving performance of aug_get() and aug_match() with large datasets

[Laine Stump]

On 10/05/2015 12:31 PM, David Lutterkort wrote:
> On Mon, Oct 5, 2015 at 6:54 AM, Laine Stump <laine at redhat.com 
> <mailto:laine at redhat.com>> wrote:
>
>
>     augeas netcf  libvirt list   dump    virt-manager
>     ------ -----  ------- ------ ------- ------------
>     1.4.0  0.2.8  1.2.20  1:37.6 13:46.6 15:37
>     upstrm 0.2.8  1.2.20  1:04.7 07:34.8 08:41
>     upstrm upstrm 1.2.20  0:03.7 06:40.3 06:46
>     upstrm upstrm upstrm  0:02.0 06:39.5 06:39
>

With your suggested query implemented in netcf:

>
>     upstrm upstrm upstrm  0:02.1 0:17.7 00:17
>

so nearly identical to the timing when I *removed that query completely*!

> This all looks very encouraging - are these times satisfactory for 
> users or do we need to look more into speeding them up further ?
>

I would say that 60x better performance looks impressive to anyone :-). 
Especially since:

1) The number of users with this many host interfaces is very small, and

2) upstream virt-manager becomes usable as soon as it has the list of 
all interface names is returned (which now takes 2sec rather than 
1m37s), with the remaining 15 sec simply consuming one CPU core while 
the user goes merrily on their way.

So I think this much improvement is good enough for now. (It would be 
nice if aug_load() could take advantage of inotify or something similar 
to avoid the large overhead of checking all files' mtimes though - even 
though it works in practice, I'm still not comfortable with the idea 
that we could occasionally have information that is up to 1 second out 
of date, as this could lead to race conditions if some other entity is 
modifying the config on disk directly.)

Thanks for all your help with this. The resulting improvement has been 
*much* better than I ever expected.

(BTW - please feel free to review/ACK the netcf patch that I posted to 
netcf-devel yesterday :-); you surely know the details behind it better 
than anyone else, and I posted it mainly so that you could verify I am 
implementing what you said. (Note that your current address has been 
whitelisted, so you don't need to subscribe)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/augeas-devel/attachments/20151008/9b7cbaf2/attachment.htm>