[augeas-devel] Match operations over long file too slow

[Josef Reidinger]

On Wed, 14 Jun 2017 14:13:31 +0200
"Štěpán Balážik" <stepan.balazik at nic.cz> wrote:

> Hi,
> 
> we want to use Augeas to programmatically parse and generate tests for 
> Deckard <https://gitlab.labs.nic.cz/knot/deckard>[1] (our test harness 
> for DNS resolvers).
> 
> I wrote a lens 
> <https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/pydnstest/deckard.aug>[2] 
> which works fine but on longer config files such as this one 
> <https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/sets/resolver/iter_badraw.rpl>[3] 
> the match operation on /scenario/step nodes takes about 3 ms per call on 
> my machine using both augtool and the python binding. This sums up to 
> over 10 s for the whole file which too slow for our purposes.
> 
> We have no idea what is the cause of this slowness. Are Augeas matches 
> always so slow on "shallow and wide" trees?
> 
> Thanks in advance for any help.
> Štěpán Balážik @ CZ.NIC
> 
> [1] https://gitlab.labs.nic.cz/knot/deckard
> [2] 
> https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/pydnstest/deckard.aug
> [3] 
> https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/sets/resolver/iter_badraw.rpl
> 

Hi Stepan,
we face similar issue when parsing /etc/hosts that are auto-filled by some blocking software ( so around 10k entries there ). In the end solution for us is to introduce caching of match results and creating own map. We use ruby, but I think it is so similar to python, that you get idea.[1]
If you need any explanation, do not hesitate to ask. Speed up in our case is approximately half of time spend at parsing.

Josef

[1] https://github.com/config-files-api/config_files_api/blob/master/lib/cfa/augeas_parser/keys_cache.rb