[augeas-devel] Match operations over long file too slow

[David Lutterkort]

Hi Štěpán,

Just to round this out, I believe the changes that cause this speedup went
into Augeas 1.5.0, and they are the commits from this PR
<https://github.com/hercules-team/augeas/pull/303>.

David

On Wed, Jun 14, 2017 at 7:27 AM, Štěpán Balážik <stepan.balazik at nic.cz>
wrote:

> Hi again,
>
> so I was using Augeas version 1.4.0 from Ubuntu repositories. Installing
> latest stable version 1.8.0 made the process (literally) 100x faster.
>
> I am sorry I was bothering you all.
>
> Štěpán
>
>
>
> On 14.6.2017 14:40, Josef Reidinger wrote:
>
>> On Wed, 14 Jun 2017 14:13:31 +0200
>> "Štěpán Balážik" <stepan.balazik at nic.cz> wrote:
>>
>> Hi,
>>>
>>> we want to use Augeas to programmatically parse and generate tests for
>>> Deckard <https://gitlab.labs.nic.cz/knot/deckard>[1] (our test harness
>>> for DNS resolvers).
>>>
>>> I wrote a lens
>>> <https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/pyd
>>> nstest/deckard.aug>[2]
>>> which works fine but on longer config files such as this one
>>> <https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/set
>>> s/resolver/iter_badraw.rpl>[3]
>>> the match operation on /scenario/step nodes takes about 3 ms per call on
>>> my machine using both augtool and the python binding. This sums up to
>>> over 10 s for the whole file which too slow for our purposes.
>>>
>>> We have no idea what is the cause of this slowness. Are Augeas matches
>>> always so slow on "shallow and wide" trees?
>>>
>>> Thanks in advance for any help.
>>> Štěpán Balážik @ CZ.NIC
>>>
>>> [1] https://gitlab.labs.nic.cz/knot/deckard
>>> [2]
>>> https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/pydn
>>> stest/deckard.aug
>>> [3]
>>> https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/sets
>>> /resolver/iter_badraw.rpl
>>>
>>> Hi Stepan,
>> we face similar issue when parsing /etc/hosts that are auto-filled by
>> some blocking software ( so around 10k entries there ). In the end solution
>> for us is to introduce caching of match results and creating own map. We
>> use ruby, but I think it is so similar to python, that you get idea.[1]
>> If you need any explanation, do not hesitate to ask. Speed up in our case
>> is approximately half of time spend at parsing.
>>
>> Josef
>>
>> [1] https://github.com/config-files-api/config_files_api/blob/
>> master/lib/cfa/augeas_parser/keys_cache.rb
>>
>
> _______________________________________________
> augeas-devel mailing list
> augeas-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/augeas-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/augeas-devel/attachments/20170614/3c1738d0/attachment.htm>