[augeas-devel] Some ideas about how to use Augeas with IPA

[Dmitri Pal]

David Lutterkort wrote:
> On Mon, 2008-05-12 at 17:57 -0400, Dmitri Pal wrote:
>   
>> I have been evaluating the Augeas library for potential use with IPA 
>> (especially for the centralized policy management) and found that though 
>> it is a pretty powerful tool there are certain features that would make 
>> it much more attractive for the IPA project.
>>     
>
> Excellent to hear.
>
>   
>> I will list the requirements with the explanation of the use case .
>>     
>
> It would help my understanding of these if you could also explain the
> specific situations in which the lack of those features leads to
> problems for FreeIPA.
>
>   
I did my best below so if you have any specific questions please let me 
know.
>> 1) When setting the value validate the provided data against the regular 
>> expression.
>> Let us say that we have a file that reads several comma separated values 
>> per rule.
>> Then the lenses will contain a definition of the "word" as any character 
>> that is not a space or a comma. The defined "word" will be used in the 
>> parsing rule. If then when use set command to update the values in the 
>> entry we provide value that contains comma the set command should fail 
>> since the data when saved will violate the lenses grammar and library 
>> will fail to parse it back.
>>     
>
> In general, it's not possible to perform those checks at any other time
> than when aug_save is called (and the caller says implicitly "the tree
> should be ok now") The regular expression that a value must match can
> change when other nodes are inserted/deleted into the tree. Though it
> will be rare, it would be very hard to detect whether the regular
> expression can change or not.
>
> Augeas does refuse to modify files if those modifications would result
> in a file that is not parseable. (Though there are probably a good
> numebr of bugs in that area ;)
>
>   
I am talking about checking the value at the moment of the set command 
or function.
The label and the token that go after it in the file (when read from 
file) are matched with the regular expression.
Use the same regular expression to check that the data that is currently 
set for this label matches the regular expression defined for this label 
in lenses.
>> 2) The match command matches the exact value of the label. It would be 
>> very useful to be able to do the regular expression searches. This would 
>> help if you have to combine two files and want to extract a subset of 
>> the entries that matches the regular expression. I also means that there 
>> should be a way to iterate through the result set.
>>     
>
> You can work around that limitation by putting '*' in the path
> expression and filtering the result set according to your needs, i.e. if
> you are interested in all nodes matching '/foo/bar/RE/baz' where RE is
> some regular expression, run aug_match on '/foo/bar/*/baz' and filter
> the resulting set of strings according to your RE.
>
> But you are right, more powerful path expressions is certainly something
> that would be interesting.
>
>   
I am talking about the regular expression in the <value> field not in 
the <path> field.
>> 3) Currently the library is bound to the physical file. There should be 
>> an option to save/read the data to/from the buffer so that the caller 
>> can take advantage of the fact that data can come from different 
>> sources. This can be used when parts of the file are stored in the 
>> external facility other than the file - for example in the DB or LDAP. 
>> Let us imagine that there is an XML data stored in the ldap field.
>>     
>
> Are you talking about the file being stored in its entirety in a DB or
> LDAP or just some parts of it, e.g. certain key values that you want to
> be able to control from your application ?
>   
File can be stored in some form for example each node on the server can 
be a database record or LDAP attribute or XML file.
The records or attributes or files that only make sense in the context 
of the specific machine will be selected and send over to the client to 
be stored in the actual file. Allowing library to spit the data for a 
node of the whole tree into the memory buffer using a provided lenses 
would solve the problem. Same is with reading data from buffer.
>   
>> The 
>> application can connect to LDAP, request the attribute, feed it to 
>> Augeas library specifying a read lenses and then save the Augeas tree in 
>> a file using a different lenses. It creates a very powerful mechanism.
>>     
>
> You can not read a file/string with one lens and write it back with a
> different lens - that completely defeats the purpose of lenses. The
> original file/string that was used to construct the initial tree is part
> of the input of the tree -> file transformation, and if you want to use
> lens l1 to go from file/string c1 to the tree, and lens l2 to write the
> file back, you'd need to come up with a file/string c2 that would
> construct the same intial tree under lens l2.
>
>   
The tree that you have is just an ordered list of the value pairs. If I 
have two different lenses that can save on and the same data in 
different formats and read it back my problem is solved. This would 
allow reading data from the database/LDAP/XML; constructing a new tree 
from pieces; saving the tree into the buffer using a data transit 
lenses; receiving data on the client; restoring tree using data transit 
lenses and saving the data into the file using the format specific to 
the file.
> Can you givea concrete example of where this use case comes from ?
>
>   
>> 4) Managing the order of the entries. Currently I do not see a way of 
>> inserting the entries in the middle of the file. Some configuration 
>> files have rules that are applied in order so capability of inserting a 
>> new entry in the middle is very important.
>>     
>
> The aug_insert call in the API lets you insert nodes in the tree in any
> order, anywhere. augtool has a command 'ins' that exposes that call. See
> tests/rec-hosts-add.rb for an example of how to insert a line in the
> middle of /etc/hosts.
>   
But you have to use the number larger than the number you already have 
in the tree, right? This adds the node to the end of the of the list and 
I do not see a way to add thing in the middle. Am I missing something 
and insert with existing number will add a new entry and shift others?
I have not tried that.
>   
>> 5) To be able to effectively merge the configuration data from different 
>> sources and apply changes to the file it might be very beneficial to be 
>> able to sort the entries in the node tree by a specific criteria. This 
>> capability will allow a much simpler implementation logic when 2 
>> different trees need to be compared and merged.
>>     
>
> What specifically do you have in mind there ? As you said, order in
> config files matter, and reordering nodes in the tree always corresponds
> to changing the file, and often changes the meaning of the file. How
> nodes should be sorted is also highly application specific - if it would
> simplify matters, we could add a aug_move call to the API to puta node
> somewhere else; for the time being, you can simulate that for the most
> part by deleting the subtree in the old place and recreating it in the
> new position.
>
>   
When you have to merge two lists the best way is to sort them and use 
traverse once.
>> 6) Create a copy of the node tree. If I have a tree that I read from the 
>> file and I want to start playing with it in therms of sorting or merging 
>> it would be nice to be able create a copy of the tree before starting 
>> the procedure so that the original tree can be preserved and would not 
>> require re-red from the disk of merge comes to an unresolvable error.
>>     
>
> Augeas will not change the underlying file in any way if saving fails
> because the tree is invalid. It doesn't (modulo bugs) need to reread the
> file to detect that a tree would lead to a file that can't be read in to
> produce the exact same tree that is being saved.
>
> If you need extra insurance, you can tell Augeas to save files with the
> extension '.augnew' by passing the '-n' option to augtool or the
> AUG_SAVE_NEWFILE flag to aug_init. In that case, Augeas will not change
> the original file at all, and your application can do the move of
> the .augnew file into the right place after doing any additional
> validation that might be needed (similar to the '.augnew' mechanism,
> there's a mechanism for overwriting the file, but preserving the
> original with a '.augsave' extension)
>
>   
I am trying to discourage the tool from being that file centric. It will 
be very convenient to use the library for the format transformation in 
the memory. The data being stored after some filtering, sorting, merging 
, serializing, deserializing ends up in the file in the order and format 
we need. With very minor changes (I think) but related to the 
disconnecting library from the file we can make it applicable to a much 
broader group of tasks that it was not originally intended for (IPA). 
Otherwise we would have to re-implement a lot of logic and functionality 
that library currently does but does not expose to the outside world.  

> David
>
>
>   


-- 
Dmitri Pal
Engineering Manager
Red Hat Inc.