[augeas-devel] Some ideas about how to use Augeas with IPA

[David Lutterkort]

On Mon, 2008-05-12 at 17:57 -0400, Dmitri Pal wrote:
> I have been evaluating the Augeas library for potential use with IPA 
> (especially for the centralized policy management) and found that though 
> it is a pretty powerful tool there are certain features that would make 
> it much more attractive for the IPA project.

Excellent to hear.

> I will list the requirements with the explanation of the use case .

It would help my understanding of these if you could also explain the
specific situations in which the lack of those features leads to
problems for FreeIPA.

> 1) When setting the value validate the provided data against the regular 
> expression.
> Let us say that we have a file that reads several comma separated values 
> per rule.
> Then the lenses will contain a definition of the "word" as any character 
> that is not a space or a comma. The defined "word" will be used in the 
> parsing rule. If then when use set command to update the values in the 
> entry we provide value that contains comma the set command should fail 
> since the data when saved will violate the lenses grammar and library 
> will fail to parse it back.

In general, it's not possible to perform those checks at any other time
than when aug_save is called (and the caller says implicitly "the tree
should be ok now") The regular expression that a value must match can
change when other nodes are inserted/deleted into the tree. Though it
will be rare, it would be very hard to detect whether the regular
expression can change or not.

Augeas does refuse to modify files if those modifications would result
in a file that is not parseable. (Though there are probably a good
numebr of bugs in that area ;)

> 2) The match command matches the exact value of the label. It would be 
> very useful to be able to do the regular expression searches. This would 
> help if you have to combine two files and want to extract a subset of 
> the entries that matches the regular expression. I also means that there 
> should be a way to iterate through the result set.

You can work around that limitation by putting '*' in the path
expression and filtering the result set according to your needs, i.e. if
you are interested in all nodes matching '/foo/bar/RE/baz' where RE is
some regular expression, run aug_match on '/foo/bar/*/baz' and filter
the resulting set of strings according to your RE.

But you are right, more powerful path expressions is certainly something
that would be interesting.

> 3) Currently the library is bound to the physical file. There should be 
> an option to save/read the data to/from the buffer so that the caller 
> can take advantage of the fact that data can come from different 
> sources. This can be used when parts of the file are stored in the 
> external facility other than the file - for example in the DB or LDAP. 
> Let us imagine that there is an XML data stored in the ldap field.

Are you talking about the file being stored in its entirety in a DB or
LDAP or just some parts of it, e.g. certain key values that you want to
be able to control from your application ?

> The 
> application can connect to LDAP, request the attribute, feed it to 
> Augeas library specifying a read lenses and then save the Augeas tree in 
> a file using a different lenses. It creates a very powerful mechanism.

You can not read a file/string with one lens and write it back with a
different lens - that completely defeats the purpose of lenses. The
original file/string that was used to construct the initial tree is part
of the input of the tree -> file transformation, and if you want to use
lens l1 to go from file/string c1 to the tree, and lens l2 to write the
file back, you'd need to come up with a file/string c2 that would
construct the same intial tree under lens l2.

Can you givea concrete example of where this use case comes from ?

> 4) Managing the order of the entries. Currently I do not see a way of 
> inserting the entries in the middle of the file. Some configuration 
> files have rules that are applied in order so capability of inserting a 
> new entry in the middle is very important.

The aug_insert call in the API lets you insert nodes in the tree in any
order, anywhere. augtool has a command 'ins' that exposes that call. See
tests/rec-hosts-add.rb for an example of how to insert a line in the
middle of /etc/hosts.

> 5) To be able to effectively merge the configuration data from different 
> sources and apply changes to the file it might be very beneficial to be 
> able to sort the entries in the node tree by a specific criteria. This 
> capability will allow a much simpler implementation logic when 2 
> different trees need to be compared and merged.

What specifically do you have in mind there ? As you said, order in
config files matter, and reordering nodes in the tree always corresponds
to changing the file, and often changes the meaning of the file. How
nodes should be sorted is also highly application specific - if it would
simplify matters, we could add a aug_move call to the API to puta node
somewhere else; for the time being, you can simulate that for the most
part by deleting the subtree in the old place and recreating it in the
new position.

> 6) Create a copy of the node tree. If I have a tree that I read from the 
> file and I want to start playing with it in therms of sorting or merging 
> it would be nice to be able create a copy of the tree before starting 
> the procedure so that the original tree can be preserved and would not 
> require re-red from the disk of merge comes to an unresolvable error.

Augeas will not change the underlying file in any way if saving fails
because the tree is invalid. It doesn't (modulo bugs) need to reread the
file to detect that a tree would lead to a file that can't be read in to
produce the exact same tree that is being saved.

If you need extra insurance, you can tell Augeas to save files with the
extension '.augnew' by passing the '-n' option to augtool or the
AUG_SAVE_NEWFILE flag to aug_init. In that case, Augeas will not change
the original file at all, and your application can do the move of
the .augnew file into the right place after doing any additional
validation that might be needed (similar to the '.augnew' mechanism,
there's a mechanism for overwriting the file, but preserving the
original with a '.augsave' extension)

David