[augeas-devel] Language revamp

[David Lutterkort]

The last todo item I want to tackle before announcing Augeas more widely
is revamp the language; what I have in mind follows Boomerang pretty
closely, which is ultimately inspired by OCaml.

One of the thigns I intensely dislike about the current state of affairs
is that it is impossible to reuse a lot of things. There's only so many
ways you can express "lines that start with a '#' are comments" as a
regexp, yet you can't reuse that from one grammar to the next.
Similarly, if you already have a regexp matching a 'word' defined, you
can't use that in a regexp that matches 'word enclosed in brackets', and
have to start from scratch - have a look at cmfm.aug for the insanity
that leads to.

The basic idea is that the language should become a simple functional
language, with primitive types 'string', 'regexp' and 'lens', some
builtin operators on those types, and the possibility to define new
operations (either simple values or functions) and use them just the
same as the builtin operations.

As an example, the file hosts.aug would look like this under the new
syntax (explanations below):

        module Hosts
        
          let sep (d:string):lens = del /[ \t]+/ d
          let sep_tab = sep "\t"
          let sep_spc = sep " "
        
          let del_str (s:string):lens = del s s
          let eol = del_str "\n"
        
          let comment = del /#.*\n/ "# "
          let word = /[^# \n\t]+/
          let record = [ seq "line" . [ label "ipaddr" . store word ] . sep_tab .
                                      [ label "canonical" . store word ] .
                                      [ label "aliases . 
                                        ( [ seq "aliases" . sep_spc . store word] ) * 
                                      ] .
                         eol ]
        
          let lens = ( comment | record ) *
        
          let files = apply lens (include "/etc/hosts" "/system/config/hosts")
        
        end
        
        autorun Hosts.files

Some explanations:
      * A 'module' is a simple grouping of names; for now, I don't think
        there's a need for nesting modules. It's just a way to keep
        names separated (so that you can have a Sshd.lens and a
        Hosts.lens)
      * String literals are enclosed in "", and regexps in //; most of
        the lens combinators look the same as in the present syntax, but
        I'd like to extend some of them to also work for strings and
        regexps so that you can say 're1 . re2' for the regexp that
        matches the concatenation of re1 and re2. Similar for 're1 |
        re2' and concatenating strings
      * The 'del' lens is new; deleting entries from the input (i.e.,
        not including them in the tree) now needs to be more explicit,
        so that a default value can be included - the default was
        previously part of the token definition.
      * The 'let' creates a new name for something. Functions need to
        declare their arguments, and the types of their arguments plus
        the return type; the 'let sep' says "sep is a function which
        takes a string and returns a lens"
      * Names need to be defined before they can be used; this also
        precludes recursion (allowing recursion would make it possible
        to write down lenses that process context-free languages, not
        just regular ones; that leads to a whole host of problems best
        left to some point in the future)

I also want to add a simple unit test facility, so that you can say
right in your .aug file something like
        test Hosts.lens put
          "127.0.0.1 localhost"
        after
          rm /0
        = ""
with the general structure "test LENS put S after COMMANDS = S'" where S
and S' are strings and COMMANDS is a list of augtool-like commands to
manipulate the tree. The test would parse S into a tree, run COMMANDS on
it and check that the result of turning the tree back into a string is
S'.

The stickiest bit about all this is how to deal with mapping files into
the tree, and how to indicate what Augeas should do by default (e.g.,
when augtool starts up) For now, the 'let files' in the above example
says "apply the lens LENS (defined on the previous line) to the contents
of file "/etc/hosts" and put the result into the tree under
"/system/config/hosts". I am not entirely clear how that should work
with globbing (like pam.aug does right now), and if files need to be
created (think of creating a new .repo file when you create a brandnew
subtree under /system/config/yum) 

Eventually, I'd also like to be able to process include directives in
files through some special lens, so that you can say "take the string
that match this regexp, read the file with that name, process it with
lens X and put the resulting tree here in the global tree"

The 'autorun Hosts.apply' is an attempt to make it explicit what Augeas
should do in standalone mode (e.g., when augtool is started without any
special instructions of what files to process - you can't do that right
now, but I'd like to add that) The 'autorun Hosts.apply' says "Remember
the Hosts.apply lens on some internal list, and if you don't know what
else to do, run all the lenses from that list"

Since this is going to be a good chunk of work, it'll be a while before
it's all implemented. I'd appreciate any comments, especially on whether
this would be a change for the better, whether you think writing file
format descriptions for Augeas will become more understandable, easier,
quicker, more maintainable etc.

David