[augeas-devel] Lens Help: disambiguating 1] regex difference 2] overlapping lenses in union.put

Fri Jan 8 13:24:45 UTC 2016

Hi Yclept,

this is a late reply and you possibly don't need it anymore, but I think I have finally understood the issue regarding "overlapping 
lenses in union.put", so I want to share my experience with you.

As you and David explained, Augeas has trouble when putting a tree into a file that (potentially) has nodes, which can be matched by 
more than one lens. This problem can be avoided in get direction, because parent nodes might be distinguishable in some way. We can 
make use of that key difference, by giving alternate node options to Augeas (as opposed to alternate expressions).

Say we have a tree

/a

/a/color = "blue"

/b

/b/color = "red"

Depending on the lens definitions for nodes "a" and "b", the "color" subnode might have to be read/written differently. Yet just from 
looking at the tree, Augeas cannot distinguish between either "color" node definition. That is what David said in his mail: "when 
Augeas sees a lens construct like 'l1|l2' it only looks at the labels of the tree node it's currently at to decide whether to use l1 
or l2". In this situation, the label is identical for both nodes, hence the conflict.

How can we avoid this problem with actually applying ugly hacks? We change the construct such that Augeas has to consider the depth of 
the tree. Without concrete examples, this will be difficult to understand, so lets expand on the above example with these lens 
definitions:

let a = key "a" . [ label "color" . store /red|blue|green|yellow/ ]

let b = key "b" . [ label "color" . del /dark-/ "dark-" . store /red|blue|green|yellow/ ]

let lns = [ a | b ]

So for "b"-nodes color strings are prefixed with "dark-". When parsing a source file, Augeas can clearly distinguish between when to 
apply lenses "a" or "b". But because in either case the resulting "color"-node is identical, saving the tree to a file fails. We make 
Augeas aware of the difference on node-level, by alternating between nodes:

let lns = [ a ]|[ b ]

I hope this was somewhat understandable and helpful to you (or others finding this mail with Google J ).

Ciao,

Xavier.

From: augeas-devel-bounces at redhat.com [mailto:augeas-devel-bounces at redhat.com] On Behalf Of Yclept Nemo
Sent: Friday, October 23, 2015 7:07 AM
To: augeas-devel
Subject: [augeas-devel] Lens Help: disambiguating 1] regex difference 2] overlapping lenses in union.put

Hi,

I'm working on a lens that has two errors:

 >>  1 (*
 >>  2 # example configuration follows:
 >>  3 # default = value
 >>  4 key = value
 >>  5
 >>  6 # default = value # inline comment
 >>  7 key = value # inline comment
 >>  8 *)
 >>  9
 >> 10 (*
 >> 11 # key/value formats:
 >> 12
 >> 13 keys
 >> 14     [a-z_]
 >> 15
 >> 16 values
 >> 17     boolean:            true/false
 >> 18     int                 0, 100
 >> 19     real                3.0
 >> 20     hex int             ffeedd
 >> 21     string              "..."
 >> 22     unquoted string     sdl, f, f2
 >> 23 *)
 >> 24
 >> 25 (*
 >> 26 # quoted string regex scratchpad
 >> 27 quoted string regex =
 >> 28     good:
 >> 29         "asdf\"qwer"
 >> 30         "asdf\\qwer"
 >> 31         "asdf\\\\\"qwer"
 >> 32     bad:
 >> 33         "asdf\\"qwer"
 >> 34         "asdf\\\\\\"qwer"
 >> 35
 >> 36     /.*/ - /.*(\\)*[^\] <file:///\\)*[%5e\%5d%22.*\> ".*/
 >> 37
 >> 38     /([^"](\")?)*/
 >> 39 *)
 >> 40
 >> 41 module ConfigObj =
 >> 42     let _               = print_regexp /"/
 >> 43     let _               = print_string "\n"
 >> 44     let var0            = "\""
 >> 45     let _               = print_string (var0 . "\n")
 >> 46     let _               = print_string "\n"
 >> 47
 >> 48     let var1            = "\"asdf\\\\\\\\"qwer\""
 >> 49     let _               = print_string (var1 . "\n")
 >> 50     let _               = print_string ((regexp_match (/\".*\"/ - /\"(.*[^\])?([\]{2})*".*\"/) var1) . "\n")
 >> 51
 >> 52     let option          = key /[a-z]([a-z_]*[a-z])?/
 >> 53     let value_bool      = store /(true|false)/ . [ label "type" . value "bool" ]
 >> 54     let value_int       = store /(0|[1-9][0-9]*)/ . [ label "type" . value "int" ]
 >> 55     let value_decimal   = store ((lens_ctype value_int) . /\.[0-9]+/) . [ label "type" . value "decimal" ]
 >> 56     let value_string    = (del "\"" "\"") . (store (/.*/ - /(.*[^\]|)([\]{2})*".*/)) . (del "\"" "\"") . [ label "type" . value 
"string" ]
 >> 57     let value_all       = value_bool | value_int | value_decimal
 >> 58     let sep_assign      = del /[ \t]*=[ \t]*/ " = "
 >> 59     let sep_comment     = del /[ \t]*#[ \t]*/ " # "
 >> 60     let record_comment  = [ label "comment" . sep_comment . store /([^ \t\n].*)?/ ]
 >> 61     let record          = option . sep_assign . value_all . record_comment?
 >> 62     let record_not      = label "comment" . store (/([^ \t\n].*)?/ - lens_ctype record)
 >> 63     let comment         = del /#[ \t]*/ "# " . ( [ record . [ label "commented" ] ] | [ record_not ] )
 >> 64     let line            = del /[ \t]*/ "" . ( comment | [ record ] ) . del "\n" "\n"
 >> 65     let lns             = line *

The first is a regex problem: I want to match double-quoted c-style strings, ie strings with escape sequences such as "\"", "\\\"" and 
so on. The regex I've crafted (line#50) evidently fits the bill - try modifying the input test values (line#48) to verify - so I don't 
understand why the derived lens (line#56) is failing, and thereby causing this "ambiguous concatenation" error:

 >> Syntax error in lens definition
 >> configobj.aug:61.4-.75:Failed to compile record
 >> configobj.aug:61.26-.75:exception: ambiguous concatenation
 >>       First regexp: /([a-z]([a-z_]*[a-z])?)([ \t]*=[ 
\t]*)((((true|false)))|(((0|[1-9][0-9]*)))|(((((0|[1-9][0-9]*)))(\\.[0-9]+)))|((")(([^\n"\\][^\n"\\]*\\\\|\\\\)([^\n\\][^\n"\\]*\\\\)*\\\\(([^\n"\\][^\n"\\]*\\\\|\\\\)([^\n\\][^\n"\\]*\\\\)*\\\\)*(([^\n"\\][^\n"\\]*\\\\|\\\\)([^\n\\][^\n"\\]*\\\\)*([^\n\\][^\n"\\]*|)|[^\n"\\][^\n"\\]*|)|([^\n"\\][^\n"\\]*\\\\|\\\\)([^\n\\][^\n"\\]*\\\\)*([^\n\\][^\n"\\]*|)|[^\n"\\][^\n"\\]*|)(")))/
 >>       Second regexp: /(([ \t]*#[ \t]*)(([^ \t\n].*)?))?/
 >>       'a="\\"#"' can be split into
 >>       'a="\\"|=|#"'
 >>
 >>      and
 >>       'a="\\"#"|=|'  <----- not possible per the regex
 >>
 >>     First lens: configobj.aug:61.26-.57:
 >>     Second lens: configobj.aug:61.60-.75:

The second problem - you'll see if the first is fixed - is a limitation of augeas' tree-text conversion, the inability to 
differentiate trees on depth. This generates the error "exception: overlapping lenses in tree union.put" and crops up in two 
locations:

 '[ record . [ label "commented" ] ]' (line#63) ----vs---- '[ record_not ]' (line#63)
    ::: The "record_not" label of "comment" is also a possible valid label of the "record" lens [line#52]. Hacky workaround is to 
change the label to an invalid key, ie "comment2", etc.
 '[ record . [ label "commented" ] ]' (line#63) ----vs---- '[ record ]' (line#64)
    ::: Unlike previous, no hacky workaround.

Obviously this can be fixed by scanning the *depth* of the tree. In both cases, one branch is guaranteed to have a subnode with the 
label "commented" unlike the other. While augeas doesn't implement this, there was a suggestion on the mailing list regarding a 
lens-based workaround[1] that I didn't understand. I'd appreciate if someone could provide a concrete example for my case.

Sincerely,

[1] https://www.redhat.com/archives/augeas-devel/2009-June/msg00088.html

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/augeas-devel/attachments/20160108/bac427c6/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6401 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/augeas-devel/attachments/20160108/bac427c6/attachment.p7s>