Audit format utility

[Steve Grubb]

Hello,

On Sunday, September 28, 2014 02:52:53 PM Burn Alting wrote:
> Had a play around with it. I am not sure about it's value in it's
> current form.

This is why I chose to make it separate for now. Its a strawman for people to 
poke at and see what's wrong before committing to something that will be 
supported long term.


> Rather than specifying the keys to print, it would be better to print
> everything in the event and only 'override' the standard formatting if there
> is an 'snode' for a key.

Sure, perhaps that is a command line option on how to use the format string.


> Further, it has a couple of immediate issues given it's using
> libauparse.
> 
> -  it is "lossy" in that it wont parse poorly formed audit events (see
> the op key value pair below)
>         [burn at swtf auformat]$ cat add_user.txt
>         node=swtf.swtf.dyndns.org type=ADD_USER
>         msg=audit(1411871714.393:47872): user pid=13455 uid=0 auid=500
>         ses=11
>         subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
>         msg='op=adding home directory id=502 exe="/usr/sbin/useradd"
>         hostname=? addr=? terminal=pts/2 res=success'
>         [burn at swtf auformat]$ ./auformat "%node %date %time %milli %
>         serial: type=%TYPE msg=%msg op=%op auid=%auid pid=%pid  path=%
>         path exe=%exe subj=%subj hostname=%hostname terminal=%terminal
>         res=%res\n" add_user.txt
>         swtf.swtf.dyndns.org 09/28/2014 12:35:14 393 47872:
>         type=ADD_USER msg= op=adding auid=500 pid=13455  path=
>         exe="/usr/sbin/useradd"
>         subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
>         hostname=? terminal=pts/2 res=success
>         [burn at swtf auformat]$
> 
> We loose the strings
>     - 'user' before the pid key

Which is meaningless in this case.


>     - op='adding home directory' becomes op'adding'
> This is particularly important for incorrectly formatted application
> level audit sent via auditd.

This is a problem in the shadow-utils package. It is the one that I'm 
currently having to re-do for this reason and many more. Upstream seems to 
have taken a stab at re-doming the audit events and pretty much used it like 
syslog.

 
> - 'rewinding' the event's cursor for each possible key, the call to
> auparse_first_record() in print_item(), is probably not what one would
> want - but then again, auformat is just a mock up at the moment.

Well, if you want your fields in a specific order and its not the order in the 
event, then we have no choice. Note that the event is alrady parsed at this 
point so we are just literally changing the position in a linked list. The 
cost is a series of strcmp calls.

 
> - one looses the parsing 'fix-up' that ausearch does in
> src/ausearch-report.c:output_interpreted_node()

Not sure what "fix-up" we are talking about. The intention is that auparse 
completely mimicks ausearch's interpretation ability (which ausearch was 
switched over to use auparse a few releases back).


> - to build a complete event, having addressed the 'rewinding' issue,
> would make the format look very messy - you would need to include every
> possible key to print all key/values.

If you wanted that, yeah. But I am thinking of cases where one may not want 
every field. For example, you might do something like this to check file access:


# ausearch --start today -m path --raw | 
		auformat 'auid=%AUID res=%SUCCESS name=%NAME\n'


> - one should add event separation so that further tools could process
> the data more easily.

I am thinking of 1 event per line. This is kind of a requirement of Map 
Reduce.


> At the moment, the only tool I'm aware of that 'correctly' parses a log
> file is ausearch. 

If there are omissions in auparse, I really want to know. It must be able to 
correctly parse events.


> Perhaps we would be better served by adding another
> output option to ausearch to print events in a much more parse-able
> format (e.g. XML, JSON)

I am sort of going that way. I am thinking about logstash/elastic search and 
Map reduce and how one might use the audit system when you have say 10,000 
systems.
 
> I am happy to work on this (either extending ausearch or working on
> auformat).

There are a couple needs at the moment are round writing a test suite to 1) 
identify new fields that suddenly show up in a record, 2) locate dangling 
values so they can be fixed.

Also, we need some performance testing and improvements of auparse. Does 
switching to jemalloc make any difference? Is a linked list the best way to do 
it? Can the field searching be smarter?

The auformat work is for now a prototype. I have near term plans to assign 
specific meaning to each event so that events are more understandable. When I 
have this working, then I think we can look at how we want to output the 
event.

-Steve