Linux audit performance impact

[Steve Grubb]

On Tuesday, February 17, 2015 01:10:21 PM Viswanath, Logeswari P wrote:
> I agree that changing the formatting of the records could break the existing
> applications  that consume them, and I didn't mean changing or eliminating
> of the formatting completely. We agree that formatting is required for
> logging the records(as buffers) into the log files. We are wondering if
> these records can be made available as RAW records so that the analytical
> programs which are capable of reading them for processing can perform
> better.

There are no analytical programs that can consume them. :-)  I'd like to see 
exactly what the bottleneck was and the correction you made. Again, this is an 
optimization for something that should rarely happen. Or if it does, its less 
than 10 a second. Additionally, the open use case is about the worst 
performing one besides connect or accept because of the large amounts of data 
that could be generated. Also, kill can generate 1000's of records in one 
syscall.

So, I'd like to see what was optimized to see if you tweaked just this one 
syscall and how different it might be for analytical programs.

-Steve


> This option of RAW mode for the events can be an additional option
> where, kauditd delivers the audit buffer without formatting. Any comments
> on this?
> 
> 
> >On Monday, February 16, 2015 11:25:57 AM Viswanath, Logeswari P wrote:
> >
> >> I configured the system to audit open system call alone instead of all 
> >> 
> > >the system calls (our loader program executes) and hence I saw the 
> > >
> >> massive improvement in performance. My fix is not causing any change 
> >> 
> > >in the performance. I wrongly communicated that the fix is causing 
> > >performance improvement. Sorry for that.
> > >
> > >
> >> As per the perf data, the format_decode is the function where most of 
> >> the time is spent i.e. formatting the record in the buffer before 
> >> 
> > >delivering the data to user space. We need to eliminate formatting 
> > >records to increase the performance. Any idea why we need to format 
> > >the record and whether can we add an option (RAW) to deliver the 
> > >record without formatting to user space?
> 
> 
> 
> >Introducing any changes to the format of the record can cause all
> >analytical programs, both open source and proprietary, to stop working
> >correctly. This cannot be changed.
 
> >I think there is room for improvement however. There are times when strings
> >are being glued together and a stpcpy works just fine. There are times
> >when a numeric hex conversion is being done and %x is very slow. Same with
> >%d.
 
> >The other issue is that the audit system's philosophy has not been to
> >optimize the formatting of the event, because events _should_ be rare.
> >Meaning that if you are getting hundred of events per second, something is
> >seriously wrong with the rules.
 
> >It has been optimized to provide as little impact as possible when _not_
> >generating events. Meaning that we want it as fast as possible in letting
> >the system operate normally.
 
> >Again, there is room for improvement in both cases of triggering and not
> >triggering events. But the format of events can't really change without a
> >lot of coordination. I have a test suite here:
 
> >http://people.redhat.com/sgrubb/audit/ausearch-test-0.5.tar.gz
> >
> >That can check that events are searchable by the main audit utility. If
> >changes cause that to fail, then its a sign you'll break the whole world.
> >
> >-Steve
> 
> 
>