Linux audit performance impact

[Richard Guy Briggs]

On 15/02/17, Viswanath, Logeswari P (MCOU OSTL) wrote:
> I agree that changing the formatting of the records could break the existing applications
> that consume them, and I didn't mean changing or eliminating of the formatting completely.
> We agree that formatting is required for logging the records(as buffers) into the log files.
> We are wondering if these records can be made available as RAW records so that the
> analytical programs which are capable of reading them for processing can perform better.

There are tools that completely ignore any of the audit userspace suite
including libaudit, so changing the formatting in the kernel and
deferring to userspace to later do that formatting is not currently an
option.

> This option of RAW mode for the events can be an additional option
> where, kauditd delivers the audit buffer without formatting. Any
> comments on this?

For a transition period if we were to consider it, it would mean
rewriting *all* places in the kernel that generate audit messages and
provide two paths switched on this RAW mode for each one of them, then
copying all that duplication to userspace libaudit.
According to Linus' decree, it would need to remain that way until we
were certain that all tools including ones we don't know about had
switched over.

> >On Monday, February 16, 2015 11:25:57 AM Viswanath, Logeswari P wrote:
> >> I configured the system to audit open system call alone instead of all 
> > >the system calls (our loader program executes) and hence I saw the 
> >> massive improvement in performance. My fix is not causing any change 
> > >in the performance. I wrongly communicated that the fix is causing 
> > >performance improvement. Sorry for that.
> > >
> >> As per the perf data, the format_decode is the function where most of 
> >> the time is spent i.e. formatting the record in the buffer before 
> > >delivering the data to user space. We need to eliminate formatting 
> > >records to increase the performance. Any idea why we need to format 
> > >the record and whether can we add an option (RAW) to deliver the 
> > >record without formatting to user space?
> 
> >Introducing any changes to the format of the record can cause all analytical programs, both open source and proprietary, to stop working correctly. This cannot be changed.
> >
> >I think there is room for improvement however. There are times when strings are being glued together and a stpcpy works just fine. There are times when a numeric hex conversion is being done and %x is very slow. Same with %d.
> >
> >The other issue is that the audit system's philosophy has not been to optimize the formatting of the event, because events _should_ be rare. Meaning that if you are getting hundred of events per second, something is seriously wrong with the rules.
> >
> >It has been optimized to provide as little impact as possible when _not_ generating events. Meaning that we want it as fast as possible in letting the system operate normally.
> >
> >Again, there is room for improvement in both cases of triggering and not triggering events. But the format of events can't really change without a lot of coordination. I have a test suite here:
> >
> >http://people.redhat.com/sgrubb/audit/ausearch-test-0.5.tar.gz
> >
> >That can check that events are searchable by the main audit utility. If changes cause that to fail, then its a sign you'll break the whole world.
> >
> >-Steve
> 
> 

- RGB

--
Richard Guy Briggs <rbriggs at redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545