Linux audit performance impact

[Viswanath, Logeswari P (MCOU OSTL)]

Hi Steve,

Thanks for the quick reply.

Please look in-line for my replies.

Regards,
Logeswari.

-----Original Message-----
From: Steve Grubb [mailto:sgrubb at redhat.com] 
Sent: Wednesday, January 28, 2015 8:46 PM
To: linux-audit at redhat.com
Cc: Viswanath, Logeswari P (MCOU OSTL)
Subject: Re: Linux audit performance impact

Hello,

On Wednesday, January 28, 2015 02:57:58 PM Viswanath, Logeswari P wrote:
> We want to know audit performance impact on RHEL and Suse linux to 
> help us evaluate linux audit as data source for our host based IDS. 
> When we ran our own performance test with a test audispd plugin, we 
> found if a system can perform 200000 open/close system calls per 
> second without auditing, system can perform only 3000 open/close 
> system calls auditing is enabled for open/close system call which is a 
> HUGE impact on the system performance. It would be great if anyone can help us answering the following questions.
> 
> 
> 1)      Is this performance impact expected? If yes, what is the reason
> behind it and can we fix it?

I'll leave this for the kernel guys to answer. That said, I think more detailed information might be helpful.

If auditd is not started and events go to syslog, does the performance change? 
To do this audit=1 on boot line and auditctl -R /etc/rules.d/your.rules

Logeswari=>System can perform 15000 open/close system calls per second which is better than earlier results.

what rules do you have loaded?

Logeswari=> # auditctl -l
LIST_RULES: exit,always syscall=open,close
 
What do you get when audit is enabled and no rules loaded?

Logeswari=> Impact is there but not major.

If you have other syscall rules loaded that are not open and openat or close, does the performance change? I suspect that if you trigger a rule, you are thrown onto the slow path. Open is perhaps the most lengthy because of multiple auxiliary records and path resolution. But we need data to tell.

Logeswari=> Yes, there is an major impact. I enabled write system call and this rule is first in the set of rules along with open/close.

That said, I know that the kernel audit path changed a couple years ago so it might be worthwhile to test against an old kernel to see if the change has affected performance.

Logeswari=> We tested with kernel 2.6.32. Should we test with old/new kernel?

-Steve

> 2)      Have anyone done any benchmarking for performance impact? If yes,
> can you please share the numbers and also the steps/programs used the 
> run the same.
> 
> 3)      Help us validating the performance test we have done in our test
> setup using the steps mentioned along with the results attached.
> 
> Attached test program (loader.c) to invoke open and close system calls.
> Attached idskerndsp is the audispd plugin program.
> We used time command to determine how much time the system took to 
> complete
> 50000 open/close system calls without (results attached 
> Without-auditing) and with auditing enabled on the system 
> (With-auditing-NOLOG-audispd-plugin
> and With-auditing-RAW)
> 
> System details:
> 
> 1 CPU machine
> 
> OS Version
> RHEL 6.5
> 
> Kernel Version
> uname -r
> 2.6.32-431.el6.x86_64
> 
> Note: auditd was occupying 35% of CPU and was sleeping for most of the 
> time whereas kauditd was occupying 20% of the CPU.
> 
> Thanks & Regards,
> Logeswari.