BIG performance hit with auditd on large systems (>64 CPUs)

Fri May 19 21:41:58 UTC 2017

Agree with Steve's suggestion re: "-S all". Also might help if you sort
your rules to put all the ones with '-F auid>=400' below a single line rule
like this:
-a never,exit -F auid<400

and remove the '-F auid>=400' from all of the rules below it.

Like so:
-a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F
key=USER_EXEC
-a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=4294967295
-F key=USER_EXEC
-a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
auid>=5000000 -F auid!=4294967295 -F key=S3DATA

-a never,exit -F auid<400
-a always,exit -F path=/etc/environment -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/login.defs -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/rsyslog.conf -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/ssh/sshd_config -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/cron.allow -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/cron.deny -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/cron.d -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/cron.daily -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/cron.hourly -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/cron.monthly -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/cron.weekly -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/aliases -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/alternatives -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/at.allow -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/at.deny -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F
key=CRIT_AUDIT
-a always,exit -F path=/etc/audisp/audispd.conf -F perm=wa -F key=CRIT_AUDIT
-a always,exit -F path=/etc/audit/auditd.conf -F perm=wa -F key=CRIT_AUDIT
-a always,exit -F path=/etc/bashrc -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/crontab -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/shells -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/default -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/depmod.conf -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/depmod.d -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/exports -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/group -F perm=wa -F key=USER_MGMT
-a always,exit -F path=/etc/passwd -F perm=wa -F key=USER_MGMT
-a always,exit -F path=/etc/shadow -F perm=wa -F key=USER_MGMT
-a always,exit -F path=/etc/inittab -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/bin -F perm=wa -F key=CRIT_PROG
-a always,exit -F dir=/sbin -F perm=wa -F key=CRIT_PROG
-a always,exit -F dir=/usr/bin -F perm=wa -F key=CRIT_PROG
-a always,exit -F dir=/usr/sbin -F perm=wa -F key=CRIT_PROG
-a always,exit -F dir=/etc/init.d -F perm=wa -F key=CRIT_PROG
-a always,exit -F path=/etc/nsswitch.conf -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/ldap.conf -F perm=wa -F key=USER_MGMT
-a always,exit -F path=/etc/sssd/sssd.conf -F perm=wa -F key=USER_MGMT
-a always,exit -F dir=/var/spool/cron -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/var/spool/atjobs -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/usr/bin/sudo -F perm=x -F key=USER_MGMT
-a always,exit -F path=/etc/sudoers -F perm=wa -F key=USER_MGMT
-a always,exit -F dir=/etc/sudoers.d -F perm=wa -F key=USER_MGMT
-a always,exit -F dir=/etc/pam.d -F perm=wa -F key=CRIT_PAM
-a always,exit -F dir=/etc/security -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/libaudit.conf -F perm=wa -F key=CRIT_AUDIT
-a always,exit -F path=/etc/init.d/auditd -F perm=wa -F key=CRIT_AUDIT
-a always,exit -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid<10000 -F
auid!=4294967295 -F key=S3DATA

On Fri, May 19, 2017 at 4:52 PM Klaus Lichtenwalder <klic at mnet-online.de>
wrote:

> Hi,
>
> we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74
> CPUs and >= 400G RAM.
> When the system is busy with large SAP jobs, it goes onto its knees with
> cpu %system up to 80%, thus making the SAP jobs run twice as long. As
> soon as you stop auditd everything returns to normal...
>
> Facts:
> RHEL6 instances on RHEL7 hosts.
> the rule set (see below) runs fine on any other system with less cpus
> (<64, maybe this is the cut off?). We have smaller systems with this
> rule set that rotate the audit file nearly every minute without any
> noticable performance hit, these SAP systems rotate once every
> 20-24hours....
>
> Anyone has an idea?
>
> Here's an excerpt from "perf top":
> with auditd running:
>
> > Samples: 28M of event 'cpu-clock', Event count (approx.): 236747914918
> > Overhead Shared Object Symbol
> > 23.13% [kernel] [k] get_task_cred
> > 10.05% [kernel] [k] audit_filter_rules
> > 4.21% [kernel] [k] _spin_unlock_irqrestore
> > 3.30% libdb2e.so.1 [.] sqlbfix
> > 2.92% [kernel] [k] finish_task_switch
> > 1.69% disp+work [.] rrol_in
> > 1.69% disp+work [.] rrol_out
> > 0.98% [kernel] [k] run_timer_softirq
> > 0.96% [kernel] [k] rcu_process_gp_end
> >
>
> auditd stopped:
>
> > Samples: 3M of event 'cpu-clock', Event count (approx.): 526535382557
> > Overhead Shared Object Symbol
> > 2.41% disp+work [.] memcmpU16
> > 2.32% disp+work [.] MmxMalloc2
> > 2.25% disp+work [.] ab_Rudi
> > 2.07% disp+work [.] rrol_out
> > 1.98% disp+work [.] rrol_in
> > 1.95% disp+work [.] ab_CompByCmpCntx
> > 1.88% libdb2e.so.1 [.] sqlbfix
> > 1.73% disp+work [.] MmxFree2
> > 1.62% [kernel] [k] run_timer_softirq
> > 1.56% [kernel] [k] __do_softirq
> > 1.39% disp+work [.] ab_InitRcDecompress
> >
> > These are the audit rules:
> > auditctl -l
> > -a always,exit -S all -F path=/etc/environment -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F
> auid>=400 -F key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F
> perm=wa -F auid>=400 -F key=CRIT_AUDIT
> > -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa -F
> auid>=400 -F key=CRIT_AUDIT
> > -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F
> auid>=400 -F key=CRIT_AUDIT
> > -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F
> key=CRIT_PROG
> > -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F
> key=CRIT_PROG
> > -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F
> key=CRIT_PROG
> > -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F
> key=CRIT_PROG
> > -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F
> key=CRIT_PROG
> > -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F
> auid>=400 -F key=USER_MGMT
> > -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F
> key=USER_EXEC
> > -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1 -F
> key=USER_EXEC
> > -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F
> key=CRIT_PAM
> > -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F auid>=400
> -F key=CRIT_AUDIT
> > -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F auid>=400
> -F key=CRIT_AUDIT
> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
> auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA
> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
> auid>=5000000 -F auid!=-1 -F key=S3DATA
> >
>
> --
> ------------------------------------------------------------------------
>  Klaus Lichtenwalder, Dipl. Inform.,  http://www.lichtenwalder.name/
>  PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA 0980
>
> --
> Linux-audit mailing list
> Linux-audit at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-audit/attachments/20170519/39c60521/attachment.htm>