BIG performance hit with auditd on large systems (>64 CPUs)

Sat May 20 07:18:13 UTC 2017

Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan <stephenwb at gmail.com>:
>Agree with Steve's suggestion re: "-S all". Also might help if you sort
>your rules to put all the ones with '-F auid>=400' below a single line
>rule
>like this:
>-a never,exit -F auid<400
>
>and remove the '-F auid>=400' from all of the rules below it.
>
>Like so:
>-a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F
>key=USER_EXEC
>-a always,exit -F arch=b64 -S execve -F auid>=5000000 -F
>auid!=4294967295
>-F key=USER_EXEC
>-a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
>auid>=5000000 -F auid!=4294967295 -F key=S3DATA
>
>-a never,exit -F auid<400
>-a always,exit -F path=/etc/environment -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/login.defs -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/rsyslog.conf -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/ssh/sshd_config -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/cron.allow -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/cron.deny -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/cron.d -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/cron.daily -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/cron.hourly -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/cron.monthly -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/cron.weekly -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/aliases -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/alternatives -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/at.allow -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/at.deny -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F
>key=CRIT_AUDIT
>-a always,exit -F path=/etc/audisp/audispd.conf -F perm=wa -F
>key=CRIT_AUDIT
>-a always,exit -F path=/etc/audit/auditd.conf -F perm=wa -F
>key=CRIT_AUDIT
>-a always,exit -F path=/etc/bashrc -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/crontab -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/shells -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/default -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/depmod.conf -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/depmod.d -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/exports -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/group -F perm=wa -F key=USER_MGMT
>-a always,exit -F path=/etc/passwd -F perm=wa -F key=USER_MGMT
>-a always,exit -F path=/etc/shadow -F perm=wa -F key=USER_MGMT
>-a always,exit -F path=/etc/inittab -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/bin -F perm=wa -F key=CRIT_PROG
>-a always,exit -F dir=/sbin -F perm=wa -F key=CRIT_PROG
>-a always,exit -F dir=/usr/bin -F perm=wa -F key=CRIT_PROG
>-a always,exit -F dir=/usr/sbin -F perm=wa -F key=CRIT_PROG
>-a always,exit -F dir=/etc/init.d -F perm=wa -F key=CRIT_PROG
>-a always,exit -F path=/etc/nsswitch.conf -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/ldap.conf -F perm=wa -F key=USER_MGMT
>-a always,exit -F path=/etc/sssd/sssd.conf -F perm=wa -F key=USER_MGMT
>-a always,exit -F dir=/var/spool/cron -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/var/spool/atjobs -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/usr/bin/sudo -F perm=x -F key=USER_MGMT
>-a always,exit -F path=/etc/sudoers -F perm=wa -F key=USER_MGMT
>-a always,exit -F dir=/etc/sudoers.d -F perm=wa -F key=USER_MGMT
>-a always,exit -F dir=/etc/pam.d -F perm=wa -F key=CRIT_PAM
>-a always,exit -F dir=/etc/security -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/libaudit.conf -F perm=wa -F key=CRIT_AUDIT
>-a always,exit -F path=/etc/init.d/auditd -F perm=wa -F key=CRIT_AUDIT
>-a always,exit -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid<10000
>-F
>auid!=4294967295 -F key=S3DATA
>
>
>On Fri, May 19, 2017 at 4:52 PM Klaus Lichtenwalder
><klic at mnet-online.de>
>wrote:
>
>> Hi,
>>
>> we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74
>> CPUs and >= 400G RAM.
>> When the system is busy with large SAP jobs, it goes onto its knees
>with
>> cpu %system up to 80%, thus making the SAP jobs run twice as long. As
>> soon as you stop auditd everything returns to normal...
>>
>> Facts:
>> RHEL6 instances on RHEL7 hosts.
>> the rule set (see below) runs fine on any other system with less cpus
>> (<64, maybe this is the cut off?). We have smaller systems with this
>> rule set that rotate the audit file nearly every minute without any
>> noticable performance hit, these SAP systems rotate once every
>> 20-24hours....
>>
>> Anyone has an idea?
>>
>> Here's an excerpt from "perf top":
>> with auditd running:
>>
>> > Samples: 28M of event 'cpu-clock', Event count (approx.):
>236747914918
>> > Overhead Shared Object Symbol
>> > 23.13% [kernel] [k] get_task_cred
>> > 10.05% [kernel] [k] audit_filter_rules
>> > 4.21% [kernel] [k] _spin_unlock_irqrestore
>> > 3.30% libdb2e.so.1 [.] sqlbfix
>> > 2.92% [kernel] [k] finish_task_switch
>> > 1.69% disp+work [.] rrol_in
>> > 1.69% disp+work [.] rrol_out
>> > 0.98% [kernel] [k] run_timer_softirq
>> > 0.96% [kernel] [k] rcu_process_gp_end
>> >
>>
>> auditd stopped:
>>
>> > Samples: 3M of event 'cpu-clock', Event count (approx.):
>526535382557
>> > Overhead Shared Object Symbol
>> > 2.41% disp+work [.] memcmpU16
>> > 2.32% disp+work [.] MmxMalloc2
>> > 2.25% disp+work [.] ab_Rudi
>> > 2.07% disp+work [.] rrol_out
>> > 1.98% disp+work [.] rrol_in
>> > 1.95% disp+work [.] ab_CompByCmpCntx
>> > 1.88% libdb2e.so.1 [.] sqlbfix
>> > 1.73% disp+work [.] MmxFree2
>> > 1.62% [kernel] [k] run_timer_softirq
>> > 1.56% [kernel] [k] __do_softirq
>> > 1.39% disp+work [.] ab_InitRcDecompress
>> >
>> > These are the audit rules:
>> > auditctl -l
>> > -a always,exit -S all -F path=/etc/environment -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F
>> auid>=400 -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F
>> perm=wa -F auid>=400 -F key=CRIT_AUDIT
>> > -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa
>-F
>> auid>=400 -F key=CRIT_AUDIT
>> > -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F
>> auid>=400 -F key=CRIT_AUDIT
>> > -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F
>> key=USER_MGMT
>> > -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400
>-F
>> key=USER_MGMT
>> > -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400
>-F
>> key=USER_MGMT
>> > -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F
>> key=CRIT_PROG
>> > -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F
>> key=CRIT_PROG
>> > -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F
>> key=CRIT_PROG
>> > -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F
>> key=CRIT_PROG
>> > -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F
>> key=CRIT_PROG
>> > -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F
>auid>=400 -F
>> key=USER_MGMT
>> > -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F
>> auid>=400 -F key=USER_MGMT
>> > -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400
>-F
>> key=USER_MGMT
>> > -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400
>-F
>> key=USER_MGMT
>> > -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400
>-F
>> key=USER_MGMT
>> > -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F
>> key=USER_EXEC
>> > -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1
>-F
>> key=USER_EXEC
>> > -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F
>> key=CRIT_PAM
>> > -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F
>auid>=400
>> -F key=CRIT_AUDIT
>> > -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F
>auid>=400
>> -F key=CRIT_AUDIT
>> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
>> auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA
>> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
>> auid>=5000000 -F auid!=-1 -F key=S3DATA
>> >
>>
>> --
>>
>------------------------------------------------------------------------
>>  Klaus Lichtenwalder, Dipl. Inform.,  http://www.lichtenwalder.name/
>>  PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA
>0980
>>
>> --
>> Linux-audit mailing list
>> Linux-audit at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-audit
>>

Thanks everybody for these valuable insights!

@Paul: it is in the support hands of the distribution provider, but there were reasons to also go here... I still have a strong indication of a problematic situation with many cpus, maybe you can make something out of it, and these tips only popped up here on this great list

Klaus
-- 
Mit K9 vom Telefon gesendet. Tippfehler und komische Worte darf der Empfänger behalten
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-audit/attachments/20170520/6e058f8f/attachment.htm>