auditd and redhat cluster

Tue Mar 1 13:57:45 UTC 2016

The kernel is  : 2.6.32-573.12.1.el6.x86_64
And the whole audit.rules file is  :
-D
-i
-b 8192
-a exit,never -F arch=b32 -F dir=/tmp/
-a exit,never -F arch=b64 -F dir=/tmp/
-a exit,never -F arch=b32 -F dir=/dev/shm/
-a exit,never -F arch=b64 -F dir=/dev/shm/
-a exit,never -F arch=b32 -F dir=/var/lock/lvm/
-a exit,never -F arch=b64 -F dir=/var/lock/lvm/
-w /sbin/agetty -p x -k console_access
-w /sbin/mingetty -p x -k console_access
-w /var/log/audit/ -k audit_logs
-w /var/log/secure -k audit_logs
-a exit,always -F arch=b32 -S creat -S mkdir -S mknod -S link -S symlink -S mkdirat -S mknodat -S linkat -S symlinkat -F uid=root -F success=1 -k creation
-a exit,always -F arch=b64 -S creat -S mkdir -S mknod -S link -S symlink -S mkdirat -S mknodat -S linkat -S symlinkat -F uid=root -F success=1 -k creation
-a exit,always -F arch=b32 -S rmdir -S unlink -S unlinkat -F uid=root -F success=1 -k deletion
-a exit,always -F arch=b64 -S rmdir -S unlink -S unlinkat -F uid=root -F success=1 -k deletion
-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S stime -k time_change
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time_change
-w /etc/localtime -p wa -k time_change
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity
-w /etc/cron.allow -p wa -k system_files
-w /etc/ntp.conf -p wa -k system_files
-w /etc/ssh/sshd_config -p wa -k system_files
-w /etc/hosts -p wa -k system_files
-w /etc/resolv.conf -p wa -k system_files
-w /etc/audit.rules -p wa -k system_files
-w /etc/auditd.conf -p wa -k system_files
-w /etc/rsyslog.conf -p wa -k system_files
-a exit,always -F arch=b32 -S sethostname -k system_locale
-a exit,always -F arch=b64 -S sethostname -k system_locale
-w /etc/issue -p wa -k system_locale
-w /etc/issue.net -p wa -k system_locale
-w /etc/hosts -p wa -k system_locale
-w /etc/sysconfig/network -p wa -k system_locale
-w /etc/sudoers -p wa -k actions
-w /root/.ssh/authorized_keys -p wa -k ssh_files
-w /home/admnet/.ssh/authorized_keys -p wa -k ssh_files
-w /home/system/.ssh/authorized_keys -p war -k ssh_files
-w /home/oper/.ssh/authorized_keys -p wa -k ssh_files
-w /home/sprod/.ssh/authorized_keys -p wa -k ssh_files
-w /home/www/.ssh/authorized_keys -p wa -k ssh_files
-w /home/integ/.ssh/authorized_keys -p wa -k ssh_files
-w /home/stat/.ssh/authorized_keys -p wa -k ssh_files
-w /home/reference/.ssh/authorized_keys -p wa -k ssh_files
-w /bin/chown -p x -k system_commands
-w /usr/local/sbin/tcpdump -p x -k system_commands
-w /usr/bin/passwd -p x -k system_commands
-w /usr/sbin/useradd -p x -k system_commands
-w /usr/sbin/usermod -p x -k system_commands
-w /bin/chgrp -p x -k system_commands
-w /sbin/route -p x -k system_commands
-w /sbin/shutdown -p x -k system_commands
-w /sbin/reboot -p x -k system_commands
-w /sbin/sysctl -p x -k system_commands
-w /sbin/ifconfig -p x -k system_commands
-w /usr/sbin/visudo -p x -k system_commands
-w /usr/bin/crontab -p x -k system_commands
-w /bin/chmod -p x -k system_commands
-w /bin/su -p x -k system_commands
-w /bin/env -p x -k system_commands
-w /sbin/auditctl -p x -k system_commands
-w /bin/mount -p x -k system_commands
-w /bin/umount -p x -k system_commands
-w /bin/ping6 -p x -k system_commands
-w /bin/ping -p x -k system_commands
-w /sbin/pam_timestamp_check -p x -k system_commands
-w /sbin/netreport -p x -k system_commands
-w /sbin/unix_chkpwd -p x -k system_commands
-w /sbin/mount.nfs -p x -k system_commands
-w /sbin/rmmod -p x -k modules
-w /sbin/modprobe -p x -k modules
-a exit,always -F arch=b64 -S init_module -S delete_module -k modules
-a exit,always -F arch=b32 -S init_module -S delete_module -k modules
-a exit,always -F arch=b64 -S open -S openat -F exit=-EPERM -k rights
-a exit,always -F arch=b32 -S open -S openat -F exit=-EPERM -k rights
-a exit,always -F arch=b64 -S ptrace -k info_scan
-a exit,always -F arch=b32 -S ptrace -k info_scan

During the hour preceding the fence we got  these events from the passive node
Key Summary Report
===========================
total  key
===========================
891  system_commands (ping)

And on the active node  :

Key Summary Report
===========================
total  key
===========================
1330  system_commands
286  deletion

I am going to follow your advice and to open a call with redhat.
Anyway, I am interested in knowing if auditd has been reported to cause trouble without generating many events.

Regards
Philippe

-----Message d'origine-----
De : Paul Moore [mailto:paul at paul-moore.com]
Envoyé : mardi 1 mars 2016 14:25
À : Maupertuis Philippe
Cc : linux-audit at redhat.com
Objet : Re: auditd and redhat cluster

On Mon, Feb 29, 2016 at 7:45 AM, Maupertuis Philippe <philippe.maupertuis at worldline.com> wrote:
> Hi list,
>
> One clusters fenced the passive node around two hours  after auditd
> was started.
>
> We have found that iowait has increased since auditd was started and
> was unusually high.
>
> Auditd wasn’t generating many messages and there were no noticeable
> added activity on the disk were the audit and syslog files were written.
>
> Besides watches, the only general rules were :
>
> # creation
> -a exit,always -F arch=b32 -S creat -S mkdir -S mknod -S link -S
> symlink -S mkdirat -S mknodat -S linkat -S symlinkat -F uid=root -F
> success=1 -k creation -a exit,always -F arch=b64 -S creat -S mkdir -S
> mknod -S link -S symlink -S mkdirat -S mknodat -S linkat -S symlinkat
> -F uid=root -F success=1 -k creation
>
> # deletion
> -a exit,always -F arch=b32 -S rmdir -S unlink -S unlinkat -F uid=root
> -F
> success=1 -k deletion
> -a exit,always -F arch=b64 -S rmdir -S unlink -S unlinkat -F uid=root
> -F
> success=1 -k deletion
>
> After the rebot we deleted all rules and didn’t notice extra iowait anymore.
>
> Could these rules be the cause of additional iowait even if not
> generating many events (around 20 in two hours) ?
>
> Is there any other auditd mechanism  that could explain this phenomenon ?
>
> I would appreciate any hints.

Hi Philippe,

First, as this is a RH cluster product, I would suggest contacting RH support with your question if you haven't already; this list is primarily for upstream development and support.

If you are able to experiment with the system, or have a test environment, I would suggest trying to narrow down the list of audit rules/watches to see which rules/watches have the most affect on the iowait times.  You've listed four rules, but you didn't list the watches you have configured.  Also, what kernel version are you using?

--
paul moore
www.paul-moore.com

!!!*************************************************************************************
"Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.!!!"