<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=Content-Type content="text/html; charset=us-ascii"> <meta name=Generator content="Microsoft Word 12 (filtered medium)"> <style>  </style>  </head> <body lang=EN-US link=blue vlink=purple> <div class=WordSection1> All:<o:p></o:p> <o:p> </o:p> I’ve seen the following situation occur on 2 machines now for a total of 3 incidents:<o:p></o:p> <o:p> </o:p> <![if !supportLists]>· <![endif]>Audisp-remote runs normally on 5 separate servers, the problem happens on two that are configured the same as the other 3.<o:p></o:p> <![if !supportLists]>· <![endif]>Audisp-remote runs normally on the problem servers for days to weeks at a time without problems. <o:p></o:p> <![if !supportLists]>· <![endif]>For an unidentified reason (nothing that I can find in any system log) audisp-remote stops sending messages to the central log server. <o:p></o:p> <![if !supportLists]>· <![endif]>Some hours or days later (depending on audit event activity) audisp-remote consumes all system memory and swap space. In my case because of the nature of my directory tree watches for my web content this usually happens when the web content is being regenerated from scratch by our build server. The memory consumption happens very rapidly. <o:p></o:p> <![if !supportLists]>· <![endif]>One server is configured with 8GB of ram and 2GB of swap, the second server has 12GB of ram and 2GB of swap. <o:p></o:p> <![if !supportLists]>· <![endif]>The system becomes completely unresponsive until enough time goes by for some critical need for memory to arise and the OOM Killer kicks in and starts reaping enough tasks to allow me to get in and shutdown auditd. <o:p></o:p> <![if !supportLists]>· <![endif]>At this point the system returns to normal, and if I restart auditd it resumes normal operation. <o:p></o:p> <o:p> </o:p> Here is a ps aux taken when it happened today on the 12GB machine:<o:p></o:p> <o:p> </o:p> USER PID %CPU %MEM VSZ RSS T TY STAT START TIME COMMAND<o:p></o:p> root 1106 0.0 0.0 0 0 ? S< Oct12 0:36 [kauditd]<o:p></o:p> root 4768 0.1 0.0 92880 500 ? S<sl Oct17 26:22 auditd<o:p></o:p> root 4770 0.2 0.1 212984 12984 ? S<sl Oct17 31:49 /sbin/audispd<o:p></o:p> root 4771 0.0 96.7 28631936 11899072 ? S< Oct17 7:52 /sbin/audisp-remote<o:p></o:p> <o:p> </o:p> Priorities for each audit task are:<o:p></o:p> <o:p> </o:p> Auditd -4<o:p></o:p> Audispd -14<o:p></o:p> Audisp-remote -4<o:p></o:p> <o:p> </o:p> <o:p> </o:p> All machines are fully current on maintenance. Running RedHat EL 5.5 x86_64 with the following audit package set:<o:p></o:p> <o:p> </o:p> <![if !supportLists]>· <![endif]>audit-libs-python-1.7.17-3.el5<o:p></o:p> <![if !supportLists]>· <![endif]>audit-libs-1.7.17-3.el5<o:p></o:p> <![if !supportLists]>· <![endif]>audit-libs-1.7.17-3.el5<o:p></o:p> <![if !supportLists]>· <![endif]>audit-1.7.17-3.el5<o:p></o:p> <![if !supportLists]>· <![endif]>audispd-plugins-1.7.17-3.el5<o:p></o:p> <o:p> </o:p> All that being said, I have the following questions:<o:p></o:p> <o:p> </o:p> <![if !supportLists]>· <![endif]>Has anyone seen this, and if so what workarounds, or fixes are available.<o:p></o:p> <![if !supportLists]>· <![endif]>What additional data should I collect that may assist in identifying the root cause of the problem? Since it can take days for this to manifest itself it seems like traces are out of the question, but perhaps there are other collection tools that can be used. <o:p></o:p> <![if !supportLists]>· <![endif]>Are there any program options or configuration options that can be used to debug this? The man pages seem to be a bit stale in this distribution?<o:p></o:p> <![if !supportLists]>· <![endif]>Does anyone have any other ideas on what I might do to get to the bottom of this?<o:p></o:p> <o:p> </o:p> I also have a separate issue, that I’m curious about. Under RedHat EL 5.5 there doesn’t seem to be any limitations on the support for audisp-remote, but I just noticed in the release notes for RedHat EL 6 Beta, this component is flagged as a Technology Preview in EL 6. Does anyone know the reason for the change in status? I was planning to use this as part of my PCI-DSS compliance efforts next year but this change may make that choice problematic. <o:p></o:p> <o:p> </o:p> Attached please find my current auditd.conf, audispd.conf, audisp-remote.conf and au-remote.conf files.<o:p></o:p> <o:p> </o:p> Beyond this query I also plan to open a support incident with RedHat, but I thought that by using feedback from this group I might be in a better position to provide support with useful information to aid in problem diagnosis. <o:p></o:p> <o:p> </o:p> Please let me know anything else that may help to get to the bottom of this. <o:p></o:p> <o:p> </o:p> Thanks in advance!<o:p></o:p> <o:p> </o:p> Jim <o:p></o:p> </div> </body> </html>