Odd memory usage in auditd

[Ross Kirk]

Doh, originally just replied to Steve rather than to the list!

Ross

> -----Original Message-----
> From: linux-audit-bounces at redhat.com [mailto:linux-audit- 
> bounces at redhat.com] On Behalf Of Steve Grubb
> Sent: 07 October 2010 20:51
> To: linux-audit at redhat.com
> Subject: Re: Odd memory usage in auditd
> 
> On Thursday, October 07, 2010 05:52:49 am Ross Kirk wrote:
> > Has anybody got any advice for the following problem? As I'm seeing
> some
> > very odd behaviour with the auditd daemon in RHEL5.2 where under
> heavy
> > system load the auditd process doesn't free any resources until all
> memory
> > is consumed and the kernel kills the process with an Out Of Memory
> error.
> 
> I seem to recall something about disk flushing causing auditd to look 
> like its the culprit. Do you have barriers enabled on ext3? You might 
> also try setting the flushing to something else like none and see if 
> that does anything.

Currently barriers are not enabled as it wasn't something I was aware of. However it does sound like something I may well want to be enabled so I will give the various flush settings a try and see if the barrier option affects anything.
So if auditd is not the culprit what do you suspect the problem is?

> > The system I have is a heavily customised RHEL5.2 with some fairly 
> > stringent auditing rules specified, the config is attached. In
> addition to
> > these rules there will be various SELinux AVCs being raised as well
> as
> > events from my own software so the auditing system is kept quite
> busy, see
> > the attached report.txt for the aureport summary .
> 
> I don't see anything terribly unusual. The audit rules didn't make it, 
> but the backlog setting is the only thing I would be interested in 
> seeing.

-b 16384

> > I can reproduce this behaviour consistently by generating a heavy
> system
> > load (CPU 100% usage) while also generating a significant number of
> audit
> > events. After about 20 minutes the auditd process will have grown
> from 8Mb
> > of memory to around 1Gb;
> >
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >
> > 3037 root      16  -3 2763m 921m   16 S  3.7 91.2   0:26.49 auditd
> >
> > If the system is kept busy eventually auditd will consume all the
> memory
> > available on the system and the process be killed by the kernel with
> an
> > Out Of Memory error.
> 
> Try playing with the disk flushing and let us know how that works out.
> There	
> are no known memory leaks in recent version of auditd. I try to keep 
> malloc down to a minimum to prevent this and memory fragmentation to 
> creep in.

With;
*  flush = none Completely different behaviour from previous, memory usage never changed for my entire test run
*  flush = data No increase in memory usage at all
*  flush = sync No increase in memory usage at all

Changing the ext3 barrier setting didn't make any changes to the above results nor to the behaviour I saw when "incremental" was set.

Well as the other settings are better behaved I can change to using "data" without any problems I believe. Arguably this is probably a better choice for me anyway!

Thanks for your help!

Ross