question on audit_backlog settings and how to prevent the sytem from hanging due to audit overload

[larry.erdahl at usbank.com]

I have a 5.4 Redhat that I'm using Snare to control the audit rules with. 
Recently this server hung on me and pointed to the SnareDispatcher as the 
cause. You can see from the samples below the dispatcher was running at 99 
- 100%.
The morning of the hang Auditd  peaked at  ~200,000 event's/hour, up from 
~50,000 events per hour. Is there away to protect the server from hanging 
during unexpected loads like this? 

I'm assuming from what I've read, I'll need to increase the audit_backlog 
level to something higher. Before increasing the number of buffers  I'd 
like to get a clearer understanding of their size and  how increasing 
these buffers my impact my over all system performance. Are there any 
recommendations on what the settings should be or a formula that I could 
use to determine the proper setting. 

I am looking into what may of caused the spike, but I'd like to know what 
my options to keep from having another system hang


Any help would be appreciated 

Sep 30 01:29:16 <servername> kernel: audit: audit_backlog=321 > 
audit_backlog_limit=320
Sep 30 01:29:16<servername> kernel: audit: audit_lost=1 audit_rate_limit=0 
audit_backlog_limit=320
Sep 30 01:29:16 <servername> kernel: audit: backlog limit exceeded
Sep 30 01:29:16 <servername> kernel: audit: audit_backlog=321 > 
audit_backlog_limit=320
Sep 30 01:29:16 <servername> auditmanager: Received wakeup signal before 
sleep finished
And this is in the process monitoring
1:16:06 4545    99.8    0       99.8    140848  3292    12      0 484   0 
0       SnareDispatchHe 4.16    12
1:21:07 4545    99.9    0       99.9    140848  3292    12      0 484   0 
0       SnareDispatchHe 4.16    12
1:26:07 4545    100     0       100     140848  3292    12      0 484   0 
0       SnareDispatchHe 4.17    12
1:31:07 4545    99.7    0       99.7    140848  3292    12      0 484   0 
0       SnareDispatchHe 4.15    12
1:36:07 4545    99.9    0       99.9    140848  3292    12      0 484   0 
0       SnareDispatchHe 4.16    12
1:41:07 4545    99.9    0       99.9    140848  3292    12      0 484   0 
0       SnareDispatchHe 4.16    12
1:46:08 4545    99.9    0       99.9    140848  3292    12      0 484   0 
0       SnareDispatchHe 4.16    12
1:51:08 4545    82.8    0       82.8    140848  3292    12      0 484   0 
0       SnareDispatchHe 3.45    12

Thanks....

Larry E. Erdahl
Information Security Services
Computer Security Incident Response Team (CSIRT)
1 Meridian Crossing 
Richfield, MN 55423
Mail Code: EP-MN-MS6I
Office Phone: (612)973-7153

U.S. BANCORP made the following annotations
---------------------------------------------------------------------
Electronic Privacy Notice. This e-mail, and any attachments, contains information that is, or may be, covered by electronic communications privacy laws, and is also confidential and proprietary in nature. If you are not the intended recipient, please be advised that you are legally prohibited from retaining, using, copying, distributing, or otherwise disclosing this information in any manner. Instead, please reply to the sender that you have received this communication in error, and then immediately delete it. Thank you in advance for your cooperation.



---------------------------------------------------------------------