malloc significally impact audit performance

[Steve Grubb]

Hello,

On Sun, 22 Oct 2017 19:12:34 +0300
Lev Olshvang <levonshe at yandex.com> wrote:

> Hello List,
>  
> I work on Ubuntu 16.10 kernel 4.8 with audit v2.7.7, and we wrote
> plugin who get its input from audispd 
> This plugin process audit messages, mostly syscalls.
>  
> In out test we saw that malloc of auditd tooks ~5% among 9% of total
> CPU time audit consumes. 
>  
> I wonder, whether libauparse indeed need to build dynamic lists, ex
>  
> auparse/nvlist.c:       nvnode* newnode = malloc(sizeof(nvnode));

Yes. Auparse library is heavy on malloc. This is known for a long time.
My main goal was to get it working nice and correct and then spend time
optimizing it. glibc malloc is slower but finds memory corruption
better. This work is on the roadmap for the audit-2.9 series of
releases. One quick and easy fix might be to recompile with jemalloc.

There are a number of possible ways to fix this that I have been
thinking about. One is lazy parsing where we only extract the timestamp
and parse the rest if needed. Another might be to just build a list of
pointers and lengths rather than mallocing each piece.


> Most audit message types have strict structure, the only name of exec
> and comm fields have variable length. 
> And besides when I do field search, I actually do not need field
> name, I can pass some ENUM. 
>  
> I even think we can live without files or path names, inode
> +filesystem id will be enough to have in log 
> This can be configurable option and we can add to ausearch /aureport
> the task to translate inode to file name. 
> What do you think?

Those things can disappear and you never know what was there. And some
people have to maintain logs for 3 years on an archive server. Its way
too late to lookup that info for activity that happened so long ago.
The whole OS may have been upgraded.

I'd be curious if switching to jemalloc helps. I think its a matter of
adding its header to any file that has malloc and then updating the
appropriate LDADD lines in Makefile.am.

-Steve