Auditd framework slowdowns (sometimes freezes) the entire system.

[Kangkook Jee]

Hi all,

I'm writing a custom user-land auditd client subscribing to kauditd to monitor a
number of system calls that we are interested. My auditd client seems to work
fine in overall but I found unexpected behavior of auditd framework which slows
down (or sometimes freezes) the entire system as the consuming rate of audit
client couldn't catch up the speed of audit message generation.

Here's the simple code snippet used to reproduce the problem.

//
// To build.
// g++ -o simple_audit -std=c++11 -L/usr/lib/x86_64-linux-gnu/  main.cpp -laudit
//
#include <libaudit.h>
#include <sys/types.h>
#include <unistd.h>

#include <cassert>
#include <iostream>

static int32_t fd = -1;
static bool au_listen_flag = true;

int main(int argc, char* argv[]) {
    struct audit_reply rep;
    uint64_t cnt = 0;

    if (argc != 2) {
        fprintf(stderr, "Invalid usage: %s <sleep_interval>\n", argv[0]);
        exit(1);
    }

    uint32_t sleep_time = atoi(argv[1]);

    fd = audit_open();
    if (fd < 0) {
        // error handling.
        std::cerr << "Invalid fd returned: " + std::to_string(fd) << std::endl;
        exit(-1);
    }
    int32_t ret = audit_set_pid(fd, getpid(), WAIT_YES);
    if (ret < 0) {
        std::cerr << "audit_set_pid failed: " + std::to_string(fd) << std::endl;
        exit(-1);
    }

    while (au_listen_flag) {
        int32_t rc = audit_get_reply(fd, &rep, GET_REPLY_BLOCKING, 0);
        if (rc > 0) {
            cnt++;
        }

        usleep(sleep_time);

        if (cnt % 10000 == 0) {
            printf ("messages %lu\n", cnt);
        }
    }
    close(fd);
}


The problem becomes more apparent as we increase the amount of sleep time that
is provided as a first command line argument (say a thousand Milli-seconds) and
simultaneously run some heavy-load tasks (i.e., kernel build). 

sudo ./simple_audit 1000

Here's the command line that we used to add system calls to be monitored and
enable.

# Adding events.
/sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect -S listen -S socket -S socketpair

# Enabling events.
/sbin/auditctl -e1 -b 102400

At the very moment, "auditctl -s" indicating that kernel buffer is filled up but
it does not throw away audit messages ('lost' is not increasing ). 

# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=29887 rate_limit=0 backlog_limit=102400 lost=270878600 backlog=102402
# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=29887 rate_limit=0 backlog_limit=102400 lost=270878600 backlog=102402

Could anyone guide me how to configure kauditd's buffer setting so that it can
dump audit messages when the buffer is filled up and user-land consumer can't
catch up the speed of audit message produce?

Thanks a lot for your help in advance!

Regards, Kangkook