[Crash-utility] feature to dump audit logs in vmcore

Thu Mar 9 01:47:48 UTC 2017

Dave,

I wrote an extension module to dump audit logs in vmcore.
How about this in crash utility as a built-in command?

    crash> extend /root/repos/crash-dumpaudit-command/src/dumpaudit.so
    /root/repos/crash-dumpaudit-command/src/dumpaudit.so: shared object loaded
    crash> dumpaudit
    type=1300 audit(1489022639.875:164489): arch=c000003e syscall=0 success=yes exit=0 a0=5 a1=7fedd3b00000 a2=400 a3=22 items=0 ppid=2575 pid=10428 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1 comm="pidof" exe="/usr/sbin/killall5" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
    type=1320 audit(1489022639.875:164489):
    type=1320 audit(1489022639.875:164487):
    type=1300 audit(1489022639.875:164490): arch=c000003e syscall=3 success=yes exit=0 a0=5 a1=1 a2=8 a3=0 items=0 ppid=2575 pid=10428 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1 comm="pidof" exe="/usr/sbin /killall5" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
    ...<cut>...

What the extension module does is so simple that it retrives audit
logs from a queue. Looking back into kernel git logs, this design that
audit logs are held in the queue, was introduced at the following
patch to introduce kauditd kernel thread and have never changed until
now:

    # git describe b7d1125817c9a46cc46f57db89d9c195e7af22f8
    v2.6.12-rc4-48-gb7d1125

    # git log -1 -p b7d1125817c9a46cc46f57db89d9c195e7af22f8
    commit b7d1125817c9a46cc46f57db89d9c195e7af22f8
    Author: David Woodhouse <dwmw2 at shinybook.infradead.org>
    Date:   Thu May 19 10:56:58 2005 +0100

        AUDIT: Send netlink messages from a separate kernel thread

        netlink_unicast() will attempt to reallocate and will free messages if
        the socket's rcvbuf limit is reached unless we give it an infinite
        timeout. So do that, from a kernel thread which is dedicated to spewing
        stuff up the netlink socket.

        Signed-off-by: David Woodhouse <dwmw2 at infradead.org>

I confirmed the comamnd works well on fedora 4.8 kernel, RHEL7.3,
RHEL6.8 and RHEL5.11; RHEL6.8 was tested both on x86_64 and x86_32.

So, I guess the design will rarely be changed also in the future for
some time.

To get vmcore full of audit logs, run the following commands:

  # auditctl -a exit,always -S all # always record exit of all system calls
  # auditctl -e 1                  # enable audit logging
  # auditctl -f 2                  # configure kernel to panic when audit buffer becomes full
  # kill -STOP $(pidof auditd)     # stop auditd that is a unique process
                                   # to read and reduce audit buffer
  # dd if=/dev/zero of=/dev/null   # issue system calls a lot

then kernel will panic when audit buffer becomes full.

To dump audit logs, simply, run dumpaudit command:

    crash> extend /root/repos/crash-dumpaudit-command/src/dumpaudit.so
    /root/repos/crash-dumpaudit-command/src/dumpaudit.so: shared object loaded
    crash> dumpaudit
    type=1300 audit(1489022639.875:164489): arch=c000003e syscall=0 success=yes exit=0 a0=5 a1=7fedd3b00000 a2=400 a3=22 items=0 ppid=2575 pid=10428 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1 comm="pidof" exe="/usr/sbin/killall5" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
    type=1320 audit(1489022639.875:164489):
    type=1320 audit(1489022639.875:164487):
    type=1300 audit(1489022639.875:164490): arch=c000003e syscall=3 success=yes exit=0 a0=5 a1=1 a2=8 a3=0 items=0 ppid=2575 pid=10428 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1 comm="pidof" exe="/usr/sbin /killall5" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
    ...<cut>...

On the vmcore created like above, log command should show the messages
ending with:

    crash> log | tail -n 21
    [253655.583070] audit: audit_backlog=321 > audit_backlog_limit=320
    [253655.583124] audit: audit_backlog=321 > audit_backlog_limit=320
    [253655.583129] audit: audit_lost=1 audit_rate_limit=0 audit_backlog_limit=320
    [253655.583139] Kernel panic - not syncing: audit: backlog limit exceeded

    [253655.583142] CPU: 0 PID: 2575 Comm: bash Not tainted 3.10.0-327.el7.x86_64 #1
    [253655.583143] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
    [253655.583147]  ffffffff81870541 000000007dd10266 ffff8800b8a43d40 ffffffff816351f1
    [253655.583149]  ffff8800b8a43dc0 ffffffff8162ea6c ffffffff00000010 ffff8800b8a43dd0
    [253655.583151]  ffff8800b8a43d70 000000007dd10266 0000000000000000 ffffffff818705e8
    [253655.583152] Call Trace:
    [253655.583168]  [<ffffffff816351f1>] dump_stack+0x19/0x1b
    [253655.583170]  [<ffffffff8162ea6c>] panic+0xd8/0x1e7
    [253655.583178]  [<ffffffff81103ce4>] audit_panic+0x64/0x70
    [253655.583180]  [<ffffffff81103d2f>] audit_log_lost+0x3f/0xd0
    [253655.583186]  [<ffffffff811049bc>] audit_log_start+0x1bc/0x4b0
    [253655.583189]  [<ffffffff810919dc>] ? do_send_sig_info+0x6c/0xa0
    [253655.583193]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
    [253655.583199]  [<ffffffff81108ce1>] audit_log_exit+0x51/0xb90
    [253655.583201]  [<ffffffff8110b7cd>] __audit_syscall_exit+0x21d/0x280
    [253655.583208]  [<ffffffff81645a2b>] sysret_audit+0x17/0x21

where audit_backlog_limit=320 means the limit number of audit logs
configured in /etc/audit.conf was 320 and audit_backlog=321 means the
audit queue, or audit backlog, became full amounting to 321.

dumpaudit should dump audit logs as many lines as the limit:

    crash> dumpaudit | wc -l
    321

--
Thanks.
HATAYAMA, Daisuke
-------------- next part --------------
A non-text attachment was scrubbed...
Name: crash-dumpaudit-1.0.1.tar.gz
Type: application/octet-stream
Size: 1676 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20170309/844d24c1/attachment.obj>