[Crash-utility] feature to dump audit logs in vmcore
Dave Anderson
anderson at redhat.com
Thu Mar 9 15:36:27 UTC 2017
----- Original Message -----
> Dave,
>
> I wrote an extension module to dump audit logs in vmcore.
> How about this in crash utility as a built-in command?
>
> crash> extend /root/repos/crash-dumpaudit-command/src/dumpaudit.so
> /root/repos/crash-dumpaudit-command/src/dumpaudit.so: shared object
> loaded
> crash> dumpaudit
> type=1300 audit(1489022639.875:164489): arch=c000003e syscall=0
> success=yes exit=0 a0=5 a1=7fedd3b00000 a2=400 a3=22 items=0 ppid=2575
> pid=10428 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
> tty=pts1 ses=1 comm="pidof" exe="/usr/sbin/killall5"
> subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
> type=1320 audit(1489022639.875:164489):
> type=1320 audit(1489022639.875:164487):
> type=1300 audit(1489022639.875:164490): arch=c000003e syscall=3
> success=yes exit=0 a0=5 a1=1 a2=8 a3=0 items=0 ppid=2575 pid=10428
> auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1
> ses=1 comm="pidof" exe="/usr/sbin /killall5"
> subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
> ...<cut>...
OK, as I understand it, this is similar in nature to the trace extension module,
in that you can display the data that happened to be in kernel memory (and didn't
make it to disk) when the kernel crashed.
Honestly, I have never seen/heard of any discussions about audit logs w/respect to
crash analysis in the past, so I'm guessing that you must have come upon a real
kernel crash that involved auditing.
Anyway, I definitely don't see it as a top-level built-in command. Perhaps you could
argue for an option to an existing command -- "ps", "log" or "sys" maybe?
Thanks,
Dave
>
> What the extension module does is so simple that it retrives audit
> logs from a queue. Looking back into kernel git logs, this design that
> audit logs are held in the queue, was introduced at the following
> patch to introduce kauditd kernel thread and have never changed until
> now:
>
> # git describe b7d1125817c9a46cc46f57db89d9c195e7af22f8
> v2.6.12-rc4-48-gb7d1125
>
> # git log -1 -p b7d1125817c9a46cc46f57db89d9c195e7af22f8
> commit b7d1125817c9a46cc46f57db89d9c195e7af22f8
> Author: David Woodhouse <dwmw2 at shinybook.infradead.org>
> Date: Thu May 19 10:56:58 2005 +0100
>
> AUDIT: Send netlink messages from a separate kernel thread
>
> netlink_unicast() will attempt to reallocate and will free messages
> if
> the socket's rcvbuf limit is reached unless we give it an infinite
> timeout. So do that, from a kernel thread which is dedicated to
> spewing
> stuff up the netlink socket.
>
> Signed-off-by: David Woodhouse <dwmw2 at infradead.org>
>
> I confirmed the comamnd works well on fedora 4.8 kernel, RHEL7.3,
> RHEL6.8 and RHEL5.11; RHEL6.8 was tested both on x86_64 and x86_32.
>
> So, I guess the design will rarely be changed also in the future for
> some time.
>
> To get vmcore full of audit logs, run the following commands:
>
> # auditctl -a exit,always -S all # always record exit of all system calls
> # auditctl -e 1 # enable audit logging
> # auditctl -f 2 # configure kernel to panic when audit
> buffer becomes full
> # kill -STOP $(pidof auditd) # stop auditd that is a unique process
> # to read and reduce audit buffer
> # dd if=/dev/zero of=/dev/null # issue system calls a lot
>
> then kernel will panic when audit buffer becomes full.
>
> To dump audit logs, simply, run dumpaudit command:
>
> crash> extend /root/repos/crash-dumpaudit-command/src/dumpaudit.so
> /root/repos/crash-dumpaudit-command/src/dumpaudit.so: shared object
> loaded
> crash> dumpaudit
> type=1300 audit(1489022639.875:164489): arch=c000003e syscall=0
> success=yes exit=0 a0=5 a1=7fedd3b00000 a2=400 a3=22 items=0 ppid=2575
> pid=10428 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
> tty=pts1 ses=1 comm="pidof" exe="/usr/sbin/killall5"
> subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
> type=1320 audit(1489022639.875:164489):
> type=1320 audit(1489022639.875:164487):
> type=1300 audit(1489022639.875:164490): arch=c000003e syscall=3
> success=yes exit=0 a0=5 a1=1 a2=8 a3=0 items=0 ppid=2575 pid=10428
> auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1
> ses=1 comm="pidof" exe="/usr/sbin /killall5"
> subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
> ...<cut>...
>
> On the vmcore created like above, log command should show the messages
> ending with:
>
> crash> log | tail -n 21
> [253655.583070] audit: audit_backlog=321 > audit_backlog_limit=320
> [253655.583124] audit: audit_backlog=321 > audit_backlog_limit=320
> [253655.583129] audit: audit_lost=1 audit_rate_limit=0
> audit_backlog_limit=320
> [253655.583139] Kernel panic - not syncing: audit: backlog limit exceeded
>
> [253655.583142] CPU: 0 PID: 2575 Comm: bash Not tainted
> 3.10.0-327.el7.x86_64 #1
> [253655.583143] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
> [253655.583147] ffffffff81870541 000000007dd10266 ffff8800b8a43d40
> ffffffff816351f1
> [253655.583149] ffff8800b8a43dc0 ffffffff8162ea6c ffffffff00000010
> ffff8800b8a43dd0
> [253655.583151] ffff8800b8a43d70 000000007dd10266 0000000000000000
> ffffffff818705e8
> [253655.583152] Call Trace:
> [253655.583168] [<ffffffff816351f1>] dump_stack+0x19/0x1b
> [253655.583170] [<ffffffff8162ea6c>] panic+0xd8/0x1e7
> [253655.583178] [<ffffffff81103ce4>] audit_panic+0x64/0x70
> [253655.583180] [<ffffffff81103d2f>] audit_log_lost+0x3f/0xd0
> [253655.583186] [<ffffffff811049bc>] audit_log_start+0x1bc/0x4b0
> [253655.583189] [<ffffffff810919dc>] ? do_send_sig_info+0x6c/0xa0
> [253655.583193] [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
> [253655.583199] [<ffffffff81108ce1>] audit_log_exit+0x51/0xb90
> [253655.583201] [<ffffffff8110b7cd>] __audit_syscall_exit+0x21d/0x280
> [253655.583208] [<ffffffff81645a2b>] sysret_audit+0x17/0x21
>
> where audit_backlog_limit=320 means the limit number of audit logs
> configured in /etc/audit.conf was 320 and audit_backlog=321 means the
> audit queue, or audit backlog, became full amounting to 321.
>
> dumpaudit should dump audit logs as many lines as the limit:
>
> crash> dumpaudit | wc -l
> 321
>
> --
> Thanks.
> HATAYAMA, Daisuke
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
More information about the Crash-utility
mailing list