Kernel 4.9.51 null pointer dereference add_rule with null key

Richard Guy Briggs rgb at redhat.com
Fri Aug 17 21:46:23 UTC 2018


On 2018-08-17 14:24, Preston Bennes wrote:
> Hey Paul,
> 
> My expectations are aligned to what you've pointed out already (not-vanilla
> old kernel, user space client other than auditd).
> 
> Primarily what I'm looking for is a sanity check on the null pointer deref
> on current vanilla kernel code because I think the rest of the issues stem
> from that. My C isn't sharp enough to try my hand at a reproduction case,
> though I would hope someone with the requisite expertise could put one
> together relatively quickly. specifically this message:
> 
> audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936
> > op="add_rule" key=(null) list=4 res=0

Not digging particularly deeply here, this fix just went upstream which
may fix the problem:

	https://www.redhat.com/archives/linux-audit/2018-July/msg00037.html

As Paul has already pointed out, the kernel you are using is a pretty
old vendor kernel, so it is difficult to speculate more than that.

> On Fri, Aug 17, 2018 at 2:12 PM Paul Moore <paul at paul-moore.com> wrote:
> > On Fri, Aug 17, 2018 at 4:26 PM Preston Bennes
> > <preston.bennes at snowflake.net> wrote:
> > > Hey Audit team,
> >
> > Hey Audit user!
> >
> > > I've got a kernel null pointer deref oops on Amazon Linux kernel
> > 4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes that
> > spawned were stuck in D wait on some audit related syscall. This exhausted
> > system file handles and the box ended up needing to be rebooted
> > out-of-bounds. The audit handle was held by
> > https://github.com/facebook/osquery/.
> > >
> > > It looks like there was an issue with audit sending to osquery
> > (netlink_unicast sending to pid 21233, error -111) followed by something
> > (presumably also osquery) attempting to 'op="add_rule" key=(null)'.
> >
> > The first thing I need to comment on is that since this is a distro
> > specific kernel (4.9.51-10.52.amzn1.x86_64) and not an upstream kernel
> > there is a limit in the amount of help we can provide.  Generally the
> > upstream list focuses on the upstream code and we leave support for
> > the distro specific builds/packages up to the distro.  If you can
> > reproduce your problem on an upstream kernel, with a strong preference
> > for a *recent* upstream kernel, we can probably be of more help.  I
> > simply don't know what patches Amazon merges into it's amzn1 kernels.
> >
> > With that out of the way, I will add that v4.9.x is a rather old
> > kernel, and we've made a number of relevant improvements since the
> > v4.9 release and it is unlikely all of them have been backported to
> > your kernel.  If possible, I would suggest upgrading to a newer kernel
> > as it may resolve your problem.
> >
> > There is also a possible issue of osquery as an auditd replacement;
> > it's unlikely this is being caused by osquery, but we don't test with
> > osquery to that save level as we do with Steve's audit userspace
> > (auditd).
> >
> > > Here's from /var/log/messages.
> > >
> > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.133904]
> > audit: netlink_unicast sending to audit_pid=21233 returned error: -111
> > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982]
> > audit_log_lost: 12 callbacks suppressed
> > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983]
> > audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096
> > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985]
> > audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233
> > auid=501 ses=102936 res=0
> > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986]
> > audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1
> > auid=501 ses=102936 res=1
> > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987]
> > audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1
> > old=1 auid=501 ses=102936 res=1
> > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989]
> > audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096
> > old=4096 auid=501 ses=102936 res=1
> > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990]
> > audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0
> > auid=501 ses=102936 res=1
> > >> Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991]
> > audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936
> > op="add_rule" key=(null) list=4 res=0
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.138007]
> > BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.143872]
> > IP: [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336]
> > PGD 0
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822]
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.151230]
> > Oops: 0000 [#1] SMP
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736]
> > Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E)
> > nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E)
> > isofs(E) ipv6(E) crc_
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.184036]
> > CPU: 0 PID: 21372 Comm: osqueryd Tainted: G            E
> >  4.9.51-10.52.amzn1.x86_64 #1
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620]
> > Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.195513]
> > task: ffff8807fa1c3b00 task.stack: ffffc900079a4000
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.200179]
> > RIP: 0010:[<ffffffff814a76da>]  [<ffffffff814a76da>]
> > netlink_unicast+0x4a/0x230
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.206822]
> > RSP: 0018:ffffc900079a7c38  EFLAGS: 00010246
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.211065]
> > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.216779]
> > RDX: 00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.222473]
> > RBP: ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.228167]
> > R10: ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.233738]
> > R13: ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.238999]
> > FS:  00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.244796]
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.249555]
> > CR2: 00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.255401]
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.260725]
> > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.266038]
> > Stack:
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.267680]
> > 00000000079a7c58 00000000000052f1 00000000000052f1 00000000ffffffa1
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.273540]
> > ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8 ffffffff811107d2
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.279379]
> > ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8 ffffffff8145a08c
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.285201]
> > Call Trace:
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.287137]
> > [<ffffffff811107d2>] audit_receive_msg+0x992/0xc20
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.291561]
> > [<ffffffff811e2f85>] ? __kmalloc_node_track_caller+0x35/0x260
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.296642]
> > [<ffffffff8145a08c>] ? release_sock+0x8c/0xa0
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.300718]
> > [<ffffffff81110ab2>] audit_receive+0x52/0xa0
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.304713]
> > [<ffffffff814a77ef>] netlink_unicast+0x15f/0x230
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.308943]
> > [<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.313236]
> > [<ffffffff81455818>] sock_sendmsg+0x38/0x50
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.317219]
> > [<ffffffff81455c4f>] SYSC_sendto+0xef/0x170
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.321142]
> > [<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.325853]
> > [<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.330464]
> > [<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.335189]
> > [<ffffffff8145673e>] SyS_sendto+0xe/0x10
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.338952]
> > [<ffffffff81003854>] do_syscall_64+0x54/0xc0
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.342995]
> > [<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.347840]
> > Code: 65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00
> > 81 c6 20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24
> > e0 01 00 00 48 89 45
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625]
> > RIP  [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.373253]
> > RSP <ffffc900079a7c38>
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.376013]
> > CR2: 00000000000001e0
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307]
> > ---[ end trace 6a41b2274729ba2a ]---
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029]
> > audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59
> > success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2
> > ppid=4556 pid=2407
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741]
> > audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date" a1="+%s"
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585]
> > audit: type=1307 audit(1534170895.379:117649235): cwd="/"
> > >> Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported 1
> > kernel oopses to Abrt
> >
> > paul moore

- RGB

--
Richard Guy Briggs <rgb at redhat.com>
Sr. S/W Engineer, Kernel Security, Base Operating Systems
Remote, Ottawa, Red Hat Canada
IRC: rgb, SunRaycer
Voice: +1.647.777.2635, Internal: (81) 32635




More information about the Linux-audit mailing list