[RFC] audit support for BPF notification

Mon Nov 4 13:41:08 UTC 2019

Hello, Jiri,

Unfortunately, I'm not with the Product Security anymore and cannot speak in
ProdSec name. My personal view is that we/customers must be able to log eBPF
program load/unload events in some way (not necessary by audit).

Best regards,
Vladis Dronov | Red Hat, Inc. | The Core Kernel | Senior Software Engineer

----- Original Message -----
> From: "Jiri Benc" <jbenc at redhat.com>
> To: "Jiri Olsa" <jolsa at redhat.com>
> Cc: "Steve Grubb" <sgrubb at redhat.com>, linux-audit at redhat.com, "Stanislav Kozina" <skozina at redhat.com>, "Yauheni
> Kaliuta" <yauheni.kaliuta at redhat.com>, "Toke Høiland-Jørgensen" <toke at redhat.com>, "Arnaldo Carvalho de Melo"
> <acme at redhat.com>, "Jesper Dangaard Brouer" <brouer at redhat.com>, "Vlad Dronov" <vdronov at redhat.com>, "Petr Matousek"
> <pmatouse at redhat.com>, "Rashid Khan" <rkhan at redhat.com>
> Sent: Monday, November 4, 2019 2:05:18 PM
> Subject: Re: [RFC] audit support for BPF notification
> 
> Seems there have been no reply to this...
> 
> Jiri, what is the current status?
> 
> Vlad, what is the Product Security's view on this? Is the audit support
> for bpf programs loading/unloading a requirement for full support of
> eBPF (as opposed to tech preview)?
> 
> Thanks,
> 
>  Jiri
> 
> On Tue, 20 Aug 2019 15:54:53 +0200, Jiri Olsa wrote:
> > cc-ing Petr Matousek
> > 
> > jirka
> > 
> > On Wed, Aug 14, 2019 at 09:33:34AM +0200, Jiri Olsa wrote:
> > > hi,
> > > Adding Vlad Dronov to the loop, because he asked
> > > about this functionality some time ago.
> > > 
> > > Vlad, the full thread can be found in here:
> > >   https://www.redhat.com/archives/linux-audit/2019-August/msg00004.html
> > > 
> > > Any thoughts on this?
> > > 
> > > thanks,
> > > jirka
> > > 
> > > On Mon, Aug 12, 2019 at 04:33:10PM +0200, Jiri Olsa wrote:
> > > > On Mon, Aug 12, 2019 at 09:49:43AM -0400, Steve Grubb wrote:
> > > > > On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote:
> > > > > > On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote:
> > > > > > > Hello,
> > > > > > > 
> > > > > > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote:
> > > > > > > > I posted initial change that allows auditd to log BPF program
> > > > > > > > 
> > > > > > > > load/unload events, it's in here:
> > > > > > > >   https://github.com/linux-audit/audit-userspace/pull/104
> > > > > > > 
> > > > > > > Thanks for the patch...but we probably should have talked a bit
> > > > > > > more
> > > > > > > before undertaking this effort. We normally do not audit from
> > > > > > > user space
> > > > > > > what happens in the kernel. Doing this can be racy and it keeps
> > > > > > > auditd
> > > > > > > from doing the one job it has - which is to grab events and
> > > > > > > record them
> > > > > > > to disk and send them out the realtime interface.
> > > > > > >   
> > > > > > > > We tried to push pure AUDIT interface for BPF program
> > > > > > > > notification,
> > > > > > > > 
> > > > > > > > but it was denied, the discussion is in here:
> > > > > > > >   https://marc.info/?t=153866123200003&r=1&w=2
> > > > > > > 
> > > > > > > Hmm. The email I remember was here:
> > > > > > > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html
> > > > > > > 
> > > > > > > and was only 2 emails long with no answer to my question. :-)
> > > > > > 
> > > > > > oops, sry about that, your question was:
> > > > > > 	> I'm not sure exactly what the issue is. You can audit for
> > > > > > 	> specific
> > > > > > 	> syscall
> > > > > > 	> and argument. So, if you want to see loads, then you can make a
> > > > > > 	> rule
> > > > > > 	> like:
> > > > > > 	> 
> > > > > > 	> -a always,exit -F arch=b64 -S bpf -F a0=5
> > > > > > 
> > > > > > The problem with above for us is that we also:
> > > > > > 
> > > > > >   - need to log also other properties of the BPF program,
> > > > > >     which are not visible from BPF syscall arguments, like
> > > > > >     its ID, JIT status
> > > > > 
> > > > > The way this is normally done is to add a supplemental record. For
> > > > > example,
> > > > > when auditing the open syscall, we also get CWD & PATH supplemental
> > > > > records.
> > > > > When auditing connect, we get a SOCKADDR supplemental record. We have
> > > > > requirements around selective audit whereby the admin is in control
> > > > > of what
> > > > > is selected for audit via audit rules. So, what one could do is set a
> > > > > rule
> > > > > for the bpf syscall and then when it triggers, we get these other
> > > > > records
> > > > > added to the bpf syscall event.
> > > > 
> > > > right, that was the initial plan, but BPF guys wanted just
> > > > single notification system without specific hooks for audit,
> > > > so we ended up with perf specific interface
> > > >   
> > > > > >     or license info
> > > > > 
> > > > > This ^^ is not a security issue.
> > > > > 
> > > > >   
> > > > > >   - need to see BPF program UNLOAD, which is not done
> > > > > >     via syscall, so those would be unvisible at all
> > > > > 
> > > > > Is there a place in the kernel where this happens? I could see
> > > > > abnormal
> > > > > termination being something we might want. Does the program go
> > > > > through
> > > > > something like an exit syscall internally?
> > > > 
> > > > it's happening in here (kernel/bpf/syscall.c):
> > > > 
> > > > 	bpf_prog_put
> > > > 	  __bpf_prog_put
> > > > 	  {
> > > > 		    if (atomic_dec_and_test(&prog->aux->refcnt)) {
> > > > 			perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_UNLOAD, 0);
> > > > 			...
> > > > 	  }
> > > > 
> > > > BPF program is released when it drops the reference count,
> > > > which is normally when its file descriptor is closed.
> > > > 
> > > > However it might get pinned and stay alive even when the initial
> > > > file descriptor is closed.. and then there's the networking world,
> > > > which might have some other specific ways.. but it all ends up
> > > > in bpf_prog_put and zero reference count.
> > > >   
> > > > > > > > The outcome of the discussion was to use perf event interface
> > > > > > > > for BPF notification and use it in some deamon.. audit was our
> > > > > > > > first choice.
> > > > > > > > 
> > > > > > > > thoughts?
> > > > > > > 
> > > > > > > I'd like to understand what the basic problem is that needs to be
> > > > > > > solved.
> > > > > > 
> > > > > > we need a way for administrators to see the history of loaded BPF
> > > > > > programs, to help investigating issues related to BPF.. and the
> > > > > > only BPF interface for this data is through perf ring buffer
> > > > > 
> > > > > That is really not the audit way. Let's keep talking to see where
> > > > > this ends
> > > > > up.
> > > > 
> > > > Would you see some other auditing daemon/app in place for this kind of
> > > > data?
> > > > 
> > > > thanks,
> > > > jirka
> 
> 