Watch Performance

[Amy Griffis]

Steve Grubb wrote:     [Tue Apr 11 2006, 05:01:23PM EDT]
> On Tuesday 11 April 2006 12:11, Amy Griffis wrote:
> > -a exit,always -S chmod -S fchmod -S chown -S fchown -S lchown
> > -S creat -S open -S truncate -S ftruncate -S mkdir -S rmdir -S unlink
> > -S rename -S link -S symlink -F watch=/etc/sysconfig/console
> >
> > Now you don't have any rules for access(), so using it as the test
> > case is much more interesting.
> 
> OK, I re-worked auditctl to use these syscalls instead of "all". I then re-ran 
> the tests on the same kernel as I was testing on since lspp.17 has slab debug 
> stuff turned on again.
> 
> rules  seconds    loss
> 0        50            0%
> 10      52            4%
> 25      56            12%
> 50      69            38%
> 75      81            62%
> 90      87            74%

Hmm, that's interesting, thanks.

> The 75 rule performance hit is now 62%. So there is some improvement in 
> performance.  RHEL4 has a 6% hit for 90 rules.

Do you mean 10 rules + 80 watches?  Or 90 rules?

> We've narrowed the difference, but I don't consider this solved.

I think there are three syscall groups for which we want to consider
the performance impact of having a large number of rules.

1. syscalls for which there are no rules

    In most use cases, the majority of syscalls will fall into this
    group.  That makes this group the most important because it likely
    has the most effect on general system performance.  Based on your
    numbers above, it looks like the impact here is unacceptable.

    For this group, there should be no filtering overhead at all.  To
    achieve this, we must have some indication per-list whether there
    are any rules for a given syscall.  If there are no rules for the
    syscall, don't even walk the list.

2. syscalls for which there are a small number of rules

    For this group we must walk the list, and some filtering overhead
    is acceptable.  How much overhead is acceptable is a factor of how
    many syscalls we would typically expect to be in this group, and
    how often we would expect those syscalls to be used.

    If we need to optimize for this case, we have a couple of options.

        a) Provide features which reduce the number of rules for a
           heavy offender.  E.g. for filesystem auditing,

           - allow multiple watches per rule, akin to multiple inodes
             per rule
           - allow a single watch on a directory to apply to many
             files

        b) Separate the rules for a heavy offender, e.g. by putting
           them in a separate list

3. syscalls for which there are a large number of rules

    For this group, the filtering overhead is the most significant and
    optimization is more difficult.  For some use cases, having a
    rules tree instead of a rules list might help.

    For filesystem auditing, when you want to audit a large number of
    inodes or watches, being able to audit an entire sub-tree with a
    single rule would help that particular use case.  However, if you
    want to audit specific inodes/watches that are spread throughout a
    filesystem, the syscall-exit-based filtering is always going to
    exact a penalty.  

    The only way I can think of to mitigate this is to hang the rule
    data off of the inodes themselves, and receive a callback for each
    filesystem event.  You can do the filtering from the callback, and
    eliminate the list traversal on syscall exit.
    
    This is basically the RHEL4 implementation, and there are a few
    reasons why we can't do it this way right now.  The first reason
    is that inotify is doing something similar, and we need to attempt
    to consolidate similar pieces of kernel functionality.  Inotify,
    however, does not support all of the filesystem events we care
    about auditing.  Additionally for those events that inotify does
    support, its hooks are placed in such a way that events are not
    produced for failed operations (which we care about).

    To use this type of implementation in audit, we must be able to
    either significantly extend inotify, or justify our need for
    having our own implementation to kernel.org.  I think the former
    is more preferable than the latter.

> I also don't like the idea of handling this by all those syscalls

Yes, it makes the rules long and ugly.

Auditctl could support keywords on the command line that map to
various groups of system calls.  That would be more user-friendly.

> or using "all" because user space tools could get out of sync with
> the kernel. On any kernel upgrade, there could be a new syscall that
> allows file system access.  The user space tools wouldn't know about
> it and wouldn't provide automatic coverage.

True, we would have to keep an eye on new syscalls.

Hope this helps,
Amy