Fw:Re:Fw:Re:Fw:Re:[RFC,v1,1/1] audit: speed up syscall rule match while exiting syscall

[yang.yang29 at zte.com.cn]

Great thanks for your suggestings, patch v3 has a better performance.

New data: 
Syscall absolute time
Test method: 
Use ktime_get_real_ts64() in do_syscall_64() to calculate time.
Run command "chmod 777 /etc/fstab" with chown rules. Each test 10times and get average.

do_syscall_64() time with 100 rules:
before this patch: 7604ns
after this patch: 5244ns, reduce 2360ns.

do_syscall_64() time with CIS rules:
before this patch: 6710ns
after this patch: 7190ns, increase 480ns.

do_syscall_64() time with CIS rules without chmod rule:
before this patch: 6710ns
after this patch: 5312ns, reduce 1398ns.

do_syscall_64() time with 10 rules:
before this patch: 5382ns
after this patch: 5217ns, reduce 165ns.

do_syscall_64() time with 1 rule:
before this patch: 5361ns
after this patch: 5242ns, reduce 119ns.

do_syscall_64() time with no rules:
before this patch: 4735ns
after this patch: 4804ns, increase 69ns

Analyse:
With no rule, this patch has no effect. The difference data is because of system jitter.
And I finger out why with CIS rules this patch has negative impact. 
Because CIS rules audit syscall chmod, and I test with "chmod syscall". 
And after I delete the 2 chmod rules in CIS, performance is much better.

Summarize: 
If system spend much more time on syscalls that don't need to be audited than syscalls need to be, this patch has obviously improvement.

------------------Original Mail------------------
Sender: PaulMoore
To: yang yang10192021;
CC: linux-audit at redhat.com;linux-kernel at vger.kernel.org;
Date: 2021/01/23 01:31
Subject: Re:Fw:Re:Fw:Re:[RFC,v1,1/1] audit: speed up syscall rule match while exiting syscall
On Thu, Jan 21, 2021 at 9:25 AM <yang.yang29 at zte.com.cn> wrote:
>
> Thanks for reply, I have sent a new patch with better performance.
> The v1 patch uses mutex() is not necessary.
>
> Performance measurements:
> 1.Environment
> CPU: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
> Linux kernel version: 5.11-rc3
> Audit version: 2.8.4

....

> 2.2 Syscall absolute time
> Test method:
> Use ktime_get_real_ts64() in do_syscall_64() to calculate time.
> Run command "chmod 777 /etc/fstab" with chown rules. Each test 10times and get average.
>
> do_syscall_64() time with 100 rules:
> before this patch: 7604ns
> after this patch: 5244ns, reduce 2360ns.
>
> do_syscall_64() time with CIS rules:
> before this patch: 6710ns
> after this patch: 7293ns, increase 583ns.
>
> do_syscall_64() time with 10 rules:
> before this patch: 5382ns
> after this patch: 5171ns, reduce 211ns.
>
> do_syscall_64() time with 1 rule:
> before this patch: 5361ns
> after this patch: 5375ns, increase 14ns.
>
> do_syscall_64() time with no rules:
> before this patch: 4735ns
> after this patch: 4804ns, increase 69ns.
>
> Analyse:
> With a few rules, performance is close.
> With 100 rules, performance is better, but with CIS rules performance regress. Maybe relevant to certain syscall.

These numbers aren't particularly good in my opinion, the negative
impact of the change to a small number of rules and to the CIS ruleset
is not a good thing.  It also should be said that you are increasing
the memory footprint, even if it is relatively small.

However, if we take a step back and look at the motivation for this
work I wonder if there are some things we can do to improve the
per-syscall rule processing performance.  On thing that jumped out
just now was this code in __audit_syscall_exit():

void __audit_syscall_exit(int success, long return_code)
{

/* ... */

/*
* we need to fix up the return code in the audit logs if the
* actual return codes are later going to be fixed up by the
* arch specific signal handlers ... */
if (unlikely(return_code <= -ERESTARTSYS) &&
(return_code >= -ERESTART_RESTARTBLOCK) &&
(return_code != -ENOIOCTLCMD))
context->return_code = -EINTR;
else
context->return_code  = return_code;

audit_filter_syscall(current, context,
&audit_filter_list[AUDIT_FILTER_EXIT]);
audit_filter_inodes(current, context);
if (context->current_state == AUDIT_RECORD_CONTEXT)
audit_log_exit();
}

.... in the snippet above the audit_filter_inodes() function/rules are
given priority over the syscall rules in
audit_filter_syscall(AUDIT_FILTER_EXIT), so why not first execute
audit_filter_inodes() and only execute
audit_filter_syscall(AUDIT_FILTER_EXIT) if necessary?  It may be that
I'm missing something on this quick look at the code, but I think it
is worth investigating.  It's also possible there are other similar
improvements to made.

There is similar code in __audit_free() but that should be less
performance critical.

--
paul moore
www.paul-moore.com