[PATCH v3] audit: report audit wait metric in audit status reply

[Max Englander]

On Tue, Jul 21, 2020 at 11:26:53AM -0400, Paul Moore wrote:
> On Wed, Jul 15, 2020 at 9:30 PM Paul Moore <paul at paul-moore.com> wrote:
> > On Wed, Jul 8, 2020 at 7:13 PM Paul Moore <paul at paul-moore.com> wrote:
> > > On Sat, Jul 4, 2020 at 11:15 AM Max Englander <max.englander at gmail.com> wrote:
> > > >
> > > > In environments where the preservation of audit events and predictable
> > > > usage of system memory are prioritized, admins may use a combination of
> > > > --backlog_wait_time and -b options at the risk of degraded performance
> > > > resulting from backlog waiting. In some cases, this risk may be
> > > > preferred to lost events or unbounded memory usage. Ideally, this risk
> > > > can be mitigated by making adjustments when backlog waiting is detected.
> > > >
> > > > However, detection can be difficult using the currently available
> > > > metrics. For example, an admin attempting to debug degraded performance
> > > > may falsely believe a full backlog indicates backlog waiting. It may
> > > > turn out the backlog frequently fills up but drains quickly.
> > > >
> > > > To make it easier to reliably track degraded performance to backlog
> > > > waiting, this patch makes the following changes:
> > > >
> > > > Add a new field backlog_wait_time_total to the audit status reply.
> > > > Initialize this field to zero. Add to this field the total time spent
> > > > by the current task on scheduled timeouts while the backlog limit is
> > > > exceeded. Reset field to zero upon request via AUDIT_SET.
> > > >
> > > > Tested on Ubuntu 18.04 using complementary changes to the
> > > > audit-userspace and audit-testsuite:
> > > > - https://github.com/linux-audit/audit-userspace/pull/134
> > > > - https://github.com/linux-audit/audit-testsuite/pull/97
> > > >
> > > > Signed-off-by: Max Englander <max.englander at gmail.com>
> > > > ---
> > > > Patch changelogs between v1 and v2:
> > > >   - Instead of printing a warning when backlog waiting occurs, add
> > > >     duration of backlog waiting to cumulative sum, and report this
> > > >     sum in audit status reply.
> > > >
> > > > Patch changelogs between v2 and v3:
> > > >  - Rename backlog_wait_sum to backlog_wait_time_actual.
> > > >  - Drop unneeded and unwanted header flags
> > > >    AUDIT_FEATURE_BITMAP_BACKLOG_WAIT_SUM and
> > > >    AUDIT_VERSION_BACKLOG_WAIT_SUM.
> > > >  - Increment backlog_wait_time_actual counter after every call to
> > > >    schedule_timeout rather than once after enqueuing (or losing) an
> > > >    audit record.
> > > >  - Add support for resetting backlog_wait_time_actual counter to zero
> > > >    upon request via AUDIT_SET.
> > > >
> > > >  include/uapi/linux/audit.h | 18 +++++++++++-------
> > > >  kernel/audit.c             | 35 +++++++++++++++++++++++++----------
> > > >  2 files changed, 36 insertions(+), 17 deletions(-)
> > >
> > > This looks okay to me, thanks for the fixes Max.
> > >
> > > Steve, does the associated userspace patch look okay to you?
> >
> > Steve, any comments on the userspace patch?  Did I miss a reply in my
> > inbox perhaps?
> >
> > If I don't see any feedback by the end of the week I'll plan on
> > merging this into audit/next.
> 
> It's been over two weeks with no comment, so I went ahead and merged
> this into audit/next.  Thanks for your patience Max!

Excellent, glad to hear it! Thank you (and Richard, Steve) for the
guidance and interesting discussion along the way.

> 
> -- 
> paul moore
> www.paul-moore.com