[PATCH v2] audit: report audit wait metric in audit status reply

[Max Englander]

On Wed, Dec 2, 2020 at 11:33 PM Joe Wulf <joe_wulf at yahoo.com> wrote:

> I would like to suggest providing a mechanism where admins can query the
> status or state of backlog issues (wait time, sums, etc...).  Maybe the
> intent is to expand the output of status checking of auditd.
>
> I believe further clarity is beneficial on the setting of the
> 'backlog_wait_sum' (or to whatever the name evolves to) initially.
> -  How it evolves over time
> -  What the conditions in the system, or auditing, would change it
> -  What conditions admins should pay attention to for informational
> understanding of status
> -  What conditions admins should realize exist such that adjustments are
> needed
>    (and suggestions to what those adjustments should be)
> -  What new guidance will admins have for building adjusting audit.rules
> around this
>
> Consider the scenario where auditing has been 'working fine' for days.
> Little to no active admin monitoring.
> Events occur to spike the auditing such that backloging of audit records
> dramatically increases.
> (for some reason) admins now come looking to investigate.
> Assuming they do:  'systemctl status auditd' the newly presented 'state'
> of the 'backlog_wait_sum' will show some evidence.
> Q:  Is that just a moment in time?
> Q:  What information here will give the perspective things are good/ok
> 'now', versus some action needs to be taken?
>
> Maybe that isn't a great scenario, or good questions----it is what occurs
> to me at the moment.
>
> Thank you.
>
> R,
> -Joe Wulf
>
>
> On Wednesday, July 1, 2020, 5:33:14 PM EDT, Max Englander <
> max.englander at gmail.com> wrote:
>
> >  In environments where the preservation of audit events and predictable
> >  usage of system memory are prioritized, admins may use a combination of
> >  --backlog_wait_time and -b options at the risk of degraded performance
> >  resulting from backlog waiting. In some cases, this risk may be
> >  preferred to lost events or unbounded memory usage. Ideally, this risk
> >  can be mitigated by making adjustments when backlog waiting is
> detected.
> >
> >  However, detection can be diffult using the currently available
> metrics.
> >  For example, an admin attempting to debug degraded performance may
> >  falsely believe a full backlog indicates backlog waiting. It may turn
> >  out the backlog frequently fills up but drains quickly.
> >
> >  To make it easier to reliably track degraded performance to backlog
> >  waiting, this patch makes the following changes:
> >
> >  Add a new field backlog_wait_sum to the audit status reply. Initialize
> >  this field to zero. Add to this field the total time spent by the
> >  current task on scheduled timeouts while the backlog limit is exceeded.
> >
> >  Tested on Ubuntu 18.04 using complementary changes to the audit
> >  userspace: https://github.com/linux-audit/audit-userspace/pull/134.
>
> <snip>
>

Hi Joe,

Not sure I can address all your points above, but the way that we monitor
Linux audit internals at my employer is to continuously monitor the audit
status
response with short evaluation windows.

- We compute a rate of change on the lost field, and alert if the there are
more than N lost records per second on average
- We compute the backlog utilization by computing backlog/backlog_limit,
and alert if that goes above 75% at any point in time
- If/when we run on a kernel that has backlog_wait_time_actual, we'll
monitor on that as well, setting thresholds around where we'd expect
growth in this value to result in service degradation.

If we get an alert, and it is just a blip that goes away and doesn't come
back,
we probably won't spend a lot of time investigating. However, if we see
that the alert is frequently active across multiple hosts, that will prompt
us
to investigate. As far as what action we would take, it would depend on
the precise values in the audit status reply, as well as other information
we
had gathered from our system. For example, if we observed elevated values
for backlog and backlog_wait_time_actual, we might first investigate other
environmental factors such as whether the auditd daemon was crashed or
starved for CPU time. If we saw that lost was high but backlog was low
that might indicate to us that the rate limit is being exceeded, or that the
kernel is out of memory.

I agree with you that it would help to expand the metrics reported in
audit status. For example, reporting the number of times an audit record was
lost due to rate limit being exceeded would help.

Not sure how responsive this is to your questions. Hope it helps some.

Thanks,
Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-audit/attachments/20201207/23286ade/attachment.htm>