Maximum Value for q_depth

[Amjad Gabbar]

Hi Steve,

Based on our discussion above, I performed some analysis as to why we were
seeing so many events. The reason seems to be due to the default rules
being triggered every time a cron job runs. We have numerous cron jobs
running per minute as a result of which multiple different events(LOGIN,
USER_END,CRED_DISP etc) are generated each time a cron job runs. As we do
not enable SELinux, disabling these thing use subj_type=crond_t is not a
viable option.

1. I have tried the following way to exclude using msg_type and exe
together and it seems to work.

-a exclude,always -F msgtype=MAC_IPSEC_EVENT -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_AUTH -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_ACCT -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=CRED_REFR -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=CRED_DISP -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=CRED_ACQ -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_START -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_END -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=SERVICE_START -F exe=/usr/sbin/cron

Just want to make sure there is nothing I am missing here and that this
only excludes the msg types for the cron executable.

2. Apart from these messages, there is a LOGIN message that gets generated
each time a cron runs. Eventhough, the LOGIN message in auditd does not
have an exe field, the following statement surprisingly seems to be working.

-a exclude,always -F msgtype=LOGIN -F exe=/usr/sbin/cron

I can still see LOGIN messages for other users but the cron LOGIN messages
seem to be suppressed. Could you provide some detail as to how this is
happening and is the expected result.

3. Is there a better way to suppress these cron messages that I am not
considering apart from the SELinux option mentioned.

Thank You.

On Thu, Dec 9, 2021 at 8:19 AM Steve Grubb <sgrubb at redhat.com> wrote:

> Hello,
>
> On Wednesday, December 8, 2021 11:00:50 PM EST Amjad Gabbar wrote:
> > Got it. Makes sense to me. Thanks for the explanation Steve.
> >
> > One last thing though based on the discussion we had, if the kernel is
> able
> > to offload events even during bursts, wouldn’t setting q_depth
> > =backlog_limit be enough?
>
> Depends on the client attached to the af_unix socket. The kernel could
> have a
> burst and then another. If the client isn't reading the af_unix socket
> fast
> enough, there could still be  overflows. That said, it is likely to be
> enough.
> But I'd look into the client app that reads the af_unix socket to see why
> it's taking so long.
>
> > The reason being if there was an overflow on the kernel side, a different
> > message would be printed in the logs but because it is all dispatch
> errors,
> > I assume the kernel is able to handle the burst which is why the
> reasoning
> > of increasing q_depth to backlog_limit.
>
> The q_depth being 90 is your likely problem. Realistically, that only
> allows
> 20 - 30 events to be enqueued.
>
> -Steve
>
> > On Wed, Dec 8, 2021 at 4:38 PM Steve Grubb <sgrubb at redhat.com> wrote:
> > > On Wednesday, December 8, 2021 4:54:18 PM EST Amjad Gabbar wrote:
> > > > 1. The version of auditd is 1:2.8.4-3 and the plugins are
> af_unix.conf
> > >
> > > and
> > >
> > > > syslog.conf for audisp. The q_depth is currently set to 80 and I
> think
> > > > it
> > > > calls for an increase but not sure if there is a way to figure out
> what
> > >
> > > the
> > >
> > > > proper number would be?
> > >
> > > There is no good calculation that I can give you. It depends on the
> > > average
> > > rate of incoming events and the rate that they can be offloaded to the
> > > plugins
> > > + some margin in case there is a burst. Looking at the 2.8.5 code, the
> > > default is 250.
> > >
> > >
> https://github.com/linux-audit/audit-userspace/blob/2.8_maintenance/init.
> > > d/ audispd.conf
> > >
> > > So, you should at least set it that high. Maybe a bit higher.
> > >
> > > > 2. Another thing I would like to follow up on is the difference
> between
> > > > q_depth and backlog_limit. My assumption was if there is any drop due
> > > > to
> > >
> > > a
> > >
> > > > burst of events it would be addressed by the backlog limit. Just
> would
> > >
> > > like
> > >
> > > > some clarification on this and how this is an event dispatcher issue?
> > >
> > > The backlog limit is inside the kernel. This is the buffer that holds
> > > events
> > > that are waiting for the audit daemon to offload them. Once the audit
> > > daemon
> > > has them, it sends it to the dispatcher which also buffers events
> because
> > > not
> > > all plugins are able to receive the events as soon as they arrive at
> the
> > > dispatcher.
> > >
> > > So, for brief bursts, the kernel backlog will handle the load. But once
> > > they
> > > are pulled out of the kernel, the q_depth controls how much to hold
> > > waiting
> > > for plugins. If this number needs to increase much, then the plugins
> are
> > > having problems. The syslog plugin should be fine. I'd look more at the
> > > af_unix plugin. The client that attaches to it needs to unload events
> > > quickly. I'd investigate the af_unix client to see if it's the problem.
> > >
> > > Cheers,
> > > -Steve
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-audit/attachments/20211220/cef61902/attachment.htm>