[PATCH] audit: improve robustness of the audit queue handling

[Richard Guy Briggs]

On 2021-12-13 13:31, Paul Moore wrote:
> If the audit daemon were ever to get stuck in a stopped state the
> kernel's kauditd_thread() could get blocked attempting to send audit
> records to the userspace audit daemon.  With the kernel thread
> blocked it is possible that the audit queue could grow unbounded as
> certain audit record generating events must be exempt from the queue
> limits else the system enter a deadlock state.
> 
> This patch resolves this problem by lowering the kernel thread's
> socket sending timeout from MAX_SCHEDULE_TIMEOUT to HZ/10 and tweaks
> the kauditd_send_queue() function to better manage the various audit
> queues when connection problems occur between the kernel and the
> audit daemon.  With this patch, the backlog may temporarily grow
> beyond the defined limits when the audit daemon is stopped and the
> system is under heavy audit pressure, but kauditd_thread() will
> continue to make progress and drain the queues as it would for other
> connection problems.  For example, with the audit daemon put into a
> stopped state and the system configured to audit every syscall it
> was still possible to shutdown the system without a kernel panic,
> deadlock, etc.; granted, the system was slow to shutdown but that is
> to be expected given the extreme pressure of recording every syscall.

I assume that in the configuration state of f=2, it would still panic if
it was not able to deliver messages.

This seems like a reasonable approach, FWIW: Reviewed-by

> The timeout value of HZ/10 was chosen primarily through
> experimentation and this developer's "gut feeling".  There is likely
> no one perfect value, but as this scenario is limited in scope (root
> privileges would be needed to send SIGSTOP to the audit daemon), it
> is likely not worth exposing this as a tunable at present.  This can
> always be done at a later date if it proves necessary.
> 
> Cc: stable at vger.kernel.org
> Fixes: 5b52330bbfe63 ("audit: fix auditd/kernel connection state tracking")
> Reported-by: Gaosheng Cui <cuigaosheng1 at huawei.com>
> Signed-off-by: Paul Moore <paul at paul-moore.com>
> ---
>  kernel/audit.c |   21 ++++++++++-----------
>  1 file changed, 10 insertions(+), 11 deletions(-)
> 
> diff --git a/kernel/audit.c b/kernel/audit.c
> index 121d37e700a6..4cebadb5f30d 100644
> --- a/kernel/audit.c
> +++ b/kernel/audit.c
> @@ -718,7 +718,7 @@ static int kauditd_send_queue(struct sock *sk, u32 portid,
>  {
>  	int rc = 0;
>  	struct sk_buff *skb;
> -	static unsigned int failed = 0;
> +	unsigned int failed = 0;
>  
>  	/* NOTE: kauditd_thread takes care of all our locking, we just use
>  	 *       the netlink info passed to us (e.g. sk and portid) */
> @@ -735,32 +735,30 @@ static int kauditd_send_queue(struct sock *sk, u32 portid,
>  			continue;
>  		}
>  
> +retry:
>  		/* grab an extra skb reference in case of error */
>  		skb_get(skb);
>  		rc = netlink_unicast(sk, skb, portid, 0);
>  		if (rc < 0) {
> -			/* fatal failure for our queue flush attempt? */
> +			/* send failed - try a few times unless fatal error */
>  			if (++failed >= retry_limit ||
>  			    rc == -ECONNREFUSED || rc == -EPERM) {
> -				/* yes - error processing for the queue */
>  				sk = NULL;
>  				if (err_hook)
>  					(*err_hook)(skb);
> -				if (!skb_hook)
> -					goto out;
> -				/* keep processing with the skb_hook */
> +				if (rc == -EAGAIN)
> +					rc = 0;
> +				/* continue to drain the queue */
>  				continue;
>  			} else
> -				/* no - requeue to preserve ordering */
> -				skb_queue_head(queue, skb);
> +				goto retry;
>  		} else {
> -			/* it worked - drop the extra reference and continue */
> +			/* skb sent - drop the extra reference and continue */
>  			consume_skb(skb);
>  			failed = 0;
>  		}
>  	}
>  
> -out:
>  	return (rc >= 0 ? 0 : rc);
>  }
>  
> @@ -1609,7 +1607,8 @@ static int __net_init audit_net_init(struct net *net)
>  		audit_panic("cannot initialize netlink socket in namespace");
>  		return -ENOMEM;
>  	}
> -	aunet->sk->sk_sndtimeo = MAX_SCHEDULE_TIMEOUT;
> +	/* limit the timeout in case auditd is blocked/stopped */
> +	aunet->sk->sk_sndtimeo = HZ / 10;
>  
>  	return 0;
>  }
> 
> --
> Linux-audit mailing list
> Linux-audit at redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-audit
> 

- RGB

--
Richard Guy Briggs <rgb at redhat.com>
Sr. S/W Engineer, Kernel Security, Base Operating Systems
Remote, Ottawa, Red Hat Canada
IRC: rgb, SunRaycer
Voice: +1.647.777.2635, Internal: (81) 32635