Auditd reconfigure using SIGHUP

[Bhagwat, Shriniketan Manjunath]

Hi Steve,

Thanks for acknowledging the issue. 

In my last email I missed mentioning the fix that I have implemented.

Issue 1) As you said, I have fixed it  by replacing the ev_signal by ev_child as below.

struct ev_child sigchld_watcher;

        ev_child_init (&sigchld_watcher, child_handler, 0, 0);
        ev_child_start (EV_DEFAULT_ &sigchld_watcher);

static void child_handler(EV_P_ ev_child *w, int revents)
{
        int pid;

        if (w->rpid == dispatcher_pid()) {
                       dispatcher_reaped();
        }
}

Issue 2) In auditd.c main(), child_handler is registered not immediately after init_dispatcher() is called. I have modified the audit to register ev_child immediately after init_dispatcher() as below. Or maybe before calling init_dispatcher(). This fixed issue 2 for me. Below extract is from documentation of libev for ev_child: 
" It is permissible to install a child watcher after the child has been forked (which implies it might have already exited), as long as the event loop isn't entered (or is continued from a watcher), i.e., forking and then immediately registering a watcher for the child is fine, but forking and registering a watcher a few event loop iterations later or in the next callback invocation is not." 

        if (init_dispatcher(&config)) {
                if (pidfile)
                        unlink(pidfile);
                tell_parent(FAILURE);
                return 1;
        }
        ev_child_init (&sigchld_watcher, child_handler, 0, 0);
        ev_child_start (EV_DEFAULT_ &sigchld_watcher);

Issue 3) With the above fix for issue 2, I did not see the issue 3 getting occurred for me. This could be because shutdown_dispatcher() is called from dispatcher_reaped() where the status on the pipe is not checked.  

Regards,
Ketan

-----Original Message-----
From: Steve Grubb [mailto:sgrubb at redhat.com] 
Sent: Friday, January 06, 2017 4:12 AM
To: linux-audit at redhat.com
Cc: Bhagwat, Shriniketan Manjunath <shriniketan.bhagwat at hpe.com>
Subject: Re: Auditd reconfigure using SIGHUP

On Thursday, January 5, 2017 11:04:55 AM EST Bhagwat, Shriniketan Manjunath
wrote:
> Greetings,
> 
> I was trying to reconfigure the Audit using SIGHUP and came across 
> with below behavior of Audit. The audit version I am using is 2.3.6.

I have been able to reproduce the issue. I am still tracing through the problem. As best I can tell, there are at least 3 problems. 1) libev eating the signal - this will be fixed by defining EV_CHILD_ENABLE to 0. This will make all child interference go away; 2) on startup audispd detects no plugins and exits before auditd can register a handler in scenario 2 - this might be fixed with a sigaction handler until libev starts processing events; 3) pipe descriptors being invalid in audispd sometimes. This one is a mystery.

Still investigating. Thanks for reporting this.

-Steve

> Scenario 1: Starting auditd with active audispd plugin.
> 1. Activate the audispd plugin by setting active = yes in 
> configuration file
> (example: /etc/audisp/plugins.d/syslog.conf) 2. Start the Auditd. 
> Starting auditd will create the both Auditd and Audispd process. 3. 
> Deactivate the audispd plugin by setting active = no in configuration 
> file. 4. Reconfigure the Auditd by sending SIGHUP to Auditd (>> kill 
> -SIGHUP <auditd_pid>). This reconfiguration stops the audispd process. 
> 5. Activate the audispd plugin by setting active = yes in 
> configuration file. 6. Reconfigure the Auditd by sending SIGHUP to 
> Auditd. This should start the audispd process, however audispd process will not be started in this reconfiguration.
> 
> Scenario2: Starting auditd with no active audispd plugin.
> 1. Start the Auditd. Starting auditd will create the only Auditd process.
> 2. Activate the audispd plugin by setting active = yes in 
> configuration file. 3. Reconfigure the Auditd by sending SIGHUP to 
> Auditd. This should start the audispd process, however in few cases 
> audispd process will not be started in this reconfiguration. 4. 
> Deactivate the audispd plugin by setting active = no in configuration 
> file. 5. Reconfigure the Auditd by sending SIGHUP to Auditd. This reconfiguration stops the audispd process.
> 6. Activate the audispd plugin by setting active = yes in 
> configuration file. 7. Reconfigure the Auditd by sending SIGHUP to 
> Auditd. This should start the audispd process, however audispd process 
> will not be started in this reconfiguration.
> 
> 
> As per the change log of audit version  2.3.2, below fix was made.
> 
> Fix: In auditd, restart dispatcher on SIGHUP if it had previously 
> exited
> 
> 
> 
> I have analyzed Auditd code of version 2.3.6 and below is my observation.
> 
> When the Auditd is started, it calls init_dispatcher() to start the Audispd.
> init_dispatcher() starts the Audispd and maintains its pid value in a 
> global variable (auditd-dispatch.c). When the audispd is terminated, 
> SIGCHLD handler of Auditd i.e child_handler() does the waitpid() to 
> remove the child process from zombie state and calls 
> dispatcher_reaped(), where this pid is set to zero. During reconfigure 
> using SIGHUP,
> reconfigure_dispatcher() checks for this pid value. If pid is valid 
> then, SIGHUP is sent to audispd otherwise init_dispatcher() is called 
> to start the audispd.
> 
> 
> Auditd uses event loop ev_signal to track the child process (SIGCHLD). 
> The event loop from libev also has child handler childcb() (in ev.c). 
> This handler childcb() also does the waitpid(). When audispd 
> terminates, libev's child handler is getting called first, waitpid() 
> removes the audispd from zombie state. Then the control is passed to 
> Auditd's child handler child_handler(). Since the audispd has been 
> already removed from process stable, waitpid() call done in 
> child_handler() will return ECHILD. Hence the dispatcher_reaped() is 
> not getting called to make the internally maintained pid variable to 
> zero. This results in subsequent SIGHUP set to Auditd end up in sending SIGHUP to non-existing audispd process.
> 
> In case of step 3 of scenario 2: When the auditd is started without 
> any active audispd plugin(step1), Auditd starts the audispd. However 
> since there are no active plugins Audispd will get terminated. In this 
> case
> shutdown_dispatcher() was called from dispatch_event() where pid value 
> was set to zero. Hence the reconfiguration in step 3 started audispd 
> process in my case.
> 
> I have put traces in auditd code to validate the above behavior. This 
> behavior is applicable for Audit version 2.6.4 as well.  Is this 
> behavior a known issue?
> 
> Regards,
> Ketan