[Linux-cluster] Logging with cluster

[isplist at logicore.net]

Solution: preprocessing logs

Great ideas, I'll look into some of these, thanks very much.



> We have a cluster of 6 machines, some running Apache, some running MySQL.
> We use shared logging successfully along with stats and post-processing
> scripts.  We also use plain-ol' logrotate with our shared logs.
> 
> We use network-enabled syslog to capture logging on every node to a single,
> master logging node (with fail-over, of course!)
> 
> For Apache, we use custom ErrorLog, CustomLog, and RewriteLog directives
> per vhost to pipe output to a custom script which greps a few undesirable
> statements out prior to logging.
> 
> Apache is sent to the local1 facility on the target syslog
> machine that holds all of our logs, where it's configured
> with something like:
> 
> /etc/syslog.conf:
> # Cluster Apache Logging
> local1.err                /var/log/shared-apache-err.log
> local1.notice                /var/log/shared-apache-access.log
> local1.debug                /var/log/shared-apache-rewrite.log
> 
> 
> And, for example, all Apache nodes use the same config akin to:
> 
> /path/to/http-vhost.conf:
> <snip>
> ErrorLog   "|/path/to/logger.pl err some_string_ID"
> CustomLog  "|/path/to/logger.pl notice some_string_ID"
> RewriteLog "|/path/to/logger.pl debug some_string_ID"
> </snip>
> 
> where logger.pl continually reads input, runs some filters
> to determine if it should indeed log the particular message,
> and then calls Sys::Syslog's "syslog()" function, and
> "some_string_ID" is a tag to identify each message in
> the shared log files.
> 
> You could really use any line-by-line filtering program
> here, but be aware that Apache executes the first argument
> after the pipe symbol directly - it doesn't run a shell or
> anything, so you don't have any expansion, piping of other
> commands, etc.
> 
> You can also use /usr/bin/logger (see "man logger") to
> send output to various facilities (localN) and informational
> levels (err, notice, debug, etc.).  This does the same
> thing as "logger.pl" above, but doesn't provide any
> filtering.
> 
> Also, we've seen syslog drop some messages under
> heavy load (hence why we filter some Apache logging
> prior to syslogging it).  I don't know the exact
> cause - maybe someone else can shed light on that for me!
> 
> 
> Hope this helps - it's what we do and it seems to work
> well enough for what we need.
> 
> Regards,
> -Brenton Rothchild