[Linux-cluster] Logging with cluster

[Brenton Rothchild]

isplist at logicore.net wrote:
> Repeat Post: Anyone have ideas on how to deal with multiple machines thinking 
> they own the shared log?
> 
> ---
> 
> It dawns on me that I'm not sure how to delete some logging that is being 
> shared in a cluster. For example, apache, the log is shared and all web 
> servers write to the same file. If I nuke the file after running a stats 
> program, the rest of the web servers don't know about it.
> 
> Should I be doing shared logging or is there a way to handle this?
> 
> Mike
> 

We have a cluster of 6 machines, some running Apache, some running MySQL.
We use shared logging successfully along with stats and post-processing
scripts.  We also use plain-ol' logrotate with our shared logs.

We use network-enabled syslog to capture logging on every node to a single,
master logging node (with fail-over, of course!)

For Apache, we use custom ErrorLog, CustomLog, and RewriteLog directives
per vhost to pipe output to a custom script which greps a few undesirable
statements out prior to logging.

Apache is sent to the local1 facility on the target syslog
machine that holds all of our logs, where it's configured
with something like:

/etc/syslog.conf:
# Cluster Apache Logging
local1.err		/var/log/shared-apache-err.log
local1.notice		/var/log/shared-apache-access.log
local1.debug		/var/log/shared-apache-rewrite.log



And, for example, all Apache nodes use the same config akin to:

/path/to/http-vhost.conf:
<snip>
   ErrorLog   "|/path/to/logger.pl err some_string_ID"
   CustomLog  "|/path/to/logger.pl notice some_string_ID"
   RewriteLog "|/path/to/logger.pl debug some_string_ID"
</snip>

where logger.pl continually reads input, runs some filters
to determine if it should indeed log the particular message,
and then calls Sys::Syslog's "syslog()" function, and
"some_string_ID" is a tag to identify each message in
the shared log files.

You could really use any line-by-line filtering program
here, but be aware that Apache executes the first argument
after the pipe symbol directly - it doesn't run a shell or
anything, so you don't have any expansion, piping of other
commands, etc.

You can also use /usr/bin/logger (see "man logger") to
send output to various facilities (localN) and informational
levels (err, notice, debug, etc.).  This does the same
thing as "logger.pl" above, but doesn't provide any
filtering.

Also, we've seen syslog drop some messages under
heavy load (hence why we filter some Apache logging
prior to syslogging it).  I don't know the exact
cause - maybe someone else can shed light on that for me!



Hope this helps - it's what we do and it seems to work
well enough for what we need.

Regards,
-Brenton Rothchild