[Freeipa-users] Logging: IPA to Rsyslog to Logstash

[Innes, Duncan]

Earlier this year I said I'd feed back how my IPA to Rsyslog to Logstash
experiments went.

They went badly.  And I didn't get much time.  Today, however, I managed
to get over my imaginary finishing line:

All systems are RHEL 6.6.

Rsyslog (rsyslog7-7.4.10) is configured to import logs from some dirsrv
files:

# cat /etc/rsyslog.d/dirsrv.conf 
module(load="imfile" PollingInterval="2")

input(type="imfile"
      File="/var/log/dirsrv/slapd-EXAMPLE-COM/access"
      Tag="dirsrv"
      StateFile="statedirsrv"
      Facility="local0")

input(type="imfile"
      File="/var/log/dirsrv/slapd-EXAMPLE-COM/errors"
      Tag="dirsrv"
      StateFile="statedirsrverr"
      Severity="error"
      Facility="local0")

#

This pulls in those log entries on a regular basis.  Rsyslog8 allows you
to use inotify for file changes, but that's not available to me.

Rsyslog is then also configured to push all logs to my Logstash servers:

# cat /etc/rsyslog.d/logstash.conf 
template(name="ls_json" type="list" option.json="on")
{ constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timegenerated"
dateFormat="rfc3339")
constant(value="\",\"@version\":\"1")
constant(value="\",\"message\":\"") property(name="msg")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"my_environment\":\"dev")
constant(value="\",\"my_project\":\"Infrastructure")
constant(value="\",\"my_use\":\"IPA")
constant(value="\",\"logsource\":\"") property(name="fromhost")
constant(value="\",\"severity_label\":\"")
property(name="syslogseverity-text")
constant(value="\",\"severity\":\"") property(name="syslogseverity")
constant(value="\",\"facility_label\":\"")
property(name="syslogfacility-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility")
constant(value="\",\"program\":\"") property(name="programname")
constant(value="\",\"pid\":\"") property(name="procid")
constant(value="\",\"rawmsg\":\"") property(name="rawmsg")
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
constant(value="\"}\n")
}

*.* @@logstash01.example.com:5500;ls_json
$ActionExecOnlyWhenPreviousIsSuspended on
& @@logstash02.example.com:5500;ls_json
& /var/log/localbuffer
$ActionExecOnlyWhenPreviousIsSuspended off

[root at lvdlvldap02 ~]#

Which pushes all logs to my logstash servers in JSON format.  Failover
is built in by using 2 logstash servers.
The client needs to have SELinux managed to allow rsyslog to write to
port 5500:

# semanage port -a -t syslogd_port_t -p tcp 5500
# semanage port -l | grep 5500

The Logstash servers are then configured to listen on this port and do
some simple groking, before sending everything to the ElasticSearch
cluster:

# cat /etc/logstash/conf.d/syslog.conf 
input {
  tcp {
    type => syslogjson
    port => 5500
    codec => "json"
  }
}

filter {
  # This replaces the host field (UDP source) with the host that
generated the message (sysloghost)
  if [sysloghost] {
    mutate {
      replace => [ "host", "%{sysloghost}" ]
      remove_field => "sysloghost" # prune the field after successfully
replacing "host"
    }
  }
  if [type] == "syslogjson" {
    grok {
      patterns_dir => "/opt/logstash/patterns"
      match => { "message" => "%{VIRGINFW}" }
      match => { "message" => "%{AUDITAVC}" }
      match => { "message" => "%{COMMONAPACHELOG}" }
      tag_on_failure => []
    }
  }

  # This filter populates the @timestamp field with the timestamp that's
in the actual message
  # dirsrv logs are currently pulled in every 2 minutes, so @timestamp
is wrong
  if [syslogtag] == "dirsrv" {
    mutate {
      remove_field => [ 'rawmsg' ]
    }
    grok {
      match => [ "message", "%{HTTPDATE:log_timestamp}" ]
    }
    date {
      match => [ "log_timestamp", "dd/MMM/YYY:HH:mm:ss Z"]
      locale => "en"
      remove_field => [ "log_timestamp" ]
    }
  }
}

output {
  elasticsearch {
    protocol => node
    node_name => "Indexer01"
  }
}
#

It works well for the most part.  I'm not performing any groking of the
actual message line as yet to pull out various bits of data into their
own separate fields, but at least I'm managing to log the access and
errors from multiple IPA servers.

The @timestamp field ends up with the timestamp from the actual message
line, so it's only down to second accuracy.  This means that multiple
log lines on the same second lose their ordering when viewed in the
Logstash/Kibana interface.  But the important thing at this point is
that they're now held centrally.

Is it feasible to alter the timestamp resolution that dirsrv uses?  This
would help separate log lines properly.

Cheers & Merry Festive Holiday thing

Duncan

This message has been checked for viruses and spam by the Virgin Money email scanning system powered by Messagelabs.

This e-mail is intended to be confidential to the recipient. If you receive a copy in error, please inform the sender and then delete this message.

Virgin Money plc - Registered in England and Wales (Company no. 6952311). Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. Virgin Money plc is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority.

The following companies also trade as Virgin Money. They are both authorised and regulated by the Financial Conduct Authority, are registered in England and Wales and have their registered office at Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482).

For further details of Virgin Money group companies please visit our website at virginmoney.com