[Freeipa-users] Logging: IPA to Rsyslog to Logstash

[Petr Spacek]

Hello Duncan,

thank you for doing this!

Could you transform this post to
http://www.freeipa.org/page/HowTos#Working_with_FreeIPA article, please? I
think that other people could use that too.

Thank you very much.
Petr^2 Spacek

On 19.12.2014 17:35, Innes, Duncan wrote:
> Earlier this year I said I'd feed back how my IPA to Rsyslog to Logstash
> experiments went.
> 
> They went badly.  And I didn't get much time.  Today, however, I managed
> to get over my imaginary finishing line:
> 
> All systems are RHEL 6.6.
> 
> Rsyslog (rsyslog7-7.4.10) is configured to import logs from some dirsrv
> files:
> 
> # cat /etc/rsyslog.d/dirsrv.conf 
> module(load="imfile" PollingInterval="2")
> 
> input(type="imfile"
>       File="/var/log/dirsrv/slapd-EXAMPLE-COM/access"
>       Tag="dirsrv"
>       StateFile="statedirsrv"
>       Facility="local0")
> 
> input(type="imfile"
>       File="/var/log/dirsrv/slapd-EXAMPLE-COM/errors"
>       Tag="dirsrv"
>       StateFile="statedirsrverr"
>       Severity="error"
>       Facility="local0")
> 
> #
> 
> This pulls in those log entries on a regular basis.  Rsyslog8 allows you
> to use inotify for file changes, but that's not available to me.
> 
> Rsyslog is then also configured to push all logs to my Logstash servers:
> 
> # cat /etc/rsyslog.d/logstash.conf 
> template(name="ls_json" type="list" option.json="on")
> { constant(value="{")
> constant(value="\"@timestamp\":\"") property(name="timegenerated"
> dateFormat="rfc3339")
> constant(value="\",\"@version\":\"1")
> constant(value="\",\"message\":\"") property(name="msg")
> constant(value="\",\"host\":\"") property(name="hostname")
> constant(value="\",\"my_environment\":\"dev")
> constant(value="\",\"my_project\":\"Infrastructure")
> constant(value="\",\"my_use\":\"IPA")
> constant(value="\",\"logsource\":\"") property(name="fromhost")
> constant(value="\",\"severity_label\":\"")
> property(name="syslogseverity-text")
> constant(value="\",\"severity\":\"") property(name="syslogseverity")
> constant(value="\",\"facility_label\":\"")
> property(name="syslogfacility-text")
> constant(value="\",\"facility\":\"") property(name="syslogfacility")
> constant(value="\",\"program\":\"") property(name="programname")
> constant(value="\",\"pid\":\"") property(name="procid")
> constant(value="\",\"rawmsg\":\"") property(name="rawmsg")
> constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
> constant(value="\"}\n")
> }
> 
> *.* @@logstash01.example.com:5500;ls_json
> $ActionExecOnlyWhenPreviousIsSuspended on
> & @@logstash02.example.com:5500;ls_json
> & /var/log/localbuffer
> $ActionExecOnlyWhenPreviousIsSuspended off
> 
> [root at lvdlvldap02 ~]#
> 
> Which pushes all logs to my logstash servers in JSON format.  Failover
> is built in by using 2 logstash servers.
> The client needs to have SELinux managed to allow rsyslog to write to
> port 5500:
> 
> # semanage port -a -t syslogd_port_t -p tcp 5500
> # semanage port -l | grep 5500
> 
> The Logstash servers are then configured to listen on this port and do
> some simple groking, before sending everything to the ElasticSearch
> cluster:
> 
> # cat /etc/logstash/conf.d/syslog.conf 
> input {
>   tcp {
>     type => syslogjson
>     port => 5500
>     codec => "json"
>   }
> }
> 
> filter {
>   # This replaces the host field (UDP source) with the host that
> generated the message (sysloghost)
>   if [sysloghost] {
>     mutate {
>       replace => [ "host", "%{sysloghost}" ]
>       remove_field => "sysloghost" # prune the field after successfully
> replacing "host"
>     }
>   }
>   if [type] == "syslogjson" {
>     grok {
>       patterns_dir => "/opt/logstash/patterns"
>       match => { "message" => "%{VIRGINFW}" }
>       match => { "message" => "%{AUDITAVC}" }
>       match => { "message" => "%{COMMONAPACHELOG}" }
>       tag_on_failure => []
>     }
>   }
> 
>   # This filter populates the @timestamp field with the timestamp that's
> in the actual message
>   # dirsrv logs are currently pulled in every 2 minutes, so @timestamp
> is wrong
>   if [syslogtag] == "dirsrv" {
>     mutate {
>       remove_field => [ 'rawmsg' ]
>     }
>     grok {
>       match => [ "message", "%{HTTPDATE:log_timestamp}" ]
>     }
>     date {
>       match => [ "log_timestamp", "dd/MMM/YYY:HH:mm:ss Z"]
>       locale => "en"
>       remove_field => [ "log_timestamp" ]
>     }
>   }
> }
> 
> output {
>   elasticsearch {
>     protocol => node
>     node_name => "Indexer01"
>   }
> }
> #
> 
> It works well for the most part.  I'm not performing any groking of the
> actual message line as yet to pull out various bits of data into their
> own separate fields, but at least I'm managing to log the access and
> errors from multiple IPA servers.
> 
> The @timestamp field ends up with the timestamp from the actual message
> line, so it's only down to second accuracy.  This means that multiple
> log lines on the same second lose their ordering when viewed in the
> Logstash/Kibana interface.  But the important thing at this point is
> that they're now held centrally.
> 
> Is it feasible to alter the timestamp resolution that dirsrv uses?  This
> would help separate log lines properly.
> 
> Cheers & Merry Festive Holiday thing
> 
> Duncan
> 
> This message has been checked for viruses and spam by the Virgin Money email scanning system powered by Messagelabs.
> 
> This e-mail is intended to be confidential to the recipient. If you receive a copy in error, please inform the sender and then delete this message.
> 
> Virgin Money plc - Registered in England and Wales (Company no. 6952311). Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. Virgin Money plc is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority.
> 
> The following companies also trade as Virgin Money. They are both authorised and regulated by the Financial Conduct Authority, are registered in England and Wales and have their registered office at Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482).
> 
> For further details of Virgin Money group companies please visit our website at virginmoney.com
> 


-- 
Petr^2 Spacek