[Freeipa-devel] Suggestion for the A part of IPA

[Martin Kosek]

On 04/28/2015 05:58 PM, Innes, Duncan wrote:
> Folks,
>  
> The A part of IPA has always been of great interest to me.  Our current
> IPA infrastructure works well at the I & P parts, giving us great
> failover abilities and connectivity through hardware firewalls without
> punching too many holes.

Good to hear :-) We recently also started investigating the Audit capabilities
for (notice I write "for" and not "in") IPA. You can check my initial nudge to
the freeipa-users list, which was unfortunately with no reply:

https://www.redhat.com/archives/freeipa-users/2015-March/msg00940.html

For the beginning, I would be interested for your use cases, if you are only
looking for a centralized log store, or you are also looking for more analytics
in the logs (like what API commands were run, user logins, etc.) or utilization
of the server core services (LDAP/Kerberos/DNS/...)

> Whilst the A part may not be solely about centralised logging, it's the
> thing I've been looking into recently.  To do this I've built a setup
> around the ELK stack using a pair of Logstash servers and an
> ElasticSearch cluster of 5 servers (overkill on the ES side perhaps, but
> this is proof of concept still).  To expand on this, I've been looking
> at running the Logstash serviceon each of our IPA servers as that gives
> us a failover pair in each part of our network.  The Logstash servers
> then connect to the ES cluster as non-data nodes.  Each client has an
> rsyslog7 (still using RHEL6 at the moment) config that writes sends the
> logs in JSON format with some extra bespoke fields added (such as
> Project, Environment, and Use to help us search better).  The sending is
> done in rsyslog's rather clunky failover method to the local pair of
> Logstash servers (with a third failover being to /dev/null).

Ah, so you are running Logstash service on each IPA service? Isn't that too
heavyweight? In our tests, we mostly simply wanted just configure rsyslog and
get the logs out of the server, to the centralized ELK/REK/EFK servers which
did the heavy lifting. After all, the IPA servers may be of different
environments (Fedora, RHEL, CentOS, ...) and with different versions of the log
processing software.

On the REK server (yes, we did not use logstash at the POC), we are able to
process the logs (make them structured), store and display them. This allows us
do searches like "list of admins which added users in the last month".

This if course required adding parsing rules to rsyslog to get the structure
out of the API logs. Are you using logstash for the parsing or did you not
start the parsing part yet?

> It struck me that this kind of setup might not be too far removed from
> some of the A part of IPA.

The centralized log processing itself is a too big task for IPA itself, it is
specialized on other things. But some integration should be added in time, I
agree. It may be minimal, from top of my head for example:

* Support of (rsyslog) configuration in ipa-client-install or ipa-server-install
* Providing the secure, GSSAPI-based log transfer to the IPA clients and
ELK/REK/EFK server
* Providing parsing templates for rsyslog or base queries for Kibana

> I'm not good at ASCII flowchart diagrams, so will leave it there for
> now.  The main point of this - does any of this idea sound reasonable to
> add in to FreeIPA?  To me it sounds like a good fit for getting (some)
> logging data back to a central point.
>  
> The Logstash indexers currently have a very low load (perhaps due to the
> incoming data already being JSON) and small memory footprint.  They run
> without issue on our IPA servers.  The ES nodes are different and I
> won't pretent to be any sort of expert in what they do.  They load up a
> bit when I shut 1 of them down, but that's the rebalancing happening.
>  
> Apologies if this is off topic, or wide of the mark.
>  
> Cheers
>  
> Duncan Innes
> 
> This message has been checked for viruses and spam by the Virgin Money email scanning system powered by Messagelabs.
> 
> This e-mail is intended to be confidential to the recipient. If you receive a copy in error, please inform the sender and then delete this message.
> 
> Virgin Money plc - Registered in England and Wales (Company no. 6952311). Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. Virgin Money plc is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority.
> 
> The following companies also trade as Virgin Money. They are both authorised and regulated by the Financial Conduct Authority, are registered in England and Wales and have their registered office at Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482).
> 
> For further details of Virgin Money group companies please visit our website at virginmoney.com
> 
> 
>