[almighty] Possible logging solutions

[Andrew Lee Rubinger]

On Tue, Sep 20, 2016 at 6:18 AM, Michael Kleinhenz <kleinhenz at redhat.com>
wrote:

> Hi all,
>
> ----
> Mgmt abstract: there are pretty common combinations for log
> aggregations available. We need to decide if we need an intermediate
> "quick-but-ugly" central logging now or if we can wait until the
> system-provided logging solution is available. Consideration points:
> only a few containers now, time to availabilty of system-logging
> (TBD). We can build the intemediate solution so that migration later
> on should be easy.
> ----
>
> I looked into some tech for aggregating logs for almighty today. This
> was a parking lot task due this week. To be clear: this is only to
> provide some insight about how we *could* achieve centralized logging,
> not to actually decide or build one of the possible solutions. The
> info is to give an overview of that stuff.
>
> Also, this currently only touches the logging for our own container
> instances, not the per-user logging of a "client-usage" of the
> pipeline. Although the solution, in the best case, should be able to
> handle both later.
>
> @KB: can you add details on the planned system-provided solution to
> the mix? Thanks!
>
> Aim is to have a single, centralized logging for all running
> containers. I used a combination of the following components before
> which turned out to be pretty neat:
>
> - Logsprout, https://github.com/gliderlabs/logspout, MIT License
> Connects to a docker daemon and collects logs from running containers
> to be sent elsewhere (like, a syslog or a rest endpoint). The ususual
> usecase is "a set of container instances run on one node, collect all
> stdout/stderr logs of all running containers and send them to a syslog
> or provide them on a single http endpoint". This one is not usable for
> us, because it involves exposing the host's docker daemon, implying
> that all containers are running on the same daemon. In our scenario,
> we would rather use a direct connection from the running instance to a
> central log sink (like Logstash, see below).
>
> - Logstash, https://github.com/elastic/logstash, Apache License (also
> Elasticsearch)
> Collects, normalizes and stores (usually in Elasticsearch) any
> incoming info. Usually used for collection and storing logs (but
> defines itself more generic). Has lots of input and output plugins for
> accepting a wide range of event/log sources and writing them to lots
> of output sinks (Elasticsearch is only one of them). Looking at the
> possible protocols, Docker and Logstash support two matching protocols
> that could be used to send logs "directly" to Logstash: syslog and
> gelf.
>
> - Kibana, https://github.com/elastic/kibana, Apache License
> Webfrontend for Elasticsearch that visualizes the data with pretty
> advanced stuff. Optional, but nice to provide diagrams. This one might
> be a candidate to look at when we are looking at the KPI stuff.
>
> For a very simple intermediate solution, we could put together a
> single container that includes most of the above components and acts
> as a "single sink" for logs. Ugly, but works. I did that one before
> (with logsprout as aggregator).
>
> Decision needed: we are currently working with a low number of
> containers. Are we even need a central logging at this stage? Or can
> we wait until the system-provided logging is available?


Let's keep design a priority. :)  We want to be in the position where we
can confidently replicate our patterns to new modules/containers as we
introduce and integrate new features.  That said, we don't need to be
perfect on the first swipe.

So anything we can do to make a single stable logging service extension SPI
now, even if it's a stupid hack whose implementation we soon swap out,
centralizes the issue for us and future work here wouldn't impact the
various systems calling upon it.

S,
ALR


>
> Anyway, a possible custom solution should be compatible with a coming
> system logging. I would think that syslog may be the
> "supported-everywhere" variant.
>
> For the above "ugly" solution, I would have some stuff already
> available that we could use to build on. I would suppose setting this
> up should not take longer than 1-2 days (has to be converted from
> Ubuntu to Centos, may need some tweaking due to new releases of
> contained software etc).
>
> So, this for this mornings findings. :-)
>
> -- Michael
>
> --
> Michael Kleinhenz
> Principal Software Engineer
>
> Red Hat Deutschland GmbH
> Werner-von-Siemens-Ring 14
> 85630 Grasbrunn
> Germany
>
> RED HAT | TRIED. TESTED. TRUSTED.
> Red Hat GmbH, www.de.redhat.com,
> Registered seat: Grasbrunn, Commercial register: Amtsgericht München,
> HRB 153243,
> Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham,
> Michael O'Neill
>
> _______________________________________________
> almighty-public mailing list
> almighty-public at redhat.com
> https://www.redhat.com/mailman/listinfo/almighty-public
>



-- 
Red Hat Developer Programs Architecture
@ALRubinger
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/almighty-public/attachments/20160920/b2b90816/attachment.htm>