[almighty] Possible logging solutions

[Michael Kleinhenz]

Hi all,

----
Mgmt abstract: there are pretty common combinations for log
aggregations available. We need to decide if we need an intermediate
"quick-but-ugly" central logging now or if we can wait until the
system-provided logging solution is available. Consideration points:
only a few containers now, time to availabilty of system-logging
(TBD). We can build the intemediate solution so that migration later
on should be easy.
----

I looked into some tech for aggregating logs for almighty today. This
was a parking lot task due this week. To be clear: this is only to
provide some insight about how we *could* achieve centralized logging,
not to actually decide or build one of the possible solutions. The
info is to give an overview of that stuff.

Also, this currently only touches the logging for our own container
instances, not the per-user logging of a "client-usage" of the
pipeline. Although the solution, in the best case, should be able to
handle both later.

@KB: can you add details on the planned system-provided solution to
the mix? Thanks!

Aim is to have a single, centralized logging for all running
containers. I used a combination of the following components before
which turned out to be pretty neat:

- Logsprout, https://github.com/gliderlabs/logspout, MIT License
Connects to a docker daemon and collects logs from running containers
to be sent elsewhere (like, a syslog or a rest endpoint). The ususual
usecase is "a set of container instances run on one node, collect all
stdout/stderr logs of all running containers and send them to a syslog
or provide them on a single http endpoint". This one is not usable for
us, because it involves exposing the host's docker daemon, implying
that all containers are running on the same daemon. In our scenario,
we would rather use a direct connection from the running instance to a
central log sink (like Logstash, see below).

- Logstash, https://github.com/elastic/logstash, Apache License (also
Elasticsearch)
Collects, normalizes and stores (usually in Elasticsearch) any
incoming info. Usually used for collection and storing logs (but
defines itself more generic). Has lots of input and output plugins for
accepting a wide range of event/log sources and writing them to lots
of output sinks (Elasticsearch is only one of them). Looking at the
possible protocols, Docker and Logstash support two matching protocols
that could be used to send logs "directly" to Logstash: syslog and
gelf.

- Kibana, https://github.com/elastic/kibana, Apache License
Webfrontend for Elasticsearch that visualizes the data with pretty
advanced stuff. Optional, but nice to provide diagrams. This one might
be a candidate to look at when we are looking at the KPI stuff.

For a very simple intermediate solution, we could put together a
single container that includes most of the above components and acts
as a "single sink" for logs. Ugly, but works. I did that one before
(with logsprout as aggregator).

Decision needed: we are currently working with a low number of
containers. Are we even need a central logging at this stage? Or can
we wait until the system-provided logging is available?

Anyway, a possible custom solution should be compatible with a coming
system logging. I would think that syslog may be the
"supported-everywhere" variant.

For the above "ugly" solution, I would have some stuff already
available that we could use to build on. I would suppose setting this
up should not take longer than 1-2 days (has to be converted from
Ubuntu to Centos, may need some tweaking due to new releases of
contained software etc).

So, this for this mornings findings. :-)

-- Michael

-- 
Michael Kleinhenz
Principal Software Engineer

Red Hat Deutschland GmbH
Werner-von-Siemens-Ring 14
85630 Grasbrunn
Germany

RED HAT | TRIED. TESTED. TRUSTED.
Red Hat GmbH, www.de.redhat.com,
Registered seat: Grasbrunn, Commercial register: Amtsgericht München,
HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham,
Michael O'Neill