[almighty] Possible logging solutions

Tue Sep 20 16:05:39 UTC 2016

This is something that should be provided as a general feature of OpenShift.

alm currently has a very low need for a central logging solution at this
point in time, but will require it later. I would hold off to check when
OpenShift can deliver something in this space before we deploy our own.
alm could be a good chance for the OpenShift solution to beta test on a
real application, assuming that is only a month or two out.

-aslak-

On Tue, Sep 20, 2016 at 3:30 AM, Andrew Lee Rubinger <alr at redhat.com> wrote:

>
>
> On Tue, Sep 20, 2016 at 6:18 AM, Michael Kleinhenz <kleinhenz at redhat.com>
> wrote:
>
>> Hi all,
>>
>> ----
>> Mgmt abstract: there are pretty common combinations for log
>> aggregations available. We need to decide if we need an intermediate
>> "quick-but-ugly" central logging now or if we can wait until the
>> system-provided logging solution is available. Consideration points:
>> only a few containers now, time to availabilty of system-logging
>> (TBD). We can build the intemediate solution so that migration later
>> on should be easy.
>> ----
>>
>> I looked into some tech for aggregating logs for almighty today. This
>> was a parking lot task due this week. To be clear: this is only to
>> provide some insight about how we *could* achieve centralized logging,
>> not to actually decide or build one of the possible solutions. The
>> info is to give an overview of that stuff.
>>
>> Also, this currently only touches the logging for our own container
>> instances, not the per-user logging of a "client-usage" of the
>> pipeline. Although the solution, in the best case, should be able to
>> handle both later.
>>
>> @KB: can you add details on the planned system-provided solution to
>> the mix? Thanks!
>>
>> Aim is to have a single, centralized logging for all running
>> containers. I used a combination of the following components before
>> which turned out to be pretty neat:
>>
>> - Logsprout, https://github.com/gliderlabs/logspout, MIT License
>> Connects to a docker daemon and collects logs from running containers
>> to be sent elsewhere (like, a syslog or a rest endpoint). The ususual
>> usecase is "a set of container instances run on one node, collect all
>> stdout/stderr logs of all running containers and send them to a syslog
>> or provide them on a single http endpoint". This one is not usable for
>> us, because it involves exposing the host's docker daemon, implying
>> that all containers are running on the same daemon. In our scenario,
>> we would rather use a direct connection from the running instance to a
>> central log sink (like Logstash, see below).
>>
>> - Logstash, https://github.com/elastic/logstash, Apache License (also
>> Elasticsearch)
>> Collects, normalizes and stores (usually in Elasticsearch) any
>> incoming info. Usually used for collection and storing logs (but
>> defines itself more generic). Has lots of input and output plugins for
>> accepting a wide range of event/log sources and writing them to lots
>> of output sinks (Elasticsearch is only one of them). Looking at the
>> possible protocols, Docker and Logstash support two matching protocols
>> that could be used to send logs "directly" to Logstash: syslog and
>> gelf.
>>
>> - Kibana, https://github.com/elastic/kibana, Apache License
>> Webfrontend for Elasticsearch that visualizes the data with pretty
>> advanced stuff. Optional, but nice to provide diagrams. This one might
>> be a candidate to look at when we are looking at the KPI stuff.
>>
>> For a very simple intermediate solution, we could put together a
>> single container that includes most of the above components and acts
>> as a "single sink" for logs. Ugly, but works. I did that one before
>> (with logsprout as aggregator).
>>
>> Decision needed: we are currently working with a low number of
>> containers. Are we even need a central logging at this stage? Or can
>> we wait until the system-provided logging is available?
>
>
> Let's keep design a priority. :)  We want to be in the position where we
> can confidently replicate our patterns to new modules/containers as we
> introduce and integrate new features.  That said, we don't need to be
> perfect on the first swipe.
>
> So anything we can do to make a single stable logging service extension
> SPI now, even if it's a stupid hack whose implementation we soon swap out,
> centralizes the issue for us and future work here wouldn't impact the
> various systems calling upon it.
>
> S,
> ALR
>
>
>>
>> Anyway, a possible custom solution should be compatible with a coming
>> system logging. I would think that syslog may be the
>> "supported-everywhere" variant.
>>
>> For the above "ugly" solution, I would have some stuff already
>> available that we could use to build on. I would suppose setting this
>> up should not take longer than 1-2 days (has to be converted from
>> Ubuntu to Centos, may need some tweaking due to new releases of
>> contained software etc).
>>
>> So, this for this mornings findings. :-)
>>
>> -- Michael
>>
>> --
>> Michael Kleinhenz
>> Principal Software Engineer
>>
>> Red Hat Deutschland GmbH
>> Werner-von-Siemens-Ring 14
>> 85630 Grasbrunn
>> Germany
>>
>> RED HAT | TRIED. TESTED. TRUSTED.
>> Red Hat GmbH, www.de.redhat.com,
>> Registered seat: Grasbrunn, Commercial register: Amtsgericht München,
>> HRB 153243,
>> Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham,
>> Michael O'Neill
>>
>> _______________________________________________
>> almighty-public mailing list
>> almighty-public at redhat.com
>> https://www.redhat.com/mailman/listinfo/almighty-public
>>
>
>
>
> --
> Red Hat Developer Programs Architecture
> @ALRubinger
>
> _______________________________________________
> almighty-public mailing list
> almighty-public at redhat.com
> https://www.redhat.com/mailman/listinfo/almighty-public
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/almighty-public/attachments/20160920/aea3f443/attachment.htm>