[almighty] Possible logging solutions

Wed Sep 21 11:58:28 UTC 2016

On 20 Sep 2016, at 18:05, Aslak Knutsen wrote:

> This is something that should be provided as a general feature of 
> OpenShift.

>
> alm currently has a very low need for a central logging solution at 
> this
> point in time, but will require it later. I would hold off to check 
> when
> OpenShift can deliver something in this space before we deploy our 
> own.
> alm could be a good chance for the OpenShift solution to beta test on 
> a
> real application, assuming that is only a month or two out.

But that is the issue we got - it does not seem to be something yet 
provided in openshift
and KB said something like 3-6 months before it would be there.

KB - can you provide the background info you got from openshift team ?

/max

> -aslak-
>
> On Tue, Sep 20, 2016 at 3:30 AM, Andrew Lee Rubinger <alr at redhat.com> 
> wrote:
>
>>
>>
>> On Tue, Sep 20, 2016 at 6:18 AM, Michael Kleinhenz 
>> <kleinhenz at redhat.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> ----
>>> Mgmt abstract: there are pretty common combinations for log
>>> aggregations available. We need to decide if we need an intermediate
>>> "quick-but-ugly" central logging now or if we can wait until the
>>> system-provided logging solution is available. Consideration points:
>>> only a few containers now, time to availabilty of system-logging
>>> (TBD). We can build the intemediate solution so that migration later
>>> on should be easy.
>>> ----
>>>
>>> I looked into some tech for aggregating logs for almighty today. 
>>> This
>>> was a parking lot task due this week. To be clear: this is only to
>>> provide some insight about how we *could* achieve centralized 
>>> logging,
>>> not to actually decide or build one of the possible solutions. The
>>> info is to give an overview of that stuff.
>>>
>>> Also, this currently only touches the logging for our own container
>>> instances, not the per-user logging of a "client-usage" of the
>>> pipeline. Although the solution, in the best case, should be able to
>>> handle both later.
>>>
>>> @KB: can you add details on the planned system-provided solution to
>>> the mix? Thanks!
>>>
>>> Aim is to have a single, centralized logging for all running
>>> containers. I used a combination of the following components before
>>> which turned out to be pretty neat:
>>>
>>> - Logsprout, https://github.com/gliderlabs/logspout, MIT License
>>> Connects to a docker daemon and collects logs from running 
>>> containers
>>> to be sent elsewhere (like, a syslog or a rest endpoint). The 
>>> ususual
>>> usecase is "a set of container instances run on one node, collect 
>>> all
>>> stdout/stderr logs of all running containers and send them to a 
>>> syslog
>>> or provide them on a single http endpoint". This one is not usable 
>>> for
>>> us, because it involves exposing the host's docker daemon, implying
>>> that all containers are running on the same daemon. In our scenario,
>>> we would rather use a direct connection from the running instance to 
>>> a
>>> central log sink (like Logstash, see below).
>>>
>>> - Logstash, https://github.com/elastic/logstash, Apache License 
>>> (also
>>> Elasticsearch)
>>> Collects, normalizes and stores (usually in Elasticsearch) any
>>> incoming info. Usually used for collection and storing logs (but
>>> defines itself more generic). Has lots of input and output plugins 
>>> for
>>> accepting a wide range of event/log sources and writing them to lots
>>> of output sinks (Elasticsearch is only one of them). Looking at the
>>> possible protocols, Docker and Logstash support two matching 
>>> protocols
>>> that could be used to send logs "directly" to Logstash: syslog and
>>> gelf.
>>>
>>> - Kibana, https://github.com/elastic/kibana, Apache License
>>> Webfrontend for Elasticsearch that visualizes the data with pretty
>>> advanced stuff. Optional, but nice to provide diagrams. This one 
>>> might
>>> be a candidate to look at when we are looking at the KPI stuff.
>>>
>>> For a very simple intermediate solution, we could put together a
>>> single container that includes most of the above components and acts
>>> as a "single sink" for logs. Ugly, but works. I did that one before
>>> (with logsprout as aggregator).
>>>
>>> Decision needed: we are currently working with a low number of
>>> containers. Are we even need a central logging at this stage? Or can
>>> we wait until the system-provided logging is available?
>>
>>
>> Let's keep design a priority. :)  We want to be in the position where 
>> we
>> can confidently replicate our patterns to new modules/containers as 
>> we
>> introduce and integrate new features.  That said, we don't need to be
>> perfect on the first swipe.
>>
>> So anything we can do to make a single stable logging service 
>> extension
>> SPI now, even if it's a stupid hack whose implementation we soon swap 
>> out,
>> centralizes the issue for us and future work here wouldn't impact the
>> various systems calling upon it.
>>
>> S,
>> ALR
>>
>>
>>>
>>> Anyway, a possible custom solution should be compatible with a 
>>> coming
>>> system logging. I would think that syslog may be the
>>> "supported-everywhere" variant.
>>>
>>> For the above "ugly" solution, I would have some stuff already
>>> available that we could use to build on. I would suppose setting 
>>> this
>>> up should not take longer than 1-2 days (has to be converted from
>>> Ubuntu to Centos, may need some tweaking due to new releases of
>>> contained software etc).
>>>
>>> So, this for this mornings findings. :-)
>>>
>>> -- Michael
>>>
>>> --
>>> Michael Kleinhenz
>>> Principal Software Engineer
>>>
>>> Red Hat Deutschland GmbH
>>> Werner-von-Siemens-Ring 14
>>> 85630 Grasbrunn
>>> Germany
>>>
>>> RED HAT | TRIED. TESTED. TRUSTED.
>>> Red Hat GmbH, www.de.redhat.com,
>>> Registered seat: Grasbrunn, Commercial register: Amtsgericht 
>>> München,
>>> HRB 153243,
>>> Managing Directors: Paul Argiry, Charles Cachera, Michael 
>>> Cunningham,
>>> Michael O'Neill
>>>
>>> _______________________________________________
>>> almighty-public mailing list
>>> almighty-public at redhat.com
>>> https://www.redhat.com/mailman/listinfo/almighty-public
>>>
>>
>>
>>
>> --
>> Red Hat Developer Programs Architecture
>> @ALRubinger
>>
>> _______________________________________________
>> almighty-public mailing list
>> almighty-public at redhat.com
>> https://www.redhat.com/mailman/listinfo/almighty-public
>>
>>

> _______________________________________________
> almighty-public mailing list
> almighty-public at redhat.com
> https://www.redhat.com/mailman/listinfo/almighty-public

/max
http://about.me/maxandersen