[Tendrl-devel] installation of monitoring and alerting

[Anmol Babu]

Please find my comments inline...

Anmol Babu
Software Engineer 
Red Hat India Pvt Ltd
M: 8884245644

----- Original Message -----
From: "Martin Bukatovic" <mbukatov at redhat.com>
To: tendrl-devel at redhat.com
Sent: Thursday, June 1, 2017 4:13:47 PM
Subject: Re: [Tendrl-devel] installation of monitoring and alerting

Hi Anmol,

On 06/01/2017 08:58 AM, Anmol Babu wrote:
> At this stage, the alerting module is light-weight when compared to the performance-monitoring application(brings in graphite + lot of periodic etcd interactions to facilitate dashboards).
> Its only responsibility is to scan through /messages/events in etcd and if it finds any new message that carries a priority "notice" It initiates a chain of actions like
> 1. Putting the validated alert in a appropriate location in etcd...
> 2. Sending out the alert to its configured destinations(currently only via email to configured users)
> The way it works is described at https://github.com/Tendrl/documentation/wiki/How-to-send-alerts-from-inside-Tendrl---for-devs
> If you find time, please review it and provide your feedback on it..
> So to summarise, from the perspective of alerting module, I currently don't see any trade-offs by virtue of where it is deployed and even the code doesn't lay any limitations on where it can be deployed.

>Thanks a lot for this explanation.

>I have few additional questions:

>Based on your description, I guess that the safe default for
>tendrl-alerting is collocation on Tendrl Server (even though
>that you can deploy it anywhere), right?

Yes you are right..

>What kind of additional processing of raw messages with priority
>notice from /messages/events are done? I see you mention
>classification and meta data processing - so do I read it right
>that the alerting process only messages from etcd and nothing
>else?

Additional processing involves:

1. Classifying alerts as node-related or cluster related and further classifying them by the node ids/cluster ids as applicable within the initial classification..
2. Anything in /messages/events is just messages which includes our logs, alerts and job updates but,
   classifying and extracting out messages corresponding to alerts into a dedicated etcd directory is done by alerting module.

These apart from the final notifying to configured users..

1 and 2 are source of alerts for the api that are shown on UI both filtered by cluster/node and unfiltered list...

>Does it mean that if you don't plan to configure email
>notifications, you don't have to deploy tendrl-alerting
>at all?

The alerting module also handles auto-acking an alert when there is a negation alert for the already existing alert in etcd.
It also facilitates user acking the alert so that he doesn't want to see it anymore in list of alerts..
It is also responsible for adding alert counters 

>Would not installing tendrl-alerting have any
>side effect on the rest of tendrl?

Not installing tendrl-alerting theoretically would leave:
1. alerts page empty
2. alerts list on dashboards empty
3. Number of alerts in each of the dashboards for each of the cards as 0..

>Does tendrl-alerting
>write some data back into etcd?

Yes in accordance with 1 and 2 above, it does write messages corresponding to alerts to dedicated location in etcd..

-- 
Martin Bukatovic
USM QE team
Red Hat

_______________________________________________
Tendrl-devel mailing list
Tendrl-devel at redhat.com
https://www.redhat.com/mailman/listinfo/tendrl-devel