From ppatiern at redhat.com Mon Aug 5 08:03:21 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Mon, 5 Aug 2019 10:03:21 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: Hi Debdeep, can you provide the Prometheus log? In the prometheus.yaml file, the ClusterRoleBinding resource provides the binding between the promethes-server service account and the cluster role but specifing "myproject" as namespace. Is you Kafka cluster running in the myproject namespace? Is the same for the Prometheus server you installed through the Prometheus operator? Thanks, Paolo. On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep wrote: > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From debdeep.das at in.unisys.com Mon Aug 5 10:00:59 2019 From: debdeep.das at in.unisys.com (Das, Debdeep) Date: Mon, 5 Aug 2019 10:00:59 +0000 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: Hello Paolo, Let me explain the environment I have and maybe there is something not setup right. I am using Rancher on Kubernetes. On the environment Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to produce the metrics of the kafka cluster in strimzi-kafka-operator name space to the prometheus instance in cattle-prometheus namespace. I am not sure what are the steps to do so. If you can help me with the steps to do so, it will be great. I could not figure that from the documentation. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com] Sent: Monday, August 5, 2019 1:33 PM To: Das, Debdeep Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Hi Debdeep, can you provide the Prometheus log? In the prometheus.yaml file, the ClusterRoleBinding resource provides the binding between the promethes-server service account and the cluster role but specifing "myproject" as namespace. Is you Kafka cluster running in the myproject namespace? Is the same for the Prometheus server you installed through the Prometheus operator? Thanks, Paolo. On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: Hello, I have a Kubernetes environment with Strimzi and Kafka deployed. I am able to produce and consume successfully. I want to use Prometheus and Grafana to report metrics. The environment already has Prometheus Operator and Grafana installed, but in a different namespace than where I installed the Strimzi + Kafka. I used the instructions here https://strimzi.io/docs/latest/#prometheus to add the rules. I used the following instructions under the namespace where Strimzi-Kafka is installed, but I don?t see the Rules under Prometheus. The metrics are already added in the YAML to deploy the Kafka(+zookeper) kubectl apply -f strimzi-service-monitor.yaml kubectl apply -f prometheus-rules.yaml kubectl apply -f prometheus.yaml Can you please help what needs to be done to get the rules. I guess because the rules are not getting to Prometheus, the Grafana is not reporting anything on the Dashboard. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 7855 bytes Desc: not available URL: From ppatiern at redhat.com Mon Aug 5 10:09:11 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Mon, 5 Aug 2019 12:09:11 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: What's not clear to me now is ... if you are using the Prometheus Operator or not. In the first email you mentioned the Prometheus operator while in the last one just Prometheus (the server) which are two different things. The first one is a Kubernetes operator for deploying Prometheus server and services to identify metrics to scrape from other applications; the second one is ... the actual Prometheus server. The steps we provide in the documentation are about having Prometheus operator running and deploying a few resources for deploying a Prometheus server and the services for monitoring Kafka metrics (all these resources are handled by the prometheus operator which than "inject" the configuration into prometheus). On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep wrote: > Hello Paolo, > > Let me explain the environment I have and maybe there is something not > setup right. I am using Rancher on Kubernetes. On the environment > Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My > Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to > produce the metrics of the kafka cluster in strimzi-kafka-operator name > space to the prometheus instance in cattle-prometheus namespace. I am not > sure what are the steps to do so. If you can help me with the steps to do > so, it will be great. I could not figure that from the documentation. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 1:33 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Hi Debdeep, > > can you provide the Prometheus log? > > In the prometheus.yaml file, the ClusterRoleBinding resource provides the > binding between the promethes-server service account and the cluster role > but specifing "myproject" as namespace. Is you Kafka cluster running in the > myproject namespace? Is the same for the Prometheus server you installed > through the Prometheus operator? > > > > Thanks, > > Paolo. > > > > On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: > > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From debdeep.das at in.unisys.com Mon Aug 5 10:13:56 2019 From: debdeep.das at in.unisys.com (Das, Debdeep) Date: Mon, 5 Aug 2019 10:13:56 +0000 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: Hello Paolo, My apologies for the confusion. I am using Prometheus operator, which is already installed at the cluster under the name space cattle-prometheus. My assumption is I do not need to install the Prometheus operator again and I can push the metrics from the kafka cluster ( deployed in strimzi-kafka-operator namespace) to the prometheus operator running in cattle-prometheus namespace. Is my assumption correct? If yes, what is the steps that I need to follow. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com] Sent: Monday, August 5, 2019 3:39 PM To: Das, Debdeep Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus What's not clear to me now is ... if you are using the Prometheus Operator or not. In the first email you mentioned the Prometheus operator while in the last one just Prometheus (the server) which are two different things. The first one is a Kubernetes operator for deploying Prometheus server and services to identify metrics to scrape from other applications; the second one is ... the actual Prometheus server. The steps we provide in the documentation are about having Prometheus operator running and deploying a few resources for deploying a Prometheus server and the services for monitoring Kafka metrics (all these resources are handled by the prometheus operator which than "inject" the configuration into prometheus). On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: Hello Paolo, Let me explain the environment I have and maybe there is something not setup right. I am using Rancher on Kubernetes. On the environment Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to produce the metrics of the kafka cluster in strimzi-kafka-operator name space to the prometheus instance in cattle-prometheus namespace. I am not sure what are the steps to do so. If you can help me with the steps to do so, it will be great. I could not figure that from the documentation. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 1:33 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Hi Debdeep, can you provide the Prometheus log? In the prometheus.yaml file, the ClusterRoleBinding resource provides the binding between the promethes-server service account and the cluster role but specifing "myproject" as namespace. Is you Kafka cluster running in the myproject namespace? Is the same for the Prometheus server you installed through the Prometheus operator? Thanks, Paolo. On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: Hello, I have a Kubernetes environment with Strimzi and Kafka deployed. I am able to produce and consume successfully. I want to use Prometheus and Grafana to report metrics. The environment already has Prometheus Operator and Grafana installed, but in a different namespace than where I installed the Strimzi + Kafka. I used the instructions here https://strimzi.io/docs/latest/#prometheus to add the rules. I used the following instructions under the namespace where Strimzi-Kafka is installed, but I don?t see the Rules under Prometheus. The metrics are already added in the YAML to deploy the Kafka(+zookeper) kubectl apply -f strimzi-service-monitor.yaml kubectl apply -f prometheus-rules.yaml kubectl apply -f prometheus.yaml Can you please help what needs to be done to get the rules. I guess because the rules are not getting to Prometheus, the Grafana is not reporting anything on the Dashboard. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 7855 bytes Desc: not available URL: From ppatiern at redhat.com Mon Aug 5 10:16:52 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Mon, 5 Aug 2019 12:16:52 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: The prometheus operator is not the prometheus server as I mentioned in the previous email. Did you use the prometheus operator to deploy the prometheus server then? Where the prometheus server is running? not the operator ;) On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep wrote: > Hello Paolo, > > My apologies for the confusion. I am using Prometheus operator, which is > already installed at the cluster under the name space cattle-prometheus. My > assumption is I do not need to install the Prometheus operator again and I > can push the metrics from the kafka cluster ( deployed in > strimzi-kafka-operator namespace) to the prometheus operator running in > cattle-prometheus namespace. Is my assumption correct? If yes, what is the > steps that I need to follow. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:39 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > What's not clear to me now is ... if you are using the Prometheus Operator > or not. > > In the first email you mentioned the Prometheus operator while in the last > one just Prometheus (the server) which are two different things. > > The first one is a Kubernetes operator for deploying Prometheus server and > services to identify metrics to scrape from other applications; the second > one is ... the actual Prometheus server. > > The steps we provide in the documentation are about having Prometheus > operator running and deploying a few resources for deploying a Prometheus > server and the services for monitoring Kafka metrics (all these resources > are handled by the prometheus operator which than "inject" the > configuration into prometheus). > > > > On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: > > Hello Paolo, > > Let me explain the environment I have and maybe there is something not > setup right. I am using Rancher on Kubernetes. On the environment > Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My > Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to > produce the metrics of the kafka cluster in strimzi-kafka-operator name > space to the prometheus instance in cattle-prometheus namespace. I am not > sure what are the steps to do so. If you can help me with the steps to do > so, it will be great. I could not figure that from the documentation. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 1:33 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Hi Debdeep, > > can you provide the Prometheus log? > > In the prometheus.yaml file, the ClusterRoleBinding resource provides the > binding between the promethes-server service account and the cluster role > but specifing "myproject" as namespace. Is you Kafka cluster running in the > myproject namespace? Is the same for the Prometheus server you installed > through the Prometheus operator? > > > > Thanks, > > Paolo. > > > > On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: > > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From debdeep.das at in.unisys.com Mon Aug 5 10:30:35 2019 From: debdeep.das at in.unisys.com (Das, Debdeep) Date: Mon, 5 Aug 2019 10:30:35 +0000 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: The prometheus server is also running in the cattle-Prometheus namespace. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com] Sent: Monday, August 5, 2019 3:47 PM To: Das, Debdeep Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus The prometheus operator is not the prometheus server as I mentioned in the previous email. Did you use the prometheus operator to deploy the prometheus server then? Where the prometheus server is running? not the operator ;) On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: Hello Paolo, My apologies for the confusion. I am using Prometheus operator, which is already installed at the cluster under the name space cattle-prometheus. My assumption is I do not need to install the Prometheus operator again and I can push the metrics from the kafka cluster ( deployed in strimzi-kafka-operator namespace) to the prometheus operator running in cattle-prometheus namespace. Is my assumption correct? If yes, what is the steps that I need to follow. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:39 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus What's not clear to me now is ... if you are using the Prometheus Operator or not. In the first email you mentioned the Prometheus operator while in the last one just Prometheus (the server) which are two different things. The first one is a Kubernetes operator for deploying Prometheus server and services to identify metrics to scrape from other applications; the second one is ... the actual Prometheus server. The steps we provide in the documentation are about having Prometheus operator running and deploying a few resources for deploying a Prometheus server and the services for monitoring Kafka metrics (all these resources are handled by the prometheus operator which than "inject" the configuration into prometheus). On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: Hello Paolo, Let me explain the environment I have and maybe there is something not setup right. I am using Rancher on Kubernetes. On the environment Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to produce the metrics of the kafka cluster in strimzi-kafka-operator name space to the prometheus instance in cattle-prometheus namespace. I am not sure what are the steps to do so. If you can help me with the steps to do so, it will be great. I could not figure that from the documentation. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 1:33 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Hi Debdeep, can you provide the Prometheus log? In the prometheus.yaml file, the ClusterRoleBinding resource provides the binding between the promethes-server service account and the cluster role but specifing "myproject" as namespace. Is you Kafka cluster running in the myproject namespace? Is the same for the Prometheus server you installed through the Prometheus operator? Thanks, Paolo. On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: Hello, I have a Kubernetes environment with Strimzi and Kafka deployed. I am able to produce and consume successfully. I want to use Prometheus and Grafana to report metrics. The environment already has Prometheus Operator and Grafana installed, but in a different namespace than where I installed the Strimzi + Kafka. I used the instructions here https://strimzi.io/docs/latest/#prometheus to add the rules. I used the following instructions under the namespace where Strimzi-Kafka is installed, but I don?t see the Rules under Prometheus. The metrics are already added in the YAML to deploy the Kafka(+zookeper) kubectl apply -f strimzi-service-monitor.yaml kubectl apply -f prometheus-rules.yaml kubectl apply -f prometheus.yaml Can you please help what needs to be done to get the rules. I guess because the rules are not getting to Prometheus, the Grafana is not reporting anything on the Dashboard. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 7855 bytes Desc: not available URL: From ppatiern at redhat.com Mon Aug 5 15:26:37 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Mon, 5 Aug 2019 17:26:37 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: Can you provide me the prometheus-operator log? >From the above steps you are missing the creation of the Secret containing the additional configs and maybe the Prometheus operator is failing to search for it. This Secret is referenced in the prometheus.yaml` file as following: additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional.yaml The Secret creation is described here: https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str (it's from master doc, what's the Strimzi version you are using?) After that, access to the prometheus web UI and let me know if you can see metrics starting with kafka_ and zookeeper_ On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep wrote: > The prometheus server is also running in the cattle-Prometheus namespace. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:47 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > The prometheus operator is not the prometheus server as I mentioned in the > previous email. > > Did you use the prometheus operator to deploy the prometheus server then? > Where the prometheus server is running? not the operator ;) > > > > On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: > > Hello Paolo, > > My apologies for the confusion. I am using Prometheus operator, which is > already installed at the cluster under the name space cattle-prometheus. My > assumption is I do not need to install the Prometheus operator again and I > can push the metrics from the kafka cluster ( deployed in > strimzi-kafka-operator namespace) to the prometheus operator running in > cattle-prometheus namespace. Is my assumption correct? If yes, what is the > steps that I need to follow. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:39 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > What's not clear to me now is ... if you are using the Prometheus Operator > or not. > > In the first email you mentioned the Prometheus operator while in the last > one just Prometheus (the server) which are two different things. > > The first one is a Kubernetes operator for deploying Prometheus server and > services to identify metrics to scrape from other applications; the second > one is ... the actual Prometheus server. > > The steps we provide in the documentation are about having Prometheus > operator running and deploying a few resources for deploying a Prometheus > server and the services for monitoring Kafka metrics (all these resources > are handled by the prometheus operator which than "inject" the > configuration into prometheus). > > > > On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: > > Hello Paolo, > > Let me explain the environment I have and maybe there is something not > setup right. I am using Rancher on Kubernetes. On the environment > Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My > Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to > produce the metrics of the kafka cluster in strimzi-kafka-operator name > space to the prometheus instance in cattle-prometheus namespace. I am not > sure what are the steps to do so. If you can help me with the steps to do > so, it will be great. I could not figure that from the documentation. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 1:33 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Hi Debdeep, > > can you provide the Prometheus log? > > In the prometheus.yaml file, the ClusterRoleBinding resource provides the > binding between the promethes-server service account and the cluster role > but specifing "myproject" as namespace. Is you Kafka cluster running in the > myproject namespace? Is the same for the Prometheus server you installed > through the Prometheus operator? > > > > Thanks, > > Paolo. > > > > On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: > > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From debdeep.das at in.unisys.com Tue Aug 6 13:43:23 2019 From: debdeep.das at in.unisys.com (Das, Debdeep) Date: Tue, 6 Aug 2019 13:43:23 +0000 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: Please find attached the Prometheus operator log. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com] Sent: Monday, August 5, 2019 8:57 PM To: Das, Debdeep Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Can you provide me the prometheus-operator log? >From the above steps you are missing the creation of the Secret containing the additional configs and maybe the Prometheus operator is failing to search for it. This Secret is referenced in the prometheus.yaml` file as following: additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional.yaml The Secret creation is described here: https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str (it's from master doc, what's the Strimzi version you are using?) After that, access to the prometheus web UI and let me know if you can see metrics starting with kafka_ and zookeeper_ On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: The prometheus server is also running in the cattle-Prometheus namespace. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:47 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus The prometheus operator is not the prometheus server as I mentioned in the previous email. Did you use the prometheus operator to deploy the prometheus server then? Where the prometheus server is running? not the operator ;) On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: Hello Paolo, My apologies for the confusion. I am using Prometheus operator, which is already installed at the cluster under the name space cattle-prometheus. My assumption is I do not need to install the Prometheus operator again and I can push the metrics from the kafka cluster ( deployed in strimzi-kafka-operator namespace) to the prometheus operator running in cattle-prometheus namespace. Is my assumption correct? If yes, what is the steps that I need to follow. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:39 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus What's not clear to me now is ... if you are using the Prometheus Operator or not. In the first email you mentioned the Prometheus operator while in the last one just Prometheus (the server) which are two different things. The first one is a Kubernetes operator for deploying Prometheus server and services to identify metrics to scrape from other applications; the second one is ... the actual Prometheus server. The steps we provide in the documentation are about having Prometheus operator running and deploying a few resources for deploying a Prometheus server and the services for monitoring Kafka metrics (all these resources are handled by the prometheus operator which than "inject" the configuration into prometheus). On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: Hello Paolo, Let me explain the environment I have and maybe there is something not setup right. I am using Rancher on Kubernetes. On the environment Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to produce the metrics of the kafka cluster in strimzi-kafka-operator name space to the prometheus instance in cattle-prometheus namespace. I am not sure what are the steps to do so. If you can help me with the steps to do so, it will be great. I could not figure that from the documentation. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 1:33 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Hi Debdeep, can you provide the Prometheus log? In the prometheus.yaml file, the ClusterRoleBinding resource provides the binding between the promethes-server service account and the cluster role but specifing "myproject" as namespace. Is you Kafka cluster running in the myproject namespace? Is the same for the Prometheus server you installed through the Prometheus operator? Thanks, Paolo. On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: Hello, I have a Kubernetes environment with Strimzi and Kafka deployed. I am able to produce and consume successfully. I want to use Prometheus and Grafana to report metrics. The environment already has Prometheus Operator and Grafana installed, but in a different namespace than where I installed the Strimzi + Kafka. I used the instructions here https://strimzi.io/docs/latest/#prometheus to add the rules. I used the following instructions under the namespace where Strimzi-Kafka is installed, but I don?t see the Rules under Prometheus. The metrics are already added in the YAML to deploy the Kafka(+zookeper) kubectl apply -f strimzi-service-monitor.yaml kubectl apply -f prometheus-rules.yaml kubectl apply -f prometheus.yaml Can you please help what needs to be done to get the rules. I guess because the rules are not getting to Prometheus, the Grafana is not reporting anything on the Dashboard. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: prometheus-operator log.log Type: application/octet-stream Size: 82402 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 7855 bytes Desc: not available URL: From ppatiern at redhat.com Tue Aug 6 14:12:21 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Tue, 6 Aug 2019 16:12:21 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: As you can see from the log as I thought you are missing the additional scrape configs creation. On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep wrote: > > > Please find attached the Prometheus operator log. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 8:57 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Can you provide me the prometheus-operator log? > > From the above steps you are missing the creation of the Secret containing > the additional configs and maybe the Prometheus operator is failing to > search for it. > > This Secret is referenced in the prometheus.yaml` file as following: > > > > additionalScrapeConfigs: > > name: additional-scrape-configs > > key: prometheus-additional.yaml > > > > The Secret creation is described here: > https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str > (it's from master doc, what's the Strimzi version you are using?) > > > > After that, access to the prometheus web UI and let me know if you can see > metrics starting with kafka_ and zookeeper_ > > > > > > On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: > > The prometheus server is also running in the cattle-Prometheus namespace. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:47 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > The prometheus operator is not the prometheus server as I mentioned in the > previous email. > > Did you use the prometheus operator to deploy the prometheus server then? > Where the prometheus server is running? not the operator ;) > > > > On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: > > Hello Paolo, > > My apologies for the confusion. I am using Prometheus operator, which is > already installed at the cluster under the name space cattle-prometheus. My > assumption is I do not need to install the Prometheus operator again and I > can push the metrics from the kafka cluster ( deployed in > strimzi-kafka-operator namespace) to the prometheus operator running in > cattle-prometheus namespace. Is my assumption correct? If yes, what is the > steps that I need to follow. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:39 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > What's not clear to me now is ... if you are using the Prometheus Operator > or not. > > In the first email you mentioned the Prometheus operator while in the last > one just Prometheus (the server) which are two different things. > > The first one is a Kubernetes operator for deploying Prometheus server and > services to identify metrics to scrape from other applications; the second > one is ... the actual Prometheus server. > > The steps we provide in the documentation are about having Prometheus > operator running and deploying a few resources for deploying a Prometheus > server and the services for monitoring Kafka metrics (all these resources > are handled by the prometheus operator which than "inject" the > configuration into prometheus). > > > > On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: > > Hello Paolo, > > Let me explain the environment I have and maybe there is something not > setup right. I am using Rancher on Kubernetes. On the environment > Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My > Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to > produce the metrics of the kafka cluster in strimzi-kafka-operator name > space to the prometheus instance in cattle-prometheus namespace. I am not > sure what are the steps to do so. If you can help me with the steps to do > so, it will be great. I could not figure that from the documentation. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 1:33 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Hi Debdeep, > > can you provide the Prometheus log? > > In the prometheus.yaml file, the ClusterRoleBinding resource provides the > binding between the promethes-server service account and the cluster role > but specifing "myproject" as namespace. Is you Kafka cluster running in the > myproject namespace? Is the same for the Prometheus server you installed > through the Prometheus operator? > > > > Thanks, > > Paolo. > > > > On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: > > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From debdeep.das at in.unisys.com Tue Aug 6 14:17:11 2019 From: debdeep.das at in.unisys.com (Das, Debdeep) Date: Tue, 6 Aug 2019 14:17:11 +0000 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: I am using Strimzi 0.12.2 I created the secret kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml The next step is to get the secret name. How to do that? Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com] Sent: Tuesday, August 6, 2019 7:42 PM To: Das, Debdeep Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus As you can see from the log as I thought you are missing the additional scrape configs creation. On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep > wrote: Please find attached the Prometheus operator log. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 8:57 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Can you provide me the prometheus-operator log? >From the above steps you are missing the creation of the Secret containing the additional configs and maybe the Prometheus operator is failing to search for it. This Secret is referenced in the prometheus.yaml` file as following: additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional.yaml The Secret creation is described here: https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str (it's from master doc, what's the Strimzi version you are using?) After that, access to the prometheus web UI and let me know if you can see metrics starting with kafka_ and zookeeper_ On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: The prometheus server is also running in the cattle-Prometheus namespace. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:47 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus The prometheus operator is not the prometheus server as I mentioned in the previous email. Did you use the prometheus operator to deploy the prometheus server then? Where the prometheus server is running? not the operator ;) On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: Hello Paolo, My apologies for the confusion. I am using Prometheus operator, which is already installed at the cluster under the name space cattle-prometheus. My assumption is I do not need to install the Prometheus operator again and I can push the metrics from the kafka cluster ( deployed in strimzi-kafka-operator namespace) to the prometheus operator running in cattle-prometheus namespace. Is my assumption correct? If yes, what is the steps that I need to follow. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:39 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus What's not clear to me now is ... if you are using the Prometheus Operator or not. In the first email you mentioned the Prometheus operator while in the last one just Prometheus (the server) which are two different things. The first one is a Kubernetes operator for deploying Prometheus server and services to identify metrics to scrape from other applications; the second one is ... the actual Prometheus server. The steps we provide in the documentation are about having Prometheus operator running and deploying a few resources for deploying a Prometheus server and the services for monitoring Kafka metrics (all these resources are handled by the prometheus operator which than "inject" the configuration into prometheus). On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: Hello Paolo, Let me explain the environment I have and maybe there is something not setup right. I am using Rancher on Kubernetes. On the environment Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to produce the metrics of the kafka cluster in strimzi-kafka-operator name space to the prometheus instance in cattle-prometheus namespace. I am not sure what are the steps to do so. If you can help me with the steps to do so, it will be great. I could not figure that from the documentation. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 1:33 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Hi Debdeep, can you provide the Prometheus log? In the prometheus.yaml file, the ClusterRoleBinding resource provides the binding between the promethes-server service account and the cluster role but specifing "myproject" as namespace. Is you Kafka cluster running in the myproject namespace? Is the same for the Prometheus server you installed through the Prometheus operator? Thanks, Paolo. On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: Hello, I have a Kubernetes environment with Strimzi and Kafka deployed. I am able to produce and consume successfully. I want to use Prometheus and Grafana to report metrics. The environment already has Prometheus Operator and Grafana installed, but in a different namespace than where I installed the Strimzi + Kafka. I used the instructions here https://strimzi.io/docs/latest/#prometheus to add the rules. I used the following instructions under the namespace where Strimzi-Kafka is installed, but I don?t see the Rules under Prometheus. The metrics are already added in the YAML to deploy the Kafka(+zookeper) kubectl apply -f strimzi-service-monitor.yaml kubectl apply -f prometheus-rules.yaml kubectl apply -f prometheus.yaml Can you please help what needs to be done to get the rules. I guess because the rules are not getting to Prometheus, the Grafana is not reporting anything on the Dashboard. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 7855 bytes Desc: not available URL: From sknot at redhat.com Tue Aug 6 14:17:12 2019 From: sknot at redhat.com (Stanislav Knot) Date: Tue, 6 Aug 2019 16:17:12 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: I can see in the log you provided `secret additional-scrape-configs could not be found`. Are you sure you have created the secrets correctly? Stanislav On Tue, Aug 6, 2019 at 4:06 PM Das, Debdeep wrote: > > > Please find attached the Prometheus operator log. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 8:57 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Can you provide me the prometheus-operator log? > > From the above steps you are missing the creation of the Secret containing > the additional configs and maybe the Prometheus operator is failing to > search for it. > > This Secret is referenced in the prometheus.yaml` file as following: > > > > additionalScrapeConfigs: > > name: additional-scrape-configs > > key: prometheus-additional.yaml > > > > The Secret creation is described here: > https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str > (it's from master doc, what's the Strimzi version you are using?) > > > > After that, access to the prometheus web UI and let me know if you can see > metrics starting with kafka_ and zookeeper_ > > > > > > On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: > > The prometheus server is also running in the cattle-Prometheus namespace. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:47 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > The prometheus operator is not the prometheus server as I mentioned in the > previous email. > > Did you use the prometheus operator to deploy the prometheus server then? > Where the prometheus server is running? not the operator ;) > > > > On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: > > Hello Paolo, > > My apologies for the confusion. I am using Prometheus operator, which is > already installed at the cluster under the name space cattle-prometheus. My > assumption is I do not need to install the Prometheus operator again and I > can push the metrics from the kafka cluster ( deployed in > strimzi-kafka-operator namespace) to the prometheus operator running in > cattle-prometheus namespace. Is my assumption correct? If yes, what is the > steps that I need to follow. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:39 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > What's not clear to me now is ... if you are using the Prometheus Operator > or not. > > In the first email you mentioned the Prometheus operator while in the last > one just Prometheus (the server) which are two different things. > > The first one is a Kubernetes operator for deploying Prometheus server and > services to identify metrics to scrape from other applications; the second > one is ... the actual Prometheus server. > > The steps we provide in the documentation are about having Prometheus > operator running and deploying a few resources for deploying a Prometheus > server and the services for monitoring Kafka metrics (all these resources > are handled by the prometheus operator which than "inject" the > configuration into prometheus). > > > > On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: > > Hello Paolo, > > Let me explain the environment I have and maybe there is something not > setup right. I am using Rancher on Kubernetes. On the environment > Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My > Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to > produce the metrics of the kafka cluster in strimzi-kafka-operator name > space to the prometheus instance in cattle-prometheus namespace. I am not > sure what are the steps to do so. If you can help me with the steps to do > so, it will be great. I could not figure that from the documentation. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 1:33 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Hi Debdeep, > > can you provide the Prometheus log? > > In the prometheus.yaml file, the ClusterRoleBinding resource provides the > binding between the promethes-server service account and the cluster role > but specifing "myproject" as namespace. Is you Kafka cluster running in the > myproject namespace? Is the same for the Prometheus server you installed > through the Prometheus operator? > > > > Thanks, > > Paolo. > > > > On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: > > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From ppatiern at redhat.com Tue Aug 6 14:21:08 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Tue, 6 Aug 2019 16:21:08 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: It has to be created in the namespace where your prometheus operator is running. The operator needs that secret for creating the prometheus instance. On Tue, Aug 6, 2019 at 4:17 PM Stanislav Knot wrote: > I can see in the log you provided `secret additional-scrape-configs could > not be found`. > Are you sure you have created the secrets correctly? > > Stanislav > > On Tue, Aug 6, 2019 at 4:06 PM Das, Debdeep > wrote: > >> >> >> Please find attached the Prometheus operator log. >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> *From:* Paolo Patierno [mailto:ppatiern at redhat.com] >> *Sent:* Monday, August 5, 2019 8:57 PM >> *To:* Das, Debdeep >> *Cc:* strimzi at redhat.com >> *Subject:* Re: [Strimzi] Strimzi with Prometheus >> >> >> >> Can you provide me the prometheus-operator log? >> >> From the above steps you are missing the creation of the Secret >> containing the additional configs and maybe the Prometheus operator is >> failing to search for it. >> >> This Secret is referenced in the prometheus.yaml` file as following: >> >> >> >> additionalScrapeConfigs: >> >> name: additional-scrape-configs >> >> key: prometheus-additional.yaml >> >> >> >> The Secret creation is described here: >> https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str >> (it's from master doc, what's the Strimzi version you are using?) >> >> >> >> After that, access to the prometheus web UI and let me know if you can >> see metrics starting with kafka_ and zookeeper_ >> >> >> >> >> >> On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep >> wrote: >> >> The prometheus server is also running in the cattle-Prometheus namespace. >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> *From:* Paolo Patierno [mailto:ppatiern at redhat.com] >> *Sent:* Monday, August 5, 2019 3:47 PM >> *To:* Das, Debdeep >> *Cc:* strimzi at redhat.com >> *Subject:* Re: [Strimzi] Strimzi with Prometheus >> >> >> >> The prometheus operator is not the prometheus server as I mentioned in >> the previous email. >> >> Did you use the prometheus operator to deploy the prometheus server then? >> Where the prometheus server is running? not the operator ;) >> >> >> >> On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep >> wrote: >> >> Hello Paolo, >> >> My apologies for the confusion. I am using Prometheus operator, which is >> already installed at the cluster under the name space cattle-prometheus. My >> assumption is I do not need to install the Prometheus operator again and I >> can push the metrics from the kafka cluster ( deployed in >> strimzi-kafka-operator namespace) to the prometheus operator running in >> cattle-prometheus namespace. Is my assumption correct? If yes, what is the >> steps that I need to follow. >> >> >> >> Regards >> >> Debdeep >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> *From:* Paolo Patierno [mailto:ppatiern at redhat.com] >> *Sent:* Monday, August 5, 2019 3:39 PM >> *To:* Das, Debdeep >> *Cc:* strimzi at redhat.com >> *Subject:* Re: [Strimzi] Strimzi with Prometheus >> >> >> >> What's not clear to me now is ... if you are using the Prometheus >> Operator or not. >> >> In the first email you mentioned the Prometheus operator while in the >> last one just Prometheus (the server) which are two different things. >> >> The first one is a Kubernetes operator for deploying Prometheus server >> and services to identify metrics to scrape from other applications; the >> second one is ... the actual Prometheus server. >> >> The steps we provide in the documentation are about having Prometheus >> operator running and deploying a few resources for deploying a Prometheus >> server and the services for monitoring Kafka metrics (all these resources >> are handled by the prometheus operator which than "inject" the >> configuration into prometheus). >> >> >> >> On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep >> wrote: >> >> Hello Paolo, >> >> Let me explain the environment I have and maybe there is something not >> setup right. I am using Rancher on Kubernetes. On the environment >> Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My >> Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to >> produce the metrics of the kafka cluster in strimzi-kafka-operator name >> space to the prometheus instance in cattle-prometheus namespace. I am not >> sure what are the steps to do so. If you can help me with the steps to do >> so, it will be great. I could not figure that from the documentation. >> >> >> >> Regards >> >> Debdeep >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> *From:* Paolo Patierno [mailto:ppatiern at redhat.com] >> *Sent:* Monday, August 5, 2019 1:33 PM >> *To:* Das, Debdeep >> *Cc:* strimzi at redhat.com >> *Subject:* Re: [Strimzi] Strimzi with Prometheus >> >> >> >> Hi Debdeep, >> >> can you provide the Prometheus log? >> >> In the prometheus.yaml file, the ClusterRoleBinding resource provides the >> binding between the promethes-server service account and the cluster role >> but specifing "myproject" as namespace. Is you Kafka cluster running in the >> myproject namespace? Is the same for the Prometheus server you installed >> through the Prometheus operator? >> >> >> >> Thanks, >> >> Paolo. >> >> >> >> On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep >> wrote: >> >> Hello, >> >> I have a Kubernetes environment with Strimzi and Kafka deployed. I am >> able to produce and consume successfully. I want to use Prometheus and >> Grafana to report metrics. The environment already has Prometheus Operator >> and Grafana installed, but in a different namespace than where I installed >> the Strimzi + Kafka. >> >> I used the instructions here https://strimzi.io/docs/latest/#prometheus >> to add the rules. I used the following instructions under the namespace >> where Strimzi-Kafka is installed, but I don?t see the Rules under >> Prometheus. The metrics are already added in the YAML to deploy the >> Kafka(+zookeper) >> >> >> >> kubectl apply -f strimzi-service-monitor.yaml >> >> kubectl apply -f prometheus-rules.yaml >> >> kubectl apply -f prometheus.yaml >> >> Can you please help what needs to be done to get the rules. I guess >> because the rules are not getting to Prometheus, the Grafana is not >> reporting anything on the Dashboard. >> >> >> >> Regards >> >> Debdeep >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> _______________________________________________ >> Strimzi mailing list >> Strimzi at redhat.com >> https://www.redhat.com/mailman/listinfo/strimzi >> >> >> >> >> -- >> >> *Paolo Patierno* >> >> Principal Software Engineer, Messaging & IoT >> >> Red Hat >> >> >> >> >> >> >> >> >> >> >> -- >> >> *Paolo Patierno* >> >> Principal Software Engineer, Messaging & IoT >> >> Red Hat >> >> >> >> >> >> >> >> >> >> >> -- >> >> *Paolo Patierno* >> >> Principal Software Engineer, Messaging & IoT >> >> Red Hat >> >> >> >> >> >> >> >> >> >> >> -- >> >> *Paolo Patierno* >> >> Principal Software Engineer, Messaging & IoT >> >> Red Hat >> >> >> >> >> >> >> _______________________________________________ >> Strimzi mailing list >> Strimzi at redhat.com >> https://www.redhat.com/mailman/listinfo/strimzi >> > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From sknot at redhat.com Tue Aug 6 14:52:53 2019 From: sknot at redhat.com (Stanislav Knot) Date: Tue, 6 Aug 2019 16:52:53 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: You can use kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n your-namespace Stanislav On Tue, Aug 6, 2019 at 4:20 PM Das, Debdeep wrote: > I am using Strimzi 0.12.2 > > > > I created the secret > > > > kubectl create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml > > > > The next step is to get the secret name. How to do that? > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 7:42 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > As you can see from the log as I thought you are missing the additional > scrape configs creation. > > > > On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep > wrote: > > > > Please find attached the Prometheus operator log. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 8:57 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Can you provide me the prometheus-operator log? > > From the above steps you are missing the creation of the Secret containing > the additional configs and maybe the Prometheus operator is failing to > search for it. > > This Secret is referenced in the prometheus.yaml` file as following: > > > > additionalScrapeConfigs: > > name: additional-scrape-configs > > key: prometheus-additional.yaml > > > > The Secret creation is described here: > https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str > (it's from master doc, what's the Strimzi version you are using?) > > > > After that, access to the prometheus web UI and let me know if you can see > metrics starting with kafka_ and zookeeper_ > > > > > > On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: > > The prometheus server is also running in the cattle-Prometheus namespace. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:47 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > The prometheus operator is not the prometheus server as I mentioned in the > previous email. > > Did you use the prometheus operator to deploy the prometheus server then? > Where the prometheus server is running? not the operator ;) > > > > On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: > > Hello Paolo, > > My apologies for the confusion. I am using Prometheus operator, which is > already installed at the cluster under the name space cattle-prometheus. My > assumption is I do not need to install the Prometheus operator again and I > can push the metrics from the kafka cluster ( deployed in > strimzi-kafka-operator namespace) to the prometheus operator running in > cattle-prometheus namespace. Is my assumption correct? If yes, what is the > steps that I need to follow. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:39 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > What's not clear to me now is ... if you are using the Prometheus Operator > or not. > > In the first email you mentioned the Prometheus operator while in the last > one just Prometheus (the server) which are two different things. > > The first one is a Kubernetes operator for deploying Prometheus server and > services to identify metrics to scrape from other applications; the second > one is ... the actual Prometheus server. > > The steps we provide in the documentation are about having Prometheus > operator running and deploying a few resources for deploying a Prometheus > server and the services for monitoring Kafka metrics (all these resources > are handled by the prometheus operator which than "inject" the > configuration into prometheus). > > > > On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: > > Hello Paolo, > > Let me explain the environment I have and maybe there is something not > setup right. I am using Rancher on Kubernetes. On the environment > Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My > Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to > produce the metrics of the kafka cluster in strimzi-kafka-operator name > space to the prometheus instance in cattle-prometheus namespace. I am not > sure what are the steps to do so. If you can help me with the steps to do > so, it will be great. I could not figure that from the documentation. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 1:33 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Hi Debdeep, > > can you provide the Prometheus log? > > In the prometheus.yaml file, the ClusterRoleBinding resource provides the > binding between the promethes-server service account and the cluster role > but specifing "myproject" as namespace. Is you Kafka cluster running in the > myproject namespace? Is the same for the Prometheus server you installed > through the Prometheus operator? > > > > Thanks, > > Paolo. > > > > On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: > > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From debdeep.das at in.unisys.com Tue Aug 6 15:28:12 2019 From: debdeep.das at in.unisys.com (Das, Debdeep) Date: Tue, 6 Aug 2019 15:28:12 +0000 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n cattle-prometheus secret/additional-scrape-configs created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl get secret -n cattle-prometheus NAME TYPE DATA AGE additional-scrape-configs Opaque 1 21m D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f strimzi-service-monitor.yaml -n cattle-prometheus servicemonitor.monitoring.coreos.com/kafka-service-monitor created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f prometheus-rules.yaml -n cattle-prometheus prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f prometheus.yaml -n cattle-prometheus clusterrole.rbac.authorization.k8s.io/prometheus-server created serviceaccount/prometheus-server created clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created prometheus.monitoring.coreos.com/prometheus created I still don?t see any metrics for Kafka. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Stanislav Knot [mailto:sknot at redhat.com] Sent: Tuesday, August 6, 2019 8:23 PM To: Das, Debdeep Cc: Paolo Patierno ; strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus You can use kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n your-namespace Stanislav On Tue, Aug 6, 2019 at 4:20 PM Das, Debdeep > wrote: I am using Strimzi 0.12.2 I created the secret kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml The next step is to get the secret name. How to do that? Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Tuesday, August 6, 2019 7:42 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus As you can see from the log as I thought you are missing the additional scrape configs creation. On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep > wrote: Please find attached the Prometheus operator log. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 8:57 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Can you provide me the prometheus-operator log? >From the above steps you are missing the creation of the Secret containing the additional configs and maybe the Prometheus operator is failing to search for it. This Secret is referenced in the prometheus.yaml` file as following: additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional.yaml The Secret creation is described here: https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str (it's from master doc, what's the Strimzi version you are using?) After that, access to the prometheus web UI and let me know if you can see metrics starting with kafka_ and zookeeper_ On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: The prometheus server is also running in the cattle-Prometheus namespace. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:47 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus The prometheus operator is not the prometheus server as I mentioned in the previous email. Did you use the prometheus operator to deploy the prometheus server then? Where the prometheus server is running? not the operator ;) On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: Hello Paolo, My apologies for the confusion. I am using Prometheus operator, which is already installed at the cluster under the name space cattle-prometheus. My assumption is I do not need to install the Prometheus operator again and I can push the metrics from the kafka cluster ( deployed in strimzi-kafka-operator namespace) to the prometheus operator running in cattle-prometheus namespace. Is my assumption correct? If yes, what is the steps that I need to follow. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:39 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus What's not clear to me now is ... if you are using the Prometheus Operator or not. In the first email you mentioned the Prometheus operator while in the last one just Prometheus (the server) which are two different things. The first one is a Kubernetes operator for deploying Prometheus server and services to identify metrics to scrape from other applications; the second one is ... the actual Prometheus server. The steps we provide in the documentation are about having Prometheus operator running and deploying a few resources for deploying a Prometheus server and the services for monitoring Kafka metrics (all these resources are handled by the prometheus operator which than "inject" the configuration into prometheus). On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: Hello Paolo, Let me explain the environment I have and maybe there is something not setup right. I am using Rancher on Kubernetes. On the environment Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to produce the metrics of the kafka cluster in strimzi-kafka-operator name space to the prometheus instance in cattle-prometheus namespace. I am not sure what are the steps to do so. If you can help me with the steps to do so, it will be great. I could not figure that from the documentation. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 1:33 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Hi Debdeep, can you provide the Prometheus log? In the prometheus.yaml file, the ClusterRoleBinding resource provides the binding between the promethes-server service account and the cluster role but specifing "myproject" as namespace. Is you Kafka cluster running in the myproject namespace? Is the same for the Prometheus server you installed through the Prometheus operator? Thanks, Paolo. On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: Hello, I have a Kubernetes environment with Strimzi and Kafka deployed. I am able to produce and consume successfully. I want to use Prometheus and Grafana to report metrics. The environment already has Prometheus Operator and Grafana installed, but in a different namespace than where I installed the Strimzi + Kafka. I used the instructions here https://strimzi.io/docs/latest/#prometheus to add the rules. I used the following instructions under the namespace where Strimzi-Kafka is installed, but I don?t see the Rules under Prometheus. The metrics are already added in the YAML to deploy the Kafka(+zookeper) kubectl apply -f strimzi-service-monitor.yaml kubectl apply -f prometheus-rules.yaml kubectl apply -f prometheus.yaml Can you please help what needs to be done to get the rules. I guess because the rules are not getting to Prometheus, the Grafana is not reporting anything on the Dashboard. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: prometheus-operator log.log Type: application/octet-stream Size: 53784 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 7855 bytes Desc: not available URL: From ppatiern at redhat.com Tue Aug 6 15:36:18 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Tue, 6 Aug 2019 17:36:18 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: Are you able to access the prometheus server web UI? Can you provide the prometheus server log? On Tue, Aug 6, 2019 at 5:28 PM Das, Debdeep wrote: > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl > create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml -n cattle-prometheus > > secret/additional-scrape-configs created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl > get secret -n cattle-prometheus > > NAME > TYPE > DATA AGE > > additional-scrape-configs > Opaque 1 21m > > > > > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f strimzi-service-monitor.yaml -n cattle-prometheus > > servicemonitor.monitoring.coreos.com/kafka-service-monitor created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f prometheus-rules.yaml -n cattle-prometheus > > prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f prometheus.yaml -n cattle-prometheus > > clusterrole.rbac.authorization.k8s.io/prometheus-server created > > serviceaccount/prometheus-server created > > clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created > > prometheus.monitoring.coreos.com/prometheus created > > > > > > I still don?t see any metrics for Kafka. > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Stanislav Knot [mailto:sknot at redhat.com] > *Sent:* Tuesday, August 6, 2019 8:23 PM > *To:* Das, Debdeep > *Cc:* Paolo Patierno ; strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > You can use > > kubectl create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml -n your-namespace > > > > Stanislav > > > > On Tue, Aug 6, 2019 at 4:20 PM Das, Debdeep > wrote: > > I am using Strimzi 0.12.2 > > > > I created the secret > > > > kubectl create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml > > > > The next step is to get the secret name. How to do that? > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 7:42 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > As you can see from the log as I thought you are missing the additional > scrape configs creation. > > > > On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep > wrote: > > > > Please find attached the Prometheus operator log. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 8:57 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Can you provide me the prometheus-operator log? > > From the above steps you are missing the creation of the Secret containing > the additional configs and maybe the Prometheus operator is failing to > search for it. > > This Secret is referenced in the prometheus.yaml` file as following: > > > > additionalScrapeConfigs: > > name: additional-scrape-configs > > key: prometheus-additional.yaml > > > > The Secret creation is described here: > https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str > (it's from master doc, what's the Strimzi version you are using?) > > > > After that, access to the prometheus web UI and let me know if you can see > metrics starting with kafka_ and zookeeper_ > > > > > > On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: > > The prometheus server is also running in the cattle-Prometheus namespace. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:47 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > The prometheus operator is not the prometheus server as I mentioned in the > previous email. > > Did you use the prometheus operator to deploy the prometheus server then? > Where the prometheus server is running? not the operator ;) > > > > On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: > > Hello Paolo, > > My apologies for the confusion. I am using Prometheus operator, which is > already installed at the cluster under the name space cattle-prometheus. My > assumption is I do not need to install the Prometheus operator again and I > can push the metrics from the kafka cluster ( deployed in > strimzi-kafka-operator namespace) to the prometheus operator running in > cattle-prometheus namespace. Is my assumption correct? If yes, what is the > steps that I need to follow. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:39 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > What's not clear to me now is ... if you are using the Prometheus Operator > or not. > > In the first email you mentioned the Prometheus operator while in the last > one just Prometheus (the server) which are two different things. > > The first one is a Kubernetes operator for deploying Prometheus server and > services to identify metrics to scrape from other applications; the second > one is ... the actual Prometheus server. > > The steps we provide in the documentation are about having Prometheus > operator running and deploying a few resources for deploying a Prometheus > server and the services for monitoring Kafka metrics (all these resources > are handled by the prometheus operator which than "inject" the > configuration into prometheus). > > > > On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: > > Hello Paolo, > > Let me explain the environment I have and maybe there is something not > setup right. I am using Rancher on Kubernetes. On the environment > Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My > Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to > produce the metrics of the kafka cluster in strimzi-kafka-operator name > space to the prometheus instance in cattle-prometheus namespace. I am not > sure what are the steps to do so. If you can help me with the steps to do > so, it will be great. I could not figure that from the documentation. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 1:33 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Hi Debdeep, > > can you provide the Prometheus log? > > In the prometheus.yaml file, the ClusterRoleBinding resource provides the > binding between the promethes-server service account and the cluster role > but specifing "myproject" as namespace. Is you Kafka cluster running in the > myproject namespace? Is the same for the Prometheus server you installed > through the Prometheus operator? > > > > Thanks, > > Paolo. > > > > On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: > > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From ppatiern at redhat.com Tue Aug 6 15:37:45 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Tue, 6 Aug 2019 17:37:45 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: One side note, let's take the strimzi-service-monitor.yaml on side for now. It should go in the Kafka cluster namespace not in the Prometheus operator namespace but then it means to enable another feature in Prometheus operator to watch for service monitor in other namespace. Btw without that service monitor you should be able to see some metrics (I just tried here). On Tue, Aug 6, 2019 at 5:36 PM Paolo Patierno wrote: > Are you able to access the prometheus server web UI? > Can you provide the prometheus server log? > > On Tue, Aug 6, 2019 at 5:28 PM Das, Debdeep > wrote: > >> D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl >> create secret generic additional-scrape-configs >> --from-file=prometheus-additional.yaml -n cattle-prometheus >> >> secret/additional-scrape-configs created >> >> >> >> D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl >> get secret -n cattle-prometheus >> >> NAME >> TYPE >> DATA AGE >> >> additional-scrape-configs >> Opaque 1 21m >> >> >> >> >> >> >> >> D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl >> apply -f strimzi-service-monitor.yaml -n cattle-prometheus >> >> servicemonitor.monitoring.coreos.com/kafka-service-monitor created >> >> >> >> D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl >> apply -f prometheus-rules.yaml -n cattle-prometheus >> >> prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created >> >> >> >> D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl >> apply -f prometheus.yaml -n cattle-prometheus >> >> clusterrole.rbac.authorization.k8s.io/prometheus-server created >> >> serviceaccount/prometheus-server created >> >> clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created >> >> prometheus.monitoring.coreos.com/prometheus created >> >> >> >> >> >> I still don?t see any metrics for Kafka. >> >> >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> *From:* Stanislav Knot [mailto:sknot at redhat.com] >> *Sent:* Tuesday, August 6, 2019 8:23 PM >> *To:* Das, Debdeep >> *Cc:* Paolo Patierno ; strimzi at redhat.com >> *Subject:* Re: [Strimzi] Strimzi with Prometheus >> >> >> >> You can use >> >> kubectl create secret generic additional-scrape-configs >> --from-file=prometheus-additional.yaml -n your-namespace >> >> >> >> Stanislav >> >> >> >> On Tue, Aug 6, 2019 at 4:20 PM Das, Debdeep >> wrote: >> >> I am using Strimzi 0.12.2 >> >> >> >> I created the secret >> >> >> >> kubectl create secret generic additional-scrape-configs >> --from-file=prometheus-additional.yaml >> >> >> >> The next step is to get the secret name. How to do that? >> >> >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> *From:* Paolo Patierno [mailto:ppatiern at redhat.com] >> *Sent:* Tuesday, August 6, 2019 7:42 PM >> *To:* Das, Debdeep >> *Cc:* strimzi at redhat.com >> *Subject:* Re: [Strimzi] Strimzi with Prometheus >> >> >> >> As you can see from the log as I thought you are missing the additional >> scrape configs creation. >> >> >> >> On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep >> wrote: >> >> >> >> Please find attached the Prometheus operator log. >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> *From:* Paolo Patierno [mailto:ppatiern at redhat.com] >> *Sent:* Monday, August 5, 2019 8:57 PM >> *To:* Das, Debdeep >> *Cc:* strimzi at redhat.com >> *Subject:* Re: [Strimzi] Strimzi with Prometheus >> >> >> >> Can you provide me the prometheus-operator log? >> >> From the above steps you are missing the creation of the Secret >> containing the additional configs and maybe the Prometheus operator is >> failing to search for it. >> >> This Secret is referenced in the prometheus.yaml` file as following: >> >> >> >> additionalScrapeConfigs: >> >> name: additional-scrape-configs >> >> key: prometheus-additional.yaml >> >> >> >> The Secret creation is described here: >> https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str >> (it's from master doc, what's the Strimzi version you are using?) >> >> >> >> After that, access to the prometheus web UI and let me know if you can >> see metrics starting with kafka_ and zookeeper_ >> >> >> >> >> >> On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep >> wrote: >> >> The prometheus server is also running in the cattle-Prometheus namespace. >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> *From:* Paolo Patierno [mailto:ppatiern at redhat.com] >> *Sent:* Monday, August 5, 2019 3:47 PM >> *To:* Das, Debdeep >> *Cc:* strimzi at redhat.com >> *Subject:* Re: [Strimzi] Strimzi with Prometheus >> >> >> >> The prometheus operator is not the prometheus server as I mentioned in >> the previous email. >> >> Did you use the prometheus operator to deploy the prometheus server then? >> Where the prometheus server is running? not the operator ;) >> >> >> >> On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep >> wrote: >> >> Hello Paolo, >> >> My apologies for the confusion. I am using Prometheus operator, which is >> already installed at the cluster under the name space cattle-prometheus. My >> assumption is I do not need to install the Prometheus operator again and I >> can push the metrics from the kafka cluster ( deployed in >> strimzi-kafka-operator namespace) to the prometheus operator running in >> cattle-prometheus namespace. Is my assumption correct? If yes, what is the >> steps that I need to follow. >> >> >> >> Regards >> >> Debdeep >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> *From:* Paolo Patierno [mailto:ppatiern at redhat.com] >> *Sent:* Monday, August 5, 2019 3:39 PM >> *To:* Das, Debdeep >> *Cc:* strimzi at redhat.com >> *Subject:* Re: [Strimzi] Strimzi with Prometheus >> >> >> >> What's not clear to me now is ... if you are using the Prometheus >> Operator or not. >> >> In the first email you mentioned the Prometheus operator while in the >> last one just Prometheus (the server) which are two different things. >> >> The first one is a Kubernetes operator for deploying Prometheus server >> and services to identify metrics to scrape from other applications; the >> second one is ... the actual Prometheus server. >> >> The steps we provide in the documentation are about having Prometheus >> operator running and deploying a few resources for deploying a Prometheus >> server and the services for monitoring Kafka metrics (all these resources >> are handled by the prometheus operator which than "inject" the >> configuration into prometheus). >> >> >> >> On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep >> wrote: >> >> Hello Paolo, >> >> Let me explain the environment I have and maybe there is something not >> setup right. I am using Rancher on Kubernetes. On the environment >> Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My >> Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to >> produce the metrics of the kafka cluster in strimzi-kafka-operator name >> space to the prometheus instance in cattle-prometheus namespace. I am not >> sure what are the steps to do so. If you can help me with the steps to do >> so, it will be great. I could not figure that from the documentation. >> >> >> >> Regards >> >> Debdeep >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> *From:* Paolo Patierno [mailto:ppatiern at redhat.com] >> *Sent:* Monday, August 5, 2019 1:33 PM >> *To:* Das, Debdeep >> *Cc:* strimzi at redhat.com >> *Subject:* Re: [Strimzi] Strimzi with Prometheus >> >> >> >> Hi Debdeep, >> >> can you provide the Prometheus log? >> >> In the prometheus.yaml file, the ClusterRoleBinding resource provides the >> binding between the promethes-server service account and the cluster role >> but specifing "myproject" as namespace. Is you Kafka cluster running in the >> myproject namespace? Is the same for the Prometheus server you installed >> through the Prometheus operator? >> >> >> >> Thanks, >> >> Paolo. >> >> >> >> On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep >> wrote: >> >> Hello, >> >> I have a Kubernetes environment with Strimzi and Kafka deployed. I am >> able to produce and consume successfully. I want to use Prometheus and >> Grafana to report metrics. The environment already has Prometheus Operator >> and Grafana installed, but in a different namespace than where I installed >> the Strimzi + Kafka. >> >> I used the instructions here https://strimzi.io/docs/latest/#prometheus >> to add the rules. I used the following instructions under the namespace >> where Strimzi-Kafka is installed, but I don?t see the Rules under >> Prometheus. The metrics are already added in the YAML to deploy the >> Kafka(+zookeper) >> >> >> >> kubectl apply -f strimzi-service-monitor.yaml >> >> kubectl apply -f prometheus-rules.yaml >> >> kubectl apply -f prometheus.yaml >> >> Can you please help what needs to be done to get the rules. I guess >> because the rules are not getting to Prometheus, the Grafana is not >> reporting anything on the Dashboard. >> >> >> >> Regards >> >> Debdeep >> >> >> >> *Debdeep Das* | Lead Engineer | Communication Products - InTC >> >> Unisys | +91 9980101186 | debdeep.das at in.unisys.com >> >> >> >> [image: unisys_logo] >> >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all devices. >> >> [image: Grey_LI] [image: >> Grey_TW] [image: Grey_YT] >> [image: Grey_FB] >> [image: Grey_Vimeo] >> [image: Grey_UB] >> >> >> >> _______________________________________________ >> Strimzi mailing list >> Strimzi at redhat.com >> https://www.redhat.com/mailman/listinfo/strimzi >> >> >> >> >> -- >> >> *Paolo Patierno* >> >> Principal Software Engineer, Messaging & IoT >> >> Red Hat >> >> >> >> >> >> >> >> >> >> >> -- >> >> *Paolo Patierno* >> >> Principal Software Engineer, Messaging & IoT >> >> Red Hat >> >> >> >> >> >> >> >> >> >> >> -- >> >> *Paolo Patierno* >> >> Principal Software Engineer, Messaging & IoT >> >> Red Hat >> >> >> >> >> >> >> >> >> >> >> -- >> >> *Paolo Patierno* >> >> Principal Software Engineer, Messaging & IoT >> >> Red Hat >> >> >> >> >> >> >> >> >> >> >> -- >> >> *Paolo Patierno* >> >> Principal Software Engineer, Messaging & IoT >> >> Red Hat >> >> >> >> >> >> >> >> _______________________________________________ >> Strimzi mailing list >> Strimzi at redhat.com >> https://www.redhat.com/mailman/listinfo/strimzi >> >> > > -- > > Paolo Patierno > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From debdeep.das at in.unisys.com Tue Aug 6 15:40:25 2019 From: debdeep.das at in.unisys.com (Das, Debdeep) Date: Tue, 6 Aug 2019 15:40:25 +0000 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: What does the prometheus.yaml do? Does is deploy a server? Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com] Sent: Tuesday, August 6, 2019 9:08 PM To: Das, Debdeep Cc: Stanislav Knot ; strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus One side note, let's take the strimzi-service-monitor.yaml on side for now. It should go in the Kafka cluster namespace not in the Prometheus operator namespace but then it means to enable another feature in Prometheus operator to watch for service monitor in other namespace. Btw without that service monitor you should be able to see some metrics (I just tried here). On Tue, Aug 6, 2019 at 5:36 PM Paolo Patierno > wrote: Are you able to access the prometheus server web UI? Can you provide the prometheus server log? On Tue, Aug 6, 2019 at 5:28 PM Das, Debdeep > wrote: D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n cattle-prometheus secret/additional-scrape-configs created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl get secret -n cattle-prometheus NAME TYPE DATA AGE additional-scrape-configs Opaque 1 21m D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f strimzi-service-monitor.yaml -n cattle-prometheus servicemonitor.monitoring.coreos.com/kafka-service-monitor created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f prometheus-rules.yaml -n cattle-prometheus prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f prometheus.yaml -n cattle-prometheus clusterrole.rbac.authorization.k8s.io/prometheus-server created serviceaccount/prometheus-server created clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created prometheus.monitoring.coreos.com/prometheus created I still don?t see any metrics for Kafka. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Stanislav Knot [mailto:sknot at redhat.com ] Sent: Tuesday, August 6, 2019 8:23 PM To: Das, Debdeep > Cc: Paolo Patierno >; strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus You can use kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n your-namespace Stanislav On Tue, Aug 6, 2019 at 4:20 PM Das, Debdeep > wrote: I am using Strimzi 0.12.2 I created the secret kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml The next step is to get the secret name. How to do that? Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Tuesday, August 6, 2019 7:42 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus As you can see from the log as I thought you are missing the additional scrape configs creation. On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep > wrote: Please find attached the Prometheus operator log. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 8:57 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Can you provide me the prometheus-operator log? >From the above steps you are missing the creation of the Secret containing the additional configs and maybe the Prometheus operator is failing to search for it. This Secret is referenced in the prometheus.yaml` file as following: additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional.yaml The Secret creation is described here: https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str (it's from master doc, what's the Strimzi version you are using?) After that, access to the prometheus web UI and let me know if you can see metrics starting with kafka_ and zookeeper_ On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: The prometheus server is also running in the cattle-Prometheus namespace. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:47 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus The prometheus operator is not the prometheus server as I mentioned in the previous email. Did you use the prometheus operator to deploy the prometheus server then? Where the prometheus server is running? not the operator ;) On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: Hello Paolo, My apologies for the confusion. I am using Prometheus operator, which is already installed at the cluster under the name space cattle-prometheus. My assumption is I do not need to install the Prometheus operator again and I can push the metrics from the kafka cluster ( deployed in strimzi-kafka-operator namespace) to the prometheus operator running in cattle-prometheus namespace. Is my assumption correct? If yes, what is the steps that I need to follow. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:39 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus What's not clear to me now is ... if you are using the Prometheus Operator or not. In the first email you mentioned the Prometheus operator while in the last one just Prometheus (the server) which are two different things. The first one is a Kubernetes operator for deploying Prometheus server and services to identify metrics to scrape from other applications; the second one is ... the actual Prometheus server. The steps we provide in the documentation are about having Prometheus operator running and deploying a few resources for deploying a Prometheus server and the services for monitoring Kafka metrics (all these resources are handled by the prometheus operator which than "inject" the configuration into prometheus). On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: Hello Paolo, Let me explain the environment I have and maybe there is something not setup right. I am using Rancher on Kubernetes. On the environment Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to produce the metrics of the kafka cluster in strimzi-kafka-operator name space to the prometheus instance in cattle-prometheus namespace. I am not sure what are the steps to do so. If you can help me with the steps to do so, it will be great. I could not figure that from the documentation. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 1:33 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Hi Debdeep, can you provide the Prometheus log? In the prometheus.yaml file, the ClusterRoleBinding resource provides the binding between the promethes-server service account and the cluster role but specifing "myproject" as namespace. Is you Kafka cluster running in the myproject namespace? Is the same for the Prometheus server you installed through the Prometheus operator? Thanks, Paolo. On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: Hello, I have a Kubernetes environment with Strimzi and Kafka deployed. I am able to produce and consume successfully. I want to use Prometheus and Grafana to report metrics. The environment already has Prometheus Operator and Grafana installed, but in a different namespace than where I installed the Strimzi + Kafka. I used the instructions here https://strimzi.io/docs/latest/#prometheus to add the rules. I used the following instructions under the namespace where Strimzi-Kafka is installed, but I don?t see the Rules under Prometheus. The metrics are already added in the YAML to deploy the Kafka(+zookeper) kubectl apply -f strimzi-service-monitor.yaml kubectl apply -f prometheus-rules.yaml kubectl apply -f prometheus.yaml Can you please help what needs to be done to get the rules. I guess because the rules are not getting to Prometheus, the Grafana is not reporting anything on the Dashboard. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 7855 bytes Desc: not available URL: From ppatiern at redhat.com Tue Aug 6 15:42:48 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Tue, 6 Aug 2019 17:42:48 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: Yes it describes the Prometheus custom resource for the Prometheus operator which then deploys the Prometheus server. On Tue, Aug 6, 2019 at 5:40 PM Das, Debdeep wrote: > What does the prometheus.yaml do? Does is deploy a server? > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 9:08 PM > *To:* Das, Debdeep > *Cc:* Stanislav Knot ; strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > One side note, let's take the strimzi-service-monitor.yaml on side for now. > > It should go in the Kafka cluster namespace not in the Prometheus operator > namespace but then it means to enable another feature in Prometheus > operator to watch for service monitor in other namespace. > > Btw without that service monitor you should be able to see some metrics (I > just tried here). > > > > On Tue, Aug 6, 2019 at 5:36 PM Paolo Patierno wrote: > > Are you able to access the prometheus server web UI? > > Can you provide the prometheus server log? > > > > On Tue, Aug 6, 2019 at 5:28 PM Das, Debdeep > wrote: > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl > create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml -n cattle-prometheus > > secret/additional-scrape-configs created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl > get secret -n cattle-prometheus > > NAME > TYPE > DATA AGE > > additional-scrape-configs > Opaque 1 21m > > > > > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f strimzi-service-monitor.yaml -n cattle-prometheus > > servicemonitor.monitoring.coreos.com/kafka-service-monitor created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f prometheus-rules.yaml -n cattle-prometheus > > prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f prometheus.yaml -n cattle-prometheus > > clusterrole.rbac.authorization.k8s.io/prometheus-server created > > serviceaccount/prometheus-server created > > clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created > > prometheus.monitoring.coreos.com/prometheus created > > > > > > I still don?t see any metrics for Kafka. > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Stanislav Knot [mailto:sknot at redhat.com] > *Sent:* Tuesday, August 6, 2019 8:23 PM > *To:* Das, Debdeep > *Cc:* Paolo Patierno ; strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > You can use > > kubectl create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml -n your-namespace > > > > Stanislav > > > > On Tue, Aug 6, 2019 at 4:20 PM Das, Debdeep > wrote: > > I am using Strimzi 0.12.2 > > > > I created the secret > > > > kubectl create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml > > > > The next step is to get the secret name. How to do that? > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 7:42 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > As you can see from the log as I thought you are missing the additional > scrape configs creation. > > > > On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep > wrote: > > > > Please find attached the Prometheus operator log. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 8:57 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Can you provide me the prometheus-operator log? > > From the above steps you are missing the creation of the Secret containing > the additional configs and maybe the Prometheus operator is failing to > search for it. > > This Secret is referenced in the prometheus.yaml` file as following: > > > > additionalScrapeConfigs: > > name: additional-scrape-configs > > key: prometheus-additional.yaml > > > > The Secret creation is described here: > https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str > (it's from master doc, what's the Strimzi version you are using?) > > > > After that, access to the prometheus web UI and let me know if you can see > metrics starting with kafka_ and zookeeper_ > > > > > > On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: > > The prometheus server is also running in the cattle-Prometheus namespace. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:47 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > The prometheus operator is not the prometheus server as I mentioned in the > previous email. > > Did you use the prometheus operator to deploy the prometheus server then? > Where the prometheus server is running? not the operator ;) > > > > On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: > > Hello Paolo, > > My apologies for the confusion. I am using Prometheus operator, which is > already installed at the cluster under the name space cattle-prometheus. My > assumption is I do not need to install the Prometheus operator again and I > can push the metrics from the kafka cluster ( deployed in > strimzi-kafka-operator namespace) to the prometheus operator running in > cattle-prometheus namespace. Is my assumption correct? If yes, what is the > steps that I need to follow. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:39 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > What's not clear to me now is ... if you are using the Prometheus Operator > or not. > > In the first email you mentioned the Prometheus operator while in the last > one just Prometheus (the server) which are two different things. > > The first one is a Kubernetes operator for deploying Prometheus server and > services to identify metrics to scrape from other applications; the second > one is ... the actual Prometheus server. > > The steps we provide in the documentation are about having Prometheus > operator running and deploying a few resources for deploying a Prometheus > server and the services for monitoring Kafka metrics (all these resources > are handled by the prometheus operator which than "inject" the > configuration into prometheus). > > > > On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: > > Hello Paolo, > > Let me explain the environment I have and maybe there is something not > setup right. I am using Rancher on Kubernetes. On the environment > Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My > Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to > produce the metrics of the kafka cluster in strimzi-kafka-operator name > space to the prometheus instance in cattle-prometheus namespace. I am not > sure what are the steps to do so. If you can help me with the steps to do > so, it will be great. I could not figure that from the documentation. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 1:33 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Hi Debdeep, > > can you provide the Prometheus log? > > In the prometheus.yaml file, the ClusterRoleBinding resource provides the > binding between the promethes-server service account and the cluster role > but specifing "myproject" as namespace. Is you Kafka cluster running in the > myproject namespace? Is the same for the Prometheus server you installed > through the Prometheus operator? > > > > Thanks, > > Paolo. > > > > On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: > > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From debdeep.das at in.unisys.com Tue Aug 6 15:44:55 2019 From: debdeep.das at in.unisys.com (Das, Debdeep) Date: Tue, 6 Aug 2019 15:44:55 +0000 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: Since my environment already has a Prometheus server, I want to add the configurations in that server for the metrics. What do I need to update in the YAML? Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com] Sent: Tuesday, August 6, 2019 9:13 PM To: Das, Debdeep Cc: Stanislav Knot ; strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Yes it describes the Prometheus custom resource for the Prometheus operator which then deploys the Prometheus server. On Tue, Aug 6, 2019 at 5:40 PM Das, Debdeep > wrote: What does the prometheus.yaml do? Does is deploy a server? Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Tuesday, August 6, 2019 9:08 PM To: Das, Debdeep > Cc: Stanislav Knot >; strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus One side note, let's take the strimzi-service-monitor.yaml on side for now. It should go in the Kafka cluster namespace not in the Prometheus operator namespace but then it means to enable another feature in Prometheus operator to watch for service monitor in other namespace. Btw without that service monitor you should be able to see some metrics (I just tried here). On Tue, Aug 6, 2019 at 5:36 PM Paolo Patierno > wrote: Are you able to access the prometheus server web UI? Can you provide the prometheus server log? On Tue, Aug 6, 2019 at 5:28 PM Das, Debdeep > wrote: D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n cattle-prometheus secret/additional-scrape-configs created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl get secret -n cattle-prometheus NAME TYPE DATA AGE additional-scrape-configs Opaque 1 21m D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f strimzi-service-monitor.yaml -n cattle-prometheus servicemonitor.monitoring.coreos.com/kafka-service-monitor created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f prometheus-rules.yaml -n cattle-prometheus prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f prometheus.yaml -n cattle-prometheus clusterrole.rbac.authorization.k8s.io/prometheus-server created serviceaccount/prometheus-server created clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created prometheus.monitoring.coreos.com/prometheus created I still don?t see any metrics for Kafka. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Stanislav Knot [mailto:sknot at redhat.com ] Sent: Tuesday, August 6, 2019 8:23 PM To: Das, Debdeep > Cc: Paolo Patierno >; strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus You can use kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n your-namespace Stanislav On Tue, Aug 6, 2019 at 4:20 PM Das, Debdeep > wrote: I am using Strimzi 0.12.2 I created the secret kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml The next step is to get the secret name. How to do that? Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Tuesday, August 6, 2019 7:42 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus As you can see from the log as I thought you are missing the additional scrape configs creation. On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep > wrote: Please find attached the Prometheus operator log. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 8:57 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Can you provide me the prometheus-operator log? >From the above steps you are missing the creation of the Secret containing the additional configs and maybe the Prometheus operator is failing to search for it. This Secret is referenced in the prometheus.yaml` file as following: additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional.yaml The Secret creation is described here: https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str (it's from master doc, what's the Strimzi version you are using?) After that, access to the prometheus web UI and let me know if you can see metrics starting with kafka_ and zookeeper_ On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: The prometheus server is also running in the cattle-Prometheus namespace. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:47 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus The prometheus operator is not the prometheus server as I mentioned in the previous email. Did you use the prometheus operator to deploy the prometheus server then? Where the prometheus server is running? not the operator ;) On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: Hello Paolo, My apologies for the confusion. I am using Prometheus operator, which is already installed at the cluster under the name space cattle-prometheus. My assumption is I do not need to install the Prometheus operator again and I can push the metrics from the kafka cluster ( deployed in strimzi-kafka-operator namespace) to the prometheus operator running in cattle-prometheus namespace. Is my assumption correct? If yes, what is the steps that I need to follow. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:39 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus What's not clear to me now is ... if you are using the Prometheus Operator or not. In the first email you mentioned the Prometheus operator while in the last one just Prometheus (the server) which are two different things. The first one is a Kubernetes operator for deploying Prometheus server and services to identify metrics to scrape from other applications; the second one is ... the actual Prometheus server. The steps we provide in the documentation are about having Prometheus operator running and deploying a few resources for deploying a Prometheus server and the services for monitoring Kafka metrics (all these resources are handled by the prometheus operator which than "inject" the configuration into prometheus). On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: Hello Paolo, Let me explain the environment I have and maybe there is something not setup right. I am using Rancher on Kubernetes. On the environment Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to produce the metrics of the kafka cluster in strimzi-kafka-operator name space to the prometheus instance in cattle-prometheus namespace. I am not sure what are the steps to do so. If you can help me with the steps to do so, it will be great. I could not figure that from the documentation. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 1:33 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Hi Debdeep, can you provide the Prometheus log? In the prometheus.yaml file, the ClusterRoleBinding resource provides the binding between the promethes-server service account and the cluster role but specifing "myproject" as namespace. Is you Kafka cluster running in the myproject namespace? Is the same for the Prometheus server you installed through the Prometheus operator? Thanks, Paolo. On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: Hello, I have a Kubernetes environment with Strimzi and Kafka deployed. I am able to produce and consume successfully. I want to use Prometheus and Grafana to report metrics. The environment already has Prometheus Operator and Grafana installed, but in a different namespace than where I installed the Strimzi + Kafka. I used the instructions here https://strimzi.io/docs/latest/#prometheus to add the rules. I used the following instructions under the namespace where Strimzi-Kafka is installed, but I don?t see the Rules under Prometheus. The metrics are already added in the YAML to deploy the Kafka(+zookeper) kubectl apply -f strimzi-service-monitor.yaml kubectl apply -f prometheus-rules.yaml kubectl apply -f prometheus.yaml Can you please help what needs to be done to get the rules. I guess because the rules are not getting to Prometheus, the Grafana is not reporting anything on the Dashboard. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 7855 bytes Desc: not available URL: From ppatiern at redhat.com Tue Aug 6 15:47:38 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Tue, 6 Aug 2019 17:47:38 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: How did you deploy this Prometheus server? Using a Prometheus operator or not? On Tue, Aug 6, 2019 at 5:45 PM Das, Debdeep wrote: > Since my environment already has a Prometheus server, I want to add the > configurations in that server for the metrics. What do I need to update in > the YAML? > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 9:13 PM > *To:* Das, Debdeep > *Cc:* Stanislav Knot ; strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Yes it describes the Prometheus custom resource for the Prometheus > operator which then deploys the Prometheus server. > > > > On Tue, Aug 6, 2019 at 5:40 PM Das, Debdeep > wrote: > > What does the prometheus.yaml do? Does is deploy a server? > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 9:08 PM > *To:* Das, Debdeep > *Cc:* Stanislav Knot ; strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > One side note, let's take the strimzi-service-monitor.yaml on side for now. > > It should go in the Kafka cluster namespace not in the Prometheus operator > namespace but then it means to enable another feature in Prometheus > operator to watch for service monitor in other namespace. > > Btw without that service monitor you should be able to see some metrics (I > just tried here). > > > > On Tue, Aug 6, 2019 at 5:36 PM Paolo Patierno wrote: > > Are you able to access the prometheus server web UI? > > Can you provide the prometheus server log? > > > > On Tue, Aug 6, 2019 at 5:28 PM Das, Debdeep > wrote: > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl > create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml -n cattle-prometheus > > secret/additional-scrape-configs created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl > get secret -n cattle-prometheus > > NAME > TYPE > DATA AGE > > additional-scrape-configs > Opaque 1 21m > > > > > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f strimzi-service-monitor.yaml -n cattle-prometheus > > servicemonitor.monitoring.coreos.com/kafka-service-monitor created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f prometheus-rules.yaml -n cattle-prometheus > > prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f prometheus.yaml -n cattle-prometheus > > clusterrole.rbac.authorization.k8s.io/prometheus-server created > > serviceaccount/prometheus-server created > > clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created > > prometheus.monitoring.coreos.com/prometheus created > > > > > > I still don?t see any metrics for Kafka. > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Stanislav Knot [mailto:sknot at redhat.com] > *Sent:* Tuesday, August 6, 2019 8:23 PM > *To:* Das, Debdeep > *Cc:* Paolo Patierno ; strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > You can use > > kubectl create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml -n your-namespace > > > > Stanislav > > > > On Tue, Aug 6, 2019 at 4:20 PM Das, Debdeep > wrote: > > I am using Strimzi 0.12.2 > > > > I created the secret > > > > kubectl create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml > > > > The next step is to get the secret name. How to do that? > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 7:42 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > As you can see from the log as I thought you are missing the additional > scrape configs creation. > > > > On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep > wrote: > > > > Please find attached the Prometheus operator log. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 8:57 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Can you provide me the prometheus-operator log? > > From the above steps you are missing the creation of the Secret containing > the additional configs and maybe the Prometheus operator is failing to > search for it. > > This Secret is referenced in the prometheus.yaml` file as following: > > > > additionalScrapeConfigs: > > name: additional-scrape-configs > > key: prometheus-additional.yaml > > > > The Secret creation is described here: > https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str > (it's from master doc, what's the Strimzi version you are using?) > > > > After that, access to the prometheus web UI and let me know if you can see > metrics starting with kafka_ and zookeeper_ > > > > > > On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: > > The prometheus server is also running in the cattle-Prometheus namespace. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:47 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > The prometheus operator is not the prometheus server as I mentioned in the > previous email. > > Did you use the prometheus operator to deploy the prometheus server then? > Where the prometheus server is running? not the operator ;) > > > > On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: > > Hello Paolo, > > My apologies for the confusion. I am using Prometheus operator, which is > already installed at the cluster under the name space cattle-prometheus. My > assumption is I do not need to install the Prometheus operator again and I > can push the metrics from the kafka cluster ( deployed in > strimzi-kafka-operator namespace) to the prometheus operator running in > cattle-prometheus namespace. Is my assumption correct? If yes, what is the > steps that I need to follow. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:39 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > What's not clear to me now is ... if you are using the Prometheus Operator > or not. > > In the first email you mentioned the Prometheus operator while in the last > one just Prometheus (the server) which are two different things. > > The first one is a Kubernetes operator for deploying Prometheus server and > services to identify metrics to scrape from other applications; the second > one is ... the actual Prometheus server. > > The steps we provide in the documentation are about having Prometheus > operator running and deploying a few resources for deploying a Prometheus > server and the services for monitoring Kafka metrics (all these resources > are handled by the prometheus operator which than "inject" the > configuration into prometheus). > > > > On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: > > Hello Paolo, > > Let me explain the environment I have and maybe there is something not > setup right. I am using Rancher on Kubernetes. On the environment > Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My > Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to > produce the metrics of the kafka cluster in strimzi-kafka-operator name > space to the prometheus instance in cattle-prometheus namespace. I am not > sure what are the steps to do so. If you can help me with the steps to do > so, it will be great. I could not figure that from the documentation. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 1:33 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Hi Debdeep, > > can you provide the Prometheus log? > > In the prometheus.yaml file, the ClusterRoleBinding resource provides the > binding between the promethes-server service account and the cluster role > but specifing "myproject" as namespace. Is you Kafka cluster running in the > myproject namespace? Is the same for the Prometheus server you installed > through the Prometheus operator? > > > > Thanks, > > Paolo. > > > > On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: > > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: From debdeep.das at in.unisys.com Tue Aug 6 15:51:48 2019 From: debdeep.das at in.unisys.com (Das, Debdeep) Date: Tue, 6 Aug 2019 15:51:48 +0000 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: I need to find that out. We are using Rancher and it comes in built with Prometheus. I need find that out from the architect and I will get back to out on that. Currently I disabled monitoring on the server and decided to set up everything with the Kafka cluster namespace. At least let me understand how it works and then I will use the platform level Prometheus. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com] Sent: Tuesday, August 6, 2019 9:18 PM To: Das, Debdeep Cc: Stanislav Knot ; strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus How did you deploy this Prometheus server? Using a Prometheus operator or not? On Tue, Aug 6, 2019 at 5:45 PM Das, Debdeep > wrote: Since my environment already has a Prometheus server, I want to add the configurations in that server for the metrics. What do I need to update in the YAML? Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Tuesday, August 6, 2019 9:13 PM To: Das, Debdeep > Cc: Stanislav Knot >; strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Yes it describes the Prometheus custom resource for the Prometheus operator which then deploys the Prometheus server. On Tue, Aug 6, 2019 at 5:40 PM Das, Debdeep > wrote: What does the prometheus.yaml do? Does is deploy a server? Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Tuesday, August 6, 2019 9:08 PM To: Das, Debdeep > Cc: Stanislav Knot >; strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus One side note, let's take the strimzi-service-monitor.yaml on side for now. It should go in the Kafka cluster namespace not in the Prometheus operator namespace but then it means to enable another feature in Prometheus operator to watch for service monitor in other namespace. Btw without that service monitor you should be able to see some metrics (I just tried here). On Tue, Aug 6, 2019 at 5:36 PM Paolo Patierno > wrote: Are you able to access the prometheus server web UI? Can you provide the prometheus server log? On Tue, Aug 6, 2019 at 5:28 PM Das, Debdeep > wrote: D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n cattle-prometheus secret/additional-scrape-configs created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl get secret -n cattle-prometheus NAME TYPE DATA AGE additional-scrape-configs Opaque 1 21m D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f strimzi-service-monitor.yaml -n cattle-prometheus servicemonitor.monitoring.coreos.com/kafka-service-monitor created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f prometheus-rules.yaml -n cattle-prometheus prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl apply -f prometheus.yaml -n cattle-prometheus clusterrole.rbac.authorization.k8s.io/prometheus-server created serviceaccount/prometheus-server created clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created prometheus.monitoring.coreos.com/prometheus created I still don?t see any metrics for Kafka. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Stanislav Knot [mailto:sknot at redhat.com ] Sent: Tuesday, August 6, 2019 8:23 PM To: Das, Debdeep > Cc: Paolo Patierno >; strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus You can use kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n your-namespace Stanislav On Tue, Aug 6, 2019 at 4:20 PM Das, Debdeep > wrote: I am using Strimzi 0.12.2 I created the secret kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml The next step is to get the secret name. How to do that? Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Tuesday, August 6, 2019 7:42 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus As you can see from the log as I thought you are missing the additional scrape configs creation. On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep > wrote: Please find attached the Prometheus operator log. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 8:57 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Can you provide me the prometheus-operator log? >From the above steps you are missing the creation of the Secret containing the additional configs and maybe the Prometheus operator is failing to search for it. This Secret is referenced in the prometheus.yaml` file as following: additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional.yaml The Secret creation is described here: https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str (it's from master doc, what's the Strimzi version you are using?) After that, access to the prometheus web UI and let me know if you can see metrics starting with kafka_ and zookeeper_ On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: The prometheus server is also running in the cattle-Prometheus namespace. Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:47 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus The prometheus operator is not the prometheus server as I mentioned in the previous email. Did you use the prometheus operator to deploy the prometheus server then? Where the prometheus server is running? not the operator ;) On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: Hello Paolo, My apologies for the confusion. I am using Prometheus operator, which is already installed at the cluster under the name space cattle-prometheus. My assumption is I do not need to install the Prometheus operator again and I can push the metrics from the kafka cluster ( deployed in strimzi-kafka-operator namespace) to the prometheus operator running in cattle-prometheus namespace. Is my assumption correct? If yes, what is the steps that I need to follow. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 3:39 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus What's not clear to me now is ... if you are using the Prometheus Operator or not. In the first email you mentioned the Prometheus operator while in the last one just Prometheus (the server) which are two different things. The first one is a Kubernetes operator for deploying Prometheus server and services to identify metrics to scrape from other applications; the second one is ... the actual Prometheus server. The steps we provide in the documentation are about having Prometheus operator running and deploying a few resources for deploying a Prometheus server and the services for monitoring Kafka metrics (all these resources are handled by the prometheus operator which than "inject" the configuration into prometheus). On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: Hello Paolo, Let me explain the environment I have and maybe there is something not setup right. I am using Rancher on Kubernetes. On the environment Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to produce the metrics of the kafka cluster in strimzi-kafka-operator name space to the prometheus instance in cattle-prometheus namespace. I am not sure what are the steps to do so. If you can help me with the steps to do so, it will be great. I could not figure that from the documentation. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. From: Paolo Patierno [mailto:ppatiern at redhat.com ] Sent: Monday, August 5, 2019 1:33 PM To: Das, Debdeep > Cc: strimzi at redhat.com Subject: Re: [Strimzi] Strimzi with Prometheus Hi Debdeep, can you provide the Prometheus log? In the prometheus.yaml file, the ClusterRoleBinding resource provides the binding between the promethes-server service account and the cluster role but specifing "myproject" as namespace. Is you Kafka cluster running in the myproject namespace? Is the same for the Prometheus server you installed through the Prometheus operator? Thanks, Paolo. On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: Hello, I have a Kubernetes environment with Strimzi and Kafka deployed. I am able to produce and consume successfully. I want to use Prometheus and Grafana to report metrics. The environment already has Prometheus Operator and Grafana installed, but in a different namespace than where I installed the Strimzi + Kafka. I used the instructions here https://strimzi.io/docs/latest/#prometheus to add the rules. I used the following instructions under the namespace where Strimzi-Kafka is installed, but I don?t see the Rules under Prometheus. The metrics are already added in the YAML to deploy the Kafka(+zookeper) kubectl apply -f strimzi-service-monitor.yaml kubectl apply -f prometheus-rules.yaml kubectl apply -f prometheus.yaml Can you please help what needs to be done to get the rules. I guess because the rules are not getting to Prometheus, the Grafana is not reporting anything on the Dashboard. Regards Debdeep Debdeep Das | Lead Engineer | Communication Products - InTC Unisys | +91 9980101186 | debdeep.das at in.unisys.com THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat _______________________________________________ Strimzi mailing list Strimzi at redhat.com https://www.redhat.com/mailman/listinfo/strimzi -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 7855 bytes Desc: not available URL: From ppatiern at redhat.com Tue Aug 6 16:02:55 2019 From: ppatiern at redhat.com (Paolo Patierno) Date: Tue, 6 Aug 2019 18:02:55 +0200 Subject: [Strimzi] Strimzi with Prometheus In-Reply-To: References: Message-ID: To be honest I don't know about Rancher but most of the users have their own Prometheus instance deployed for monitoring the Kafka cluster in the same namespace. On Tue, Aug 6, 2019 at 5:52 PM Das, Debdeep wrote: > I need to find that out. We are using Rancher and it comes in built with > Prometheus. I need find that out from the architect and I will get back to > out on that. Currently I disabled monitoring on the server and decided to > set up everything with the Kafka cluster namespace. At least let me > understand how it works and then I will use the platform level Prometheus. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 9:18 PM > *To:* Das, Debdeep > *Cc:* Stanislav Knot ; strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > How did you deploy this Prometheus server? Using a Prometheus operator or > not? > > > > On Tue, Aug 6, 2019 at 5:45 PM Das, Debdeep > wrote: > > Since my environment already has a Prometheus server, I want to add the > configurations in that server for the metrics. What do I need to update in > the YAML? > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 9:13 PM > *To:* Das, Debdeep > *Cc:* Stanislav Knot ; strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Yes it describes the Prometheus custom resource for the Prometheus > operator which then deploys the Prometheus server. > > > > On Tue, Aug 6, 2019 at 5:40 PM Das, Debdeep > wrote: > > What does the prometheus.yaml do? Does is deploy a server? > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 9:08 PM > *To:* Das, Debdeep > *Cc:* Stanislav Knot ; strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > One side note, let's take the strimzi-service-monitor.yaml on side for now. > > It should go in the Kafka cluster namespace not in the Prometheus operator > namespace but then it means to enable another feature in Prometheus > operator to watch for service monitor in other namespace. > > Btw without that service monitor you should be able to see some metrics (I > just tried here). > > > > On Tue, Aug 6, 2019 at 5:36 PM Paolo Patierno wrote: > > Are you able to access the prometheus server web UI? > > Can you provide the prometheus server log? > > > > On Tue, Aug 6, 2019 at 5:28 PM Das, Debdeep > wrote: > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl > create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml -n cattle-prometheus > > secret/additional-scrape-configs created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\additional-properties>kubectl > get secret -n cattle-prometheus > > NAME > TYPE > DATA AGE > > additional-scrape-configs > Opaque 1 21m > > > > > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f strimzi-service-monitor.yaml -n cattle-prometheus > > servicemonitor.monitoring.coreos.com/kafka-service-monitor created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f prometheus-rules.yaml -n cattle-prometheus > > prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created > > > > D:\DD\Office\Delivery\Projects\Falcon\kafka\strimzi-kafka-operator\metrics\examples\prometheus\install>kubectl > apply -f prometheus.yaml -n cattle-prometheus > > clusterrole.rbac.authorization.k8s.io/prometheus-server created > > serviceaccount/prometheus-server created > > clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created > > prometheus.monitoring.coreos.com/prometheus created > > > > > > I still don?t see any metrics for Kafka. > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Stanislav Knot [mailto:sknot at redhat.com] > *Sent:* Tuesday, August 6, 2019 8:23 PM > *To:* Das, Debdeep > *Cc:* Paolo Patierno ; strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > You can use > > kubectl create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml -n your-namespace > > > > Stanislav > > > > On Tue, Aug 6, 2019 at 4:20 PM Das, Debdeep > wrote: > > I am using Strimzi 0.12.2 > > > > I created the secret > > > > kubectl create secret generic additional-scrape-configs > --from-file=prometheus-additional.yaml > > > > The next step is to get the secret name. How to do that? > > > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Tuesday, August 6, 2019 7:42 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > As you can see from the log as I thought you are missing the additional > scrape configs creation. > > > > On Tue, Aug 6, 2019 at 3:43 PM Das, Debdeep > wrote: > > > > Please find attached the Prometheus operator log. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 8:57 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Can you provide me the prometheus-operator log? > > From the above steps you are missing the creation of the Secret containing > the additional configs and maybe the Prometheus operator is failing to > search for it. > > This Secret is referenced in the prometheus.yaml` file as following: > > > > additionalScrapeConfigs: > > name: additional-scrape-configs > > key: prometheus-additional.yaml > > > > The Secret creation is described here: > https://strimzi.io/docs/latest/#proc-metrics-deploying-prometheus-str > (it's from master doc, what's the Strimzi version you are using?) > > > > After that, access to the prometheus web UI and let me know if you can see > metrics starting with kafka_ and zookeeper_ > > > > > > On Mon, Aug 5, 2019 at 12:30 PM Das, Debdeep > wrote: > > The prometheus server is also running in the cattle-Prometheus namespace. > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:47 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > The prometheus operator is not the prometheus server as I mentioned in the > previous email. > > Did you use the prometheus operator to deploy the prometheus server then? > Where the prometheus server is running? not the operator ;) > > > > On Mon, Aug 5, 2019 at 12:14 PM Das, Debdeep > wrote: > > Hello Paolo, > > My apologies for the confusion. I am using Prometheus operator, which is > already installed at the cluster under the name space cattle-prometheus. My > assumption is I do not need to install the Prometheus operator again and I > can push the metrics from the kafka cluster ( deployed in > strimzi-kafka-operator namespace) to the prometheus operator running in > cattle-prometheus namespace. Is my assumption correct? If yes, what is the > steps that I need to follow. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 3:39 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > What's not clear to me now is ... if you are using the Prometheus Operator > or not. > > In the first email you mentioned the Prometheus operator while in the last > one just Prometheus (the server) which are two different things. > > The first one is a Kubernetes operator for deploying Prometheus server and > services to identify metrics to scrape from other applications; the second > one is ... the actual Prometheus server. > > The steps we provide in the documentation are about having Prometheus > operator running and deploying a few resources for deploying a Prometheus > server and the services for monitoring Kafka metrics (all these resources > are handled by the prometheus operator which than "inject" the > configuration into prometheus). > > > > On Mon, Aug 5, 2019 at 12:01 PM Das, Debdeep > wrote: > > Hello Paolo, > > Let me explain the environment I have and maybe there is something not > setup right. I am using Rancher on Kubernetes. On the environment > Prometheus and Grafana runs on a separate namespace ?cattle-prometheus?. My > Kafka cluster is deployed in strimzi-kafka-operator namespace. I want to > produce the metrics of the kafka cluster in strimzi-kafka-operator name > space to the prometheus instance in cattle-prometheus namespace. I am not > sure what are the steps to do so. If you can help me with the steps to do > so, it will be great. I could not figure that from the documentation. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > *From:* Paolo Patierno [mailto:ppatiern at redhat.com] > *Sent:* Monday, August 5, 2019 1:33 PM > *To:* Das, Debdeep > *Cc:* strimzi at redhat.com > *Subject:* Re: [Strimzi] Strimzi with Prometheus > > > > Hi Debdeep, > > can you provide the Prometheus log? > > In the prometheus.yaml file, the ClusterRoleBinding resource provides the > binding between the promethes-server service account and the cluster role > but specifing "myproject" as namespace. Is you Kafka cluster running in the > myproject namespace? Is the same for the Prometheus server you installed > through the Prometheus operator? > > > > Thanks, > > Paolo. > > > > On Mon, Jul 29, 2019 at 5:36 PM Das, Debdeep > wrote: > > Hello, > > I have a Kubernetes environment with Strimzi and Kafka deployed. I am able > to produce and consume successfully. I want to use Prometheus and Grafana > to report metrics. The environment already has Prometheus Operator and > Grafana installed, but in a different namespace than where I installed the > Strimzi + Kafka. > > I used the instructions here https://strimzi.io/docs/latest/#prometheus > to add the rules. I used the following instructions under the namespace > where Strimzi-Kafka is installed, but I don?t see the Rules under > Prometheus. The metrics are already added in the YAML to deploy the > Kafka(+zookeper) > > > > kubectl apply -f strimzi-service-monitor.yaml > > kubectl apply -f prometheus-rules.yaml > > kubectl apply -f prometheus.yaml > > Can you please help what needs to be done to get the rules. I guess > because the rules are not getting to Prometheus, the Grafana is not > reporting anything on the Dashboard. > > > > Regards > > Debdeep > > > > *Debdeep Das* | Lead Engineer | Communication Products - InTC > > Unisys | +91 9980101186 | debdeep.das at in.unisys.com > > > > [image: unisys_logo] > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all devices. > > [image: Grey_LI] [image: > Grey_TW] [image: Grey_YT] > [image: Grey_FB] > [image: Grey_Vimeo] > [image: Grey_UB] > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > _______________________________________________ > Strimzi mailing list > Strimzi at redhat.com > https://www.redhat.com/mailman/listinfo/strimzi > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > > > > > -- > > *Paolo Patierno* > > Principal Software Engineer, Messaging & IoT > > Red Hat > > > > > > > -- Paolo Patierno Principal Software Engineer, Messaging & IoT Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4550 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 778 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 755 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 1737 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 791 bytes Desc: not available URL: