<div dir="ltr">Sorry, I somehow missed this email earlier. I will have a look at the article - we are still working on the encryption and security support.<div><br></div><div>As for the second point ... the usual setup is that a single Kafka node is shared between multiple topics / partitions. Unless you have some exceptionally busy topic, this should work fine. AFAIK it is quite normal to have 100s of partitions on a single node. </div><div><br></div><div>The storage type question is a bit more complicated. We plan to have support for local SSD disks which are used quite often (Kafka replication can take care of the fault tolerance). Network attached SSDs such as Amazon EBS volumes should also work fine. Things like Gluster or NFS are a bit more complicated. I don't think NFS would work well for Kafka. With Gluster we are yet to do some testing. The challenge with GlusterFS will be that it normally shares the same network for storage as is used by Kafka. So throughput requirements on the network are multiplied.</div><div><br></div><div>Thanks & Regards</div><div>Jakub</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 7, 2018 at 12:26 PM, Budzik, Przemyslaw <span dir="ltr"><<a href="mailto:Przemyslaw.Budzik@sabre.com" target="_blank">Przemyslaw.Budzik@sabre.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div lang="EN-US" link="blue" vlink="purple"> <div class="m_-9012839331702852034WordSection1"> <p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%">Thank you Jakub. It worked for us, but we are not sure what may be potentially missing. <u></u><u></u></p> <p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%">Two new questions:<u></u><u></u></p> <ol style="margin-top:0in" start="1" type="1"> <li class="m_-9012839331702852034MsoListParagraph" style="margin-bottom:10.0pt;margin-left:0in;line-height:115%"> What do you think about using Istio for TLS and AuthN? Liike so: <a href="https://medium.com/jayanthg/transparently-securing-kafka-istio-style-with-up-to-300-higher-performance-than-native-tls-in-aa95731b5d40" target="_blank"> https://medium.com/jayanthg/<wbr>transparently-securing-kafka-<wbr>istio-style-with-up-to-300-<wbr>higher-performance-than-<wbr>native-tls-in-aa95731b5d40</a><u></u><u></u></li><li class="m_-9012839331702852034MsoListParagraph" style="margin-bottom:10.0pt;margin-left:0in;line-height:115%"> I suppose ideally each topic partition is located on a separate physical node/storage. Otherwise we have concurrent writes/reads to the same underlying storage (in different files). However in an environment like the one we are building, there may be hundreds of topics multiplied by its replicas (hence very many files) and for sure there will be many residing on the same physical node (and more than 1 accessed a time). I am not able to assess how this would perform given different setup specifics like SSD disks, GlusterFS/NFS etc. etc. Do you see any issues with this approach <u></u><u></u></li></ol> <p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%">Przemek<u></u><u></u></p> <p class="MsoNormal"><u></u> <u></u></p> <p class="MsoNormal"><span class=""><b>From:</b> Jakub Scholz [mailto:<a href="mailto:jakub@scholz.cz" target="_blank">jakub@scholz.cz</a>] <br> </span><b>Sent:</b> Monday, April 30, 2018 12:44 PM<br> <b>To:</b> Budzik, Przemyslaw <<a href="mailto:Przemyslaw.Budzik@sabre.com" target="_blank">Przemyslaw.Budzik@sabre.com</a>><br> <b>Cc:</b> <a href="mailto:strimzi@redhat.com" target="_blank">strimzi@redhat.com</a>; Pietrucha, Piotr <<a href="mailto:Piotr.Pietrucha@sabre.com" target="_blank">Piotr.Pietrucha@sabre.com</a>>; Pachowicz, Krzysztof <<a href="mailto:Krzysztof.Pachowicz@sabre.com" target="_blank">Krzysztof.Pachowicz@sabre.com</a><wbr>></p><div><div class="h5"><br> <b>Subject:</b> Re: [Strimzi] RC1 of Strimzi 0.3.0 available<u></u><u></u></div></div><p></p><div><div class="h5"> <p class="MsoNormal"><u></u> <u></u></p> <div> <p class="MsoNormal">Hi Przemek,<u></u><u></u></p> <div> <p class="MsoNormal"><u></u> <u></u></p> </div> <div> <p class="MsoNormal">The permissions in this role are all needed by Strimzi. They are not cluster wide, but on OpenShift you need cluster admin to create the role. As an alternative, you can use the existing role "edit" which should already exist in your cluster. To do that, just ignore the 02-role.yaml file and modify the 03-binding.yaml file. In this file, replace the reference to the role "strimzi-cluster-controller-<wbr>role" with the role named "edit" and create the binding. That should give Strimzi and its service account all the permissions it needs.<u></u><u></u></p> </div> <div> <p class="MsoNormal"><u></u> <u></u></p> </div> <div> <p class="MsoNormal">As for the number of ZK nodes - in the examples folder we have two config maps. The config map <a href="https://github.com/strimzi/strimzi/blob/master/examples/configmaps/cluster-controller/kafka-ephemeral.yaml" target="_blank"> kafka-ephemeral.yaml</a> is using non-persisitent emptyDir storage and is more or less supposed to be used for development ad testing purposes. It has by default only 1 ZK node. You can always modify the ConfigMap to add more ZK nodes - normally you should use odd number of ZK nodes, so increasing it to 3 or 5 would be reasonable. There is another ConfigMap in the examples folder - <a href="https://github.com/strimzi/strimzi/blob/master/examples/configmaps/cluster-controller/kafka-persistent.yaml" target="_blank"> kafka-persistent.yaml</a>. This one is using persistent volumes and has by default 3 Zookeeper nodes. But again, even here you can modify it according to your needs.<u></u><u></u></p> </div> <div> <p class="MsoNormal"><u></u> <u></u></p> </div> <div> <p class="MsoNormal">Thanks & Regards<u></u><u></u></p> </div> <div> <p class="MsoNormal">Jakub<u></u><u></u></p> </div> </div> <div> <p class="MsoNormal"><u></u> <u></u></p> <div> <p class="MsoNormal">On Mon, Apr 30, 2018 at 12:24 PM, Budzik, Przemyslaw <<a href="mailto:Przemyslaw.Budzik@sabre.com" target="_blank">Przemyslaw.Budzik@sabre.com</a>> wrote:<u></u><u></u></p> <blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> <div> <div> <p class="MsoNormal">Jakub,<u></u><u></u></p> <p class="MsoNormal"><span style="color:black"> </span><u></u><u></u></p> <p class="MsoNormal"><span style="color:black">We are trying to install Strmzi in our DEV cluster. The thing is that there are certain restrictions and we’ve come across a problem:</span><u></u><u></u></p> <p class="MsoNormal"><span style="color:black"> </span><u></u><u></u></p> <p class="MsoNormal">$ oc create -f 02-role.yaml <u></u><u></u></p> <p class="MsoNormal">Error from server (Forbidden): error when creating "02-role.yaml": <a href="http://roles.rbac.authorization.k8s.io" target="_blank">roles.rbac.authorization.k8s.<wbr>io</a> "strimzi-cluster-controller-<wbr>role" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["<wbr>events"], APIGroups:[""], Verbs:["create"]} PolicyRule{Resources:["<wbr>replicationcontrollers"], APIGroups:["extensions"], Verbs:["get"]} PolicyRule{Resources:["<wbr>replicationcontrollers"], APIGroups:["extensions"], Verbs:["list"]} PolicyRule{Resources:["<wbr>replicationcontrollers"], APIGroups:["extensions"], Verbs:["watch"]} PolicyRule{Resources:["<wbr>replicationcontrollers"], APIGroups:["extensions"], Verbs:["create"]} PolicyRule{Resources:["<wbr>replicationcontrollers"], APIGroups:["extensions"], Verbs:["delete"]} PolicyRule{Resources:["<wbr>replicationcontrollers"], APIGroups:["extensions"], Verbs:["patch"]} PolicyRule{Resources:["<wbr>replicationcontrollers"], APIGroups:["extensions"], Verbs:["update"]} PolicyRule{Resources:["<wbr>deploymentconfigs/status"], APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["create"]} PolicyRule{Resources:["<wbr>deploymentconfigs/status"], APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["delete"]} PolicyRule{Resources:["<wbr>deploymentconfigs/status"], APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["patch"]} PolicyRule{Resources:["<wbr>deploymentconfigs/status"], APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["update"]} PolicyRule{Resources:["<wbr>deploymentconfigs/finalizers"]<wbr>, APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["get"]} PolicyRule{Resources:["<wbr>deploymentconfigs/finalizers"]<wbr>, APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["list"]} PolicyRule{Resources:["<wbr>deploymentconfigs/finalizers"]<wbr>, APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["watch"]} PolicyRule{Resources:["<wbr>deploymentconfigs/finalizers"]<wbr>, APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["create"]} PolicyRule{Resources:["<wbr>deploymentconfigs/finalizers"]<wbr>, APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["delete"]} PolicyRule{Resources:["<wbr>deploymentconfigs/finalizers"]<wbr>, APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["patch"]} PolicyRule{Resources:["<wbr>deploymentconfigs/finalizers"]<wbr>, APIGroups:["<a href="http://apps.openshift.io" target="_blank">apps.openshift.io</a>"<wbr>], Verbs:["update"]} PolicyRule{Resources:["<wbr>imagestreams/status"], APIGroups:["<a href="http://image.openshift.io" target="_blank">image.openshift.io</a><wbr>"], Verbs:["create"]} PolicyRule{Resources:["<wbr>imagestreams/status"], APIGroups:["<a href="http://image.openshift.io" target="_blank">image.openshift.io</a><wbr>"], Verbs:["delete"]} PolicyRule{Resources:["<wbr>imagestreams/status"], APIGroups:["<a href="http://image.openshift.io" target="_blank">image.openshift.io</a><wbr>"], Verbs:["patch"]} PolicyRule{Resources:["<wbr>imagestreams/status"], APIGroups:["<a href="http://image.openshift.io" target="_blank">image.openshift.io</a><wbr>"], Verbs:["update"]}] user=&{SG0556477 …<u></u><u></u></p> <p class="MsoNormal"><span style="color:black"> </span><u></u><u></u></p> <p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%"> <span style="color:black">What we are wondering is if all permissions (<a href="https://github.com/strimzi/strimzi/blob/master/examples/install/cluster-controller/02-role.yaml" target="_blank">https://github.com/strimzi/<wbr>strimzi/blob/master/examples/<wbr>install/cluster-controller/02-<wbr>role.yaml</a>) are required? For example we don’t need to install Strimzi in other namespaces. We want to start with only operating within one namespace. Do you think we may reduce permissions (which?) and hence solve the issue (temporarily until our admins give us cluster admin role). </span><u></u><u></u></p> <p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%"> Also, we observed that after installing Strimzi, there is one ZK pod. Should it not be more e.g. 3 for HA ? What topology would you recommend for a start (3+3 ? ) <u></u><u></u></p> <p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%"> Piotr, Krzysztof – please add if needed. <u></u><u></u></p> <p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%"> Przemek<u></u><u></u></p> <p class="MsoNormal"> <u></u><u></u></p> <p class="MsoNormal"><b>From:</b> Jakub Scholz [mailto:<a href="mailto:jakub@scholz.cz" target="_blank">jakub@scholz.cz</a>] <br> <b>Sent:</b> Tuesday, April 17, 2018 6:42 PM<u></u><u></u></p> <div> <div> <p class="MsoNormal"><br> <b>To:</b> Budzik, Przemyslaw <<a href="mailto:Przemyslaw.Budzik@sabre.com" target="_blank">Przemyslaw.Budzik@sabre.com</a>><br> <b>Cc:</b> <a href="mailto:strimzi@redhat.com" target="_blank">strimzi@redhat.com</a>; Chylek, Artur <<a href="mailto:Artur.Chylek@sabre.com" target="_blank">Artur.Chylek@sabre.com</a>>; Wnuk, Norbert <<a href="mailto:Norbert.Wnuk@sabre.com" target="_blank">Norbert.Wnuk@sabre.com</a>><br> <b>Subject:</b> Re: [Strimzi] RC1 of Strimzi 0.3.0 available<u></u><u></u></p> </div> </div> <div> <div> <p class="MsoNormal"> <u></u><u></u></p> <div> <p class="MsoNormal">One of the problems is that the config map is a namespaced resource. But if you want a Kafka cluster which serves multiple namespaces then Kafka topic is not namespaced resource. That means a lot of complications. For example what should we do when the same topic is configured within different namespaces? So I do not think we can see the presence of the config map for given topic in the namespace as a proof of ownership for the topic.<u></u><u></u></p> <div> <p class="MsoNormal"> <u></u><u></u></p> </div> <div> <p class="MsoNormal">I think that the general idea is quite good. But the implementation might not be trivial. I think I will need some more time to think about it. Either way this is far beyond the 0.3.0 scope.<u></u><u></u></p> </div> <div> <p class="MsoNormal"> <u></u><u></u></p> </div> <div> <p class="MsoNormal">Thanks<u></u><u></u></p> </div> <div> <p class="MsoNormal">Jakub<u></u><u></u></p> </div> <div> <p class="MsoNormal"> <u></u><u></u></p> </div> </div> <div> <p class="MsoNormal"> <u></u><u></u></p> <div> <p class="MsoNormal">On Mon, Apr 16, 2018 at 10:24 AM, Budzik, Przemyslaw <<a href="mailto:Przemyslaw.Budzik@sabre.com" target="_blank">Przemyslaw.Budzik@sabre.com</a>> wrote:<u></u><u></u></p> <blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> <div> <div> <p class="MsoNormal">Thank you Jakub. I have a question. Is there any way to bind namespace with topic in terms of authorization? Consider a scenario.<u></u><u></u></p> <p class="MsoNormal"><span style="color:black"> </span><u></u><u></u></p> <ol start="1" type="1"> <li class="MsoNormal" style="color:black"> Team A creates a topic called “reservations”. They own the config map and of course they can publish to this topic.<u></u><u></u></li><li class="MsoNormal" style="color:black"> Team B is interested in consuming A’s messages. I am guessing if they know the name of the topic, they can use it ? Ideally since they are in the namespace B, we’d like to be able to say they can, but others can’t. Also I mean we give them read, but not write. Most likely it’s the creator of topic writing and other namespaces (or the same) consuming. <u></u><u></u></li></ol> <p class="MsoNormal"> <u></u><u></u></p> <p class="MsoNormal">If that is not supported, is that on your roadmap ?<u></u><u></u></p> <p class="MsoNormal"> <u></u><u></u></p> <p class="MsoNormal"><b>From:</b> Jakub Scholz [mailto:<a href="mailto:jakub@scholz.cz" target="_blank">jakub@scholz.cz</a>] <br> <b>Sent:</b> Friday, April 13, 2018 2:44 PM<br> <b>To:</b> Budzik, Przemyslaw <<a href="mailto:Przemyslaw.Budzik@sabre.com" target="_blank">Przemyslaw.Budzik@sabre.com</a>><br> <b>Cc:</b> <a href="mailto:strimzi@redhat.com" target="_blank">strimzi@redhat.com</a>; Chylek, Artur <<a href="mailto:Artur.Chylek@sabre.com" target="_blank">Artur.Chylek@sabre.com</a>>; Wnuk, Norbert <<a href="mailto:Norbert.Wnuk@sabre.com" target="_blank">Norbert.Wnuk@sabre.com</a>><br> <b>Subject:</b> Re: [Strimzi] RC1 of Strimzi 0.3.0 available<u></u><u></u></p> <p class="MsoNormal"> <u></u><u></u></p> <div> <p class="MsoNormal">Hi,<u></u><u></u></p> <div> <p class="MsoNormal"> <u></u><u></u></p> </div> <div> <p class="MsoNormal">Well, it is a release candidate. So while I hope that there will be no issues, there is always possibility. But feel free to give it a try. It doesn't bring any significant new features, but it improves a bit the resilience with which the controller reacts to different problems and situations. If you didn't experienced any issues with 0.2.0, you can just stay with it.<u></u><u></u></p> </div> <div> <p class="MsoNormal"> <u></u><u></u></p> </div> <div> <p class="MsoNormal">I hope the final release will be done early next week.<u></u><u></u></p> </div> <div> <p class="MsoNormal"> <u></u><u></u></p> </div> <div> <p class="MsoNormal">Thanks & Regards<u></u><u></u></p> </div> <div> <p class="MsoNormal">Jakub<u></u><u></u></p> </div> </div> <div> <p class="MsoNormal"> <u></u><u></u></p> <div> <p class="MsoNormal">On Fri, Apr 13, 2018 at 1:10 AM, Budzik, Przemyslaw <<a href="mailto:Przemyslaw.Budzik@sabre.com" target="_blank">Przemyslaw.Budzik@sabre.com</a>> wrote:<u></u><u></u></p> <blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> <div> <div> <p class="MsoNormal">Jakub,<u></u><u></u></p> <p class="MsoNormal"><span style="color:black"> </span><u></u><u></u></p> <p class="MsoNormal"><span style="color:black">Which version would you recommend using in our case ? One of the teams is now installing Strimzi in our DEV environment.</span><u></u><u></u></p> <p class="MsoNormal"> <u></u><u></u></p> <p class="MsoNormal">Thanks,<u></u><u></u></p> <p class="MsoNormal">Przemek<u></u><u></u></p> <p class="MsoNormal"> <u></u><u></u></p> <p class="MsoNormal"><b>From:</b> <a href="mailto:strimzi-bounces@redhat.com" target="_blank">strimzi-bounces@redhat.com</a> [mailto:<a href="mailto:strimzi-bounces@redhat.com" target="_blank">strimzi-bounces@<wbr>redhat.com</a>] <b>On Behalf Of </b>Jakub Scholz<br> <b>Sent:</b> Friday, April 13, 2018 4:32 AM<br> <b>To:</b> <a href="mailto:strimzi@redhat.com" target="_blank">strimzi@redhat.com</a><br> <b>Subject:</b> [Strimzi] RC1 of Strimzi 0.3.0 available<u></u><u></u></p> <div> <div> <p class="MsoNormal"> <u></u><u></u></p> <div> <p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222;background:white">Hi,</span><u></u><u></u></p> <div> <p class="MsoNormal" style="background:white"> <span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><u></u><u></u></p> </div> <div> <p class="MsoNormal" style="background:white"> <span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222">The Release Candidate 1 for the first Strimzi 0.3.0 is now available. </span><u></u><u></u></p> </div> <div> <p class="MsoNormal" style="background:white"> <span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><u></u><u></u></p> </div> <div> <p class="MsoNormal" style="background:white"> <span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222">If you are interested have a look at the release candidate and give us your feedback:</span><u></u><u></u></p> </div> <div> <p class="MsoNormal" style="background:white"> <span style="font-size:9.5pt"><a href="https://github.com/strimzi/strimzi/releases/tag/0.3.0-rc1" target="_blank">https://github.com/strimzi/<wbr>strimzi/releases/tag/0.3.0-rc1</a></span><u></u><u></u></p> </div> <div> <p class="MsoNormal" style="background:white"> <span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><u></u><u></u></p> </div> <div> <p class="MsoNormal" style="background:white"> <span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222">Thanks & Regards</span><u></u><u></u></p> </div> <p class="MsoNormal">Jakub<u></u><u></u></p> </div> </div> </div> </div> </div> </blockquote> </div> <p class="MsoNormal"> <u></u><u></u></p> </div> </div> </div> </blockquote> </div> <p class="MsoNormal"> <u></u><u></u></p> </div> </div> </div> </div> </div> </blockquote> </div> <p class="MsoNormal"><u></u> <u></u></p> </div> </div></div></div> </div> </blockquote></div><br></div>