<div dir="ltr">HI <a class="gmail_plusreply" id="plusReplyChip-5" href="mailto:supittma@redhat.com" tabindex="-1">@Summers Pittman</a>,<div><br></div><div>The watches are implemented in the controller as you can check here for an example[1]. See here[2] their implementation.</div><div>I understand that you are using an enmasse operator then the best approach would be introduced the watches there. I mean contribute with a PR. However, I did not try to do it but shows that would be possible impl watches in the UPS Oper controllers for the enmasse's objects as well. In order to do it, you will use the attributes here[3] for example. </div><div><br></div><div>I hope that it is what you are looking for.<br></div><div><br></div><div>[1] - <a href="https://github.com/aerogear/mobile-security-service-operator/blob/master/pkg/controller/mobilesecurityservice/controller.go#L59">https://github.com/aerogear/mobile-security-service-operator/blob/master/pkg/controller/mobilesecurityservice/controller.go#L59</a></div><div>[2] - <a href="https://github.com/aerogear/mobile-security-service-operator/blob/master/pkg/controller/mobilesecurityservice/watches.go#L15">https://github.com/aerogear/mobile-security-service-operator/blob/master/pkg/controller/mobilesecurityservice/watches.go#L15</a></div><div>[3] - <a href="https://github.com/aerogear/mobile-security-service-operator/blob/master/pkg/controller/mobilesecurityservice/watches.go#L17-L18">https://github.com/aerogear/mobile-security-service-operator/blob/master/pkg/controller/mobilesecurityservice/watches.go#L17-L18</a></div><div> <br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="background-color:rgb(253,253,253)"><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;background-color:rgb(255,255,255);font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>CAMILA</span> <span>MACEDO</span></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;background-color:rgb(255,255,255);font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>SR. SOFTWARE ENGINEER, </span><span style="color:rgb(170,170,170);margin:0px"></span>RED HAT CLOUD SERVICES</p><p style="font-family:overpass,sans-serif;background-color:rgb(255,255,255);margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>UK</span></a></p><p style="font-family:overpass,sans-serif;background-color:rgb(255,255,255);margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)"><span>IM: <span>cmacedo</span></span></p><p style="font-family:overpass,sans-serif;background-color:rgb(255,255,255);margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)">Phone: +44 7853500035</p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium;background-color:rgb(255,255,255)"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img src="https://www.redhat.com/files/brand/email/sig-redhat.png" width="90" height="auto"></a></td></tr></tbody></table><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 31, 2019 at 4:10 PM Summers Pittman <<a href="mailto:supittma@redhat.com">supittma@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 31, 2019 at 8:41 AM Jens Reimann <<a href="mailto:jreimann@redhat.com" target="_blank">jreimann@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 31, 2019 at 1:27 PM Camila Macedo <<a href="mailto:cmacedo@redhat.com" target="_blank">cmacedo@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi @Summers, <div><br></div><div>Regards the items 2 and 3, could we not implement these actions in the UPS operators instead of the expected implementation on enmasse?</div><div>Regards item 4 shows that a new release was made 10 hours ago with this fix[1](0.29.0[2]). So, could not it be solved by using this version? </div></div></blockquote><div><br></div><div>I would hope we would not do that. Simply because that means a bit (or a lot) of effort, but only solves to the problem for one consumer of enmasse. However others (even the IoT components of enmasse) face the same issue. And we would need to invest some time there as well.</div><div><br></div><div>So I would prefer some solution, which fixes the root cause, rather than working around the issue in various other locations.<br></div></div></div></blockquote><div><br></div><div>For the watch issue there are probably some low hanging fruits to add to enmasse. The easiest and probably pretty important one I would say is make the get operations for the various controllers return an error if a watch is requested. Currently the endpoints return a list of their resources.</div><div><br></div><div>In k8s itself, here is the error they return is a watch isn't supported : </div><div>1: <a href="https://github.com/kubernetes/kubernetes/blob/1ff857e1269fc100c8dda64287180cbfb5b2e657/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/get.go#L239" target="_blank">https://github.com/kubernetes/kubernetes/blob/1ff857e1269fc100c8dda64287180cbfb5b2e657/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/get.go#L239</a></div><div>2: <a href="https://github.com/kubernetes/apimachinery/blob/master/pkg/api/errors/errors.go#L283" target="_blank">https://github.com/kubernetes/apimachinery/blob/master/pkg/api/errors/errors.go#L283</a></div><div><br></div><div>I'm not sure how that will interact with client-go, but an explicit error will make it easier for developers to know this is not a problem in their code. </div><div><br></div><div><br></div><div><br></div><div>For configuring redelivery and DLQs, that should probably be some properties in Address custom resources. But I'm not really a messaging expert so I don't have a lot of strong ideas or opinions.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>[1] - <a href="https://github.com/EnMasseProject/enmasse/issues/2927" target="_blank">https://github.com/EnMasseProject/enmasse/issues/2927</a></div><div>[2] - <a href="https://github.com/EnMasseProject/enmasse/releases/tag/0.29.0" target="_blank">https://github.com/EnMasseProject/enmasse/releases/tag/0.29.0</a></div><div> <br clear="all"><div><div dir="ltr" class="gmail-m_-3604847869306135027gmail-m_2931351075916016513gmail-m_-9190718942072786810m_-276311094956139300m_-6047242088729779332m_406782978233792488gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="background-color:rgb(253,253,253)"><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;background-color:rgb(255,255,255);font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>CAMILA</span> <span>MACEDO</span></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;background-color:rgb(255,255,255);font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>SR. SOFTWARE ENGINEER, </span><span style="color:rgb(170,170,170);margin:0px"></span>RED HAT CLOUD SERVICES</p><p style="font-family:overpass,sans-serif;background-color:rgb(255,255,255);margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>UK</span></a></p><p style="font-family:overpass,sans-serif;background-color:rgb(255,255,255);margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)"><span>IM: <span>cmacedo</span></span></p><p style="font-family:overpass,sans-serif;background-color:rgb(255,255,255);margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)">Phone: +44 7853500035</p><table style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium;background-color:rgb(255,255,255)" border="0"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img width="90" height="auto"></a></td></tr></tbody></table><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 29, 2019 at 4:07 PM Summers Pittman <<a href="mailto:supittma@redhat.com" target="_blank">supittma@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I'm cross posting this on the AeroGear(<a href="https://groups.google.com/forum/#!forum/aerogear" target="_blank">https://groups.google.com/forum/#!forum/aerogear</a>) and the enmasse (<a href="https://www.redhat.com/mailman/listinfo/enmasse" target="_blank">https://www.redhat.com/mailman/listinfo/enmasse</a>) mailing lists. <br><br>In the past few months we've been adding the ability to use an external broker with the Unified Push Server[UPS] (<a href="https://github.com/aerogear/aerogear-unifiedpush-server" target="_blank">https://github.com/aerogear/aerogear-unifiedpush-server</a>). We've added support for external AMQP connections to the UPS container image, as well as added support for connecting to enmasse using the UPS operator (<a href="https://github.com/aerogear/unifiedpush-operator" target="_blank">https://github.com/aerogear/unifiedpush-operator</a>). Following are our experiences with enmasse as well as some problems we encountered.<div><br></div><div>1: The creation of enmasse resources using the UPS operator (based on client-go <a href="https://github.com/kubernetes/client-go" target="_blank">https://github.com/kubernetes/client-go</a>) worked pretty much as expected. The only downside was that the golang bindings from enmasse were out of date, but the team accepted our PR's and released a new version with updated bindings. Much appreciated!</div><div><br></div><div>2: The enmasse controllers for address, addresspaces, and messagingusers do not implement k8s watches, nor do they throw an exception if you try to watch a resource. This causes a lot of errors to be logged in the UPS operator as we need to watch and maintain those resources to keep our service running. There is an open enmasse issue here that describes the issue : <a href="https://github.com/EnMasseProject/enmasse/issues/1280" target="_blank">https://github.com/EnMasseProject/enmasse/issues/1280</a></div><div><br></div><div>3: Enmasse doesn't block owner deletion nor implement a finalizer to handle being orphaned. This means that when our operator deletes a UPS server, the addresses and addressspace resources we create don't get deleted, nor do we get a notification that they aren't being deleted. </div><div><br></div><div>4: Enmasse has no way to configure delivery failures. Usually with a message broker we want failed messages to be retried, then retried with a backoff, and eventually retired to a dead letter queue. In enmasse the default behavior is to retry immediately and infinitely. There is very little we can use the broker for to get an alert that this is happening. In the case of a production error this means that enmasse will death star our service with an infinite stream of doomed messages. We are researching workarounds, but per this issue (<a href="https://github.com/EnMasseProject/enmasse/issues/2927" target="_blank">https://github.com/EnMasseProject/enmasse/issues/2927</a>) it seems like there is little we can do at the level of address resource configuration.</div><div><br></div><div>5: Creation of resources (addresses, addressspaces, messaging users) and getting their respective information into our deployment using our operator was really straight forward. Like amazingly simple and directly straight forward.<br><br>Feel free to reply inline to the points. Now for my actual question : <br><br>We're trying to find workaround to #4 and are actively soliciting ideas. Right now we're looking at implementing more aggressive messaging handling so messages will always be consumed even if we would prefer them to be retried (for instance if the network connection to our push service was unavailable for a moment, a credential expired etc). <br><br>This doesn't help for unexpected errors in the UPS code (who doesn't love a good NPE), and for that we might want to have the UPS operator keep an eye out for extreme message re deliveries and auto-manually delete those messages, but that would be better handled by a DLQ mechanism.<br><br>Thoughts on the potential workarounds?</div></div> <p></p> -- <br> You received this message because you are subscribed to the Google Groups "Aerogear" group.<br> To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:aerogear+unsubscribe@googlegroups.com" target="_blank">aerogear+unsubscribe@googlegroups.com</a>.<br> To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/aerogear/CAEQz2CtEDGth1cWj3nKPNmZDBg6cSvLW_0D6nfhgmyX6ZW6t3w%40mail.gmail.com?utm_medium=email&utm_source=footer" target="_blank">https://groups.google.com/d/msgid/aerogear/CAEQz2CtEDGth1cWj3nKPNmZDBg6cSvLW_0D6nfhgmyX6ZW6t3w%40mail.gmail.com</a>.<br> </blockquote></div> _______________________________________________<br> enmasse mailing list<br> <a href="mailto:enmasse@redhat.com" target="_blank">enmasse@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/enmasse" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/enmasse</a><br> </blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_-3604847869306135027gmail-m_2931351075916016513gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Jens Reimann<br>Principal Software Engineer / EMEA ENG Middleware<br>Werner-von-Siemens-Ring 14<br>85630 Grasbrunn<br>Germany<br>phone: +49 89 2050 71286<br>_____________________________________________________________________________<br><br>Red Hat GmbH, <a href="http://www.de.redhat.com" target="_blank">www.de.redhat.com</a>,<br>Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243,<br>Managing Directors: Paul Argiry, Charles Cachera, Tom Savage, Michael O'Neill<br></div></div></div></div></div></div></div> </blockquote></div></div> <p></p> -- <br> You received this message because you are subscribed to the Google Groups "Aerogear" group.<br> To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:aerogear+unsubscribe@googlegroups.com" target="_blank">aerogear+unsubscribe@googlegroups.com</a>.<br> To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/aerogear/CAEQz2Cs4WHPux9LFdHB_E8ajzj-npXzw_FSn9rtL1o%2B%3DUefUQg%40mail.gmail.com?utm_medium=email&utm_source=footer" target="_blank">https://groups.google.com/d/msgid/aerogear/CAEQz2Cs4WHPux9LFdHB_E8ajzj-npXzw_FSn9rtL1o%2B%3DUefUQg%40mail.gmail.com</a>.<br> </blockquote></div>