[EnMasse] Enmasse + UnifiedPush + Operator Feedback and request for help

[Summers Pittman]

Sorry Camila, I didn't do a reply-all.
On Tue, Jul 30, 2019 at 6:07 AM Camila Macedo <cmacedo at redhat.com> wrote:

> Hi @Summers,
>
> Regards the items 2 and 3, could we not implement these actions in the UPS
> operators instead of the expected implementation on enmasse?
>

We could but it would take a lot of work and the results would be less
optimal than having those features in enmasse proper.

I've seen a few tutorials online, we'd end up writing a polling cache to
get something love waits, but I don't know how well they would integrate
into the normal client go workflow.

Finalizers are a whole different story that I haven't begin researching the
best way to third party myself into. The picture becomes more muddy because
addresses existing doesn't block deletion of their owners. Addresses stick
around but the ups instances are deleted with out waiting.

> Regards item 4 shows that a new release was made 10 hours ago with this
> fix[1](0.29.0[2]). So, could not it be solved by using this version?
>

No. This pr creates an infinite stream of doomed messages that death star
UPS if there is an exception and the message is retried.



>
> [1] - https://github.com/EnMasseProject/enmasse/issues/2927
[2] - https://github.com/EnMasseProject/enmasse/releases/tag/0.29.0


CAMILA MACEDO

SR. SOFTWARE ENGINEER, RED HAT CLOUD SERVICES

Red Hat UK <https://www.redhat.com/>

IM: cmacedo

Phone: +44 7853500035
<https://red.ht/sig>


On Mon, Jul 29, 2019 at 4:07 PM Summers Pittman <supittma at redhat.com> wrote:

> I'm cross posting this on the AeroGear(
> https://groups.google.com/forum/#!forum/aerogear) and the enmasse (
> https://www.redhat.com/mailman/listinfo/enmasse) mailing lists.
>
> In the past few months we've been adding the ability to use an external
> broker with the Unified Push Server[UPS] (
> https://github.com/aerogear/aerogear-unifiedpush-server).  We've added
> support for external AMQP connections to the UPS container image, as well
> as added support for connecting to enmasse using the UPS operator (
> https://github.com/aerogear/unifiedpush-operator).  Following are our
> experiences with enmasse as well as some problems we encountered.
>
> 1: The creation of enmasse resources using the UPS operator (based on
> client-go https://github.com/kubernetes/client-go) worked pretty much as
> expected.  The only downside was that the golang bindings from enmasse were
> out of date, but the team accepted our PR's and released a new version with
> updated bindings.  Much appreciated!
>
> 2: The enmasse controllers for address, addresspaces, and messagingusers
> do not implement k8s watches, nor do they throw an exception if you try to
> watch a resource.  This causes a lot of errors to be logged in the UPS
> operator as we need to watch and maintain those resources to keep our
> service running.  There is an open enmasse issue here that describes the
> issue : https://github.com/EnMasseProject/enmasse/issues/1280
>
> 3: Enmasse doesn't block owner deletion nor implement a finalizer to
> handle being orphaned.  This means that when our operator deletes a UPS
> server, the addresses and addressspace resources we create don't get
> deleted, nor do we get a notification that they aren't being deleted.
>
> 4: Enmasse has no way to configure delivery failures.  Usually with a
> message broker we want failed messages to be retried, then retried with a
> backoff, and eventually retired to a dead letter queue.  In enmasse the
> default behavior is to retry immediately and infinitely.  There is very
> little we can use the broker for to get an alert that this is happening.
> In the case of a production error this means that enmasse will death star
> our service with an infinite stream of doomed messages.  We are researching
> workarounds, but per this issue (
> https://github.com/EnMasseProject/enmasse/issues/2927) it seems like
> there is little we can do at the level of address resource configuration.
>
> 5: Creation of resources (addresses, addressspaces, messaging users) and
> getting their respective information into our deployment using our operator
> was really straight forward.  Like amazingly simple and directly straight
> forward.
>
> Feel free to reply inline to the points.  Now for my actual question :
>
> We're trying to find workaround to #4 and are actively soliciting ideas.
> Right now we're looking at implementing more aggressive messaging handling
> so messages will always be consumed even if we would prefer them to be
> retried (for instance if the network connection to our push service was
> unavailable for a moment, a credential expired etc).
>
> This doesn't help for unexpected errors in the UPS code (who doesn't love
> a good NPE), and for that we might want to have the UPS operator keep an
> eye out for extreme message re deliveries and auto-manually delete those
> messages, but that would be better handled by a DLQ mechanism.
>
> Thoughts on the potential workarounds?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Aerogear" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to aerogear+unsubscribe at googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/aerogear/CAEQz2CtEDGth1cWj3nKPNmZDBg6cSvLW_0D6nfhgmyX6ZW6t3w%40mail.gmail.com
> <https://groups.google.com/d/msgid/aerogear/CAEQz2CtEDGth1cWj3nKPNmZDBg6cSvLW_0D6nfhgmyX6ZW6t3w%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190730/71dbcbdd/attachment.htm>