[katello-devel] Organization deletion bug, orchestration, testing

[Dmitri Dolguikh]

On 17/12/12 11:36 AM, Lukas Zapletal wrote:
> On Mon, Dec 17, 2012 at 10:52:33AM +0100, Martin Bacovsky wrote:
>> to handle communication between services. Today I came across
>> article about messaging [1] and to me it seems a way to go. I know
>> it would be a *big* change, but I think it is worth to at least
>> consider it, especially when we have messaging broker (QPID) already
>> set up in katello.
> Well, messaging is not the only way to orchestrate things, but it looks
> like a good option since developers trying to achieve orchestration tend
> to write state machines or queue-based processors.
>
> I see the biggest advantage in easy de-coupling. We need to get
> orchestration out of the model, out of Katello completely. With
> messaging between Katello and OrchestrationNG, it will be possible to
> run many lightweight Ruby processes/threads for orchestration without
> Rails in the memory.
>
> Having standalone broker also allows services to communicate directly to
> each other - for request-reply read-only lookups.
>
>> - consistency - support for transactions, proven messaging patterns
>> would help us to keep data consistent across subsystems. Like DB
>> server is good at keeping data healthy (FKs, constraints,
>> transactions), similarly  message broker is good at keeping
>> communication between system parts healthy. (lets learn from our
>> previous mistakes)
> While messaging systems (including AMQP-based) supports both local and
> distributed transactions, I'd rather stick with transaction-compensation
> support. We would need to have distributed transaction support to be
> implemented in all three backend engines which is not feasible right
> now. Transaction compensation can be easier to implement if we strictly
> divide messaging exchanges.
>
> There are many other advantages like performance, durability, better
> error recovery and other things. The ultimate goal should be to drop
> REST via HTTP and integrate using REST via AMQP. But for the first phase,
> Katello still can have simple bridge between AMQP and HTTP for each
> backend engine.
>

I agree with the general sentiment expressed so far - I think we need to 
revise our approach to orchestration.

Let's avoid architectural discussions in this thread, but rather 
concentrate on what we need from the new engine, stuff like:

Crash-only semantics (this is pretty big - as things stand right now we 
won't recover from any crash should it happen in the middle of an 
orchestration)
Simpler definition of orchestrations
Ability to orchestrate any operation, whether it includes AR model or not
Ability to recover from certain types of failures (500, 503 type of errors)
Easier testability of orchestrations

As a side note, I think we need to dispose of the idea of synchronous 
orchestrations altogether (it makes error recovery possible without 
breaking the semantics of the call among other things), and aim for 
eventual consistency between Katello (primary) and its subsystems.

-d