[Freeipa-devel] [PATCH 0064] Create ipa-otp-decrement 389DS plugin

[Nathaniel McCallum]

On Wed, 2014-10-08 at 13:53 -0600, Rich Megginson wrote:
> On 10/08/2014 01:45 PM, thierry bordaz wrote:
> > On 10/08/2014 07:30 PM, Nathaniel McCallum wrote:
> >> On Wed, 2014-10-08 at 17:30 +0200, thierry bordaz wrote:
> >>> On 10/07/2014 06:00 PM, Nathaniel McCallum wrote:
> >>>> Attached is the latest patch. I believe this includes all of our
> >>>> discussions up until this point. However, a few bits of additional
> >>>> information are needed.
> >>>>
> >>>> First, I have renamed the plugin to ipa-otp-counter. I believe all
> >>>> replay prevention work can land inside this plugin, so the name is
> >>>> appropriate.
> >>>>
> >>>> Second, I uncovered a bug in 389 which prevents me from validating the
> >>>> non-replication request in bepre. This is the reason for the 
> >>>> additional
> >>>> betxnpre callback. If the upstream 389 bug is fixed, we can merge this
> >>>> check back into bepre. https://fedorahosted.org/389/ticket/47919
> >>>>
> >>>> Third, I believe we are now handling replication correct. An error is
> >>>> never returned. When a replication would cause the counter to 
> >>>> decrease,
> >>>> we remove all counter/watermark related mods from the operation. This
> >>>> will allow the replication to apply without decrementing the value.
> >>>> There is also a new bepost method which check to see if the 
> >>>> replication
> >>>> was discarded (via CSN) while having a higher counter value. If so, we
> >>>> apply the higher counter value.
> >>> For me the code is good. It took me some time to understand the benefit
> >>> of removing mods in preop.
> >>> In fact I think it is a good idea, as it prevents extra repair ops and
> >>> also make more easy the computation of the value to set in repair mod.
> >>>> Here is the scenario. Server X receives two quick authentications;
> >>>> replications A and B are sent to server Y. Before server Y can process
> >>>> server X's replications, an authentication is performed on server Y;
> >>>> replication C is sent to server X. The following logic holds true:
> >>>>    * csnA < csnB < csnC
> >>>>    * valueA = valueC, valueB > valueC
> >>>>
> >>>> When server X receives replication C, ipa-otp-counter will strip 
> >>>> out all
> >>>> counter mod operations; applying the update but not the lower 
> >>>> value. The
> >>>> value of replication B is retained. This is the correct behavior.
> >>>>
> >>>> When server Y processes replications A and B, ipa-otp-counter will
> >>>> detect that a higher value has a lower CSN and will manually set the
> >>>> higher value (in bepost). This initiates replication D, which is 
> >>>> sent to
> >>>> server X. Here is the logic:
> >>>>    * csnA < csnB < csnC < csnD
> >>>>    * valueA = valueC, valueB = valueD, valueD > valueC
> >>>>
> >>>> Server X receives replication D. D has the highest CSN. It has the 
> >>>> same
> >>>> value as replication B (server X's current value). Because the values
> >>>> are the same, ipa-otp-counter will strip all counter mod operations.
> >>>> This reduces counter write contention which might become a problem in
> >>>> N-way replication when N>2.
> >>> I think we should rather let the mods going on. So  that the full
> >>> topology will have
> >>> valueD (or valueB)/csnD rather having a set of servers having
> >>> valueD/csnB and an other set valueD/csnD.
> >> I think you misunderstand. The value for csnD is only discarded when the
> >> server already has valueB (valueB == valueD). Only the value is
> >> discarded, so csnD is still applied. The full topology will have either
> >> valueB w/ csnD or valueD w/ csnD. Since, valueB always equals valueD, by
> >> substitution, all servers have valueD w/ csnD.
> >>
> >> Nathaniel
> >>
> >
> > There are several parts where the CSN are stored.
> > One is used to allow replication protocol to send the approriate 
> > updates. This part is stored into a dedicated entry: RUV.
> > In fact when the update valudD/CSND will be received and applied, the 
> > RUV will be updated with csnD.
> >
> > An other part is the attribute/attribute values. An attribute value 
> > contains the actual value and the CSN associated to that value.
> > This CSN is updated by entry_apply_mod_wsi when it decides which value 
> > to keep and which CSN is associated to this value.
> >
> > In the example above, on the server X, the counter attribute has 
> > valueB/csnB. Then it receives the update ValueD/csnD it discard this 
> > update because valueD=ValueB. That means that on serverX we will have 
> > valueB/csnB.
> >
> > Now if on an other server we receive the updates in the reverse order: 
> > valueD/csnD first then valueB/csnB.
> > This server will apply and valueD/csnD then will discard valueB/csnB.
> >
> > ValueD and ValueB being identical it is not a big issue. But we will 
> > have some server with csnD and others with csnB.
> 
> The CSN is also the key in the changelog database.

Right. We *never* discard a replication operation. We only discard some
of its mods if and only if those mods would result in either no change
at all or an illegal change. If an illegal change would occur, we also
issue a new fixup replication request so that everyone quickly gets
consistency.

Nathaniel