[Freeipa-users] replication again :-(

[Rob Crittenden]

Janelle wrote:
> On 5/21/15 6:46 AM, Ludwig Krispenz wrote:
>>
>> On 05/21/2015 03:28 PM, Janelle wrote:
>>> I think I found the problem.
>>>
>>> There was a lone replica running in another DC. It was installed as a
>>> replica some time ago with all the others.  Think of this -- the
>>> original config had 5 servers, one of them was this server. Then the
>>> other 4 servers were RE-BUILT from scratch, so all the replication
>>> agreements were changed AND - this is the important part - the 5th
>>> server was never added back in. BUT - the 5th server was left running
>>> and never told it that it was not a member anymore. It still thought
>>> it had a replication agreement with original "server 1", but server 1
>>> knew otherwise.
>>>
>>> Now, although the first 4 servers were rebuilt, the same domain,
>>> realm, AND passwords were used.
>>>
>>> I am guessing that somehow, this 5th server keeps trying to interject
>>> its info into the ring of 4 servers, kind of forcing its way in.
>>> Somehow, because the original credentials still work (but certs are
>>> all different) is leaving the first 4 servers with a "can't decode"
>>> issue.
>>>
>>> There should be some security checks so this can't happen. It should
>>> also be easy to replicate.
>>>
>>> Now I have to go re-initialize all the servers from a good server, so
>>> everyone is happy again. The "problem" server has been shutdown
>>> completely. (and yes, there were actually 3 of them in my scenario -
>>> I just used 1 to simplify my example - but that explains the 3 CSNs
>>> that just kept "appearing")
>>>
>>> What concerns me most about this - were the servers outside of the
>>> "good ring" somehow able to inject data into replication which might
>>> have been causing bad data??? This is bad if it is true.
>> it depends a bit on what you mean by rebuilt from scratch.
>> A replication session needs to meet three conditions to be able to
>> send data:
>> - the supplier side needs to be able to authenticate and the
>> authenticated users has to be in the list of binddns of the replica
>> -  the data generation of supplier and consumer side need to be the
>> same (they all have to have the same common origin)
>> - the supplier needs to have the changes (CSNs) to be able to position
>> in its changelog to send updates
>>
>> now if you have 5 servers, forget about one of them and do not change
>> the credentials in the others and do not reinitialize the database by
>> an ldif import to generate a new database generation, the fifth server
>> will still be able to connect and eventually send updates - how should
>> the other servers know that this one is no longer a "good" one
>>>
>>> ~Janelle
>>>
>>
> To clarify "rebuilt from scratch":
>
> ipa-server-install --uninstall --unattended
> yum remove 389-ds-* certmonger-* freeipa-*
> reboot
> yum install freeipa-server
> ipa-server-install
>
> The reason for the uninstalls is to clear out all "leftover" folders and
> certs.
> Also run are:
> rm -f /var/lib/sss/db/*
> rm -rf /etc/ipa
> rm -rf /var/log/dirsrv/
>
> But those are before the re-install.
>
> So the 5th server was left running, and yes, the "admin" username and PW
> are the same, so that may be how it is happening, regardless of
> certificates being different.

The certs don't matter as replication doesn't use TLS and doesn't use 
simple bind.

It uses GSSAPI, so any connection still should have failed because the 
keys wouldn't match, if they existed at all. So I'd look for BIND from 
this master in the access logs of the re-installed ones.

rob