[Freeipa-users] replication again :-(

[Janelle]

On 5/19/15 12:17 AM, Ludwig Krispenz wrote:
>
> On 05/19/2015 08:58 AM, thierry bordaz wrote:
>> On 05/19/2015 07:47 AM, Martin Kosek wrote:
>>> On 05/19/2015 03:23 AM, Janelle wrote:
>>>> Once again, replication/sync has been lost. I really wish the 
>>>> product was more
>>>> stable, it is so much potential and yet.
>>>>
>>>> Servers running for 6 days no issues. No new accounts or changes 
>>>> (maybe a few
>>>> users changing passwords) and again, 5 out of 16 servers are no 
>>>> longer in sync.
>>>>
>>>> I can test it easily by adding an account and then waiting a few 
>>>> minutes, then
>>>> run "ipa  user-show --all username" on all the servers, and only a 
>>>> few of them
>>>> have the account.  I have now waited 15 minutes, still no luck.
>>>>
>>>> Oh well.. I guess I will go look at alternatives. I had such high 
>>>> hopes for
>>>> this tool. Thanks so much everyone for all your help in trying to 
>>>> get things
>>>> stable, but for whatever reason, there is a random loss of sync 
>>>> among the
>>>> servers and obviously this is not acceptable.
>>>
>>> Hello Janelle,
>>>
>>> I am very sorry to hear about your troubles. Would you be still OK 
>>> with helping us (mostly Ludwig and Thierry) investigate what is the 
>>> root cause of the instability of the replication agreements? This is 
>>> obviously something that should not be happening at this rate as in 
>>> your deployment, so I would really like to be able to identity and 
>>> fix this issue in the 389 DS.
>> Hello Janelle,
>>
>> I can only join my voice to Martin to say how I am sorry to read this.
>> Would you turn on replication logging level (8192) on the 
>> master/consumer and provide us the logs(access/error) and config 
>> (dse.ldif).
>> The master is the instance where you can see the update and the that 
>> is linked (replica agreement) to a replica(aka consumer) where the 
>> update is not received.
> what puzzles me most, is that replication is working for quite some 
> time and then breaks, so we need to find out about the dynamics which 
> lead to that state. You reported errors about invalid credentials or 
> about a bind dn entry not found, these credentials don't get changed 
> by ds or entries are not deleted by ds, so what triggers these changes.
> also for the suggestion by Thierry to debug, we need to determine 
> where replication breaks, if you add an account and it is propageted 
> to some servers and not to others, where does it stop ? This depends 
> on your replication topology, you said in anotehr post that you have a 
> ring topology, does it mean all 16 servers are conencted in a ring 
> only, and if two links break the topology is disconnected ?
>>
>> thanks
>> thierry
>
Let me see about getting some debug logs going to provide more info.  As 
for topology -- yes, ring, but also within the DC - the 3 servers are 
connected in an internal ring. There have been no outages on the WAN 
connections, as I have logs showing network data at all times, so this 
is not an issue. If I did lose a WAN, dozens of other inter-DC apps 
would blow up too, and they have not.

However, I guess you are right, I have not provided enough logging data 
to help diagnose this. Let me see what I can do.  Not sure if this helps 
-- I do try and do all updates from a single master, never from 
different ones. Users are also forced to the same master to change 
passwords and update things. So the "source" of changes is always the same.

Time to go do some log enabling...

~J

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20150519/3fddafaf/attachment.htm>