[Freeipa-users] Problem with replication

Wed May 6 09:10:26 UTC 2015

Hi,

ipactl stops working after dirsrv-stop/start.

There are many changes in the changelog:
from 39399 to 44397

(…)
# 44393, changelog
dn: changenumber=44393,cn=changelog

# 44394, changelog
dn: changenumber=44394,cn=changelog

# 44395, changelog
dn: changenumber=44395,cn=changelog

# 44396, changelog
dn: changenumber=44396,cn=changelog

# 44397, changelog
dn: changenumber=44397,cn=changelog

# search result
search: 2
result: 11 Administrative limit exceeded

# numResponses: 5001
# numEntries: 5000

After some seconds dirsrv stops responding.

In error log:
[06/May/2015:11:00:04 +0200] agmt="cn=cloneAgreement1-replica09.local-pki-tomcat" (replica08:389) - Can't locate CSN 55100d8c0000069f0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[06/May/2015:11:00:04 +0200] - ldbm_back_seq deadlock retry BAD 1601, err=0 BDB0062 Successful return: 0

ldapsearch hangs. Dirsrv is not responding now.

This replica is on virtual machine (ganeti). We had problems with replication to vm, but after force-sync all was fine. On physical servers all works fine.

Lukasz Jaworski 'Ender'

Wiadomość napisana przez Ludwig Krispenz <lkrispen at redhat.com> w dniu 6 maj 2015, o godz. 10:52:

> Hi,
> 
> there seem to be different issues,
> - I don't know what the ipactl status is looking for when it generates the error message about no matching master,
> but I don't think it is related to the retro changelog.
> 
> - the retro changelog errors for adding and deleting
> -- the add failures are about aborted transactions because a page cannot be accessed, this maybe caused by concurrent mods on different backends, which want to update teh shared retro cl database.
> the changenumber reprted seems to be increasing, one error is about changenumber 44975, the next about 45577, so it looks like changes into the changelog are written and teh changenumber increases
> -- i'm not sure about the delete errors, but normally trimming would go on after such an error message, the changenumber attempted to delete are increasing.
> Could you verify which changes are in the changelog, and if these are changing:
> ldapsearch -b "cn=changelog" dn
> 
> On 05/06/2015 09:52 AM, Łukasz Jaworski wrote:
>> Hi,
>> 
>> One of our replica hanged up morning. Error log after dirsrv restart:
>> [06/May/2015:09:28:15 +0200] - Retry count exceeded in delete
>> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 38376 (rc: 51)
>> [06/May/2015:09:28:15 +0200] - Operation error fetching Null DN (6368aeb7-f3c111e4-ae70ce39-9b469c1f), error -30993.
>> [06/May/2015:09:28:15 +0200] - dn2entry_ext: Failed to get id for changenumber=44975,cn=changelog from entryrdn index (-30993)
>> [06/May/2015:09:28:15 +0200] - Operation error fetching changenumber=44975,cn=changelog (null), error -30993.
>> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - replog: an error occured while adding change number 44975, dn = changenumber=44975,cn=changelog: Operations error.
>> [06/May/2015:09:28:15 +0200] retrocl-plugin - retrocl_postob: operation failure [1]
>> [06/May/2015:09:28:15 +0200] - ldbm_back_seq deadlock retry BAD 1601, err=0 BDB0062 Successful return: 0
>> [06/May/2015:09:30:03 +0200] - ldbm_back_seq deadlock retry BAD 1601, err=0 BDB0062 Successful return: 0
>> [06/May/2015:09:30:06 +0200] - Retry count exceeded in delete
>> [06/May/2015:09:30:06 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39297 (rc: 51)
>> 
>> I did "re-initialize" from other replica.
>> 
>> Now ipactl doesn't work. Shows: Configured hostname 'replica09.local' does not match any master server in LDAP. On lists replica09 is exists (twice)
>> 
>> # ipactl status
>> Failed to get list of services to probe status!
>> Configured hostname 'replica09.local' does not match any master server in LDAP:
>> replica01.local
>> replica02.local
>> replica03.local
>> replica04.local
>> replica05.local
>> replica06.local
>> replica07.local
>> replica08.local
>> replica09.local
>> replica10.local
>> replica09.local
>> 
>> After dirsrv stop/start:
>> 
>> In error logs there are many:
>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39290 (rc: 32)
>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39291 (rc: 32)
>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39292 (rc: 32)
>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39293 (rc: 32)
>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39294 (rc: 32)
>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39295 (rc: 32)
>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39296 (rc: 32)
>> etc.
>> 
>> [06/May/2015:09:51:08 +0200] - Operation error fetching Null DN (9f51430a-f3c411e4-927ece39-9b469c1f), error -30993.
>> [06/May/2015:09:51:08 +0200] - dn2entry_ext: Failed to get id for changenumber=45577,cn=changelog from entryrdn index (-30993)
>> [06/May/2015:09:51:08 +0200] - Operation error fetching changenumber=45577,cn=changelog (null), error -30993.
>> [06/May/2015:09:51:08 +0200] DSRetroclPlugin - replog: an error occured while adding change number 45577, dn = changenumber=45577,cn=changelog: Operations error.
>> [06/May/2015:09:51:08 +0200] retrocl-plugin - retrocl_postob: operation failure [1]
>> [06/May/2015:09:51:08 +0200] - ldbm_back_seq deadlock retry BAD 1601, err=0 BDB0062 Successful return: 0
>> 
>> Packages:
>> freeipa-server-4.1.3-2.fc21.x86_64
>> 389-ds-base-1.3.3.8-1.fc21.x86_64
>> 389-ds-base-libs-1.3.3.8-1.fc21.x86_64
>> 
>> Best regards,
>> Ender
>> 
> 
> -- 
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project