[Freeipa-users] Problem with replication

[Ludwig Krispenz]

Hi,

there seem to be different issues,
- I don't know what the ipactl status is looking for when it generates 
the error message about no matching master,
but I don't think it is related to the retro changelog.

- the retro changelog errors for adding and deleting
-- the add failures are about aborted transactions because a page cannot 
be accessed, this maybe caused by concurrent mods on different backends, 
which want to update teh shared retro cl database.
the changenumber reprted seems to be increasing, one error is about 
changenumber 44975, the next about 45577, so it looks like changes into 
the changelog are written and teh changenumber increases
-- i'm not sure about the delete errors, but normally trimming would go 
on after such an error message, the changenumber attempted to delete are 
increasing.
Could you verify which changes are in the changelog, and if these are 
changing:
ldapsearch -b "cn=changelog" dn

On 05/06/2015 09:52 AM, Łukasz Jaworski wrote:
> Hi,
>
> One of our replica hanged up morning. Error log after dirsrv restart:
> [06/May/2015:09:28:15 +0200] - Retry count exceeded in delete
> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 38376 (rc: 51)
> [06/May/2015:09:28:15 +0200] - Operation error fetching Null DN (6368aeb7-f3c111e4-ae70ce39-9b469c1f), error -30993.
> [06/May/2015:09:28:15 +0200] - dn2entry_ext: Failed to get id for changenumber=44975,cn=changelog from entryrdn index (-30993)
> [06/May/2015:09:28:15 +0200] - Operation error fetching changenumber=44975,cn=changelog (null), error -30993.
> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - replog: an error occured while adding change number 44975, dn = changenumber=44975,cn=changelog: Operations error.
> [06/May/2015:09:28:15 +0200] retrocl-plugin - retrocl_postob: operation failure [1]
> [06/May/2015:09:28:15 +0200] - ldbm_back_seq deadlock retry BAD 1601, err=0 BDB0062 Successful return: 0
> [06/May/2015:09:30:03 +0200] - ldbm_back_seq deadlock retry BAD 1601, err=0 BDB0062 Successful return: 0
> [06/May/2015:09:30:06 +0200] - Retry count exceeded in delete
> [06/May/2015:09:30:06 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39297 (rc: 51)
>
> I did "re-initialize" from other replica.
>
> Now ipactl doesn't work. Shows: Configured hostname 'replica09.local' does not match any master server in LDAP. On lists replica09 is exists (twice)
>
> # ipactl status
> Failed to get list of services to probe status!
> Configured hostname 'replica09.local' does not match any master server in LDAP:
> replica01.local
> replica02.local
> replica03.local
> replica04.local
> replica05.local
> replica06.local
> replica07.local
> replica08.local
> replica09.local
> replica10.local
> replica09.local
>
> After dirsrv stop/start:
>
> In error logs there are many:
> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39290 (rc: 32)
> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39291 (rc: 32)
> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39292 (rc: 32)
> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39293 (rc: 32)
> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39294 (rc: 32)
> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39295 (rc: 32)
> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39296 (rc: 32)
> etc.
>
> [06/May/2015:09:51:08 +0200] - Operation error fetching Null DN (9f51430a-f3c411e4-927ece39-9b469c1f), error -30993.
> [06/May/2015:09:51:08 +0200] - dn2entry_ext: Failed to get id for changenumber=45577,cn=changelog from entryrdn index (-30993)
> [06/May/2015:09:51:08 +0200] - Operation error fetching changenumber=45577,cn=changelog (null), error -30993.
> [06/May/2015:09:51:08 +0200] DSRetroclPlugin - replog: an error occured while adding change number 45577, dn = changenumber=45577,cn=changelog: Operations error.
> [06/May/2015:09:51:08 +0200] retrocl-plugin - retrocl_postob: operation failure [1]
> [06/May/2015:09:51:08 +0200] - ldbm_back_seq deadlock retry BAD 1601, err=0 BDB0062 Successful return: 0
>
> Packages:
> freeipa-server-4.1.3-2.fc21.x86_64
> 389-ds-base-1.3.3.8-1.fc21.x86_64
> 389-ds-base-libs-1.3.3.8-1.fc21.x86_64
>
> Best regards,
> Ender
>