[Freeipa-users] General status of my FreeIPA servers - is there a method for cleaning them?

Fri Apr 13 20:41:40 UTC 2012

On 04/13/2012 02:30 PM, Dan Scott wrote:
> On Fri, Apr 13, 2012 at 15:24, Rich Megginson<rmeggins at redhat.com>  wrote:
>> On 04/13/2012 01:03 PM, Dan Scott wrote:
>>>>> If I'm interpreting this correctly, it can't be deleted because it's
>>>>> not a leaf node, but it doesn't have any sub-entries that I can delete
>>>>> first.
>>>> You are correct.  Try this:
>>>>
>>>> ldapsearch -D 'cn=directory manager' -W -v -b
>>>>
>>>> 'cn=fileserver5.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu'
>>>> '(|(objectclass=nstombstone)(objectclass=*))'
>>> Ahh, so there are some 'child' entries:
>>>
> [snip]
>
>>> Is it safe to delete them?
>> Yes.
> I deleted them, but it's still complaining about a non-leaf:
>
> [root at fileserver1 ~]# ldapmodify -f rmfileserver5.ldif -D
> 'cn=directory manager' -W
> Enter LDAP Password:
> deleting entry "cn=fileserver5.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu"
> ldap_delete: Operation not allowed on non-leaf (66)
>
> [root at fileserver1 ~]# ldapsearch -D 'cn=directory manager' -W -b
> 'cn=fileserver5.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu'
> '(|(objectclass=nstombstone)(objectclass=*))'
> Enter LDAP Password:
> # extended LDIF
> #
> # LDAPv3
> # base<cn=fileserver5.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu>
> with scope subtree
> # filter: (|(objectclass=nstombstone)(objectclass=*))
> # requesting: ALL
> #
>
> # fileserver5.ecg.mit.edu, masters, ipa, etc, ecg.mit.edu
> dn: cn=fileserver5.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu
> cn: fileserver5.ecg.mit.edu
> objectClass: top
> objectClass: nsContainer
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 2
> # numEntries: 1
> [root at fileserver1 ~]#

Wow - never seen this one before

>
>>>>>>> I also see
>>>>>>> inconsistent replication states on the servers. i.e. server1 shows
>>>>>>> that it's replicating with server2 but server2 does not show that it's
>>>>>>> replicating with server1.
>>>>>>
>>>>>> Do you have errors in the server2 log showing that it is attempting to
>>>>>> replicate with server1 but failing with some error?
>>>>> [root at fileserver1 ~]# ipa-csreplica-manage list -v
>>>>> fileserver1.ecg.mit.edu
>>>>> Directory Manager password:
>>>>>
>>>>> fileserver2.ecg.mit.edu
>>>>>    last init status: None
>>>>>    last init ended: None
>>>>>    last update status: 0 Replica acquired successfully: Incremental
>>>>> update succeeded
>>>>>    last update ended: 2012-04-13 17:57:39+00:00
>>>>> [root at fileserver1 ~]# ipa-csreplica-manage list -v
>>>>> fileserver2.ecg.mit.edu
>>>>> Directory Manager password:
>>>>>
>>>>> fileserver1.ecg.mit.edu
>>>>>    last init status: None
>>>>>    last init ended: None
>>>>>    last update status: 0 Replica acquired successfully: Incremental
>>>>> update succeeded
>>>>>    last update ended: 2012-04-13 17:57:41+00:00
>>>>> fileserver3.ecg.mit.edu
>>>>>    last init status: None
>>>>>    last init ended: None
>>>>>    last update status: 0 Replica acquired successfully: Incremental
>>>>> update succeeded
>>>>>    last update ended: 2012-04-13 17:57:41+00:00
>>>>> [root at fileserver1 ~]# ipa-csreplica-manage list -v
>>>>> fileserver3.ecg.mit.edu
>>>>> Directory Manager password:
>>>>>
>>>>> fileserver2.ecg.mit.edu
>>>>>    last init status: None
>>>>>    last init ended: None
>>>>>    last update status: 0 Replica acquired successfully: Incremental
>>>>> update succeeded
>>>>>    last update ended: 2012-04-13 17:57:44+00:00
>>>>> fileserver1.ecg.mit.edu
>>>>>    last init status: None
>>>>>    last init ended: None
>>>>>    last update status: 0 Replica acquired successfully: Incremental
>>>>> update succeeded
>>>>>    last update ended: 2012-04-13 17:57:43+00:00
>>>>> [root at fileserver1 ~]#
>>>>>
>>>>> fileserver1's (and fileserver2s) /var/log/dirsrv/slapd-PKI-IPA/errors
>>>>> contains lots of:
>>>>> [13/Apr/2012:13:57:43 -0400] NSMMReplicationPlugin -
>>>>> repl_set_mtn_referrals: could not set referrals for replica o=ipaca:
>>>>> 20
>>>>
>>>> This error usually means a replica was deleted and the RUV needs to be
>>>> cleaned.
>>>> see http://port389.org/wiki/Howto:CLEANRUV
>>>> and
>>>> https://fedorahosted.org/freeipa/ticket/2303
>>>> and
>>>> https://fedorahosted.org/389/ticket/337
>>> OK, I've seen this before - is it important to remove them? I've had
>>> to add and remove replicas so much that I don't really want to do it
>>> unless it's necessary. I'm happy to live with them if it's not a
>>> problem.
>>
>> It's not a problem until it's a problem :-)  I would go ahead and run
>> CLEANRUV.
> I cleaned up a load of these entries, but now I think I've broken the
> replication between fileserver1 and 3:
>
> fileserver1:/var/log/dirsrv/slapd-ECG-MIT-EDU/errors
> [13/Apr/2012:15:57:56 -0400] NSMMReplicationPlugin - changelog program
> - agmt="cn=meTofileserver3.ecg.mit.edu" (fileserver3:389): CSN
> 4f5039960000002b0000 not found, we aren't as up to date, or we purged
> [13/Apr/2012:15:57:56 -0400] NSMMReplicationPlugin -
> agmt="cn=meTofileserver3.ecg.mit.edu" (fileserver3:389): Data required
> to update replica has been purged. The replica must be reinitialized.
> [13/Apr/2012:15:57:56 -0400] NSMMReplicationPlugin -
> agmt="cn=meTofileserver3.ecg.mit.edu" (fileserver3:389): Incremental
> update failed and requires administrator action
>
> fileserver3:/var/log/dirsrv/slapd-ECG-MIT-EDU/errors
> [13/Apr/2012:16:19:38 -0400] NSMMReplicationPlugin - changelog program
> - agmt="cn=meTofileserver1.ecg.mit.edu" (fileserver1:389): CSN
> 4f031e76001d000b0000 not found, we aren't as up to date, or we purged
>
> Is it safe to run:
> [root at fileserver3 ~]# ipa-replica-manage re-initialize --from
> fileserver1.ecg.mit.edu
>
> I want to make sure I get it the correct way round!

Are you sure that fileserver1 has the correct data?

>
>>>>> fileserver3's /var/log/dirsrv/slapd-PKI-IPA/errors contains lots of:
>>>>> [13/Apr/2012:13:52:50 -0400] slapi_ldap_bind - Error: could not send
>>>>> startTLS request: error -1 (Can't contact LDAP server) errno 107
>>>>> (Transport endpoint is not connected)
>>>>
>>>> This is a real connection error - could be cert or hostname lookup
>>>> related.
>>> How do I find out if it's cert or hostname lookup? Which hostname?
>>> Fileserver3 runs DNS, and it seems to be working fine.
>>
>> Try ldapsearch - on server3
>>
>> LDAPTLS_CACERTDIR=/etc/dirsrv/slapd-PKI-IPA ldapsearch -x -ZZ -H
>> ldap://server2.fqdn -D "cn=directory manager" -W -s base -b ""
>>
>> If that works, check to make sure the replication agreement has the correct
>> server2.fqdn
>>
>> If that doesn't work, use ldapsearch -d 1 -x ..... to get further debugging
>> information.
> The replication agreements (according to ipa-replica-manage) all have
> the correct host names - I'm not sure what ldapsearch command to run
> to check the replication agreements.

ipa-replica-manage --list?  or something like that?

>
>>> The /var/log/dirsrv/slapd-ECG-MIT-EDU/errors is
>>> now full of:
>>>
>>> [13/Apr/2012:14:59:19 -0400] NSMMReplicationPlugin - conn=1 op=571
>>> csn=4f70a9e5000100060000: Can't created glue entry
>>> cn=fileserver4.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu
>>> uniqueid=6949d104-775b11e1-abce82a1-a45dd3c3, error 68
>>>
>>> Should I delete the LDAP entry which is trying to replicate
>>> fileserver2 with fileserver4?
>>
>> Yes.  And it may be due to the fact that the entry it is trying to delete
>> has those tombstone children that have to be deleted too.
> OK, I'll see how this goes, once the tombstones are gone.
>
> Thanks,
>
> Dan