[Freeipa-users] Stock with a Master in read-only mode

Mon May 26 08:22:27 UTC 2014

On 05/25/2014 09:44 PM, Davis Goodman wrote:
> On Wed, May 21, 2014 at 12:06 PM, Martin Kosek <mkosek at redhat.com> wrote:
> 
>> On 05/21/2014 01:31 PM, Davis Goodman wrote:
>>>
>>>
>>>
>>>
>>> <http://www.digital-district.ca/>
>>>
>>> On May 21, 2014, at 6:54 , Martin Kosek <mkosek at redhat.com
>>> <mailto:mkosek at redhat.com>> wrote:
>>>
>>>> On 05/21/2014 09:12 AM, Davis Goodman wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On May 21, 2014, at 2:45 , Martin Kosek <mkosek at redhat.com
>>>>> <mailto:mkosek at redhat.com>> wrote:
>>>>>
>>>>>> On 05/21/2014 08:36 AM, Davis Goodman wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Lately I’ve been having issues of replication between my server and
>> my 2
>>>>>>> replicas.
>>>>>>>
>>>>>>> I decided I was going to delete my 2 replicas and start over keeping
>> my
>>>>>>> master intact.
>>>>>>>
>>>>>>> I wasn`t successfull in getting all 3 servers to replicate to each
>> other. (
>>>>>>> it used to work)
>>>>>>>
>>>>>>> I tried deleting  1 replica after the other one  to always keep one
>> of the
>>>>>>> two available.
>>>>>>>
>>>>>>> I had to delete manually the replica host on the master with a bunch
>> of
>>>>>>> ldapdelete command which worked fine.
>>>>>>>
>>>>>>> But after many unsuccessful trials of getting everyone to sync I
>> decided to
>>>>>>> delete my two replicas.
>>>>>>>
>>>>>>> I went back to my master to use the ldapdelete to remove both host`s
>>>>>>> records so that I could start over.
>>>>>>>
>>>>>>> Unfortunately now I’m getting this error.
>>>>>>>
>>>>>>> ldapdelete -x -D "cn=Directory Manager" -W
>>>>>>>   cn=DNS,cn=freeipa02.mtl.domain.int
>> ,cn=masters,cn=ipa,cn=etc,dc=domain,dc=int
>>>>>>> Enter LDAP Password:
>>>>>>> ldap_delete: Server is unwilling to perform (53)
>>>>>>> additional info: database is read-only
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I’m kinda stuck now with no replicas and no DNS. I could restore the
>> backup
>>>>>>> prior to the start of the operation but with a master in read-only
>> mode it
>>>>>>> wouldn’t of much help.
>>>>>>>
>>>>>>> Any insights would be more than welcome.
>>>>>>>
>>>>>>>
>>>>>>> Davis
>>>>>>
>>>>>> Hi Davis, did maybe some of your ipa-replica-manage crashed in a
>> middle of an
>>>>>> operation or an upgrade was interrupted  and left the database put in
>> read only
>>>>>> mode?
>>>>>>
>>>>>> You can find out with this ldapsearch:
>>>>>>
>>>>>> ldapsearch -h `hostname` -D "cn=Directory Manager" -x -w kokos123 -b
>>>>>> 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config' -s base
>>>>>>
>>>>>> Check for nsslapd-readonly, it should be put to "off" in normal
>> operation.
>>>>>>
>>>>>> Martin
>>>>> Ok finally managed to modify the read-only flag.
>>>>>
>>>>> Could prepare my replicas and get them going.
>>>>>
>>>>> Everything seems fine but I’m getting this error while setting up the
>>>>> replicas. Should I be concerned about this one:
>>>>>
>>>>> Update in progress
>>>>> Update in progress
>>>>> Update in progress
>>>>> Update in progress
>>>>> Update in progress
>>>>> Update in progress
>>>>> Update succeeded
>>>>>  [23/31]: adding replication acis
>>>>>  [24/31]: setting Auto Member configuration
>>>>>  [25/31]: enabling S4U2Proxy delegation
>>>>> ipa         : CRITICAL Failed to load replica-s4u2proxy.ldif: Command
>>>>> '/usr/bin/ldapmodify -v -f /tmp/tmplpfMNG -H
>>>>> ldap://freeipa02.mtl.ddistrict.int:389 -x -D cn=Directory Manager -y
>>>>> /tmp/tmp4Svn9k' returned non-zero exit status 20
>>>>>  [26/31]: initializing group membership
>>>>>  [27/31]: adding master entry
>>>>>  [28/31]: configuring Posix uid/gid generation
>>>>>
>>>>>
>>>>>
>>>>> the rest seems to work fine.
>>>>
>>>> You need to check ipareplica-install.log to see the real error.
>>>>
>>>> I wonder if "cn=ipa-http-delegation,cn=s4u2proxy,cn=etc,YOUR-SUFFIX" and
>>>> "cn=ipa-ldap-delegation,cn=s4u2proxy,cn=etc,YOUR-SUFFIX" exist.
>>>>
>>>> Martin
>>>>
>>>
>>> The first one is there:
>>>
>>> ldapsearch -D "cn=Directory Manager” -W -LLL -x -b
>>> cn=ipa-http-delegation,cn=s4u2proxy,cn=etc,dc=ddistrict,dc=int""
>>> dn: cn=ipa-http-delegation,cn=s4u2proxy,cn=etc,dc=ddistrict,dc=int
>>> ipaAllowedTarget:
>> cn=ipa-ldap-delegation-targets,cn=s4u2proxy,cn=etc,dc=ddistr
>>>   ict,dc=int
>>> ipaAllowedTarget:
>> cn=ipa-cifs-delegation-targets,cn=s4u2proxy,cn=etc,dc=ddistr
>>>   ict,dc=int
>>> memberPrincipal: HTTP/freeipa01.prs.ddistrict.int at DDISTRICT.INT
>>> <mailto:HTTP/freeipa01.prs.ddistrict.int at DDISTRICT.INT>
>>> memberPrincipal: HTTP/freeipa02.prs.ddistrict.int at DDISTRICT.INT
>>> <mailto:HTTP/freeipa02.prs.ddistrict.int at DDISTRICT.INT>
>>> memberPrincipal: HTTP/freeipa02.mtl.ddistrict.int at DDISTRICT.INT
>>> <mailto:HTTP/freeipa02.mtl.ddistrict.int at DDISTRICT.INT>
>>> memberPrincipal: HTTP/freeipa01.chr.ddistrict.int at DDISTRICT.INT
>>> <mailto:HTTP/freeipa01.chr.ddistrict.int at DDISTRICT.INT>
>>> memberPrincipal: HTTP/freeipa01.bxl.ddistrict.int at DDISTRICT.INT
>>> <mailto:HTTP/freeipa01.bxl.ddistrict.int at DDISTRICT.INT>
>>> memberPrincipal: HTTP/freeipa01.mtl.ddistrict.int at DDISTRICT.INT
>>> <mailto:HTTP/freeipa01.mtl.ddistrict.int at DDISTRICT.INT>
>>> cn: ipa-http-delegation
>>> objectClass: ipaKrb5DelegationACL
>>> objectClass: groupOfPrincipals
>>> objectClass: top
>>>
>>>
>>> But not the second one:
>>>
>>> ldapsearch -D "cn=Directory Manager” -W -LLL -x -b
>>> cn=ipa-ldap-delegation,cn=s4u2proxy,cn=etc,dc=ddistrict,dc=int""
>>> No such object (32)
>>> Matched DN: cn=s4u2proxy,cn=etc,dc=ddistrict,dc=int
>>>
>>>
>>> Also what is strange is that I got the error only on one of the
>> replicas, the
>>> other one went through without any hiccups.
>>
>> Ok, I think I misguided you with the second DN, the real DN should be
>> "cn=ipa-ldap-delegation-targets,cn=s4u2proxy,cn=etc,dc=ddistrict,dc=int",
>> see
>> /usr/share/ipa/replica-s4u2proxy.ldif for the LDIF that is being loaded.
>>
>> The key here is to check the error message of ldapmodify that was run on
>> the
>> failing replica, try to search in /var/log/ipareplica-install.log.
>>
>> Martin
>>
> 
> Hi Martin,
> 
> Finally got back on this problem.
> 
> I seem to have a huge mess in my replication agreements between my servers.
> if I run the "ipa-replica-manage list-ruv on my master which is
> freeipa01.prs,
> 
> I get this:
> [root at freeipa01 ~]# ipa-replica-manage list-ruv
> freeipa01.prs.ddistrict.int:389: 4
> freeipa01.mtl.ddistrict.int:389: 16
> freeipa01.mtl.ddistrict.int:389: 13
> freeipa01.mtl.ddistrict.int:389: 12
> freeipa01.bxl.ddistrict.int:389: 10
> freeipa01.chr.ddistrict.int:389: 8
> freeipa01.mtl.ddistrict.int:389: 6
> freeipa02.prs.ddistrict.int:389: 3
> freeipa01.chr.ddistrict.int:389: 9
> freeipa02.mtl.ddistrict.int:389: 17
> freeipa02.mtl.ddistrict.int:389: 7
> freeipa02.mtl.ddistrict.int:389: 11
> freeipa02.mtl.ddistrict.int:389: 14
> freeipa02.mtl.ddistrict.int:389: 15
> [root at freeipa01 ~]#
> 
> 
> I've tried to do the ipa-replica-manage clean-ruv on all ID's relating to
> freeipa02.mtl which is the one I'm having the most problems with and would
> like to start from scratch.
> 
> running the ipa-replica-manage list-clean-ruv gives me this:
> 
> [root at freeipa01 slapd-DDISTRICT-INT]# ipa-replica-manage list-clean-ruv
> CLEANALLRUV tasks
> RID 11: Not all replicas online, retrying in 160 seconds...
> RID 17: Not all replicas online, retrying in 640 seconds...
> RID 7: Waiting to process all the updates from the deleted replica...
> 
> No abort CLEANALLRUV tasks running
> [root at freeipa01 slapd-DDISTRICT-INT]#
> 
> I'm kinda stuck in a loop and not sure which way to go.

Check "ipa-replica-manage list" - some of the replicas listed here are not
active. You may have uninstalled a replica which is still pointed in this list.

I think /var/log/dirsrv/slapd-YOUR-REALM/errors contain additional information
which replica is really not accessible.

> 
> I'm also stuck with a orphaned user in the WebUI which I see but can not
> delete, giving me the user doesn't exist.
> 
> If I do an ldapsearch it seems incomplete:
> [root at freeipa01 ~]# ldapsearch -xLLL -D "cn=directory manager" -w XXXXXXX
> -b dc=ddistrict,dc=int | grep -i arobitaille
> dn: cn=arobitaille,cn=groups,cn=compat,dc=ddistrict,dc=int
> cn: arobitaille
> memberUid: arobitaille
> dn: uid=arobitaille,cn=users,cn=compat,dc=ddistrict,dc=int
> homeDirectory: /home/arobitaille
> uid: arobitaille
> member:
> nsuniqueid=08165a01-dd3311e3-8982f534-a4dfcf64+uid=arobitaille,cn=user
> member:
> nsuniqueid=08165a01-dd3311e3-8982f534-a4dfcf64+uid=arobitaille,cn=user
> dn:
> nsuniqueid=08165a01-dd3311e3-8982f534-a4dfcf64+uid=arobitaille,cn=users,cn
> homeDirectory: /home/arobitaille
> mepManagedEntry: cn=arobitaille,cn=groups,cn=accounts,dc=ddistrict,dc=int
> mail: arobitaille at digital-district.ca
> krbPrincipalName: arobitaille at DDISTRICT.INT
> uid: arobitaille
> dn:
> cn=arobitaille+nsuniqueid=08165a02-dd3311e3-8982f534-a4dfcf64,cn=groups,cn
> cn: arobitaille
> description: User private group for arobitaille
> mepManagedBy: uid=arobitaille,cn=users,cn=accounts,dc=ddistrict,dc=int

This is a Directory Server replication conflict entry (notice the
nsuniqueid=08165a02-dd3311e3-8982f534-a4dfcf64 part), FreeIPA cannot manipulate
those. You can try deleting this record with ldapdelete utility or any LDAP gui
of choice.

Martin