[Freeipa-users] be_pam_handler_callback Backend returned: (3, 4, <NULL>) [Internal Error (System error)]

[Ludwig Krispenz]

On 01/18/2017 08:13 AM, Harald Dunkel wrote:
> Hi Ludwig,
>
> On 01/17/17 17:01, Ludwig Krispenz wrote:
>> On 01/17/2017 04:48 PM, Harald Dunkel wrote:
>>> On 01/17/17 16:12, Harald Dunkel wrote:
>>>> On 01/17/17 11:38, Sumit Bose wrote:
>>>>> On Tue, Jan 17, 2017 at 10:44:14AM +0100, Harald Dunkel wrote:
>>>>>> It seems something got corrupted in my ipa setup. I found this in the
>>>>>> sssd log file on Wheezy:
>>>>>>
>>>>>> (Tue Jan 17 10:19:02 2017) [hbac_shost_attrs_to_rule] (0x0400): Processing source hosts for rule [allow_all]
>>>>>> (Tue Jan 17 10:19:02 2017) [hbac_eval_user_element] (0x0080): Parse error on [cn=System: Manage Host Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de]
>>>>> Looks like there was a replication conflict, please see
>>>>> https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html
>>>>> how to resolve it.
>>>>>
>>>> % ldapsearch -D "cn=directory manager" -w secret -b "dc=example,dc=de" "nsds5ReplConflict=*" \* nsds5ReplConflict | grep nsds5ReplConflict | wc -l
>>>> 26
>>>>
>>> PS:
>>>
>>> nsds5ReplConflict: namingConflict cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=domain,cn=topology,cn=ipa,cn=etc,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=locations,cn=etc,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=dns administrators,cn=privileges,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=dns servers,cn=privileges,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=cas,cn=ca,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=custodia,cn=ipa,cn=etc,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=dogtag,cn=custodia,cn=ipa,cn=etc,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=system: add ca,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=system: delete ca,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=system: modify ca,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=system: read cas,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=system: modify dns servers configuration,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=system: read dns servers configuration,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=System: Manage Host Principals,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=System: Add IPA Locations,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=System: Modify IPA Locations,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=System: Read IPA Locations,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=System: Remove IPA Locations,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=System: Read Locations of IPA Servers,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=System: Read Status of Services on IPA Servers,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=System: Manage Service Principals,cn=permissions,cn=pbac,dc=example,dc=de
>>> nsds5ReplConflict: namingConflict cn=System: Manage User Principals,cn=permissions,cn=pbac,dc=example,dc=de
>>>
>>> This looks like a problem of ipa-server-install. These entries were created
>>> in the very first seconds.
>> Conflict entries are created if an entry is added on different servers at the "same time", where same time means it is created on instance x before the add of the entry on instance y was replicated to x. This can happen if you run things in parallel, eg upgrades.
>>
> You mean Freeipa has a race condition? I use tools like clusterssh to
> install or upgrade several hosts in parallel (n <= 49 due to available
> screen and font size). The "same time" is built in.
>
> Of course I understand that Freeipa is a special case, because it is
> network application, but it should be able to handle n = 2.
it is handling it (389-ds does), but the way it is handling it has the 
conflict entries as a result.

If you add an entry to an instance it is assigned a unique identifier, 
the nsuniqeuid. The dn of an entry can change by modrdn, but the 
nsuniqueid is not changed. The nsuniqueid is also used in replication to 
identify which is the entry the replicated op has to be applied to.If 
you add entries with the same dn in parallel you have entries with the 
same dn but different nsuniqueids, this cannot be prevented, on instance 
has no control of what happens on another instance.
But if these entries are replicated you have two entries with the same 
dn but different nsuniqueids on the same server. This violates the ldap 
data model and ds needs to handle it. The way we do it is to transform 
one of the entries (always the same on all instances) into a conflict entry.

There have been several proposal to change this and there is work in 
progress to  make the conflict enties invisible (only visible on 
explicite request) to client operations. But this is no trivial task, 
the main problem is that you can not only add entries with the same dn 
in parallel, but you can also add children to these entries, or rename 
them, or delete them before they are replicated and then, if you are 
hiding them entries will have no visible parent or there will be 
problems in deletion.
I definitely want to resolve this, but all proposed and tried solutions 
ended up in some unresolved issues, corner case but still existent.

>
>> There is no simple way to get rid of them, you need to delete them one by one, so do:
>> ldapmodify .......
>> dn:  cn=System: Manage Host Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de
>> changetype: delete
>>
>> for all of your conflict entries
> I am surely no specialist for ldap, so hopefully its allowed to ask
> a question:
>
> This is a tree-like structure. If I delete a conflicting node, what
> happens to the leafs? Is there any indication that these leafs
> contain information that is not needed anymore? Isn't it possible
> that server b created a huge tree with tons of subnodes and leafs
> before the conflict is detected?
if you try to delete a node which has children the deletion will be 
rejected. The bad thing is that you could have conflict entries with 
children. In that case you needd to decide if you want to delete them or 
move them to the "valid" entries before deleting the conflict entry
>
>
>
> Every helpful comment is highly appreciated
> Harri
>

-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander