[Freeipa-devel] topology issues
thierry bordaz
tbordaz at redhat.com
Wed Jun 10 12:42:27 UTC 2015
On 06/10/2015 02:19 PM, Ludwig Krispenz wrote:
>
> On 06/10/2015 02:13 PM, thierry bordaz wrote:
>> On 06/10/2015 10:51 AM, Ludwig Krispenz wrote:
>>>
>>> On 06/10/2015 10:41 AM, Martin Basti wrote:
>>>> On 10/06/15 09:13, Ludwig Krispenz wrote:
>>>>> Hi,
>>>>>
>>>>> there seems to be somethin going wrong in the code to delete the
>>>>> services.
>>>>>
>>>>> The code is:
>>>>>
>>>>> # delete master entry with all active services
>>>>> try:
>>>>> dn = DN(('cn', replica), ('cn', 'masters'), ('cn', 'ipa'),
>>>>> ('cn', 'etc'), self.suffix)
>>>>> entries = self.conn.get_entries(dn, ldap.SCOPE_SUBTREE)
>>>>> if entries:
>>>>> entries.sort(key=lambda x: len(x.dn), reverse=True)
>>>>> for entry in entries:
>>>>> self.conn.delete_entry(entry)
>>>>> except errors.NotFound:
>>>>> pass
>>>>> except Exception, e:
>>>>> if not force:
>>>>> raise e
>>>>> elif not err:
>>>>> err = e
>>>>>
>>>>> In the access log we see:
>>>>>
>>>>>
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=52 SRCH
>>>>> base="cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>> scope=2 filter="(objectClass=*)" attrs=ALL
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=52 RESULT err=0 tag=101
>>>>> nentries=8 etime=0 notes=U
>>>>>
>>>>> this was the get_entries, it returns 8 entries
>>>>>
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=53 DEL
>>>>> dn="cn=KDC,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=53 RESULT err=0 tag=107
>>>>> nentries=0 etime=0 csn=5576dceb000600040000
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=54 DEL
>>>>> dn="cn=KPASSWD,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=54 RESULT err=0 tag=107
>>>>> nentries=0 etime=0 csn=5576dceb000700040000
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=55 DEL
>>>>> dn="cn=MEMCACHE,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>> [09/Jun/2015:08:32:43 -0400] conn=150 op=55 RESULT err=0 tag=107
>>>>> nentries=0 etime=1 csn=5576dcec000100040000
>>>>>
>>>>> here it stops after deleting three entries, and it should do it in
>>>>> reverse order of the dn length, but KDC is deleted before MEMCACHE
>>
>> Something surprising is that according to the code
>>
>> # delete master entry with all active services
>> try:
>> dn = DN(('cn', replica), ('cn', 'masters'), ('cn',
>> 'ipa'),
>> ('cn', 'etc'), self.suffix)
>> entries = self.conn.get_entries(dn, ldap.SCOPE_SUBTREE)
>> if entries:
>> entries.sort(key=lambda x: len(x.dn), reverse=True)
>> for entry in entries:
>> self.conn.delete_entry(entry)
>> except errors.NotFound:
>> pass
>> except Exception, e:
>> if not force:
>> raise e
>> elif not err:
>> err = e
>>
>> try:
>> entry = *self.conn.get_entry*(
>> DN(('cn', 'ipa'), ('cn', 'etc'), self.suffix),
>> ['aci'])
>>
>> sub = {'suffix': self.suffix, 'fqdn': replica}
>> ...
>>
>> We should see a search on cn=ipa,cn=etc,SUFFIX following the deletion
>> of those entries.
>> But the next op is an UNBIND.
> yes, that is strange, maybe we hit an exception and the connection was
> closed
With UNBIND logged, it looks like the closure is triggered by the CLI.
I agree it should be some exception but my understanding is that 'force'
was set. so when it started deleting entries any exception is caught and
we should do the following search.
>> Is that the code executed by ipa-replica-manage ?
> I think so, yes.
>>
>>
>>>>> [09/Jun/2015:08:32:43 -0400] conn=150 op=56 UNBIND
>>>>>
>>>>>
>>>>> Are there any ideas what is going on or how to debug it ?
>>>>>
>>>> Actually, the both DNs of KDC and MEMCACHE has the same length.
>>>> IPA implements own DN class, where length is the number of AVA/RDN
>>>> parts (mixed in code, but it means the 'cn=user' has length 1, and
>>>> 'cn=user,cn=accounts' has length 2)
>>>>
>>>> def __len__(self):
>>>> return len(self.rdns)
>>>>
>>>> This reverse sort guarantees the child entries will be removed
>>>> before the parent entries.
>>> thanks, then it is ok, but it does not explain why not all services
>>> and the master were not deleted.
>>>>
>>>> To debug, maybe print the entries from IPA code, before sort and
>>>> after sort might help.
>>> yep, but so far only Oleg reprted this, and he's not here today, I
>>> haven't reproduced the issue
>>>>
>>>> Martin^2
>>>>
>>>>>
>>>>> On 06/09/2015 05:32 PM, Ludwig Krispenz wrote:
>>>>>> Hi Oleg,
>>>>>> thanks for access to your machine, the replication agreements are
>>>>>> still there - and that is expected since the server was not removed.
>>>>>>
>>>>>> In the access log I see:
>>>>>>
>>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=52 SRCH
>>>>>> base="cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>>> scope=2 filter="(objectClass=*)" attrs=ALL
>>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=52 RESULT err=0 tag=101
>>>>>> nentries=8 etime=0 notes=U
>>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=53 DEL
>>>>>> dn="cn=KDC,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=53 RESULT err=0 tag=107
>>>>>> nentries=0 etime=0 csn=5576dceb000600040000
>>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=54 DEL
>>>>>> dn="cn=KPASSWD,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=54 RESULT err=0 tag=107
>>>>>> nentries=0 etime=0 csn=5576dceb000700040000
>>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=55 DEL
>>>>>> dn="cn=MEMCACHE,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>>> [09/Jun/2015:08:32:43 -0400] conn=150 op=55 RESULT err=0 tag=107
>>>>>> nentries=0 etime=1 csn=5576dcec000100040000
>>>>>> [09/Jun/2015:08:32:43 -0400] conn=150 op=56 UNBIND
>>>>>>
>>>>>> the search for cn=f22replica1.bagam.net,cn=masters,.... returns 8
>>>>>> entries, which then should be deleted, but only 3 ae deleted and the
>>>>>> cn=f22replica1.bagam.net,cn=masters,... entry is not deleted, so
>>>>>> the topology segments are not deleted, and the agreement is not
>>>>>> removed.
>>>>>>
>>>>>> I don't know why ipa-replica-manage del does stop deleting
>>>>>> services and the master entry
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 06/09/2015 04:25 PM, Oleg Fayans wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 06/09/2015 04:19 PM, Ludwig Krispenz wrote:
>>>>>>>>
>>>>>>>> On 06/09/2015 04:14 PM, Oleg Fayans wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 06/09/2015 04:04 PM, Ludwig Krispenz wrote:
>>>>>>>>>>
>>>>>>>>>> On 06/09/2015 03:55 PM, Oleg Fayans wrote:
>>>>>>>>>>> Hi everybody,
>>>>>>>>>>>
>>>>>>>>>>> The current status of Topology plugin testing is as follows:
>>>>>>>>>>>
>>>>>>>>>>> 1. There is still no proper way of removing the replica.
>>>>>>>>>>> Standard procedure using `ipa-replica-manage del` throws
>>>>>>>>>>> "Server is unwilling to perform: Entry is managed by
>>>>>>>>>>> topology plugin.Deletion not allowed.".
>>>>>>>>>> yes, that is for the first attempt to directly remove the
>>>>>>>>>> agreement, but when the server is removed the agreements
>>>>>>>>>> should be removed
>>>>>>>>> We should probably think of less threatening error message in
>>>>>>>>> this case. Just from reading the command output one might
>>>>>>>>> conclude that replica removal failed.
>>>>>>>>>>> The replication agreement though does get deleted,
>>>>>>>>>> then it is ok,
>>>>>>>>>>> but the topology information does not get updated.
>>>>>>>>>> what do you mean, where do you check ? in the "remaining"
>>>>>>>>>> topology the shared tree should be updated, for the removed
>>>>>>>>>> replica it will not, but this should be uninstalled anyway
>>>>>>>>> The problem here, is that the topology information does not
>>>>>>>>> get updated on master as well.
>>>>>>>> could you be a bit more precise. what do you still see ? the
>>>>>>>> agreement will be only removed if the segment is removed, and
>>>>>>>> this should be reoplicated to all severs in the remaining
>>>>>>>> topology - if you don't disconnect it by removing the replica.
>>>>>>>> and what was the topology structure and which replica did you
>>>>>>>> remove, on which server did you remove it?
>>>>>>> So, Here is the results of the `topologysegment-find` command
>>>>>>> before replica removal:
>>>>>>> root at f22master:/var/log/dirsrv/slapd-BAGAM-NET]$ ipa
>>>>>>> topologysegment-find
>>>>>>> Suffix name: realm
>>>>>>> ------------------
>>>>>>> 2 segments matched
>>>>>>> ------------------
>>>>>>> Segment name: f22master.bagam.net-to-f22replica1.bagam.net
>>>>>>> Left node: f22master.bagam.net
>>>>>>> Right node: f22replica1.bagam.net
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name: f22master.bagam.net-to-f22replica2.bagam.net
>>>>>>> Left node: f22master.bagam.net
>>>>>>> Right node: f22replica2.bagam.net
>>>>>>> Connectivity: both
>>>>>>> ----------------------------
>>>>>>> Number of entries returned 2
>>>>>>> ----------------------------
>>>>>>> Then, after issuing `ipa-replica-manage-del f2replica1.bagam.net
>>>>>>> --force` on the master, the same command on master still shows
>>>>>>> exactly the same topology:
>>>>>>>
>>>>>>> root at f22master:/var/log/dirsrv/slapd-BAGAM-NET]$ ipa
>>>>>>> topologysegment-find
>>>>>>> Suffix name: realm
>>>>>>> ------------------
>>>>>>> 2 segments matched
>>>>>>> ------------------
>>>>>>> Segment name: f22master.bagam.net-to-f22replica1.bagam.net
>>>>>>> Left node: f22master.bagam.net
>>>>>>> Right node: f22replica1.bagam.net
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name: f22master.bagam.net-to-f22replica2.bagam.net
>>>>>>> Left node: f22master.bagam.net
>>>>>>> Right node: f22replica2.bagam.net
>>>>>>> Connectivity: both
>>>>>>> ----------------------------
>>>>>>> Number of entries returned 2
>>>>>>> ----------------------------
>>>>>>>
>>>>>>>>>>> When I then issue `ipa topologysegment-del`, it fails due to
>>>>>>>>>>> "ipa: ERROR: Server is unwilling to perform: Removal of
>>>>>>>>>>> Segment disconnects topology.Deletion not allowed."
>>>>>>>>>> correct, you can only do it after removal of the server
>>>>>>>>> I do not get it. Master still thinks it has the replica, it
>>>>>>>>> displays it both in CLI using `ipa topologysegment-find` and
>>>>>>>>> in the web-ui. (although it does not show it using `ipa
>>>>>>>>> host-find`, which is correct), and there is no way to manually
>>>>>>>>> make it change it's mind?
>>>>>>>>>>>
>>>>>>>>>>> I tried to disable the segment first and then delete it, but
>>>>>>>>>>> with the segment properly disabled, the attempt to delete it
>>>>>>>>>>> raised a GSS error: "ipa: ERROR: Kerberos error: Kerberos
>>>>>>>>>>> error: ('Unspecified GSS failure. Minor code may provide
>>>>>>>>>>> more information', 851968)/('KDC returned error string:
>>>>>>>>>>> PROCESS_TGS', -1765328324)/". I am not sure, where to search
>>>>>>>>>>> for corresponding logs. The session transcript is attached.
>>>>>>>>>>>
>>>>>>>>>>> 2. The following is probably unrelated to the topology plugin:
>>>>>>>>>>> I installed a replica with --setup-ca option. Then, on this
>>>>>>>>>>> replica tried to prepare another replica:
>>>>>>>>>>> -------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> root at f22replica2:/home/ofayans/f22]$ ipa-replica-prepare
>>>>>>>>>>> --ip-address 192.168.122.141 f22replica3.bagam.net
>>>>>>>>>>> Directory Manager (existing master) password:
>>>>>>>>>>>
>>>>>>>>>>> Preparing replica for f22replica3.bagam.net from
>>>>>>>>>>> f22replica2.bagam.net
>>>>>>>>>>> Creating SSL certificate for the Directory Server
>>>>>>>>>>> Certificate issuance failed
>>>>>>>>>>> -------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> The corresponding line in the dirsrv log:
>>>>>>>>>>> [09/Jun/2015:09:54:46 -0400] - Entry
>>>>>>>>>>> "uid=admin,ou=people,o=ipaca" -- attribute "krbExtraData"
>>>>>>>>>>> not allowed
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Oleg Fayans
>>>>>>>>> Quality Engineer
>>>>>>>>> FreeIPA team
>>>>>>>>> RedHat.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Oleg Fayans
>>>>>>> Quality Engineer
>>>>>>> FreeIPA team
>>>>>>> RedHat.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Martin Basti
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20150610/2aef1eb7/attachment.htm>
More information about the Freeipa-devel
mailing list