[Freeipa-devel] topology issues
Ludwig Krispenz
lkrispen at redhat.com
Wed Jun 10 12:19:22 UTC 2015
On 06/10/2015 02:13 PM, thierry bordaz wrote:
> On 06/10/2015 10:51 AM, Ludwig Krispenz wrote:
>>
>> On 06/10/2015 10:41 AM, Martin Basti wrote:
>>> On 10/06/15 09:13, Ludwig Krispenz wrote:
>>>> Hi,
>>>>
>>>> there seems to be somethin going wrong in the code to delete the
>>>> services.
>>>>
>>>> The code is:
>>>>
>>>> # delete master entry with all active services
>>>> try:
>>>> dn = DN(('cn', replica), ('cn', 'masters'), ('cn', 'ipa'),
>>>> ('cn', 'etc'), self.suffix)
>>>> entries = self.conn.get_entries(dn, ldap.SCOPE_SUBTREE)
>>>> if entries:
>>>> entries.sort(key=lambda x: len(x.dn), reverse=True)
>>>> for entry in entries:
>>>> self.conn.delete_entry(entry)
>>>> except errors.NotFound:
>>>> pass
>>>> except Exception, e:
>>>> if not force:
>>>> raise e
>>>> elif not err:
>>>> err = e
>>>>
>>>> In the access log we see:
>>>>
>>>>
>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=52 SRCH
>>>> base="cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>> scope=2 filter="(objectClass=*)" attrs=ALL
>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=52 RESULT err=0 tag=101
>>>> nentries=8 etime=0 notes=U
>>>>
>>>> this was the get_entries, it returns 8 entries
>>>>
>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=53 DEL
>>>> dn="cn=KDC,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=53 RESULT err=0 tag=107
>>>> nentries=0 etime=0 csn=5576dceb000600040000
>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=54 DEL
>>>> dn="cn=KPASSWD,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=54 RESULT err=0 tag=107
>>>> nentries=0 etime=0 csn=5576dceb000700040000
>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=55 DEL
>>>> dn="cn=MEMCACHE,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>> [09/Jun/2015:08:32:43 -0400] conn=150 op=55 RESULT err=0 tag=107
>>>> nentries=0 etime=1 csn=5576dcec000100040000
>>>>
>>>> here it stops after deleting three entries, and it should do it in
>>>> reverse order of the dn length, but KDC is deleted before MEMCACHE
>
> Something surprising is that according to the code
>
> # delete master entry with all active services
> try:
> dn = DN(('cn', replica), ('cn', 'masters'), ('cn', 'ipa'),
> ('cn', 'etc'), self.suffix)
> entries = self.conn.get_entries(dn, ldap.SCOPE_SUBTREE)
> if entries:
> entries.sort(key=lambda x: len(x.dn), reverse=True)
> for entry in entries:
> self.conn.delete_entry(entry)
> except errors.NotFound:
> pass
> except Exception, e:
> if not force:
> raise e
> elif not err:
> err = e
>
> try:
> entry = *self.conn.get_entry*(
> DN(('cn', 'ipa'), ('cn', 'etc'), self.suffix),
> ['aci'])
>
> sub = {'suffix': self.suffix, 'fqdn': replica}
> ...
>
> We should see a search on cn=ipa,cn=etc,SUFFIX following the deletion
> of those entries.
> But the next op is an UNBIND.
yes, that is strange, maybe we hit an exception and the connection was
closed
> Is that the code executed by ipa-replica-manage ?
I think so, yes.
>
>
>>>> [09/Jun/2015:08:32:43 -0400] conn=150 op=56 UNBIND
>>>>
>>>>
>>>> Are there any ideas what is going on or how to debug it ?
>>>>
>>> Actually, the both DNs of KDC and MEMCACHE has the same length.
>>> IPA implements own DN class, where length is the number of AVA/RDN
>>> parts (mixed in code, but it means the 'cn=user' has length 1, and
>>> 'cn=user,cn=accounts' has length 2)
>>>
>>> def __len__(self):
>>> return len(self.rdns)
>>>
>>> This reverse sort guarantees the child entries will be removed
>>> before the parent entries.
>> thanks, then it is ok, but it does not explain why not all services
>> and the master were not deleted.
>>>
>>> To debug, maybe print the entries from IPA code, before sort and
>>> after sort might help.
>> yep, but so far only Oleg reprted this, and he's not here today, I
>> haven't reproduced the issue
>>>
>>> Martin^2
>>>
>>>>
>>>> On 06/09/2015 05:32 PM, Ludwig Krispenz wrote:
>>>>> Hi Oleg,
>>>>> thanks for access to your machine, the replication agreements are
>>>>> still there - and that is expected since the server was not removed.
>>>>>
>>>>> In the access log I see:
>>>>>
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=52 SRCH
>>>>> base="cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>> scope=2 filter="(objectClass=*)" attrs=ALL
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=52 RESULT err=0 tag=101
>>>>> nentries=8 etime=0 notes=U
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=53 DEL
>>>>> dn="cn=KDC,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=53 RESULT err=0 tag=107
>>>>> nentries=0 etime=0 csn=5576dceb000600040000
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=54 DEL
>>>>> dn="cn=KPASSWD,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=54 RESULT err=0 tag=107
>>>>> nentries=0 etime=0 csn=5576dceb000700040000
>>>>> [09/Jun/2015:08:32:42 -0400] conn=150 op=55 DEL
>>>>> dn="cn=MEMCACHE,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net"
>>>>> [09/Jun/2015:08:32:43 -0400] conn=150 op=55 RESULT err=0 tag=107
>>>>> nentries=0 etime=1 csn=5576dcec000100040000
>>>>> [09/Jun/2015:08:32:43 -0400] conn=150 op=56 UNBIND
>>>>>
>>>>> the search for cn=f22replica1.bagam.net,cn=masters,.... returns 8
>>>>> entries, which then should be deleted, but only 3 ae deleted and the
>>>>> cn=f22replica1.bagam.net,cn=masters,... entry is not deleted, so
>>>>> the topology segments are not deleted, and the agreement is not
>>>>> removed.
>>>>>
>>>>> I don't know why ipa-replica-manage del does stop deleting
>>>>> services and the master entry
>>>>>
>>>>>
>>>>>
>>>>> On 06/09/2015 04:25 PM, Oleg Fayans wrote:
>>>>>>
>>>>>>
>>>>>> On 06/09/2015 04:19 PM, Ludwig Krispenz wrote:
>>>>>>>
>>>>>>> On 06/09/2015 04:14 PM, Oleg Fayans wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06/09/2015 04:04 PM, Ludwig Krispenz wrote:
>>>>>>>>>
>>>>>>>>> On 06/09/2015 03:55 PM, Oleg Fayans wrote:
>>>>>>>>>> Hi everybody,
>>>>>>>>>>
>>>>>>>>>> The current status of Topology plugin testing is as follows:
>>>>>>>>>>
>>>>>>>>>> 1. There is still no proper way of removing the replica.
>>>>>>>>>> Standard procedure using `ipa-replica-manage del` throws
>>>>>>>>>> "Server is unwilling to perform: Entry is managed by topology
>>>>>>>>>> plugin.Deletion not allowed.".
>>>>>>>>> yes, that is for the first attempt to directly remove the
>>>>>>>>> agreement, but when the server is removed the agreements
>>>>>>>>> should be removed
>>>>>>>> We should probably think of less threatening error message in
>>>>>>>> this case. Just from reading the command output one might
>>>>>>>> conclude that replica removal failed.
>>>>>>>>>> The replication agreement though does get deleted,
>>>>>>>>> then it is ok,
>>>>>>>>>> but the topology information does not get updated.
>>>>>>>>> what do you mean, where do you check ? in the "remaining"
>>>>>>>>> topology the shared tree should be updated, for the removed
>>>>>>>>> replica it will not, but this should be uninstalled anyway
>>>>>>>> The problem here, is that the topology information does not get
>>>>>>>> updated on master as well.
>>>>>>> could you be a bit more precise. what do you still see ? the
>>>>>>> agreement will be only removed if the segment is removed, and
>>>>>>> this should be reoplicated to all severs in the remaining
>>>>>>> topology - if you don't disconnect it by removing the replica.
>>>>>>> and what was the topology structure and which replica did you
>>>>>>> remove, on which server did you remove it?
>>>>>> So, Here is the results of the `topologysegment-find` command
>>>>>> before replica removal:
>>>>>> root at f22master:/var/log/dirsrv/slapd-BAGAM-NET]$ ipa
>>>>>> topologysegment-find
>>>>>> Suffix name: realm
>>>>>> ------------------
>>>>>> 2 segments matched
>>>>>> ------------------
>>>>>> Segment name: f22master.bagam.net-to-f22replica1.bagam.net
>>>>>> Left node: f22master.bagam.net
>>>>>> Right node: f22replica1.bagam.net
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name: f22master.bagam.net-to-f22replica2.bagam.net
>>>>>> Left node: f22master.bagam.net
>>>>>> Right node: f22replica2.bagam.net
>>>>>> Connectivity: both
>>>>>> ----------------------------
>>>>>> Number of entries returned 2
>>>>>> ----------------------------
>>>>>> Then, after issuing `ipa-replica-manage-del f2replica1.bagam.net
>>>>>> --force` on the master, the same command on master still shows
>>>>>> exactly the same topology:
>>>>>>
>>>>>> root at f22master:/var/log/dirsrv/slapd-BAGAM-NET]$ ipa
>>>>>> topologysegment-find
>>>>>> Suffix name: realm
>>>>>> ------------------
>>>>>> 2 segments matched
>>>>>> ------------------
>>>>>> Segment name: f22master.bagam.net-to-f22replica1.bagam.net
>>>>>> Left node: f22master.bagam.net
>>>>>> Right node: f22replica1.bagam.net
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name: f22master.bagam.net-to-f22replica2.bagam.net
>>>>>> Left node: f22master.bagam.net
>>>>>> Right node: f22replica2.bagam.net
>>>>>> Connectivity: both
>>>>>> ----------------------------
>>>>>> Number of entries returned 2
>>>>>> ----------------------------
>>>>>>
>>>>>>>>>> When I then issue `ipa topologysegment-del`, it fails due to
>>>>>>>>>> "ipa: ERROR: Server is unwilling to perform: Removal of
>>>>>>>>>> Segment disconnects topology.Deletion not allowed."
>>>>>>>>> correct, you can only do it after removal of the server
>>>>>>>> I do not get it. Master still thinks it has the replica, it
>>>>>>>> displays it both in CLI using `ipa topologysegment-find` and in
>>>>>>>> the web-ui. (although it does not show it using `ipa
>>>>>>>> host-find`, which is correct), and there is no way to manually
>>>>>>>> make it change it's mind?
>>>>>>>>>>
>>>>>>>>>> I tried to disable the segment first and then delete it, but
>>>>>>>>>> with the segment properly disabled, the attempt to delete it
>>>>>>>>>> raised a GSS error: "ipa: ERROR: Kerberos error: Kerberos
>>>>>>>>>> error: ('Unspecified GSS failure. Minor code may provide
>>>>>>>>>> more information', 851968)/('KDC returned error string:
>>>>>>>>>> PROCESS_TGS', -1765328324)/". I am not sure, where to search
>>>>>>>>>> for corresponding logs. The session transcript is attached.
>>>>>>>>>>
>>>>>>>>>> 2. The following is probably unrelated to the topology plugin:
>>>>>>>>>> I installed a replica with --setup-ca option. Then, on this
>>>>>>>>>> replica tried to prepare another replica:
>>>>>>>>>> -------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> root at f22replica2:/home/ofayans/f22]$ ipa-replica-prepare
>>>>>>>>>> --ip-address 192.168.122.141 f22replica3.bagam.net
>>>>>>>>>> Directory Manager (existing master) password:
>>>>>>>>>>
>>>>>>>>>> Preparing replica for f22replica3.bagam.net from
>>>>>>>>>> f22replica2.bagam.net
>>>>>>>>>> Creating SSL certificate for the Directory Server
>>>>>>>>>> Certificate issuance failed
>>>>>>>>>> -------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> The corresponding line in the dirsrv log:
>>>>>>>>>> [09/Jun/2015:09:54:46 -0400] - Entry
>>>>>>>>>> "uid=admin,ou=people,o=ipaca" -- attribute "krbExtraData" not
>>>>>>>>>> allowed
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Oleg Fayans
>>>>>>>> Quality Engineer
>>>>>>>> FreeIPA team
>>>>>>>> RedHat.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Oleg Fayans
>>>>>> Quality Engineer
>>>>>> FreeIPA team
>>>>>> RedHat.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Martin Basti
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20150610/b1bdaa5d/attachment.htm>
More information about the Freeipa-devel
mailing list