<html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 06/10/2015 02:13 PM, thierry bordaz wrote: </div> <blockquote cite="mid:557829D9.10901@redhat.com" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 06/10/2015 10:51 AM, Ludwig Krispenz wrote: </div> <blockquote cite="mid:5577FA98.9070808@redhat.com" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 06/10/2015 10:41 AM, Martin Basti wrote: </div> <blockquote cite="mid:5577F830.8020208@redhat.com" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 10/06/15 09:13, Ludwig Krispenz wrote: </div> <blockquote cite="mid:5577E3A4.90309@redhat.com" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> Hi, there seems to be somethin going wrong in the code to delete the services. The code is: # delete master entry with all active services try: dn = DN(('cn', replica), ('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'), self.suffix) entries = self.conn.get_entries(dn, ldap.SCOPE_SUBTREE) if entries: entries.sort(key=lambda x: len(x.dn), reverse=True) for entry in entries: self.conn.delete_entry(entry) except errors.NotFound: pass except Exception, e: if not force: raise e elif not err: err = e In the access log we see: [09/Jun/2015:08:32:42 -0400] conn=150 op=52 SRCH base="cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net" scope=2 filter="(objectClass=*)" attrs=ALL [09/Jun/2015:08:32:42 -0400] conn=150 op=52 RESULT err=0 tag=101 nentries=8 etime=0 notes=U this was the get_entries, it returns 8 entries [09/Jun/2015:08:32:42 -0400] conn=150 op=53 DEL dn="cn=KDC,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net" [09/Jun/2015:08:32:42 -0400] conn=150 op=53 RESULT err=0 tag=107 nentries=0 etime=0 csn=5576dceb000600040000 [09/Jun/2015:08:32:42 -0400] conn=150 op=54 DEL dn="cn=KPASSWD,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net" [09/Jun/2015:08:32:42 -0400] conn=150 op=54 RESULT err=0 tag=107 nentries=0 etime=0 csn=5576dceb000700040000 [09/Jun/2015:08:32:42 -0400] conn=150 op=55 DEL dn="cn=MEMCACHE,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net" [09/Jun/2015:08:32:43 -0400] conn=150 op=55 RESULT err=0 tag=107 nentries=0 etime=1 csn=5576dcec000100040000 here it stops after deleting three entries, and it should do it in reverse order of the dn length, but KDC is deleted before MEMCACHE </blockquote> </blockquote> </blockquote> Something surprising is that according to the code <blockquote><tt> # delete master entry with all active services</tt> <tt> try:</tt> <tt> dn = DN(('cn', replica), ('cn', 'masters'), ('cn', 'ipa'),</tt> <tt> ('cn', 'etc'), self.suffix)</tt> <tt> entries = self.conn.get_entries(dn, ldap.SCOPE_SUBTREE)</tt> <tt> if entries:</tt> <tt> entries.sort(key=lambda x: len(x.dn), reverse=True)</tt> <tt> for entry in entries:</tt> <tt> self.conn.delete_entry(entry)</tt> <tt> except errors.NotFound:</tt> <tt> pass</tt> <tt> except Exception, e:</tt> <tt> if not force:</tt> <tt> raise e</tt> <tt> elif not err:</tt> <tt> err = e</tt> <tt> try:</tt> <tt> entry = self.conn.get_entry(</tt> <tt> DN(('cn', 'ipa'), ('cn', 'etc'), self.suffix), ['aci'])</tt> <tt> sub = {'suffix': self.suffix, 'fqdn': replica}</tt> <tt> ...</tt> </blockquote> We should see a search on cn=ipa,cn=etc,SUFFIX following the deletion of those entries. But the next op is an UNBIND. </blockquote> yes, that is strange, maybe we hit an exception and the connection was closed <blockquote cite="mid:557829D9.10901@redhat.com" type="cite"> Is that the code executed by ipa-replica-manage ? </blockquote> I think so, yes. <blockquote cite="mid:557829D9.10901@redhat.com" type="cite"> <blockquote cite="mid:5577FA98.9070808@redhat.com" type="cite"> <blockquote cite="mid:5577F830.8020208@redhat.com" type="cite"> <blockquote cite="mid:5577E3A4.90309@redhat.com" type="cite"> [09/Jun/2015:08:32:43 -0400] conn=150 op=56 UNBIND Are there any ideas what is going on or how to debug it ? </blockquote> Actually, the both DNs of KDC and MEMCACHE has the same length. IPA implements own DN class, where length is the number of AVA/RDN parts (mixed in code, but it means the 'cn=user' has length 1, and 'cn=user,cn=accounts' has length 2) def __len__(self): return len(self.rdns) This reverse sort guarantees the child entries will be removed before the parent entries. </blockquote> thanks, then it is ok, but it does not explain why not all services and the master were not deleted. <blockquote cite="mid:5577F830.8020208@redhat.com" type="cite"> To debug, maybe print the entries from IPA code, before sort and after sort might help. </blockquote> yep, but so far only Oleg reprted this, and he's not here today, I haven't reproduced the issue <blockquote cite="mid:5577F830.8020208@redhat.com" type="cite"> Martin^2 <blockquote cite="mid:5577E3A4.90309@redhat.com" type="cite"> <div class="moz-cite-prefix">On 06/09/2015 05:32 PM, Ludwig Krispenz wrote: </div> <blockquote cite="mid:55770703.8030202@redhat.com" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> Hi Oleg, thanks for access to your machine, the replication agreements are still there - and that is expected since the server was not removed. In the access log I see: [09/Jun/2015:08:32:42 -0400] conn=150 op=52 SRCH base="cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net" scope=2 filter="(objectClass=*)" attrs=ALL [09/Jun/2015:08:32:42 -0400] conn=150 op=52 RESULT err=0 tag=101 nentries=8 etime=0 notes=U [09/Jun/2015:08:32:42 -0400] conn=150 op=53 DEL dn="cn=KDC,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net" [09/Jun/2015:08:32:42 -0400] conn=150 op=53 RESULT err=0 tag=107 nentries=0 etime=0 csn=5576dceb000600040000 [09/Jun/2015:08:32:42 -0400] conn=150 op=54 DEL dn="cn=KPASSWD,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net" [09/Jun/2015:08:32:42 -0400] conn=150 op=54 RESULT err=0 tag=107 nentries=0 etime=0 csn=5576dceb000700040000 [09/Jun/2015:08:32:42 -0400] conn=150 op=55 DEL dn="cn=MEMCACHE,cn=f22replica1.bagam.net,cn=masters,cn=ipa,cn=etc,dc=bagam,dc=net" [09/Jun/2015:08:32:43 -0400] conn=150 op=55 RESULT err=0 tag=107 nentries=0 etime=1 csn=5576dcec000100040000 [09/Jun/2015:08:32:43 -0400] conn=150 op=56 UNBIND the search for cn=f22replica1.bagam.net,cn=masters,.... returns 8 entries, which then should be deleted, but only 3 ae deleted and the cn=f22replica1.bagam.net,cn=masters,... entry is not deleted, so the topology segments are not deleted, and the agreement is not removed. I don't know why ipa-replica-manage del does stop deleting services and the master entry <div class="moz-cite-prefix">On 06/09/2015 04:25 PM, Oleg Fayans wrote: </div> <blockquote cite="mid:5576F755.7080809@redhat.com" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 06/09/2015 04:19 PM, Ludwig Krispenz wrote: </div> <blockquote cite="mid:5576F5E6.2030502@redhat.com" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 06/09/2015 04:14 PM, Oleg Fayans wrote: </div> <blockquote cite="mid:5576F4D8.80907@redhat.com" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 06/09/2015 04:04 PM, Ludwig Krispenz wrote: </div> <blockquote cite="mid:5576F26C.7010802@redhat.com" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 06/09/2015 03:55 PM, Oleg Fayans wrote: </div> <blockquote cite="mid:5576F055.2060603@redhat.com" type="cite">Hi everybody, The current status of Topology plugin testing is as follows: 1. There is still no proper way of removing the replica. Standard procedure using `ipa-replica-manage del` throws "Server is unwilling to perform: Entry is managed by topology plugin.Deletion not allowed.". </blockquote> yes, that is for the first attempt to directly remove the agreement, but when the server is removed the agreements should be removed </blockquote> We should probably think of less threatening error message in this case. Just from reading the command output one might conclude that replica removal failed. <blockquote cite="mid:5576F26C.7010802@redhat.com" type="cite"> <blockquote cite="mid:5576F055.2060603@redhat.com" type="cite">The replication agreement though does get deleted, </blockquote> then it is ok, <blockquote cite="mid:5576F055.2060603@redhat.com" type="cite">but the topology information does not get updated. </blockquote> what do you mean, where do you check ? in the "remaining" topology the shared tree should be updated, for the removed replica it will not, but this should be uninstalled anyway </blockquote> The problem here, is that the topology information does not get updated on master as well. </blockquote> could you be a bit more precise. what do you still see ? the agreement will be only removed if the segment is removed, and this should be reoplicated to all severs in the remaining topology - if you don't disconnect it by removing the replica. and what was the topology structure and which replica did you remove, on which server did you remove it? </blockquote> So, Here is the results of the `topologysegment-find` command before replica removal: root@f22master:/var/log/dirsrv/slapd-BAGAM-NET]$ ipa topologysegment-find Suffix name: realm ------------------ 2 segments matched ------------------ Segment name: f22master.bagam.net-to-f22replica1.bagam.net Left node: f22master.bagam.net Right node: f22replica1.bagam.net Connectivity: both Segment name: f22master.bagam.net-to-f22replica2.bagam.net Left node: f22master.bagam.net Right node: f22replica2.bagam.net Connectivity: both ---------------------------- Number of entries returned 2 ---------------------------- Then, after issuing `ipa-replica-manage-del f2replica1.bagam.net --force` on the master, the same command on master still shows exactly the same topology: root@f22master:/var/log/dirsrv/slapd-BAGAM-NET]$ ipa topologysegment-find Suffix name: realm ------------------ 2 segments matched ------------------ Segment name: f22master.bagam.net-to-f22replica1.bagam.net Left node: f22master.bagam.net Right node: f22replica1.bagam.net Connectivity: both Segment name: f22master.bagam.net-to-f22replica2.bagam.net Left node: f22master.bagam.net Right node: f22replica2.bagam.net Connectivity: both ---------------------------- Number of entries returned 2 ---------------------------- <blockquote cite="mid:5576F5E6.2030502@redhat.com" type="cite"> <blockquote cite="mid:5576F4D8.80907@redhat.com" type="cite"> <blockquote cite="mid:5576F26C.7010802@redhat.com" type="cite"> <blockquote cite="mid:5576F055.2060603@redhat.com" type="cite">When I then issue `ipa topologysegment-del`, it fails due to "ipa: ERROR: Server is unwilling to perform: Removal of Segment disconnects topology.Deletion not allowed." </blockquote> correct, you can only do it after removal of the server </blockquote> I do not get it. Master still thinks it has the replica, it displays it both in CLI using `ipa topologysegment-find` and in the web-ui. (although it does not show it using `ipa host-find`, which is correct), and there is no way to manually make it change it's mind? <blockquote cite="mid:5576F26C.7010802@redhat.com" type="cite"> <blockquote cite="mid:5576F055.2060603@redhat.com" type="cite"> I tried to disable the segment first and then delete it, but with the segment properly disabled, the attempt to delete it raised a GSS error: "ipa: ERROR: Kerberos error: Kerberos error: ('Unspecified GSS failure. Minor code may provide more information', 851968)/('KDC returned error string: PROCESS_TGS', -1765328324)/". I am not sure, where to search for corresponding logs. The session transcript is attached. 2. The following is probably unrelated to the topology plugin: I installed a replica with --setup-ca option. Then, on this replica tried to prepare another replica: ------------------------------------------------------------------------------------------------------------------------------------------------- root@f22replica2:/home/ofayans/f22]$ ipa-replica-prepare --ip-address 192.168.122.141 f22replica3.bagam.net Directory Manager (existing master) password: Preparing replica for f22replica3.bagam.net from f22replica2.bagam.net Creating SSL certificate for the Directory Server Certificate issuance failed ------------------------------------------------------------------------------------------------------------------------------------------------- The corresponding line in the dirsrv log: [09/Jun/2015:09:54:46 -0400] - Entry "uid=admin,ou=people,o=ipaca" -- attribute "krbExtraData" not allowed <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> <pre class="moz-signature" cols="72">-- Oleg Fayans Quality Engineer FreeIPA team RedHat.</pre> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> <pre class="moz-signature" cols="72">-- Oleg Fayans Quality Engineer FreeIPA team RedHat.</pre> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> <pre class="moz-signature" cols="72">-- Martin Basti</pre> </blockquote> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> </blockquote> </body> </html>