From todd.nine at onwebconsulting.com Sun Mar 2 22:44:08 2008 From: todd.nine at onwebconsulting.com (Todd Nine) Date: Mon, 3 Mar 2008 11:44:08 +1300 Subject: [Fedora-directory-devel] Creating an OSX Port Message-ID: <996eb2230803021444n1400206cuec696680cd5ec7d6@mail.gmail.com> Hi all, I'm a J2EE Developer, and an amateur system admin. I absolutely love Fedora Directory, and I'd like to attempt to port the console to Mac and create a simple install DMG. I read on the requirements that a Sun JDK is required. Has anyone successfully run on an OS X jdk. Is it even possible, or does it require linux specific gtk libraries in java? Thanks, Todd -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmeggins at redhat.com Mon Mar 3 15:34:24 2008 From: rmeggins at redhat.com (Rich Megginson) Date: Mon, 03 Mar 2008 08:34:24 -0700 Subject: [Fedora-directory-devel] Creating an OSX Port In-Reply-To: <996eb2230803021444n1400206cuec696680cd5ec7d6@mail.gmail.com> References: <996eb2230803021444n1400206cuec696680cd5ec7d6@mail.gmail.com> Message-ID: <47CC1A80.2010501@redhat.com> Todd Nine wrote: > Hi all, > I'm a J2EE Developer, and an amateur system admin. I absolutely > love Fedora Directory, and I'd like to attempt to port the console to > Mac and create a simple install DMG. I read on the requirements that > a Sun JDK is required. No, IBM should work as well. Others work too (e.g. the console works on HP-UX using the HP provided JRE). > Has anyone successfully run on an OS X jdk. Is it even possible, or > does it require linux specific gtk libraries in java? It does not depend on specific gtk libraries. The GUI uses the Swing toolkit which, as far as the console code is concerned, is platform independent. So it might just work with the OSX jre. > > Thanks, > Todd > ------------------------------------------------------------------------ > > -- > Fedora-directory-devel mailing list > Fedora-directory-devel at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-devel > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3245 bytes Desc: S/MIME Cryptographic Signature URL: From nkinder at redhat.com Mon Mar 3 16:27:39 2008 From: nkinder at redhat.com (Nathan Kinder) Date: Mon, 03 Mar 2008 08:27:39 -0800 Subject: [Fedora-directory-devel] Creating an OSX Port In-Reply-To: <996eb2230803021444n1400206cuec696680cd5ec7d6@mail.gmail.com> References: <996eb2230803021444n1400206cuec696680cd5ec7d6@mail.gmail.com> Message-ID: <47CC26FB.8060007@redhat.com> Todd Nine wrote: > Hi all, > I'm a J2EE Developer, and an amateur system admin. I absolutely > love Fedora Directory, and I'd like to attempt to port the console to > Mac and create a simple install DMG. I read on the requirements that > a Sun JDK is required. Has anyone successfully run on an OS X jdk. > Is it even possible, or does it require linux specific gtk libraries > in java? I think you'll have no problem with the main Console jars. I think I've been able to at least launch the Console up on OS X a long time ago, but things have changed with the Console since then, so it might just be easier now. You can likely just copy the jars over from a Linux system. In order to use any feature which requires crypto (SSL, importing certificates, etc.) in the Console, you will need to build JSS, which has a JNI library. It will require you to compile NSS and NSPR as well. All of these components have been ported to OS X previously, so it should be quite doable. -NGK > > Thanks, > Todd > ------------------------------------------------------------------------ > > -- > Fedora-directory-devel mailing list > Fedora-directory-devel at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-devel > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3241 bytes Desc: S/MIME Cryptographic Signature URL: From nkinder at redhat.com Mon Mar 3 17:35:25 2008 From: nkinder at redhat.com (Nathan Kinder) Date: Mon, 03 Mar 2008 09:35:25 -0800 Subject: [Fedora-directory-devel] Please Review: (435730) Allow fractional replication between masters Message-ID: <47CC36DD.7070107@redhat.com> https://bugzilla.redhat.com/show_bug.cgi?id=435730 Resolves: bug 435730 Bug Description: Fractional replication is only currently allowed when the agreement destination is a read-only replica. We should allow fraction replication when the destination is a master or a hub, with the caveat that the attributes replicated for all agreements between masters are exactly the same. This change is needed to support attributes that are dynamically generated by each server instance, as is done when using the recently added memberOf plug-in. Reviewed by: ??? Files: see diff Branch: HEAD, Directory_Server_8_0_Branch Fix Description: The fix simply removes the check to see if we're trying to use a fractional agreement with a non-read-only replica when we acquire the replica. Platforms tested: RHEL5.1 i386 Flag Day: no Doc impact: Yes. We need to document that agreements need to replicate the same attributes in both directions when using fractional between masters. https://bugzilla.redhat.com/attachment.cgi?id=296630&action=diff -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3241 bytes Desc: S/MIME Cryptographic Signature URL: From nhosoi at redhat.com Thu Mar 13 20:42:37 2008 From: nhosoi at redhat.com (Noriko Hosoi) Date: Thu, 13 Mar 2008 12:42:37 -0800 Subject: [Fedora-directory-devel] Please review: [Bug 436397] New: LDAPI: move default LDAPI UNIX socket from /var/run/dirsrv/slapd-ID.socket to /var/run/slapd-ID.socket In-Reply-To: References: Message-ID: <47D991BD.5000509@redhat.com> After the discussion, we agreed to move the LDAPI UNIX socket from RHDS/FDS run_dir (/var/run/dirsrv, by default) to its parent directory. Thanks, --noriko https://bugzilla.redhat.com/show_bug.cgi?id=436397 Summary: LDAPI: move default LDAPI UNIX socket from /var/run/dirsrv/slapd-ID.socket to /var/run/slapd- ID.socket Product: Fedora Directory Server Version: 1.1.0 Platform: All OS/Version: Linux Status: NEW Severity: low Priority: low Component: Directory Server AssignedTo: nhosoi at redhat.com ReportedBy: nhosoi at redhat.com QAContact: ckannan at redhat.com Estimated Hours: 0.0 Description of problem: * If fedora-ds-base is installed by root, the mode of /var/run/dirsrv is 0750, which prevents ordinary users to access the UNIX socket. Should the mode be 0755? Or we don't allow non-root/non-nobody users to use LDAPI? drwxr-x--- 2 nobody nobody 4096 Mar 5 13:57 /var/run/dirsrv/ It's set by makeDSDirs in DSCreate.pm. rmeggins wrote: > > We should see what OpenLDAP does - they use /var/run/ldapi by default - what > mode is that by default? It's about the intermediate directory's permission. OpenLDAP just has /var and /var/run. ldapi is already the socket, isn't it? rmeggins wrote: > > Yes. > We have one more level /var/run/dirsrv, which is hiding the socket from non-root and non-nobody... But yes, I have to install openldap and investigate more. rmeggins wrote: > > Hmm - we probably don't want to open up /var/run/dirsrv if we don't have to - > maybe we should move the socket into /var/run? e.g. /var/run/slapd-instance.socket? I think that's a good idea. One thing I'd like to make sure is we have to worry about RHDS/FDS coexisting with OpenLDAP server on one host? Something like, if port 389 is already taken, our setup-ds offers alternative. Do we need to do something similar for LDAPI socket? rmeggins wrote: > > If there is already a /var/run/ldapi and it is in use by openldap (or another > redhat/fedora ds) we probably don't want to use it. nalin wrote: > > When OpenLDAP's libldap gets 'ldapi:///' as a URI, it tries to connect > > to '/var/run/ldapi'. Perhaps we should just use that? > > > > Nalin > ------- Additional Comments From nhosoi at redhat.com 2008-03-13 16:36 EST ------- Created an attachment (id=297983) --> (https://bugzilla.redhat.com/attachment.cgi?id=297983&action=view) cvs diff DSCreate.pm.in Description: create an LDAPI UNIX socket at the parent dir of run_dir (/var/run/dirsrv, by default). Test result. Installed by root and the server's owner is nobody. # ls -l /var/run/slapd-*socket srw-rw-rw- 1 root root 0 Mar 13 10:28 /var/run/slapd-laputa1.socket [..] - Red Hat-Directory/8.0.0 B2008.073.1814 starting up [..] - slapd started. Listening on All Interfaces port 10391 for LDAP requests [..] - Listening on /var/run/slapd-laputa1.socket for LDAPI requests -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3237 bytes Desc: S/MIME Cryptographic Signature URL: From nhosoi at redhat.com Fri Mar 14 22:33:16 2008 From: nhosoi at redhat.com (Noriko Hosoi) Date: Fri, 14 Mar 2008 14:33:16 -0800 Subject: [Fedora-directory-devel] Please review: [Bug 436388] LDAPI: introduce --enable-autobind to support AUTOBIND In-Reply-To: <200803142219.m2EMJGCp022296@bz-web1.app.phx.redhat.com> References: <200803142219.m2EMJGCp022296@bz-web1.app.phx.redhat.com> Message-ID: <47DAFD2C.2050605@redhat.com> Summary: LDAPI: introduce --enable-autobind to support AUTOBIND https://bugzilla.redhat.com/show_bug.cgi?id=436388 Description of problem: * Auto bind codes are all in the ENABLE_AUTOBIND macro. Should we enable it and support the functionality? rmeggins wrote: > > Yes, but turned off by default. > Okay. then should we add --enable-autobind to configure.ac? rmeggins wrote: > > Yes. > Or should ENABLE_AUTOBIND be part of LDAPI? I feel autobind is tightly coupled with LDAPI, ENABLE_AUTOBIND could be replaced with ENABLE_LDAPI and merge template-ldapi-autobind into template-ldapi-default? rmeggins wrote: > > I think there may be some security conscious people who will not want to > enable autobind at all and will want to build without it. ------- Additional Comments From nhosoi at redhat.com 2008-03-14 18:19 EST ------- autoconf gets uid # and gid # from the LDAPI UNIX socket and retrieve the matched entry from the backend to bind the server. For example, Assume these are my uid # and gid # on the test system: $ id uid=12345(nhosoi) gid=12345(nhosoi) Add this posix account to the server: dn: uid=nhosoi, dc=example,dc=com objectclass: top objectclass: posixAccount cn: noriko hosoi uid: nhosoi uidNumber: 12345 gidNumber: 12345 homeDirectory: /home/nhosoi loginShell: bash userPassword: nhosoi Then, run the search against LDAPI UNIX socket without the bind user. Autobind internally searches an entry with the filter (&(uidNumber=12345)(gidNumber=12345)) and binds using the found entry. $ ldapsearch -H ldapi://%2fvar%2frun%2fslapd-laputa.socket/ -w nhosoi -Y DIGEST-MD5 -b "dc=example,dc=com" "(cn=*)" SASL/DIGEST-MD5 authentication started SASL username: nhosoi SASL SSF: 128 SASL installing layers [...] Tested on RHEL4. To use autobind, ldapi, autobind, and maptoentries need to be turned on. nsslapd-ldapifilepath: /var/run/slapd-laputa.socket nsslapd-ldapilisten: on nsslapd-ldapiautobind: on nsslapd-ldapimaprootdn: cn=Directory Manager nsslapd-ldapimaptoentries: on nsslapd-ldapiuidnumbertype: uidNumber nsslapd-ldapigidnumbertype: gidNumber nsslapd-ldapientrysearchbase: dc=example,dc=com nsslapd-ldapiautodnsuffix: cn=peercred,cn=external,cn=auth ------- Additional Comments From nhosoi at redhat.com 2008-03-14 18:30 EST ------- Created an attachment (id=298099) --> (https://bugzilla.redhat.com/attachment.cgi?id=298099&action=view) cvs diff configure.ac Makefile.am -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3237 bytes Desc: S/MIME Cryptographic Signature URL: From abartlet at samba.org Wed Mar 19 10:05:19 2008 From: abartlet at samba.org (Andrew Bartlett) Date: Wed, 19 Mar 2008 21:05:19 +1100 Subject: [Fedora-directory-devel] Problems creating a Samba4 LDAP Backend Message-ID: <1205921119.22616.52.camel@naomi> Over the past few weeks, I have been testing OpenLDAP as a backend for Samba4. I've been working with the OpenLDAP team on my requirements, and there has been some really good outcomes - the memberOf module has been improved, as has the refint module. However, I seem to have hit a brick wall, in the form of (internal) transaction support. I need an LDAP backend to support internal transactions - that is, when for example a 'member' modification is made, all the memberOf attributes must be updated before the call returns. Similarly, if a user or group moves, all the member/memberOf attributes that link the user to their groups must also move, before the modrdn returns. The Samba4 test ldap.js tests this behaviour extensively, because I want to be sure it works. As understand the discussion I've had with the OpenLDAP team, OpenLDAP does not support this, and will not support it for perhaps some time. Similarly, from discussions with the Fedora DS team at the CIFS developer days, I understand that it is similarly very unlikely that Fedora DS will support internal transactions. (It also does not support subtree renames, which we also need). The fact that LDAP does not expose a transaction API was always going to be a difficult part of having Samba4 use an LDAP backend, but I always assumed that if we pushed the really hard bit - updating linked attributes - into the LDAP server that we could at least always have a consistent DB. (It turns out this is one of the primary uses of transactions anyway.) But without that consistency, and without knowing as a caller if all the updates succeed, I'm worried about how we can safely move forward. This is especially disappointing because I was hoping that these free, replicating LDAP servers might solve the backed replication problem for me, without needing to use AD replication. Does anybody have any ideas or suggestions on how I could get around this? Should we drop the LDAP backend as a nice idea, but not going to work, and focus on DRS or some other form of replication? Can someone imagine a sane way to reconstruct the DN links, including subtree renames, without the help of the LDAP backend? Could we ban subtree renames (as Fedora DS does), and try to handle this ourselves (with pre/post checks and a good deal of prayer)? Thanks, Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Red Hat Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From abartlet at samba.org Wed Mar 19 10:13:40 2008 From: abartlet at samba.org (Andrew Bartlett) Date: Wed, 19 Mar 2008 21:13:40 +1100 Subject: [Fedora-directory-devel] Where to note Samba4 required features? Message-ID: <1205921620.22616.55.camel@naomi> Given my previous mail about the suitability of OpenLDAP and Fedora DS as a Samba4, can someone remind me where I need to list my feature requests? I realise that things like subtree renames and transactions might be positioned somewhere between 'unlikely' and 'never' (give discussions on the technical complexity in the codebase), but I figure I should list them somewhere... Thanks, Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Red Hat Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From hyc at symas.com Wed Mar 19 11:26:47 2008 From: hyc at symas.com (Howard Chu) Date: Wed, 19 Mar 2008 04:26:47 -0700 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <1205921119.22616.52.camel@naomi> References: <1205921119.22616.52.camel@naomi> Message-ID: <47E0F877.10103@symas.com> Andrew Bartlett wrote: > Over the past few weeks, I have been testing OpenLDAP as a backend for > Samba4. > > I've been working with the OpenLDAP team on my requirements, and there > has been some really good outcomes - the memberOf module has been > improved, as has the refint module. > > However, I seem to have hit a brick wall, in the form of (internal) > transaction support. I need an LDAP backend to support internal > transactions - that is, when for example a 'member' modification is > made, all the memberOf attributes must be updated before the call > returns. Similarly, if a user or group moves, all the member/memberOf > attributes that link the user to their groups must also move, before the > modrdn returns. > > The Samba4 test ldap.js tests this behaviour extensively, because I want > to be sure it works. > > As understand the discussion I've had with the OpenLDAP team, OpenLDAP > does not support this, and will not support it for perhaps some time. I may have overstated the problem in the previous discussion of our refint module. In fact, RE24 was already changed to work around any potential deadlock issues a long time ago. But to give some context: the refint module was originally written to operate synchronously (back in 2004). Some time later it was changed (2006) to asynchronous mode because users didn't want their clients to be stuck waiting for all the cascaded updates to complete. Most clients don't know or care that a particular change has side-effects. We could introduce a config keyword to select synch vs asynch behavior here, but I have a feeling that will still leave some group of users unhappy no matter how you set it. > Similarly, from discussions with the Fedora DS team at the CIFS > developer days, I understand that it is similarly very unlikely that > Fedora DS will support internal transactions. (It also does not support > subtree renames, which we also need). > > The fact that LDAP does not expose a transaction API You mean draft-zeilenga-ldap-txn ? > was always going to > be a difficult part of having Samba4 use an LDAP backend, but I always > assumed that if we pushed the really hard bit - updating linked > attributes - into the LDAP server that we could at least always have a > consistent DB. (It turns out this is one of the primary uses of > transactions anyway.) > But without that consistency, and without knowing as a caller if all the > updates succeed, I'm worried about how we can safely move forward. > This is especially disappointing because I was hoping that these free, > replicating LDAP servers might solve the backed replication problem for > me, without needing to use AD replication. > > Does anybody have any ideas or suggestions on how I could get around > this? There are other ways to guarantee consistency. The simplest approach is to just not store one end of the linked attributes, and always generate them dynamically when they're referenced. In the old Symas Connexitor EMS product we used (the equivalent of) a UUIDAndOptionalName syntax for all references. In that case the DN was essentially just window-dressing; we always used the ID to actually reference entries and we updated the DNs whenever we found that they didn't match. As such, referential integrity was pretty simple - you never had anything pointing to the wrong entry; the worst that would happen is that you occasionally had dangling references to deleted entries stored in the DB but no one ever saw them because they were cleaned out whenever the entry containing the reference was read. > Should we drop the LDAP backend as a nice idea, but not going to work, > and focus on DRS or some other form of replication? > > Can someone imagine a sane way to reconstruct the DN links, including > subtree renames, without the help of the LDAP backend? Could we ban > subtree renames (as Fedora DS does), and try to handle this ourselves > (with pre/post checks and a good deal of prayer)? Banning subtree renames seems like a non-starter, and it only eliminates one case; the overall problem still remains. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ From andrey.ivanov at polytechnique.fr Wed Mar 19 12:12:55 2008 From: andrey.ivanov at polytechnique.fr (Andrey Ivanov) Date: Wed, 19 Mar 2008 13:12:55 +0100 Subject: [Fedora-directory-devel] Where to note Samba4 required features? In-Reply-To: <1205921620.22616.55.camel@naomi> References: <1205921620.22616.55.camel@naomi> Message-ID: <1601b8650803190512x3a54db8dm9265c9396f0b9e92@mail.gmail.com> Hi, I think you can do it on bugzilla.redhat.com. I have already listed (partially, just to enable the entries move between subtrees) the feature request for the subtree rename : https://bugzilla.redhat.com/show_bug.cgi?id=429005 2008/3/19, Andrew Bartlett : > Given my previous mail about the suitability of OpenLDAP and Fedora DS > as a Samba4, can someone remind me where I need to list my feature > requests? I realise that things like subtree renames and transactions > might be positioned somewhere between 'unlikely' and 'never' (give > discussions on the technical complexity in the codebase), but I figure I > should list them somewhere... > > Thanks, > > Andrew Bartlett > > -- > Andrew Bartlett > http://samba.org/~abartlet/ > Authentication Developer, Samba Team http://samba.org > Samba Developer, Red Hat Inc. From abartlet at samba.org Wed Mar 19 23:23:02 2008 From: abartlet at samba.org (Andrew Bartlett) Date: Thu, 20 Mar 2008 10:23:02 +1100 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <47E0F877.10103@symas.com> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> Message-ID: <1205968982.14751.3.camel@naomi> On Wed, 2008-03-19 at 04:26 -0700, Howard Chu wrote: > Andrew Bartlett wrote: > > Over the past few weeks, I have been testing OpenLDAP as a backend for > > Samba4. > > > > I've been working with the OpenLDAP team on my requirements, and there > > has been some really good outcomes - the memberOf module has been > > improved, as has the refint module. > > > > However, I seem to have hit a brick wall, in the form of (internal) > > transaction support. I need an LDAP backend to support internal > > transactions - that is, when for example a 'member' modification is > > made, all the memberOf attributes must be updated before the call > > returns. Similarly, if a user or group moves, all the member/memberOf > > attributes that link the user to their groups must also move, before the > > modrdn returns. > > > > The Samba4 test ldap.js tests this behaviour extensively, because I want > > to be sure it works. > > > > As understand the discussion I've had with the OpenLDAP team, OpenLDAP > > does not support this, and will not support it for perhaps some time. > > I may have overstated the problem in the previous discussion of our refint > module. In fact, RE24 was already changed to work around any potential > deadlock issues a long time ago. But to give some context: the refint module > was originally written to operate synchronously (back in 2004). Some time > later it was changed (2006) to asynchronous mode because users didn't want > their clients to be stuck waiting for all the cascaded updates to complete. > Most clients don't know or care that a particular change has side-effects. We > could introduce a config keyword to select synch vs asynch behavior here, but > I have a feeling that will still leave some group of users unhappy no matter > how you set it. Great. If run sync, will it error out correctly if I make an invalid modification (say target not present etc). > > Similarly, from discussions with the Fedora DS team at the CIFS > > developer days, I understand that it is similarly very unlikely that > > Fedora DS will support internal transactions. (It also does not support > > subtree renames, which we also need). > > > > The fact that LDAP does not expose a transaction API > > You mean draft-zeilenga-ldap-txn ? I suppose I should have said 'The free LDAP implemetnations I'm looking at don't expose a transaction API'. What did end up happening with that draft? > > was always going to > > be a difficult part of having Samba4 use an LDAP backend, but I always > > assumed that if we pushed the really hard bit - updating linked > > attributes - into the LDAP server that we could at least always have a > > consistent DB. (It turns out this is one of the primary uses of > > transactions anyway.) > > > But without that consistency, and without knowing as a caller if all the > > updates succeed, I'm worried about how we can safely move forward. > > > This is especially disappointing because I was hoping that these free, > > replicating LDAP servers might solve the backed replication problem for > > me, without needing to use AD replication. > > > > Does anybody have any ideas or suggestions on how I could get around > > this? > > There are other ways to guarantee consistency. The simplest approach is to > just not store one end of the linked attributes, and always generate them > dynamically when they're referenced. > > In the old Symas Connexitor EMS product we used (the equivalent of) a > UUIDAndOptionalName syntax for all references. In that case the DN was > essentially just window-dressing; we always used the ID to actually reference > entries and we updated the DNs whenever we found that they didn't match. As > such, referential integrity was pretty simple - you never had anything > pointing to the wrong entry; the worst that would happen is that you > occasionally had dangling references to deleted entries stored in the DB but > no one ever saw them because they were cleaned out whenever the entry > containing the reference was read. Do you think the LDAP backend could/should handle this, or will Samba4 have to do the GUID -> DN and DN -> GUID translations before passing things to the backend? > > Should we drop the LDAP backend as a nice idea, but not going to work, > > and focus on DRS or some other form of replication? > > > > Can someone imagine a sane way to reconstruct the DN links, including > > subtree renames, without the help of the LDAP backend? Could we ban > > subtree renames (as Fedora DS does), and try to handle this ourselves > > (with pre/post checks and a good deal of prayer)? > > Banning subtree renames seems like a non-starter, and it only eliminates one > case; the overall problem still remains. Indeed. Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Red Hat Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From hyc at symas.com Thu Mar 20 00:09:25 2008 From: hyc at symas.com (Howard Chu) Date: Wed, 19 Mar 2008 17:09:25 -0700 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <1205968982.14751.3.camel@naomi> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> Message-ID: <47E1AB35.7050800@symas.com> Andrew Bartlett wrote: > On Wed, 2008-03-19 at 04:26 -0700, Howard Chu wrote: >> Most clients don't know or care that a particular change has side-effects. We >> could introduce a config keyword to select synch vs asynch behavior here, but >> I have a feeling that will still leave some group of users unhappy no matter >> how you set it. > > Great. If run sync, will it error out correctly if I make an invalid > modification (say target not present etc). No. We specifically don't validate the original modification here - the server's number one job is always to try to do exactly what the client specified. Also, you can't reliably detect an "invalid" name - it may simply be the name of a non-local entry. >>> The fact that LDAP does not expose a transaction API >> You mean draft-zeilenga-ldap-txn ? > > I suppose I should have said 'The free LDAP implemetnations I'm looking > at don't expose a transaction API'. What did end up happening with > that draft? It's stalled, like a lot of other things... It is partially implemented in OpenLDAP. Eventually we'll have to complete the implementation, it's too important to leave dangling. >>> Does anybody have any ideas or suggestions on how I could get around >>> this? >> There are other ways to guarantee consistency. The simplest approach is to >> just not store one end of the linked attributes, and always generate them >> dynamically when they're referenced. >> >> In the old Symas Connexitor EMS product we used (the equivalent of) a >> UUIDAndOptionalName syntax for all references. In that case the DN was >> essentially just window-dressing; we always used the ID to actually reference >> entries and we updated the DNs whenever we found that they didn't match. As >> such, referential integrity was pretty simple - you never had anything >> pointing to the wrong entry; the worst that would happen is that you >> occasionally had dangling references to deleted entries stored in the DB but >> no one ever saw them because they were cleaned out whenever the entry >> containing the reference was read. > > Do you think the LDAP backend could/should handle this, or will Samba4 > have to do the GUID -> DN and DN -> GUID translations before passing > things to the backend? That would only be practical if you had LDAP transaction support.... It strikes me that the best approach is to just dynamically generate the memberOf values instead of storing them statically. But that also depends on your usage patterns. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ From abartlet at samba.org Thu Mar 20 00:17:09 2008 From: abartlet at samba.org (Andrew Bartlett) Date: Thu, 20 Mar 2008 11:17:09 +1100 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <47E1AB35.7050800@symas.com> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> Message-ID: <1205972229.14751.11.camel@naomi> On Wed, 2008-03-19 at 17:09 -0700, Howard Chu wrote: > Andrew Bartlett wrote: > > On Wed, 2008-03-19 at 04:26 -0700, Howard Chu wrote: > >> Most clients don't know or care that a particular change has side-effects. We > >> could introduce a config keyword to select synch vs asynch behavior here, but > >> I have a feeling that will still leave some group of users unhappy no matter > >> how you set it. > > > > Great. If run sync, will it error out correctly if I make an invalid > > modification (say target not present etc). > > No. We specifically don't validate the original modification here - the > server's number one job is always to try to do exactly what the client > specified. Also, you can't reliably detect an "invalid" name - it may simply > be the name of a non-local entry. This is going to get painful then... > >>> The fact that LDAP does not expose a transaction API > >> You mean draft-zeilenga-ldap-txn ? > > > > I suppose I should have said 'The free LDAP implemetnations I'm looking > > at don't expose a transaction API'. What did end up happening with > > that draft? > > It's stalled, like a lot of other things... It is partially implemented in > OpenLDAP. Eventually we'll have to complete the implementation, it's too > important to leave dangling. OK. > >>> Does anybody have any ideas or suggestions on how I could get around > >>> this? > >> There are other ways to guarantee consistency. The simplest approach is to > >> just not store one end of the linked attributes, and always generate them > >> dynamically when they're referenced. > >> > >> In the old Symas Connexitor EMS product we used (the equivalent of) a > >> UUIDAndOptionalName syntax for all references. In that case the DN was > >> essentially just window-dressing; we always used the ID to actually reference > >> entries and we updated the DNs whenever we found that they didn't match. As > >> such, referential integrity was pretty simple - you never had anything > >> pointing to the wrong entry; the worst that would happen is that you > >> occasionally had dangling references to deleted entries stored in the DB but > >> no one ever saw them because they were cleaned out whenever the entry > >> containing the reference was read. > > > > Do you think the LDAP backend could/should handle this, or will Samba4 > > have to do the GUID -> DN and DN -> GUID translations before passing > > things to the backend? > > That would only be practical if you had LDAP transaction support.... > > It strikes me that the best approach is to just dynamically generate the > memberOf values instead of storing them statically. But that also depends on > your usage patterns. Well, searches on member and memberOf are common, so it seems we would quickly find ourselves downloading the whole backend DB and filtering locally... I would hate to give up on this area of work - if only because I've put a lot of time into it, and it seems like a really useful way to use Samba4, but it also seems that I've found a rat-hole, with far more to it than I ever imagined. Any further thoughts on how we can resurrect this would be most appreciated. Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Red Hat Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From hyc at symas.com Thu Mar 20 00:33:24 2008 From: hyc at symas.com (Howard Chu) Date: Wed, 19 Mar 2008 17:33:24 -0700 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <1205972229.14751.11.camel@naomi> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> Message-ID: <47E1B0D4.9070102@symas.com> Andrew Bartlett wrote: > On Wed, 2008-03-19 at 17:09 -0700, Howard Chu wrote: >> It strikes me that the best approach is to just dynamically generate the >> memberOf values instead of storing them statically. But that also depends on >> your usage patterns. > > Well, searches on member and memberOf are common, so it seems we would > quickly find ourselves downloading the whole backend DB and filtering > locally... Searching on memberOf doesn't make a lot of sense to me, when you could simply read the group object directly. When is this actually a useful thing to do? An alternative would be to make the memberOf overlay intercept these filters and rewrite them in terms of member. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ From idra at samba.org Thu Mar 20 03:54:31 2008 From: idra at samba.org (simo) Date: Wed, 19 Mar 2008 23:54:31 -0400 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <47E1B0D4.9070102@symas.com> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> Message-ID: <1205985271.12916.61.camel@localhost.localdomain> On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote: > Andrew Bartlett wrote: > > On Wed, 2008-03-19 at 17:09 -0700, Howard Chu wrote: > >> It strikes me that the best approach is to just dynamically generate the > >> memberOf values instead of storing them statically. But that also depends on > >> your usage patterns. > > > > Well, searches on member and memberOf are common, so it seems we would > > quickly find ourselves downloading the whole backend DB and filtering > > locally... > > Searching on memberOf doesn't make a lot of sense to me, when you could simply > read the group object directly. When is this actually a useful thing to do? An > alternative would be to make the memberOf overlay intercept these filters and > rewrite them in terms of member. Premise: here I am thinking beyond what AD is doing as I use the memberOf concept in another project. >From my usage memberOf makes it very simple to find all the groups a member is part of even if that membership derives from nested grouping. It's very clear that most of the time you have an identity and you want to know what this Identity is part of, not the other way. So you really want to do a single search on one entry, rather than a huge search on the whole directory to find out (including local calculation for nesting) what groups include that identity as member, by parsing all groups one by one. It is as simple as that. Now for what concern the Samba4 problem, I think we should be more creative and first understand in which cases we might hit a problem with plugins like memberOf. I am sure some of these cases are just normal possible inconsistencies that can happen even in a normal AD server if you do many modifications at the same time. For these cases we just have to try not to make them more probable or problematic than what they are now. In other cases we might think of doing aggressive caching/prediction in our internal transactions. It might require some more work, but it could be a viable option, and also drive some more performance as dealing with an external LDAP is necessarily slower. Finally, if caching/prediction is not possible, we can think of writing overlays/slapi plugins directly for the LDAP server of choice be it OpenLDAP or Fedora Directory Server or anything else. This third option would require some more work and will be server specific, and perhaps involve some creative thinking wrt licensing, but it is certainly a viable option we should not discard. After all, these LDAP servers have a plugin system with defined APIs exactly to solve those problems that cannot be solved merely by external interaction. Simo. -- Simo Sorce Samba Team GPL Compliance Officer Senior Software Engineer at Red Hat Inc. From hyc at symas.com Fri Mar 21 12:40:33 2008 From: hyc at symas.com (Howard Chu) Date: Fri, 21 Mar 2008 05:40:33 -0700 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <1205985271.12916.61.camel@localhost.localdomain> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> Message-ID: <47E3ACC1.2020507@symas.com> simo wrote: > On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote: >> Searching on memberOf doesn't make a lot of sense to me, when you could simply >> read the group object directly. When is this actually a useful thing to do? An >> alternative would be to make the memberOf overlay intercept these filters and >> rewrite them in terms of member. > > Premise: here I am thinking beyond what AD is doing as I use the > memberOf concept in another project. > >> From my usage memberOf makes it very simple to find all the groups a > member is part of even if that membership derives from nested grouping. Ah, an interesting point, but probably a separate discussion. Note that the OpenLDAP memberOf overlay doesn't handle nested groups. > It's very clear that most of the time you have an identity and you want > to know what this Identity is part of, not the other way. No, not clear at all. A very common application of "groups" is for things like email lists. In that case, an MTA knows a specific group (the name of the email list), and needs to know all of the members. Other times (e.g. access control) you know an identity (current user) and want to know if the identity belongs to a particular group (for an authorization check). In that case, it is an equal amount of work to look in the user's entry for a memberOf value as to look in the group entry for the particular member value. In practice, because groups may be referenced frequently for multiple users, the group entries will be hot in the server cache and so the member lookup is actually cheaper. The frequency with which the question "what groups do I belong to" is asked is extremely low in most applications. The frequency with which the question "is XX a member of this group" is asked is very high in most applications. E.g., the libc initgroups() call needs to know what groups you belong to; that call typically occurs only once at the beginning of a login session. After that the result is essentially cached by the kernel. The result is cached in the kernel because the subsequent "is XX a member of group YY" questions happen so frequently as a process accesses system resources. > So you really want to do a single search on one entry, rather than a > huge search on the whole directory to find out (including local > calculation for nesting) what groups include that identity as member, by > parsing all groups one by one. > It is as simple as that. No, not simple at all. Yes, ideally you would like to be able to look in a single place and get the answer to "what privileges does user X have" but that doesn't actually mean what you're implying. In particular, the "single place" you're looking isn't necessarily that user's own entry. In most cases, that's the worst place to use because users generally have full write privileges to their own data, and the data comprising their set of privileges really belongs to the sysadmin, not to the individual user. Ideally, you write to the privilege set and read from the privilege set in the same way, in the same place. As an administrator, this simple consistency makes life easier. The memberOf concept is fundamentally broken if you actually rely on it for privilege determination because it is one step removed from how privileges are actually assigned by the sysadmin. (I.e., the sysadmin doesn't assign membership privileges to a user by writing to the memberOf attribute, therefore memberOf is not authoritative.) > Now for what concern the Samba4 problem, I think we should be more > creative and first understand in which cases we might hit a problem with > plugins like memberOf. I am sure some of these cases are just normal > possible inconsistencies that can happen even in a normal AD server if > you do many modifications at the same time. For these cases we just have > to try not to make them more probable or problematic than what they are > now. > In other cases we might think of doing aggressive caching/prediction in > our internal transactions. It might require some more work, but it could > be a viable option, and also drive some more performance as dealing with > an external LDAP is necessarily slower. > Finally, if caching/prediction is not possible, we can think of writing > overlays/slapi plugins directly for the LDAP server of choice be it > OpenLDAP or Fedora Directory Server or anything else. This third option > would require some more work and will be server specific, and perhaps > involve some creative thinking wrt licensing, but it is certainly a > viable option we should not discard. After all, these LDAP servers have > a plugin system with defined APIs exactly to solve those problems that > cannot be solved merely by external interaction. Agreed. And frankly, there's already an existence proof that this approach is viable. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ From lukeh at padl.com Fri Mar 21 22:41:10 2008 From: lukeh at padl.com (Luke Howard) Date: Sat, 22 Mar 2008 09:41:10 +1100 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <1206137972.12916.145.camel@localhost.localdomain> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> <1206137972.12916.145.camel@localhost.localdomain> Message-ID: <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> >>>> >> To me, this says that the directory does an internal group search to >> generate the isMemberOf attribute on the fly. I believe this is >> the way >> Active Directory handles the memberOf attribute as well. > > IIRC in AD memberOf is a linked attribute, stored permanently. From memory, AD stores the entry IDs of the link tuples in a separate table, so both "member" and "memberOf" are ostensibly generated on the fly. But this is really an implementation decision (although things do start to get interesting when dealing with references across partition boundaries). -- Luke From michael at stroeder.com Sat Mar 22 11:03:24 2008 From: michael at stroeder.com (=?ISO-8859-1?Q?Michael_Str=F6der?=) Date: Sat, 22 Mar 2008 12:03:24 +0100 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> <1206137972.12916.145.camel@localhost.localdomain> <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> Message-ID: Andrew Morgan wrote: > > I noticed that the memberOf tab in AD Users and Computers does not show > group membership for groups that are not in the same domain as the user. > Access controls were still correctly applied though, so those interfaces > must be enumerating group membership some other way. How about the 'tokenGroups' attribute? Samba would have to emulate this too. http://msdn2.microsoft.com/en-us/library/ms680275(VS.85).aspx Ciao, Michael. From lukeh at padl.com Sat Mar 22 15:25:09 2008 From: lukeh at padl.com (Luke Howard) Date: Sun, 23 Mar 2008 02:25:09 +1100 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> <1206137972.12916.145.camel@localhost.localdomain> <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> Message-ID: <09E067FB-14C0-4273-BA91-4916AA2AA38E@padl.com> On 22/03/2008, at 10:17 AM, Andrew Morgan wrote: > On Sat, 22 Mar 2008, Luke Howard wrote: > >>>> To me, this says that the directory does an internal group search >>>> to >>>> generate the isMemberOf attribute on the fly. I believe this is >>>> the way >>>> Active Directory handles the memberOf attribute as well. >>> IIRC in AD memberOf is a linked attribute, stored permanently. >> >> >> From memory, AD stores the entry IDs of the link tuples in a >> separate table, so both "member" and "memberOf" are ostensibly >> generated on the fly. But this is really an implementation decision >> (although things do start to get interesting when dealing with >> references across partition boundaries). > > I noticed that the memberOf tab in AD Users and Computers does not > show group membership for groups that are not in the same domain as > the user. Access controls were still correctly applied though, so > those interfaces must be enumerating group membership some other way. Right, this is because it's expensive (I think they'll show such memberships if you query through the GC port [3268] but that requires you talk to a GC). When a non-GC KDC builds a PAC, it will contact a GC over RPC to complete the user's token (cf. IDL_DRSGetMemberships). -- Luke From lukeh at padl.com Sun Mar 23 06:17:34 2008 From: lukeh at padl.com (Luke Howard) Date: Sun, 23 Mar 2008 17:17:34 +1100 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <09E067FB-14C0-4273-BA91-4916AA2AA38E@padl.com> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> <1206137972.12916.145.camel@localhost.localdomain> <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> <09E067FB-14C0-4273-BA91-4916AA2AA38E@padl.com> Message-ID: <99D111C5-1BD9-456D-B9F5-28B28850A083@padl.com> FYI here's how we did it in XAD, page 19ff. http://www.openldap.org/conf/odd-wien-2003/luke.pdf Implemented in OpenLDAP completely at the SLAPI layer. Without help from the underlying DB, though, it wasn't transaction safe; so the approach of storing "member" and computing "memberOf" is probably better. -- Luke On 23/03/2008, at 2:25 AM, Luke Howard wrote: > > On 22/03/2008, at 10:17 AM, Andrew Morgan wrote: >> On Sat, 22 Mar 2008, Luke Howard wrote: >> >>>>> To me, this says that the directory does an internal group >>>>> search to >>>>> generate the isMemberOf attribute on the fly. I believe this is >>>>> the way >>>>> Active Directory handles the memberOf attribute as well. >>>> IIRC in AD memberOf is a linked attribute, stored permanently. >>> >>> >>> From memory, AD stores the entry IDs of the link tuples in a >>> separate table, so both "member" and "memberOf" are ostensibly >>> generated on the fly. But this is really an implementation >>> decision (although things do start to get interesting when dealing >>> with references across partition boundaries). >> >> I noticed that the memberOf tab in AD Users and Computers does not >> show group membership for groups that are not in the same domain as >> the user. Access controls were still correctly applied though, so >> those interfaces must be enumerating group membership some other way. > > Right, this is because it's expensive (I think they'll show such > memberships if you query through the GC port [3268] but that > requires you talk to a GC). > > When a non-GC KDC builds a PAC, it will contact a GC over RPC to > complete the user's token (cf. IDL_DRSGetMemberships). > > -- Luke > > -- > Fedora-directory-devel mailing list > Fedora-directory-devel at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-devel > -- www.padl.com | www.fghr.net From lukeh at padl.com Fri Mar 21 22:41:10 2008 From: lukeh at padl.com (Luke Howard) Date: Sat, 22 Mar 2008 09:41:10 +1100 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <1206137972.12916.145.camel@localhost.localdomain> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> <1206137972.12916.145.camel@localhost.localdomain> Message-ID: <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> >>>> >> To me, this says that the directory does an internal group search to >> generate the isMemberOf attribute on the fly. I believe this is >> the way >> Active Directory handles the memberOf attribute as well. > > IIRC in AD memberOf is a linked attribute, stored permanently. From memory, AD stores the entry IDs of the link tuples in a separate table, so both "member" and "memberOf" are ostensibly generated on the fly. But this is really an implementation decision (although things do start to get interesting when dealing with references across partition boundaries). -- Luke From andrey.ivanov at polytechnique.fr Sun Mar 23 22:36:03 2008 From: andrey.ivanov at polytechnique.fr (Andrey Ivanov) Date: Sun, 23 Mar 2008 23:36:03 +0100 Subject: [Fedora-directory-devel] memberOf plug-in memory leaks and bugs, proposals of patch Message-ID: <1601b8650803231536n28f67a10q9af194e2e4c5d188@mail.gmail.com> Hi, I have tried to adapt the memberOf plugin as it exists in today in the CVS to our production environment - it is a very nice and USEFUL addition to server features, especially if one has to manage direct and indirect groups. We have several relatively large groups (~ 3000 entries). It turned out that at each full group members delete and recreation I've had huge memory leaks (~ 100 Mb per each full delete/recreation). I've tried to narrow it down in the plug-in. here is what i have found (i structured it in paragraphs) : ?1.There are several slapi_search_internal_get_entry() calls that return a copy of Slapi_Entry *e that is never released by slapi_entry_free(). Here are the proposed changes : --- /tmp/fedora-ds-base-cvs/ldap/servers/plugins/memberof/memberof.c 2008-02-19 07:04:56.000000000 +0100 +++ /tmp/memberof-patched.c 2008-03-23 22:13:00.000000000 +0100 @@ -1019,6 +1019,7 @@ } bail: + slapi_entry_free(e); return rc; } @@ -1310,6 +1311,7 @@ } slapi_sdn_free(&sdn); + slapi_entry_free(group_e); return rc; } @@ -1826,6 +1828,8 @@ bail: slapi_ch_free_string(&filter_str); + slapi_entry_free(group_e); + slapi_entry_free(opto_e); slapi_log_error( SLAPI_LOG_TRACE, MEMBEROF_PLUGIN_SUBSYSTEM, "<-- memberof_is_legit_member\n" ); ?2. I don't know whether the arrays pre_array and post_array in memberofd_replace_list() function should be freed. I think they should because they are allocated by slapi_ch_malloc(), but i'm not sure and i don't know how i should free them - i am not a specialist of the plugin API. Anyway, apparently this part of code is never used by our group management applications so i don't have a memory leak caused by this function. ?3. If we make MODRDN on an INDIRECT group the original indirect group DN is deleted from memberof but the renamed indirect group DN is not added. It may be fixed by running every once in a while the memberOf fixup task BUT in this particular case the attribute will never be fixed - because of the problem in ?4 : ?4. I propose to delete all the memberOf attribute values at the beginning of the fix thread. Otherwise if we have in the entry a "memberOf" attribute value equal to a DN of a direct group, the memeberOf fix task (the function memberof_fix_memberof_callback()) will NEVER add the indirect groupsDNs that depend on the abovementioned direct group, so this thread is not really a FULL FIX thread :) Here are the proposed changes to avoid it: --- /tmp/fedora-ds-base-cvs/ldap/servers/plugins/memberof/memberof.c 2008-02-19 07:04:56.000000000 +0100 +++ /tmp/memberof-patched.c 2008-03-23 22:13:00.000000000 +0100 @@ -2006,6 +2010,7 @@ /* memberof_fix_memberof_callback() * Add initial and/or fix up broken group list in entry * + * 0. Delete all the memberOf attribute values * 1. Make sure direct membership groups are in the entry * 2. Add all groups that current group list allows through nested membership * 3. Trim groups that have no relationship to entry @@ -2016,6 +2021,9 @@ char *dn = slapi_entry_get_dn(e); memberof_add_groups data = {dn, dn}; + /* step 0. */ + slapi_entry_attr_delete(e, MEMBEROF_ATTR); + /* step 1. and step 2. */ rc = memberof_call_foreach_dn(0, dn, MEMBEROF_GROUP_ATTR, memberof_add_groups_search_callback, &data); ?5. After the code changes proposed in "?1" i still have ~600 bytes memory increase per 5000 modifications (2500 DEL & 2500 ADDs to group membership, the memory increases mainly at DEL operations). I don't know whether it is due to connection management in slapd code or to the plug-in itself but it is a much more acceptable value for me than 100Mb :) ?6. #define MEMBEROF_GROUP_ATTR_TYPE "uid" is declared but never used in the plug-in code... Thanks once more for this plug-in! From idra at samba.org Fri Mar 21 14:36:48 2008 From: idra at samba.org (simo) Date: Fri, 21 Mar 2008 10:36:48 -0400 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <47E3ACC1.2020507@symas.com> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> Message-ID: <1206110208.12916.113.camel@localhost.localdomain> On Fri, 2008-03-21 at 05:40 -0700, Howard Chu wrote: > simo wrote: > > On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote: > >> Searching on memberOf doesn't make a lot of sense to me, when you could simply > >> read the group object directly. When is this actually a useful thing to do? An > >> alternative would be to make the memberOf overlay intercept these filters and > >> rewrite them in terms of member. > > > > Premise: here I am thinking beyond what AD is doing as I use the > > memberOf concept in another project. > > > >> From my usage memberOf makes it very simple to find all the groups a > > member is part of even if that membership derives from nested grouping. > > Ah, an interesting point, but probably a separate discussion. Note that the > OpenLDAP memberOf overlay doesn't handle nested groups. Interesting. To be honest, if memberOf does not solve the nesting group problem, I find it rather useless, and that may explain your comments below. > > It's very clear that most of the time you have an identity and you want > > to know what this Identity is part of, not the other way. > > No, not clear at all. A very common application of "groups" is for things like > email lists. In that case, an MTA knows a specific group (the name of the > email list), and needs to know all of the members. My bad, I should have specified that this is the case I meet/care in my work. It is not, in general, the most common case, I agree. > Other times (e.g. access control) you know an identity (current user) and want > to know if the identity belongs to a particular group (for an authorization > check). In that case, it is an equal amount of work to look in the user's > entry for a memberOf value as to look in the group entry for the particular > member value. In practice, because groups may be referenced frequently for > multiple users, the group entries will be hot in the server cache and so the > member lookup is actually cheaper. This is true only if you do member checking by looking at groups. But if you don't they won't, so I am not sure this really is a reason unless there is another reason groups are frequently checked. But given the fact that groups may have huge memberships, actually wasting a lot of cache memory to keep around the members could also be a reason not to do it that way. Of course this is just cheap talk of me, without numbers related to a particular use case (and the use case here is more identity management, not much MTAs or such, even if, in the end, the same Directory can be used for many applications, so groups can actually be really in cache already). > The frequency with which the question "what groups do I belong to" is asked is > extremely low in most applications. The frequency with which the question "is > XX a member of this group" is asked is very high in most applications. E.g., > the libc initgroups() call needs to know what groups you belong to; that call > typically occurs only once at the beginning of a login session. After that the > result is essentially cached by the kernel. The result is cached in the kernel > because the subsequent "is XX a member of group YY" questions happen so > frequently as a process accesses system resources. Well, initgroups() is a special case, it asks for all groups. And I agree that the question: "is XX a member of group YY" is the most common. > > So you really want to do a single search on one entry, rather than a > > huge search on the whole directory to find out (including local > > calculation for nesting) what groups include that identity as member, by > > parsing all groups one by one. > > > It is as simple as that. > > No, not simple at all. Points of view :-) > Yes, ideally you would like to be able to look in a single place and get the > answer to "what privileges does user X have" but that doesn't actually mean > what you're implying. Checking privileges is a whole different business, you still have the "is XX a member of group YY" to solve before that. > In particular, the "single place" you're looking isn't > necessarily that user's own entry. In most cases, that's the worst place to > use because users generally have full write privileges to their own data, and > the data comprising their set of privileges really belongs to the sysadmin, > not to the individual user. Well obviously memberOf need to be protected from user manipulation. Esp. in Identity Management applications user entries are usually very well constrained as they hold other information about the user that the user is not supposed to be able to modify. That's not a big problem, we have ACIs just for that. > Ideally, you write to the privilege set and read from the privilege set in the > same way, in the same place. This may be theoretically a sound principle, but in this case I think it is not as ideal as it sounds. > As an administrator, this simple consistency makes life easier. As long as the Admin knows how the mechanism work, I honestly see no problem, and you can still just query the members of a group when member/memberOf is in use. > The memberOf concept is fundamentally broken if you > actually rely on it for privilege determination because it is one step removed > from how privileges are actually assigned by the sysadmin. (I.e., the sysadmin > doesn't assign membership privileges to a user by writing to the memberOf > attribute, therefore memberOf is not authoritative.) I do not agree with the "fundamentally broken", I think that's exaggeration. Esp. when memberOf solves the "unrolling nested memberships" problem, it actually is, I'd say it is "fundamentally better" than requiring clients to do it and for 2 reasons: 1.) The server choose what nested groups mean, if you want to change that mechanism you are free to do it without having to care what legacy clients may be doing. *** <-- This is *extremely* important IMHO *** 1.1.) You also do not have to beg client app implementers to fix bugs, or change the code if you need to change something, and as group membership can be critical it means you make it simpler for client apps not to do the wrong thing. 2.) The operation is done only once, and is verifiable for the admin. Nested groups can be very complicated to grok, but, by querying memberOf, the Administrator can empirically check that what he did actually lead to the expected results. So in this sense I see memberOf as a real aid for admins. Of course all this discussion is almost meaningless if you do not support group nesting which is, for me, the whole point of using memberOf. Finally, while I agree that, technically, memberOf is not authoritative, as what admins change is really the groups's "member" attribute, I fail to see how this can be a problem unless you have a buggy implementation. Memberships do not change often enough to consider the mechanism not worth of trust imo, and there are many ways to implement the memberOf attribute, you can even make it a virtual attribute generated by a search, but this would add a substantial cost if you consider nested groups, so you would very quick end up caching it aggressively which is actually what calculating and storing it ends up being. If you think of memberOf as a cached search that tells you the user's set of groups (including nested...) I think it makes a lot of sense. > > Now for what concern the Samba4 problem, I think we should be more > > creative and first understand in which cases we might hit a problem with > > plugins like memberOf. I am sure some of these cases are just normal > > possible inconsistencies that can happen even in a normal AD server if > > you do many modifications at the same time. For these cases we just have > > to try not to make them more probable or problematic than what they are > > now. > > In other cases we might think of doing aggressive caching/prediction in > > our internal transactions. It might require some more work, but it could > > be a viable option, and also drive some more performance as dealing with > > an external LDAP is necessarily slower. > > Finally, if caching/prediction is not possible, we can think of writing > > overlays/slapi plugins directly for the LDAP server of choice be it > > OpenLDAP or Fedora Directory Server or anything else. This third option > > would require some more work and will be server specific, and perhaps > > involve some creative thinking wrt licensing, but it is certainly a > > viable option we should not discard. After all, these LDAP servers have > > a plugin system with defined APIs exactly to solve those problems that > > cannot be solved merely by external interaction. > > Agreed. And frankly, there's already an existence proof that this approach is > viable. Indeed. Simo. -- Simo Sorce Samba Team GPL Compliance Officer Senior Software Engineer at Red Hat Inc. From idra at samba.org Fri Mar 21 22:19:32 2008 From: idra at samba.org (simo) Date: Fri, 21 Mar 2008 18:19:32 -0400 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> Message-ID: <1206137972.12916.145.camel@localhost.localdomain> On Fri, 2008-03-21 at 10:51 -0700, Andrew Morgan wrote: > On Fri, 21 Mar 2008, simo wrote: > > > On Fri, 2008-03-21 at 05:40 -0700, Howard Chu wrote: > >> simo wrote: > >>> On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote: > >>>> Searching on memberOf doesn't make a lot of sense to me, when you could simply > >>>> read the group object directly. When is this actually a useful thing to do? An > >>>> alternative would be to make the memberOf overlay intercept these filters and > >>>> rewrite them in terms of member. > >>> > >>> Premise: here I am thinking beyond what AD is doing as I use the > >>> memberOf concept in another project. > >>> > >>>> From my usage memberOf makes it very simple to find all the groups a > >>> member is part of even if that membership derives from nested grouping. > >> > >> Ah, an interesting point, but probably a separate discussion. Note that the > >> OpenLDAP memberOf overlay doesn't handle nested groups. > > > > Interesting. > > To be honest, if memberOf does not solve the nesting group problem, I > > find it rather useless, and that may explain your comments below. > > > >>> It's very clear that most of the time you have an identity and you want > >>> to know what this Identity is part of, not the other way. > >> > >> No, not clear at all. A very common application of "groups" is for things like > >> email lists. In that case, an MTA knows a specific group (the name of the > >> email list), and needs to know all of the members. > > > > My bad, I should have specified that this is the case I meet/care in my > > work. It is not, in general, the most common case, I agree. > > > >> Other times (e.g. access control) you know an identity (current user) and want > >> to know if the identity belongs to a particular group (for an authorization > >> check). In that case, it is an equal amount of work to look in the user's > >> entry for a memberOf value as to look in the group entry for the particular > >> member value. In practice, because groups may be referenced frequently for > >> multiple users, the group entries will be hot in the server cache and so the > >> member lookup is actually cheaper. > > > > This is true only if you do member checking by looking at groups. But if > > you don't they won't, so I am not sure this really is a reason unless > > there is another reason groups are frequently checked. But given the > > fact that groups may have huge memberships, actually wasting a lot of > > cache memory to keep around the members could also be a reason not to do > > it that way. Of course this is just cheap talk of me, without numbers > > related to a particular use case (and the use case here is more identity > > management, not much MTAs or such, even if, in the end, the same > > Directory can be used for many applications, so groups can actually be > > really in cache already). > > We run a medium sized directory here at Oregon State University which > contains user objects for all students, staff, and faculty (approximately > 35,000 entries). It also contains a few static groups (not nested) for > categorizing users. We are using Sun Directory Server 5.2. > > The documentation for Sun Directory Server 6.2 says: > > The isMemberOf attribute is calculated and added to the user entry at > the start of the search. It is then removed again after the search has > finished. This functionality provides easy management of groups, and > fast read access. Hmm this seem just a description of a virtual attribute. > To me, this says that the directory does an internal group search to > generate the isMemberOf attribute on the fly. I believe this is the way > Active Directory handles the memberOf attribute as well. IIRC in AD memberOf is a linked attribute, stored permanently. > Most directory servers suggest tuning the caches so that the entire > directory fits in RAM, which isn't too hard these days with the price of > RAM. Directory entries are typically only a few kilobytes each. Yes it is a good idea to do so. > From a performance standpoint, it may not matter which method of group > enumeration you use. Directory servers are optimized for reads, and a > simple LDAP query like: > > (&(objectclass=groupofuniquenames)(uniquemember=uid=doej)) > > can be just as fast as: > > (&(objectclass=posixaccount)(uid=doej)) It matters if you start thinking of nested groups as in that case the number of searches may increase considerably, although, if all the data is in cache it may not have such a big impact depending on the frequency of such searches. > I'm not familiar with the background of this whole discussion. It sounds > like the memberOf/isMemberOf implementations are relatively new. Unless > you want Samba to require the latest and greatest versions of these > directory servers, you may need to avoid memberOf/isMemberOf and use the > standard methods of querying group membership. Samba4 has very peculiar needs as the aim is to provide an AD compatible interface to the clients. This is because MS clients are tightly coupled and expect certain behaviors that are AD specific. We can't just run a generic directory in this case, we either proxy and translate to a generic directory or we heavily modify one. So far we have basically worked on 2 fronts. One was to make our own LDAP implementation based on our LDB database so that we could freely experiment and discover exactly what the clients need/want. At the same side we developed LDB to be a proxy to a backend directory and created a module called ldb_map that could do translation on the fly. This second approach is showing the inherent limits it was clear we would have met since we started. The point is to decide to what degree of compatibility we need to get to. Personally I am ok with just Windows clients being able to join using Kerberos and our Directory, even if some Windows admin tools don't work. We have reached this point some time ago already, it is just a matter of deciding when we are compatible enough to be satisfied, and start stabilizing the code. Simo. -- Simo Sorce Samba Team GPL Compliance Officer Senior Software Engineer at Red Hat Inc. From idra at samba.org Sat Mar 22 04:01:11 2008 From: idra at samba.org (simo) Date: Sat, 22 Mar 2008 00:01:11 -0400 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> <1206137972.12916.145.camel@localhost.localdomain> <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> Message-ID: <1206158471.12916.149.camel@localhost.localdomain> On Fri, 2008-03-21 at 16:17 -0700, Andrew Morgan wrote: > On Sat, 22 Mar 2008, Luke Howard wrote: > > >>>>> > >>> To me, this says that the directory does an internal group search to > >>> generate the isMemberOf attribute on the fly. I believe this is the way > >>> Active Directory handles the memberOf attribute as well. > >> > >> IIRC in AD memberOf is a linked attribute, stored permanently. > > > > > > From memory, AD stores the entry IDs of the link tuples in a separate table, > > so both "member" and "memberOf" are ostensibly generated on the fly. But this > > is really an implementation decision (although things do start to get > > interesting when dealing with references across partition boundaries). > > I noticed that the memberOf tab in AD Users and Computers does not show > group membership for groups that are not in the same domain as the user. > Access controls were still correctly applied though, so those interfaces > must be enumerating group membership some other way. AD clients get user memberships from the PAC (the kerberos extension) when the user authenticates. Windows clients never do LDAP calls to get group membership of a user. > I've always viewed the memberOf attribute as a convenient tool for human > use, but not something I would use programmatically. All the benefits > provided by memberOf can be attained other ways by the LDAP client, > although the LDAP client may need to know more about the directory > structure to do it. This is my point in using memberOf, less knowledge need to be put in the client the better. Simo. -- Simo Sorce Samba Team GPL Compliance Officer Senior Software Engineer at Red Hat Inc. From idra at samba.org Fri Mar 21 22:19:32 2008 From: idra at samba.org (simo) Date: Fri, 21 Mar 2008 18:19:32 -0400 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> Message-ID: <1206137972.12916.145.camel@localhost.localdomain> On Fri, 2008-03-21 at 10:51 -0700, Andrew Morgan wrote: > On Fri, 21 Mar 2008, simo wrote: > > > On Fri, 2008-03-21 at 05:40 -0700, Howard Chu wrote: > >> simo wrote: > >>> On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote: > >>>> Searching on memberOf doesn't make a lot of sense to me, when you could simply > >>>> read the group object directly. When is this actually a useful thing to do? An > >>>> alternative would be to make the memberOf overlay intercept these filters and > >>>> rewrite them in terms of member. > >>> > >>> Premise: here I am thinking beyond what AD is doing as I use the > >>> memberOf concept in another project. > >>> > >>>> From my usage memberOf makes it very simple to find all the groups a > >>> member is part of even if that membership derives from nested grouping. > >> > >> Ah, an interesting point, but probably a separate discussion. Note that the > >> OpenLDAP memberOf overlay doesn't handle nested groups. > > > > Interesting. > > To be honest, if memberOf does not solve the nesting group problem, I > > find it rather useless, and that may explain your comments below. > > > >>> It's very clear that most of the time you have an identity and you want > >>> to know what this Identity is part of, not the other way. > >> > >> No, not clear at all. A very common application of "groups" is for things like > >> email lists. In that case, an MTA knows a specific group (the name of the > >> email list), and needs to know all of the members. > > > > My bad, I should have specified that this is the case I meet/care in my > > work. It is not, in general, the most common case, I agree. > > > >> Other times (e.g. access control) you know an identity (current user) and want > >> to know if the identity belongs to a particular group (for an authorization > >> check). In that case, it is an equal amount of work to look in the user's > >> entry for a memberOf value as to look in the group entry for the particular > >> member value. In practice, because groups may be referenced frequently for > >> multiple users, the group entries will be hot in the server cache and so the > >> member lookup is actually cheaper. > > > > This is true only if you do member checking by looking at groups. But if > > you don't they won't, so I am not sure this really is a reason unless > > there is another reason groups are frequently checked. But given the > > fact that groups may have huge memberships, actually wasting a lot of > > cache memory to keep around the members could also be a reason not to do > > it that way. Of course this is just cheap talk of me, without numbers > > related to a particular use case (and the use case here is more identity > > management, not much MTAs or such, even if, in the end, the same > > Directory can be used for many applications, so groups can actually be > > really in cache already). > > We run a medium sized directory here at Oregon State University which > contains user objects for all students, staff, and faculty (approximately > 35,000 entries). It also contains a few static groups (not nested) for > categorizing users. We are using Sun Directory Server 5.2. > > The documentation for Sun Directory Server 6.2 says: > > The isMemberOf attribute is calculated and added to the user entry at > the start of the search. It is then removed again after the search has > finished. This functionality provides easy management of groups, and > fast read access. Hmm this seem just a description of a virtual attribute. > To me, this says that the directory does an internal group search to > generate the isMemberOf attribute on the fly. I believe this is the way > Active Directory handles the memberOf attribute as well. IIRC in AD memberOf is a linked attribute, stored permanently. > Most directory servers suggest tuning the caches so that the entire > directory fits in RAM, which isn't too hard these days with the price of > RAM. Directory entries are typically only a few kilobytes each. Yes it is a good idea to do so. > From a performance standpoint, it may not matter which method of group > enumeration you use. Directory servers are optimized for reads, and a > simple LDAP query like: > > (&(objectclass=groupofuniquenames)(uniquemember=uid=doej)) > > can be just as fast as: > > (&(objectclass=posixaccount)(uid=doej)) It matters if you start thinking of nested groups as in that case the number of searches may increase considerably, although, if all the data is in cache it may not have such a big impact depending on the frequency of such searches. > I'm not familiar with the background of this whole discussion. It sounds > like the memberOf/isMemberOf implementations are relatively new. Unless > you want Samba to require the latest and greatest versions of these > directory servers, you may need to avoid memberOf/isMemberOf and use the > standard methods of querying group membership. Samba4 has very peculiar needs as the aim is to provide an AD compatible interface to the clients. This is because MS clients are tightly coupled and expect certain behaviors that are AD specific. We can't just run a generic directory in this case, we either proxy and translate to a generic directory or we heavily modify one. So far we have basically worked on 2 fronts. One was to make our own LDAP implementation based on our LDB database so that we could freely experiment and discover exactly what the clients need/want. At the same side we developed LDB to be a proxy to a backend directory and created a module called ldb_map that could do translation on the fly. This second approach is showing the inherent limits it was clear we would have met since we started. The point is to decide to what degree of compatibility we need to get to. Personally I am ok with just Windows clients being able to join using Kerberos and our Directory, even if some Windows admin tools don't work. We have reached this point some time ago already, it is just a matter of deciding when we are compatible enough to be satisfied, and start stabilizing the code. Simo. -- Simo Sorce Samba Team GPL Compliance Officer Senior Software Engineer at Red Hat Inc. From idra at samba.org Sat Mar 22 04:01:11 2008 From: idra at samba.org (simo) Date: Sat, 22 Mar 2008 00:01:11 -0400 Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> <1206137972.12916.145.camel@localhost.localdomain> <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> Message-ID: <1206158471.12916.149.camel@localhost.localdomain> On Fri, 2008-03-21 at 16:17 -0700, Andrew Morgan wrote: > On Sat, 22 Mar 2008, Luke Howard wrote: > > >>>>> > >>> To me, this says that the directory does an internal group search to > >>> generate the isMemberOf attribute on the fly. I believe this is the way > >>> Active Directory handles the memberOf attribute as well. > >> > >> IIRC in AD memberOf is a linked attribute, stored permanently. > > > > > > From memory, AD stores the entry IDs of the link tuples in a separate table, > > so both "member" and "memberOf" are ostensibly generated on the fly. But this > > is really an implementation decision (although things do start to get > > interesting when dealing with references across partition boundaries). > > I noticed that the memberOf tab in AD Users and Computers does not show > group membership for groups that are not in the same domain as the user. > Access controls were still correctly applied though, so those interfaces > must be enumerating group membership some other way. AD clients get user memberships from the PAC (the kerberos extension) when the user authenticates. Windows clients never do LDAP calls to get group membership of a user. > I've always viewed the memberOf attribute as a convenient tool for human > use, but not something I would use programmatically. All the benefits > provided by memberOf can be attained other ways by the LDAP client, > although the LDAP client may need to know more about the directory > structure to do it. This is my point in using memberOf, less knowledge need to be put in the client the better. Simo. -- Simo Sorce Samba Team GPL Compliance Officer Senior Software Engineer at Red Hat Inc. From morgan at orst.edu Fri Mar 21 17:51:03 2008 From: morgan at orst.edu (Andrew Morgan) Date: Fri, 21 Mar 2008 10:51:03 -0700 (PDT) Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <1206110208.12916.113.camel@localhost.localdomain> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> Message-ID: On Fri, 21 Mar 2008, simo wrote: > On Fri, 2008-03-21 at 05:40 -0700, Howard Chu wrote: >> simo wrote: >>> On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote: >>>> Searching on memberOf doesn't make a lot of sense to me, when you could simply >>>> read the group object directly. When is this actually a useful thing to do? An >>>> alternative would be to make the memberOf overlay intercept these filters and >>>> rewrite them in terms of member. >>> >>> Premise: here I am thinking beyond what AD is doing as I use the >>> memberOf concept in another project. >>> >>>> From my usage memberOf makes it very simple to find all the groups a >>> member is part of even if that membership derives from nested grouping. >> >> Ah, an interesting point, but probably a separate discussion. Note that the >> OpenLDAP memberOf overlay doesn't handle nested groups. > > Interesting. > To be honest, if memberOf does not solve the nesting group problem, I > find it rather useless, and that may explain your comments below. > >>> It's very clear that most of the time you have an identity and you want >>> to know what this Identity is part of, not the other way. >> >> No, not clear at all. A very common application of "groups" is for things like >> email lists. In that case, an MTA knows a specific group (the name of the >> email list), and needs to know all of the members. > > My bad, I should have specified that this is the case I meet/care in my > work. It is not, in general, the most common case, I agree. > >> Other times (e.g. access control) you know an identity (current user) and want >> to know if the identity belongs to a particular group (for an authorization >> check). In that case, it is an equal amount of work to look in the user's >> entry for a memberOf value as to look in the group entry for the particular >> member value. In practice, because groups may be referenced frequently for >> multiple users, the group entries will be hot in the server cache and so the >> member lookup is actually cheaper. > > This is true only if you do member checking by looking at groups. But if > you don't they won't, so I am not sure this really is a reason unless > there is another reason groups are frequently checked. But given the > fact that groups may have huge memberships, actually wasting a lot of > cache memory to keep around the members could also be a reason not to do > it that way. Of course this is just cheap talk of me, without numbers > related to a particular use case (and the use case here is more identity > management, not much MTAs or such, even if, in the end, the same > Directory can be used for many applications, so groups can actually be > really in cache already). We run a medium sized directory here at Oregon State University which contains user objects for all students, staff, and faculty (approximately 35,000 entries). It also contains a few static groups (not nested) for categorizing users. We are using Sun Directory Server 5.2. The documentation for Sun Directory Server 6.2 says: The isMemberOf attribute is calculated and added to the user entry at the start of the search. It is then removed again after the search has finished. This functionality provides easy management of groups, and fast read access. To me, this says that the directory does an internal group search to generate the isMemberOf attribute on the fly. I believe this is the way Active Directory handles the memberOf attribute as well. Most directory servers suggest tuning the caches so that the entire directory fits in RAM, which isn't too hard these days with the price of RAM. Directory entries are typically only a few kilobytes each. >From a performance standpoint, it may not matter which method of group enumeration you use. Directory servers are optimized for reads, and a simple LDAP query like: (&(objectclass=groupofuniquenames)(uniquemember=uid=doej)) can be just as fast as: (&(objectclass=posixaccount)(uid=doej)) I'm not familiar with the background of this whole discussion. It sounds like the memberOf/isMemberOf implementations are relatively new. Unless you want Samba to require the latest and greatest versions of these directory servers, you may need to avoid memberOf/isMemberOf and use the standard methods of querying group membership. Andy From morgan at orst.edu Fri Mar 21 23:17:17 2008 From: morgan at orst.edu (Andrew Morgan) Date: Fri, 21 Mar 2008 16:17:17 -0700 (PDT) Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> <1206137972.12916.145.camel@localhost.localdomain> <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> Message-ID: On Sat, 22 Mar 2008, Luke Howard wrote: >>>>> >>> To me, this says that the directory does an internal group search to >>> generate the isMemberOf attribute on the fly. I believe this is the way >>> Active Directory handles the memberOf attribute as well. >> >> IIRC in AD memberOf is a linked attribute, stored permanently. > > > From memory, AD stores the entry IDs of the link tuples in a separate table, > so both "member" and "memberOf" are ostensibly generated on the fly. But this > is really an implementation decision (although things do start to get > interesting when dealing with references across partition boundaries). I noticed that the memberOf tab in AD Users and Computers does not show group membership for groups that are not in the same domain as the user. Access controls were still correctly applied though, so those interfaces must be enumerating group membership some other way. I've always viewed the memberOf attribute as a convenient tool for human use, but not something I would use programmatically. All the benefits provided by memberOf can be attained other ways by the LDAP client, although the LDAP client may need to know more about the directory structure to do it. Andy From morgan at orst.edu Fri Mar 21 23:17:17 2008 From: morgan at orst.edu (Andrew Morgan) Date: Fri, 21 Mar 2008 16:17:17 -0700 (PDT) Subject: [Fedora-directory-devel] Re: Problems creating a Samba4 LDAP Backend In-Reply-To: <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> References: <1205921119.22616.52.camel@naomi> <47E0F877.10103@symas.com> <1205968982.14751.3.camel@naomi> <47E1AB35.7050800@symas.com> <1205972229.14751.11.camel@naomi> <47E1B0D4.9070102@symas.com> <1205985271.12916.61.camel@localhost.localdomain> <47E3ACC1.2020507@symas.com> <1206110208.12916.113.camel@localhost.localdomain> <1206137972.12916.145.camel@localhost.localdomain> <88833F6A-C677-4521-9C17-5840BD8AF36C@padl.com> Message-ID: On Sat, 22 Mar 2008, Luke Howard wrote: >>>>> >>> To me, this says that the directory does an internal group search to >>> generate the isMemberOf attribute on the fly. I believe this is the way >>> Active Directory handles the memberOf attribute as well. >> >> IIRC in AD memberOf is a linked attribute, stored permanently. > > > From memory, AD stores the entry IDs of the link tuples in a separate table, > so both "member" and "memberOf" are ostensibly generated on the fly. But this > is really an implementation decision (although things do start to get > interesting when dealing with references across partition boundaries). I noticed that the memberOf tab in AD Users and Computers does not show group membership for groups that are not in the same domain as the user. Access controls were still correctly applied though, so those interfaces must be enumerating group membership some other way. I've always viewed the memberOf attribute as a convenient tool for human use, but not something I would use programmatically. All the benefits provided by memberOf can be attained other ways by the LDAP client, although the LDAP client may need to know more about the directory structure to do it. Andy From nkinder at redhat.com Mon Mar 24 16:09:17 2008 From: nkinder at redhat.com (Nathan Kinder) Date: Mon, 24 Mar 2008 09:09:17 -0700 Subject: [Fedora-directory-devel] Re: memberOf plug-in memory leaks and bugs, proposals of patch In-Reply-To: <1601b8650803231536n28f67a10q9af194e2e4c5d188@mail.gmail.com> References: <1601b8650803231536n28f67a10q9af194e2e4c5d188@mail.gmail.com> Message-ID: <47E7D22D.2000200@redhat.com> Andrey Ivanov wrote: > Hi, > > I have tried to adapt the memberOf plugin as it exists in today in > the CVS to our production environment - it is a very nice and USEFUL > addition to server features, especially if one has to manage direct > and indirect groups. > > We have several relatively large groups (~ 3000 entries). It turned > out that at each full group members delete and recreation I've had > huge memory leaks (~ 100 Mb per each full delete/recreation). I've > tried to narrow it down in the plug-in. here is what i have found (i > structured it in paragraphs) : > > ?1.There are several slapi_search_internal_get_entry() calls that > return a copy of Slapi_Entry *e that is never released by > slapi_entry_free(). Here are the proposed changes : > --- /tmp/fedora-ds-base-cvs/ldap/servers/plugins/memberof/memberof.c > 2008-02-19 07:04:56.000000000 +0100 > +++ /tmp/memberof-patched.c 2008-03-23 22:13:00.000000000 +0100 > @@ -1019,6 +1019,7 @@ > } > > bail: > + slapi_entry_free(e); > return rc; > } > > @@ -1310,6 +1311,7 @@ > } > > slapi_sdn_free(&sdn); > + slapi_entry_free(group_e); > return rc; > } > > @@ -1826,6 +1828,8 @@ > > bail: > slapi_ch_free_string(&filter_str); > + slapi_entry_free(group_e); > + slapi_entry_free(opto_e); > > slapi_log_error( SLAPI_LOG_TRACE, MEMBEROF_PLUGIN_SUBSYSTEM, > "<-- memberof_is_legit_member\n" ); > > ?2. I don't know whether the arrays pre_array and post_array in > memberofd_replace_list() function should be freed. I think they should > because they are allocated by slapi_ch_malloc(), but i'm not sure and > i don't know how i should free them - i am not a specialist of the > plugin API. Anyway, apparently this part of code is never used by our > group management applications so i don't have a memory leak caused by > this function. > > ?3. If we make MODRDN on an INDIRECT group the original indirect group > DN is deleted from memberof but the renamed indirect group DN is not > added. It may be fixed by running every once in a while the memberOf > fixup task BUT in this particular case the attribute will never be > fixed - because of the problem in ?4 : > > ?4. I propose to delete all the memberOf attribute values at the > beginning of the fix thread. Otherwise if we have in the entry a > "memberOf" attribute value equal to a DN of a direct group, the > memeberOf fix task (the function memberof_fix_memberof_callback()) > will NEVER add the indirect groupsDNs that depend on the > abovementioned direct group, so this thread is not really a FULL FIX > thread :) Here are the proposed changes to avoid it: > --- /tmp/fedora-ds-base-cvs/ldap/servers/plugins/memberof/memberof.c > 2008-02-19 07:04:56.000000000 +0100 > +++ /tmp/memberof-patched.c 2008-03-23 22:13:00.000000000 +0100 > @@ -2006,6 +2010,7 @@ > /* memberof_fix_memberof_callback() > * Add initial and/or fix up broken group list in entry > * > + * 0. Delete all the memberOf attribute values > * 1. Make sure direct membership groups are in the entry > * 2. Add all groups that current group list allows through nested membership > * 3. Trim groups that have no relationship to entry > @@ -2016,6 +2021,9 @@ > char *dn = slapi_entry_get_dn(e); > memberof_add_groups data = {dn, dn}; > > + /* step 0. */ > + slapi_entry_attr_delete(e, MEMBEROF_ATTR); > + > /* step 1. and step 2. */ > rc = memberof_call_foreach_dn(0, dn, MEMBEROF_GROUP_ATTR, > memberof_add_groups_search_callback, &data); > > ?5. After the code changes proposed in "?1" i still have ~600 bytes > memory increase per 5000 modifications (2500 DEL & 2500 ADDs to group > membership, the memory increases mainly at DEL operations). I don't > know whether it is due to connection management in slapd code or to > the plug-in itself but it is a much more acceptable value for me than > 100Mb :) > > ?6. #define MEMBEROF_GROUP_ATTR_TYPE "uid" is declared but never used > in the plug-in code... > > Thanks once more for this plug-in! > Thank you for the information Andrey. I'm currently doing some cleanup on the memberOf plug-in around the implementation of the cleanup task. Once I am finished with that work, I'll look into the issues that you raised. We are working on making the memberOf plug-in more robust, as it's currently in a sort of "alpha" stage as you've noticed. If you are interested in helping with this effort (or just following it), take a look at the design document for this plug-in on our wiki (http://directory.fedoraproject.org/wiki/MemberOf_Plugin). Thanks again! -NGK -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3241 bytes Desc: S/MIME Cryptographic Signature URL: From vinod3942 at gmail.com Wed Mar 26 06:40:20 2008 From: vinod3942 at gmail.com (vinod gupta) Date: Wed, 26 Mar 2008 12:10:20 +0530 Subject: [Fedora-directory-devel] skew time error Message-ID: <75a3b3850803252340v63c6c83fm8771539754761182@mail.gmail.com> hi, we installed FDS 1.04 on red hat linux servers with MMR. Everything seems to be working fine till 4-5 months and suddenly all the MMR broken and says too excessive skew time and replication aborting. I have tried all the possible ways that has been mentioned in some of the posts but it din't help. After some time the same errors come again and again. I have checked my server time they are all in sync. I am clueless now. Can anybody suggest a way to help me out of this problem. Regards Vinod -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmeggins at redhat.com Mon Mar 31 17:57:44 2008 From: rmeggins at redhat.com (Rich Megginson) Date: Mon, 31 Mar 2008 11:57:44 -0600 Subject: [Fedora-directory-devel] skew time error In-Reply-To: <75a3b3850803252340v63c6c83fm8771539754761182@mail.gmail.com> References: <75a3b3850803252340v63c6c83fm8771539754761182@mail.gmail.com> Message-ID: <47F12618.3070605@redhat.com> vinod gupta wrote: > hi, > > we installed FDS 1.04 on red hat linux servers with MMR. Everything > seems to be working fine till 4-5 months and suddenly all the MMR > broken and says too excessive skew time and replication aborting. I > have tried all the possible ways that has been mentioned in some of > the posts but it din't help. After some time the same errors come > again and again. I have checked my server time they are all in sync. I > am clueless now. Can anybody suggest a way to help me out of this problem. 32-bit or 64-bit? What platform? > > Regards > Vinod > ------------------------------------------------------------------------ > > -- > Fedora-directory-devel mailing list > Fedora-directory-devel at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-devel > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3245 bytes Desc: S/MIME Cryptographic Signature URL: From vinod3942 at gmail.com Mon Mar 31 18:04:39 2008 From: vinod3942 at gmail.com (vinod gupta) Date: Mon, 31 Mar 2008 23:34:39 +0530 Subject: [Fedora-directory-devel] skew time error In-Reply-To: <47F12618.3070605@redhat.com> References: <75a3b3850803252340v63c6c83fm8771539754761182@mail.gmail.com> <47F12618.3070605@redhat.com> Message-ID: <75a3b3850803311104g38be3e04ieea85ecf70de2f5b@mail.gmail.com> its 64 bit machine. Regds Vinod On Mon, Mar 31, 2008 at 11:27 PM, Rich Megginson wrote: > vinod gupta wrote: > > hi, > > > > we installed FDS 1.04 on red hat linux servers with MMR. Everything > > seems to be working fine till 4-5 months and suddenly all the MMR > > broken and says too excessive skew time and replication aborting. I > > have tried all the possible ways that has been mentioned in some of > > the posts but it din't help. After some time the same errors come > > again and again. I have checked my server time they are all in sync. I > > am clueless now. Can anybody suggest a way to help me out of this > problem. > 32-bit or 64-bit? What platform? > > > > Regards > > Vinod > > ------------------------------------------------------------------------ > > > > -- > > Fedora-directory-devel mailing list > > Fedora-directory-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/fedora-directory-devel > > > > > -- > Fedora-directory-devel mailing list > Fedora-directory-devel at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinod3942 at gmail.com Mon Mar 31 18:07:23 2008 From: vinod3942 at gmail.com (vinod gupta) Date: Mon, 31 Mar 2008 23:37:23 +0530 Subject: [Fedora-directory-devel] skew time error In-Reply-To: <47F12618.3070605@redhat.com> References: <75a3b3850803252340v63c6c83fm8771539754761182@mail.gmail.com> <47F12618.3070605@redhat.com> Message-ID: <75a3b3850803311107i783b77d9r419a5e9724162d10@mail.gmail.com> Linux mihmumludb02 2.6.9-34.0.2.ELsmp #1 SMP Fri Jun 30 10:33:58 EDT 2006 i686 i686 i386 GNU/Linux Above is the detail abt the system where FDS 1.04 is installed. Regards Vinod On Mon, Mar 31, 2008 at 11:27 PM, Rich Megginson wrote: > vinod gupta wrote: > > hi, > > > > we installed FDS 1.04 on red hat linux servers with MMR. Everything > > seems to be working fine till 4-5 months and suddenly all the MMR > > broken and says too excessive skew time and replication aborting. I > > have tried all the possible ways that has been mentioned in some of > > the posts but it din't help. After some time the same errors come > > again and again. I have checked my server time they are all in sync. I > > am clueless now. Can anybody suggest a way to help me out of this > problem. > 32-bit or 64-bit? What platform? > > > > Regards > > Vinod > > ------------------------------------------------------------------------ > > > > -- > > Fedora-directory-devel mailing list > > Fedora-directory-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/fedora-directory-devel > > > > > -- > Fedora-directory-devel mailing list > Fedora-directory-devel at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nkinder at redhat.com Mon Mar 31 22:45:58 2008 From: nkinder at redhat.com (Nathan Kinder) Date: Mon, 31 Mar 2008 15:45:58 -0700 Subject: [Fedora-directory-devel] Please Review: (439907) Need to improve and expose SLAPI task API Message-ID: <47F169A6.5030606@redhat.com> https://bugzilla.redhat.com/show_bug.cgi?id=439907 Resolves: bug 439907 Bug Description: The current SLAPI task API is lacking in numerous ways. One of the main issues is that not all functions and datastructures are exposed via SLAPI that are needed for writing custom task plug-ins. Reviewed by: ??? Files: see diff Branch: HEAD Fix Description: A detailed design document for this work is located at: http://directory.fedoraproject.org/wiki/Slapi_Task_API Much of the existing API was reused, but new API and cleanup was performed as well. I converted the existing server and plug-in tasks to use the new task API as much as possible. The exceptions to this are some of the server internal tasks that call into the backend plug-in. These tasks work a bit differently since they can be called as modes of ns-slapd. Platforms tested: RHEL5.1 x86_64, F8 x86_64 Flag Day: Yes! Doc impact: Yes. We need to add the new API to the Plug-In Programmer's Guide https://bugzilla.redhat.com/attachment.cgi?id=299782&action=diff -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3241 bytes Desc: S/MIME Cryptographic Signature URL: