From jderose at redhat.com Tue Dec 1 01:26:44 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Mon, 30 Nov 2009 18:26:44 -0700 Subject: [Freeipa-devel] [PATCH] 318 add PKCS#10 parser In-Reply-To: <4B0C4D5B.40103@redhat.com> References: <4B0C4D5B.40103@redhat.com> Message-ID: <1259630804.2515.0.camel@jgd-dsk> On Tue, 2009-11-24 at 16:17 -0500, Rob Crittenden wrote: > The pyOpenSSL PKCS#10 parser doesn't provide a way to get to attributes > so we can't get the subject alt names (or other interesting bits). This > pyasn1-based parser adds that support. > > I'm also switching to the pyasn1 X509v3 support because older releases > of pyOpenSSL lacked the get_components() method on subjects making it > difficult to get a usable subject. > > This PKCS#10 parser cannot handle all possible attribute types. It > should be robust enough to not blow up if it gets something it knows > nothing about. > > If a subjectaltname extension is present in a CSR we: > > - require that the host(s) exist in IPA > - If the requestor is a machine then the alt names must be present in > the services managedBy attribute. This is so we can control what > hosts(s) a machine can request a cert for. > > I'm working on a way to be able to set the service principal within the > reuqest. Nalin's certmonger program will set it as an otherName in the > GeneralNames attribute. We should be able to make principal an optional > argument to cert-request and use the value from the CSR (and blow up if > we get it neither way). > > rob ack. pushed to master. From jderose at redhat.com Tue Dec 1 01:27:10 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Mon, 30 Nov 2009 18:27:10 -0700 Subject: [Freeipa-devel] [PATCH] 319 add -s option to ipa-join In-Reply-To: <4B0D5D55.7050905@redhat.com> References: <4B0D5D55.7050905@redhat.com> Message-ID: <1259630830.2515.1.camel@jgd-dsk> On Wed, 2009-11-25 at 11:37 -0500, Rob Crittenden wrote: > In ipa-client-install we do the ipa-join before creating any of the > configuration files. I added a -s option to ipa-join to specify the IPA > server since it won't be defined in /etc/ipa/default.conf yet. > > I discovered to my chagrin that previous testing of this worked because > /etc/ipa/default.conf isn't owned by our packages. I'll fix this in a > future patch. > > rob ack. pushed to master. From jderose at redhat.com Tue Dec 1 01:28:20 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Mon, 30 Nov 2009 18:28:20 -0700 Subject: [Freeipa-devel] [PATCH] 320 remove /etc/ipa/ipa.conf In-Reply-To: <4B0DB321.1020104@redhat.com> References: <4B0DB321.1020104@redhat.com> Message-ID: <1259630900.2515.2.camel@jgd-dsk> On Wed, 2009-11-25 at 17:43 -0500, Rob Crittenden wrote: > The configuration file /etc/ipa/ipa.conf was used by the v1 clients and > servers to manually set realm, domain and server(s). This has been > renamed to /etc/ipa/default.conf in v2. > > Some old utilities still referenced this old file and we still created > it. This patch should completely remove it. > > rob This isn't applying to the current master: Applying: Replace /etc/ipa/ipa.conf with /etc/ipa/default.conf error: patch failed: ipa.spec.in:473 error: ipa.spec.in: patch does not apply Patch failed at 0001 Replace /etc/ipa/ipa.conf with /etc/ipa/default.conf From pzuna at redhat.com Tue Dec 1 13:04:55 2009 From: pzuna at redhat.com (=?UTF-8?B?UGF2ZWwgWsWvbmE=?=) Date: Tue, 01 Dec 2009 14:04:55 +0100 Subject: [Freeipa-devel] [PATCH] Add {user, host, sourcehost}Category to HBAC and make accessTime multivalue. In-Reply-To: <4B14103F.6040303@redhat.com> References: <4B0425BB.3030700@redhat.com> <4B0AFDB8.5080407@redhat.com> <4B0EA112.5050202@redhat.com> <4B14103F.6040303@redhat.com> Message-ID: <4B151477.9070901@redhat.com> Rob Crittenden wrote: > Pavel Zuna wrote: >> Rob Crittenden wrote: >>> Pavel Zuna wrote: >>>> Due to the format of accessTime (it has commas and spaces in it), we >>>> can't use the List parameter type. I made it so that accessTime >>>> values have to be entered one by one using new commands. >>>> >>>> We also agreed, that we're going to rename GeneralizedTime parameter >>>> to AccessTime to prevent confusion with RFC 4517 standard. I >>>> attached a separate patch for clarity. >>>> >>>> Pavel >>> >>> A couple of questions: >>> >>> - Would it make sense to leave time in as an option that takes a >>> singular value? If someone wants multiple times they can use the new >>> add interface, right? >> It would and I think it's a good idea, updated patch attached. >> >>> - What are these new enums for? If there is only one choice do you >>> really have a choice? >> Well for now, we only have the 'all' in categories, but the list is >> expected to grow. At first I didn't include categories in the plugin, >> because of this, but Sumit wanted it to be complete. >> >>> - We still need some tests for GeneralizedTime/AccessTime. >> Ok, added to my TODO list. > > The patch isn't applying for me: > > $ patch -p1 --dry-run < 0003-Fix-takes_options-in-automount-plugin.patch > patching file ipalib/plugins/hbac.py > patching file tests/test_xmlrpc/test_hbac_plugin.py > Hunk #1 FAILED at 52. > Hunk #2 FAILED at 84. > 2 out of 3 hunks FAILED -- saving rejects to file > tests/test_xmlrpc/test_hbac_plugin.py.rej > > Since you have to mess with this anyway, can you: > > - add another test to also test adding the access time on the add. You > added back the capability but the tests are still removed AFAICT. > > - add a FUTURE or FIXME comment indicating that the enumerators are > future-proofing things by making them a 1-option enumerator for now? > > rob Fixed patch attached. Pavel -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-user-host-sourcehost-Category-to-HBAC-and-make.patch Type: application/mbox Size: 6791 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Rename-GeneralizedTime-to-AccessTime.patch Type: application/mbox Size: 3299 bytes Desc: not available URL: From pzuna at redhat.com Tue Dec 1 13:05:52 2009 From: pzuna at redhat.com (=?UTF-8?B?UGF2ZWwgWsWvbmE=?=) Date: Tue, 01 Dec 2009 14:05:52 +0100 Subject: [Freeipa-devel] [PATCH] Change object_class of group object. In-Reply-To: <4B141087.5020106@redhat.com> References: <4B0EA157.90007@redhat.com> <4B141087.5020106@redhat.com> Message-ID: <4B1514B0.80602@redhat.com> Rob Crittenden wrote: > Pavel Zuna wrote: >> Some groups created by default don't have ipaUserGroup and won't show >> up in searches. >> >> Pavel >> > > nack, isn't the better approach to fix up the groups that are created by > default without the ipaUserGroup objectclass? It is. Fixed patch attached. > rob Pavel -------------- next part -------------- A non-text attachment was scrubbed... Name: 0014-Add-ipaUserGroup-objectClass-to-default-groups-where.patch Type: application/mbox Size: 990 bytes Desc: not available URL: From pzuna at redhat.com Tue Dec 1 13:27:51 2009 From: pzuna at redhat.com (=?UTF-8?B?UGF2ZWwgWsWvbmE=?=) Date: Tue, 01 Dec 2009 14:27:51 +0100 Subject: [Freeipa-devel] [PATCH] jderose 027 Extensible return values In-Reply-To: <4B0D9C99.5030607@redhat.com> References: <1259163317.24813.84.camel@jgd-dsk> <4B0D63D9.9010602@redhat.com> <1259172995.24813.115.camel@jgd-dsk> <4B0D9C99.5030607@redhat.com> Message-ID: <4B1519D7.8040207@redhat.com> Rob Crittenden wrote: > Jason Gerard DeRose wrote: >> On Wed, 2009-11-25 at 12:05 -0500, Rob Crittenden wrote: >>> This is purely from reading the patch, I haven't applied and tested >>> it yet. >>> >>> ipalib/output.py: >>> +primary_key = Output('primary_key', unicode, >>> + 'The primary key of the deleted entry' >>> +) >>> >>> This isn't only for deleted entries, right? >> >> Ah, yeah, that should be made more generic. This doc message is only >> used by developers, though. >> >>> This import doesn't seem to be used: >>> from inspect import getdoc >>> >>> What is dont_output_for_cli()? Is this an effort to make things work >>> while we're in transition? >> >> Yeah, I just renamed some methods so we can reference how they were >> implemented. Temporary. >> >>> You seem to have disabled the raw option in LDAPSearch, was that >>> intentional? >> >> Originally I got the impression we weren't going to keep both --raw and >> --all, but this can be changed. >> >>> Is cli_name being dropped for label? I'm ok with that but should we >>> remove it from all the plugins? >> >> No, here is how they work: >> >> `cli_name` is used for the optparse names and defaults to Param.name, >> like: >> >> --first >> >> `label` is a human readable, translatable string. It's used in the >> webUI, and to prompt show entries on cli, like: >> >> First name: John Doe >> >> `doc` is human readable help passed to optparse.make_option(help=doc). >> It default to the value of the label. It's used like this: >> >> --uid=INT UID (use this option to set it manually) >> >> In the above case the `label` is "UID" (not shown) but the `doc` is this >> longer string. >> >> The user plugins provide good examples of how I think these should be >> used. >> >> I'll submit a patch later documented these different string uses. >> >>> rob >> > > We'll also need to determine what we'll do about all the plugins. The > cert plugin, for example, isn't ported to this new return value system > and blows up in many places. > > There are also some labels missing, such as for fqdn in the host plugin. > > These are both quite easy to fix, I think we just need to coordinate > things. Perhaps if Pavel and I split up the plugins and fix anything > that needs fixing and commit all the patches at one time to avoid any > period of breakage. > > rob Just did a fast forward through the big patch. It looks mostly OK, but as Rob said - it breaks a few things. I don't mind fixing all the plugins - it shouldn't be too hard, because at this point most of them are just extensions of baseldap.py classes. I'm going to apply the patch on my tree and see what I can do in the second half of this week. One thing I noticed: + return dict( + result=entry_attrs, + primary_key=keys[0], + ) This will work on most plugins, but you should use keys[-1], because keys might contain parent object keys as well. The last key is always the primary key of the object in question. Pavel From pzuna at redhat.com Tue Dec 1 13:50:32 2009 From: pzuna at redhat.com (=?UTF-8?B?UGF2ZWwgWsWvbmE=?=) Date: Tue, 01 Dec 2009 14:50:32 +0100 Subject: [Freeipa-devel] [PATCHES] Migration wrap-up. Message-ID: <4B151F28.60305@redhat.com> Okey, I think my migration patches are ready for submission. What's new? - No more forced password change after migration, unless the password doesn't meet IPA password policy. Expiration time sets correctly (hooray!). - Migration mode (adding entries with pre-hashed passwords) can now be turned ON/OFF using the ipaMigrationEnabled attribute in ipaConfig entry. - New fancy password migration page using HTML form based authentication. (CSS and looks in general will probably have to change to visually go with the rest of the webUI.) - Better error/log messages and some general code clean up. I didn't change the migration plugin to use IPA commands. Believe me, I tried. There's just too much overhead and additional work: - We need to sanitize data from DS before we feed it to the IPA commands and it's not just converting them to unicode. - There are attributes our commands do not accept as parameters and setattr/addattr doesn't really help that much there. It's going to be even worst when custom schemas kick in. Our commands also make some assumptions about attributes - like givenName/sn being required etc. It's just too hard to do it properly in a generic way. - Using IPA commands generates at least 4 times more LDAP requests. - The code is also longer. The migration plugin might still need some work and I'm thinking of ways to make it better, more readable and maintainable, but if the other patches pass and there's no big problems with it, I say we should push it, so that QE can do some testing. I'm currently writing a wiki page with step by step migration guide, but I left it open at the office and I'm sick at home at the moment, so I'm going to resume when back. I will also setup a testing environment on the blades for DS to IPA migration. Pavel -------------- next part -------------- A non-text attachment was scrubbed... Name: 0009-Add-enable-migration-option-in-config-plugin.patch Type: application/mbox Size: 2235 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0006-Allow-adding-entries-with-pre-hashed-passwords-but.patch Type: application/mbox Size: 2196 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0007-Add-BIND-pre-op-for-DS-IPA-password-migration-to-ip.patch Type: application/mbox Size: 16285 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0008-Add-DS-migration-plugin-and-password-migration-page.patch Type: application/mbox Size: 22598 bytes Desc: not available URL: From mnagy at redhat.com Tue Dec 1 15:10:32 2009 From: mnagy at redhat.com (Martin Nagy) Date: Tue, 01 Dec 2009 16:10:32 +0100 Subject: [Freeipa-devel] [PATCH] Remove unnecessary "error: " prefixes Message-ID: <1259680232.3326.25.camel@wolverine.englab.brq.redhat.com> Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Remove-unnecessary-error-prefixes.patch Type: text/x-patch Size: 3312 bytes Desc: not available URL: From mnagy at redhat.com Tue Dec 1 15:12:37 2009 From: mnagy at redhat.com (Martin Nagy) Date: Tue, 01 Dec 2009 16:12:37 +0100 Subject: [Freeipa-devel] [PATCH] Ask the user before overwriting /etc/named.conf Message-ID: <1259680357.3326.27.camel@wolverine.englab.brq.redhat.com> Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Ask-the-user-before-overwriting-etc-named.conf.patch Type: text/x-patch Size: 3095 bytes Desc: not available URL: From mnagy at redhat.com Tue Dec 1 15:13:44 2009 From: mnagy at redhat.com (Martin Nagy) Date: Tue, 01 Dec 2009 16:13:44 +0100 Subject: [Freeipa-devel] [PATCH] Add idnsUpdatePolicy into the dns plug-in Message-ID: <1259680424.3326.28.camel@wolverine.englab.brq.redhat.com> Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-idnsUpdatePolicy-into-the-dns-plug-in.patch Type: text/x-patch Size: 1217 bytes Desc: not available URL: From rcritten at redhat.com Tue Dec 1 15:12:27 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 10:12:27 -0500 Subject: [Freeipa-devel] [PATCH] Remove unnecessary "error: " prefixes In-Reply-To: <1259680232.3326.25.camel@wolverine.englab.brq.redhat.com> References: <1259680232.3326.25.camel@wolverine.englab.brq.redhat.com> Message-ID: <4B15325B.2020905@redhat.com> Martin Nagy wrote: > Martin ack From rcritten at redhat.com Tue Dec 1 15:15:21 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 10:15:21 -0500 Subject: [Freeipa-devel] [PATCH] Ask the user before overwriting /etc/named.conf In-Reply-To: <1259680357.3326.27.camel@wolverine.englab.brq.redhat.com> References: <1259680357.3326.27.camel@wolverine.englab.brq.redhat.com> Message-ID: <4B153309.4010709@redhat.com> Martin Nagy wrote: > Martin > ack. As an aside, it might be nice if the actual package name(s) were used to make it easier for the user to know exactly what they are missing for BIND and the BIND LDAP plug-in. rob From rcritten at redhat.com Tue Dec 1 15:17:21 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 10:17:21 -0500 Subject: [Freeipa-devel] [PATCH] Add idnsUpdatePolicy into the dns plug-in In-Reply-To: <1259680424.3326.28.camel@wolverine.englab.brq.redhat.com> References: <1259680424.3326.28.camel@wolverine.englab.brq.redhat.com> Message-ID: <4B153381.9030604@redhat.com> Martin Nagy wrote: > Martin > Should there be a validator on idnsUpdatePolicy to ensure that each policy is terminated by a ;? If one wants to have multiple policies is it set with idnspolicy="policy1;policy2;policy3;"? Should the formatting be included in the doc message, or an example of usage be added? rob From rcritten at redhat.com Tue Dec 1 15:36:00 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 10:36:00 -0500 Subject: [Freeipa-devel] [PATCH] 320 remove /etc/ipa/ipa.conf In-Reply-To: <1259630900.2515.2.camel@jgd-dsk> References: <4B0DB321.1020104@redhat.com> <1259630900.2515.2.camel@jgd-dsk> Message-ID: <4B1537E0.8090804@redhat.com> Jason Gerard DeRose wrote: > On Wed, 2009-11-25 at 17:43 -0500, Rob Crittenden wrote: >> The configuration file /etc/ipa/ipa.conf was used by the v1 clients and >> servers to manually set realm, domain and server(s). This has been >> renamed to /etc/ipa/default.conf in v2. >> >> Some old utilities still referenced this old file and we still created >> it. This patch should completely remove it. >> >> rob > > This isn't applying to the current master: > > Applying: Replace /etc/ipa/ipa.conf with /etc/ipa/default.conf > error: patch failed: ipa.spec.in:473 > error: ipa.spec.in: patch does not apply > Patch failed at 0001 Replace /etc/ipa/ipa.conf > with /etc/ipa/default.conf > > Boy that spec file trips me up ever time. New patch attached. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-320.2-conf.patch Type: application/mbox Size: 9318 bytes Desc: not available URL: From mnagy at redhat.com Tue Dec 1 15:41:30 2009 From: mnagy at redhat.com (Martin Nagy) Date: Tue, 01 Dec 2009 16:41:30 +0100 Subject: [Freeipa-devel] [PATCH] Add idnsUpdatePolicy into the dns plug-in In-Reply-To: <4B153381.9030604@redhat.com> References: <1259680424.3326.28.camel@wolverine.englab.brq.redhat.com> <4B153381.9030604@redhat.com> Message-ID: <1259682090.3326.33.camel@wolverine.englab.brq.redhat.com> On Tue, 2009-12-01 at 10:17 -0500, Rob Crittenden wrote: > Martin Nagy wrote: > > Martin > > > > Should there be a validator on idnsUpdatePolicy to ensure that each > policy is terminated by a ;? If one wants to have multiple policies is > it set with idnspolicy="policy1;policy2;policy3;"? > > Should the formatting be included in the doc message, or an example of > usage be added? That might not be that easy to do, we would probably need to do more than that, e.g. make sure bind can accept the policy string. For now, I'm only adding the idnsupdatepolicy into the dns plugin so that I can use it to create zones with it during installation (patch will follow soon). Might I add the other bits later after I'm done with this? Martin From rcritten at redhat.com Tue Dec 1 15:40:14 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 10:40:14 -0500 Subject: [Freeipa-devel] [PATCH] Add {user, host, sourcehost}Category to HBAC and make accessTime multivalue. In-Reply-To: <4B151477.9070901@redhat.com> References: <4B0425BB.3030700@redhat.com> <4B0AFDB8.5080407@redhat.com> <4B0EA112.5050202@redhat.com> <4B14103F.6040303@redhat.com> <4B151477.9070901@redhat.com> Message-ID: <4B1538DE.8020306@redhat.com> Pavel Z?na wrote: > Rob Crittenden wrote: >> Pavel Zuna wrote: >>> Rob Crittenden wrote: >>>> Pavel Zuna wrote: >>>>> Due to the format of accessTime (it has commas and spaces in it), >>>>> we can't use the List parameter type. I made it so that accessTime >>>>> values have to be entered one by one using new commands. >>>>> >>>>> We also agreed, that we're going to rename GeneralizedTime >>>>> parameter to AccessTime to prevent confusion with RFC 4517 >>>>> standard. I attached a separate patch for clarity. >>>>> >>>>> Pavel >>>> >>>> A couple of questions: >>>> >>>> - Would it make sense to leave time in as an option that takes a >>>> singular value? If someone wants multiple times they can use the new >>>> add interface, right? >>> It would and I think it's a good idea, updated patch attached. >>> >>>> - What are these new enums for? If there is only one choice do you >>>> really have a choice? >>> Well for now, we only have the 'all' in categories, but the list is >>> expected to grow. At first I didn't include categories in the plugin, >>> because of this, but Sumit wanted it to be complete. >>> >>>> - We still need some tests for GeneralizedTime/AccessTime. >>> Ok, added to my TODO list. >> >> The patch isn't applying for me: >> >> $ patch -p1 --dry-run < 0003-Fix-takes_options-in-automount-plugin.patch >> patching file ipalib/plugins/hbac.py >> patching file tests/test_xmlrpc/test_hbac_plugin.py >> Hunk #1 FAILED at 52. >> Hunk #2 FAILED at 84. >> 2 out of 3 hunks FAILED -- saving rejects to file >> tests/test_xmlrpc/test_hbac_plugin.py.rej >> >> Since you have to mess with this anyway, can you: >> >> - add another test to also test adding the access time on the add. You >> added back the capability but the tests are still removed AFAICT. >> >> - add a FUTURE or FIXME comment indicating that the enumerators are >> future-proofing things by making them a 1-option enumerator for now? >> >> rob > Fixed patch attached. > > Pavel > ack x2, push master x2 From rcritten at redhat.com Tue Dec 1 15:42:21 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 10:42:21 -0500 Subject: [Freeipa-devel] [PATCH] Change object_class of group object. In-Reply-To: <4B1514B0.80602@redhat.com> References: <4B0EA157.90007@redhat.com> <4B141087.5020106@redhat.com> <4B1514B0.80602@redhat.com> Message-ID: <4B15395D.6080200@redhat.com> Pavel Z?na wrote: > Rob Crittenden wrote: >> Pavel Zuna wrote: >>> Some groups created by default don't have ipaUserGroup and won't show >>> up in searches. >>> >>> Pavel >>> >> >> nack, isn't the better approach to fix up the groups that are created >> by default without the ipaUserGroup objectclass? > It is. Fixed patch attached. > >> rob > > Pavel > ack, pushed to master. NOTE: we should probably revisit the editors group to see if it is needed/wanted in the new UI. rob From mnagy at redhat.com Tue Dec 1 15:47:44 2009 From: mnagy at redhat.com (Martin Nagy) Date: Tue, 01 Dec 2009 16:47:44 +0100 Subject: [Freeipa-devel] [PATCH] Ask the user before overwriting /etc/named.conf In-Reply-To: <4B153309.4010709@redhat.com> References: <1259680357.3326.27.camel@wolverine.englab.brq.redhat.com> <4B153309.4010709@redhat.com> Message-ID: <1259682464.3326.34.camel@wolverine.englab.brq.redhat.com> On Tue, 2009-12-01 at 10:15 -0500, Rob Crittenden wrote: > Martin Nagy wrote: > > Martin > > > > ack. > > As an aside, it might be nice if the actual package name(s) were used to > make it easier for the user to know exactly what they are missing for > BIND and the BIND LDAP plug-in. Yeah, I guess you're right. New patch attached. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Ask-the-user-before-overwriting-etc-named.conf.patch Type: text/x-patch Size: 3117 bytes Desc: not available URL: From jderose at redhat.com Tue Dec 1 17:01:10 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Tue, 01 Dec 2009 10:01:10 -0700 Subject: [Freeipa-devel] [PATCH] 320 remove /etc/ipa/ipa.conf In-Reply-To: <4B1537E0.8090804@redhat.com> References: <4B0DB321.1020104@redhat.com> <1259630900.2515.2.camel@jgd-dsk> <4B1537E0.8090804@redhat.com> Message-ID: <1259686870.2515.3.camel@jgd-dsk> On Tue, 2009-12-01 at 10:36 -0500, Rob Crittenden wrote: > Jason Gerard DeRose wrote: > > On Wed, 2009-11-25 at 17:43 -0500, Rob Crittenden wrote: > >> The configuration file /etc/ipa/ipa.conf was used by the v1 clients and > >> servers to manually set realm, domain and server(s). This has been > >> renamed to /etc/ipa/default.conf in v2. > >> > >> Some old utilities still referenced this old file and we still created > >> it. This patch should completely remove it. > >> > >> rob > > > > This isn't applying to the current master: > > > > Applying: Replace /etc/ipa/ipa.conf with /etc/ipa/default.conf > > error: patch failed: ipa.spec.in:473 > > error: ipa.spec.in: patch does not apply > > Patch failed at 0001 Replace /etc/ipa/ipa.conf > > with /etc/ipa/default.conf > > > > > > Boy that spec file trips me up ever time. New patch attached. > > rob ack. pushed to master. From jderose at redhat.com Tue Dec 1 17:01:42 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Tue, 01 Dec 2009 10:01:42 -0700 Subject: [Freeipa-devel] [PATCH] 321 better LDAP error handling in client In-Reply-To: <4B142FFA.3040104@redhat.com> References: <4B142FFA.3040104@redhat.com> Message-ID: <1259686902.2515.4.camel@jgd-dsk> On Mon, 2009-11-30 at 15:50 -0500, Rob Crittenden wrote: > This improves the LDAP exception handling in the client. The existing > code spit out a slew of deprecation errors because of the use of the > message attribute. > > rob ack. pushed to master. From rcritten at redhat.com Tue Dec 1 19:04:14 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 14:04:14 -0500 Subject: [Freeipa-devel] [PATCH] 322 set minimum level of python-pyasn1 Message-ID: <4B1568AE.4050105@redhat.com> Update the spec to set minimum version of python-pyasn1 to 0.0.9a so we can have the ASN.1 Any type needed by the PKCS#10 parser. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-322-pyasn1.patch Type: application/mbox Size: 1139 bytes Desc: not available URL: From rcritten at redhat.com Tue Dec 1 20:30:40 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 15:30:40 -0500 Subject: [Freeipa-devel] [PATCH] Add idnsUpdatePolicy into the dns plug-in In-Reply-To: <1259682090.3326.33.camel@wolverine.englab.brq.redhat.com> References: <1259680424.3326.28.camel@wolverine.englab.brq.redhat.com> <4B153381.9030604@redhat.com> <1259682090.3326.33.camel@wolverine.englab.brq.redhat.com> Message-ID: <4B157CF0.3020105@redhat.com> Martin Nagy wrote: > On Tue, 2009-12-01 at 10:17 -0500, Rob Crittenden wrote: >> Martin Nagy wrote: >>> Martin >>> >> Should there be a validator on idnsUpdatePolicy to ensure that each >> policy is terminated by a ;? If one wants to have multiple policies is >> it set with idnspolicy="policy1;policy2;policy3;"? >> >> Should the formatting be included in the doc message, or an example of >> usage be added? > > That might not be that easy to do, we would probably need to do more > than that, e.g. make sure bind can accept the policy string. For now, > I'm only adding the idnsupdatepolicy into the dns plugin so that I can > use it to create zones with it during installation (patch will follow > soon). Might I add the other bits later after I'm done with this? > > Martin > Sure, that makes sense. ack. rob From rcritten at redhat.com Tue Dec 1 20:31:17 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 15:31:17 -0500 Subject: [Freeipa-devel] [PATCH] Ask the user before overwriting /etc/named.conf In-Reply-To: <1259682464.3326.34.camel@wolverine.englab.brq.redhat.com> References: <1259680357.3326.27.camel@wolverine.englab.brq.redhat.com> <4B153309.4010709@redhat.com> <1259682464.3326.34.camel@wolverine.englab.brq.redhat.com> Message-ID: <4B157D15.4020604@redhat.com> Martin Nagy wrote: > On Tue, 2009-12-01 at 10:15 -0500, Rob Crittenden wrote: >> Martin Nagy wrote: >>> Martin >>> >> ack. >> >> As an aside, it might be nice if the actual package name(s) were used to >> make it easier for the user to know exactly what they are missing for >> BIND and the BIND LDAP plug-in. > > Yeah, I guess you're right. New patch attached. > > Martin > Cool, lots better! ack rob From rcritten at redhat.com Tue Dec 1 22:20:22 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 17:20:22 -0500 Subject: [Freeipa-devel] [PATCH] 323 type argument for x509.load_certificate() Message-ID: <4B1596A6.5060208@redhat.com> Add a type argument (PEM or DER) for x509.load_certificate(). Certs are coming out of LDAP as binary so we need to be able to handle that too. Seems more sane to add an argument that to base64-encode it. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-323-cert.patch Type: application/mbox Size: 2066 bytes Desc: not available URL: From rcritten at redhat.com Tue Dec 1 22:23:27 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 17:23:27 -0500 Subject: [Freeipa-devel] [PATCH] 324 add errors.NotImplementedError Message-ID: <4B15975F.6030401@redhat.com> This deprecates a similar patch from John last month. The server-side baseclass rabase defines a framework for CA plugins. When I added this code I set it up to return errors.NotImplementedError but didn't actually include that error class in the commit. I'm adding that in now, favoring it over the python built-in exception of the same name because it is more friendly to the client (they get a "command not implemented" instead of an InternalError. Ideally we should not register commands that aren't implemented, I'll tackle that soon but for now this will fill in the gap. This also wraps the call to cert_revoke() in the service plugin to not blow up if using the selfsign CA which doesn't implement revocation. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-324-errors.patch Type: application/mbox Size: 2595 bytes Desc: not available URL: From rcritten at redhat.com Wed Dec 2 04:19:53 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 01 Dec 2009 23:19:53 -0500 Subject: [Freeipa-devel] [PATCH] 325 test for cert plugin Message-ID: <4B15EAE9.4070001@redhat.com> An extremely basic test for the cert plugin. Only tests the cert-request command but it's a start. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-325-certtest.patch Type: application/mbox Size: 4406 bytes Desc: not available URL: From jderose at redhat.com Wed Dec 2 06:15:55 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Tue, 01 Dec 2009 23:15:55 -0700 Subject: [Freeipa-devel] [PATCH] 322 set minimum level of python-pyasn1 In-Reply-To: <4B1568AE.4050105@redhat.com> References: <4B1568AE.4050105@redhat.com> Message-ID: <1259734555.22165.1.camel@jgd-dsk> On Tue, 2009-12-01 at 14:04 -0500, Rob Crittenden wrote: > Update the spec to set minimum version of python-pyasn1 to 0.0.9a so we > can have the ASN.1 Any type needed by the PKCS#10 parser. > > rob nack. This introduces a bug in the spec: error: line 89: Unknown tag: pequires: libcap From jderose at redhat.com Wed Dec 2 06:27:30 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Tue, 01 Dec 2009 23:27:30 -0700 Subject: [Freeipa-devel] [PATCH] 323 type argument for x509.load_certificate() In-Reply-To: <4B1596A6.5060208@redhat.com> References: <4B1596A6.5060208@redhat.com> Message-ID: <1259735250.22165.2.camel@jgd-dsk> On Tue, 2009-12-01 at 17:20 -0500, Rob Crittenden wrote: > Add a type argument (PEM or DER) for x509.load_certificate(). Certs are > coming out of LDAP as binary so we need to be able to handle that too. > Seems more sane to add an argument that to base64-encode it. > > rob ack. pushed to master. From jderose at redhat.com Wed Dec 2 06:28:04 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Tue, 01 Dec 2009 23:28:04 -0700 Subject: [Freeipa-devel] [PATCH] 324 add errors.NotImplementedError In-Reply-To: <4B15975F.6030401@redhat.com> References: <4B15975F.6030401@redhat.com> Message-ID: <1259735284.22165.3.camel@jgd-dsk> On Tue, 2009-12-01 at 17:23 -0500, Rob Crittenden wrote: > This deprecates a similar patch from John last month. The server-side > baseclass rabase defines a framework for CA plugins. When I added this > code I set it up to return errors.NotImplementedError but didn't > actually include that error class in the commit. > > I'm adding that in now, favoring it over the python built-in exception > of the same name because it is more friendly to the client (they get a > "command not implemented" instead of an InternalError. > > Ideally we should not register commands that aren't implemented, I'll > tackle that soon but for now this will fill in the gap. > > This also wraps the call to cert_revoke() in the service plugin to not > blow up if using the selfsign CA which doesn't implement revocation. > > rob ack. pushed to master. From pzuna at redhat.com Wed Dec 2 11:24:39 2009 From: pzuna at redhat.com (=?UTF-8?B?UGF2ZWwgWsWvbmE=?=) Date: Wed, 02 Dec 2009 12:24:39 +0100 Subject: [Freeipa-devel] Re: [PATCHES] Migration wrap-up. In-Reply-To: <4B151F28.60305@redhat.com> References: <4B151F28.60305@redhat.com> Message-ID: <4B164E77.7080507@redhat.com> Pavel Z?na wrote: > Okey, I think my migration patches are ready for submission. > > What's new? > > - No more forced password change after migration, unless the password > doesn't meet IPA password policy. Expiration time sets correctly (hooray!). > - Migration mode (adding entries with pre-hashed passwords) can now be > turned ON/OFF using the ipaMigrationEnabled attribute in ipaConfig entry. > - New fancy password migration page using HTML form based > authentication. (CSS and looks in general will probably have to change > to visually go with the rest of the webUI.) > - Better error/log messages and some general code clean up. > > I didn't change the migration plugin to use IPA commands. Believe me, I > tried. There's just too much overhead and additional work: > > - We need to sanitize data from DS before we feed it to the IPA commands > and it's not just converting them to unicode. > - There are attributes our commands do not accept as parameters and > setattr/addattr doesn't really help that much there. It's going to be > even worst when custom schemas kick in. Our commands also make some > assumptions about attributes - like givenName/sn being required etc. > It's just too hard to do it properly in a generic way. > - Using IPA commands generates at least 4 times more LDAP requests. > - The code is also longer. > > The migration plugin might still need some work and I'm thinking of ways > to make it better, more readable and maintainable, but if the other > patches pass and there's no big problems with it, I say we should push > it, so that QE can do some testing. > > I'm currently writing a wiki page with step by step migration guide, but > I left it open at the office and I'm sick at home at the moment, so I'm > going to resume when back. I will also setup a testing environment on > the blades for DS to IPA migration. > > Pavel Oups, I forgot to change the spec file. Patch attached. Pavel -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-password-migration-page-files-to-the-spec-file.patch Type: application/mbox Size: 912 bytes Desc: not available URL: From mnagy at redhat.com Wed Dec 2 12:04:51 2009 From: mnagy at redhat.com (Martin Nagy) Date: Wed, 02 Dec 2009 13:04:51 +0100 Subject: [Freeipa-devel] [PATCH] Make ldap2.convert_attr_synonyms more robust against schema lookup fails. In-Reply-To: <4B06A872.8080608@redhat.com> References: <4AE9C376.9090906@redhat.com> <4B044C9D.7020803@redhat.com> <4B05584A.50603@redhat.com> <4B059E60.1030606@redhat.com> <4B068D15.7030904@redhat.com> <4B06A872.8080608@redhat.com> Message-ID: <1259755491.3326.44.camel@wolverine.englab.brq.redhat.com> On Fri, 2009-11-20 at 09:32 -0500, Rob Crittenden wrote: > Pavel Zuna wrote: > > Rob Crittenden wrote: > >> Pavel Zuna wrote: > >>> Rob Crittenden wrote: > >>>> Pavel Zuna wrote: > >>>>> Rob Crittenden wrote: > >>>>>> The user plugin is crapping out on line 317 of ldap2.py because > >>>>>> attr is coming back None. The attribute it is looking for is member. > >>>>>> > >>>>>> I think the fix involves setting member_attributes = ['member'] to > >>>>>> the user plugin. > >>>>>> > >>>>>> I wonder if we need to make the ldap2 plugin a bit more robust too > >>>>>> so it can handle it better if the schema lookup returns None. > >>>>>> > >>>>>> rob > >>>>> This should fix the issue. > >>>>> > >>>> > >>>> Yes, this will fix it (I did a similar fix to work around it) but > >>>> what does it mean if there is no attribute found? Is that possible? > >>>> > >>>> Should we catch it and return a more specific error message instead? > >>>> > >>>> rob > >>> > >>> If it doesn't find the attribute, PROBABLY nothing will happen... > >>> > >>> Fortunately, we don't have to worry about it anymore. I played with > >>> python-ldap a bit today and it seems to have the > >>> convert_attr_synonyms functionality built-in. :) > >>> > >>> Here's a replacement patch. > >>> > >>> Pavel > >> > >> nack. I don't see where python-ldap is replacing it. We weren't seeing > >> it done before were we? > > That's because we were doing it wrong. > > > > We were requesting all attributes ('*') + ACIs ('aci'). After this patch > > we explicitly request all attributes in the new entry (i.e. all > > attributes that are going to be updated) and python-ldap will always > > return them named as they were requested. In other words: If we request > > localityName as l, python-ldap will return it as l, if we request it as > > localityName, python-ldap will return it as localityName. > > > >> Also, we need to request the 'aci' attribute for the aci plugin to work. > > And we do so, because after this patch, we're requesting all attributes > > explicitly. > > > > Well, no, you're requesting all attributes in the current entry. The > code looked like this once before and caused the aci plugin to break. I > guess some other change fixed that, things are working as expected. > > ack > > rob Pushed to master. Martin From mnagy at redhat.com Wed Dec 2 12:08:15 2009 From: mnagy at redhat.com (Martin Nagy) Date: Wed, 02 Dec 2009 13:08:15 +0100 Subject: [Freeipa-devel] [PATCH] Ask the user before overwriting /etc/named.conf In-Reply-To: <4B157D15.4020604@redhat.com> References: <1259680357.3326.27.camel@wolverine.englab.brq.redhat.com> <4B153309.4010709@redhat.com> <1259682464.3326.34.camel@wolverine.englab.brq.redhat.com> <4B157D15.4020604@redhat.com> Message-ID: <1259755695.3326.51.camel@wolverine.englab.brq.redhat.com> On Tue, 2009-12-01 at 15:31 -0500, Rob Crittenden wrote: > Martin Nagy wrote: > > On Tue, 2009-12-01 at 10:15 -0500, Rob Crittenden wrote: > >> Martin Nagy wrote: > >>> Martin > >>> > >> ack. > >> > >> As an aside, it might be nice if the actual package name(s) were used to > >> make it easier for the user to know exactly what they are missing for > >> BIND and the BIND LDAP plug-in. > > > > Yeah, I guess you're right. New patch attached. > > > > Martin > > > > Cool, lots better! ack > > rob Pushed to master. Martin From mnagy at redhat.com Wed Dec 2 12:08:18 2009 From: mnagy at redhat.com (Martin Nagy) Date: Wed, 02 Dec 2009 13:08:18 +0100 Subject: [Freeipa-devel] [PATCH] Add idnsUpdatePolicy into the dns plug-in In-Reply-To: <4B157CF0.3020105@redhat.com> References: <1259680424.3326.28.camel@wolverine.englab.brq.redhat.com> <4B153381.9030604@redhat.com> <1259682090.3326.33.camel@wolverine.englab.brq.redhat.com> <4B157CF0.3020105@redhat.com> Message-ID: <1259755698.3326.52.camel@wolverine.englab.brq.redhat.com> On Tue, 2009-12-01 at 15:30 -0500, Rob Crittenden wrote: > Martin Nagy wrote: > > On Tue, 2009-12-01 at 10:17 -0500, Rob Crittenden wrote: > >> Martin Nagy wrote: > >>> Martin > >>> > >> Should there be a validator on idnsUpdatePolicy to ensure that each > >> policy is terminated by a ;? If one wants to have multiple policies is > >> it set with idnspolicy="policy1;policy2;policy3;"? > >> > >> Should the formatting be included in the doc message, or an example of > >> usage be added? > > > > That might not be that easy to do, we would probably need to do more > > than that, e.g. make sure bind can accept the policy string. For now, > > I'm only adding the idnsupdatepolicy into the dns plugin so that I can > > use it to create zones with it during installation (patch will follow > > soon). Might I add the other bits later after I'm done with this? > > > > Martin > > > > Sure, that makes sense. ack. > > rob Pushed to master. Martin From mnagy at redhat.com Wed Dec 2 12:08:19 2009 From: mnagy at redhat.com (Martin Nagy) Date: Wed, 02 Dec 2009 13:08:19 +0100 Subject: [Freeipa-devel] [PATCH] Remove unnecessary "error: " prefixes In-Reply-To: <4B15325B.2020905@redhat.com> References: <1259680232.3326.25.camel@wolverine.englab.brq.redhat.com> <4B15325B.2020905@redhat.com> Message-ID: <1259755699.3326.53.camel@wolverine.englab.brq.redhat.com> On Tue, 2009-12-01 at 10:12 -0500, Rob Crittenden wrote: > Martin Nagy wrote: > > Martin > > ack Pushed to master. Martin From pzuna at redhat.com Wed Dec 2 12:15:31 2009 From: pzuna at redhat.com (=?UTF-8?B?UGF2ZWwgWsWvbmE=?=) Date: Wed, 02 Dec 2009 13:15:31 +0100 Subject: [Freeipa-devel] Re: [PATCHES] Migration wrap-up. In-Reply-To: <4B164E77.7080507@redhat.com> References: <4B151F28.60305@redhat.com> <4B164E77.7080507@redhat.com> Message-ID: <4B165A63.6020702@redhat.com> Pavel Z?na wrote: > Oups, I forgot to change the spec file. Patch attached. > > Pavel There was a missing * to handle .pyc/.pyo files. Updated patch attached. Pavel -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-password-migration-page-files-to-the-spec-file.patch Type: application/mbox Size: 913 bytes Desc: not available URL: From mnagy at redhat.com Wed Dec 2 13:38:58 2009 From: mnagy at redhat.com (Martin Nagy) Date: Wed, 02 Dec 2009 14:38:58 +0100 Subject: [Freeipa-devel] Problem with ipa installation: certutil Message-ID: <1259761138.3326.60.camel@wolverine.englab.brq.redhat.com> Hi, I'm trying to install ipa and am getting a python traceback (attached). It seems that running certutil didn't succeed so I added a debugging print before it's execution and tried to run it manually. This is what I get: # /usr/bin/certutil -d /etc/httpd/alias -S -n 'CA certificate' -s 'cn=IPA Test Certificate Authority' -x -t 'CT,,C' -1 -2 -5 -m 1056 -v 120 -z /etc/httpd/alias/noise.txt -f /etc/httpd/alias/pwdfile.txt certutil -o: unable to open "tempcertreq" for writing (-5950, 2) Exit 255 (The "Exit 255" is from my shell saying that certutil exited returning 255). I did a git grep tempcertreq in freeipa git tree but didn't find anything, so I'm assuming we weren't creating it or anything. Does anyone know what might be causing this error? Martin -------------- next part -------------- ipa: DEBUG: [Errno 32] Broken pipe File "/usr/sbin/ipa-server-install", line 791, in sys.exit(main()) File "/usr/sbin/ipa-server-install", line 673, in main ds.create_instance(ds_user, realm_name, host_name, domain_name, dm_password, self_signed_ca=not options.ca, uidstart=options.uidstart, gidstart=options.gidstart) File "/usr/lib/python2.6/site-packages/ipaserver/install/dsinstance.py", line 193, in create_instance self.start_creation("Configuring directory server:") File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py", line 171, in start_creation method() File "/usr/lib/python2.6/site-packages/ipaserver/install/dsinstance.py", line 342, in __enable_ssl cadb.create_self_signed() File "/usr/lib/python2.6/site-packages/ipaserver/install/certs.py", line 826, in create_self_signed self.create_ca_cert() File "/usr/lib/python2.6/site-packages/ipaserver/install/certs.py", line 357, in create_ca_cert p.stdin.write("0\n1\n5\n9\ny\n") From rcritten at redhat.com Wed Dec 2 14:12:36 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 02 Dec 2009 09:12:36 -0500 Subject: [Freeipa-devel] [PATCH] 322 set minimum level of python-pyasn1 In-Reply-To: <1259734555.22165.1.camel@jgd-dsk> References: <4B1568AE.4050105@redhat.com> <1259734555.22165.1.camel@jgd-dsk> Message-ID: <4B1675D4.1020809@redhat.com> Jason Gerard DeRose wrote: > On Tue, 2009-12-01 at 14:04 -0500, Rob Crittenden wrote: >> Update the spec to set minimum version of python-pyasn1 to 0.0.9a so we >> can have the ASN.1 Any type needed by the PKCS#10 parser. >> >> rob > > nack. This introduces a bug in the spec: > > error: line 89: Unknown tag: pequires: libcap > > Uh....yeah, it's a new directive when doing a python requires, hence pequires instead of Requires. That's my story and I'm sticking with it. New patch attached. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-322.2-pyasn1.patch Type: application/mbox Size: 1071 bytes Desc: not available URL: From rcritten at redhat.com Wed Dec 2 14:38:22 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 02 Dec 2009 09:38:22 -0500 Subject: [Freeipa-devel] Problem with ipa installation: certutil In-Reply-To: <1259761138.3326.60.camel@wolverine.englab.brq.redhat.com> References: <1259761138.3326.60.camel@wolverine.englab.brq.redhat.com> Message-ID: <4B167BDE.6060303@redhat.com> Martin Nagy wrote: > Hi, > I'm trying to install ipa and am getting a python traceback (attached). > It seems that running certutil didn't succeed so I added a debugging > print before it's execution and tried to run it manually. This is what I > get: > > # /usr/bin/certutil -d /etc/httpd/alias -S -n 'CA certificate' -s > 'cn=IPA Test Certificate Authority' -x -t 'CT,,C' -1 -2 -5 -m 1056 -v > 120 -z /etc/httpd/alias/noise.txt -f /etc/httpd/alias/pwdfile.txt > certutil -o: unable to open "tempcertreq" for writing (-5950, 2) > Exit 255 > > (The "Exit 255" is from my shell saying that certutil exited returning > 255). I did a git grep tempcertreq in freeipa git tree but didn't find > anything, so I'm assuming we weren't creating it or anything. Does > anyone know what might be causing this error? > > Martin This message comes directly from certutil itself. It tries to open the file "tempcertreq" in the cwd. Odd since you are installing this as root, right? Perhaps you are in a directory that no longer exists? I seem to recall running into this in v1 as well and though we did a chdir(). Maybe we do that in some places and not others. rob From jdennis at redhat.com Wed Dec 2 17:49:57 2009 From: jdennis at redhat.com (John Dennis) Date: Wed, 02 Dec 2009 12:49:57 -0500 Subject: [Freeipa-devel] [PATCH] 325 test for cert plugin In-Reply-To: <4B15EAE9.4070001@redhat.com> References: <4B15EAE9.4070001@redhat.com> Message-ID: <4B16A8C5.9070400@redhat.com> On 12/01/2009 11:19 PM, Rob Crittenden wrote: > An extremely basic test for the cert plugin. Only tests the cert-request > command but it's a start. I think the test should also check for the correct return type. For instance shouldn't assert res['subject'] == 'CN=ipatestcert.greyoak.com' by followed (or preceded by) assert type(res['subject']) is unicode Also, is this going to deprecate checks/check_ra.py? -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From rcritten at redhat.com Wed Dec 2 18:11:11 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 02 Dec 2009 13:11:11 -0500 Subject: [Freeipa-devel] [PATCH] 325 test for cert plugin In-Reply-To: <4B16A8C5.9070400@redhat.com> References: <4B15EAE9.4070001@redhat.com> <4B16A8C5.9070400@redhat.com> Message-ID: <4B16ADBF.1000208@redhat.com> John Dennis wrote: > On 12/01/2009 11:19 PM, Rob Crittenden wrote: >> An extremely basic test for the cert plugin. Only tests the cert-request >> command but it's a start. > > I think the test should also check for the correct return type. For > instance shouldn't > > assert res['subject'] == 'CN=ipatestcert.greyoak.com' > > by followed (or preceded by) > > assert type(res['subject']) is unicode > > Also, is this going to deprecate checks/check_ra.py? > Ah, excellent point. What it probably really should do is call xmlrpc_test.assert_attr_equal() which should do the unicode enforcement. If we need additional types we can add an expected type argument, defaulting to unicode. I'm reluctant to tackle this just yet with Jason's big patch looming. It contains a bunch of changes to the test infrastructure to handle the new return values. Perhaps I'll shelve this patch for a few days until we can get his patch committed, then rebase this and add in the type enforcement and resubmit. rob From jdennis at redhat.com Wed Dec 2 19:08:30 2009 From: jdennis at redhat.com (John Dennis) Date: Wed, 02 Dec 2009 14:08:30 -0500 Subject: [Freeipa-devel] multiple plugin loads? Message-ID: <4B16BB2E.50705@redhat.com> I haven't had a chance to look at the source code for an explanation yet but I'm wondering if what I see in the debug logs is correct. I see loading all plugin modules in xxx/ipalib/plugins ... and loading all plugin modules in xxx/ipaserver/plugins ... 3 or 4 times when the server initializes (not always the identical list). Some plugins do get loaded multiple times. Is that expected? -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From rcritten at redhat.com Wed Dec 2 19:09:17 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 02 Dec 2009 14:09:17 -0500 Subject: [Freeipa-devel] multiple plugin loads? In-Reply-To: <4B16BB2E.50705@redhat.com> References: <4B16BB2E.50705@redhat.com> Message-ID: <4B16BB5D.1070206@redhat.com> John Dennis wrote: > I haven't had a chance to look at the source code for an explanation yet > but I'm wondering if what I see in the debug logs is correct. I see > > loading all plugin modules in xxx/ipalib/plugins ... > > and > > loading all plugin modules in xxx/ipaserver/plugins ... > > 3 or 4 times when the server initializes (not always the identical > list). Some plugins do get loaded multiple times. Is that expected? > If you're seeing it in the Apache logs this is expected. It loads for each Apache process loading the server. rob From jderose at redhat.com Wed Dec 2 20:14:46 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Wed, 02 Dec 2009 13:14:46 -0700 Subject: [Freeipa-devel] [PATCH] 322 set minimum level of python-pyasn1 In-Reply-To: <4B1675D4.1020809@redhat.com> References: <4B1568AE.4050105@redhat.com> <1259734555.22165.1.camel@jgd-dsk> <4B1675D4.1020809@redhat.com> Message-ID: <1259784886.22165.64.camel@jgd-dsk> On Wed, 2009-12-02 at 09:12 -0500, Rob Crittenden wrote: > Jason Gerard DeRose wrote: > > On Tue, 2009-12-01 at 14:04 -0500, Rob Crittenden wrote: > >> Update the spec to set minimum version of python-pyasn1 to 0.0.9a so we > >> can have the ASN.1 Any type needed by the PKCS#10 parser. > >> > >> rob > > > > nack. This introduces a bug in the spec: > > > > error: line 89: Unknown tag: pequires: libcap > > > > > > Uh....yeah, it's a new directive when doing a python requires, hence > pequires instead of Requires. That's my story and I'm sticking with it. > > New patch attached. > > rob ack. pushed to master. From rcritten at redhat.com Wed Dec 2 21:26:41 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 02 Dec 2009 16:26:41 -0500 Subject: [Freeipa-devel] [PATCH] 326 bump IPA install version Message-ID: <4B16DB91.1010508@redhat.com> We store a rough version of IPA at install time in the base object, bump this up to V2.0 rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-326-version.patch Type: application/mbox Size: 682 bytes Desc: not available URL: From jdennis at redhat.com Thu Dec 3 00:01:03 2009 From: jdennis at redhat.com (John Dennis) Date: Wed, 02 Dec 2009 19:01:03 -0500 Subject: [Freeipa-devel] [PATCH] dogtag clean-up Message-ID: <4B16FFBF.1010002@redhat.com> The essence of this patch is to return the correct types from certificate plugins and avoid scraping Javascript from dogtag (CMS) html responses with better error handling. Instead we ask CMS to always return our data as XML documents which can be much more robustly parsed (including properly handling issues such as character encoding, escapes, etc.). Fundamentally the process is split into two parts. A parsing routine which returns a dict with all the values from CMS in the correct Python types for IPA. The possible values returned from CMS are fully documented and can easily be read via the documentation link in HTML posted at the bottom (plus in the code of course). The command plugin invokes the parsing routine and picks out from the parse result dict the values it wants to return (and may optionaly convert the type as needed for XMLRPC, this is fully documented, in particular serial numbers need special handling in XMLRPC). This model allows us to use different parsing methods without disturbing the logic in the command plugin should that ever be necessary (i.e. clear separation of responsibilities). Status results are never returned in the command result. Instead we use the defined exception handling logic for IPA XMLRPC. If the command fails in some fashion we return a CertificateOperationError exception. On the receiving end if no exception has been thrown it knows the values returned are valid. Careful attention has been paid to the types being used. Strings are always unicode, integral values are represented as either int or long objects. No longer are integral values represented as strings with confusion as to thier radix representation (with the notable exception of serial numbers which must be passed through XMLRPC as decimal strings, the rules for this are fully documented). The logic in the selfsign and dogtag plugins have been brought into alignment. Much more extensive error checking has been added to selfsign to handle issues concering serial number operations. A new error exception has been added (CertificateOperationError). Error messages have been localized. The check_ra.py test was updated (unfortunately this test requires a configured server so I used my test server). Extensive documentation has been added to many of the routines. Easy to browse HTML documentation for the dogtag plugin can be found here (for the time being) http://jdennis.fedorapeople.org/ipa/dogtag I've noticed we have a bit of code duplication going on with CMS interactions. In the future we shold consolodate all CMS code in one library (module). This patch has been lingering in my private repo too long. I no longer want to keep merging as others modify the same code :-) So here it is. Other components of the fixes have already been posted as patches. -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-dogtag-clean-up URL: From rcritten at redhat.com Thu Dec 3 04:15:31 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 02 Dec 2009 23:15:31 -0500 Subject: [Freeipa-devel] service record conundrum Message-ID: <4B173B63.2000506@redhat.com> Here is sort of a tricky problem, need some advice (LONG). When we bootstrap an IPA server we create a number of principals for the server itself. We create a host/, HTTP/ and ldap/ principal using kadmin.local. By using kadmin.local this entry is put into cn=kerberos,dc=example,dc=com. This has the nice side effect of making these records not appear as service entries so they are unmodifiable by anyone, meaning an admin will have a really hard time hosing their server. The downside is that these records do not appear as service entries, so if you search for services on the IPA server you'll get nothing. Even worse it means you can't request certificates for these services, because they don't exist. Not that one really should since we also generate certificates for these at bootstrap, but we don't store them anywhere because there isn't any place to put them. This also means that we can't track expiration of these. To make things even more fun we have the DS uniqueness plugin configured so there can be no duplication in principal names. Since this is in the RDN of service records we can't even create a bit of a bogus entry to still protect the principals and yet be able to store certificates. Remember too that these records are creating during installation, effectively bootstrapping the real services (httpd, dirsrv), so we have limited options for how to generate them to begin with. One idea I had is to continue to use kadmin.local to create the principals and then move them out of cn=kerberos into cn=services, adding whatever additional data we need. This way we would maintain the principalkeys. Then we'd need to insert the certificates we generate. Unfortunately 389-DS doesn't seem to support newsuperior so I guess we'd have to move it ourselves via delete and re-add. So I'm basically stuck right now. rob From jderose at redhat.com Thu Dec 3 07:25:19 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Thu, 03 Dec 2009 07:25:19 +0000 Subject: [Freeipa-devel] [PATCH] jderose 028 Lossless datetime round-trip Message-ID: <1259825119.9039.161.camel@jgd-dsk> As per John's request, this patch allows lossless round-tripping of Python datetime.datetime objects. Unfortunately, the xmlrpclib dumps() and loads() functions use funny wrapper objects like xmlrpclib.DateTime rather than directly serializing to/from standard Python types like datetime.datetime. This makes lossless round-tripping pretty cumbersome to implement. Doing a loads(foo, use_datetime=True) would work, but the `use_datetime` kwarg is only available in Python2.5 and newer, so I instead extended my xml_wrap() and xml_unwrap() functions. -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jderose-028-Lossless-datetime-round-trip.patch Type: text/x-patch Size: 4035 bytes Desc: not available URL: From mnagy at redhat.com Thu Dec 3 09:53:26 2009 From: mnagy at redhat.com (Martin Nagy) Date: Thu, 03 Dec 2009 10:53:26 +0100 Subject: [Freeipa-devel] Problem with ipa installation: certutil In-Reply-To: <4B167BDE.6060303@redhat.com> References: <1259761138.3326.60.camel@wolverine.englab.brq.redhat.com> <4B167BDE.6060303@redhat.com> Message-ID: <1259834006.3326.83.camel@wolverine.englab.brq.redhat.com> On Wed, 2009-12-02 at 09:38 -0500, Rob Crittenden wrote: > Martin Nagy wrote: > > Hi, > > I'm trying to install ipa and am getting a python traceback (attached). > > It seems that running certutil didn't succeed so I added a debugging > > print before it's execution and tried to run it manually. This is what I > > get: > > > > # /usr/bin/certutil -d /etc/httpd/alias -S -n 'CA certificate' -s > > 'cn=IPA Test Certificate Authority' -x -t 'CT,,C' -1 -2 -5 -m 1056 -v > > 120 -z /etc/httpd/alias/noise.txt -f /etc/httpd/alias/pwdfile.txt > > certutil -o: unable to open "tempcertreq" for writing (-5950, 2) > > Exit 255 > > > > (The "Exit 255" is from my shell saying that certutil exited returning > > 255). I did a git grep tempcertreq in freeipa git tree but didn't find > > anything, so I'm assuming we weren't creating it or anything. Does > > anyone know what might be causing this error? > > > > Martin > > This message comes directly from certutil itself. It tries to open the > file "tempcertreq" in the cwd. > > Odd since you are installing this as root, right? Perhaps you are in a > directory that no longer exists? Correct. I was in my freeipa git directory when I executed ipa-server-install but had to delete it and clone again in other terminal. > I seem to recall running into this in v1 as well and though we did a > chdir(). Maybe we do that in some places and not others. Should we make a patch to prevent any future problems like this (even if they are rare)? Maybe at the beginning we could chdir() to our current directory to make sure, and abort if that fails. Martin From rmeggins at redhat.com Thu Dec 3 15:19:08 2009 From: rmeggins at redhat.com (Rich Megginson) Date: Thu, 03 Dec 2009 08:19:08 -0700 Subject: [Freeipa-devel] service record conundrum In-Reply-To: <4B173B63.2000506@redhat.com> References: <4B173B63.2000506@redhat.com> Message-ID: <4B17D6EC.4060203@redhat.com> Rob Crittenden wrote: > Here is sort of a tricky problem, need some advice (LONG). > > When we bootstrap an IPA server we create a number of principals for > the server itself. We create a host/, HTTP/ and ldap/ principal using > kadmin.local. By using kadmin.local this entry is put into > cn=kerberos,dc=example,dc=com. > > This has the nice side effect of making these records not appear as > service entries so they are unmodifiable by anyone, meaning an admin > will have a really hard time hosing their server. > > The downside is that these records do not appear as service entries, > so if you search for services on the IPA server you'll get nothing. > > Even worse it means you can't request certificates for these services, > because they don't exist. Not that one really should since we also > generate certificates for these at bootstrap, but we don't store them > anywhere because there isn't any place to put them. This also means > that we can't track expiration of these. > > To make things even more fun we have the DS uniqueness plugin > configured so there can be no duplication in principal names. Since > this is in the RDN of service records we can't even create a bit of a > bogus entry to still protect the principals and yet be able to store > certificates. > > Remember too that these records are creating during installation, > effectively bootstrapping the real services (httpd, dirsrv), so we > have limited options for how to generate them to begin with. > > One idea I had is to continue to use kadmin.local to create the > principals and then move them out of cn=kerberos into cn=services, > adding whatever additional data we need. This way we would maintain > the principalkeys. Then we'd need to insert the certificates we generate. > > Unfortunately 389-DS doesn't seem to support newsuperior so I guess > we'd have to move it ourselves via delete and re-add. 389 will soon support new superior. But yes, delete then add is the only way to do that at present. If you want to preserve the uuid of the entry, you could dump to ldif, edit the DN to "move" it, then ldif2db back - kind of painful, but would preserve the uuid of the entries. > > So I'm basically stuck right now. > > rob > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3258 bytes Desc: S/MIME Cryptographic Signature URL: From mnagy at redhat.com Thu Dec 3 16:25:19 2009 From: mnagy at redhat.com (Martin Nagy) Date: Thu, 03 Dec 2009 17:25:19 +0100 Subject: [Freeipa-devel] [PATCHES] Use the dns plugin during installation Message-ID: <1259857519.3326.153.camel@wolverine.englab.brq.redhat.com> Hi, these three patches should make sure that we add dns records the right way. It will also serve for the ipa-dns-install command that's almost ready, patch will be coming soon. Thanks Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Move-api-finalization-in-ipa-server-install-after-wr.patch Type: text/x-patch Size: 3562 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Use-the-dns-plug-in-for-addition-of-records-during-i.patch Type: text/x-patch Size: 10674 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Only-add-an-NTP-SRV-record-if-we-really-are-setting.patch Type: text/x-patch Size: 4525 bytes Desc: not available URL: From dpal at redhat.com Thu Dec 3 16:30:57 2009 From: dpal at redhat.com (Dmitri Pal) Date: Thu, 03 Dec 2009 11:30:57 -0500 Subject: [Freeipa-devel] service record conundrum In-Reply-To: <4B173B63.2000506@redhat.com> References: <4B173B63.2000506@redhat.com> Message-ID: <4B17E7C1.5050501@redhat.com> Rob Crittenden wrote: > Here is sort of a tricky problem, need some advice (LONG). > > When we bootstrap an IPA server we create a number of principals for > the server itself. We create a host/, HTTP/ and ldap/ principal using > kadmin.local. By using kadmin.local this entry is put into > cn=kerberos,dc=example,dc=com. > > This has the nice side effect of making these records not appear as > service entries so they are unmodifiable by anyone, meaning an admin > will have a really hard time hosing their server. > > The downside is that these records do not appear as service entries, > so if you search for services on the IPA server you'll get nothing. > How do we search? What base DN we use? One of the solutions might be to install these principals as is and only later apply ipaService object class to them so that the search for services would find them. Would be a bit ugly since as far as I understand these services are in a different location in the tree but this approach might be less painfull than LDIF and delete and add. I hope that we will get the RDN renames pretty soon so that this would not be an issue but it might not be soon enough for v2. -- Thank you, Dmitri Pal Engineering Manager IPA project, Red Hat Inc. ------------------------------- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From rcritten at redhat.com Thu Dec 3 16:55:53 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 03 Dec 2009 11:55:53 -0500 Subject: [Freeipa-devel] service record conundrum In-Reply-To: <4B17E7C1.5050501@redhat.com> References: <4B173B63.2000506@redhat.com> <4B17E7C1.5050501@redhat.com> Message-ID: <4B17ED99.1070701@redhat.com> Dmitri Pal wrote: > Rob Crittenden wrote: >> Here is sort of a tricky problem, need some advice (LONG). >> >> When we bootstrap an IPA server we create a number of principals for >> the server itself. We create a host/, HTTP/ and ldap/ principal using >> kadmin.local. By using kadmin.local this entry is put into >> cn=kerberos,dc=example,dc=com. >> >> This has the nice side effect of making these records not appear as >> service entries so they are unmodifiable by anyone, meaning an admin >> will have a really hard time hosing their server. >> >> The downside is that these records do not appear as service entries, >> so if you search for services on the IPA server you'll get nothing. >> > > How do we search? What base DN we use? One of the solutions might be to > install these principals as is and only later apply ipaService object > class to them so that the search for services would find them. Would be > a bit ugly since as far as I understand these services are in a > different location in the tree but this approach might be less painfull > than LDIF and delete and add. > I hope that we will get the RDN renames pretty soon so that this would > not be an issue but it might not be soon enough for v2. > We search in the baseDN of the type of object is is, so cn=services, cn=computers, cn=users, etc. We also filter on the objectclasses that should be in that object. Searching in 2 places is possible just not something we currently do. I'm leaning towards moving the entries, more so since I haven't gotten any "that is the dumbest idea I've heard all week" responses :-) We store a list of the IPA masters in the DIT somewhere, I'll have to see if I can find a way to maintain protection of the principals using that. rob From rcritten at redhat.com Thu Dec 3 16:56:13 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 03 Dec 2009 11:56:13 -0500 Subject: [Freeipa-devel] [PATCH] jderose 028 Lossless datetime round-trip In-Reply-To: <1259825119.9039.161.camel@jgd-dsk> References: <1259825119.9039.161.camel@jgd-dsk> Message-ID: <4B17EDAD.8080908@redhat.com> Jason Gerard DeRose wrote: > As per John's request, this patch allows lossless round-tripping of > Python datetime.datetime objects. > > Unfortunately, the xmlrpclib dumps() and loads() functions use funny > wrapper objects like xmlrpclib.DateTime rather than directly serializing > to/from standard Python types like datetime.datetime. This makes > lossless round-tripping pretty cumbersome to implement. > > Doing a loads(foo, use_datetime=True) would work, but the `use_datetime` > kwarg is only available in Python2.5 and newer, so I instead extended my > xml_wrap() and xml_unwrap() functions. > What should this do it if the incoming DateTime value is not parsed correctly by datetime.datetime()? rob From jdennis at redhat.com Thu Dec 3 17:05:09 2009 From: jdennis at redhat.com (John Dennis) Date: Thu, 03 Dec 2009 12:05:09 -0500 Subject: [Freeipa-devel] [PATCH] jderose 028 Lossless datetime round-trip In-Reply-To: <4B17EDAD.8080908@redhat.com> References: <1259825119.9039.161.camel@jgd-dsk> <4B17EDAD.8080908@redhat.com> Message-ID: <4B17EFC5.4010707@redhat.com> On 12/03/2009 11:56 AM, Rob Crittenden wrote: > What should this do it if the incoming DateTime value is not parsed > correctly by datetime.datetime()? Well in theory this should never happen if the parameter value complies with the specification which demands it be in iso8601 format. I would imagine it would raise a ValueError just like any other ill formed parameter. -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From dpal at redhat.com Thu Dec 3 17:07:32 2009 From: dpal at redhat.com (Dmitri Pal) Date: Thu, 03 Dec 2009 12:07:32 -0500 Subject: [Freeipa-devel] service record conundrum In-Reply-To: <4B17ED99.1070701@redhat.com> References: <4B173B63.2000506@redhat.com> <4B17E7C1.5050501@redhat.com> <4B17ED99.1070701@redhat.com> Message-ID: <4B17F054.5090304@redhat.com> Rob Crittenden wrote: > Dmitri Pal wrote: >> Rob Crittenden wrote: >>> Here is sort of a tricky problem, need some advice (LONG). >>> >>> When we bootstrap an IPA server we create a number of principals for >>> the server itself. We create a host/, HTTP/ and ldap/ principal using >>> kadmin.local. By using kadmin.local this entry is put into >>> cn=kerberos,dc=example,dc=com. >>> >>> This has the nice side effect of making these records not appear as >>> service entries so they are unmodifiable by anyone, meaning an admin >>> will have a really hard time hosing their server. >>> >>> The downside is that these records do not appear as service entries, >>> so if you search for services on the IPA server you'll get nothing. >>> >> >> How do we search? What base DN we use? One of the solutions might be to >> install these principals as is and only later apply ipaService object >> class to them so that the search for services would find them. Would be >> a bit ugly since as far as I understand these services are in a >> different location in the tree but this approach might be less painfull >> than LDIF and delete and add. >> I hope that we will get the RDN renames pretty soon so that this would >> not be an issue but it might not be soon enough for v2. >> > > We search in the baseDN of the type of object is is, so cn=services, > cn=computers, cn=users, etc. > > We also filter on the objectclasses that should be in that object. > > Searching in 2 places is possible just not something we currently do. > > I'm leaning towards moving the entries, more so since I haven't gotten > any "that is the dumbest idea I've heard all week" responses :-) > > We store a list of the IPA masters in the DIT somewhere, I'll have to > see if I can find a way to maintain protection of the principals using > that. > Will it affect expectations of the KDC on where it searches for its entries? Would we have to also update KDC DAL configuration to search the records in a different place? > rob -- Thank you, Dmitri Pal Engineering Manager IPA project, Red Hat Inc. ------------------------------- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From jderose at redhat.com Thu Dec 3 17:14:38 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Thu, 03 Dec 2009 10:14:38 -0700 Subject: [Freeipa-devel] [PATCH] jderose 028 Lossless datetime round-trip In-Reply-To: <4B17EDAD.8080908@redhat.com> References: <1259825119.9039.161.camel@jgd-dsk> <4B17EDAD.8080908@redhat.com> Message-ID: <1259860478.9039.167.camel@jgd-dsk> On Thu, 2009-12-03 at 11:56 -0500, Rob Crittenden wrote: > Jason Gerard DeRose wrote: > > As per John's request, this patch allows lossless round-tripping of > > Python datetime.datetime objects. > > > > Unfortunately, the xmlrpclib dumps() and loads() functions use funny > > wrapper objects like xmlrpclib.DateTime rather than directly serializing > > to/from standard Python types like datetime.datetime. This makes > > lossless round-tripping pretty cumbersome to implement. > > > > Doing a loads(foo, use_datetime=True) would work, but the `use_datetime` > > kwarg is only available in Python2.5 and newer, so I instead extended my > > xml_wrap() and xml_unwrap() functions. > > > > What should this do it if the incoming DateTime value is not parsed > correctly by datetime.datetime()? > > rob I don't believe this can happen... DateTime and datetime are both stored in a time.struct_time, so if the XML contains an invalid date, things will have already blown-up when the DateTime was created. I image xmlrpclib will raise a ProtocolError error, but I can add a test for this. From jderose at redhat.com Thu Dec 3 17:16:35 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Thu, 03 Dec 2009 10:16:35 -0700 Subject: [Freeipa-devel] [PATCH] 325 test for cert plugin In-Reply-To: <4B16ADBF.1000208@redhat.com> References: <4B15EAE9.4070001@redhat.com> <4B16A8C5.9070400@redhat.com> <4B16ADBF.1000208@redhat.com> Message-ID: <1259860595.9039.168.camel@jgd-dsk> On Wed, 2009-12-02 at 13:11 -0500, Rob Crittenden wrote: > John Dennis wrote: > > On 12/01/2009 11:19 PM, Rob Crittenden wrote: > >> An extremely basic test for the cert plugin. Only tests the cert-request > >> command but it's a start. > > > > I think the test should also check for the correct return type. For > > instance shouldn't > > > > assert res['subject'] == 'CN=ipatestcert.greyoak.com' > > > > by followed (or preceded by) > > > > assert type(res['subject']) is unicode > > > > Also, is this going to deprecate checks/check_ra.py? > > > > Ah, excellent point. What it probably really should do is call > xmlrpc_test.assert_attr_equal() which should do the unicode enforcement. > If we need additional types we can add an expected type argument, > defaulting to unicode. > > I'm reluctant to tackle this just yet with Jason's big patch looming. It > contains a bunch of changes to the test infrastructure to handle the new > return values. Perhaps I'll shelve this patch for a few days until we > can get his patch committed, then rebase this and add in the type > enforcement and resubmit. > > rob ack. pushed to master. From jderose at redhat.com Thu Dec 3 17:16:48 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Thu, 03 Dec 2009 10:16:48 -0700 Subject: [Freeipa-devel] [PATCH] 326 bump IPA install version In-Reply-To: <4B16DB91.1010508@redhat.com> References: <4B16DB91.1010508@redhat.com> Message-ID: <1259860608.9039.169.camel@jgd-dsk> On Wed, 2009-12-02 at 16:26 -0500, Rob Crittenden wrote: > We store a rough version of IPA at install time in the base object, bump > this up to V2.0 > > rob ack. pushed to master. From jderose at redhat.com Thu Dec 3 17:37:33 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Thu, 03 Dec 2009 10:37:33 -0700 Subject: [Freeipa-devel] [PATCH] dogtag clean-up In-Reply-To: <4B16FFBF.1010002@redhat.com> References: <4B16FFBF.1010002@redhat.com> Message-ID: <1259861853.9039.173.camel@jgd-dsk> Patch looks great, thanks for including such detailed documentation. But it's not applying to the current master: Falling back to patching base and 3-way merge... Auto-merging ipalib/plugins/service.py CONFLICT (content): Merge conflict in ipalib/plugins/service.py Auto-merging ipalib/x509.py Failed to merge in the changes. Patch failed at 0001 dogtag clean-up Can you take a look? On Wed, 2009-12-02 at 19:01 -0500, John Dennis wrote: > The essence of this patch is to return the correct types from > certificate plugins and avoid scraping Javascript from dogtag (CMS) > html responses with better error handling. Instead we ask CMS to > always return our data as XML documents which can be much more > robustly parsed (including properly handling issues such as character > encoding, escapes, etc.). > > Fundamentally the process is split into two parts. A parsing routine > which returns a dict with all the values from CMS in the correct > Python types for IPA. The possible values returned from CMS are fully > documented and can easily be read via the documentation link in HTML > posted at the bottom (plus in the code of course). The command plugin > invokes the parsing routine and picks out from the parse result dict > the values it wants to return (and may optionaly convert the type as > needed for XMLRPC, this is fully documented, in particular serial > numbers need special handling in XMLRPC). This model allows us to use > different parsing methods without disturbing the logic in the command > plugin should that ever be necessary (i.e. clear separation of > responsibilities). > > Status results are never returned in the command result. Instead we > use the defined exception handling logic for IPA XMLRPC. If the > command fails in some fashion we return a CertificateOperationError > exception. On the receiving end if no exception has been thrown it > knows the values returned are valid. > > Careful attention has been paid to the types being used. Strings are > always unicode, integral values are represented as either int or long > objects. No longer are integral values represented as strings with > confusion as to thier radix representation (with the notable exception > of serial numbers which must be passed through XMLRPC as decimal > strings, the rules for this are fully documented). > > The logic in the selfsign and dogtag plugins have been brought into > alignment. > > Much more extensive error checking has been added to selfsign to > handle issues concering serial number operations. > > A new error exception has been added (CertificateOperationError). > > Error messages have been localized. > > The check_ra.py test was updated (unfortunately this test requires a > configured server so I used my test server). > > Extensive documentation has been added to many of the routines. > > Easy to browse HTML documentation for the dogtag plugin can be found > here (for the time being) > > http://jdennis.fedorapeople.org/ipa/dogtag > > I've noticed we have a bit of code duplication going on with CMS > interactions. In the future we shold consolodate all CMS code in one > library (module). > > This patch has been lingering in my private repo too long. I no longer > want to keep merging as others modify the same code :-) So here it > is. Other components of the fixes have already been posted as patches. > > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel From rcritten at redhat.com Fri Dec 4 14:49:58 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 04 Dec 2009 09:49:58 -0500 Subject: [Freeipa-devel] [PATCH] jderose 028 Lossless datetime round-trip In-Reply-To: <4B17EFC5.4010707@redhat.com> References: <1259825119.9039.161.camel@jgd-dsk> <4B17EDAD.8080908@redhat.com> <4B17EFC5.4010707@redhat.com> Message-ID: <4B192196.5070604@redhat.com> John Dennis wrote: > On 12/03/2009 11:56 AM, Rob Crittenden wrote: > >> What should this do it if the incoming DateTime value is not parsed >> correctly by datetime.datetime()? > > Well in theory this should never happen if the parameter value complies > with the specification which demands it be in iso8601 format. I would > imagine it would raise a ValueError just like any other ill formed > parameter. > That's not the case I'm concerned with. I'm wondering about Joe hacker who wants to see what happens when bad data is sent to IPA. I don't think we'll catastrophically blow up but I also don't think we'd return a useful error message. rob From jdennis at redhat.com Fri Dec 4 15:12:10 2009 From: jdennis at redhat.com (John Dennis) Date: Fri, 04 Dec 2009 10:12:10 -0500 Subject: [Freeipa-devel] [PATCH] jderose 028 Lossless datetime round-trip In-Reply-To: <4B192196.5070604@redhat.com> References: <1259825119.9039.161.camel@jgd-dsk> <4B17EDAD.8080908@redhat.com> <4B17EFC5.4010707@redhat.com> <4B192196.5070604@redhat.com> Message-ID: <4B1926CA.2030208@redhat.com> On 12/04/2009 09:49 AM, Rob Crittenden wrote: > John Dennis wrote: >> On 12/03/2009 11:56 AM, Rob Crittenden wrote: >> >>> What should this do it if the incoming DateTime value is not parsed >>> correctly by datetime.datetime()? >> >> Well in theory this should never happen if the parameter value >> complies with the specification which demands it be in iso8601 format. >> I would imagine it would raise a ValueError just like any other ill >> formed parameter. >> > > That's not the case I'm concerned with. I'm wondering about Joe hacker > who wants to see what happens when bad data is sent to IPA. I don't > think we'll catastrophically blow up but I also don't think we'd return > a useful error message. I agree we should test with bad input and make sure we gracefully handle it. Just pointing out its no different than any other data type, you could declare the parameter type to be an integer and send junk too, or a completely malformed xml document. A graceful response to all these cases would be a good thing to ensure. -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From jderose at redhat.com Fri Dec 4 15:50:32 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Fri, 04 Dec 2009 08:50:32 -0700 Subject: [Freeipa-devel] Declarative tests Message-ID: <1259941832.6706.129.camel@jgd-dsk> I'm still not quite finished with my revised 027 patch, but I wanted to give everyone a quick status update on the testing enhancements I've been working on. I wrote a new test base class xmlrpc_test.Declarative (which borrows from Rob's XMLRPC_test). It allows you to write tests completely (surprise!) declaratively by using simple data structures to describe a sequence of api.Command.* calls and the expected return value of each. The first step is to define a `cleanup_commands` class attribute, something like this: class test_user(Declarative): cleanup_commands = [ ['user_del', [u'tuser'], {}], ['group_del', [u'tgroup'], {}], ] The `cleanup_commands` attribute is a list of commands (and their arguments and options) that, when run, should delete any entries your tests might have created but not deleted (because of a partial failure or whatever). The `Declarative` class will run these commands before and after your tests run. It also runs each cleanup command in a try/except, so trying to delete a non-existent entry wont botch things up. The next (and only other) step is to define a `tests` class attribute, which is a simple list of tests in the order they should be run, something like this: tests = [ { 'desc': 'Create a user', 'command': [ 'user_add', [u'tuser'], {'givenname': u'Test', 'sn': u'User'} ), 'expected': { 'result': { 'uid': u'tuser', 'givenname': u'Test', 'sn': u'User', }, 'summary': 'Added user "tuser"', }, 'ignore_values': ['ipauniqueid'] }, { 'desc': 'More cowbell!', 'command': ['user_find', [], {}], 'expected': [...], }, ] Each test in the `tests` list is a dictionary, which must contain at least the `command` and `expected` members. The `expected` member is the exact data structure the command should return... couldn't be simpler. `Declarative` takes care of running the test and comparing the expected value to the actual value, and reporting errors when they don't match up. The `desc` member is an optional human-readable description of the test. If something goes wrong, this description will show up in the trace to help orient you more quickly. The other optional member is `ignore_values`, which is a list of entry attributes whose values you don't want `Declarative` to match. The attributes still must be present, but they are removed before comparing with your `expected` value. This is for random values like `ipauniqueid`... they need to be present, but you don't care about the exact value. For example, if your test had an `expected` and `ignore_values` like this: { 'expected': { 'result': {'foo': u'bar'}, }, 'ignore_values': ['bee'], } Then a return value of {'result': {'foo': u'bar', 'bee': u'bop'}} is fine as is {'result': {'foo': u'bar', 'bee': u'movie'}}. But if `bee` is absent, then the test will fail. It's also easy to declare tests in which an exception is raised. In this case, your `expected` should be a PublicException instance, like this: { 'desc': 'Non existent user', 'command': ['user_show', [u'nope], {}], 'expected': errors.NotFound(reason='no way!'), } I attached the new declarative tests for the user plugins if anyone wants to see more examples. This definitely makes the tests faster to write easier to read. But this is just the beginning. The plugin framework has kept things pretty regular and it will be easy to automatically generate streams of truly abusive tests for `Declarative` to run. We really need more coverage on the plugin side, I feel, and this is a fast, dirty, and fun way to get there. -Jason "patch soon" DeRose -------------- next part -------------- A non-text attachment was scrubbed... Name: test_user_plugin.py Type: text/x-python Size: 7666 bytes Desc: not available URL: From jdennis at redhat.com Fri Dec 4 15:52:57 2009 From: jdennis at redhat.com (John Dennis) Date: Fri, 04 Dec 2009 10:52:57 -0500 Subject: [Freeipa-devel] Declarative tests In-Reply-To: <1259941832.6706.129.camel@jgd-dsk> References: <1259941832.6706.129.camel@jgd-dsk> Message-ID: <4B193059.8070403@redhat.com> Cool, that looks really nice Jason. -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From rcritten at redhat.com Fri Dec 4 17:06:06 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 04 Dec 2009 12:06:06 -0500 Subject: [Freeipa-devel] Re: [PATCHES] Migration wrap-up. In-Reply-To: <4B151F28.60305@redhat.com> References: <4B151F28.60305@redhat.com> Message-ID: <4B19417E.3000107@redhat.com> Pavel Z?na wrote: > Okey, I think my migration patches are ready for submission. > > What's new? > > - No more forced password change after migration, unless the password > doesn't meet IPA password policy. Expiration time sets correctly (hooray!). > - Migration mode (adding entries with pre-hashed passwords) can now be > turned ON/OFF using the ipaMigrationEnabled attribute in ipaConfig entry. > - New fancy password migration page using HTML form based > authentication. (CSS and looks in general will probably have to change > to visually go with the rest of the webUI.) > - Better error/log messages and some general code clean up. > > I didn't change the migration plugin to use IPA commands. Believe me, I > tried. There's just too much overhead and additional work: > > - We need to sanitize data from DS before we feed it to the IPA commands > and it's not just converting them to unicode. > - There are attributes our commands do not accept as parameters and > setattr/addattr doesn't really help that much there. It's going to be > even worst when custom schemas kick in. Our commands also make some > assumptions about attributes - like givenName/sn being required etc. > It's just too hard to do it properly in a generic way. > - Using IPA commands generates at least 4 times more LDAP requests. > - The code is also longer. > > The migration plugin might still need some work and I'm thinking of ways > to make it better, more readable and maintainable, but if the other > patches pass and there's no big problems with it, I say we should push > it, so that QE can do some testing. > > I'm currently writing a wiki page with step by step migration guide, but > I left it open at the office and I'm sick at home at the moment, so I'm > going to resume when back. I will also setup a testing environment on > the blades for DS to IPA migration. > > Pavel A few comments: - The comment block in ipapwd_pre_bind() is incorrect. It says that it will generate a principal name. - You check for the existence of userPassword in the entry. Since you've already made sure a simple bind was successful I don't think this is necessary, it is implicit, right? I suppose it doesn't hurt anything. - Under what conditions would the bind password not be found in the userPassword attribute? - Why the formatting change to ipaEscrowKeyCertificate and ipaEscrowKey? - There are a number of typos on the migration HTML pages: - There was a problem with you request. - If the problem persists, contact you administrator. - migrated to a new Indentity management solution - Upon successfull login - There are a number of ways to get redirected to /ipa/migration/error.html and invalid.html. Should some logging be added so an admin can debug why failures occur? - Can you add a validator for the LDAP uri and perhaps an example somewhere? - When migrating users/groups do we want an option to maintain existing uid/gid? - Is there a reason you're using pure ldap calls instead of the ldap2 plugin? rob From jdennis at redhat.com Fri Dec 4 19:02:45 2009 From: jdennis at redhat.com (John Dennis) Date: Fri, 04 Dec 2009 14:02:45 -0500 Subject: [Freeipa-devel] Re: [PATCH] dogtag clean-up In-Reply-To: <4B16FFBF.1010002@redhat.com> References: <4B16FFBF.1010002@redhat.com> Message-ID: <4B195CD5.8000103@redhat.com> On 12/02/2009 07:01 PM, John Dennis wrote: > Easy to browse HTML documentation for the dogtag plugin can be found > here (for the time being) > > http://jdennis.fedorapeople.org/ipa/dogtag opps ... I just realized this html was not generated properly and most of the formatting, footnotes, and cross-references were missing. I've fixed that now. -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From rcritten at redhat.com Fri Dec 4 21:29:40 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 04 Dec 2009 16:29:40 -0500 Subject: [Freeipa-devel] [PATCH] 305 remove a principal from a keytab In-Reply-To: <1257273157.16763.440.camel@jgd-dsk> References: <4AEB4CD1.9010707@redhat.com> <1257273157.16763.440.camel@jgd-dsk> Message-ID: <4B197F44.2090502@redhat.com> Jason Gerard DeRose wrote: > On Fri, 2009-10-30 at 16:30 -0400, Rob Crittenden wrote: >> I wasn't able to find a command-line program to remove principals from a >> keytab so I wrote my own. ktutil can do it but it doesn't take >> command-line arguments. Java ships a utility named ktab but adding a >> huge dependency for one app seem a bit much :-) >> >> In any case, this program has 2 modes: >> >> 1. Given a keytab and a principal, remove all entries of that principal >> from the keytab. This removes all versions and encryption types. >> >> 2. Given a realm remove all principals in that realm. I cheat a little >> and insert an @ before the principal name because all this really does >> is a strstr() to see if the principal in the keytab is in the realm >> provided. >> >> This utility will be added to the ipa-client-uninstall script at some >> point to clean up /etc/krb5.keytab. >> >> rob > > ack. Rob walked me through its use on #freeipa, and it works as > advertised. > pushed to master From rcritten at redhat.com Fri Dec 4 21:40:36 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 04 Dec 2009 16:40:36 -0500 Subject: [Freeipa-devel] [PATCH] 327 pass debug option in client installer Message-ID: <4B1981D4.10003@redhat.com> Pass along the debug option in the client installer to the ipa-join command. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-327-join.patch Type: application/mbox Size: 958 bytes Desc: not available URL: From jdennis at redhat.com Fri Dec 4 22:39:14 2009 From: jdennis at redhat.com (John Dennis) Date: Fri, 04 Dec 2009 17:39:14 -0500 Subject: [Freeipa-devel] [PATCH] 327 pass debug option in client installer In-Reply-To: <4B1981D4.10003@redhat.com> References: <4B1981D4.10003@redhat.com> Message-ID: <4B198F92.6050907@redhat.com> On 12/04/2009 04:40 PM, Rob Crittenden wrote: > Pass along the debug option in the client installer to the ipa-join > command. ACK -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From pzuna at redhat.com Mon Dec 7 15:04:30 2009 From: pzuna at redhat.com (Pavel Zuna) Date: Mon, 07 Dec 2009 16:04:30 +0100 Subject: [Freeipa-devel] Re: [PATCHES] Migration wrap-up. In-Reply-To: <4B19417E.3000107@redhat.com> References: <4B151F28.60305@redhat.com> <4B19417E.3000107@redhat.com> Message-ID: <4B1D197E.1040609@redhat.com> Rob Crittenden wrote: > Pavel Z?na wrote: >> Okey, I think my migration patches are ready for submission. >> >> What's new? >> >> - No more forced password change after migration, unless the password >> doesn't meet IPA password policy. Expiration time sets correctly >> (hooray!). >> - Migration mode (adding entries with pre-hashed passwords) can now be >> turned ON/OFF using the ipaMigrationEnabled attribute in ipaConfig entry. >> - New fancy password migration page using HTML form based >> authentication. (CSS and looks in general will probably have to change >> to visually go with the rest of the webUI.) >> - Better error/log messages and some general code clean up. >> >> I didn't change the migration plugin to use IPA commands. Believe me, >> I tried. There's just too much overhead and additional work: >> >> - We need to sanitize data from DS before we feed it to the IPA >> commands and it's not just converting them to unicode. >> - There are attributes our commands do not accept as parameters and >> setattr/addattr doesn't really help that much there. It's going to be >> even worst when custom schemas kick in. Our commands also make some >> assumptions about attributes - like givenName/sn being required etc. >> It's just too hard to do it properly in a generic way. >> - Using IPA commands generates at least 4 times more LDAP requests. >> - The code is also longer. >> >> The migration plugin might still need some work and I'm thinking of >> ways to make it better, more readable and maintainable, but if the >> other patches pass and there's no big problems with it, I say we >> should push it, so that QE can do some testing. >> >> I'm currently writing a wiki page with step by step migration guide, >> but I left it open at the office and I'm sick at home at the moment, >> so I'm going to resume when back. I will also setup a testing >> environment on the blades for DS to IPA migration. >> >> Pavel > > A few comments: > > - The comment block in ipapwd_pre_bind() is incorrect. It says that it > will generate a principal name. Ok, I forgot to update it. > - You check for the existence of userPassword in the entry. Since you've > already made sure a simple bind was successful I don't think this is > necessary, it is implicit, right? I suppose it doesn't hurt anything. No, I think that the pre-bind plugin is called anyway. This shows up in the DS log, when I try to bind with an invalid password: ipapwd_pre_bind - invalid BIND password for user entry: uid=someuser,cn=users,cn=accounts,dc=example,dc=com > - Under what conditions would the bind password not be found in the > userPassword attribute? If unauthenticated BINDs were allowed. I know they're not, but we better make sure. :) > - Why the formatting change to ipaEscrowKeyCertificate and ipaEscrowKey? Don't know, guess I'm a bit OCD about code formatting. :) > - There are a number of typos on the migration HTML pages: > - There was a problem with you request. > - If the problem persists, contact you administrator. > - migrated to a new Indentity management solution > - Upon successfull login Ok, I need to get spell checking in vim! Anyway I think someone better at english/user friendliness should review those messages anyway, but I'll fix the typos for the next patch. > - There are a number of ways to get redirected to > /ipa/migration/error.html and invalid.html. Should some logging be added > so an admin can debug why failures occur? Ok, I'm going to make the migration plugin generate a stand alone log file anyway, because when migrating a lot of users and groups, something is bound to fail :) and it's not good enough to have errors printed out to stdout/stderr only. > - Can you add a validator for the LDAP uri and perhaps an example > somewhere? Sure. > - When migrating users/groups do we want an option to maintain existing > uid/gid? I don't know, I do keep them for now - some services might depend on them and it wouldn't be nice if they changed after migration. So, I think I'd rather add an option to generate new UID/GIDs. There might also be collision with already existing users/groups, so we need to add some checks. > - Is there a reason you're using pure ldap calls instead of the ldap2 > plugin? Yes, I wanted to use ldap2, I really did. But unfortunately ldap2 is closely tied to the rest of the API (it's a plugin after all) and it is currently impossible to even spawn a new ldap2 instance and make it connect to an arbitrary LDAP server, because it makes assumptions about the base DN and such. I really need to make it a bit more flexible, because it would be a shame not to use it in these situations. I'll make it a high priority task for myself. > rob Pavel From rcritten at redhat.com Tue Dec 8 04:06:04 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 07 Dec 2009 23:06:04 -0500 Subject: [Freeipa-devel] [PATCH] 328 force deletion of replica Message-ID: <4B1DD0AC.4080305@redhat.com> This adds an option to ipa-replica-manage, --force, that will let you force the deletion of a replication agreement. Before this both ends had to be up and running for this to work, so that the agreement could be removed on both sides. But what if the remote has already been destroyed, either through an uninstall or the host went bye bye. This will let you force remove it from the local instance. I run into this a lot with replication testing because I always forget to remove the agreement before destroying a replica installation. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-328-replica.patch Type: application/mbox Size: 2957 bytes Desc: not available URL: From rcritten at redhat.com Tue Dec 8 04:21:41 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 07 Dec 2009 23:21:41 -0500 Subject: [Freeipa-devel] [PATCH] 329 real services Message-ID: <4B1DD455.7080309@redhat.com> Make the IPA server host and its services "real" IPA entries We use kadmin.local to bootstrap the creation of the kerberos principals for the IPA server machine: host, HTTP and ldap. This works fine and has the side-effect of protecting the services from modification by an admin (which would likely break the server). Unfortunately this also means that the services can't be managed by useful utilities such as certmonger. So we have to create them as "real" services instead. This is a relatively manual process so if the schema for hosts or services changes this may require updates as well. There remains a minor problem. If you create a replica, during the installation of that replica it will create host and service entries too. But if you retire this replica those entries will remain. The next time you try to install the replica it will fail with dupliate entries. I'll address this in the future as the easy workaround is to run `ipa host-del replica.example.com` and re-install the replica. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-329-services.patch Type: application/mbox Size: 23216 bytes Desc: not available URL: From jdennis at redhat.com Tue Dec 8 22:06:06 2009 From: jdennis at redhat.com (John Dennis) Date: Tue, 08 Dec 2009 17:06:06 -0500 Subject: [Freeipa-devel] [PATCH] dogtag clean-up In-Reply-To: <4B16FFBF.1010002@redhat.com> References: <4B16FFBF.1010002@redhat.com> Message-ID: <4B1ECDCE.8050708@redhat.com> On 12/02/2009 07:01 PM, John Dennis wrote: > The essence of this patch is to return the correct types from > certificate plugins and avoid scraping Javascript from dogtag (CMS) > html responses with better error handling. Instead we ask CMS to > always return our data as XML documents which can be much more > robustly parsed (including properly handling issues such as character > encoding, escapes, etc.). > > Fundamentally the process is split into two parts. A parsing routine > which returns a dict with all the values from CMS in the correct > Python types for IPA. The possible values returned from CMS are fully > documented and can easily be read via the documentation link in HTML > posted at the bottom (plus in the code of course). The command plugin > invokes the parsing routine and picks out from the parse result dict > the values it wants to return (and may optionaly convert the type as > needed for XMLRPC, this is fully documented, in particular serial > numbers need special handling in XMLRPC). This model allows us to use > different parsing methods without disturbing the logic in the command > plugin should that ever be necessary (i.e. clear separation of > responsibilities). > > Status results are never returned in the command result. Instead we > use the defined exception handling logic for IPA XMLRPC. If the > command fails in some fashion we return a CertificateOperationError > exception. On the receiving end if no exception has been thrown it > knows the values returned are valid. > > Careful attention has been paid to the types being used. Strings are > always unicode, integral values are represented as either int or long > objects. No longer are integral values represented as strings with > confusion as to thier radix representation (with the notable exception > of serial numbers which must be passed through XMLRPC as decimal > strings, the rules for this are fully documented). > > The logic in the selfsign and dogtag plugins have been brought into > alignment. > > Much more extensive error checking has been added to selfsign to > handle issues concering serial number operations. > > A new error exception has been added (CertificateOperationError). > > Error messages have been localized. > > The check_ra.py test was updated (unfortunately this test requires a > configured server so I used my test server). > > Extensive documentation has been added to many of the routines. > > Easy to browse HTML documentation for the dogtag plugin can be found > here (for the time being) > > http://jdennis.fedorapeople.org/ipa/dogtag > > I've noticed we have a bit of code duplication going on with CMS > interactions. In the future we shold consolodate all CMS code in one > library (module). > > This patch has been lingering in my private repo too long. I no longer > want to keep merging as others modify the same code :-) So here it > is. Other components of the fixes have already been posted as patches. The rebased patch is attached. May the gods of patchdom shine upon my face and we'll celebrate it's successful application :-) -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-rebase-dogtag-clean-up-patch.patch Type: text/x-patch Size: 105064 bytes Desc: not available URL: From davido at redhat.com Tue Dec 8 23:53:25 2009 From: davido at redhat.com (David O'Brien) Date: Wed, 09 Dec 2009 09:53:25 +1000 Subject: [Freeipa-devel] Re: [PATCHES] Migration wrap-up. In-Reply-To: <4B1D197E.1040609@redhat.com> References: <4B151F28.60305@redhat.com> <4B19417E.3000107@redhat.com> <4B1D197E.1040609@redhat.com> Message-ID: <4B1EE6F5.6070403@redhat.com> Pavel Zuna wrote: > Rob Crittenden wrote: >> Pavel Z??na wrote: >>> Okey, I think my migration patches are ready for submission. >>> >>> What's new? >>> >>> - No more forced password change after migration, unless the password >>> doesn't meet IPA password policy. Expiration time sets correctly >>> (hooray!). >>> - Migration mode (adding entries with pre-hashed passwords) can now >>> be turned ON/OFF using the ipaMigrationEnabled attribute in ipaConfig >>> entry. >>> - New fancy password migration page using HTML form based >>> authentication. (CSS and looks in general will probably have to >>> change to visually go with the rest of the webUI.) >>> - Better error/log messages and some general code clean up. >>> >>> I didn't change the migration plugin to use IPA commands. Believe me, >>> I tried. There's just too much overhead and additional work: >>> >>> - We need to sanitize data from DS before we feed it to the IPA >>> commands and it's not just converting them to unicode. >>> - There are attributes our commands do not accept as parameters and >>> setattr/addattr doesn't really help that much there. It's going to be >>> even worst when custom schemas kick in. Our commands also make some >>> assumptions about attributes - like givenName/sn being required etc. >>> It's just too hard to do it properly in a generic way. >>> - Using IPA commands generates at least 4 times more LDAP requests. >>> - The code is also longer. >>> >>> The migration plugin might still need some work and I'm thinking of >>> ways to make it better, more readable and maintainable, but if the >>> other patches pass and there's no big problems with it, I say we >>> should push it, so that QE can do some testing. >>> >>> I'm currently writing a wiki page with step by step migration guide, >>> but I left it open at the office and I'm sick at home at the moment, >>> so I'm going to resume when back. I will also setup a testing >>> environment on the blades for DS to IPA migration. >>> >>> Pavel >> >> A few comments: >> >> - The comment block in ipapwd_pre_bind() is incorrect. It says that it >> will generate a principal name. > Ok, I forgot to update it. > >> - You check for the existence of userPassword in the entry. Since >> you've already made sure a simple bind was successful I don't think >> this is necessary, it is implicit, right? I suppose it doesn't hurt >> anything. > No, I think that the pre-bind plugin is called anyway. > > This shows up in the DS log, when I try to bind with an invalid password: > > ipapwd_pre_bind - invalid BIND password for user entry: > uid=someuser,cn=users,cn=accounts,dc=example,dc=com > >> - Under what conditions would the bind password not be found in the >> userPassword attribute? > If unauthenticated BINDs were allowed. I know they're not, but we better > make sure. :) > >> - Why the formatting change to ipaEscrowKeyCertificate and ipaEscrowKey? > Don't know, guess I'm a bit OCD about code formatting. :) > >> - There are a number of typos on the migration HTML pages: >> - There was a problem with you request. >> - If the problem persists, contact you administrator. >> - migrated to a new Indentity management solution >> - Upon successfull login > Ok, I need to get spell checking in vim! Anyway I think someone better > at english/user friendliness should review those messages anyway, but > I'll fix the typos for the next patch. If there's an easy way for me to get access to the messages and anything else that gets put in front of the user I'm happy to review it. I haven't done much (read "almost nothing") in the way of patch reviews, so I'm not sure if that's the best way. Maybe I should just learn to read patches... I'm also looking forward to seeing the wiki page with the migration steps; something else for IPA doc. David > >> - There are a number of ways to get redirected to >> /ipa/migration/error.html and invalid.html. Should some logging be >> added so an admin can debug why failures occur? > Ok, I'm going to make the migration plugin generate a stand alone log > file anyway, because when migrating a lot of users and groups, something > is bound to fail :) and it's not good enough to have errors printed out > to stdout/stderr only. > >> - Can you add a validator for the LDAP uri and perhaps an example >> somewhere? > Sure. > >> - When migrating users/groups do we want an option to maintain >> existing uid/gid? > I don't know, I do keep them for now - some services might depend on > them and it wouldn't be nice if they changed after migration. So, I > think I'd rather add an option to generate new UID/GIDs. There might > also be collision with already existing users/groups, so we need to add > some checks. > >> - Is there a reason you're using pure ldap calls instead of the ldap2 >> plugin? > Yes, I wanted to use ldap2, I really did. But unfortunately ldap2 is > closely tied to the rest of the API (it's a plugin after all) and it is > currently impossible to even spawn a new ldap2 instance and make it > connect to an arbitrary LDAP server, because it makes assumptions about > the base DN and such. I really need to make it a bit more flexible, > because it would be a shame not to use it in these situations. I'll make > it a high priority task for myself. > >> rob > > Pavel > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel -- David O'Brien Red Hat Asia Pacific +61 7 3514 8189 http://freeipa.org/page/DocumentationPortal http://git.fedorahosted.org/git/ipadocs.git He who asks is a fool for five minutes, but he who does not ask remains a fool forever." ~ Chinese proverb From jderose at redhat.com Wed Dec 9 09:52:14 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Wed, 09 Dec 2009 02:52:14 -0700 Subject: [Freeipa-devel] [PATCH] dogtag clean-up In-Reply-To: <4B1ECDCE.8050708@redhat.com> References: <4B16FFBF.1010002@redhat.com> <4B1ECDCE.8050708@redhat.com> Message-ID: <1260352334.2482.4.camel@jgd-dsk> On Tue, 2009-12-08 at 17:06 -0500, John Dennis wrote: > On 12/02/2009 07:01 PM, John Dennis wrote: > > The essence of this patch is to return the correct types from > > certificate plugins and avoid scraping Javascript from dogtag (CMS) > > html responses with better error handling. Instead we ask CMS to > > always return our data as XML documents which can be much more > > robustly parsed (including properly handling issues such as character > > encoding, escapes, etc.). > > > > Fundamentally the process is split into two parts. A parsing routine > > which returns a dict with all the values from CMS in the correct > > Python types for IPA. The possible values returned from CMS are fully > > documented and can easily be read via the documentation link in HTML > > posted at the bottom (plus in the code of course). The command plugin > > invokes the parsing routine and picks out from the parse result dict > > the values it wants to return (and may optionaly convert the type as > > needed for XMLRPC, this is fully documented, in particular serial > > numbers need special handling in XMLRPC). This model allows us to use > > different parsing methods without disturbing the logic in the command > > plugin should that ever be necessary (i.e. clear separation of > > responsibilities). > > > > Status results are never returned in the command result. Instead we > > use the defined exception handling logic for IPA XMLRPC. If the > > command fails in some fashion we return a CertificateOperationError > > exception. On the receiving end if no exception has been thrown it > > knows the values returned are valid. > > > > Careful attention has been paid to the types being used. Strings are > > always unicode, integral values are represented as either int or long > > objects. No longer are integral values represented as strings with > > confusion as to thier radix representation (with the notable exception > > of serial numbers which must be passed through XMLRPC as decimal > > strings, the rules for this are fully documented). > > > > The logic in the selfsign and dogtag plugins have been brought into > > alignment. > > > > Much more extensive error checking has been added to selfsign to > > handle issues concering serial number operations. > > > > A new error exception has been added (CertificateOperationError). > > > > Error messages have been localized. > > > > The check_ra.py test was updated (unfortunately this test requires a > > configured server so I used my test server). > > > > Extensive documentation has been added to many of the routines. > > > > Easy to browse HTML documentation for the dogtag plugin can be found > > here (for the time being) > > > > http://jdennis.fedorapeople.org/ipa/dogtag > > > > I've noticed we have a bit of code duplication going on with CMS > > interactions. In the future we shold consolodate all CMS code in one > > library (module). > > > > This patch has been lingering in my private repo too long. I no longer > > want to keep merging as others modify the same code :-) So here it > > is. Other components of the fixes have already been posted as patches. > > The rebased patch is attached. May the gods of patchdom shine upon my > face and we'll celebrate it's successful application :-) ack. pushed to master. Very high quality patch, thanks John! From jderose at redhat.com Wed Dec 9 16:12:46 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Wed, 09 Dec 2009 09:12:46 -0700 Subject: [Freeipa-devel] [PATCH] jderose 027 Extensible return values In-Reply-To: <1259163317.24813.84.camel@jgd-dsk> References: <1259163317.24813.84.camel@jgd-dsk> Message-ID: <1260375166.2482.293.camel@jgd-dsk> Okay, here's a revised patch. Significant additions/changes from the previous version are: 1. The return value dict now includes a 'summary' value, something like 'Added user "jdoe"'. This summary is used by the CLI and webUI. Previously I was generating the summary in the CLI and webUI separately. This removes the duplication and allows the commands to easily produce arbitrary summaries (before they were limited a single summary format like 'Added user "%(primary_key)s"'. This also makes it easier for 3rd-party tools to provide UIs without having to introspect the Python API (because they happen to be written in PHP, whatever). 2. I renamed the 'primary_key' member in the return value dict to 'value'. This is simpler and will be will be easier on translators ('Added user "%(primary_key)s"' vs 'Added user "%(value)s"'). I'm also thinking of returning the name of the primary_key (e.g., 'uid') when returning an entry or a list of entries, so this opens the door for me to use 'key' in the future without confusion. Note this change is only relative to my previous proposed patch. The use of the return value dict hasn't yet hit master. 3. XMLRPC_test.setUp() no longer tests for server availability with `user-show notfound` prior to each test running. Instead, I try to connect to the server just once when the `xmlrcp_test` module loads, which sets the `server_available` module attribute. XMLRPC_test.setUp() will still raise nose.SkipTest for each test as before. This change helps the XMLRPC tests run much faster and also makes problems easier to debug server-side as there isn't all the `user-show notfound` background noise. 4. This adds my new `Declarative` base class for the XMLRPC tests which allows you to define the XMLRPC tests using simple data structures, letting the base class do the tedious stuff. IHMO, the tests are considerably faster and easier to write this way, but just as important is the fact that Declarative takes care of reporting the errors when a command's return value doesn't match what we expected. We have pretty good coverage in the XMLRCP tests, but we don't have very good reporting when something goes wrong. I've put a lot of effort into making sure typical error reports contain the information needed to quickly focus in on the problem. The most important part of the error reporting is in the new tests.util.assert_deepequal() function, which can be used by any test to compare two nested data structures. Currently only the test_user_plugin and test_group_plugin tests are using `Declarative`, but the rest will follow. 5. I rewrote the make-test script in Python and added a feature John asked for and one I wanted. John wanted the ability to easily run only the tests in one or more modules. You can now be specifying the module in Python notation or the module file. For example: ./make-test tests.test_xmlrpc.test_user_plugin Or equivalently: ./make-test tests/test_xmlrpc/test_user_plugin.py I wanted an easy way to use the nosetests --stop option, which causes the testing to abort upon reaching the first error, which I have found useful when updating plugins to one of my incompatible API changes. Use it like this: ./make-test --stop Yup, big! May my patch reviewers one day forgive me. -Jason On Wed, 2009-11-25 at 08:35 -0700, Jason Gerard DeRose wrote: > This patch is big. There are some significant changes, but easily 50% > of this patch is made up of small changes to plugins and their unit > tests to port them to the lower-level API changes. All unit tests pass > assuming you install without the --ca option. I didn't touch the > cert/ra plugins as I know John has pending changes there. > > Here's a summary: > > 1. Our standard return value is now a dict, which gives us a nice way to > extend the API without breaking clients/code. So in this respect, the > return values are more deeply nested. Basically if you returned 'foo' > before, now you return {'result': 'foo'}. > > > 2. Previously nested data-structures have been flattened. Most > importantly, these have been changed: > > Entry: [dn, entry] => {'result': entry} > Entries: [entries, truncated] => {'result': entries, 'truncated': truncated} > > This was mostly done so that an entry/row/object/whatever can be > consistently represented as a dict... whether it's from LDAP, or a > database, or whatever else IPA might get glued into. Likewise, lists > are now consistently used to represent lists of things, rather than also > a struct-like pair, triple, etc. > > > 3. We now have a way to specify a return value schema, and this schema > is enforced. The top-level schema of the return value dict is described > using the Param-like classes in ipalib/output.py. For example: > > from ipalib import output > > class user_add(LDAPCreate): > > has_output = ( > output.Entry('result'), > output.Output('primary_key', unicode, 'The user uid') > ) > > This hypothetical `user_add` expects you to return something like this: > > dict( > result=dict(givenname=u'Foo', sn=u'Bar', uid=u'fbar'), > primary_key=u'fbar', > ) > > A few commands have C clients whose API would break with these changes. > If your command is one of them, you can disable the return value > validation for now by setting the 'use_output_validation' attribute to > False, like this: > > class join(Command): > use_output_validation = False > > I already did this for join so the installer doesn't break. > > The plugin browser now shows the output schema. Just fire up the > in-tree server like this: > > ./lite-server.py > > And then go to http://localhost:8888/ipa/ui/Command > > > 4. I'm using the return values themselves to drive CLI and webUI output. > As I've worked on extending meta-data for the webUI, I've realized a lot > of it can be delivered in the return values themselves. This is good > because we display the output more on what the commands actually return, > less on what they merely say they will return... this keeps us honest > and is another way to ensure we live up to our API contracts. > > My goal is to have a single base output_for_cli() implementation, and > this patch get's us 90% there already. Not all the commands have been > updated, but this will continue in additional patches (or be delegated). > > More webUI functionality has been enabled also, but not as much as I > hoped for as work on the command plugins took so much time. The > commands return a page now instead of JSON. > > > 5. I stubbed out the remained missing gettext features we need. They > aren't hooked up to gettext yet, but for plugin writers, the API is now > complete. See ipalib/text.py for the implementation and > ipalib/plugins/user.py for a good example of how they are used. > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jderose-027.2-Extensible-return-values.patch Type: text/x-patch Size: 217565 bytes Desc: not available URL: From rcritten at redhat.com Wed Dec 9 22:17:19 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 09 Dec 2009 17:17:19 -0500 Subject: [Freeipa-devel] [PATCH] 327 pass debug option in client installer In-Reply-To: <4B198F92.6050907@redhat.com> References: <4B1981D4.10003@redhat.com> <4B198F92.6050907@redhat.com> Message-ID: <4B2021EF.7080804@redhat.com> John Dennis wrote: > On 12/04/2009 04:40 PM, Rob Crittenden wrote: >> Pass along the debug option in the client installer to the ipa-join >> command. > > ACK > pushed to master From rcritten at redhat.com Thu Dec 10 04:08:00 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 09 Dec 2009 23:08:00 -0500 Subject: [Freeipa-devel] [PATCH] jderose 027 Extensible return values In-Reply-To: <1260375166.2482.293.camel@jgd-dsk> References: <1259163317.24813.84.camel@jgd-dsk> <1260375166.2482.293.camel@jgd-dsk> Message-ID: <4B207420.6000704@redhat.com> Jason Gerard DeRose wrote: > Okay, here's a revised patch. > > Significant additions/changes from the previous version are: > > 1. The return value dict now includes a 'summary' value, something like > 'Added user "jdoe"'. This summary is used by the CLI and webUI. > Previously I was generating the summary in the CLI and webUI separately. > This removes the duplication and allows the commands to easily produce > arbitrary summaries (before they were limited a single summary format > like 'Added user "%(primary_key)s"'. This also makes it easier for > 3rd-party tools to provide UIs without having to introspect the Python > API (because they happen to be written in PHP, whatever). > > > 2. I renamed the 'primary_key' member in the return value dict to > 'value'. This is simpler and will be will be easier on translators > ('Added user "%(primary_key)s"' vs 'Added user "%(value)s"'). I'm also > thinking of returning the name of the primary_key (e.g., 'uid') when > returning an entry or a list of entries, so this opens the door for me > to use 'key' in the future without confusion. Note this change is only > relative to my previous proposed patch. The use of the return value > dict hasn't yet hit master. > > > 3. XMLRPC_test.setUp() no longer tests for server availability with > `user-show notfound` prior to each test running. Instead, I try to > connect to the server just once when the `xmlrcp_test` module loads, > which sets the `server_available` module attribute. XMLRPC_test.setUp() > will still raise nose.SkipTest for each test as before. This change > helps the XMLRPC tests run much faster and also makes problems easier to > debug server-side as there isn't all the `user-show notfound` background > noise. > > > 4. This adds my new `Declarative` base class for the XMLRPC tests which > allows you to define the XMLRPC tests using simple data structures, > letting the base class do the tedious stuff. IHMO, the tests are > considerably faster and easier to write this way, but just as important > is the fact that Declarative takes care of reporting the errors when a > command's return value doesn't match what we expected. We have pretty > good coverage in the XMLRCP tests, but we don't have very good reporting > when something goes wrong. I've put a lot of effort into making sure > typical error reports contain the information needed to quickly focus in > on the problem. The most important part of the error reporting is in > the new tests.util.assert_deepequal() function, which can be used by any > test to compare two nested data structures. Currently only the > test_user_plugin and test_group_plugin tests are using `Declarative`, > but the rest will follow. > > > 5. I rewrote the make-test script in Python and added a feature John > asked for and one I wanted. John wanted the ability to easily run only > the tests in one or more modules. You can now be specifying the module > in Python notation or the module file. For example: > > ./make-test tests.test_xmlrpc.test_user_plugin > > Or equivalently: > > ./make-test tests/test_xmlrpc/test_user_plugin.py > > I wanted an easy way to use the nosetests --stop option, which causes > the testing to abort upon reaching the first error, which I have found > useful when updating plugins to one of my incompatible API changes. Use > it like this: > > ./make-test --stop > > > Yup, big! May my patch reviewers one day forgive me. > > -Jason Ack. There are a couple of things we need to address such as porting the rest of the plugins to work with this new return value scheme but we can do that post-push. IMHO it is better to get this in now and clean up the few remaining items than to delay any further. We also need to try to avoid hardcoding domains in the tests. A couple of user tests look for dc=example,dc=com instead of api.env.basedn. rob From pzuna at redhat.com Thu Dec 10 13:29:52 2009 From: pzuna at redhat.com (Pavel Zuna) Date: Thu, 10 Dec 2009 14:29:52 +0100 Subject: [Freeipa-devel] IPA man page Message-ID: <4B20F7D0.30105@redhat.com> Okey, here's my first shot at the ipa man page. I didn't post it as a patch, so it's easier to review. You can use 'man ./ipa.1.gz' to read it from anywhere, just in case you didn't know - I didn't. :) Pavel -------------- next part -------------- A non-text attachment was scrubbed... Name: ipa.1.gz Type: application/x-gzip Size: 1810 bytes Desc: not available URL: From pzuna at redhat.com Thu Dec 10 13:33:44 2009 From: pzuna at redhat.com (Pavel Zuna) Date: Thu, 10 Dec 2009 14:33:44 +0100 Subject: [Freeipa-devel] Re: [PATCHES] Migration wrap-up. In-Reply-To: <4B1EE6F5.6070403@redhat.com> References: <4B151F28.60305@redhat.com> <4B19417E.3000107@redhat.com> <4B1D197E.1040609@redhat.com> <4B1EE6F5.6070403@redhat.com> Message-ID: <4B20F8B8.1030407@redhat.com> David O'Brien wrote: > If there's an easy way for me to get access to the messages and anything > else that gets put in front of the user I'm happy to review it. I > haven't done much (read "almost nothing") in the way of patch reviews, > so I'm not sure if that's the best way. Maybe I should just learn to > read patches... Oh no, patches are terrible to read! :) Anyway from now on, I'll try to send you things for review in non-patch form. > I'm also looking forward to seeing the wiki page with the migration > steps; something else for IPA doc. Sent in a private email. > David > Pavel From jderose at redhat.com Thu Dec 10 15:30:42 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Thu, 10 Dec 2009 08:30:42 -0700 Subject: [Freeipa-devel] [PATCH] jderose 027 Extensible return values In-Reply-To: <4B207420.6000704@redhat.com> References: <1259163317.24813.84.camel@jgd-dsk> <1260375166.2482.293.camel@jgd-dsk> <4B207420.6000704@redhat.com> Message-ID: <1260459042.21262.0.camel@jgd-dsk> On Wed, 2009-12-09 at 23:08 -0500, Rob Crittenden wrote: > Jason Gerard DeRose wrote: > > Okay, here's a revised patch. > > > > Significant additions/changes from the previous version are: > > > > 1. The return value dict now includes a 'summary' value, something like > > 'Added user "jdoe"'. This summary is used by the CLI and webUI. > > Previously I was generating the summary in the CLI and webUI separately. > > This removes the duplication and allows the commands to easily produce > > arbitrary summaries (before they were limited a single summary format > > like 'Added user "%(primary_key)s"'. This also makes it easier for > > 3rd-party tools to provide UIs without having to introspect the Python > > API (because they happen to be written in PHP, whatever). > > > > > > 2. I renamed the 'primary_key' member in the return value dict to > > 'value'. This is simpler and will be will be easier on translators > > ('Added user "%(primary_key)s"' vs 'Added user "%(value)s"'). I'm also > > thinking of returning the name of the primary_key (e.g., 'uid') when > > returning an entry or a list of entries, so this opens the door for me > > to use 'key' in the future without confusion. Note this change is only > > relative to my previous proposed patch. The use of the return value > > dict hasn't yet hit master. > > > > > > 3. XMLRPC_test.setUp() no longer tests for server availability with > > `user-show notfound` prior to each test running. Instead, I try to > > connect to the server just once when the `xmlrcp_test` module loads, > > which sets the `server_available` module attribute. XMLRPC_test.setUp() > > will still raise nose.SkipTest for each test as before. This change > > helps the XMLRPC tests run much faster and also makes problems easier to > > debug server-side as there isn't all the `user-show notfound` background > > noise. > > > > > > 4. This adds my new `Declarative` base class for the XMLRPC tests which > > allows you to define the XMLRPC tests using simple data structures, > > letting the base class do the tedious stuff. IHMO, the tests are > > considerably faster and easier to write this way, but just as important > > is the fact that Declarative takes care of reporting the errors when a > > command's return value doesn't match what we expected. We have pretty > > good coverage in the XMLRCP tests, but we don't have very good reporting > > when something goes wrong. I've put a lot of effort into making sure > > typical error reports contain the information needed to quickly focus in > > on the problem. The most important part of the error reporting is in > > the new tests.util.assert_deepequal() function, which can be used by any > > test to compare two nested data structures. Currently only the > > test_user_plugin and test_group_plugin tests are using `Declarative`, > > but the rest will follow. > > > > > > 5. I rewrote the make-test script in Python and added a feature John > > asked for and one I wanted. John wanted the ability to easily run only > > the tests in one or more modules. You can now be specifying the module > > in Python notation or the module file. For example: > > > > ./make-test tests.test_xmlrpc.test_user_plugin > > > > Or equivalently: > > > > ./make-test tests/test_xmlrpc/test_user_plugin.py > > > > I wanted an easy way to use the nosetests --stop option, which causes > > the testing to abort upon reaching the first error, which I have found > > useful when updating plugins to one of my incompatible API changes. Use > > it like this: > > > > ./make-test --stop > > > > > > Yup, big! May my patch reviewers one day forgive me. > > > > -Jason > > Ack. There are a couple of things we need to address such as porting the > rest of the plugins to work with this new return value scheme but we can > do that post-push. IMHO it is better to get this in now and clean up the > few remaining items than to delay any further. > > We also need to try to avoid hardcoding domains in the tests. A couple > of user tests look for dc=example,dc=com instead of api.env.basedn. > > rob Thanks. Pushed to master. I'll get on porting the few remaining plugins. From rcritten at redhat.com Fri Dec 11 22:39:57 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 11 Dec 2009 17:39:57 -0500 Subject: [Freeipa-devel] [PATCH] 330 remove delegation patch Message-ID: <4B22CA3D.6000009@redhat.com> The delegation patch was migrated from v1 and pretty much deprecated from the get-go. Lets finally put this thing down. It was replaced by the aci plugin. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-330-delegation.patch Type: application/mbox Size: 2680 bytes Desc: not available URL: From rcritten at redhat.com Fri Dec 11 22:41:14 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 11 Dec 2009 17:41:14 -0500 Subject: [Freeipa-devel] [PATCH] 331 add more options to make-test Message-ID: <4B22CA8A.60404@redhat.com> I like using the --pdb and --pdb-failures options with make-test. Add these to the make-test script to be passed along to nosetests. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-331-test.patch Type: application/mbox Size: 1114 bytes Desc: not available URL: From rcritten at redhat.com Fri Dec 11 22:42:13 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 11 Dec 2009 17:42:13 -0500 Subject: [Freeipa-devel] [PATCH] 332 aci return values Message-ID: <4B22CAC5.6070905@redhat.com> Convert the aci plugin to understand the new return values system. I had to do some hacks here because the aci plugin returns a single unicode value back representing the aci, not a set of attributes. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-332-aci.patch Type: application/mbox Size: 10301 bytes Desc: not available URL: From rcritten at redhat.com Fri Dec 11 22:42:52 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 11 Dec 2009 17:42:52 -0500 Subject: [Freeipa-devel] [PATCH] 333 add some labels Message-ID: <4B22CAEC.9080705@redhat.com> The hostgroup and netgroup plugins were missing some labels in their Params. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-333-labels.patch Type: application/mbox Size: 1707 bytes Desc: not available URL: From rcritten at redhat.com Fri Dec 11 22:43:21 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 11 Dec 2009 17:43:21 -0500 Subject: [Freeipa-devel] [PATCH] 334 add aci tests Message-ID: <4B22CB09.2080206@redhat.com> Add an extremely simple set of tests for the aci plugin. At this point something is better than nothing. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-334-acitest.patch Type: application/mbox Size: 3178 bytes Desc: not available URL: From jderose at redhat.com Sat Dec 12 06:05:01 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Fri, 11 Dec 2009 23:05:01 -0700 Subject: [Freeipa-devel] [PATCH] 328 force deletion of replica In-Reply-To: <4B1DD0AC.4080305@redhat.com> References: <4B1DD0AC.4080305@redhat.com> Message-ID: <1260597901.18590.0.camel@jgd-dsk> On Mon, 2009-12-07 at 23:06 -0500, Rob Crittenden wrote: > This adds an option to ipa-replica-manage, --force, that will let you > force the deletion of a replication agreement. > > Before this both ends had to be up and running for this to work, so that > the agreement could be removed on both sides. But what if the remote has > already been destroyed, either through an uninstall or the host went bye > bye. This will let you force remove it from the local instance. > > I run into this a lot with replication testing because I always forget > to remove the agreement before destroying a replica installation. > > rob ack. pushed to master. From jderose at redhat.com Sat Dec 12 06:06:39 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Fri, 11 Dec 2009 23:06:39 -0700 Subject: [Freeipa-devel] [PATCH] 330 remove delegation patch In-Reply-To: <4B22CA3D.6000009@redhat.com> References: <4B22CA3D.6000009@redhat.com> Message-ID: <1260597999.18590.1.camel@jgd-dsk> On Fri, 2009-12-11 at 17:39 -0500, Rob Crittenden wrote: > The delegation patch was migrated from v1 and pretty much deprecated > from the get-go. Lets finally put this thing down. It was replaced by > the aci plugin. > > rob ack. pushed to master. From jderose at redhat.com Sat Dec 12 06:07:09 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Fri, 11 Dec 2009 23:07:09 -0700 Subject: [Freeipa-devel] [PATCH] 331 add more options to make-test In-Reply-To: <4B22CA8A.60404@redhat.com> References: <4B22CA8A.60404@redhat.com> Message-ID: <1260598029.18590.2.camel@jgd-dsk> On Fri, 2009-12-11 at 17:41 -0500, Rob Crittenden wrote: > I like using the --pdb and --pdb-failures options with make-test. Add > these to the make-test script to be passed along to nosetests. > > rob Thanks for adding this, Rob. ack. pushed to master. From jderose at redhat.com Sat Dec 12 07:26:12 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Sat, 12 Dec 2009 00:26:12 -0700 Subject: [Freeipa-devel] [PATCH] 329 real services In-Reply-To: <4B1DD455.7080309@redhat.com> References: <4B1DD455.7080309@redhat.com> Message-ID: <1260602772.18590.3.camel@jgd-dsk> On Mon, 2009-12-07 at 23:21 -0500, Rob Crittenden wrote: > Make the IPA server host and its services "real" IPA entries > > We use kadmin.local to bootstrap the creation of the kerberos principals > for the IPA server machine: host, HTTP and ldap. This works fine and has > the side-effect of protecting the services from modification by an admin > (which would likely break the server). > > Unfortunately this also means that the services can't be managed by > useful utilities such as certmonger. So we have to create them as "real" > services instead. > > This is a relatively manual process so if the schema for hosts or > services changes this may require updates as well. > > There remains a minor problem. If you create a replica, during the > installation of that replica it will create host and service entries > too. But if you retire this replica those entries will remain. The next > time you try to install the replica it will fail with dupliate entries. > I'll address this in the future as the easy workaround is to run `ipa > host-del replica.example.com` and re-install the replica. > > rob ack. pushed to master. From ssorce at redhat.com Sat Dec 12 17:47:17 2009 From: ssorce at redhat.com (Simo Sorce) Date: Sat, 12 Dec 2009 12:47:17 -0500 Subject: [Freeipa-devel] [OUTAGE] Web site outage Message-ID: <1260640037.2498.3.camel@willson.li.ssimo.org> Sorry for the late communication. Our main website will be down this weekend for a scheduled outage that involves moving the hardware to a different location. The web site should be back online Monday. Simo. -- Simo Sorce * Red Hat, Inc * New York From jderose at redhat.com Mon Dec 14 20:37:15 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Mon, 14 Dec 2009 13:37:15 -0700 Subject: [Freeipa-devel] [PATCH] jderose 029 host and hostgroup messages, tests Message-ID: <1260823035.2548.4.camel@jgd-dsk> This patch: * Adds correct translatable `msg_summary` attributes on the host and hostgroup plugins * Rewrites the host and hostgroup unit tests as `Declarative` based tests and expands there coverage somewhat * Adds new tests.test_xmlrpc.objectclasses module where we can define the expected object classes is a single location -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jderose-029-host-hostgroup.pach Type: application/mbox Size: 26510 bytes Desc: not available URL: From jderose at redhat.com Mon Dec 14 20:46:53 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Mon, 14 Dec 2009 13:46:53 -0700 Subject: [Freeipa-devel] [PATCH] jderose 029 host and hostgroup messages, tests In-Reply-To: <1260823035.2548.4.camel@jgd-dsk> References: <1260823035.2548.4.camel@jgd-dsk> Message-ID: <1260823613.2548.5.camel@jgd-dsk> I attached this again in case the incorrect ".pach" extension caused problems for anyone. On Mon, 2009-12-14 at 13:37 -0700, Jason Gerard DeRose wrote: > This patch: > > * Adds correct translatable `msg_summary` attributes on the host and > hostgroup plugins > > * Rewrites the host and hostgroup unit tests as `Declarative` based > tests and expands there coverage somewhat > > * Adds new tests.test_xmlrpc.objectclasses module where we can define > the expected object classes is a single location > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jderose-029-host-hostgroup.patch Type: text/x-patch Size: 26510 bytes Desc: not available URL: From jderose at redhat.com Tue Dec 15 03:26:38 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Mon, 14 Dec 2009 20:26:38 -0700 Subject: [Freeipa-devel] [PATCH] 332 aci return values In-Reply-To: <4B22CAC5.6070905@redhat.com> References: <4B22CAC5.6070905@redhat.com> Message-ID: <1260847598.4187.0.camel@jgd-dsk> On Fri, 2009-12-11 at 17:42 -0500, Rob Crittenden wrote: > Convert the aci plugin to understand the new return values system. > > I had to do some hacks here because the aci plugin returns a single > unicode value back representing the aci, not a set of attributes. > > rob ack. pushed to master. From jderose at redhat.com Tue Dec 15 03:26:47 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Mon, 14 Dec 2009 20:26:47 -0700 Subject: [Freeipa-devel] [PATCH] 333 add some labels In-Reply-To: <4B22CAEC.9080705@redhat.com> References: <4B22CAEC.9080705@redhat.com> Message-ID: <1260847607.4187.1.camel@jgd-dsk> On Fri, 2009-12-11 at 17:42 -0500, Rob Crittenden wrote: > The hostgroup and netgroup plugins were missing some labels in their Params. > > rob ack. pushed to master. From jderose at redhat.com Tue Dec 15 03:26:58 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Mon, 14 Dec 2009 20:26:58 -0700 Subject: [Freeipa-devel] [PATCH] 334 add aci tests In-Reply-To: <4B22CB09.2080206@redhat.com> References: <4B22CB09.2080206@redhat.com> Message-ID: <1260847618.4187.2.camel@jgd-dsk> On Fri, 2009-12-11 at 17:43 -0500, Rob Crittenden wrote: > Add an extremely simple set of tests for the aci plugin. At this point > something is better than nothing. > > rob ack. pushed to master. From rcritten at redhat.com Tue Dec 15 17:22:58 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 15 Dec 2009 12:22:58 -0500 Subject: [Freeipa-devel] freeIPA wiki status Message-ID: <4B27C5F2.4040501@redhat.com> The machine hosting the freeIPA wiki was moved to a new datacenter this weekend. The move was successful and the machine is up and operating, the problem is that DNS hasn't been updated to reflect the new IP address. We are working on resolving this but at this time have no ETA on when that will be. We apologize for any inconvenience. rob From rcritten at redhat.com Tue Dec 15 17:51:47 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 15 Dec 2009 12:51:47 -0500 Subject: [Freeipa-devel] IPA man page In-Reply-To: <4B20F7D0.30105@redhat.com> References: <4B20F7D0.30105@redhat.com> Message-ID: <4B27CCB3.4070701@redhat.com> Pavel Zuna wrote: > Okey, here's my first shot at the ipa man page. I didn't post it as a > patch, so it's easier to review. > > You can use 'man ./ipa.1.gz' to read it from anywhere, just in case you > didn't know - I didn't. :) > > Pavel This is a great start. I have the following comments: I'd make clear that -e doesn't set shell environment variables, but IPA environment variables. Looks like -v and -n got mixed up in the description. I like your explanation at the end about getting specific help, can you add an example of it for clarity? I wonder if we need a section 5 man page for the default.conf format. rob From jderose at redhat.com Tue Dec 15 20:48:34 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Tue, 15 Dec 2009 13:48:34 -0700 Subject: [Freeipa-devel] [PATCH] Updated: jderose 029.2 host and hostgroup messages, tests Message-ID: <1260910114.4893.4.camel@jgd-dsk> This patch replaces my 029 patch. There was a conflict with Rob's freeipa-333-labels.patch, so I rebased my patch. I also fixed several Declarative tests that were broken with changes to baseldap (the 'dn' is now always returned). And from my original email: This patch: * Adds correct translatable `msg_summary` attributes on the host and hostgroup plugins * Rewrites the host and hostgroup unit tests as `Declarative` based tests and expands there coverage somewhat * Adds new tests.test_xmlrpc.objectclasses module where we can define the expected object classes is a single location -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jderose-029.2-host-hostgroup.patch Type: text/x-patch Size: 27586 bytes Desc: not available URL: From rcritten at redhat.com Wed Dec 16 21:15:15 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 16 Dec 2009 16:15:15 -0500 Subject: [Freeipa-devel] [PATCH] 336 remove debugging Message-ID: <4B294DE3.8010009@redhat.com> Remove a left-over debugging statement. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-336-debug.patch Type: application/mbox Size: 815 bytes Desc: not available URL: From rcritten at redhat.com Wed Dec 16 21:15:48 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 16 Dec 2009 16:15:48 -0500 Subject: [Freeipa-devel] [PATCH] 337 fix selinux contexts Message-ID: <4B294E04.8040909@redhat.com> Fix up the selinux context of some files needed by the selfsign CA. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-337-selinux.patch Type: application/mbox Size: 1366 bytes Desc: not available URL: From rcritten at redhat.com Wed Dec 16 21:16:46 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 16 Dec 2009 16:16:46 -0500 Subject: [Freeipa-devel] [PATCH] 338 make hosts more like IPA services Message-ID: <4B294E3E.3050401@redhat.com> Since the host entry contains the host/ principal it needs to look a bit more like a service in order to be able to store certificates in it. This should make IPA work better with certmonger. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-338-join.patch Type: application/mbox Size: 11081 bytes Desc: not available URL: From rcritten at redhat.com Wed Dec 16 22:28:55 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 16 Dec 2009 17:28:55 -0500 Subject: [Freeipa-devel] [PATCH] Updated: jderose 029.2 host and hostgroup messages, tests In-Reply-To: <1260910114.4893.4.camel@jgd-dsk> References: <1260910114.4893.4.camel@jgd-dsk> Message-ID: <4B295F27.8080407@redhat.com> Jason Gerard DeRose wrote: > This patch replaces my 029 patch. There was a conflict with Rob's > freeipa-333-labels.patch, so I rebased my patch. I also fixed several > Declarative tests that were broken with changes to baseldap (the 'dn' is > now always returned). And from my original email: > > This patch: > > * Adds correct translatable `msg_summary` attributes on the host and > hostgroup plugins > > * Rewrites the host and hostgroup unit tests as `Declarative` based > tests and expands there coverage somewhat > > * Adds new tests.test_xmlrpc.objectclasses module where we can define > the expected object classes is a single location Ack For your FIXME, the localityname is an alias for l, with l taking precedence. rob From jderose at redhat.com Wed Dec 16 22:55:10 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Wed, 16 Dec 2009 15:55:10 -0700 Subject: [Freeipa-devel] [PATCH] Updated: jderose 029.2 host and hostgroup messages, tests In-Reply-To: <4B295F27.8080407@redhat.com> References: <1260910114.4893.4.camel@jgd-dsk> <4B295F27.8080407@redhat.com> Message-ID: <1261004110.9013.0.camel@jgd-dsk> On Wed, 2009-12-16 at 17:28 -0500, Rob Crittenden wrote: > Jason Gerard DeRose wrote: > > This patch replaces my 029 patch. There was a conflict with Rob's > > freeipa-333-labels.patch, so I rebased my patch. I also fixed several > > Declarative tests that were broken with changes to baseldap (the 'dn' is > > now always returned). And from my original email: > > > > This patch: > > > > * Adds correct translatable `msg_summary` attributes on the host and > > hostgroup plugins > > > > * Rewrites the host and hostgroup unit tests as `Declarative` based > > tests and expands there coverage somewhat > > > > * Adds new tests.test_xmlrpc.objectclasses module where we can define > > the expected object classes is a single location > > Ack > > For your FIXME, the localityname is an alias for l, with l taking > precedence. > pushed to master. From jderose at redhat.com Thu Dec 17 02:52:23 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Wed, 16 Dec 2009 19:52:23 -0700 Subject: [Freeipa-devel] [PATCH] 336 remove debugging In-Reply-To: <4B294DE3.8010009@redhat.com> References: <4B294DE3.8010009@redhat.com> Message-ID: <1261018343.9013.1.camel@jgd-dsk> On Wed, 2009-12-16 at 16:15 -0500, Rob Crittenden wrote: > Remove a left-over debugging statement. > > rob ack. pushed to master. sorry for forgetting this in there. From jderose at redhat.com Thu Dec 17 02:52:35 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Wed, 16 Dec 2009 19:52:35 -0700 Subject: [Freeipa-devel] [PATCH] 337 fix selinux contexts In-Reply-To: <4B294E04.8040909@redhat.com> References: <4B294E04.8040909@redhat.com> Message-ID: <1261018355.9013.2.camel@jgd-dsk> On Wed, 2009-12-16 at 16:15 -0500, Rob Crittenden wrote: > Fix up the selinux context of some files needed by the selfsign CA. > > rob ack. pushed to master. From jderose at redhat.com Thu Dec 17 02:52:49 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Wed, 16 Dec 2009 19:52:49 -0700 Subject: [Freeipa-devel] [PATCH] 338 make hosts more like IPA services In-Reply-To: <4B294E3E.3050401@redhat.com> References: <4B294E3E.3050401@redhat.com> Message-ID: <1261018369.9013.3.camel@jgd-dsk> On Wed, 2009-12-16 at 16:16 -0500, Rob Crittenden wrote: > Since the host entry contains the host/ principal it needs to look a bit > more like a service in order to be able to store certificates in it. > > This should make IPA work better with certmonger. > > rob ack. pushed to master. From jderose at redhat.com Thu Dec 17 13:17:55 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Thu, 17 Dec 2009 06:17:55 -0700 Subject: [Freeipa-devel] [PATCH] jderose 030 Fuzzy feelings Message-ID: <1261055875.9013.298.camel@jgd-dsk> This patch: == 1. == Updates the host and hostgroup XMLRPC tests now that hosts have a 'managedby' attribute and the 'ipaservice' objectclass. == 2. == Changes assert_deepequal() so it normalizes tuples into lists. I did this because using lists in the Declarative tests makes things much more readable: uid=[u'fbar'], vs. uid=(u'fbar',), For readability I replaced the tuples in the existing Declarative tests with lists, although they will work either way. == 3. == Removes use of 'ignored_values' from the Declarative tests. After further thought, I decided against using the 'ignore_values' list because it wasn't intuitive, didn't read well, and led us to testing the ignored_values more loosely than needed. In its place is the new tests.util.Fuzzy class for doing loose equality tests. For example, if previously I would do this: expected=dict( result=dict( uid=[u'fbar'], ), ), ignore_values=['ipauniqueid', 'uidnumber'], I would now do this: expected=dict( result=dict( uid=[u'fbar'], ipauniqueid=[Fuzzy()], uidnumber=[Fuzzy()], ), ), By default a `Fuzzy` instance is equal to everything (so it only tests for existence in an entry). Although it seems pretty wired, these all evaluate to True: Fuzzy() == False 7 == Fuzzy() # Order doesn't matter Fuzzy() == u'Hello False, Lucky 7!' The first optional argument is a regular expression pattern to match (which implies the `unicode` type). You could match a phone number like this: u'123-456-7890' == Fuzzy('\d{3}-\d{3}-\d{4}') # True But because it sets an implied type of `unicode`, this evaluates to False: '123-456-7890' == Fuzzy('\d{3}-\d{3}-\d{4}') # False The 'type' kwarg allows you to specify a type constraint, like this: 42 == Fuzzy(type=int) # True 42.0 == Fuzzy(type=int) # False 42.0 == Fuzzy(type=(int, float)) # True Finally the 'test' kwarg is an optional callable that will be called to perform the loose equality test, like this: 42 == Fuzzy(test=lambda other: other > 42) # False 43 == Fuzzy(test=lambda other: other > 42) # True You can also use a type constraint with the 'test' kwarg, like this: 43 == Fuzzy(type=float, test=lambda other: other > 42) # False 42.5 == Fuzzy(type=float, test=lambda other: other > 42) # True So far `Fuzzy` only needs to be used for 'ipauniqueid', 'uidnumber', and 'gidnumber'. xmlrpc_test.py contains two pre-defined `Fuzzy` instances. `fuzzy_uuid` matches the 'ipauniqueid' using this burly regular expression: fuzzy_uuid = Fuzzy( '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' ) `fuzzy_digits` will match 'uidnumber' and 'gidnumber' (which at the moment are being returned as `str` instances, which I regard as a bug): fuzzy_digits = Fuzzy('^\d+$', type=str) == 4. == Removes all use of 'ignore_values' by replacing it with Fuzzy instances, cleans up the declarative tests a bit, and adds slightly more coverage. -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jderose-030-Fuzzy-feelings.patch Type: text/x-patch Size: 61119 bytes Desc: not available URL: From rcritten at redhat.com Thu Dec 17 16:22:45 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 17 Dec 2009 11:22:45 -0500 Subject: [Freeipa-devel] [PATCH] jderose 030 Fuzzy feelings In-Reply-To: <1261055875.9013.298.camel@jgd-dsk> References: <1261055875.9013.298.camel@jgd-dsk> Message-ID: <4B2A5AD5.6040807@redhat.com> Jason Gerard DeRose wrote: > This patch: > > == > 1. > == > Updates the host and hostgroup XMLRPC tests now that hosts have a > 'managedby' attribute and the 'ipaservice' objectclass. > > > == > 2. > == > Changes assert_deepequal() so it normalizes tuples into lists. I did > this because using lists in the Declarative tests makes things much more > readable: > > uid=[u'fbar'], > > vs. > > uid=(u'fbar',), > > For readability I replaced the tuples in the existing Declarative tests > with lists, although they will work either way. > > > == > 3. > == > Removes use of 'ignored_values' from the Declarative tests. After > further thought, I decided against using the 'ignore_values' list > because it wasn't intuitive, didn't read well, and led us to testing the > ignored_values more loosely than needed. In its place is the new > tests.util.Fuzzy class for doing loose equality tests. For example, if > previously I would do this: > > expected=dict( > result=dict( > uid=[u'fbar'], > ), > ), > ignore_values=['ipauniqueid', 'uidnumber'], > > I would now do this: > > expected=dict( > result=dict( > uid=[u'fbar'], > ipauniqueid=[Fuzzy()], > uidnumber=[Fuzzy()], > ), > ), > > By default a `Fuzzy` instance is equal to everything (so it only tests > for existence in an entry). Although it seems pretty wired, these all > evaluate to True: > > Fuzzy() == False > 7 == Fuzzy() # Order doesn't matter > Fuzzy() == u'Hello False, Lucky 7!' > > The first optional argument is a regular expression pattern to match > (which implies the `unicode` type). You could match a phone number like > this: > > u'123-456-7890' == Fuzzy('\d{3}-\d{3}-\d{4}') # True > > But because it sets an implied type of `unicode`, this evaluates to > False: > > '123-456-7890' == Fuzzy('\d{3}-\d{3}-\d{4}') # False > > The 'type' kwarg allows you to specify a type constraint, like this: > > 42 == Fuzzy(type=int) # True > 42.0 == Fuzzy(type=int) # False > 42.0 == Fuzzy(type=(int, float)) # True > > Finally the 'test' kwarg is an optional callable that will be called to > perform the loose equality test, like this: > > 42 == Fuzzy(test=lambda other: other > 42) # False > 43 == Fuzzy(test=lambda other: other > 42) # True > > You can also use a type constraint with the 'test' kwarg, like this: > > 43 == Fuzzy(type=float, test=lambda other: other > 42) # False > 42.5 == Fuzzy(type=float, test=lambda other: other > 42) # True > > So far `Fuzzy` only needs to be used for 'ipauniqueid', 'uidnumber', and > 'gidnumber'. xmlrpc_test.py contains two pre-defined `Fuzzy` instances. > > `fuzzy_uuid` matches the 'ipauniqueid' using this burly regular > expression: > > fuzzy_uuid = Fuzzy( > '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' > ) > > `fuzzy_digits` will match 'uidnumber' and 'gidnumber' (which at the > moment are being returned as `str` instances, which I regard as a bug): > > fuzzy_digits = Fuzzy('^\d+$', type=str) > > > == > 4. > == > Removes all use of 'ignore_values' by replacing it with Fuzzy instances, > cleans up the declarative tests a bit, and adds slightly more coverage. ack, pushed to master. rob From rcritten at redhat.com Thu Dec 17 16:32:34 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 17 Dec 2009 11:32:34 -0500 Subject: [Freeipa-devel] [PATCH] 339 fix some certificate issues Message-ID: <4B2A5D22.90805@redhat.com> Found a few problems with certificate handling with certmonger. Add a try/except to handle base64-encoded certificates more gracefully. I had also missed a function import causing things to blow up in some cases. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-339-cert.patch Type: application/mbox Size: 1978 bytes Desc: not available URL: From rcritten at redhat.com Thu Dec 17 19:31:53 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 17 Dec 2009 14:31:53 -0500 Subject: [Freeipa-devel] [PATCH] 340 suspect looping when deleting principals from keytab Message-ID: <4B2A8729.4060102@redhat.com> ipa-rmkeytab stopped working in F12. Turns out I'm supposed to disable looping when removing a keytab entry. Not sure why this worked in F-11 though, luck perhaps. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-340-rmkeytab.patch Type: application/mbox Size: 974 bytes Desc: not available URL: From jderose at redhat.com Fri Dec 18 07:14:21 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Fri, 18 Dec 2009 00:14:21 -0700 Subject: [Freeipa-devel] [PATCH] jderose 031 Fuzzy docstrings + spelling Message-ID: <1261120461.18974.8.camel@jgd-dsk> This patch: 1. Adds the '--doctest-tests' nosetests option in our ./make-test script so the doctests are run on all the modules in tests/* also. 2. Adds detailed examples in the Fuzzy class docstring... at Rob's request, I adapted the content from my last patch email, which now also gets doc-tested because of --doctest-tests 3. Fixes the broken assert_deepequal() docstring example (see, testing is good) 4. Fixes my rampant miss-spelling of 'existent' as 'existant'. In my defense, a few others made the mistake too, which I also fixed. ;) -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jderose-031-Fuzzy-docstring-plus-spelling.patch Type: text/x-patch Size: 12847 bytes Desc: not available URL: From jderose at redhat.com Fri Dec 18 11:52:29 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Fri, 18 Dec 2009 04:52:29 -0700 Subject: [Freeipa-devel] [PATCH] jderose 032 Add messages, declarative tests for rolegroup, taskgroup plugins Message-ID: <1261137149.18974.194.camel@jgd-dsk> This patch: 1. Adds translatable `msg_summary` attributes to the rolegroup and taskgroup plugins 2. Replaces the previous rolegroup and taskgroup XML-RCP tests with equivalent Declarative tests 3. Adds more rolegroup and taskgroup search tests, generally expands their test coverage -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jderose-032-rolegroup-taskgroup-messages-tests.patch Type: text/x-patch Size: 39052 bytes Desc: not available URL: From jderose at redhat.com Fri Dec 18 12:45:44 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Fri, 18 Dec 2009 05:45:44 -0700 Subject: [Freeipa-devel] [PATCH] 339 fix some certificate issues In-Reply-To: <4B2A5D22.90805@redhat.com> References: <4B2A5D22.90805@redhat.com> Message-ID: <1261140344.18974.248.camel@jgd-dsk> On Thu, 2009-12-17 at 11:32 -0500, Rob Crittenden wrote: > Found a few problems with certificate handling with certmonger. Add a > try/except to handle base64-encoded certificates more gracefully. I had > also missed a function import causing things to blow up in some cases. > > rob ack. pushed to master. From jderose at redhat.com Fri Dec 18 12:46:01 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Fri, 18 Dec 2009 05:46:01 -0700 Subject: [Freeipa-devel] [PATCH] 340 suspect looping when deleting principals from keytab In-Reply-To: <4B2A8729.4060102@redhat.com> References: <4B2A8729.4060102@redhat.com> Message-ID: <1261140361.18974.249.camel@jgd-dsk> On Thu, 2009-12-17 at 14:31 -0500, Rob Crittenden wrote: > ipa-rmkeytab stopped working in F12. Turns out I'm supposed to disable > looping when removing a keytab entry. Not sure why this worked in F-11 > though, luck perhaps. > > rob ack. pushed to master. From jdennis at redhat.com Fri Dec 18 13:32:11 2009 From: jdennis at redhat.com (John Dennis) Date: Fri, 18 Dec 2009 08:32:11 -0500 Subject: [Freeipa-devel] [PATCH] 339 fix some certificate issues In-Reply-To: <1261140344.18974.248.camel@jgd-dsk> References: <4B2A5D22.90805@redhat.com> <1261140344.18974.248.camel@jgd-dsk> Message-ID: <4B2B845B.4070800@redhat.com> On 12/18/2009 07:45 AM, Jason Gerard DeRose wrote: > On Thu, 2009-12-17 at 11:32 -0500, Rob Crittenden wrote: >> Found a few problems with certificate handling with certmonger. Add a >> try/except to handle base64-encoded certificates more gracefully. I had >> also missed a function import causing things to blow up in some cases. >> >> rob > > ack. pushed to master. Hmm... maybe this should have been NAK'ed. The issues were under active discussion. I don't think the patch is doing any harm but I'm not sure it's the right solution. Maybe the patch shouldn't have been applied. We have to be careful with our data types. The patch effectively was trying to determine if a certificate was encoded in binary DER format as opposed to base64 encoded PEM format by trying to base64 decode the certificate, if it successfully decoded it was assumed to be PEM. That's not the right way to handle this IMHO. We either need to: * adopt the convention that all certificates are in pem format when exchanged at an interface boundary * Have a method to unambiguously identify the certificate encoding, this could be done in one of two ways. 1. Always associate an encoding format attribute with the certificate 2. We do have the ability to unambiguously distinguish between binary objects and text objects. We could adopt the convention that if the data type of the certificate object is binary it is in DER format and if the data type of the certificate is TEXT then it's in PEM format. The distinction between binary and text is based on whether the object is a str class or a unicode class. The downside of this approach is we've haven't been rigorous with enforcing the correct data types, a problem compounded by the fact Python happily converts between str and unicode silently. Provided we're careful with using the right data type then the following would work: if type(cert) is unicode: cert_der = base64.b64decode(cert) else: cert_der = cert -or- if type(cert) is str: cert_pem = cert else: cert_pem = der_cert_to_pem(cert) What we don't want to do is start employing heuristics to guess the encoding, format, or data type of objects, it's not robust defensive coding practice. -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From jderose at redhat.com Fri Dec 18 13:36:33 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Fri, 18 Dec 2009 06:36:33 -0700 Subject: [Freeipa-devel] [PATCH] 339 fix some certificate issues In-Reply-To: <4B2B845B.4070800@redhat.com> References: <4B2A5D22.90805@redhat.com> <1261140344.18974.248.camel@jgd-dsk> <4B2B845B.4070800@redhat.com> Message-ID: <1261143393.18974.302.camel@jgd-dsk> On Fri, 2009-12-18 at 08:32 -0500, John Dennis wrote: > On 12/18/2009 07:45 AM, Jason Gerard DeRose wrote: > > On Thu, 2009-12-17 at 11:32 -0500, Rob Crittenden wrote: > >> Found a few problems with certificate handling with certmonger. Add a > >> try/except to handle base64-encoded certificates more gracefully. I had > >> also missed a function import causing things to blow up in some cases. > >> > >> rob > > > > ack. pushed to master. > > Hmm... maybe this should have been NAK'ed. The issues were under active > discussion. I don't think the patch is doing any harm but I'm not sure > it's the right solution. Maybe the patch shouldn't have been applied. Ah, sorry about that... I got the impression that this was an innocent stop-gap till we decide upon the details here. > We have to be careful with our data types. > > The patch effectively was trying to determine if a certificate was > encoded in binary DER format as opposed to base64 encoded PEM format by > trying to base64 decode the certificate, if it successfully decoded it > was assumed to be PEM. That's not the right way to handle this IMHO. > > We either need to: > > * adopt the convention that all certificates are in pem format when > exchanged at an interface boundary > > * Have a method to unambiguously identify the certificate encoding, this > could be done in one of two ways. > > 1. Always associate an encoding format attribute with the certificate > > 2. We do have the ability to unambiguously distinguish between binary > objects and text objects. We could adopt the convention that if the data > type of the certificate object is binary it is in DER format and if the > data type of the certificate is TEXT then it's in PEM format. > > The distinction between binary and text is based on whether the object > is a str class or a unicode class. The downside of this approach is > we've haven't been rigorous with enforcing the correct data types, a > problem compounded by the fact Python happily converts between str and > unicode silently. Provided we're careful with using the right data type > then the following would work: > > if type(cert) is unicode: > cert_der = base64.b64decode(cert) > else: > cert_der = cert > > -or- > > if type(cert) is str: > cert_pem = cert > else: > cert_pem = der_cert_to_pem(cert) > > What we don't want to do is start employing heuristics to guess the > encoding, format, or data type of objects, it's not robust defensive > coding practice. > From rcritten at redhat.com Fri Dec 18 15:56:34 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 18 Dec 2009 10:56:34 -0500 Subject: [Freeipa-devel] [PATCH] jderose 031 Fuzzy docstrings + spelling In-Reply-To: <1261120461.18974.8.camel@jgd-dsk> References: <1261120461.18974.8.camel@jgd-dsk> Message-ID: <4B2BA632.3010806@redhat.com> Jason Gerard DeRose wrote: > This patch: > > 1. Adds the '--doctest-tests' nosetests option in our ./make-test script > so the doctests are run on all the modules in tests/* also. > > 2. Adds detailed examples in the Fuzzy class docstring... at Rob's > request, I adapted the content from my last patch email, which now also > gets doc-tested because of --doctest-tests > > 3. Fixes the broken assert_deepequal() docstring example (see, testing > is good) > > 4. Fixes my rampant miss-spelling of 'existent' as 'existant'. In my > defense, a few others made the mistake too, which I also fixed. ;) ack, pushed to master rob From rcritten at redhat.com Fri Dec 18 15:56:46 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 18 Dec 2009 10:56:46 -0500 Subject: [Freeipa-devel] [PATCH] jderose 032 Add messages, declarative tests for rolegroup, taskgroup plugins In-Reply-To: <1261137149.18974.194.camel@jgd-dsk> References: <1261137149.18974.194.camel@jgd-dsk> Message-ID: <4B2BA63E.201@redhat.com> Jason Gerard DeRose wrote: > This patch: > > 1. Adds translatable `msg_summary` attributes to the rolegroup and > taskgroup plugins > > 2. Replaces the previous rolegroup and taskgroup XML-RCP tests with > equivalent Declarative tests > > 3. Adds more rolegroup and taskgroup search tests, generally expands > their test coverage ack, pushed to master From rcritten at redhat.com Fri Dec 18 15:59:25 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 18 Dec 2009 10:59:25 -0500 Subject: [Freeipa-devel] [PATCH] 341 remove hardcoded example.com domain name Message-ID: <4B2BA6DD.6030704@redhat.com> Remove hardcoded domain name so tests will pass on systems not configured with example.com. Note that I left example.com as the nisdomainname in the netgroup test because this won't affect the tests themselves. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-341-tests.patch Type: application/mbox Size: 3246 bytes Desc: not available URL: From rcritten at redhat.com Fri Dec 18 16:05:22 2009 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 18 Dec 2009 11:05:22 -0500 Subject: [Freeipa-devel] [PATCH] 342 control the certificate subject in dogtag Message-ID: <4B2BA842.5040305@redhat.com> Use the caIPAserviceCert profile for issuing service certs. This profile enables subject validation and ensures that the subject that the CA issues is uniform. The client can only request a specific CN, the rest of the subject is fixed. This is the first step of allowing the subject to be set at installation time. Also fix 2 more issues related to the return results migration. Note that with the selfsign plugin it will still issue the subject that was in the CSR. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-342-dogtag.patch Type: application/mbox Size: 2454 bytes Desc: not available URL: From jderose at redhat.com Fri Dec 18 16:43:53 2009 From: jderose at redhat.com (Jason Gerard DeRose) Date: Fri, 18 Dec 2009 09:43:53 -0700 Subject: [Freeipa-devel] [PATCH] 341 remove hardcoded example.com domain name In-Reply-To: <4B2BA6DD.6030704@redhat.com> References: <4B2BA6DD.6030704@redhat.com> Message-ID: <1261154633.18974.473.camel@jgd-dsk> On Fri, 2009-12-18 at 10:59 -0500, Rob Crittenden wrote: > Remove hardcoded domain name so tests will pass on systems not > configured with example.com. > > Note that I left example.com as the nisdomainname in the netgroup test > because this won't affect the tests themselves. > > rob ack. pushed to master. From davido at redhat.com Sun Dec 20 22:28:13 2009 From: davido at redhat.com (David O'Brien) Date: Mon, 21 Dec 2009 08:28:13 +1000 Subject: [Freeipa-devel] [PATCHES] Migration wrap-up. In-Reply-To: <4B20F8B8.1030407@redhat.com> References: <4B151F28.60305@redhat.com> <4B19417E.3000107@redhat.com> <4B1D197E.1040609@redhat.com> <4B1EE6F5.6070403@redhat.com> <4B20F8B8.1030407@redhat.com> Message-ID: <4B2EA4FD.9030406@redhat.com> Pavel Zuna wrote: > David O'Brien wrote: >> If there's an easy way for me to get access to the messages and >> anything else that gets put in front of the user I'm happy to review >> it. I haven't done much (read "almost nothing") in the way of patch >> reviews, so I'm not sure if that's the best way. Maybe I should just >> learn to read patches... > Oh no, patches are terrible to read! :) Anyway from now on, I'll try to > send you things for review in non-patch form. > >> I'm also looking forward to seeing the wiki page with the migration >> steps; something else for IPA doc. > Sent in a private email. > >> David >> > > Pavel > Thanks Pavel. I've been out of commission for several days and out of touch. I'll keep an eye out for all this. -- David O'Brien Red Hat Asia Pacific +61 7 3514 8189 http://freeipa.org/page/DocumentationPortal http://git.fedorahosted.org/git/ipadocs.git He who asks is a fool for five minutes, but he who does not ask remains a fool forever." ~ Chinese proverb