[Pki-devel] replication of new/modified profiles

Wed Jul 2 18:09:16 UTC 2014

In general, I think we all agree that putting profiles on ldap  is a 
good idea.  However, in practice, I think we want to allow admins to 
make changes to local files as they see fit.
  Please see my in-line response below.

On 07/02/2014 10:17 AM, Ade Lee wrote:
> On Tue, 2014-07-01 at 12:42 +1000, Fraser Tweedale wrote:
>> On Mon, Jun 30, 2014 at 02:28:09PM -0500, Endi Sukma Dewata wrote:
>>> Hi, I have some comments about the design:
>>> http://pki.fedoraproject.org/wiki/LDAP_Profile_Storage
>>>
>>> I think all system/default profiles should remain file-based and all custom
>>> profiles should be LDAP-based. It will make a clean separation: system
>>> profiles are owned by us (Dogtag developers), custom profiles are owned by
>>> the admin.
>>>
>>> The system profiles will be read-only. This way we will be able to update
>>> the system profiles without writing any upgrade scripts because the files
>>> will be updated automatically by RPM. Just one requirement, all server
>>> instances must be upgraded to the same version.
>>>
>>> If the admin wants to change a system profile, they can clone it into a
>>> custom profile and make the changes there. The custom profiles cannot have
>>> the same names as the system profiles, so there's won't be any
>>> conflict/confusion, and no need to support a "restore" command. In general
>>> we won't need to write upgrade scripts for custom profiles except if we
>>> change the LDAP schema.
>>>
>> This was my original proposal, with the exception of making
>> system/default profiles read-only and cloning them to make
>> modifications.  I think this is a good modification to that
>> proposal.
>>
>> Ade was in favour of moving to LDAP only, but perhaps this variation
>> makes using a mix of LDAP and file-based profiles more palatable?
>> It would be good to hear from more people on this part of the design
>> - to see if there's a clear winner.
>>
Perhaps we can view files v.s. ldap as "localized" v.s. "centralized" 
instead of "system" v.s. "customized".
Again, the benefit of putting profiles in ldap is so that it can be 
configured once and utilized by many.
My comments below are based on such view.

> The problem with the above proposal is one of migration.  Customers are
> used to both creating new profiles and modifying existing profiles to
> suit their needs.  If we go along with this mechanism, we would have to
> tell these customers that they would need to rename their existing
> customized profiles (and update all the relevant clients etc.)
>
> There is another problem, and that is that it is not clear that we want
> updates to the default profiles to be propagated to existing instances.
> I have looked at the profiles and there have been only a handful of
> changes over the last 7 years.  Those changes include things like
> updating the default signing algorithms or the default validity.  More
> likely than not, admins would prefer that we not change the behavior of
> profiles in existing instances underneath them.
>
> The changes that I have found are all behavioral - and therefore things
> that admin can opt out of -- or would prefer to do on their own
> schedule.  There have been no structural changes.
>
> If there are structural changes, then we need to (and can) provide an
> upgrade script which would run with the automatic upgrade.  An example
> of this would be a schema upgrade as we sort out how to represent
> profiles in LDAP.
>
> So, I think we should do the following:
> 1. Continue to provide the system profiles in files.  These files will
> be parsed and stored in LDAP when an instance is created.
if the ldap copies are the "centralized" ones, then we don't want to 
override them at each installation of an instance. We could give admins 
an option to override what's stored in ldap during configuration.
When a profile with the same id exists in both ldap and local system, 
then by default the one on ldap takes precedence.  Although we could add 
"per profile" config param for each instance to change the view 
(remember the unix ns/yp config where you can set the order of the 
source? so if "file" is 1st in the order, then the local ones have 
priority, etc.)
With this, a site that has carefully crafted their profiles and stored 
in ldap to share among all ca instances don't have to bother changing 
anything for each ca instance created (because they are default to the 
ldap).
Upgrade will also not affect the ldap copies inadvertently.
>
> 2. All profiles for an instance should live in LDAP.  This makes it
> simple - no need to check to see if a profile is in LDAP or files or
> both, and which has priority etc.  Tools will be provided to manage/
> create/delete profiles etc.
It might be simple, but we need to think about what's the value of them 
existing in ldap and not being shared by others.
>
> 3. Updates to system profile files will not affect the existing LDAP
> profiles.  We can provide update scripts or manual instructions for
> admins to run when they opt to do so.  This will be for behavioral
> changes.
so, in the view I specified above, update to the "local" profiles will 
only affect the  instance where they exist, however, we can allow option 
for admins to "publish" to ldap.
>
> 4. Structural changes will be done using upgrade scripts using the
> database upgrade mechanism.  This framework is something that we had
> planned to do already in 10.3.  We already have a model on how to do
> this in our current upgrade framework.
>   
>>> About the schema itself, I suppose we want to have something that resembles
>>> the actual Profile data structure (see ProfileData Java class). There should
>>> be an LDAP attribute for each single-valued Java attribute (e.g. name,
>>> description, enabled, visible). This way the profile is more manageable and
>>> can be queried based on these attributes. For collection attributes (e.g.
>>> inputs, outputs, policySets) we can use child LDAP entries to represent
>>> them.
>>>
>>> About the names, right now we use "profile" to refer to both cert profile in
>>> CA and token profile in TPS. We probably should use separate terms to
>>> distinguish them (e.g. CertProfile and TokenProfile). This applies to
>>> Java/Python class names, REST interface, CLI, LDAP schema, etc.
>>>
>> Good point.  I've s/profile/certProfile/ in the schema.
>>
>>> So the LDAP entries for a profile may look like this:
>>>
>>> dn: cn=MyUserCert,ou=Profiles,ou=CA,{suffix}
>>> objectClass: certProfile
>>> cn: MyUserCert
>>> displayName: Manual User Dual-Use Certificate Enrollment
>>> description: This certificate profile is for enrolling user certificates.
>>> visible: true
>>> enabled: true
>>> enabledBy: admin
>>>
>>> dn: ou=Inputs,cn=MyUserCert,ou=Profiles,ou=CA,{suffix}
>>> objectClass: organizationalUnit
>>> ou: Inputs
>>>
>>> dn: cn=i1,ou=Inputs,cn=MyUserCert,ou=Profiles,ou=CA,o=pki-tomcat
>>> objectClass: pkiCertProfileInput
>>> cn: i1
>>> classId: keyGenInputImpl
>>>
>> Introducing this schema (and the more fine-grained functionality it
>> would support) could happen later.  I see the benefits but I think
>> it's better to just get the profiles in LDAP as soon as possible,
>> then look at how we can break them down to enable richer features.
>>
> I would prefer to see if we could design something more fine grained at
> first, rather than assuming an upgrade.  If nothing else, you will need
> to separate out whether the profile is visible and enabled and by whom.
>
>> Also, if we retain file-based profiles as you suggest, whatever
>> queries LDAP makes easier, we would still need to be able to do
>> those same queries across the file-based profiles.  So while
>> file-based profiles remain, a finer-grained LDAP profile schema
>> doesn't buy us anything in terms of reducing complexity or the
>> amount of code to write.
>>
>>> About the REST interface & CLI, since this will be the primary way to edit
>>> profiles, we might want to have more granular commands to modify parts of
>>> the profile. Right now with ca-profile-mod command you need to send the
>>> entire profile in a file. It would be nice to be able to specify some
>>> parameters to change certain attributes only, or use separate commands to
>>> manage the inputs/outputs.
>>>
>>> We'll also need an interface to find existing cert records that use a
>>> certain profile and bulk modify them to use a different profile. This will
>>> be useful when you create a clone to change the system profile.
>>>
>> Again, I agree with the benefits here, but feel that this may be out
>> of scope for the initial changeover to LDAP profiles, given how much
>> we need that for FreeIPA.
>>
> Agreed. I think this can be a later development.
>
>>> I'm not sure if we should support 10.2 -> 10.3 cloning. When we release 10.3
>>> the 10.2 will still be fairly new so it might be reasonable to require all
>>> clones to be upgraded. It will reduce the amount of testing requirement too.
>>>
>> I'll incorporate this comment (and all your others) in the design
>> proposal; you should see the updated document soon.
>>
>> Cheers,
>>
>> Fraser
>>
>>> -- 
>>> Endi S. Dewata
>>>
>>> On 6/18/2014 2:44 AM, Fraser Tweedale wrote:
>>>> Hi all,
>>>>
>>>> A requirement from the FreeIPA side is the ability to add and
>>>> customise CA profiles.  Dogtag's current profile creation behaviour
>>>> writes the new profile to the filesystem beside the standard
>>>> profiles (as well as making the appropriate update to the registry,
>>>> etc.)
>>>>
>>>> There does not seem to be a mechanism to distribute new/modified
>>>> profiles to replicas - though perhaps I have missed something.
>>>>
>>>> Because this behaviour is required, unless I have overlooked
>>>> something or there is a better way (in which case please shout out),
>>>> I think it makes sense to begin a design proposal for an LDAP-based
>>>> profile store.
>>>>
>>>> Finally, a brief mention of some tickets related to profile storage
>>>> that could be good to tackle simultaneously should the proposed
>>>> change go ahead:
>>>>
>>>> - https://fedorahosted.org/pki/ticket/778
>>>> - https://fedorahosted.org/freeipa/ticket/4002
>>>>
>>>> _______________________________________________
>>>> Pki-devel mailing list
>>>> Pki-devel at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pki-devel
>>>>
>
> _______________________________________________
> Pki-devel mailing list
> Pki-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/pki-devel