[Freeipa-devel] ds-migrate feature enhancements

Tue Dec 9 12:33:03 UTC 2014

On 12/05/2014 12:32 AM, Alan Evans wrote:
> Hello Martin,
> 
> I have spent some time looking into how to address
> https://fedorahosted.org/freeipa/ticket/3709.  I thought I would look at
> --setattr --addattr from user-add and others to see if I could reuse code
> for --user-attr-default.  At least for parsing the commandline arguments
> anyway.  It does not look like it would be easy to re-purpose the code in
> _convert_2_dict() from ipalib/plugins/baseldap.py (without blatantly
> copying it).  What are your thoughts about creating a Dict paramater type
> in ipalib/parameters.py to to user for --setaddr --addattr --delattr and to
> use for --user-attr-default?

Ah, I did not expect any integration with the standard baseldap.py --*attr
functions at all. Though reusing _convert_2_dict sounds as a good idea!

> Also I was looking at string.Template() vs string.Formatter().  I think
> string.Template is probably sufficient.  But I don't follow PEPs so I don't
> know if that's Python3 future proof or not.  Thoughts?

You do not need to worry with Python3 at this moment too much, string.Template
is already used by FreeIPA anyway.

> # Pseudocode
> (attr, default) = args['userdefaultattr'][0].split('=',2)
> entry_attrs[attr] = default.safe_substitute(entry_attrs.toDict())
> 
> In looking at ipalib/plugins/baseldap.py, I was wondering if It might make
> sense to make migrate more in line with the --*attr paramaters.
> 
> Consider:
> ipa migrate-ds --user-delattr ou
> 
> Instead of:
> ipa-migrate-ds --user-ignore-attribute ou

Hm, I would be afraid that you would get into a too much complexity if you try
to integrate the new functionality which was supposed to be rather simple with
*attr parameters processed process_attr_options that even do LDAP search do
process the merge the *attr options with existing LDAP entry. You do not need
to worry about that, you always work only on top of the migrated user entry
that you have available. That simplifies the processing a lot.

> Then we could also add user-{setattr,addattr,delattr,attrdefault} like:
> ipa migrate-ds --user-attrdefault "mail=$uid at example.com" --user-setattr
> "cn=$givenname $sn" --user-addattr description="Migrated $(date)"
> --user-delattr ou
> Which would add a mail address if one wasn't present, replace the CN with
> the first and last name, add a description attribute and get rid of the ou
> attribute.

This functionality may make sense as an extension, eventually. Currently, we
have nothing so even having

--user-attr-default
and
--user-attr-set

would go a long way.

> Another thought I had was about the behaivor of the search on the source
> directory.  Currently one provides --{user,group}-continer and
> --{user,group}-objectclass and the scope is statically defined in the
> script.
> 
> 
> --{user,group}-filter
> --------------------------
> 
> What about adding --{user,group}-scope ONE|SUB?

This is actually a very good idea and a valid option to add! We even have a
ticket in the FreeIPA Trac that no one brave enough took so far.

https://fedorahosted.org/freeipa/ticket/2547

If you decide to contribute this one, I would recommend doing in a separate
patch from your template work as it is parallel effort and would make the
review easier.

> 
> My directory for example is like:
> 
> dn: uid=me,ou=ops,ou=people,dc=example,dc=com
> dn: uid=joe,ou=eng,ou=people,dc=example,dc=com
> 
> So currently I have to use the --continue switch and change my
> --user-container to get all of my users.  If the scope weren't limited to
> ONELEVEL then I could get everyone in one pass.
> 
> ipa migrate-ds --user-container ou=People,dc=example,dc=com --user-scope SUB

This makes a lot of sense, +1 for adding the switches (should be very easy to
implement)

> Lastly I was thinking that --user-objectclass and --exclude-users could be
> made more powerful with --user-filter
> 
> ipa migrate-ds --user-container ou=People,dc=example,dc=com
> --user-objectclass posixAccount --user-filter '(!(loginShell=/bin/false))'
> or we could fire all of sales --user-filter '(!(ou=Sales))'
> 
> Which gets concatenated internally in the search to
> "(&(objectClass=posixAccount)(!(loginShell=/bin/false)))"

Another great idea. Currently we only create automated filter in

def construct_filter(template, oc_list):
    oc_subfilter = ''.join([ '(objectclass=%s)' % oc for oc in oc_list])
    return template % oc_subfilter

based on objectclass. Your proposal would make it much more flexible. This one
is also completely parallel so it should be in a separate patch compared to the
template work.

I am really looking forward to your patches! :-)

> On Thu, Dec 4, 2014 at 4:59 AM, Martin Kosek <mkosek at redhat.com> wrote:
> 
>> On 11/26/2014 11:52 AM, Martin Kosek wrote:
>>> On 11/24/2014 11:24 PM, Alan Evans wrote:
>>>> I am in the midst of preparing for a migration from OpenLDAP to FreeIPA.
>>>> ds-migrate wasn't going to fill all of my needs so I thought I would
>> use it
>>>> for most and then make up some LDIF's and massage them to do the last
>> bit
>>>> of migration.
>>>>
>>>> I have instead decided to extend ds-migrate and I think that my features
>>>> might be of use to others so I would like to contribute them.
>>>
>>> Great idea! :-) IMO, enhancing FreeIPA migration capability and making
>> it more
>>> even more pleasant experience for user users is a good goal.
>>>
>>>> Before I get
>>>> too for I wanted to get some input from the community.
>>>>
>>>> Here are MY original goals:
>>>> * Migrate ssh public keys
>>>>   The openssh-lpk schema is used in my tree so objectClass:
>> ldapPublicKey
>>>> attribute: sshPublicKey
>>>> * Migrate disabled accounts as disabled
>>>>   We 'disable' usere by setting their shadowExpire to a date in the past
>>>> and setting their shell to /bin/false
>>>>
>>>> I realized that the ssh-public key problem is more generally an
>> attribute
>>>> mapping problem and dealing with disabled users could be more
>> generalized
>>>> too.
>>>>
>>>> Here are instead the new features I would provide.
>>>>
>>>> * Attribute mapping
>>>>   Feature should check the new syntax exists and is the same as the old
>>>> syntax (perhaps further check for compatible syntax)
>>>>   --user-attribute-map=oldAttribute=newAttribute
>>>>   --group-attribute-map=foo=bar
>>>>   Should I drop user/group and just make it --attribute-map and apply
>> it to
>>>> both?
>>>>   Should certain attributes be mapped by default, i.e.
>>>> sshPublicKey=ipaSSHPubKey (this means we also need to ignore the
>>>> objectClass ldapPublicKey by default)  Maybe make a separate switch
>>>> --with-ssh-keys that automatically adds a map and an ignore?
>>>
>>> If the mapping sshPublicKey -> ipaSSHPubKey is clear, I would do it by
>> default
>>> and maybe add a switch to skip these advanced migrations. Just make sure
>> the
>>> expected format matches (ipa requires all 3 pieces of the public key,
>> not just
>>> the blob)
>>>
>>> I think it would be very useful to combine user attribute mapping with
>> the
>>> ideas from
>>>
>>> https://fedorahosted.org/freeipa/ticket/3709
>>>
>>> i.e. filling attribute with values from original entry or a default.
>> This may
>>> lead to following syntax:
>>>
>>> --user-attribute-default "sn=notdefined" --user-attribute-default
>>> "description=User with cn $cn"
>>>
>>> which would only use the pattern if attribute is not defined in original
>> entry
>>>
>>> and
>>>
>>> --user-attribute-map "sn=$cn"
>>>
>>> which would fill overwrite the attribute even if it was defined in the
>> original
>>> entry.
>>>
>>>>
>>>> * Handling disabled users
>>>>   1. How to identify disabled users?
>>>>     a. shadowExpire < now()
>>>>         --use-disable-shadow-expire
>>>
>>> How would you like to disable users then? With nsAccountLock attribute?
>>> Wouldn't it be better to instead map shadowExpire to
>> krbPrincipalExpiration
>>> attribute which should have sematically the same meaning?
>>>
>>>>     b. loginShell is one of configurable shells
>>>>         --use-disable-login-shell
>>>>         --disabled-shell=/bin/false --disabled-shell=/sbin/nologin
>> (these
>>>
>>> Is this really a general mechanism? I think Kerberos principal
>> expiration is
>>> the way to go:
>>>
>>> # ipa user-mod fbar --principal-expiration 20140101000000Z
>>>
>>> # ssh fbar@`hostname`
>>> fbar at vm-067.idm.lab.bos.redhat.com's password:
>>> Permission denied, please try again.
>>>
>>> # tail /var/log/krb5kdc.log
>>> ...
>>> Nov 26 04:56:51 vm-067.idm.lab.bos.redhat.com krb5kdc[19615](info):
>> AS_REQ (6
>>> etypes {18 17 16 23 25 26}) 10.16.78.67: CLIENT EXPIRED:
>>> fbar at IDM.LAB.BOS.REDHAT.COM for
>>> krbtgt/IDM.LAB.BOS.REDHAT.COM at IDM.LAB.BOS.REDHAT.COM, Client's entry in
>>> database has expired
>>>
>>> It is respected by Kerberos authentication and SSSD logins, at minimum.
>>>
>>>
>>>> two would be the defaults)
>>>>     c. nsAccountLocked (though that would be straight copied by the
>>>> migrator anyway
>>>
>>> +1 for copying.
>>>
>>>>     d. From Open DJ the attribute ds-pwp-account-disabled can be used to
>>>> identify disabled users
>>>>     (are there others?)
>>>>   2.  What do do with disabled users (in my case migrate and disable)
>>>>     a. Migrate them and don't touch nsAccountLocked
>>>>     b. Migrate them and set nsAccountLocked = true
>>>>        --disable-users
>>>>     c. Do not migrate them
>>>>        --skip-disabled-users
>>>>     d. Which is the default?  Migrate and disable?  If so which are the
>>>> default methods for identifying them?  All methods?
>>>
>>> I may be missing something, but to me following would make sense:
>>>
>>> - properly migrate and fill krbPrincipalExpiration and nsACcountLock
>> attributes
>>> - optionally implement --skip-disabled-users. Though it may be
>> challenging to
>>> implement the "is_user_disabled" function right so that it works on all
>> sort of
>>> DS schemes.
>>>
>>>> So is there anything I'm missing?  Any suggestions on the switches? I'm
>> not
>>>> entirely sure I like them the way they are.
>>>>
>>>> I have code to cover about 60% of the above already.  The user-attr-map
>>>> feature is working and the --disabled-users and disabled-shells options
>> are
>>>> working.
>>>>
>>>> Regards,
>>>> -Alan
>>
>> Hello Alan,
>>
>> Are there any updates in this noble effort? Are you stuck? Can I or the
>> team
>> help you in any way?
>>
>> Thanks,
>> Martin
>>
> 