[Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available

Fri Jan 27 18:42:21 UTC 2012

Sigbjorn Lie wrote:
> On 01/27/2012 06:15 PM, Sigbjorn Lie wrote:
>> On 01/27/2012 03:55 PM, Rob Crittenden wrote:
>>> Sigbjorn Lie wrote:
>>>>
>>>> On Fri, January 27, 2012 15:37, Rob Crittenden wrote:
>>>>> Stephen Gallagher wrote:
>>>>>
>>>>>> On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>
>>>>>>> The first naming context returned from the LDAP server is always
>>>>>>> chosen
>>>>>>> when using migrate-ds. This makes my import fail when I attempt
>>>>>>> to import users and groups from
>>>>>>> a previous LDAP server having more than 1 naming contexts available.
>>>>>>>
>>>>>>> The migrate-ds script should accept an option to specify what
>>>>>>> base_dn I
>>>>>>> would like to import from.
>>>>>>>
>>>>>>> Is there such an option today? I cannot find it...
>>>>>>>
>>>>>
>>>>> Not currently. I noticed this earlier in the week and opened a
>>>>> ticket on
>>>>> it, https://fedorahosted.org/freeipa/ticket/2314
>>>>>
>>>>>>
>>>>>> Just to add to this request, if the original LDAP server has a
>>>>>> defaultNamingContext attribute, it should be honored for
>>>>>> auto-detecting which base to migrate.
>>>>>
>>>>> I'll update the 2314 to ensure we don't forget about this. 389-ds just
>>>>> added support for defaultNamingContext.
>>>>>
>>>>
>>>> Ok, thank you.
>>>>
>>>> Anything I can do to work around this issue today? I suppose there
>>>> is just a file that need to be
>>>> hacked to set a set a value instead of the auto-detected value... ?
>>>>
>>>
>>> /usr/lib/python*/site-packages/ipalib/plugins/migration.py
>>>
>>> ~line 620 you'll see a block starting with the comment "retrieve DS
>>> base DN".
>>>
>>> Comment out the next 8 lines by prefixing them with # (these query to
>>> get the namingContext then pull the first value out).
>>>
>>> Add:
>>>
>>> ds_base_dn = 'dc=yourbasedn,dc=com'
>>>
>>> Alternatively you could always just add the above line to override
>>> what is detected. Commenting out just saves an LDAP lookup.
>>>
>>> Restart Apache.
>>
>>
>> I already found that file and did that earlier today, however I was
>> restarting tomcat6, not httpd... my bad. :)
>>
>> I have to specify --group-objectclass=posixGroup to get groups
>> imported, that's fine. But I only get a few users imported. I see that
>> by default it seem to be looking for objectclass=person. Only a few
>> user accounts have that objectclass associated, so I add
>> --user-objectclass=posixAccount as all users have this objectclass
>> associated with their account.
>>
>> $ ipa migrate-ds --user-container='ou=people'
>> --group-container='ou=group' --bind-dn='cn=directory manager'
>> --user-objectclass=account --group-objectclass=posixGroup
>> --schema=RFC2307 --continue ldap://ldapserver:399
>> ipa: ERROR: an internal error has occurred
>>
>> Not good. I look in the /var/log/httpd/error_log file, and I find:
>>
>> [Fri Jan 27 18:12:51 2012] [error] ipa: INFO: admin at NONE: ping(): SUCCESS
>> [Fri Jan 27 18:12:52 2012] [error] ipa: ERROR: non-public:
>> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1:
>> invalid continuation byte
>> [Fri Jan 27 18:12:52 2012] [error] Traceback (most recent call last):
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 228,
>> in wsgi_execute
>> [Fri Jan 27 18:12:52 2012] [error] result = self.Command[name](*args,
>> **options)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 432, in
>> __call__
>> [Fri Jan 27 18:12:52 2012] [error] ret = self.run(*args, **options)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 738, in run
>> [Fri Jan 27 18:12:52 2012] [error] return self.execute(*args, **options)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line
>> 634, in execute
>> [Fri Jan 27 18:12:52 2012] [error] ldap, config, ds_ldap, ds_base_dn,
>> options
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line
>> 513, in migrate
>> [Fri Jan 27 18:12:52 2012] [error] search_refs=True # migrated DS may
>> contain search references
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 188, in new_f
>> [Fri Jan 27 18:12:52 2012] [error] return f(*new_args, **kwargs)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 199, in new_f
>> [Fri Jan 27 18:12:52 2012] [error] return args[0].decode(f(*args,
>> **kwargs))
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode
>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>> in var)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
>> <genexpr>
>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>> in var)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in decode
>> [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var]
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode
>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>> in var)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
>> <genexpr>
>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>> in var)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 157, in decode
>> [Fri Jan 27 18:12:52 2012] [error] dct[k] = self._decode_dict_val(k, v)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 64, in
>> _decode_dict_val
>> [Fri Jan 27 18:12:52 2012] [error] return self.decode(val)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in decode
>> [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var]
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 132, in decode
>> [Fri Jan 27 18:12:52 2012] [error]
>> var.decode(self.encoder_settings.decode_from)
>> [Fri Jan 27 18:12:52 2012] [error] File
>> "/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode
>> [Fri Jan 27 18:12:52 2012] [error] return codecs.utf_8_decode(input,
>> errors, True)
>> [Fri Jan 27 18:12:52 2012] [error] UnicodeDecodeError: 'utf8' codec
>> can't decode byte 0xe5 in position 1: invalid continuation byte
>> [Fri Jan 27 18:12:52 2012] [error] ipa: INFO: admin at NONE:
>> migrate_ds(u'ldap://svg-p-idm02.none:389', u'********',
>> binddn=u'cn=directory manager', usercontainer=u'ou=people',
>> groupcontainer=u'ou=group', userobjectclass=(u'account',),
>> groupobjectclass=(u'posixGroup',), userignoreobjectclass=None,
>> userignoreattribute=None, groupignoreobjectclass=None,
>> groupignoreattribute=None, groupoverwritegid=False, schema=u'RFC2307',
>> continue=False, exclude_groups=None, exclude_users=None):
>> UnicodeDecodeError
>>
>>
>> Any suggestions?
>>
>>
> Oh yes and of course I've already looked for accounts with any non-utf8
> chars in any of the output of an ldapsearch of the same ldap tree I'm
> trying to import from...

This came up yesterday internally too. I don't believe a bug or ticket 
has been filed yet.

My best guess on what is happening, based on what I saw with our own 
case, is this:

A migrated attribute is coming in that IPA doesn't know about. We 
default to treating all attributes that don't require special treatment 
(like binary values) as utf-8. I'm guessing that an unknown binary 
attribute is being migrated, we try to utf-8 encode it and kablooey.

There is no easy workaround for the above problem right now because that 
happens before the --ignore* options are considered.

One idea that has come to mind but is untested is to grab the missing 
schema in the form of a discrete 389-ds-compatible schema file. Stop 
dirsrv, drop the file into /etc/dirsrv/slapd-YOURINSTANCE/schema, start 
dirsrv

Then retry the migration.

Use --ignore* if you don't want/need these attributes, then you stop 
dirsrv, remove the schema file, start dirsrv.

regards

rob