[Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
Rob Crittenden
rcritten at redhat.com
Mon Jan 30 14:06:34 UTC 2012
Sigbjorn Lie wrote:
> On 01/27/2012 07:42 PM, Rob Crittenden wrote:
>> Sigbjorn Lie wrote:
>>> On 01/27/2012 06:15 PM, Sigbjorn Lie wrote:
>>>> On 01/27/2012 03:55 PM, Rob Crittenden wrote:
>>>>> Sigbjorn Lie wrote:
>>>>>>
>>>>>> On Fri, January 27, 2012 15:37, Rob Crittenden wrote:
>>>>>>> Stephen Gallagher wrote:
>>>>>>>
>>>>>>>> On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The first naming context returned from the LDAP server is always
>>>>>>>>> chosen
>>>>>>>>> when using migrate-ds. This makes my import fail when I attempt
>>>>>>>>> to import users and groups from
>>>>>>>>> a previous LDAP server having more than 1 naming contexts
>>>>>>>>> available.
>>>>>>>>>
>>>>>>>>> The migrate-ds script should accept an option to specify what
>>>>>>>>> base_dn I
>>>>>>>>> would like to import from.
>>>>>>>>>
>>>>>>>>> Is there such an option today? I cannot find it...
>>>>>>>>>
>>>>>>>
>>>>>>> Not currently. I noticed this earlier in the week and opened a
>>>>>>> ticket on
>>>>>>> it, https://fedorahosted.org/freeipa/ticket/2314
>>>>>>>
>>>>>>>>
>>>>>>>> Just to add to this request, if the original LDAP server has a
>>>>>>>> defaultNamingContext attribute, it should be honored for
>>>>>>>> auto-detecting which base to migrate.
>>>>>>>
>>>>>>> I'll update the 2314 to ensure we don't forget about this. 389-ds
>>>>>>> just
>>>>>>> added support for defaultNamingContext.
>>>>>>>
>>>>>>
>>>>>> Ok, thank you.
>>>>>>
>>>>>> Anything I can do to work around this issue today? I suppose there
>>>>>> is just a file that need to be
>>>>>> hacked to set a set a value instead of the auto-detected value... ?
>>>>>>
>>>>>
>>>>> /usr/lib/python*/site-packages/ipalib/plugins/migration.py
>>>>>
>>>>> ~line 620 you'll see a block starting with the comment "retrieve DS
>>>>> base DN".
>>>>>
>>>>> Comment out the next 8 lines by prefixing them with # (these query to
>>>>> get the namingContext then pull the first value out).
>>>>>
>>>>> Add:
>>>>>
>>>>> ds_base_dn = 'dc=yourbasedn,dc=com'
>>>>>
>>>>> Alternatively you could always just add the above line to override
>>>>> what is detected. Commenting out just saves an LDAP lookup.
>>>>>
>>>>> Restart Apache.
>>>>
>>>>
>>>> I already found that file and did that earlier today, however I was
>>>> restarting tomcat6, not httpd... my bad. :)
>>>>
>>>> I have to specify --group-objectclass=posixGroup to get groups
>>>> imported, that's fine. But I only get a few users imported. I see that
>>>> by default it seem to be looking for objectclass=person. Only a few
>>>> user accounts have that objectclass associated, so I add
>>>> --user-objectclass=posixAccount as all users have this objectclass
>>>> associated with their account.
>>>>
>>>> $ ipa migrate-ds --user-container='ou=people'
>>>> --group-container='ou=group' --bind-dn='cn=directory manager'
>>>> --user-objectclass=account --group-objectclass=posixGroup
>>>> --schema=RFC2307 --continue ldap://ldapserver:399
>>>> ipa: ERROR: an internal error has occurred
>>>>
>>>> Not good. I look in the /var/log/httpd/error_log file, and I find:
>>>>
>>>> [Fri Jan 27 18:12:51 2012] [error] ipa: INFO: admin at NONE: ping():
>>>> SUCCESS
>>>> [Fri Jan 27 18:12:52 2012] [error] ipa: ERROR: non-public:
>>>> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1:
>>>> invalid continuation byte
>>>> [Fri Jan 27 18:12:52 2012] [error] Traceback (most recent call last):
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 228,
>>>> in wsgi_execute
>>>> [Fri Jan 27 18:12:52 2012] [error] result = self.Command[name](*args,
>>>> **options)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 432, in
>>>> __call__
>>>> [Fri Jan 27 18:12:52 2012] [error] ret = self.run(*args, **options)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 738, in run
>>>> [Fri Jan 27 18:12:52 2012] [error] return self.execute(*args,
>>>> **options)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line
>>>> 634, in execute
>>>> [Fri Jan 27 18:12:52 2012] [error] ldap, config, ds_ldap, ds_base_dn,
>>>> options
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line
>>>> 513, in migrate
>>>> [Fri Jan 27 18:12:52 2012] [error] search_refs=True # migrated DS may
>>>> contain search references
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 188, in
>>>> new_f
>>>> [Fri Jan 27 18:12:52 2012] [error] return f(*new_args, **kwargs)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 199, in
>>>> new_f
>>>> [Fri Jan 27 18:12:52 2012] [error] return args[0].decode(f(*args,
>>>> **kwargs))
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
>>>> decode
>>>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>>>> in var)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
>>>> <genexpr>
>>>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>>>> in var)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in
>>>> decode
>>>> [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var]
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
>>>> decode
>>>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>>>> in var)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
>>>> <genexpr>
>>>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>>>> in var)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 157, in
>>>> decode
>>>> [Fri Jan 27 18:12:52 2012] [error] dct[k] = self._decode_dict_val(k, v)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 64, in
>>>> _decode_dict_val
>>>> [Fri Jan 27 18:12:52 2012] [error] return self.decode(val)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in
>>>> decode
>>>> [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var]
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 132, in
>>>> decode
>>>> [Fri Jan 27 18:12:52 2012] [error]
>>>> var.decode(self.encoder_settings.decode_from)
>>>> [Fri Jan 27 18:12:52 2012] [error] File
>>>> "/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode
>>>> [Fri Jan 27 18:12:52 2012] [error] return codecs.utf_8_decode(input,
>>>> errors, True)
>>>> [Fri Jan 27 18:12:52 2012] [error] UnicodeDecodeError: 'utf8' codec
>>>> can't decode byte 0xe5 in position 1: invalid continuation byte
>>>> [Fri Jan 27 18:12:52 2012] [error] ipa: INFO: admin at NONE:
>>>> migrate_ds(u'ldap://svg-p-idm02.none:389', u'********',
>>>> binddn=u'cn=directory manager', usercontainer=u'ou=people',
>>>> groupcontainer=u'ou=group', userobjectclass=(u'account',),
>>>> groupobjectclass=(u'posixGroup',), userignoreobjectclass=None,
>>>> userignoreattribute=None, groupignoreobjectclass=None,
>>>> groupignoreattribute=None, groupoverwritegid=False, schema=u'RFC2307',
>>>> continue=False, exclude_groups=None, exclude_users=None):
>>>> UnicodeDecodeError
>>>>
>>>>
>>>> Any suggestions?
>>>>
>>>>
>>> Oh yes and of course I've already looked for accounts with any non-utf8
>>> chars in any of the output of an ldapsearch of the same ldap tree I'm
>>> trying to import from...
>>
>> This came up yesterday internally too. I don't believe a bug or ticket
>> has been filed yet.
>>
>> My best guess on what is happening, based on what I saw with our own
>> case, is this:
>>
>> A migrated attribute is coming in that IPA doesn't know about. We
>> default to treating all attributes that don't require special
>> treatment (like binary values) as utf-8. I'm guessing that an unknown
>> binary attribute is being migrated, we try to utf-8 encode it and
>> kablooey.
>>
>> There is no easy workaround for the above problem right now because
>> that happens before the --ignore* options are considered.
>>
>> One idea that has come to mind but is untested is to grab the missing
>> schema in the form of a discrete 389-ds-compatible schema file. Stop
>> dirsrv, drop the file into /etc/dirsrv/slapd-YOURINSTANCE/schema,
>> start dirsrv
>>
>> Then retry the migration.
>>
>> Use --ignore* if you don't want/need these attributes, then you stop
>> dirsrv, remove the schema file, start dirsrv.
>>
>
> I did use the ignore flags, ended up with:
> --user-ignore-attribute=memberOf,mail,telephoneNumber,userPassword
> --user-ignore-objectclass=organizationalPerson,posixAccount,shadowAccount,inetOrgPerson,organizationalPerson
>
>
> Still no luck, same error.
>
> According to your testing: is there still some value that should have
> been ignored I've overlooked ?
>
> On the other hand, I've only been looking at entries within the ou's
> that contains users and groups. Will this issue also occur if there is
> non-UTF8 chars in other parts of the directory?
Like I said, this error is triggered before ignore is evaluated so if
an unknown binary attribute is getting decoded it will cause this
failure. The only solutions we have right now is to either load the
schema into IPA temporarily for the migration, rremove it on the remote
side or you could modify the query we make to find the remote entries to
pull only certain attributes. This last one would be tricky to get right.
The code looks like:
(entries, truncated) = ds_ldap.find_entries(
search_filter, ['*'], search_bases[ldap_obj_name],
ds_ldap.SCOPE_ONELEVEL,
time_limit=0, size_limit=-1,
search_refs=True # migrated DS may contain
search references
)
You'd want to replace ['*'] with ['attr1','attr2','attr3',...]. It would
be a rather long list and would need to cover both users and groups.
rob
More information about the Freeipa-users
mailing list