[Fedora-directory-users] Re: Migration from i-planet 52
Richard Megginson
rmeggins at redhat.com
Thu Dec 21 19:16:49 UTC 2006
Eddie C wrote:
> >>Any errors?
> No as far as I can tell the import goes smooth. Our upstream
> applications seem to have no problem with the data it seems to be all
> imported.
>
> I am really troubleshooting and index and performance issues, I figure
> out of order keys could be slowing searches down. This a fairly large
> db 160000 entires) some objects have upwards of 20 attributes.
>
> db_verify and db_2index both run fairly quickly. It just seems like
> they cant both agree on what clean data looks like.
>
> FWI I upgraded to the latest 1.0.4 before all this testing.
Just to confirm, was the server running when you ran db_verify?
>
> [18/Dec/2006:15:29:50 -0500] - WARNING: Import is running with
> nsslapd-db-private-import-mem on; No other process is allowed to
> access the database
> [18/Dec/2006:15:29:50 -0500] - import idsk_services: Beginning import
> job...
> [18/Dec/2006:15:29:50 -0500] - import idsk_services: Index buffering
> enabled with bucket size 15
> [18/Dec/2006:15:29:50 -0500] - import idsk_services: Processing file
> "/opt/idsk/downloads/idsk.services.com.ldif"
> [18/Dec/2006:15:29:50 -0500] - import idsk_services: Finished scanning
> file "/opt/idsk/downloads/idsk.services.com.ldif" (2037 entries)
> [18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers finished;
> cleaning up...
> [18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers cleaned up.
> [18/Dec/2006:15:29:51 -0500] - import idsk_services: Cleaning up
> producer thread...
> [18/Dec/2006:15:29:51 -0500] - import idsk_services: Indexing
> complete. Post-processing...
> [18/Dec/2006:15:29:51 -0500] - import idsk_services: Flushing caches...
> [18/Dec/2006:15:29:51 -0500] - import idsk_services: Closing files...
> [18/Dec/2006:15:29:51 -0500] - import idsk_services: Import complete.
> Processed 2037 entries in 1 seconds. ( 2037.00 entries/sec)
> [18/Dec/2006:15:30:28 -0500] - Bringing idsk_data offline...
> [18/Dec/2006:15:30:28 -0500] - WARNING: Import is running with
> nsslapd-db-private-import-mem on; No other process is allowed to
> access the database
> [18/Dec/2006:15:30:28 -0500] - import idsk_data: Beginning import job...
> [18/Dec/2006:15:30:28 -0500] - import idsk_data: Index buffering
> enabled with bucket size 15
> [18/Dec/2006:15:30:28 -0500] - import idsk_data: Processing file
> "/opt/idsk/downloads/idsk.com.ldif"
> [18/Dec/2006:15:30:48 -0500] - import idsk_data: Processed 77288
> entries -- average rate 3680.4/sec, recent rate 3680.3/sec, hit ratio 0%
> [18/Dec/2006:15:31:13 -0500] - import idsk_data: Processed 141956
> entries -- average rate 3086.0/sec, recent rate 3086.0/sec, hit ratio 100%
> [18/Dec/2006:15:31:35 -0500] - import idsk_data: Finished scanning
> file "/opt/idsk/downloads/idsk.com.ldif" (163684 entries)
> [18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers finished;
> cleaning up...
> [18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers cleaned up.
> [18/Dec/2006:15:31:49 -0500] - import idsk_data: Cleaning up producer
> thread...
> [18/Dec/2006:15:31:49 -0500] - import idsk_data: Indexing complete.
> Post-processing...
> [18/Dec/2006:15:32:20 -0500] - import idsk_data: Flushing caches...
> [18/Dec/2006:15:32:23 -0500] - import idsk_data: Closing files...
> [18/Dec/2006:15:32:24 -0500] - import idsk_data: Import complete.
> Processed 163684 entries in 116 seconds. ( 1411.07 entries/sec)
>
> Edward
>
> On 12/18/06, *Noriko Hosoi* <nhosoi at redhat.com
> <mailto:nhosoi at redhat.com>> wrote:
>
> Eddie C wrote:
> > All,
> >
> > I tested this from scratch.
> > I used the ldif_2db function which did work much faster! 112
> > seconds...rather then about 20 minutes.
> > However I think the verify_db and db2_index functions are not in
> > agreement.
> >
> > Create indexes.
> > After my initial import.
> > [root at ldap3 slapd-ldap3]# more out.after.final.txt
> > *****************************************************************
> > verify-db: This tool should only be run if recovery start fails
> > and the server is down. If you run this tool while the server is
> > running, you may get false reports of corrupted files or other
> > false errors.
> > *****************************************************************
> > Verify log files in db ... Good
> > Verify db/o_idsk_com/id2entry.db4 ... Good
> > Verify db/userRoot/ancestorid.db4 ... Good
> > Verify db/userRoot/entrydn.db4 ... Good
> > Verify db/userRoot/cn.db4 ... Good
> > Verify db/userRoot/numsubordinates.db4 ... Good
> > Verify db/userRoot/aci.db4 ... Good
> > Verify db/userRoot/parentid.db4 ... Good
> > Verify db/userRoot/objectclass.db4 ... Good
> > Verify db/userRoot/id2entry.db4 ... Good
> > Verify db/userRoot/nsUniqueId.db4 ... Good
> > Verify db/idsk_services/ancestorid.db4 ...
> > DB ERROR: db_verify: Page 4: out-of-order key at entry 252
> > DB ERROR: db_verify: Page 7: out-of-order key at entry 194
> > DB ERROR: db_verify: Page 7: out-of-order key at entry 450
> > DB ERROR: db_verify: Page 11: out-of-order key at entry 69
> > DB ERROR: db_verify: Page 11: out-of-order key at entry 325
> > DB ERROR: db_verify: Page 11: out-of-order key at entry 581
> > DB ERROR: db_verify: Page 12: out-of-order key at entry 22
> > DB ERROR: db_verify: Page 16: out-of-order key at entry 249
> > DB ERROR: db_verify: Page 16: out-of-order key at entry 498
> > DB ERROR: db_verify: Page 16: out-of-order key at entry 754
> > DB ERROR: db_verify: Page 17: out-of-order key at entry 195
> > DB ERROR: db_verify: Page 17: out-of-order key at entry 451
> > DB ERROR: db_verify: Page 17: out-of-order key at entry 707
> > DB ERROR: db_verify: Page 18: out-of-order key at entry 148
> > DB ERROR: db_verify: Page 21: out-of-order key at entry 254
> > DB ERROR: db_verify: Page 21: out-of-order key at entry 510
> > DB ERROR: db_verify: Page 21: out-of-order key at entry 766
> > DB ERROR: db_verify: Page 22: out-of-order key at entry 207
> > DB ERROR: db_verify: Page 22: out-of-order key at entry 463
> > DB ERROR: db_verify: Page 22: out-of-order key at entry 719
> > DB ERROR: db_verify: Page 23: out-of-order key at entry 160
> > DB ERROR: db_verify: DB->verify: db/idsk_services/ancestorid.db4:
> > DB_VERIFY_BAD: Database verification failed
> > Secondary index file ancestorid.db4 in db/idsk_services is
> corrupted.
> > Please run db2index(.pl) for reindexing.
> >
> Is this after you imported your ldif file to the backend
> 'idsk_services'?
>
> When you imported the ldif file, did you get any errors from ldif2db?
> This is an sample output when we import Example.ldif. Did you see any
> messages other than these?
> [18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096,
> pages: 1009265, procpages: 23476
> [18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache:
> 204800k
> [18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50,
> import_pages: 51200, pagesize: 4096
> [18/Dec/2006:04:00:48 -0800] - WARNING: Import is running with
> nsslapd-db-private-import-mem on; No other process is allowed to
> access
> the database
> [18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096,
> pages: 1009265, procpages: 23476
> [18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache: 204800k
> [18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50,
> import_pages: 51200, pagesize: 4096
> [18/Dec/2006:04:00:49 -0800] - import example: Beginning import job...
> [18/Dec/2006:04:00:49 -0800] - import example: Index buffering enabled
> with bucket size 100
> [18/Dec/2006:04:00:49 -0800] - import example: Processing file
> "/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif"
> [18/Dec/2006:04:00:49 -0800] - import example: Finished scanning file
> "/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif"
> (160
> entries)
> [18/Dec/2006:04:00:50 -0800] - import example: Workers finished;
> cleaning up...
> [18/Dec/2006:04:00:50 -0800] - import example: Workers cleaned up.
> [18/Dec/2006:04:00:50 -0800] - import example: Cleaning up producer
> thread...
> [18/Dec/2006:04:00:50 -0800] - import example: Indexing complete.
> Post-processing...
> [18/Dec/2006:04:00:50 -0800] - import example: Flushing caches...
> [18/Dec/2006:04:00:50 -0800] - import example: Closing files...
> [18/Dec/2006:04:00:51 -0800] - All database threads now stopped
> [18/Dec/2006:04:00:51 -0800] - import example: Import complete.
> Processed 160 entries in 2 seconds. (80.00 entries/sec)
>
> Thanks,
> --noriko
> > So then i reran db2index....verify_db again...same result.
> >
> > Edward
> >
> >
> > On 12/15/06, *Eddie C* <edlinuxguru at gmail.com
> <mailto:edlinuxguru at gmail.com>
> > <mailto: edlinuxguru at gmail.com <mailto:edlinuxguru at gmail.com>>>
> wrote:
> >
> > I recently did an ldif backup of our iplanet 52 database. Its
> > about an 88 MB ldif file.
> > I took this to a new FDS server Dell 850 3 ghz duel core 2 sata
> > hard disks.
> > I ran an ldapadd the data imported perfectly.
> > Then I tried to cutover some systems and give the database
> some load.
> >
> > System went 200% processor
> >
> > Eventually I realized I was missing indexes so I added them
> > through the graphical tool.
> >
> > The log seemed to do something like this
> > generating index 1%
> > generating index 2%
> > ....
> > generating index 49%
> > Done
> > Seemed weird that they would jump from 49% to Done
> > At this point the new system was running at 100% processor
> > But the queries are running faster on our old 440 MHZ sparc t1
> > server52 database
> >
> > I ran
> > DB ERROR: db_verify: Page 30: out-of-order key at entry 498
> > DB ERROR: db_verify: DB->verify:
> db/o_com/channelcontentowner.db4:
> > DB_VERIFY_BAD: Database verification failed
> >
> > then I tried db2_index. The program seemed to be in a tight loop
> > complaining about 1 missing entry.
> >
> > I do not realize how the data can be so corrupted right after an
> > import.
> >
> > These are someone generic symptoms. Any ideas? Thanks
> >
> >
> >
> ------------------------------------------------------------------------
> >
> > --
> > Fedora-directory-users mailing list
> > Fedora-directory-users at redhat.com
> <mailto:Fedora-directory-users at redhat.com>
> > https://www.redhat.com/mailman/listinfo/fedora-directory-users
> <https://www.redhat.com/mailman/listinfo/fedora-directory-users>
> >
>
>
>
> --
> Fedora-directory-users mailing list
> Fedora-directory-users at redhat.com
> <mailto:Fedora-directory-users at redhat.com>
> https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
>
>
>
> ------------------------------------------------------------------------
>
> --
> Fedora-directory-users mailing list
> Fedora-directory-users at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3245 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://listman.redhat.com/archives/fedora-directory-users/attachments/20061221/594179e6/attachment.bin>
More information about the Fedora-directory-users
mailing list