<div>>>Any errors?</div> <div>No as far as I can tell the import goes smooth. Our upstream applications seem to have no problem with the data it seems to be all imported. </div> <div> </div> <div>I am really troubleshooting and index and performance issues, I figure out of order keys could be slowing searches down. This a fairly large db 160000 entires) some objects have upwards of 20 attributes. </div> <div> </div> <div>db_verify and db_2index both run fairly quickly. It just seems like they cant both agree on what clean data looks like.</div> <div> </div> <div>FWI I upgraded to the latest 1.0.4 before all this testing.</div> <div> </div> <div>[18/Dec/2006:15:29:50 -0500] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [18/Dec/2006:15:29:50 -0500] - import idsk_services: Beginning import job... [18/Dec/2006:15:29:50 -0500] - import idsk_services: Index buffering enabled with bucket size 15 [18/Dec/2006:15:29:50 -0500] - import idsk_services: Processing file "/opt/idsk/downloads/idsk.services.com.ldif" [18/Dec/2006:15:29:50 -0500] - import idsk_services: Finished scanning file "/opt/idsk/downloads/idsk.services.com.ldif" (2037 entries) [18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers finished; cleaning up... [18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers cleaned up. [18/Dec/2006:15:29:51 -0500] - import idsk_services: Cleaning up producer thread... [18/Dec/2006:15:29:51 -0500] - import idsk_services: Indexing complete. Post-processing... [18/Dec/2006:15:29:51 -0500] - import idsk_services: Flushing caches... [18/Dec/2006:15:29:51 -0500] - import idsk_services: Closing files... [18/Dec/2006:15:29:51 -0500] - import idsk_services: Import complete. Processed 2037 entries in 1 seconds. ( 2037.00 entries/sec) [18/Dec/2006:15:30:28 -0500] - Bringing idsk_data offline... [18/Dec/2006:15:30:28 -0500] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [18/Dec/2006:15:30:28 -0500] - import idsk_data: Beginning import job... [18/Dec/2006:15:30:28 -0500] - import idsk_data: Index buffering enabled with bucket size 15 [18/Dec/2006:15:30:28 -0500] - import idsk_data: Processing file "/opt/idsk/downloads/idsk.com.ldif" [18/Dec/2006:15:30:48 -0500] - import idsk_data: Processed 77288 entries -- average rate 3680.4/sec, recent rate 3680.3/sec, hit ratio 0% [18/Dec/2006:15:31:13 -0500] - import idsk_data: Processed 141956 entries -- average rate 3086.0/sec, recent rate 3086.0/sec, hit ratio 100% [18/Dec/2006:15:31:35 -0500] - import idsk_data: Finished scanning file "/opt/idsk/downloads/idsk.com.ldif" (163684 entries) [18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers finished; cleaning up... [18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers cleaned up. [18/Dec/2006:15:31:49 -0500] - import idsk_data: Cleaning up producer thread... [18/Dec/2006:15:31:49 -0500] - import idsk_data: Indexing complete. Post-processing... [18/Dec/2006:15:32:20 -0500] - import idsk_data: Flushing caches... [18/Dec/2006:15:32:23 -0500] - import idsk_data: Closing files... [18/Dec/2006:15:32:24 -0500] - import idsk_data: Import complete. Processed 163684 entries in 116 seconds. ( 1411.07 entries/sec) Edward </div> <div>On 12/18/06, Noriko Hosoi <<a href="mailto:nhosoi@redhat.com">nhosoi@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">Eddie C wrote: > All, > > I tested this from scratch. > I used the ldif_2db function which did work much faster! 112 > seconds...rather then about 20 minutes. > However I think the verify_db and db2_index functions are not in > agreement. > > Create indexes. > After my initial import. > [root@ldap3 slapd-ldap3]# more out.after.final.txt > ***************************************************************** > verify-db: This tool should only be run if recovery start fails > and the server is down. If you run this tool while the server is > running, you may get false reports of corrupted files or other > false errors. > ***************************************************************** > Verify log files in db ... Good > Verify db/o_idsk_com/id2entry.db4 ... Good > Verify db/userRoot/ancestorid.db4 ... Good > Verify db/userRoot/entrydn.db4 ... Good > Verify db/userRoot/cn.db4 ... Good > Verify db/userRoot/numsubordinates.db4 ... Good > Verify db/userRoot/aci.db4 ... Good > Verify db/userRoot/parentid.db4 ... Good > Verify db/userRoot/objectclass.db4 ... Good > Verify db/userRoot/id2entry.db4 ... Good > Verify db/userRoot/nsUniqueId.db4 ... Good > Verify db/idsk_services/ancestorid.db4 ... > DB ERROR: db_verify: Page 4: out-of-order key at entry 252 > DB ERROR: db_verify: Page 7: out-of-order key at entry 194 > DB ERROR: db_verify: Page 7: out-of-order key at entry 450 > DB ERROR: db_verify: Page 11: out-of-order key at entry 69 > DB ERROR: db_verify: Page 11: out-of-order key at entry 325 > DB ERROR: db_verify: Page 11: out-of-order key at entry 581 > DB ERROR: db_verify: Page 12: out-of-order key at entry 22 > DB ERROR: db_verify: Page 16: out-of-order key at entry 249 > DB ERROR: db_verify: Page 16: out-of-order key at entry 498 > DB ERROR: db_verify: Page 16: out-of-order key at entry 754 > DB ERROR: db_verify: Page 17: out-of-order key at entry 195 > DB ERROR: db_verify: Page 17: out-of-order key at entry 451 > DB ERROR: db_verify: Page 17: out-of-order key at entry 707 > DB ERROR: db_verify: Page 18: out-of-order key at entry 148 > DB ERROR: db_verify: Page 21: out-of-order key at entry 254 > DB ERROR: db_verify: Page 21: out-of-order key at entry 510 > DB ERROR: db_verify: Page 21: out-of-order key at entry 766 > DB ERROR: db_verify: Page 22: out-of-order key at entry 207 > DB ERROR: db_verify: Page 22: out-of-order key at entry 463 > DB ERROR: db_verify: Page 22: out-of-order key at entry 719 > DB ERROR: db_verify: Page 23: out-of-order key at entry 160 > DB ERROR: db_verify: DB->verify: db/idsk_services/ancestorid.db4: > DB_VERIFY_BAD: Database verification failed > Secondary index file ancestorid.db4 in db/idsk_services is corrupted. > Please run db2index(.pl) for reindexing. > Is this after you imported your ldif file to the backend 'idsk_services'? When you imported the ldif file, did you get any errors from ldif2db? This is an sample output when we import Example.ldif. Did you see any messages other than these? [18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096, pages: 1009265, procpages: 23476 [18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache: 204800k [18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50, import_pages: 51200, pagesize: 4096 [18/Dec/2006:04:00:48 -0800] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096, pages: 1009265, procpages: 23476 [18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache: 204800k [18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50, import_pages: 51200, pagesize: 4096 [18/Dec/2006:04:00:49 -0800] - import example: Beginning import job... [18/Dec/2006:04:00:49 -0800] - import example: Index buffering enabled with bucket size 100 [18/Dec/2006:04:00:49 -0800] - import example: Processing file "/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif" [18/Dec/2006:04:00:49 -0800] - import example: Finished scanning file "/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif" (160 entries) [18/Dec/2006:04:00:50 -0800] - import example: Workers finished; cleaning up... [18/Dec/2006:04:00:50 -0800] - import example: Workers cleaned up. [18/Dec/2006:04:00:50 -0800] - import example: Cleaning up producer thread... [18/Dec/2006:04:00:50 -0800] - import example: Indexing complete. Post-processing... [18/Dec/2006:04:00:50 -0800] - import example: Flushing caches... [18/Dec/2006:04:00:50 -0800] - import example: Closing files... [18/Dec/2006:04:00:51 -0800] - All database threads now stopped [18/Dec/2006:04:00:51 -0800] - import example: Import complete. Processed 160 entries in 2 seconds. (80.00 entries/sec) Thanks, --noriko > So then i reran db2index....verify_db again...same result. > > Edward > > > On 12/15/06, *Eddie C* <<a href="mailto:edlinuxguru@gmail.com">edlinuxguru@gmail.com</a> > <mailto: <a href="mailto:edlinuxguru@gmail.com">edlinuxguru@gmail.com</a>>> wrote: > > I recently did an ldif backup of our iplanet 52 database. Its > about an 88 MB ldif file. > I took this to a new FDS server Dell 850 3 ghz duel core 2 sata > hard disks. > I ran an ldapadd the data imported perfectly. > Then I tried to cutover some systems and give the database some load. > > System went 200% processor > > Eventually I realized I was missing indexes so I added them > through the graphical tool. > > The log seemed to do something like this > generating index 1% > generating index 2% > .... > generating index 49% > Done > Seemed weird that they would jump from 49% to Done > At this point the new system was running at 100% processor > But the queries are running faster on our old 440 MHZ sparc t1 > server52 database > > I ran > DB ERROR: db_verify: Page 30: out-of-order key at entry 498 > DB ERROR: db_verify: DB->verify: db/o_com/channelcontentowner.db4: > DB_VERIFY_BAD: Database verification failed > > then I tried db2_index. The program seemed to be in a tight loop > complaining about 1 missing entry. > > I do not realize how the data can be so corrupted right after an > import. > > These are someone generic symptoms. Any ideas? Thanks > > > ------------------------------------------------------------------------ > > -- > Fedora-directory-users mailing list > <a href="mailto:Fedora-directory-users@redhat.com">Fedora-directory-users@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/fedora-directory-users">https://www.redhat.com/mailman/listinfo/fedora-directory-users </a> > -- Fedora-directory-users mailing list <a href="mailto:Fedora-directory-users@redhat.com">Fedora-directory-users@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/fedora-directory-users"> https://www.redhat.com/mailman/listinfo/fedora-directory-users</a> </blockquote></div>