Best practice for bulk data transfer

[Robin Laing]

Andrew wrote:
> I just need to transfer a couple hundred thousand files off a **REALLY** 
> slow server to a faster server and to be able to document that file x on 
> slow server is identical to file x on faster server. It is not a 
> forensics problem per se -- but I do need to prove/document the process 
> -- so I'm thinking md5sums.
> 
> /** begin simplification **/
> 
> Procedure 1: The slow option
> 
> [slow server] calc md5sums > list && nc list > faster_server
> [slow server] rsync files to faster at server over a 10T crossover cable
> [faster server] check md5sums against list
> 
> Procedure 2: The faster option (repeated for each hard drive)
> 
> [slow server] halt
> pull hard drives off slow_server and place in removable tray on 
> faster_server
> [faster server] dd if= /dev/slowdrive ibs=4096 conv=notruc,noerror > 
> slowdrive.img
> [faster server] mount slowdrive.img && calc md5sums > list
> [faster server] rsync /mnt/slowdrive/ /newdir/
> [faster server] cd /newdir/ && check md5sums against list
> 
> /** end simplification **/
> 
> -The slow server is 6 years old -- no cdrom, no floppy :-)
> -Data corruption is totally unacceptable.
> 
> Questions:
> 1) Theory question: will Option 2 produce identical results to Option 1 
> only quicker ??
> 2) Practical question: is Option 2 "too risky" given the age of  the 
> drives ?? I have this fear of possibly harming a hard drive no matter 
> how careful .. is my fear justified or baseless ??
> 3) The better idea is to ... I'm always up for the better idea !!!
> 
> Thanks again and best wishes to everyone.
> 
> Andrew
> 

Are you afraid of a reboot on the old machine?

If not, why not put a new drive in the old machine to copy all the 
data to.  Then move the new drive to the new machine.  Similar to your 
option 2.

Drives are cheap and you probably wouldn't need that big of a drive.

Another option would be to use par2 and then you can compare and 
repair any damaged files.  This works for Usenet so why not in your 
case.  Heck, I am starting to do my backups this way.


-- 
Robin Laing

Holidays start today, Yea. :)