Copy with SCP - low performance

[Rodolfo J. Paiz]

At 04:28 3/30/2004, you wrote:
>>Horacio Reyes wrote:
>>  >>- The files are copied in batch of 100 tif images, in a directory with
>>  >>more than 5 million files.
>>  >>
>>I have found thet SCP does not like lots of small files, if you can tar
>>the files together into on big file it will probably transfer much quicker..
>
>Do you know a command to tar through SSH which is the same as scp but 
>kepps better permissions and is quicker?
>Here it is:
>
>You have bar/foo and you'd like to copy dir foo to server blabla:
>   cd bar && tar c foo | ssh user at blabla "cd /dir_on_blabla && tar x"

Good Lord, that's complicated. A simpler suggestion:

# rsync -ave ssh /home/user/bar user at otherserver:/home/user/

If you don't have the rsync command available, you can install it from the 
rsync package with "up2date rsync". This command will take the bar 
directory (and all its subdirectories and files) and copy it over to 
otherserver using the "user" account. The "bar" directory will be placed 
*inside /home/user* (note the trailing slash). Rsync intelligently checks 
files to see whether they've been previously copied, and copies just the 
changed files so it is usually used to synchronize or mirror directories in 
different places; and of course, the command shown keeps all permissions 
and ownership intact.

Once you have the initial copy performed, you can put this command into a 
cron job every hour and keep your directories synchronized permanently. 
Rsync is also the basis for the excellent "rsnapshot" backup project on 
SourceForge.

Cheers,


-- 
Rodolfo J. Paiz
rpaiz at simpaticus.com
http://www.simpaticus.com