Copy with SCP - low performance

[duncan brown]

you may be better off encapsulating the first part, and if you were doing a tar across a wan, you'd want to use bzip2 or gzip compression on your two tar command lines.

(cd bar && tar c foo) | ssh user at blabla "cd /dir_on_blabla && tar x"

[though i do this] :

(cd /var/www/html/images && tar cfj - ./) | ssh user at system "cd /var/www/html/images && tar xvfj -"

same thing, basically... just different methods... and the v in the second tar gives you verbose output (which you may not want on a few million files)

-d

----- Original Message -----
From: Jean-Eric Cuendet <jec at rptec.ch>
Date: Tue, 30 Mar 2004 12:28:38 +0200
To: fedora-list at redhat.com
Subject: Re: Copy with SCP - low performance

> 
> > Horacio Reyes wrote:
> >
> >  >>- The files are copied in batch of 100 tif images, in a directory with
> >  >>more than 5 million files.
> >  >>- the server that initiate the scp is a 1.2Ghz P3 with 2G in ram, the
> >  >>receiver is a double Xeon 2.6ghz with 2G in ram. The first server takes
> >  >>no time to send the file, but the receiver takes more than 200segs to
> >  >>copy. They are in a 100mb lan.
> >  >>- the top comand shows 98% free cpu before the scp, when scp start it
> >  >>takes 100% of the 2 xeon cpu. The problem is not the ram, both servers
> >  >>have more than 1G free when scp starts.
> >  >>
> > 
> > I have found thet SCP does not like lots of small files, if you can tar
> > the files together into on big file it will probably transfer much 
> > quicker..
> 
> Do you know a command to tar through SSH which is the same as scp but 
> kepps better permissions and is quicker?
> Here it is:
> 
> You have bar/foo and you'd like to copy dir foo to server blabla:
>    cd bar && tar c foo | ssh user at blabla "cd /dir_on_blabla && tar x"
> 
> Great isn't it?
> -jec
> 
> 
> -- 
> fedora-list mailing list
> fedora-list at redhat.com
> To unsubscribe: http://www.redhat.com/mailman/listinfo/fedora-list
> 



+( duncan brown
+( duncanbrown at email.com
+( http://www.linuxadvocate.net

she's crashing her car in an intersection
she did not see the light
she's walking round and round in a shopping mall
as if she had no sight
dreams are only clouds that form and dissipate
the sky is a highway for metal birds
and land is real estate
she does not understand
her world depends upon a history of lies
she walks right by all her old high school friends 
for she has dollar signs in her eyes                                                                                                                             
  - the dead milkmen, "dollar signs in her eyes"