[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: ext2/ext3 directory handling

On Sunday 18 May 2003 19:09, Michael Harris wrote:

htrees should speed up your directories i think, but they are not stable yet, 
as far as i know.

I think putting up the files via scp or ftp and not nfs or smb should also 
speed up things, because no directory-listing is made.

Perhaps there is a method to prevent samba from showing the directory-content 
to the clients, so the directory-listing would not happen.

> > On Sunday 18 May 2003 18:01, Michael Harris wrote:
> > > Just to add, I can attest that moving the files from the old dir to the
> > > new as described improves performance on my machines dramatically. In
> > > our service we end up with directories of 150k+ files which are
> > > generally touched only as they are added, though every file will be
> > > touched several times over a month. The files are each around 50kB.
> > > When the directory entry gets to be about 4MB it begins to take a long
> > > time for remote machines to copy files into the directory, maybe 4
> > > seconds for a 50kB file on a switched 100 base network. The performance
> > > hit is worst with remote machines using SMB. Compressing the directory
> > > entry with mkdir new cp old/* new/
> > >  	rm -rf old
> > >  	mv old new
> > > definitely improves things, but generally when there gets to be more
> > > than 200k files we have to roll over to a new directory to keep things
> > > moving. I suspect the remote machines are effectively downloading the
> > > directory entry with each copy to the server, but I also see the smbd
> > > tasks pegging on the server as well, but never really investigated it.
> > > We see this with ext2 and ext3. Not really looking for a solution here
> > > but just offering the info, but if anyone has a quick fix please share
> > > it. I may try resiserfs someday but for now we just use thousands of
> > > directories for the files. Mike
> >
> > which way do you normaly use to push the files when you don't use smb?
> NFS, though virtually all of the uploads to the directory are via SMB from
> NT machines. The local machine will write files as well, maybe 5% of those
> in the directory overall. Most of the file reads are from the local machine
> which takes a little hit with big directories, though reads from the NT
> machines over SMB get very slow but are infrequent. Splitting the files
> into many directories has been an acceptable fix.

e-admin internet gmbh
Andreas Gietl                                            tel +49 941 3810884
Ludwig-Thoma-Strasse 35                      fax +49 89 244329104
93051 Regensburg                                  mobil +49 171 6070008

PGP/GPG-Key unter http://www.e-admin.de/gpg.html

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]