problems with large directories?

[Ric Wheeler]

On 03/08/2010 08:23 PM, Mitch Trachtenberg wrote:
> Hi,
>
> I have an application that deals with 100,000 to 1,000,000 image files.
>
> I initially structured it to use multiple directories, so that file
> 123456 would be stored in /12/34/123456.  I'm now wondering if that's
> pointless, as it would simplify things to simply store the file in /123456.
>
> Can anyone indicate whether I'm gaining anything by using smaller
> directories in ext3/ext4?  Thanks.
>
> Mitch
>

I think that breaking up your files into subdirectories makes it easier to 
navigate the tree and find files from a human point of view. Even better if the 
bytes reflect something like year/month/day/hour/min (assuming your pathname has 
a date based guid or similar encoding).

You can have a million files in one large directory, but be careful to iterate 
and copy them in a sorted order (sorted by inode) to avoid nasty performance 
issues that are side effects of the way we hash file names in ext3/4.

Good luck!

Ric