[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: problems with large directories?

On 03/08/2010 08:23 PM, Mitch Trachtenberg wrote:

I have an application that deals with 100,000 to 1,000,000 image files.

I initially structured it to use multiple directories, so that file
123456 would be stored in /12/34/123456.  I'm now wondering if that's
pointless, as it would simplify things to simply store the file in /123456.

Can anyone indicate whether I'm gaining anything by using smaller
directories in ext3/ext4?  Thanks.


I think that breaking up your files into subdirectories makes it easier to navigate the tree and find files from a human point of view. Even better if the bytes reflect something like year/month/day/hour/min (assuming your pathname has a date based guid or similar encoding).

You can have a million files in one large directory, but be careful to iterate and copy them in a sorted order (sorted by inode) to avoid nasty performance issues that are side effects of the way we hash file names in ext3/4.

Good luck!


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]