optimising filesystem for many small files

[Peter Grandi]

>>>> Hi, System : Fedora 11 x86_64 Current Filesystem: 150G ext4
>>>> (formatted with "-T small" option) Number of files: 50
>>>> Million, 1 to 30K png images We are generating these files
>>>> using a python programme and getting very slow IO
>>>> performance. While generation there in only write, no
>>>> read. After generation there is heavy read and no write.
>>>> I am looking for best practices/recommendation to get a
>>>> better performance.  Any suggestions of the above are
>>>> greatly appreciated.

The first suggestion is to report issues in a less vague way.

Perhaps you are actually getting very good performance, but it is
not sufficient for the needs of your application; or perhaps you
are really getting poor performance, and emoving the cause would
make it sufficient for the needs of your application. But no
information on what is the current and what is the desired
performance is available, and that should have been the first
thing stated.

>>> [ ... ] these files are not in a single directory, this is a
>>> pyramid structure. There are total 15 pyramids and coming down
>>> from top to bottom the sub directories and files are
>>> multiplied by a factor of 4. The IO is scattered all over!!!!
>>> [ ... ]

Is that a surprise? First one creates a marvellously pessimized
data storage scheme, and then "surprise!" IO is totally random
(and it is likely to be somewhat random at the application level
too).

>> [ ... ] What is the application trying to do, at a high level?
>> Sometimes it's not possible to optimize a filesystem against a
>> badly designed application.  :-( [ ... ]

> The application is reading the gis data from a data source and
> plotting the map tiles (256x256, png images) for different zoom
> levels. The tree output of the first zoom level is as follows in
> each zoom level the fourth level directories are multiplied by a
> factor of four. Also the number of png images are multiplied by
> the same number. Some times a single image is taking around 20
> sec to create. [ ... ]

Once upon a time in the Land of the Silly Fools some folks wanted
to store many small records, and being silly fools they worried
about complicated nonsense like locality of access, index fanout,
compact representations, caching higher index tree levels, and
studied indexed files and DBMSes; and others who wanted to store
large images with various LODs studied ways to have compact,
progressive representations of those images.

As any really clever programmer and sysadm knows, all those silly
fools wasted a lot of time because it is very easy indeed instead
to just represent large small-record image collections as files
scattered in a many-level directory tree, and LODs as files of
varying size in subdirectories of that tree. :-)

[ ... ]

>> With a sufficiently bad access patterns, there may not be a lot
>> you can do, other than (a) throw hardware at the problem, or
>> (b) fix or redesign the application to be more intelligent (if
>> possible).

"if possible" here is a big understatement. :-)

> The file system is crated with "-i 1024 -b 1024" for larger
> inode number, 50% of the total images are less than 10KB.
> I have disabled access time and given a large value to the
> commit also.

These are likely to be irrelevant or counteproductive, and do not
address the two main issues, the acess pattern profile of the
application and how stunningly pessimized the current setup is.

> Do you have any other recommendation of the file system
> creation?

Get the application and the system redeveloped by some not-clever
programmers and sysadms who come from the Land of the Silly Fools
and thus have heard of indexed files and databases and LOD image
representations and know why silly fools use them. :-)

Since that recommendation is unlikely to happen (turkeys don't
vote for Christmas...), the main alternative is use some kind of
SLC SSD (e.g. recent Intel 160GB one) so as to minimize the impact
of a breathtakingly pessimized design thanks to a storage device
that can do very many more IOP/s than a hard disk. On a flash SSD
I would suggest using 'ext2' (or NILFS2, and I wish that UDF were
in a better state):

  http://www.storagesearch.com/ssd.html
  http://club.cdfreaks.com/f138/ssd-faq-297856/
  http://www.anandtech.com/cpuchipsets/showdoc.aspx?i=3607&p=4

Actually I would use an SSD even with an indexed file or a DBMS
and a LOD friendly image representation, because while that will
avoid the pessimized 15-layer index tree of directories, even in
the best of cases the app looks like having extremly low locality
of reference for data, and odds are that there will be 1-3 IOPs
per image cess (while probably currently there are many more).

The minor alternative is to use a file system like ReiserFS that
uses index trees internally and handles particularly well file
"tails", and also spread the excitingly pessimized IOP load across
a RAID5 (this application seems one of the only 2 cases where a
RAID5 makes sense), not a single disk. A nice set of low access
time 2.5" SAS drives might be the best choice. But considering the
cost of a flash 160GB SSD today, I'd go for a flash SSD drive (or
a small RAID of those) and a suitable fs.