ext3 filesystem performance issues

[Andrew Kelly]

On Thu, 2007-09-13 at 08:39 -0500, Bruno Wolff III wrote:
> On Wed, Sep 12, 2007 at 20:07:42 -0400,
>   Matthew Flaschen <matthew.flaschen at gatech.edu> wrote:
> > > 
> > > I have directories with several million files in them.
> > 
> > Just curious...what for?
> 
> Legacy, they need to be fixed.
> 
> I am tracking IP addresses that sent me spam and blocking them and I save
> a file for each IP address with the headers in it in case I want to check
> on things later. When I started doing that it took a long time to get
> hit bit 100K unique addresses. Now it takes a couple of months to get hit
> by over a million unique addresses. I don't check as often as I used to because
> even doing ls -f | wc takes several minutes to run. Not too mention that each
> file takes up 8K of disk space on ext3 file systems.
> I have been meaning to rewrite my tracking system to put everything in a
> single (or perhaps several) files to make dealing with the data less
> burdensome. I just haven't gotten round to it yet.


Bruno, 
this is screaming "database me".

Andy