extremely slow "ls" on a cleared fatty ext3 directory on FC4/5

[Brian Chadwick]

Robinson Tiemuqinke wrote:
> Hi,
>
>  A stupid flat directory /tmp holding 5 millon files,
> the directory locates on a ext3 file system with
> dir_index feature turned on. The running Linux are FC4
> and FC5.
>  
>  The files are just directly under /tmp, not in any
> subdirectories -- they are results of mis-operations
> of users.
>
>  Then a 'ls' or 'find' command will take one hour to
> finish, a lot of other applications on the computer
> boxes are affected. 
>
> I managed to have deleted the files one by one with a
> 'find . |xargs rm -rf' similar command in about 10
> hours. but after a file system sync, it still take me
> 20 minutes to list the cleaned /tmp directory again --
> even now the directory holds only 8 files total.
>
> so I try to 'ls' the directory itself (not any files
> and subdirectories on it) and find that its size is
> stupidly large (it is 131M even after deletion)
> compared with 4K for normal directories.
>
> -bash-3.00# ls -alFdh /tmp*
> drwxrwxrwt  4 root  staff 4.0K Aug 12 23:17 new_tmp/
> drwxrwxrwt  4 root  staff 131M Aug 12 20:30 tmp/
>
> Anyone know why the former fatty directory still looks
> unchanged and takes hours to traverse even after
> 99.999999% files got removed? 
>
> If there are any ways to fix this kind of problem
> without rebooting machine? I'm afraid of the commands
> "rsync -avHn /tmp/ /new_tmp/; rm -rf /tmp/ && mv
> /new_tmp/ /tmp" because other applications are
> accessing /tmp/ as well.
>
> Please help. Thanks a lot.
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
>
>   
i wonder ... i have noted that the journal file can get pretty big ... 
for instance on a devoted /tmp partition, do a "df" on the directory 
after deleting everything and note that there is still substantial use 
of space which is the journal... but i havent had zillions of files 
deleted before to be able to see wot happens  to the size of this 
journal... but i think its something to do with that big journal ..... 
you can test this by remounting the partition as ext2, not ext3 ... then 
the journal is ignored.. i am betting ls would be quick..  if your /tmp 
is a separate partition i would "umount" it, then reformat it with 
"mke2fs -j /dev/hdx", the remount it ....

thats my hard fix ....but i am no expert .... i look forward to more 
knowledgable critters to shed light.

no responsibility accepted :) ... but i am sure my HARD fix will work 
... but maybe not shed light