Huge amount of used inode handlers

[Janne Pikkarainen <jaba mikrobitti fi>]

Hi all,

Any ideas, why is this happening or how to debug this? After random time 
(usually one to three days uptime) inode-sz reported by sar jumps way up its 
normal level and <24 hours after this bug has happened server will crash. 

Server used to work fine, but started to do this around beginning of June.

---
08:20:00 AM dentunusd   file-sz  %file-sz  inode-sz  super-sz %super-sz  
dquot-sz %dquot-sz  rtsig-sz %rtsig-sz
08:30:00 AM     74716       335      0.32     28189         0      0.00         
0      0.00         0      0.00
08:40:00 AM     76795       380      0.36     28383         0      0.00         
0      0.00         0      0.00
08:50:00 AM     55700       370      0.35     10806         0      0.00         
0      0.00         0      0.00
09:00:01 AM     18770       291      0.28 4294948543         0      0.00         
0      0.00         0      0.00
09:10:00 AM     16385       291      0.28 4294947285         0      0.00         
0      0.00         0      0.00
09:20:00 AM       982       275      0.26 4294939963         0      0.00         
0      0.00         0      0.00
---

Disk activity stops in a very weird way: 

I can ...

- create new files with commands like "echo test >test" and read them.
- move around in directory structure. 
- use "df" and such commands.

But ...

- all kind of syslog activity freezes, no new stuff to /var/log/, although all 
the processes seem to run as usual.
- Apache seems to serve pages it already has been serving, but trying to surf 
to the yet unvisited page, it will hang.
- restarting any service via "/etc/rc.d/init.d/x restart" fails (will hang 
when service is stopping).

Server is a dual P3 Xeon 700 MHz / 1 GB RAM / 200 GB (hardware) RAID-5 SCSI.

Distribution used is a fresh Red Hat 7.3 install, updated with up2date 
whenever necessary. 

Kernel version:

Linux xxx.xxxxxxxxx.xx 2.4.18-4smp #1 SMP Thu May 2 18:32:34 EDT 2002 i686 
unknown

Best regards,

Janne Pikkarainen