IO lockups and ext3 readonly filecorruption on RHEL4 (pre and postU4)

[Wolber, Richard C]

We're using the same systems with the same OS (well okay, actually
CentOS 4) and aren't seeing the same thing. 

2.6.14.3 #1 SMP PREEMPT Thu Dec 8 10:34:08 PST 2005 i686 i686 i386
GNU/Linux

..Chuck..
 

> -----Original Message-----
> From: tweeks [mailto:tweeks at rackspace.com] 
> Sent: Tuesday, September 05, 2006 1:54 PM
> To: ext3-users at redhat.com
> Subject: IO lockups and ext3 readonly filecorruption on RHEL4 
> (pre and postU4)
> 
> Has anyone been seeing IO lockup problems on EL4?  
> 
> I've tried multiple IO scheduler options (elevator=) in the 
> boot... I'm seeing the same behavior regardless.  Independent 
> of hardware.  Whitebox ATA, HA enclosure with dedicated SCSI, 
> megaraid RAID hardware, Dell 2850s... same
> behavior:
> 
> A semi-busy system will suddenly go into some kind of IO 
> la-la land where nothing can be written to disk for >1hour.  
> Of course when this happens, the
> ext3 kernel module freaks out and remounts all the 
> filesystems as readonly.  
> Then when the system is rebooted, if the system is allowed to 
> fsck, the journal is hosed and the filesystem eats itself.  
> Moving them off the RH kernel all together seems to fix the 
> problem, but I have not found a way to reproduce the problem 
> yet (burning and stress testing doesn't seem to make it 
> appear), so real re-testing is difficult at best.
> 
> It's become so big of a problem that we're moving some 
> customers that require rock solid systems either over to 
> RHEL3, or off RH and over to SLES or other distro with a 
> non-RH kernel.  
> 
> Just the ext3 problem (minus the IO lockup part) can be seen 
> in other BZ
> tickets:
> 	https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=175877
> (when the filesystem fills up)
> 
> Has anyone seen these type of IO lockups + ext3 corruption on RHEL4?  
> Can you reproduce it?
> 
> Tweeks
> 
> _______________________________________________
> Ext3-users mailing list
> Ext3-users at redhat.com
> https://www.redhat.com/mailman/listinfo/ext3-users
>