Ext3 filesystem aborting journal at random times (Maxtor 300GB disk)

Thu Jul 22 22:35:23 UTC 2004

Hi,

Sorry if this is an old and fixed issue but I can't really seem to find 
a definitive explanation/fix to this in the mail lists.  Ever since I 
upgraded to FC2 my server randomly decides that it can no longer access 
the disk and "aborts" the ext3 journal leaving it read-only as 
expected.  I've seen the same errors on other mail list, explained away 
as disk or disk cable issues, but in my case all the disks are all 
brand new 300GB Maxtors.

Does anyone have any ideas?  Is it because the current kernel can't 
handle a 288GB partition?  BTW the server is completely up to date as 
per the up2date command including the kernel (2.6.6-1.435.2.3).

Start of fsck:

=================================================
[root at server /]# fsck -y -t ext3 /dev/hdc1
fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
/dev/hdc1: recovering journal
/dev/hdc1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error reading block 55214498 (Attempt to read block from filesystem 
resulted in short read) while doing inode scan.  Ignore error? yes

Force rewrite? yes

Error reading block 55214499 (Attempt to read block from filesystem 
resulted in short read) while doing inode scan.  Ignore error? yes

Force rewrite? yes

Error reading block 55214500 (Attempt to read block from filesystem 
resulted in short read) while doing inode scan.  Ignore error? yes

Force rewrite? yes

Error reading block 61177858 (Attempt to read block from filesystem 
resulted in short read) while doing inode scan.  Ignore error? yes

Force rewrite? yes

<<SNIP>>

Pass 2: Checking directory structure
Entry 'ghostscript' in /OLD/usr/share (32112642) has deleted/unused 
inode 32915539.  Clear? yes

Entry 'projects' in /OLD/usr/share/gettext (32295519) has 
deleted/unused inode 32915521.  Clear? yes

Entry 'disk' in /OLD/usr/include (32608468) has deleted/unused inode 
33473295.  Clear? yes

Entry 'cyrus' in /OLD/usr/include (32608468) has deleted/unused inode 
33473299.  Clear? yes

Entry 'parted' in /OLD/usr/include (32608468) has deleted/unused inode 
33492290.  Clear? yes

Entry 'php' in /OLD/usr/include (32608468) has deleted/unused inode 
33492303.  Clear? yes

Entry '..' in ??? (32915564) has deleted/unused inode 32915540.  Clear? 
yes

Entry '..' in ??? (32915786) has deleted/unused inode 32915540.  Clear? 
yes

Entry '..' in ??? (32915801) has deleted/unused inode 32915539.  Clear? 
yes

<<SNIP>>

Pass 3: Checking directory connectivity
Unconnected directory inode 32915564 (...)
Connect to /lost+found? yes

/lost+found not found.  Create? yes

Error reading block 69730304 (Attempt to read block from filesystem 
resulted in short read) while reading inode and block bitmaps.  Ignore 
error? yes

Force rewrite? yes

Unconnected directory inode 32915786 (...)
Connect to /lost+found? yes

Unconnected directory inode 32915801 (...)
Connect to /lost+found? yes

Unconnected directory inode 33492357 (...)
Connect to /lost+found? yes

Unconnected directory inode 33492461 (...)
Connect to /lost+found? yes

Unconnected directory inode 33492497 (...)
Connect to /lost+found? yes

Unconnected directory inode 33769637 (Error reading block 67534919 
(Attempt to read block from filesystem resulted in short read).  Ignore 
error? yes

Force rewrite? yes

???)
Connect to /lost+found? yes

Couldn't fix parent of inode 33769637: Ext2 inode is not a directory

Unconnected directory inode 33769657 (???)
Connect to /lost+found? yes

Couldn't fix parent of inode 33769657: Ext2 inode is not a directory

<<SNIP>>

Pass 4: Checking reference counts
Inode 32112642 ref count is 316, should be 315.  Fix? yes

Inode 32295519 ref count is 5, should be 4.  Fix? yes

Inode 32608468 ref count is 250, should be 246.  Fix? yes

Unattached inode 32915553
Connect to /lost+found? yes

WARNING: PROGRAMMING BUG IN E2FSCK!
         OR SOME BONEHEAD (YOU) IS CHECKING A MOUNTED (LIVE) FILESYSTEM.
inode_link_info[33769637] is 3, inode.i_links_count is 1.  They should 
be the same!
Inode 33769637 ref count is 1, should be 2.  Fix? yes

WARNING: PROGRAMMING BUG IN E2FSCK!
         OR SOME BONEHEAD (YOU) IS CHECKING A MOUNTED (LIVE) FILESYSTEM.
inode_link_info[33769657] is 3, inode.i_links_count is 1.  They should 
be the same!
Inode 33769657 ref count is 1, should be 2.  Fix? yes

<<SNIP>>

Pass 5: Checking group summary information
Block bitmap differences:  -(65853587--65853668) -(66711497--66711572) 
-(66711644--66711989) -(66968848--66968889) -(67007184--67007226) 
-(67565768--67566720) +(69730304--69730817)
Fix? yes

Free blocks count wrong for group #2009 (22014, counted=22096).
Fix? yes

Free blocks count wrong for group #2035 (21906, counted=22328).
Fix? yes

Free blocks count wrong for group #2043 (22014, counted=22056).
Fix? yes

Free blocks count wrong for group #2044 (21880, counted=21923).
Fix? yes

Free blocks count wrong for group #2061 (22014, counted=22967).
Fix? yes

Free blocks count wrong (64428562, counted=64430104).
Fix? yes

Inode bitmap differences:  -(32915521--32915552) -(33342497--33342560) 
-(33342625--33342944) -(33473281--33473312) -(33492289--33492320) 
-(33769505--33769632)
Fix? yes
Free inodes count wrong for group #2009 (13080, counted=13112).
Fix? yes

/dev/hdc1: ***** FILE SYSTEM WAS MODIFIED *****

/dev/hdc1: ********** WARNING: Filesystem still has errors **********

/dev/hdc1: 939701/36634624 files (0.3% non-contiguous), 
8812231/73242335 blocks
[root at server /]#

=================================================

Errors in messages:

=================================================
Jul 20 04:09:13 server kernel: hdc: dma_intr: status=0x51 { DriveReady 
SeekComplete Error }
Jul 20 04:09:13 server kernel: hdc: dma_intr: error=0x40 { 
UncorrectableError }, LBAsect=526647391, high=31, low=6553695, 
sector=526647391
Jul 20 04:09:13 server kernel: end_request: I/O error, dev hdc, sector 
526647391
Jul 20 04:09:13 server kernel: EXT3-fs error (device hdc1): 
ext3_get_inode_loc: unable to read inode block - inode=32915539, 
block=65830916
Jul 20 04:09:13 server kernel: Aborting journal on device hdc1.
Jul 20 04:09:13 server kernel: ext3_abort called.
Jul 20 04:09:13 server kernel: EXT3-fs abort (device hdc1): 
ext3_journal_start: Detected aborted journal
Jul 20 04:09:13 server kernel: Remounting filesystem read-only
Jul 20 04:10:13 server kernel: hdc: dma_intr: status=0x51 { DriveReady 
SeekComplete Error }Jul 20 04:10:13 server kernel: hdc: dma_intr: 
error=0x40 { UncorrectableError }, LBAsect=526647391, high=31, 
low=6553695, sector=526647391Jul 20 04:10:13 server kernel: 
end_request: I/O error, dev hdc, sector 526647391
Jul 20 04:10:13 server kernel: EXT3-fs error (device hdc1): 
ext3_get_inode_loc: unable to read inode block - inode=32915521, 
block=65830916
Jul 20 04:10:29 server kernel: hdc: dma_intr: status=0x51 { DriveReady 
SeekComplete Error }
Jul 20 04:10:29 server kernel: hdc: dma_intr: error=0x40 { 
UncorrectableError }, LBAsect=531366999, high=31, low=11273303, 
sector=531366999
Jul 20 04:10:29 server kernel: end_request: I/O error, dev hdc, sector 
531366999Jul 20 04:10:29 server kernel: EXT3-fs error (device hdc1): 
ext3_get_inode_loc: unable to read inode block - inode=33214528, 
block=66420867
Jul 20 04:10:31 server kernel: hdc: dma_intr: status=0x51 { DriveReady 
SeekComplete Error }
Jul 20 04:10:31 server kernel: hdc: dma_intr: error=0x40 { 
UncorrectableError }, LBAsect=531366999, high=31, low=11273303, 
sector=531366999Jul 20 04:10:31 server kernel: end_request: I/O error, 
dev hdc, sector 531366999
Jul 20 04:10:31 server kernel: EXT3-fs error (device hdc1): 
ext3_get_inode_loc: unable to read inode block - inode=33214497, 
block=66420867
Jul 20 04:10:39 server kernel: hdc: dma_intr: status=0x51 { DriveReady 
SeekComplete Error }Jul 20 04:10:39 server kernel: hdc: dma_intr: 
error=0x40 { UncorrectableError }, LBAsect=533463199, high=31, 
low=13369503, sector=533463199
Jul 20 04:10:39 server kernel: end_request: I/O error, dev hdc, sector 
533463199Jul 20 04:10:39 server kernel: EXT3-fs error (device hdc1): 
ext3_get_inode_loc: unable to read inode block - inode=33341770, 
block=66682892
Jul 20 04:10:44 server kernel: hdc: dma_intr: status=0x51 { DriveReady 
SeekComplete Error }Jul 20 04:10:44 server kernel: hdc: dma_intr: 
error=0x40 { UncorrectableError }, LBAsect=533463441, high=31, 
low=13369745, sector=533463439Jul 20 04:10:44 server kernel: 
end_request: I/O error, dev hdc, sector 533463439
Jul 20 04:10:44 server kernel: EXT3-fs error (device hdc1): 
ext3_get_inode_loc: unable to read inode block - inode=33342730, 
block=66682922
=================================================

Thanks in advance,
Harvi