[Linux-cluster]GFS Problem

Wed May 18 05:39:08 UTC 2005

Hardware Configuration:  Six node cluster, each node has a LSI Fibre Channel
Host Adapter interface to a SAN.
Software Configuration: The kernel is 2.4.21-20.EL with GFS-6.0.2-25
Problem: While four nodes are simultaneously accessing the SAN, if a 5th
node attempts to access the SAN, one of the nodes will kernel panic.
              The node that crashes seems to be random.  All the crashes
have the same error as follows:
 
May 17 21:53:52 compute-0-2.local kernel: mptscsih: ioc0: WARNING - Device
(0:0:1) reported QUEUE_FULL! 
May 17 21:53:52 compute-0-2.local kernel: SCSI disk error : host 0 channel 0
id 0 lun 1 return code = 440b0000 
May 17 21:53:52 compute-0-2.local kernel: I/O error: dev 08:12, sector
139961968 
May 17 21:53:52 compute-0-2.local kernel: Pool: IO request to device, (8,18)
blk #139961968, failed. 
May 17 21:53:52 compute-0-2.local kernel: GFS: fsid=p2-2:gfs1.3: read error
on block 17495244 
May 17 21:53:52 compute-0-2.local kernel: Panicking because of read error on
block 17495244 
May 17 21:53:52 compute-0-2.local kernel: f3d33b98 f8a2f2a2 00000032
00000031 c01217d2 0000000a 00000400 f8a4f7f5 
May 17 21:53:52 compute-0-2.local kernel: f3d33be8 f3740370 010af4cc
f3740370 00000020 00000000 f8a5c000 00000031 
May 17 21:53:52 compute-0-2.local kernel: f8a1419e f8a4d692 f8a4d57a
0000024f 00000013 f8a5c000 f8a5c000 f3d33c3c 
May 17 21:53:52 compute-0-2.local kernel: Call Trace: [<f8a2f2a2>]
gfs_asserti [gfs] 0x32 (0xf3d33b9c) 
May 17 21:53:52 compute-0-2.local kernel: [<c01217d2>] printk [kernel] 0x122
(0xf3d33ba8) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a4f7f5>] .rodata.str1.4 [gfs]
0x249 (0xf3d33bb4) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a1419e>] gfs_dreread [gfs]
0x12e (0xf3d33bd8) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a4d692>] .rodata.str1.1 [gfs]
0x1e6 (0xf3d33bdc) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a4d57a>] .rodata.str1.1 [gfs]
0xce (0xf3d33be0) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a13ff9>] gfs_dread [gfs] 0x49
(0xf3d33bfc) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a1513f>] gfs_get_meta_buffer
[gfs] 0x9f (0xf3d33c18) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a22fc2>] get_metablock [gfs]
0xb2 (0xf3d33c50) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a233db>] gfs_block_map [gfs]
0x2eb (0xf3d33c70) 
May 17 21:53:52 compute-0-2.local kernel: [<c016ad48>] init_buffer_head
[kernel] 0x38 (0xf3d33cb8) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a1cdae>] get_block [gfs] 0x9e
(0xf3d33d28) 
May 17 21:53:52 compute-0-2.local kernel: [<c01567db>] __block_prepare_write
[kernel] 0x19b (0xf3d33d64) 
May 17 21:53:52 compute-0-2.local kernel: [<c014a0d0>] __alloc_pages_limit
[kernel] 0x60 (0xf3d33d94) 
May 17 21:53:52 compute-0-2.local kernel: [<c0157139>] block_prepare_write
[kernel] 0x39 (0xf3d33da8) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a1cd10>] get_block [gfs] 0x0
(0xf3d33dbc) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a1d41c>] gfs_prepare_write
[gfs] 0x11c (0xf3d33dc8) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a1cd10>] get_block [gfs] 0x0
(0xf3d33dd8) 
May 17 21:53:52 compute-0-2.local kernel: [<c013f5d5>] do_generic_file_write
[kernel] 0x1d5 (0xf3d33df0) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a1790b>] do_do_write [gfs]
0x2ab (0xf3d33e44) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a17d4b>] do_write [gfs] 0x18b
(0xf3d33e90) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a15c89>] gfs_walk_vma [gfs]
0x129 (0xf3d33ecc) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a1f112>] gfs_sync_page [gfs]
0x52 (0xf3d33eec) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a319b7>] gfs_glock_nq_init
[gfs] 0x37 (0xf3d33f30) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a319f3>] gfs_glock_dq_uninit
[gfs] 0x13 (0xf3d33f40) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a187e1>] gfs_sync_file [gfs]
0x61 (0xf3d33f4c) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a17e20>] gfs_write [gfs] 0x90
(0xf3d33f6c) 
May 17 21:53:52 compute-0-2.local kernel: [<f8a17bc0>] do_write [gfs] 0x0
(0xf3d33f80) 
May 17 21:53:52 compute-0-2.local kernel: [<c0153a53>] sys_write [kernel]
0xa3 (0xf3d33f94) 
May 17 21:53:52 compute-0-2.local kernel: 
May 17 21:53:52 compute-0-2.local kernel: Kernel panic: GFS: Assertion
failed on line 591 of file linux_dio.c 
May 17 21:53:52 compute-0-2.local kernel: GFS: assertion: "FALSE" 
May 17 21:53:52 compute-0-2.local kernel: GFS: time = 1116388432 
May 17 21:53:52 compute-0-2.local kernel: GFS: fsid=p2-2:gfs1.3 
May 17 21:53:52 compute-0-2.local kernel: 
 
Frank L. Setinsek
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20050517/e7c20fa8/attachment.htm>