<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META http-equiv=Content-Type content="text/html; charset=us-ascii"> <META content="MSHTML 6.00.2900.2627" name=GENERATOR></HEAD> <BODY> <DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2>Hardware Configuration: <SPAN class=033581305-18052005>Six node cluster, each node has a LSI Fibre Channel Host Adapter interface to a SAN.</SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2>Software Configuration: <SPAN class=033581305-18052005>The kernel is 2.4.21-20.EL with GFS-6.0.2-25</SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2>Problem: <SPAN class=033581305-18052005>While four nodes are simultaneously accessing the SAN, if a 5th node attempts to access the SAN, one of the nodes will kernel panic.</SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005> The node that crashes seems to be random. All the crashes have the same error as follows:</SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2></FONT></SPAN></FONT></FONT></SPAN> </DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: mptscsih: ioc0: WARNING - Device (0:0:1) reported QUEUE_FULL! </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: SCSI disk error : host 0 channel 0 id 0 lun 1 return code = 440b0000 </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: I/O error: dev 08:12, sector 139961968 </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: Pool: IO request to device, (8,18) blk #139961968, failed. </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: GFS: fsid=p2-2:gfs1.3: read error on block 17495244 </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: Panicking because of read error on block 17495244 </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: f3d33b98 f8a2f2a2 00000032 00000031 c01217d2 0000000a 00000400 f8a4f7f5 </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: f3d33be8 f3740370 010af4cc f3740370 00000020 00000000 f8a5c000 00000031 </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: f8a1419e f8a4d692 f8a4d57a 0000024f 00000013 f8a5c000 f8a5c000 f3d33c3c </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: Call Trace: [<f8a2f2a2>] gfs_asserti [gfs] 0x32 (0xf3d33b9c) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<c01217d2>] printk [kernel] 0x122 (0xf3d33ba8) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a4f7f5>] .rodata.str1.4 [gfs] 0x249 (0xf3d33bb4) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a1419e>] gfs_dreread [gfs] 0x12e (0xf3d33bd8) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a4d692>] .rodata.str1.1 [gfs] 0x1e6 (0xf3d33bdc) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a4d57a>] .rodata.str1.1 [gfs] 0xce (0xf3d33be0) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a13ff9>] gfs_dread [gfs] 0x49 (0xf3d33bfc) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a1513f>] gfs_get_meta_buffer [gfs] 0x9f (0xf3d33c18) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a22fc2>] get_metablock [gfs] 0xb2 (0xf3d33c50) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a233db>] gfs_block_map [gfs] 0x2eb (0xf3d33c70) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<c016ad48>] init_buffer_head [kernel] 0x38 (0xf3d33cb8) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a1cdae>] get_block [gfs] 0x9e (0xf3d33d28) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<c01567db>] __block_prepare_write [kernel] 0x19b (0xf3d33d64) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<c014a0d0>] __alloc_pages_limit [kernel] 0x60 (0xf3d33d94) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<c0157139>] block_prepare_write [kernel] 0x39 (0xf3d33da8) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a1cd10>] get_block [gfs] 0x0 (0xf3d33dbc) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a1d41c>] gfs_prepare_write [gfs] 0x11c (0xf3d33dc8) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a1cd10>] get_block [gfs] 0x0 (0xf3d33dd8) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<c013f5d5>] do_generic_file_write [kernel] 0x1d5 (0xf3d33df0) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a1790b>] do_do_write [gfs] 0x2ab (0xf3d33e44) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a17d4b>] do_write [gfs] 0x18b (0xf3d33e90) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a15c89>] gfs_walk_vma [gfs] 0x129 (0xf3d33ecc) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a1f112>] gfs_sync_page [gfs] 0x52 (0xf3d33eec) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a319b7>] gfs_glock_nq_init [gfs] 0x37 (0xf3d33f30) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a319f3>] gfs_glock_dq_uninit [gfs] 0x13 (0xf3d33f40) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a187e1>] gfs_sync_file [gfs] 0x61 (0xf3d33f4c) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a17e20>] gfs_write [gfs] 0x90 (0xf3d33f6c) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<f8a17bc0>] do_write [gfs] 0x0 (0xf3d33f80) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: [<c0153a53>] sys_write [kernel] 0xa3 (0xf3d33f94) </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: Kernel panic: GFS: Assertion failed on line 591 of file linux_dio.c </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: GFS: assertion: "FALSE" </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: GFS: time = 1116388432 </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: GFS: fsid=p2-2:gfs1.3 </FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV><SPAN class=454292304-12052005><FONT face=Arial><FONT size=2><SPAN class=033581305-18052005><FONT size=2>May 17 21:53:52 compute-0-2.local kernel: </DIV></FONT></SPAN></FONT></FONT></SPAN></DIV> <DIV align=left><FONT face=Arial size=2></FONT> </DIV> <DIV align=left><FONT face=Arial size=2>Frank L. Setinsek</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV></BODY></HTML>