[Linux-cluster] Problem with gfs2 - waiting for i/o

[Jean-Marc jmc]

Hi list,
I am new to cluster and gfs so my apologies if I missed something obvious.
I am having problems using gfs2 filesystems where writing processes hang
forever waiting for i/o. It does not seem to be locking (or even cluster)
related since its reproducible on a local gfs2 filesystem with 'lock_nolock'
as locking protocol.

I can create and mount the gfs2 file system without any problems. Creating
and writing to files works as well, but when I write larger amounts of data
the process gets stuck waiting for i/o. It doesn't matter whether I amd
using lvm or not thre results are the same. The hardware is working and I
have no problem writing to an ext3 filesystem on the same device.

The problem can be reproduced by doing the following:

-------------------------------------------------------------------------------------------------
[root at indigo ~]# mkfs.gfs2 -p lock_nolock /dev/cciss/c0d0p3
This will destroy any data on /dev/cciss/c0d0p3.
  It appears to contain a gfs2 filesystem.

Are you sure you want to proceed? [y/n] y

Device:                    /dev/cciss/c0d0p3
Blocksize:                 4096
Device Size                644.44 GB (168935523 blocks)
Filesystem Size:           644.44 GB (168935523 blocks)
Journals:                  1
Resource Groups:           2578
Locking Protocol:          "lock_nolock"
Lock Table:                ""

[root at indigo ~]# mount -o locktable=xyz /dev/cciss/c0d0p3 /export

[root at indigo ~]# mount|grep /export
/dev/cciss/c0d0p3 on /export type gfs2
(rw,locktable=xyz,localflocks,localcaching)

[root at indigo ~]# dd if=/dev/zero of=/export/8GB bs=1024k count=8000

## A partial file is created, but the process never returns, it is forever
waiting for i/o:
[root at indigo ~]# ls -l /export/;ps auxww|grep 8GB
total 1995152
-rw-r--r-- 1 root root 2039009280 Jun  1 07:12 8GB
root      3857 42.8  0.0   4880  1584 pts/0    D    07:12   0:07 dd if
/dev/zero of /export/8GB bs 1024k count 8000
----------------------------------------------------------------------------------------------------

It's still possible to create new files to the file system but these
processes are also waiting for i/o after a while.
After the mounting there is nothing else appearing in dmesg or messages log.
Since the processes are uninterruptible we are forced to reboot the machine
in order to resolve the lockup.

Basic info on the system setup:

HP dl380 G5 2 xeon cpu with dual cores each, 12 GB RAM
local raid controller is p400 but we also reproduced the behaviour on the
san disk (Emulex card)
OS is RH 5 (32 bit), output of uname is:
    Linux indigo 2.6.18-8.1.4.el5xen #1 SMP Fri May 4 22:42:50 EDT 2007 i686
i686 i386 GNU/Linux

The same problem could also be reproduced on RH 4 and Ubuntu 7.04

Thanks in advance for your help!

/jmc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20070606/d9fbbc81/attachment.htm>