[Linux-cluster] strange slowness of ls with 1 newly created file on gfs 1 or 2

[David Teigland]

On Wed, Jul 11, 2007 at 01:46:57AM +0200, Pavel Stano wrote:
> Hello,
> 
> i am testing gfs and its very slow, please look at this if it is normal
> or i miss something
> 
> i have 2 node cluster, nodes are connected via SAS to disk array promise
> e310s, when i run dd on attached block device i have cca 150MBps
> throughput on both nodes
> there is debian etch, i compile cluster-2.00.00 with gfs1 module
> i create one 475GB logical volume (i dont use clvmd, just normal lvm),
> create gfs1 on it
> gfs_mkfs -t cluster1:data0 -p lock_dlm -j 2 /dev/vgdata0/lvdata0
> mount that lv on both nodes to directory /d/0/, run df on both nodes
> and then run touch on node 1:
> serpico# touch /d/0/test
> 
> and ls on node 2:
> dinorscio:~# time ls /d/0/
> test
> 
> real    0m9.486s
> user    0m0.000s
> sys     0m0.004s
> 
> it took almost 10 seconds to display 1 file on that filesystem
> when i again create other file via touch(node1) and run ls (node2) it
> took again cca 10 seconds
> i monitor activity with dstat and there is 50% iowait on node where run
> ls (50% because 2 core cpu on node), but no disk activity
> nodes are connected via 1gbps idle ethernet
> 
> and when ls is runing, i look at wchan with ps
> ps axf -o pid,wchan:20,cmd|grep ls
> 6387 sync_buffer                               \_ ls --color=auto /d/0/
> i run ps many times there is still sync_buffer, i dont see other kernel
> function

Have you found the problem here?  10 seconds definately means something
has gone very wrong somewhere.  I don't think it's in the dlm or gfs,
though.  My guess is that the problem is in the i/o layer (sync_buffer is
waiting for a write to complete).

Could you partition your disk, create an ext3 fs on each partition,
have each node mount one of the ext3's, and run some tests?

Dave