[Linux-cluster] GFS2 cluster node is running very slow

[Steven Whitehouse]

Hi,

On Wed, 2011-03-30 at 01:34 -0400, David Hill wrote:
> Hi guys,
> 
>  
> 
> I’ve found this in /sys/kernel/debug/gfs2/fsname/glocks
> 
>  
> 
> H: s:EX f:tW e:0 p:22591 [jsvc] gfs2_inplace_reserve_i+0x451/0x69a
> [gfs2]
> 
> H: s:EX f:tW e:0 p:22591 [jsvc] gfs2_inplace_reserve_i+0x451/0x69a
> [gfs2]
> 
> H: s:EX f:W e:0 p:806 [pdflush] gfs2_write_inode+0x57/0x152 [gfs2]
> 
This doesn't mean anything without a bit more context. Were these all
queued against the same glock? If so which glock was it?

>  
> 
> The application running is confluence and has 184 thread.   The other
> nodes work fine but that specific node is having issues obtaining
> locks when it’s time to write?
> 
That does sound a bit strange. Are you using a different network card on
the slow node? Have you checked to see if there is too much traffic on
that network link?

Also, how full was the filesystem and which version of GFS2 are you
using (i.e. RHELx, Fedora X or CentOS or....)?


Steve.

>  
> 
> Dave
> 
>  
> 
>  
> 
>  
> 
>  
> 
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of David Hill
> Sent: 29 mars 2011 21:00
> To: linux-cluster at redhat.com
> Subject: [Linux-cluster] GFS2 cluster node is running very slow
> 
> 
>  
> 
> Hi guys,
> 
>  
> 
>                 We have a GFS2 cluster consisting of 3 nodes.  At this
> point, everything is going smooth.  Now, we add a new node with more
> CPUs with the
> 
> exact same configuration but all transactions on the mount run very
> slow.
> 
>  
> 
> Copying a file to the mount is done at about 25kb/s when on the three
> other nodes, everything goes smooth at about 7MB/s.
> 
> CPU on all nodes is idling at some point, all cluster process are kind
> of sleeping. 
> 
>  
> 
> We’ve tried the ping_pong.c from apache and it seems to be able to
> write/read lock files at a decent rate.
> 
>  
> 
> There’s other mounts on the system using the same fc
> card/fibers/switches/san and all these are also working at a decent
> speed...
> 
>  
> 
> I’ve been reading a good part of the day, and I can’t seem to find a
> solution.
> 
>  
> 
>  
> 
>  
> 
>  
> 
> ubisoft_logo
> 
> David C. Hill
> 
> Linux System Administrator - Enterprise
> 
> 514-490-2000#5655
> 
> http://www.ubi.com
> 
>  
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster