[Linux-cluster] GFS + CORAID Performance Problem

[Wendy Cheng]

bigendian+gfs at gmail.com wrote:

> I've just set up a new two-node GFS cluster on a CORAID sr1520 
> ATA-over-Ethernet.  My nodes are each quad dual-core Opteron CPU 
> systems with 32GB RAM each.  The CORAID unit exports a 1.6TB block 
> device that I have a GFS file system on.
>
> I seem to be having performance issues where certain read system calls 
> take up to three seconds to complete.  My test app is bonnie++, and 
> the slow-downs appear to be happen in the "Rewriting" portion of the 
> test, though I'm not sure if this is exclusive.  If I watch top and 
> iostat for the device in question, I see activity on the device, then 
> long (up to three second) periods of no apparent I/O.  During the 
> periods of no I/O the bonnie++ process is blocked on disk I/O, so it 
> seems that the system it trying to do something.  Network traces seem 
> to show that the host machine is not waiting on the RAID array, and 
> the packet following the dead-period seems to always be sent from the 
> host to the coraid device.  Unfortunately, I don't know how to dig in 
> any deeper to figure out what the problem is.
>
I think we know about this issue. Note that bonnie++ doesn't keep the 
file size within the benchmark's local memory, it always invokes a 
"stat" system call to poll for the file size before it can do read and 
rewrite. GFS1 has a known performance issue with stat system call (that 
we hope it can be addressed by GFS2) and since file size in bonnie++ 
tend to be small, the stat() call overhead becomes very obvious. It will 
be worse in your case due to the filesystem size.

The issue here is whether bonnie++ faithfully represents your 
application ? More specifically, does your application need to do a 
"stat" system call before each and every IO ? On the other hand, there 
are few tunables that could help out (e.g., increasing statfs_slots via 
"gfs_tool settune" command). Will work with support group to document 
the detailed tuning steps into Red Hat Knowledgebase 
(http://kbase.redhat.com/faq/). When they're ready, we'll post here.

-- Wendy