[Linux-cluster] rhcs + gfs performance issues

[Wendy Cheng]

Hopefully the following provide some relieves ...
 
1. Enable lock trimming tunable. It is particularly relevant if NFS-GFS 
is used by development type of workloads (editing, compiling, build, 
etc) and/or after filesystem backup. Unlike fast statfs, this tunable is 
per-node base (you don't need to have the same value on each of the 
nodes and a mix of on-off within the same cluster is ok). Make the 
trimming very aggressive on backup node (> 50% where you run backup) and 
moderate on your active node (< 50%). Try to experiment with different 
values to fit the workload. Googling "gfs lock trimming wcheng" to pick 
up the technical background if needed.

shell> gfs_tool settune <mount_point> glock_purge <percentage>
         (e.g. gfs_tool settune /mnt/gfs1 glock_purge 50)

2. Turn on readahead tunable. It is effective for large file (stream IO) 
read performance. As I recalled, there was a cluster (with IPTV 
application) used val=256 for 400-500M files. Another one with 2G file 
size used val=2048. Again, it is per-node base so different values are 
ok on different nodes.

shell> gfs_tool settune <mount> seq_readahead <val>
         (e.g. gfs_tool settune /mnt/gfs1 seq_readahead 2048)

3. Fast statfs tunable - you have this on already ? Make sure they need 
to be same across cluster nodes.

4. Understand the risks and performance implications of NFS server's 
"async" vs. "sync" options. Linux NFS server "sync" options are 
controlled by two different mechanisms - mount and export. By default, 
mount is "aysnc" and export is "sync". Even with specific "async" mount 
request, Linux server uses "sync" export as default that is particularly 
troublesome for gfs. I don't plan to give an example and/or suggest the 
exact export option here - hopefully this will force folks to do more 
researches to fully understand the ramifications between performance and 
data liability. Most of the proprietary NFS servers in the market today 
utilize hardware features to relieve this performance and data integrity 
conflicts. Mainline linux servers (and RHEL) are totally software-base 
so it generally has problem in this regard.

Gfs1 in general doesn't do well in "sync" performance (journal layer is 
too bulky). Gfs2 has potentials to do better (but I'm not sure).

There are also few other things that worth mentioned but my flight is 
call for boarding .. I'll stop here .......

-- Wendy