<br><font size=2 face="sans-serif">Hi,</font>
<br>
<br><font size=2 face="sans-serif">Can anyone provide techniques or suggestions
to improve GFS performance?</font>
<br>
<br><font size=2 face="sans-serif">The problem we have is best summarized
by the output of a perl script that one of our frustrated developers has
written:</font>
<br>
<br><font size=2 face="sans-serif">The script simply creates a file, then
reads it, then removes it and prints out the time it takes for each of
these instructions to complete. Sometimes it takes only a second
to do all three, while sometimes it takes as long as 15 seconds to do these
3 simple instructions. I've run the script on all of the five GFS
file systems and see the same response characteristics. I've run
the script when the systems and when they are fairly quiet and still see
the same issue.</font>
<br>
<br>
<br><font size=2 face="sans-serif">To summarize the environment:</font>
<br>
<br><font size=2 face="sans-serif">RH ES3 update 6 (GFS 6.0) 11 node cluster
environment with 3 redundant lock managers and five GFS file systems mounted
on each participating servers. The GFS file systems range from 100GB
to 1.5 TB.</font>
<br>
<br><font size=2 face="sans-serif">The storage array is an EMC CX700 attached
to a dual redundant SAN consisting four 2GB Brocade 3900 SAN switches.
</font>
<br>
<br><font size=2 face="sans-serif">The HBA's in all servers are Qlogic
qla2340's Firmware version: 3.03.14, Driver version 7.07.00</font>
<br>
<br><font size=2 face="sans-serif">The servers are all HP DL585 64bit AMD
opteron server class machines each configured with between 8GB and 32GB
of memory.</font>
<br>
<br>
<br><font size=2 face="sans-serif">.....................................................................................................................................................................................................................................................</font>
<br>
<br><font size=2 face="sans-serif">I've raised a support call with RedHat
but according to their experts our configuration seems already to be set
for optimum performance.</font>
<br>
<br><font size=2 face="sans-serif">RedHat provide a utility to get and
set tunable gfs file systems parameters but there is next to no supporting
documentation. </font>
<br>
<br><font size=2 face="sans-serif">So, is there anything I can do or am
I missing something obvious that is just plainly mis-configured? </font>
<br>
<br><font size=2 face="sans-serif">Shown below is the GFS configuration
summary derived from lock_gulmd -C and gfs_tool df for each file system.</font>
<br>
<br><font size=2 face="sans-serif">I'll be happy to supply any other information
if it will help.</font>
<br>
<br><font size=2 face="sans-serif">Thanks to all in advance</font>
<br>
<br><font size=2 face="sans-serif">Paul McDowell</font>
<br>
<br>
<br>
<br><font size=2 face="sans-serif">lock_gulmd -C</font>
<br>
<br><font size=2 face="sans-serif"># hashed: 0x44164246</font>
<br><font size=2 face="sans-serif">cluster {</font>
<br><font size=2 face="sans-serif"> name = "cra_gfs"</font>
<br><font size=2 face="sans-serif"> lock_gulm {</font>
<br><font size=2 face="sans-serif"> heartbeat_rate = 15.000</font>
<br><font size=2 face="sans-serif"> allowed_misses = 3</font>
<br><font size=2 face="sans-serif"> coreport = 40040</font>
<br><font size=2 face="sans-serif"> new_connection_timeout = 15.000</font>
<br><font size=2 face="sans-serif"> # server cnt: 3</font>
<br><font size=2 face="sans-serif"> # servers = ["iclc1g.cra.applera.net",
"iclc2g.cra.applera.net", "ccf001g.cra.applera.net"]</font>
<br><font size=2 face="sans-serif"> servers = ["172.20.8.21",
"172.20.8.22", "172.20.8.51"]</font>
<br><font size=2 face="sans-serif"> lt_partitions = 4</font>
<br><font size=2 face="sans-serif"> lt_base_port = 41040</font>
<br><font size=2 face="sans-serif"> lt_high_locks = 20971520</font>
<br><font size=2 face="sans-serif"> lt_drop_req_rate = 300</font>
<br><font size=2 face="sans-serif"> prealloc_locks = 5000000</font>
<br><font size=2 face="sans-serif"> prealloc_holders = 11000000</font>
<br><font size=2 face="sans-serif"> prealloc_lkrqs = 60</font>
<br><font size=2 face="sans-serif"> ltpx_port = 40042</font>
<br><font size=2 face="sans-serif"> </font>
<br>
<br><font size=2 face="sans-serif">#gfs_tool df</font>
<br>
<br>
<br><font size=2 face="sans-serif">/crx:</font>
<br><font size=2 face="sans-serif"> SB lock proto = "lock_gulm"</font>
<br><font size=2 face="sans-serif"> SB lock table = "cra_gfs:cra_crx"</font>
<br><font size=2 face="sans-serif"> SB ondisk format = 1308</font>
<br><font size=2 face="sans-serif"> SB multihost format = 1401</font>
<br><font size=2 face="sans-serif"> Block size = 4096</font>
<br><font size=2 face="sans-serif"> Journals = 11</font>
<br><font size=2 face="sans-serif"> Resource Groups = 1988</font>
<br><font size=2 face="sans-serif"> Mounted lock proto = "lock_gulm"</font>
<br><font size=2 face="sans-serif"> Mounted lock table = "cra_gfs:cra_crx"</font>
<br><font size=2 face="sans-serif"> Mounted host data = ""</font>
<br><font size=2 face="sans-serif"> Journal number = 0</font>
<br><font size=2 face="sans-serif"> Lock module flags = async </font>
<br><font size=2 face="sans-serif"> Local flocks = FALSE</font>
<br><font size=2 face="sans-serif"> Local caching = FALSE</font>
<br>
<br><font size=2 face="sans-serif"> Type
Total Used
Free use%
</font>
<br><font size=2 face="sans-serif"> ------------------------------------------------------------------------</font>
<br><font size=2 face="sans-serif"> inodes
933593 933593 0
100%</font>
<br><font size=2 face="sans-serif"> metadata
943899 121868 822031
13%</font>
<br><font size=2 face="sans-serif"> data
128274180 58546879 69727301
46%</font>
<br>
<br>
<br><font size=2 face="sans-serif">[root@iclc1g tmp]# gfs_tool df /crx/data</font>
<br><font size=2 face="sans-serif">/crx/data:</font>
<br><font size=2 face="sans-serif"> SB lock proto = "lock_gulm"</font>
<br><font size=2 face="sans-serif"> SB lock table = "cra_gfs:cra_crxdata"</font>
<br><font size=2 face="sans-serif"> SB ondisk format = 1308</font>
<br><font size=2 face="sans-serif"> SB multihost format = 1401</font>
<br><font size=2 face="sans-serif"> Block size = 4096</font>
<br><font size=2 face="sans-serif"> Journals = 11</font>
<br><font size=2 face="sans-serif"> Resource Groups = 5970</font>
<br><font size=2 face="sans-serif"> Mounted lock proto = "lock_gulm"</font>
<br><font size=2 face="sans-serif"> Mounted lock table = "cra_gfs:cra_crxdata"</font>
<br><font size=2 face="sans-serif"> Mounted host data = ""</font>
<br><font size=2 face="sans-serif"> Journal number = 0</font>
<br><font size=2 face="sans-serif"> Lock module flags = async </font>
<br><font size=2 face="sans-serif"> Local flocks = FALSE</font>
<br><font size=2 face="sans-serif"> Local caching = FALSE</font>
<br>
<br><font size=2 face="sans-serif"> Type
Total Used
Free use%
</font>
<br><font size=2 face="sans-serif"> ------------------------------------------------------------------------</font>
<br><font size=2 face="sans-serif"> inodes
3296091 3296091 0
100%</font>
<br><font size=2 face="sans-serif"> metadata
2649271 616186 2033085
23%</font>
<br><font size=2 face="sans-serif"> data
385236382 310495360 74741022
81%</font>
<br>
<br><font size=2 face="sans-serif">[root@iclc1g tmp]# gfs_tool df /crx/home</font>
<br><font size=2 face="sans-serif">/crx/home:</font>
<br><font size=2 face="sans-serif"> SB lock proto = "lock_gulm"</font>
<br><font size=2 face="sans-serif"> SB lock table = "cra_gfs:cra_crxhome"</font>
<br><font size=2 face="sans-serif"> SB ondisk format = 1308</font>
<br><font size=2 face="sans-serif"> SB multihost format = 1401</font>
<br><font size=2 face="sans-serif"> Block size = 4096</font>
<br><font size=2 face="sans-serif"> Journals = 11</font>
<br><font size=2 face="sans-serif"> Resource Groups = 3978</font>
<br><font size=2 face="sans-serif"> Mounted lock proto = "lock_gulm"</font>
<br><font size=2 face="sans-serif"> Mounted lock table = "cra_gfs:cra_crxhome"</font>
<br><font size=2 face="sans-serif"> Mounted host data = ""</font>
<br><font size=2 face="sans-serif"> Journal number = 0</font>
<br><font size=2 face="sans-serif"> Lock module flags = async </font>
<br><font size=2 face="sans-serif"> Local flocks = FALSE</font>
<br><font size=2 face="sans-serif"> Local caching = FALSE</font>
<br>
<br><font size=2 face="sans-serif"> Type
Total Used
Free use%
</font>
<br><font size=2 face="sans-serif"> ------------------------------------------------------------------------</font>
<br><font size=2 face="sans-serif"> inodes
3477487 3477487 0
100%</font>
<br><font size=2 face="sans-serif"> metadata
3162164 341627 2820537
11%</font>
<br><font size=2 face="sans-serif"> data
254032093 157709829 96322264
62%</font>
<br>
<br><font size=2 face="sans-serif">[root@iclc1g tmp]# gfs_tool df /usr/local</font>
<br><font size=2 face="sans-serif">/usr/local:</font>
<br><font size=2 face="sans-serif"> SB lock proto = "lock_gulm"</font>
<br><font size=2 face="sans-serif"> SB lock table = "cra_gfs:cra_usrlocal"</font>
<br><font size=2 face="sans-serif"> SB ondisk format = 1308</font>
<br><font size=2 face="sans-serif"> SB multihost format = 1401</font>
<br><font size=2 face="sans-serif"> Block size = 4096</font>
<br><font size=2 face="sans-serif"> Journals = 11</font>
<br><font size=2 face="sans-serif"> Resource Groups = 394</font>
<br><font size=2 face="sans-serif"> Mounted lock proto = "lock_gulm"</font>
<br><font size=2 face="sans-serif"> Mounted lock table = "cra_gfs:cra_usrlocal"</font>
<br><font size=2 face="sans-serif"> Mounted host data = ""</font>
<br><font size=2 face="sans-serif"> Journal number = 0</font>
<br><font size=2 face="sans-serif"> Lock module flags = async </font>
<br><font size=2 face="sans-serif"> Local flocks = FALSE</font>
<br><font size=2 face="sans-serif"> Local caching = FALSE</font>
<br>
<br><font size=2 face="sans-serif"> Type
Total Used
Free use%
</font>
<br><font size=2 face="sans-serif"> ------------------------------------------------------------------------</font>
<br><font size=2 face="sans-serif"> inodes
765762 765762 0
100%</font>
<br><font size=2 face="sans-serif"> metadata
582989 22854 560135
4%</font>
<br><font size=2 face="sans-serif"> data
24393837 9477084 14916753
39%</font>
<br>
<br><font size=2 face="sans-serif">[root@iclc1g tmp]# gfs_tool df /data</font>
<br><font size=2 face="sans-serif">/data:</font>
<br><font size=2 face="sans-serif"> SB lock proto = "lock_gulm"</font>
<br><font size=2 face="sans-serif"> SB lock table = "cra_gfs:cra_GQ"</font>
<br><font size=2 face="sans-serif"> SB ondisk format = 1308</font>
<br><font size=2 face="sans-serif"> SB multihost format = 1401</font>
<br><font size=2 face="sans-serif"> Block size = 4096</font>
<br><font size=2 face="sans-serif"> Journals = 11</font>
<br><font size=2 face="sans-serif"> Resource Groups = 1298</font>
<br><font size=2 face="sans-serif"> Mounted lock proto = "lock_gulm"</font>
<br><font size=2 face="sans-serif"> Mounted lock table = "cra_gfs:cra_GQ"</font>
<br><font size=2 face="sans-serif"> Mounted host data = ""</font>
<br><font size=2 face="sans-serif"> Journal number = 0</font>
<br><font size=2 face="sans-serif"> Lock module flags = async </font>
<br><font size=2 face="sans-serif"> Local flocks = FALSE</font>
<br><font size=2 face="sans-serif"> Local caching = FALSE</font>
<br>
<br><font size=2 face="sans-serif"> Type
Total Used
Free use%
</font>
<br><font size=2 face="sans-serif"> ------------------------------------------------------------------------</font>
<br><font size=2 face="sans-serif"> inodes
10026 10026
0 100%</font>
<br><font size=2 face="sans-serif"> metadata
282680 189037 93643
67%</font>
<br><font size=2 face="sans-serif"> data
103761726 94277221 9484505
91%</font>
<br>