[Linux-cluster] GFS on CentOS 5.2
jeff.sturm at eprize.com
Tue Dec 9 04:16:52 UTC 2008
Interesting. Can you share more details of your application, such as
the access patterns of the files/directories involved, and the directory
We faced some similar challenges in our cluster, in which we use GFS for
web session storage. Performance was improved dramatically by several
changes we experimented with:
- GFS tunables: glock_purge, demote_secs, etc.)
- Network optimization: using different interfaces tuned for DLM/OpenAIS
traffic vs. iSCSI
- Locality: Having each node create session files in a directory
dedicated to that node. Different nodes can read files created on other
nodes, but are not permitted to create files outside their own
- Session affinity: Having repeat session hits come back to the same
node when possible.
We also found we needed to specify "nodiratime" in addition to
"noatime", though this contradicts certain documentation that suggests
the two are redundant.
GFS2 may significantly outpeform GFS for these usage patterns. As RHEL
5.3 is still in beta and the implementation of GFS2 in 5.2 is not
production-ready, we have not had much opportunity to test GFS2
Some general hints we found:
- If you have much temporal locality in your application, tune GFS to
extend the lifetime of locks, such as increasing demote_secs.
Consecutive accesses by the same node to the same file over GFS may
perform nearly as well as local storage.
- Avoid having different nodes create files in a common directory at
once, if possible. This creates a ping-pong effect as the directory
lock bounces between nodes.
- If you have a particularly large number of files, try setting
glock_purge to a value greater than zero. This may help especially if
you are experiencing high system CPU usage.
- Pay attention to statistics such as you'll see with "gfs_tool
counters" to understand how many locks are acquired by your app over
time, and how often they are accessed. Any change to decrease the
frequency of lock traffic is a performance win.
GFS can be a bit of a fuss to configure, but once running our experience
with 5.2 and GFS has been rock-steady. Good luck.
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Shaun Mccullagh
> Sent: Monday, December 08, 2008 3:00 PM
> To: linux clustering
> Subject: [Linux-cluster] GFS on CentOS 5.2
> A few weeks ago I started to setup a 6 node GFS cluster
> connected to a SAN with Fiber HBAs. Each node is connected to
> 2 gigabit switches for redundancy.
> Up until a few days ago things were going very well, but
> intensive testing showed that we have a bit of a performance problem.
> Perhaps I should explain that we want to use GFS in a LAMP
> environment to speed up a customer website.
> The old system consisted of six nodes which shared files
> using glorious NFS.
> But we have found to our dismay that Apache is taking much
> longer to access files via GFS than NFS. Strace shows that
> the Apache workers are waiting on fstat calls. They take 4
> times longer to complete on GFS than NFS
> I have tried the following:
> - disabled atime updating: mount option 'noatime'
> - disabled quota: mount option 'noquota'
> - increased statfs_slots: gfs_tool settune <mountpoint>
> statfs_slots 128
> - enabled statfs_fast (gfs_tool settune <mountpoint> statfs_fast 1)
> These improved things slightly but GFS is still much slower than NFS
> The CPU load is much higher with GFS, vmstat shows that
> Context switching is > 5 greater than that of an NFS client.
> The GFS filesystem is only 10GB.
> We are getting a bit desperate so I'd grateful for any
> pointers as to what might be going wrong.
> Op dit e-mailbericht is een disclaimer van toepassing, welke
> te vinden is op http://www.espritxb.nl/disclaimer
> NB: Vanaf heden zijn al onze mailadressen gewijzigd in
> ... at espritxb.nl! Pas dit svp aan in uw adresboek.
> Linux-cluster mailing list
> Linux-cluster at redhat.com
More information about the Linux-cluster