[Linux-cluster] GFS is *very* slow when NFS exported
Stephen Willey
stephen.willey at framestore-cfc.com
Fri Jul 23 10:53:46 UTC 2004
We are looking at using GFS for load balanced NFS. Going with GNBD
exported GFS isn't really an option since we're talking about over 1000
machines needing to access the storage. For this reason we were looking
at providing a relatively small cluster of GFS machines serving the
storage via NFS. The clients would then use round-robin DNS to load
balance across these servers.
The results we've got back from our tests are shown below:
-= Setup ========================================-
Two machines with GFS filesystem mounted from a
dual-port RAID, connected to ethernet via GigE and
serving NFS as follows:
/mnt/gfs *(no_root_squash,rw,insecure,async)
-= End of Setup =================================-
-= Local GFS Filesystem Access ==================-
The tests show Mb/s computed by using 10Gb dd
1 machine, write (gfstest1):
166
1 machine, read (gfstest2 reading file gfstest1 created):
139.5
2 machines, sim writes (different files):
113.7 (gfstest1) 108 (gfstest2) - 221.7aggr
2 machines, sim reads (different files):
101.5 (gfstest1) 101.6 (gfstest2) - 203aggr
2 machines, sim reads (same file):
130 (gfstest1) 134 (gfstest2) - 264aggr
-= End of Local GFS Filesystem Access ===========-
-= NFS/GFS Access ===============================-
1 write: (client1 to gfstest1)
42.3
1 read: (client1 from gfstest1)
Varies enormously between 10-35Mb/s)
Simultaneous writes and reads done as follows:
client1 NFS mounting gfstest1
clients2 NFS mounting gfstest2
2 sim writes (different files):
39.6 (client1) 42.4 (client2) - 82aggr
2 sim reads (different files):
11.3 (client1) 11.2 (client2) - 22.5aggr
-= End of NFS/GFS Access ========================-
-= NFS/XFS Access ===============================-
Done for comparison of XFS & GFS export speeds
1 write: 65.2
1 read: 73.5
-= End of NFS/XFS Access ========================-
We know NFS isn't the highest performing thing in the world, but it's a
concern that the NFS performance of a GFS mounted filesystem is so much
lower than that of an XFS system. There are of course, the clustering
overheads, but this would affect local performance as well. Anyone got
any ideas as to why this might be and how to get more performance?
Thanks,
Stephen
More information about the Linux-cluster
mailing list