[Linux-cluster] nfs/smb slow on gfs2

[Arwin L Tugade]

For me at least on gfs, I'm using lock_dlm and I was getting like 1.3-1.5 MB/s transfer rates on samba.  I know you said you using lock_nolock but I say give ping_pong a shot:

http://wiki.samba.org/index.php/Ping_pong


Another thing to try to see if locking is the culprit is in smb.conf set posix locking = no and try a transfer.  That's why my transfer rates were so bad.  With ping_pong I noticed 94-100 locks per seconds.  Turns out gfs_controld has 100 locks per/s as the default and can be lifted by -l0 flag.  With that on, I then got like 25-30k locks per/s and my transfer rates went up with samba to like 8 - 9 MB/s (on 100mbit lan).  I also tried the plock optimization that I believe was introduced not to many versions ago and the rate when up to like 100k and transfer rate got a little better.  It's probably best to put some lower limit as a side effect is more network load and lock coordination amongst the nodes, I'm trying to find a good balance with my setup.  I tried all this in a test environment btw and I'm still tweaking/learning about it :).  I mean that was my personal experience with gfs (not gfs2, don't know if it's the same).  Hope this helps.

Regards,
Arwin

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Andrew A. Neuschwander
Sent: Thursday, April 23, 2009 11:03 AM
To: linux clustering
Subject: [Linux-cluster] nfs/smb slow on gfs2

I have a local only (lock_nolock) gfs2 filesystem which is served to 
clients via smb and nfs v3. Crawling around the filesystem is extremely 
slow over smb and nfs. Stat, ls and find over the network take many 
orders of magnitude longer than I'd expect.

The same tasks are very fast locally on the gfs2 filesystem. Also, for 
comparison, I created an ext3 on the same server and exported it via smb 
and nfs. The same task over nfs or samba on ext3 is nearly as fast as 
doing the task locally.

Is there any way to optimized the interaction between smb/nfs and gfs2?

I'm using gfs2 because I needed a filesystem over 8TB. I'm on CentOS5.3 
(2.6.18-128.1.6.el5).

The storage is locally attached. It is two hardware RAID6 (11 disks each 
+ 1 hotspare) arrays striped together with software raid, then managed 
with lvm2. Both the ext3 and gfs2 are on this storage. Large file 
sequential reads are fast. For example, dd if=file-on-nfs of=/dev/null 
will where file-on-nfs is an 80GB file stored on the gfs2 will max out a 
GigE connection.

Any insight or advice on speeding up these operations would be appreciated.

Thanks,
-Andrew
--
Andrew A. Neuschwander, RHCE
Systems/Software Engineer
College of Forestry and Conservation
The University of Montana
http://www.ntsg.umt.edu
andrew at ntsg.umt.edu - 406.243.6310

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster