[Linux-cluster] GFS Performance Problems (RHEL5)

[Paul Risenhoover]

Hi James,

Like I said in my last email, my M500i has been swell so far, but I'm 
only using one interface.  In regards to your problems though, did you 
ever call Promise to get help?  I haven't had a big need to call them in 
the past, but when I have, they've been extremely helpful.

My thinking is that this is related to the glock problem that I 
mentioned in my previous email.   The reason I think this is because 
there are two large filesystems on two separate NAS's that I'm trying to 
combine into one.  One NAS has a directory structure with a max of about 
1000 files per directory.  When I transferred those files via rsync I 
got great performance with no stalling.

The other NAS has directories with 5000+ files, and it was only after 
those directories started showing up that the system began stalling.  
The problem is, now that those directories are here, the whole 
filesystem stalls, whether I'm accessing those directories or not.

The thing that's confusing to me is that, according to the writeup I 
read 
(http://people.redhat.com/wcheng/Patches/GFS/readme.gfs_glock_trimming.R4) 
the problem is with glocks.  From what I understand, a certain number of 
glocks are allocated and then the system needs to "garbage collect," 
which causes the pauses.  This indicates to me that there is a glock 
table that must be cleaned up on a regular basis.  Apparently I'm 
running out of lock space and that's why my system is pausing. 

My question is, how big is this table?  Can it be increased?  I've got a 
machine with 4G of ram and 2.5G are free.  I would think that if I could 
allocate another gig for the glock table and simply let gfs_scand run 
constantly at 10% of CPU, I might be able to prevent the stalls.

(I'm not asking you, James, for the answer, but I'm wondering if anybody 
else can chime in.)

Paul


James Chamberlain wrote:
> Hi Paul,
>
> In my experience with the VTrak M500i, it didn't seem like it could 
> handle active multipathing.  When I tried to use both interfaces 
> simultaneously rather than fail over between them, my throughput to 
> the disks dropped to less than 1 MB/s.  It looks like they've made 
> some improvements in the VTrak M610i, such as link aggregation, so 
> this may not be applicable for your newer hardware.
>
> You might want to check the management interface of your VTrak and see 
> if anything interesting is going on when everything freezes up on 
> you.  You might also want to check dmesg and your syslogs to see if 
> you see anything about the iSCSI session being lost and reestablished.
>
> Regards,
>
> James Chamberlain