[Linux-cluster] GFS load average and locking

[Treece, Britt]

Wendy,

Unfortunately our customer has (for the time being) moved their PHP
sessions off of the GFS filesystem because of the instability.  Our GFS
performance has returned to normal, but our customer expects us to fix
GFS so that they can have the PHP sessions on GFS.  I'm *attempting* to
reproduce the behavior on a lab GFS cluster.  Assuming I can
successfully do this I will send strace's of the issue as it occurs.

Is Redhat aware of any issues with GFS and flock syscalls?  

Regarding the U7 kernel suggestion you made previously, is this going to
help with the flock issue or is it strictly for keeping the number of
cached locks down?

Britt


-----Original Message-----
From: Wendy Cheng [mailto:wcheng at redhat.com] 
Sent: Thursday, March 09, 2006 2:33 PM
To: linux clustering
Cc: Stanley, Jon; Treece, Britt
Subject: Re: [Linux-cluster] GFS load average and locking

Marc Grimme wrote:

>Although the strace does not show the output I know of the problem
description 
>sounds like a deja vu.
>We had loads of problems with having sessions on GFS and httpd s ending
up 
>with "D" state for some time (at high load times we had ServerLimit
httpd in 
>D per node which ended up in the service not being available). 
>As I posted already we think it is because of the "bad" locking of
sessions 
>with php (as php sessions are on gfs and strace showed those timeouts
with 
>the session files). When you issue a "session_start" or what ever that 
>function is called, the session_file is locked via an flock syscall.
That 
>lock is held until you end the session which is implicitly done when
the tcp 
>connection to the client is ended. Now comes another http process (on 
>whatever node) and calls a "session start" and trys an flock on that
session 
>while another process already holds that lock. The process might end up
in 
>the seen timeouts (30-60secs) which (as far as I remember relates to
the 
>timeout of the tcp connection defined in the httpd.conf or some timeout
in 
>the php.ini) - there is an explanation on this but I cannot rember ;-)
). 
>Nevertheless in our scenario the problems were the "bad" session
handling by 
>php. We have made a patch for the phplib where you can disable the
locking, 
>or just implicitly do locking and therefore keep consitency while
session 
>data is read or written. We could make apache work as expected and now
we 
>don't see any "D" process anymore since a year.
>Oh yes the patch can be found at
>www.opensharedroot.org in the download section.
>
>Besides: You will never encounter this on a localfilesystem or nfs (as
nfs 
>ignores flocks). As nfs does not support flocks and silently ignores
them.
>
>  
>
Hi,

This does look like the problem description sent out by savvis.net folks

during our off-list email exchanges. However, without actually looking 
at the thread traces (when they are in D state), it is difficult to be 
sure. One way to obtain the exact thread trace is using "crash" tool to 
do a back trace (e.g. "bt <pid>", you need kernel debuginfo RPM though).

Britt, do let us know whether this php patch helps and/or using crash 
command to obtain the thread trace output.

On the other hand, I don't understand how a local (non-cluster) 
filesystem can be immune from this problem ?

-- Wendy