[Linux-cluster] Locking in GFS

[Wendy Cheng]

Chris Joelly wrote:
> Hello,
>
> i have a question on locking issues on GFS:
>
> how do GFS lock files on the filesystem. I have found one posting to
> this list which states that locking occurs "more or less" on file 
> level. Is this true? or does some kind of locking occur on directory
> level too?
>   

You may view GFS(1) internal lock granularity is on system call level - 
that is, when either a write or read (say pwrite() or pread()) is 
issued, the associated file is locked until the system call returns. 
There are few simple things that will be helpful if you keep them in mind:

1. a write requires an exclusive lock (i.e., when there is a write going 
on, every access to that file has to wait).
2. a read needs a shared lock (i.e.  many reads to the same file will 
not be stalled).
3. a write may involve directory lock (e.g. a "create" would need a 
write lock of the parent directory).
4. local locking (two writes compete the same lock on the same node) is 
always much better than inter-node (different nodes) locking (ping-pong 
the same write lock between different nodes is very expensive).
5. standard APIs (such as fcntl() and flock()) precedes GFS(1) internal 
locking if used correctly (e.g. upon obtaining an exclusive flock, other 
access to that file will be stalled, assuming every instance of the 
executables running on different nodes has the correct flock implemented 
and honored).

One exception to (5) is posix byte-range locks. Say there are two 
processes running on different nodes, each obtaining its own byte range 
locks. Process A locks byte offset 0 to 10K; process B locks byte 10K+1 
to 40K. When both have writes issued, one of them has to wait until 
other's write system call completes before it can continue - a result of 
its posix locking implementation that is done by a separate module 
outside its internal filesystem locking.

> The question arrose because i was thinking on whats would happen when
> using GFS on an file server with home directories for lets say 1000
> users. how do i setup this directory structure to avoid locking issues
> as best as possible.
>
> is a directory layout like the following ok when /home is a GFS file 
> system:
>
> /home/a/b/c/abc
> /home/d/e/f/def
> /home/g/h/i/ghi
> ...
> /home/x/y/z/xyz
>
>   

Hope above statements have helped you understanding that on GFS(1),

1. A short-and-fat directory structure will work (much) better than 
tall-and-skinny ones.
2. If possible, the directory setup should avoid ping-pong directory 
and/or write locks between different nodes

For GFS2, after browsing thru its newest source code few minutes ago - 
it reminds me of a "sea horse" shape with linux page locks on its curly 
"belly" :). It is difficult (for me) to describe so I'll skip.

-- Wendy