[Linux-cluster] Locking in GFS

[Wendy Cheng]

Wendy Cheng wrote:
> Chris Joelly wrote:
>> Hello,
>>
>> i have a question on locking issues on GFS:
>>
>> how do GFS lock files on the filesystem. I have found one posting to
>> this list which states that locking occurs "more or less" on file 
>> level. Is this true? or does some kind of locking occur on directory
>> level too?
>>   
>
> You may view GFS(1) internal lock granularity is on system call level 
> - that is, when either a write or read (say pwrite() or pread()) is 
> issued, the associated file is locked until the system call returns. 
> There are few simple things that will be helpful if you keep them in 
> mind:
>
> 1. a write requires an exclusive lock (i.e., when there is a write 
> going on, every access to that file has to wait).
> 2. a read needs a shared lock (i.e.  many reads to the same file will 
> not be stalled).
> 3. a write may involve directory lock (e.g. a "create" would need a 
> write lock of the parent directory).
> 4. local locking (two writes compete the same lock on the same node) 
> is always much better than inter-node (different nodes) locking 
> (ping-pong the same write lock between different nodes is very 
> expensive).
> 5. standard APIs (such as fcntl() and flock()) precedes GFS(1) 
> internal locking if used correctly (e.g. upon obtaining an exclusive 
> flock, other access to that file will be stalled, assuming every 
> instance of the executables running on different nodes has the correct 
> flock implemented and honored).
>
> One exception to (5) is posix byte-range locks. Say there are two 
> processes running on different nodes, each obtaining its own byte 
> range locks. Process A locks byte offset 0 to 10K; process B locks 
> byte 10K+1 to 40K. When both have writes issued, one of them has to 
> wait until other's write system call completes before it can continue 
> - a result of its posix locking implementation that is done by a 
> separate module outside its internal filesystem locking.
>
>> The question arrose because i was thinking on whats would happen when
>> using GFS on an file server with home directories for lets say 1000
>> users. how do i setup this directory structure to avoid locking issues
>> as best as possible.
>>
>> is a directory layout like the following ok when /home is a GFS file 
>> system:
>>
>> /home/a/b/c/abc
>> /home/d/e/f/def
>> /home/g/h/i/ghi
>> ...
>> /home/x/y/z/xyz
>>
>>   
>
> Hope above statements have helped you understanding that on GFS(1),
>
> 1. A short-and-fat directory structure will work (much) better than 
> tall-and-skinny ones.
> 2. If possible, the directory setup should avoid ping-pong directory 
> and/or write locks between different nodes

Really think about it, (1) is *not* a right description at all. What I 
meant to say is that, if possible, try not to put everyone in the very 
same directory if there are lots of write activities that will cause 
directory lock(s) contention, particularly if it (they) has (have) to 
get passed around different nodes.

Sorry !

-- Wendy