[Linux-cluster] GFS tuning for combined batch / interactive use

[Kevin Maguire]

Hi

> You can get a glock dump via debugfs which may show up contention, looks 
> for type 2 glocks which have lots of lock requests queued but not 
> granted. The lock requests (holders) are tagged with the relevant 
> process.

Note I am currently using GFS, not GFS2. And before going further I ran 
the ping_pong test on my cluster and see only about 100 locks/second even 
on just 1 node.  So maybe I should look at plock_rate_limit parameter, 
though not sure if that is our core problem.

Anyways, As I write this my test cluster is being heavily used with batch 
jobs, and thus I have a window of opportunity to study it under load (but 
not change it).  I have debugfs mounted. There are 10 nodes in this test 
cluster. My filesystem is called mygfs, and was created via

mkfs.gfs -O -t dfoxen-cluster:mygfs -p lock_dlm -j 10 -r 2048 /dev/mapper/vggfs-lvgfs

This is what I have in debugfs:

# find /sys/kernel/debug/ -type f -exec wc -l {} \;
2309 /sys/kernel/debug/dlm/mygfs_locks
0 /sys/kernel/debug/dlm/mygfs_waiters
16258 /sys/kernel/debug/dlm/mygfs
2 /sys/kernel/debug/dlm/clvmd_locks
0 /sys/kernel/debug/dlm/clvmd_waiters
7 /sys/kernel/debug/dlm/clvmd

The lock dump file has content like:

# cat /sys/kernel/debug/dlm/mygfs_locks
id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name
14f19eb 0 0 1038 0 0 0 2 3 -1 0 0 24 "       5         cec3e6d"
3da1a67 0 0 31861 0 0 0 2 3 -1 0 0 24 "       5         a0fafc2"
1120003 1 16f0019 3552 0 408 0 2 0 -1 0 1 24 "       3        2d8b9091"
af0002 1 10024 3552 0 408 0 2 0 -1 0 1 24 "       3        2053fbf8"
...

But I don't really see how to work our which type of lock is which from 
this file - sorry. Given $2 is the nodeid I can work our who has locks and 
that leads to a minor strangeness

node1 # awk 'NR>1{print $2}' /sys/kernel/debug/dlm/mygfs_locks | sort | uniq -c | sort -k +2n
    2142 0
    1619 2
    2001 3
    1586 4
    1566 5
    1624 6
    1610 7
    1733 8
    1592 9
    1612 10

These numbers are much bigger than the counts on the 9 other nodes, e.g.

node2 # awk 'NR>1{print $2}' /sys/kernel/debug/dlm/mygfs_locks | sort | uniq -c | sort -k +2n
     441 0
    1630 1
      75 3
       2 4
      10 5
      25 7
      15 8
      38 10

Is that normal ?

Using gfs_tool's lockdump I see

node1 # gfs_tool lockdump /newcache | egrep '^Glock' | sed 's?(\([0-9]*\).*)?\1?g' | sort | uniq -c
       3 Glock 1
     308 Glock 2
    1538 Glock 3
       2 Glock 4
     233 Glock 5
       2 Glock 8

Only type 2 and type 5 counts seem to change. Across the cluster there is 
one node with a lot more (10x more) Glock type 2 and Glock type 5 locks.

# gfs_tool counters /newcache

                                   locks 2313
                              locks held 781
                            freeze count 0
                           incore inodes 230
                        metadata buffers 1061
                         unlinked inodes 28
                               quota IDs 2
                      incore log buffers 28
                          log space used 1.46%
               meta header cache entries 1304
                      glock dependencies 185
                  glocks on reclaim list 0
                               log wraps 91
                    outstanding LM calls 0
                   outstanding BIO calls 0
                        fh2dentry misses 0
                        glocks reclaimed 2125924
                          glock nq calls 801437507
                          glock dq calls 796261692
                    glock prefetch calls 319835
                           lm_lock calls 6396763
                         lm_unlock calls 1031709
                            lm callbacks 7669741
                      address operations 1267096416
                       dentry operations 35815146
                       export operations 0
                         file operations 233333825
                        inode operations 61818196
                        super operations 148712313
                           vm operations 87114
                         block I/O reads 0
                        block I/O writes 0

Not sure if anyone can make anything from all these numbers ...

Thanks,
Kevin