[Linux-cluster] Performance tuning help sought

[Kadlecsik Jozsef]

Hi,

On Tue, 11 Mar 2008, Ferenc Wagner wrote:

> Kadlecsik Jozsef <kadlec at sunserv.kfki.hu> writes:
> 
> > Which knobs should we tune in order to achieve the fastest lock 
> > processing?
> 
> Try starting gfs_controld with -l0, 

We run our GFS cluster with 'gfs_controld -l0' from the beginning, thanks 
to your messages on the mailing list :-).

> and maybe decrease demote_secs (you don't access the same file 
> repeatedly after all).

Increasing demote_secs to 600 or decreasing it to 150 did not help. 
We set glock_purge to 50 and it did not increase the performance either.
The volumes are mounted with noatime, but quota is enabled.

There are max ~100 users (including POP/IMAP) using the cluster at peak 
and sometimes it takes 30-50s to load in a 30MB folder. As one of our 
users expressed "It is so slow as if I connected to the cluster via mobile 
GPRS." The blind users who use jaws screen reader complain that the reader 
simply goes out of sync at waiting to be displayed something to read.

This is cluster-2.01.00. Some stats:

# gfs_tool df /gfs/home
/gfs/home:
  SB lock proto = "lock_dlm"
  SB lock table = "kfki:home"
  SB ondisk format = 1309
  SB multihost format = 1401
  Block size = 4096
  Journals = 16
  Resource Groups = 4088
  Mounted lock proto = "lock_dlm"
  Mounted lock table = "kfki:home"
  Mounted host data = "jid=2:id=196609:first=0"
  Journal number = 2
  Lock module flags = 0
  Local flocks = FALSE
  Local caching = FALSE
  Oopses OK = FALSE

  Type           Total          Used           Free           use%           
  ------------------------------------------------------------------------
  inodes         592913         592913         0              100%
  metadata       2748600        42475          2706125        2%
  data           264538979      12255826       252283153      5%

And a counter-snapshot:

# gfs_tool counters /gfs/home

                                  locks 5778
                             locks held 4913
                           freeze count 0
                          incore inodes 737
                       metadata buffers 884
                        unlinked inodes 0
                              quota IDs 0
                     incore log buffers 0
                         log space used 0.20%
              meta header cache entries 0
                     glock dependencies 0
                 glocks on reclaim list 0
                              log wraps 332
                   outstanding LM calls 0
                  outstanding BIO calls 0
                       fh2dentry misses 0
                       glocks reclaimed 244939160
                         glock nq calls 382928115
                         glock dq calls 376758954
                   glock prefetch calls 101297055
                          lm_lock calls 155817206
                        lm_unlock calls 148752006
                           lm callbacks 304980555
                     address operations 50855951
                      dentry operations 71076079
                      export operations 0
                        file operations 56213431
                       inode operations 114720407
                       super operations 207151835
                          vm operations 9
                        block I/O reads 0
                       block I/O writes 0

Is there anything which indicates what's wrong or what could be tuned?

Best regards,
Jozsef
--
E-mail : kadlec at mail.kfki.hu, kadlec at blackhole.kfki.hu
PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address: KFKI Research Institute for Particle and Nuclear Physics
         H-1525 Budapest 114, POB. 49, Hungary