[Linux-cluster] performace tuning

[gordan at bobich.net]

Try mounting the FS with noatime,nodiratime,noquota.

Gordan

On Mon, 4 Feb 2008, Shawn Hood wrote:

> Hey all,
>
> My company has gone live with a GFS cluster this morning.  It is a 4
> node RHEL4U6 cluster, running RHCS and GFS.  It mounts an Apple 4.5TB
> XRAID configured as RAID5, whose physical volumes are combined into
> one large volume group.  From this volume group, five striped LVMs
> (striped across the two physical volumes of the XRAID) were created.
> Five GFS filesystems were created, one on each logical volume. Even
> though there are currently four nodes, there are 12 journals for each
> filesystem to allow for planned cluster growth.
>
> Currently, each filesystem is mounted noatime, and tunebles
> quota_enforce and quota_account are set to 0.  I have posted the
> results of gfs_tool gettune /hq-san/nlp/nlp_qa below.  We have an
> application which depends heavily upon a find command that lists a
> number of files.  It looks something like this:
> find $delta_home -name summary -maxdepth 2 -type f
>
> Its output consists of thousands of files that exist on
> /hq-san/nlp/nlp_qa.  This command is CRAWLING at the moment.  An ext3
> filesystem would output hundreds of matches a second.  This GFS
> filesystem is currently outputting 100-200/minutes.  This is crippling
> one of our applications.  Any advice on tuning this filesystem for
> this kind of access would be greatly appreciated.
>
> Output from a gfs_tool df /hq-san/nlp_qa:
>
> odin / # gfs_tool df /hq-san/nlp/nlp_qa
> /hq-san/nlp/nlp_qa:
>  SB lock proto = "lock_dlm"
>  SB lock table = "boson:nlp_qa"
>  SB ondisk format = 1309
>  SB multihost format = 1401
>  Block size = 4096
>  Journals = 12
>  Resource Groups = 3009
>  Mounted lock proto = "lock_dlm"
>  Mounted lock table = "boson:nlp_qa"
>  Mounted host data = ""
>  Journal number = 1
>  Lock module flags =
>  Local flocks = FALSE
>  Local caching = FALSE
>  Oopses OK = FALSE
>
>  Type           Total          Used           Free           use%
>  ------------------------------------------------------------------------
>  inodes         15167101       15167101       0              100%
>  metadata       868298         750012         118286         86%
>  data           219476789      192088469      27388320       88%
>
>
>
> Output from a df -h:
>
> /dev/mapper/hq--san-cam_development 499G  201G  298G  41%
> /hq-san/nlp/cam_development
> /dev/mapper/hq--san-nlp_qa 899G  794G  105G  89% /hq-san/nlp/nlp_qa
> /dev/mapper/hq--san-svn_users 1.5T  1.3T  282G  82% /hq-san/nlp/svn_users
> /dev/mapper/hq--san-development 499G  373G  126G  75% /hq-san/nlp/development
> /dev/mapper/hq--san-prod_reports 1023G  680G  343G  67% /hq-san/nlp/prod_reports
>
> odin / # gfs_tool gettune /hq-san/nlp/nlp_qa
> ilimit1 = 100
> ilimit1_tries = 3
> ilimit1_min = 1
> ilimit2 = 500
> ilimit2_tries = 10
> ilimit2_min = 3
> demote_secs = 300
> incore_log_blocks = 1024
> jindex_refresh_secs = 60
> depend_secs = 60
> scand_secs = 5
> recoverd_secs = 60
> logd_secs = 1
> quotad_secs = 5
> inoded_secs = 15
> glock_purge = 0
> quota_simul_sync = 64
> quota_warn_period = 10
> atime_quantum = 3600
> quota_quantum = 60
> quota_scale = 1.0000   (1, 1)
> quota_enforce = 0
> quota_account = 0
> new_files_jdata = 0
> new_files_directio = 0
> max_atomic_write = 4194304
> max_readahead = 262144
> lockdump_size = 131072
> stall_secs = 600
> complain_secs = 10
> reclaim_limit = 5000
> entries_per_readdir = 32
> prefetch_secs = 10
> statfs_slots = 64
> max_mhc = 10000
> greedy_default = 100
> greedy_quantum = 25
> greedy_max = 250
> rgrp_try_threshold = 100
> statfs_fast = 0
> seq_readahead = 0
>
>
> Shawn
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>