[Linux-cluster] Slowness above 500 RRDs

[David Teigland]

On Wed, Mar 28, 2007 at 11:27:27AM -0500, David Teigland wrote:
> On Wed, Mar 28, 2007 at 05:48:10PM +0200, Wagner Ferenc wrote:
> > Ferenc Wagner <wferi at niif.hu> writes:
> > 
> > > There's a good bunch of RRDs in a directory.  A script scans them for
> > > their last modification times, and then updates each in turn for a
> > > couple of times.  The number of files scanned and the length of the
> > > update rounds are printed.  The results are much different for 500 and
> > > 501 files:
> > >
> > > filecount=501
> > >   iteration=0 elapsed time=10.425568 s
> > >   iteration=1 elapsed time= 9.766178 s
> > >   iteration=2 elapsed time=20.14514 s
> > >   iteration=3 elapsed time= 2.991397 s
> > >   iteration=4 elapsed time=20.496422 s
> > > total elapsed time=63.824705 s
> > >
> > > filecount=500
> > >   iteration=0 elapsed time=6.560811 s
> > >   iteration=1 elapsed time=0.229375 s
> > >   iteration=2 elapsed time=0.202973 s
> > >   iteration=3 elapsed time=0.203439 s
> > >   iteration=4 elapsed time=0.203095 s
> > > total elapsed time=7.399693 s
> > 
> > Following up to myself with one more data point: raising
> > SHRINK_CACHE_MAX from 1000 to 20000 in gfs/dlm/lock_dlm.h helps

> SHRINK_CACHE_MAX is related to fcntl posix locks, did you intend to change
> the app to use flock (which is much faster that fcntl)?

> > significantly, but still isn't enough.  Besides, I don't know what I'm
> > doing.  Should I tweak the surrounding #defines, too?

lock_dlm caches dlm locks for old plocks for a while in an attempt to
improve performance and reduce thrashing the dlm -- SHRINK_CACHE_MAX is
the max level of caching, it's fine to change it as you've done.  The fact
that you're hitting it, though, indicates that your app is using plocks
more heavily than gfs/dlm are suited to handle.  Switching to flock will
obviate all of this.  (Or switching to the new dlm and cluster
infrastructure which has a completely different and far better approach to
plocks).

Dave