[Linux-cluster] Slowness above 500 RRDs

[Ferenc Wagner]

David Teigland <teigland at redhat.com> writes:

> On Wed, Mar 28, 2007 at 11:27:27AM -0500, David Teigland wrote:
>> On Wed, Mar 28, 2007 at 05:48:10PM +0200, Wagner Ferenc wrote:
>>> Ferenc Wagner <wferi at niif.hu> writes:
>>> 
>>>> There's a good bunch of RRDs in a directory.  A script scans them for
>>>> their last modification times, and then updates each in turn for a
>>>> couple of times.  The number of files scanned and the length of the
>>>> update rounds are printed.  The results are much different for 500 and
>>>> 501 files:
>>>>
>>>> filecount=501
>>>>   iteration=0 elapsed time=10.425568 s
>>>>   iteration=1 elapsed time= 9.766178 s
>>>>   iteration=2 elapsed time=20.14514 s
>>>>   iteration=3 elapsed time= 2.991397 s
>>>>   iteration=4 elapsed time=20.496422 s
>>>> total elapsed time=63.824705 s
>>>>
>>>> filecount=500
>>>>   iteration=0 elapsed time=6.560811 s
>>>>   iteration=1 elapsed time=0.229375 s
>>>>   iteration=2 elapsed time=0.202973 s
>>>>   iteration=3 elapsed time=0.203439 s
>>>>   iteration=4 elapsed time=0.203095 s
>>>> total elapsed time=7.399693 s
>>> 
>>> Following up to myself with one more data point: raising
>>> SHRINK_CACHE_MAX from 1000 to 20000 in gfs/dlm/lock_dlm.h helps
>
>> SHRINK_CACHE_MAX is related to fcntl posix locks, did you intend to change
>> the app to use flock (which is much faster that fcntl)?

I modified librrd to use flock instead if the LIBRRD_USE_FLOCK
environmental variable is defined.  It makes this case evenly slow
(with SHRINK_CACHE_MAX et al. left at their defaults):

filecount=500
  iteration=0 elapsed time=10.335737 s
  iteration=1 elapsed time=10.426273 s
  iteration=2 elapsed time=10.634286 s
  iteration=3 elapsed time=10.342333 s
  iteration=4 elapsed time=10.458371 s
total elapsed time=52.197 s

(The same for 501 files).  strace -T reveals an irregular pattern here:

filecount=5
flock(3, LOCK_EX)                       = 0 <0.000116>
flock(3, LOCK_EX)                       = 0 <0.000261>
flock(3, LOCK_EX)                       = 0 <0.000113>
flock(3, LOCK_EX)                       = 0 <0.037657>
flock(3, LOCK_EX)                       = 0 <0.000093>
  iteration=0 elapsed time=0.04244 s
flock(3, LOCK_EX)                       = 0 <0.000094>
flock(3, LOCK_EX)                       = 0 <0.037314>
flock(3, LOCK_EX)                       = 0 <0.000105>
flock(3, LOCK_EX)                       = 0 <0.038323>
flock(3, LOCK_EX)                       = 0 <0.000087>
  iteration=1 elapsed time=0.079957 s
[...]

So I've got the impression that flock is slower than fcntl locking.
The difference probably erodes with greater number of files, both
being equally slow.  I must be missing something, it's just the
opposite of what you suggest.

>> > significantly, but still isn't enough.  Besides, I don't know what I'm
>> > doing.  Should I tweak the surrounding #defines, too?
>
> lock_dlm caches dlm locks for old plocks for a while in an attempt to
> improve performance and reduce thrashing the dlm -- SHRINK_CACHE_MAX is
> the max level of caching, it's fine to change it as you've done.  The fact
> that you're hitting it, though, indicates that your app is using plocks
> more heavily than gfs/dlm are suited to handle.  Switching to flock will
> obviate all of this.  (Or switching to the new dlm and cluster
> infrastructure which has a completely different and far better approach to
> plocks).

The lock usage of my application is very simple, at least in this
testing phase, like I described at the start of the thread.  I'd be
glad to go into more detail if you are willing to assess my use case.
Since flock doesn't seem to help at all, I guess something is missing
from the picture.

Also, what's that new infrastructure?  Do you mean GFS2?  I read it
was not production quality yet, so I didn't mean to try it.  But again
you may have got something else in your head...
-- 
Thanks,
Feri.