[Linux-cluster] How does caching work in GFS1?

[Peter Schobel]

Is anyone else able to comment on this thread?

Thanks,

Peter
~

On Thu, Aug 12, 2010 at 9:25 AM, Peter Schobel <pschobel at 1iopen.net> wrote:
> In this cluster the nodes only have 4G of RAM and 4G of swap. Running
> top indicates that there are about 3G used and 1G free and nothing is
> swapping.
>
> So running gfs_tool counters shows me that there are around 200000
> locks and around 150000-160000 locks held.
>
> Glocks reclaimed is 22764386 and the rate is around 0-1000/s
>
> When running the test on the large directory, I see that the number of
> locks and locks held stays pretty much the same (below 200000). The
> number of glocks reclaimed fluctuates between 0-15000/s. The number of
> glock nq calls and glock dq calls is between 3000-6000/s.
>
> Currently I have glock_purge = 0 and reclaim_limit = 500000 so I don't
> understand why there are any glocks being reclaimed at all. This is
> what I have been struggling with since I don't feel as though the
> tunable parameter changes are doing anything.
>
> When running the test on the smaller directory, I see that the the
> usage patterns are pretty much the same initially but following that,
> the glocks reclaimed drops to between 0-5000/s and then on subsequent
> runs, the du completes so quickly and I can't see any glocks being
> reclaimed at all which is the behavior I would expect to see.
>
> It's a bit perplexing.
>
> Peter
> ~
>
> On Wed, Aug 11, 2010 at 2:35 PM, Jeff Sturm <jeff.sturm at eprize.com> wrote:
>>> -----Original Message-----
>>> From: linux-cluster-bounces at redhat.com
>> [mailto:linux-cluster-bounces at redhat.com]
>>> On Behalf Of Peter Schobel
>>> Sent: Wednesday, August 11, 2010 3:28 PM
>>> To: linux clustering
>>> Subject: Re: [Linux-cluster] How does caching work in GFS1?
>>>
>>> Increasing demote_secs did not seem to have an appreciable effect.
>>
>> We run some hosts with demote_secs=86400, for what it's worth.  They
>> tend to go through a "cold start" each morning, but are responsive for
>> the remainder of the day.
>>
>>> The du command is a simplification of the use case. Our developers run
>>> scripts which make tags in source code directories which require
>>> stat'ing the files.
>>
>> Gotcha.  I don't have many good suggestions for version control, but I
>> can offer commiseration.  Some systems are worse than others.
>> (Subversion for instance tends to create lots of little lock files, and
>> performs very poorly on just about every filesystem we've tried.)
>>
>> How much RAM do you have?  All filesystems like plenty of cache.
>>
>> One thing you can do is run "gfs_tool counters <mount-point>" a few
>> times during your 20GB test, that may give you some insight.  For
>> example, does the number of locks increase steadily or does it plateau?
>> Does it abruptly drop following the test?  Does the "glocks reclaimed"
>> number accumulate rapidly?  When locks are held, stat() operations tend
>> to be very fast.  When a lock has to be obtained, that's when they are
>> slow.
>>
>> (Any cluster engineers out there, feel free to tell me if any of this is
>> right or wrong--I've had to base my understanding of GFS on a lot of
>> experimentation and empirical evidence, not on a deep understanding of
>> the software.)
>>
>> -Jeff
>>
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>
>
>
> --
> Peter Schobel
> ~
>



-- 
Peter Schobel
~