[Cluster-devel] [GFS2 PATCH] GFS2: Don't brelse rgrp buffer_heads every allocation

[Steven Whitehouse]

Hi,

On 15/06/15 14:56, Bob Peterson wrote:
> ----- Original Message -----
>>>>> If you compare the two vmstat outputs in the bugzilla #1154782, you'll
>>>>> see no significant difference in memory usage nor cpu usage. So I assume
>>>>> the page lookup is the "slow" part; not because it's such a slow thing
>>>>> but because it's done 33 times per read-reference-invalidate (33 pages
>>>>> to look up per rgrp).
>>>>>
>>>>> Regards,
>>>>>
>>>>> Bob Peterson
>>>>> Red Hat File Systems
>>>> Thats true, however, as I understand the problem here, the issue is not
>>>> reading in the blocks for the rgrp that is eventually selected to use,
>>>> but the reading in of those blocks for the rgrps that we reject, for
>>>> whatever reason (full, or congested, or whatever). So with rgrplvb
>>>> enabled, we don't then read those rgrps in off disk at all in most cases
>>>> - so I was wondering whether that solves the problem without needing
>>>> this change?
> Actually, I believe the problem is reading in the blocks for the rgrps we
> use, not the ones we reject. In this case, I think the rejected rgrps are
> pretty minimal.
>
>>> The rgrplvb mount option only helps if the file system is using lock_dlm.
>>> For lock_nolock, it's still just as slow because lock_nolock has no
>>> knowledge
>>> of lvbs. Now, granted, that's an unusual case because GFS2 is normally used
>>> with lock_dlm.
>> That sounds like a bug... it should work in the same way, even with
>> lock_nolock.
> Perhaps it is a bug in the rgrplvb code. I'll investigate the possibility.
> Until I look into the matter, all I can tell you is that the lvb option doesn't
> come near the performance of this patch. Here are some example runs:
>
> Stock kernel with -r128:
>                kB  reclen    write
>           2097152      32   213428
>           2097152      64   199363
>           2097152     128   202046
>           2097152     256   212355
>           2097152     512   228691
>           2097152    1024   216815
>
> Stock kernel with -r2048:
>                kB  reclen    write
>           2097152      32   150471
>           2097152      64   166858
>           2097152     128   165517
>           2097152     256   168206
>           2097152     512   163427
>           2097152    1024   158296
>
> Stock kernel with -r2048 and -o rgrplvb:
>                kB  reclen    write
>           2097152      32   167268
>           2097152      64   165654
>           2097152     128   166783
>           2097152     256   164070
>           2097152     512   166561
>           2097152    1024   166933
>
> With my patch and -r2048:
>                kB  reclen    write
>           2097152      32   196209
>           2097152      64   224383
>           2097152     128   223108
>           2097152     256   228552
>           2097152     512   224295
>           2097152    1024   229110
>
> With my patch and -r2048 and -o rgrplvb:
>                kB  reclen    write
>           2097152      32   214281
>           2097152      64   227061
>           2097152     128   226949
>           2097152     256   229651
>           2097152     512   229196
>           2097152    1024   226651
>
> I'll see if I can track down why the rgrplvb option isn't performing as well.
> I suspect the matter goes back to my first comment above. Namely, that the
> slowdown goes back to the slowness of page cache lookup for the buffers of the
> rgrps we are using (not rejected ones).
I'm assuming that these figures are bandwidth rather than times, since 
that appears to show that the patch makes quite a large difference. 
However the reclen is rather small. In the 32 bytes case, thats 128 
writes for each new block thats being allocated, unless of course that 
is 32k?

Steve.

> Regards,
>
> Bob Peterson
> Red Hat File Systems