[Cluster-devel] [GFS2 PATCH] GFS2: Separate lru lists by glock type, use with glops

[Andreas Gruenbacher]

Hi Bob,

On Fri, Jul 21, 2017 at 5:19 PM, Bob Peterson <rpeterso at redhat.com> wrote:
> Hi,
>
> On 03 July I posted a patch called:
>
> "GFS2: Convert LRU list to use a hash table to reduce contention"
>
> Steve Whitehouse didn't like it for several reasons, one of them
> being that the information was no longer age-ordered. Point taken.
>
> The following is a new patch I wrote as a potential replacement.
>
> The idea here is simply to reduce LRU contention by breaking
> the lru_list into different lists, by glock type. All lru lists
> are kept in order within the glock type. Thus, all inode glocks
> are ordered properly.
>
> The logic to manage the LRU lists is basically the same, except:
>
> 1. There are now multiple lists
> 2. The lists are managed by functions hooked into the glock ops
>    (glops).
> 3. The GLOF_LRU bit is no longer needed, since we can now simply
>    check whether the glops has the appropriate lru list function.
> 4. In function gfs2_clear_rgrpd, when the rgrps are being
>    dismantled for unmount, it was calling gfs2_glock_add_to_lru()
>    for the rgrp glocks. This is incorrect, since rgrps are never
>    added to the lru lists (They didn't even have the GLOF_LRU bit).
>    In the pre-patch kernel, it was treated as a no-op, but calling
>    the new gl_op is an error. So this line was removed.
> 5. Another notable exception: In gfs2_evict_inode, if this is a
>    "final" glock reference "put" there's no point in putting the
>    glock onto the lru list only to remove it again a few
>    microseconds later. This is a new optimization.
>
> I asked one of our performance testing guys to see whether this
> patch speeds things up. As the attached graph shows, yes it does.
> The specsfs test runs anywhere from 9 to 20 percent faster than
> the same (pre-release rhel7.4) kernel without the patch.
>
> So given that the LRU logic is preserved, this is well worth doing.

this still doesn't address why there is so much contention on the
glock LRU spinlock(s). If you compare the glock LRU list with the
inode LRU list, you'll notice that inodes are not constantly added and
removed from their LRU list and that they can remain in the same list
position for a long time. Have a look at inode_lru_isolate and
__list_lru_walk_one to see how inodes age before being reclaimed
depending on whether they are in use and on the I_REFERENCED flag.

The constant adding and removing of glocks on their LRU list should
explain a lot of the contention you're seeing. Has the list_lru
approach been tried instead? I have a strong suspicion that it will be
just good enough.

Thanks,
Andreas