[Cluster-devel] gfs2 vmscan warnings (was: GFS2 Errors)

[FeldHost™ Admin]

Hello, I switched to Fedora 26 and see again that errors on kernel 4.12.x.

> On 4 Apr 2018, at 17:28, Andrew Price <anprice at redhat.com> wrote:
> 
> On 19/07/17 10:09, Steven Whitehouse wrote:>> On 2017-07-18 07:25 PM, Kristián Feldsam wrote:
>>>> Hello, I see today GFS2 errors in log and nothing about that is on net,
>>>> so I writing to this mailing list.
>>>> 
>>>> node2    19.07.2017 01:11:55    kernel    kern    err    vmscan: shrink_slab:
>>>> gfs2_glock_shrink_scan+0x0/0x2f0 [gfs2] negative objects to delete
>>>> nr=-4549568322848002755
>>>> node2    19.07.2017 01:10:56    kernel    kern    err    vmscan: shrink_slab:
>>>> gfs2_glock_shrink_scan+0x0/0x2f0 [gfs2] negative objects to delete
>>>> nr=-8191295421473926116
>>>> node2    19.07.2017 01:10:48    kernel    kern    err    vmscan: shrink_slab:
>>>> gfs2_glock_shrink_scan+0x0/0x2f0 [gfs2] negative objects to delete
>>>> nr=-8225402411152149004
>>>> node2    19.07.2017 01:10:47    kernel    kern    err    vmscan: shrink_slab:
>>>> gfs2_glock_shrink_scan+0x0/0x2f0 [gfs2] negative objects to delete
>>>> nr=-8230186816585019317
>>>> node2    19.07.2017 01:10:45    kernel    kern    err    vmscan: shrink_slab:
> <snip>
>> This looks like a bug to me, since the object count should never be negative. The glock shrinker is not (yet) zone aware, although the quota shrinker is. Not sure if that is related, but perhaps.... certainly something we'd like to investigate further. That said the messages in themselves are harmless, but it will likely indicate a less than optimal use of memory. If there are any details that can be shared about the use case, and how to reproduce that will be very helpful for us to know. Also what kernel version was this?
> 
> I can reproduce this (by running fsstress for a while, unmounting and running fsck.gfs2) with the latest mainline kernel. As far as I can tell, gfs2's lru_count is positive until unmount but then gets decremented a lot across more than one code path at unmount. It's decremented at:
> 
> gfs2_glock_remove_from_lru
>  clear_glock+0x50/0x60
>  glock_hash_walk+0xe1/0xf0
>  gfs2_gl_hash_clear+0x40/0x120
>  ? gfs2_jindex_free+0x106/0x140
>  gfs2_put_super+0x131/0x1d0
>  generic_shutdown_super+0x69/0x110
>  kill_block_super+0x21/0x50
>  deactivate_locked_super+0x39/0x70
>  cleanup_mnt+0x3b/0x70
>  task_work_run+0x82/0xb0
>  exit_to_usermode_loop+0x87/0x90
>  do_syscall_64+0x194/0x1a0
>  entry_SYSCALL_64_after_hwframe+0x42/0xb7
> 
> Workqueue: glock_workqueue glock_work_func
> gfs2_glock_remove_from_lru
>  __gfs2_glock_put+0x108/0x1c0
>  process_one_work+0x20c/0x660
>  worker_thread+0x3a/0x390
>  ? process_one_work+0x660/0x660
>  kthread+0x11c/0x140
>  ? kthread_delayed_work_timer_fn+0x90/0x90
>  ret_from_fork+0x24/0x30
> 
> gfs2_glock_remove_from_lru
>  __gfs2_glock_put+0x108/0x1c0
>  gfs2_evict_inode+0x2f3/0x650
>  ? find_held_lock+0x2d/0x90
>  ? evict+0xba/0x190
>  ? evict+0xcd/0x190
>  evict+0xcd/0x190
>  dispose_list+0x51/0x80
>  evict_inodes+0x1a5/0x1b0
>  generic_shutdown_super+0x3f/0x110
>  kill_block_super+0x21/0x50
>  deactivate_locked_super+0x39/0x70
>  cleanup_mnt+0x3b/0x70
>  task_work_run+0x82/0xb0
>  exit_to_usermode_loop+0x87/0x90
>  do_syscall_64+0x194/0x1a0
>  entry_SYSCALL_64_after_hwframe+0x42/0xb7
> 
> gfs2_scan_glock_lru
>  gfs2_glock_shrink_scan
>  shrink_slab.part.44+0x1c6/0x5e0
>  shrink_node+0x350/0x360
>  kswapd+0x2d9/0x8e0
>  ? mem_cgroup_shrink_node+0x310/0x310
>  kthread+0x11c/0x140
>  ? kthread_delayed_work_timer_fn+0x90/0x90
>  ret_from_fork+0x24/0x30
> 
> I tried a scripted bisect on it and it threw up 'mm: use sc->priority for slab shrink targets' (9092c71bb724dba2ecba849eae69e5c9d39bd3d2) but that's fairly recent and Kristián's report was in an older kernel so I'm not sure how reliable that is.
> 
> My expectations are uncalibrated wrt the lru list at unmount so I'm not seeing the root cause at the moment.
> 
> Andy