[Cluster-devel] [syzbot] KASAN: use-after-free Read in gfs2_glock_shrink_scan

Hillf Danton hdanton at sina.com
Tue May 18 08:46:25 UTC 2021 

On Mon, 17 May 2021 23:13:16 -0700
>Hello,
>
>syzbot found the following issue on:
>
>HEAD commit:    315d9931 Merge tag 'pm-5.13-rc2' of git://git.kernel.org/p..
>git tree:       upstream
>console output: https://syzkaller.appspot.com/x/log.txt?x=126d17b3d00000
>kernel config:  https://syzkaller.appspot.com/x/.config?x=4e950b1ffed48778
>dashboard link: https://syzkaller.appspot.com/bug?extid=34ba7ddbf3021981a228
>userspace arch: i386
>
>Unfortunately, I don't have any reproducer for this issue yet.
>
>IMPORTANT: if you fix the issue, please add the following tag to the commit:
>Reported-by: syzbot+34ba7ddbf3021981a228 at syzkaller.appspotmail.com
>
>==================================================================
>BUG: KASAN: use-after-free in __list_del_entry_valid+0xcc/0xf0 lib/list_debug.c:42
>Read of size 8 at addr ffff888074ee8f20 by task khugepaged/1669
>
>CPU: 0 PID: 1669 Comm: khugepaged Not tainted 5.13.0-rc1-syzkaller #0
>Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014
>Call Trace:
> __dump_stack lib/dump_stack.c:79 [inline]
> dump_stack+0x141/0x1d7 lib/dump_stack.c:120
> print_address_description.constprop.0.cold+0x5b/0x2f8 mm/kasan/report.c:233
> __kasan_report mm/kasan/report.c:419 [inline]
> kasan_report.cold+0x7c/0xd8 mm/kasan/report.c:436
> __list_del_entry_valid+0xcc/0xf0 lib/list_debug.c:42
> __list_del_entry include/linux/list.h:132 [inline]
> list_del_init include/linux/list.h:204 [inline]
> gfs2_dispose_glock_lru fs/gfs2/glock.c:1777 [inline]
> gfs2_scan_glock_lru fs/gfs2/glock.c:1832 [inline]
> gfs2_glock_shrink_scan fs/gfs2/glock.c:1843 [inline]
> gfs2_glock_shrink_scan+0x69f/0xa80 fs/gfs2/glock.c:1838
> do_shrink_slab+0x42d/0xbd0 mm/vmscan.c:709
> shrink_slab+0x17f/0x6f0 mm/vmscan.c:869
> shrink_node_memcgs mm/vmscan.c:2852 [inline]
> shrink_node+0x8d1/0x1de0 mm/vmscan.c:2967
> shrink_zones mm/vmscan.c:3170 [inline]
> do_try_to_free_pages+0x388/0x14b0 mm/vmscan.c:3225
> try_to_free_pages+0x29f/0x750 mm/vmscan.c:3464
> __perform_reclaim mm/page_alloc.c:4430 [inline]
> __alloc_pages_direct_reclaim mm/page_alloc.c:4451 [inline]
> __alloc_pages_slowpath.constprop.0+0x84e/0x2140 mm/page_alloc.c:4855
> __alloc_pages+0x422/0x500 mm/page_alloc.c:5213
> __alloc_pages_node include/linux/gfp.h:549 [inline]
> khugepaged_alloc_page+0xa0/0x170 mm/khugepaged.c:882
> collapse_huge_page mm/khugepaged.c:1085 [inline]
> khugepaged_scan_pmd mm/khugepaged.c:1368 [inline]
> khugepaged_scan_mm_slot mm/khugepaged.c:2137 [inline]
> khugepaged_do_scan mm/khugepaged.c:2218 [inline]
> khugepaged+0x312b/0x5530 mm/khugepaged.c:2263
> kthread+0x3b1/0x4a0 kernel/kthread.c:313
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
>
>Allocated by task 10231:
> kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
> kasan_set_track mm/kasan/common.c:46 [inline]
> set_alloc_info mm/kasan/common.c:428 [inline]
> __kasan_slab_alloc+0x84/0xa0 mm/kasan/common.c:461
> kasan_slab_alloc include/linux/kasan.h:236 [inline]
> slab_post_alloc_hook mm/slab.h:524 [inline]
> slab_alloc_node mm/slub.c:2912 [inline]
> slab_alloc mm/slub.c:2920 [inline]
> kmem_cache_alloc+0x152/0x3a0 mm/slub.c:2925
> gfs2_glock_get+0x20e/0x1100 fs/gfs2/glock.c:1027
> gfs2_inode_lookup+0x2c9/0xb10 fs/gfs2/inode.c:149
> gfs2_dir_search+0x20f/0x2c0 fs/gfs2/dir.c:1665
> gfs2_lookupi+0x475/0x640 fs/gfs2/inode.c:332
> gfs2_lookup_simple+0x99/0xe0 fs/gfs2/inode.c:273
> init_inodes+0x1c79/0x2610 fs/gfs2/ops_fstype.c:880
> gfs2_fill_super+0x1b4a/0x2680 fs/gfs2/ops_fstype.c:1204
> get_tree_bdev+0x440/0x760 fs/super.c:1293
> gfs2_get_tree+0x4a/0x270 fs/gfs2/ops_fstype.c:1273
> vfs_get_tree+0x89/0x2f0 fs/super.c:1498
> do_new_mount fs/namespace.c:2905 [inline]
> path_mount+0x132a/0x1fa0 fs/namespace.c:3235
> do_mount fs/namespace.c:3248 [inline]
> __do_sys_mount fs/namespace.c:3456 [inline]
> __se_sys_mount fs/namespace.c:3433 [inline]
> __ia32_sys_mount+0x27e/0x300 fs/namespace.c:3433
> do_syscall_32_irqs_on arch/x86/entry/common.c:78 [inline]
> __do_fast_syscall_32+0x67/0xe0 arch/x86/entry/common.c:143
> do_fast_syscall_32+0x2f/0x70 arch/x86/entry/common.c:168
> entry_SYSENTER_compat_after_hwframe+0x4d/0x5c
>
>Freed by task 8886:
> kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
> kasan_set_track+0x1c/0x30 mm/kasan/common.c:46
> kasan_set_free_info+0x20/0x30 mm/kasan/generic.c:357
> ____kasan_slab_free mm/kasan/common.c:360 [inline]
> ____kasan_slab_free mm/kasan/common.c:325 [inline]
> __kasan_slab_free+0xfb/0x130 mm/kasan/common.c:368
> kasan_slab_free include/linux/kasan.h:212 [inline]
> slab_free_hook mm/slub.c:1581 [inline]
> slab_free_freelist_hook+0xdf/0x240 mm/slub.c:1606
> slab_free mm/slub.c:3166 [inline]
> kmem_cache_free+0x8a/0x740 mm/slub.c:3182
> gfs2_glock_dealloc+0xcc/0x150 fs/gfs2/glock.c:130
> rcu_do_batch kernel/rcu/tree.c:2558 [inline]
> rcu_core+0x7ab/0x13b0 kernel/rcu/tree.c:2793
> __do_softirq+0x29b/0x9f6 kernel/softirq.c:559
>
>Last potentially related work creation:
> kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
> kasan_record_aux_stack+0xe5/0x110 mm/kasan/generic.c:345
> __call_rcu kernel/rcu/tree.c:3038 [inline]
> call_rcu+0xb1/0x750 kernel/rcu/tree.c:3113
> gfs2_glock_free+0x7cf/0x1080 fs/gfs2/glock.c:170
> glock_work_func+0x2bb/0x3f0 fs/gfs2/glock.c:949
> process_one_work+0x98d/0x1600 kernel/workqueue.c:2275
> worker_thread+0x64c/0x1120 kernel/workqueue.c:2421
> kthread+0x3b1/0x4a0 kernel/kthread.c:313
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
>
>Second to last potentially related work creation:
> kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
> kasan_record_aux_stack+0xe5/0x110 mm/kasan/generic.c:345
> insert_work+0x48/0x370 kernel/workqueue.c:1331
> __queue_work+0x5c1/0xed0 kernel/workqueue.c:1497
> __queue_delayed_work+0x1c8/0x270 kernel/workqueue.c:1644
> queue_delayed_work_on+0x105/0x120 kernel/workqueue.c:1680
> queue_delayed_work include/linux/workqueue.h:522 [inline]
> __gfs2_glock_queue_work+0x2a/0xb0 fs/gfs2/glock.c:245
> gfs2_glock_queue_work fs/gfs2/glock.c:259 [inline]
> do_xmote+0x833/0xbc0 fs/gfs2/glock.c:702
> run_queue+0x323/0x680 fs/gfs2/glock.c:766
> glock_work_func+0xff/0x3f0 fs/gfs2/glock.c:933
> process_one_work+0x98d/0x1600 kernel/workqueue.c:2275
> worker_thread+0x64c/0x1120 kernel/workqueue.c:2421
> kthread+0x3b1/0x4a0 kernel/kthread.c:313
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
>
>The buggy address belongs to the object at ffff888074ee8e10
> which belongs to the cache gfs2_glock(aspace) of size 1072
>The buggy address is located 272 bytes inside of
> 1072-byte region [ffff888074ee8e10, ffff888074ee9240)
>The buggy address belongs to the page:
>page:ffffea0001d3ba00 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x74ee8
>head:ffffea0001d3ba00 order:3 compound_mapcount:0 compound_pincount:0
>flags: 0x4fff00000010200(slab|head|node=1|zone=1|lastcpupid=0x7ff)
>raw: 04fff00000010200 dead000000000100 dead000000000122 ffff888043ead180
>raw: 0000000000000000 00000000801b001b 00000001ffffffff 0000000000000000
>page dumped because: kasan: bad access detected
>page_owner tracks the page as allocated
>page last allocated via order 3, migratetype Unmovable, gfp_mask 0xd2040(__GFP_IO|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 10231, ts 369366273371, free_ts 369360520013
> prep_new_page mm/page_alloc.c:2358 [inline]
> get_page_from_freelist+0x1033/0x2b60 mm/page_alloc.c:3994
> __alloc_pages_slowpath.constprop.0+0x2ef/0x2140 mm/page_alloc.c:4762
> __alloc_pages+0x422/0x500 mm/page_alloc.c:5213
> alloc_pages+0x18c/0x2a0 mm/mempolicy.c:2272
> alloc_slab_page mm/slub.c:1644 [inline]
> allocate_slab+0x2c5/0x4c0 mm/slub.c:1784
> new_slab mm/slub.c:1847 [inline]
> new_slab_objects mm/slub.c:2593 [inline]
> ___slab_alloc+0x44c/0x7a0 mm/slub.c:2756
> __slab_alloc.constprop.0+0xa7/0xf0 mm/slub.c:2796
> slab_alloc_node mm/slub.c:2878 [inline]
> slab_alloc mm/slub.c:2920 [inline]
> kmem_cache_alloc+0x34b/0x3a0 mm/slub.c:2925
> gfs2_glock_get+0x20e/0x1100 fs/gfs2/glock.c:1027
> gfs2_inode_lookup+0x2c9/0xb10 fs/gfs2/inode.c:149
> gfs2_dir_search+0x20f/0x2c0 fs/gfs2/dir.c:1665
> gfs2_lookupi+0x475/0x640 fs/gfs2/inode.c:332
> gfs2_lookup_simple+0x99/0xe0 fs/gfs2/inode.c:273
> init_statfs fs/gfs2/ops_fstype.c:649 [inline]
> init_journal fs/gfs2/ops_fstype.c:805 [inline]
> init_inodes+0xd8f/0x2610 fs/gfs2/ops_fstype.c:874
> gfs2_fill_super+0x1b4a/0x2680 fs/gfs2/ops_fstype.c:1204
> get_tree_bdev+0x440/0x760 fs/super.c:1293
>page last free stack trace:
> reset_page_owner include/linux/page_owner.h:24 [inline]
> free_pages_prepare mm/page_alloc.c:1298 [inline]
> __free_pages_ok+0x476/0xce0 mm/page_alloc.c:1572
> qlink_free mm/kasan/quarantine.c:146 [inline]
> qlist_free_all+0x5a/0xc0 mm/kasan/quarantine.c:165
> kasan_quarantine_reduce+0x180/0x200 mm/kasan/quarantine.c:272
> __kasan_slab_alloc+0x8e/0xa0 mm/kasan/common.c:438
> kasan_slab_alloc include/linux/kasan.h:236 [inline]
> slab_post_alloc_hook mm/slab.h:524 [inline]
> slab_alloc_node mm/slub.c:2912 [inline]
> slab_alloc mm/slub.c:2920 [inline]
> __kmalloc+0x1f7/0x330 mm/slub.c:4063
> kmalloc include/linux/slab.h:561 [inline]
> tomoyo_realpath_from_path+0xc3/0x620 security/tomoyo/realpath.c:254
> tomoyo_get_realpath security/tomoyo/file.c:151 [inline]
> tomoyo_path_number_perm+0x1d5/0x590 security/tomoyo/file.c:723
> security_path_chmod+0xe0/0x150 security/security.c:1205
> chmod_common+0x156/0x440 fs/open.c:580
> vfs_fchmod fs/open.c:601 [inline]
> __do_sys_fchmod fs/open.c:610 [inline]
> __se_sys_fchmod fs/open.c:604 [inline]
> __x64_sys_fchmod+0x10e/0x190 fs/open.c:604
> do_syscall_64+0x3a/0xb0 arch/x86/entry/common.c:47
> entry_SYSCALL_64_after_hwframe+0x44/0xae
>
>Memory state around the buggy address:
> ffff888074ee8e00: fc fc fa fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff888074ee8e80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>>ffff888074ee8f00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>                               ^
> ffff888074ee8f80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff888074ee9000: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>==================================================================

When put, glock is removed from lru by calling gfs2_glock_remove_from_lru()
in __gfs2_glock_put(), and check GLF_LRU under lru_lock.

On the shrink scan path, GLF_LRU is cleared under lru_lock but because of
cond_resched_lock(&lru_lock) in gfs2_dispose_glock_lru() progress on the put
side can be made without deleting glock from lru.

Keep GLF_LRU across the race window opened by cond_resched_lock(&lru_lock) to
ensure correct behavior on both sides - clrear GLF_LRU after list_del under
lru_lock.

+++ x/fs/gfs2/glock.c
@@ -1772,6 +1772,7 @@ __acquires(&lru_lock)
 	while(!list_empty(list)) {
 		gl = list_first_entry(list, struct gfs2_glock, gl_lru);
 		list_del_init(&gl->gl_lru);
+		clear_bit(GLF_LRU, &gl->gl_flags);
 		if (!spin_trylock(&gl->gl_lockref.lock)) {
 add_back_to_lru:
 			list_add(&gl->gl_lru, &lru_list);
@@ -1817,7 +1818,6 @@ static long gfs2_scan_glock_lru(int nr)
 		if (!test_bit(GLF_LOCK, &gl->gl_flags)) {
 			list_move(&gl->gl_lru, &dispose);
 			atomic_dec(&lru_count);
-			clear_bit(GLF_LRU, &gl->gl_flags);
 			freed++;
 			continue;
 		}

More information about the Cluster-devel mailing list