<div dir="ltr">Deleting from glusterfs or xfs direct is cost same time.<div>But glusterfs will stop all io request and hang all the cluster system. I do not what case this.</div><div>I think IO ops maybe different with local delete , glusterfs deleteing.</div></div> <div class="gmail_quote"><div dir="ltr">On Thu, Nov 1, 2018 at 12:33 PM Achilles Gaikwad <<a href="mailto:agaikwad@redhat.com">agaikwad@redhat.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Just my two cents. Can you also test if deleting a file from glusterfs takes time or deleting it from xfs takes time? Xfs is known to take a long time deleting files which are fragmented or have holes within the files (sparse files). It would be helpful if sanity check could be done just to make sure that the file removal from xfs is causing issues. </div> <div class="gmail_quote"><div dir="ltr">On Thu, 1 Nov, 2018, 2:23 AM Bryan Gurney <<a href="mailto:bgurney@redhat.com" target="_blank">bgurney@redhat.com</a> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I recommend remounting the XFS filesystem without the online discard option (i.e.: without "-o discard"), and scheduling fstrim runs on the mountpoint. On the first fstrim run, monitor the system to see if there's any more I/O stalls. Thanks, Bryan On Mon, Oct 29, 2018 at 10:52 PM, 劼磊周 <<a href="mailto:loli.miko@gmail.com" rel="noreferrer" target="_blank">loli.miko@gmail.com</a>> wrote: > 1. What is the version of the Linux kernel running on this system? > linux 4.14 kenel. two set have different version. > one is 4.14.32 > two is 4.14.68 > 2. What type of storage device is the VDO volume stored on? Is it > rotational (hard-disk drive), or nonrotational (flash storage, > solid-state drive)? Does it use RAID (either hardware or software > RAID)? > two set has different disk > one 1T *8 raid5 > two 8T * 10 raid6 and i add a bcache on it. (dell h730p raid card) > all above is in hardware raid. > > > 3. Can you run "dmsetup ls --tree -o ascii" and/or "lsblk -i" to show > the positioning of each element in the storage stack? > one is > dmsetup ls --tree -o ascii > sdc (253:3) > `- (8:32) > vg00-lv01 (253:1) > `- (8:2) > sdb (253:2) > `- (8:16) > vg00-lv00 (253:0) > `- (8:2) > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sdb 8:16 0 2.7T 0 disk > `-sdb 253:2 0 4T 0 vdo /exports/sdb > sdc 8:32 0 2.7T 0 disk > `-sdc 253:3 0 4T 0 vdo /exports/sdc > sda 8:0 0 102.4G 0 disk > |-sda2 8:2 0 101.9G 0 part > | |-vg00-lv01 253:1 0 15.6G 0 lvm [SWAP] > | `-vg00-lv00 253:0 0 86.2G 0 lvm / > `-sda1 8:1 0 512M 0 part /boot > > two is > dmsetup ls --tree -o ascii > image_store (253:3) > `- (252:0) > centos-home (253:2) > `- (8:34) > centos-swap (253:1) > `- (8:34) > centos-root (253:0) > `- (8:34) > > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sdb 8:16 0 372G 0 disk > `-bcache0 252:0 0 58.2T 0 disk > `-image_store 253:3 0 116T 0 vdo /exports/disk0 > sdc 8:32 0 931G 0 disk > |-sdc2 8:34 0 930G 0 part > | |-centos-swap 253:1 0 4G 0 lvm [SWAP] > | |-centos-home 253:2 0 876G 0 lvm /home > | `-centos-root 253:0 0 50G 0 lvm / > `-sdc1 8:33 0 1G 0 part /boot > sda 8:0 0 58.2T 0 disk > `-bcache0 252:0 0 58.2T 0 disk > `-image_store 253:3 0 116T 0 vdo /exports/disk0 > > > On Tue, Oct 30, 2018 at 2:08 AM Bryan Gurney <<a href="mailto:bgurney@redhat.com" rel="noreferrer" target="_blank">bgurney@redhat.com</a>> wrote: >> >> Thank you; these call traces are exactly what I was looking for. >> >> If possible, can you provide a few more items of information about this >> system? >> >> 1. What is the version of the Linux kernel running on this system? >> (i.e.: "uname -r") >> >> 2. What type of storage device is the VDO volume stored on? Is it >> rotational (hard-disk drive), or nonrotational (flash storage, >> solid-state drive)? Does it use RAID (either hardware or software >> RAID)? >> >> 3. Can you run "dmsetup ls --tree -o ascii" and/or "lsblk -i" to show >> the positioning of each element in the storage stack? >> >> >> Thanks, >> >> Bryan >> >> On Mon, Oct 29, 2018 at 2:51 AM, 劼磊周 <<a href="mailto:loli.miko@gmail.com" rel="noreferrer" target="_blank">loli.miko@gmail.com</a>> wrote: >> > hi bryan >> > Sorry for replay not in mail list.So in here.I to report again >> > For now the problem is where I delete big file in vdo with glusterfs all >> > io >> > request will be hang. >> > removing from local vdo mount. will not case this problem. >> > >> > this a two infomation from my two glusterfs. >> > I do not known why this will happen. >> > >> > Oct 29 06:42:40 image-store21003 kernel: sysrq: SysRq : Show Blocked >> > State >> > Oct 29 06:42:40 image-store21003 kernel: task PC >> > stack pid father >> > Oct 29 06:42:40 image-store21003 kernel: kworker/6:1H D 0 958 >> > 2 >> > 0x80000080 >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-log/dm-3 >> > xfs_buf_ioend_work [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace: >> > Oct 29 06:42:40 image-store21003 kernel: initializeCompletion+0x23/0x40 >> > [kvdo] >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: io_schedule+0x16/0x40 >> > Oct 29 06:42:40 image-store21003 kernel: limiterWaitForOneFree+0x84/0xe0 >> > [kvdo] >> > Oct 29 06:42:40 image-store21003 kernel: ? remove_wait_queue+0x60/0x60 >> > Oct 29 06:42:40 image-store21003 kernel: kvdoMapBio+0x164/0x270 [kvdo] >> > Oct 29 06:42:40 image-store21003 kernel: vdoMapBio+0x12/0x20 [kvdo] >> > Oct 29 06:42:40 image-store21003 kernel: __map_bio+0xc9/0x210 [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: ? dm_blk_close+0x70/0x70 >> > [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: >> > __send_changing_extent_only+0x115/0x159 [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > get_num_write_zeroes_bios+0x10/0x10 [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > dm_remap_zone_report+0x10/0x10 >> > [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: >> > __split_and_process_non_flush+0x240/0x270 [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: >> > __split_and_process_bio+0x11b/0x270 >> > [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: dm_make_request+0x7b/0xd0 >> > [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: >> > generic_make_request+0x110/0x2e0 >> > Oct 29 06:42:40 image-store21003 kernel: submit_bio+0x75/0x140 >> > Oct 29 06:42:40 image-store21003 kernel: next_bio+0x38/0x40 >> > Oct 29 06:42:40 image-store21003 kernel: >> > __blkdev_issue_discard+0x13b/0x1e0 >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_committed+0x1d7/0x340 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xlog_state_do_callback+0x165/0x2b0 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xlog_state_done_syncing+0x72/0x90 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xlog_iodone+0x7d/0xb0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend+0x9d/0x1e0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend_work+0x15/0x20 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380 >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0 >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140 >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0 >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40 >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild/dm-3 D 0 9867 >> > 2 >> > 0x80000080 >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace: >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870 >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x1b0/0x280 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: ? xfsaild+0x19a/0x780 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild+0x19a/0x780 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140 >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > xfs_trans_ail_cursor_first+0x90/0x90 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0 >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40 >> > Oct 29 06:42:40 image-store21003 kernel: glusteriotwr7 D 0 32733 >> > 1 >> > 0x00000080 >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace: >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870 >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320 >> > Oct 29 06:42:40 image-store21003 kernel: ? xlog_sync+0x2c9/0x3d0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: __down_common+0xfc/0x132 >> > Oct 29 06:42:40 image-store21003 kernel: ? _xfs_buf_find+0x2c1/0x510 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: __down+0x1d/0x1f >> > Oct 29 06:42:40 image-store21003 kernel: down+0x41/0x50 >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_lock+0x3c/0xf0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_buf_find+0x2c1/0x510 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_get_map+0x2a/0x280 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > xfs_free_ag_extent+0x3be/0x7a0 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_trans_get_buf_map+0x116/0x190 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_btree_get_bufs+0x5f/0x80 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_alloc_fix_freelist+0x1f8/0x520 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_free_extent_fix_freelist+0x75/0xc0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_free_extent+0x5c/0x150 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_trans_free_extent+0x4f/0x110 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_trans_add_item+0x5d/0x90 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_extent_free_finish_item+0x26/0x40 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_defer_finish+0x13f/0x410 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_itruncate_extents+0x127/0x2e0 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_inactive_truncate+0xad/0x100 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_inactive+0xec/0x110 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_fs_destroy_inode+0xa4/0x210 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: destroy_inode+0x3b/0x60 >> > Oct 29 06:42:40 image-store21003 kernel: evict+0x142/0x1a0 >> > Oct 29 06:42:40 image-store21003 kernel: iput+0x17d/0x1d0 >> > Oct 29 06:42:40 image-store21003 kernel: do_unlinkat+0x194/0x2b0 >> > Oct 29 06:42:40 image-store21003 kernel: SyS_unlink+0x16/0x20 >> > Oct 29 06:42:40 image-store21003 kernel: do_syscall_64+0x73/0x1b0 >> > Oct 29 06:42:40 image-store21003 kernel: >> > entry_SYSCALL_64_after_hwframe+0x3d/0xa2 >> > Oct 29 06:42:40 image-store21003 kernel: RIP: 0033:0x7fcbd64c9417 >> > Oct 29 06:42:40 image-store21003 kernel: RSP: 002b:00007fcb8c3b6208 >> > EFLAGS: >> > 00000246 ORIG_RAX: 0000000000000057 >> > Oct 29 06:42:40 image-store21003 kernel: RAX: ffffffffffffffda RBX: >> > 00007fcb8c3b6250 RCX: 00007fcbd64c9417 >> > Oct 29 06:42:40 image-store21003 kernel: RDX: 00007fcb8c3b6050 RSI: >> > 00007fcb8c3b6050 RDI: 00007fcb8c3b63d0 >> > Oct 29 06:42:40 image-store21003 kernel: RBP: 00007fcb7c047cf0 R08: >> > 65342d323362662d R09: 00007fcbd642516d >> > Oct 29 06:42:40 image-store21003 kernel: R10: 6532656634383938 R11: >> > 0000000000000246 R12: 00007fcb8c3b64c8 >> > Oct 29 06:42:40 image-store21003 kernel: R13: 00007fcbc4008610 R14: >> > 00007fcb8c3b63d0 R15: 0000000000000000 >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:3 D 0 2447 >> > 2 >> > 0x80000080 >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-sync/dm-3 >> > xfs_log_worker [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace: >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870 >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320 >> > Oct 29 06:42:40 image-store21003 kernel: wait_for_completion+0x121/0x180 >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: flush_work+0x11d/0x1a0 >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_worker+0x30/0x30 >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_force_lsn+0x8d/0x210 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > generic_make_request+0x110/0x2e0 >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x86/0x280 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > __bch_btree_node_write+0x340/0x660 [bcache] >> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_log_worker+0x36/0x100 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_worker+0x36/0x100 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380 >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0 >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140 >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0 >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40 >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:32 D 0 11430 >> > 2 >> > 0x80000080 >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-cil/dm-3 >> > xlog_cil_push_work [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace: >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870 >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: >> > xlog_state_get_iclog_space+0x105/0x300 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: xlog_write+0x19d/0x6c0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? kmem_zone_alloc+0xa4/0x110 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push+0x28c/0x460 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? __switch_to_asm+0x34/0x70 >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push_work+0x15/0x20 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380 >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x1d8/0x3e0 >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140 >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0 >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40 >> > >> > >> > >> > two is >> > >> > Oct 29 06:42:40 image-store21003 kernel: sysrq: SysRq : Show Blocked >> > State >> > Oct 29 06:42:40 image-store21003 kernel: task PC >> > stack pid father >> > Oct 29 06:42:40 image-store21003 kernel: kworker/6:1H D 0 958 >> > 2 >> > 0x80000080 >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-log/dm-3 >> > xfs_buf_ioend_work [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace: >> > Oct 29 06:42:40 image-store21003 kernel: initializeCompletion+0x23/0x40 >> > [kvdo] >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: io_schedule+0x16/0x40 >> > Oct 29 06:42:40 image-store21003 kernel: limiterWaitForOneFree+0x84/0xe0 >> > [kvdo] >> > Oct 29 06:42:40 image-store21003 kernel: ? remove_wait_queue+0x60/0x60 >> > Oct 29 06:42:40 image-store21003 kernel: kvdoMapBio+0x164/0x270 [kvdo] >> > Oct 29 06:42:40 image-store21003 kernel: vdoMapBio+0x12/0x20 [kvdo] >> > Oct 29 06:42:40 image-store21003 kernel: __map_bio+0xc9/0x210 [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: ? dm_blk_close+0x70/0x70 >> > [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: >> > __send_changing_extent_only+0x115/0x159 [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > get_num_write_zeroes_bios+0x10/0x10 [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > dm_remap_zone_report+0x10/0x10 >> > [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: >> > __split_and_process_non_flush+0x240/0x270 [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: >> > __split_and_process_bio+0x11b/0x270 >> > [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: dm_make_request+0x7b/0xd0 >> > [dm_mod] >> > Oct 29 06:42:40 image-store21003 kernel: >> > generic_make_request+0x110/0x2e0 >> > Oct 29 06:42:40 image-store21003 kernel: submit_bio+0x75/0x140 >> > Oct 29 06:42:40 image-store21003 kernel: next_bio+0x38/0x40 >> > Oct 29 06:42:40 image-store21003 kernel: >> > __blkdev_issue_discard+0x13b/0x1e0 >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_committed+0x1d7/0x340 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xlog_state_do_callback+0x165/0x2b0 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xlog_state_done_syncing+0x72/0x90 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xlog_iodone+0x7d/0xb0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend+0x9d/0x1e0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend_work+0x15/0x20 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380 >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0 >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140 >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0 >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40 >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild/dm-3 D 0 9867 >> > 2 >> > 0x80000080 >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace: >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870 >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x1b0/0x280 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: ? xfsaild+0x19a/0x780 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild+0x19a/0x780 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140 >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > xfs_trans_ail_cursor_first+0x90/0x90 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0 >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40 >> > Oct 29 06:42:40 image-store21003 kernel: glusteriotwr7 D 0 32733 >> > 1 >> > 0x00000080 >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace: >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870 >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320 >> > Oct 29 06:42:40 image-store21003 kernel: ? xlog_sync+0x2c9/0x3d0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: __down_common+0xfc/0x132 >> > Oct 29 06:42:40 image-store21003 kernel: ? _xfs_buf_find+0x2c1/0x510 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: __down+0x1d/0x1f >> > Oct 29 06:42:40 image-store21003 kernel: down+0x41/0x50 >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_lock+0x3c/0xf0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_buf_find+0x2c1/0x510 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_get_map+0x2a/0x280 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > xfs_free_ag_extent+0x3be/0x7a0 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_trans_get_buf_map+0x116/0x190 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_btree_get_bufs+0x5f/0x80 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_alloc_fix_freelist+0x1f8/0x520 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_free_extent_fix_freelist+0x75/0xc0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_free_extent+0x5c/0x150 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_trans_free_extent+0x4f/0x110 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_trans_add_item+0x5d/0x90 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_extent_free_finish_item+0x26/0x40 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_defer_finish+0x13f/0x410 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_itruncate_extents+0x127/0x2e0 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: >> > xfs_inactive_truncate+0xad/0x100 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_inactive+0xec/0x110 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_fs_destroy_inode+0xa4/0x210 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: destroy_inode+0x3b/0x60 >> > Oct 29 06:42:40 image-store21003 kernel: evict+0x142/0x1a0 >> > Oct 29 06:42:40 image-store21003 kernel: iput+0x17d/0x1d0 >> > Oct 29 06:42:40 image-store21003 kernel: do_unlinkat+0x194/0x2b0 >> > Oct 29 06:42:40 image-store21003 kernel: SyS_unlink+0x16/0x20 >> > Oct 29 06:42:40 image-store21003 kernel: do_syscall_64+0x73/0x1b0 >> > Oct 29 06:42:40 image-store21003 kernel: >> > entry_SYSCALL_64_after_hwframe+0x3d/0xa2 >> > Oct 29 06:42:40 image-store21003 kernel: RIP: 0033:0x7fcbd64c9417 >> > Oct 29 06:42:40 image-store21003 kernel: RSP: 002b:00007fcb8c3b6208 >> > EFLAGS: >> > 00000246 ORIG_RAX: 0000000000000057 >> > Oct 29 06:42:40 image-store21003 kernel: RAX: ffffffffffffffda RBX: >> > 00007fcb8c3b6250 RCX: 00007fcbd64c9417 >> > Oct 29 06:42:40 image-store21003 kernel: RDX: 00007fcb8c3b6050 RSI: >> > 00007fcb8c3b6050 RDI: 00007fcb8c3b63d0 >> > Oct 29 06:42:40 image-store21003 kernel: RBP: 00007fcb7c047cf0 R08: >> > 65342d323362662d R09: 00007fcbd642516d >> > Oct 29 06:42:40 image-store21003 kernel: R10: 6532656634383938 R11: >> > 0000000000000246 R12: 00007fcb8c3b64c8 >> > Oct 29 06:42:40 image-store21003 kernel: R13: 00007fcbc4008610 R14: >> > 00007fcb8c3b63d0 R15: 0000000000000000 >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:3 D 0 2447 >> > 2 >> > 0x80000080 >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-sync/dm-3 >> > xfs_log_worker [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace: >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870 >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320 >> > Oct 29 06:42:40 image-store21003 kernel: wait_for_completion+0x121/0x180 >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: flush_work+0x11d/0x1a0 >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_worker+0x30/0x30 >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_force_lsn+0x8d/0x210 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > generic_make_request+0x110/0x2e0 >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x86/0x280 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? >> > __bch_btree_node_write+0x340/0x660 [bcache] >> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_log_worker+0x36/0x100 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_worker+0x36/0x100 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380 >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0 >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140 >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0 >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40 >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:32 D 0 11430 >> > 2 >> > 0x80000080 >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-cil/dm-3 >> > xlog_cil_push_work [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace: >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870 >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: >> > xlog_state_get_iclog_space+0x105/0x300 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: xlog_write+0x19d/0x6c0 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? kmem_zone_alloc+0xa4/0x110 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push+0x28c/0x460 [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: ? __switch_to_asm+0x34/0x70 >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push_work+0x15/0x20 >> > [xfs] >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380 >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x1d8/0x3e0 >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140 >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80 >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0 >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40 >> > >> > >> > >> > On Sat, Oct 27, 2018 at 3:01 AM Bryan Gurney <<a href="mailto:bgurney@redhat.com" rel="noreferrer" target="_blank">bgurney@redhat.com</a>> wrote: >> >> >> >> >> >> >> >> On Fri, Oct 26, 2018 at 12:03 AM, 劼磊周 <<a href="mailto:loli.miko@gmail.com" rel="noreferrer" target="_blank">loli.miko@gmail.com</a>> wrote: >> >>> >> >>> >> >>> >> >>> I have a big problem >> >>> >> >>> I use glusterfs over vdo on xfs in case. >> >>> my vdo version >> >>> Kernel module: >> >>> Loaded: true >> >>> Name: kvdo >> >>> Version information: >> >>> kvdo version: 6.2.0.219 >> >>> >> >>> In my case We have many big file over than 500G some is 1T or 2T. >> >>> if it be deleteing. Vdo will hang all of io reqeust. >> >>> Some small file like 20g or 50g. if size over 100G. this problem will >> >>> be >> >>> very frequency. >> >>> >> >>> this is come information on my system.I use iostat to record >> >>> >> >>> your will see dev utils is 100% and it will hang all io request >> >>> >> >>> iostats: dm-3 is vdo device >> >>> >> >>> avg-cpu: %user %nice %system %iowait %steal %idle >> >>> 0.38 0.00 28.95 0.64 0.00 70.03 >> >>> >> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util >> >>> sdb 0.00 218.00 0.00 15055.00 0.00 61152.00 >> >>> 8.12 0.54 0.04 0.00 0.04 0.03 49.00 >> >>> sda 0.00 205.00 1.00 16104.00 8.00 65908.00 >> >>> 8.19 0.78 0.05 0.00 0.05 0.04 70.00 >> >>> sdc 0.00 4.00 0.00 125.00 0.00 644.00 >> >>> 10.30 0.06 0.50 0.00 0.50 0.01 0.10 >> >>> dm-0 0.00 0.00 0.00 129.00 0.00 644.00 >> >>> 9.98 0.06 0.50 0.00 0.50 0.01 0.10 >> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> bcache0 0.00 0.00 0.00 15980.00 0.00 61144.00 >> >>> 7.65 0.87 0.05 0.00 0.05 0.05 75.40 >> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00 >> >>> >> >>> avg-cpu: %user %nice %system %iowait %steal %idle >> >>> 0.51 0.00 29.62 0.76 0.00 69.11 >> >>> >> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util >> >>> sdb 0.00 193.00 0.00 15245.00 0.00 61764.00 >> >>> 8.10 0.53 0.03 0.00 0.03 0.03 49.90 >> >>> sda 0.00 192.00 1.00 16268.00 4.00 66512.00 >> >>> 8.18 0.77 0.05 0.00 0.05 0.04 69.60 >> >>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> bcache0 0.00 0.00 0.00 16137.00 0.00 61756.00 >> >>> 7.65 0.88 0.05 0.00 0.05 0.05 74.60 >> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00 >> >>> >> >>> avg-cpu: %user %nice %system %iowait %steal %idle >> >>> 0.39 0.00 28.35 0.64 0.00 70.62 >> >>> >> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util >> >>> sdb 0.00 143.00 0.00 15014.00 0.00 60648.00 >> >>> 8.08 0.54 0.04 0.00 0.04 0.03 50.20 >> >>> sda 0.00 142.00 1.00 15996.00 4.00 65224.00 >> >>> 8.16 0.74 0.05 0.00 0.05 0.04 67.30 >> >>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> bcache0 0.00 0.00 0.00 15830.00 0.00 60648.00 >> >>> 7.66 0.87 0.05 0.00 0.05 0.05 73.30 >> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 >> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00 >> >>> >> >>> >> >>> top info >> >>> >> >>> Tasks: 1929 total, 2 running, 140 sleeping, 0 stopped, 0 zombie >> >>> %Cpu(s): 2.8 us, 17.0 sy, 0.0 ni, 71.7 id, 8.0 wa, 0.0 hi, 0.4 >> >>> si, >> >>> 0.0 st >> >>> KiB Mem : 13158505+total, 7327428 free, 10939488+used, 14862756 >> >>> buff/cache >> >>> KiB Swap: 4194300 total, 4194300 free, 0 used. 8015328 avail >> >>> Mem >> >>> >> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> >>> COMMAND >> >>> 2144 root 20 0 0 0 0 S 19.8 0.0 3767:26 >> >>> kvdo0:journalQ >> >>> 2150 root 20 0 0 0 0 S 15.8 0.0 4701:00 >> >>> kvdo0:physQ0 >> >>> 18560 root 20 0 2429296 40120 9004 S 12.2 0.0 1585:55 >> >>> glusterfsd >> >>> 2161 root 20 0 0 0 0 R 11.2 0.0 3579:51 >> >>> kvdo0:cpuQ0 >> >>> 2162 root 20 0 0 0 0 S 11.2 0.0 3572:51 >> >>> kvdo0:cpuQ1 >> >>> 17807 root 0 -20 0 0 0 D 10.2 0.0 109:02.36 >> >>> kworker/5:0H >> >>> 2146 root 20 0 0 0 0 S 7.6 0.0 1678:45 >> >>> kvdo0:logQ0 >> >>> 2147 root 20 0 0 0 0 S 7.3 0.0 1674:34 >> >>> kvdo0:logQ1 >> >>> 2148 root 20 0 0 0 0 S 7.3 0.0 1674:09 >> >>> kvdo0:logQ2 >> >>> 2149 root 20 0 0 0 0 S 7.3 0.0 1672:51 >> >>> kvdo0:logQ3 >> >>> 18567 root 20 0 2369988 31936 9068 S 5.9 0.0 483:05.83 >> >>> glusterfsd >> >>> 2145 root 20 0 0 0 0 S 4.0 0.0 1572:49 >> >>> kvdo0:packerQ >> >>> 2151 root 20 0 0 0 0 S 4.0 0.0 1446:38 >> >>> kvdo0:hashQ0 >> >>> 2152 root 20 0 0 0 0 S 4.0 0.0 1442:42 >> >>> kvdo0:hashQ1 >> >>> 2156 root 20 0 0 0 0 S 2.6 0.0 798:50.26 >> >>> kvdo0:bioQ0 >> >>> 2157 root 20 0 0 0 0 S 2.6 0.0 779:48.42 >> >>> kvdo0:bioQ1 >> >>> 2158 root 20 0 0 0 0 S 2.6 0.0 778:43.52 >> >>> kvdo0:bioQ2 >> >>> 2159 root 20 0 0 0 0 S 2.6 0.0 776:37.81 >> >>> kvdo0:bioQ3 >> >>> 2160 root 20 0 0 0 0 S 2.6 0.0 974:01.15 >> >>> kvdo0:ackQ >> >>> >> >>> >> >>> -- >> >>> zhou-jielei >> >>> >> >>> >> >> >> >> Hello, >> >> >> >> In your "top" output, there is a process "kworker/5:0H" with a status >> >> of >> >> "D" (uninterruptible sleep). If you send "w" to /proc/sysrq-trigger, >> >> it >> >> will show the "SysRq : Show Blocked State" output to print a call trace >> >> on >> >> the blocked task. >> >> >> >> (There may already be a call trace in a "task ... blocked for more than >> >> ... seconds" message in the kernel messages for this system.) >> >> >> >> However, as noted in your github issue, you set the "discards_limit" >> >> for >> >> this VDO volume (in the sysfs setting >> >> /sys/kvdo/<vdoname>/discards_limit) >> >> from the default of 1500 to 100. While this will allow for more I/O >> >> operations for non-discard operations, it will result in slower discard >> >> performance. >> >> >> >> You can adjust the "discards_limit" setting while the VDO volume is >> >> running. If you set it back to 1500, does the discard performance >> >> improve? >> >> >> >> Additionally, I posted a vdo-devel list email on different strategies >> >> for >> >> discards [1]. Depending on the level of filesystem activity (and the >> >> size >> >> and/or frequency of the files being discarded), it may be better to use >> >> a >> >> "scheduled fstrim" approach, instead of "mount -o discard". >> >> >> >> >> >> Thanks, >> >> >> >> Bryan Gurney >> >> Senior Software Engineer, VDO >> >> Red Hat >> >> >> >> [1] >> >> <a href="https://www.redhat.com/archives/vdo-devel/2018-February/msg00009.html" rel="noreferrer noreferrer" target="_blank">https://www.redhat.com/archives/vdo-devel/2018-February/msg00009.html</a> >> > >> > >> > >> > -- >> > 周劼磊 >> > > > > > -- > 周劼磊 > _______________________________________________ Vdo-devel mailing list <a href="mailto:Vdo-devel@redhat.com" rel="noreferrer" target="_blank">Vdo-devel@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/vdo-devel" rel="noreferrer noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vdo-devel</a> </blockquote></div> </blockquote></div> <div> </div>-- <div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"> 周劼磊 </div>