[Vdo-devel] Big problem on deleting big file on vdo. It will stop all io request
Bryan Gurney
bgurney at redhat.com
Mon Oct 29 17:08:20 UTC 2018
Thank you; these call traces are exactly what I was looking for.
If possible, can you provide a few more items of information about this system?
1. What is the version of the Linux kernel running on this system?
(i.e.: "uname -r")
2. What type of storage device is the VDO volume stored on? Is it
rotational (hard-disk drive), or nonrotational (flash storage,
solid-state drive)? Does it use RAID (either hardware or software
RAID)?
3. Can you run "dmsetup ls --tree -o ascii" and/or "lsblk -i" to show
the positioning of each element in the storage stack?
Thanks,
Bryan
On Mon, Oct 29, 2018 at 2:51 AM, 劼磊周 <loli.miko at gmail.com> wrote:
> hi bryan
> Sorry for replay not in mail list.So in here.I to report again
> For now the problem is where I delete big file in vdo with glusterfs all io
> request will be hang.
> removing from local vdo mount. will not case this problem.
>
> this a two infomation from my two glusterfs.
> I do not known why this will happen.
>
> Oct 29 06:42:40 image-store21003 kernel: sysrq: SysRq : Show Blocked State
> Oct 29 06:42:40 image-store21003 kernel: task PC
> stack pid father
> Oct 29 06:42:40 image-store21003 kernel: kworker/6:1H D 0 958 2
> 0x80000080
> Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-log/dm-3
> xfs_buf_ioend_work [xfs]
> Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> Oct 29 06:42:40 image-store21003 kernel: initializeCompletion+0x23/0x40
> [kvdo]
> Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> Oct 29 06:42:40 image-store21003 kernel: io_schedule+0x16/0x40
> Oct 29 06:42:40 image-store21003 kernel: limiterWaitForOneFree+0x84/0xe0
> [kvdo]
> Oct 29 06:42:40 image-store21003 kernel: ? remove_wait_queue+0x60/0x60
> Oct 29 06:42:40 image-store21003 kernel: kvdoMapBio+0x164/0x270 [kvdo]
> Oct 29 06:42:40 image-store21003 kernel: vdoMapBio+0x12/0x20 [kvdo]
> Oct 29 06:42:40 image-store21003 kernel: __map_bio+0xc9/0x210 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: ? dm_blk_close+0x70/0x70 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel:
> __send_changing_extent_only+0x115/0x159 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: ?
> get_num_write_zeroes_bios+0x10/0x10 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: ? dm_remap_zone_report+0x10/0x10
> [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel:
> __split_and_process_non_flush+0x240/0x270 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: __split_and_process_bio+0x11b/0x270
> [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: dm_make_request+0x7b/0xd0 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: generic_make_request+0x110/0x2e0
> Oct 29 06:42:40 image-store21003 kernel: submit_bio+0x75/0x140
> Oct 29 06:42:40 image-store21003 kernel: next_bio+0x38/0x40
> Oct 29 06:42:40 image-store21003 kernel: __blkdev_issue_discard+0x13b/0x1e0
> Oct 29 06:42:40 image-store21003 kernel: xlog_cil_committed+0x1d7/0x340
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xlog_state_do_callback+0x165/0x2b0
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xlog_state_done_syncing+0x72/0x90
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xlog_iodone+0x7d/0xb0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend+0x9d/0x1e0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend_work+0x15/0x20 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> Oct 29 06:42:40 image-store21003 kernel: xfsaild/dm-3 D 0 9867 2
> 0x80000080
> Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x1b0/0x280 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: ? xfsaild+0x19a/0x780 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfsaild+0x19a/0x780 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> Oct 29 06:42:40 image-store21003 kernel: ?
> xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> Oct 29 06:42:40 image-store21003 kernel: glusteriotwr7 D 0 32733 1
> 0x00000080
> Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> Oct 29 06:42:40 image-store21003 kernel: ? xlog_sync+0x2c9/0x3d0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: __down_common+0xfc/0x132
> Oct 29 06:42:40 image-store21003 kernel: ? _xfs_buf_find+0x2c1/0x510 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: __down+0x1d/0x1f
> Oct 29 06:42:40 image-store21003 kernel: down+0x41/0x50
> Oct 29 06:42:40 image-store21003 kernel: xfs_buf_lock+0x3c/0xf0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: _xfs_buf_find+0x2c1/0x510 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_buf_get_map+0x2a/0x280 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? xfs_free_ag_extent+0x3be/0x7a0
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_trans_get_buf_map+0x116/0x190
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_btree_get_bufs+0x5f/0x80 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_alloc_fix_freelist+0x1f8/0x520
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel:
> xfs_free_extent_fix_freelist+0x75/0xc0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_free_extent+0x5c/0x150 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_trans_free_extent+0x4f/0x110
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? xfs_trans_add_item+0x5d/0x90
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel:
> xfs_extent_free_finish_item+0x26/0x40 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_defer_finish+0x13f/0x410 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_itruncate_extents+0x127/0x2e0
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_inactive_truncate+0xad/0x100
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_inactive+0xec/0x110 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_fs_destroy_inode+0xa4/0x210
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: destroy_inode+0x3b/0x60
> Oct 29 06:42:40 image-store21003 kernel: evict+0x142/0x1a0
> Oct 29 06:42:40 image-store21003 kernel: iput+0x17d/0x1d0
> Oct 29 06:42:40 image-store21003 kernel: do_unlinkat+0x194/0x2b0
> Oct 29 06:42:40 image-store21003 kernel: SyS_unlink+0x16/0x20
> Oct 29 06:42:40 image-store21003 kernel: do_syscall_64+0x73/0x1b0
> Oct 29 06:42:40 image-store21003 kernel:
> entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> Oct 29 06:42:40 image-store21003 kernel: RIP: 0033:0x7fcbd64c9417
> Oct 29 06:42:40 image-store21003 kernel: RSP: 002b:00007fcb8c3b6208 EFLAGS:
> 00000246 ORIG_RAX: 0000000000000057
> Oct 29 06:42:40 image-store21003 kernel: RAX: ffffffffffffffda RBX:
> 00007fcb8c3b6250 RCX: 00007fcbd64c9417
> Oct 29 06:42:40 image-store21003 kernel: RDX: 00007fcb8c3b6050 RSI:
> 00007fcb8c3b6050 RDI: 00007fcb8c3b63d0
> Oct 29 06:42:40 image-store21003 kernel: RBP: 00007fcb7c047cf0 R08:
> 65342d323362662d R09: 00007fcbd642516d
> Oct 29 06:42:40 image-store21003 kernel: R10: 6532656634383938 R11:
> 0000000000000246 R12: 00007fcb8c3b64c8
> Oct 29 06:42:40 image-store21003 kernel: R13: 00007fcbc4008610 R14:
> 00007fcb8c3b63d0 R15: 0000000000000000
> Oct 29 06:42:40 image-store21003 kernel: kworker/2:3 D 0 2447 2
> 0x80000080
> Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-sync/dm-3
> xfs_log_worker [xfs]
> Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> Oct 29 06:42:40 image-store21003 kernel: wait_for_completion+0x121/0x180
> Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: flush_work+0x11d/0x1a0
> Oct 29 06:42:40 image-store21003 kernel: ? wake_up_worker+0x30/0x30
> Oct 29 06:42:40 image-store21003 kernel: xlog_cil_force_lsn+0x8d/0x210 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? generic_make_request+0x110/0x2e0
> Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x86/0x280 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ?
> __bch_btree_node_write+0x340/0x660 [bcache]
> Oct 29 06:42:40 image-store21003 kernel: ? xfs_log_worker+0x36/0x100 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_log_worker+0x36/0x100 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> Oct 29 06:42:40 image-store21003 kernel: kworker/2:32 D 0 11430 2
> 0x80000080
> Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-cil/dm-3
> xlog_cil_push_work [xfs]
> Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> Oct 29 06:42:40 image-store21003 kernel:
> xlog_state_get_iclog_space+0x105/0x300 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: xlog_write+0x19d/0x6c0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? kmem_zone_alloc+0xa4/0x110 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push+0x28c/0x460 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? __switch_to_asm+0x34/0x70
> Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push_work+0x15/0x20 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x1d8/0x3e0
> Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
>
>
>
> two is
>
> Oct 29 06:42:40 image-store21003 kernel: sysrq: SysRq : Show Blocked State
> Oct 29 06:42:40 image-store21003 kernel: task PC
> stack pid father
> Oct 29 06:42:40 image-store21003 kernel: kworker/6:1H D 0 958 2
> 0x80000080
> Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-log/dm-3
> xfs_buf_ioend_work [xfs]
> Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> Oct 29 06:42:40 image-store21003 kernel: initializeCompletion+0x23/0x40
> [kvdo]
> Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> Oct 29 06:42:40 image-store21003 kernel: io_schedule+0x16/0x40
> Oct 29 06:42:40 image-store21003 kernel: limiterWaitForOneFree+0x84/0xe0
> [kvdo]
> Oct 29 06:42:40 image-store21003 kernel: ? remove_wait_queue+0x60/0x60
> Oct 29 06:42:40 image-store21003 kernel: kvdoMapBio+0x164/0x270 [kvdo]
> Oct 29 06:42:40 image-store21003 kernel: vdoMapBio+0x12/0x20 [kvdo]
> Oct 29 06:42:40 image-store21003 kernel: __map_bio+0xc9/0x210 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: ? dm_blk_close+0x70/0x70 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel:
> __send_changing_extent_only+0x115/0x159 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: ?
> get_num_write_zeroes_bios+0x10/0x10 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: ? dm_remap_zone_report+0x10/0x10
> [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel:
> __split_and_process_non_flush+0x240/0x270 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: __split_and_process_bio+0x11b/0x270
> [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: dm_make_request+0x7b/0xd0 [dm_mod]
> Oct 29 06:42:40 image-store21003 kernel: generic_make_request+0x110/0x2e0
> Oct 29 06:42:40 image-store21003 kernel: submit_bio+0x75/0x140
> Oct 29 06:42:40 image-store21003 kernel: next_bio+0x38/0x40
> Oct 29 06:42:40 image-store21003 kernel: __blkdev_issue_discard+0x13b/0x1e0
> Oct 29 06:42:40 image-store21003 kernel: xlog_cil_committed+0x1d7/0x340
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xlog_state_do_callback+0x165/0x2b0
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xlog_state_done_syncing+0x72/0x90
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xlog_iodone+0x7d/0xb0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend+0x9d/0x1e0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend_work+0x15/0x20 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> Oct 29 06:42:40 image-store21003 kernel: xfsaild/dm-3 D 0 9867 2
> 0x80000080
> Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x1b0/0x280 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: ? xfsaild+0x19a/0x780 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfsaild+0x19a/0x780 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> Oct 29 06:42:40 image-store21003 kernel: ?
> xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> Oct 29 06:42:40 image-store21003 kernel: glusteriotwr7 D 0 32733 1
> 0x00000080
> Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> Oct 29 06:42:40 image-store21003 kernel: ? xlog_sync+0x2c9/0x3d0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: __down_common+0xfc/0x132
> Oct 29 06:42:40 image-store21003 kernel: ? _xfs_buf_find+0x2c1/0x510 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: __down+0x1d/0x1f
> Oct 29 06:42:40 image-store21003 kernel: down+0x41/0x50
> Oct 29 06:42:40 image-store21003 kernel: xfs_buf_lock+0x3c/0xf0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: _xfs_buf_find+0x2c1/0x510 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_buf_get_map+0x2a/0x280 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? xfs_free_ag_extent+0x3be/0x7a0
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_trans_get_buf_map+0x116/0x190
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_btree_get_bufs+0x5f/0x80 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_alloc_fix_freelist+0x1f8/0x520
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel:
> xfs_free_extent_fix_freelist+0x75/0xc0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_free_extent+0x5c/0x150 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_trans_free_extent+0x4f/0x110
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? xfs_trans_add_item+0x5d/0x90
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel:
> xfs_extent_free_finish_item+0x26/0x40 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_defer_finish+0x13f/0x410 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_itruncate_extents+0x127/0x2e0
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_inactive_truncate+0xad/0x100
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_inactive+0xec/0x110 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_fs_destroy_inode+0xa4/0x210
> [xfs]
> Oct 29 06:42:40 image-store21003 kernel: destroy_inode+0x3b/0x60
> Oct 29 06:42:40 image-store21003 kernel: evict+0x142/0x1a0
> Oct 29 06:42:40 image-store21003 kernel: iput+0x17d/0x1d0
> Oct 29 06:42:40 image-store21003 kernel: do_unlinkat+0x194/0x2b0
> Oct 29 06:42:40 image-store21003 kernel: SyS_unlink+0x16/0x20
> Oct 29 06:42:40 image-store21003 kernel: do_syscall_64+0x73/0x1b0
> Oct 29 06:42:40 image-store21003 kernel:
> entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> Oct 29 06:42:40 image-store21003 kernel: RIP: 0033:0x7fcbd64c9417
> Oct 29 06:42:40 image-store21003 kernel: RSP: 002b:00007fcb8c3b6208 EFLAGS:
> 00000246 ORIG_RAX: 0000000000000057
> Oct 29 06:42:40 image-store21003 kernel: RAX: ffffffffffffffda RBX:
> 00007fcb8c3b6250 RCX: 00007fcbd64c9417
> Oct 29 06:42:40 image-store21003 kernel: RDX: 00007fcb8c3b6050 RSI:
> 00007fcb8c3b6050 RDI: 00007fcb8c3b63d0
> Oct 29 06:42:40 image-store21003 kernel: RBP: 00007fcb7c047cf0 R08:
> 65342d323362662d R09: 00007fcbd642516d
> Oct 29 06:42:40 image-store21003 kernel: R10: 6532656634383938 R11:
> 0000000000000246 R12: 00007fcb8c3b64c8
> Oct 29 06:42:40 image-store21003 kernel: R13: 00007fcbc4008610 R14:
> 00007fcb8c3b63d0 R15: 0000000000000000
> Oct 29 06:42:40 image-store21003 kernel: kworker/2:3 D 0 2447 2
> 0x80000080
> Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-sync/dm-3
> xfs_log_worker [xfs]
> Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> Oct 29 06:42:40 image-store21003 kernel: wait_for_completion+0x121/0x180
> Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: flush_work+0x11d/0x1a0
> Oct 29 06:42:40 image-store21003 kernel: ? wake_up_worker+0x30/0x30
> Oct 29 06:42:40 image-store21003 kernel: xlog_cil_force_lsn+0x8d/0x210 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? generic_make_request+0x110/0x2e0
> Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x86/0x280 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ?
> __bch_btree_node_write+0x340/0x660 [bcache]
> Oct 29 06:42:40 image-store21003 kernel: ? xfs_log_worker+0x36/0x100 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xfs_log_worker+0x36/0x100 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> Oct 29 06:42:40 image-store21003 kernel: kworker/2:32 D 0 11430 2
> 0x80000080
> Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-cil/dm-3
> xlog_cil_push_work [xfs]
> Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> Oct 29 06:42:40 image-store21003 kernel:
> xlog_state_get_iclog_space+0x105/0x300 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: xlog_write+0x19d/0x6c0 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? kmem_zone_alloc+0xa4/0x110 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push+0x28c/0x460 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: ? __switch_to_asm+0x34/0x70
> Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push_work+0x15/0x20 [xfs]
> Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x1d8/0x3e0
> Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
>
>
>
> On Sat, Oct 27, 2018 at 3:01 AM Bryan Gurney <bgurney at redhat.com> wrote:
>>
>>
>>
>> On Fri, Oct 26, 2018 at 12:03 AM, 劼磊周 <loli.miko at gmail.com> wrote:
>>>
>>>
>>>
>>> I have a big problem
>>>
>>> I use glusterfs over vdo on xfs in case.
>>> my vdo version
>>> Kernel module:
>>> Loaded: true
>>> Name: kvdo
>>> Version information:
>>> kvdo version: 6.2.0.219
>>>
>>> In my case We have many big file over than 500G some is 1T or 2T.
>>> if it be deleteing. Vdo will hang all of io reqeust.
>>> Some small file like 20g or 50g. if size over 100G. this problem will be
>>> very frequency.
>>>
>>> this is come information on my system.I use iostat to record
>>>
>>> your will see dev utils is 100% and it will hang all io request
>>>
>>> iostats: dm-3 is vdo device
>>>
>>> avg-cpu: %user %nice %system %iowait %steal %idle
>>> 0.38 0.00 28.95 0.64 0.00 70.03
>>>
>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>>> sdb 0.00 218.00 0.00 15055.00 0.00 61152.00
>>> 8.12 0.54 0.04 0.00 0.04 0.03 49.00
>>> sda 0.00 205.00 1.00 16104.00 8.00 65908.00
>>> 8.19 0.78 0.05 0.00 0.05 0.04 70.00
>>> sdc 0.00 4.00 0.00 125.00 0.00 644.00
>>> 10.30 0.06 0.50 0.00 0.50 0.01 0.10
>>> dm-0 0.00 0.00 0.00 129.00 0.00 644.00
>>> 9.98 0.06 0.50 0.00 0.50 0.01 0.10
>>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> bcache0 0.00 0.00 0.00 15980.00 0.00 61144.00
>>> 7.65 0.87 0.05 0.00 0.05 0.05 75.40
>>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
>>>
>>> avg-cpu: %user %nice %system %iowait %steal %idle
>>> 0.51 0.00 29.62 0.76 0.00 69.11
>>>
>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>>> sdb 0.00 193.00 0.00 15245.00 0.00 61764.00
>>> 8.10 0.53 0.03 0.00 0.03 0.03 49.90
>>> sda 0.00 192.00 1.00 16268.00 4.00 66512.00
>>> 8.18 0.77 0.05 0.00 0.05 0.04 69.60
>>> sdc 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> bcache0 0.00 0.00 0.00 16137.00 0.00 61756.00
>>> 7.65 0.88 0.05 0.00 0.05 0.05 74.60
>>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
>>>
>>> avg-cpu: %user %nice %system %iowait %steal %idle
>>> 0.39 0.00 28.35 0.64 0.00 70.62
>>>
>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>>> sdb 0.00 143.00 0.00 15014.00 0.00 60648.00
>>> 8.08 0.54 0.04 0.00 0.04 0.03 50.20
>>> sda 0.00 142.00 1.00 15996.00 4.00 65224.00
>>> 8.16 0.74 0.05 0.00 0.05 0.04 67.30
>>> sdc 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> bcache0 0.00 0.00 0.00 15830.00 0.00 60648.00
>>> 7.66 0.87 0.05 0.00 0.05 0.05 73.30
>>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
>>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
>>>
>>>
>>> top info
>>>
>>> Tasks: 1929 total, 2 running, 140 sleeping, 0 stopped, 0 zombie
>>> %Cpu(s): 2.8 us, 17.0 sy, 0.0 ni, 71.7 id, 8.0 wa, 0.0 hi, 0.4 si,
>>> 0.0 st
>>> KiB Mem : 13158505+total, 7327428 free, 10939488+used, 14862756
>>> buff/cache
>>> KiB Swap: 4194300 total, 4194300 free, 0 used. 8015328 avail
>>> Mem
>>>
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>>> COMMAND
>>> 2144 root 20 0 0 0 0 S 19.8 0.0 3767:26
>>> kvdo0:journalQ
>>> 2150 root 20 0 0 0 0 S 15.8 0.0 4701:00
>>> kvdo0:physQ0
>>> 18560 root 20 0 2429296 40120 9004 S 12.2 0.0 1585:55
>>> glusterfsd
>>> 2161 root 20 0 0 0 0 R 11.2 0.0 3579:51
>>> kvdo0:cpuQ0
>>> 2162 root 20 0 0 0 0 S 11.2 0.0 3572:51
>>> kvdo0:cpuQ1
>>> 17807 root 0 -20 0 0 0 D 10.2 0.0 109:02.36
>>> kworker/5:0H
>>> 2146 root 20 0 0 0 0 S 7.6 0.0 1678:45
>>> kvdo0:logQ0
>>> 2147 root 20 0 0 0 0 S 7.3 0.0 1674:34
>>> kvdo0:logQ1
>>> 2148 root 20 0 0 0 0 S 7.3 0.0 1674:09
>>> kvdo0:logQ2
>>> 2149 root 20 0 0 0 0 S 7.3 0.0 1672:51
>>> kvdo0:logQ3
>>> 18567 root 20 0 2369988 31936 9068 S 5.9 0.0 483:05.83
>>> glusterfsd
>>> 2145 root 20 0 0 0 0 S 4.0 0.0 1572:49
>>> kvdo0:packerQ
>>> 2151 root 20 0 0 0 0 S 4.0 0.0 1446:38
>>> kvdo0:hashQ0
>>> 2152 root 20 0 0 0 0 S 4.0 0.0 1442:42
>>> kvdo0:hashQ1
>>> 2156 root 20 0 0 0 0 S 2.6 0.0 798:50.26
>>> kvdo0:bioQ0
>>> 2157 root 20 0 0 0 0 S 2.6 0.0 779:48.42
>>> kvdo0:bioQ1
>>> 2158 root 20 0 0 0 0 S 2.6 0.0 778:43.52
>>> kvdo0:bioQ2
>>> 2159 root 20 0 0 0 0 S 2.6 0.0 776:37.81
>>> kvdo0:bioQ3
>>> 2160 root 20 0 0 0 0 S 2.6 0.0 974:01.15
>>> kvdo0:ackQ
>>>
>>>
>>> --
>>> zhou-jielei
>>>
>>>
>>
>> Hello,
>>
>> In your "top" output, there is a process "kworker/5:0H" with a status of
>> "D" (uninterruptible sleep). If you send "w" to /proc/sysrq-trigger, it
>> will show the "SysRq : Show Blocked State" output to print a call trace on
>> the blocked task.
>>
>> (There may already be a call trace in a "task ... blocked for more than
>> ... seconds" message in the kernel messages for this system.)
>>
>> However, as noted in your github issue, you set the "discards_limit" for
>> this VDO volume (in the sysfs setting /sys/kvdo/<vdoname>/discards_limit)
>> from the default of 1500 to 100. While this will allow for more I/O
>> operations for non-discard operations, it will result in slower discard
>> performance.
>>
>> You can adjust the "discards_limit" setting while the VDO volume is
>> running. If you set it back to 1500, does the discard performance improve?
>>
>> Additionally, I posted a vdo-devel list email on different strategies for
>> discards [1]. Depending on the level of filesystem activity (and the size
>> and/or frequency of the files being discarded), it may be better to use a
>> "scheduled fstrim" approach, instead of "mount -o discard".
>>
>>
>> Thanks,
>>
>> Bryan Gurney
>> Senior Software Engineer, VDO
>> Red Hat
>>
>> [1] https://www.redhat.com/archives/vdo-devel/2018-February/msg00009.html
>
>
>
> --
> 周劼磊
>
More information about the vdo-devel
mailing list