[Vdo-devel] Big problem on deleting big file on vdo. It will stop all io request
Bryan Gurney
bgurney at redhat.com
Wed Oct 31 20:50:27 UTC 2018
I recommend remounting the XFS filesystem without the online discard
option (i.e.: without "-o discard"), and scheduling fstrim runs on the
mountpoint. On the first fstrim run, monitor the system to see if
there's any more I/O stalls.
Thanks,
Bryan
On Mon, Oct 29, 2018 at 10:52 PM, 劼磊周 <loli.miko at gmail.com> wrote:
> 1. What is the version of the Linux kernel running on this system?
> linux 4.14 kenel. two set have different version.
> one is 4.14.32
> two is 4.14.68
> 2. What type of storage device is the VDO volume stored on? Is it
> rotational (hard-disk drive), or nonrotational (flash storage,
> solid-state drive)? Does it use RAID (either hardware or software
> RAID)?
> two set has different disk
> one 1T *8 raid5
> two 8T * 10 raid6 and i add a bcache on it. (dell h730p raid card)
> all above is in hardware raid.
>
>
> 3. Can you run "dmsetup ls --tree -o ascii" and/or "lsblk -i" to show
> the positioning of each element in the storage stack?
> one is
> dmsetup ls --tree -o ascii
> sdc (253:3)
> `- (8:32)
> vg00-lv01 (253:1)
> `- (8:2)
> sdb (253:2)
> `- (8:16)
> vg00-lv00 (253:0)
> `- (8:2)
>
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sdb 8:16 0 2.7T 0 disk
> `-sdb 253:2 0 4T 0 vdo /exports/sdb
> sdc 8:32 0 2.7T 0 disk
> `-sdc 253:3 0 4T 0 vdo /exports/sdc
> sda 8:0 0 102.4G 0 disk
> |-sda2 8:2 0 101.9G 0 part
> | |-vg00-lv01 253:1 0 15.6G 0 lvm [SWAP]
> | `-vg00-lv00 253:0 0 86.2G 0 lvm /
> `-sda1 8:1 0 512M 0 part /boot
>
> two is
> dmsetup ls --tree -o ascii
> image_store (253:3)
> `- (252:0)
> centos-home (253:2)
> `- (8:34)
> centos-swap (253:1)
> `- (8:34)
> centos-root (253:0)
> `- (8:34)
>
>
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sdb 8:16 0 372G 0 disk
> `-bcache0 252:0 0 58.2T 0 disk
> `-image_store 253:3 0 116T 0 vdo /exports/disk0
> sdc 8:32 0 931G 0 disk
> |-sdc2 8:34 0 930G 0 part
> | |-centos-swap 253:1 0 4G 0 lvm [SWAP]
> | |-centos-home 253:2 0 876G 0 lvm /home
> | `-centos-root 253:0 0 50G 0 lvm /
> `-sdc1 8:33 0 1G 0 part /boot
> sda 8:0 0 58.2T 0 disk
> `-bcache0 252:0 0 58.2T 0 disk
> `-image_store 253:3 0 116T 0 vdo /exports/disk0
>
>
> On Tue, Oct 30, 2018 at 2:08 AM Bryan Gurney <bgurney at redhat.com> wrote:
>>
>> Thank you; these call traces are exactly what I was looking for.
>>
>> If possible, can you provide a few more items of information about this
>> system?
>>
>> 1. What is the version of the Linux kernel running on this system?
>> (i.e.: "uname -r")
>>
>> 2. What type of storage device is the VDO volume stored on? Is it
>> rotational (hard-disk drive), or nonrotational (flash storage,
>> solid-state drive)? Does it use RAID (either hardware or software
>> RAID)?
>>
>> 3. Can you run "dmsetup ls --tree -o ascii" and/or "lsblk -i" to show
>> the positioning of each element in the storage stack?
>>
>>
>> Thanks,
>>
>> Bryan
>>
>> On Mon, Oct 29, 2018 at 2:51 AM, 劼磊周 <loli.miko at gmail.com> wrote:
>> > hi bryan
>> > Sorry for replay not in mail list.So in here.I to report again
>> > For now the problem is where I delete big file in vdo with glusterfs all
>> > io
>> > request will be hang.
>> > removing from local vdo mount. will not case this problem.
>> >
>> > this a two infomation from my two glusterfs.
>> > I do not known why this will happen.
>> >
>> > Oct 29 06:42:40 image-store21003 kernel: sysrq: SysRq : Show Blocked
>> > State
>> > Oct 29 06:42:40 image-store21003 kernel: task PC
>> > stack pid father
>> > Oct 29 06:42:40 image-store21003 kernel: kworker/6:1H D 0 958
>> > 2
>> > 0x80000080
>> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-log/dm-3
>> > xfs_buf_ioend_work [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
>> > Oct 29 06:42:40 image-store21003 kernel: initializeCompletion+0x23/0x40
>> > [kvdo]
>> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: io_schedule+0x16/0x40
>> > Oct 29 06:42:40 image-store21003 kernel: limiterWaitForOneFree+0x84/0xe0
>> > [kvdo]
>> > Oct 29 06:42:40 image-store21003 kernel: ? remove_wait_queue+0x60/0x60
>> > Oct 29 06:42:40 image-store21003 kernel: kvdoMapBio+0x164/0x270 [kvdo]
>> > Oct 29 06:42:40 image-store21003 kernel: vdoMapBio+0x12/0x20 [kvdo]
>> > Oct 29 06:42:40 image-store21003 kernel: __map_bio+0xc9/0x210 [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel: ? dm_blk_close+0x70/0x70
>> > [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > __send_changing_extent_only+0x115/0x159 [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > get_num_write_zeroes_bios+0x10/0x10 [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > dm_remap_zone_report+0x10/0x10
>> > [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > __split_and_process_non_flush+0x240/0x270 [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > __split_and_process_bio+0x11b/0x270
>> > [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel: dm_make_request+0x7b/0xd0
>> > [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > generic_make_request+0x110/0x2e0
>> > Oct 29 06:42:40 image-store21003 kernel: submit_bio+0x75/0x140
>> > Oct 29 06:42:40 image-store21003 kernel: next_bio+0x38/0x40
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > __blkdev_issue_discard+0x13b/0x1e0
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_committed+0x1d7/0x340
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xlog_state_do_callback+0x165/0x2b0
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xlog_state_done_syncing+0x72/0x90
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_iodone+0x7d/0xb0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend+0x9d/0x1e0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend_work+0x15/0x20
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
>> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
>> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
>> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
>> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
>> > Oct 29 06:42:40 image-store21003 kernel: xfsaild/dm-3 D 0 9867
>> > 2
>> > 0x80000080
>> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
>> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
>> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x1b0/0x280
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: ? xfsaild+0x19a/0x780 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfsaild+0x19a/0x780 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
>> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
>> > Oct 29 06:42:40 image-store21003 kernel: glusteriotwr7 D 0 32733
>> > 1
>> > 0x00000080
>> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
>> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
>> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
>> > Oct 29 06:42:40 image-store21003 kernel: ? xlog_sync+0x2c9/0x3d0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: __down_common+0xfc/0x132
>> > Oct 29 06:42:40 image-store21003 kernel: ? _xfs_buf_find+0x2c1/0x510
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: __down+0x1d/0x1f
>> > Oct 29 06:42:40 image-store21003 kernel: down+0x41/0x50
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_lock+0x3c/0xf0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: _xfs_buf_find+0x2c1/0x510 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_get_map+0x2a/0x280
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > xfs_free_ag_extent+0x3be/0x7a0
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_trans_get_buf_map+0x116/0x190
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_btree_get_bufs+0x5f/0x80
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_alloc_fix_freelist+0x1f8/0x520
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_free_extent_fix_freelist+0x75/0xc0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_free_extent+0x5c/0x150
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_trans_free_extent+0x4f/0x110
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_trans_add_item+0x5d/0x90
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_extent_free_finish_item+0x26/0x40 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_defer_finish+0x13f/0x410
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_itruncate_extents+0x127/0x2e0
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_inactive_truncate+0xad/0x100
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_inactive+0xec/0x110 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_fs_destroy_inode+0xa4/0x210
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: destroy_inode+0x3b/0x60
>> > Oct 29 06:42:40 image-store21003 kernel: evict+0x142/0x1a0
>> > Oct 29 06:42:40 image-store21003 kernel: iput+0x17d/0x1d0
>> > Oct 29 06:42:40 image-store21003 kernel: do_unlinkat+0x194/0x2b0
>> > Oct 29 06:42:40 image-store21003 kernel: SyS_unlink+0x16/0x20
>> > Oct 29 06:42:40 image-store21003 kernel: do_syscall_64+0x73/0x1b0
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > entry_SYSCALL_64_after_hwframe+0x3d/0xa2
>> > Oct 29 06:42:40 image-store21003 kernel: RIP: 0033:0x7fcbd64c9417
>> > Oct 29 06:42:40 image-store21003 kernel: RSP: 002b:00007fcb8c3b6208
>> > EFLAGS:
>> > 00000246 ORIG_RAX: 0000000000000057
>> > Oct 29 06:42:40 image-store21003 kernel: RAX: ffffffffffffffda RBX:
>> > 00007fcb8c3b6250 RCX: 00007fcbd64c9417
>> > Oct 29 06:42:40 image-store21003 kernel: RDX: 00007fcb8c3b6050 RSI:
>> > 00007fcb8c3b6050 RDI: 00007fcb8c3b63d0
>> > Oct 29 06:42:40 image-store21003 kernel: RBP: 00007fcb7c047cf0 R08:
>> > 65342d323362662d R09: 00007fcbd642516d
>> > Oct 29 06:42:40 image-store21003 kernel: R10: 6532656634383938 R11:
>> > 0000000000000246 R12: 00007fcb8c3b64c8
>> > Oct 29 06:42:40 image-store21003 kernel: R13: 00007fcbc4008610 R14:
>> > 00007fcb8c3b63d0 R15: 0000000000000000
>> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:3 D 0 2447
>> > 2
>> > 0x80000080
>> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-sync/dm-3
>> > xfs_log_worker [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
>> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
>> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
>> > Oct 29 06:42:40 image-store21003 kernel: wait_for_completion+0x121/0x180
>> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: flush_work+0x11d/0x1a0
>> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_worker+0x30/0x30
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_force_lsn+0x8d/0x210
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > generic_make_request+0x110/0x2e0
>> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x86/0x280 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > __bch_btree_node_write+0x340/0x660 [bcache]
>> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_log_worker+0x36/0x100
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_worker+0x36/0x100 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
>> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
>> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
>> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
>> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
>> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:32 D 0 11430
>> > 2
>> > 0x80000080
>> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-cil/dm-3
>> > xlog_cil_push_work [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
>> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
>> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xlog_state_get_iclog_space+0x105/0x300 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_write+0x19d/0x6c0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? kmem_zone_alloc+0xa4/0x110
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push+0x28c/0x460 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? __switch_to_asm+0x34/0x70
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push_work+0x15/0x20
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
>> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x1d8/0x3e0
>> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
>> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
>> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
>> >
>> >
>> >
>> > two is
>> >
>> > Oct 29 06:42:40 image-store21003 kernel: sysrq: SysRq : Show Blocked
>> > State
>> > Oct 29 06:42:40 image-store21003 kernel: task PC
>> > stack pid father
>> > Oct 29 06:42:40 image-store21003 kernel: kworker/6:1H D 0 958
>> > 2
>> > 0x80000080
>> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-log/dm-3
>> > xfs_buf_ioend_work [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
>> > Oct 29 06:42:40 image-store21003 kernel: initializeCompletion+0x23/0x40
>> > [kvdo]
>> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: io_schedule+0x16/0x40
>> > Oct 29 06:42:40 image-store21003 kernel: limiterWaitForOneFree+0x84/0xe0
>> > [kvdo]
>> > Oct 29 06:42:40 image-store21003 kernel: ? remove_wait_queue+0x60/0x60
>> > Oct 29 06:42:40 image-store21003 kernel: kvdoMapBio+0x164/0x270 [kvdo]
>> > Oct 29 06:42:40 image-store21003 kernel: vdoMapBio+0x12/0x20 [kvdo]
>> > Oct 29 06:42:40 image-store21003 kernel: __map_bio+0xc9/0x210 [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel: ? dm_blk_close+0x70/0x70
>> > [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > __send_changing_extent_only+0x115/0x159 [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > get_num_write_zeroes_bios+0x10/0x10 [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > dm_remap_zone_report+0x10/0x10
>> > [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > __split_and_process_non_flush+0x240/0x270 [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > __split_and_process_bio+0x11b/0x270
>> > [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel: dm_make_request+0x7b/0xd0
>> > [dm_mod]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > generic_make_request+0x110/0x2e0
>> > Oct 29 06:42:40 image-store21003 kernel: submit_bio+0x75/0x140
>> > Oct 29 06:42:40 image-store21003 kernel: next_bio+0x38/0x40
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > __blkdev_issue_discard+0x13b/0x1e0
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_committed+0x1d7/0x340
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xlog_state_do_callback+0x165/0x2b0
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xlog_state_done_syncing+0x72/0x90
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_iodone+0x7d/0xb0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend+0x9d/0x1e0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend_work+0x15/0x20
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
>> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
>> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
>> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
>> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
>> > Oct 29 06:42:40 image-store21003 kernel: xfsaild/dm-3 D 0 9867
>> > 2
>> > 0x80000080
>> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
>> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
>> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x1b0/0x280
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: ? xfsaild+0x19a/0x780 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfsaild+0x19a/0x780 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
>> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
>> > Oct 29 06:42:40 image-store21003 kernel: glusteriotwr7 D 0 32733
>> > 1
>> > 0x00000080
>> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
>> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
>> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
>> > Oct 29 06:42:40 image-store21003 kernel: ? xlog_sync+0x2c9/0x3d0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: __down_common+0xfc/0x132
>> > Oct 29 06:42:40 image-store21003 kernel: ? _xfs_buf_find+0x2c1/0x510
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: __down+0x1d/0x1f
>> > Oct 29 06:42:40 image-store21003 kernel: down+0x41/0x50
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_lock+0x3c/0xf0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: _xfs_buf_find+0x2c1/0x510 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_get_map+0x2a/0x280
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > xfs_free_ag_extent+0x3be/0x7a0
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_trans_get_buf_map+0x116/0x190
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_btree_get_bufs+0x5f/0x80
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_alloc_fix_freelist+0x1f8/0x520
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_free_extent_fix_freelist+0x75/0xc0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_free_extent+0x5c/0x150
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_trans_free_extent+0x4f/0x110
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_trans_add_item+0x5d/0x90
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_extent_free_finish_item+0x26/0x40 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_defer_finish+0x13f/0x410
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_itruncate_extents+0x127/0x2e0
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xfs_inactive_truncate+0xad/0x100
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_inactive+0xec/0x110 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_fs_destroy_inode+0xa4/0x210
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: destroy_inode+0x3b/0x60
>> > Oct 29 06:42:40 image-store21003 kernel: evict+0x142/0x1a0
>> > Oct 29 06:42:40 image-store21003 kernel: iput+0x17d/0x1d0
>> > Oct 29 06:42:40 image-store21003 kernel: do_unlinkat+0x194/0x2b0
>> > Oct 29 06:42:40 image-store21003 kernel: SyS_unlink+0x16/0x20
>> > Oct 29 06:42:40 image-store21003 kernel: do_syscall_64+0x73/0x1b0
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > entry_SYSCALL_64_after_hwframe+0x3d/0xa2
>> > Oct 29 06:42:40 image-store21003 kernel: RIP: 0033:0x7fcbd64c9417
>> > Oct 29 06:42:40 image-store21003 kernel: RSP: 002b:00007fcb8c3b6208
>> > EFLAGS:
>> > 00000246 ORIG_RAX: 0000000000000057
>> > Oct 29 06:42:40 image-store21003 kernel: RAX: ffffffffffffffda RBX:
>> > 00007fcb8c3b6250 RCX: 00007fcbd64c9417
>> > Oct 29 06:42:40 image-store21003 kernel: RDX: 00007fcb8c3b6050 RSI:
>> > 00007fcb8c3b6050 RDI: 00007fcb8c3b63d0
>> > Oct 29 06:42:40 image-store21003 kernel: RBP: 00007fcb7c047cf0 R08:
>> > 65342d323362662d R09: 00007fcbd642516d
>> > Oct 29 06:42:40 image-store21003 kernel: R10: 6532656634383938 R11:
>> > 0000000000000246 R12: 00007fcb8c3b64c8
>> > Oct 29 06:42:40 image-store21003 kernel: R13: 00007fcbc4008610 R14:
>> > 00007fcb8c3b63d0 R15: 0000000000000000
>> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:3 D 0 2447
>> > 2
>> > 0x80000080
>> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-sync/dm-3
>> > xfs_log_worker [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
>> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
>> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
>> > Oct 29 06:42:40 image-store21003 kernel: wait_for_completion+0x121/0x180
>> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: flush_work+0x11d/0x1a0
>> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_worker+0x30/0x30
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_force_lsn+0x8d/0x210
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > generic_make_request+0x110/0x2e0
>> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x86/0x280 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ?
>> > __bch_btree_node_write+0x340/0x660 [bcache]
>> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_log_worker+0x36/0x100
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_worker+0x36/0x100 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
>> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
>> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
>> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
>> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
>> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:32 D 0 11430
>> > 2
>> > 0x80000080
>> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-cil/dm-3
>> > xlog_cil_push_work [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
>> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
>> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
>> > Oct 29 06:42:40 image-store21003 kernel:
>> > xlog_state_get_iclog_space+0x105/0x300 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_write+0x19d/0x6c0 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? kmem_zone_alloc+0xa4/0x110
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push+0x28c/0x460 [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: ? __switch_to_asm+0x34/0x70
>> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push_work+0x15/0x20
>> > [xfs]
>> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
>> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x1d8/0x3e0
>> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
>> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
>> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
>> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
>> >
>> >
>> >
>> > On Sat, Oct 27, 2018 at 3:01 AM Bryan Gurney <bgurney at redhat.com> wrote:
>> >>
>> >>
>> >>
>> >> On Fri, Oct 26, 2018 at 12:03 AM, 劼磊周 <loli.miko at gmail.com> wrote:
>> >>>
>> >>>
>> >>>
>> >>> I have a big problem
>> >>>
>> >>> I use glusterfs over vdo on xfs in case.
>> >>> my vdo version
>> >>> Kernel module:
>> >>> Loaded: true
>> >>> Name: kvdo
>> >>> Version information:
>> >>> kvdo version: 6.2.0.219
>> >>>
>> >>> In my case We have many big file over than 500G some is 1T or 2T.
>> >>> if it be deleteing. Vdo will hang all of io reqeust.
>> >>> Some small file like 20g or 50g. if size over 100G. this problem will
>> >>> be
>> >>> very frequency.
>> >>>
>> >>> this is come information on my system.I use iostat to record
>> >>>
>> >>> your will see dev utils is 100% and it will hang all io request
>> >>>
>> >>> iostats: dm-3 is vdo device
>> >>>
>> >>> avg-cpu: %user %nice %system %iowait %steal %idle
>> >>> 0.38 0.00 28.95 0.64 0.00 70.03
>> >>>
>> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>> >>> sdb 0.00 218.00 0.00 15055.00 0.00 61152.00
>> >>> 8.12 0.54 0.04 0.00 0.04 0.03 49.00
>> >>> sda 0.00 205.00 1.00 16104.00 8.00 65908.00
>> >>> 8.19 0.78 0.05 0.00 0.05 0.04 70.00
>> >>> sdc 0.00 4.00 0.00 125.00 0.00 644.00
>> >>> 10.30 0.06 0.50 0.00 0.50 0.01 0.10
>> >>> dm-0 0.00 0.00 0.00 129.00 0.00 644.00
>> >>> 9.98 0.06 0.50 0.00 0.50 0.01 0.10
>> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> bcache0 0.00 0.00 0.00 15980.00 0.00 61144.00
>> >>> 7.65 0.87 0.05 0.00 0.05 0.05 75.40
>> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
>> >>>
>> >>> avg-cpu: %user %nice %system %iowait %steal %idle
>> >>> 0.51 0.00 29.62 0.76 0.00 69.11
>> >>>
>> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>> >>> sdb 0.00 193.00 0.00 15245.00 0.00 61764.00
>> >>> 8.10 0.53 0.03 0.00 0.03 0.03 49.90
>> >>> sda 0.00 192.00 1.00 16268.00 4.00 66512.00
>> >>> 8.18 0.77 0.05 0.00 0.05 0.04 69.60
>> >>> sdc 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> bcache0 0.00 0.00 0.00 16137.00 0.00 61756.00
>> >>> 7.65 0.88 0.05 0.00 0.05 0.05 74.60
>> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
>> >>>
>> >>> avg-cpu: %user %nice %system %iowait %steal %idle
>> >>> 0.39 0.00 28.35 0.64 0.00 70.62
>> >>>
>> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>> >>> sdb 0.00 143.00 0.00 15014.00 0.00 60648.00
>> >>> 8.08 0.54 0.04 0.00 0.04 0.03 50.20
>> >>> sda 0.00 142.00 1.00 15996.00 4.00 65224.00
>> >>> 8.16 0.74 0.05 0.00 0.05 0.04 67.30
>> >>> sdc 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> bcache0 0.00 0.00 0.00 15830.00 0.00 60648.00
>> >>> 7.66 0.87 0.05 0.00 0.05 0.05 73.30
>> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
>> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
>> >>>
>> >>>
>> >>> top info
>> >>>
>> >>> Tasks: 1929 total, 2 running, 140 sleeping, 0 stopped, 0 zombie
>> >>> %Cpu(s): 2.8 us, 17.0 sy, 0.0 ni, 71.7 id, 8.0 wa, 0.0 hi, 0.4
>> >>> si,
>> >>> 0.0 st
>> >>> KiB Mem : 13158505+total, 7327428 free, 10939488+used, 14862756
>> >>> buff/cache
>> >>> KiB Swap: 4194300 total, 4194300 free, 0 used. 8015328 avail
>> >>> Mem
>> >>>
>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> >>> COMMAND
>> >>> 2144 root 20 0 0 0 0 S 19.8 0.0 3767:26
>> >>> kvdo0:journalQ
>> >>> 2150 root 20 0 0 0 0 S 15.8 0.0 4701:00
>> >>> kvdo0:physQ0
>> >>> 18560 root 20 0 2429296 40120 9004 S 12.2 0.0 1585:55
>> >>> glusterfsd
>> >>> 2161 root 20 0 0 0 0 R 11.2 0.0 3579:51
>> >>> kvdo0:cpuQ0
>> >>> 2162 root 20 0 0 0 0 S 11.2 0.0 3572:51
>> >>> kvdo0:cpuQ1
>> >>> 17807 root 0 -20 0 0 0 D 10.2 0.0 109:02.36
>> >>> kworker/5:0H
>> >>> 2146 root 20 0 0 0 0 S 7.6 0.0 1678:45
>> >>> kvdo0:logQ0
>> >>> 2147 root 20 0 0 0 0 S 7.3 0.0 1674:34
>> >>> kvdo0:logQ1
>> >>> 2148 root 20 0 0 0 0 S 7.3 0.0 1674:09
>> >>> kvdo0:logQ2
>> >>> 2149 root 20 0 0 0 0 S 7.3 0.0 1672:51
>> >>> kvdo0:logQ3
>> >>> 18567 root 20 0 2369988 31936 9068 S 5.9 0.0 483:05.83
>> >>> glusterfsd
>> >>> 2145 root 20 0 0 0 0 S 4.0 0.0 1572:49
>> >>> kvdo0:packerQ
>> >>> 2151 root 20 0 0 0 0 S 4.0 0.0 1446:38
>> >>> kvdo0:hashQ0
>> >>> 2152 root 20 0 0 0 0 S 4.0 0.0 1442:42
>> >>> kvdo0:hashQ1
>> >>> 2156 root 20 0 0 0 0 S 2.6 0.0 798:50.26
>> >>> kvdo0:bioQ0
>> >>> 2157 root 20 0 0 0 0 S 2.6 0.0 779:48.42
>> >>> kvdo0:bioQ1
>> >>> 2158 root 20 0 0 0 0 S 2.6 0.0 778:43.52
>> >>> kvdo0:bioQ2
>> >>> 2159 root 20 0 0 0 0 S 2.6 0.0 776:37.81
>> >>> kvdo0:bioQ3
>> >>> 2160 root 20 0 0 0 0 S 2.6 0.0 974:01.15
>> >>> kvdo0:ackQ
>> >>>
>> >>>
>> >>> --
>> >>> zhou-jielei
>> >>>
>> >>>
>> >>
>> >> Hello,
>> >>
>> >> In your "top" output, there is a process "kworker/5:0H" with a status
>> >> of
>> >> "D" (uninterruptible sleep). If you send "w" to /proc/sysrq-trigger,
>> >> it
>> >> will show the "SysRq : Show Blocked State" output to print a call trace
>> >> on
>> >> the blocked task.
>> >>
>> >> (There may already be a call trace in a "task ... blocked for more than
>> >> ... seconds" message in the kernel messages for this system.)
>> >>
>> >> However, as noted in your github issue, you set the "discards_limit"
>> >> for
>> >> this VDO volume (in the sysfs setting
>> >> /sys/kvdo/<vdoname>/discards_limit)
>> >> from the default of 1500 to 100. While this will allow for more I/O
>> >> operations for non-discard operations, it will result in slower discard
>> >> performance.
>> >>
>> >> You can adjust the "discards_limit" setting while the VDO volume is
>> >> running. If you set it back to 1500, does the discard performance
>> >> improve?
>> >>
>> >> Additionally, I posted a vdo-devel list email on different strategies
>> >> for
>> >> discards [1]. Depending on the level of filesystem activity (and the
>> >> size
>> >> and/or frequency of the files being discarded), it may be better to use
>> >> a
>> >> "scheduled fstrim" approach, instead of "mount -o discard".
>> >>
>> >>
>> >> Thanks,
>> >>
>> >> Bryan Gurney
>> >> Senior Software Engineer, VDO
>> >> Red Hat
>> >>
>> >> [1]
>> >> https://www.redhat.com/archives/vdo-devel/2018-February/msg00009.html
>> >
>> >
>> >
>> > --
>> > 周劼磊
>> >
>
>
>
> --
> 周劼磊
>
More information about the vdo-devel
mailing list