[Vdo-devel] Big problem on deleting big file on vdo. It will stop all io request
Achilles Gaikwad
agaikwad at redhat.com
Thu Nov 1 03:33:37 UTC 2018
Just my two cents. Can you also test if deleting a file from glusterfs
takes time or deleting it from xfs takes time? Xfs is known to take a long
time deleting files which are fragmented or have holes within the files
(sparse files). It would be helpful if sanity check could be done just to
make sure that the file removal from xfs is causing issues.
On Thu, 1 Nov, 2018, 2:23 AM Bryan Gurney <bgurney at redhat.com wrote:
> I recommend remounting the XFS filesystem without the online discard
> option (i.e.: without "-o discard"), and scheduling fstrim runs on the
> mountpoint. On the first fstrim run, monitor the system to see if
> there's any more I/O stalls.
>
>
> Thanks,
>
> Bryan
>
> On Mon, Oct 29, 2018 at 10:52 PM, 劼磊周 <loli.miko at gmail.com> wrote:
> > 1. What is the version of the Linux kernel running on this system?
> > linux 4.14 kenel. two set have different version.
> > one is 4.14.32
> > two is 4.14.68
> > 2. What type of storage device is the VDO volume stored on? Is it
> > rotational (hard-disk drive), or nonrotational (flash storage,
> > solid-state drive)? Does it use RAID (either hardware or software
> > RAID)?
> > two set has different disk
> > one 1T *8 raid5
> > two 8T * 10 raid6 and i add a bcache on it. (dell h730p raid card)
> > all above is in hardware raid.
> >
> >
> > 3. Can you run "dmsetup ls --tree -o ascii" and/or "lsblk -i" to show
> > the positioning of each element in the storage stack?
> > one is
> > dmsetup ls --tree -o ascii
> > sdc (253:3)
> > `- (8:32)
> > vg00-lv01 (253:1)
> > `- (8:2)
> > sdb (253:2)
> > `- (8:16)
> > vg00-lv00 (253:0)
> > `- (8:2)
> >
> > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> > sdb 8:16 0 2.7T 0 disk
> > `-sdb 253:2 0 4T 0 vdo /exports/sdb
> > sdc 8:32 0 2.7T 0 disk
> > `-sdc 253:3 0 4T 0 vdo /exports/sdc
> > sda 8:0 0 102.4G 0 disk
> > |-sda2 8:2 0 101.9G 0 part
> > | |-vg00-lv01 253:1 0 15.6G 0 lvm [SWAP]
> > | `-vg00-lv00 253:0 0 86.2G 0 lvm /
> > `-sda1 8:1 0 512M 0 part /boot
> >
> > two is
> > dmsetup ls --tree -o ascii
> > image_store (253:3)
> > `- (252:0)
> > centos-home (253:2)
> > `- (8:34)
> > centos-swap (253:1)
> > `- (8:34)
> > centos-root (253:0)
> > `- (8:34)
> >
> >
> > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> > sdb 8:16 0 372G 0 disk
> > `-bcache0 252:0 0 58.2T 0 disk
> > `-image_store 253:3 0 116T 0 vdo /exports/disk0
> > sdc 8:32 0 931G 0 disk
> > |-sdc2 8:34 0 930G 0 part
> > | |-centos-swap 253:1 0 4G 0 lvm [SWAP]
> > | |-centos-home 253:2 0 876G 0 lvm /home
> > | `-centos-root 253:0 0 50G 0 lvm /
> > `-sdc1 8:33 0 1G 0 part /boot
> > sda 8:0 0 58.2T 0 disk
> > `-bcache0 252:0 0 58.2T 0 disk
> > `-image_store 253:3 0 116T 0 vdo /exports/disk0
> >
> >
> > On Tue, Oct 30, 2018 at 2:08 AM Bryan Gurney <bgurney at redhat.com> wrote:
> >>
> >> Thank you; these call traces are exactly what I was looking for.
> >>
> >> If possible, can you provide a few more items of information about this
> >> system?
> >>
> >> 1. What is the version of the Linux kernel running on this system?
> >> (i.e.: "uname -r")
> >>
> >> 2. What type of storage device is the VDO volume stored on? Is it
> >> rotational (hard-disk drive), or nonrotational (flash storage,
> >> solid-state drive)? Does it use RAID (either hardware or software
> >> RAID)?
> >>
> >> 3. Can you run "dmsetup ls --tree -o ascii" and/or "lsblk -i" to show
> >> the positioning of each element in the storage stack?
> >>
> >>
> >> Thanks,
> >>
> >> Bryan
> >>
> >> On Mon, Oct 29, 2018 at 2:51 AM, 劼磊周 <loli.miko at gmail.com> wrote:
> >> > hi bryan
> >> > Sorry for replay not in mail list.So in here.I to report again
> >> > For now the problem is where I delete big file in vdo with glusterfs
> all
> >> > io
> >> > request will be hang.
> >> > removing from local vdo mount. will not case this problem.
> >> >
> >> > this a two infomation from my two glusterfs.
> >> > I do not known why this will happen.
> >> >
> >> > Oct 29 06:42:40 image-store21003 kernel: sysrq: SysRq : Show Blocked
> >> > State
> >> > Oct 29 06:42:40 image-store21003 kernel: task
> PC
> >> > stack pid father
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/6:1H D 0 958
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-log/dm-3
> >> > xfs_buf_ioend_work [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel:
> initializeCompletion+0x23/0x40
> >> > [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: io_schedule+0x16/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel:
> limiterWaitForOneFree+0x84/0xe0
> >> > [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? remove_wait_queue+0x60/0x60
> >> > Oct 29 06:42:40 image-store21003 kernel: kvdoMapBio+0x164/0x270 [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: vdoMapBio+0x12/0x20 [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: __map_bio+0xc9/0x210 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? dm_blk_close+0x70/0x70
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __send_changing_extent_only+0x115/0x159 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > get_num_write_zeroes_bios+0x10/0x10 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > dm_remap_zone_report+0x10/0x10
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __split_and_process_non_flush+0x240/0x270 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __split_and_process_bio+0x11b/0x270
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: dm_make_request+0x7b/0xd0
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > generic_make_request+0x110/0x2e0
> >> > Oct 29 06:42:40 image-store21003 kernel: submit_bio+0x75/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: next_bio+0x38/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __blkdev_issue_discard+0x13b/0x1e0
> >> > Oct 29 06:42:40 image-store21003 kernel:
> xlog_cil_committed+0x1d7/0x340
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_do_callback+0x165/0x2b0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_done_syncing+0x72/0x90
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_iodone+0x7d/0xb0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend+0x9d/0x1e0
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend_work+0x15/0x20
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild/dm-3 D 0 9867
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x1b0/0x280
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xfsaild+0x19a/0x780 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild+0x19a/0x780 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: glusteriotwr7 D 0 32733
> >> > 1
> >> > 0x00000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xlog_sync+0x2c9/0x3d0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: __down_common+0xfc/0x132
> >> > Oct 29 06:42:40 image-store21003 kernel: ? _xfs_buf_find+0x2c1/0x510
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: __down+0x1d/0x1f
> >> > Oct 29 06:42:40 image-store21003 kernel: down+0x41/0x50
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_lock+0x3c/0xf0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_buf_find+0x2c1/0x510
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_get_map+0x2a/0x280
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > xfs_free_ag_extent+0x3be/0x7a0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_trans_get_buf_map+0x116/0x190
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_btree_get_bufs+0x5f/0x80
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_alloc_fix_freelist+0x1f8/0x520
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_free_extent_fix_freelist+0x75/0xc0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_free_extent+0x5c/0x150
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_trans_free_extent+0x4f/0x110
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> xfs_trans_add_item+0x5d/0x90
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_extent_free_finish_item+0x26/0x40 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_defer_finish+0x13f/0x410
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_itruncate_extents+0x127/0x2e0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_inactive_truncate+0xad/0x100
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_inactive+0xec/0x110 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> xfs_fs_destroy_inode+0xa4/0x210
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: destroy_inode+0x3b/0x60
> >> > Oct 29 06:42:40 image-store21003 kernel: evict+0x142/0x1a0
> >> > Oct 29 06:42:40 image-store21003 kernel: iput+0x17d/0x1d0
> >> > Oct 29 06:42:40 image-store21003 kernel: do_unlinkat+0x194/0x2b0
> >> > Oct 29 06:42:40 image-store21003 kernel: SyS_unlink+0x16/0x20
> >> > Oct 29 06:42:40 image-store21003 kernel: do_syscall_64+0x73/0x1b0
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> >> > Oct 29 06:42:40 image-store21003 kernel: RIP: 0033:0x7fcbd64c9417
> >> > Oct 29 06:42:40 image-store21003 kernel: RSP: 002b:00007fcb8c3b6208
> >> > EFLAGS:
> >> > 00000246 ORIG_RAX: 0000000000000057
> >> > Oct 29 06:42:40 image-store21003 kernel: RAX: ffffffffffffffda RBX:
> >> > 00007fcb8c3b6250 RCX: 00007fcbd64c9417
> >> > Oct 29 06:42:40 image-store21003 kernel: RDX: 00007fcb8c3b6050 RSI:
> >> > 00007fcb8c3b6050 RDI: 00007fcb8c3b63d0
> >> > Oct 29 06:42:40 image-store21003 kernel: RBP: 00007fcb7c047cf0 R08:
> >> > 65342d323362662d R09: 00007fcbd642516d
> >> > Oct 29 06:42:40 image-store21003 kernel: R10: 6532656634383938 R11:
> >> > 0000000000000246 R12: 00007fcb8c3b64c8
> >> > Oct 29 06:42:40 image-store21003 kernel: R13: 00007fcbc4008610 R14:
> >> > 00007fcb8c3b63d0 R15: 0000000000000000
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:3 D 0 2447
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-sync/dm-3
> >> > xfs_log_worker [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> >> > Oct 29 06:42:40 image-store21003 kernel:
> wait_for_completion+0x121/0x180
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: flush_work+0x11d/0x1a0
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_worker+0x30/0x30
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_force_lsn+0x8d/0x210
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > generic_make_request+0x110/0x2e0
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x86/0x280
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > __bch_btree_node_write+0x340/0x660 [bcache]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_log_worker+0x36/0x100
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_worker+0x36/0x100
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:32 D 0 11430
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-cil/dm-3
> >> > xlog_cil_push_work [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_get_iclog_space+0x105/0x300 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_write+0x19d/0x6c0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? kmem_zone_alloc+0xa4/0x110
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push+0x28c/0x460
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __switch_to_asm+0x34/0x70
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push_work+0x15/0x20
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x1d8/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> >
> >> >
> >> >
> >> > two is
> >> >
> >> > Oct 29 06:42:40 image-store21003 kernel: sysrq: SysRq : Show Blocked
> >> > State
> >> > Oct 29 06:42:40 image-store21003 kernel: task
> PC
> >> > stack pid father
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/6:1H D 0 958
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-log/dm-3
> >> > xfs_buf_ioend_work [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel:
> initializeCompletion+0x23/0x40
> >> > [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: io_schedule+0x16/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel:
> limiterWaitForOneFree+0x84/0xe0
> >> > [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? remove_wait_queue+0x60/0x60
> >> > Oct 29 06:42:40 image-store21003 kernel: kvdoMapBio+0x164/0x270 [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: vdoMapBio+0x12/0x20 [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: __map_bio+0xc9/0x210 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? dm_blk_close+0x70/0x70
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __send_changing_extent_only+0x115/0x159 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > get_num_write_zeroes_bios+0x10/0x10 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > dm_remap_zone_report+0x10/0x10
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __split_and_process_non_flush+0x240/0x270 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __split_and_process_bio+0x11b/0x270
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: dm_make_request+0x7b/0xd0
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > generic_make_request+0x110/0x2e0
> >> > Oct 29 06:42:40 image-store21003 kernel: submit_bio+0x75/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: next_bio+0x38/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __blkdev_issue_discard+0x13b/0x1e0
> >> > Oct 29 06:42:40 image-store21003 kernel:
> xlog_cil_committed+0x1d7/0x340
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_do_callback+0x165/0x2b0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_done_syncing+0x72/0x90
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_iodone+0x7d/0xb0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend+0x9d/0x1e0
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend_work+0x15/0x20
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild/dm-3 D 0 9867
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x1b0/0x280
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xfsaild+0x19a/0x780 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild+0x19a/0x780 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: glusteriotwr7 D 0 32733
> >> > 1
> >> > 0x00000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xlog_sync+0x2c9/0x3d0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: __down_common+0xfc/0x132
> >> > Oct 29 06:42:40 image-store21003 kernel: ? _xfs_buf_find+0x2c1/0x510
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: __down+0x1d/0x1f
> >> > Oct 29 06:42:40 image-store21003 kernel: down+0x41/0x50
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_lock+0x3c/0xf0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_buf_find+0x2c1/0x510
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_get_map+0x2a/0x280
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > xfs_free_ag_extent+0x3be/0x7a0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_trans_get_buf_map+0x116/0x190
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_btree_get_bufs+0x5f/0x80
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_alloc_fix_freelist+0x1f8/0x520
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_free_extent_fix_freelist+0x75/0xc0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_free_extent+0x5c/0x150
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_trans_free_extent+0x4f/0x110
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> xfs_trans_add_item+0x5d/0x90
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_extent_free_finish_item+0x26/0x40 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_defer_finish+0x13f/0x410
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_itruncate_extents+0x127/0x2e0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_inactive_truncate+0xad/0x100
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_inactive+0xec/0x110 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> xfs_fs_destroy_inode+0xa4/0x210
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: destroy_inode+0x3b/0x60
> >> > Oct 29 06:42:40 image-store21003 kernel: evict+0x142/0x1a0
> >> > Oct 29 06:42:40 image-store21003 kernel: iput+0x17d/0x1d0
> >> > Oct 29 06:42:40 image-store21003 kernel: do_unlinkat+0x194/0x2b0
> >> > Oct 29 06:42:40 image-store21003 kernel: SyS_unlink+0x16/0x20
> >> > Oct 29 06:42:40 image-store21003 kernel: do_syscall_64+0x73/0x1b0
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> >> > Oct 29 06:42:40 image-store21003 kernel: RIP: 0033:0x7fcbd64c9417
> >> > Oct 29 06:42:40 image-store21003 kernel: RSP: 002b:00007fcb8c3b6208
> >> > EFLAGS:
> >> > 00000246 ORIG_RAX: 0000000000000057
> >> > Oct 29 06:42:40 image-store21003 kernel: RAX: ffffffffffffffda RBX:
> >> > 00007fcb8c3b6250 RCX: 00007fcbd64c9417
> >> > Oct 29 06:42:40 image-store21003 kernel: RDX: 00007fcb8c3b6050 RSI:
> >> > 00007fcb8c3b6050 RDI: 00007fcb8c3b63d0
> >> > Oct 29 06:42:40 image-store21003 kernel: RBP: 00007fcb7c047cf0 R08:
> >> > 65342d323362662d R09: 00007fcbd642516d
> >> > Oct 29 06:42:40 image-store21003 kernel: R10: 6532656634383938 R11:
> >> > 0000000000000246 R12: 00007fcb8c3b64c8
> >> > Oct 29 06:42:40 image-store21003 kernel: R13: 00007fcbc4008610 R14:
> >> > 00007fcb8c3b63d0 R15: 0000000000000000
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:3 D 0 2447
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-sync/dm-3
> >> > xfs_log_worker [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> >> > Oct 29 06:42:40 image-store21003 kernel:
> wait_for_completion+0x121/0x180
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: flush_work+0x11d/0x1a0
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_worker+0x30/0x30
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_force_lsn+0x8d/0x210
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > generic_make_request+0x110/0x2e0
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x86/0x280
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > __bch_btree_node_write+0x340/0x660 [bcache]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_log_worker+0x36/0x100
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_worker+0x36/0x100
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:32 D 0 11430
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-cil/dm-3
> >> > xlog_cil_push_work [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_get_iclog_space+0x105/0x300 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_write+0x19d/0x6c0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? kmem_zone_alloc+0xa4/0x110
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push+0x28c/0x460
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __switch_to_asm+0x34/0x70
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push_work+0x15/0x20
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x1d8/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> >
> >> >
> >> >
> >> > On Sat, Oct 27, 2018 at 3:01 AM Bryan Gurney <bgurney at redhat.com>
> wrote:
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Oct 26, 2018 at 12:03 AM, 劼磊周 <loli.miko at gmail.com> wrote:
> >> >>>
> >> >>>
> >> >>>
> >> >>> I have a big problem
> >> >>>
> >> >>> I use glusterfs over vdo on xfs in case.
> >> >>> my vdo version
> >> >>> Kernel module:
> >> >>> Loaded: true
> >> >>> Name: kvdo
> >> >>> Version information:
> >> >>> kvdo version: 6.2.0.219
> >> >>>
> >> >>> In my case We have many big file over than 500G some is 1T or 2T.
> >> >>> if it be deleteing. Vdo will hang all of io reqeust.
> >> >>> Some small file like 20g or 50g. if size over 100G. this problem
> will
> >> >>> be
> >> >>> very frequency.
> >> >>>
> >> >>> this is come information on my system.I use iostat to record
> >> >>>
> >> >>> your will see dev utils is 100% and it will hang all io request
> >> >>>
> >> >>> iostats: dm-3 is vdo device
> >> >>>
> >> >>> avg-cpu: %user %nice %system %iowait %steal %idle
> >> >>> 0.38 0.00 28.95 0.64 0.00 70.03
> >> >>>
> >> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >> >>> sdb 0.00 218.00 0.00 15055.00 0.00 61152.00
> >> >>> 8.12 0.54 0.04 0.00 0.04 0.03 49.00
> >> >>> sda 0.00 205.00 1.00 16104.00 8.00 65908.00
> >> >>> 8.19 0.78 0.05 0.00 0.05 0.04 70.00
> >> >>> sdc 0.00 4.00 0.00 125.00 0.00 644.00
> >> >>> 10.30 0.06 0.50 0.00 0.50 0.01 0.10
> >> >>> dm-0 0.00 0.00 0.00 129.00 0.00 644.00
> >> >>> 9.98 0.06 0.50 0.00 0.50 0.01 0.10
> >> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> bcache0 0.00 0.00 0.00 15980.00 0.00 61144.00
> >> >>> 7.65 0.87 0.05 0.00 0.05 0.05 75.40
> >> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
> >> >>>
> >> >>> avg-cpu: %user %nice %system %iowait %steal %idle
> >> >>> 0.51 0.00 29.62 0.76 0.00 69.11
> >> >>>
> >> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >> >>> sdb 0.00 193.00 0.00 15245.00 0.00 61764.00
> >> >>> 8.10 0.53 0.03 0.00 0.03 0.03 49.90
> >> >>> sda 0.00 192.00 1.00 16268.00 4.00 66512.00
> >> >>> 8.18 0.77 0.05 0.00 0.05 0.04 69.60
> >> >>> sdc 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> bcache0 0.00 0.00 0.00 16137.00 0.00 61756.00
> >> >>> 7.65 0.88 0.05 0.00 0.05 0.05 74.60
> >> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
> >> >>>
> >> >>> avg-cpu: %user %nice %system %iowait %steal %idle
> >> >>> 0.39 0.00 28.35 0.64 0.00 70.62
> >> >>>
> >> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >> >>> sdb 0.00 143.00 0.00 15014.00 0.00 60648.00
> >> >>> 8.08 0.54 0.04 0.00 0.04 0.03 50.20
> >> >>> sda 0.00 142.00 1.00 15996.00 4.00 65224.00
> >> >>> 8.16 0.74 0.05 0.00 0.05 0.04 67.30
> >> >>> sdc 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> bcache0 0.00 0.00 0.00 15830.00 0.00 60648.00
> >> >>> 7.66 0.87 0.05 0.00 0.05 0.05 73.30
> >> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
> >> >>>
> >> >>>
> >> >>> top info
> >> >>>
> >> >>> Tasks: 1929 total, 2 running, 140 sleeping, 0 stopped, 0
> zombie
> >> >>> %Cpu(s): 2.8 us, 17.0 sy, 0.0 ni, 71.7 id, 8.0 wa, 0.0 hi, 0.4
> >> >>> si,
> >> >>> 0.0 st
> >> >>> KiB Mem : 13158505+total, 7327428 free, 10939488+used, 14862756
> >> >>> buff/cache
> >> >>> KiB Swap: 4194300 total, 4194300 free, 0 used. 8015328
> avail
> >> >>> Mem
> >> >>>
> >> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> >> >>> COMMAND
> >> >>> 2144 root 20 0 0 0 0 S 19.8 0.0 3767:26
> >> >>> kvdo0:journalQ
> >> >>> 2150 root 20 0 0 0 0 S 15.8 0.0 4701:00
> >> >>> kvdo0:physQ0
> >> >>> 18560 root 20 0 2429296 40120 9004 S 12.2 0.0 1585:55
> >> >>> glusterfsd
> >> >>> 2161 root 20 0 0 0 0 R 11.2 0.0 3579:51
> >> >>> kvdo0:cpuQ0
> >> >>> 2162 root 20 0 0 0 0 S 11.2 0.0 3572:51
> >> >>> kvdo0:cpuQ1
> >> >>> 17807 root 0 -20 0 0 0 D 10.2 0.0 109:02.36
> >> >>> kworker/5:0H
> >> >>> 2146 root 20 0 0 0 0 S 7.6 0.0 1678:45
> >> >>> kvdo0:logQ0
> >> >>> 2147 root 20 0 0 0 0 S 7.3 0.0 1674:34
> >> >>> kvdo0:logQ1
> >> >>> 2148 root 20 0 0 0 0 S 7.3 0.0 1674:09
> >> >>> kvdo0:logQ2
> >> >>> 2149 root 20 0 0 0 0 S 7.3 0.0 1672:51
> >> >>> kvdo0:logQ3
> >> >>> 18567 root 20 0 2369988 31936 9068 S 5.9 0.0 483:05.83
> >> >>> glusterfsd
> >> >>> 2145 root 20 0 0 0 0 S 4.0 0.0 1572:49
> >> >>> kvdo0:packerQ
> >> >>> 2151 root 20 0 0 0 0 S 4.0 0.0 1446:38
> >> >>> kvdo0:hashQ0
> >> >>> 2152 root 20 0 0 0 0 S 4.0 0.0 1442:42
> >> >>> kvdo0:hashQ1
> >> >>> 2156 root 20 0 0 0 0 S 2.6 0.0 798:50.26
> >> >>> kvdo0:bioQ0
> >> >>> 2157 root 20 0 0 0 0 S 2.6 0.0 779:48.42
> >> >>> kvdo0:bioQ1
> >> >>> 2158 root 20 0 0 0 0 S 2.6 0.0 778:43.52
> >> >>> kvdo0:bioQ2
> >> >>> 2159 root 20 0 0 0 0 S 2.6 0.0 776:37.81
> >> >>> kvdo0:bioQ3
> >> >>> 2160 root 20 0 0 0 0 S 2.6 0.0 974:01.15
> >> >>> kvdo0:ackQ
> >> >>>
> >> >>>
> >> >>> --
> >> >>> zhou-jielei
> >> >>>
> >> >>>
> >> >>
> >> >> Hello,
> >> >>
> >> >> In your "top" output, there is a process "kworker/5:0H" with a status
> >> >> of
> >> >> "D" (uninterruptible sleep). If you send "w" to /proc/sysrq-trigger,
> >> >> it
> >> >> will show the "SysRq : Show Blocked State" output to print a call
> trace
> >> >> on
> >> >> the blocked task.
> >> >>
> >> >> (There may already be a call trace in a "task ... blocked for more
> than
> >> >> ... seconds" message in the kernel messages for this system.)
> >> >>
> >> >> However, as noted in your github issue, you set the "discards_limit"
> >> >> for
> >> >> this VDO volume (in the sysfs setting
> >> >> /sys/kvdo/<vdoname>/discards_limit)
> >> >> from the default of 1500 to 100. While this will allow for more I/O
> >> >> operations for non-discard operations, it will result in slower
> discard
> >> >> performance.
> >> >>
> >> >> You can adjust the "discards_limit" setting while the VDO volume is
> >> >> running. If you set it back to 1500, does the discard performance
> >> >> improve?
> >> >>
> >> >> Additionally, I posted a vdo-devel list email on different strategies
> >> >> for
> >> >> discards [1]. Depending on the level of filesystem activity (and the
> >> >> size
> >> >> and/or frequency of the files being discarded), it may be better to
> use
> >> >> a
> >> >> "scheduled fstrim" approach, instead of "mount -o discard".
> >> >>
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Bryan Gurney
> >> >> Senior Software Engineer, VDO
> >> >> Red Hat
> >> >>
> >> >> [1]
> >> >>
> https://www.redhat.com/archives/vdo-devel/2018-February/msg00009.html
> >> >
> >> >
> >> >
> >> > --
> >> > 周劼磊
> >> >
> >
> >
> >
> > --
> > 周劼磊
> >
>
> _______________________________________________
> Vdo-devel mailing list
> Vdo-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/vdo-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vdo-devel/attachments/20181101/dc49943a/attachment.htm>
More information about the vdo-devel
mailing list