[Vdo-devel] Big problem on deleting big file on vdo. It will stop all io request
劼磊周
loli.miko at gmail.com
Thu Nov 1 04:27:23 UTC 2018
Yesterday I change mount without discard
And use fstrim .
But It is really need a very long time to run. It is running about 28hour
and not end yet.
On Thu, Nov 1, 2018 at 5:50 AM Bryan Gurney <bgurney at redhat.com> wrote:
> I recommend remounting the XFS filesystem without the online discard
> option (i.e.: without "-o discard"), and scheduling fstrim runs on the
> mountpoint. On the first fstrim run, monitor the system to see if
> there's any more I/O stalls.
>
>
> Thanks,
>
> Bryan
>
> On Mon, Oct 29, 2018 at 10:52 PM, 劼磊周 <loli.miko at gmail.com> wrote:
> > 1. What is the version of the Linux kernel running on this system?
> > linux 4.14 kenel. two set have different version.
> > one is 4.14.32
> > two is 4.14.68
> > 2. What type of storage device is the VDO volume stored on? Is it
> > rotational (hard-disk drive), or nonrotational (flash storage,
> > solid-state drive)? Does it use RAID (either hardware or software
> > RAID)?
> > two set has different disk
> > one 1T *8 raid5
> > two 8T * 10 raid6 and i add a bcache on it. (dell h730p raid card)
> > all above is in hardware raid.
> >
> >
> > 3. Can you run "dmsetup ls --tree -o ascii" and/or "lsblk -i" to show
> > the positioning of each element in the storage stack?
> > one is
> > dmsetup ls --tree -o ascii
> > sdc (253:3)
> > `- (8:32)
> > vg00-lv01 (253:1)
> > `- (8:2)
> > sdb (253:2)
> > `- (8:16)
> > vg00-lv00 (253:0)
> > `- (8:2)
> >
> > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> > sdb 8:16 0 2.7T 0 disk
> > `-sdb 253:2 0 4T 0 vdo /exports/sdb
> > sdc 8:32 0 2.7T 0 disk
> > `-sdc 253:3 0 4T 0 vdo /exports/sdc
> > sda 8:0 0 102.4G 0 disk
> > |-sda2 8:2 0 101.9G 0 part
> > | |-vg00-lv01 253:1 0 15.6G 0 lvm [SWAP]
> > | `-vg00-lv00 253:0 0 86.2G 0 lvm /
> > `-sda1 8:1 0 512M 0 part /boot
> >
> > two is
> > dmsetup ls --tree -o ascii
> > image_store (253:3)
> > `- (252:0)
> > centos-home (253:2)
> > `- (8:34)
> > centos-swap (253:1)
> > `- (8:34)
> > centos-root (253:0)
> > `- (8:34)
> >
> >
> > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> > sdb 8:16 0 372G 0 disk
> > `-bcache0 252:0 0 58.2T 0 disk
> > `-image_store 253:3 0 116T 0 vdo /exports/disk0
> > sdc 8:32 0 931G 0 disk
> > |-sdc2 8:34 0 930G 0 part
> > | |-centos-swap 253:1 0 4G 0 lvm [SWAP]
> > | |-centos-home 253:2 0 876G 0 lvm /home
> > | `-centos-root 253:0 0 50G 0 lvm /
> > `-sdc1 8:33 0 1G 0 part /boot
> > sda 8:0 0 58.2T 0 disk
> > `-bcache0 252:0 0 58.2T 0 disk
> > `-image_store 253:3 0 116T 0 vdo /exports/disk0
> >
> >
> > On Tue, Oct 30, 2018 at 2:08 AM Bryan Gurney <bgurney at redhat.com> wrote:
> >>
> >> Thank you; these call traces are exactly what I was looking for.
> >>
> >> If possible, can you provide a few more items of information about this
> >> system?
> >>
> >> 1. What is the version of the Linux kernel running on this system?
> >> (i.e.: "uname -r")
> >>
> >> 2. What type of storage device is the VDO volume stored on? Is it
> >> rotational (hard-disk drive), or nonrotational (flash storage,
> >> solid-state drive)? Does it use RAID (either hardware or software
> >> RAID)?
> >>
> >> 3. Can you run "dmsetup ls --tree -o ascii" and/or "lsblk -i" to show
> >> the positioning of each element in the storage stack?
> >>
> >>
> >> Thanks,
> >>
> >> Bryan
> >>
> >> On Mon, Oct 29, 2018 at 2:51 AM, 劼磊周 <loli.miko at gmail.com> wrote:
> >> > hi bryan
> >> > Sorry for replay not in mail list.So in here.I to report again
> >> > For now the problem is where I delete big file in vdo with glusterfs
> all
> >> > io
> >> > request will be hang.
> >> > removing from local vdo mount. will not case this problem.
> >> >
> >> > this a two infomation from my two glusterfs.
> >> > I do not known why this will happen.
> >> >
> >> > Oct 29 06:42:40 image-store21003 kernel: sysrq: SysRq : Show Blocked
> >> > State
> >> > Oct 29 06:42:40 image-store21003 kernel: task
> PC
> >> > stack pid father
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/6:1H D 0 958
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-log/dm-3
> >> > xfs_buf_ioend_work [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel:
> initializeCompletion+0x23/0x40
> >> > [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: io_schedule+0x16/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel:
> limiterWaitForOneFree+0x84/0xe0
> >> > [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? remove_wait_queue+0x60/0x60
> >> > Oct 29 06:42:40 image-store21003 kernel: kvdoMapBio+0x164/0x270 [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: vdoMapBio+0x12/0x20 [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: __map_bio+0xc9/0x210 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? dm_blk_close+0x70/0x70
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __send_changing_extent_only+0x115/0x159 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > get_num_write_zeroes_bios+0x10/0x10 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > dm_remap_zone_report+0x10/0x10
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __split_and_process_non_flush+0x240/0x270 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __split_and_process_bio+0x11b/0x270
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: dm_make_request+0x7b/0xd0
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > generic_make_request+0x110/0x2e0
> >> > Oct 29 06:42:40 image-store21003 kernel: submit_bio+0x75/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: next_bio+0x38/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __blkdev_issue_discard+0x13b/0x1e0
> >> > Oct 29 06:42:40 image-store21003 kernel:
> xlog_cil_committed+0x1d7/0x340
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_do_callback+0x165/0x2b0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_done_syncing+0x72/0x90
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_iodone+0x7d/0xb0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend+0x9d/0x1e0
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend_work+0x15/0x20
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild/dm-3 D 0 9867
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x1b0/0x280
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xfsaild+0x19a/0x780 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild+0x19a/0x780 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: glusteriotwr7 D 0 32733
> >> > 1
> >> > 0x00000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xlog_sync+0x2c9/0x3d0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: __down_common+0xfc/0x132
> >> > Oct 29 06:42:40 image-store21003 kernel: ? _xfs_buf_find+0x2c1/0x510
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: __down+0x1d/0x1f
> >> > Oct 29 06:42:40 image-store21003 kernel: down+0x41/0x50
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_lock+0x3c/0xf0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_buf_find+0x2c1/0x510
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_get_map+0x2a/0x280
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > xfs_free_ag_extent+0x3be/0x7a0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_trans_get_buf_map+0x116/0x190
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_btree_get_bufs+0x5f/0x80
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_alloc_fix_freelist+0x1f8/0x520
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_free_extent_fix_freelist+0x75/0xc0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_free_extent+0x5c/0x150
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_trans_free_extent+0x4f/0x110
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> xfs_trans_add_item+0x5d/0x90
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_extent_free_finish_item+0x26/0x40 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_defer_finish+0x13f/0x410
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_itruncate_extents+0x127/0x2e0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_inactive_truncate+0xad/0x100
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_inactive+0xec/0x110 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> xfs_fs_destroy_inode+0xa4/0x210
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: destroy_inode+0x3b/0x60
> >> > Oct 29 06:42:40 image-store21003 kernel: evict+0x142/0x1a0
> >> > Oct 29 06:42:40 image-store21003 kernel: iput+0x17d/0x1d0
> >> > Oct 29 06:42:40 image-store21003 kernel: do_unlinkat+0x194/0x2b0
> >> > Oct 29 06:42:40 image-store21003 kernel: SyS_unlink+0x16/0x20
> >> > Oct 29 06:42:40 image-store21003 kernel: do_syscall_64+0x73/0x1b0
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> >> > Oct 29 06:42:40 image-store21003 kernel: RIP: 0033:0x7fcbd64c9417
> >> > Oct 29 06:42:40 image-store21003 kernel: RSP: 002b:00007fcb8c3b6208
> >> > EFLAGS:
> >> > 00000246 ORIG_RAX: 0000000000000057
> >> > Oct 29 06:42:40 image-store21003 kernel: RAX: ffffffffffffffda RBX:
> >> > 00007fcb8c3b6250 RCX: 00007fcbd64c9417
> >> > Oct 29 06:42:40 image-store21003 kernel: RDX: 00007fcb8c3b6050 RSI:
> >> > 00007fcb8c3b6050 RDI: 00007fcb8c3b63d0
> >> > Oct 29 06:42:40 image-store21003 kernel: RBP: 00007fcb7c047cf0 R08:
> >> > 65342d323362662d R09: 00007fcbd642516d
> >> > Oct 29 06:42:40 image-store21003 kernel: R10: 6532656634383938 R11:
> >> > 0000000000000246 R12: 00007fcb8c3b64c8
> >> > Oct 29 06:42:40 image-store21003 kernel: R13: 00007fcbc4008610 R14:
> >> > 00007fcb8c3b63d0 R15: 0000000000000000
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:3 D 0 2447
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-sync/dm-3
> >> > xfs_log_worker [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> >> > Oct 29 06:42:40 image-store21003 kernel:
> wait_for_completion+0x121/0x180
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: flush_work+0x11d/0x1a0
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_worker+0x30/0x30
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_force_lsn+0x8d/0x210
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > generic_make_request+0x110/0x2e0
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x86/0x280
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > __bch_btree_node_write+0x340/0x660 [bcache]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_log_worker+0x36/0x100
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_worker+0x36/0x100
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:32 D 0 11430
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-cil/dm-3
> >> > xlog_cil_push_work [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_get_iclog_space+0x105/0x300 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_write+0x19d/0x6c0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? kmem_zone_alloc+0xa4/0x110
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push+0x28c/0x460
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __switch_to_asm+0x34/0x70
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push_work+0x15/0x20
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x1d8/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> >
> >> >
> >> >
> >> > two is
> >> >
> >> > Oct 29 06:42:40 image-store21003 kernel: sysrq: SysRq : Show Blocked
> >> > State
> >> > Oct 29 06:42:40 image-store21003 kernel: task
> PC
> >> > stack pid father
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/6:1H D 0 958
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-log/dm-3
> >> > xfs_buf_ioend_work [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel:
> initializeCompletion+0x23/0x40
> >> > [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: io_schedule+0x16/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel:
> limiterWaitForOneFree+0x84/0xe0
> >> > [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? remove_wait_queue+0x60/0x60
> >> > Oct 29 06:42:40 image-store21003 kernel: kvdoMapBio+0x164/0x270 [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: vdoMapBio+0x12/0x20 [kvdo]
> >> > Oct 29 06:42:40 image-store21003 kernel: __map_bio+0xc9/0x210 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? dm_blk_close+0x70/0x70
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __send_changing_extent_only+0x115/0x159 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > get_num_write_zeroes_bios+0x10/0x10 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > dm_remap_zone_report+0x10/0x10
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __split_and_process_non_flush+0x240/0x270 [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __split_and_process_bio+0x11b/0x270
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel: dm_make_request+0x7b/0xd0
> >> > [dm_mod]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > generic_make_request+0x110/0x2e0
> >> > Oct 29 06:42:40 image-store21003 kernel: submit_bio+0x75/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: next_bio+0x38/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > __blkdev_issue_discard+0x13b/0x1e0
> >> > Oct 29 06:42:40 image-store21003 kernel:
> xlog_cil_committed+0x1d7/0x340
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_do_callback+0x165/0x2b0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_done_syncing+0x72/0x90
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_iodone+0x7d/0xb0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend+0x9d/0x1e0
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_ioend_work+0x15/0x20
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild/dm-3 D 0 9867
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x1b0/0x280
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xfsaild+0x19a/0x780 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfsaild+0x19a/0x780 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: glusteriotwr7 D 0 32733
> >> > 1
> >> > 0x00000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xlog_sync+0x2c9/0x3d0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: __down_common+0xfc/0x132
> >> > Oct 29 06:42:40 image-store21003 kernel: ? _xfs_buf_find+0x2c1/0x510
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: __down+0x1d/0x1f
> >> > Oct 29 06:42:40 image-store21003 kernel: down+0x41/0x50
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_lock+0x3c/0xf0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_buf_find+0x2c1/0x510
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_buf_get_map+0x2a/0x280
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > xfs_free_ag_extent+0x3be/0x7a0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_trans_get_buf_map+0x116/0x190
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_btree_get_bufs+0x5f/0x80
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_alloc_fix_freelist+0x1f8/0x520
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_free_extent_fix_freelist+0x75/0xc0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_free_extent+0x5c/0x150
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_trans_free_extent+0x4f/0x110
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> xfs_trans_add_item+0x5d/0x90
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_extent_free_finish_item+0x26/0x40 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_defer_finish+0x13f/0x410
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_itruncate_extents+0x127/0x2e0
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xfs_inactive_truncate+0xad/0x100
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_inactive+0xec/0x110 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel:
> xfs_fs_destroy_inode+0xa4/0x210
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: destroy_inode+0x3b/0x60
> >> > Oct 29 06:42:40 image-store21003 kernel: evict+0x142/0x1a0
> >> > Oct 29 06:42:40 image-store21003 kernel: iput+0x17d/0x1d0
> >> > Oct 29 06:42:40 image-store21003 kernel: do_unlinkat+0x194/0x2b0
> >> > Oct 29 06:42:40 image-store21003 kernel: SyS_unlink+0x16/0x20
> >> > Oct 29 06:42:40 image-store21003 kernel: do_syscall_64+0x73/0x1b0
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> >> > Oct 29 06:42:40 image-store21003 kernel: RIP: 0033:0x7fcbd64c9417
> >> > Oct 29 06:42:40 image-store21003 kernel: RSP: 002b:00007fcb8c3b6208
> >> > EFLAGS:
> >> > 00000246 ORIG_RAX: 0000000000000057
> >> > Oct 29 06:42:40 image-store21003 kernel: RAX: ffffffffffffffda RBX:
> >> > 00007fcb8c3b6250 RCX: 00007fcbd64c9417
> >> > Oct 29 06:42:40 image-store21003 kernel: RDX: 00007fcb8c3b6050 RSI:
> >> > 00007fcb8c3b6050 RDI: 00007fcb8c3b63d0
> >> > Oct 29 06:42:40 image-store21003 kernel: RBP: 00007fcb7c047cf0 R08:
> >> > 65342d323362662d R09: 00007fcbd642516d
> >> > Oct 29 06:42:40 image-store21003 kernel: R10: 6532656634383938 R11:
> >> > 0000000000000246 R12: 00007fcb8c3b64c8
> >> > Oct 29 06:42:40 image-store21003 kernel: R13: 00007fcbc4008610 R14:
> >> > 00007fcb8c3b63d0 R15: 0000000000000000
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:3 D 0 2447
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-sync/dm-3
> >> > xfs_log_worker [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule_timeout+0x1e6/0x320
> >> > Oct 29 06:42:40 image-store21003 kernel:
> wait_for_completion+0x121/0x180
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: flush_work+0x11d/0x1a0
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_worker+0x30/0x30
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_force_lsn+0x8d/0x210
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > generic_make_request+0x110/0x2e0
> >> > Oct 29 06:42:40 image-store21003 kernel: _xfs_log_force+0x86/0x280
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ?
> >> > __bch_btree_node_write+0x340/0x660 [bcache]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? xfs_log_worker+0x36/0x100
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_force+0x2c/0x80 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xfs_log_worker+0x36/0x100
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x4d/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> > Oct 29 06:42:40 image-store21003 kernel: kworker/2:32 D 0 11430
> >> > 2
> >> > 0x80000080
> >> > Oct 29 06:42:40 image-store21003 kernel: Workqueue: xfs-cil/dm-3
> >> > xlog_cil_push_work [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: Call Trace:
> >> > Oct 29 06:42:40 image-store21003 kernel: __schedule+0x28a/0x870
> >> > Oct 29 06:42:40 image-store21003 kernel: schedule+0x36/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel:
> >> > xlog_state_get_iclog_space+0x105/0x300 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? wake_up_q+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_write+0x19d/0x6c0 [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? kmem_zone_alloc+0xa4/0x110
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push+0x28c/0x460
> [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __switch_to_asm+0x34/0x70
> >> > Oct 29 06:42:40 image-store21003 kernel: xlog_cil_push_work+0x15/0x20
> >> > [xfs]
> >> > Oct 29 06:42:40 image-store21003 kernel: process_one_work+0x157/0x380
> >> > Oct 29 06:42:40 image-store21003 kernel: worker_thread+0x1d8/0x3e0
> >> > Oct 29 06:42:40 image-store21003 kernel: kthread+0x109/0x140
> >> > Oct 29 06:42:40 image-store21003 kernel: ? max_active_store+0x80/0x80
> >> > Oct 29 06:42:40 image-store21003 kernel: ? __kthread_parkme+0xa0/0xa0
> >> > Oct 29 06:42:40 image-store21003 kernel: ret_from_fork+0x35/0x40
> >> >
> >> >
> >> >
> >> > On Sat, Oct 27, 2018 at 3:01 AM Bryan Gurney <bgurney at redhat.com>
> wrote:
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Oct 26, 2018 at 12:03 AM, 劼磊周 <loli.miko at gmail.com> wrote:
> >> >>>
> >> >>>
> >> >>>
> >> >>> I have a big problem
> >> >>>
> >> >>> I use glusterfs over vdo on xfs in case.
> >> >>> my vdo version
> >> >>> Kernel module:
> >> >>> Loaded: true
> >> >>> Name: kvdo
> >> >>> Version information:
> >> >>> kvdo version: 6.2.0.219
> >> >>>
> >> >>> In my case We have many big file over than 500G some is 1T or 2T.
> >> >>> if it be deleteing. Vdo will hang all of io reqeust.
> >> >>> Some small file like 20g or 50g. if size over 100G. this problem
> will
> >> >>> be
> >> >>> very frequency.
> >> >>>
> >> >>> this is come information on my system.I use iostat to record
> >> >>>
> >> >>> your will see dev utils is 100% and it will hang all io request
> >> >>>
> >> >>> iostats: dm-3 is vdo device
> >> >>>
> >> >>> avg-cpu: %user %nice %system %iowait %steal %idle
> >> >>> 0.38 0.00 28.95 0.64 0.00 70.03
> >> >>>
> >> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >> >>> sdb 0.00 218.00 0.00 15055.00 0.00 61152.00
> >> >>> 8.12 0.54 0.04 0.00 0.04 0.03 49.00
> >> >>> sda 0.00 205.00 1.00 16104.00 8.00 65908.00
> >> >>> 8.19 0.78 0.05 0.00 0.05 0.04 70.00
> >> >>> sdc 0.00 4.00 0.00 125.00 0.00 644.00
> >> >>> 10.30 0.06 0.50 0.00 0.50 0.01 0.10
> >> >>> dm-0 0.00 0.00 0.00 129.00 0.00 644.00
> >> >>> 9.98 0.06 0.50 0.00 0.50 0.01 0.10
> >> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> bcache0 0.00 0.00 0.00 15980.00 0.00 61144.00
> >> >>> 7.65 0.87 0.05 0.00 0.05 0.05 75.40
> >> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
> >> >>>
> >> >>> avg-cpu: %user %nice %system %iowait %steal %idle
> >> >>> 0.51 0.00 29.62 0.76 0.00 69.11
> >> >>>
> >> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >> >>> sdb 0.00 193.00 0.00 15245.00 0.00 61764.00
> >> >>> 8.10 0.53 0.03 0.00 0.03 0.03 49.90
> >> >>> sda 0.00 192.00 1.00 16268.00 4.00 66512.00
> >> >>> 8.18 0.77 0.05 0.00 0.05 0.04 69.60
> >> >>> sdc 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> bcache0 0.00 0.00 0.00 16137.00 0.00 61756.00
> >> >>> 7.65 0.88 0.05 0.00 0.05 0.05 74.60
> >> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
> >> >>>
> >> >>> avg-cpu: %user %nice %system %iowait %steal %idle
> >> >>> 0.39 0.00 28.35 0.64 0.00 70.62
> >> >>>
> >> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >> >>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >> >>> sdb 0.00 143.00 0.00 15014.00 0.00 60648.00
> >> >>> 8.08 0.54 0.04 0.00 0.04 0.03 50.20
> >> >>> sda 0.00 142.00 1.00 15996.00 4.00 65224.00
> >> >>> 8.16 0.74 0.05 0.00 0.05 0.04 67.30
> >> >>> sdc 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> bcache0 0.00 0.00 0.00 15830.00 0.00 60648.00
> >> >>> 7.66 0.87 0.05 0.00 0.05 0.05 73.30
> >> >>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00
> >> >>> 0.00 1.00 0.00 0.00 0.00 0.00 100.00
> >> >>>
> >> >>>
> >> >>> top info
> >> >>>
> >> >>> Tasks: 1929 total, 2 running, 140 sleeping, 0 stopped, 0
> zombie
> >> >>> %Cpu(s): 2.8 us, 17.0 sy, 0.0 ni, 71.7 id, 8.0 wa, 0.0 hi, 0.4
> >> >>> si,
> >> >>> 0.0 st
> >> >>> KiB Mem : 13158505+total, 7327428 free, 10939488+used, 14862756
> >> >>> buff/cache
> >> >>> KiB Swap: 4194300 total, 4194300 free, 0 used. 8015328
> avail
> >> >>> Mem
> >> >>>
> >> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> >> >>> COMMAND
> >> >>> 2144 root 20 0 0 0 0 S 19.8 0.0 3767:26
> >> >>> kvdo0:journalQ
> >> >>> 2150 root 20 0 0 0 0 S 15.8 0.0 4701:00
> >> >>> kvdo0:physQ0
> >> >>> 18560 root 20 0 2429296 40120 9004 S 12.2 0.0 1585:55
> >> >>> glusterfsd
> >> >>> 2161 root 20 0 0 0 0 R 11.2 0.0 3579:51
> >> >>> kvdo0:cpuQ0
> >> >>> 2162 root 20 0 0 0 0 S 11.2 0.0 3572:51
> >> >>> kvdo0:cpuQ1
> >> >>> 17807 root 0 -20 0 0 0 D 10.2 0.0 109:02.36
> >> >>> kworker/5:0H
> >> >>> 2146 root 20 0 0 0 0 S 7.6 0.0 1678:45
> >> >>> kvdo0:logQ0
> >> >>> 2147 root 20 0 0 0 0 S 7.3 0.0 1674:34
> >> >>> kvdo0:logQ1
> >> >>> 2148 root 20 0 0 0 0 S 7.3 0.0 1674:09
> >> >>> kvdo0:logQ2
> >> >>> 2149 root 20 0 0 0 0 S 7.3 0.0 1672:51
> >> >>> kvdo0:logQ3
> >> >>> 18567 root 20 0 2369988 31936 9068 S 5.9 0.0 483:05.83
> >> >>> glusterfsd
> >> >>> 2145 root 20 0 0 0 0 S 4.0 0.0 1572:49
> >> >>> kvdo0:packerQ
> >> >>> 2151 root 20 0 0 0 0 S 4.0 0.0 1446:38
> >> >>> kvdo0:hashQ0
> >> >>> 2152 root 20 0 0 0 0 S 4.0 0.0 1442:42
> >> >>> kvdo0:hashQ1
> >> >>> 2156 root 20 0 0 0 0 S 2.6 0.0 798:50.26
> >> >>> kvdo0:bioQ0
> >> >>> 2157 root 20 0 0 0 0 S 2.6 0.0 779:48.42
> >> >>> kvdo0:bioQ1
> >> >>> 2158 root 20 0 0 0 0 S 2.6 0.0 778:43.52
> >> >>> kvdo0:bioQ2
> >> >>> 2159 root 20 0 0 0 0 S 2.6 0.0 776:37.81
> >> >>> kvdo0:bioQ3
> >> >>> 2160 root 20 0 0 0 0 S 2.6 0.0 974:01.15
> >> >>> kvdo0:ackQ
> >> >>>
> >> >>>
> >> >>> --
> >> >>> zhou-jielei
> >> >>>
> >> >>>
> >> >>
> >> >> Hello,
> >> >>
> >> >> In your "top" output, there is a process "kworker/5:0H" with a status
> >> >> of
> >> >> "D" (uninterruptible sleep). If you send "w" to /proc/sysrq-trigger,
> >> >> it
> >> >> will show the "SysRq : Show Blocked State" output to print a call
> trace
> >> >> on
> >> >> the blocked task.
> >> >>
> >> >> (There may already be a call trace in a "task ... blocked for more
> than
> >> >> ... seconds" message in the kernel messages for this system.)
> >> >>
> >> >> However, as noted in your github issue, you set the "discards_limit"
> >> >> for
> >> >> this VDO volume (in the sysfs setting
> >> >> /sys/kvdo/<vdoname>/discards_limit)
> >> >> from the default of 1500 to 100. While this will allow for more I/O
> >> >> operations for non-discard operations, it will result in slower
> discard
> >> >> performance.
> >> >>
> >> >> You can adjust the "discards_limit" setting while the VDO volume is
> >> >> running. If you set it back to 1500, does the discard performance
> >> >> improve?
> >> >>
> >> >> Additionally, I posted a vdo-devel list email on different strategies
> >> >> for
> >> >> discards [1]. Depending on the level of filesystem activity (and the
> >> >> size
> >> >> and/or frequency of the files being discarded), it may be better to
> use
> >> >> a
> >> >> "scheduled fstrim" approach, instead of "mount -o discard".
> >> >>
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Bryan Gurney
> >> >> Senior Software Engineer, VDO
> >> >> Red Hat
> >> >>
> >> >> [1]
> >> >>
> https://www.redhat.com/archives/vdo-devel/2018-February/msg00009.html
> >> >
> >> >
> >> >
> >> > --
> >> > 周劼磊
> >> >
> >
> >
> >
> > --
> > 周劼磊
> >
>
--
周劼磊
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vdo-devel/attachments/20181101/6da2266b/attachment.htm>
More information about the vdo-devel
mailing list