[dm-devel] dm-thin: Several Questions on dm-thin performance.

[Eric Wheeler]

On Mon, 2 Dec 2019, JeffleXu wrote:
> 在 2019/11/23 上午2:55, Joe Thornber 写道:
> 
> > On Fri, Nov 22, 2019 at 11:14:15AM +0800, JeffleXu wrote:
> >
> > > The first question is what's the purpose of data cell? In thin_bio_map(),
> > > normal bio will be packed as a virtual cell and data cell. I can
> > > understand
> > > that virtual cell is used to prevent discard bio and non-discard bio
> > > targeting the same block from being processed at the same time. I find it
> > > was added in commit     e8088073c9610af017fd47fddd104a2c3afb32e8 (dm thin:
> > > fix race between simultaneous io and discards to same block), but I'm
> > > still
> > > confused about the use of data cell.
> > As you are aware there are two address spaces for the locks.  The 'virtual'
> > one
> > refers to cells in the logical address space of the thin devices, and the
> > 'data' one
> > refers to the underlying data device.  There are certain conditions where we
> > unfortunately need to hold both of these (eg, to prevent a data block being
> > reprovisioned
> > before an io to it has completed).
> >
> > > The second question is the impact of virtual cell and data cell on IO
> > > performance. If $data_block_size is large for example 1G, in multithread
> > > fio
> > > test, most bio will be buffered in cell->bios list and then be processed
> > > by
> > > worker thread asynchronously, even when there's no discard bio. Thus the
> > > original parallel IO is processed by worker thread serially now. As the
> > > number of fio test threads increase, the single worker thread can easily
> > > get
> > > CPU 100%, and thus become the bottleneck of the performance since dm-thin
> > > workqueue is ordered unbound.
> > Yep, this is a big issue.  Take a look at dm-bio-prison-v2.h, this is the
> > new interface that we need to move dm-thin across to use (dm-cache already
> > uses it).
> > It allows concurrent holders of a cell (ie, read locks), so we'll be able to
> > remap
> > much more io without handing it off to a worker thread.  Once this is done I
> > want
> > to add an extra field to cells that will cache the mapping, this way if you
> > acquire a
> > cell that is already held then you can avoid the expensive btree lookup.
> > Together
> > these changes should make a huge difference to the performance.
> >
> > If you've got some spare coding cycles I'd love some help with this ;)

Hi Joe,

I'm not sure if I will have the time but thought I would start the 
research and ask a few questions. I looked at the v1/v2 .h files and some 
of the functions just change suffix to _v2 and maybe calling 
convention/structure field changes.

However, there appear to be some design changes, too:

* dm_deferred_set - These appear to be used a bit in dm-thin.c.  
The dm_deferred_set calls don't seem to reference anything prison-related, 
but they are defined in dm-bio-prison-v1.h.  Can you provide direction on 
how these would be refactored, or if they can just remain as-is?
  Call counts in dm-thin.c:
      2 dm_deferred_entry_dec
      2 dm_deferred_set_create
      3 dm_deferred_entry_inc
      3 dm_deferred_set_add_work
      4 dm_deferred_set_destroy

* dm_bio_detain - is this replaced by dm_cell_get_v2?
	- It looks like dm_bio_detain() returns 1 if already held, but 
	  dm_cell_get_v2() returns true if the lock is granted.  How might 
	  this be handled?
	- What are the lock_levels?
	- What in dm-thin.c would then call dm_cell_put_v2?

* dm_cell_release(_no_holder) - is this replaced by dm_cell_unlock_v2?
	- How would the _no_holder version be refactored?

* dm_cell_visit_release - This function uses a callback, but none of the 
v2 functions have such a callback.  Do we need to write a helper function 
with get/unlock(?) around the cell?


* dm_cell_error - no equivalent v2 implementation.  Suggestions?


What other considerations might there be in the v2 port?

Thanks!

--
Eric Wheeler


> >
> > - Joe
> >
> > --
> > dm-devel mailing list
> > dm-devel at redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> 
> 
> --
> dm-devel mailing list
> dm-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>