[dm-devel] [PATCH 0/4] dm: reduce memory overhead of DM devices

[Mike Snitzer]

On Fri, Oct 03 2014 at  9:23am -0400,
Mike Snitzer <snitzer at redhat.com> wrote:

> On Fri, Oct 03 2014 at  7:48am -0400,
> Junichi Nomura <j-nomura at ce.jp.nec.com> wrote:
> 
> > This series of patches reduce the memory overhead of device-mapper
> > device mainly by removing unused mempool for bio vecs.
> > 
> > DM creates per-device bioset to ensure forward progress under
> > low memory situation and bioset always includes mempool for bvec.
> > 
> > However, with the introduction of immutable bvec by Kent Overstreet,
> > dm core now uses bio_clone_fast for creating a clone bio, and no longer
> > needs bvecs for it.
> > 
> > For example, when you create 10,000 bio-based DM devices and 1,000
> > request-based DM devices, memory usage of biovec under no load is:
> >   # grep biovec /proc/slabinfo
> >   biovec-256        418068 418068   4096  ...
> >   biovec-128             0      0   2048  ...
> >   biovec-64              0      0   1024  ...
> >   biovec-16              0      0    256  ...
> > 
> > With this patch series applied, the usage becomes:
> >   # grep biovec /proc/slabinfo
> >   biovec-256           116    116   4096  ...
> >   biovec-128             0      0   2048  ...
> >   biovec-64              0      0   1024  ...
> >   biovec-16              0      0    256  ...
> > 
> > So 4096 * (418068 - 116) = 1.6GB of memory is saved in this example.
> 
> Very nice Junichi!
> 
> Kent definitely left us some low hanging fruit that offers a huge win
> (and fixes the really long-standing problem of excessive memory reserves
> for each request-based DM device!)
> 
> Jens, this is obviously an awkward patchset given it touches both block
> and DM (block more so than DM).  I'd be fine with you pulling it into
> block (or vice-versa if you're OK with me staging it for 3.18 via
> linux-dm.git with your Acked-by).

I picked up this patchset, refactored the patches based on my earlier
suggested s/bioset_nobvec_create/bioset_create_nobvec/ rename and
tweaked headers a bit.  I've staged the result in the for-next branch of
linux-dm.git for v3.18 inclusion, see:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=for-next

b83bb63 dm: remove nr_iovecs parameter from alloc_tio()
b875215 block: use bio_clone_fast() in blk_rq_prep_clone()
cd595c9 block: add bioset_create_nobvec()
3611594 dm: use bioset_create_nobvec()

Jens, I can easily yield this patchset to you if you care.  Or I can
backfill your Acked-by in the above 2 block commits (I obviously won't
just send block changes upstream without your OK!).

Please advise, thanks!
Mike