[dm-devel] slab-nomerge (was Re: [git pull] device mapper changes for 4.3)

[Dave Chinner]

On Thu, Sep 03, 2015 at 08:02:40AM -0700, Linus Torvalds wrote:
> On Wed, Sep 2, 2015 at 11:02 PM, Dave Chinner <dchinner at redhat.com> wrote:
> > On Wed, Sep 02, 2015 at 06:21:02PM -0700, Linus Torvalds wrote:
> > Right, it's not xyzzy-specific where 'xyzzy' is a subsystem. The
> > flag application is actually *object specific*. That is, the use of
> > the individual objects that determines whether it should be merged
> > or not.
> 
> Yes.
> 
> I do agree that something like SLAB_NO_MERGE can make sense on an
> actual object-specific level, if you have very specific allocation
> pattern knowledge and can show that the merging actually hurts.

There are generic cases where it hurts, so no justification should
be needed for those cases...

> > e.g. Slab fragmentation levels are affected more than anything by
> > mixing objects with different life times in the same slab.  i.e. if
> > we free all the short lived objects from a page but there is one
> > long lived object on the page then that page is pinned and we free
> > no memory. Do that to enough pages in the slab, and we end up with a
> > badly fragmented slab.
> 
> The thing is, *if* you can show that kind of behavior for a particular
> slab, and have numbers for it, then mark that slab as no-merge, and
> document why you did it.

The double standard is the problem here. No notification, proof,
discussion or review was needed to turn on slab merging for
everyone, but you're setting a very high bar to jump if anyone wants
to turn it off in their code.

> And quite frankly, I don't actually think you have the numbers to show
> that theoretical bad behavior.

I don't keep numbers close handy. I've been dealing with these
problems for ten years, to I just know what workloads demonstrate
this "theoretical bad behaviour" within specific slabs and test them
when relevant. I'll do a couple of quick "merging is better"
verification tests this afternoon, but other than that I don't have
time in the next couple of weeks...

But speaking of workloads, internal inode cache slab fragmentation
is simple to reproduce on any filesystem. XFS just happens to be the
only one that really actively manages it as a result of long term
developer awareness of the problem. I first tripped over it in early
2005 with SpecSFS, and then with other similar NFS benchmarks like
filebench.  That's where Christoph Lameter was introduced to the
problem, too:

https://lwn.net/Articles/371892/

" The problem is that sparse use of objects in slab caches can cause
large amounts of memory to become unusable. The first ideas to
address this were developed in 2005 by various people."

FYI, with appropriate manual "drop slab" hacks during the benchmark,
we could get 20-25% higher throughput from the NFS server because
dropping the entire slab cache before the measurement phase meant we
avoided the slab fragmentation issue and had ~50% more free memory
to use for the page cache during the measurement period...

Similar problems have been reported over the years by users with
backup programs or scripts that used find, rsync and/or 'cp -R' on
large filesystems. It used to be easy to cause these sorts of
problems in the XFS inode cache. There's quite a few other
workloads, but it easily to reproduce inode slab fragmetnation with
find, bulkstat and cp. Basically all you need to do is populate the
inode cache, randomise the LRU order, then trigger combined inode
cache and memory demand.  It's that simple.

The biggest problem with using a workload like this to "prove" that
slab merging degrades behaviour is that we don't know what slabs
have been merged. Hence it's extremely hard to generate a workload
definition that demonstrates it. Indeed, change kernel config
options, structures change size and the slab is merged with
different objects, so the workload that generates problems has to be
changed, too.  And it doesn't even need to be a kernel with a
different config - just a different set of modules loaded because
the hardware and software config is different will change what slabs
are merged.

IOWs, what produces a problem on one kernel on one machine will not
reproduce the same problem on a different kernel or machine. Numbers
are a crapshoot here, especially as the cause of the problem is
trivially easy to understand.

Linus, you always say that at some point you've just got to step
back, read the code and understand the underlying issue that is
being dealt with because some things are way too complex to
reproduce reliably. This is one of those cases - it's obvious that
slab merging does not fix or prevent internal slab cache
fragmentation and that it only serves to minimise the impact of
fragmentation by amortising it across multiple similar slabs.
Really, this is the best we can do with passive slab caches where
you can't control freeing patterns.

However, we also have actively managed slab caches, and they can and
do work to prevent fragmetnation and clear it quickly when it
happens. Merging these actively managed slabs with other passive
slab is just a bad idea because the passive slab objects can only
reduce the effectiveness of the active management algorithms. We
don't need numbers to understand this - it's clear and obvious from
an algorithmic point of view.

> In contrast, there really *are*
> numbers to show the advantages of merging.

I have never denied that. Please listen to what I'm saying.

> So the fragmentation argument has been shown to generally be in favor
> of merging, _not_ in favor of that "no-merge" behavior.

Yes, all the numbers and research I've seen has been on passive
slab cache behaviour. I *agree* that passive slab caches should be
merged, but I don't recall anyone documenting the behavioural
distinction between active/passive slabs before now, even though
it's been something I've had in my head for several years. Actively
managed slabs are very different in their behaviour to passive
slabs, and so what holds true for passive slabs is not necessarily
true for actively managed slabs.

Really, we don't need some stupidly high bar to jump over here -
whether merging should be allowed can easily be answered with a
simple question: "Does the slab have a shrinker or does it back a
mempool?" If the answer is yes then using SLAB_SHRINKER or
SLAB_MEMPOOL to trigger the no-merge case doesn't need any more
justification from subsystem maintainers at all.

Cheers,

Dave.
-- 
Dave Chinner
dchinner at redhat.com