[dm-devel] bcache gets stuck flushing writeback cache when used in combination with LUKS/dm-crypt and non-default bucket size

['Eric Wheeler']

On Fri, 20 May 2016, James Johnston wrote:

> > On Mon, 16 May 2016, Tim Small wrote:
> > 
> > > On 08/05/16 19:39, James Johnston wrote:
> > > > I've run into a problem where the bcache writeback cache can't be flushed to
> > > > disk when the backing device is a LUKS / dm-crypt device and the cache set has
> > > > a non-default bucket size.  Basically, only a few megabytes will be flushed to
> > > > disk, and then it gets stuck.  Stuck means that the bcache writeback task
> > > > thrashes the disk by constantly reading hundreds of MB/second from the cache set
> > > > in an infinite loop, while not actually progressing (dirty_data never decreases
> > > > beyond a certain point).
> > >
> > > > [...]
> > >
> > > > The situation is basically unrecoverable as far as I can tell: if you attempt
> > > > to detach the cache set then the cache set disk gets thrashed extra-hard
> > > > forever, and it's impossible to actually get the cache set detached.  The only
> > > > solution seems to be to back up the data and destroy the volume...
> > >
> > > You can boot an older kernel to flush the device without destroying it
> > > (I'm guessing that's because older kernels split down the big requests
> > > which are failing on the 4.4 kernel).  Once flushed you could put the
> > > cache into writethrough mode, or use a smaller bucket size.
> > 
> > Indeed, can someone test 4.1.y and see if the problem persists with a 2M
> > bucket size?  (If someone has already tested 4.1, then appologies as I've
> > not yet seen that report.)
> > 
> > If 4.1 works, then I think a bisect is in order.  Such a bisect would at
> > least highlight the problem and might indicate a (hopefully trivial) fix.
> 
> To help narrow this down, I tested the following generic pre-compiled mainline kernels
> on Ubuntu 15.10:
> 
>  * WORKS:  http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3.6-wily/
>  * DOES NOT WORK:  http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4-rc1+cod1-wily/
> 
> I also tried the default & latest distribution-provided 4.2 kernel.  It worked.
> This one also worked:
> 
>  * WORKS:  http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.2.8-wily/
> 
> So it seems to me that it is a regression from 4.3.6 kernel to any 4.4 kernel.  That
> should help save time with bisection...

Below is the patchlist for md and block that might help with a place to 
start.  Are there any other places in the Linux tree where we should watch 
for changes?

I'm wondering if it might be in dm-4.4-changes since this is dm-crypt
related, but it could be ac322de which was quite large.

James or Tim,

Can you try building ac322de?  If that produces the problem, then there 
are only 3 more to try (unless this was actually a problem in 4.3 which 
was fixed in 4.3.y, but hopefully that isn't so). 

ccf21b6 is probably the next to test to rule out neil's big md patch, 
which Linus abreviated in the commit log so it must be quite long.  OTOH, 
if dm-4.4-changes works, then I'm not sure what commit might produce the 
problem because the rest are not obviously relevant to the issue that are 
more recent.  

-Eric

]# git log --oneline v4.3~1..v4.4-rc1 drivers/md/ block/ Makefile | egrep -v 'md-cluster|raid5|blk-mq'

 8005c49 Linux 4.4-rc1
 ccc2600 block: fix blk-core.c kernel-doc warning
 c34e6e0 Merge branch 'kbuild' of git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild
 3419b45 Merge branch 'for-4.4/io-poll' of git://git.kernel.dk/linux-block
 3934bbc Merge tag 'md/4.4-rc0-fix' of git://neil.brown.name/md
 ad804a0 Merge branch 'akpm' (patches from Andrew)
 75021d2 Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
 05229be block: add block polling support
 dece163 block: change ->make_request_fn() and users to return a queue cookie
 8639b46 pidns: fix set/getpriority and ioprio_set/get in PRIO_USER mode
 71baba4 mm, page_alloc: rename __GFP_WAIT to __GFP_RECLAIM
 d0164ad mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd
 8d090f4 bcache: Really show state of work pending bit
 933425fb Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
 5ebe0ee Merge tag 'docs-for-linus' of git://git.lwn.net/linux
 69234ac Merge branch 'for-4.4' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
 e0700ce Merge tag 'dm-4.4-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm
 ac322de Merge tag 'md/4.4' of git://neil.brown.name/md
 ccf21b6 Merge branch 'for-4.4/reservations' of git://git.kernel.dk/linux-block
 527d152 Merge branch 'for-4.4/integrity' of git://git.kernel.dk/linux-block
 d9734e0 Merge branch 'for-4.4/core' of git://git.kernel.dk/linux-block
 6a13feb Linux 4.3


--
Eric Wheeler



> 
> James
> 
> 
>