[dm-devel] dm-cache: ensuring cache is flushed in timely manner

[Thanos Makatos]

Occasionally I need to flush a writeback cache in order to execute
some other operation (e.g. take a back up). I do this by first setting
the policy to cleaner, then wait for the number of dirty blocks to
reach 0, and finally set the mode to writethrough. After that it is
safe to execute the operation I want. Once I'm finished I set the
policy to mq/smq and mode to writeback, job done. (Just a sanity check
here, is my flush procedure correct?)

The problem is that when I/O is particularly heavy (e.g. lots of
concurrent random writes) and the backing device is too slow (e.g. the
HDD is some remote, virtualised block device), the number of dirty
blocks not only fails to decrease, it even increases from time to
time. This is happening on CentOS 3.18.44-20.el6.x86_64.

I was able to locally reproduce this by priming the cache, setting
migration_threshold to 0, dirtying it, rate-limiting the backing
device (e.g. to 1 MB/s), running fio with random writes and a queue
depth of 32, and then finally restoring migration_threshold and
setting the policy to cleaner. The number of dirty blocks generally
fluctuates but never seems to reach 0.

I understand that I can solve the problem by rate-limiting the cache
target such that there are more I/O resources for the flush to work
with, but AFAIK the only way to do this is via cgroups and it requires
an absolute number (e.g. MB/s or IOPS). Providing such an absolute
number doesn't seem attractive as it always boils down to how heavily
the cache target is used vs. the backing device I/O performance.

Ideally I'd like to use something like ionice where I could prioritise
flushing (even slightly) over serving regular I/O, but that doesn't
look like an option here as I don't see separate threads for handling
regular I/O and flushing I/O.

Any ideas how I could solve this problem?
-- 
Thanos Makatos