[dm-devel] dm-cache with zero hit rate

[Mike Snitzer]

On Fri, Aug 02 2013 at  4:49am -0400,
Steinar H. Gunderson <sgunderson at bigfoot.com> wrote:

> On Wed, Jul 31, 2013 at 04:25:22PM -0400, Mike Snitzer wrote:
> >> After a fair amount of fighting with udev and initramfs-tools, plus upping
> >> the block size to 2048 since 512 complained about not enough RAM (on a 24GB
> >> machine!), this seems to boot up and work, but I seem to get absolutely zero
> >> cache hits.
> > Kudos to you for hacking that to work.
> 
> It's actually not that hard; once things stabilize and I know this is
> actually the solution I want to have, I plan to sit down and document it, and
> it's not going to be a long document.
> 
> > Please see the recent "dm-cache warming thread":
> > https://www.redhat.com/archives/dm-devel/2013-July/msg00133.html
> > 
> > That isn't to say we cannot take steps to be more aggressive; but we'll
> > need more context for what you're doing.
> > 
> > A normal system boot is likely predominantly read once, and/or (as Joe
> > pointed out) the page cache could be masking subsequent reads.
> > 
> > If you're doing write heavy workloads are they being elided by the
> > sequential_threshold?
> > 
> > Try a git checkout, and switch branches a few times (e.g. checkout v3.1,
> > then v3.8, then v3.2, then v3.9, then v3.1, etc).
> 
> OK, so context: This is my personal do-everything server, so it does all
> kinds of things; web serving (both pages, scripts and mirrors), large files,
> database work, email (local, not IMAP), backup through rsync, etc.. Although
> it has 24GB of RAM, the disk set is 8TB (and the cache disks ~450GB), so one
> would think it found _something_ relevant to cache. In particular, my
> Maildirs would seem like a prime candidate :-) There's I/O going on all the
> time, and I'm a bit surprised that not even any writes are cached. 

Yes, it is surprising.  Curious to know if the promotions aren't
happening due to the IO scheduler somehow merging all your random small
IO.  We don't yet have a descrete counter to show the number of
migrations that were skipped due to sequential_threshold but that is
something we can add.

But you can effectively disable the sequential_threshold by setting it
really high, e.g.:

dmsetup message cache 0 sequential_threshold 16384

> I did the git checkout as you asked, 10–15 times with different version
> numbers, and although the git command seemed to create I/O (ie., the page
> cache wasn't gobbling it all up, although it was mostly write I/O), but it
> still didn't put anything in the cache.

Please write a file that is smaller than your specified
sequential_threshold, and then read it numerous times via direct IO,
e.g.:

dd if=<your file> of=/dev/null iflag=direct bs=16K
 
> I see the thread you're referring to says “too low”, but I really have zero,
> even after almost 40 hours of normal operation:
> 
>   cache: 0 23440891904 cache 913/8192 0 6888144 0 1653677 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 512
> 
> Could the large block size be related to this?

No, a large cache block size just brings a lot more into the cache when
something was deemed worthy of promotion.

We need to better understand/diagnose _why_ nothing is getting promoted,
to that end Heinz and Joe will collaborate on the development of a
"cache_dump" utility to allow visibility into the mq policy's hit counts
and hints associated with each block.

But in the meantime, cranking up sequential_threshold may help.

Mike