[dm-devel] dm-cache questions

[Paul B. Henson]

I'm building a small virtualization server for which I'd like to avail of
ssd caching to increase performance. While there seems to be an increasing
plethora of options for ssd caching under linux, I'd like to stick with
something that's part of the mainline kernel, which I think restricts the
playing field down to bcache or dm-cache.

After reviewing the dm-cache documentation and mailing list archives, I had
a few questions I hope somebody might be able to answer; I apologize in
advance if any of them are silly or something I should've already found on
my own.

I've got four WD RE4 2TB drives that I plan to configure as RAID10 for the
data device, and two Samsung 840 Pro 256GB SSD's that I plan to configure as
RAID1 for the cache device. I'd like to set up write back caching to improve
both read and write performance. I was going to set up lvm on top of the
cached device and then use lv's as the backing store for kvm virtual
machines.

Is dm-cache considered ready for production deployment? From what I
understand, there are plans to add support for managing dm-cache to lvm2,
and without that it's a bit cryptic to use/set up. I see that Fedora has
deferred including support for dm-cache into their distribution pending that
lvm2 support, but other than easing configuration/management, are there any
reasons not to go ahead and deploy dm-cache in production now working with
it directly rather than through lvm2?

What is the recommended kernel version for using dm-cache? Would 3.10LTS be
suitable, or would it be better at this point to be running the latest
stable, eg 3.12.x now, and then 3.13.x once 3.12 goes EOL, to be sure to
have the latest bug fixes and performance enhancements?

>From reviewing the documentation, in addition to the origin/backing device
and the cache device, a third device is necessary for metadata. Per the
documentation the rationale for having a separate device for metadata rather
than simply using the cache device is so that the metadevice can be
configured with different redundancy; the example given is that perhaps it
could be mirrored. I'm confused though as to what utility there is an having
a metadata device with a different level of redundancy than the cache
device. If the metadata device is mirrored, and the cache device is not, you
will still be able to access the metadata should the cache device fail, but
given the cache device has failed, what are you going to do with it?
Conversely, if the cache device is mirrored, and the metadata device is not,
should the metadata device fail, how are you going to use your cache? I can
see potentially having the origin device redundant, and the cache device
not, assuming you are not using write back caching, but I don't initially
see a scenario where you would configure a cache device and a metadevice
with different availability characteristics.

What are the performance requirements of the metadevice? For my system, I
can either put it on the cache device, on the origin device, or I have
another mirror of two USB sticks used for /boot that it could go on.
Intuitively it seems the metadata device should be fast/low latency, so my
first guess would be the best location would be on the SSD mirror I'm using
for cache. Based on the examples I've seen, you can either partition the
device into two pieces to separate metadata from cache, or use dm-linear,
I'm thinking I'll go with partitioning as that seems simpler and I'm more
familiar with it, although I suppose that will result in a little bit of
waste for the partition table and alignment.

With bcache, they recommend selecting the bucket size and block size based
on the specifications of your SSD, is there any similar recommended
alignment with the underlying SSD for selecting dm-cache block size? The SSD
I am using has a 1024k erase block size and an 8k page size. Or should be
block size be tuned based more on the size of the origin device relative to
the cache device and your expected I/O sizes, with no particular regard for
the physical characteristics of your SSD ?

>From what I've read, the rule of thumb algorithm for sizing your metadata
device is 4 MB + ( 16 bytes * nr_blocks ). Is that still accurate? So, if I
hypothetically selected a 256k block size, I would calculate it as:

# blockdev --getsize64 /dev/md2          (ssd mirror)
255926140928

4194304 + (16 * 255926140928 / 262144)  = 19814796

So I would need to make a partition of size approximately 19MB for the
metadata? Then, assuming I partitioned md2 into md2p1 (metadata) and md2p2
(cache), and my origin device was md3, I could create the cache device via:

# blockdev --getsz /dev/md3 
7813531648

# dmsetup create md3-cached --table '0 7813531648 cache /dev/md2p1
/dev/md2p2 /dev/md3 512 1 writeback default 0'

For shutdown, you should then arrange to run 'dmsetup suspend md3-cached'
at reboot/halt so it goes down cleanly? From what I read, dm-cache should be
reasonably robust in the face of a crash/panic, so this is really more of an
optimization as opposed to a hard requirement?

Just a couple more miscellaneous questions :), is there any way to switch
between modes/policies without downtime on the cache device? For example, if
one of the SSD's failed and you wanted to switch to write through mode
rather than write back until you replaced it and the mirror was healthy
again?

Is there any support or integration with SSD TRIM for the cache device? Not
necessarily in real-time, as that can degrade performance, but occasionally
in batch ala fstrim for filesystems, to get dm-cache to TRIM all of the not
in use blocks at that time in order to optimize the SSD garbage collector?

If you have read this far, thank you very much :), I'm sorry for such a long
message, but I'm trying to wrap my head around this and be sure I have a
good understanding before using it.

Thanks.