[dm-devel] dm-cache - should origin device and cache be identical when quiescent?
Alex Sudakar
alex.sudakar at gmail.com
Fri Jan 23 02:14:33 UTC 2015
I'm running Centos7 - kernel 3.10.0.x86_64 - and am testing dm-cache
in a libvert/KVM virtual machine. I plan to eventually use dm-cache
with a 256GB SSD and an origin device of a 2TB md-raid1 device.
I've written a couple of test programs and am concerned to find that
they are suggesting that my test cache in my VM becomes corrupted over
time. But before I panic I would like to ask a couple of very basic
questions. I've tried to find the answers in the kernel-doc and
elsewhere but haven't so far found explanations to what I'm seeing.
I'm setting up a cache using three partitions of a 1GB test disk -
/dev/vdb - in my VM. vdb1 is the 900MB origin device, vdb2 is the
90MB cache device and vdb3 the 30MB metadata device. I set up the
cache as follows:
dd if=/dev/zero of=/dev/vdb3 bs=32k; dd if=/dev/zero of=/dev/vdb2
bs=32k; dd if=/dev/zero of=/dev/vdb1 bs=32k
dmsetup create cache --table '0 1843200 cache /dev/vdb3 /dev/vdb2
/dev/vdb1 64 1 writeback default 0
At this point the cache and the origin device all have zero bytes
content, so a simple comparison works:
cmp /dev/mapper/cache /dev/vdb1
Then I invoke my little test program which writes random sectors to
/dev/mapper/cache. After a minute I terminate the program and try
another comparison:
# cmp /dev/mapper/cache /dev/vdb1
/dev/mapper/cache /dev/vdb1 differ: byte 35841, line 1
Using 'od' I confirm that sector #70 in the origin device is still all
zeroes, whereas the sector on the cache device is correct (agreeing
with the values last written by my test program).
Even after 5 minutes a comparison still shows that the sector is full
of zeroes on the origin device.
I shift to the cleaner policy:
dmsetup suspend cache
dmsetup reload cache --table '0 1843200 cache /dev/vdb3 /dev/vdb2
/dev/vdb1 64 0 cleaner 0'
dmsetup resume cache
wait a few seconds (a script monitoring 'dmsetup status cache' shows a
small short burst of reads/writes, then no activity) and then switch
back to the default/mq policy:
dmsetup suspend cache
dmsetup reload cache --table '0 1843200 cache /dev/vdb3 /dev/vdb2
/dev/vdb1 64 1 writeback default 0'
dmsetup resume cache
But the cmp still fails, saying that same sector is different between
origin and cache devices. Even after I 'dmsetup remove cache' the
origin device's sector is still all zeroes rather than the data
written by the test program.
I'm sure I'm missing something super-basic, but I would appreciate
advice on why this discrepancy exists. I thought I'd read that dirty
blocks are ultimately written out in a matter of seconds to the origin
device. The kernel-doc says that the cleaner 'writes back all dirty
blocks in a cache to decommission it'. I'm at a loss as to why that
apparently isn't happening.
While I'm here I'd like to ask about my 'corruption' problem which I'm
still investigating. Basically I leave my test program running for a
longer time - an hour or more - and then I find that the sectors in my
*cache device* don't have the data that the test program (which writes
'sector X last written with bytes of value Y' information to a state
file) says it should have. While the test program is writing random
sectors I have a separate script which approximately every ten minutes
switches from mq to cleaner policy and then, after another period
(random 0 - 10 minutes) switches back again (using the commands listed
above).
My ultimate intent is to switch the cache into using the cleaner
policy during various night-time batch runs which scan all the files
on the system (backups, tripwire, etc) and would typically 'de-prime'
the cache from the state it had acquired during my typical daytime
activities. I've done some simple tests with mq/cleaner and it
appeared to me that cleaner left the cached blocks alone - after a lot
of data reads with cleaner a switch back to mq would find that the
blocks held in the cache immediately before the move to cleaner were
still cached. Since I don't believe dm-cache has a 'blacklist pid'
facility like Flashcache (I wish it did!) I thought using cleaner
would be a the next best thing.
But in my hours-long test I experienced the corruption I mentioned
(*not* a discrepancy between origin device and cache, as per my first
question, but between the cache device and what my test program said
it had written). While I'm double-checking my test program I thought
I'd ask if my practice of switching from mq to cleaner and back again
as a regular operational procedure is supported or if I'm doing
something stupid.
Many thanks for any advice. I have tried to research
answers/solutions; if I've missed something obvious I'd appreciate any
references.
Thanks!
More information about the dm-devel
mailing list