[linux-lvm] dm-cache / lvmcache - no cleanup for dirty data in writeback/cleaner mode
coolthecold at gmail.com
Mon Jun 6 17:33:17 UTC 2022
Main issue as I see it: dirty data size grows, cached data size
declines, no automagic flushing happens (or much slower than growing
Sub issue: cleaner policy flushes data back to origin device very slowly.
LVM version: 2.03.07(2) (2019-11-30)
Kernel 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022
x86_64 x86_64 x86_64 GNU/Linux
Sizes: 2TB origin/700GB cache device
[cachedata_cpool] vg0 Cwi---C--- 700.00g
99.99 21.21 72.70
[cachedata_cpool_cdata] vg0 Cwi-ao---- 700.00g
[cachedata_cpool_cmeta] vg0 ewi-ao---- 20.00m
Let's start from cleaner mode, as it what is happening on server right now:
root at TasteWP2:~# lvs -o+cache_policy,cache_settings
LV VG Attr LSize Pool Origin Data%
Meta% Move Log Cpy%Sync Convert CachePolicy CacheSettings
data vg0 Cwi-aoC--- 2.00t [cachedata_cpool] [data_corig] 99.99
21.21 72.70 cleaner
and Cpy%Sync moving down very slowly - over 4 hours, it moved from
72.74% to 72.70% only.
The origin device (a couple of HDDs in mdadm RAID1) is almost idling,
5-15% busy based on iostat.
graphical representation from collectd/rrd - https://snipboard.io/WNdGhz.jpg
What I've tried to do: play around setting migration_threshhold, but
see no changes
lvchange --cachesettings 'migration_threshold=65536' vg0/data
lvchange --cachesettings 'migration_threshold=8192' vg0/data
I think this subissue leads to the main issue with dirty growing up,
not leaving space for "real" data.
My very limited knowledge, checking through history on github
back to 2019 (time of kernel release)
have not provided any hints on fixes/changes added here, so I'm bit
lost. Help is appreciated.
More information about the linux-lvm