<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style></style></head><body lang=EN-US link=blue vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Gentle reminder… I would sincerely appreciate clarification for the below.</p><p class=MsoNormal><br>Regards</p><p class=MsoNormal>LN</p><p class=MsoNormal>Sent from <a href="https://go.microsoft.com/fwlink/?LinkId=550986">Mail</a> for Windows 10</p><p class=MsoNormal><o:p> </o:p></p><div style='mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='border:none;padding:0in'><b>From: </b><a href="mailto:lns@portworx.com">Lakshmi Narasimhan Sundararajan</a><br><b>Sent: </b>Wednesday, August 7, 2019 5:44 PM<br><b>To: </b><a href="mailto:lvm-devel@redhat.com">LVM2 development</a><br><b>Subject: </b>RE: [lvm-devel] lvmcache in writeback mode gets stuckflushingdirtyblocks</p></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi Nikhil,<o:p></o:p></p><p class=MsoNormal>So far with migration_threshold set to 20480 from original 2048 has not seen this problem. I shall keep you posted on further internal testing on this.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>But I would like to understand more on the tunables we have with lvmcache. Can you please help refine the definitions and my understanding of the below.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Defaults:<o:p></o:p></p><p class=MsoNormal><span style='font-family:Consolas;color:#494E52;background:#FAFAFA'>migration_threshold 2048 <o:p></o:p></span></p><p class=MsoNormal><span style='font-family:Consolas;color:#494E52;background:#FAFAFA'>random_threshold 4 <o:p></o:p></span></p><p class=MsoNormal><span style='font-family:Consolas;color:#494E52;background:#FAFAFA'>sequential_threshold 512</span><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><ol style='margin-top:0in' start=1 type=1><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'>Migration_threshold: This tunable controls how many sectors (512B) of data are pulled in or pushed out of cache. So all flush/writeback operations from the cache device operates in multiples of this threshold. There is no migration ever in writethrough cache. Larger the number of sectors will help in moving larger context into/out of cache immediately and improve sequential performance, but shall adversely affect random performance.<o:p></o:p></li><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'>Sequential_threshold: This tunable is a count of IO requests that have to be contiguous (start from last IO end) to treat incoming IO as sequential. Each IO can be of any size. As long as the next IO is contiguous it shall get counted. All IOs only after hitting the sequential_threshold shall be bypassed from cache. Even if one IO misses the sequential pattern from last IO, the threshold gets reset to zero? And all intervening IO are cached?<o:p></o:p></li><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'>Random_threshold: This tunable is a count of IO requests that miss sequential condition to be considered as a random IO. In default condition, first 4 IO requests in the stream can never get cached. All IO between 4 and 512 requests in the stream get cached. And only after 512 requests does the caching module recognize incoming IO as sequential and stop caching further.<o:p></o:p></li></ol><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Outside of this I also see 3 other tunables.<o:p></o:p></p><p class=MsoNormal> "read_promote_adjustment",<o:p></o:p></p><p class=MsoNormal> "write_promote_adjustment",<o:p></o:p></p><p class=MsoNormal> "discard_promote_adjustment"<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>To which I do not understand how this needs to be configured.<o:p></o:p></p><p class=MsoNormal>Are there any other tunables that I am not aware of.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Can you please help clarify on the same.<o:p></o:p></p><p class=MsoNormal><br>Regards<o:p></o:p></p><p class=MsoNormal>LN<o:p></o:p></p><p class=MsoNormal>Sent from <a href="https://go.microsoft.com/fwlink/?LinkId=550986">Mail</a> for Windows 10<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From: </b><a href="mailto:nkshirsa@redhat.com">Nikhil Kshirsagar</a><br><b>Sent: </b>Monday, August 5, 2019 2:42 PM<br><b>To: </b><a href="mailto:lvm-devel@redhat.com">LVM2 development</a><br><b>Subject: </b>Re: [lvm-devel] lvmcache in writeback mode gets stuck flushingdirtyblocks<o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Can you try increasing migration threshold through the device mapper commands and check if this gets rid of the infinite flushes ?<o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Fri, 2 Aug, 2019, 5:14 PM Nikhil Kshirsagar, <<a href="mailto:nkshirsa@redhat.com">nkshirsa@redhat.com</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-left:4.8pt'>Hello,<o:p></o:p></p><div><p class=MsoNormal style='margin-left:4.8pt'><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'>You are welcome.<o:p></o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'>The migration threshold is in terms of chunks, I think.. So it should be at least one chunk so the looping forever won't happen. The bug we found was if chunksize goes beyond a certain value triggered by larger than one tb sized cached lv, it ends up with migration threshold hard coded to lower than the increased chunksize.<o:p></o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'>Yes migration threshold right now needs better documentation and explanations. Also the ability to see it from lvm commands just like we can see chunksize. We are working on it through the bzs mentioned earlier. (See the bz about migration threshold needing better documentation in the man pages)<o:p></o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'>I think right now you can get it only at the device mapper layer, will check..<o:p></o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'>Regards,<o:p></o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'>Nikhil.<o:p></o:p></p></div><div><p class=MsoNormal style='margin-left:4.8pt'><o:p> </o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:4.8pt'><o:p> </o:p></p><div><div><p class=MsoNormal style='margin-left:4.8pt'>On Fri, 2 Aug, 2019, 5:09 PM Lakshmi Narasimhan Sundararajan, <<a href="mailto:lns@portworx.com" target="_blank">lns@portworx.com</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>Hi Nikhil,<br>Thank you for your email. Much appreciated.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>In my environment, Chunksize is fixed at 1M irrespective of the pool size. This may take the number of entries over 1M and result in kernel warning. But the class of systems we are using are huge, and so the memory and cpu bottlenecks does not seem to be a factor in our testing.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>I looked up at the bugs. The first one about chunksize > 1M, we should be safe on that given our chunksize is fixed at 1MB.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>The other one about migration threshold is interesting, I will have to validate this again.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>What would be the unit of migration threshold? Is it the number of 512 byte sectors? And what exactly is its definition?<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>And also curiously this does not seem to be exported through lvm cli, need to fetch this only through dmsetup?<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>Thanks<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>LN<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>Sent from <a href="https://go.microsoft.com/fwlink/?LinkId=550986" target="_blank">Mail</a> for Windows 10<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'> <o:p></o:p></p><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'><b>From: </b><a href="mailto:nkshirsa@redhat.com" target="_blank">Nikhil Kshirsagar</a><br><b>Sent: </b>Wednesday, July 31, 2019 3:04 PM<br><b>To: </b><a href="mailto:lvm-devel@redhat.com" target="_blank">LVM2 development</a><br><b>Subject: </b>Re: [lvm-devel] lvmcache in writeback mode gets stuck flushing dirtyblocks<o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'> <o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>This used to happen if the chunksize increased as a result of needing to use more than a million chunks to store the size of the cached lv. What is the size of the pool?<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>Regards,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>Nikhil.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'> <o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'>On Tue, 30 Jul, 2019, 1:25 PM Lakshmi Narasimhan Sundararajan, <<a href="mailto:lns@portworx.com" target="_blank">lns@portworx.com</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>Hi Team,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>A very good day to all.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'><br>I am using lvmcache in writeback mode. When there are dirty blocks still in the lv, and if needs to be destroyed or flushed, then<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>It seems to me that there are some conditions under which the dirty data flush gets stuck forever.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>As an example:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>root@pdc4-sm35:~# lvremove -f pwx0/pool<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> 367 blocks must still be flushed.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> 367 blocks must still be flushed.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> 367 blocks must still be flushed.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> 367 blocks must still be flushed.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> 367 blocks must still be flushed.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> 367 blocks must still be flushed.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>^C<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>root@pdc4-sm35:~#<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>I am running these version:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>root@pdc4-sm35:~# lvm version<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> LVM version: 2.02.133(2) (2015-10-30)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> Library version: 1.02.110 (2015-10-30)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> Driver version: 4.34.0<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>root@pdc4-sm35:~#<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>This issue seems old and reported multiple places. There have been some acknowledgement that this issue is resolved in 2.02.133, but still I see it. Also, I have seen some posts report it in 2.02.170+ as well (here: <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=878441" target="_blank">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=878441</a>) (<span style='font-size:10.0pt;font-family:"Courier New";color:black;background:#FEFEFE'>Package: lvm2 Version: 2.02.173-1 Severity: normal)</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>I filed one here myself, <a href="https://github.com/lvmteam/lvm2/issues/22" target="_blank">https://github.com/lvmteam/lvm2/issues/22</a>, trying to understand from you experts where we are on this?<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>I would sincerely appreciate your help in understanding the state of this issue in more detail.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>Best regards<br>LN<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'>Sent from <a href="https://go.microsoft.com/fwlink/?LinkId=550986" target="_blank">Mail</a> for Windows 10<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:19.2pt'> <o:p></o:p></p></div></div></blockquote></div></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.2in'>--<br>lvm-devel mailing list<br><a href="mailto:lvm-devel@redhat.com" target="_blank">lvm-devel@redhat.com</a><br><a href="https://www.redhat.com/mailman/listinfo/lvm-devel" target="_blank">https://www.redhat.com/mailman/listinfo/lvm-devel</a><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:9.6pt'> <o:p></o:p></p></div></div></blockquote></div></div></blockquote></div><p class=MsoNormal style='margin-left:9.6pt'>--<br>lvm-devel mailing list<br><a href="mailto:lvm-devel@redhat.com" target="_blank">lvm-devel@redhat.com</a><br><a href="https://www.redhat.com/mailman/listinfo/lvm-devel" target="_blank">https://www.redhat.com/mailman/listinfo/lvm-devel</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>