<div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 24, 2019 at 9:08 AM Ryan Norwood <<a href="mailto:ryan.p.norwood@gmail.com">ryan.p.norwood@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Thank you for your help.<div><br></div><div>You are correct, it appears that the problem occurs when there is a RAID 5 or RAID 50 volume beneath VDO.</div><div><br></div><div>NAME KNAME RA SIZE ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC RQ-SIZE SCHED WSAME<br></div><div><div><font face="monospace, monospace">sdh sdh 128 977.5G 0 512 0 512 512 128 deadline 0B</font></div><div><font face="monospace, monospace">└─sed6 dm-6 128 977.5G 0 512 0 512 512 128 0B</font></div><div><font face="monospace, monospace"> └─md127 md127 12288 5.7T 0 1048576 6291456 512 512 128 0B</font></div><div><font face="monospace, monospace"> └─vdo_data dm-17 128 5.7T 0 1048576 6291456 512 512 128 0B</font></div><div><font face="monospace, monospace"> └─vdo dm-18 128 57.3T 0 4096 4096 4096 4096 128 0B</font></div></div><div><font face="monospace, monospace"><br></font></div><div><div><div><b>/sys/block/md126/queue/max_hw_sectors_kb:2147483647</b></div><div>/sys/block/md126/queue/max_integrity_segments:0</div><div><b>/sys/block/md126/queue/max_sectors_kb:512</b></div><div>/sys/block/md126/queue/max_segments:64</div><div>/sys/block/md126/queue/max_segment_size:4096</div></div><div><br></div><div><b>/sys/block/dm-17/queue/max_hw_sectors_kb:512</b></div><div>/sys/block/dm-17/queue/max_integrity_segments:0</div><div><b>/sys/block/dm-17/queue/max_sectors_kb:512</b></div><div>/sys/block/dm-17/queue/max_segments:64</div><div>/sys/block/dm-17/queue/max_segment_size:4096</div><div><br></div><div><b>/sys/block/dm-18/queue/max_hw_sectors_kb:4</b></div><div>/sys/block/dm-18/queue/max_integrity_segments:0</div><div><b>/sys/block/dm-18/queue/max_sectors_kb:4</b></div><div>/sys/block/dm-18/queue/max_segments:64</div><div>/sys/block/dm-18/queue/max_segment_size:4096</div></div><div><br></div><div>NAME KNAME RA SIZE ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC RQ-SIZE SCHED WSAME<br></div><div><div><font face="monospace, monospace">sdq sdq 128 977.5G 0 512 0 512 512 128 deadline 0B</font></div><div><font face="monospace, monospace">└─sed15 dm-15 128 977.5G 0 512 0 512 512 128 0B</font></div><div><font face="monospace, monospace"> └─vdo dm-16 128 57.3T 0 4096 4096 4096 4096 128 0B</font></div></div><div><font face="monospace, monospace"><br></font></div><div><div><div><font face="arial, helvetica, sans-serif"><b>/sys/block/sdq/queue/max_hw_sectors_kb:256</b></font></div><div><font face="arial, helvetica, sans-serif">/sys/block/sdq/queue/max_integrity_segments:0</font></div><div><font face="arial, helvetica, sans-serif"><b>/sys/block/sdq/queue/max_sectors_kb:256</b></font></div><div><font face="arial, helvetica, sans-serif">/sys/block/sdq/queue/max_segments:64</font></div><div><font face="arial, helvetica, sans-serif">/sys/block/sdq/queue/max_segment_size:65536</font></div></div><div><div><font face="arial, helvetica, sans-serif"><br class="gmail-m_7391193115041869426gmail-Apple-interchange-newline"><b>/sys/block/dm-15/queue/max_hw_sectors_kb:256</b></font></div><div><font face="arial, helvetica, sans-serif">/sys/block/dm-15/queue/max_integrity_segments:0</font></div><div><font face="arial, helvetica, sans-serif"><b>/sys/block/dm-15/queue/max_sectors_kb:256</b></font></div><div><font face="arial, helvetica, sans-serif">/sys/block/dm-15/queue/max_segments:64</font></div><div><font face="arial, helvetica, sans-serif">/sys/block/dm-15/queue/max_segment_size:4096</font></div></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif"><b>/sys/block/dm-16/queue/max_hw_sectors_kb:256</b></font></div><div><font face="arial, helvetica, sans-serif">/sys/block/dm-16/queue/max_integrity_segments:0</font></div><div><font face="arial, helvetica, sans-serif"><b>/sys/block/dm-16/queue/max_sectors_kb:256</b></font></div><div><font face="arial, helvetica, sans-serif">/sys/block/dm-16/queue/max_segments:64</font></div><div><font face="arial, helvetica, sans-serif">/sys/block/dm-16/queue/max_segment_size:4096</font></div><div><br></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><br></div><div style="font-family:monospace,monospace"><br></div></div><div><br></div><div><br></div><div><br></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 23, 2019 at 9:11 PM Sweet Tea Dorminy <<a href="mailto:sweettea@redhat.com" target="_blank">sweettea@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">One piece of this that I'm not following:<br> <br> > Now fast forward to VDO. Normally the IO size is determined by the max_sectors_kb setting in /sys/block/DEVICE/queue. This value is inherited for stacked DM devices and can be modified by the user up to the hardware limit max_hw_sectors_kb, which also appears to be inherited for stacked DM devices. VDO sets this value to 4k which in turn forces all layers stacked above it to also have a 4k maximum. If you take my previous example but place VDO beneath the dm-thin volume, all IO sequential or otherwise will be split down to 4k which will completely eliminate all the performance optimizations that dm-thin provides.<br> <br> I am unable to find a place that VDO is setting max_sectors, and<br> indeed I cannot reproduce this -- I stack VDO atop various disks of<br> max_hw_sectors_kb of 256, 512, or 1280, and VDO reports max_sectors_kb<br> of [underlying max_hw_sectors_kb]. I'm suspicious that it's some other<br> setting that is going wonky... can you recheck whether max_sectors_kb<br> is changing between (device under VDO) and (VDO device)?<br> </blockquote></div> </blockquote></div>