<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="ProgId" content="Word.Document"> <meta name="Generator" content="Microsoft Word 14"> <meta name="Originator" content="Microsoft Word 14"> <link rel="File-List" href="cid:filelist.xml@01CFD752.35ED7BC0"><link rel="Edit-Time-Data" href="cid:editdata.mso"><style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="tab-interval:.5in"> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";mso-bidi-font-family:"Times New Roman";color:#1F497D">I think the reason behind this is <span class="SpellE">read_ahead_kb</span> which is set to 128 by default. Can you try passing “<span class="SpellE">iflag</span>=direct” for <span class="SpellE">dd</span> command and verify.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";mso-bidi-font-family:"Times New Roman";color:#1F497D">With <span class="SpellE">sd</span> device prefetching might be causing for better performance. As you are doing sequential I/O, with round-robin path selector, alternate<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";mso-bidi-font-family:"Times New Roman";color:#1F497D">path will be selected for each I/O ( <span class="SpellE">rr_min_io_rq</span> set to 1 ) and prefetching will not help. This is due to the nature of this test ( 128K I/O ), you are seeing this.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";mso-bidi-font-family:"Times New Roman";color:#1F497D">In real scenario, there will be an overlap of I/O’s due to multiple instances of I/O threads and will benefit from read prefetching on <span class="SpellE">dm</span> device.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";mso-bidi-font-family:"Times New Roman";color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";mso-bidi-font-family:"Times New Roman";color:#1F497D">-Shiva<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";mso-bidi-font-family:"Times New Roman";color:#1F497D"><o:p> </o:p></span></p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-font-family:"Times New Roman"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-font-family:"Times New Roman""> dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] <b>On Behalf Of </b>Ali Poursamadi<br> <b>Sent:</b> Tuesday, September 23, 2014 2:57 PM<br> <b>To:</b> device-mapper development<br> <b>Subject:</b> Re: [dm-devel] DM-Multipath limiting IO to 127.000 MB/Sec<o:p></o:p></span></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <div> <div> <div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Thanks ShivaKrishna,<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">max_sectors_kb , max_segments, max_segment_size is identical on both sdx-s and the relevan dm-x device and are as follows : <o:p></o:p></p> </div> <p class="MsoNormal">max_sectors_kb : 512<br> max_segments : 1024<br> max_segment_size : 65536<o:p></o:p></p> </div> <div> <p class="MsoNormal"><br> I did the iostat tests, turns out that average Request Size show close readings ( 244,251 for dm-x, sdx respectively ) but the Average Queue Size is close to 1.9 for sdx vs 1.1 for dm-x and also the device utilization (%util) is ~99% on sd-x vs ~60% on dm-xx. It seems that dm multipath is under-utilizing the underlaying devices.<o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-bottom:12.0pt">is there a way to change this behavior ? <o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-bottom:12.0pt">-Ali Poursamadi<o:p></o:p></p> </div> </div> <div> <div> <div> <div> <div> <p class="MsoNormal" style="margin-bottom:12.0pt"><br style="mso-special-character:line-break"> <![if !supportLineBreakNewLine]><br style="mso-special-character:line-break"> <![endif]><o:p></o:p></p> </div> </div> </div> </div> </div> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal">On Tue, Sep 23, 2014 at 7:45 AM, Merla, ShivaKrishna <<a href="mailto:ShivaKrishna.Merla@netapp.com" target="_blank">ShivaKrishna.Merla@netapp.com</a>> wrote:<o:p></o:p></p> <div> <div> <p class="MsoNormal" style="margin-bottom:12.0pt">> Hi Gurus,<br> <br> > I'm experiencing an issue with DM-Multipath on Centos 6.5.<br> > DD reads 2.5 times faster when reading from a device directly compared to Multipath-ed device.<br> > I'm using device-mapper-multipath-0.4.9-72.el6_5.2.x86_64.<br> > Apparently Multipath is limiting throughput to 127.000 MB/s even though the underlying device is capable of more throughput which to me it looks like a software imposed limit. <br> <br> > [root@JS2 ~]# multipath -l mpathaf<br> > mpathaf (--------) dm-89 Rorke,G4S-16L-4F8<br> > size=7.3T features='0' hwhandler='0' wp=rw<br> > `-+- policy='round-robin 0' prio=0 status=active<br> > |- 0:0:0:1 sdb 8:16 active undef running<br> > |- 0:0:10:2 sdbb 67:80 active undef running<br> > |- 1:0:11:1 sddl 71:48 active undef running<br> > `- 1:0:12:2 sddp 71:112 active undef running<br> <br> > [root@JS2 ~]#dd if=/dev/sddl of=/dev/null bs=128k count=32000 skip=14000<br> > 32000+0 records in<br> > 32000+0 records out<br> > <a href="tel:4194304000">4194304000</a> bytes (4.2 GB) copied, 11.8166 s, 355 MB/s<br> <br> > [root@JS2 ~]# dd if=/dev/dm-89 of=/dev/null bs=128k count=32000 skip=84000 # skipping 2GB array cache data <br> > 32000+0 records in<br> > 32000+0 records out<br> > <a href="tel:4194304000">4194304000</a> bytes (4.2 GB) copied, 31.4959 s, 133 MB/s<br> <br> <br> > This is on a machine connected over 2x8Gb/s links to a fiber switch and then to an HDX4 disk array, exporting the volume to 2 of it's 4 8Gbps fiber ports . <br> <br> > Multipath.conf looks like this : <br> ><br> > defaults {<br> > user_friendly_names yes<br> > }<br> <br> > devices {<br> ><br> > device { # normal HDX4 Disk arrays<br> > vendor "Rorke"<br> > product "G4S-16L-4F8"<br> > path_grouping_policy multibus<br> > }<br> ><br> > }<br> <br> > blacklist {<br> > #blacklisting almost everything<br> > devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"<br> > devnode "^hd[a-z][0-9]*"<br> > }<o:p></o:p></p> </div> </div> <p class="MsoNormal">Check the request queue parameters for both dm and sd devices. Parameters that could affect this are<br> <br> #cat /sys/block/<dm-x>/queue/max_sectors_kb<br> #cat /sys/block/<dm-x>/queue/max_segments<br> #cat /sys/block/<dm-x>/queue/max_segment_size<br> <br> #cat /sys/block/<sdx>/queue/max_sectors_kb<br> #cat /sys/block/<sdx>/queue/max_segments<br> #cat /sys/block/<sdx>/queue/max_segment_size<br> <br> You can monitor the average request size(avgrq-sz) on both dm and sd device using<br> #iostat –xm –d /dev/dm-x /dev/sdx –t 2<br> <br> Compare this with the request size while running I/O only sd device directly. There was a patch to fix<br> block stacking limits on dm devices upstream.<br> <br> <br> <br> <br> <br> --<br> dm-devel mailing list<br> <a href="mailto:dm-devel@redhat.com">dm-devel@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/dm-devel" target="_blank">https://www.redhat.com/mailman/listinfo/dm-devel</a><o:p></o:p></p> </div> <p class="MsoNormal"><br> <br clear="all"> <br> -- <o:p></o:p></p> <div> <p class="MsoNormal"><o:p> </o:p></p> <table class="MsoNormalTable" border="0" cellpadding="0" style="mso-cellspacing:1.5pt;mso-yfti-tbllook:1184"> <tbody> <tr style="mso-yfti-irow:0;mso-yfti-firstrow:yes;mso-yfti-lastrow:yes"> <td style="padding:.75pt .75pt .75pt .75pt"></td> </tr> </tbody> </table> <p class="MsoNormal"><span style="display:none;mso-hide:all"><o:p> </o:p></span></p> <table class="MsoNormalTable" border="0" cellpadding="0" style="mso-cellspacing:1.5pt;mso-yfti-tbllook:1184"> <tbody> <tr style="mso-yfti-irow:0;mso-yfti-firstrow:yes;mso-yfti-lastrow:yes"> <td valign="top" style="padding:.75pt .75pt .75pt .75pt"> <p><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><a href="http://firstperson.is/" target="_blank"><span style="border:none windowtext 1.0pt;mso-border-alt:none windowtext 0in;padding:0in;text-decoration:none;text-underline:none"><img border="0" id="_x0000_i1025" src="http://westernized.com/email_assets/fp_email_sig-fp_logo-2x.png" alt="First Person Logo"></span></a><o:p></o:p></span></p> </td> <td valign="top" style="padding:.75pt .75pt .75pt .75pt"> <p><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><img border="0" id="_x0000_i1026" src="http://westernized.com/email_assets/signatures/fp_email_sig-ali_poursamadi-2x.png" alt="Signature"><o:p></o:p></span></p> <p><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><a href="https://www.facebook.com/pages/First-Person/677554662257977" target="_blank"><span style="border:none windowtext 1.0pt;mso-border-alt:none windowtext 0in;padding:0in;text-decoration:none;text-underline:none"><img border="0" id="_x0000_i1027" src="http://westernized.com/email_assets/fp_email_sig-facebook-2x.png" alt="Facebook"> </span></a><a href="http://www.linkedin.com/company/first-person" target="_blank"><span style="border:none windowtext 1.0pt;mso-border-alt:none windowtext 0in;padding:0in;text-decoration:none;text-underline:none"><img border="0" id="_x0000_i1028" src="http://westernized.com/email_assets/fp_email_sig-linkedin-2x.png" alt="LinkedIn"> </span></a><a href="https://twitter.com/FirstPersonsf" target="_blank"><span style="border:none windowtext 1.0pt;mso-border-alt:none windowtext 0in;padding:0in;text-decoration:none;text-underline:none"><img border="0" id="_x0000_i1029" src="http://westernized.com/email_assets/fp_email_sig-twitter-2x.png" alt="Twitter"></span></a><o:p></o:p></span></p> </td> </tr> </tbody> </table> <p class="MsoNormal"><o:p> </o:p></p> </div> </div> </div> </div> </body> </html>