<div dir="ltr">Reply to self:<div><br></div><div>Could it be explained by the difference in max_segments between the different devices and the dm device?</div><div>Sounds like <a href="https://bugzilla.redhat.com/show_bug.cgi?id=755046" style="font-family:Calibri,sans-serif">https://bugzilla.redhat.com/show_bug.cgi?id=755046</a> which is supposed to be fixed in 6.4, I reckon:</div>
<div><p></p>
<p> </p>
<p>3514f0c5615a00003 dm-3 XtremIO,XtremApp</p>
<p>size=1.0T features='0' hwhandler='0' wp=rw</p>
<p>`-+- policy='queue-length 0' prio=1 status=active</p>
<p> |- 0:0:2:2 sdi 8:128 active ready running</p>
<p> |- 0:0:3:2 sdl 8:176 active ready running</p>
<p> |- 0:0:1:2 sdf 8:80 active ready running</p>
<p> |- 0:0:0:2 sdc 8:32 active ready running</p>
<p> |- 1:0:0:2 sds 65:32 active ready running</p>
<p> |- 1:0:3:2 sdab 65:176 active ready running</p>
<p> |- 1:0:2:2 sdy 65:128 active ready running</p>
<p> `- 1:0:1:2 sdv 65:80 active ready running</p>
<p> </p>
<p>[root@lg545 ~]# cat /sys/class/block/dm-3/queue/max_segments</p>
<p>128</p>
<p>[root@lg545 ~]# cat /sys/class/block/sdi/queue/max_segments</p>
<p>1024</p>
<p>[root@lg545 ~]# cat /sys/class/block/sdl/queue/max_segments</p>
<p>1024</p>
<p>[root@lg545 ~]# cat /sys/class/block/sdf/queue/max_segments</p>
<p>1024</p>
<p>[root@lg545 ~]# cat /sys/class/block/sdc/queue/max_segments</p>
<p>1024</p>
<p>[root@lg545 ~]# cat /sys/class/block/sds/queue/max_segments</p>
<p>1024</p>
<p>[root@lg545 ~]# cat /sys/class/block/sdab/queue/max_segments</p>
<p>1024</p>
<p>[root@lg545 ~]# cat /sys/class/block/sdy/queue/max_segments</p>
<p>1024</p>
<p>[root@lg545 ~]# cat /sys/class/block/sdv/queue/max_segments</p>
<p>1024</p></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jul 22, 2013 at 2:47 PM, Kaul <span dir="ltr"><<a href="mailto:mykaul@gmail.com" target="_blank">mykaul@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">We are seeing a substantial difference in performance when we perform a read/write to /dev/mapper/... vs. the specific device (/dev/sdXX)<div>
What can we do to further isolate the issue?</div><div><br></div>
<div>We are using CentOS 6.4, with all updates, 2 CPUs, 4 FC ports:</div><div>Here's a table comparing the results:</div><div><table border="0" cellspacing="0" cellpadding="0" style="border-collapse:collapse">
<tbody><tr style="min-height:15pt">
<td nowrap valign="bottom" style="border-style:solid solid solid none;border-top-color:windowtext;border-right-color:windowtext;border-bottom-color:windowtext;border-top-width:1pt;border-right-width:1pt;border-bottom-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style># of LUNs</span></p>
</td>
<td nowrap valign="bottom" style="border-style:solid solid solid none;border-top-color:windowtext;border-right-color:windowtext;border-bottom-color:windowtext;border-top-width:1pt;border-right-width:1pt;border-bottom-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style># of Paths per device</span></p>
</td>
<td nowrap valign="bottom" style="border-style:solid solid solid none;border-top-color:windowtext;border-right-color:windowtext;border-bottom-color:windowtext;border-top-width:1pt;border-right-width:1pt;border-bottom-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>Native Multipath Device</span></p>
</td>
<td nowrap valign="bottom" style="border-style:solid solid solid none;border-top-color:windowtext;border-right-color:windowtext;border-bottom-color:windowtext;border-top-width:1pt;border-right-width:1pt;border-bottom-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>IO Pattern</span></p>
</td>
<td nowrap valign="bottom" style="border-style:solid solid solid none;border-top-color:windowtext;border-right-color:windowtext;border-bottom-color:windowtext;border-top-width:1pt;border-right-width:1pt;border-bottom-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>IOPS</span></p>
</td>
<td nowrap valign="bottom" style="border-style:solid solid solid none;border-top-color:windowtext;border-right-color:windowtext;border-bottom-color:windowtext;border-top-width:1pt;border-right-width:1pt;border-bottom-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>Latency Micro</span></p>
</td>
<td nowrap valign="bottom" style="border-style:solid solid solid none;border-top-color:windowtext;border-right-color:windowtext;border-bottom-color:windowtext;border-top-width:1pt;border-right-width:1pt;border-bottom-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>BW KBps</span></p>
</td>
</tr>
<tr style="min-height:15pt">
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>4</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>16</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>No</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>100% Read</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>605,661.4</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>3,381</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>2,420,736</span></p>
</td>
</tr>
<tr style="min-height:15pt">
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>4</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>16</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>No</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>100% Write</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>477,515.1</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>4,288</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>1,908,736</span></p>
</td>
</tr>
<tr style="min-height:15pt">
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>8</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>16</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>No</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>100% Read</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>663,339.4</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>6,174</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>2,650,112</span></p>
</td>
</tr>
<tr style="min-height:15pt">
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>8</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>16</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>No</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>100% Write</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>536,936.9</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>7,628</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>2,146,304</span></p>
</td>
</tr>
<tr style="min-height:15pt">
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>4</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>16</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>Yes</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>100% Read</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>456,108.9</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>1,122</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>1,824,256</span></p>
</td>
</tr>
<tr style="min-height:15pt">
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>4</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>16</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>Yes</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>100% Write</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>371,665.8</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>1,377</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>1,486,336</span></p>
</td>
</tr>
<tr style="min-height:15pt">
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>8</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>16</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>Yes</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>100% Read</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>519,450.2</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>1,971</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>2,077,696</span></p>
</td>
</tr>
<tr style="min-height:15pt">
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>8</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>16</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>Yes</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p><span style>100% Write</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>448,840.4</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>2,281</span></p>
</td>
<td nowrap valign="bottom" style="border-style:none solid solid none;border-bottom-color:windowtext;border-bottom-width:1pt;border-right-color:windowtext;border-right-width:1pt;padding:0cm 5.4pt;min-height:15pt">
<p align="right" style="text-align:right"><span style>1,795,072</span></p>
</td>
</tr>
</tbody></table></div></div>
</blockquote></div><br></div>