[dm-devel] [Multipath] Round-robin performance limit

[Adam Chasen]

Malahal,
After your mentioning bio vs request based I attempted to determine if
my kernel contains the request based mpath. It seems in 2.6.31 all
mpath was switched to request based. I have a kernel 2.6.31+ (actually
.35 and .38), so I believe I have requrest-based mpath.

All,
There also appears to be a new multipath configuration option
documented in the RHEL 6 beta documentation:
rr_min_io_rq 	Specifies the number of I/O requests to route to a path
before switching to the next path in the current path group, using
request-based device-mapper-multipath. This setting should be used on
systems running current kernels. On systems running kernels older than
2.6.31, use rr_min_io. The default value is 1.

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6-Beta/html/DM_Multipath/config_file_multipath.html

I have not tested using this setting vs rr_min_io yet or even if my
system supports the configuration directive.

If I trust some of the claims of several VMware ESX iscsi multipath
setups, it is possible (possibly using different software) to gain a
multiplicative throughput by adding additional Ethernet links. This
makes me hopeful that we can do this with open-iscsi and dm-mulitpath
as well.

It could be something obvious I am missing, but it appears a lot of
people experience this same issue.

Thanks,
Adam

On Tue, May 3, 2011 at 6:12 AM, John A. Sullivan III
<jsullivan at opensourcedevel.com> wrote:
> On Mon, 2011-05-02 at 22:04 -0700, Malahal Naineni wrote:
>> John A. Sullivan III [jsullivan at opensourcedevel.com] wrote:
>> > I'm also very curious about your findings on rr_min_io.  I cannot find
>> > my benchmarks but we tested various settings heavily.  I do not recall
>> > if we saw more even scaling with 10 or 100.  I remember being surprised
>> > that performance with it set to 1 was poor.  I would have thought that,
>> > in a bonded environment, changing paths per iSCSI command would give
>> > optimal performance.  Can anyone explain why it does not?
>>
>> rr_min_io of 1 will give poor performance if your multipath kernel
>> module doesn't support request based multipath. In those BIO based
>> multipath, multipath receives 4KB requests. Such requests can't be
>> coalesced if they are sent on different paths.
> <snip>
> Ah, that makes perfect sense and why 3 seems to be the magic number in
> Linux (4000 / 1460 (or whatever IP payload is)).  Does that change with
> Jumbo frames? In fact, how would that be optimized in Linux?
>
> 9KB seems to be a reasonable common jumbo frame value for various
> vendors and that should contain two pages but, I would guess, Linux
> can't utilize it as each block must be independently acknowledged. Is
> that correct? Thus a frame size of a little over 4KB would be optimal
> for Linux?
>
> Would that mean that rr_min_io of 1 would become optimal? However, if
> each block needs to be acknowledged before the next is sent, I would
> think we are still latency bound, i.e., even if I can send four requests
> down four separate paths, I cannot send the second until the first has
> been acknowledged and since I can easily place four packets on the same
> path within the latency period of four packets, multibus gives me
> absolutely no performance advantage for a single iSCSI stream and only
> proves useful as I start multiplexing multiple iSCSI streams.
>
> Is that analysis correct? If so, what constitutes a separate iSCSI
> stream? Are two separate file requests from the same file systems to the
> same iSCSI device considered two iSCSI streams and thus can be
> multiplexed and benefit from multipath or are they considered all part
> of the same iSCSI stream? If they are considered one, do they become two
> if they reside on different partitions and thus different file systems?
> If not, then do we only see multibus performance gains between a single
> file system host and a single iSCSI host when we use virtualization each
> with their own iSCSI connection (as opposed to using iSCSI connections
> in the underlying host and exposing them to the virtual machines as
> local storage)?
>
> I hope I'm not hijacking this thread and realize I've asked some
> convoluted questions but optimizing multibus through bonded links for
> single large hosts is still a bit of a mystery to me.  Thanks - John
>
> --
> dm-devel mailing list
> dm-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>