[dm-devel] HDS multipathing prioritizer not doing what it should

[Christian Schausberger]

On 05/10/2012 09:28 AM, Christian Schausberger wrote:
>>  Hi all,
>>
>>
>>  I think I found a bug in the HDS prioritizer module at
>>  http://git.kernel.org/gitweb.cgi?p=linux/storage/multipath/hare/multipath-tools.git;a=blob_plain;f=libmultipath/prioritizers/hds.c;hb=HEAD
>>
>>  In there the following is stated for assigning the priority:
>>
>>  * CONTROLLER ODD and LDEV ODD: PRIORITY 1
>>  * CONTROLLER ODD and LDEV EVEN: PRIORITY 0
>>  * CONTROLLER EVEN and LDEV ODD: PRIORITY 0
>>  * CONTROLLER EVEN and LDEV EVEN: PRIORITY 1
>>
>>  When watching multipathing with debug output one can see that the
>>  controllers returned are 1 and 2:
>>
>>  May 08 14:44:00 | sdo: hds prio: VENDOR:  HITACHI
>>  May 08 14:44:00 | sdo: hds prio: PRODUCT: DF600F
>>  May 08 14:44:00 | sdo: hds prio: SERIAL:  0x0089
>>  May 08 14:44:00 | sdo: hds prio: LDEV:    0x0004
>>  May 08 14:44:00 | sdo: hds prio: CTRL:    1
>>  <= This is really controller 0
>>  May 08 14:44:00 | sdo: hds prio: PORT:    C
>>  May 08 14:44:00 | sdo: hds prio: CTRL ODD, LDEV EVEN, PRIO 0
>>  May 08 14:44:00 | sdo: hds prio = 0
>>
>>  May 08 14:44:00 | sdk: hds prio: VENDOR:  HITACHI
>>  May 08 14:44:00 | sdk: hds prio: PRODUCT: DF600F
>>  May 08 14:44:00 | sdk: hds prio: SERIAL:  0x0089
>>  May 08 14:44:00 | sdk: hds prio: LDEV:    0x0004
>>  May 08 14:44:00 | sdk: hds prio: CTRL:    2
>>  <= This is really controller 1
>>  May 08 14:44:00 | sdk: hds prio: PORT:    C
>>  May 08 14:44:00 | sdk: hds prio: CTRL EVEN, LDEV EVEN, PRIO 1
>>  May 08 14:44:00 | sdk: hds prio = 1
>>
>>  This looks fine, but afaik HDS starts counting controllers from 0
>>  (so actually I have 0 and 1). So when assigning LUN ownership in the
>>  storage, a LUN with an active/passive path will actually always be
>>  accessed through the wrong controller. This has a huge performance
>  penalty when the system is under stress, because of the additional
>  overhead generated by this.
>>

>Have you tested whether the situation improves when the priority is
>reversed?

>I'd be very much surprised if it did, though.

>I suspect more the internal queue size of the Hitachi to be a
>problem here. I've seen instances where we overload the internal
>queue size, causing the array to seize up.

>Cheers,

>Hannes

Yes, with the priority reversed within the storage throughput goes from 4.5 GB/s to 6 GB/s. Mind you, this is without any other changes to the host or storage.

I agree, that in normal operation the load balancing and the active/active mode of the storage iron this out. But in this setup (Lustre filesystem trimmed for
sustainable bandwidth) those features actually decrease performance and are not used. That's why the false priority makes such a difference.

Christian