[dm-devel] 4.4-rc4 multipath not working?

[E V]

Ok stop and start of multipathd now with path_selector "queue-length
0" in defaults changes the map, but things still fail. Would a reboot
be in order, or just the start stop of multipath should clear it up?

mpathc: selector = queue-length 0 (config file default)
mpathc: features = 0 (config file default)
mpathc: hwhandler = 0 (internal default)
mpathc: rr_weight = 1 (internal default)
mpathc: minio = 1 rq (config file default)
mpathc: no_path_retry = 0 (internal default)
mpathc: fast_io_fail_tmo = 5 (config file default)
mpathc: retain_attached_hw_handler = 1 (config file default)
sdd: No SAS end device for 'end_device-1:0'
sdab: No SAS end device for 'end_device-1:1'
mpathc: remove queue_if_no_path from '0'
mpathc: assembled map [0 0 2 1 queue-length 0 1 1 8:48 1 queue-length
0 1 1 65:176 1]
mpathc: set ACT_CREATE (map does not exist)
mpathc: domap (0) failure for create/reload map
mpathc: ignoring map

On Mon, Dec 14, 2015 at 2:54 PM, Benjamin Marzinski <bmarzins at redhat.com> wrote:
> Mike is correct about the malformed table. looking at your ouput, I see
>
>> > mpathc: selector = queue_length (config file default)
>
> This means that you likely have
>
> path_selector "queue_length"
>
> in the defaults section of /etc/multipath.conf
>
> This should be
>
> path_selector "queue-length 0"
>
> With both a dash instead of an underscore, and a zero to let the kernel
> know that this selector takes no arguments.
>
> multipath does some sanity checking on the config values, but it
> definitely could be better.  For instance, on the values like this, that
> it simply passes to the kernel as a string, it doesn't do any.
>
> -Ben
>
> On Mon, Dec 14, 2015 at 01:51:18PM -0500, Mike Snitzer wrote:
>> On Mon, Dec 14 2015 at 11:55am -0500,
>> E V <eliventer at gmail.com> wrote:
>>
>> > On Fri, Dec 11, 2015 at 4:00 PM, Mike Snitzer <snitzer at redhat.com> wrote:
>> > > On Fri, Dec 11 2015 at 12:23pm -0500,
>> > > E V <eliventer at gmail.com> wrote:
>> > >
>> > >> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
>> > >> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
>> > >>
>> > >> And multipath doesn't seem to be working, dmesg has lots of these:
>> > >>
>> > >> device-mapper: table: 254:2: multipath: unknown path selector type
>> > >> device-mapper: ioctl: error adding target to table
>> > >
>> > > You compile the path selector module you're attempting to use?
>> > Yes as far as I can tell, here are the _DM_ config options in my config:
>> > CONFIG_BLK_DEV_DM_BUILTIN=y
>> > # CONFIG_DM_MQ_DEFAULT is not set
>> > # CONFIG_DM_DEBUG is not set
>> > # CONFIG_DM_CRYPT is not set
>> > # CONFIG_DM_SNAPSHOT is not set
>> > # CONFIG_DM_THIN_PROVISIONING is not set
>> > # CONFIG_DM_CACHE is not set
>> > # CONFIG_DM_ERA is not set
>> > # CONFIG_DM_MIRROR is not set
>> > CONFIG_DM_RAID=m
>> > # CONFIG_DM_ZERO is not set
>> > CONFIG_DM_MULTIPATH=y
>> > CONFIG_DM_MULTIPATH_QL=y
>> > # CONFIG_DM_MULTIPATH_ST is not set
>> > # CONFIG_DM_DELAY is not set
>> > CONFIG_DM_UEVENT=y
>> > # CONFIG_DM_FLAKEY is not set
>> > # CONFIG_DM_VERITY is not set
>> > # CONFIG_DM_SWITCH is not set
>> > # CONFIG_DM_LOG_WRITES is not set
>> >
>> >
>> > >> Is it expected to work? and any suggestions on getting it to work,
>> > >
>> > > Yes, no reason dm-multipath shouldn't work (other than user error).
>> > Entirely possible, this is my first multipath setup.
>> >
>> > > Figure out what the multipath table load line that failed to load
>> > > (verbose logging with multipathd?) and make sure you have the
>> > > corresponding path selector modules enabled in your .config
>> >
>> > Running multipathd with -v3 shows this in it's output, trimming 1st
>> > few columns for the 1st mpath device in the output. Doesn't mean much
>> > to me, other then I see the queue_length selector is set as expected:
>> > read /etc/multipath.conf
>> > multipath.conf line 8, duplicate keyword: defaults
>> > loading /lib/multipath/libcheckdirectio.so checker
>> > loading /lib/multipath/libprioconst.so prioritizer
>> > set open fds limit to 1048576/1048576
>> > sdab: ownership set to mpathc
>> > sdab: not found in pathvec
>> > sdab: mask = 0xc
>> > sdab: path state = running
>> > sdab: get_state
>> > directio: starting new request
>> > directio: io finished 4096/0
>> > sdab: state = up
>> > sdab: const prio = 1
>> > mpathc: pgfailback = -1 (config file default)
>> > mpathc: pgpolicy = failover (internal default)
>> > mpathc: selector = queue_length (config file default)
>> > mpathc: features = 0 (config file default)
>> > mpathc: hwhandler = 0 (internal default)
>> > mpathc: rr_weight = 1 (internal default)
>> > mpathc: minio = 1 rq (config file default)
>> > mpathc: no_path_retry = 0 (internal default)
>> > mpathc: fast_io_fail_tmo = 5 (config file default)
>> > mpathc: retain_attached_hw_handler = 1 (config file default)
>> > sdd: No SAS end device for 'end_device-1:0'
>> > sdab: No SAS end device for 'end_device-1:1'
>> > mpathc: remove queue_if_no_path from '0'
>> > mpathc: assembled map [0 0 2 1 queue_length 1 1 8:48 1 queue_length 1 1 65:176 1]
>>
>> This table looks malformed.
>>
>> Expected format is (from drivers/md/dm-mpath.c):
>>
>> /*-----------------------------------------------------------------
>>  * Constructor/argument parsing:
>>  * <#multipath feature args> [<arg>]*
>>  * <#hw_handler args> [hw_handler [<arg>]*]
>>  * <#priority groups>
>>  * <initial priority group>
>>  *     [<selector> <#selector args> [<arg>]*
>>  *      <#paths> <#per-path selector args>
>>  *         [<path> [<arg>]* ]+ ]+
>>  *---------------------------------------------------------------*/
>>
>> So in your case, breaking the map into logical sections, it is:
>>
>> 0 0 2 1
>>
>>   <#multipath feature args> = 0
>>   <#hw_handler args> = 0
>>   <#priority groups> = 2
>>   <initial priority group> = 1
>>
>> queue_length 1 1 8:48 1
>>
>>      <selector> = queue_length
>>      <#selector args> = 1 <BUT NO SELECTOR ARG!?>
>>      <#paths> = 1 <path> = 8:48 <arg> = 1
>>
>> queue_length 1 1 65:176 1
>>
>>      <selector> = queue_length <#selector args> = 1 <BUT NO SELECTOR ARG!?>
>>      <#paths> = 1 <path> = 65:176 <arg> = 1
>>
>> What do you have in your multipath.conf for 'path_selector'?
>>
>> AFAIK, it should be: path_selector queue-length 0
>>
>> (cc'ing Ben in case you need more multipath.conf help)