[dm-devel] 4.4-rc4 multipath not working?

[E V]

Ok, makes sense. Looks like mdadm auto assembled my devices on boot
without the dm-? devices. So mdadm -S freed up the devices and then
restarting multipath tools, and mdadm -A /dev/md/0 /dev/dm-5+...
worked perfect. Guess I just need to figure out how to make the mdadm
assemble take place after multipathd is up on boot. Thanks a lot.

On Mon, Dec 14, 2015 at 3:36 PM, Benjamin Marzinski <bmarzins at redhat.com> wrote:
> On Mon, Dec 14, 2015 at 03:09:50PM -0500, E V wrote:
>> Ok stop and start of multipathd now with path_selector "queue-length
>> 0" in defaults changes the map, but things still fail. Would a reboot
>> be in order, or just the start stop of multipath should clear it up?
>
> You don't need to reboot. You can see from the output that the change
> has already taken effect.  Could you post the output of
>
> # tail /var/log/messages
>
> right after you run the multipath command.  Most likely you will see
> messages like
>
> Dec 14 08:49:56 ask-07 kernel: device-mapper: table: 253:7: multipath:
> error getting device
> Dec 14 08:49:56 ask-07 kernel: device-mapper: ioctl: error adding target
> to table
>
> This usually means that one of the paths devices (In your case 8:48 or
> 65:176) or a partition of one of those devices is already in use.  It
> could be a partition in use by another device-mapper device, or possibly
> a filesystem is already mounted on the device. At any rate, if you free
> up the device from whatever has exclusively grabbed it, multipath should
> be able to set up on it.
>
> -Ben
>
>>
>> mpathc: selector = queue-length 0 (config file default)
>> mpathc: features = 0 (config file default)
>> mpathc: hwhandler = 0 (internal default)
>> mpathc: rr_weight = 1 (internal default)
>> mpathc: minio = 1 rq (config file default)
>> mpathc: no_path_retry = 0 (internal default)
>> mpathc: fast_io_fail_tmo = 5 (config file default)
>> mpathc: retain_attached_hw_handler = 1 (config file default)
>> sdd: No SAS end device for 'end_device-1:0'
>> sdab: No SAS end device for 'end_device-1:1'
>> mpathc: remove queue_if_no_path from '0'
>> mpathc: assembled map [0 0 2 1 queue-length 0 1 1 8:48 1 queue-length
>> 0 1 1 65:176 1]
>> mpathc: set ACT_CREATE (map does not exist)
>> mpathc: domap (0) failure for create/reload map
>> mpathc: ignoring map
>>
>> On Mon, Dec 14, 2015 at 2:54 PM, Benjamin Marzinski <bmarzins at redhat.com> wrote:
>> > Mike is correct about the malformed table. looking at your ouput, I see
>> >
>> >> > mpathc: selector = queue_length (config file default)
>> >
>> > This means that you likely have
>> >
>> > path_selector "queue_length"
>> >
>> > in the defaults section of /etc/multipath.conf
>> >
>> > This should be
>> >
>> > path_selector "queue-length 0"
>> >
>> > With both a dash instead of an underscore, and a zero to let the kernel
>> > know that this selector takes no arguments.
>> >
>> > multipath does some sanity checking on the config values, but it
>> > definitely could be better.  For instance, on the values like this, that
>> > it simply passes to the kernel as a string, it doesn't do any.
>> >
>> > -Ben
>> >
>> > On Mon, Dec 14, 2015 at 01:51:18PM -0500, Mike Snitzer wrote:
>> >> On Mon, Dec 14 2015 at 11:55am -0500,
>> >> E V <eliventer at gmail.com> wrote:
>> >>
>> >> > On Fri, Dec 11, 2015 at 4:00 PM, Mike Snitzer <snitzer at redhat.com> wrote:
>> >> > > On Fri, Dec 11 2015 at 12:23pm -0500,
>> >> > > E V <eliventer at gmail.com> wrote:
>> >> > >
>> >> > >> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
>> >> > >> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
>> >> > >>
>> >> > >> And multipath doesn't seem to be working, dmesg has lots of these:
>> >> > >>
>> >> > >> device-mapper: table: 254:2: multipath: unknown path selector type
>> >> > >> device-mapper: ioctl: error adding target to table
>> >> > >
>> >> > > You compile the path selector module you're attempting to use?
>> >> > Yes as far as I can tell, here are the _DM_ config options in my config:
>> >> > CONFIG_BLK_DEV_DM_BUILTIN=y
>> >> > # CONFIG_DM_MQ_DEFAULT is not set
>> >> > # CONFIG_DM_DEBUG is not set
>> >> > # CONFIG_DM_CRYPT is not set
>> >> > # CONFIG_DM_SNAPSHOT is not set
>> >> > # CONFIG_DM_THIN_PROVISIONING is not set
>> >> > # CONFIG_DM_CACHE is not set
>> >> > # CONFIG_DM_ERA is not set
>> >> > # CONFIG_DM_MIRROR is not set
>> >> > CONFIG_DM_RAID=m
>> >> > # CONFIG_DM_ZERO is not set
>> >> > CONFIG_DM_MULTIPATH=y
>> >> > CONFIG_DM_MULTIPATH_QL=y
>> >> > # CONFIG_DM_MULTIPATH_ST is not set
>> >> > # CONFIG_DM_DELAY is not set
>> >> > CONFIG_DM_UEVENT=y
>> >> > # CONFIG_DM_FLAKEY is not set
>> >> > # CONFIG_DM_VERITY is not set
>> >> > # CONFIG_DM_SWITCH is not set
>> >> > # CONFIG_DM_LOG_WRITES is not set
>> >> >
>> >> >
>> >> > >> Is it expected to work? and any suggestions on getting it to work,
>> >> > >
>> >> > > Yes, no reason dm-multipath shouldn't work (other than user error).
>> >> > Entirely possible, this is my first multipath setup.
>> >> >
>> >> > > Figure out what the multipath table load line that failed to load
>> >> > > (verbose logging with multipathd?) and make sure you have the
>> >> > > corresponding path selector modules enabled in your .config
>> >> >
>> >> > Running multipathd with -v3 shows this in it's output, trimming 1st
>> >> > few columns for the 1st mpath device in the output. Doesn't mean much
>> >> > to me, other then I see the queue_length selector is set as expected:
>> >> > read /etc/multipath.conf
>> >> > multipath.conf line 8, duplicate keyword: defaults
>> >> > loading /lib/multipath/libcheckdirectio.so checker
>> >> > loading /lib/multipath/libprioconst.so prioritizer
>> >> > set open fds limit to 1048576/1048576
>> >> > sdab: ownership set to mpathc
>> >> > sdab: not found in pathvec
>> >> > sdab: mask = 0xc
>> >> > sdab: path state = running
>> >> > sdab: get_state
>> >> > directio: starting new request
>> >> > directio: io finished 4096/0
>> >> > sdab: state = up
>> >> > sdab: const prio = 1
>> >> > mpathc: pgfailback = -1 (config file default)
>> >> > mpathc: pgpolicy = failover (internal default)
>> >> > mpathc: selector = queue_length (config file default)
>> >> > mpathc: features = 0 (config file default)
>> >> > mpathc: hwhandler = 0 (internal default)
>> >> > mpathc: rr_weight = 1 (internal default)
>> >> > mpathc: minio = 1 rq (config file default)
>> >> > mpathc: no_path_retry = 0 (internal default)
>> >> > mpathc: fast_io_fail_tmo = 5 (config file default)
>> >> > mpathc: retain_attached_hw_handler = 1 (config file default)
>> >> > sdd: No SAS end device for 'end_device-1:0'
>> >> > sdab: No SAS end device for 'end_device-1:1'
>> >> > mpathc: remove queue_if_no_path from '0'
>> >> > mpathc: assembled map [0 0 2 1 queue_length 1 1 8:48 1 queue_length 1 1 65:176 1]
>> >>
>> >> This table looks malformed.
>> >>
>> >> Expected format is (from drivers/md/dm-mpath.c):
>> >>
>> >> /*-----------------------------------------------------------------
>> >>  * Constructor/argument parsing:
>> >>  * <#multipath feature args> [<arg>]*
>> >>  * <#hw_handler args> [hw_handler [<arg>]*]
>> >>  * <#priority groups>
>> >>  * <initial priority group>
>> >>  *     [<selector> <#selector args> [<arg>]*
>> >>  *      <#paths> <#per-path selector args>
>> >>  *         [<path> [<arg>]* ]+ ]+
>> >>  *---------------------------------------------------------------*/
>> >>
>> >> So in your case, breaking the map into logical sections, it is:
>> >>
>> >> 0 0 2 1
>> >>
>> >>   <#multipath feature args> = 0
>> >>   <#hw_handler args> = 0
>> >>   <#priority groups> = 2
>> >>   <initial priority group> = 1
>> >>
>> >> queue_length 1 1 8:48 1
>> >>
>> >>      <selector> = queue_length
>> >>      <#selector args> = 1 <BUT NO SELECTOR ARG!?>
>> >>      <#paths> = 1 <path> = 8:48 <arg> = 1
>> >>
>> >> queue_length 1 1 65:176 1
>> >>
>> >>      <selector> = queue_length <#selector args> = 1 <BUT NO SELECTOR ARG!?>
>> >>      <#paths> = 1 <path> = 65:176 <arg> = 1
>> >>
>> >> What do you have in your multipath.conf for 'path_selector'?
>> >>
>> >> AFAIK, it should be: path_selector queue-length 0
>> >>
>> >> (cc'ing Ben in case you need more multipath.conf help)