[dm-devel] 4.4-rc4 multipath not working?

Mon Dec 14 20:36:51 UTC 2015

On Mon, Dec 14, 2015 at 03:09:50PM -0500, E V wrote:
> Ok stop and start of multipathd now with path_selector "queue-length
> 0" in defaults changes the map, but things still fail. Would a reboot
> be in order, or just the start stop of multipath should clear it up?

You don't need to reboot. You can see from the output that the change
has already taken effect.  Could you post the output of 

# tail /var/log/messages

right after you run the multipath command.  Most likely you will see
messages like

Dec 14 08:49:56 ask-07 kernel: device-mapper: table: 253:7: multipath:
error getting device
Dec 14 08:49:56 ask-07 kernel: device-mapper: ioctl: error adding target
to table

This usually means that one of the paths devices (In your case 8:48 or
65:176) or a partition of one of those devices is already in use.  It
could be a partition in use by another device-mapper device, or possibly
a filesystem is already mounted on the device. At any rate, if you free
up the device from whatever has exclusively grabbed it, multipath should
be able to set up on it.

-Ben

> 
> mpathc: selector = queue-length 0 (config file default)
> mpathc: features = 0 (config file default)
> mpathc: hwhandler = 0 (internal default)
> mpathc: rr_weight = 1 (internal default)
> mpathc: minio = 1 rq (config file default)
> mpathc: no_path_retry = 0 (internal default)
> mpathc: fast_io_fail_tmo = 5 (config file default)
> mpathc: retain_attached_hw_handler = 1 (config file default)
> sdd: No SAS end device for 'end_device-1:0'
> sdab: No SAS end device for 'end_device-1:1'
> mpathc: remove queue_if_no_path from '0'
> mpathc: assembled map [0 0 2 1 queue-length 0 1 1 8:48 1 queue-length
> 0 1 1 65:176 1]
> mpathc: set ACT_CREATE (map does not exist)
> mpathc: domap (0) failure for create/reload map
> mpathc: ignoring map
> 
> On Mon, Dec 14, 2015 at 2:54 PM, Benjamin Marzinski <bmarzins at redhat.com> wrote:
> > Mike is correct about the malformed table. looking at your ouput, I see
> >
> >> > mpathc: selector = queue_length (config file default)
> >
> > This means that you likely have
> >
> > path_selector "queue_length"
> >
> > in the defaults section of /etc/multipath.conf
> >
> > This should be
> >
> > path_selector "queue-length 0"
> >
> > With both a dash instead of an underscore, and a zero to let the kernel
> > know that this selector takes no arguments.
> >
> > multipath does some sanity checking on the config values, but it
> > definitely could be better.  For instance, on the values like this, that
> > it simply passes to the kernel as a string, it doesn't do any.
> >
> > -Ben
> >
> > On Mon, Dec 14, 2015 at 01:51:18PM -0500, Mike Snitzer wrote:
> >> On Mon, Dec 14 2015 at 11:55am -0500,
> >> E V <eliventer at gmail.com> wrote:
> >>
> >> > On Fri, Dec 11, 2015 at 4:00 PM, Mike Snitzer <snitzer at redhat.com> wrote:
> >> > > On Fri, Dec 11 2015 at 12:23pm -0500,
> >> > > E V <eliventer at gmail.com> wrote:
> >> > >
> >> > >> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
> >> > >> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
> >> > >>
> >> > >> And multipath doesn't seem to be working, dmesg has lots of these:
> >> > >>
> >> > >> device-mapper: table: 254:2: multipath: unknown path selector type
> >> > >> device-mapper: ioctl: error adding target to table
> >> > >
> >> > > You compile the path selector module you're attempting to use?
> >> > Yes as far as I can tell, here are the _DM_ config options in my config:
> >> > CONFIG_BLK_DEV_DM_BUILTIN=y
> >> > # CONFIG_DM_MQ_DEFAULT is not set
> >> > # CONFIG_DM_DEBUG is not set
> >> > # CONFIG_DM_CRYPT is not set
> >> > # CONFIG_DM_SNAPSHOT is not set
> >> > # CONFIG_DM_THIN_PROVISIONING is not set
> >> > # CONFIG_DM_CACHE is not set
> >> > # CONFIG_DM_ERA is not set
> >> > # CONFIG_DM_MIRROR is not set
> >> > CONFIG_DM_RAID=m
> >> > # CONFIG_DM_ZERO is not set
> >> > CONFIG_DM_MULTIPATH=y
> >> > CONFIG_DM_MULTIPATH_QL=y
> >> > # CONFIG_DM_MULTIPATH_ST is not set
> >> > # CONFIG_DM_DELAY is not set
> >> > CONFIG_DM_UEVENT=y
> >> > # CONFIG_DM_FLAKEY is not set
> >> > # CONFIG_DM_VERITY is not set
> >> > # CONFIG_DM_SWITCH is not set
> >> > # CONFIG_DM_LOG_WRITES is not set
> >> >
> >> >
> >> > >> Is it expected to work? and any suggestions on getting it to work,
> >> > >
> >> > > Yes, no reason dm-multipath shouldn't work (other than user error).
> >> > Entirely possible, this is my first multipath setup.
> >> >
> >> > > Figure out what the multipath table load line that failed to load
> >> > > (verbose logging with multipathd?) and make sure you have the
> >> > > corresponding path selector modules enabled in your .config
> >> >
> >> > Running multipathd with -v3 shows this in it's output, trimming 1st
> >> > few columns for the 1st mpath device in the output. Doesn't mean much
> >> > to me, other then I see the queue_length selector is set as expected:
> >> > read /etc/multipath.conf
> >> > multipath.conf line 8, duplicate keyword: defaults
> >> > loading /lib/multipath/libcheckdirectio.so checker
> >> > loading /lib/multipath/libprioconst.so prioritizer
> >> > set open fds limit to 1048576/1048576
> >> > sdab: ownership set to mpathc
> >> > sdab: not found in pathvec
> >> > sdab: mask = 0xc
> >> > sdab: path state = running
> >> > sdab: get_state
> >> > directio: starting new request
> >> > directio: io finished 4096/0
> >> > sdab: state = up
> >> > sdab: const prio = 1
> >> > mpathc: pgfailback = -1 (config file default)
> >> > mpathc: pgpolicy = failover (internal default)
> >> > mpathc: selector = queue_length (config file default)
> >> > mpathc: features = 0 (config file default)
> >> > mpathc: hwhandler = 0 (internal default)
> >> > mpathc: rr_weight = 1 (internal default)
> >> > mpathc: minio = 1 rq (config file default)
> >> > mpathc: no_path_retry = 0 (internal default)
> >> > mpathc: fast_io_fail_tmo = 5 (config file default)
> >> > mpathc: retain_attached_hw_handler = 1 (config file default)
> >> > sdd: No SAS end device for 'end_device-1:0'
> >> > sdab: No SAS end device for 'end_device-1:1'
> >> > mpathc: remove queue_if_no_path from '0'
> >> > mpathc: assembled map [0 0 2 1 queue_length 1 1 8:48 1 queue_length 1 1 65:176 1]
> >>
> >> This table looks malformed.
> >>
> >> Expected format is (from drivers/md/dm-mpath.c):
> >>
> >> /*-----------------------------------------------------------------
> >>  * Constructor/argument parsing:
> >>  * <#multipath feature args> [<arg>]*
> >>  * <#hw_handler args> [hw_handler [<arg>]*]
> >>  * <#priority groups>
> >>  * <initial priority group>
> >>  *     [<selector> <#selector args> [<arg>]*
> >>  *      <#paths> <#per-path selector args>
> >>  *         [<path> [<arg>]* ]+ ]+
> >>  *---------------------------------------------------------------*/
> >>
> >> So in your case, breaking the map into logical sections, it is:
> >>
> >> 0 0 2 1
> >>
> >>   <#multipath feature args> = 0
> >>   <#hw_handler args> = 0
> >>   <#priority groups> = 2
> >>   <initial priority group> = 1
> >>
> >> queue_length 1 1 8:48 1
> >>
> >>      <selector> = queue_length
> >>      <#selector args> = 1 <BUT NO SELECTOR ARG!?>
> >>      <#paths> = 1 <path> = 8:48 <arg> = 1
> >>
> >> queue_length 1 1 65:176 1
> >>
> >>      <selector> = queue_length <#selector args> = 1 <BUT NO SELECTOR ARG!?>
> >>      <#paths> = 1 <path> = 65:176 <arg> = 1
> >>
> >> What do you have in your multipath.conf for 'path_selector'?
> >>
> >> AFAIK, it should be: path_selector queue-length 0
> >>
> >> (cc'ing Ben in case you need more multipath.conf help)