[dm-devel] multipath failover with 2x Emulex LP9802, setup, WWIDs

Thu Nov 25 22:21:53 UTC 2004

Hi,

On Thursday 25 November 2004 10:30, christophe varoqui wrote:
> Le jeudi 25 novembre 2004 Ã  08:55 +0100, Andreas Huck a Ã©crit :
> > Hi,
> > 
> > I struggle to setup multipath devices by dmsetup, multipath-tools
> > and multipathd. I followed the description at
> > http://christophe.varoqui.free.fr/multipath.html
> > and refered to all I could find on the web, but some questions
> > remain. Maybe you could give some advices?
> > 
> I'd rather remove this doc as it out of date and generate more trouble
> than enlightenment.
Don't do that. Its really OK for the idea, quite comprehensive and also
gives some examples. I'll give you my thoughts on it when I have some
minutes left.

> > First, how to get a WWID?  I have 4 devices with 2 paths each.
> > `scsidev -s` should provide it but doesn't:
> > 
> what is the output of "multipath -v2" ?

/~# multipath -v 2
# all paths :
# all multipaths :
# device maps :

It's strange that the tool does not show anything although it works basically
as I could see with iostat on the underlying scsi devices: 
After unplugging one FC-cable the traffic switched over onto the failover paths.
The traffic was heavy drbd full-sync and a  `dd if=/dev/zero bs=1M...` on top
of ext3 on top of LVM2 on top of drbd on top of multipath :-) 
There is, of course, another storage involved.

> 
> > 
> > So how can I provide WWIDs for /etc/multipath.conf ?
> > 
> That is only necessary for aliases.
> You can safely leave the config file multipath section empty.
> 
> > How to identify the devices if  for XXX=[a-h]:
> > # /sbin/scsi_id -g -p 0x83 -s /block/sdXXX
> > 220000003390011d3
> >
> And without "-p 0x83" ?

fsrv1:# for i in b c d f g h; do echo sd$i: `/sbin/scsi_id -g -p 0x83 -s /block/sd$i`; done
sdb: 2200000033900128b
sdc: 2200000033900128b
sdd: 2200000033900128b
sdf: 2200000033900128b
sdg: 2200000033900128b
sdh: 2200000033900128b
fsrv1:# for i in b c d f g h; do echo sd$i: `/sbin/scsi_id -g -s /block/sd$i`; done
sdb: 2200000033900128b
sdc: 2200000033900128b
sdd: 2200000033900128b
sdf: 2200000033900128b
sdg: 2200000033900128b
sdh: 2200000033900128b
fsrv1:# for i in b c d f g h; do echo sd$i: `/sbin/scsi_id -g -p 0x80 -s /block/sd$i`; done
sdb: SEUROLOGCFC2502 0001cc7420000080e512cc740000000000000000
sdc: SEUROLOGCFC2502 0002cc7420000080e512cc740000000000000000
sdd: SEUROLOGCFC2502 0003cc7420000080e512cc740000000000000000
sdf: SEUROLOGCFC2502 0001cc7420000080e512cc740000000000000000
sdg: SEUROLOGCFC2502 0002cc7420000080e512cc740000000000000000
sdh: SEUROLOGCFC2502 0003cc7420000080e512cc740000000000000000

YES! Thanks. I replaced it but what does that help? Should UUID==WWID? 
Should I pipe it through ´cut -n -b 17-´? 

(LMB: the provided "-u" option of scsi_id does not not work "out of the SLES-box".)

> 
> > I.e. all IDs are equal (but configured right, i.e.
> > a=e,b=f,c=g,d=h).
> > 
> > Is it wise to take the unique path as ID? :
> > # ls -l /sys/block/sdb/device
> > /sys/block/sdb/device -> ../../devices/pci0000:04/0000:04:08.0/host0/0:0:0:1
> > 
> No : we want a unique LU id, not a unique path id.
OK, be on the save side.

> 
> > The hardware is not listed in setup_default_hwtable and I wonder
> > whether I can define it by multipath.conf at all since sysfs does
> > provide the vendor but NOT the "product":
> > 
> Sane defaults are applied when there is no corresponding entry in the
> config file nor in setup_default_hwtable. Notably the failover pgpolicy
> is selected, which is what you want.
> 
> > # ls  /sys/block/sdb/device/
> > .  ..  block  delete  detach_state  device_blocked  generic  model
> > online  power  queue_depth  rescan  rev  scsi_level  timeout  type  vendor
> > 
> > Instead I could use 'model "FC2502          "' - but that's not implemented.
> > 
> This is just terminology : "vendor / product /rev" used by most scsi
> people, "vendor / model / rev" chosen by dreaming sysfs developers :)
> 
> model == product, so you can add the configlet if you want.
> 
> > I tried a direct approach by dmsetup, but didn't get a device:
> > 
> > # echo "0 274968576 multipath 10 2 1 round-robin 1 0 \
> > /dev/sdb 1 round-robin 1 0 /dev/sdf" | dmsetup create /dev/dm-0
> > k_set_name: Device /dev/dm-0 not found
> > Command failed
> > 
> # echo "0 274968576 multipath 10 2 1 round-robin 1 0 \
> /dev/sdb 1 round-robin 1 0 /dev/sdf" | dmsetup create
> just_the_map_name_you_want_not_a_path
> 
> > Maybe I am not too familiar with hotplug and udev.
> > I already did add a udev rule:
> > # grep dm /etc/udev/udev.rules
> > KERNEL="dm-[0-9]*", PROGRAM="/sbin/devmap_name %M %m", NAME="%k", SYMLINK="%c"
> > 
> > but
> > 
> > # /sbin/devmap_name 8 16
> > No devices found
> > 
> major:minor are devmap's one, as listed in "dmsetup ls" ?
> And the map must exist for the command to work.
> 
> > Although the devices exist:
> > brw-rw----  1 root disk 8, 0 Jun 30 21:04 /dev/sda
> > brw-rw----  1 root disk 8, 16 Jun 30 21:04 /dev/sdb
> > brw-rw----  1 root disk 8, 32 Jun 30 21:04 /dev/sdc
> > brw-rw----  1 root disk 8, 48 Jun 30 21:04 /dev/sdd
> > brw-rw----  1 root disk 8, 64 Jun 30 21:04 /dev/sde
> > brw-rw----  1 root disk 8, 80 Jun 30 21:04 /dev/sdf
> > brw-rw----  1 root disk 8, 96 Jun 30 21:04 /dev/sdg
> > brw-rw----  1 root disk 8, 112 Jun 30 21:04 /dev/sdh
> > 
> > Do I overlook something important here?
> > 
> certainly :)
> 
> > Last but not least, I use
> > kernel-smp-2.6.5-7.111
> > udev-021-36.32
> > multipath-tools-0.2.1-1.5
> > device-mapper-1.00.09-17.5
> > from SLES9 and also played around with multipath-tools-0.3.7 from source.
> > 
> I guess you'll have more feedback from Suse folk then.
> I don't really know how the combination in SLES9 behave and what patch
> pile they applied to the listed products.
> 
> > So if you have some hints for me that would be great.
> > 
> Hope it is enough to get you a bit further.
Yes, thanks a lot!

One single problem remains: unplugging something in the aktive path leads
to a nice failover. How to deal with the correction of the disturbance?
If I replug everything, the Emulex reports bravely:

Nov 25 16:46:31 fsrv1 kernel: lpfc 0000:04:08.0: 0:1303:LKe:Link Up Event x3 received Data: x3 x0 x8 x0

But just unplugging one of the other (now still active) path components won't
bring up the "restored" path but leads to a complete failure of both paths.
Is there any way to reset and reuse the revitalized path? I think its similar to
the mail from Jims yesterday.

Regards,
Andreas Huck

> 
> regards,