[dm-devel] failed multipaths are never re-validated

[Guy Coates]

Hi,

I have a dual fabric / dual contoller setup which I've got partially to
work with the multipath device mapper personality. I can create a
multipath device and I can get multipathd to remove failed paths to the
storage.  However, once failed paths become active again, multipathd never
adds the paths back into the devicemap.



The multipath tools correctly identify the four paths to the storage and
created the appropraite map.


ia64g:~# multipath -v -d
# all paths :
600508b40001045d0001c00006de0000 (2 0 0 1) /dev/sdc [HSV110 (C)COMPAQ]
600508b40001045d0001c00006de0000 (2 0 1 1) /dev/sdd [HSV110 (C)COMPAQ]
600508b40001045d0001c00006de0000 (3 0 0 1) /dev/sde [HSV110 (C)COMPAQ]
600508b40001045d0001c00006de0000 (3 0 1 1) /dev/sdf [HSV110 (C)COMPAQ]
# all multipaths :
600508b40001045d0001c00006de0000
 \_(2 0 0 1) /dev/sdc [HSV110 (C)COMPAQ]
 \_(2 0 1 1) /dev/sdd [HSV110 (C)COMPAQ]
 \_(3 0 0 1) /dev/sde [HSV110 (C)COMPAQ]
 \_(3 0 1 1) /dev/sdf [HSV110 (C)COMPAQ]

ia64g:~# multipath
U:600508b40001045d0001c00006de0000:0 209715200 multipath 2 round-robin 2 0
/dev/sdc /dev/sde round-robin 2 0 /dev/sdd /dev/sdf

ia64g:~# dmsetup  status
600508b40001045d0001c00006de0000: 0 209715200 multipath 2 2 0 8:32 A 1 8:64 A 1 2 0 8:48 A 1 8:80 A 1


I can format the multipath device and IO  does indeed go down both SAN
fabrics.

If I pull one of the fabrics, multipathd correctly reconfigures the
device, and IO continues down a single fabric.

<syslog>
May 28 15:16:46 ia64g kernel: qla2300 0000:40:01.1: LOOP DOWN detected.
May 28 15:16:54 ia64g kernel: SCSI error : <3 0 0 1> return code = 0x10000
May 28 15:16:54 ia64g kernel: end_request: I/O error, dev sde, sector 33967656

May 28 15:15:33 ia64g /usr/bin/multipathd: [event thread] refresh failpaths list
May 28 15:15:33 ia64g /usr/bin/multipathd: [event thread] start up event loops
May 28 15:16:59 ia64g /usr/bin/multipathd: [checker thread] reconfigure 600508b40001045d0001c00006de


ia64g:~# dmsetup  status
600508b40001045d0001c00006de0000: 0 209715200 multipath 2 2 0 8:32 A 1 8:64 F 0 2 0 8:48 A 1 8:80 A 1

ia64g:~# multipath -d -v
# all paths :
600508b40001045d0001c00006de0000 (2 0 0 1) /dev/sdc [HSV110 (C)COMPAQ]
600508b40001045d0001c00006de0000 (2 0 1 1) /dev/sdd [HSV110 (C)COMPAQ]
00000000000000000000000000000000 (3 0 0 1) /dev/sde [HSV110 (C)COMPAQ]
00000000000000000000000000000000 (3 0 1 1) /dev/sdf [HSV110 (C)COMPAQ]
# all multipaths :
600508b40001045d0001c00006de0000
 \_(2 0 0 1) /dev/sdc [HSV110 (C)COMPAQ]
 \_(2 0 1 1) /dev/sdd [HSV110 (C)COMPAQ]
00000000000000000000000000000000
 \_(3 0 0 1) /dev/sde [HSV110 (C)COMPAQ]
 \_(3 0 1 1) /dev/sdf [HSV110 (C)COMPAQ]


When I reconnect the seconds fabric, the newly active path does not get
automatically added back into the device map.

May 28 15:19:02 ia64g kernel: qla2300 0000:40:01.1: LIP reset occured (f7f7).
May 28 15:19:02 ia64g kernel: qla2300 0000:40:01.1: LOOP UP detected (2 Gbps).

ia64g:~# dmsetup  status
600508b40001045d0001c00006de0000: 0 209715200 multipath 2 2 0 8:32 A 1 8:64 F 0 2 0 8:48 A 1 8:80 A 1


I can force a reconfiguration manually by issuing the "multipath" command
again.

ia64g:~# multipath
U:600508b40001045d0001c00006de0000:0 209715200 multipath 2 round-robin 2 0
/dev/sdc /dev/sde round-robin 2 0 /dev/sdd /dev/sdf

ia64g:~# dmsetup status
600508b40001045d0001c00006de0000: 0 209715200 multipath 2 2 0 8:32 A 1
8:64 A 1 2 0 8:48 A 1 8:80 A 1

Does anyone have any suggestions as to how to get multipathd to
automatically act on the LOOP UP event?


I running on kernel 2.6.6 on IA64 with 2.6.6-udm1 device mapper patchset
with v0.2.0 of the multipath tools. The HBA is using QLogic Fibre Channel
HBA Driver: 8.00.00b11-k and the storage targets are dual Compaq HSV110s.


Cheers,

Guy Coates

-- 
Guy Coates,  Informatics System Group
The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK
Tel: +44 (0)1223 834244 ex 7199