<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML> <HEAD> <META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=utf-8"> <TITLE></TITLE> <META NAME="GENERATOR" CONTENT="OpenOffice.org 1.1.0 (Linux)"> <META NAME="CREATED" CONTENT="20040215;13342400"> <META NAME="CHANGED" CONTENT="20040216;13450400"> <STYLE>  </STYLE> </HEAD> <BODY LANG="en-US" DIR="LTR"> <H1>The Linux multipath implementation</H1> Original author : Christophe Varoqui, Feb 2004 This document is shared under the OpenContent Licence (<A HREF="http://www.opencontent.org/opl.shtml">http://www.opencontent.org/opl.shtml</A>) Introduction The most common multipathed environment today is a Fibre Channel (FC) Storage Area Network (SAN). This beasts can be found in most Datacenters. The lego blocks forming a SAN are : <UL> <LI>FC switchs : core switchs (multiprotocols chassis and FC boards), or stacked switchs linked by Inter Switch Links (ISL). This layer, the Fabric layer, can be subdived in two major fabric types : <UL> <LI>Simple fabrics : all storage ports can be routed to all hosts ports. Hosts an storage controlers can have a single attachement to the the fabric. <LI>Dual independent Fabrics : two sets of switchs are completely segregated (no ISL). They form two independent naming domains. The hosts and storage controlers must be attached the the two fabrics to assure redondancy. This technology is used to provide the maximum availability as one fabric can be shut, for planned or unplanned reasons, without pertubation seen on the other one. </UL> <LI>FC storage controlers : most of them provide multiple ports to attach to the switchs layer. The physical storage they drive is arranged in virtual drives we will refer to as Logical Units (LU). Each LU is provided by its host controler a unique identifier as per the SCSI standard. We will refer to this identifier as the World Wide Identifier (WWID) or World Wide Name (WWN). <LI>Host Bus Controlers (HBA) : the PCI / FC coupling adapters. A server can embed multiple HBA. </UL> The multipath term simply means that a host can acess a LU by muliple paths, the path being a route from one host HBA port to one storage controler port. <TABLE WIDTH=100% BORDER=1 CELLPADDING=4 CELLSPACING=3 STYLE="page-break-inside: avoid"> <COL WIDTH=256*> <THEAD> <TR> <TD WIDTH=100% VALIGN=TOP BGCOLOR="#cccccc"> Examples : <UL> <LI>A host with 2 HBA attached to a single fabric is presented a LU by a 4 ports storage controler. The host then see 8 paths to the LU <LI>A host with 2 HBA attached to a dual independent fabric (1 HBA on eache fabric) is presented a LU by a 4 ports storage controler (2 ports on each fabric). The host then see 4 paths to the LU : 2 paths through fabric A, plus 2 through the fabric B. </UL> </TD> </TR> </THEAD> </TABLE> The Linux kernel choose not the mask the individual paths, that appears as normal SCSI Disks (SD). Multipath awareness and support for an operating system can be described as : <UL> <LI>Provide a single block device node for a multipathed LU <LI>Ensure that IO are re-routed to available paths when a loss of path occurs, with no userspace process disruption other than an short pause. <LI>Ensure that failed paths get revalidated as soon as possible <LI>Ensure stability of the naming of that node <LI>Configure the multipaths to maximize performance : spread IO when possible path switching is free, and not spread when it's costly. <LI>Configure the multipaths automatically early at boot to permit OS install on a multipathed LU <LI>Reconfigure the multipaths automatically when events occur <LI>The multipath must be partitionable <LI>In the Linux way : simple and hardware vendor agnostic </UL> All these goals are met by leveraging a set of userspace tools ans kernel subsystems : <UL> <LI>the kernel device mapper <LI>the hotplug kernel subsystem <LI>the udev device naming tool <LI>the multipath userspace configuration tool <LI>the kpartx userspace configuration tool <LI>the early userspace Linux kernel boot environment </UL> The rest of this document describes these individual tools and subsystems and their interactions. Device Mapper Starting with Linux kernel 2.6, a new lightweight block subsystem named Device Mapper enables advanced storage management with style. This component features a plugable design. At the time of this writing available plugins are : <UL> <LI>segments concatenation <LI>segment striping <LI>segment snapshoting <LI>segment mirroring, with and without persistance <LI>segment on-the-fly encryption <LI>segment multipathing </UL> This last policy is the core component of the multipath tool chain. It is not included in the main kernel tree as of linux-2.6.3. It is part of a patchset maintained by Joe Thornber (thornber at redhat dot com) that can be downloaded at <A HREF="http://people.sistina.com/~thornber/dm/">http://people.sistina.com/~thornber/dm/</A> This component fills the following requirements : <UL> <LI>Provide a single block device node for a multipathed LU <LI>Ensure that IO are re-routed to available paths when a loss of path occurs, with no userspace process disruption other than an short pause. <LI>Ensure that failed paths get revalidated as soon as possible </UL> So, let's see how it works. The Device Mapper is configured one map at a time. A device map, also referred to as a table, is a list of segments in the form of : <TABLE WIDTH=100% BORDER=1 CELLPADDING=4 CELLSPACING=3 STYLE="page-break-inside: avoid"> <COL WIDTH=256*> <THEAD> <TR> <TD WIDTH=100% VALIGN=TOP BGCOLOR="#cccccc"> 0 35258368 linear 8:48 65920 35258368 35258368 linear 8:32 65920 70516736 17694720 linear 8:16 17694976 88211456 17694720 linear 8:16 256 </TD> </TR> </THEAD> </TABLE> The first 2 parameters of each line are the segment starting block in the virtual device and the lengh of the segment. The next keyword ist the target policy (linear). The rest of the line is the target parameters. The Device Mapper can be fed its tables through the use of a library : libdevmapper. Dmsetup, LVM2, the multipath configuration tool and kpartx all link this lib. A table setup boils down to sprintf'ing the right segment definitions in a char *. Should the DM user-kernel interface change from being ioctl based to a pseudo filesystem, the libdevmapper API should remain stable. <TABLE WIDTH=100% BORDER=1 CELLPADDING=4 CELLSPACING=3 STYLE="page-break-inside: avoid"> <COL WIDTH=256*> <THEAD> <TR> <TD WIDTH=100% VALIGN=TOP BGCOLOR="#cccccc"> Here is an example of a multipath target : 0 27262976 multipath 10 2 1 round-robin 2 0 /dev/sda /dev/sdk 1 round-robin 2 0 /dev/sdc /dev/sdm </TD> </TR> </THEAD> </TABLE> The multipath target parameters are : <UL> <LI>$0 : the “multipath” keyword <LI>$1 : the test IO posting interval, used to revalidate failed paths, in seconds <LI>$2 : the number of priority groups for the segment <LI>$3 : the first priority group parameters : <UL> <LI>$3.1 : the priority of this PG <LI>$3.2 : the scheduler to be used to spread IO inside the PG <LI>$3.3 : the number of paths in the PG <LI>$3.4 : the number of paths parameters (usualy 0) <LI>$3.5 : the paths list for this PG </UL> <LI>$4 : next priority group </UL> <TABLE WIDTH=100% BORDER=1 CELLPADDING=4 CELLSPACING=3 STYLE="page-break-inside: avoid"> <COL WIDTH=256*> <THEAD> <TR> <TD WIDTH=100% VALIGN=TOP BGCOLOR="#cccccc"> For completion, here is an example of a pure failover target definition for the same LU : 0 27262976 multipath 10 4 1 round-robin 1 0 /dev/sda 1 round-robin 1 0 /dev/sdc 1 round-robin 1 0 /dev/sdk 1 round-robin 1 0 /dev/sdm And a full spread (multibus) target one : 0 27262976 multipath 10 1 1 round-robin 4 0 /dev/sda /dev/sdc /dev/sdk /dev/sdm </TD> </TR> </THEAD> </TABLE> Upon device map creating, a new block kernel object named dm-[0-9]* is instantiated, and a hotplug call is triggered. Each device map can be assigned a symbolic name when created through libdevmapper, but this name won't be available anywhere but through a libdevmapper request. hotplug subsystem and udev Starting with Linux kernel 2.6, the hotplug callbacks are commonized through the presence of a new pseudo filesystem : sysfs. This filesystem presents to userspace kernel objects like bus, driver instances or block devices in a hierarchicaly and homogenous manner. The hotplug subsystem is leveraged by triggering a /sbin/hotplug call upon file creation and deletion in the sysfs filesystem. For our needs this facility provides : <UL> <LI>userspace callbacks upon paths additions and suppressions <LI>userspace callbacks upon device maps additions and suppressions </UL> It may also provide in the future callbacks upon FC transport events like a “Port Database Rescan”. This callbacks could then be used to trigger SCSI Bus Rescan to bring a fully dynamic storage layer. Here is how we use this callbacks for the multipath implementation : <UL> <LI>The paths additions and suppressions callbacks are routed to the multipath userspace configuration tool described later. This tool ensure the multipath maps are always up-to-date with the fabric topology, and this ensure optimal performance by adding new paths to the existing maps as soon as they become available. <LI>The udev userspace tool is triggered upon every block sysfs entry creation and suppression, and assume the responsibility of the associated device node creation and naming. Udev default naming policies can be complemented by add-on scripts or binaries. As it does not currently have a default policy for device maps naming, we plug a little tool named devmap_name that resolve the sysfs dm-[0-9]* names in map names as set at map creation time. Provided the map naming is rightly done, this plugin provides the naming stability and meaningfullness required for a proper multipath implementation. <LI>The userspace callbacks upon device maps additions and suppressions also triggers the kpartx tool to create the device maps over eventual partitions </UL> Udev is a reimplementation in userspace of the devfs kernel facility. It provides a dynamic /dev space, with an agnostic naming policy. Greg Kroah-Hartman is the main developer and maintainer of this package. It can be found at <A HREF="http://ftp.kernel.org/pub/linux/utils/kernel/hotplug/">http://ftp.kernel.org/pub/linux/utils/kernel/hotplug/</A> To synthetize what implementation details these subsystems fill : <UL> <LI>Ensure stability of the naming of that node <LI>Reconfigure the multipaths automatically when events occur <LI>The multipath must be partitionable </UL> multipath userspace config tool This tool is responsible for the paths coalescing and device maps creation. As seen earlier, it is triggered by the hotplug calls on new paths additions and suppressions. It must deal with hardware specifics and abstract them for the others subsystems. Here is how it works : <UL> <LI>draw a list of all available devices in the system through a sysfs scan. For each device, get a bunch of information : <UL> <LI>Host / Bus / Target / Lun tuple <LI>SCSI Device Strings : Vendor / Model / Revision <LI>SCSI Serial String </UL> <LI>Considering the informations fetched, elect a LU WWID method and an IO spreading policy. Ie deal with hardware specifics. <LI>Get the LU WWID with the elected method. This method defaults to the standard 128 bit EUID found in the EVPD 0x83 inquiry page of the device. <LI>Coalesce the paths to form the multipath structs <LI>Create and name the device maps associated with the multipath structs with the selected IO spreading policy </UL> There are currently 3 spreading policy implemented : <UL> <LI>failover : 1 path per priority group. IO thus get routed to one path only. <LI>multibus : 1 priority group containing all paths to the LU. Brings the maximum spreading, but assumes that all paths are sollicitable without penalty. <LI>group_by_serial : 1 priority group per storage controler (serial), paths through one controler are assigned to the associated PG. This policy applies to controlers that impose a latency penalty on LU management hand-over between a pair of redondant controlers. </UL> The device maps naming policy is “name by LU WWID”. <TABLE WIDTH=100% BORDER=1 CELLPADDING=4 CELLSPACING=3 STYLE="page-break-inside: avoid"> <COL WIDTH=256*> <THEAD> <TR> <TD WIDTH=100% VALIGN=TOP BGCOLOR="#cccccc"> To illustrate this synopsis, here is an example verbose output : <DL> <DD>xa-s03:~/udev-016/extras# multipath -v</DD><DD> 600508b4000156d700012000000b0000 (0 0 1 1) /dev/sda [HSV110 (C)COMPAQ]</DD><DD> 600508b4000156c30001200000210000 (0 0 1 2) /dev/sdb [HSV110 (C)COMPAQ]</DD><DD> 600508b4000156d700012000000b0000 (0 0 2 1) /dev/sdc [HSV110 (C)COMPAQ]</DD><DD> 600508b4000156c30001200000210000 (0 0 2 2) /dev/sdd [HSV110 (C)COMPAQ]</DD><DD> 60001fe1000bdad0000903507109004b (0 0 3 1) /dev/sde [HSG80 ]</DD><DD> 60001fe1000bdad000090371312100bf (0 0 3 2) /dev/sdf [HSG80 ]</DD><DD> 60001fe1000bdad000090371312100c2 (0 0 3 3) /dev/sdg [HSG80 ]</DD><DD> 60001fe1000bdad00009037131210067 (0 0 4 1) /dev/sdh [HSG80 ]</DD><DD> 60001fe1000bdad000090371312100b3 (0 0 4 2) /dev/sdi [HSG80 ]</DD><DD> 60001fe1000bdad00009035071090024 (0 0 4 3) /dev/sdj [HSG80 ]</DD><DD> 600508b4000156d700012000000b0000 (1 0 1 1) /dev/sdk [HSV110 (C)COMPAQ]</DD><DD> 600508b4000156c30001200000210000 (1 0 1 2) /dev/sdl [HSV110 (C)COMPAQ]</DD><DD> 600508b4000156d700012000000b0000 (1 0 2 1) /dev/sdm [HSV110 (C)COMPAQ]</DD><DD> 600508b4000156c30001200000210000 (1 0 2 2) /dev/sdn [HSV110 (C)COMPAQ]</DD><DD> 600508b4000156d700012000000b0000</DD><DD> \_(0 0 1 1) /dev/sda [HSV110 (C)COMPAQ]</DD><DD> \_(0 0 2 1) /dev/sdc [HSV110 (C)COMPAQ]</DD><DD> \_(1 0 1 1) /dev/sdk [HSV110 (C)COMPAQ]</DD><DD> \_(1 0 2 1) /dev/sdm [HSV110 (C)COMPAQ]</DD><DD> 600508b4000156c30001200000210000</DD><DD> \_(0 0 1 2) /dev/sdb [HSV110 (C)COMPAQ]</DD><DD> \_(0 0 2 2) /dev/sdd [HSV110 (C)COMPAQ]</DD><DD> \_(1 0 1 2) /dev/sdl [HSV110 (C)COMPAQ]</DD><DD> \_(1 0 2 2) /dev/sdn [HSV110 (C)COMPAQ]</DD><DD> 60001fe1000bdad0000903507109004b</DD><DD> \_(0 0 3 1) /dev/sde [HSG80 ]</DD><DD> 60001fe1000bdad000090371312100bf</DD><DD> \_(0 0 3 2) /dev/sdf [HSG80 ]</DD><DD> 60001fe1000bdad000090371312100c2</DD><DD> \_(0 0 3 3) /dev/sdg [HSG80 ]</DD><DD> 60001fe1000bdad00009037131210067</DD><DD> \_(0 0 4 1) /dev/sdh [HSG80 ]</DD><DD> 60001fe1000bdad000090371312100b3</DD><DD> \_(0 0 4 2) /dev/sdi [HSG80 ]</DD><DD> 60001fe1000bdad00009035071090024</DD><DD> \_(0 0 4 3) /dev/sdj [HSG80 ]</DD><DD> U:600508b4000156d700012000000b0000:0 27262976 multipath 10 2 1 round-robin 2 0 /dev/sda /dev/sdk 1 round-robin 2 0 /dev/sdc /dev/sdm</DD><DD> U:600508b4000156c30001200000210000:0 31457280 multipath 10 2 1 round-robin 2 0 /dev/sdb /dev/sdl 1 round-robin 2 0 /dev/sdd /dev/sdn</DD><DD> U:60001fe1000bdad0000903507109004b:0 106669167 multipath 10 1 1 round-robin 1 0 /dev/sde</DD><DD> U:60001fe1000bdad000090371312100bf:0 142229246 multipath 10 1 1 round-robin 1 0 /dev/sdf</DD><DD> U:60001fe1000bdad000090371312100c2:0 142229246 multipath 10 1 1 round-robin 1 0 /dev/sdg</DD><DD> U:60001fe1000bdad00009037131210067:0 213338334 multipath 10 1 1 round-robin 1 0 /dev/sdh</DD><DD> U:60001fe1000bdad000090371312100b3:0 213338334 multipath 10 1 1 round-robin 1 0 /dev/sdi</DD><DD> U:60001fe1000bdad00009035071090024:0 71114623 multipath 10 1 1 round-robin 1 0 /dev/sdj</DD></DL> </TD> </TR> </THEAD> </TABLE> The first section shows the list of all paths detected on the host. The second shows the multipath structs produced by the coalescing logic. The third shows the device maps submited to the Device Mapper. Of interest is the creation of device maps for single path LU : this enable to system to operate normaly when booted in a degraded SAN context. The missing paths will be added to the maps when they become available. This tool is packaged with udev, in the extras/ section. The devmap_name tool is distributed in the same tree. The implementation requirements filled by this tool are : <UL> <LI>Ensure stability of the naming of that node (in complement of udev) <LI>Configure the multipaths to maximize performance : spread IO when possible path switching is free, and not spread when it's costly. <LI>Configure the multipaths automatically at boot <LI>Reconfigure the multipaths automatically when events occur ( in complement of hotplug) </UL> kpartx userspace config tool This tool, derived from util-linux' partx, reads partition tables on specified device and create device maps over partitions segments detected. It is called from hotplug upon device maps creation and deletion. For now kpartx can be found at <A HREF="http://dsit.free.fr/">http://dsit.free.fr/</A> Early userspace Starting with Linux kernel 2.6, an early userspace execution environment is available in the name of initramfs. The grand plan is to package a set of tools in a cpio archive concatanated to the kernel. This archive is expanded in an in-memory filesystem early at boot and the tools are called to assume logics that previously belonged in the kernel : dhcp requests and setups, nfsroot stuffing ... Being concatenated to the kernel, the size of this archive matters a lot. A slim libc implementation is required and provided in the name of klibc, maintained by Hans Peter Anvin. The multipath implementation toolchain fits in this early userspace definition. Udev, multipath and kpartx are linked against klibc and can be packaged with the cpio archive to bring up the multipathed device early enough to boot on. So is met the last multipath implementation requirement. Quick installation guide <UL> <LI>Download and unarchive a recent 2.6 kernel <LI>Download, unarchive and apply the associated udm patchset from <A HREF="http://people.sistina.com/~thornber/dm/">http://people.sistina.com/~thornber/dm/</A> <LI>Configure, compile and install the kernel, reconfigure the bootloader if needed <LI>Download udev, at least 017, from <A HREF="http://ftp.kernel.org/pub/linux/utils/kernel/hotplug/">http://ftp.kernel.org/pub/linux/utils/kernel/hotplug/</A> <LI>Compile udev with klibc linking (edit the Makefile to uncomment KLIBC=true). Make install. <LI>In udev/extras/multipath, make and make install <LI>Reboot under the new kernel and see the magic operate </UL> </BODY> </HTML>