[Linux-cluster] Multipath or Oracle RAC ASM issue?

[Joel Becker]

On Thu, Jun 19, 2008 at 12:20:03AM +0800, sunhux G wrote:
> We have SAN disk partitions presented to 2 Redhat Linux servers,  running
> Oracle RAC (cluster) ASM.

Hey there,
	First off, you're using ASMLib, so you don't need to worry about
device naming - ASMLib makes sure you are accessing the same device.
But read on for furthur discussion.

> As suggested by Linux "man multipath", I've made the
> /var/lib/multipath/bindings the same
> on both servers "to ensure That the multipath devices have the same names on
> all nodes
>                             accessing  them" :
> # Format:
> # alias wwid
> mpath0 360a98000567244396334493370345055

	This works as you expect.  The names in
/dev/(mapper|mpath)/mpathX should be identical on each server.
 
> However /proc/partition   on both servers  showed differences on the
>  dm-x  devices  (but not on the /dev/sdxx devices) :

	The dm-X devices are not expected to be the same.  You see, the
system creates the dm-X devices in the order they are seen - this can be
different.  After they are created, the udev subsystem uses the
information in /var/lib/multipath/bindings to map the mpathX name to the
appropriate dm-X device.  This ensures that the mpathX names are
identical on each system.

> Q1 :
> Do mpath0, mpath1, ...   shown in "multipath -ll" listing refer to
>   /dev/mapper/mpathx  or  to  /dev/mpath/mpathx ?

	If there are names /dev/mapper/mpathx and /dev/mpath/mpathx,
they should be refer to the same thing.

> Q2 :
> How do we make it such that the dm-x  devices are accessing the same
> SAN LUNs across the servers?  I believe they are not the same based
> on the observations of the disk spaces associated with each of the dm-x
> shown in /proc/partition

	You don't.  You don't care.  If you want a /dev name, you use
the mpathX name that you know maps correctly.  Since you are using
ASMLib, you don't worry about that either.  The LANDx names are read by
ASMLib to ensure ASM sees the correct devices no matter what /dev name
they have.

> [root at landnet-rac1-temp mpath]# /etc/init.d/oracleasm querydisk land4
> Disk "LAND4" is a valid ASM disk on device [253, 0]On Server 1 :

	That maps to dm-0.

> [root at landnet-rac2-temp dev]# /etc/init.d/oracleasm scandisks
> Scanning system for ASM disks:   OK  ]
> [root at landnet-rac2-temp dev]# /etc/init.d/oracleasm querydisk land4
> Disk "LAND4" is a valid ASM disk on device [8, 32] <<== note the number

	And that is /dev/sdc.  Your problem is SCANORDER.  ASMLib's disk
scan is seeing the scsi device (one of the paths) before the multipath
device.  This is actually safe, it just doesn't take advantage of the
multipath.  You want to use the multipath of course, so you want to
configure it correctly.
	See the instructions at
http://www.oracle.com/technology/tech/linux/asmlib/multipath.html to
configure ASMLib to see dm devices before scsi devices.  Then your
scandisks will always see the multipath devices.

> We logged a call to Oracle who responded :
> If we are using device mapper and ASMLib then, we need to use disks from
> /dev/dm-* disks
> instead of disks from  /dev/mapper/mpath*

	That's not the issue, and I'm sorry Oracle support gave you the
wrong answer.  You can createdisk against any name that's correct
(/dev/dm-X, /dev/mapper/mpathX, /dev/mpath/mpathX).  scandisks doesn't
even need a name - it just needs the correct order in your case.

Joel

-- 

"War doesn't determine who's right; war determines who's left."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127