[Linux-cluster] GNBD and CLVM and multipath oh my

[Benjamin Marzinski]

On Wed, Jun 01, 2005 at 08:12:22PM +0100, David Golden wrote:
> Hi,
> 
> I have a 4-node GFS setup on top of a CLVM volume across a 
> 14-shared-block-device SAN.   I'd like to try farming out devices 
> at a block level from the four head nodes to an, ahem, larger 
> number of worker nodes, to see how GFS works across said larger 
> number of nodes, otherwise it's back to NFS-ing out from the head 
> nodes, which works but, jeez, it's NFS.
> 
> (I can delete and recreate the GFS filesystem at this
> stage, so no worries about journals count etc.)
> 
> Soo... question, in case anyone else has tried this:
> 
> Which is most likely to work? 
> 
> 
> (a) gnbd export the logical volume block device from the 
> four head nodes to the worker nodes, (maybe multipath
> the four imported gnbds on each worker node), then run gfs on top.
> 
> or
> 
> (b) gnbd export the underlying block devices (maybe multipath
> them all into 14 multipath devices on each node)
> and run  CLVM across the larger number of nodes,
> then run gfs.

First, GNBD does not currently work with multipath. The multipath tools
are too SCSI-centric. Sorry.

either (a) or (b) should work. I would bet that (b) gets you better performance
on the worker nodes. gnbd has one server per exported device per client.
If you have one logical volume, in (a) you only have one thread serving requests
for each client. in (b) you have 14 threads.  Of course, (b) will slow down
your head nodes more. So that's the tradeoff. Hope that helps 

Just a reminder: If you are exporting the same device from multiple nodes,
even if you are not multipathing between the devices, you need to run gnbd
in uncached mode.

-Ben

> I guess this comes down to  whether and how GFS and CLVM are 
> coupled... Naively, I'd imagine both "should" work, but (b) 
> would cause unnecessary locking overhead, so (a) would be 
> preferable, so I'll try (a) first unless someone says 
> "no, you fool, (a) won't work because XYZ" before I get around
> to it tomorrow.
> 
> ("maybe multipath" because I'm using a RHEL4oid and AFAIK wouldn't
> be able to make a working multipath-tools without a custom kernel...)
> 
> Best Regards,
> 
> David Golden
> 
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> http://www.redhat.com/mailman/listinfo/linux-cluster