[Linux-cluster] gfs mounted but not working

[Benjamin Marzinski]

On Mon, Nov 06, 2006 at 01:38:40AM -0300, romero.cl at gmail.com wrote:
> Hi.
> 
> Now i'm trying this and it works! for now...
> 
> Two nodes: node3 & node4
> node4 export his /dev/sdb2 with gnbd_export as "node4_sdb2"
> node3 import node4's /dev/sdb2 with gnbd_import (new /dev/gnbd/node4_sdb2)
> 
> on node3:  gfs_mkfs -p lock_dlm -t node3:node3_gfs -j 4 /dev/gnbd/node4_sdb2
>                  mount -t gfs /dev/gnbd/node4_gfs /users/home
> 
> on node4: mount -t gfs /dev/sdb2 /users/home
> 
> and both nodes can read an write ths same files on /users/home!!!
> 
> Now i'm going for this:
> 
> 4 nodes on a dedicated 3com 1Gbit ethernet switch:
> 
> node2 exporting with gnbd_export /dev/sdb2 as "node2_sdb2"
> node3 exporting with gnbd_export /dev/sdb2 as "node3_sdb2"
> node4 exporting with gnbd_export /dev/sdb2 as "node4_sdb2"
> 
> node1 (main) will import all "nodeX_sdb2" and create a logical volume named
> "main_lv" including:
> 
>     /dev/sdb2 (his own)
>     /dev/gnbd/node2_sdb2
>     /dev/gnbd/node3_sdb2
>     /dev/gnbd/node4_sdb2
> 
> Next I will try to export the new big logical volume with "gnbd_export" and
> then do gnbd_import on each node.
> With that each node will see "main_lv", then mount it on /users/home as gfs
> and get a big shared filesystem to work toghether.
> 
> Is this the correct way to do the work??? possibly a deadlock???

Sorry. This will not work. There are a couple of problems.

1. A node shouldn't ever gnbd import a device it has exported.  This can
cause memory deadlock. When memory pressure is high nodes try to write
their buffers to disk. Once the buffer is written to disk, the node can drop
it from memory, reducing memory pressure. When you do this over gnbd, for every
buffer that you write out on the client, a new buffer request come into the gnbd
server. If you import a device you have exported (even indirectly through the
logical volume on node1 in this setup) that new request just comes back to you.
This means that you suddenly double your buffers in memory, just when memory was
running low.

The solution is to only access the local device directly, but never through
gnbd. Oh, just a note, if you are planning on accessing the local device
directly, you must not use the "-c" option when you are exporting the device.
This will eventually lead to corruption. The "-c" option is only for dedicated
gnbd servers.

2. Theoretically, you could just have every node export the devices to every
other node, and then build a logical volume on top of all the devices on each
node, but you should not do this. It totally destroys the benefit of having
a cluster. Since your GFS filesystem would then depend on having access to the
block devices of every machine, if ANY machine in your cluster went down, the
whole cluster would crash, because a piece of your filesystem would just
disappear.


Without shared storage, your gnbd server will be a single point of failure.
The most common way that people set up gnbd is with one dedicated gnbd server
machine, that is only used to serve gnbd blocks, so that it is unlikely to
crash.
 
> Sorry if my english isn't very good ;)
> 
> ----- Original Message ----- 
> From: "Kevin Anderson" <kanderso at redhat.com>
> To: "linux clustering" <linux-cluster at redhat.com>
> Sent: Sunday, November 05, 2006 10:12 PM
> Subject: Re: [Linux-cluster] gfs mounted but not working
> 
> 
> > On 11/5/06, *romero.cl at gmail.com <mailto:romero.cl at gmail.com>*
> > <romero.cl at gmail.com <mailto:romero.cl at gmail.com>> wrote:
> > >
> > >     Hi.
> > >
> > >     I'm trying your method, but still have a problem:
> > >
> > >     Note: /dev/db2/ is a local partition on my second SCSI hard drive
> > >     (no RAID)
> > >     runing on HP ProLiant.
> > >
> > GFS requires that all storage is equally accessible by all nodes in the
> > cluster.  Your other nodes have no path to the storage you set up so it
> > is impossible for them to share the data.
> >
> > Kevin
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster