[Linux-cluster] gfs mounted but not working

Tue Nov 7 06:03:21 UTC 2006

On Tue, Nov 07, 2006 at 12:19:50AM -0300, romero.cl at gmail.com wrote:
> Hi.
> 
> Thanks for the tips.
> 
> But, if there is a deadlock problem with memory, what wold be a good
> solution to get each one of the drives on my 4 nodes look like one big
> shared drive using gfs?

The easiest way to get a cluster with GFS is to have one of your nodes export
its block device with gnbd to the other nodes. There is no good way to take all
the storage from all your nodes and make it look like one big file system
that all the nodes can use.  Like I said before, you can do this with GNBD.
It's just a bad idea. If you make a big shared drive that uses all the disks
from all the machines, when any machine goes down, a piece of that big shared
drive will disappear, because it was located on the machine that went down.
Since there's no way that any software could access a machine's local storage
when the machine is crashed, there is no possible way to avoid this problem.

We will shortly be releasing cluster-mirror software, that will allow you to
make a mirror using all of the gnbd_exported devices from all of the machines.
However, since you are mirroring the devices, instead of concatenating or
striping them. the shared device will not be the size of the sum of all the
devices.  However this does mean that there is no gnbd server machine that
is a single point of failure. This code is already available in cvs, but I'm
not sure that there are any rpms yet.

> I think that I have a wrong idea of what GFS is, can you explain it to me
> please? is only a file system ? is a way to share storage with distributed
> lock?

GFS is simply a filesystem that uses locking to allow you to access shared
storage from multiple machines at the same time. GFS does not give you
shared storage. To use GFS in a cluster you must either have a SAN, or you
must use software like GNBD or iSCSI. 

> 
> 
> 
> ----- Original Message ----- 
> From: "Benjamin Marzinski" <bmarzins at redhat.com>
> To: "linux clustering" <linux-cluster at redhat.com>
> Sent: Monday, November 06, 2006 11:01 PM
> Subject: Re: [Linux-cluster] gfs mounted but not working
> 
> 
> > On Mon, Nov 06, 2006 at 01:38:40AM -0300, romero.cl at gmail.com wrote:
> > > Hi.
> > >
> > > Now i'm trying this and it works! for now...
> > >
> > > Two nodes: node3 & node4
> > > node4 export his /dev/sdb2 with gnbd_export as "node4_sdb2"
> > > node3 import node4's /dev/sdb2 with gnbd_import (new
> /dev/gnbd/node4_sdb2)
> > >
> > > on node3:  gfs_mkfs -p lock_dlm -t node3:node3_gfs -j 4
> /dev/gnbd/node4_sdb2
> > >                  mount -t gfs /dev/gnbd/node4_gfs /users/home
> > >
> > > on node4: mount -t gfs /dev/sdb2 /users/home
> > >
> > > and both nodes can read an write ths same files on /users/home!!!
> > >
> > > Now i'm going for this:
> > >
> > > 4 nodes on a dedicated 3com 1Gbit ethernet switch:
> > >
> > > node2 exporting with gnbd_export /dev/sdb2 as "node2_sdb2"
> > > node3 exporting with gnbd_export /dev/sdb2 as "node3_sdb2"
> > > node4 exporting with gnbd_export /dev/sdb2 as "node4_sdb2"
> > >
> > > node1 (main) will import all "nodeX_sdb2" and create a logical volume
> named
> > > "main_lv" including:
> > >
> > >     /dev/sdb2 (his own)
> > >     /dev/gnbd/node2_sdb2
> > >     /dev/gnbd/node3_sdb2
> > >     /dev/gnbd/node4_sdb2
> > >
> > > Next I will try to export the new big logical volume with "gnbd_export"
> and
> > > then do gnbd_import on each node.
> > > With that each node will see "main_lv", then mount it on /users/home as
> gfs
> > > and get a big shared filesystem to work toghether.
> > >
> > > Is this the correct way to do the work??? possibly a deadlock???
> >
> > Sorry. This will not work. There are a couple of problems.
> >
> > 1. A node shouldn't ever gnbd import a device it has exported.  This can
> > cause memory deadlock. When memory pressure is high nodes try to write
> > their buffers to disk. Once the buffer is written to disk, the node can
> drop
> > it from memory, reducing memory pressure. When you do this over gnbd, for
> every
> > buffer that you write out on the client, a new buffer request come into
> the gnbd
> > server. If you import a device you have exported (even indirectly through
> the
> > logical volume on node1 in this setup) that new request just comes back to
> you.
> > This means that you suddenly double your buffers in memory, just when
> memory was
> > running low.
> >
> > The solution is to only access the local device directly, but never
> through
> > gnbd. Oh, just a note, if you are planning on accessing the local device
> > directly, you must not use the "-c" option when you are exporting the
> device.
> > This will eventually lead to corruption. The "-c" option is only for
> dedicated
> > gnbd servers.
> >
> > 2. Theoretically, you could just have every node export the devices to
> every
> > other node, and then build a logical volume on top of all the devices on
> each
> > node, but you should not do this. It totally destroys the benefit of
> having
> > a cluster. Since your GFS filesystem would then depend on having access to
> the
> > block devices of every machine, if ANY machine in your cluster went down,
> the
> > whole cluster would crash, because a piece of your filesystem would just
> > disappear.
> >
> >
> > Without shared storage, your gnbd server will be a single point of
> failure.
> > The most common way that people set up gnbd is with one dedicated gnbd
> server
> > machine, that is only used to serve gnbd blocks, so that it is unlikely to
> > crash.
> >
> > > Sorry if my english isn't very good ;)
> > >
> > > ----- Original Message ----- 
> > > From: "Kevin Anderson" <kanderso at redhat.com>
> > > To: "linux clustering" <linux-cluster at redhat.com>
> > > Sent: Sunday, November 05, 2006 10:12 PM
> > > Subject: Re: [Linux-cluster] gfs mounted but not working
> > >
> > >
> > > > On 11/5/06, *romero.cl at gmail.com <mailto:romero.cl at gmail.com>*
> > > > <romero.cl at gmail.com <mailto:romero.cl at gmail.com>> wrote:
> > > > >
> > > > >     Hi.
> > > > >
> > > > >     I'm trying your method, but still have a problem:
> > > > >
> > > > >     Note: /dev/db2/ is a local partition on my second SCSI hard
> drive
> > > > >     (no RAID)
> > > > >     runing on HP ProLiant.
> > > > >
> > > > GFS requires that all storage is equally accessible by all nodes in
> the
> > > > cluster.  Your other nodes have no path to the storage you set up so
> it
> > > > is impossible for them to share the data.
> > > >
> > > > Kevin
> > > >
> > > > --
> > > > Linux-cluster mailing list
> > > > Linux-cluster at redhat.com
> > > > https://www.redhat.com/mailman/listinfo/linux-cluster
> > >
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster