[Linux-cluster] GNBD & Network Outage

[Benjamin Marzinski]

On Tue, Jan 25, 2005 at 10:43:36AM +0000, Nigel Jewell wrote:
> Dear all,
> 
> We've been looking at the issues of using GNBD to provide access to a 
> block device on a secondary installation and we've hit a brick wall.  I 
> was wondering if anyone had seen the same behaviour
> 
> On host "A" we do:
> 
>    gnbd_export -d /dev/sda2 -e foo -c
> 
> On host "B" we do:
> 
>    gnbd_import -i A
> 
> ... and as you would expect /dev/gnbd/foo appears on B and is usable.
> 
> We have no other aspects of GFS in use.
> 
> Now - in order for this to be useful, we've been testing the effects of 
> using GNBD if there is a LAN outage.  If we write a big file to a 
> mounted file system on B:/dev/gnbd/foo and pull out the LAN cable 
> halfway through the data being synced to A, host B never gives up trying 
> to contact A.  In fact, if you plug in the cable 10 minutes later the 
> sync recovers.
> 
> Now - on the surface - this doesn't seem like a big problem, but it is 
> when you try and use the imported device alongside software RAID or when 
> you want to do something "normal" like reboot the box.  Rebooting just 
> stops when it trys to unmount the file systems.
> 
> We want to use B:/dev/gndb/foo alongside a local partition on B and 
> create a RAID-1 using mdadm.  In the same scenario (where the LAN cable 
> is pulled), the md device on B completely stops all of the IO on the 
> machine because (presumably) the md software is trying to write to the 
> gnbd device ... which is forever trying to contact host A ... and of 
> course never gives up.  It would be nice if it did give up and the md 
> software continued the md device in degraded mode.
> 
> So the question is this (got there in the end).  Can anyone suggest a 
> solution and/or alternative/workaround?  Is it possible to specify a 
> time-out for the GNBD import/export for when the LAN does die?

Sure. You see the -c in you export line.  Don't put it there.  That puts
the device in (the very poorly named) uncached mode.  This does two things.
One: It causes the server to use direct IO to write to the exported device,
so your read performance will take a hit.  Two: It will time out after
a period (default to 10 sec).  After gnbd times out, it must be able to fence
the server before it will let the requests fail.  This is so that you know
that the server isn't simply stalled and might write out the requests later
(if gnbd failed out, and the requests were rerouted to the backend storage over
another gnbd server, if the first server wrote it's requests out later, it
could cause data corruption).

This means that to run in uncached mode, you need to have a cluster manager and
fencing devices, which I'm not certain that you have.

I've got some questions about your setup.  Will this be part of a clustered
filesystem setup? If it will, I see some problems with your mirror.  When
other nodes (including the gnbd server node A) write to the exported device,
these writes will not appear on the local partion of B.  So won't your mirror
get out of sync?  If only B will write to the exported device, (and that's
the only way I see this working) you can probably get by with nbd, which
simply fails out if it loses connection.

There is a cluster mirror project in the works.  When that is done, you would
be able to have node B gnbd export it's local partition, and then run a mirror
on top of the device exported from A and the device exported from B, which
all nodes could access and would stay in sync. But this project isn't finished
yet.
 
-Ben

> Any ideas?
> 
> Regards,
> 
> -- 
> Nige.
> 
> PixExcel Limited
> URL: http://www.pixexcel.co.uk
> MSN: nigel.jewell at pixexcel.co.uk
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> http://www.redhat.com/mailman/listinfo/linux-cluster