[Linux-cluster] AW: GNBD multipath with devicemapper? -- possible solution

[Benjamin Marzinski]

On Fri, Apr 15, 2005 at 04:24:21PM +0200, Hansjoerg.Maurer at dlr.de wrote:
> Hi
> 
> I found a solution for the problem descriped below,
> but I am not sure if it is the right way.
> 
> - importing the two gnbd's (wich point to the same device) from two servers
> -> /dev/gnbd0 and /dev/gnbd1 on the client
> 
> - creating a multipath device with something like this:
> echo "0 167772160  multipath 0 0 1 1 round-robin 0 2 1 251:0 1000 251:1 1000 " | dmsetup create dm0
>  (251:0 ist the major:minor id of /dev/gnbd0)
> 
> - mounting the created device
> eg:
> mount -t gfs /dev/mapper/dm0 /mnt/lvol0
> 
> If I do a write on /mnt/lvol0 the gnbd_server task on both gnbd_servers start (with a noticeable speedup)
> 
> If one gnbd_server fails dm removes that path with the following log
> kernel: device-mapper: dm-multipath: Failing path 251:0.
> 
> I was able to add it again with
> 
> dmsetup  message dm0 0  reinstate_path 251:0
> 
> 
> I was able to deactivate a path manually with
> 
> dmsetup  message dm0 0  fail_path 251:0 
> 
> But I can not unimport the underlying gnbd
> 
> gnbd_import: ERROR cannot disconnect device #1 : Device or resource busy
> 
> 
> Is there a way to remove a gnbd, which is bunndled in a dm-multipath device?
> (might be necessary, if one gnbd server must be rebooted)
> 
> How can I reimport an gnbd on the client in state disconnected?
> (I had to manually start 
> gnbd_recvd -d 0  to do so)
> 
> Is the descriped solution for gnbd multipath the right one?

Um... It's a really ugly one. Unfortunately it's the only one that works, since
multipath-tools do not currently support non-scsi devices.

There are also some bugs in gnbd that make multipathing even more annoying.

But to answer your question, in order to remove gnbd, you must first get it
out of the multipath table, otherwise dm-multipath will still have it open.

To do this, after dmsetup status shows that the path is failed, you run:

# echo "0 167772160  multipath 0 0 1 1 round-robin 0 1 1 251:1 1000 " | dmsetup reload dm0
# dmsetup resume dm0

This removes the gnbd from the path.

However, if you use the gnbd code from the cvs head, it is no longer necessary
to do this to reimport the device.  In the stable branch, gnbd_monitor waits
until all users close the device before setting it to restartable. In the head
code, this happens as soon as the device is successfully fenced. So, if you
loose a gnbd server, reboot it, and reexport the device, gnbd_monitor should
automatically reimport the device, and you can simply run 

# dmsetup  message dm0 0  reinstate_path 251:0

and you should never need to remove the gnbd device with the method I described
above.


-Ben

> Thank you very much
> 
> Greetings from munich
> 
> Hansjörg
> 
> 
> 
> 
> 
> 
> >Hi
> >
> >I am trying to set up gnbd with multipath.
> >Accoding to the gnbd_usage.txt file, I understand, that this should work with
> >dm-multipath.
> >But unfortunatly only the gfs part of the setup is descriped there.
> >
> >Has anybody experiance with this setup, especially how to set up
> >multipath with multiple /dev/gnbd* and how to setup the multipath.conf file
> >
> >
> >Thank you very much
> >
> >Hansjörg Maurer
> -- 
> _________________________________________________________________
> 
> Dr.  Hansjoerg Maurer           | LAN- & System-Manager
>                                 |
> Deutsches Zentrum               | DLR Oberpfaffenhofen
>   f. Luft- und Raumfahrt e.V.   |
> Institut f. Robotik             |
> Postfach 1116                   | Muenchner Strasse 20
> 82230 Wessling                  | 82234 Wessling
> Germany                         |
>                                 |
> Tel: 08153/28-2431              | E-mail: Hansjoerg.Maurer at dlr.de
> Fax: 08153/28-1134              | WWW: http://www.robotic.dlr.de/
> __________________________________________________________________
> 
> 
> There are 10 types of people in this world, 
> those who understand binary and those who don't.
> 
> 
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> http://www.redhat.com/mailman/listinfo/linux-cluster