[Linux-cluster] GNBD/GFS question

[James Fait]

I have run into the following problem using GNBD/GFS on a 4 node test
cluster I have set up.  I have been testing failure scenarios to make
sure that things will be handled correctly when a node is fenced using
fence_gnbd.
The cluster is a set of 4 dell 1u servers, with dual 1GB network
connections, one to the intranet, and the other to a storage net.  The
cluster communications all happen on the storage net.  Each machine has
a dns name, which is decoded to the hostname for logs, which is the
intranet address. The storage net also has name to ip resolution via the
hosts file, to names of the form cluster*.  Cluster29 is the gnbd server
node, and does not mount the gfs volume at all.  The other three nodes
do mount the gfs volume.

This is the test that is currently causing the problem.
        I kill the network link for cluster28(com28) using the network
        switch to temporarily disable the link.
        
        Cluster28 is fenced successfully. One of the other nodes handles
        the gfs cleanup.
        
        The network link is restored.
        
        Cluster28 attempts an orderly shutdown, and hangs on umount of
        gfs.  Later, it is power cycled to force reboot.
        
        Cluster28 rejoins the cluster, and attempts to mount gfs.  The
        gnbd_import command fails with the error messages:
                gnbd_recvd: ERROR login refused by the server,
                quitting : Operation not permitted
                gnbd_import: ERROR gnbd_recvd failed
        
        Cluster29 reports the error:
                com29 gnbd_serv[4794]: ERROR [gserv.c:468] client
                cluster28 is banned. Canceling login

What is happening, and what can I do about it?

Sincerly

Jim
-- 
James Fait, Ph.D.
Beamline Scientist, SER-CAT
APS, building 436B-008
Argonne National Laboratory
9700 S Cass Ave
Argonne, IL 60439
phone 630-252-0644
fax   630-252-0652
email fait at anl.gov