[Linux-cluster] 10 Node Installation - Loosing Heartbeat

[Richard Mayhew]

Hi All,
 
I have been able to successfully setup a 10 node GFS cluster, 4 Locking
Servers and 6 Clients. Each server is a Dell PowerEdge 1750 with 1GB
RAM, Dual P4 2.8 HT running RedHat Enterprise Server V3 Update 4. The
first ethernet interface is used for normal network traffic and the
second ethernet interface is used for GFS heatbeats only. Both
interfaces run at 1GB-FD on 2 separate switches. I have installed the
GFS-6.0.2-25 from http://bender.it.swin.edu.au/centos-3/RHGFS/i686/ on
the latest RedHat ES Kernel.
 
The storage is made available using a EMC CX600 SAN. (4 x 50GB) running
over a McData Fiber Switch. The pool service uses the storage from the
san over the EMC Powepath software using the emcpower pseudo devices.
Pool is able to assemble the pools with no problem. The ccsd service is
able to retrieve the gfs archive with no problems. The lock gulm server
loads with no problem and communicates with the master lock server with
out any errors or missed beats. GFS mounts with out any errors etc. I am
able to access the storage on each server with no problem on throughput.

 
The problem I am experiencing is as follows.
 
Once the GFS system has been running for a few hours with some usage on
each of the servers, some of the servers start missing beats. I
increased the heartbeat rate to test every 60 seconds and to fail after
10 tries. This just prolonged the servers being fenced. The only thing I
can come up with is that the locking server is buggy and stops
responding to heartbeats. On the master server when it detects that the
server has skipped the required number of beats, it tries to fence it
and fails. I have setup the fencing to use the mcdata module and I have
specified the correct login details. When the server that was fenced has
had its lock server restarted it tries to relog in to the master lock
server. This fails for obvious reasons as the master will refuse to
allow it to reconnect due to the previous fencing failures. Manual
fencing works without any problems but I have only tried this on the cmd
line.
 
Does anyone have an idea as to why the locking servers are hanging up
when it comes to sending heartbeat beats and possibly why the fencing
isnt working?
 
Here are my configs with some of the privileged information changed.
 
fence.css
fence_devices {

	EMC_Switch_01 {
	agent = "fence_mcdata"
	ipaddr = "xxx.xxx.xxx.xxx"
	login = "XXXXXXXX"
	password = "xxxxxx"
	}

I	EMC_Switch_02 {
	agent = "fence_mcdata"
	ipaddr = "xxx.xxx.xxx.xxx"
	login = "XXXXXXXXXX"
	password = "xxxxxx"
	}
}


Cluster.ccs
cluster {
        name = "mail"
        lock_gulm 
        servers =
["store-01.mc.mweb.net","store-02.mc.mweb.net","store-03.mc.mweb.net","s
tore-04.mc.mweb.net"]
        heartbeat_rate = 60
        allowed_misses = 10
        }
}

Nodes.ccs
nodes   {
   store-01.mc.mweb.net    {
       ip_interfaces   {
             eth1 = "xxx.xxx.xxx.xxx"
       }
       fence {
         san {
          EMC_Switch_01   {
              port = 3
          }
         }
       }
    }

   store-02.mc.mweb.net  {
        ip_interfaces {
              eth1 = "xxx.xxx.xxx.xxx"
        }
        fence {
          san {
            EMC_Switch_01 {
                port = 27
            }
          }
        }
    }

   store-03.mc.mweb.net  {
        ip_interfaces {
              eth1 = "xxx.xxx.xxx.xxx"
        }
        fence {
          san {
            EMC_Switch_01 {
                port = 9
            }
          }
        }
    }
   store-04.mc.mweb.net  {
        ip_interfaces {
              eth1 = "xxx.xxx.xxx.xxx"
        }
        fence {
          san {
            EMC_Switch_01 {
                port = 31
            }
          }
        }
    }

        serv-01.mc.mweb.net {
                                ip_interfaces {
                                                        eth1 =
"xxx.xxx.xxx.xxx"
                                }
                                fence {
                                        san {
                                                        EMC_Switch_01 {
                                                                port =
19
                                                        }
                                        }
                                }
                        }
        serv-02.mc.mweb.net {
                                ip_interfaces {
                                                eth1 = "xxx.xxx.xxx.xxx"
                                }
                                fence {
                                        san {
                                                        EMC_Switch_01 {
                                                                port =
27
                                        }
                               }
                        }
                }
        serv-03.mc.mweb.net {
                        ip_interfaces {
                                                eth1 = "xxx.xxx.xxx.xxx"
                        }
                        fence {
                                san {
                                                EMC_Switch_01 {
                                                        port = 31
                                                }
                                        }
                                }
                        }
        serv-04.mc.mweb.net {
                        ip_interfaces {
                                                eth1 = "xxx.xxx.xxx.xxx"
                        }
                        fence {
                                san {
                                                EMC_Switch_02 {
                                                        port = 3
                                                }
                                        }
                                }
                        }
        serv-05.mc.mweb.net {
                        ip_interfaces {
                                                eth1 = "xxx.xxx.xxx.xxx"
                        }
                        fence {
                                san {
                                                EMC_Switch_02 {
                                                        port = 9
                                                }
                       ip_interfaces {
                                                eth1 = "xxx.xxx.xxx.xxx"
                        }
                        fence {
                                san {
                                                EMC_Switch_02 {
                                                        port = 3
                                                }
                                        }
                                }
                        }
        serv-05.mc.mweb.net {
                        ip_interfaces {
                                                eth1 = "xxx.xxx.xxx.xxx"
                        }
                        fence {
                                san {
                                                EMC_Switch_02 {
                                                        port = 9
                                                }
                                        }
                                }
                        }
        serv-06.mc.mweb.net {
                        ip_interfaces {
                                        eth1 = "xxx.xxx.xxx.xxx"
                        }
                        fence {
                                san {
                                                EMC_Switch_02 {
                                                                port =
19
                                                }
                                        }
                                }
                        }
}
--

Regards

Richard Mayhew
Unix Specialist

MWEB Business
Tel:  + 27 11 340 7200
Fax:  + 27 11 340 7288
Website: www.mwebbusiness.co.za