[Linux-cluster] Adding a new node to rh cluster + GFS2

[Arturo Gonzalez Ferrer]

Dear all,

I'm in trouble with adding a new node to an existing cluster of three nodes
(so I want to have four), because it somehow doesn't let me access the
cluster infrastructure.

These 3 nodes were set up as http servers, sharing a GFS2 volume (physical:
vg_cluster, logical: lv_cluster) where data is stored.

I want to set up the new node to access the same GFS2 volume, with the idea
of exporting the data via NFS, so that a remote backup library can be
configured to backup nightly the data, by connecting to the new node.

I've tried a lot of things, always getting same kind of errors.

Running "cman_tool status" on any of the 3 nodes i get:

Version: 6.2.0
Config Version: 70
Cluster Name: campusvirtual
Cluster Id: 45794
Cluster Member: Yes
Cluster Generation: 1136
Membership state: Cluster-Member
Nodes: 3
Expected votes: 4
Total votes: 3
Quorum: 3
Active subsystems: 9
Flags: Dirty
Ports Bound: 0 11 177
Node name: cev01
Node ID: 2
Multicast addresses: 239.192.178.149
Node addresses: 150.214.243.20


while running "cman_tool status" on the new node:

Version: 6.2.0
Config Version: 70
Cluster Name: campusvirtual
Cluster Id: 45794
Cluster Member: Yes
Cluster Generation: 1124
Membership state: Cluster-Member
Nodes: 1
Expected votes: 4
Total votes: 1
Quorum: 3 Activity blocked
Active subsystems: 2
Flags:
Ports Bound: 0
Node name: cevstream.ugr.es
Node ID: 4
Multicast addresses: 239.192.178.149
Node addresses: 150.214.243.19


Running "fence_tool_dump" on the three nodes:

[root at cev01 ~]# fence_tool dump
dump read: Success
1260778939 our_nodeid 2 our_name cev01.ugr.es
1260778939 listen 4 member 5 groupd 7
1260778964 client 3: join default
1260778964 delay post_join 3s post_fail 0s
1260778964 added 4 nodes from ccs
1260778964 setid default 65538
1260778964 start default 1 members 2
1260778964 do_recovery stop 0 start 1 finish 0
1260778964 node "cevstream.ugr.es" not a cman member, cn 1
1260778964 add first victim cevstream.ugr.es
1260778965 node "cevstream.ugr.es" not a cman member, cn 1
1260778966 node "cevstream.ugr.es" not a cman member, cn 1
1260778967 node "cevstream.ugr.es" not a cman member, cn 1
1260778967 delay of 3s leaves 1 victims
1260778967 node "cevstream.ugr.es" not a cman member, cn 1
1260778967 node "cevstream.ugr.es" has not been fenced
1260778967 fencing node cevstream.ugr.es
1260778971 finish default 1
1260778971 stop default
1260778971 start default 2 members 3 2
1260778971 do_recovery stop 1 start 2 finish 1
1260778971 finish default 2
1260778971 stop default
1260778971 start default 3 members 1 3 2
1260778971 do_recovery stop 2 start 3 finish 2
1260778971 finish default 3
1260779876 client 3: dump

while running it in the new node:

[root at cevstream ~]# fence_tool dump
fence_tool: can't communicate with fenced

I get a lot of errors telling me that cluster is not quorate:

Dec 14 09:39:20 cevstream ccsd[3668]: Cluster is not quorate.  Refusing
connection.
Dec 14 09:39:20 cevstream ccsd[3668]: Error while processing connect:
Connection refused

Printing the superblock on any of the three nodes:

[root at cev01 ~]# gfs2_tool sb /dev/vg_cluster/lv_cluster all
  mh_magic = 0x01161970
  mh_type = 1
  mh_format = 100
  sb_fs_format = 1801
  sb_multihost_format = 1900
  sb_bsize = 4096
  sb_bsize_shift = 12
  no_formal_ino = 2
  no_addr = 23
  no_formal_ino = 1
  no_addr = 22
  sb_lockproto = lock_dlm
  sb_locktable = campusvirtual:gfs_cluster01
  uuid = C6A9FBB4-A881-2128-2AB8-1AB8547C7F30


I've tried something i saw in some forums, deactivating and even removing
the logical volume (with lvremove), because supposedly the new node could
need this operation in order to access the gfs2 volume.

Running lvcreate on the new node, with all the other nodes deactivates and
removed, i still get the error:

[root at cevstream ~]# lvcreate -l 100%FREE -n lv_cluster vg_cluster
  connect() failed on local socket: Conexión rehusada
  WARNING: Falling back to local file-based locking.
  Volume Groups with the clustered attribute will be inaccessible.
  Skipping clustered volume group vg_cluster

Find attached the configuration of cluster.conf.

I'm pretty desperate with this situation, i really don't know how to deal
with the adition of a new node.

Best regards,
Arturo.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20091214/a24fda90/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cluster.conf
Type: application/octet-stream
Size: 3463 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20091214/a24fda90/attachment.obj>