From alban.crequy at seanodes.com Fri Jul 1 08:36:46 2005 From: alban.crequy at seanodes.com (Alban Crequy) Date: Fri, 01 Jul 2005 10:36:46 +0200 Subject: [Linux-cluster] Fencing the gulm master node: problem Message-ID: <42C5009E.4080108@seanodes.com> Hello, I want to test the fencing capability of GFS by unplugging the network on a node. But I experience some problems when the node I unplug is the gulm master. I am using the RPM: - GFS-6.0.2.20-2 - GFS-modules-smp-6.0.2.20-2 I have a 8-nodes cluster (sam21, sam22, ..., sam28). I mount a GFS filesystem on all nodes on /mnt/gfs My config is: ----->8-------->8-------->8-------->8--- # fence.ccs fence_devices { admin { agent="fence_manual" } } # cluster.ccs cluster { name="sam" lock_gulm { servers=["sam21", "sam22", "sam23", "sam24", "sam25"] } } # nodes.ccs nodes { sam21.toulouse { ip_interfaces { eth0 = "192.168.0.121" } fence { human { admin { ipaddr = "192.168.0.121" } } } } # etc. for sam22 ... sam28 ----->8-------->8-------->8-------->8--- I want to check that the unplugged node is fenced and its locks are released when I run "fence_ack_manual" (and only when I run fence_ack_manual", not before). In order to know when the locks are released, I wrote a small program: ----->8-------->8-------->8-------->8--- // lock.c fd = open("/mnt/gfs/lock-test.tmp", O_RDWR|O_CREAT, S_IREAD|S_IWRITE); if (fd == -1) { printf("ERROR: open failed.\n"); return 1; } error = flock(fd, LOCK_EX); if (error == -1) { printf("ERROR: lock failed.\n"); return 1; } while (1) { printf("writing... pid %d : %d\n", pid, counter++); buf[0]=0; p = sprintf(buf, "pid %d : %d\n", pid, counter); write(fd, buf, p); sleep(1); } ----->8-------->8-------->8-------->8--- First test (which works): - I run my lock.c program on sam26 (not a gulm server) - The lock is acquired on sam26 - I run my lock.c program on all other nodes - The other nodes wait for the lock - I unplug sam26 and wait until the gulm master (sam21) want to fence sam26 - The gulm master (sam21) want me to run fence_ack_manual - The lock is not taken on other node - I run fence_ack_manual - The lock is released on the unplugged node (sam26) and taken by another => So when I unplug a node which is not a gulm server, all work correctly. Second test (which doesn't work): - I run my lock.c program on sam21 (the gulm master) - The lock is acquired on sam21 - I run my lock.c program on all other nodes - The other nodes wait for the lock - I unplug sam21 and wait until a new gulm master (sam22) want to fence the old master (sam21) - The new gulm master (sam22) want me to run fence_ack_manual BUT the lock is released immediately. I did not run fence_ack_manual and the lock is already released. This is my problem. I read the bug reports [1][2] and the advisory RHBA-2005:466-11 [3] which says ? Fixed a problem in which a gulm lock server ran on GFS clients after the master server died. ? But I use GFS-6.0.2.20-2. [1] https://bugzilla.redhat.com/beta/show_bug.cgi?id=148029 [2] https://bugzilla.redhat.com/beta/show_bug.cgi?id=149119 [3] http://rhn.redhat.com/errata/RHBA-2005-466.html Is this a bug? Or a misunderstanding of the fencing mechanism? The syslogs on the new gulm master (sam22) are: ----->8-------->8-------->8-------->8--- Jul 1 09:06:52 sam22 lock_gulmd_core[4195]: Failed to receive a timely heartbeat reply from Master. (t:1120201612489192 mb:1) Jul 1 09:07:07 sam22 lock_gulmd_core[4195]: Failed to receive a timely heartbeat reply from Master. (t:1120201627509192 mb:2) Jul 1 09:07:22 sam22 lock_gulmd_core[4195]: Failed to receive a timely heartbeat reply from Master. (t:1120201642529191 mb:3) Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: I see no Masters, So I am Arbitrating until enough Slaves talk to me. Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send quorum update to slave sam23.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send quorum update to slave sam28.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send quorum update to slave sam26.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send quorum update to slave sam25.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send quorum update to slave sam24.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send quorum update to slave sam22.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send quorum update to slave sam27.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: LastMaster sam21.toulouse:192.168.0.121, is being marked Expired. Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send membership update "Expired" about sam21.toulouse to slave sam23.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send membership update "Expired" about sam21.toulouse to slave sam28.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send membership update "Expired" about sam21.toulouse to slave sam26.toulouse Jul 1 09:07:37 sam22 lock_gulmd_LTPX[4197]: New Master at sam22.toulouse:192.168.0.122 Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send membership update "Expired" about sam21.toulouse to slave sam25.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send membership update "Expired" about sam21.toulouse to slave sam24.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send membership update "Expired" about sam21.toulouse to slave sam22.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Could not send membership update "Expired" about sam21.toulouse to slave sam27.toulouse Jul 1 09:07:37 sam22 lock_gulmd_core[4195]: Forked [4882] fence_node sam21.toulouse with a 0 pause. Jul 1 09:07:37 sam22 lock_gulmd_core[4882]: Gonna exec fence_node sam21.toulouse Jul 1 09:07:37 sam22 fence_node[4882]: Performing fence method, human, on sam21.toulouse. Jul 1 09:07:37 sam22 fence_manual: Node 192.168.0.121 requires hard reset. Run "fence_ack_manual -s 192.168.0.121" after power cycling the machine. Jul 1 09:07:38 sam22 lock_gulmd_core[4195]: Still in Arbitrating: Have 1, need 3 for quorum. Jul 1 09:07:39 sam22 lock_gulmd_core[4195]: Still in Arbitrating: Have 2, need 3 for quorum. Jul 1 09:07:39 sam22 lock_gulmd_core[4195]: New Client: idx:5 fd:10 from (192.168.0.124:sam24.toulouse) Jul 1 09:07:39 sam22 lock_gulmd_core[4195]: Member update message Logged in about sam24.toulouse to sam23.toulouse is lost because node is in OM Jul 1 09:07:39 sam22 lock_gulmd_core[4195]: Member update message Logged in about sam24.toulouse to sam28.toulouse is lost because node is in OM Jul 1 09:07:39 sam22 lock_gulmd_core[4195]: Member update message Logged in about sam24.toulouse to sam25.toulouse is lost because node is in OM Jul 1 09:07:39 sam22 lock_gulmd_core[4195]: Member update message Logged in about sam24.toulouse to sam27.toulouse is lost because node is in OM Jul 1 09:07:39 sam22 lock_gulmd_LT000[4196]: Attached slave sam24.toulouse:192.168.0.124 idx:2 fd:7 (soff:3 connected:0x8) Jul 1 09:07:39 sam22 lock_gulmd_core[4195]: Still in Arbitrating: Have 2, need 3 for quorum. Jul 1 09:07:39 sam22 lock_gulmd_core[4195]: Still in Arbitrating: Have 2, need 3 for quorum. Jul 1 09:07:41 sam22 lock_gulmd_core[4195]: Now have Slave quorum, going full Master. Jul 1 09:07:41 sam22 lock_gulmd_core[4195]: New Client: idx:6 fd:11 from (192.168.0.123:sam23.toulouse) Jul 1 09:07:41 sam22 lock_gulmd_core[4195]: Member update message Logged in about sam23.toulouse to sam25.toulouse is lost because node is in OM Jul 1 09:07:41 sam22 lock_gulmd_LTPX[4197]: Logged into LT000 at sam22.toulouse:192.168.0.122 Jul 1 09:07:41 sam22 lock_gulmd_LT000[4196]: New Client: idx 3 fd 8 from (192.168.0.122:sam22.toulouse) Jul 1 09:07:41 sam22 lock_gulmd_LTPX[4197]: Finished resending to LT000 Jul 1 09:07:41 sam22 lock_gulmd_LT000[4196]: Attached slave sam23.toulouse:192.168.0.123 idx:4 fd:9 (soff:2 connected:0xc) Jul 1 09:07:41 sam22 lock_gulmd_LT000[4196]: New Client: idx 5 fd 10 from (192.168.0.123:sam23.toulouse) Jul 1 09:07:41 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=0: Trying to acquire journal lock... Jul 1 09:07:41 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=0: Looking at journal... Jul 1 09:07:41 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=0: Acquiring the transaction lock... Jul 1 09:07:42 sam22 lock_gulmd_LT000[4196]: New Client: idx 6 fd 11 from (192.168.0.124:sam24.toulouse) Jul 1 09:07:45 sam22 lock_gulmd_core[4195]: New Client: idx:3 fd:7 from (192.168.0.126:sam26.toulouse) Jul 1 09:07:45 sam22 lock_gulmd_core[4195]: Member update message Logged in about sam26.toulouse to sam25.toulouse is lost because node is in OM Jul 1 09:07:45 sam22 lock_gulmd_LT000[4196]: New Client: idx 7 fd 12 from (192.168.0.126:sam26.toulouse) Jul 1 09:07:46 sam22 lock_gulmd_core[4195]: New Client: idx:7 fd:12 from (192.168.0.128:sam28.toulouse) Jul 1 09:07:46 sam22 lock_gulmd_core[4195]: Member update message Logged in about sam28.toulouse to sam25.toulouse is lost because node is in OM Jul 1 09:07:46 sam22 lock_gulmd_LT000[4196]: New Client: idx 8 fd 13 from (192.168.0.128:sam28.toulouse) Jul 1 09:07:47 sam22 lock_gulmd_core[4195]: New Client: idx:8 fd:13 from (192.168.0.125:sam25.toulouse) Jul 1 09:07:47 sam22 lock_gulmd_LT000[4196]: Attached slave sam25.toulouse:192.168.0.125 idx:9 fd:14 (soff:1 connected:0xe) Jul 1 09:07:47 sam22 lock_gulmd_LT000[4196]: New Client: idx 10 fd 15 from (192.168.0.125:sam25.toulouse) Jul 1 09:07:49 sam22 lock_gulmd_core[4195]: New Client: idx:9 fd:14 from (192.168.0.127:sam27.toulouse) Jul 1 09:07:49 sam22 lock_gulmd_LT000[4196]: New Client: idx 11 fd 16 from (192.168.0.127:sam27.toulouse) Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=0: Replaying journal... Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=0: Replayed 0 of 2 blocks Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=0: replays = 0, skips = 1, sames = 1 Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=0: Journal replayed in 9s Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=0: Done Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=7: Trying to acquire journal lock... Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=7: Busy Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=6: Trying to acquire journal lock... Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=6: Busy Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=5: Trying to acquire journal lock... Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=5: Busy Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=7: Trying to acquire journal lock... Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=7: Busy Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=6: Trying to acquire journal lock... Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=6: Busy Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=5: Trying to acquire journal lock... Jul 1 09:07:50 sam22 kernel: GFS: fsid=sam:grp_gfs.1: jid=5: Busy ----->8-------->8-------->8-------->8--- Sincerely, Alban Crequy From gwood at dragonhold.org Fri Jul 1 09:31:50 2005 From: gwood at dragonhold.org (gwood at dragonhold.org) Date: Fri, 1 Jul 2005 10:31:50 +0100 (BST) Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <42C429C7.3070200@possibilityforge.com> References: <1119649059.13563.18.camel@ayanami.boston.redhat.com> <42BF9FCF.7060000@redhat.com> <42C24610.2@redhat.com> <42C2AD19.5000601@redhat.com> <42C39DFC.7070209@redhat.com> <42C3FD0A.4040800@possibilityforge.com> <42C4002C.80403@redhat.com> <42C402E9.2080502@possibilityforge.com> <42C4067F.6000903@redhat.com> <42C40F58.4040401@oakland.edu> <21339.208.178.77.200.1120145458.squirrel@208.178.77.200> <42C42352.3000600@possibilityforge.com> <12897.208.178.77.200.1120151133.squirrel@208.178.! 77.200> <42C429C7.3070200@possibilityforge.com> Message-ID: <16201.208.178.77.200.1120210310.squirrel@208.178.77.200> > node 1 accesses gfs mount/some_dir > node 2 accesses gfs mount/some_other_dir > node 3 accesses gfs mount/yet_some_other_dir This won't totally solve the same problem - for two reasons. Firstly, the metadata for "some_dir" and "some_other_dir" are going to be stored in the same place ("mount"'s directory) - which means that access time, permission changes, and other metadata will be cached within each of the split brains quite happily. The more serious problem is new/deleted files - since the 2 mini-clusters will think they are allocating disk space from a device they are solely using, they will each start allocating from the same place (e.g. nodes 1-3 could use block offset 12345 for new file /mount/some_dir/file1, at the same time as nodes 4-6 usr block offset 12345 for new file /mount/yet_some_other_dir/file2) While each half uses the cached data, you'll just get data corruption (the contents of the file will be the last one written). However, as soon as they go back to the disk to look things up they may notice the cross linked files, and either whinge or die. And when you next fsck, it'll split them out and that's the first time you'll definitely know. If you're just after sharing the space on disk (and not sharing the data within the partitions) then clvm may be the answer - the only thing you won't be able to do (from memory) without quorum is resize the partitions. Run ext3 on top of that for each node? From jscottorn at possibilityforge.com Fri Jul 1 13:46:48 2005 From: jscottorn at possibilityforge.com (Jon Scottorn) Date: Fri, 01 Jul 2005 07:46:48 -0600 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <42C46EAF.9050703@possibilityforge.com> References: <42C46EAF.9050703@possibilityforge.com> Message-ID: <42C54948.9070404@possibilityforge.com> Does any one have any thoughts about my problem? I have been fighting with fenced for the last couple of days now. Jon Scottorn wrote: >Thanks, > >Another little issue, I can't get fenced to start. > >I issue fenced -cD and it gives > >1120169493 our_nodeid 1 our_name data >1120169493 group_init error 0 111 > >What have I done wrong here? > >Thanks, > >Jon > >Dan B. Phung wrote: > > > >>while [ 1 ]; then >> cman_tool expected -e 1 >> sleep 600 >>done >> >> >>On 30, Jun, 2005, Jon Scottorn declared: >> >> >> >> >> >>>How would something like that look? >>> >>>would you use cman_tool to do it? >>> >>> >>> >>>Lon Hohberger wrote: >>> >>> >>> >>> >>> >>>>On Thu, 2005-06-30 at 10:52 -0600, Jon Scottorn wrote: >>>> >>>> >>>> >>>> >>>> >>>> >>>>>Ok, so let me reiterate: >>>>> >>>>> If I don't even care about quorum and the cluster. I just want a >>>>>filesystem that will server out a block device, which is what gfs does. >>>>>I'm not worried about "split brain" issues. If we need to have quorum, >>>>>I want the number of nodes for quorum to be set at 1, which will be the >>>>>main server containing the data. Any other node that connectes can just >>>>>access the data or go offline without causeing any quorum issues. >>>>>Is this functionality going to be possible, I want to use GFS for this >>>>>because if not, our other option is enbd but then we are limiting >>>>>ourselves very much. We would have to create seperate partitions for >>>>>each node to mount, etc... a major PAIN to go that way but, it is not as >>>>>painful of having quorum fail and cause all of our nodes to go down. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>Set a cronjob to reset expected votes every 10 minutes... >>>> >>>>Good luck. >>>> >>>>-- Lon >>>> >>>> >>>> >>>>-- >>>>Linux-cluster mailing list >>>>Linux-cluster at redhat.com >>>>http://www.redhat.com/mailman/listinfo/linux-cluster >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>-- >>>Linux-cluster mailing list >>>Linux-cluster at redhat.com >>>http://www.redhat.com/mailman/listinfo/linux-cluster >>> >>> >>> >>> >>> >> >> >> >> > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster > > > From pcaulfie at redhat.com Fri Jul 1 14:07:35 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Fri, 01 Jul 2005 15:07:35 +0100 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <42C54948.9070404@possibilityforge.com> References: <42C46EAF.9050703@possibilityforge.com> <42C54948.9070404@possibilityforge.com> Message-ID: <42C54E27.1020303@redhat.com> Jon Scottorn wrote: > Does any one have any thoughts about my problem? I have been fighting > with fenced for the last couple of days now. > > Jon Scottorn wrote: > > >>Thanks, >> >>Another little issue, I can't get fenced to start. >> >>I issue fenced -cD and it gives >> >>1120169493 our_nodeid 1 our_name data >>1120169493 group_init error 0 111 >> >>What have I done wrong here? >> Are you using head of CVS? that's not really ready yet. -- patrick From jscottorn at possibilityforge.com Fri Jul 1 14:13:26 2005 From: jscottorn at possibilityforge.com (Jon Scottorn) Date: Fri, 01 Jul 2005 08:13:26 -0600 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <42C54E27.1020303@redhat.com> References: <42C46EAF.9050703@possibilityforge.com> <42C54948.9070404@possibilityforge.com> <42C54E27.1020303@redhat.com> Message-ID: <42C54F86.8010402@possibilityforge.com> Yes, I am. Which one should I use than? Patrick Caulfield wrote: >Jon Scottorn wrote: > > >>Does any one have any thoughts about my problem? I have been fighting >>with fenced for the last couple of days now. >> >>Jon Scottorn wrote: >> >> >> >> >>>Thanks, >>> >>>Another little issue, I can't get fenced to start. >>> >>>I issue fenced -cD and it gives >>> >>>1120169493 our_nodeid 1 our_name data >>>1120169493 group_init error 0 111 >>> >>>What have I done wrong here? >>> >>> >>> > >Are you using head of CVS? that's not really ready yet. > > > From jscottorn at possibilityforge.com Fri Jul 1 14:24:50 2005 From: jscottorn at possibilityforge.com (Jon Scottorn) Date: Fri, 01 Jul 2005 08:24:50 -0600 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <42C54E27.1020303@redhat.com> References: <42C46EAF.9050703@possibilityforge.com> <42C54948.9070404@possibilityforge.com> <42C54E27.1020303@redhat.com> Message-ID: <42C55232.40808@possibilityforge.com> Hi, Thanks for the response. I was using head but now I went back to the version I was using before that was working. Now I get this error fenced -cD fenced: 1120227794 our_nodeid 1 our_name data fenced: 1120227794 delay post_join 6s post_fail 0s fenced: 1120227794 clean start, skipping initial nodes fenced: 1120227794 group_init error 0 111 free(): invalid pointer 0x8051300! Any Ideas? Thanks again, Jon Patrick Caulfield wrote: >Jon Scottorn wrote: > > >>Does any one have any thoughts about my problem? I have been fighting >>with fenced for the last couple of days now. >> >>Jon Scottorn wrote: >> >> >> >> >>>Thanks, >>> >>>Another little issue, I can't get fenced to start. >>> >>>I issue fenced -cD and it gives >>> >>>1120169493 our_nodeid 1 our_name data >>>1120169493 group_init error 0 111 >>> >>>What have I done wrong here? >>> >>> >>> > >Are you using head of CVS? that's not really ready yet. > > > From mtilstra at redhat.com Fri Jul 1 15:12:50 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Fri, 1 Jul 2005 10:12:50 -0500 Subject: [Linux-cluster] Fencing the gulm master node: problem In-Reply-To: <42C5009E.4080108@seanodes.com> References: <42C5009E.4080108@seanodes.com> Message-ID: <20050701151250.GA4400@redhat.com> On Fri, Jul 01, 2005 at 10:36:46AM +0200, Alban Crequy wrote: > I want to test the fencing capability of GFS by unplugging the network on a > node. But I experience some problems when the node I unplug is the gulm > master. > > I am using the RPM: > - GFS-6.0.2.20-2 > - GFS-modules-smp-6.0.2.20-2 > > I have a 8-nodes cluster (sam21, sam22, ..., sam28). I mount a GFS > filesystem on all nodes on /mnt/gfs [snip] > Is this a bug? Or a misunderstanding of the fencing mechanism? No, you've got the right idea. It looks like a bug. ick. So can you file a bugzilla for this? Also, can you try things where there is no gfs mounted on the gulm master, then unplug that node and see if the cluster behaves? thanks. -- Michael Conrad Tadpol Tilstra Any technology, no matter how primitive, is magic to those who don't understand it. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From pcaulfie at redhat.com Fri Jul 1 15:30:51 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Fri, 01 Jul 2005 16:30:51 +0100 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <42C54F86.8010402@possibilityforge.com> References: <42C46EAF.9050703@possibilityforge.com> <42C54948.9070404@possibilityforge.com> <42C54E27.1020303@redhat.com> <42C54F86.8010402@possibilityforge.com> Message-ID: <42C561AB.7070309@redhat.com> Jon Scottorn wrote: > Yes, I am. Which one should I use than? > STABLE is probably the best CVS tag to use if you want something that actually works - rather than testing new stuff. Patrick From lhh at redhat.com Fri Jul 1 15:49:06 2005 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 01 Jul 2005 11:49:06 -0400 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <16201.208.178.77.200.1120210310.squirrel@208.178.77.200> References: <1119649059.13563.18.camel@ayanami.boston.redhat.com> <42BF9FCF.7060000@redhat.com> <42C24610.2@redhat.com> <42C2AD19.5000601@redhat.com> <42C39DFC.7070209@redhat.com> <42C3FD0A.4040800@possibilityforge.com> <42C4002C.80403@redhat.com> <42C402E9.2080502@possibilityforge.com> <42C4067F.6000903@redhat.com> <42C40F58.4040401@oakland.edu> <21339.208.178.77.200.1120145458.squirrel@208.178.77.200> <42C42352.3000600@possibilityforge.com> <12897.208.178.77.200.1120151133.squirrel@208.178.! 77.200> <42C429C7.3070200@possibilityforge.com> <16201.208.178.77.200.1120210310.squirrel@208.178.77.200> Message-ID: <1120232946.13773.30.camel@ayanami.boston.redhat.com> On Fri, 2005-07-01 at 10:31 +0100, gwood at dragonhold.org wrote: > If you're just after sharing the space on disk (and not sharing the data > within the partitions) then clvm may be the answer - the only thing you > won't be able to do (from memory) without quorum is resize the partitions. You shouldn't even need CLVM if you don't intend to muddle with partitions or cross-mount the file systems. You'll lose resizing, but in doing so, your clients no longer need to be cluster participants. Partition once, and just make sure *everybody* mounts their own partition. -- Lon From gwood at dragonhold.org Fri Jul 1 15:53:45 2005 From: gwood at dragonhold.org (gwood at dragonhold.org) Date: Fri, 1 Jul 2005 16:53:45 +0100 (BST) Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <1120232946.13773.30.camel@ayanami.boston.redhat.com> References: <1119649059.13563.18.camel@ayanami.boston.redhat.com> <42BF9FCF.7060000@redhat.com> <42C24610.2@redhat.com> <42C2AD19.5000601@redhat.com> <42C39DFC.7070209@redhat.com> <42C3FD0A.4040800@possibilityforge.com> <42C4002C.80403@redhat.com> <42C402E9.2080502@possibilityforge.com> <42C4067F.6000903@redhat.com> <42C40F58.4040401@oakland.edu> <21339.208.178.77.200.1120145458.squirrel@208.178.77.200> <42C42352.3000600@possibilityforge.com> <12897.208.178.77.200.1120151133.squirrel@208.178.! 77.200> <42C429C7.3070200@possibilityforge.com> <16201.208.178.77.200.1120210310.squirrel@208.178.77.200> <1120232946.13773.30.camel@ayanami.boston.redhat.com> Message-ID: <14504.208.178.77.200.1120233225.squirrel@208.178.77.200> > You shouldn't even need CLVM if you don't intend to muddle with > partitions or cross-mount the file systems. You'll lose resizing, but > in doing so, your clients no longer need to be cluster participants. Oh certainly... I was just trying to think of why someone would want to use GFS to do this, and the idea of the machines not have pre-fixed partition sizes was about the only thing I could think of. From lhh at redhat.com Fri Jul 1 15:58:44 2005 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 01 Jul 2005 11:58:44 -0400 Subject: [Linux-cluster] ccsd performance / local sockets patch for STABLE / RHEL4 branches In-Reply-To: <1120087941.28218.40.camel@ayanami.boston.redhat.com> References: <1120087941.28218.40.camel@ayanami.boston.redhat.com> Message-ID: <1120233524.13773.32.camel@ayanami.boston.redhat.com> Committed to STABLE and HEAD. -- Lon From sasmaz at itu.edu.tr Mon Jul 4 15:38:05 2005 From: sasmaz at itu.edu.tr (Aydin SASMAZ) Date: Mon, 4 Jul 2005 18:38:05 +0300 Subject: [Linux-cluster] How to test GFS-6.0.2.20-1 on RHELAS3.0 ? Message-ID: Hi I would like to be sure which method to test GFS-6.0.2.20-1installed 3 node file cluster doesn't corrupt data while multiple nodes writing on the same file. At the same time how LOCK_GULM manage filesystem ? Is there any known test? or benchmark other than bonnie++ . I would like to deploy 3 filesystem, each one on one gfs server and export these file systems with smb to other smb clients. I plan to use 3 smb instance for all 3 filesystem on every server node and load balancer in front of them. Platform : 3x HP-DL380 G4 OP : RedHat Enterprise Linux Advanced Server 3.0 Update 4 Cluster Suite : 3.0, clumanager-1.2.22-2 redhat-config-cluster-1.0.3-1.noarch.rpm GFS : GFS-devel-6.0.2.20-1 GFS-modules-smp-6.0.2.20-1 GFS-6.0.2.20-1 Fencing Dev : fence_ilo Any advice would be appreciated. Some notes about system : ------------------------------------------ [root at gfs-test2 root]# gfs_tool df /users/lnxsrv1: SB lock proto = "lock_gulm" SB lock table = "gfs-test:lnxsrv1" SB ondisk format = 1308 SB multihost format = 1401 Block size = 4096 Journals = 8 Resource Groups = 1596 Mounted lock proto = "lock_gulm" Mounted lock table = "gfs-test:lnxsrv1" Mounted host data = "" Journal number = 1 Lock module flags = async Local flocks = FALSE Local caching = FALSE Type Total Used Free use% ------------------------------------------------------------------------ inodes 8 8 0 100% metadata 92131 60723 31408 66% data 104492113 30341120 74150993 29% /users/lnxsrv2: SB lock proto = "lock_gulm" SB lock table = "gfs-test:lnxsrv2" SB ondisk format = 1308 SB multihost format = 1401 Block size = 4096 Journals = 8 Resource Groups = 1596 Mounted lock proto = "lock_gulm" Mounted lock table = "gfs-test:lnxsrv2" Mounted host data = "" Journal number = 1 Lock module flags = async Local flocks = FALSE Local caching = FALSE Type Total Used Free use% ------------------------------------------------------------------------ inodes 5 5 0 100% metadata 38 38 0 100% data 104584209 0 104584209 0% /users/lnxsrv3: SB lock proto = "lock_gulm" SB lock table = "gfs-test:lnxsrv3" SB ondisk format = 1308 SB multihost format = 1401 Block size = 4096 Journals = 8 Resource Groups = 396 Mounted lock proto = "lock_gulm" Mounted lock table = "gfs-test:lnxsrv3" Mounted host data = "" Journal number = 1 Lock module flags = async Local flocks = FALSE Local caching = FALSE Type Total Used Free use% ------------------------------------------------------------------------ inodes 5 5 0 100% metadata 10 10 0 100% data 25949437 0 25949437 0% [root at gfs-test1 root]# gulm_tool nodeinfo gfs-test{3,1}.----- Name: gfs-test1.----- ip = 160.75.100.22 state = Logged in mode = Slave missed beats = 0 last beat = 1120235791271378 delay avg = 6672475 max delay = 9459850 [root at gfs-test1 root]# gulm_tool nodeinfo gfs-test{3,2}.------ Name: gfs-test2.------ ip = 160.75.100.23 state = Logged in mode = Slave missed beats = 0 last beat = 1120235802888768 delay avg = 6678723 max delay = 6880217 Aydin SASMAZ System Support Engineer ITU BIDB Phone: +90 212 2853930 Fax : +90 212 2856936 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Birger.Wathne at ift.uib.no Mon Jul 4 20:16:01 2005 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Mon, 04 Jul 2005 22:16:01 +0200 Subject: [Linux-cluster] NFS exports disappeared Message-ID: <42C99901.8090805@uib.no> I currently have a situation on my FC4 system where all exports for one of the file systems disappeared. Other file systems using the same nfsclient entries are still there, but I have not yet checked that all entries are there for all file systems. The one file system that had lost all exports was so obvious I spent my time trying to debug that one. I tried bumping up the version number of cluster.conf and ran ccs_tool update with no messages that look like errors, but still no exports. I tried killing the clurmtabd process for that file system and kill -HUP clurgmgrd. It restarted the process, but still no exports. I then removed two nfsclient lines, bumped up the version, ran ccs_tool, restored the missing lines, bumped up the version and ran ccs_tool again. Now those two exports are back, but there others for that file system are still missing. One client had the file system mounted and had stale NFS handles. After the above procedure the file system was ok without any remount, so that is good. How can I find all info needed to debug this? What do you want? From haydar2906 at hotmail.com Tue Jul 5 04:49:30 2005 From: haydar2906 at hotmail.com (haydar Ali) Date: Tue, 05 Jul 2005 00:49:30 -0400 Subject: [Linux-cluster] RedHat Cluster suite installation Message-ID: Hi, I'm looking for an installing and configuring procedure for RedHat Cluster suite (examples). We have 2 servers HP Proliant 380 G3 (RedHat Advanced Server 2.1) attached by fiber optic to the storage area network SAN HP MSA1000 and we want to install and configure Cluster suite to allow 2 servers to simultaneously read and write to a single shared file system by NFS (Word documents located into /u04) on the SAN. Thanks. Haydar From npf at eurotux.com Tue Jul 5 09:18:53 2005 From: npf at eurotux.com (Nuno Pais Fernandes) Date: Tue, 5 Jul 2005 10:18:53 +0100 Subject: [Linux-cluster] RedHat Cluster suite installation In-Reply-To: References: Message-ID: <200507051018.57454.npf@eurotux.com> Hi On Tuesday 05 July 2005 05:49, haydar Ali wrote: > Hi, > > I'm looking for an installing and configuring procedure for RedHat Cluster > suite (examples). > We have 2 servers HP Proliant 380 G3 (RedHat Advanced Server 2.1) attached > by fiber optic to the storage area network SAN HP MSA1000 and we want to > install and configure Cluster suite to allow 2 servers to simultaneously > read and write to a single shared file system by NFS (Word documents Read and write isn't possible with cluster suite.. you have to use GFS. Nuno Fernandes > located into /u04) on the SAN. > > Thanks. > > Haydar > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -- -------------------------- Nuno Miguel Pais Fernandes -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From yazan at ccs.com.jo Tue Jul 5 12:06:52 2005 From: yazan at ccs.com.jo (Yazan Al-Sheyyab) Date: Tue, 5 Jul 2005 14:06:52 +0200 Subject: [Linux-cluster] RedHat Cluster suite installation References: Message-ID: <001c01c5815a$07a65c30$69050364@yazanz> You have to use GFS in order to reach the shared from the two nodes at the same time. and about the documents , You can find many of them in the redhat network or on the net. Regards ------------------------------------------------- Yazan ----- Original Message ----- From: "haydar Ali" To: Sent: Tuesday, July 05, 2005 6:49 AM Subject: [Linux-cluster] RedHat Cluster suite installation > Hi, > > I'm looking for an installing and configuring procedure for RedHat > Cluster suite (examples). > We have 2 servers HP Proliant 380 G3 (RedHat Advanced Server 2.1) attached > by fiber optic to the storage area network SAN HP MSA1000 and we want to > install and configure Cluster suite to allow 2 servers to simultaneously > read and write to a single shared file system by NFS (Word documents > located into /u04) on the SAN. > > Thanks. > > Haydar > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From haydar2906 at hotmail.com Tue Jul 5 12:26:44 2005 From: haydar2906 at hotmail.com (haydar Ali) Date: Tue, 05 Jul 2005 08:26:44 -0400 Subject: [Linux-cluster] RedHat Cluster suite installation In-Reply-To: <001c01c5815a$07a65c30$69050364@yazanz> Message-ID: Hi Yazan, Yes I know that I have to use GFS to reach the shared from the two nodes at the same time but the RedHat technical support said me that I need 3 nodes (3 servers) connected to my SAN and we have only 2 nodes connected by 2 Fiber channel each. My question : can I use only 2 nodes in GFS solution?? Thanks Haydar >From: "Yazan Al-Sheyyab" >Reply-To: linux clustering >To: "linux clustering" >Subject: Re: [Linux-cluster] RedHat Cluster suite installation >Date: Tue, 5 Jul 2005 14:06:52 +0200 > > >You have to use GFS in order to reach the shared from the two nodes at the >same time. > > and about the documents , You can find many of them in the redhat network >or on the net. > > > >Regards >------------------------------------------------- > >Yazan > > >----- Original Message ----- From: "haydar Ali" >To: >Sent: Tuesday, July 05, 2005 6:49 AM >Subject: [Linux-cluster] RedHat Cluster suite installation > > >>Hi, >> >>I'm looking for an installing and configuring procedure for RedHat Cluster >>suite (examples). >>We have 2 servers HP Proliant 380 G3 (RedHat Advanced Server 2.1) attached >>by fiber optic to the storage area network SAN HP MSA1000 and we want to >>install and configure Cluster suite to allow 2 servers to simultaneously >>read and write to a single shared file system by NFS (Word documents >>located into /u04) on the SAN. >> >>Thanks. >> >>Haydar >> >> >>-- >>Linux-cluster mailing list >>Linux-cluster at redhat.com >>http://www.redhat.com/mailman/listinfo/linux-cluster > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster From REdelman at netstandard.com Tue Jul 5 13:20:42 2005 From: REdelman at netstandard.com (Rich Edelman) Date: Tue, 5 Jul 2005 08:20:42 -0500 Subject: [Linux-cluster] Fencing Problems Message-ID: <06BAD0EE281D8C4F96DBCEC41260D47BEE8AF0@nsimail03.NetStandard.inc> Hi all. I've got a 2 node FC4 system here, connected via a Brocade FC switch to FC SAN. I've followed usage.txt to a T, but seem to have some fencing problems. What happens is if one of the 2 nodes loses network connectivity, that node immediately gets fenced by the remaining node, which is all fine and good. The problem here is the fenced node (the one without network connectivity) starts trying to fence the remaining node, and when network connectivity is restored, succeeds in fencing the other node. Now both my nodes are fenced. And not only that, but I have the 'split brain' scenario, where each node thinks it is the only member of a similarly named cluster. Any ideas how to fix this? Thanks, Rich Edelman From pcaulfie at redhat.com Tue Jul 5 13:32:23 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 05 Jul 2005 14:32:23 +0100 Subject: [Linux-cluster] Fencing Problems In-Reply-To: <06BAD0EE281D8C4F96DBCEC41260D47BEE8AF0@nsimail03.NetStandard.inc> References: <06BAD0EE281D8C4F96DBCEC41260D47BEE8AF0@nsimail03.NetStandard.inc> Message-ID: <42CA8BE7.4060701@redhat.com> Rich Edelman wrote: > Hi all. > > I've got a 2 node FC4 system here, connected via a Brocade FC switch to > FC SAN. I've followed usage.txt to a T, but seem to have some fencing > problems. > > What happens is if one of the 2 nodes loses network connectivity, that > node immediately gets fenced by the remaining node, which is all fine > and good. The problem here is the fenced node (the one without network > connectivity) starts trying to fence the remaining node, and when > network connectivity is restored, succeeds in fencing the other node. > Now both my nodes are fenced. And not only that, but I have the 'split > brain' scenario, where each node thinks it is the only member of a > similarly named cluster. > > Any ideas how to fix this? > For a start you need to set the 2-node option to cman in the /etc/cluster/cluster.conf file. I think that 2-node option only works with power-switch fencing. It relies on the fact that the two nodes will race to fence each other and only the first one suceeds because the second is then dead. -- patrick From REdelman at netstandard.com Tue Jul 5 13:44:32 2005 From: REdelman at netstandard.com (Rich Edelman) Date: Tue, 5 Jul 2005 08:44:32 -0500 Subject: [Linux-cluster] Fencing Problems Message-ID: <06BAD0EE281D8C4F96DBCEC41260D47BEE8AF3@nsimail03.NetStandard.inc> Ah, yes, I did set the two-node option to cman in /etc/cluster/cluster.conf. Perhaps some documentation should be updated... But would it not be better to have some kind of timeout value on the nodes when trying to fence the others? Seems to me that in a two-node scenario, if one node is repeatedly trying to fence the other, and after say 10 tries it can't, it should just give up and assume it is the problem node. Then, hopefully, everything else would go quite well when that node came back. Rich -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Patrick Caulfield Sent: Tuesday, July 05, 2005 8:32 AM To: linux clustering Subject: Re: [Linux-cluster] Fencing Problems Rich Edelman wrote: > Hi all. > > I've got a 2 node FC4 system here, connected via a Brocade FC switch > to FC SAN. I've followed usage.txt to a T, but seem to have some > fencing problems. > > What happens is if one of the 2 nodes loses network connectivity, that > node immediately gets fenced by the remaining node, which is all fine > and good. The problem here is the fenced node (the one without network > connectivity) starts trying to fence the remaining node, and when > network connectivity is restored, succeeds in fencing the other node. > Now both my nodes are fenced. And not only that, but I have the 'split > brain' scenario, where each node thinks it is the only member of a > similarly named cluster. > > Any ideas how to fix this? > For a start you need to set the 2-node option to cman in the /etc/cluster/cluster.conf file. I think that 2-node option only works with power-switch fencing. It relies on the fact that the two nodes will race to fence each other and only the first one suceeds because the second is then dead. -- patrick -- Linux-cluster mailing list Linux-cluster at redhat.com http://www.redhat.com/mailman/listinfo/linux-cluster From pcaulfie at redhat.com Tue Jul 5 14:25:27 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 05 Jul 2005 15:25:27 +0100 Subject: [Linux-cluster] Fencing Problems In-Reply-To: <06BAD0EE281D8C4F96DBCEC41260D47BEE8AF3@nsimail03.NetStandard.inc> References: <06BAD0EE281D8C4F96DBCEC41260D47BEE8AF3@nsimail03.NetStandard.inc> Message-ID: <42CA9857.9050507@redhat.com> Rich Edelman wrote: > Ah, yes, I did set the two-node option to cman in > /etc/cluster/cluster.conf. > > Perhaps some documentation should be updated... But would it not be > better to have some kind of timeout value on the nodes when trying to > fence the others? Seems to me that in a two-node scenario, if one node > is repeatedly trying to fence the other, and after say 10 tries it > can't, it should just give up and assume it is the problem node. Then, > hopefully, everything else would go quite well when that node came back. > Data integrity requires that the remaining node knows for sure that the other node has been removed from the cluster. Without that assurance your data is toast. -- patrick From forgue at oakland.edu Tue Jul 5 14:30:45 2005 From: forgue at oakland.edu (Andrew Forgue) Date: Tue, 05 Jul 2005 10:30:45 -0400 Subject: [Linux-cluster] Fencing Problems In-Reply-To: <42CA9857.9050507@redhat.com> References: <06BAD0EE281D8C4F96DBCEC41260D47BEE8AF3@nsimail03.NetStandard.inc> <42CA9857.9050507@redhat.com> Message-ID: <42CA9995.5000503@oakland.edu> Patrick Caulfield wrote: > >Data integrity requires that the remaining node knows for sure that the other >node has been removed from the cluster. Without that assurance your data is toast. > Would this be possible if GFS implemented an on-disk quota system? -- Andrew J. Forgue Systems Programmer II :: Oakland University forgue at oakland.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 256 bytes Desc: OpenPGP digital signature URL: From pcaulfie at redhat.com Tue Jul 5 14:48:07 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 05 Jul 2005 15:48:07 +0100 Subject: [Linux-cluster] Fencing Problems In-Reply-To: <42CA9995.5000503@oakland.edu> References: <06BAD0EE281D8C4F96DBCEC41260D47BEE8AF3@nsimail03.NetStandard.inc> <42CA9857.9050507@redhat.com> <42CA9995.5000503@oakland.edu> Message-ID: <42CA9DA7.10804@redhat.com> Andrew Forgue wrote: > Patrick Caulfield wrote: > > >>Data integrity requires that the remaining node knows for sure that the other >>node has been removed from the cluster. Without that assurance your data is toast. >> > > > Would this be possible if GFS implemented an on-disk quota system? > GFS does have quotas but I don't see what they have to do with fencing. If you meant a "quorum disk" then the answer is "yes". But we don't have a quorum disk facility. -- patrick From lhh at redhat.com Tue Jul 5 14:49:47 2005 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 05 Jul 2005 10:49:47 -0400 Subject: [Linux-cluster] Fencing Problems In-Reply-To: <42CA9995.5000503@oakland.edu> Message-ID: <20050705.i7H.30528600@egw.corp.redhat.com> Andrew Forgue (forgue at oakland.edu) wrote: > Would this be possible if GFS implemented an on-disk quota system? (Quorum?) In either case (quorum *or* quota), fencing is still required, as there's no assurance that a rogue node can't flush buffers. Fencing *forcefully* prevents nodes from flushing any outstanding buffers to disk after we've declared them out of the cluster, regardless of the membership/quorum model used. -- Lon From lhh at redhat.com Tue Jul 5 15:02:50 2005 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 05 Jul 2005 11:02:50 -0400 Subject: [Linux-cluster] Fencing Problems In-Reply-To: <42CA9DA7.10804@redhat.com> Message-ID: <20050705.Lw9.35654900@egw.corp.redhat.com> Patrick Caulfield (pcaulfie at redhat.com) wrote: > > Would this be possible if GFS implemented an on-disk quota system? > > > > GFS does have quotas but I don't see what they have to do with fencing. If you > meant a "quorum disk" then the answer is "yes". But we don't have a quorum disk > facility. How does the presence or use of a quorum disk prevent queued buffers from flying out and hitting the disk? Typically, quorum disks have been used to prevent (or detect) multiple partitions, as a primary form of cluster membership, etc. A SCSI reservation as a secondary quorum determinant is more like "fencing" than a traditional quorum disk, but that's open to argument of course ;) -- Lon From haydar2906 at hotmail.com Tue Jul 5 16:22:15 2005 From: haydar2906 at hotmail.com (haydar Ali) Date: Tue, 05 Jul 2005 12:22:15 -0400 Subject: [Linux-cluster] GFS installation In-Reply-To: <20050616153316.39456.qmail@web52910.mail.yahoo.com> Message-ID: Hi Igor, Thanks for this URL. My question is: Have I to use 3 nodes to achieve GFS solution? We have 2 servers HP Proliant 380 G3 (RedHat Advanced Server 2.1) attached by 2 fiber channels each to the storage area network SAN HP MSA1000 and we want to install and configure GFS to allow 2 servers to simultaneously read and write to a single shared file system (Word documents located into /u04) located on the Storage area network SAN HP MSA1000. I read the example that you have sent to me and I see 3 nodes, 2 client nodes share a directory mounted on the 3d server node, but our solution the directory is located in the SAN. Have you any explanation or ideas for our request? Thanks Haydar >From: Igor >Reply-To: linux clustering >To: linux clustering >Subject: Re: [Linux-cluster] GFS installation >Date: Thu, 16 Jun 2005 08:33:16 -0700 (PDT) > >Look at this URL that David suggested to me: > >http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/min-gfs.txt?rev=1.3&content-type=text/x-cvsweb-markup&cvsroot=cluster > >it's pretty good. > >--- haydar Ali wrote: > > > Hi, > > > > I'm looking for an installing and configuring > > procedure for GFS (examples). > > We have 2 servers HP Proliant 380 G3 (RedHat > > Advanced Server 2.1) attached > > by fiber optic to the storage area network SAN HP > > MSA1000 and we want to > > install and configure GFS to allow 2 servers to > > simultaneously read and > > write to a single shared file system (Word documents > > located into /u04) on > > the SAN. > > > > Thanks. > > > > Haydar > > > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > >__________________________________ >Discover Yahoo! >Stay in touch with email, IM, photo sharing and more. Check it out! >http://discover.yahoo.com/stayintouch.html > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster From jscottorn at possibilityforge.com Tue Jul 5 17:04:30 2005 From: jscottorn at possibilityforge.com (Jon Scottorn) Date: Tue, 05 Jul 2005 11:04:30 -0600 Subject: [Linux-cluster] GFS freezing Message-ID: <42CABD9E.8040404@possibilityforge.com> Hi, I am wondering what is going wrong with my setup. I am running a min-gfs setup with 2 nodes accessing the storage server. The node that is the storage server is also acting as a client node. So I have 2 physical systems only. I have brought up the cluster and it works fine. Both systems can access the gfs share and what not. When I go to copy data to the share with rsync, or scp or cp, they gfs system freezes and doesn't finish putting the data on gfs. It will get about half way through and that's it. Have I missed something in the setup or what else would I be missing? Thanks, Jon From Scott.Money at lycos-inc.com Tue Jul 5 18:32:00 2005 From: Scott.Money at lycos-inc.com (Scott.Money at lycos-inc.com) Date: Tue, 5 Jul 2005 14:32:00 -0400 Subject: [Linux-cluster] GFS freezing In-Reply-To: <42CABD9E.8040404@possibilityforge.com> Message-ID: We are seeing a similar issue. We have a 3 node gfs system that uses a gnbd server as storage. We originally ran into this problem quite frequently, but hard-setting our NICs to 100Mbit full duplex has limited the system freezes to "large" data transfers. (e.g. copying 500mb files via scp or creating 500mb Oracle tablespaces). The good news is that the fencing works ;-) Let me know if you get any information about this. $cott Jon Scottorn Sent by: linux-cluster-bounces at redhat.com 07/05/2005 01:04 PM Please respond to linux clustering To linux clustering cc Subject [Linux-cluster] GFS freezing Hi, I am wondering what is going wrong with my setup. I am running a min-gfs setup with 2 nodes accessing the storage server. The node that is the storage server is also acting as a client node. So I have 2 physical systems only. I have brought up the cluster and it works fine. Both systems can access the gfs share and what not. When I go to copy data to the share with rsync, or scp or cp, they gfs system freezes and doesn't finish putting the data on gfs. It will get about half way through and that's it. Have I missed something in the setup or what else would I be missing? Thanks, Jon -- Linux-cluster mailing list Linux-cluster at redhat.com http://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From mtilstra at redhat.com Tue Jul 5 19:06:25 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Tue, 5 Jul 2005 14:06:25 -0500 Subject: [Linux-cluster] GFS freezing In-Reply-To: References: <42CABD9E.8040404@possibilityforge.com> Message-ID: <20050705190625.GA16854@redhat.com> On Tue, Jul 05, 2005 at 02:32:00PM -0400, Scott.Money at lycos-inc.com wrote: > We are seeing a similar issue. We have a 3 node gfs system that > uses a gnbd server as storage. We originally ran into this problem > quite frequently, but hard-setting our NICs to 100Mbit full duplex > has limited the system freezes to "large" data transfers. (e.g. > copying 500mb files via scp or creating 500mb Oracle tablespaces). > The good news is that the fencing works ;-) > Let me know if you get any information about this. What you describe here sounds more like flooding of the network. If you send too much data over the same network device as the heartbeat&locking traffic, you can starve out the heatbeats. There was a bunch of emails about this already on this list. The way to deal with it is one of 1: don't ever flood the network, 2: use a provate network for heartbeats & lock traffic, 3: use the traffic shaping kernel modules to provide a garunteed bandwidth for the heartbeat & locking traffic. -- Michael Conrad Tadpol Tilstra What is your one purpose in life? To explode of course! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jscottorn at possibilityforge.com Tue Jul 5 19:40:25 2005 From: jscottorn at possibilityforge.com (Jon Scottorn) Date: Tue, 05 Jul 2005 13:40:25 -0600 Subject: [Linux-cluster] GFS freezing In-Reply-To: <20050705190625.GA16854@redhat.com> References: <42CABD9E.8040404@possibilityforge.com> <20050705190625.GA16854@redhat.com> Message-ID: <42CAE229.6000207@possibilityforge.com> We already have #2 setup that way. Also we are not flooding the network. I am copying data from with the storage server. Also. Here is what my gfs_tool df displays. I am wondering if the inodes and such being so low will cause an issue. I would think that is shouldn't be what it is. SB lock proto = "lock_dlm" SB lock table = "SAN1:VserversFS" SB ondisk format = 1309 SB multihost format = 1401 Block size = 4096 Journals = 4 Resource Groups = 2794 Mounted lock proto = "lock_dlm" Mounted lock table = "SAN1:VserversFS" Mounted host data = "" Journal number = 0 Lock module flags = Local flocks = FALSE Local caching = FALSE Oopses OK = FALSE Type Total Used Free use% ------------------------------------------------------------------------ inodes 5 5 0 100% metadata 66 66 0 100% data 182996209 0 182996209 0% Michael Conrad Tadpol Tilstra wrote: >On Tue, Jul 05, 2005 at 02:32:00PM -0400, Scott.Money at lycos-inc.com wrote: > > >> We are seeing a similar issue. We have a 3 node gfs system that >> uses a gnbd server as storage. We originally ran into this problem >> quite frequently, but hard-setting our NICs to 100Mbit full duplex >> has limited the system freezes to "large" data transfers. (e.g. >> copying 500mb files via scp or creating 500mb Oracle tablespaces). >> The good news is that the fencing works ;-) >> Let me know if you get any information about this. >> >> > >What you describe here sounds more like flooding of the network. If you >send too much data over the same network device as the heartbeat&locking >traffic, you can starve out the heatbeats. There was a bunch of emails >about this already on this list. The way to deal with it is one of >1: don't ever flood the network, 2: use a provate network for heartbeats >& lock traffic, 3: use the traffic shaping kernel modules to provide a >garunteed bandwidth for the heartbeat & locking traffic. > > > >------------------------------------------------------------------------ > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster > From mtilstra at redhat.com Tue Jul 5 20:22:28 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Tue, 5 Jul 2005 15:22:28 -0500 Subject: [Linux-cluster] GFS freezing In-Reply-To: <42CAE229.6000207@possibilityforge.com> References: <42CABD9E.8040404@possibilityforge.com> <20050705190625.GA16854@redhat.com> <42CAE229.6000207@possibilityforge.com> Message-ID: <20050705202228.GA17623@redhat.com> On Tue, Jul 05, 2005 at 01:40:25PM -0600, Jon Scottorn wrote: > We already have #2 setup that way. Also we are not flooding the > network. I am copying data from with the storage server. Also. Here > is what my gfs_tool df displays. I am wondering if the inodes and such > being so low will cause an issue. I would think that is shouldn't be > what it is. yeah, I was responding specifically to Scott.Money at lycos-inc.com, sorry for not being clear on that. -- Michael Conrad Tadpol Tilstra Too many errors on one line (make fewer) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jscottorn at possibilityforge.com Tue Jul 5 22:35:36 2005 From: jscottorn at possibilityforge.com (Jon Scottorn) Date: Tue, 05 Jul 2005 16:35:36 -0600 Subject: [Linux-cluster] Network problems Message-ID: <42CB0B38.7060000@possibilityforge.com> Hi, How does a node rejoin the cluster if network connectivity is lost? I have 3 nodes that are in the cluster. I have unplugged the network connection from one of the nodes, for testing purposes, the cluster sees the node disappear, when I plug the network connection back in it does not rejoin the cluster. I looks as though it forms it's own cluster. ie... cat /proc/cluster/nodes shows two of the nodes with the third missing. cat /proc/cluster/nodes on the system that lost network connectivity shows itself in the clusterand the other two nodes missing and the quorum is locked as well. Can this issue be prevented when network connectivity is lost or when that happens do I just have to reboot the system? Jon From fajar at telkom.net.id Wed Jul 6 02:25:59 2005 From: fajar at telkom.net.id (Fajar A. Nugraha) Date: Wed, 06 Jul 2005 09:25:59 +0700 Subject: [Linux-cluster] RedHat Cluster suite installation In-Reply-To: References: Message-ID: <42CB4137.30204@telkom.net.id> haydar Ali wrote: > Hi Yazan, > > Yes I know that I have to use GFS to reach the shared from the two > nodes at the same time but the RedHat technical support said me that I > need 3 nodes (3 servers) connected to my SAN and we have only 2 nodes > connected by 2 Fiber channel each. > My question : can I use only 2 nodes in GFS solution?? > You can with RHEL4 and http://sources.redhat.com/cluster/ . I believe the official (supported) version was released recently but I can't find the PR stuff yet. Regards, Fajar From jason at selu.edu Wed Jul 6 03:52:09 2005 From: jason at selu.edu (Jason Lanclos) Date: Tue, 5 Jul 2005 22:52:09 -0500 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <14504.208.178.77.200.1120233225.squirrel@208.178.77.200> References: <1120232946.13773.30.camel@ayanami.boston.redhat.com> <14504.208.178.77.200.1120233225.squirrel@208.178.77.200> Message-ID: <200507052252.09456.jason@selu.edu> > > You shouldn't even need CLVM if you don't intend to muddle with > > partitions or cross-mount the file systems. You'll lose resizing, but > > in doing so, your clients no longer need to be cluster participants. > CLVM is cool, but its pretty much useless until LVM2 actually implements pvresize or a pvextend. One of the main advantages of having a SAN is being able to add space to a volume (LUN) Currently when we expand a volume on the san, we have to unmount the filesystem, rescan the LUNs, then run fdisk on that volume, delete the partition entry, and recreate it to use all the space.. then at that point we can run ext2online / gfs_grow to resize the filesystem. I would be VERY nice if pvresize / pvextend existed, that way one could expand the volume on the SAN, rescan LUNs on each cluster member, run pvresize / pvextend, run lvextend and then gfs_grow and call it a day. I mentioned this at the RedHat summit, and got a few puzzled looks, but never got an answer on the matter. > Oh certainly... > > I was just trying to think of why someone would want to use GFS to do > this, and the idea of the machines not have pre-fixed partition sizes was > about the only thing I could think of. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From fajar at telkom.net.id Wed Jul 6 04:04:08 2005 From: fajar at telkom.net.id (Fajar A. Nugraha) Date: Wed, 06 Jul 2005 11:04:08 +0700 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <200507052252.09456.jason@selu.edu> References: <1120232946.13773.30.camel@ayanami.boston.redhat.com> <14504.208.178.77.200.1120233225.squirrel@208.178.77.200> <200507052252.09456.jason@selu.edu> Message-ID: <42CB5838.4060802@telkom.net.id> Jason Lanclos wrote: >>>You shouldn't even need CLVM if you don't intend to muddle with >>>partitions or cross-mount the file systems. You'll lose resizing, but >>>in doing so, your clients no longer need to be cluster participants. >>> >>> > >CLVM is cool, but its pretty much useless until LVM2 actually implements >pvresize or a pvextend. > What's wrong with vgextend? >One of the main advantages of having a SAN is being >able to add space to a volume (LUN) Currently when we expand a volume on >the san, we have to unmount the filesystem, rescan the LUNs, then run fdisk >on that volume, delete the partition entry, and recreate it to use all the >space.. then at that point we can run ext2online / gfs_grow to resize the >filesystem. > > > A simpler method will be create a NEW LUN, scan it, pvcreate, add it to existing VG with vgextend, and extend your volume lvextend. At least you don't have to mess with existing partitions. >I would be VERY nice if pvresize / pvextend existed, that way one could expand >the volume on the SAN, > My IBM SAN can't expand exisiting volume on the SAN. It's not like I need it though. The above solution works better. >rescan LUNs on each cluster member, run pvresize / >pvextend, run lvextend and then gfs_grow and call it a day. > > > My only problem (for now) is I can't rescan LUNs without removing HBA modules (thus unmounting filesystems or restarting the server). If you can rescan LUNs online, it's a simple method of pvcreate, vgextend, lvextend, gfs_grow, and call it a day. >I mentioned this at the RedHat summit, and got a few puzzled looks, > I wonder why :-D Regards, Fajar From JACOB_LIBERMAN at Dell.com Wed Jul 6 04:04:56 2005 From: JACOB_LIBERMAN at Dell.com (JACOB_LIBERMAN at Dell.com) Date: Tue, 5 Jul 2005 23:04:56 -0500 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc Message-ID: Jason, LVM2 has a vgextend command that can be used to add a new PV to a vg. There is also an lvextend and ext2online so you can increase the size of the lv without dismounting it. This is slightly different than a pvextend, but I believe it achieves the same end result. So a process you might be able to try is: 1. extend the LUN on the SAN 2. use fdisk to create a new partition on the LUN 3. use partprobe -s to dynamically redetect the new partition 4. make a new filesystem on the partition 5. use vgextend to add the new PV to the VG 6. use lvextend to add new space to the LV 7. use ext2online of gfs_grow to tarck out the filesystem on the new space *without* dismounting or rebooting I've used to above procedure under RHEL4 with LVM2 and an ext3 filesystem. I have not tried it w/GFS yet because our reference lab is down for maintenance. Thanks, jacob > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Jason Lanclos > Sent: Tuesday, July 05, 2005 10:52 PM > To: linux clustering > Subject: Re: [Linux-cluster] A few GFS newbie questions: journals, etc > > > > > You shouldn't even need CLVM if you don't intend to muddle with > > > partitions or cross-mount the file systems. You'll lose > resizing, > > > but in doing so, your clients no longer need to be > cluster participants. > > > > CLVM is cool, but its pretty much useless until LVM2 actually > implements pvresize or a pvextend. One of the main advantages > of having a SAN is being > able to add space to a volume (LUN) Currently when we > expand a volume on > the san, we have to unmount the filesystem, rescan the LUNs, > then run fdisk on that volume, delete the partition entry, > and recreate it to use all the space.. then at that point we > can run ext2online / gfs_grow to resize the filesystem. > > I would be VERY nice if pvresize / pvextend existed, that way > one could expand the volume on the SAN, rescan LUNs on each > cluster member, run pvresize / pvextend, run lvextend and > then gfs_grow and call it a day. > > I mentioned this at the RedHat summit, and got a few puzzled > looks, but never got an answer on the matter. > > > > > Oh certainly... > > > > I was just trying to think of why someone would want to use > GFS to do > > this, and the idea of the machines not have pre-fixed > partition sizes was > > about the only thing I could think of. > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From fabbione at fabbione.net Wed Jul 6 04:49:15 2005 From: fabbione at fabbione.net (Fabio Massimo Di Nitto) Date: Wed, 06 Jul 2005 06:49:15 +0200 Subject: [Linux-cluster] [PATCH] add syscall missing includes in rgmanager/src/clulib/gettid.c Message-ID: <42CB62CB.408@fabbione.net> Hi guys, patch is pretty self-explanatory, when we did change the way in which we use syscall, we forgot to add the relevant includes. Patch is against the STABLE branch, but i am pretty sure it applies all over. As a consequence it also shuts up a warning a build time. Please apply. Cheers Fabio -- no signature file found. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: add_missing_syscall_includes.dpatch URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 256 bytes Desc: OpenPGP digital signature URL: From fabbione at fabbione.net Wed Jul 6 05:00:19 2005 From: fabbione at fabbione.net (Fabio Massimo Di Nitto) Date: Wed, 06 Jul 2005 07:00:19 +0200 Subject: [Linux-cluster] [PATCH] add syscall missing includes in rgmanager/src/clulib/gettid.c In-Reply-To: <42CB62CB.408@fabbione.net> References: <42CB62CB.408@fabbione.net> Message-ID: <42CB6563.5050007@fabbione.net> Fabio Massimo Di Nitto wrote: > Hi guys, > patch is pretty self-explanatory, when we did change the way in which we use > syscall, we forgot to add the relevant includes. > Patch is against the STABLE branch, but i am pretty sure it applies all over. > As a consequence it also shuts up a warning a build time. > > Please apply. > > Cheers > Fabio whops.. wrong patch.. sorry.. here is the good one.. Fabio -- no signature file found. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: add_missing_syscall_includes.dpatch URL: From fabbione at fabbione.net Wed Jul 6 05:25:08 2005 From: fabbione at fabbione.net (Fabio Massimo Di Nitto) Date: Wed, 06 Jul 2005 07:25:08 +0200 Subject: [Linux-cluster] Suggestion for the STABLE branch Message-ID: <42CB6B34.6090800@fabbione.net> Hi guys, given the nature of the branch, wouldn't make sense to setup the release info to something more natural than: RELEASE_MAJOR = DEVEL RELEASE_MINOR = DATE ? Perhaps start traking versions, libs soname and stuff like will give different vendors the possibility to allign their package names and be less confusing for the end users. Thanks. Fabio -- no signature file found. From yazan at ccs.com.jo Wed Jul 6 06:37:57 2005 From: yazan at ccs.com.jo (Yazan Al-Sheyyab) Date: Wed, 6 Jul 2005 08:37:57 +0200 Subject: [Linux-cluster] GFS installation References: Message-ID: <010401c581f5$3ecc1560$69050364@yazanz> hi haydar, my friend i was having the same case : which is a two ml 370 hp proliant nodes and a shared storage msa500 and the two nodes connected to the shared by SCSI cable , the problem was with the lock server when implementing the GFS. im using RHEL_ES_V3_U4 and a GFS for V3 U4 also, but here in this GFS release you have to have an odd number of lock server , i mean that when you have two server so you have to have a three lock server , but in my case i have used only the first node as lock server and then i reached a poor redunduncy in the cluster cause i still studying the purchase of a third server., i heared that there is a release of GFS which is GFS 6.1 working with RHEL_V4 which will work without the need of the locking service ..... some body correct me if that is not right. but some body told me that i can use the third lock server as logical not physical server , he meant that i dont need another HardWare like a server but my question now is there any body on our list can explain me the use of the third server as logical way , is it as unix and hp-ux as an area of the disk used for locking or what? can we manage one of the two cpu on the server as virtual to get a solution without another server. Sorry for the long Email , but we have this last problem. Regards ------------------------------------------------- Yazan --------------------------- ----- Original Message ----- From: "haydar Ali" To: Sent: Tuesday, July 05, 2005 6:22 PM Subject: Re: [Linux-cluster] GFS installation > Hi Igor, > > Thanks for this URL. > My question is: Have I to use 3 nodes to achieve GFS solution? > We have 2 servers HP Proliant 380 G3 (RedHat Advanced Server 2.1) attached > by 2 fiber channels each to the storage area network SAN HP MSA1000 and we > want to install and configure GFS to allow 2 servers to simultaneously > read and write to a single shared file system (Word documents located into > /u04) located on the Storage area network SAN HP MSA1000. > I read the example that you have sent to me and I see 3 nodes, 2 client > nodes share a directory mounted on the 3d server node, but our solution > the directory is located in the SAN. > > Have you any explanation or ideas for our request? > Thanks > > Haydar > > >>From: Igor >>Reply-To: linux clustering >>To: linux clustering >>Subject: Re: [Linux-cluster] GFS installation >>Date: Thu, 16 Jun 2005 08:33:16 -0700 (PDT) >> >>Look at this URL that David suggested to me: >> >>http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/min-gfs.txt?rev=1.3&content-type=text/x-cvsweb-markup&cvsroot=cluster >> >>it's pretty good. >> >>--- haydar Ali wrote: >> >> > Hi, >> > >> > I'm looking for an installing and configuring >> > procedure for GFS (examples). >> > We have 2 servers HP Proliant 380 G3 (RedHat >> > Advanced Server 2.1) attached >> > by fiber optic to the storage area network SAN HP >> > MSA1000 and we want to >> > install and configure GFS to allow 2 servers to >> > simultaneously read and >> > write to a single shared file system (Word documents >> > located into /u04) on >> > the SAN. >> > >> > Thanks. >> > >> > Haydar >> > >> > >> > -- >> > Linux-cluster mailing list >> > Linux-cluster at redhat.com >> > http://www.redhat.com/mailman/listinfo/linux-cluster >> > >> >> >> >> >>__________________________________ >>Discover Yahoo! >>Stay in touch with email, IM, photo sharing and more. Check it out! >>http://discover.yahoo.com/stayintouch.html >> >>-- >>Linux-cluster mailing list >>Linux-cluster at redhat.com >>http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From JACOB_LIBERMAN at Dell.com Wed Jul 6 06:06:00 2005 From: JACOB_LIBERMAN at Dell.com (JACOB_LIBERMAN at Dell.com) Date: Wed, 6 Jul 2005 01:06:00 -0500 Subject: [Linux-cluster] GFS installation Message-ID: Yazan, You can implement RLM (Redundant Lock Manager) on RHEL3 with 3 servers. All of the lock servers do NOT need access to the shared storage, although they can have access to the shared storage. The file locking in all handled over the heartbeat LAN. I believe that at least 1 of the lockservers needs access to the shared storage, but I have not tried any less than X -1 if X is the total number of lock servers. The only important rule of thumb to keep in mind is that when the lockserver has access to the shared storage, it must be fenced from BOTH the disk and the network. If the lockserver only has access to the network, it should be fenced from the network only. In both cases, a network power switch is the preferred fencing method. If a non-lockserver GFS server (ie -- node) is accessing the shared storage, it can be fenced from the storage via a fiber switch. The method for implementing a non-disk attached RLM lockserver is to make a copy of the current CCS information on a disk-attached lockserver, copy it to the non-disk attached lockserver, and then make a CCS file archive on the non-disk attached lockserver with ccs_tool. Incidentally, it is far easier to set this up if you make one of the disk-attached lockservers the master lockserver. Once you have copied the cluster config to a directory on the non-disk attached lockserver, edit the /etc/sysconfig/ccsd CCS_ARCHIVE parameter so that it points to the local directory that contains the config files. Then, wafter you have copied the files to the non-disk attached lockserver, all you need to do to generate the files is run cc_tool -v create /etc/gfs/data0 /etc/gfs/data0.cca. (of course, your pathnames will vary.) Next, start ccsd on the non-disk lockserver with ccsd -f /path/to/file.cca and then start lock_gulmd. You can verify the new lockserver in /var/log/messages on the master lock server. Then test fencing and you are done! I hope this helps. Cheers, jacob > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Yazan > Al-Sheyyab > Sent: Wednesday, July 06, 2005 1:38 AM > To: linux clustering > Subject: Re: [Linux-cluster] GFS installation > > hi haydar, > > my friend i was having the same case : which is a two ml > 370 hp proliant nodes and a shared storage msa500 and the two > nodes connected to the shared by SCSI cable , the problem was > with the lock server when implementing the GFS. > > im using RHEL_ES_V3_U4 and a GFS for V3 U4 also, but here in > this GFS release you have to have an odd number of lock > server , i mean that when you have two server so you have to > have a three lock server , but in my case i have used only > the first node as lock server and then i reached a poor > redunduncy in the cluster cause i still studying the purchase > of a third server., i heared that there is a release of GFS > which is GFS 6.1 working with RHEL_V4 which will work without > the need of the locking service ..... > some body correct me if that is not right. > > but some body told me that i can use the third lock server as > logical not physical server , he meant that i dont need > another HardWare like a server > > but my question now is there any body on our list can explain > me the use of the third server as logical way , is it as unix > and hp-ux as an area of the disk used for locking or what? > > can we manage one of the two cpu on the server as virtual to > get a solution without another server. > > Sorry for the long Email , but we have this last problem. > > > > Regards > ------------------------------------------------- > > Yazan > --------------------------- > > ----- Original Message ----- > From: "haydar Ali" > To: > Sent: Tuesday, July 05, 2005 6:22 PM > Subject: Re: [Linux-cluster] GFS installation > > > > Hi Igor, > > > > Thanks for this URL. > > My question is: Have I to use 3 nodes to achieve GFS solution? > > We have 2 servers HP Proliant 380 G3 (RedHat Advanced > Server 2.1) attached > > by 2 fiber channels each to the storage area network SAN HP > MSA1000 and we > > want to install and configure GFS to allow 2 servers to > simultaneously > > read and write to a single shared file system (Word > documents located into > > /u04) located on the Storage area network SAN HP MSA1000. > > I read the example that you have sent to me and I see 3 > nodes, 2 client > > nodes share a directory mounted on the 3d server node, but > our solution > > the directory is located in the SAN. > > > > Have you any explanation or ideas for our request? > > Thanks > > > > Haydar > > > > > >>From: Igor > >>Reply-To: linux clustering > >>To: linux clustering > >>Subject: Re: [Linux-cluster] GFS installation > >>Date: Thu, 16 Jun 2005 08:33:16 -0700 (PDT) > >> > >>Look at this URL that David suggested to me: > >> > >>http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/min -gfs.txt?rev=1.3&content-type=text/x-cvsweb-markup&cvsroot=cluster >> >>it's pretty good. >> >>--- haydar Ali wrote: >> >> > Hi, >> > >> > I'm looking for an installing and configuring >> > procedure for GFS (examples). >> > We have 2 servers HP Proliant 380 G3 (RedHat >> > Advanced Server 2.1) attached >> > by fiber optic to the storage area network SAN HP >> > MSA1000 and we want to >> > install and configure GFS to allow 2 servers to >> > simultaneously read and >> > write to a single shared file system (Word documents >> > located into /u04) on >> > the SAN. >> > >> > Thanks. >> > >> > Haydar >> > >> > >> > -- >> > Linux-cluster mailing list >> > Linux-cluster at redhat.com >> > http://www.redhat.com/mailman/listinfo/linux-cluster >> > >> >> >> >> >>__________________________________ >>Discover Yahoo! >>Stay in touch with email, IM, photo sharing and more. Check it out! >>http://discover.yahoo.com/stayintouch.html >> >>-- >>Linux-cluster mailing list >>Linux-cluster at redhat.com >>http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com http://www.redhat.com/mailman/listinfo/linux-cluster From schlegel at riege.com Wed Jul 6 06:30:51 2005 From: schlegel at riege.com (Gunther Schlegel) Date: Wed, 06 Jul 2005 08:30:51 +0200 Subject: [Linux-cluster] RHEL3 Cluster network hangup Message-ID: <42CB7A9B.30304@riege.com> Hi, I am running RHEL3 ES with the RedHat Cluster Suite (not GFS, simply failover cluster). The clustered application does a lot of printing (lprng), faxing(hylafax) and mailing(sendmail). It uses shell scripts to pass the jobs to the operating systems daemons. The client programs of these daemons, which pass jobs to the daemons using network connections to localhost start to behave irregular when the cluster is up for about 2 weeks. Examples: - hylafax faxstat stops listing the transmitted faxes in the middle of the list ( but always at the same job ) - sendmail opens a connection to the local daemon but does not transfer the message. Both processes sit there and wait, after some time the server closes the connection because of missing input from the clients side. - same with lpr. I assume that something locks up in the ip stack. Not all services are affected at the same time. I guess this is related to the cluster software as we run that application on a lot of servers which all do not show this behaviour and that are all not clustered. Any hints? regards, Gunther -------------- next part -------------- A non-text attachment was scrubbed... Name: schlegel.vcf Type: text/x-vcard Size: 345 bytes Desc: not available URL: From pcaulfie at redhat.com Wed Jul 6 07:08:39 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 06 Jul 2005 08:08:39 +0100 Subject: [Linux-cluster] Network problems In-Reply-To: <42CB0B38.7060000@possibilityforge.com> References: <42CB0B38.7060000@possibilityforge.com> Message-ID: <42CB8377.2090504@redhat.com> Jon Scottorn wrote: > Hi, > > How does a node rejoin the cluster if network connectivity is lost? > > I have 3 nodes that are in the cluster. I have unplugged the network > connection from one of the nodes, for testing purposes, the cluster sees > the node disappear, when I plug the network connection back in it does > not rejoin the cluster. I looks as though it forms it's own cluster. ie... > > cat /proc/cluster/nodes shows two of the nodes with the third missing. > cat /proc/cluster/nodes on the system that lost network connectivity > shows itself in the clusterand the other two nodes missing and the > quorum is locked as well. > > Can this issue be prevented when network connectivity is lost or when > that happens do I just have to reboot the system? You'll need to reboot the system - or at least restart all the cluster services which probably amnounts to the same thing. When a cluster is partitioned, the partitions (obviously) don't know what has been going on in the other partition, so it can't just rejoin the cluster because the whole lockspace/filesystem state may have completely changed. When network connectivity is re-established the quorate partition (if there is one) should fence the inquorate one. If it is an even partition where neither has quorum then they will just stare sullenly at each other for ever, unfortunately. -- patrick From Birger.Wathne at ift.uib.no Wed Jul 6 10:00:12 2005 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Wed, 06 Jul 2005 12:00:12 +0200 Subject: [Linux-cluster] NFS exports disappeared In-Reply-To: <42C99901.8090805@uib.no> References: <42C99901.8090805@uib.no> Message-ID: <42CBABAC.4000909@uib.no> I'm still not certain how I lost the nfs export list in the first place, but I think I see why they never got 'fixed'. I added some logging inside /usr/share/cluster/nfsclient.sh, and it seems like status is only checked for a few of my exports. Is this a bug? Shouldn't all nfs exports get checked regularly? -- birger From Birger.Wathne at ift.uib.no Wed Jul 6 10:12:22 2005 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Wed, 06 Jul 2005 12:12:22 +0200 Subject: [Linux-cluster] Problem rebooting cluster node - CMANsendmsg failed: -101 Message-ID: <42CBAE86.3090101@uib.no> When I try to reboot or shut down my cluster node (a 2 node cluster with only one node present. Still waiting for delivery of the second node) it stops answering to ping, and then seems to loop forever with the message CMANsendmsg failed: -101. Perhaps it times out in the end, but I have ended up power cycling it to get up again. -- birger From Birger.Wathne at ift.uib.no Wed Jul 6 10:16:34 2005 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Wed, 06 Jul 2005 12:16:34 +0200 Subject: [Linux-cluster] NFS configuration question Message-ID: <42CBAF82.8050900@uib.no> When setting up a configuration for NFS server failover, can I ensure the IP address comes up after exports are ready, and more importantly that the IP address goes down before unexporting by placeing the entry at the end of the service declaration? When taking down the nfs service I want the server to 'disappear' before it starts unexporting, as i want clients to hang waiting for the service to come back up instead of getting io errors. I am unable to test this, since I had to go into production a bit too quickly. -- birger From pcaulfie at redhat.com Wed Jul 6 10:36:08 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 06 Jul 2005 11:36:08 +0100 Subject: [Linux-cluster] Problem rebooting cluster node - CMANsendmsg failed: -101 In-Reply-To: <42CBAE86.3090101@uib.no> References: <42CBAE86.3090101@uib.no> Message-ID: <42CBB418.7020303@redhat.com> Birger Wathne wrote: > When I try to reboot or shut down my cluster node (a 2 node cluster with > only one node present. Still waiting for delivery of the second node) it > stops answering to ping, and then seems to loop forever with the message > CMANsendmsg failed: -101. > I have seen this a small number of times - though only on test machines that don't invoke the cman shutdown script. -- patrick From Birger.Wathne at ift.uib.no Wed Jul 6 11:29:53 2005 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Wed, 06 Jul 2005 13:29:53 +0200 Subject: [Linux-cluster] Problem rebooting cluster node - CMANsendmsg failed: -101 In-Reply-To: <42CBB418.7020303@redhat.com> References: <42CBAE86.3090101@uib.no> <42CBB418.7020303@redhat.com> Message-ID: <42CBC0B1.7040309@uib.no> Patrick Caulfield wrote: > I have seen this a small number of times - though only on test machines that > don't invoke the cman shutdown script. > I have installed from source using the FC4 branch. It seems like a make install installs startup and shutdown links for clvmd_init and ccsd. for cman, rgmanager and fenced it only installs links for startup. I'll fix my installation. Thanks for the tip. Is this a bug in the Makefile? -- birger From rkenna at redhat.com Wed Jul 6 13:43:54 2005 From: rkenna at redhat.com (Rob Kenna) Date: Wed, 06 Jul 2005 09:43:54 -0400 Subject: [Linux-cluster] RedHat Cluster suite installation In-Reply-To: <42CB4137.30204@telkom.net.id> References: <42CB4137.30204@telkom.net.id> Message-ID: <1120657434.4311.7.camel@rkenna-laptop> GFS 6.1 for RHEL 4 shipped two weeks ago: http://www.redhat.com/software/rha/gfs/ (General info page) http://www.redhat.com/docs/manuals/csgfs/ (Documentation Pages) GFS 6.1 can indeed run in 2 node configurations. It includes the DLM (Distributed Lock Manager) which "spreads" the locking across all GFS nodes w/o separate lock servers. Though DLM is now the default, GULM remains as an alternative lock manager. - Rob On Wed, 2005-07-06 at 09:25 +0700, Fajar A. Nugraha wrote: > haydar Ali wrote: > > > Hi Yazan, > > > > Yes I know that I have to use GFS to reach the shared from the two > > nodes at the same time but the RedHat technical support said me that I > > need 3 nodes (3 servers) connected to my SAN and we have only 2 nodes > > connected by 2 Fiber channel each. > > My question : can I use only 2 nodes in GFS solution?? > > > You can with RHEL4 and http://sources.redhat.com/cluster/ . I believe > the official (supported) version was released recently but I can't find > the PR stuff yet. > > Regards, > > Fajar > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -- Robert Kenna / Red Hat Sr Product Mgr - Storage o: (978) 392-2410 (x22410) c: (978) 771-6314 rkenna at redhat.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason at selu.edu Wed Jul 6 14:17:32 2005 From: jason at selu.edu (Jason Lanclos) Date: Wed, 6 Jul 2005 09:17:32 -0500 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <42CB5838.4060802@telkom.net.id> References: <200507052252.09456.jason@selu.edu> <42CB5838.4060802@telkom.net.id> Message-ID: <200507060917.32412.Jason@selu.edu> On Tuesday 05 July 2005 11:04 pm, Fajar A. Nugraha wrote: > Jason Lanclos wrote: > > >>>You shouldn't even need CLVM if you don't intend to muddle with > >>>partitions or cross-mount the file systems. You'll lose resizing, but > >>>in doing so, your clients no longer need to be cluster participants. > >>> > >>> > > > >CLVM is cool, but its pretty much useless until LVM2 actually implements > >pvresize or a pvextend. > > > What's wrong with vgextend? Nothings wrong with vgextend.. but its useless when i want to extend a pv. > > >One of the main advantages of having a SAN is being > >able to add space to a volume (LUN) Currently when we expand a volume on > >the san, we have to unmount the filesystem, rescan the LUNs, then run fdisk > >on that volume, delete the partition entry, and recreate it to use all the > >space.. then at that point we can run ext2online / gfs_grow to resize the > >filesystem. > > > > > > > A simpler method will be create a NEW LUN, scan it, pvcreate, add it to > existing VG with vgextend, and extend your volume lvextend. At least you > don't have to mess with existing partitions. Ok.. what happens when you reach the max LUNs or max partitions??? And I can only imagine the administration nightmare it would be to manage with several volumes spread over multiple LUNs. The whole point of a SAN is to simplify management of data storage. And what about copy/swap? Spreading a volume over multiple LUNs would make using this feature of the SAN impossible / very dangerous to do while volumes are mounted. So.. One Volume, One LUN, One partition is the easiest way. There are prolly a thousand different ways to "make it work", but I wouldn't consider that part of the whole "Red Hat Enterprise Linux" concept. I know I'm not the 1st person to run into this issue.. so rather than having numerous work arounds, why not just implement pvresize/pvextend to do this? https://www.redhat.com/archives/linux-lvm/2003-July/msg00038.html Oh.. and there's already a man page for it.. even though its not implemented in LVM2: http://mandoc.etopian.net/man/linux/8/pvresize And a work around that I'd be reluctant using on a production system: http://www.redhat.com/archives/linux-lvm/2004-December/msg00049.html > > >I would be VERY nice if pvresize / pvextend existed, that way one could expand > >the volume on the SAN, > > > My IBM SAN can't expand exisiting volume on the SAN. It's not like I > need it though. The above solution works better. XioTech Magnitude expands volumes with no problem.. Its one of the reasons we chose it for our SAN. > > >rescan LUNs on each cluster member, run pvresize / > >pvextend, run lvextend and then gfs_grow and call it a day. > > > > > > > My only problem (for now) is I can't rescan LUNs without removing HBA > modules (thus unmounting filesystems or restarting the server). I'm not sure if it will work with your FC card, but I've used the following script a few times with a qlogic card and it worked. http://www.fifi.org/cgi-bin/man2html/usr/share/man/man8/rescan-scsi-bus.sh.8.gz > If you > can rescan LUNs online, it's a simple method of pvcreate, vgextend, > lvextend, gfs_grow, and call it a day. > > >I mentioned this at the RedHat summit, and got a few puzzled looks, > > > I wonder why :-D > > Regards, > > Fajar > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > -- Jason Lanclos Systems Administrator Red Hat Certified Engineer Southeastern Louisiana University From amanthei at redhat.com Wed Jul 6 16:04:33 2005 From: amanthei at redhat.com (Adam Manthei) Date: Wed, 6 Jul 2005 11:04:33 -0500 Subject: [Linux-cluster] Problem rebooting cluster node - CMANsendmsg failed: -101 In-Reply-To: <42CBC0B1.7040309@uib.no> References: <42CBAE86.3090101@uib.no> <42CBB418.7020303@redhat.com> <42CBC0B1.7040309@uib.no> Message-ID: <20050706160433.GC15320@redhat.com> On Wed, Jul 06, 2005 at 01:29:53PM +0200, Birger Wathne wrote: > Patrick Caulfield wrote: > > >I have seen this a small number of times - though only on test machines > >that > >don't invoke the cman shutdown script. > > > > I have installed from source using the FC4 branch. It seems like a make > install installs startup and shutdown links for clvmd_init and ccsd. for > cman, rgmanager and fenced it only installs links for startup. > > I'll fix my installation. Thanks for the tip. Is this a bug in the Makefile? This is not a bug. The Makefiles are responsible for installing the init scripts and binaries onto your system. They are not responsible for configuring your system to start and stop services; that is the responsibility of the rpm. -- Adam Manthei From jason at selu.edu Wed Jul 6 18:10:42 2005 From: jason at selu.edu (Jason Lanclos) Date: Wed, 6 Jul 2005 13:10:42 -0500 Subject: [Linux-cluster] Problem rebooting cluster node - CMANsendmsg failed: -101 In-Reply-To: <20050706160433.GC15320@redhat.com> References: <42CBAE86.3090101@uib.no> <42CBC0B1.7040309@uib.no> <20050706160433.GC15320@redhat.com> Message-ID: <200507061310.42642.Jason@selu.edu> On Wednesday 06 July 2005 11:04 am, Adam Manthei wrote: > On Wed, Jul 06, 2005 at 01:29:53PM +0200, Birger Wathne wrote: > > Patrick Caulfield wrote: > > > > >I have seen this a small number of times - though only on test machines > > >that > > >don't invoke the cman shutdown script. > > > > > > > I have installed from source using the FC4 branch. It seems like a make > > install installs startup and shutdown links for clvmd_init and ccsd. for > > cman, rgmanager and fenced it only installs links for startup. > > > I'll fix my installation. Thanks for the tip. Is this a bug in the Makefile? > > This is not a bug. The Makefiles are responsible for installing the init > scripts and binaries onto your system. They are not responsible for > configuring your system to start and stop services; that is the > responsibility of the rpm. > I've run into this also.. For some reason any scripts that don't have the word "daemon" in them, don't run when shutting the system down. And sometimes the install didn't setup the correct links in rc6.d. To resolve this I just used the following script to put the word daemon in each script, and remove, and recreate the rc links. for i in ccsd cman fenced gfs rgmanager do echo "# daemon" >> /etc/init.d/$i chkconfig --del $i chkconfig --add $i done -- Jason Lanclos Systems Administrator Red Hat Certified Engineer Southeastern Louisiana University From alewis at redhat.com Wed Jul 6 19:01:28 2005 From: alewis at redhat.com (AJ Lewis) Date: Wed, 6 Jul 2005 14:01:28 -0500 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <200507060917.32412.Jason@selu.edu> References: <200507052252.09456.jason@selu.edu> <42CB5838.4060802@telkom.net.id> <200507060917.32412.Jason@selu.edu> Message-ID: <20050706190128.GB15005@null.msp.redhat.com> On Wed, Jul 06, 2005 at 09:17:32AM -0500, Jason Lanclos wrote: > > A simpler method will be create a NEW LUN, scan it, pvcreate, add it to > > existing VG with vgextend, and extend your volume lvextend. At least you > > don't have to mess with existing partitions. > > Ok.. what happens when you reach the max LUNs or max partitions??? > And I can only imagine the administration nightmare it would be to manage with several > volumes spread over multiple LUNs. The whole point of a SAN is to simplify management of data storage. > And what about copy/swap? Spreading a volume over multiple LUNs would make using this feature of the SAN impossible / very > dangerous to do while volumes are mounted. > So.. One Volume, One LUN, One partition is the easiest way. > There are prolly a thousand different ways to "make it work", > but I wouldn't consider that part of the whole "Red Hat Enterprise Linux" concept. Just out of curiousity - if you're using your array to do volume management anyway - why use lvm at all? -- AJ Lewis Voice: 612-638-0500 Red Hat E-Mail: alewis at redhat.com One Main Street SE, Suite 209 Minneapolis, MN 55414 Current GPG fingerprint = D9F8 EDCE 4242 855F A03D 9B63 F50C 54A8 578C 8715 Grab the key at: http://people.redhat.com/alewis/gpg.html or one of the many keyservers out there... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jason at selu.edu Wed Jul 6 20:08:19 2005 From: jason at selu.edu (Jason Lanclos) Date: Wed, 6 Jul 2005 15:08:19 -0500 Subject: [Linux-cluster] A few GFS newbie questions: journals, etc In-Reply-To: <20050706190128.GB15005@null.msp.redhat.com> References: <200507060917.32412.Jason@selu.edu> <20050706190128.GB15005@null.msp.redhat.com> Message-ID: <200507061508.20046.Jason@selu.edu> On Wednesday 06 July 2005 02:01 pm, AJ Lewis wrote: > Just out of curiousity - if you're using your array to do volume management > anyway - why use lvm at all? Currently, the linux cluster connected to the SAN is RedHat AS 2.1, ext3 is the filesystem. Each volume is on its own LUN for failover reasons. However once we upgrade to RHEL4 and start using GFS, we would like to consolidate some of the smaller filesystems to one LUN and use CLVM, but at the same time avoid having a volume group spread over multiple LUNS. right now, the 2 servers we have are identical.. but as we add new servers to the cluster, the device names might be different. using CLVM would allow usage of /dev// mappings instead of /dev/sd?? The only reason we wouldn't use CLVM is because it doesn't support extending PV size. -- Jason Lanclos Systems Administrator Red Hat Certified Engineer Southeastern Louisiana University From natecars at natecarlson.com Wed Jul 6 20:08:38 2005 From: natecars at natecarlson.com (Nate Carlson) Date: Wed, 6 Jul 2005 15:08:38 -0500 (CDT) Subject: [Linux-cluster] cluster.conf changes without restarting cluster? Message-ID: Hey all, I'm adding a new node to my play cluster, and made the config changes with ccs_tool (added a few new fence devices, removed a node, added another, etc.) On the node I made the changes on, I get this in syslog: Jul 6 15:03:48 xen1 ccsd[2052]: Update of cluster.conf complete (version 15 -> 16). Jul 6 15:04:12 xen1 ccsd[2052]: Update of cluster.conf complete (version 16 -> 17). Jul 6 15:04:21 xen1 ccsd[2052]: Update of cluster.conf complete (version 17 -> 18). Jul 6 15:04:26 xen1 ccsd[2052]: Update of cluster.conf complete (version 18 -> 19). Jul 6 15:05:01 xen1 ccsd[2052]: Update of cluster.conf complete (version 19 -> 20). Jul 6 15:05:22 xen1 ccsd[2052]: Update of cluster.conf complete (version 20 -> 21). The rest of the nodes also have the updated config file. However, /proc/cluster/status on all boxes still says: Config version: 15 Is there any way to get ccs to load the new config file without restarting everything? I'm running STABLE as of a few days ago. ------------------------------------------------------------------------ | nate carlson | natecars at natecarlson.com | http://www.natecarlson.com | | depriving some poor village of its idiot since 1981 | ------------------------------------------------------------------------ From phung at cs.columbia.edu Wed Jul 6 20:11:06 2005 From: phung at cs.columbia.edu (Dan B. Phung) Date: Wed, 6 Jul 2005 16:11:06 -0400 (EDT) Subject: [Linux-cluster] cluster.conf changes without restarting cluster? In-Reply-To: Message-ID: first run ccs_tool update /etc/clutser/cluster.conf then cman_tool version -r On 6, Jul, 2005, Nate Carlson declared: > Hey all, > > I'm adding a new node to my play cluster, and made the config changes with > ccs_tool (added a few new fence devices, removed a node, added another, > etc.) > > On the node I made the changes on, I get this in syslog: > > Jul 6 15:03:48 xen1 ccsd[2052]: Update of cluster.conf complete (version 15 -> 16). > Jul 6 15:04:12 xen1 ccsd[2052]: Update of cluster.conf complete (version 16 -> 17). > Jul 6 15:04:21 xen1 ccsd[2052]: Update of cluster.conf complete (version 17 -> 18). > Jul 6 15:04:26 xen1 ccsd[2052]: Update of cluster.conf complete (version 18 -> 19). > Jul 6 15:05:01 xen1 ccsd[2052]: Update of cluster.conf complete (version 19 -> 20). > Jul 6 15:05:22 xen1 ccsd[2052]: Update of cluster.conf complete (version 20 -> 21). > > The rest of the nodes also have the updated config file. However, > /proc/cluster/status on all boxes still says: > > Config version: 15 > > Is there any way to get ccs to load the new config file without restarting > everything? > > I'm running STABLE as of a few days ago. > > ------------------------------------------------------------------------ > | nate carlson | natecars at natecarlson.com | http://www.natecarlson.com | > | depriving some poor village of its idiot since 1981 | > ------------------------------------------------------------------------ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > -- email: phung at cs.columbia.edu www: http://www.cs.columbia.edu/~phung phone: 646-775-6090 office: CS Dept. 520, 1214 Amsterdam Ave., MC 0401, New York, NY 10027 From natecars at natecarlson.com Wed Jul 6 20:15:02 2005 From: natecars at natecarlson.com (Nate Carlson) Date: Wed, 6 Jul 2005 15:15:02 -0500 (CDT) Subject: [Linux-cluster] cluster.conf changes without restarting cluster? In-Reply-To: References: Message-ID: On Wed, 6 Jul 2005, Dan B. Phung wrote: > first run > ccs_tool update /etc/clutser/cluster.conf > then > cman_tool version -r Ah, very good - thanks! ------------------------------------------------------------------------ | nate carlson | natecars at natecarlson.com | http://www.natecarlson.com | | depriving some poor village of its idiot since 1981 | ------------------------------------------------------------------------ From eric at bootseg.com Wed Jul 6 21:29:12 2005 From: eric at bootseg.com (Eric Kerin) Date: Wed, 06 Jul 2005 17:29:12 -0400 Subject: [Linux-cluster] NFS configuration question In-Reply-To: <42CBAF82.8050900@uib.no> References: <42CBAF82.8050900@uib.no> Message-ID: <1120685352.5704.9.camel@auh5-0478> On Wed, 2005-07-06 at 12:16 +0200, Birger Wathne wrote: > When taking down the nfs service I want the server to 'disappear' before it > starts unexporting, as i want clients to hang waiting for the service to > come back up instead of getting io errors. It should already work this way. Look in /usr/share/cluster/service.sh, there is a block of XML data that defines the service resource agent. Twords the end of the block of XML is a "special" tag this defines the child node types for that agent. You'll notice each of the child nodes has a start and stop number. These define the order that the given children are started and stopped You'll see filesystems are started at level 2, and ip addresses are started at 3. Since a nfs export is defined as a child of a fs agent, the nfs exports are turned on after mounting the filesystem, and before the IP address is active. Then they are stopped using the order defined by the stop attribute. So IP address is turned off, then filesystems are stopped, (nfs exports are turned off, and then the filesystem is unmounted.) Hope this helps. -- Eric Kerin From Birger.Wathne at ift.uib.no Wed Jul 6 23:44:07 2005 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Thu, 07 Jul 2005 01:44:07 +0200 Subject: [Linux-cluster] Problem rebooting cluster node - CMANsendmsg failed: -101 In-Reply-To: <200507061310.42642.Jason@selu.edu> References: <42CBAE86.3090101@uib.no> <42CBC0B1.7040309@uib.no> <20050706160433.GC15320@redhat.com> <200507061310.42642.Jason@selu.edu> Message-ID: <42CC6CC7.9040904@uib.no> Jason Lanclos wrote: > I've run into this also.. For some reason any scripts that don't have the word "daemon" in them, > don't run when shutting the system down. And sometimes the install didn't setup the correct links in rc6.d. > To resolve this I just used the following script to put the word daemon in each script, and remove, and recreate the rc links. > > for i in ccsd cman fenced gfs rgmanager > do > echo "# daemon" >> /etc/init.d/$i > chkconfig --del $i > chkconfig --add $i > done > I know I didn't install the startup links, as I would have used chkconfig to do it. I used chkconfig to add the links, and it did the job as expected. I first installed the rpm's when fc4 was released, but they had too many old bugs. I uninstalled the rpm's and installed from cvs (FC4 branch). Since then the rpm's have been updated, so I will probably convert to those to ease management when I have the other cluster node available. Anyway. Problem solved, and just to be safe I have also added the daemon lines in the scripts. -- birger From Birger.Wathne at ift.uib.no Wed Jul 6 23:53:07 2005 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Thu, 07 Jul 2005 01:53:07 +0200 Subject: [Linux-cluster] NFS configuration question In-Reply-To: <1120685352.5704.9.camel@auh5-0478> References: <42CBAF82.8050900@uib.no> <1120685352.5704.9.camel@auh5-0478> Message-ID: <42CC6EE3.6020005@uib.no> Eric Kerin wrote: > On Wed, 2005-07-06 at 12:16 +0200, Birger Wathne wrote: > >>When taking down the nfs service I want the server to 'disappear' before it >>starts unexporting, as i want clients to hang waiting for the service to >>come back up instead of getting io errors. > > > It should already work this way. Look in /usr/share/cluster/service.sh, > there is a block of XML data that defines the service resource agent. . . . Thanks. That is reassuring. I'll have to read all of those files so I really know what goes on. :-) -- birger From liangs at cse.ohio-state.edu Thu Jul 7 01:53:25 2005 From: liangs at cse.ohio-state.edu (Shuang Liang) Date: Wed, 06 Jul 2005 21:53:25 -0400 Subject: [Linux-cluster] installing GFS In-Reply-To: References: Message-ID: <42CC8B15.7040705@cse.ohio-state.edu> Hi, I am trying to install GFS on 3 cluster nodes with redhat linux-2.6.9-5EL using gnbd. I met with some problems the mount, which says "# mount -t gfs /dev/gnbd0 /mnt/gfs/ mount: Connection refused" I have tried the installation with different order and configuration all end up with this error, hope sb can help solve this puzzle. Here are some step I have taken: 1.Insert gfs-lock and gfs kernel modules at the GFS node. Insert gnbd module at GFS node 2. run gnbd_serv -n at GNBD server; ./gnbd_export -d /dev/hda8 -e mygnbd -c 3. ./gnbd_import -i k35-as4 -n at GFS node 4. Now I want to create logic volume at GFS node: it failed saying [root at k32-as4 sbin]# pvcreate /dev/gnbd0 Device /dev/gnbd0 not found. So I choose to do the logic volume stuff at the GNBD server side and gfs_mkfs at the GFS side, it succeeded. But it fails again when mounting. Then I tried to mkfs at GNBD server and mount directly through gnbd device, same error again. So I am wondering what should be the correct configuration trick. Thanks for reading this far! From haydar2906 at hotmail.com Thu Jul 7 02:44:47 2005 From: haydar2906 at hotmail.com (haydar Ali) Date: Wed, 06 Jul 2005 22:44:47 -0400 Subject: [Linux-cluster] GFS installation In-Reply-To: <010401c581f5$3ecc1560$69050364@yazanz> Message-ID: Hi Yazan, But if the 3d node (lock server) will be down. will GFS offer the shared file system?? Thanks Haydar >From: "Yazan Al-Sheyyab" >Reply-To: linux clustering >To: "linux clustering" >Subject: Re: [Linux-cluster] GFS installation >Date: Wed, 6 Jul 2005 08:37:57 +0200 > >hi haydar, > > my friend i was having the same case : which is a two ml 370 hp proliant >nodes and a shared storage msa500 and the two nodes connected to the shared >by SCSI cable , the problem was with the lock server when implementing the >GFS. > >im using RHEL_ES_V3_U4 and a GFS for V3 U4 also, but here in this GFS >release you have to have an odd number of lock server , i mean that when >you have two server so you have to have a three lock server , but in my >case i have used only the first node as lock server and then i reached a >poor redunduncy in the cluster cause i still studying the purchase of a >third server., i heared that there is a release of GFS which is GFS 6.1 >working with RHEL_V4 which will work without the need of the locking >service ..... some body correct me if that is not right. > >but some body told me that i can use the third lock server as logical not >physical server , he meant that i dont need another HardWare like a server > >but my question now is there any body on our list can explain me the use of >the third server as logical way , is it as unix and hp-ux as an area of >the disk used for locking or what? > >can we manage one of the two cpu on the server as virtual to get a solution >without another server. > >Sorry for the long Email , but we have this last problem. > > > >Regards >------------------------------------------------- > >Yazan >--------------------------- > >----- Original Message ----- From: "haydar Ali" >To: >Sent: Tuesday, July 05, 2005 6:22 PM >Subject: Re: [Linux-cluster] GFS installation > > >>Hi Igor, >> >>Thanks for this URL. >>My question is: Have I to use 3 nodes to achieve GFS solution? >>We have 2 servers HP Proliant 380 G3 (RedHat Advanced Server 2.1) attached >>by 2 fiber channels each to the storage area network SAN HP MSA1000 and we >>want to install and configure GFS to allow 2 servers to simultaneously >>read and write to a single shared file system (Word documents located into >>/u04) located on the Storage area network SAN HP MSA1000. >>I read the example that you have sent to me and I see 3 nodes, 2 client >>nodes share a directory mounted on the 3d server node, but our solution >>the directory is located in the SAN. >> >>Have you any explanation or ideas for our request? >>Thanks >> >>Haydar >> >> >>>From: Igor >>>Reply-To: linux clustering >>>To: linux clustering >>>Subject: Re: [Linux-cluster] GFS installation >>>Date: Thu, 16 Jun 2005 08:33:16 -0700 (PDT) >>> >>>Look at this URL that David suggested to me: >>> >>>http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/min-gfs.txt?rev=1.3&content-type=text/x-cvsweb-markup&cvsroot=cluster >>> >>>it's pretty good. >>> >>>--- haydar Ali wrote: >>> >>> > Hi, >>> > >>> > I'm looking for an installing and configuring >>> > procedure for GFS (examples). >>> > We have 2 servers HP Proliant 380 G3 (RedHat >>> > Advanced Server 2.1) attached >>> > by fiber optic to the storage area network SAN HP >>> > MSA1000 and we want to >>> > install and configure GFS to allow 2 servers to >>> > simultaneously read and >>> > write to a single shared file system (Word documents >>> > located into /u04) on >>> > the SAN. >>> > >>> > Thanks. >>> > >>> > Haydar >>> > >>> > >>> > -- >>> > Linux-cluster mailing list >>> > Linux-cluster at redhat.com >>> > http://www.redhat.com/mailman/listinfo/linux-cluster >>> > >>> >>> >>> >>> >>>__________________________________ >>>Discover Yahoo! >>>Stay in touch with email, IM, photo sharing and more. Check it out! >>>http://discover.yahoo.com/stayintouch.html >>> >>>-- >>>Linux-cluster mailing list >>>Linux-cluster at redhat.com >>>http://www.redhat.com/mailman/listinfo/linux-cluster >> >> >>-- >>Linux-cluster mailing list >>Linux-cluster at redhat.com >>http://www.redhat.com/mailman/listinfo/linux-cluster > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster From Hansjoerg.Maurer at dlr.de Thu Jul 7 06:26:39 2005 From: Hansjoerg.Maurer at dlr.de (=?ISO-8859-15?Q?Hansj=F6rg_Maurer?=) Date: Thu, 07 Jul 2005 08:26:39 +0200 Subject: [Linux-cluster] GFS for RHEL4U1 (2.6.9-11)? Message-ID: <42CCCB1F.7080602@dlr.de> Hi, are there any GFS SRPM's available for RHEL4U1 (2.6.9-11)? The cluster webpage http://sources.redhat.com/cluster/ says, that there is a RHEL4 CVS, which builds against 2.6.9-5. Is GFS-6.1 already officially released? Are there any plans for making stable SRPM's available, like there are for RHEL3? Thank you very much Hansj?rg -- _________________________________________________________________ Dr. Hansjoerg Maurer | LAN- & System-Manager | Deutsches Zentrum | DLR Oberpfaffenhofen f. Luft- und Raumfahrt e.V. | Institut f. Robotik | Postfach 1116 | Muenchner Strasse 20 82230 Wessling | 82234 Wessling Germany | | Tel: 08153/28-2431 | E-mail: Hansjoerg.Maurer at dlr.de Fax: 08153/28-1134 | WWW: http://www.robotic.dlr.de/ __________________________________________________________________ There are 10 types of people in this world, those who understand binary and those who don't. From birger at uib.no Thu Jul 7 10:43:12 2005 From: birger at uib.no (Birger Wathne) Date: Thu, 07 Jul 2005 12:43:12 +0200 Subject: [Linux-cluster] NFS configuration question In-Reply-To: <1120685352.5704.9.camel@auh5-0478> References: <42CBAF82.8050900@uib.no> <1120685352.5704.9.camel@auh5-0478> Message-ID: <42CD0740.8040102@uib.no> Eric Kerin wrote: > >It should already work this way. Look in /usr/share/cluster/service.sh, >there is a block of XML data that defines the service resource agent. >Twords the end of the block of XML is a "special" tag this defines the >child node types for that agent. You'll notice each of the child nodes >has a start and stop number. These define the order that the given >children are started and stopped You'll see filesystems are started at >level 2, and ip addresses are started at 3. Since a nfs export is >defined as a child of a fs agent, the nfs exports are turned on after >mounting the filesystem, and before the IP address is active. > It kind of works this way, but still it doesn't... I just did a clusvcadm -R and I also tried with -s followed by -e. What happens is exacly what you describe. Exports come up all in one go, then the IP address. But then, a split second later all exports except the one I have in /etc/exports are gone. It's as if something has done 'exportfs -r'. I'll have to look into this. Could be my own config problem, as I restart the lockd when bringing up the service. However the exports are all there when I reboot and let the services come up automatically, and if my script is the culprit it should behave the same way then, shouldn't it? After this, the export lines get added again when the cluster tests for them and finds them missing. The problem there is that not all the export lines get tested. Of my 9 export entries in cluster.conf only 5 get tested and reexported after disappearing. As I said they are all there if I reboot and let the service come up automatically. -- birger From mtilstra at redhat.com Thu Jul 7 13:40:00 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Thu, 7 Jul 2005 08:40:00 -0500 Subject: [Linux-cluster] installing GFS In-Reply-To: <42CC8B15.7040705@cse.ohio-state.edu> References: <42CC8B15.7040705@cse.ohio-state.edu> Message-ID: <20050707134000.GA27833@redhat.com> On Wed, Jul 06, 2005 at 09:53:25PM -0400, Shuang Liang wrote: > Hi, > I am trying to install GFS on 3 cluster nodes with redhat > linux-2.6.9-5EL using gnbd. I met with some problems the mount, which says > "# mount -t gfs /dev/gnbd0 /mnt/gfs/ > mount: Connection refused" > > I have tried the installation with different order and configuration all > end up with this error, hope sb can help solve this puzzle. Here are > some step I have taken: > 1.Insert gfs-lock and gfs kernel modules at the GFS node. > Insert gnbd module at GFS node > 2. run gnbd_serv -n at GNBD server; > ./gnbd_export -d /dev/hda8 -e mygnbd -c > 3. ./gnbd_import -i k35-as4 -n at GFS node > 4. Now I want to create logic volume at GFS node: it failed saying > [root at k32-as4 sbin]# pvcreate /dev/gnbd0 > Device /dev/gnbd0 not found. > > So I choose to do the logic volume stuff at the GNBD server side and > gfs_mkfs at the GFS side, it succeeded. But it fails again when mounting. > Then I tried to mkfs at GNBD server and mount directly through gnbd > device, same error again. > > So I am wondering what should be the correct configuration trick. is cman/dlm loaded and running? -- Michael Conrad Tadpol Tilstra I can see clearly now, the brain is gone... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jscottorn at possibilityforge.com Thu Jul 7 14:33:14 2005 From: jscottorn at possibilityforge.com (Jon Scottorn) Date: Thu, 07 Jul 2005 08:33:14 -0600 Subject: [Linux-cluster] HELP! Diapered block device Message-ID: <42CD3D2A.2050108@possibilityforge.com> Hi, I have a major issue. I was running fine with a 4 node cluster for a few days and then I copied some data to the GFS share and within a few hours the gfs share went down. I restarted the cluster and now I am getting this error from dmesg and it also says removing device node /dev/diaperedmd0: GFS: Trying to join cluster "fsck_dlm", "SAN1:VserversFS" lock_harness: can't find protocol fsck_dlm GFS: can't mount proto = fsck_dlm, table = SAN1:VserversFS, hostdata = I have removed the storage server from the cluster and shutdown gnbd and I have been running gfs_fsck for about 24 hours and it still hasn't completed. I have a 750GB raid 5 setup and I can't remount or anything. What do I need to do to fix this? Here is what gfs_fsck reports: Initializing fsck Initializing lists... Initializing special inodes... Setting block ranges... Creating a block list of size 183146926... Clearing journals (this may take a while) Clearing journal 0 Clearing journal 1 Clearing journal 2 Clearing journal 3 Cleared journals Starting pass1 Checking metadata in Resource Group 0 Checking metadata in Resource Group 1 Checking metadata in Resource Group 2 ...... Skipped to conserve space ............. Checking metadata in Resource Group 2791 Checking metadata in Resource Group 2792 Checking metadata in Resource Group 2793 Pass1 complete Starting pass1b Looking for duplicate blocks... Found dup block at 61573000 Found dup block at 61573014 Found dup block at 61573015 Found dup block at 61573016 Found dup block at 61573017 Found dup block at 61573018 Found dup block at 61573019 Found dup block at 61573020 Found dup block at 61573021 Found dup block at 61573022 Found dup block at 61573024 Found dup block at 61573047 Found dup block at 61573048 Found dup block at 61573052 Found dup block at 61623032 Found dup block at 61623033 Found dup block at 61623034 Found dup block at 61623035 Scanning filesystem for inodes containing duplicate blocks... Any help is greatly appreciated. Thanks, Jon From jason at selu.edu Thu Jul 7 15:02:01 2005 From: jason at selu.edu (Jason Lanclos) Date: Thu, 7 Jul 2005 10:02:01 -0500 Subject: [Linux-cluster] HELP! Diapered block device In-Reply-To: <42CD3D2A.2050108@possibilityforge.com> References: <42CD3D2A.2050108@possibilityforge.com> Message-ID: <200507071002.01130.Jason@selu.edu> On Thursday 07 July 2005 09:33 am, Jon Scottorn wrote: > GFS: Trying to join cluster "fsck_dlm", "SAN1:VserversFS" > lock_harness: ?can't find protocol fsck_dlm > GFS: can't mount proto = fsck_dlm, table = SAN1:VserversFS, hostdata = > the lock protocol should be lock_dlm not fsck_dlm you can change this with gfs_tool gfs_tool sb proto lock_dlm -- Jason Lanclos Systems Administrator Red Hat Certified Engineer Southeastern Louisiana University From jscottorn at possibilityforge.com Thu Jul 7 15:16:45 2005 From: jscottorn at possibilityforge.com (Jon Scottorn) Date: Thu, 07 Jul 2005 09:16:45 -0600 Subject: [Linux-cluster] HELP! Diapered block device In-Reply-To: <200507071002.01130.Jason@selu.edu> References: <42CD3D2A.2050108@possibilityforge.com> <200507071002.01130.Jason@selu.edu> Message-ID: <42CD475D.3060609@possibilityforge.com> Thanks, That made it so I can mount it from the other nodes, but now I can't mount it on the storage server. This is what dmesg reports: GFS: fsid=SAN1:VserversFS.0: Joined cluster. Now mounting FS... GFS: fsid=SAN1:VserversFS.0: jid=0: Trying to acquire journal lock... GFS: fsid=SAN1:VserversFS.0: jid=0: Looking at journal... GFS: fsid=SAN1:VserversFS.0: jid=0: Done GFS: fsid=SAN1:VserversFS.0: jid=1: Trying to acquire journal lock... GFS: fsid=SAN1:VserversFS.0: jid=1: Looking at journal... GFS: fsid=SAN1:VserversFS.0: jid=1: Done GFS: fsid=SAN1:VserversFS.0: jid=2: Trying to acquire journal lock... GFS: fsid=SAN1:VserversFS.0: jid=2: Looking at journal... GFS: fsid=SAN1:VserversFS.0: jid=2: Done GFS: fsid=SAN1:VserversFS.0: jid=3: Trying to acquire journal lock... GFS: fsid=SAN1:VserversFS.0: jid=3: Looking at journal... GFS: fsid=SAN1:VserversFS.0: jid=3: Done GFS: fsid=SAN1:VserversFS.0: fatal: filesystem consistency error GFS: fsid=SAN1:VserversFS.0: inode = 52732294/52732294 GFS: fsid=SAN1:VserversFS.0: function = gfs_change_nlink GFS: fsid=SAN1:VserversFS.0: file = /usr/src/cluster/cluster.STABLE.20050701/gfs-kernel/src/gfs/inode.c, line = 843 GFS: fsid=SAN1:VserversFS.0: time = 1120748944 GFS: fsid=SAN1:VserversFS.0: about to withdraw from the cluster GFS: fsid=SAN1:VserversFS.0: waiting for outstanding I/O GFS: fsid=SAN1:VserversFS.0: telling LM to withdraw lock_dlm: withdraw abandoned memory GFS: fsid=SAN1:VserversFS.0: withdrawn mh_magic = 0x01161970 mh_type = 4 mh_generation = 13 mh_format = 400 mh_incarn = 4 no_formal_ino = 52732294 no_addr = 52732294 di_mode = 0600 di_uid = 31 di_gid = 32 di_nlink = 0 di_size = 0 di_blocks = 1 di_atime = 1120690913 di_mtime = 1120690938 di_ctime = 1120690938 di_major = 0 no_formal_ino = 0 no_addr = 0 di_eattr = 0 di_reserved = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 I'm not sure what else I am missing. Also, the mount I am doing from the clients are done with mounting the gnbd device ie.. /dev/gnbd/fs On the storage server itself, I am mounting the actual device ie.. /dev/md0 Don't know if that causes an issue doing it that way or not. Any other thoughts as to how to get my errors fixed. Thanks, Jon Jason Lanclos wrote: >On Thursday 07 July 2005 09:33 am, Jon Scottorn wrote: > > >>GFS: Trying to join cluster "fsck_dlm", "SAN1:VserversFS" >>lock_harness: can't find protocol fsck_dlm >>GFS: can't mount proto = fsck_dlm, table = SAN1:VserversFS, hostdata = >> >> >> > > the lock protocol should be lock_dlm not fsck_dlm > > you can change this with gfs_tool > gfs_tool sb proto lock_dlm > > > From eric at bootseg.com Thu Jul 7 15:22:03 2005 From: eric at bootseg.com (Eric Kerin) Date: Thu, 07 Jul 2005 11:22:03 -0400 Subject: [Linux-cluster] NFS configuration question In-Reply-To: <42CD0740.8040102@uib.no> References: <42CBAF82.8050900@uib.no> <1120685352.5704.9.camel@auh5-0478> <42CD0740.8040102@uib.no> Message-ID: <1120749723.5534.7.camel@auh5-0478> On Thu, 2005-07-07 at 12:43 +0200, Birger Wathne wrote: > Eric Kerin wrote: > > > > >It should already work this way. Look in /usr/share/cluster/service.sh, > >there is a block of XML data that defines the service resource agent. > >Twords the end of the block of XML is a "special" tag this defines the > >child node types for that agent. You'll notice each of the child nodes > >has a start and stop number. These define the order that the given > >children are started and stopped You'll see filesystems are started at > >level 2, and ip addresses are started at 3. Since a nfs export is > >defined as a child of a fs agent, the nfs exports are turned on after > >mounting the filesystem, and before the IP address is active. > > > It kind of works this way, but still it doesn't... > What happens is exacly what you describe. Exports come up all in one go, > then the IP address. > But then, a split second later all exports except the one I have in > /etc/exports are gone. It's as if something has done 'exportfs -r'. I'll > have to look into this. Could be my own config problem, as I restart the > lockd when bringing up the service. However the exports are all there > when I reboot and let the services come up automatically, and if my > script is the culprit it should behave the same way then, shouldn't it? > Just because I'm curious, why do you restart lockd? Are you restarting any other nfs related services from rgmanager? > Of my 9 export entries in cluster.conf only 5 > get tested and reexported after disappearing. As I said they are all > there if I reboot and let the service come up automatically. > It'd be interesting to see the relevant section of your cluster.conf file. Also you don't have any of the filesystems you are exporting in the cluster setup in /etc/exports, do you? -- Eric Kerin From alewis at redhat.com Thu Jul 7 15:34:35 2005 From: alewis at redhat.com (AJ Lewis) Date: Thu, 7 Jul 2005 10:34:35 -0500 Subject: [Linux-cluster] HELP! Diapered block device In-Reply-To: <42CD475D.3060609@possibilityforge.com> References: <42CD3D2A.2050108@possibilityforge.com> <200507071002.01130.Jason@selu.edu> <42CD475D.3060609@possibilityforge.com> Message-ID: <20050707153435.GD15005@null.msp.redhat.com> On Thu, Jul 07, 2005 at 09:16:45AM -0600, Jon Scottorn wrote: > Thanks, > > That made it so I can mount it from the other nodes, but now I can't > mount it on the storage server. Gah! Is the fsck still running? You *CANNOT* run the fsck while other nodes have the fs mounted. The fsck changes the lock protocol to prevent others from mounting after the fsck starts. It will be changed back after completion. The fsck can take a while in the duplicate block code - could you tell if it was still accessing storage? If you have lots of inodes in the system, it's gonna take a while to work through them in the dup block handling code. Regards, -- AJ Lewis Voice: 612-638-0500 Red Hat E-Mail: alewis at redhat.com One Main Street SE, Suite 209 Minneapolis, MN 55414 Current GPG fingerprint = D9F8 EDCE 4242 855F A03D 9B63 F50C 54A8 578C 8715 Grab the key at: http://people.redhat.com/alewis/gpg.html or one of the many keyservers out there... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jscottorn at possibilityforge.com Thu Jul 7 15:40:49 2005 From: jscottorn at possibilityforge.com (Jon Scottorn) Date: Thu, 07 Jul 2005 09:40:49 -0600 Subject: [Linux-cluster] HELP! Diapered block device In-Reply-To: <20050707153435.GD15005@null.msp.redhat.com> References: <42CD3D2A.2050108@possibilityforge.com> <200507071002.01130.Jason@selu.edu> <42CD475D.3060609@possibilityforge.com> <20050707153435.GD15005@null.msp.redhat.com> Message-ID: <42CD4D01.20803@possibilityforge.com> When I ran the fsck, i had everything unmounted as well as the gnbd serv stopped. I let it run for almost 24 hours and it was still running. That seems a little long for me. Should I let it run again and see what happens. My main problem is I can't have the FS down for that long. Thanks, Jon AJ Lewis wrote: >On Thu, Jul 07, 2005 at 09:16:45AM -0600, Jon Scottorn wrote: > > >>Thanks, >> >> That made it so I can mount it from the other nodes, but now I can't >>mount it on the storage server. >> >> > >Gah! Is the fsck still running? You *CANNOT* run the fsck while other nodes >have the fs mounted. The fsck changes the lock protocol to prevent others >from mounting after the fsck starts. It will be changed back after >completion. > >The fsck can take a while in the duplicate block code - could you tell if it >was still accessing storage? If you have lots of inodes in the system, it's >gonna take a while to work through them in the dup block handling code. > >Regards, > > >------------------------------------------------------------------------ > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster > From lhh at redhat.com Thu Jul 7 15:49:17 2005 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 07 Jul 2005 11:49:17 -0400 Subject: [Linux-cluster] RHEL3 Cluster network hangup In-Reply-To: <42CB7A9B.30304@riege.com> References: <42CB7A9B.30304@riege.com> Message-ID: <1120751357.15658.40.camel@ayanami.boston.redhat.com> On Wed, 2005-07-06 at 08:30 +0200, Gunther Schlegel wrote: > The clustered application does a lot of printing (lprng), > faxing(hylafax) and mailing(sendmail). It uses shell scripts to pass the > jobs to the operating systems daemons. > The client programs of these daemons, which pass jobs to the daemons > using network connections to localhost start to behave irregular when > the cluster is up for about 2 weeks. > Examples: > - hylafax faxstat stops listing the transmitted faxes in the middle of > the list ( but always at the same job ) > - sendmail opens a connection to the local daemon but does not transfer > the message. Both processes sit there and wait, after some time the > server closes the connection because of missing input from the clients side. > - same with lpr. > > I assume that something locks up in the ip stack. Not all services are > affected at the same time. > > I guess this is related to the cluster software as we run that > application on a lot of servers which all do not show this behaviour and > that are all not clustered. I doubt it, but it's not out of the realm of possibility. The cluster software does three things mostly: (a) figures out who's online (b) shoots nodes (c) manages services using shell scripts The shell scripts call standard utilities (ifconfig, route, etc.). Now -- here's the thing. Earlier versions of clumanager (<1.2.22) had a problem where sometimes (and randomly!), services would get a bogus status return and restart on the same node. Also, the most recent errata fixed a signal handling problem which broke JVMs from running under it. Either of these may have caused the problems on your cluster, I don't know. The former would have associated log messages; the latter wouldn't. I'd try the latest release from RHN (clumanager-1.2.26.1-1). If that doesn't work, I'd call Red Hat Support... -- Lon From lhh at redhat.com Thu Jul 7 15:58:20 2005 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 07 Jul 2005 11:58:20 -0400 Subject: [Linux-cluster] NFS exports disappeared In-Reply-To: <42CBABAC.4000909@uib.no> References: <42C99901.8090805@uib.no> <42CBABAC.4000909@uib.no> Message-ID: <1120751900.15658.48.camel@ayanami.boston.redhat.com> On Wed, 2005-07-06 at 12:00 +0200, Birger Wathne wrote: > I'm still not certain how I lost the nfs export list in the first place, but > I think I see why they never got 'fixed'. > > I added some logging inside /usr/share/cluster/nfsclient.sh, and it seems > like status is only checked for a few of my exports. > > Is this a bug? Shouldn't all nfs exports get checked regularly? > Yes, they should be. Can you include the relevant parts of your cluster.conf when you file the bugzilla? -- Lon From lhh at redhat.com Thu Jul 7 16:01:07 2005 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 07 Jul 2005 12:01:07 -0400 Subject: [Linux-cluster] NFS configuration question In-Reply-To: <42CBAF82.8050900@uib.no> References: <42CBAF82.8050900@uib.no> Message-ID: <1120752067.15658.54.camel@ayanami.boston.redhat.com> On Wed, 2005-07-06 at 12:16 +0200, Birger Wathne wrote: > When setting up a configuration for NFS server failover, can I ensure the IP > address comes up after exports are ready, and more importantly that the IP > address goes down before unexporting by placeing the entry at the > end of the service declaration? The IP is stopped before disk devices are stopped. See the "child types" in service.sh. IPs are started next-to-last (scripts are started last). If you do a typical configuration where the NFS exports are children of file systems, and clients are children of exports, your IP will start after NFS exports are complete and the IP will be taken off before the NFS exports are unexported. > When taking down the nfs service I want the server to 'disappear' before it > starts unexporting, as i want clients to hang waiting for the service to > come back up instead of getting io errors. That's how NFS failover needs to be done, actually. -- Lon From lhh at redhat.com Thu Jul 7 16:05:16 2005 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 07 Jul 2005 12:05:16 -0400 Subject: [Linux-cluster] NFS configuration question In-Reply-To: <1120749723.5534.7.camel@auh5-0478> References: <42CBAF82.8050900@uib.no> <1120685352.5704.9.camel@auh5-0478> <42CD0740.8040102@uib.no> <1120749723.5534.7.camel@auh5-0478> Message-ID: <1120752316.15658.59.camel@ayanami.boston.redhat.com> On Thu, 2005-07-07 at 11:22 -0400, Eric Kerin wrote: > > What happens is exacly what you describe. Exports come up all in one go, > > then the IP address. > > But then, a split second later all exports except the one I have in > > /etc/exports are gone. It's as if something has done 'exportfs -r'. I'll > > have to look into this. Could be my own config problem, as I restart the > > lockd when bringing up the service. However the exports are all there > > when I reboot and let the services come up automatically, and if my > > script is the culprit it should behave the same way then, shouldn't it? > > > Just because I'm curious, why do you restart lockd? Are you restarting > any other nfs related services from rgmanager? > > > Of my 9 export entries in cluster.conf only 5 > > get tested and reexported after disappearing. As I said they are all > > there if I reboot and let the service come up automatically. > > > It'd be interesting to see the relevant section of your cluster.conf > file. Also you don't have any of the filesystems you are exporting in > the cluster setup in /etc/exports, do you? ... and /etc/exports. If the cluster is for some reason unexporting stuff in /etc/exports which it shouldn't, it's a bug. e.g.: If you have /mnt/clusterexport in cluster.conf as an NFS export, and you have "/tmp" as an export in /etc/exports - and the "/tmp" export is dissappearing, that's a bug. If you have /mnt/clusterexport in both cluster.conf and /etc/exports, ... that's a config problem. Let the cluster manage the stuff you intend to export from the cluster. =) -- Lon From jscottorn at possibilityforge.com Thu Jul 7 16:17:46 2005 From: jscottorn at possibilityforge.com (Jon Scottorn) Date: Thu, 07 Jul 2005 10:17:46 -0600 Subject: [Linux-cluster] HELP! Diapered block device In-Reply-To: <42CD4D01.20803@possibilityforge.com> References: <42CD3D2A.2050108@possibilityforge.com> <200507071002.01130.Jason@selu.edu> <42CD475D.3060609@possibilityforge.com> <20050707153435.GD15005@null.msp.redhat.com> <42CD4D01.20803@possibilityforge.com> Message-ID: <42CD55AA.2040004@possibilityforge.com> Ok, so I am rerunning gfs_fsck again. I have everything unmounted and the storage server not even in the cluster. It has been running now for 30 mins again and it gets to this point and just doesn't look like it is doing anything. This is where is stayed after running it for 24 hours yesterday: Here is what the verbose output from gfs_fsck: Initializing fsck Initializing lists... Initializing special inodes... Setting block ranges... Creating a block list of size 183146926... Clearing journals (this may take a while) Clearing journal 0 Clearing journal 1 Clearing journal 2 Clearing journal 3 Cleared journals Starting pass1 Checking metadata in Resource Group 0 Checking metadata in Resource Group 1 Checking metadata in Resource Group 2 Checking metadata in Resource Group 3 ........Omitted lines for space......................... Checking metadata in Resource Group 2790 Checking metadata in Resource Group 2791 Checking metadata in Resource Group 2792 Checking metadata in Resource Group 2793 Pass1 complete Starting pass1b Looking for duplicate blocks... Found dup block at 61573000 Found dup block at 61573014 Found dup block at 61573015 Found dup block at 61573016 Found dup block at 61573017 Found dup block at 61573018 Found dup block at 61573019 Found dup block at 61573020 Found dup block at 61573021 Found dup block at 61573022 Found dup block at 61573024 Found dup block at 61573047 Found dup block at 61573048 Found dup block at 61573052 Found dup block at 61623032 Found dup block at 61623033 Found dup block at 61623034 Found dup block at 61623035 Scanning filesystem for inodes containing duplicate blocks... Once it gets to this point it just sits there. gfs_fsck is using 99% of the CPU for the whole time it runs. What else can I do to get this fixed? Thanks, Jon Jon Scottorn wrote: >When I ran the fsck, i had everything unmounted as well as the gnbd serv >stopped. I let it run for almost 24 hours and it was still running. >That seems a little long for me. Should I let it run again and see what >happens. My main problem is I can't have the FS down for that long. > >Thanks, > >Jon > >AJ Lewis wrote: > > > >>On Thu, Jul 07, 2005 at 09:16:45AM -0600, Jon Scottorn wrote: >> >> >> >> >>>Thanks, >>> >>> That made it so I can mount it from the other nodes, but now I can't >>>mount it on the storage server. >>> >>> >>> >>> >>Gah! Is the fsck still running? You *CANNOT* run the fsck while other nodes >>have the fs mounted. The fsck changes the lock protocol to prevent others >> >> >>from mounting after the fsck starts. It will be changed back after > > >>completion. >> >>The fsck can take a while in the duplicate block code - could you tell if it >>was still accessing storage? If you have lots of inodes in the system, it's >>gonna take a while to work through them in the dup block handling code. >> >>Regards, >> >> >>------------------------------------------------------------------------ >> >>-- >>Linux-cluster mailing list >>Linux-cluster at redhat.com >>http://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster > > > From lhh at redhat.com Thu Jul 7 16:20:45 2005 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 07 Jul 2005 12:20:45 -0400 Subject: [Linux-cluster] [PATCH] add syscall missing includes in rgmanager/src/clulib/gettid.c In-Reply-To: <42CB6563.5050007@fabbione.net> References: <42CB62CB.408@fabbione.net> <42CB6563.5050007@fabbione.net> Message-ID: <1120753245.15658.67.camel@ayanami.boston.redhat.com> On Wed, 2005-07-06 at 07:00 +0200, Fabio Massimo Di Nitto wrote: > Fabio Massimo Di Nitto wrote: > > Hi guys, > > patch is pretty self-explanatory, when we did change the way in which we use > > syscall, we forgot to add the relevant includes. > > Patch is against the STABLE branch, but i am pretty sure it applies all over. > > As a consequence it also shuts up a warning a build time. > > > > Please apply. > > > > Cheers > > Fabio Thanks -- Lon From phung at cs.columbia.edu Thu Jul 7 22:38:16 2005 From: phung at cs.columbia.edu (Dan B. Phung) Date: Thu, 7 Jul 2005 18:38:16 -0400 (EDT) Subject: [Linux-cluster] kernel versions supported by -rSTABLE? Message-ID: Just wondering, which kernel versions are known to work with gfs? -dan From phung at cs.columbia.edu Fri Jul 8 01:01:33 2005 From: phung at cs.columbia.edu (Dan B. Phung) Date: Thu, 7 Jul 2005 21:01:33 -0400 (EDT) Subject: [Linux-cluster] kernel versions supported by -rSTABLE? In-Reply-To: Message-ID: to be more specific, will it work for 2.6.11? On 7, Jul, 2005, Dan B. Phung declared: > Just wondering, which kernel versions are known to > work with gfs? > > -dan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > -- email: phung at cs.columbia.edu www: http://www.cs.columbia.edu/~phung phone: 646-775-6090 office: CS Dept. 520, 1214 Amsterdam Ave., MC 0401, New York, NY 10027 From schlegel at riege.com Fri Jul 8 06:27:13 2005 From: schlegel at riege.com (Gunther Schlegel) Date: Fri, 08 Jul 2005 08:27:13 +0200 Subject: [Linux-cluster] RHEL3 Cluster network hangup In-Reply-To: <1120751357.15658.40.camel@ayanami.boston.redhat.com> References: <42CB7A9B.30304@riege.com> <1120751357.15658.40.camel@ayanami.boston.redhat.com> Message-ID: <42CE1CC1.2070200@riege.com> Lon, > I doubt it, but it's not out of the realm of possibility. The cluster > software does three things mostly: > > (a) figures out who's online > (b) shoots nodes > (c) manages services using shell scripts > > The shell scripts call standard utilities (ifconfig, route, etc.). From the theory you are right ( and you probably know your software ). But what about this: The software we run has "background job managers", which are started by the script I made up for the cluster. When I run lsof on such a bcj-process, it looks like this: [root at tim root]# lsof -p 22993 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME plb 22993 rsi cwd DIR 8,19 24576 2949121 /opt/rsi/de/ham/data plb 22993 rsi rtd DIR 8,10 4096 2 / plb 22993 rsi txt REG 8,19 1043972 10944608 /opt/rsi/plb90f/plb plb 22993 rsi mem REG 8,10 1571824 102450 /lib/tls/libc-2.3.2.so plb 22993 rsi mem REG 8,10 14868 74214 /lib/libdl-2.3.2.so plb 22993 rsi mem REG 8,10 97712 104282 /lib/tls/libpthread-0.60.so plb 22993 rsi mem REG 8,10 23388 73133 /lib/libcrypt-2.3.2.so plb 22993 rsi mem REG 8,6 52584 637864 /usr/lib/libz.so.1.1.4 plb 22993 rsi mem REG 8,10 213508 104281 /lib/tls/libm-2.3.2.so plb 22993 rsi mem REG 8,10 106912 73243 /lib/ld-2.3.2.so plb 22993 rsi 0r CHR 1,3 60122 /dev/null plb 22993 rsi 1w CHR 1,3 60122 /dev/null plb 22993 rsi 2w CHR 1,3 60122 /dev/null plb 22993 rsi 3uw REG 8,19 0 10944633 /opt/rsi/plb90f/.^A^A^A^A^A^A^A^B plb 22993 rsi 4u REG 8,19 2048 2965515 /opt/rsi/de/ham/data/cook.isi plb 22993 rsi 5u REG 8,19 4509 2949322 /opt/rsi/de/ham/data/cook.txt plb 22993 rsi 6u REG 8,19 2304 3134942 /opt/rsi/de/ham/scra/bcjmgr301100000009640.par plb 22993 rsi 7u REG 8,19 2048 868402 /opt/rsi/de/data/bct.isi plb 22993 rsi 8u REG 8,19 1512 869372 /opt/rsi/de/data/bct.txt ... data files shortened ... If the BCJ-Process has been started by the cluster, it also has IP-sockets listet by lsof. The application is that old that it has no clue about IP, therefore it will not open IP sockets itself. Would have added another lsof output, but I disabled the cluster software on customers demand. Another weird fact is that the applications index file get broken if I use ext3. ext2 is fine. Applikation is technically old fashioned and stores data just is hundreds of text files with external index files and concurrent access. Without the cluster ext3 is fine as well. ( Same with LVM, besides. ) > Now -- here's the thing. Earlier versions of clumanager (<1.2.22) had a I have been running 1.2.22. > status return and restart on the same node. Also, the most recent > errata fixed a signal handling problem which broke JVMs from running > under it. Either of these may have caused the problems on your cluster, > I don't know. The former would have associated log messages; the latter > wouldn't. There have not been any log messages. > I'd try the latest release from RHN (clumanager-1.2.26.1-1). Hmm, I will probably not start up the cluster again... :( > If that doesn't work, I'd call Red Hat Support... While calling support is always on option, I am pretty much sure that it will not lead to a solution. In the end they will not be able to reproduce it and I can't test on a customers production system. Do not point me to test systems -- they are there, but they do not have the problem. Seems to be related to the workload of the machine, which is hard to simulate. regards, Gunther -------------- next part -------------- A non-text attachment was scrubbed... Name: schlegel.vcf Type: text/x-vcard Size: 331 bytes Desc: not available URL: From birger at uib.no Fri Jul 8 07:57:12 2005 From: birger at uib.no (Birger Wathne) Date: Fri, 08 Jul 2005 09:57:12 +0200 Subject: [Linux-cluster] NFS configuration question In-Reply-To: <1120749723.5534.7.camel@auh5-0478> References: <42CBAF82.8050900@uib.no> <1120685352.5704.9.camel@auh5-0478> <42CD0740.8040102@uib.no> <1120749723.5534.7.camel@auh5-0478> Message-ID: <42CE31D8.4090206@uib.no> Eric Kerin wrote: >Just because I'm curious, why do you restart lockd? Are you restarting >any other nfs related services from rgmanager? > > I think I found the bug. A line had sneaked (snuck?) back into cluster.conf from an early experiment. I have just been blind to this line in the config file. I once had it set up so that the service started NFS. Later I changed it so NFS gets started at boot with just a dummy export in my exports file. Now, the line starting NFS was back in my config file. Must have been a copy/paste error at some time. The lockd/statd that start at boot shouldn't really have anything to do, so when the service starts I simply run a modified version of /etc/init.d/nfslock that kills the existing daemons and start them again with some parameters set... I set the hostname to the one that follows the nfs service, and redirect the state dir to a gfs disk. When the service stops, these daemons get stopped, and the default ones started again. This way I hope to get the HA service to also take over locks. Here is a snippet of my config file showing the resources and the nfs service. Of the listed exports, the first one for users01 and all 3 for iftscratch don't seem to get status checks, and thus never came up again.