From schuan2 at gmail.com Wed Dec 1 04:48:52 2004 From: schuan2 at gmail.com (Shih-Che Huang) Date: Tue, 30 Nov 2004 23:48:52 -0500 Subject: [Linux-cluster] GNBD+RAID5 Message-ID: Hi Shih-Che, If I understand correctly you have three nodes (PC's) with local harddisks of size 35GB. You want to make a raid-5 array using these three harddisks and at the sametime make this data accessible from all the three nodes. Is it correct? I am sorry but I didn't try GNBD myself yet. I don't know how much I can help you. But let me try >#gfs_mkfs -p lock_gulm -t alpha:gfstest -j 3 /dev/pool/storage >#mount -t gfs /dev/pool/storage /gfs1 Instead of making one big pool of 70GB make three pools /dev/pool/storage1,2, 3. and then you can mount them onto /gfs1,2,3. You can then use Linux MD (Meta Devices) driver to create raid-5 array (software raid). You will edit /etc/raidtab, check this link: http://www.tldp.org/HOWTO/Software-RAID-HOWTO-5.html#ss5.8 It's just my thought but I don't know if it works. You may want to post this question and my reply to linux-cluster and see if any experts can help you out. Also, it looks like PVFS (Parallel Virtual File System) may address your problem correctly. Check this: Look at what is PVFS? http://www.parl.clemson.edu/pvfs/pvfs-faq.html#what Hope this helps! Raj On Tue, 30 Nov 2004 Shih-Che Huang wrote : - Hide quoted text - >Hi Raj, > >Following is my idea. >Do you think that I can do it under GNBD? > >I have two nodes and each of them has 35GB, and then I want to combine two >storage together to be 70GB. > >Under this condition, I can use this 70GB storage. > > Master /gfs1 (70 GB) > | > | > / \ > / \ > Kh00 Kh01 > 35GB 35GB > >I import GNDB from Kh00 and Kh01 and then I had following storage.cfg > >poolname stogage >minor subpools 2 >subpool 0 128 1 gfs_data >subpool 1 128 1 gfs_data >pooldevice 0 0 /dev/gnbd/gfstest >pooldevice 1 0 /dev/gnbd/gfstest1 > >======================= >#gfs_mkfs -p lock_gulm -t alpha:gfstest -j 3 /dev/pool/storage >#mount -t gfs /dev/pool/storage /gfs1 > >After I mounted it, I can see 70GB storage and I can used it. > >we also want some >RAID1 or RAID5-like redundancy in case of a hard drive failure. > >So it would be more like this: >We have three nodes, kh00, kh01, kh02, each with 35 GB, that together >give us a 70 GB GFS with 35 GB of parity/redundancy. >It's more likely to be like this: >We have four nodes, kh00, kh01, kh02, kh03, each with 35 GB, that are >paired up in redundant copies, yielding 70 GB of usable storage. > >What should I do by using GNBD in GFS? >Could you give me some suggestion? > >Shih-Che -- Shih-Che Huang From linux-cluster at spam.dragonhold.org Wed Dec 1 15:43:42 2004 From: linux-cluster at spam.dragonhold.org (Graham Wood) Date: Wed, 1 Dec 2004 15:43:42 +0000 Subject: [Linux-cluster] GNBD+RAID5 In-Reply-To: References: Message-ID: <20041201154342.GA5330@dragonhold.org> On Tue, Nov 30, 2004 at 11:48:52PM -0500, Shih-Che Huang wrote: > Hi Shih-Che, > > >#gfs_mkfs -p lock_gulm -t alpha:gfstest -j 3 /dev/pool/storage > >#mount -t gfs /dev/pool/storage /gfs1 > Instead of making one big pool of 70GB make three pools > /dev/pool/storage1,2, 3. and then you can mount them onto /gfs1,2,3. > You can then use Linux MD (Meta Devices) driver to create raid-5 array > (software raid). I'm not sure what you're suggesting here, but I don't think it will work - at all. Linux MD runs on top of block devices, not filesystems. Therefore to do anything on top of the GFS, you'd be looking at using the loopback device to mount files under the filesystems. Whilst this will give you block devices on shared storage, it won't help at all with respect to data integrity & management, since you'll still have the underlying problem with the lack of support for clustering in the MD driver. To give you an example - as soon as you attempt to start the MD on the second node, it's going to notice (from the metainfo stored within it) that the device is still open already. This could easily start a RAID rebuild - which is going to hammer your IO. The 1st node is likely to notice that the rebuild has started, and since it doesn't see any reason for it, is likely to panic or at least complain that the metainfo has been written to by someone else. I don't know enough about the MD driver to know if it would cope with this - and that's just starting up the 2nd node - if you add a 3rd (assuming that a rebuild happened successfully and then nodes 1 & 2 both accepted it was all happy), then you get even more problems. Also you have to allow for the underlying GFS partition to fail, and be made available again. As soon as you do that, you need to tell all the nodes that it is available - and they are then all going to start resyncing - so your IO is going to be massacred. And, of course, you'd have to run GFS over the top as well, since otherwise you get filesystem caching problems. GFS running on MD running on GFS running on GNBD - that's a recipe for disaster. > >Could you give me some suggestion? The latest CVS code apparently has some mirroring support via the device mapper, which is probably the best place for it *grin*, but it's early days at the moment, so it is likely that the system will change. At the moment I don't think there's an easy way to do it. There may be a way to do it by using GNBD directly on top of MD, and only sharing it out from one server at a time (and using something like heartbeat to shift the IP to another machine should that one fail). That way you've only got one machine accessing the block devices at a time. GNBD can export md devices, so this /should/ work. However, without actually testing it, I wouldn't want to rely on it in a live system. Graham From vahram at Broadspire.com Wed Dec 1 17:29:55 2004 From: vahram at Broadspire.com (vahram) Date: Wed, 01 Dec 2004 09:29:55 -0800 Subject: [Linux-cluster] having problems trying to setup a two node cluster Message-ID: <41ADFF93.5030308@broadspire.com> Hi all, I'm trying to set up a little two node demo cluster, but I'm having some weird issues. Here's my cluster.conf: I have both nodeA and nodeB defined in /etc/hosts on both boxes, and I launched ccsd on both machines. All the necessary modules are also loaded. When I run ccs_test connect, I get a connection refused message. I then tried to run cman_tool join on both boxes. nodeB successfully receives a copy of cluster.conf, but it then tries to form it's own cluster. In other words...nodeA sees this: [root at nodeA ~]# cat /proc/cluster/status Version: 3.0.1 Config version: 1 Cluster name: demo Cluster ID: 40925 Membership state: Cluster-Member Nodes: 1 Expected_votes: 1 Total_votes: 1 Quorum: 1 Active subsystems: 0 Node addresses: 192.168.2.203 [root at nodeA ~]# cat /proc/cluster/nodes Node Votes Exp Sts Name 1 1 1 M nodeA and nodeB sees this: [root at nodeB ~]# cat /proc/cluster/status Version: 3.0.1 Config version: 1 Cluster name: demo Cluster ID: 40925 Membership state: Cluster-Member Nodes: 1 Expected_votes: 1 Total_votes: 1 Quorum: 1 Active subsystems: 0 Node addresses: 192.168.2.204 [root at nodeB ~]# cat /proc/cluster/nodes Node Votes Exp Sts Name 1 1 1 M nodeB Both boxes are running FC3, and I used the SRPM's from http://people.redhat.com/cfeist/cluster/SRPMS/ Please let me know what I'm doing wrong here...thanks! -vahram From rstevens at vitalstream.com Wed Dec 1 18:08:49 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Wed, 01 Dec 2004 10:08:49 -0800 Subject: [Linux-cluster] having problems trying to setup a two node cluster In-Reply-To: <41ADFF93.5030308@broadspire.com> References: <41ADFF93.5030308@broadspire.com> Message-ID: <41AE08B1.6020803@vitalstream.com> vahram wrote: > Hi all, > > I'm trying to set up a little two node demo cluster, but I'm having some > weird issues. > > Here's my cluster.conf: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have both nodeA and nodeB defined in /etc/hosts on both boxes, and I > launched ccsd on both machines. All the necessary modules are also loaded. > > When I run ccs_test connect, I get a connection refused message. I then > tried to run cman_tool join on both boxes. nodeB successfully receives > a copy of cluster.conf, but it then tries to form it's own cluster. In > other words...nodeA sees this: > > [root at nodeA ~]# cat /proc/cluster/status > Version: 3.0.1 > Config version: 1 > Cluster name: demo > Cluster ID: 40925 > Membership state: Cluster-Member > Nodes: 1 > Expected_votes: 1 > Total_votes: 1 > Quorum: 1 > Active subsystems: 0 > Node addresses: 192.168.2.203 > [root at nodeA ~]# cat /proc/cluster/nodes > Node Votes Exp Sts Name > 1 1 1 M nodeA > > and nodeB sees this: > > [root at nodeB ~]# cat /proc/cluster/status > Version: 3.0.1 > Config version: 1 > Cluster name: demo > Cluster ID: 40925 > Membership state: Cluster-Member > Nodes: 1 > Expected_votes: 1 > Total_votes: 1 > Quorum: 1 > Active subsystems: 0 > Node addresses: 192.168.2.204 > [root at nodeB ~]# cat /proc/cluster/nodes > Node Votes Exp Sts Name > 1 1 1 M nodeB > > Both boxes are running FC3, and I used the SRPM's from > http://people.redhat.com/cfeist/cluster/SRPMS/ > > Please let me know what I'm doing wrong here...thanks! I had a similar issue. The problem was with the multicast routing. I was using two NICs on each node...one public (eth0) and one private (eth1), with the default gateway going out eth0. The route for the multicast (224.x.x.x) was going out the default gateway and not reaching the other machine. By putting in a fixed route in for multicast: route add -net 224.0.0.0/8 dev eth1 it all started working. This was my fix, it may not work for you. Also, I use the CVS code from http://sources.redhat.com/cluster and not the source RPMs from where you specified. ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Veni, Vidi, VISA: I came, I saw, I did a little shopping. - ---------------------------------------------------------------------- From vahram at Broadspire.com Wed Dec 1 18:25:08 2004 From: vahram at Broadspire.com (vahram) Date: Wed, 01 Dec 2004 10:25:08 -0800 Subject: [Linux-cluster] having problems trying to setup a two node cluster In-Reply-To: <41AE08B1.6020803@vitalstream.com> References: <41ADFF93.5030308@broadspire.com> <41AE08B1.6020803@vitalstream.com> Message-ID: <41AE0C84.9090504@broadspire.com> Rick Stevens wrote: > > I had a similar issue. The problem was with the multicast routing. > I was using two NICs on each node...one public (eth0) and one private > (eth1), with the default gateway going out eth0. > > The route for the multicast (224.x.x.x) was going out the default > gateway and not reaching the other machine. By putting in a fixed route > in for multicast: > > route add -net 224.0.0.0/8 dev eth1 > > it all started working. This was my fix, it may not work for you. > Also, I use the CVS code from http://sources.redhat.com/cluster and > not the source RPMs from where you specified. > ---------------------------------------------------------------------- > - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - > - VitalStream, Inc. http://www.vitalstream.com - > - - > - Veni, Vidi, VISA: I came, I saw, I did a little shopping. - > ---------------------------------------------------------------------- > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster Yeap, both boxes have two NICs. eth0 is public, and eth1 is private (192.168.2.x). I tried adding the route, and that didn't fix it. I've also tried disabling the private NIC before and running with one public NIC, and that didn't fix it either. One other interesting thing I noticed...when I run cman_tool join on nodeA, netstat shows ccsd trying to do this: tcp 0 0 127.0.0.1:50006 127.0.0.1:739 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:738 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:737 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:736 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:743 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:742 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:741 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:740 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:727 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:731 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:730 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:729 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:728 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:735 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:734 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:733 TIME_WAIT - tcp 0 0 127.0.0.1:50006 127.0.0.1:732 TIME_WAIT - -vahram From rstevens at vitalstream.com Wed Dec 1 18:39:49 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Wed, 01 Dec 2004 10:39:49 -0800 Subject: [Linux-cluster] having problems trying to setup a two node cluster In-Reply-To: <41AE0C84.9090504@broadspire.com> References: <41ADFF93.5030308@broadspire.com> <41AE08B1.6020803@vitalstream.com> <41AE0C84.9090504@broadspire.com> Message-ID: <41AE0FF5.6080908@vitalstream.com> vahram wrote: > Rick Stevens wrote: > >> >> I had a similar issue. The problem was with the multicast routing. >> I was using two NICs on each node...one public (eth0) and one private >> (eth1), with the default gateway going out eth0. >> >> The route for the multicast (224.x.x.x) was going out the default >> gateway and not reaching the other machine. By putting in a fixed route >> in for multicast: >> >> route add -net 224.0.0.0/8 dev eth1 >> >> it all started working. This was my fix, it may not work for you. >> Also, I use the CVS code from http://sources.redhat.com/cluster and >> not the source RPMs from where you specified. >> ---------------------------------------------------------------------- >> - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - >> - VitalStream, Inc. http://www.vitalstream.com - >> - - >> - Veni, Vidi, VISA: I came, I saw, I did a little shopping. - >> ---------------------------------------------------------------------- >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> http://www.redhat.com/mailman/listinfo/linux-cluster > > > Yeap, both boxes have two NICs. eth0 is public, and eth1 is private > (192.168.2.x). I tried adding the route, and that didn't fix it. I've > also tried disabling the private NIC before and running with one public > NIC, and that didn't fix it either. One other interesting thing I > noticed...when I run cman_tool join on nodeA, netstat shows ccsd trying > to do this: > > tcp 0 0 127.0.0.1:50006 127.0.0.1:739 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:738 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:737 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:736 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:743 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:742 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:741 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:740 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:727 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:731 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:730 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:729 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:728 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:735 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:734 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:733 > TIME_WAIT - > tcp 0 0 127.0.0.1:50006 127.0.0.1:732 > TIME_WAIT - > Looking back at your cluster.conf, I see you're using broadcast. I used multicast because, in the first CVS checkout I did, broadcast didn't work properly. It's possible your SRPMs also have that flaw. Why not try multicast and see if that works. Add that route I mentioned and here's my cluster.conf which you can crib: ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - What's small, yellow and very, VERY dangerous? The root canary! - ---------------------------------------------------------------------- From owen at isrl.uiuc.edu Wed Dec 1 18:49:20 2004 From: owen at isrl.uiuc.edu (Brynnen R Owen) Date: Wed, 1 Dec 2004 12:49:20 -0600 Subject: [Linux-cluster] having problems trying to setup a two node cluster In-Reply-To: <41AE0FF5.6080908@vitalstream.com> References: <41ADFF93.5030308@broadspire.com> <41AE08B1.6020803@vitalstream.com> <41AE0C84.9090504@broadspire.com> <41AE0FF5.6080908@vitalstream.com> Message-ID: <20041201184919.GC1232@iwork57.lis.uiuc.edu> One possibility may be that the hostnames used in your cluster.conf file resolve to 127.0.0.1 in /etc/hosts. Then the system will try to broadcast to the lo device. By the way, we're using CVS from Nov 21 with broadcast on dual-nic's. On Wed, Dec 01, 2004 at 10:39:49AM -0800, Rick Stevens wrote: > vahram wrote: > >Rick Stevens wrote: > > > >> > >>I had a similar issue. The problem was with the multicast routing. > >>I was using two NICs on each node...one public (eth0) and one private > >>(eth1), with the default gateway going out eth0. > >> > >>The route for the multicast (224.x.x.x) was going out the default > >>gateway and not reaching the other machine. By putting in a fixed route > >>in for multicast: > >> > >> route add -net 224.0.0.0/8 dev eth1 > >> > >>it all started working. This was my fix, it may not work for you. > >>Also, I use the CVS code from http://sources.redhat.com/cluster and > >>not the source RPMs from where you specified. > >>---------------------------------------------------------------------- > >>- Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - > >>- VitalStream, Inc. http://www.vitalstream.com - > >>- - > >>- Veni, Vidi, VISA: I came, I saw, I did a little shopping. - > >>---------------------------------------------------------------------- > >> > >>-- > >>Linux-cluster mailing list > >>Linux-cluster at redhat.com > >>http://www.redhat.com/mailman/listinfo/linux-cluster > > > > > >Yeap, both boxes have two NICs. eth0 is public, and eth1 is private > >(192.168.2.x). I tried adding the route, and that didn't fix it. I've > >also tried disabling the private NIC before and running with one public > >NIC, and that didn't fix it either. One other interesting thing I > >noticed...when I run cman_tool join on nodeA, netstat shows ccsd trying > >to do this: > > > >tcp 0 0 127.0.0.1:50006 127.0.0.1:739 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:738 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:737 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:736 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:743 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:742 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:741 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:740 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:727 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:731 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:730 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:729 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:728 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:735 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:734 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:733 > >TIME_WAIT - > >tcp 0 0 127.0.0.1:50006 127.0.0.1:732 > >TIME_WAIT - > > > > Looking back at your cluster.conf, I see you're using broadcast. I used > multicast because, in the first CVS checkout I did, broadcast didn't > work properly. It's possible your SRPMs also have that flaw. Why not > try multicast and see if that works. Add that route I mentioned and > here's my cluster.conf which you can crib: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - > - VitalStream, Inc. http://www.vitalstream.com - > - - > - What's small, yellow and very, VERY dangerous? The root canary! - > ---------------------------------------------------------------------- > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -- <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> <> Brynnen Owen ( this space for rent )<> <> owen at uiuc.edu ( )<> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> From jbrassow at redhat.com Wed Dec 1 19:15:14 2004 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Wed, 1 Dec 2004 13:15:14 -0600 Subject: [Linux-cluster] having problems trying to setup a two node cluster In-Reply-To: <41ADFF93.5030308@broadspire.com> References: <41ADFF93.5030308@broadspire.com> Message-ID: <53FFF3B2-43CD-11D9-86A2-000A957BB1F6@redhat.com> try to do a 'ccs_test connect' after doing the cman_tool join, and see what happens ccs_test will not allow you to connect to an inquorate cluster, unless you use the force option. Since your nodes become quorate after cman_tool join, you should be able to connect... If you can not, it suggests that the magma-plugins are not installed/correct. brassow On Dec 1, 2004, at 11:29 AM, vahram wrote: > Hi all, > > I'm trying to set up a little two node demo cluster, but I'm having > some weird issues. > > Here's my cluster.conf: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have both nodeA and nodeB defined in /etc/hosts on both boxes, and I > launched ccsd on both machines. All the necessary modules are also > loaded. > > When I run ccs_test connect, I get a connection refused message. I > then tried to run cman_tool join on both boxes. nodeB successfully > receives a copy of cluster.conf, but it then tries to form it's own > cluster. In other words...nodeA sees this: > > [root at nodeA ~]# cat /proc/cluster/status > Version: 3.0.1 > Config version: 1 > Cluster name: demo > Cluster ID: 40925 > Membership state: Cluster-Member > Nodes: 1 > Expected_votes: 1 > Total_votes: 1 > Quorum: 1 > Active subsystems: 0 > Node addresses: 192.168.2.203 > [root at nodeA ~]# cat /proc/cluster/nodes > Node Votes Exp Sts Name > 1 1 1 M nodeA > > and nodeB sees this: > > [root at nodeB ~]# cat /proc/cluster/status > Version: 3.0.1 > Config version: 1 > Cluster name: demo > Cluster ID: 40925 > Membership state: Cluster-Member > Nodes: 1 > Expected_votes: 1 > Total_votes: 1 > Quorum: 1 > Active subsystems: 0 > Node addresses: 192.168.2.204 > [root at nodeB ~]# cat /proc/cluster/nodes > Node Votes Exp Sts Name > 1 1 1 M nodeB > > Both boxes are running FC3, and I used the SRPM's from > http://people.redhat.com/cfeist/cluster/SRPMS/ > > Please let me know what I'm doing wrong here...thanks! > > -vahram > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From vahram at Broadspire.com Wed Dec 1 19:16:21 2004 From: vahram at Broadspire.com (vahram) Date: Wed, 01 Dec 2004 11:16:21 -0800 Subject: [Linux-cluster] having problems trying to setup a two node cluster In-Reply-To: <41AE0FF5.6080908@vitalstream.com> References: <41ADFF93.5030308@broadspire.com> <41AE08B1.6020803@vitalstream.com> <41AE0C84.9090504@broadspire.com> <41AE0FF5.6080908@vitalstream.com> Message-ID: <41AE1885.7060401@broadspire.com> this did the trick...thanks! Rick Stevens wrote: > Looking back at your cluster.conf, I see you're using broadcast. I used > multicast because, in the first CVS checkout I did, broadcast didn't > work properly. It's possible your SRPMs also have that flaw. Why not > try multicast and see if that works. Add that route I mentioned and > here's my cluster.conf which you can crib: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - > - VitalStream, Inc. http://www.vitalstream.com - > - - > - What's small, yellow and very, VERY dangerous? The root canary! - > ---------------------------------------------------------------------- > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From rstevens at vitalstream.com Wed Dec 1 21:40:34 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Wed, 01 Dec 2004 13:40:34 -0800 Subject: [Linux-cluster] having problems trying to setup a two node cluster In-Reply-To: <41AE1885.7060401@broadspire.com> References: <41ADFF93.5030308@broadspire.com> <41AE08B1.6020803@vitalstream.com> <41AE0C84.9090504@broadspire.com> <41AE0FF5.6080908@vitalstream.com> <41AE1885.7060401@broadspire.com> Message-ID: <41AE3A52.5050701@vitalstream.com> vahram wrote: > this did the trick...thanks! Glad to hear it. You might want to try the CVS version of the code rather than the SRPM versions. The CVS is generally more current and I think they've fixed the broadcast option in it. > Rick Stevens wrote: > >> Looking back at your cluster.conf, I see you're using broadcast. I used >> multicast because, in the first CVS checkout I did, broadcast didn't >> work properly. It's possible your SRPMs also have that flaw. Why not >> try multicast and see if that works. Add that route I mentioned and >> here's my cluster.conf which you can crib: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ---------------------------------------------------------------------- >> - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - >> - VitalStream, Inc. http://www.vitalstream.com - >> - - >> - What's small, yellow and very, VERY dangerous? The root canary! - >> ---------------------------------------------------------------------- >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > -- ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Errors have occurred. We won't tell you where or why. We have - - lazy programmers. - ---------------------------------------------------------------------- From daniel at osdl.org Thu Dec 2 00:55:56 2004 From: daniel at osdl.org (Daniel McNeil) Date: Wed, 01 Dec 2004 16:55:56 -0800 Subject: [Linux-cluster] Some simple GFS test scripts In-Reply-To: <20041128044443.17162.qmail@webmail18.rediffmail.com> References: <20041128044443.17162.qmail@webmail18.rediffmail.com> Message-ID: <1101948956.10109.20.camel@ibm-c.pdx.osdl.net> I've posted my simple test scripts I've been using to test GFS on the osdl developer web page: http://developer.osdl.org/daniel/gfs_tests/ So far, I have not been able to run these overnight without hitting some problem. I am upgrading to the latest cvs code and will be running these again. I'll post any problems. Daniel From ben.m.cahill at intel.com Thu Dec 2 19:40:55 2004 From: ben.m.cahill at intel.com (Cahill, Ben M) Date: Thu, 2 Dec 2004 11:40:55 -0800 Subject: [Linux-cluster] [PATCH] YAROC (yet another round of comments) Message-ID: <0604335B7764D141945E202153105960033E26DE@orsmsx404.amr.corp.intel.com> FYI, I just sent another GFS comments patch to Ken for review. -- Ben -- Opinions are mine, not Intel's From rstevens at vitalstream.com Thu Dec 2 21:27:00 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Thu, 02 Dec 2004 13:27:00 -0800 Subject: [Linux-cluster] [PATCH] YAROC (yet another round of comments) In-Reply-To: <0604335B7764D141945E202153105960033E26DE@orsmsx404.amr.corp.intel.com> References: <0604335B7764D141945E202153105960033E26DE@orsmsx404.amr.corp.intel.com> Message-ID: <41AF88A4.2070807@vitalstream.com> Cahill, Ben M wrote: > FYI, > > I just sent another GFS comments patch to Ken for review. Just comments? ;-) ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Careful! Ugly strikes 9 out of 10 people! - ---------------------------------------------------------------------- From amanthei at redhat.com Thu Dec 2 21:40:22 2004 From: amanthei at redhat.com (Adam Manthei) Date: Thu, 2 Dec 2004 15:40:22 -0600 Subject: [Linux-cluster] Device missing In-Reply-To: <834F55E6F1BE3B488AD3AFC927A0970007B29A@EMAILSRV1.exad.net> References: <834F55E6F1BE3B488AD3AFC927A0970007B29A@EMAILSRV1.exad.net> Message-ID: <20041202214022.GW15503@redhat.com> use "pool_assemble" to activate the pool device. use "pool_assemble -r" to deactivate the deivce. This will also remove the the device node. use "pool_info" to see what pools are loaded use "pool_tool -s" to scan the system to see what pools are there On Tue, Nov 30, 2004 at 06:44:02PM -0500, Vincent Aniello wrote: > > One of my GFS file system is no longer mountable because the device > associated with it has disappeared from my system. The file system is > named /gfs02 and the device it points to is /dev/pool/pool_gfs02. > > [root at dvblkc01a gfs]# mount /gfs02 > mount: special device /dev/pool/pool_gfs02 does not exist > [root at dvblkc01a gfs]# > > The device /dev/pool/pool_gfs02 is missing from my /dev/pool directory. > It used to be there. > > [root at dvblkc01a gfs]# ls -l /dev/pool > total 0 > brw------- 2 root root 254, 65 Nov 29 18:22 dvcluster_cca > brw------- 2 root root 254, 66 Nov 29 18:22 pool_gfs01 > brw------- 2 root root 254, 67 Nov 29 18:22 pool_gfs03 > [root at dvblkc01a gfs]# > > The device /dev/sdc2 is associated with /dev/pool/pool_gfs02: > > poolname pool_gfs02 > subpools 1 > subpool 0 0 1 > pooldevice 0 0 /dev/sdc2 > > The device /dev/sdc2 seems to be available to the system: > > Disk /dev/sdc: 71.7 GB, 71728889856 bytes > 255 heads, 63 sectors/track, 8720 cylinders > Units = cylinders of 16065 * 512 = 8225280 bytes > > Device Boot Start End Blocks Id System > /dev/sdc1 1 2 16033+ 83 Linux > /dev/sdc2 3 8720 70027335 83 Linux > > > So, how do I get this device and file system back so I can mount it and > what would make it dissapear in the first place? > > --Vincent > > > > This e-mail and/or its attachments may contain confidential and/or privileged information. If you are not the intended recipient(s) or have received this e-mail in error, please notify the sender immediately and delete this e-mail and its attachments from your computer and files. Any unauthorized copying, disclosure or distribution of the material contained herein is strictly forbidden. Pipeline Trading Systems, LLC - Member NASD & SIPC. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -- Adam Manthei From Vincent.Aniello at PipelineTrading.com Fri Dec 3 19:01:55 2004 From: Vincent.Aniello at PipelineTrading.com (Vincent Aniello) Date: Fri, 3 Dec 2004 14:01:55 -0500 Subject: [Linux-cluster] GFS with Cluster Suite Message-ID: <834F55E6F1BE3B488AD3AFC927A0970007B4D1@EMAILSRV1.exad.net> Is Redhat Cluster Suite required for using GFS? If not, what would be the advantage of running Cluster Suite with GFS? --Vincent This e-mail and/or its attachments may contain confidential and/or privileged information. If you are not the intended recipient(s) or have received this e-mail in error, please notify the sender immediately and delete this e-mail and its attachments from your computer and files. Any unauthorized copying, disclosure or distribution of the material contained herein is strictly forbidden. Pipeline Trading Systems, LLC - Member NASD & SIPC. From mbrookov at mines.edu Fri Dec 3 22:47:55 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Fri, 03 Dec 2004 15:47:55 -0700 Subject: [Linux-cluster] GFS no version for lm_mount foudn: kernel tainted Message-ID: <1102114075.6916.8.camel@merlin.Mines.EDU> I am trying to set up GFS on a 2 node cluster running Fedora core 3. The GFS kernel module is generating this error: "FATAL: Error inserting gfs (/lib/modules/2.6.9/kernel/fs/gfs/gfs.ko): Unknown symbol in module, or unknown parameter (see dmesg)". I am also getting a "gfs: no version for "lm_mount" found: kernel tainted." and "gfs: Unknown symbol panic_on_oops" errors in the logs. The lock_dlm, dlm, cman and lock_harness modules do load, but the gfs module does not. Any ideas? The source code for the cluster, device-mapper and LVM2 were pulled from the CVS server today (12/3/2004). I have tried the default kernel, 2.6.9-1.681_FC3 using the source code in the SRPM and I also tried a copy of linux-2.6.9 from kernel.org. I have included a copy of the 2.6.9 kernel dmesg, appropriate log entries from /var/log/messages, the commands that I typed, and a copy of the cluster.conf. The logical volumes and GFS file systems were not created with this version of GFS, but from a version of GFS that I set up almost 2 months ago. I doubt that the logical volumes and file systems are part of this problem because I never got to the point where I was loading clvmd. Thanks for you help Matthew B. Brookover Colorado School of Mines mbrookov at mines.edu Commands: [root at fouroften GFS-FEDORA-try2]# modprobe dm-mod [root at fouroften GFS-FEDORA-try2]#./device-mapper/scripts/devmap_mknod.sh Creating /dev/mapper/control character device with major:10 minor:63. [root at fouroften GFS-FEDORA-try2]# modprobe gfs FATAL: Error inserting gfs (/lib/modules/2.6.9/kernel/fs/gfs/gfs.ko): Unknown symbol in module, or unknown parameter (see dmesg) [root at fouroften GFS-FEDORA-try2]# modprobe lock_dlm [root at fouroften GFS-FEDORA-try2]# dmesg > dmesg.out [root at fouroften GFS-FEDORA-try2]# ccsd [root at fouroften GFS-FEDORA-try2]# cman_tool join cman_tool: local node name "fouroften" not found in cluster.conf [root at fouroften GFS-FEDORA-try2]# Logs: Dec 3 14:56:02 fouroften kernel: Lock_Harness (built Dec 3 2004 14:18:33) installed Dec 3 14:56:02 fouroften kernel: gfs: no version for "lm_mount" found: kernel tainted. Dec 3 14:56:02 fouroften kernel: gfs: Unknown symbol panic_on_oops Dec 3 14:56:07 fouroften ntpd[2188]: synchronized to LOCAL(0), stratum 10 Dec 3 14:56:07 fouroften ntpd[2188]: kernel time sync disabled 0041 Dec 3 14:56:13 fouroften kernel: CMAN (built Dec 3 2004 14:17:50) installed Dec 3 14:56:13 fouroften kernel: NET: Registered protocol family 30 Dec 3 14:56:13 fouroften kernel: DLM (built Dec 3 2004 14:18:20) installed Dec 3 14:56:13 fouroften kernel: Lock_DLM (built Dec 3 2004 14:18:43) installed cluster.conf: dmesg output for 2.6.9: Linux version 2.6.9 (mbrookov at fouroften.mines.edu) (gcc version 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)) #1 Fri Dec 3 11:05:35 MST 2004 BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable) BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved) BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 000000001ffc0000 (usable) BIOS-e820: 000000001ffc0000 - 000000001fff8000 (ACPI data) BIOS-e820: 000000001fff8000 - 0000000020000000 (ACPI NVS) BIOS-e820: 00000000ffb80000 - 00000000ffc00000 (reserved) BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved) 0MB HIGHMEM available. 511MB LOWMEM available. On node 0 totalpages: 131008 DMA zone: 4096 pages, LIFO batch:1 Normal zone: 126912 pages, LIFO batch:16 HighMem zone: 0 pages, LIFO batch:1 DMI 2.3 present. ACPI: RSDP (v000 AMI ) @ 0x000ff980 ACPI: RSDT (v001 D815EA D815EPFV 0x20010404 MSFT 0x00001011) @ 0x1fff0000 ACPI: FADT (v001 D815EA EA81510A 0x20010404 MSFT 0x00001011) @ 0x1fff1000 ACPI: DSDT (v001 D815E2 EA81520A 0x00000021 MSFT 0x0100000b) @ 0x00000000 ACPI: PM-Timer IO Port: 0x408 Built 1 zonelists Kernel command line: ro root=LABEL=/ rhgb quiet Initializing CPU#0 PID hash table entries: 2048 (order: 11, 32768 bytes) Detected 996.487 MHz processor. Using pmtmr for high-res timesource Console: colour VGA+ 80x25 Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) Memory: 514296k/524032k available (2512k kernel code, 9208k reserved, 672k data, 168k init, 0k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. Calibrating delay loop... 1978.36 BogoMIPS (lpj=989184) Security Scaffold v1.0.0 initialized SELinux: Initializing. SELinux: Starting in permissive mode There is already a security framework initialized, register_security failed. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 512 (order: 0, 4096 bytes) CPU: After generic identify, caps: 0383f9ff 00000000 00000000 00000000 CPU: After vendor identify, caps: 0383f9ff 00000000 00000000 00000000 CPU: L1 I cache: 16K, L1 D cache: 16K CPU: L2 cache: 256K CPU: After all inits, caps: 0383f9ff 00000000 00000000 00000040 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#0. CPU: Intel Pentium III (Coppermine) stepping 0a Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Checking 'hlt' instruction... OK. ACPI: IRQ9 SCI: Level Trigger. checking if image is initramfs... it is Freeing initrd memory: 574k freed NET: Registered protocol family 16 PCI: PCI BIOS revision 2.10 entry at 0xfda95, last bus=2 PCI: Using configuration type 1 mtrr: v2.0 (20020519) ACPI: Subsystem revision 20040816 ACPI: Interpreter enabled ACPI: Using PIC for interrupt routing ACPI: PCI Root Bridge [PCI0] (00:00) PCI: Probing PCI hardware (bus 00) PCI: Transparent bridge - 0000:00:1e.0 ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT] ACPI: Power Resource [URP1] (off) ACPI: Power Resource [URP2] (off) ACPI: Power Resource [FDDP] (off) ACPI: Power Resource [LPTP] (off) ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12) ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 *10 11 12) ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 11 12) *0, disabled. ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11 12) ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 *11 12) ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12) *0, disabled. ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 *9 10 11 12) ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11 12) Linux Plug and Play Support v0.97 (c) Adam Belay usbcore: registered new driver usbfs usbcore: registered new driver hub PCI: Using ACPI for IRQ routing ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 ACPI: PCI interrupt 0000:00:1f.2[D] -> GSI 11 (level, low) -> IRQ 11 ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 ACPI: PCI interrupt 0000:00:1f.3[B] -> GSI 10 (level, low) -> IRQ 10 ACPI: PCI Interrupt Link [LNKH] enabled at IRQ 11 ACPI: PCI interrupt 0000:00:1f.4[C] -> GSI 11 (level, low) -> IRQ 11 ACPI: PCI interrupt 0000:00:1f.5[B] -> GSI 10 (level, low) -> IRQ 10 ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 ACPI: PCI interrupt 0000:02:00.0[A] -> GSI 11 (level, low) -> IRQ 11 ACPI: PCI Interrupt Link [LNKE] enabled at IRQ 11 ACPI: PCI interrupt 0000:01:08.0[A] -> GSI 11 (level, low) -> IRQ 11 ACPI: PCI Interrupt Link [LNKG] enabled at IRQ 9 ACPI: PCI interrupt 0000:01:0a.0[A] -> GSI 9 (level, low) -> IRQ 9 apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac) apm: overridden by ACPI. audit: initializing netlink socket (disabled) audit(1102085502.573:0): initialized Total HugeTLB memory allocated, 0 VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) SELinux: Registering netfilter hooks Initializing Cryptographic API pci_hotplug: PCI Hot Plug PCI Core version: 0.5 vesafb: probe of vesafb0 failed with error -6 ACPI: Processor [CPU1] (supports C1) isapnp: Scanning for PnP cards... isapnp: No Plug & Play device found Real Time Clock Driver v1.12 Linux agpgart interface v0.100 (c) Dave Jones agpgart: Detected an Intel i815 Chipset. agpgart: Maximum main memory to use for agp memory: 439M agpgart: AGP aperture is 64M @ 0xf8000000 serio: i8042 AUX port at 0x60,0x64 irq 12 serio: i8042 KBD port at 0x60,0x64 irq 1 Serial: 8250/16550 driver $Revision: 1.90 $ 8 ports, IRQ sharing enabled ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize divert: not allocating divert_blk for non-ethernet device lo Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx ICH2: IDE controller at PCI slot 0000:00:1f.1 ICH2: chipset revision 17 ICH2: not 100% native mode: will probe irqs later ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:DMA ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:DMA Probing IDE interface ide0... hda: QUANTUM FIREBALLP AS40.0, ATA DISK drive hdb: QUANTUM FIREBALLP AS40.0, ATA DISK drive Using anticipatory io scheduler ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 Probing IDE interface ide1... hdc: IOMEGA ZIP 250 ATAPI, ATAPI FLOPPY drive hdd: SONY CD-RW CRX160E, ATAPI CD/DVD-ROM drive ide1 at 0x170-0x177,0x376 on irq 15 Probing IDE interface ide2... ide2: Wait for ready failed before probe ! Probing IDE interface ide3... ide3: Wait for ready failed before probe ! Probing IDE interface ide4... ide4: Wait for ready failed before probe ! Probing IDE interface ide5... ide5: Wait for ready failed before probe ! hda: max request size: 128KiB hda: 78177792 sectors (40027 MB) w/1902KiB Cache, CHS=65535/16/63, UDMA(100) hda: cache flushes not supported hda: hda1 hda2 hda3 hda4 < hda5 hda6 hda7 hda8 hda9 > hdb: max request size: 128KiB hdb: 78177792 sectors (40027 MB) w/1902KiB Cache, CHS=65535/16/63, UDMA(100) hdb: cache flushes not supported hdb: hdb1 hdd: ATAPI 32X CD-ROM CD-R/RW drive, 4096kB Cache, UDMA(33) Uniform CD-ROM driver Revision: 3.20 ide-floppy driver 0.99.newide hdc: No disk in drive hdc: 0kB, 0/64/32 CHS, 4096 kBps, 512 sector size, 2941 rpm usbcore: registered new driver hiddev usbcore: registered new driver usbhid drivers/usb/input/hid-core.c: v2.0:USB HID core driver mice: PS/2 mouse device common for all mice input: AT Translated Set 2 keyboard on isa0060/serio0 input: ImExPS/2 Generic Explorer Mouse on isa0060/serio1 md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27 NET: Registered protocol family 2 IP: routing cache hash table of 1024 buckets, 32Kbytes TCP: Hash tables configured (established 32768 bind 9362) Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 17 ACPI: (supports S0 S1 S4 S5) ACPI wakeup devices: SBTN PCI1 UAR1 USB USB2 AC9 SMB Freeing unused kernel memory: 168k freed SCSI subsystem initialized ACPI: PCI interrupt 0000:01:0a.0[A] -> GSI 9 (level, low) -> IRQ 9 scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36 aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs (scsi0:A:0): 20.000MB/s transfers (20.000MHz, offset 15) Vendor: SEAGATE Model: ST173404LW Rev: 0003 Type: Direct-Access ANSI SCSI revision: 03 scsi0:A:0:0: Tagged Queuing enabled. Depth 4 SCSI device sda: 143374738 512-byte hdwr sectors (73408 MB) SCSI device sda: drive cache: write through sda: Attached scsi disk sda at scsi0, channel 0, id 0, lun 0 (scsi0:A:1): 40.000MB/s transfers (20.000MHz, offset 8, 16bit) Vendor: SEAGATE Model: ST173404LW Rev: 0003 Type: Direct-Access ANSI SCSI revision: 03 scsi0:A:1:0: Tagged Queuing enabled. Depth 4 SCSI device sdb: 143374738 512-byte hdwr sectors (73408 MB) SCSI device sdb: drive cache: write through sdb: Attached scsi disk sdb at scsi0, channel 0, id 1, lun 0 device-mapper: 4.1.0-ioctl (2003-12-10) initialised: dm at uk.sistina.com kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. SELinux: Disabled at runtime. SELinux: Unregistering netfilter hooks inserting floppy driver for 2.6.9 Floppy drive(s): fd0 is 1.44M FDC 0 is a post-1991 82077 e100: Intel(R) PRO/100 Network Driver, 3.0.27-k2-NAPI e100: Copyright(c) 1999-2004 Intel Corporation ACPI: PCI interrupt 0000:01:08.0[A] -> GSI 11 (level, low) -> IRQ 11 divert: allocating divert_blk for eth0 e100: eth0: e100_probe: addr 0xfc9fe000, irq 11, MAC addr 00:03:47:A0:AC:36 ACPI: PCI interrupt 0000:00:1f.5[B] -> GSI 10 (level, low) -> IRQ 10 PCI: Setting latency timer of device 0000:00:1f.5 to 64 intel8x0_measure_ac97_clock: measured 49927 usecs intel8x0: clocking to 41142 USB Universal Host Controller Interface driver v2.2 ACPI: PCI interrupt 0000:00:1f.2[D] -> GSI 11 (level, low) -> IRQ 11 uhci_hcd 0000:00:1f.2: UHCI Host Controller PCI: Setting latency timer of device 0000:00:1f.2 to 64 uhci_hcd 0000:00:1f.2: irq 11, io base 0000ef40 uhci_hcd 0000:00:1f.2: new USB bus registered, assigned bus number 1 hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected ACPI: PCI interrupt 0000:00:1f.4[C] -> GSI 11 (level, low) -> IRQ 11 uhci_hcd 0000:00:1f.4: UHCI Host Controller PCI: Setting latency timer of device 0000:00:1f.4 to 64 uhci_hcd 0000:00:1f.4: irq 11, io base 0000ef80 uhci_hcd 0000:00:1f.4: new USB bus registered, assigned bus number 2 hub 2-0:1.0: USB hub found hub 2-0:1.0: 2 ports detected md: Autodetecting RAID arrays. md: autorun ... md: ... autorun DONE. usb 2-2: new full speed USB device using address 2 hub 2-2:1.0: USB hub found hub 2-2:1.0: 4 ports detected ACPI: Power Button (FF) [PWRF] ACPI: Sleep Button (CM) [SBTN] EXT3 FS on hda3, internal journal ide-floppy: hdc: I/O error, pc = 0, key = 2, asc = 3a, ascq = 0 ide-floppy: hdc: I/O error, pc = 1b, key = 2, asc = 3a, ascq = 0 hdc: No disk in drive cdrom: open failed. kjournald starting. Commit interval 5 seconds EXT3 FS on hda1, internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS on hda5, internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS on hdb1, internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS on hda7, internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS on hda8, internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS on hda6, internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS on hda9, internal journal EXT3-fs: mounted filesystem with ordered data mode. NET: Registered protocol family 10 Disabled Privacy Extensions on device c03dc960(lo) IPv6 over IPv4 tunneling driver divert: not allocating divert_blk for non-ethernet device sit0 Adding 2047744k swap on /dev/hda2. Priority:-1 extents:1 parport0: PC-style at 0x378 [PCSPP,TRISTATE,EPP] ip_tables: (C) 2000-2002 Netfilter core team ip_conntrack version 2.1 (4094 buckets, 32752 max) - 356 bytes per conntrack e100: eth0: e100_watchdog: link up, 100Mbps, full-duplex i2c /dev entries driver parport0: PC-style at 0x378 [PCSPP,TRISTATE,EPP] lp0: using parport0 (polling). lp0: console ready eth0: no IPv6 routers present Installing knfsd (copyright (C) 1996 okir at monad.swb.de). Lock_Harness (built Dec 3 2004 14:18:33) installed gfs: no version for "lm_mount" found: kernel tainted. gfs: Unknown symbol panic_on_oops CMAN (built Dec 3 2004 14:17:50) installed NET: Registered protocol family 30 DLM (built Dec 3 2004 14:18:20) installed Lock_DLM (built Dec 3 2004 14:18:43) installed From mtilstra at redhat.com Fri Dec 3 23:06:29 2004 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Fri, 3 Dec 2004 17:06:29 -0600 Subject: [Linux-cluster] GFS no version for lm_mount foudn: kernel tainted In-Reply-To: <1102114075.6916.8.camel@merlin.Mines.EDU> References: <1102114075.6916.8.camel@merlin.Mines.EDU> Message-ID: <20041203230629.GA3941@redhat.com> On Fri, Dec 03, 2004 at 03:47:55PM -0700, Matthew B. Brookover wrote: > I am trying to set up GFS on a 2 node cluster running Fedora core 3. > The GFS kernel module is generating this error: "FATAL: Error inserting > gfs (/lib/modules/2.6.9/kernel/fs/gfs/gfs.ko): Unknown symbol in > module, or unknown parameter (see dmesg)". I am also getting a "gfs: > no version for "lm_mount" found: kernel tainted." and "gfs: Unknown > symbol panic_on_oops" errors in the logs. The lock_dlm, dlm, cman and > lock_harness modules do load, but the gfs module does not. > > Any ideas? new patch in cluster/gfs-kernel/patches/2.6.9/00006.patch that exports the panic_on_oops symbol. Need to patch your kernel, then build teh gfs module again. -- Michael Conrad Tadpol Tilstra Like a cool breeze through a freshly napalmed forest. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From daniel at osdl.org Fri Dec 3 23:08:00 2004 From: daniel at osdl.org (Daniel McNeil) Date: Fri, 03 Dec 2004 15:08:00 -0800 Subject: [Linux-cluster] GFS 2 node hang in rm test Message-ID: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> I ran my test script (http://developer.osdl.org/daniel/gfs_tests/test.sh) overnight. It ran 17 test runs before hanging in a rm during a 2 node test. The /gfs_stripe5 is mounted on cl030 and cl031. process 28723 (rm) on cl030 is hung. process 29693 (updatedb) is also hung on cl030. process 29537 (updatedb) is hung on cl031. I have stack traces and lockdump and lock debug output from both nodes here: http://developer.osdl.org/daniel/GFS/gfs_2node_rm_hang/ gfs_tool/decipher_lockstate_dump cl030.lockdump shows: Glock (inode[2], 39860) gl_flags = gl_count = 6 gl_state = shared[3] lvb_count = 0 object = yes aspace = 2 reclaim = no Holder owner = 28723 gh_state = shared[3] gh_flags = atime[9] error = 0 gh_iflags = promote[1] holder[6] first[7] Waiter2 owner = none[-1] gh_state = unlocked[0] gh_flags = try[0] error = 0 gh_iflags = demote[2] alloced[4] dealloc[5] Waiter3 owner = 29693 gh_state = shared[3] gh_flags = any[3] error = 0 gh_iflags = promote[1] Inode: busy gfs_tool/decipher_lockstate_dump cl031.lockdump shows: Glock (inode[2], 39860) gl_flags = lock[1] gl_count = 5 gl_state = shared[3] lvb_count = 0 object = yes aspace = 1 reclaim = no Request owner = 29537 gh_state = exclusive[1] gh_flags = local_excl[5] atime[9] error = 0 gh_iflags = promote[1] Waiter3 owner = 29537 gh_state = exclusive[1] gh_flags = local_excl[5] atime[9] error = 0 gh_iflags = promote[1] Inode: busy Is there any documentation on what these fields are? What is the difference between Waiter2 and Waiter3? If I understand this correctly, the updatedb (29537) on cl031 is trying to go from shared -> exclusive while the rm (28723) on cl030 is holding the glock shared and the updatedb (29693) on cl030 is waiting to get the glock shared. Questions: How does one know which node is the master for a lock? Shouldn't the cl030 know (bast) that the updatedb on cl031 is trying to go shared->exclusive? What does the gfs_tool/parse_lockdump script do? I have include the output from /proc/cluster/lock_dlm/debug, but I have no idea what that data is. Any hints? Anything else I can do to debug this further? Thanks, Daniel From daniel at osdl.org Sat Dec 4 00:36:31 2004 From: daniel at osdl.org (Daniel McNeil) Date: Fri, 03 Dec 2004 16:36:31 -0800 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> Message-ID: <1102120591.2706.53.camel@ibm-c.pdx.osdl.net> On Fri, 2004-12-03 at 15:08, Daniel McNeil wrote: > I ran my test script > (http://developer.osdl.org/daniel/gfs_tests/test.sh) overnight. > > It ran 17 test runs before hanging in a rm during a 2 node test. > The /gfs_stripe5 is mounted on cl030 and cl031. > > process 28723 (rm) on cl030 is hung. > process 29693 (updatedb) is also hung on cl030. > > process 29537 (updatedb) is hung on cl031. > > I have stack traces and lockdump and lock debug output > from both nodes here: > > http://developer.osdl.org/daniel/GFS/gfs_2node_rm_hang/ > > > gfs_tool/decipher_lockstate_dump cl030.lockdump shows: > > Glock (inode[2], 39860) > gl_flags = > gl_count = 6 > gl_state = shared[3] > lvb_count = 0 > object = yes > aspace = 2 > reclaim = no > Holder > owner = 28723 > gh_state = shared[3] > gh_flags = atime[9] > error = 0 > gh_iflags = promote[1] holder[6] first[7] > Waiter2 > owner = none[-1] > gh_state = unlocked[0] > gh_flags = try[0] > error = 0 > gh_iflags = demote[2] alloced[4] dealloc[5] > Waiter3 > owner = 29693 > gh_state = shared[3] > gh_flags = any[3] > error = 0 > gh_iflags = promote[1] > Inode: busy > > gfs_tool/decipher_lockstate_dump cl031.lockdump shows: > > Glock (inode[2], 39860) > gl_flags = lock[1] > gl_count = 5 > gl_state = shared[3] > lvb_count = 0 > object = yes > aspace = 1 > reclaim = no > Request > owner = 29537 > gh_state = exclusive[1] > gh_flags = local_excl[5] atime[9] > error = 0 > gh_iflags = promote[1] > Waiter3 > owner = 29537 > gh_state = exclusive[1] > gh_flags = local_excl[5] atime[9] > error = 0 > gh_iflags = promote[1] > Inode: busy > > Is there any documentation on what these fields are? > > What is the difference between Waiter2 and Waiter3? > > If I understand this correctly, the updatedb (29537) on > cl031 is trying to go from shared -> exclusive while the > rm (28723) on cl030 is holding the glock shared and the > updatedb (29693) on cl030 is waiting to get the glock shared. > Looking at the stack traces, what I said above does not makes sense. So now I am really confused. updatedb should only need the glock shared since it is only doing a readdir. But the stack trace on the rm cl030 shows that it is in readdir as well. So what does the Request, gh_state = exclusive mean? Still looks like it is trying to go exclusive, but I cannot tell why. Thanks for any help, Daniel From daniel at osdl.org Sat Dec 4 00:58:12 2004 From: daniel at osdl.org (Daniel McNeil) Date: Fri, 03 Dec 2004 16:58:12 -0800 Subject: [Linux-cluster] hung cluster question Message-ID: <1102121892.2706.67.camel@ibm-c.pdx.osdl.net> Another thing I am a bit confused by. After hitting the rm hang describe before, I expected that reset one of the nodes of the cluster would clear up the problem since recovery should clean up the DLM lock state. So I reset cl031. cl030 still had the gfs file system mounted and cl032 was a member of the cluster, but did not a gfs file system mounted. When I reset cl031, both other nodes printed CMAN: no HELLO from cl031a, removing from the cluster Since I had configured manual fencing, I expected that I would see a message on one of the nodes saying I needed to ack the fencing, but I never saw any message. After that running, cat /proc/cluster/services hung. I reset cl031 and cl032 got: CMAN: no HELLO from cl030a, removing from the cluster CMAN: quorum lost, blocking activity SM: 00000001 process_recovery_barrier status=-104 Does the SM: message mean anything. After rebooting the other 2 nodes, they rejoined the cluster ok, but there were message vi /var/log/messages : Dec 3 16:53:44 cl032 fenced[17168]: fencing node "cl030a" Dec 3 16:53:44 cl032 fenced[17168]: fence "cl030a" failed So I'm not sure my manual fencing is working correctly. Any suggestions? Thanks, Daniel From teigland at redhat.com Sat Dec 4 05:56:54 2004 From: teigland at redhat.com (David Teigland) Date: Sat, 4 Dec 2004 13:56:54 +0800 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> Message-ID: <20041204055654.GA8271@redhat.com> On Fri, Dec 03, 2004 at 03:08:00PM -0800, Daniel McNeil wrote: > I ran my test script > (http://developer.osdl.org/daniel/gfs_tests/test.sh) overnight. > > It ran 17 test runs before hanging in a rm during a 2 node test. > The /gfs_stripe5 is mounted on cl030 and cl031. > > process 28723 (rm) on cl030 is hung. > process 29693 (updatedb) is also hung on cl030. > > process 29537 (updatedb) is hung on cl031. > > I have stack traces and lockdump and lock debug output > from both nodes here: > > http://developer.osdl.org/daniel/GFS/gfs_2node_rm_hang/ There's evidently atime updates happening. That's not necessarily a killer, but you might check the system times on your nodes to verify they're the same (or were the same when the tests were running). If they're different enough, then atime updates could take a portion of the blame. > How does one know which node is the master for a lock? echo "name of lockspace" >> /proc/cluster/dlm_locks cat /proc/cluster/dlm_locks > dlm_locks.txt gives a list of all the dlm locks that node knows about. > I have include the output from /proc/cluster/lock_dlm/debug, > but I have no idea what that data is. Any hints? It's a small circular buffer of lock_dlm activity. It can be helpful debugging some problems, but usually not unless you're looking for something specific and recent. -- Dave Teigland From kpreslan at redhat.com Mon Dec 6 19:45:21 2004 From: kpreslan at redhat.com (Ken Preslan) Date: Mon, 6 Dec 2004 13:45:21 -0600 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> Message-ID: <20041206194521.GA6120@potassium.msp.redhat.com> On Fri, Dec 03, 2004 at 03:08:00PM -0800, Daniel McNeil wrote: > Is there any documentation on what these fields are? Ben Cahill is doing a good job adding comments. So, you might want to look in cluster/gfs-kernels/src/gfs/incore.h and cluster/gfs-kernels/src/gfs/glock.h. > What is the difference between Waiter2 and Waiter3? The waiter2 list is for requests to drop the glock. The waiter3 list is for processes waiting to acquire the glock. > Shouldn't the cl030 know (bast) that the updatedb on cl031 > is trying to go shared->exclusive? It does, that's why there is something on the waiter2 list. > What does the gfs_tool/parse_lockdump script do? parse_lockdump takes the output of decipher_lockstate_dump from multiple machines and looks for groups of locks and processes that might be deadlocking together. The interesting one it found for your situation is: 2 chain: 30.ld Glock (inode[2], 39860) gl_flags = gl_count = 6 gl_state = shared[3] lvb_count = 0 object = yes aspace = 2 reclaim = no Holder owner = 28723 gh_state = shared[3] gh_flags = atime[9] error = 0 gh_iflags = promote[1] holder[6] first[7] Waiter2 owner = none[-1] gh_state = unlocked[0] gh_flags = try[0] error = 0 gh_iflags = demote[2] alloced[4] dealloc[5] Waiter3 owner = 29693 gh_state = shared[3] gh_flags = any[3] error = 0 gh_iflags = promote[1] Inode: busy 31.ld Glock (inode[2], 39860) gl_flags = lock[1] gl_count = 5 gl_state = shared[3] lvb_count = 0 object = yes aspace = 1 reclaim = no Request owner = 29537 gh_state = exclusive[1] gh_flags = local_excl[5] atime[9] error = 0 gh_iflags = promote[1] Waiter3 owner = 29537 gh_state = exclusive[1] gh_flags = local_excl[5] atime[9] error = 0 gh_iflags = promote[1] Inode: busy 1 requests It shows machine 31 is asking for the lock exclusively to perform an atime update. Machine 30 has received the callback asking for the lock to be released, but it can't because process 28723 is actively using the lock. Looking at the backtrace for process 28723, we see: rm D 00000008 0 28723 28711 (NOTLB) caa0ab60 00000082 caa0ab50 00000008 00000001 00000000 00000008 00000000 c1addb80 00000246 00002f57 caa0ab6c c0105d5e ddb18f90 00000000 caa0ab48 c170df60 00000000 00003c53 44ddcfe7 00002f57 ea82cd70 ea82ced0 f8b80260 Call Trace: [] rwsem_down_write_failed+0x9c/0x18e [] .text.lock.locking+0x72/0x1c9 [dlm] [] dlm_lock+0x319/0x3b0 [dlm] [] do_dlm_lock+0x11a/0x200 [lock_dlm] [] lm_dlm_lock+0x6c/0x80 [lock_dlm] [] gfs_glock_xmote_th+0xc0/0x200 [gfs] [] gfs_glock_prefetch+0xf9/0x1e0 [gfs] [] gfs_glock_prefetch_num+0x6a/0xc0 [gfs] [] filldir_reg_func+0xbb/0x180 [gfs] [] do_filldir_main+0x1b4/0x240 [gfs] [] do_filldir_single+0xc8/0x110 [gfs] [] dir_e_read+0x31f/0x360 [gfs] [] gfs_dir_read+0x3c/0x90 [gfs] [] readdir_reg+0xa5/0xc0 [gfs] [] gfs_readdir+0x6b/0x90 [gfs] [] vfs_readdir+0x9b/0xc0 [] sys_getdents64+0x70/0xaf [] sysenter_past_esp+0x52/0x71 So, inode 39860 is a directory. That process acquired the lock on the directory to perform a readdir. While it does the readdir, it sends out prefetch requests for the locks in the directory. Those requests are asynchronous tries, so they shouldn't ever block for long. The prefetch is for this glock: 1 chain: 30.ld Glock (inode[2], 39883) gl_flags = lock[1] prefetch[3] gl_count = 2 gl_state = unlocked[0] lvb_count = 0 object = no aspace = 0 reclaim = no 0 requests So, the question is: Why is the DLM stuck on that request? -- Ken Preslan From daniel at osdl.org Tue Dec 7 00:13:50 2004 From: daniel at osdl.org (Daniel McNeil) Date: Mon, 06 Dec 2004 16:13:50 -0800 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <20041206194521.GA6120@potassium.msp.redhat.com> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <20041206194521.GA6120@potassium.msp.redhat.com> Message-ID: <1102378430.2706.83.camel@ibm-c.pdx.osdl.net> On Mon, 2004-12-06 at 11:45, Ken Preslan wrote: > On Fri, Dec 03, 2004 at 03:08:00PM -0800, Daniel McNeil wrote: > > Is there any documentation on what these fields are? > > Ben Cahill is doing a good job adding comments. So, you might want > to look in cluster/gfs-kernels/src/gfs/incore.h and > cluster/gfs-kernels/src/gfs/glock.h. > > > What is the difference between Waiter2 and Waiter3? > > The waiter2 list is for requests to drop the glock. The waiter3 list > is for processes waiting to acquire the glock. > > > Shouldn't the cl030 know (bast) that the updatedb on cl031 > > is trying to go shared->exclusive? > > It does, that's why there is something on the waiter2 list. > > > What does the gfs_tool/parse_lockdump script do? > > parse_lockdump takes the output of decipher_lockstate_dump from multiple > machines and looks for groups of locks and processes that might be > deadlocking together. The interesting one it found for your situation is: > > 2 chain: > 30.ld Glock (inode[2], 39860) > gl_flags = > gl_count = 6 > gl_state = shared[3] > lvb_count = 0 > object = yes > aspace = 2 > reclaim = no > Holder > owner = 28723 > gh_state = shared[3] > gh_flags = atime[9] > error = 0 > gh_iflags = promote[1] holder[6] first[7] > Waiter2 > owner = none[-1] > gh_state = unlocked[0] > gh_flags = try[0] > error = 0 > gh_iflags = demote[2] alloced[4] dealloc[5] > Waiter3 > owner = 29693 > gh_state = shared[3] > gh_flags = any[3] > error = 0 > gh_iflags = promote[1] > Inode: busy > 31.ld Glock (inode[2], 39860) > gl_flags = lock[1] > gl_count = 5 > gl_state = shared[3] > lvb_count = 0 > object = yes > aspace = 1 > reclaim = no > Request > owner = 29537 > gh_state = exclusive[1] > gh_flags = local_excl[5] atime[9] > error = 0 > gh_iflags = promote[1] > Waiter3 > owner = 29537 > gh_state = exclusive[1] > gh_flags = local_excl[5] atime[9] > error = 0 > gh_iflags = promote[1] > Inode: busy > 1 requests > > It shows machine 31 is asking for the lock exclusively to perform an > atime update. Machine 30 has received the callback asking for the > lock to be released, but it can't because process 28723 is actively > using the lock. > > Looking at the backtrace for process 28723, we see: > > rm D 00000008 0 28723 28711 (NOTLB) > caa0ab60 00000082 caa0ab50 00000008 00000001 00000000 00000008 00000000 > c1addb80 00000246 00002f57 caa0ab6c c0105d5e ddb18f90 00000000 caa0ab48 > c170df60 00000000 00003c53 44ddcfe7 00002f57 ea82cd70 ea82ced0 f8b80260 > Call Trace: > [] rwsem_down_write_failed+0x9c/0x18e > [] .text.lock.locking+0x72/0x1c9 [dlm] > [] dlm_lock+0x319/0x3b0 [dlm] > [] do_dlm_lock+0x11a/0x200 [lock_dlm] > [] lm_dlm_lock+0x6c/0x80 [lock_dlm] > [] gfs_glock_xmote_th+0xc0/0x200 [gfs] > [] gfs_glock_prefetch+0xf9/0x1e0 [gfs] > [] gfs_glock_prefetch_num+0x6a/0xc0 [gfs] > [] filldir_reg_func+0xbb/0x180 [gfs] > [] do_filldir_main+0x1b4/0x240 [gfs] > [] do_filldir_single+0xc8/0x110 [gfs] > [] dir_e_read+0x31f/0x360 [gfs] > [] gfs_dir_read+0x3c/0x90 [gfs] > [] readdir_reg+0xa5/0xc0 [gfs] > [] gfs_readdir+0x6b/0x90 [gfs] > [] vfs_readdir+0x9b/0xc0 > [] sys_getdents64+0x70/0xaf > [] sysenter_past_esp+0x52/0x71 > > So, inode 39860 is a directory. That process acquired the lock on the > directory to perform a readdir. While it does the readdir, it sends out > prefetch requests for the locks in the directory. Those requests > are asynchronous tries, so they shouldn't ever block for long. > The prefetch is for this glock: > > 1 chain: > 30.ld Glock (inode[2], 39883) > gl_flags = lock[1] prefetch[3] > gl_count = 2 > gl_state = unlocked[0] > lvb_count = 0 > object = no > aspace = 0 > reclaim = no > 0 requests > > So, the question is: Why is the DLM stuck on that request? Looking at the stack trace above and dissabling dlm.ko, it looks like dlm_lock+0x319 is the call to dlm_lock_stage1(). looking at dlm_lock_stage1(), it looks like it is sleeping on down_write(&rsb->res_lock) So now I have to find who is holding the res_lock. Any ideas on how to debug this? It is reproducible. My test script was on 1st 17th time through the test loop when it hit this. I hit it again over the weekend in only 4 times through the loop, but this time on a 3 node test. Unfortunately, when I got the stack trace on the 3rd node, the other nodes thought it died and kicked it out of the cluster. The other nodes got stuck spinning in the kernel along with 70 other 'df' processes that cron was periodically firing off. I do more poking around on this one and see if I can get more info out on the next hang also. Daniel From dmorgan at gmi-mr.com Tue Dec 7 02:52:02 2004 From: dmorgan at gmi-mr.com (Duncan Morgan) Date: Mon, 6 Dec 2004 18:52:02 -0800 Subject: [Linux-cluster] D Apache processes Message-ID: <000c01c4dc07$bb05d560$6701a8c0@DMorganMobile> Hello, We have apache running on 14 GFS nodes where the web roots are shared via GFS. Occasionally we see that the load on all nodes rises dramatically (to 150+) and all httpd processes become dead (D). I know this is a little lacking in details but does anyone have any insight into this? We suspected perhaps a cron job was running simultaneously against the GFS file system on all nodes but have virtually ruled this out. Please help - this is very alarming. Thanks in advance, Duncan Morgan From Axel.Thimm at ATrpms.net Tue Dec 7 03:14:57 2004 From: Axel.Thimm at ATrpms.net (Axel Thimm) Date: Tue, 7 Dec 2004 04:14:57 +0100 Subject: [Linux-cluster] Re: D Apache processes In-Reply-To: <000c01c4dc07$bb05d560$6701a8c0@DMorganMobile> References: <000c01c4dc07$bb05d560$6701a8c0@DMorganMobile> Message-ID: <20041207031457.GA28697@neu.nirvana> On Mon, Dec 06, 2004 at 06:52:02PM -0800, Duncan Morgan wrote: > Hello, > > We have apache running on 14 GFS nodes where the web roots are shared > via GFS. Occasionally we see that the load on all nodes rises > dramatically (to 150+) and all httpd processes become dead (D). I know > this is a little lacking in details but does anyone have any insight > into this? We suspected perhaps a cron job was running simultaneously > against the GFS file system on all nodes but have virtually ruled this > out. I am getting the same behaviour on a non-clustered, non GFS system (dual opteron on FC2/x86_64). There is a peak of almost all httpd processes in D-state (that's not really dead, but "uninterruptible sleep", e.g. when the kernel does IO for the userland process). A few seconds later the number of D-processes fall down to less than a dozen and you can watch your load exponentially decrease. Note that 150 is the default MaxClient setting for apache, that's why you get slightly more than 150 load. I guess that's a kernel issue with too many processes accessing the same files. Nothing to do directly with GFS. > Please help - this is very alarming. > > Thanks in advance, > Duncan Morgan > -- Axel.Thimm at ATrpms.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From dmorgan at gmi-mr.com Tue Dec 7 03:25:32 2004 From: dmorgan at gmi-mr.com (Duncan Morgan) Date: Mon, 6 Dec 2004 19:25:32 -0800 Subject: [Linux-cluster] Re: D Apache processes In-Reply-To: <20041207031457.GA28697@neu.nirvana> Message-ID: <001601c4dc0c$68bf6af0$6701a8c0@DMorganMobile> Unfortunately we don't see the load decrease at all and have to reboot all servers in the cluster - causing mass chaos :)) so I'm not sure we are talking about the same issue. Thanks for your input. Duncan -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Axel Thimm Sent: Monday, December 06, 2004 7:15 PM To: linux clistering Subject: [Linux-cluster] Re: D Apache processes On Mon, Dec 06, 2004 at 06:52:02PM -0800, Duncan Morgan wrote: > Hello, > > We have apache running on 14 GFS nodes where the web roots are shared > via GFS. Occasionally we see that the load on all nodes rises > dramatically (to 150+) and all httpd processes become dead (D). I know > this is a little lacking in details but does anyone have any insight > into this? We suspected perhaps a cron job was running simultaneously > against the GFS file system on all nodes but have virtually ruled this > out. I am getting the same behaviour on a non-clustered, non GFS system (dual opteron on FC2/x86_64). There is a peak of almost all httpd processes in D-state (that's not really dead, but "uninterruptible sleep", e.g. when the kernel does IO for the userland process). A few seconds later the number of D-processes fall down to less than a dozen and you can watch your load exponentially decrease. Note that 150 is the default MaxClient setting for apache, that's why you get slightly more than 150 load. I guess that's a kernel issue with too many processes accessing the same files. Nothing to do directly with GFS. > Please help - this is very alarming. > > Thanks in advance, > Duncan Morgan > -- Axel.Thimm at ATrpms.net From Axel.Thimm at ATrpms.net Tue Dec 7 03:36:03 2004 From: Axel.Thimm at ATrpms.net (Axel Thimm) Date: Tue, 7 Dec 2004 04:36:03 +0100 Subject: [Linux-cluster] Re: D Apache processes In-Reply-To: <001601c4dc0c$68bf6af0$6701a8c0@DMorganMobile> References: <20041207031457.GA28697@neu.nirvana> <001601c4dc0c$68bf6af0$6701a8c0@DMorganMobile> Message-ID: <20041207033603.GA31169@neu.nirvana> On Mon, Dec 06, 2004 at 07:25:32PM -0800, Duncan Morgan wrote: > Unfortunately we don't see the load decrease at all and have to reboot > all servers in the cluster - causing mass chaos :)) so I'm not sure we > are talking about the same issue. Then you have enough time to examine the problem more thoroughly. Use lsof to check which resources are accessed by the processes in D-state. You will probably find that a larger share is accessing the same files. > Thanks for your input. > > Duncan > > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Axel Thimm > Sent: Monday, December 06, 2004 7:15 PM > To: linux clistering > Subject: [Linux-cluster] Re: D Apache processes > > On Mon, Dec 06, 2004 at 06:52:02PM -0800, Duncan Morgan wrote: > > Hello, > > > > We have apache running on 14 GFS nodes where the web roots are shared > > via GFS. Occasionally we see that the load on all nodes rises > > dramatically (to 150+) and all httpd processes become dead (D). I know > > this is a little lacking in details but does anyone have any insight > > into this? We suspected perhaps a cron job was running simultaneously > > against the GFS file system on all nodes but have virtually ruled this > > out. > > I am getting the same behaviour on a non-clustered, non GFS system > (dual opteron on FC2/x86_64). There is a peak of almost all httpd > processes in D-state (that's not really dead, but "uninterruptible > sleep", e.g. when the kernel does IO for the userland process). A few > seconds later the number of D-processes fall down to less than a dozen > and you can watch your load exponentially decrease. > > Note that 150 is the default MaxClient setting for apache, that's why > you get slightly more than 150 load. > > I guess that's a kernel issue with too many processes accessing the > same files. Nothing to do directly with GFS. > > > Please help - this is very alarming. > > > > Thanks in advance, > > Duncan Morgan > > > -- Axel.Thimm at ATrpms.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From crh at ubiqx.mn.org Tue Dec 7 05:56:21 2004 From: crh at ubiqx.mn.org (Christopher R. Hertel) Date: Mon, 6 Dec 2004 23:56:21 -0600 Subject: [Linux-cluster] Samba Technical thread Message-ID: <20041207055621.GG23207@Favog.ubiqx.mn.org> There's a thread started on the Samba-Technical mailing list that has some discussion regarding cluster filesystems. I'm learning fast, but I'm not the right one to answer this: http://lists.samba.org/archive/samba-technical/2004-December/038326.html An excerpt: The other network-like filesystems - Lustre, SANFS, GPFS, and RedHat's GFS do differ a little.. They differ in that they would attempt stricter posix semantics and therefore view themselves as "cluster" rather than "network" filesystems (an odd distinction ... why shouldn't a network filesystem simply consider "cluster" in effect a mount option which would optimize for higher performance to nearby hosts in the cluster and stricter POSIX file semantics rather than relaxed "nfs file semantics"). If they had a good standards story with the IETF and were inkernel in 2.6, perhaps no one would care, but it seems odd - when you can make AFS or CIFS or NFSv4 do the same with rather more trivial changes. Somehow I think that the above doesn't quite capture what GFS is all about. I'm not trying to start a flamewar, but I'd certainly like to see someone provide a clearer explanation than I could do. Chris -)----- -- "Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X Samba Team -- http://www.samba.org/ -)----- Christopher R. Hertel jCIFS Team -- http://jcifs.samba.org/ -)----- ubiqx development, uninq. ubiqx Team -- http://www.ubiqx.org/ -)----- crh at ubiqx.mn.org OnLineBook -- http://ubiqx.org/cifs/ -)----- crh at ubiqx.org From teigland at redhat.com Tue Dec 7 07:29:32 2004 From: teigland at redhat.com (David Teigland) Date: Tue, 7 Dec 2004 15:29:32 +0800 Subject: [Linux-cluster] Samba Technical thread In-Reply-To: <20041207055621.GG23207@Favog.ubiqx.mn.org> References: <20041207055621.GG23207@Favog.ubiqx.mn.org> Message-ID: <20041207072932.GA11901@redhat.com> On Mon, Dec 06, 2004 at 11:56:21PM -0600, Christopher R. Hertel wrote: > There's a thread started on the Samba-Technical mailing list that has some > discussion regarding cluster filesystems. I'm learning fast, but I'm not > the right one to answer this: > > http://lists.samba.org/archive/samba-technical/2004-December/038326.html > > An excerpt: > > The other network-like filesystems - Lustre, SANFS, GPFS, and RedHat's > GFS do differ a little.. They differ in that they would attempt > stricter posix semantics and therefore view themselves as "cluster" > rather than "network" filesystems (an odd distinction ... why shouldn't > a network filesystem simply consider "cluster" in effect a mount option > which would optimize for higher performance to nearby hosts in the > cluster and stricter POSIX file semantics rather than relaxed "nfs file > semantics"). If they had a good standards story with the IETF and were > inkernel in 2.6, perhaps no one would care, but it seems odd - when you > can make AFS or CIFS or NFSv4 do the same with rather more trivial > changes. > > Somehow I think that the above doesn't quite capture what GFS is all > about. I'm not trying to start a flamewar, but I'd certainly like to see > someone provide a clearer explanation than I could do. There seem to be at least three different things there that can be considered separately: 1. SAN usage CIFS/NFS aren't interested in exploiting SAN access from clients while others like GFS are. 2. server role (symmetric vs asymmetric) GFS aims to be server-less, NFS/CIFS are very server-based, and others can fall somewhere in between. (If you consider using GFS above iscsi or nbd then the differences become even more subtle.) 3. POSIX semantics GFS semantics aim to copy those of a local fs exactly, while others like NFS don't, although there's nothing precluding that (NFS4 can be close if not exact). -- Dave Teigland From crh at ubiqx.mn.org Tue Dec 7 07:51:17 2004 From: crh at ubiqx.mn.org (Christopher R. Hertel) Date: Tue, 7 Dec 2004 01:51:17 -0600 Subject: [Linux-cluster] Samba Technical thread In-Reply-To: <20041207072932.GA11901@redhat.com> References: <20041207055621.GG23207@Favog.ubiqx.mn.org> <20041207072932.GA11901@redhat.com> Message-ID: <20041207075117.GI23207@Favog.ubiqx.mn.org> On Tue, Dec 07, 2004 at 03:29:32PM +0800, David Teigland wrote: : > 3. POSIX semantics > GFS semantics aim to copy those of a local fs exactly, while others > like NFS don't, although there's nothing precluding that (NFS4 can > be close if not exact). Question on this. Is there any interest in clustered block media? Eg., what if five systems all had access to the same chunk of (virtual) disk via iSCSI. Is there a way that one of those five could create an EXT3 filesystem on that raw block-space such that all of the five clustered systems would see an EXT3 partition (and be able to use it concurrently)? Oh... and a side-comment. CIFS tries, and gets fairly close, to copy lovsl FS semantics. The problem is that those semantics are DOS, OS/2, and NTFS semantics. The protocol has support for all of these, and there are some folks who have tried to add Unix semantics as well. Chris -)----- -- "Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X Samba Team -- http://www.samba.org/ -)----- Christopher R. Hertel jCIFS Team -- http://jcifs.samba.org/ -)----- ubiqx development, uninq. ubiqx Team -- http://www.ubiqx.org/ -)----- crh at ubiqx.mn.org OnLineBook -- http://ubiqx.org/cifs/ -)----- crh at ubiqx.org From pcaulfie at redhat.com Tue Dec 7 09:38:10 2004 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 7 Dec 2004 09:38:10 +0000 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <1102378430.2706.83.camel@ibm-c.pdx.osdl.net> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <20041206194521.GA6120@potassium.msp.redhat.com> <1102378430.2706.83.camel@ibm-c.pdx.osdl.net> Message-ID: <20041207093810.GC29001@tykepenguin.com> On Mon, Dec 06, 2004 at 04:13:50PM -0800, Daniel McNeil wrote: > On Mon, 2004-12-06 at 11:45, Ken Preslan wrote: > > On Fri, Dec 03, 2004 at 03:08:00PM -0800, Daniel McNeil wrote: > > > Looking at the stack trace above and dissabling dlm.ko, > it looks like dlm_lock+0x319 is the call to dlm_lock_stage1(). > looking at dlm_lock_stage1(), it looks like it is sleeping on > down_write(&rsb->res_lock) > > So now I have to find who is holding the res_lock. That's consistent with the hang you reported before - in fact it's almost certainly the same thing. My guess is thet there is a dealock on res_lock somewhere . In which case I suspect it's going to be easier to find that one by reading code rather than running tests. res_lock should never be held for any extended period of time, but in your last set of tracebacks there was nothing obviously holding it - so I suspect something is sleeping with it. Patrick From ed.mann at choicepoint.com Mon Dec 6 22:24:06 2004 From: ed.mann at choicepoint.com (Edward Mann) Date: Mon, 06 Dec 2004 16:24:06 -0600 Subject: [Linux-cluster] gfs fam notify Message-ID: <1102371846.16796.15.camel@storm.cp-direct.com> I have searched the news-group but have not found anyone that has asked this question. If it has been and i have missed it please point me in the right direction. I am using FAM on a box that waits for files to be placed in folders. Once the it is notified about a new file it will run a script that i have created. However if the file shows up on another box it is never notified about the new file. ----- ----- - M - link - M - - 1 - ---------- - 2 - ----- ----- So machine 1 (M1) runs the script and the storage is shared between it and Machine 2 (M2). If M2 get's the file than M1 does not know about it and does not run the script. Is there some method that M2 can notify M1 via gfs methods that a file was created/changed? I can just move the script to the other box, i just want to be sure that there is not something that i can set to get this to work the way i want. I know this has something to do with the kernel. if i do a touch on the file on M1 than the script will launch. I hope i have explained this well enough. Thanks for your time. From daniel at osdl.org Tue Dec 7 16:53:14 2004 From: daniel at osdl.org (Daniel McNeil) Date: Tue, 07 Dec 2004 08:53:14 -0800 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <20041207093810.GC29001@tykepenguin.com> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <20041206194521.GA6120@potassium.msp.redhat.com> <1102378430.2706.83.camel@ibm-c.pdx.osdl.net> <20041207093810.GC29001@tykepenguin.com> Message-ID: <1102438394.2706.103.camel@ibm-c.pdx.osdl.net> On Tue, 2004-12-07 at 01:38, Patrick Caulfield wrote: > On Mon, Dec 06, 2004 at 04:13:50PM -0800, Daniel McNeil wrote: > > On Mon, 2004-12-06 at 11:45, Ken Preslan wrote: > > > On Fri, Dec 03, 2004 at 03:08:00PM -0800, Daniel McNeil wrote: > > > > > > Looking at the stack trace above and dissabling dlm.ko, > > it looks like dlm_lock+0x319 is the call to dlm_lock_stage1(). > > looking at dlm_lock_stage1(), it looks like it is sleeping on > > down_write(&rsb->res_lock) > > > > So now I have to find who is holding the res_lock. > > That's consistent with the hang you reported before - in fact it's almost > certainly the same thing. My guess is thet there is a dealock on res_lock > somewhere . In which case I suspect it's going to be easier to find that one by > reading code rather than running tests. res_lock should never be held for any > extended period of time, but in your last set of tracebacks there was nothing > obviously holding it - so I suspect something is sleeping with it. > > I looked through the stack traces and did not see any other processes that might be holding the lock. There were only 3 other processes with stack traces in the dlm module and they do not look like they are holding it. That is confusing. I can think of 3 possibilites: 1. forgetting to up the semaphore somewhere 2. a process spinning in the kernel is holding it 3. freed the structure containing the res_lock. All of these seem unlikely to me. I reviewed the code last evening, the the up's and down's are closed together and nothing looked obviously wrong. I'll think about adding more debug output. I ran it again last night and it ran 27 loops until 7am this morning before hanging. I'm still collecting info from this hang. At least it is reproducible. Daniel From mbrookov at mines.edu Tue Dec 7 18:01:43 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Tue, 07 Dec 2004 11:01:43 -0700 Subject: [Linux-cluster] GFS no version for lm_mount foudn: kernel tainted In-Reply-To: <20041203230629.GA3941@redhat.com> References: <1102114075.6916.8.camel@merlin.Mines.EDU> <20041203230629.GA3941@redhat.com> Message-ID: <1102442503.20577.6.camel@merlin.Mines.EDU> The new patch fixed the lm_mount and unknown symbol problems. There is a new problem now, cman_tool join returns this message: cman_tool: local node name "fouroften" not found in cluster.conf I have double checked my host name, the entry in cluster.conf, /etc/hosts, and DNS and they look ok. /proc/cluster/nodes does not list any hosts. Any ideas? thanks Matt mbrookov at mines.edu On Fri, 2004-12-03 at 16:06, Michael Conrad Tadpol Tilstra wrote: > On Fri, Dec 03, 2004 at 03:47:55PM -0700, Matthew B. Brookover wrote: > > I am trying to set up GFS on a 2 node cluster running Fedora core 3. > > The GFS kernel module is generating this error: "FATAL: Error inserting > > gfs (/lib/modules/2.6.9/kernel/fs/gfs/gfs.ko): Unknown symbol in > > module, or unknown parameter (see dmesg)". I am also getting a "gfs: > > no version for "lm_mount" found: kernel tainted." and "gfs: Unknown > > symbol panic_on_oops" errors in the logs. The lock_dlm, dlm, cman and > > lock_harness modules do load, but the gfs module does not. > > > > Any ideas? > > new patch in cluster/gfs-kernel/patches/2.6.9/00006.patch that exports > the panic_on_oops symbol. Need to patch your kernel, then build teh gfs > module again. > From jbrassow at redhat.com Tue Dec 7 19:42:20 2004 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Tue, 7 Dec 2004 13:42:20 -0600 Subject: [Linux-cluster] GFS no version for lm_mount foudn: kernel tainted In-Reply-To: <1102442503.20577.6.camel@merlin.Mines.EDU> References: <1102114075.6916.8.camel@merlin.Mines.EDU> <20041203230629.GA3941@redhat.com> <1102442503.20577.6.camel@merlin.Mines.EDU> Message-ID: <1C325738-4888-11D9-B3C0-000A957BB1F6@redhat.com> if you are using the most current cvs, the tags in /etc/cluster/cluster.conf have changed. https://www.redhat.com/archives/linux-cluster/2004-November/ msg00157.html brassow On Dec 7, 2004, at 12:01 PM, Matthew B. Brookover wrote: > The new patch fixed the lm_mount and unknown symbol problems. There is > a new problem now, cman_tool join returns this message: > cman_tool: local node name "fouroften" not found in cluster.conf > > I have double checked my host name, the entry in cluster.conf, > /etc/hosts, and DNS and they look ok. /proc/cluster/nodes does not > list > any hosts. > > Any ideas? > > thanks > > Matt > mbrookov at mines.edu > > On Fri, 2004-12-03 at 16:06, Michael Conrad Tadpol Tilstra wrote: >> On Fri, Dec 03, 2004 at 03:47:55PM -0700, Matthew B. Brookover wrote: >>> I am trying to set up GFS on a 2 node cluster running Fedora core 3. >>> The GFS kernel module is generating this error: "FATAL: Error >>> inserting >>> gfs (/lib/modules/2.6.9/kernel/fs/gfs/gfs.ko): Unknown symbol in >>> module, or unknown parameter (see dmesg)". I am also getting a "gfs: >>> no version for "lm_mount" found: kernel tainted." and "gfs: Unknown >>> symbol panic_on_oops" errors in the logs. The lock_dlm, dlm, cman >>> and >>> lock_harness modules do load, but the gfs module does not. >>> >>> Any ideas? >> >> new patch in cluster/gfs-kernel/patches/2.6.9/00006.patch that >> exports >> the panic_on_oops symbol. Need to patch your kernel, then build teh >> gfs >> module again. >> > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From jn at it.swin.edu.au Tue Dec 7 22:48:33 2004 From: jn at it.swin.edu.au (John Newbigin) Date: Wed, 08 Dec 2004 09:48:33 +1100 Subject: [Linux-cluster] Samba Technical thread In-Reply-To: <20041207075117.GI23207@Favog.ubiqx.mn.org> References: <20041207055621.GG23207@Favog.ubiqx.mn.org> <20041207072932.GA11901@redhat.com> <20041207075117.GI23207@Favog.ubiqx.mn.org> Message-ID: <41B63341.6080303@it.swin.edu.au> Christopher R. Hertel wrote: > On Tue, Dec 07, 2004 at 03:29:32PM +0800, David Teigland wrote: > : > >>3. POSIX semantics >> GFS semantics aim to copy those of a local fs exactly, while others >> like NFS don't, although there's nothing precluding that (NFS4 can >> be close if not exact). > > > Question on this. Is there any interest in clustered block media? > > Eg., what if five systems all had access to the same chunk of (virtual) > disk via iSCSI. Is there a way that one of those five could create an > EXT3 filesystem on that raw block-space such that all of the five > clustered systems would see an EXT3 partition (and be able to use it > concurrently)? No (well not for a writable filesystem). You still need a cluster filesystem so you don't trash the filesystem. I believe that work is underway for a cluster raid to run gfs on top of. This would be good for people without shared storage hardware but still want to use GFS. > > Oh... and a side-comment. CIFS tries, and gets fairly close, to copy > lovsl FS semantics. The problem is that those semantics are DOS, OS/2, > and NTFS semantics. The protocol has support for all of these, and there > are some folks who have tried to add Unix semantics as well. I did the initial implementation of the Unix semantics in the smbfs driver. POSIX semantics just aren't possible over CIFS. Unlinking an open file for example. I tried for weeks to implement that in smbfs and samba. Windows does not have such a concept so it is very difficult to make it work over CIFS. This means that programs that do stupid NFS locking workarounds (like gconfd) won't run with CIFS mounted home directories. John. > > Chris -)----- > -- John Newbigin Computer Systems Officer Faculty of Information and Communication Technologies Swinburne University of Technology Melbourne, Australia http://www.it.swin.edu.au/staff/jnewbigin From crh at ubiqx.mn.org Tue Dec 7 23:40:20 2004 From: crh at ubiqx.mn.org (Christopher R. Hertel) Date: Tue, 7 Dec 2004 17:40:20 -0600 Subject: [Linux-cluster] Samba Technical thread In-Reply-To: <41B63341.6080303@it.swin.edu.au> References: <20041207055621.GG23207@Favog.ubiqx.mn.org> <20041207072932.GA11901@redhat.com> <20041207075117.GI23207@Favog.ubiqx.mn.org> <41B63341.6080303@it.swin.edu.au> Message-ID: <20041207234020.GG29240@Favog.ubiqx.mn.org> On Wed, Dec 08, 2004 at 09:48:33AM +1100, John Newbigin wrote: > Christopher R. Hertel wrote: : > >Question on this. Is there any interest in clustered block media? > > > >Eg., what if five systems all had access to the same chunk of (virtual) > >disk via iSCSI. Is there a way that one of those five could create an > >EXT3 filesystem on that raw block-space such that all of the five > >clustered systems would see an EXT3 partition (and be able to use it > >concurrently)? > No (well not for a writable filesystem). You still need a cluster > filesystem so you don't trash the filesystem. I believe that work is > underway for a cluster raid to run gfs on top of. This would be good > for people without shared storage hardware but still want to use GFS. I imagine that it would be doable, somehow. :) Possibly not with GFS, but... You would certainly need to have the cluster layer, it would present block storage to the kernel. I'm thinking along the lines of clustered iSCSI. Dunno if that's practical, but I like the mental exersize. > >Oh... and a side-comment. CIFS tries, and gets fairly close, to copy > >local FS semantics. The problem is that those semantics are DOS, OS/2, > >and NTFS semantics. The protocol has support for all of these, and there > >are some folks who have tried to add Unix semantics as well. > I did the initial implementation of the Unix semantics in the smbfs > driver. POSIX semantics just aren't possible over CIFS. Unlinking an > open file for example. I tried for weeks to implement that in smbfs and > samba. Windows does not have such a concept so it is very difficult to > make it work over CIFS. This means that programs that do stupid NFS > locking workarounds (like gconfd) won't run with CIFS mounted home > directories. Well... In the most course-grained fashion you could add a new dialect string to the NegProt and then run an entirely different protocol if both systems agree to do so. That's not what's happening, however. The folks working on this are adding new Infolevels and suchlike over and above the original work by Byron at HP. I don't know how far that will go. Sounds like you've got more experience there. Thanks! Chris -)----- -- "Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X Samba Team -- http://www.samba.org/ -)----- Christopher R. Hertel jCIFS Team -- http://jcifs.samba.org/ -)----- ubiqx development, uninq. ubiqx Team -- http://www.ubiqx.org/ -)----- crh at ubiqx.mn.org OnLineBook -- http://ubiqx.org/cifs/ -)----- crh at ubiqx.org From daniel at osdl.org Wed Dec 8 00:13:12 2004 From: daniel at osdl.org (Daniel McNeil) Date: Tue, 07 Dec 2004 16:13:12 -0800 Subject: [Linux-cluster] GFS 3 node hang in rm test In-Reply-To: <1102438394.2706.103.camel@ibm-c.pdx.osdl.net> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <20041206194521.GA6120@potassium.msp.redhat.com> <1102378430.2706.83.camel@ibm-c.pdx.osdl.net> <20041207093810.GC29001@tykepenguin.com> <1102438394.2706.103.camel@ibm-c.pdx.osdl.net> Message-ID: <1102464791.2706.122.camel@ibm-c.pdx.osdl.net> The latest hang is a 3 node remove hang. I have stack traces, lockdump output from gfs_tool lockdump, and dlm_locks output from all 3 nodes. Except for lockdump output on node cl032 -- it is stuck in: gfs_tool D 00000008 0 20033 2778 (NOTLB) f70c1d90 00000086 f70c1d7c 00000008 00000001 c03d8315 00000008 00000001 d857ddc0 00001000 f70c1d8c c0180832 f689b2d8 e14890d0 00000000 c170e8c0 c170df60 00000000 000975b2 6d2af78e 000044d3 e08b6ef0 e08b7050 00000000 Call Trace: [] wait_for_completion+0xa4/0xe0 [] glock_wait_internal+0x3b/0x270 [gfs] [] gfs_glock_nq+0x86/0x130 [gfs] [] gfs_glock_nq_init+0x34/0x50 [gfs] [] gfs_permission+0x4a/0x90 [gfs] [] permission+0x47/0x50 [] may_open+0x5f/0x220 [] open_namei+0xa7/0x6e0 [] filp_open+0x41/0x70 [] sys_open+0x46/0xa0 [] sysenter_past_esp+0x52/0x71 The problem looks like it is on cl032, but is a little different: dlm_recvd D C170DF98 0 19721 4 19722 19720 (L-TLB) c7a3dd30 00000046 eb6f1450 c170df98 0000399e c5cbd712 00000008 0000399e f5208dc0 c5d11e5d 0000399e c170df98 0000000a eb6f1450 00000000 c170e8c0 c170df60 00000000 00000971 c5d17c58 0000399e d50488b0 d5048a10 00000000 Call Trace: [] rwsem_down_write_failed+0x9c/0x18e [] .text.lock.locking+0xa6/0x1c9 [dlm] [] dlm_lock_stage2+0x60/0xd0 [dlm] [] process_lockqueue_reply+0x3aa/0x770 [dlm] [] process_cluster_request+0x816/0xeb0 [dlm] [] midcomms_process_incoming_buffer+0x167/0x270 [dlm] [] receive_from_sock+0x189/0x2e0 [dlm] [] process_sockets+0x76/0xc0 [dlm] [] dlm_recvd+0x86/0xa0 [dlm] [] kthread+0xba/0xc0 [] kernel_thread_helper+0x5/0x10 There are also a bunch of 'df' processes from cron which are looping forever in the kernel. They are looping in stat_gfs_async(). So the problem is similar, a process stuck on a down_write of a res_lock. I'm assuming that is causing all the other problems. All the info is available here: http://developer.osdl.org/daniel/GFS/rm.hang.07dec2004/ I've include the dlm_debug output also, but I do not know how read the output. I'm planning rebooting with a kernel with more DEBUG options turned on (DEBUG_SLAB) to be sure that it is not accessing freed memory. Any other ideas on debugging? Daniel From phillips at redhat.com Wed Dec 8 06:10:02 2004 From: phillips at redhat.com (Daniel Phillips) Date: Wed, 8 Dec 2004 01:10:02 -0500 Subject: [Linux-cluster] Samba Technical thread In-Reply-To: <20041207055621.GG23207@Favog.ubiqx.mn.org> References: <20041207055621.GG23207@Favog.ubiqx.mn.org> Message-ID: <200412080110.02502.phillips@redhat.com> Hi Chris! On Tuesday 07 December 2004 00:56, Christopher R. Hertel wrote: > There's a thread started on the Samba-Technical mailing list that has > some discussion regarding cluster filesystems. I'm learning fast, > but I'm not the right one to answer this: > > http://lists.samba.org/archive/samba-technical/2004-December/038326.h >tml > > An excerpt: > > The other network-like filesystems - Lustre, SANFS, GPFS, and > RedHat's GFS do differ a little.. They differ in that they would > attempt stricter posix semantics A filesystem either complies with Posix semantics (understood to mean local filesystem semantics) or it doesn't, there is no "stricter". The filesystems listed all do attempt to achieve local filesystem semantics for a filesystem shared by a cluster (and they all rely on shared disk access). > and therefore view themselves as > "cluster" rather than "network" filesystems (an odd distinction ... > why shouldn't a network filesystem simply consider "cluster" in > effect a mount option which would optimize for higher performance to > nearby hosts in the cluster and stricter POSIX file semantics rather > than relaxed "nfs file semantics"). Whoops, we've gotten off track mark here. I think the author may have confused "cluster" with "distributed" filesystems (Coda, AFS) which do have a notion of "nearby", and support disconnected operation. Cluster filesystems do not support disconnected operation: a disconnected machine can't see the shared filesystem at all, and any data it may have cached is invalid. In return for giving up disconnected operation, cluster filesystems offer inexpensive scaling of bandwidth and compute power, and they offer local filesystem semantics. > If they had a good standards > story with the IETF and were inkernel in 2.6, perhaps no one would > care, I'm not sure what he means. The only standard that matters to a cluster filesystem is Posix/SUS. Perhaps he is thinking about operating a shared filesystem over the internet, which is explicitly not a goal of any cluster filesystem that I know of. > but it seems odd - when you can make AFS or CIFS or NFSv4 do > the same with rather more trivial changes. Ahem. the task of adding local filesystem semantics to any existing network filesystem is far from trivial. Also, these are all really stacking filesystems: each of them exports portions of a local filesystem. A cluster filesystem is not stacked: it directly accesses a shared storage device without using some other filesystem and an intermediary. (Lustre actually uses a modified Ext3 on the storage device, however, this intermediate filesystem is used only to keep track of data blocks and does not provide namespace operations.) So, cluster and network filesystems are very different animals. > Somehow I think that the above doesn't quite capture what GFS is all > about. Indeed. The importance of GFS to Samba is that you can stack a clustered Samba on top of it, though as we have discussed, GFS will need some hacking to support things like oplocks and weirdo Windows naming semantics. Have we gotten to the point of discussing exactly how yet, or are we still digesting concepts? > I'm not trying to start a flamewar, but I'd certainly like to > see someone provide a clearer explanation than I could do. > > Chris -)----- Hope this helps, and that I haven't moosed anything up :-) Regards, Daniel From yazan at ccs.com.jo Wed Dec 8 20:45:34 2004 From: yazan at ccs.com.jo (Yazan Bakheit) Date: Wed, 8 Dec 2004 12:45:34 -0800 Subject: [Linux-cluster] need a help Message-ID: hi im new in the field , we bought the cluster suit , and i have problem making the cluster, here i will explain my problem : i have two nodes each has RedHat Enterprise linux V3 intalled, and i have a shared storage coneccted with the two nodes, the shared has 4 HD (each 72GB) and was managed as RAID 5 , each node can see the shared storage, and from the first node i made a partion with 90GB on the shared storage, my problem is that i mounted the new partition on the two nodes on the /u01 folder, now the new partition can be seen by the two nodes from the /u01 mount point, but when i make any directory from the first node1 then i couldnt see the new directory from the other node until i make a umount and then a mount for the /u01 to see the changes. this is my problem , so i need to know what option can i use with the mount command to see the changes directly on each node, ( i used [mount -o sync,atim,......] from the man page of mount , i tried all of the options but with no benifit). sorry for long E-mail , but i really need to know how the changes can be seen on the two nodes without the procedure of (umount then mount to see the result). Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tech.gif Type: image/gif Size: 862 bytes Desc: not available URL: From linux-cluster at spam.dragonhold.org Wed Dec 8 10:57:23 2004 From: linux-cluster at spam.dragonhold.org (linux-cluster at spam.dragonhold.org) Date: Wed, 8 Dec 2004 10:57:23 +0000 Subject: [Linux-cluster] need a help In-Reply-To: References: Message-ID: <20041208105723.GA1138@dragonhold.org> On Wed, Dec 08, 2004 at 12:45:34PM -0800, Yazan Bakheit wrote: > this is my problem , so i need to know what option can i use with the > mount command to see the changes directly on each node, ( i used [mount -o > sync,atim,......] from the man page of mount , i tried all of the options > but with no benifit). You need to use GFS or similar "cluster" file system - other file systems cache things internally, and therefore you will get this sort of problem. For example, if you do an ls on each machine, they will cache the blocks that represent that directory - then you won't see any changes as long as that buffer is still available - hence the need to unmount or remount. There is no way (that I know of) of using a "standard" filesystem locally mounted to both - the entire reason for the existence of GFS is this issue. From a.pugachev at pcs-net.net Wed Dec 8 13:33:12 2004 From: a.pugachev at pcs-net.net (Anatoly Pugachev) Date: Wed, 8 Dec 2004 16:33:12 +0300 Subject: [Linux-cluster] GFS and vanilla kernel In-Reply-To: <20041129212849.GA2609@dragonhold.org> References: <20041129160248.GK2721@proxy-ttk.pcs-net.net> <20041129212849.GA2609@dragonhold.org> Message-ID: <20041208133312.GX2721@proxy-ttk.pcs-net.net> Hi On Mon, Nov 29, 2004 at 09:28:49PM +0000, Graham Wood wrote: > On Mon, Nov 29, 2004 at 07:02:48PM +0300, Anatoly Pugachev wrote: > > Hello! > > > > Any chance to install GFS over vanilla kernel ? > > Installation guide will be usefull too. > > > > Thanks. > > If you're looking at the CVS, the instructions in > > http://sources.redhat.com/cluster/doc/usage.txt > > are the steps you need to take. > > To summarise: > > use cvs to download latest source. help. i'm stuck on the first stage compiling kernel. here's my problem description: (using http://sources.redhat.com/cluster/gfs/ as guide) $ cvs -d :pserver:cvs at sources.redhat.com:/cvs/cluster login cvs $ cvs -d :pserver:cvs at sources.redhat.com:/cvs/cluster checkout cluster/gfs $ cvs -d :pserver:cvs at sources.redhat.com:/cvs/cluster checkout cluster/gfs-kernel gfs kernel patch sources in $HOME/cluster now. $ cd /usr/src/v2.6.10-rc3 $ find ~/cluster -name *.patch | xargs cat | patch -p1 patching file fs/Kconfig Hunk #1 succeeded at 1800 with fuzz 2 (offset 4 lines). can't find file to patch at input line 21 Perhaps you used the wrong -p or --strip option? The text leading up to this was: -------------------------- |diff -urN -p linux-2.6.9/fs/gfs_locking/Makefile linux/fs/gfs_locking/Makefile |--- linux-2.6.9/fs/gfs_locking/Makefile 2004-10-27 02:12:05.867100243 -0500 |+++ linux/fs/gfs_locking/Makefile 2004-10-27 02:12:05.878097148 -0500 -------------------------- File to patch: this what i get trying to apply cvs gfs kernel patches to kernel sources from http://www.kernel.org/pub/linux/kernel/v2.6/testing/linux-2.6.10-rc3.tar.bz2 same with 2.6.9 kernel sources. It seems to me that this cvs patches are from already someway altered/patched kernel. If you will make simple test case, how work 'diff -urN dir1 dir2 > code.patch' first nonexistant file from dir1, in code.patch will be with time 1 Jan 1970, and as shown above with fs/gfs_locking/Makefile 2004-10-27 02:12:05.867100243 -0500 Anyone to clear situation? Please help. > run configure in the relevant directories - telling it where the kernel source is. > Run make install > Create /etc/cluster/cluster.conf > startup cluster > create filesystems > mount filesystems. > > I rebuilt mine a few days ago - from start to finish (including debian sarge install) it was a couple of hours from start to finish. -- Anatoly P. Pugachev -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From rstevens at vitalstream.com Tue Dec 7 17:34:59 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Tue, 07 Dec 2004 09:34:59 -0800 Subject: [Linux-cluster] gfs fam notify In-Reply-To: <1102371846.16796.15.camel@storm.cp-direct.com> References: <1102371846.16796.15.camel@storm.cp-direct.com> Message-ID: <41B5E9C3.3000108@vitalstream.com> Edward Mann wrote: > I have searched the news-group but have not found anyone that has asked > this question. If it has been and i have missed it please point me in > the right direction. > > > I am using FAM on a box that waits for files to be placed in folders. > Once the it is notified about a new file it will run a script that i > have created. However if the file shows up on another box it is never > notified about the new file. > > ----- ----- > - M - link - M - > - 1 - ---------- - 2 - > ----- ----- > > So machine 1 (M1) runs the script and the storage is shared between it > and Machine 2 (M2). If M2 get's the file than M1 does not know about it > and does not run the script. Is there some method that M2 can notify M1 > via gfs methods that a file was created/changed? I can just move the > script to the other box, i just want to be sure that there is not > something that i can set to get this to work the way i want. I know this > has something to do with the kernel. if i do a touch on the file on M1 > than the script will launch. I hope i have explained this well enough. > > Thanks for your time. FAM is triggered by reads, writes and such from the kernel. If M2 receives the file (I'm assuming over something such as FTP), then M1's kernel won't know about it, and thus will not launch the script. You'd need to run FAM on both nodes to do this. BTW, this is not GFS-related. You'll see the same behaviour on any file system. ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Jimmie crack corn and I don't care...what kind of lousy attitude - - is THAT to have, huh? -- Dennis Miller - ---------------------------------------------------------------------- From mmatus at dinha.acms.arizona.edu Tue Dec 7 19:43:48 2004 From: mmatus at dinha.acms.arizona.edu (Marcelo Matus) Date: Tue, 07 Dec 2004 12:43:48 -0700 Subject: [Linux-cluster] GFS, how many nodes do you have? In-Reply-To: <000c01c4dc07$bb05d560$6701a8c0@DMorganMobile> References: <000c01c4dc07$bb05d560$6701a8c0@DMorganMobile> Message-ID: <41B607F4.3070906@acms.arizona.edu> Dear GFS list members: We are planning to install GFS in our new Dual Opteron cluster (20 nodes, ie 40cpus) and probably to serve some workstation (another 10-20), considering one GFS server and using gndb. I have seen in the list that there were some problems for more than 10 nodes, but now that is fixed. Still, the question is, does someone has experience with ~40 clients + 1 GFS server?, is that too much? Marcelo From linux-cluster at spam.dragonhold.org Wed Dec 8 14:19:09 2004 From: linux-cluster at spam.dragonhold.org (Graham Wood) Date: Wed, 8 Dec 2004 14:19:09 +0000 Subject: [Linux-cluster] GFS and vanilla kernel In-Reply-To: <20041208133312.GX2721@proxy-ttk.pcs-net.net> References: <20041129160248.GK2721@proxy-ttk.pcs-net.net> <20041129212849.GA2609@dragonhold.org> <20041208133312.GX2721@proxy-ttk.pcs-net.net> Message-ID: <20041208141909.GA6269@dragonhold.org> On Wed, Dec 08, 2004 at 04:33:12PM +0300, Anatoly Pugachev wrote: > Hi > help. i'm stuck on the first stage compiling kernel. Ahh - my fault. I should have pointed out that patching the kernel isn't necessarily going to work at the moment - according to someone on the list a few weeks back, the patches there are not necessarily kept up to date with the code in the module form > $ cvs -d :pserver:cvs at sources.redhat.com:/cvs/cluster login cvs > $ cvs -d :pserver:cvs at sources.redhat.com:/cvs/cluster checkout cluster/gfs > $ cvs -d :pserver:cvs at sources.redhat.com:/cvs/cluster checkout cluster/gfs-kernel Do you already have the rest of the framework? GFS runs on top of the cluster stuff, therefore you're going to need the other cluster stuff - just cvs checkout the entire cluster tree - it's not that big, and may help you avoid other problems down the line. > $ cd /usr/src/v2.6.10-rc3 > $ find ~/cluster -name *.patch | xargs cat | patch -p1 This is going to find all the patches in that directory - I've not done a cvs co recently, but last time I checked it had multiple versions of the kernel in it. Also since the patches are closely linked to a specific kernel version, you'd need patches for the 2.6.10-rc3 version - not the 2.6.9 patches. > Anyone to clear situation? Please help. Again, apologies for not being explicit - I forgot about the patching method, since I got told a few weeks back that it wouldn't work. Download & configure/compile/install the new kernel - you shouldn't need to reboot into it, but it means you're less likely to hit any conflicts later with different versions. Ignore the reference in the instructions about patching the kernel, and work through usage.txt - or min-gfs.txt if you're using gnbd only - to get everything installed. From mbrookov at mines.edu Wed Dec 8 15:49:47 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Wed, 08 Dec 2004 08:49:47 -0700 Subject: [Linux-cluster] GFS and vanilla kernel In-Reply-To: <20041208141909.GA6269@dragonhold.org> References: <20041129160248.GK2721@proxy-ttk.pcs-net.net> <20041129212849.GA2609@dragonhold.org> <20041208133312.GX2721@proxy-ttk.pcs-net.net> <20041208141909.GA6269@dragonhold.org> Message-ID: <1102520987.23672.12.camel@merlin.Mines.EDU> On Wed, 2004-12-08 at 07:19, Graham Wood wrote: > On Wed, Dec 08, 2004 at 04:33:12PM +0300, Anatoly Pugachev wrote: > > Hi > > help. i'm stuck on the first stage compiling kernel. > Ahh - my fault. I should have pointed out that patching the kernel > isn't necessarily going > to work at the moment - according to someone on the list a few weeks > back, the patches there > are not necessarily kept up to date with the code in the module form It seems that there is some conflicting information. Patching the kernel is necessary. Without the patches you will get errors like: FATAL: Error inserting gfs (/lib/modules/2.6.9/kernel/fs/gfs/gfs.ko): Unknown symbol in module, or unknown parameter (see dmesg) gfs: no version for "lm_mount" found: kernel tainted. gfs: Unknown symbol panic_on_oops" errors in the logs. The patches for the 2.6.9 kernel include the panic_on_oops symbol as well as others. > From: Michael Conrad Tadpol Tilstra > Reply-To: mtilstra at redhat.com, linux clistering > To: linux clistering > Subject: Re: [Linux-cluster] GFS no version for lm_mount foudn: kernel tainted > Date: Fri, 3 Dec 2004 17:06:29 -0600 > On Fri, Dec 03, 2004 at 03:47:55PM -0700, Matthew B. Brookover wrote: > > I am trying to set up GFS on a 2 node cluster running Fedora core 3. > > The GFS kernel module is generating this error: "FATAL: Error inserting > > gfs (/lib/modules/2.6.9/kernel/fs/gfs/gfs.ko): Unknown symbol in > > module, or unknown parameter (see dmesg)". I am also getting a "gfs: > > no version for "lm_mount" found: kernel tainted." and "gfs: Unknown > > symbol panic_on_oops" errors in the logs. The lock_dlm, dlm, cman and > > lock_harness modules do load, but the gfs module does not. > > > > Any ideas? > > new patch in cluster/gfs-kernel/patches/2.6.9/00006.patch that exports > the panic_on_oops symbol. Need to patch your kernel, then build teh gfs > module again. > > > -- > Michael Conrad Tadpol Tilstra > Like a cool breeze through a freshly napalmed forest. Matt From mtilstra at redhat.com Wed Dec 8 16:32:33 2004 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Wed, 8 Dec 2004 10:32:33 -0600 Subject: [Linux-cluster] GFS and vanilla kernel In-Reply-To: <1102520987.23672.12.camel@merlin.Mines.EDU> References: <20041129160248.GK2721@proxy-ttk.pcs-net.net> <20041129212849.GA2609@dragonhold.org> <20041208133312.GX2721@proxy-ttk.pcs-net.net> <20041208141909.GA6269@dragonhold.org> <1102520987.23672.12.camel@merlin.Mines.EDU> Message-ID: <20041208163233.GA32145@redhat.com> On Wed, Dec 08, 2004 at 08:49:47AM -0700, Matthew B. Brookover wrote: > On Wed, 2004-12-08 at 07:19, Graham Wood wrote: > > On Wed, Dec 08, 2004 at 04:33:12PM +0300, Anatoly Pugachev wrote: > > > Hi > > > help. i'm stuck on the first stage compiling kernel. > > Ahh - my fault. I should have pointed out that patching the kernel > > isn't necessarily going > > to work at the moment - according to someone on the list a few weeks > > back, the patches there > > are not necessarily kept up to date with the code in the module form > > It seems that there is some conflicting information. Patching the > kernel is necessary. Without the patches you will get errors like: > > FATAL: Error inserting gfs (/lib/modules/2.6.9/kernel/fs/gfs/gfs.ko): > Unknown symbol in module, or unknown parameter (see dmesg) > > gfs: no version for "lm_mount" found: kernel tainted. > > gfs: Unknown symbol panic_on_oops" errors in the logs. > > The patches for the 2.6.9 kernel include the panic_on_oops symbol as > well as others. There is now a patch that is required. It wasn't there before. All this patch does is export the panic_on_oops symbol. cluster/gfs-kernel/patches/2.6.9/00006.patch You can either apply the patches to the kernel, or apply 00006.patch and build the modules from their source dirs. Both ways should work. I said a few weeks back that I forget to update the lock_gulm.patch when I make changes to the module's code sometimes. Ken is much better at keeping his patches uptodate. And in either case, you can always make new ones with the make files in the source dirs for the modules. (peek at the Makefile, you need to have the kernel source in the right place for the make patches command to work.) Sorry for confusing things. -- Michael Conrad Tadpol Tilstra Computers are not intelligent. They only think they are. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jmjoseph at andrew.cmu.edu Wed Dec 8 16:53:00 2004 From: jmjoseph at andrew.cmu.edu (Jacob Joseph) Date: Wed, 08 Dec 2004 11:53:00 -0500 Subject: [Linux-cluster] GFS, how many nodes do you have? In-Reply-To: <41B607F4.3070906@acms.arizona.edu> References: <000c01c4dc07$bb05d560$6701a8c0@DMorganMobile> <41B607F4.3070906@acms.arizona.edu> Message-ID: <41B7316C.2030904@andrew.cmu.edu> Hi. I run a very similar dual opteron cluster here at CMU and would be very interested in any responses you might have recieved off of the list. Additionally, do you happen to know of any benchmarks of gnbd performance over gigabit? Thanks, -Jacob Marcelo Matus wrote: > Dear GFS list members: > > We are planning to install GFS in our new Dual Opteron cluster > (20 nodes, ie 40cpus) and probably to serve some workstation > (another 10-20), considering one GFS server and using gndb. > > I have seen in the list that there were some problems for more > than 10 nodes, but now that is fixed. > > Still, the question is, does someone has experience with > ~40 clients + 1 GFS server?, is that too much? > > Marcelo > > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From zab at zabbo.net Wed Dec 8 18:34:45 2004 From: zab at zabbo.net (Zach Brown) Date: Wed, 08 Dec 2004 10:34:45 -0800 Subject: [Linux-cluster] Samba Technical thread In-Reply-To: <200412080110.02502.phillips@redhat.com> References: <20041207055621.GG23207@Favog.ubiqx.mn.org> <200412080110.02502.phillips@redhat.com> Message-ID: <41B74945.6060800@zabbo.net> >>but it seems odd - when you can make AFS or CIFS or NFSv4 do >>the same with rather more trivial changes. > > Ahem. the task of adding local filesystem semantics to any existing > network filesystem is far from trivial. Ahem, indeed. Daniel's measured response here is admirable :) - z From chicagoboy12001 at yahoo.com Wed Dec 8 19:08:46 2004 From: chicagoboy12001 at yahoo.com (Chicago Boy) Date: Wed, 8 Dec 2004 11:08:46 -0800 (PST) Subject: [Linux-cluster] GFS with Cluster Suite In-Reply-To: <834F55E6F1BE3B488AD3AFC927A0970007B4D1@EMAILSRV1.exad.net> Message-ID: <20041208190847.39901.qmail@web50401.mail.yahoo.com> > Is Redhat Cluster Suite required for using GFS? I believe we don't need the cluster suite. I was able to install and make few GFS tests on two nodes with out cluster suite software- but I am experiencing some stability issues. Is the cluster suite required. > If not, what would be the advantage of running > Cluster Suite with GFS? I would be interested to know about this too.. Thanks much! S --- Vincent Aniello wrote: > > Is Redhat Cluster Suite required for using GFS? > > If not, what would be the advantage of running > Cluster Suite with GFS? > > --Vincent > > > This e-mail and/or its attachments may contain > confidential and/or privileged information. If you > are not the intended recipient(s) or have received > this e-mail in error, please notify the sender > immediately and delete this e-mail and its > attachments from your computer and files. Any > unauthorized copying, disclosure or distribution of > the material contained herein is strictly forbidden. > Pipeline Trading Systems, LLC - Member NASD & SIPC. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > __________________________________ Do you Yahoo!? Take Yahoo! Mail with you! Get it on your mobile phone. http://mobile.yahoo.com/maildemo From thomsonr at ucalgary.ca Wed Dec 8 20:09:20 2004 From: thomsonr at ucalgary.ca (Ryan Thomson) Date: Wed, 08 Dec 2004 13:09:20 -0700 Subject: [Linux-cluster] LVM2 over GNBD import? Message-ID: <20041208.H51.41685700@squirrel.bio.ucalgary.ca> Hello. I am currently playing around with the GFS and the cluster tools which I checked out of CVS on December 3rd. I've created a mock setup using three cluster nodes, one of which is exporting a disk via GNBD over a private network to the other two nodes. They physical setup looks like this: nfs-serv1------| |----pool-serv1 nfs-serv2------| What I thought I wanted to do was export /dev/hdc from pool-serv1 over GNBD, import it on one nfs-serv node, use LVM2 to create some logical volumes, mount it on the other nfs-serv node and then slap GFS over the logical volumes so both nfs-servers can utilize the same LV concurrently. I either get a "Device /dev/gnbd/pool1 not found" error or a metadata error depending on the state of the partition table on the said block device. I get "Device /dev/gnbd/pool1 not found" when there are no partitions on the disk and I get the metadata error when I use 'dd' to zero out the partition table. I can execute pvcreate on pool-serv1 sometimes, but not others and I can't figure out under which situations it "works" and which it doesn't exact. Either way, when it "works", none of the other nodes seem to see the PV, VG or LGs I create locally on pool-serv1. What I've done so far on every node: ccsd cman_tool join fence_tool join clvmd After those commands, every node seems to join the cluster perfectly. Looking in /proc/cluster confirms this. My /etc/cluster/cluster.conf is as follow: I'm thinking this might have something to do with fencing since I've read that you need to fence GNBD nodes using fence_gnbd but I have no actual foundation for that assumption. Also, my understanding of fencing is... poor. I suppose my major question is this: How should I be setting this up? I want the two nfs-servers to both import the same GNBD export (shared storage), see the same LVs on that GNBD block device and put GFS on the LVs so both nfs-servers can read/write to the GNBD block device at the same time. If I'm going about this totally the wrong way, please advise. Any insight would be helpful. -- Ryan Thomson -- Ryan Thomson, Systems Administrator University Of Calgary, Biocomputing Phone: (403) 220-2264 Email: thomsonr ucalgary ca From thomsonr at ucalgary.ca Wed Dec 8 21:50:16 2004 From: thomsonr at ucalgary.ca (Ryan Thomson) Date: Wed, 08 Dec 2004 14:50:16 -0700 Subject: =?iso-8859-1?Q?=5BLinux=2Dcluster=5D?= LVM2 over GNBD import? In-Reply-To: <20041208.H51.41685700@squirrel.bio.ucalgary.ca> Message-ID: <20041208.Q9u.34664300@squirrel.bio.ucalgary.ca> More information: When I use 'dd' to zero out the partition table I get this error afterwards when trying to use pvcreate on the GNBD import: [root at wolverine ~]# pvcreate /dev/gnbd/pool1 Failed to wipe new metadata area /dev/gnbd/pool1: Format-specific setup of physical volume failed. Failed to setup physical volume "/dev/gnbd/pool1" blockdev gives me this error when I try to re-read it after I zero the partition table with'dd' (unmounted, non-gnbd exported): [root at pool-serv1 ~]# blockdev --rereadpt /dev/hdc BLKRRPART: Input/output error I know I'm doing something wrong, but I just can't seem to figure it out. -- Ryan Ryan Thomson (thomsonr at ucalgary.ca) wrote: > > Hello. > > I am currently playing around with the GFS and the cluster tools which I > checked out of CVS on December 3rd. I've created a mock setup using three > cluster nodes, one of which is exporting a disk via GNBD over a private > network to the other two nodes. > > They physical setup looks like this: > > nfs-serv1------| > |----pool-serv1 > nfs-serv2------| > > What I thought I wanted to do was export /dev/hdc from pool-serv1 over GNBD, > import it on one nfs-serv node, use LVM2 to create some logical volumes, mount > it on the other nfs-serv node and then slap GFS over the logical volumes so > both nfs-servers can utilize the same LV concurrently. I either get a "Device > /dev/gnbd/pool1 not found" error or a metadata error depending on the state of > the partition table on the said block device. I get "Device /dev/gnbd/pool1 > not found" when there are no partitions on the disk and I get the metadata > error when I use 'dd' to zero out the partition table. > > I can execute pvcreate on pool-serv1 sometimes, but not others and I can't > figure out under which situations it "works" and which it doesn't exact. > Either way, when it "works", none of the other nodes seem to see the PV, VG or > LGs I create locally on pool-serv1. > > What I've done so far on every node: > > ccsd > cman_tool join > fence_tool join > clvmd > > After those commands, every node seems to join the cluster perfectly. Looking > in /proc/cluster confirms this. > > My /etc/cluster/cluster.conf is as follow: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm thinking this might have something to do with fencing since I've read that > you need to fence GNBD nodes using fence_gnbd but I have no actual foundation > for that assumption. Also, my understanding of fencing is... poor. > > I suppose my major question is this: > > How should I be setting this up? I want the two nfs-servers to both import the > same GNBD export (shared storage), see the same LVs on that GNBD block device > and put GFS on the LVs so both nfs-servers can read/write to the GNBD block > device at the same time. If I'm going about this totally the wrong way, please > advise. > > Any insight would be helpful. From thomsonr at ucalgary.ca Thu Dec 9 03:09:47 2004 From: thomsonr at ucalgary.ca (Ryan Thomson) Date: Wed, 08 Dec 2004 20:09:47 -0700 Subject: =?iso-8859-1?Q?=5BLinux=2Dcluster=5D?= LVM2 over GNBD import? In-Reply-To: <20041208.Q9u.34664300@squirrel.bio.ucalgary.ca> Message-ID: <20041208.rsF.24010100@squirrel.bio.ucalgary.ca> And it turns out to be something stupid. Appearently you need to run 'pvscan' after importing the GNBD device, to think of the time I wasted on this... at least I learned something :) -- Ryan Ryan Thomson (thomsonr at ucalgary.ca) wrote: > > More information: > > When I use 'dd' to zero out the partition table I get this error afterwards > when trying to use pvcreate on the GNBD import: > > [root at wolverine ~]# pvcreate /dev/gnbd/pool1 > Failed to wipe new metadata area > /dev/gnbd/pool1: Format-specific setup of physical volume failed. > Failed to setup physical volume "/dev/gnbd/pool1" > > blockdev gives me this error when I try to re-read it after I zero the > partition table with'dd' (unmounted, non-gnbd exported): > > [root at pool-serv1 ~]# blockdev --rereadpt /dev/hdc > BLKRRPART: Input/output error > > I know I'm doing something wrong, but I just can't seem to figure it out. > > > Ryan Thomson (thomsonr at ucalgary.ca) wrote: > > > > Hello. > > > > I am currently playing around with the GFS and the cluster tools which I > > checked out of CVS on December 3rd. I've created a mock setup using three > > cluster nodes, one of which is exporting a disk via GNBD over a private > > network to the other two nodes. > > > > They physical setup looks like this: > > > > nfs-serv1------| > > |----pool-serv1 > > nfs-serv2------| > > > > What I thought I wanted to do was export /dev/hdc from pool-serv1 over GNBD, > > import it on one nfs-serv node, use LVM2 to create some logical volumes, mount > > it on the other nfs-serv node and then slap GFS over the logical volumes so > > both nfs-servers can utilize the same LV concurrently. I either get a "Device > > /dev/gnbd/pool1 not found" error or a metadata error depending on the state of > > the partition table on the said block device. I get "Device /dev/gnbd/pool1 > > not found" when there are no partitions on the disk and I get the metadata > > error when I use 'dd' to zero out the partition table. > > > > I can execute pvcreate on pool-serv1 sometimes, but not others and I can't > > figure out under which situations it "works" and which it doesn't exact. > > Either way, when it "works", none of the other nodes seem to see the PV, VG or > > LGs I create locally on pool-serv1. > > > > What I've done so far on every node: > > > > ccsd > > cman_tool join > > fence_tool join > > clvmd > > > > After those commands, every node seems to join the cluster perfectly. Looking > > in /proc/cluster confirms this. > > > > My /etc/cluster/cluster.conf is as follow: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm thinking this might have something to do with fencing since I've read that > > you need to fence GNBD nodes using fence_gnbd but I have no actual foundation > > for that assumption. Also, my understanding of fencing is... poor. > > > > I suppose my major question is this: > > > > How should I be setting this up? I want the two nfs-servers to both import the > > same GNBD export (shared storage), see the same LVs on that GNBD block device > > and put GFS on the LVs so both nfs-servers can read/write to the GNBD block > > device at the same time. If I'm going about this totally the wrong way, please > > advise. > > > > Any insight would be helpful. > > > From stanley.wang at linux.intel.com Thu Dec 9 16:41:36 2004 From: stanley.wang at linux.intel.com (Stanley Wang) Date: Thu, 09 Dec 2004 16:41:36 +0000 Subject: [Linux-cluster] Simple wrap for SAF AIS Lock API Message-ID: <1102610496.4843.16.camel@manticore.sh.intel.com> Hi all, The attached patch provides SAF AIS lock APIs support based on current GDLM. It's just simplest wrap of GDLM's user mode api and didn't touch GDLM's codes. I think it can be a good complementarity to GDLM. The patch is against lastest CVS codes. Any interests or comments? Best Regards, Stan -- Opinions expressed are those of the author and do not represent Intel Corporation "gpg --recv-keys --keyserver wwwkeys.pgp.net E1390A7F" {E1390A7F:3AD1 1B0C 2019 E183 0CFF 55E8 369A 8B75 E139 0A7F} -------------- next part -------------- A non-text attachment was scrubbed... Name: saf_lock.diff Type: text/x-patch Size: 20314 bytes Desc: not available URL: From cian.cullinan at gmail.com Thu Dec 9 11:00:21 2004 From: cian.cullinan at gmail.com (Cian Cullinan) Date: Thu, 9 Dec 2004 11:00:21 +0000 Subject: [Linux-cluster] oracle 9i on redhat cluster suite Message-ID: Hi, I'm looking to set up oracle 9i on a two RH ES3 machines running Redhat Cluster suite. There's a short section in the Cluster suite manual on running Oracle but it's a bit lacking in details. Has anyone done this before and would be willing to answer a few questions, or know of some decent on-line documentation? A couple of quick questions off the top of my head: 1) Do I install Oracle on each machine (as is normal for clustered applications) or just once on a shared disk? 2) If on a shared disk, do I need to copy any files from /etc, /home/oracle between the two machines before starting? Thanks, Cian From pcaulfie at redhat.com Thu Dec 9 14:43:13 2004 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Thu, 9 Dec 2004 14:43:13 +0000 Subject: [Linux-cluster] GFS 3 node hang in rm test In-Reply-To: <1102464791.2706.122.camel@ibm-c.pdx.osdl.net> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <20041206194521.GA6120@potassium.msp.redhat.com> <1102378430.2706.83.camel@ibm-c.pdx.osdl.net> <20041207093810.GC29001@tykepenguin.com> <1102438394.2706.103.camel@ibm-c.pdx.osdl.net> <1102464791.2706.122.camel@ibm-c.pdx.osdl.net> Message-ID: <20041209144313.GC15822@tykepenguin.com> There are a few straws being clutched at but can you try this patch please? -- patrick RCS file: /cvs/cluster/cluster/dlm-kernel/src/locking.c,v retrieving revision 1.49 diff -u -r1.49 locking.c --- locking.c 8 Dec 2004 11:46:05 -0000 1.49 +++ locking.c 9 Dec 2004 14:42:57 -0000 @@ -529,8 +529,8 @@ retry: if (rsb->res_nodeid == -1) { if (get_directory_nodeid(rsb) != our_nodeid()) { - remote_stage(lkb, GDLM_LQSTATE_WAIT_RSB); up_write(&rsb->res_lock); + remote_stage(lkb, GDLM_LQSTATE_WAIT_RSB); return 0; } From rstevens at vitalstream.com Thu Dec 9 16:57:32 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Thu, 09 Dec 2004 08:57:32 -0800 Subject: [Linux-cluster] Two-node problems Message-ID: <41B883FC.7020201@vitalstream.com> Hi, gang. We're trying to do a GFS test on a two node cluster. In our offices, the test worked very well and we're trying to put it up for "real world" testing. We're sorting out an issue with the hardware in one of the nodes (it keeps crashing). However, when it dies, node 2 loses quorum and activity on the cluster ceases. I thought the purpose of was to prevent a loss of quorum on a two node cluster when one node goes down. Can someone enlighten me? ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Polygon: A dead parrot (With apologies to John Cleese) - ---------------------------------------------------------------------- From mbrookov at mines.edu Thu Dec 9 17:08:41 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Thu, 09 Dec 2004 10:08:41 -0700 Subject: [Linux-cluster] cluster is not quorate. refusing connection Message-ID: <1102612121.4846.24.camel@merlin.Mines.EDU> Getting closer. The modules load and run, and after a few minutes GFS will work. The problem is ccsd logs this error "Cluster is not quorate. Refusing connection. Error while processing connect: Connection refused" in syslog. This error will be generated by ccsd every second for 5 minutes, then fence_tool will exit, fenced starts up and I can mount file systems. Cman on the other node will generate this log messages: CMAN: WARNING no listener for port 11 on node fiveoften. This is a 2 node cluster using SCSI disks on Fedora Core 3 and Linux 2.6.9 from kernel.org. I have attached the logs from both hosts and the cluter.conf. Any ideas? The source code was pulled from CVS on Monday. Thank you for your help. Matt mbrookov at mines.edu Logs from fiveoften: Dec 9 09:00:28 fiveoften kernel: Lock_Harness (built Dec 6 2004 16:42:08) installed Dec 9 09:00:28 fiveoften kernel: GFS (built Dec 6 2004 16:41:25) installed Dec 9 09:00:28 fiveoften kernel: CMAN (built Dec 7 2004 08:47:39) installed Dec 9 09:00:28 fiveoften kernel: NET: Registered protocol family 30 Dec 9 09:00:28 fiveoften kernel: DLM (built Dec 7 2004 08:48:06) installed Dec 9 09:00:28 fiveoften kernel: Lock_DLM (built Dec 6 2004 16:41:47) installed Dec 9 09:00:29 fiveoften kernel: CMAN: Waiting to join or form a Linux-cluster Dec 9 09:01:01 fiveoften kernel: CMAN: forming a new cluster Dec 9 09:01:01 fiveoften kernel: CMAN: quorum regained, resuming activity Dec 9 09:01:01 fiveoften kernel: CMAN: got node fouroften Dec 9 09:01:01 fiveoften ccsd[3417]: Cluster is not quorate. Refusing connection. Dec 9 09:01:01 fiveoften ccsd[3417]: Error while processing connect: Connection refused Dec 9 09:01:01 fiveoften crond(pam_unix)[3454]: session opened for user root by (uid=0) Dec 9 09:01:01 fiveoften crond(pam_unix)[3454]: session closed for user root Dec 9 09:01:02 fiveoften ccsd[3417]: Cluster is not quorate. Refusing connection. Dec 9 09:01:02 fiveoften ccsd[3417]: Error while processing connect: Connection refused Dec 9 09:01:03 fiveoften ccsd[3417]: Cluster is not quorate. Refusing connection. Dec 9 09:01:03 fiveoften ccsd[3417]: Error while processing connect: Connection refused Dec 9 09:01:04 fiveoften ccsd[3417]: Cluster is not quorate. Refusing connection. Dec 9 09:01:04 fiveoften ccsd[3417]: Error while processing connect: Connection refused Dec 9 09:01:05 fiveoften ccsd[3417]: Cluster is not quorate. Refusing connection. . . . . Dec 9 09:05:59 fiveoften ccsd[3417]: Error while processing connect: Connection refused Dec 9 09:06:00 fiveoften ccsd[3417]: Cluster is not quorate. Refusing connection. Dec 9 09:06:00 fiveoften ccsd[3417]: Error while processing connect: Connection refused Dec 9 09:06:01 fiveoften kernel: dlm: clvmd: recover event 2 (first) Dec 9 09:06:01 fiveoften kernel: dlm: clvmd: add nodes Dec 9 09:06:01 fiveoften kernel: dlm: connecting to 2 Dec 9 09:06:01 fiveoften kernel: dlm: clvmd: total nodes 2 Dec 9 09:06:01 fiveoften kernel: dlm: clvmd: rebuild resource directory Dec 9 09:06:01 fiveoften kernel: dlm: clvmd: rebuilt 1 resources Dec 9 09:06:02 fiveoften kernel: dlm: clvmd: recover event 2 done Dec 9 09:06:02 fiveoften kernel: dlm: clvmd: process held requests Dec 9 09:06:02 fiveoften kernel: dlm: clvmd: processed 0 requests Dec 9 09:06:02 fiveoften kernel: dlm: clvmd: recover event 2 finished Logs from fouroften: Dec 9 09:00:28 fouroften kernel: Lock_Harness (built Dec 6 2004 16:42:08) installed Dec 9 09:00:28 fouroften kernel: GFS (built Dec 6 2004 16:41:25) installed Dec 9 09:00:28 fouroften kernel: CMAN (built Dec 7 2004 08:47:39) installed Dec 9 09:00:28 fouroften kernel: NET: Registered protocol family 30 Dec 9 09:00:28 fouroften kernel: DLM (built Dec 7 2004 08:48:06) installed Dec 9 09:00:28 fouroften kernel: Lock_DLM (built Dec 6 2004 16:41:47) installed Dec 9 09:00:29 fouroften kernel: CMAN: Waiting to join or form a Linux-cluster Dec 9 09:00:58 fouroften kernel: CMAN: sending membership request Dec 9 09:00:58 fouroften kernel: CMAN: got node fiveoften Dec 9 09:00:58 fouroften kernel: CMAN: quorum regained, resuming activity Dec 9 09:01:00 fouroften kernel: dlm: clvmd: recover event 2 (first) Dec 9 09:01:00 fouroften kernel: dlm: clvmd: add nodes Dec 9 09:01:01 fouroften crond(pam_unix)[3198]: session opened for user root by (uid=0) Dec 9 09:01:01 fouroften crond(pam_unix)[3198]: session closed for user root Dec 9 09:01:02 fouroften kernel: dlm: clvmd: total nodes 1 Dec 9 09:01:02 fouroften kernel: dlm: clvmd: rebuild resource directory Dec 9 09:01:02 fouroften kernel: dlm: clvmd: rebuilt 0 resources Dec 9 09:01:02 fouroften kernel: dlm: clvmd: recover event 2 done Dec 9 09:01:02 fouroften kernel: CMAN: WARNING no listener for port 11 on node fiveoften Dec 9 09:01:03 fouroften kernel: dlm: clvmd: process held requests Dec 9 09:01:03 fouroften kernel: dlm: clvmd: processed 0 requests Dec 9 09:01:03 fouroften kernel: dlm: clvmd: recover event 2 finished . . . . Dec 9 09:05:58 fouroften kernel: dlm: got connection from 1 Dec 9 09:05:58 fouroften kernel: dlm: clvmd: recover event 4 Dec 9 09:05:58 fouroften kernel: dlm: clvmd: add node 1 Dec 9 09:05:59 fouroften kernel: dlm: clvmd: total nodes 2 Dec 9 09:05:59 fouroften kernel: dlm: clvmd: rebuild resource directory Dec 9 09:05:59 fouroften kernel: dlm: clvmd: rebuilt 2 resources Dec 9 09:05:59 fouroften kernel: dlm: clvmd: purge requests Dec 9 09:05:59 fouroften kernel: dlm: clvmd: purged 0 requests Dec 9 09:05:59 fouroften kernel: dlm: clvmd: mark waiting requests Dec 9 09:05:59 fouroften kernel: dlm: clvmd: marked 0 requests Dec 9 09:05:59 fouroften kernel: dlm: clvmd: recover event 4 done Dec 9 09:06:00 fouroften kernel: dlm: clvmd: process held requests Dec 9 09:06:00 fouroften kernel: dlm: clvmd: processed 1 requests Dec 9 09:06:00 fouroften kernel: dlm: clvmd: resend marked requests Dec 9 09:06:00 fouroften kernel: dlm: clvmd: resent 0 requests Dec 9 09:06:00 fouroften kernel: dlm: clvmd: recover event 4 finished Cluster.conf: From phillips at redhat.com Thu Dec 9 21:52:42 2004 From: phillips at redhat.com (Daniel Phillips) Date: Thu, 9 Dec 2004 16:52:42 -0500 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <20041207093810.GC29001@tykepenguin.com> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <1102378430.2706.83.camel@ibm-c.pdx.osdl.net> <20041207093810.GC29001@tykepenguin.com> Message-ID: <200412091652.42408.phillips@redhat.com> On Tuesday 07 December 2004 04:38, Patrick Caulfield wrote: > On Mon, Dec 06, 2004 at 04:13:50PM -0800, Daniel McNeil wrote: > > On Mon, 2004-12-06 at 11:45, Ken Preslan wrote: > > > On Fri, Dec 03, 2004 at 03:08:00PM -0800, Daniel McNeil wrote: > > > > Looking at the stack trace above and dissabling dlm.ko, > > it looks like dlm_lock+0x319 is the call to dlm_lock_stage1(). > > looking at dlm_lock_stage1(), it looks like it is sleeping on > > down_write(&rsb->res_lock) > > > > So now I have to find who is holding the res_lock. > > That's consistent with the hang you reported before - in fact it's > almost certainly the same thing. My guess is thet there is a dealock > on res_lock somewhere . In which case I suspect it's going to be > easier to find that one by reading code rather than running tests. > res_lock should never be held for any extended period of time, but in > your last set of tracebacks there was nothing obviously holding it - > so I suspect something is sleeping with it. Hi Patrick, Last week I had a bug in the cluster snapshot failover code that exposed a bug in dlm or libdlm I think. My code inadvertently acquired a lock twice, first in PW mode, then later in CR mode (because I wasn't checking to see if it already had the PW lock). This caused dlm_unlock_wait to wait forever. Are these locks supposed to be recursive or not? In any event, waiting forever has got to be a bug. It might have something to do with a lkid tangle, since I never provided separate lkids for the unlock. This should be easily reproducible. Regards, Daniel From rstevens at vitalstream.com Thu Dec 9 22:31:29 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Thu, 09 Dec 2004 14:31:29 -0800 Subject: [Linux-cluster] Hard lockups when writing a lot to GFS Message-ID: <41B8D241.6020309@vitalstream.com> I have a two-node setup on a dual-port SCSI SAN. Note this is just for test purposes. Part of the SAN is a GFS filesystem shared between the two nodes. When we fetch content to the GFS filesystem via an rsync pull (well, several rsync pulls) on node 1, it runs for a while then node 1 hard locks (nothing on the console, network dies, console dies, it's frozen solid). Of course, node 2 notices it and marks node 1 down (/proc/cluster/nodes shows an "X" for node 1 under "Sts"). So the cluster behaviour is OK. If I "fence-ack-manual -n node1" on node 2, it runs along happily. I can reboot node 1 and everything returns to normalcy. The problem is, why is node 1 dying like this? It is important that this get sorted out as we have a LOT of data to synchronize (rsync is just the test case--we'll probably use a different scheme on deployment), and I suspect it's heavy write activity on that node that's causing the crash. Oh, both nodes have the GFS filesystem mounted with "-o rw,noatime". Any ideas would be GREATLY appreciated! ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Do you know how to save five drowning lawyers? No? GOOD! - ---------------------------------------------------------------------- From daniel at osdl.org Thu Dec 9 23:47:02 2004 From: daniel at osdl.org (Daniel McNeil) Date: Thu, 09 Dec 2004 15:47:02 -0800 Subject: [Linux-cluster] dlm patch to fix referencing freed memory In-Reply-To: <20041209144313.GC15822@tykepenguin.com> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <20041206194521.GA6120@potassium.msp.redhat.com> <1102378430.2706.83.camel@ibm-c.pdx.osdl.net> <20041207093810.GC29001@tykepenguin.com> <1102438394.2706.103.camel@ibm-c.pdx.osdl.net> <1102464791.2706.122.camel@ibm-c.pdx.osdl.net> <20041209144313.GC15822@tykepenguin.com> Message-ID: <1102636022.19646.17.camel@ibm-c.pdx.osdl.net> I turned on CONFIG_DEBUG_SLAB was hitting slab corruption. I added a debug magic number to the rw_semaphore struct and then added a BUG_ON() in the down_write() and up_write() and got a stack trace showing the up_write() was referencing free memory: EIP is at dlm_unlock_stage2+0x126/0x2a0 [dlm] eax: 00000025 ebx: e5b34d98 ecx: c0456c0c edx: 00000286 esi: e5b34d34 edi: e5b2eb28 ebp: ea3f2e64 esp: ea3f2e48 ds: 007b es: 007b ss: 0068 Process gfs_glockd (pid: 3693, threadinfo=ea3f2000 task=e8c20ed0) Stack: e5b2eb28 00000005 00000000 00000000 e5b2eb28 e7ae85a4 e7ae8654 ea3f2ea0 f8b305dd e5b2eb28 e5b34d34 00000000 000e00a3 00040000 00000000 00000000 e5b34dcd ffffffea e5b34d34 00000000 e5b40678 e5b33d38 ea3f2ecc f8b4fcd6 Call Trace: [] show_stack+0x7f/0xa0 [] show_registers+0x15e/0x1d0 [] die+0xfe/0x190 [] error_code+0x2d/0x38 [] dlm_unlock+0x2ad/0x3c0 [dlm] [] do_dlm_unlock+0x86/0x120 [lock_dlm] [] lm_dlm_unlock+0x18/0x30 [lock_dlm] [] gfs_glock_drop_th+0x93/0x1a0 [gfs] [] rq_demote+0xbb/0xe0 [gfs] [] run_queue+0x88/0xe0 [gfs] [] unlock_on_glock+0x2b/0x40 [gfs] [] gfs_reclaim_glock+0x132/0x1b0 [gfs] [] gfs_glockd+0x11a/0x130 [gfs] [] kernel_thread_helper+0x5/0x10 Code: 2a 89 d8 ba ff ff 00 00 f0 0f c1 10 0f 85 4c 12 00 00 ba 9a 3a b4 f8 e8 c9 bf 6e c7 e8 54 b8 ff ff 83 c4 10 31 c0 5b 5e 5f 5d c3 <0f> 0b e8 00 2e 3a b4 f8 eb cc ba 88 3a b4 f8 89 d8 e8 a4 bf 6e Looking through the code, I found when that a call to queue_ast(lkb, AST_COMP | AST_DEL, 0); will lead to process_asts() which will free the dlm_rsb. So there is a race where the rsb can be freed BEFORE we do the up_write(rsb->res_lock); The fix is simple, do the up_write() before the queue_ast(). Here's a patch that fixed this problem: --- cluster.orig/dlm-kernel/src/locking.c 2004-12-09 15:23:13.789834384 -0800 +++ cluster/dlm-kernel/src/locking.c 2004-12-09 15:24:51.809742940 -0800 @@ -687,8 +687,13 @@ void dlm_lock_stage3(struct dlm_lkb *lkb lkb->lkb_retstatus = -EAGAIN; if (lkb->lkb_lockqueue_flags & DLM_LKF_NOQUEUEBAST) send_blocking_asts_all(rsb, lkb); + /* + * up the res_lock before queueing ast, since the AST_DEL will + * cause the rsb to be released and that can happen anytime. + */ + up_write(&rsb->res_lock); queue_ast(lkb, AST_COMP | AST_DEL, 0); - goto out; + return; } /* @@ -888,7 +893,13 @@ int dlm_unlock_stage2(struct dlm_lkb *lk lkb->lkb_retstatus = flags & DLM_LKF_CANCEL ? -DLM_ECANCEL:-DLM_EUNLOCK; if (!remote) { + /* + * up the res_lock before queueing ast, since the AST_DEL will + * cause the rsb to be released and that can happen anytime. + */ + up_write(&rsb->res_lock); queue_ast(lkb, AST_COMP | AST_DEL, 0); + goto out2; } else { up_write(&rsb->res_lock); release_lkb(rsb->res_ls, lkb); This did not fix my other hang, I'll try out Patrick's simple patch and see what happens. Thanks, Daniel From daniel at osdl.org Fri Dec 10 01:16:58 2004 From: daniel at osdl.org (Daniel McNeil) Date: Thu, 09 Dec 2004 17:16:58 -0800 Subject: [Linux-cluster] A slightly different hang on 2 node remove In-Reply-To: <20041209144313.GC15822@tykepenguin.com> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <20041206194521.GA6120@potassium.msp.redhat.com> <1102378430.2706.83.camel@ibm-c.pdx.osdl.net> <20041207093810.GC29001@tykepenguin.com> <1102438394.2706.103.camel@ibm-c.pdx.osdl.net> <1102464791.2706.122.camel@ibm-c.pdx.osdl.net> <20041209144313.GC15822@tykepenguin.com> Message-ID: <1102641418.19646.21.camel@ibm-c.pdx.osdl.net> I reran the test with my patch and what looks like a different hang in glock and/or dlm. The info is here: http://developer.osdl.org/daniel/GFS/rm.hang.09dec2004/ Daniel PS I'm running with patrick's small patch now. On Thu, 2004-12-09 at 06:43, Patrick Caulfield wrote: > There are a few straws being clutched at but can you try this patch > please? From teigland at redhat.com Fri Dec 10 05:41:18 2004 From: teigland at redhat.com (David Teigland) Date: Fri, 10 Dec 2004 13:41:18 +0800 Subject: [Linux-cluster] Two-node problems In-Reply-To: <41B883FC.7020201@vitalstream.com> References: <41B883FC.7020201@vitalstream.com> Message-ID: <20041210054118.GB8758@redhat.com> On Thu, Dec 09, 2004 at 08:57:32AM -0800, Rick Stevens wrote: > Hi, gang. > > We're trying to do a GFS test on a two node cluster. In our offices, > the test worked very well and we're trying to put it up for "real world" > testing. > > We're sorting out an issue with the hardware in one of the nodes (it > keeps crashing). However, when it dies, node 2 loses quorum and > activity on the cluster ceases. I thought the purpose of > > > > Does your cluster.conf also have a hyphenated two-node (instead of two_node)? -- Dave Teigland From teigland at redhat.com Fri Dec 10 06:22:49 2004 From: teigland at redhat.com (David Teigland) Date: Fri, 10 Dec 2004 14:22:49 +0800 Subject: [Linux-cluster] Hard lockups when writing a lot to GFS In-Reply-To: <41B8D241.6020309@vitalstream.com> References: <41B8D241.6020309@vitalstream.com> Message-ID: <20041210062249.GC8758@redhat.com> On Thu, Dec 09, 2004 at 02:31:29PM -0800, Rick Stevens wrote: > I have a two-node setup on a dual-port SCSI SAN. Note this is just > for test purposes. Part of the SAN is a GFS filesystem shared between > the two nodes. > > When we fetch content to the GFS filesystem via an rsync pull (well, > several rsync pulls) on node 1, it runs for a while then node 1 hard > locks (nothing on the console, network dies, console dies, it's frozen > solid). Of course, node 2 notices it and marks node 1 down > (/proc/cluster/nodes shows an "X" for node 1 under "Sts"). So the > cluster behaviour is OK. If I "fence-ack-manual -n node1" on node 2, > it runs along happily. I can reboot node 1 and everything returns to > normalcy. > > The problem is, why is node 1 dying like this? It is important that > this get sorted out as we have a LOT of data to synchronize (rsync is > just the test case--we'll probably use a different scheme on > deployment), and I suspect it's heavy write activity on that node > that's causing the crash. > > Oh, both nodes have the GFS filesystem mounted with "-o rw,noatime". > > Any ideas would be GREATLY appreciated! You might want to verify it's not the hardware giving you problems. SCSI RAID devices might behave badly when heavily loaded from more than one initiator at a time. I'd partition the device or create two LUN's, put ext3 on each, have each node mount one of the ext3's, and run the same kind of load test. Since you don't see any SCSI errors on either machine it might seem unlikely, but it's a start. -- Dave Teigland From teigland at redhat.com Fri Dec 10 07:17:45 2004 From: teigland at redhat.com (David Teigland) Date: Fri, 10 Dec 2004 15:17:45 +0800 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: <1102612121.4846.24.camel@merlin.Mines.EDU> References: <1102612121.4846.24.camel@merlin.Mines.EDU> Message-ID: <20041210071745.GD8758@redhat.com> On Thu, Dec 09, 2004 at 10:08:41AM -0700, Matthew B. Brookover wrote: > Getting closer. The modules load and run, and after a few minutes GFS > will work. The problem is ccsd logs this error "Cluster is not > quorate. Refusing connection. Error while processing connect: > Connection refused" in syslog. This error will be generated by ccsd > every second for 5 minutes, then fence_tool will exit, fenced starts up > and I can mount file systems. This sounds similar to a problem I have if I run fence_tool without ccsd running. Check /proc/cluster/status while it's waiting to see if the cluster actually has quorum or not. Also, I've added some extra checking and debugging to fence_tool that should help narrow down where things are stuck. Please update from cvs and rebuild at least the stuff in cluster/fence; then use "fence_tool join -D". Usually things get stuck talking to ccs when ccs/magma libraries are out of sync, but this case sounds different. -- Dave Teigland From mmatus at dinha.acms.arizona.edu Fri Dec 10 09:09:27 2004 From: mmatus at dinha.acms.arizona.edu (Marcelo Matus) Date: Fri, 10 Dec 2004 02:09:27 -0700 Subject: [Linux-cluster] GFS, how many nodes do you have? In-Reply-To: <41B7316C.2030904@andrew.cmu.edu> References: <000c01c4dc07$bb05d560$6701a8c0@DMorganMobile> <41B607F4.3070906@acms.arizona.edu> <41B7316C.2030904@andrew.cmu.edu> Message-ID: <41B967C7.4030208@acms.arizona.edu> I saw some benchmarks over 100 Mbs/ ethernet, they got around *5MB/s (single IDE disk?).* https://open.datacore.ch/DCwiki.open/Wiki.jsp?page=GFS.Bonnie Version 1.03 - - - Sequential Output - - - --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP System2-GNBD-GFS 1G 4190 27 7185 11 2777 5 4165 26 5029 4 96.4 3 - - - Sequential Create - - - - - - - Random Create - - - - -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 262 38 12996 35 147 11 203 31 557 7 150 12 * We use Infiniband and we are getting around 35MB/s (SCSI raid 5/SGI TP9100).* Bonnie 1.4: File './Bonnie.8320', size: 1048576000, volumes: 1 Writing with putc()... done: 34619 kB/s 90.6 %CPU Rewriting... done: 459206 kB/s 99.6 %CPU Writing intelligently... done: 213785 kB/s 41.5 %CPU Reading with getc()... done: 35544 kB/s 100.0 %CPU Reading intelligently... done: 1066698 kB/s 100.0 %CPU I guess you should get a similar performance over Gigabit. For comparison, our Infiniband network provides only around 2 times the bandwith over TCP/IP (1800 Mb/s) than our GB network (800Mb/s). Marcelo Jacob Joseph wrote: > Hi. I run a very similar dual opteron cluster here at CMU and would > be very interested in any responses you might have recieved off of the > list. Additionally, do you happen to know of any benchmarks of gnbd > performance over gigabit? > > Thanks, > -Jacob > > Marcelo Matus wrote: > >> Dear GFS list members: >> >> We are planning to install GFS in our new Dual Opteron cluster >> (20 nodes, ie 40cpus) and probably to serve some workstation >> (another 10-20), considering one GFS server and using gndb. >> >> I have seen in the list that there were some problems for more >> than 10 nodes, but now that is fixed. >> >> Still, the question is, does someone has experience with >> ~40 clients + 1 GFS server?, is that too much? >> >> Marcelo >> >> >> >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From pcaulfie at redhat.com Fri Dec 10 09:14:53 2004 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Fri, 10 Dec 2004 09:14:53 +0000 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <200412091652.42408.phillips@redhat.com> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <1102378430.2706.83.camel@ibm-c.pdx.osdl.net> <20041207093810.GC29001@tykepenguin.com> <200412091652.42408.phillips@redhat.com> Message-ID: <20041210091453.GD14897@tykepenguin.com> On Thu, Dec 09, 2004 at 04:52:42PM -0500, Daniel Phillips wrote: > On Tuesday 07 December 2004 04:38, Patrick Caulfield wrote: > > On Mon, Dec 06, 2004 at 04:13:50PM -0800, Daniel McNeil wrote: > > > On Mon, 2004-12-06 at 11:45, Ken Preslan wrote: > > > > On Fri, Dec 03, 2004 at 03:08:00PM -0800, Daniel McNeil wrote: > > > > > > Looking at the stack trace above and dissabling dlm.ko, > > > it looks like dlm_lock+0x319 is the call to dlm_lock_stage1(). > > > looking at dlm_lock_stage1(), it looks like it is sleeping on > > > down_write(&rsb->res_lock) > > > > > > So now I have to find who is holding the res_lock. > > > > That's consistent with the hang you reported before - in fact it's > > almost certainly the same thing. My guess is thet there is a dealock > > on res_lock somewhere . In which case I suspect it's going to be > > easier to find that one by reading code rather than running tests. > > res_lock should never be held for any extended period of time, but in > > your last set of tracebacks there was nothing obviously holding it - > > so I suspect something is sleeping with it. > > Hi Patrick, > > Last week I had a bug in the cluster snapshot failover code that exposed > a bug in dlm or libdlm I think. My code inadvertently acquired a lock > twice, first in PW mode, then later in CR mode (because I wasn't > checking to see if it already had the PW lock). This caused > dlm_unlock_wait to wait forever. Are these locks supposed to be > recursive or not? In any event, waiting forever has got to be a bug. > > It might have something to do with a lkid tangle, since I never provided > separate lkids for the unlock. > > This should be easily reproducible. Thanks, I'll have a look at that. It's certainly a bug in your code that's causing it but, as you say, a hang in the library is still a bug in the library. If you call dlm_lock() with the same parameters twice then you will get two seperate locks - in that case the CR will wait for the PW to get out of the way. What is really confusing the issue is that both locks are sharing a lock status block - so it's possible that one lock is being marked INPROGRESS immediately after the first has completed, in that case the library will have a lot of difficulty in digging you out of your own hole I'm afraid as it can't disentangle the status of your two locks. Anyway, I'll see if I can see what's happening. -- patrick From owen at isrl.uiuc.edu Fri Dec 10 18:15:29 2004 From: owen at isrl.uiuc.edu (Brynnen R Owen) Date: Fri, 10 Dec 2004 12:15:29 -0600 Subject: [Linux-cluster] NFS + GFS = kernel panic In-Reply-To: <20041203173442.GA14481@potassium.msp.redhat.com> References: <20041130203210.GC1163@iwork57.lis.uiuc.edu> <20041203173442.GA14481@potassium.msp.redhat.com> Message-ID: <20041210181529.GA9880@iwork57.lis.uiuc.edu> > > More from my testing. I have the CVS versions of GFS, ccs, and > > lock_gulm, all from Nov 21, with a stock 2.6.9 kernel from > > kernel.org. Things seem to run fine until NFS is used. NFS appears > > to work for a bit, but either after a few hours or when there is a > > known stale NFS handle, the kernel panics with (Sorry, only a fragment > > of the error) > > > > Failed assertion fs/gfs/glock.c line 1366. > > > > Any ideas on this one? > > Hi, > > [logging instructions] A quick update from about 2 weeks ago. As of yet, I have been stuck with NFS problems which appear to be related to the 2.6.9 vanilla kernel. I'm working on getting a kernel that works with NFS and ext3 before completing the GFS tests. Is there a problem with applying the 00006.patch for kernel oopses to other kernels than 2.6.9? -- <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> <> Brynnen Owen ( this space for rent )<> <> owen at uiuc.edu ( )<> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> From phillips at redhat.com Fri Dec 10 18:45:45 2004 From: phillips at redhat.com (Daniel Phillips) Date: Fri, 10 Dec 2004 13:45:45 -0500 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <20041210091453.GD14897@tykepenguin.com> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <200412091652.42408.phillips@redhat.com> <20041210091453.GD14897@tykepenguin.com> Message-ID: <200412101345.45229.phillips@redhat.com> Hi Patrick, On Friday 10 December 2004 04:14, Patrick Caulfield wrote: > On Thu, Dec 09, 2004 at 04:52:42PM -0500, Daniel Phillips wrote: > If you call dlm_lock() with the same parameters twice then you will get > two seperate locks - in that case the CR will wait for the PW to get out > of the way. What is really confusing the issue is that both locks are > sharing a lock status block - so it's possible that one lock is being > marked INPROGRESS immediately after the first has completed, in that case > the library will have a lot of difficulty in digging you out of your own > hole I'm afraid as it can't disentangle the status of your two locks. For robustness, dlm_unlock(_wait) ought to return an error in that case. But are these locks supposed to be recursive or not? Regards, Daniel From kpreslan at redhat.com Fri Dec 10 19:12:15 2004 From: kpreslan at redhat.com (Ken Preslan) Date: Fri, 10 Dec 2004 13:12:15 -0600 Subject: [Linux-cluster] NFS + GFS = kernel panic In-Reply-To: <20041210181529.GA9880@iwork57.lis.uiuc.edu> References: <20041130203210.GC1163@iwork57.lis.uiuc.edu> <20041203173442.GA14481@potassium.msp.redhat.com> <20041210181529.GA9880@iwork57.lis.uiuc.edu> Message-ID: <20041210191215.GA24554@potassium.msp.redhat.com> On Fri, Dec 10, 2004 at 12:15:29PM -0600, Brynnen R Owen wrote: > A quick update from about 2 weeks ago. As of yet, I have been stuck > with NFS problems which appear to be related to the 2.6.9 vanilla > kernel. I'm working on getting a kernel that works with NFS and ext3 > before completing the GFS tests. Is there a problem with applying the > 00006.patch for kernel oopses to other kernels than 2.6.9? Derek was able to reproduce your problem with his own tests. It should be fixed in CVS now. Let us know if you see more problems when you get your test setup. I'm pretty sure you should be able to apply 00006.patch to any 2.6-based kernel. -- Ken Preslan From tinuscubensis at gmail.com Fri Dec 10 20:39:50 2004 From: tinuscubensis at gmail.com (fausto) Date: Fri, 10 Dec 2004 15:39:50 -0500 Subject: [Linux-cluster] innactive nodes after cluster installation Message-ID: <1afa7d8c04121012392ff9cf8b@mail.gmail.com> Hi there, We are trying to setup a two-nodes cluster using clumanager. After add the two member to the cluster and create the rawdevices, all the member are inactived, the clustat's output is: Cluster Status - mail-cluster 15:04:38 Incarnation #0 (This member is not part of the cluster quorum) Member Status ------------------ ---------- 192.168.0.151 Inactive 192.168.0.152 Inactive <-- You are here No Quorum - Service States Unknown ------------------------------------------------------------------- Both member have access to the shared raw partition in the storage, here is the clurawtest's output: Starting exercises... 1 iterations run in 0 days, 00:00:05 Write Exercise Passes: 1 * 269 successful writes Read Exercise Passes: 3 * 621 successful reads * 617 successful data integrity checks Overall result: Pass ------------------------------------------------------------------- and these are the entries in message: Dec 10 15:06:33 atmail-1 clumanager: cluquorumd startup succeeded Dec 10 15:06:33 atmail-1 cluquorumd[6347]: STONITH: No drivers configured for host '192.168.0.151'! Dec 10 15:06:33 atmail-1 cluquorumd[6347]: STONITH: Data integrity may be compromised! Dec 10 15:06:33 atmail-1 cluquorumd[6347]: STONITH: No drivers configured for host '192.168.0.152'! Dec 10 15:06:33 atmail-1 cluquorumd[6347]: STONITH: Data integrity may be compromised! At this time we don't have any service created just the two cluster's member. Any idea? Regards, fausto From rajkum2002 at rediffmail.com Fri Dec 10 21:48:26 2004 From: rajkum2002 at rediffmail.com (Raj Kumar) Date: 10 Dec 2004 21:48:26 -0000 Subject: [Linux-cluster] GFS after kernel update Message-ID: <20041210214826.3676.qmail@webmail27.rediffmail.com> Hello All, Are we required to compile and install GFS after kernel updates? If yes, is there a way to update kernels without having to install GFS after each update? Thanks! Raj -------------- next part -------------- An HTML attachment was scrubbed... URL: From owen at isrl.uiuc.edu Fri Dec 10 21:59:25 2004 From: owen at isrl.uiuc.edu (Brynnen R Owen) Date: Fri, 10 Dec 2004 15:59:25 -0600 Subject: [Linux-cluster] CVS compile with 2.6.10-rc3 Message-ID: <20041210215924.GA11520@iwork57.lis.uiuc.edu> Hi all, This may be off your radar still, but it appears that the CVS source fails to compile with vanilla 2.6.10-rc3. The smoking source file is cluster/gfs-kernel/src/gfs/quota.c: CC [M] /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.o /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.c: In function `print_quota_message': /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.c:956: warning: passing arg 3 of pointer to function makes integer from pointer without a cast /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.c:956: too many arguments to function Did the kernel API for tty access change? -- <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> <> Brynnen Owen ( this space for rent )<> <> owen at uiuc.edu ( )<> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> From kpreslan at redhat.com Fri Dec 10 22:14:12 2004 From: kpreslan at redhat.com (Ken Preslan) Date: Fri, 10 Dec 2004 16:14:12 -0600 Subject: [Linux-cluster] CVS compile with 2.6.10-rc3 In-Reply-To: <20041210215924.GA11520@iwork57.lis.uiuc.edu> References: <20041210215924.GA11520@iwork57.lis.uiuc.edu> Message-ID: <20041210221412.GA26453@potassium.msp.redhat.com> It looks like every other driver in the rc3 patch just drops the "0" argument to that function. Go ahead and try it and see what you get. On Fri, Dec 10, 2004 at 03:59:25PM -0600, Brynnen R Owen wrote: > Hi all, > > This may be off your radar still, but it appears that the CVS source > fails to compile with vanilla 2.6.10-rc3. The smoking source file is > cluster/gfs-kernel/src/gfs/quota.c: > > CC [M] /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.o > /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.c: > In function `print_quota_message': > /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.c:956: > warning: passing arg 3 of pointer to function makes integer from > pointer without a cast > /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.c:956: > too many arguments to function > > Did the kernel API for tty access change? > > -- > <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> > <> Brynnen Owen ( this space for rent )<> > <> owen at uiuc.edu ( )<> > <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -- Ken Preslan From mbrookov at mines.edu Fri Dec 10 22:16:01 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Fri, 10 Dec 2004 15:16:01 -0700 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: <20041210071745.GD8758@redhat.com> References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> Message-ID: <1102716961.9174.3.camel@merlin.Mines.EDU> On Fri, 2004-12-10 at 00:17, David Teigland wrote: > This sounds similar to a problem I have if I run fence_tool without ccsd > running. > > Check /proc/cluster/status while it's waiting to see if the cluster > actually has quorum or not. Also, I've added some extra checking and > debugging to fence_tool that should help narrow down where things are > stuck. Please update from cvs and rebuild at least the stuff in > cluster/fence; then use "fence_tool join -D". > > Usually things get stuck talking to ccs when ccs/magma libraries are out > of sync, but this case sounds different. Ok, I pulled the updates from CVS and rebuilt the code and the kernel. On node fiveoften, fence_tool printed out some errors and exited with status 1. The errors are below. On node fouroften, fence_tool did not print any messages and it did not exit. I am guessing that fence_tool did not exit because of a feature of the -D flag. Fence_tool did startup fenced on fouroften. Ccsd started up and is running on both nodes. According to /proc/cluster/status and nodes on both fouroften and fiveoften, the cluster is up and has quorum. fence_tool printed these messages on node fiveoften: + fence_tool join -D fence_tool: cannot connect to ccs -111 fence_tool: wait for quorum fence_tool: waiting for cluster quorum fence_tool: waiting for cluster quorum fence_tool: waiting for cluster quorum fence_tool: get our node name fence_tool: connect to ccs fence_tool: waiting for ccs connection -111 fence_tool: waiting for ccs connection -111 fence_tool: waiting for ccs connection -111 fence_tool: waiting for ccs connection -111 fence_tool: waiting for ccs connection -111 fence_tool: waiting for ccs connection -111 fence_tool: waiting for ccs connection -111 fence_tool: waiting for ccs connection -111 fence_tool: waiting for ccs connection -111 fence_tool: waiting for ccs connection -111 Log entries on node fiveoften: Dec 10 14:08:39 fiveoften kernel: Lock_Harness (built Dec 10 2004 09:14:45) installed Dec 10 14:08:39 fiveoften kernel: GFS (built Dec 10 2004 09:14:04) installed Dec 10 14:08:39 fiveoften kernel: CMAN (built Dec 10 2004 09:51:59) installed Dec 10 14:08:39 fiveoften kernel: NET: Registered protocol family 30 Dec 10 14:08:39 fiveoften kernel: DLM (built Dec 10 2004 09:52:25) installed Dec 10 14:08:39 fiveoften kernel: Lock_DLM (built Dec 10 2004 09:14:25) installed Dec 10 14:08:40 fiveoften kernel: CMAN: Waiting to join or form a Linux-cluster Dec 10 14:09:11 fiveoften kernel: CMAN: sending membership request Dec 10 14:09:11 fiveoften kernel: CMAN: got node fouroften Dec 10 14:09:11 fiveoften kernel: CMAN: quorum regained, resuming activity Dec 10 14:09:11 fiveoften ccsd[3391]: Cluster is not quorate. Refusing connection. Dec 10 14:09:11 fiveoften ccsd[3391]: Error while processing connect: Connection refused Dec 10 14:09:12 fiveoften ccsd[3391]: Cluster is not quorate. Refusing connection. Dec 10 14:09:12 fiveoften ccsd[3391]: Error while processing connect: Connection refused Dec 10 14:09:13 fiveoften ccsd[3391]: Cluster is not quorate. Refusing connection. Dec 10 14:09:13 fiveoften ccsd[3391]: Error while processing connect: Connection refused Dec 10 14:09:14 fiveoften ccsd[3391]: Cluster is not quorate. Refusing connection. Dec 10 14:09:14 fiveoften ccsd[3391]: Error while processing connect: Connection refused Dec 10 14:09:15 fiveoften ccsd[3391]: Cluster is not quorate. Refusing connection. Dec 10 14:09:15 fiveoften ccsd[3391]: Error while processing connect: Connection refused Dec 10 14:09:16 fiveoften ccsd[3391]: Cluster is not quorate. Refusing connection. Dec 10 14:09:16 fiveoften ccsd[3391]: Error while processing connect: Connection refused Dec 10 14:09:17 fiveoften ccsd[3391]: Cluster is not quorate. Refusing connection. Dec 10 14:09:17 fiveoften ccsd[3391]: Error while processing connect: Connection refused Dec 10 14:09:18 fiveoften ccsd[3391]: Cluster is not quorate. Refusing connection. Dec 10 14:09:18 fiveoften ccsd[3391]: Error while processing connect: Connection refused Dec 10 14:09:19 fiveoften ccsd[3391]: Cluster is not quorate. Refusing connection. Dec 10 14:09:19 fiveoften ccsd[3391]: Error while processing connect: Connection refused Dec 10 14:09:20 fiveoften ccsd[3391]: Cluster is not quorate. Refusing connection. Dec 10 14:09:20 fiveoften ccsd[3391]: Error while processing connect: Connection refused The logs stopped when fence_tool exited. On node fiveoften, /proc/cluster/status and /proc/cluster/nodes contain: [mbrookov at fiveoften ~]$ more /proc/cluster/status Protocol version: 4.0.1 Config version: 6 Cluster name: CSMTEST Cluster ID: 9374 Membership state: Cluster-Member Nodes: 2 Expected_votes: 1 Total_votes: 2 Quorum: 1 Active subsystems: 0 Node addresses: 138.67.4.25 [mbrookov at fiveoften ~]$ more /proc/cluster/nodes Node Votes Exp Sts Name 1 1 1 M fouroften 2 1 1 M fiveoften [mbrookov at fiveoften ~]$ On node fouroften, /proc/cluster/status and /proc/cluster/nodes contain: [mbrookov at fouroften ~]$ more /proc/cluster/status Protocol version: 4.0.1 Config version: 6 Cluster name: CSMTEST Cluster ID: 9374 Membership state: Cluster-Member Nodes: 2 Expected_votes: 1 Total_votes: 2 Quorum: 1 Active subsystems: 1 Node addresses: 138.67.4.24 [mbrookov at fouroften ~]$ more /proc/cluster/nodes Node Votes Exp Sts Name 1 1 1 M fouroften 2 1 1 M fiveoften Log entries on node fouroften: Dec 10 14:08:36 fouroften kernel: Lock_Harness (built Dec 10 2004 09:14:45) installed Dec 10 14:08:36 fouroften kernel: GFS (built Dec 10 2004 09:14:04) installed Dec 10 14:08:36 fouroften kernel: CMAN (built Dec 10 2004 09:51:59) installed Dec 10 14:08:36 fouroften kernel: NET: Registered protocol family 30 Dec 10 14:08:36 fouroften kernel: DLM (built Dec 10 2004 09:52:25) installed Dec 10 14:08:36 fouroften kernel: Lock_DLM (built Dec 10 2004 09:14:25) installed Dec 10 14:08:37 fouroften kernel: CMAN: Waiting to join or form a Linux-cluster Dec 10 14:09:09 fouroften kernel: CMAN: forming a new cluster Dec 10 14:09:09 fouroften kernel: CMAN: quorum regained, resuming activity Dec 10 14:09:09 fouroften kernel: CMAN: got node fiveoften /etc/cluster/cluster.conf: Both nodes are running Fedora Core 3 with the 2.6.9 kernel from kernel.org. Thanks for you time! Matt mbrookov at mines.edu From jbrassow at redhat.com Fri Dec 10 22:36:01 2004 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Fri, 10 Dec 2004 16:36:01 -0600 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: <1102716961.9174.3.camel@merlin.Mines.EDU> References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> <1102716961.9174.3.camel@merlin.Mines.EDU> Message-ID: starting ccsd with the -v flag should print to the log the plugin it is using to connect to the cluster infrastructure. Something on the order of: "Connected to cluster infrastructure via: CMAN/SM Plugin v1.1" This will at least tell you that ccsd is able to connect to the cluster manager - and therefore, should know whether the cluster is quorate or not. brassow On Dec 10, 2004, at 4:16 PM, Matthew B. Brookover wrote: > On Fri, 2004-12-10 at 00:17, David Teigland wrote: >> This sounds similar to a problem I have if I run fence_tool without >> ccsd >> running. >> >> Check /proc/cluster/status while it's waiting to see if the cluster >> actually has quorum or not. Also, I've added some extra checking and >> debugging to fence_tool that should help narrow down where things are >> stuck. Please update from cvs and rebuild at least the stuff in >> cluster/fence; then use "fence_tool join -D". >> >> Usually things get stuck talking to ccs when ccs/magma libraries are >> out >> of sync, but this case sounds different. > > Ok, I pulled the updates from CVS and rebuilt the code and the kernel. > > On node fiveoften, fence_tool printed out some errors and exited with > status 1. The errors are below. > > On node fouroften, fence_tool did not print any messages and it did > not exit. I am guessing that fence_tool did not exit because of a > feature of the -D flag. Fence_tool did startup fenced on fouroften. > > Ccsd started up and is running on both nodes. > > According to /proc/cluster/status and nodes on both fouroften and > fiveoften, the cluster is up and has quorum. > > fence_tool printed these messages on node fiveoften: > + fence_tool join -D > fence_tool: cannot connect to ccs -111 > > fence_tool: wait for quorum > fence_tool: waiting for cluster quorum > fence_tool: waiting for cluster quorum > fence_tool: waiting for cluster quorum > fence_tool: get our node name > fence_tool: connect to ccs > fence_tool: waiting for ccs connection -111 > fence_tool: waiting for ccs connection -111 > fence_tool: waiting for ccs connection -111 > fence_tool: waiting for ccs connection -111 > fence_tool: waiting for ccs connection -111 > fence_tool: waiting for ccs connection -111 > fence_tool: waiting for ccs connection -111 > fence_tool: waiting for ccs connection -111 > fence_tool: waiting for ccs connection -111 > fence_tool: waiting for ccs connection -111 > > Log entries on node fiveoften: > Dec 10 14:08:39 fiveoften kernel: Lock_Harness (built Dec 10 2004 > 09:14:45) installed > Dec 10 14:08:39 fiveoften kernel: GFS (built Dec 10 2004 > 09:14:04) > installed > Dec 10 14:08:39 fiveoften kernel: CMAN (built Dec 10 2004 > 09:51:59) installed > Dec 10 14:08:39 fiveoften kernel: NET: Registered protocol family 30 > Dec 10 14:08:39 fiveoften kernel: DLM (built Dec 10 2004 > 09:52:25) > installed > Dec 10 14:08:39 fiveoften kernel: Lock_DLM (built Dec 10 2004 09:14:25) > installed > Dec 10 14:08:40 fiveoften kernel: CMAN: Waiting to join or form a > Linux-cluster > Dec 10 14:09:11 fiveoften kernel: CMAN: sending membership request > Dec 10 14:09:11 fiveoften kernel: CMAN: got node fouroften > Dec 10 14:09:11 fiveoften kernel: CMAN: quorum regained, resuming > activity > Dec 10 14:09:11 fiveoften ccsd[3391]: Cluster is not quorate. Refusing > connection. > Dec 10 14:09:11 fiveoften ccsd[3391]: Error while processing connect: > Connection refused > Dec 10 14:09:12 fiveoften ccsd[3391]: Cluster is not quorate. Refusing > connection. > Dec 10 14:09:12 fiveoften ccsd[3391]: Error while processing connect: > Connection refused > Dec 10 14:09:13 fiveoften ccsd[3391]: Cluster is not quorate. Refusing > connection. > Dec 10 14:09:13 fiveoften ccsd[3391]: Error while processing connect: > Connection refused > Dec 10 14:09:14 fiveoften ccsd[3391]: Cluster is not quorate. Refusing > connection. > Dec 10 14:09:14 fiveoften ccsd[3391]: Error while processing connect: > Connection refused > Dec 10 14:09:15 fiveoften ccsd[3391]: Cluster is not quorate. Refusing > connection. > Dec 10 14:09:15 fiveoften ccsd[3391]: Error while processing connect: > Connection refused > Dec 10 14:09:16 fiveoften ccsd[3391]: Cluster is not quorate. Refusing > connection. > Dec 10 14:09:16 fiveoften ccsd[3391]: Error while processing connect: > Connection refused > Dec 10 14:09:17 fiveoften ccsd[3391]: Cluster is not quorate. Refusing > connection. > Dec 10 14:09:17 fiveoften ccsd[3391]: Error while processing connect: > Connection refused > Dec 10 14:09:18 fiveoften ccsd[3391]: Cluster is not quorate. Refusing > connection. > Dec 10 14:09:18 fiveoften ccsd[3391]: Error while processing connect: > Connection refused > Dec 10 14:09:19 fiveoften ccsd[3391]: Cluster is not quorate. Refusing > connection. > Dec 10 14:09:19 fiveoften ccsd[3391]: Error while processing connect: > Connection refused > Dec 10 14:09:20 fiveoften ccsd[3391]: Cluster is not quorate. Refusing > connection. > Dec 10 14:09:20 fiveoften ccsd[3391]: Error while processing connect: > Connection refused > > The logs stopped when fence_tool exited. > > On node fiveoften, /proc/cluster/status and /proc/cluster/nodes > contain: > > [mbrookov at fiveoften ~]$ more /proc/cluster/status > Protocol version: 4.0.1 > Config version: 6 > Cluster name: CSMTEST > Cluster ID: 9374 > Membership state: Cluster-Member > Nodes: 2 > Expected_votes: 1 > Total_votes: 2 > Quorum: 1 > Active subsystems: 0 > Node addresses: 138.67.4.25 > > [mbrookov at fiveoften ~]$ more /proc/cluster/nodes > Node Votes Exp Sts Name > 1 1 1 M fouroften > 2 1 1 M fiveoften > [mbrookov at fiveoften ~]$ > > On node fouroften, /proc/cluster/status and /proc/cluster/nodes > contain: > > [mbrookov at fouroften ~]$ more /proc/cluster/status > Protocol version: 4.0.1 > Config version: 6 > Cluster name: CSMTEST > Cluster ID: 9374 > Membership state: Cluster-Member > Nodes: 2 > Expected_votes: 1 > Total_votes: 2 > Quorum: 1 > Active subsystems: 1 > Node addresses: 138.67.4.24 > > [mbrookov at fouroften ~]$ more /proc/cluster/nodes > Node Votes Exp Sts Name > 1 1 1 M fouroften > 2 1 1 M fiveoften > > Log entries on node fouroften: > Dec 10 14:08:36 fouroften kernel: Lock_Harness (built Dec 10 2004 > 09:14:45) installed > Dec 10 14:08:36 fouroften kernel: GFS (built Dec 10 2004 > 09:14:04) > installed > Dec 10 14:08:36 fouroften kernel: CMAN (built Dec 10 2004 > 09:51:59) installed > Dec 10 14:08:36 fouroften kernel: NET: Registered protocol family 30 > Dec 10 14:08:36 fouroften kernel: DLM (built Dec 10 2004 > 09:52:25) > installed > Dec 10 14:08:36 fouroften kernel: Lock_DLM (built Dec 10 2004 09:14:25) > installed > Dec 10 14:08:37 fouroften kernel: CMAN: Waiting to join or form a > Linux-cluster > Dec 10 14:09:09 fouroften kernel: CMAN: forming a new cluster > Dec 10 14:09:09 fouroften kernel: CMAN: quorum regained, resuming > activity > Dec 10 14:09:09 fouroften kernel: CMAN: got node fiveoften > > > /etc/cluster/cluster.conf: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Both nodes are running Fedora Core 3 with the 2.6.9 kernel from > kernel.org. > > Thanks for you time! > > Matt > mbrookov at mines.edu > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From mbrookov at mines.edu Fri Dec 10 23:05:21 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Fri, 10 Dec 2004 16:05:21 -0700 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> <1102716961.9174.3.camel@merlin.Mines.EDU> Message-ID: <1102719921.9174.13.camel@merlin.Mines.EDU> On Fri, 2004-12-10 at 15:36, Jonathan E Brassow wrote: > starting ccsd with the -v flag should print to the log the plugin it is > using to connect to the cluster infrastructure. Something on the order > of: > "Connected to cluster infrastructure via: CMAN/SM Plugin v1.1" > > This will at least tell you that ccsd is able to connect to the cluster > manager - and therefore, should know whether the cluster is quorate or > not. > > brassow > > On Dec 10, 2004, at 4:16 PM, Matthew B. Brookover wrote: > > > On Fri, 2004-12-10 at 00:17, David Teigland wrote: > >> This sounds similar to a problem I have if I run fence_tool without > >> ccsd > >> running. > >> > >> Check /proc/cluster/status while it's waiting to see if the cluster > >> actually has quorum or not. Also, I've added some extra checking and > >> debugging to fence_tool that should help narrow down where things are > >> stuck. Please update from cvs and rebuild at least the stuff in > >> cluster/fence; then use "fence_tool join -D". > >> > >> Usually things get stuck talking to ccs when ccs/magma libraries are > >> out > >> of sync, but this case sounds different. > > I tried -v, and the cluster is quorate. Only the fencing is not starting up correctly. Here are the logs with ccsd -v: Added -v flag to ccsd, producing this log entry on fouroften: Dec 10 15:50:00 fouroften kernel: Lock_Harness (built Dec 10 2004 09:14:45) installed Dec 10 15:50:00 fouroften kernel: GFS (built Dec 10 2004 09:14:04) installed Dec 10 15:50:00 fouroften kernel: CMAN (built Dec 10 2004 09:51:59) installed Dec 10 15:50:00 fouroften kernel: NET: Registered protocol family 30 Dec 10 15:50:00 fouroften kernel: DLM (built Dec 10 2004 09:52:25) installed Dec 10 15:50:00 fouroften kernel: Lock_DLM (built Dec 10 2004 09:14:25) installed Dec 10 15:50:00 fouroften ccsd[3379]: Starting ccsd DEVEL.1102700899: Dec 10 15:50:01 fouroften ccsd[3379]: Built: Dec 10 2004 10:50:57 Dec 10 15:50:01 fouroften ccsd[3379]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Dec 10 15:50:01 fouroften ccsd[3379]: Verbose Flag:: SET Dec 10 15:50:01 fouroften ccsd[3379]: cluster.conf (cluster name = CSMTEST, version = 6) found. Dec 10 15:50:01 fouroften kernel: CMAN: Waiting to join or form a Linux-cluster Dec 10 15:50:01 fouroften crond(pam_unix)[3388]: session opened for user root by (uid=0) Dec 10 15:50:01 fouroften crond(pam_unix)[3387]: session opened for user root by (uid=0) Dec 10 15:50:02 fouroften ccsd[3379]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1 Dec 10 15:50:02 fouroften ccsd[3379]: Initial status:: Inquorate Dec 10 15:50:02 fouroften crond(pam_unix)[3388]: session closed for user root Dec 10 15:50:03 fouroften crond(pam_unix)[3387]: session closed for user root Dec 10 15:50:33 fouroften kernel: CMAN: forming a new cluster Dec 10 15:50:33 fouroften kernel: CMAN: quorum regained, resuming activity Dec 10 15:50:33 fouroften kernel: CMAN: got node fiveoften Logs on fiveoften: Dec 10 15:50:07 fiveoften kernel: Lock_Harness (built Dec 10 2004 09:14:45) installed Dec 10 15:50:07 fiveoften kernel: GFS (built Dec 10 2004 09:14:04) installed Dec 10 15:50:07 fiveoften crond(pam_unix)[3354]: session closed for user root Dec 10 15:50:07 fiveoften kernel: CMAN (built Dec 10 2004 09:51:59) installed Dec 10 15:50:07 fiveoften kernel: NET: Registered protocol family 30 Dec 10 15:50:07 fiveoften kernel: DLM (built Dec 10 2004 09:52:25) installed Dec 10 15:50:07 fiveoften kernel: Lock_DLM (built Dec 10 2004 09:14:25) installed Dec 10 15:50:08 fiveoften ccsd[3381]: Starting ccsd DEVEL.1102700899: Dec 10 15:50:08 fiveoften ccsd[3381]: Built: Dec 10 2004 10:50:57 Dec 10 15:50:08 fiveoften ccsd[3381]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Dec 10 15:50:08 fiveoften ccsd[3381]: Verbose Flag:: SET Dec 10 15:50:08 fiveoften ccsd[3381]: cluster.conf (cluster name = CSMTEST, version = 6) found. Dec 10 15:50:08 fiveoften kernel: CMAN: Waiting to join or form a Linux-cluster Dec 10 15:50:35 fiveoften kernel: CMAN: sending membership request Dec 10 15:50:35 fiveoften kernel: CMAN: got node fouroften Dec 10 15:50:35 fiveoften kernel: CMAN: quorum regained, resuming activity Dec 10 15:50:35 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:35 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:36 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:36 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:37 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:37 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:38 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:38 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:39 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:39 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:40 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:40 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:41 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:41 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:42 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:42 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:43 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:43 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:44 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:44 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:53:26 fiveoften ccsd[3381]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1 Dec 10 15:53:26 fiveoften ccsd[3381]: Initial status:: Quorate Matt mbrookov at mines.edu From jbrassow at redhat.com Fri Dec 10 23:50:41 2004 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Fri, 10 Dec 2004 17:50:41 -0600 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: <1102719921.9174.13.camel@merlin.Mines.EDU> References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> <1102716961.9174.3.camel@merlin.Mines.EDU> <1102719921.9174.13.camel@merlin.Mines.EDU> Message-ID: <4D0E2418-4B06-11D9-B3C0-000A957BB1F6@redhat.com> on fouroften, ccsd starts up and reports it's initial status (inquorate), this happens after it connects to the cluster infrastructure. A few moments later, CMAN reports that the cluster is quorate. On fiveoften, ccsd starts up, but doesn't report its initial status (or connect to the cluster infrastructure) until after the error messages - 3min and 18sec after it has started up... At this point, I'm not sure what the hold-up is. Once ccsd has reported that it is connected to the cluster infrastructure, everything is fine right? I'll try to find out what the hold up is. brassow On Dec 10, 2004, at 5:05 PM, Matthew B. Brookover wrote: > > On Fri, 2004-12-10 at 15:36, Jonathan > E Brassow wrote: >> starting ccsd with the -v flag should print to the log the plugin it >> is >> using to connect to the cluster infrastructure. Something on the >> order >> of: >> "Connected to cluster infrastructure via: CMAN/SM Plugin v1.1" >> >> This will at least tell you that ccsd is able to connect to the >> cluster >> manager - and therefore, should know whether the cluster is quorate or >> not. >> >> brassow >> >> On Dec 10, 2004, at 4:16 PM, Matthew B. Brookover wrote: >> >>> On Fri, 2004-12-10 at 00:17, David Teigland wrote: >>>> This sounds similar to a problem I have if I run fence_tool without >>>> ccsd >>>> running. >>>> >>>> Check /proc/cluster/status while it's waiting to see if the cluster >>>> actually has quorum or not. Also, I've added some extra checking >>>> and >>>> debugging to fence_tool that should help narrow down where things >>>> are >>>> stuck. Please update from cvs and rebuild at least the stuff in >>>> cluster/fence; then use "fence_tool join -D". >>>> >>>> Usually things get stuck talking to ccs when ccs/magma libraries are >>>> out >>>> of sync, but this case sounds different. >>> > > I tried -v, and the cluster is quorate. Only the fencing is not > starting up correctly. > > Here are the logs with ccsd -v: > Added -v flag to ccsd, producing this log entry on fouroften: > Dec 10 15:50:00 fouroften kernel: Lock_Harness (built Dec 10 2004 > 09:14:45) installed > Dec 10 15:50:00 fouroften kernel: GFS (built Dec 10 2004 > 09:14:04) > installed > Dec 10 15:50:00 fouroften kernel: CMAN (built Dec 10 2004 > 09:51:59) installed > Dec 10 15:50:00 fouroften kernel: NET: Registered protocol family 30 > Dec 10 15:50:00 fouroften kernel: DLM (built Dec 10 2004 > 09:52:25) > installed > Dec 10 15:50:00 fouroften kernel: Lock_DLM (built Dec 10 2004 09:14:25) > installed > Dec 10 15:50:00 fouroften ccsd[3379]: Starting ccsd DEVEL.1102700899: > Dec 10 15:50:01 fouroften ccsd[3379]: Built: Dec 10 2004 10:50:57 > Dec 10 15:50:01 fouroften ccsd[3379]: Copyright (C) Red Hat, Inc. > 2004 All rights reserved. > Dec 10 15:50:01 fouroften ccsd[3379]: Verbose Flag:: SET > Dec 10 15:50:01 fouroften ccsd[3379]: cluster.conf (cluster name = > CSMTEST, version = 6) found. > Dec 10 15:50:01 fouroften kernel: CMAN: Waiting to join or form a > Linux-cluster > Dec 10 15:50:01 fouroften crond(pam_unix)[3388]: session opened for > user > root by (uid=0) > Dec 10 15:50:01 fouroften crond(pam_unix)[3387]: session opened for > user > root by (uid=0) > Dec 10 15:50:02 fouroften ccsd[3379]: Connected to cluster > infrastruture > via: CMAN/SM Plugin v1.1 > Dec 10 15:50:02 fouroften ccsd[3379]: Initial status:: Inquorate > Dec 10 15:50:02 fouroften crond(pam_unix)[3388]: session closed for > user > root > Dec 10 15:50:03 fouroften crond(pam_unix)[3387]: session closed for > user > root > Dec 10 15:50:33 fouroften kernel: CMAN: forming a new cluster > Dec 10 15:50:33 fouroften kernel: CMAN: quorum regained, resuming > activity > Dec 10 15:50:33 fouroften kernel: CMAN: got node fiveoften > > Logs on fiveoften: > Dec 10 15:50:07 fiveoften kernel: Lock_Harness (built Dec 10 2004 > 09:14:45) installed > Dec 10 15:50:07 fiveoften kernel: GFS (built Dec 10 2004 > 09:14:04) > installed > Dec 10 15:50:07 fiveoften crond(pam_unix)[3354]: session closed for > user > root > Dec 10 15:50:07 fiveoften kernel: CMAN (built Dec 10 2004 > 09:51:59) installed > Dec 10 15:50:07 fiveoften kernel: NET: Registered protocol family 30 > Dec 10 15:50:07 fiveoften kernel: DLM (built Dec 10 2004 > 09:52:25) > installed > Dec 10 15:50:07 fiveoften kernel: Lock_DLM (built Dec 10 2004 09:14:25) > installed > Dec 10 15:50:08 fiveoften ccsd[3381]: Starting ccsd DEVEL.1102700899: > Dec 10 15:50:08 fiveoften ccsd[3381]: Built: Dec 10 2004 10:50:57 > Dec 10 15:50:08 fiveoften ccsd[3381]: Copyright (C) Red Hat, Inc. > 2004 All rights reserved. > Dec 10 15:50:08 fiveoften ccsd[3381]: Verbose Flag:: SET > Dec 10 15:50:08 fiveoften ccsd[3381]: cluster.conf (cluster name = > CSMTEST, version = 6) found. > Dec 10 15:50:08 fiveoften kernel: CMAN: Waiting to join or form a > Linux-cluster > Dec 10 15:50:35 fiveoften kernel: CMAN: sending membership request > Dec 10 15:50:35 fiveoften kernel: CMAN: got node fouroften > Dec 10 15:50:35 fiveoften kernel: CMAN: quorum regained, resuming > activity > Dec 10 15:50:35 fiveoften ccsd[3381]: Cluster is not quorate. Refusing > connection. > Dec 10 15:50:35 fiveoften ccsd[3381]: Error while processing connect: > Connection refused > Dec 10 15:50:36 fiveoften ccsd[3381]: Cluster is not quorate. Refusing > connection. > Dec 10 15:50:36 fiveoften ccsd[3381]: Error while processing connect: > Connection refused > Dec 10 15:50:37 fiveoften ccsd[3381]: Cluster is not quorate. Refusing > connection. > Dec 10 15:50:37 fiveoften ccsd[3381]: Error while processing connect: > Connection refused > Dec 10 15:50:38 fiveoften ccsd[3381]: Cluster is not quorate. Refusing > connection. > Dec 10 15:50:38 fiveoften ccsd[3381]: Error while processing connect: > Connection refused > Dec 10 15:50:39 fiveoften ccsd[3381]: Cluster is not quorate. Refusing > connection. > Dec 10 15:50:39 fiveoften ccsd[3381]: Error while processing connect: > Connection refused > Dec 10 15:50:40 fiveoften ccsd[3381]: Cluster is not quorate. Refusing > connection. > Dec 10 15:50:40 fiveoften ccsd[3381]: Error while processing connect: > Connection refused > Dec 10 15:50:41 fiveoften ccsd[3381]: Cluster is not quorate. Refusing > connection. > Dec 10 15:50:41 fiveoften ccsd[3381]: Error while processing connect: > Connection refused > Dec 10 15:50:42 fiveoften ccsd[3381]: Cluster is not quorate. Refusing > connection. > Dec 10 15:50:42 fiveoften ccsd[3381]: Error while processing connect: > Connection refused > Dec 10 15:50:43 fiveoften ccsd[3381]: Cluster is not quorate. Refusing > connection. > Dec 10 15:50:43 fiveoften ccsd[3381]: Error while processing connect: > Connection refused > Dec 10 15:50:44 fiveoften ccsd[3381]: Cluster is not quorate. Refusing > connection. > Dec 10 15:50:44 fiveoften ccsd[3381]: Error while processing connect: > Connection refused > Dec 10 15:53:26 fiveoften ccsd[3381]: Connected to cluster > infrastruture > via: CMAN/SM Plugin v1.1 > Dec 10 15:53:26 fiveoften ccsd[3381]: Initial status:: Quorate > > Matt > mbrookov at mines.edu > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From owen at isrl.uiuc.edu Sat Dec 11 00:21:54 2004 From: owen at isrl.uiuc.edu (Brynnen R Owen) Date: Fri, 10 Dec 2004 18:21:54 -0600 Subject: [Linux-cluster] CVS compile with 2.6.10-rc3 In-Reply-To: <20041210221412.GA26453@potassium.msp.redhat.com> References: <20041210215924.GA11520@iwork57.lis.uiuc.edu> <20041210221412.GA26453@potassium.msp.redhat.com> Message-ID: <20041211002154.GA11972@iwork57.lis.uiuc.edu> Looks like it compiles now, and may even be working. I'll know in a few days. > It looks like every other driver in the rc3 patch just drops the "0" > argument to that function. Go ahead and try it and see what you get. > On Fri, Dec 10, 2004 at 03:59:25PM -0600, Brynnen R Owen wrote: > > Hi all, > > > > This may be off your radar still, but it appears that the CVS source > > fails to compile with vanilla 2.6.10-rc3. The smoking source file is > > cluster/gfs-kernel/src/gfs/quota.c: > > > > CC [M] /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.o > > /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.c: > > In function `print_quota_message': > > /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.c:956: > > warning: passing arg 3 of pointer to function makes integer from > > pointer without a cast > > /mnt/install/src-2.6.10-rc3-gfs32-1/cluster/gfs-kernel/src/gfs/quota.c:956: > > too many arguments to function > > > > Did the kernel API for tty access change? -- <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> <> Brynnen Owen ( this space for rent )<> <> owen at uiuc.edu ( )<> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> From daniel at osdl.org Sat Dec 11 01:07:42 2004 From: daniel at osdl.org (Daniel McNeil) Date: Fri, 10 Dec 2004 17:07:42 -0800 Subject: [Linux-cluster] 2 node rm hang more info - dlm hang? Message-ID: <1102727261.19646.39.camel@ibm-c.pdx.osdl.net> I reran up test with my dlm patch to up_write() before the queue_ast() patch and Patrick's up_write() before remote_stage() patch. I am running with SLAB_DEBUG and I am not hitting any referencing free memory problem any more. Now it looks like I am hitting some dlm hang. This is a 2 node hang with cl030 and cl032. On cl030: rm D C1716F9C 0 31078 31054 (NOTLB) da393b0c 00000086 f5dae710 c1716f9c 000046f3 c1716f9c 00000020 00000004 f5cf2318 55a020b1 000046f3 a5871cab c188e304 00000286 ba5871ca da393b08 c1716f60 00000001 0000fde0 55a24b8b 000046f3 c1b2a090 c1b2a1f8 30313765 Call Trace: [] wait_for_completion+0xa4/0xe0 [] glock_wait_internal+0x3b/0x270 [gfs] [] gfs_glock_nq+0x86/0x130 [gfs] [] gfs_glock_nq_m+0x166/0x1a0 [gfs] [] do_strip+0x22f/0x5c0 [gfs] [] recursive_scan+0xbe/0x2a0 [gfs] [] gfs_shrink+0x3c5/0x490 [gfs] [] inode_dealloc+0x18d/0x2b0 [gfs] [] inode_dealloc_init+0x9b/0xe0 [gfs] [] gfs_unlinked_limit+0x6d/0xd0 [gfs] [] gfs_unlink+0x39/0x190 [gfs] [] vfs_unlink+0x18a/0x220 [] sys_unlink+0xf8/0x160 [] sysenter_past_esp+0x52/0x71 And cl032: rm D 00000008 0 23528 23520 (NOTLB) e81eeb0c 00000086 e81eeafc 00000008 00000001 00000000 00000008 00000004 ce619378 c024c8e2 0000000d bb775aa9 c1aa3188 00000286 ddbbad54 e81eeb08 c170ef60 00000000 000100cc 5e2b3227 00008c7f e2f85050 e2f851b8 30306539 Call Trace: [] wait_for_completion+0xa4/0xe0 [] glock_wait_internal+0x3b/0x270 [gfs] [] gfs_glock_nq+0x86/0x130 [gfs] [] gfs_glock_nq_m+0x166/0x1a0 [gfs] [] do_strip+0x22f/0x5c0 [gfs] [] recursive_scan+0xbe/0x2a0 [gfs] [] gfs_shrink+0x3c5/0x490 [gfs] [] inode_dealloc+0x18d/0x2b0 [gfs] [] inode_dealloc_init+0x9b/0xe0 [gfs] [] gfs_unlinked_limit+0x6d/0xd0 [gfs] [] gfs_unlink+0x39/0x190 [gfs] [] vfs_unlink+0x18a/0x220 [] sys_unlink+0xf8/0x160 [] sysenter_past_esp+0x52/0x71 I ran the decipher and parse scrips from gfs_tool/ directory and it looks like the problem is on cl032. Here's the parse output: cl032.ld.decipher Glock (rgrp[3], 17) gl_flags = lock[1] dirty[5] gl_count = 6 gl_state = exclusive[1] lvb_count = 1 object = yes aspace = 5 reclaim = no Request owner = none[-1] gh_state = unlocked[0] gh_flags = try[0] error = 0 gh_iflags = demote[2] alloced[4] dealloc[5] Waiter2 owner = none[-1] gh_state = unlocked[0] gh_flags = try[0] error = 0 gh_iflags = demote[2] alloced[4] dealloc[5] Waiter3 owner = 23528 gh_state = exclusive[1] gh_flags = local_excl[5] error = 0 gh_iflags = promote[1] Looking that the output from /proc/cluster/dlm_locks, this lock looks interesting: Resource d6e2a5cc (parent 00000000). Name (len=24) " 3 11" Local Copy, Master is node 3 Granted Queue 0022031d NL Master: 001b004c Conversion Queue Waiting Queue 0036020c -- (EX) Master: 00330164 LQ: 0,0x8 Is there an easy way to know which resource name matches with glock? AFAIKT, the glock is waiting for the unlock to happen. The DLM (if this is the matching dlm lock) is NL waiting to grant to EX, but it not doing it. Thoughts? Is my analysis correct? The full info is available here: http://developer.osdl.org/daniel/GFS/rm.hang.10dec2004/ Daniel PS Is this the fun part? :) From kpreslan at redhat.com Sat Dec 11 02:20:36 2004 From: kpreslan at redhat.com (Ken Preslan) Date: Fri, 10 Dec 2004 20:20:36 -0600 Subject: [Linux-cluster] 2 node rm hang more info - dlm hang? In-Reply-To: <1102727261.19646.39.camel@ibm-c.pdx.osdl.net> References: <1102727261.19646.39.camel@ibm-c.pdx.osdl.net> Message-ID: <20041211022036.GA28649@potassium.msp.redhat.com> On Fri, Dec 10, 2004 at 05:07:42PM -0800, Daniel McNeil wrote: > cl032.ld.decipher Glock (rgrp[3], 17) > gl_flags = lock[1] dirty[5] > gl_count = 6 > gl_state = exclusive[1] > lvb_count = 1 > object = yes > aspace = 5 > reclaim = no > Request > owner = none[-1] > gh_state = unlocked[0] > gh_flags = try[0] > error = 0 > gh_iflags = demote[2] alloced[4] dealloc[5] > Waiter2 > owner = none[-1] > gh_state = unlocked[0] > gh_flags = try[0] > error = 0 > gh_iflags = demote[2] alloced[4] dealloc[5] > Waiter3 > owner = 23528 > gh_state = exclusive[1] > gh_flags = local_excl[5] > error = 0 > gh_iflags = promote[1] Given the the dirty bit is still set on the glock, this is probably stuck in between the time that GFS marks request as being in progress and when it calls down into the lock module. I've looked at the code and I don't see anything obviously wrong. But... You wouldn't be able to install KDB or something that will let you get backtraces of the processes in the runnable state, would you? -- Ken Preslan From dmorgan at gmi-mr.com Sat Dec 11 06:12:29 2004 From: dmorgan at gmi-mr.com (Duncan Morgan) Date: Fri, 10 Dec 2004 22:12:29 -0800 Subject: [Linux-cluster] A very basic question Message-ID: <000301c4df48$652b1c60$2200570a@DMorganMobile> Hello, I know some might consider this a very stupid question but I will ask anyways because its answer will help me understand GFS a little bit better. If all nodes in a SAN environment to only read (no writes), would GFS actually be needed? Sorry for the question in advance :)) Duncan From dmorgan at gmi-mr.com Sat Dec 11 06:14:01 2004 From: dmorgan at gmi-mr.com (Duncan Morgan) Date: Fri, 10 Dec 2004 22:14:01 -0800 Subject: [Linux-cluster] GFS consultant? Message-ID: <000401c4df48$9baef3b0$2200570a@DMorganMobile> Hello, Does anyone know of a very good GFS consultant in the Pacific Northwest? Regards, Duncan From dmorgan at gmi-mr.com Sat Dec 11 18:27:58 2004 From: dmorgan at gmi-mr.com (Duncan Morgan) Date: Sat, 11 Dec 2004 10:27:58 -0800 Subject: [Linux-cluster] Heavy writing to GFS partition bringing down GFS cluster Message-ID: <011c01c4dfaf$23e0c360$2200570a@DMorganMobile> We are not 100% certain of this but think that very heavy reads/writes on a GFS mount elevate the load on the individual nodes considerably and result in crashing the entire (14-node) cluster. It appears that a shared PHP session ID directory is the primary cause of this. Does anyone have any experience similar to this? Thanks in advance, Duncan Morgan From teigland at redhat.com Mon Dec 13 04:56:42 2004 From: teigland at redhat.com (David Teigland) Date: Mon, 13 Dec 2004 12:56:42 +0800 Subject: [Linux-cluster] 2 node rm hang more info - dlm hang? In-Reply-To: <1102727261.19646.39.camel@ibm-c.pdx.osdl.net> References: <1102727261.19646.39.camel@ibm-c.pdx.osdl.net> Message-ID: <20041213045642.GA4191@redhat.com> > cl032.ld.decipher Glock (rgrp[3], 17) > Resource d6e2a5cc (parent 00000000). Name (len=24) " 3 11" > Local Copy, Master is node 3 > Granted Queue > 0022031d NL Master: 001b004c > Conversion Queue > Waiting Queue > 0036020c -- (EX) Master: 00330164 LQ: 0,0x8 > Is there an easy way to know which resource name matches with > glock? These are the same, gfs prints as 3,17; dlm prints in hex 3,11 > AFAIKT, the glock is waiting for the unlock to happen. > The DLM (if this is the matching dlm lock) is NL waiting > to grant to EX, but it not doing it. > > Thoughts? Is my analysis correct? cl030 already has a NL lock granted and is requesting a second lock (EX). To get the full dlm picture you need to look at 3,11 on both nodes: cl030 (nodeid 1) ---------------- Resource d6e2a5cc (parent 00000000). Name (len=24) " 3 11" Local Copy, Master is node 3 Granted Queue 0022031d NL Master: 001b004c Conversion Queue Waiting Queue 0036020c -- (EX) Master: 00330164 LQ: 0,0x8 cl032 (nodeid 3) ---------------- Resource ddac08e4 (parent 00000000). Name (len=24) " 3 11" Master Copy LVB: 01 16 19 70 00 00 10 28 00 00 42 1f 00 00 00 00 00 00 00 a1 00 00 10 8d 00 00 00 00 00 00 00 00 Granted Queue 002b014b EX 001b004c NL Remote: 1 0022031d 001d027b NL Conversion Queue Waiting Queue 003300be -- (EX) Remote: 1 002d009b LQ: 0,0x8 I don't see why the remote lock id's aren't correct for cl030's EX lock: cl030_lkid 0036020c != cl032_remote_lkid 002d009b cl032_lkid 003300be != cl030 remote_lkid 00330164 Compare with cl030's NL lock for which the lkid's are correct. For something that basic there must be something really obvious I'm missing. -- Dave Teigland From mlkb at x-tend.be Fri Dec 10 18:37:07 2004 From: mlkb at x-tend.be (Kris Buytaert) Date: Fri, 10 Dec 2004 19:37:07 +0100 Subject: [Linux-cluster] Call For Participation , Cluster room at Fosdem 2005 Message-ID: <1102703827.3342.1.camel@mine2> Call For Participation : Cluster Developers room at Fosdem 2005 The fifth Free and Open source Software Developers' European Meeting is a 2 days event, organized by volunteers, to promote the widespread use of Free and Open Source software. Taking place in the beautiful city of Brussels, FOSDEM is a free and non commercial event organized for and by the developer and user community. Its goal is to provide Free and Open Source developers and their user communities a place to: * meet with other developers and projects * be informed about the latest developments in the Free and Open Source world * attend interesting talks and presentations held project leaders and developers * promote the development and the benefits of Free and Open Source solutions. This year Fosdem will have a Developers room dedicated to Clustering. Therefore we are looking for people willing to share their experiences/knowledge with different Clustering technologies (HPC,HA,Grid etc..) either as a as talk, workhop, BOF etc. If you have ideas, for this event please let us know. Kris Buytaert buytaert at x-tend.be PS. Feel free to forward this mail to other appropriate lists/people. -- Kris Buytaert From felipemaya at admatel.com Sat Dec 11 16:07:56 2004 From: felipemaya at admatel.com (Felipe Maya) Date: Sat, 11 Dec 2004 11:07:56 -0500 Subject: [Linux-cluster] IDEA: Cluster Developers room at Fosdem 2005 Message-ID: <000a01c4df9b$a758b4f0$0100a8c0@lammt6okcai5l7> HI Kris Buytaert. I'm Colombian's Linux Developer. I have developed a system called GNEAX (http://www.admatel.com). with It software the Organizations can obtain telephonic control. About the MySQL cluster NDB: Figure out two or more PBX in severals Organizations that want integrate databases of calls and have the same information to. About Openmosix: Figure out severals computers with linux and balancing Apache loads. In General, the databases are big because contain very information about the calls. This case is EMTELSA-Manizales-Colombia that has several builds with different PBX . THANKS -------------- next part -------------- An HTML attachment was scrubbed... URL: From sunjw at onewaveinc.com Sun Dec 12 02:55:16 2004 From: sunjw at onewaveinc.com (=?gb2312?B?y++/oc6w?=) Date: Sun, 12 Dec 2004 10:55:16 +0800 Subject: [Linux-cluster] cluster is not quorate. refusing connection Message-ID: Hi,all: I got the same problem.I checked out the code from cvs in Dec 3 2004. The OS is FC3 ,with it's own kernel,I only have one node which is DELL 1850. After the cluster was builded with "cman_tool join", I got Version: 3.0.1 Config version: 1 Cluster name: alpha Cluster ID: 3169 Membership state: Cluster-Member Nodes: 1 Expected_votes: 1 Total_votes: 1 Quorum: 1 Active subsystems: 0 Node addresses: 192.168.200.71 I executed the command "fence_tool join", then , I saw the words "cluster is not quorate. refusing connection" in file /var/log/messages. Dec 11 15:28:39 OW-SRV-01 kernel: lock_harness: no version for "struct_module" found: kernel tainted. Dec 11 15:28:39 OW-SRV-01 kernel: Lock_Harness (built Dec 3 2004 15:51:08) installed Dec 11 15:28:39 OW-SRV-01 kernel: GFS (built Dec 3 2004 15:51:30) installed Dec 11 15:28:39 OW-SRV-01 kernel: CMAN (built Dec 3 2004 15:50:49) installed Dec 11 15:28:39 OW-SRV-01 kernel: NET: Registered protocol family 30 Dec 11 15:28:39 OW-SRV-01 kernel: DLM (built Dec 3 2004 15:51:02) installed Dec 11 15:28:39 OW-SRV-01 kernel: Lock_DLM (built Dec 3 2004 15:51:10) installed Dec 11 15:29:10 OW-SRV-01 kernel: CMAN: Waiting to join or form a Linux-cluster Dec 11 15:29:42 OW-SRV-01 kernel: CMAN: forming a new cluster Dec 11 15:29:42 OW-SRV-01 kernel: CMAN: quorum regained, resuming activity Dec 11 15:30:11 OW-SRV-01 ccsd[3084]: Cluster is not quorate. Refusing connection. Dec 11 15:30:11 OW-SRV-01 ccsd[3084]: Error while processing connect: Connection refused Dec 11 15:30:12 OW-SRV-01 ccsd[3084]: Cluster is not quorate. Refusing connection. Dec 11 15:30:12 OW-SRV-01 ccsd[3084]: Error while processing connect: Connection refused ... And the command was blocked there for a long time. Then I striked "CTRL + c" . the messages was also stopped. So what's the matter? From a.pugachev at pcs-net.net Mon Dec 13 15:46:13 2004 From: a.pugachev at pcs-net.net (Anatoly Pugachev) Date: Mon, 13 Dec 2004 18:46:13 +0300 Subject: [Linux-cluster] A very basic question In-Reply-To: <000301c4df48$652b1c60$2200570a@DMorganMobile> References: <000301c4df48$652b1c60$2200570a@DMorganMobile> Message-ID: <20041213154612.GE5562@proxy-ttk.pcs-net.net> On Fri, Dec 10, 2004 at 10:12:29PM -0800, Duncan Morgan wrote: > Hello, hi > If all nodes in a SAN environment to only read (no writes), would GFS > actually be needed? It depends, you should say first what is needed from SAN. Maybe any other network filesystem will fit your requirements, like NFS/SMB ? Or i'm not understood question. -- Anatoly P. Pugachev -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From a.pugachev at pcs-net.net Mon Dec 13 15:52:10 2004 From: a.pugachev at pcs-net.net (Anatoly Pugachev) Date: Mon, 13 Dec 2004 18:52:10 +0300 Subject: [Linux-cluster] GFS and vanilla kernel In-Reply-To: <20041208163233.GA32145@redhat.com> References: <20041129160248.GK2721@proxy-ttk.pcs-net.net> <20041129212849.GA2609@dragonhold.org> <20041208133312.GX2721@proxy-ttk.pcs-net.net> <20041208141909.GA6269@dragonhold.org> <1102520987.23672.12.camel@merlin.Mines.EDU> <20041208163233.GA32145@redhat.com> Message-ID: <20041213155210.GF5562@proxy-ttk.pcs-net.net> On Wed, Dec 08, 2004 at 10:32:33AM -0600, Michael Conrad Tadpol Tilstra wrote: > On Wed, Dec 08, 2004 at 08:49:47AM -0700, Matthew B. Brookover wrote: > > On Wed, 2004-12-08 at 07:19, Graham Wood wrote: > > > On Wed, Dec 08, 2004 at 04:33:12PM +0300, Anatoly Pugachev wrote: > > > > Hi > > > > help. i'm stuck on the first stage compiling kernel. > > > Ahh - my fault. I should have pointed out that patching the kernel > > > isn't necessarily going > > > to work at the moment - according to someone on the list a few weeks > > > back, the patches there > > > are not necessarily kept up to date with the code in the module form > > > > It seems that there is some conflicting information. Patching the > > kernel is necessary. Without the patches you will get errors like: > > > > FATAL: Error inserting gfs (/lib/modules/2.6.9/kernel/fs/gfs/gfs.ko): > > Unknown symbol in module, or unknown parameter (see dmesg) > > > > gfs: no version for "lm_mount" found: kernel tainted. > > > > gfs: Unknown symbol panic_on_oops" errors in the logs. > > > > The patches for the 2.6.9 kernel include the panic_on_oops symbol as > > well as others. > > There is now a patch that is required. It wasn't there before. All > this patch does is export the panic_on_oops symbol. > cluster/gfs-kernel/patches/2.6.9/00006.patch > > You can either apply the patches to the kernel, or apply 00006.patch > and build the modules from their source dirs. Both ways should work. > > I said a few weeks back that I forget to update the lock_gulm.patch when > I make changes to the module's code sometimes. Ken is much better at > keeping his patches uptodate. And in either case, you can always make > new ones with the make files in the source dirs for the modules. (peek > at the Makefile, you need to have the kernel source in the right place > for the make patches command to work.) > > Sorry for confusing things. well, patched 2.6.9 kernel successfully. But still unable to compile. Compiling kernel modules: ... CC [M] fs/gfs/glops.o CC [M] fs/gfs/inode.o fs/gfs/inode.c: In function `inode_init_and_link': fs/gfs/inode.c:1277: invalid lvalue in unary `&' fs/gfs/inode.c: In function `inode_alloc_hidden': fs/gfs/inode.c:2092: invalid lvalue in unary `&' make[2]: *** [fs/gfs/inode.o] Error 1 make[1]: *** [fs/gfs] Error 2 make: *** [fs] Error 2 [mator at Test4 v9]$ gcc --version 2.96 [mator at Test4 v9]$ cat /etc/redhat-release Red Hat Linux release 7.3 (Valhalla) [mator at Test4 v9]$ and another check on my current gentoo with same kernel sources. [mator at gentoo mator]$ gcc --version gcc (GCC) 3.3.4 20040623 (Gentoo Linux 3.3.4-r1, ssp-3.3.2-2, pie-8.7.6) Copyright (C) 2003 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. [mator at gentoo mator]$ gcc --version CC [M] fs/gfs_locking/lock_dlm/main.o In file included from fs/gfs_locking/lock_dlm/main.c:14: fs/gfs_locking/lock_dlm/lock_dlm.h:28:28: cluster/cnxman.h: No such file or directory fs/gfs_locking/lock_dlm/lock_dlm.h:29:29: cluster/service.h: No such file or directory fs/gfs_locking/lock_dlm/lock_dlm.h:30:25: cluster/dlm.h: No such file or directory In file included from fs/gfs_locking/lock_dlm/main.c:14: fs/gfs_locking/lock_dlm/lock_dlm.h:89: error: syntax error before "dlm_lockspace_t" fs/gfs_locking/lock_dlm/lock_dlm.h:89: warning: no semicolon at end of struct or union fs/gfs_locking/lock_dlm/lock_dlm.h:96: error: `complete' redeclared as different kind of symbol include/linux/completion.h:31: error: previous declaration of `complete' fs/gfs_locking/lock_dlm/lock_dlm.h:125: error: syntax error before '}' token fs/gfs_locking/lock_dlm/lock_dlm.h:175: error: field `lksb' has incomplete type fs/gfs_locking/lock_dlm/lock_dlm.h:289: warning: `struct dlm_range' declared inside parameter list fs/gfs_locking/lock_dlm/lock_dlm.h:289: warning: its scope is only this definition or declaration, which is pr obably not what you want fs/gfs_locking/lock_dlm/lock_dlm.h:290: warning: `struct dlm_range' declared inside parameter list make[3]: *** [fs/gfs_locking/lock_dlm/main.o] Error 1 make[2]: *** [fs/gfs_locking/lock_dlm] Error 2 make[1]: *** [fs/gfs_locking] Error 2 make: *** [fs] Error 2 where do i missed cluster/cnxman.h , cluster/service.h and cluster/dlm.h ? What should be installed prior to compiling patched kernel ? Actually i'm interested in possibility of running GFS over redhat 7.3. But if it really hard to do, or unmaintainable, or even impossible, let's go with more fresh distro (read compiler/glibc). Comments are greatly appreciated. -- Anatoly P. Pugachev -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From coughlan at redhat.com Mon Dec 13 15:54:31 2004 From: coughlan at redhat.com (Tom Coughlan) Date: Mon, 13 Dec 2004 10:54:31 -0500 Subject: [Linux-cluster] Hard lockups when writing a lot to GFS In-Reply-To: <41B8D241.6020309@vitalstream.com> References: <41B8D241.6020309@vitalstream.com> Message-ID: <1102953270.14284.23066.camel@bianchi.boston.redhat.com> On Thu, 2004-12-09 at 17:31, Rick Stevens wrote: > I have a two-node setup on a dual-port SCSI SAN. Note this is just > for test purposes. Part of the SAN is a GFS filesystem shared between > the two nodes. > > When we fetch content to the GFS filesystem via an rsync pull (well, > several rsync pulls) on node 1, it runs for a while then node 1 hard > locks (nothing on the console, network dies, console dies, it's frozen > solid). Try putting "nmi_watchdog=1" on the kernel command line. This will hopefully cause the hung machine to crash, producing a stack trace on the console. This may provide clues as to the cause. If you are running PowerPath, or anything like that, try a test without it. Tom From mbrookov at mines.edu Mon Dec 13 16:30:38 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Mon, 13 Dec 2004 09:30:38 -0700 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: <4D0E2418-4B06-11D9-B3C0-000A957BB1F6@redhat.com> References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> <1102716961.9174.3.camel@merlin.Mines.EDU> <1102719921.9174.13.camel@merlin.Mines.EDU> <4D0E2418-4B06-11D9-B3C0-000A957BB1F6@redhat.com> Message-ID: <1102955438.17696.48.camel@merlin.Mines.EDU> On Fri, 2004-12-10 at 16:50, Jonathan E Brassow wrote: > on fouroften, ccsd starts up and reports it's initial status > (inquorate), this happens after it connects to the cluster > infrastructure. A few moments later, CMAN reports that the cluster is > quorate. > > On fiveoften, ccsd starts up, but doesn't report its initial status (or > connect to the cluster infrastructure) until after the error messages - > 3min and 18sec after it has started up... At this point, I'm not sure > what the hold-up is. > > Once ccsd has reported that it is connected to the cluster > infrastructure, everything is fine right? Not quite right, fenced is not starting up on fiveoften. fence_tool has some extra checking and debug output from Dave Tiegland, and I believe it is not starting up when it finds an error. If I try to mount a GFS partition on fiveoften, I get a permission denied. Foruoften starts up fenced and I can mount up GFS. Clvmd will start up and seems to run fine on both hosts. The first time I activate the logical volumes, the directory in /dev/ for the volume group is not created. I have to turn the volume group off again and then turn it back on to get the directory created: [root at fiveoften mbrookov]# vgchange -aly 3 logical volume(s) in volume group "test_vg" now active [root at fiveoften mbrookov]# ls /dev/test_vg ls: /dev/test_vg: No such file or directory [root at fiveoften mbrookov]# vgchange -aln /dev/test_vg: opendir failed: No such file or directory /dev/test_vg: opendir failed: No such file or directory /dev/test_vg: opendir failed: No such file or directory 0 logical volume(s) in volume group "test_vg" now active [root at fiveoften mbrookov]# vgchange -aly 3 logical volume(s) in volume group "test_vg" now active [root at fiveoften mbrookov]# ls /dev/test_vg testext3@ test_lv@ www_lv@ [root at fiveoften mbrookov]# mount -t gfs /dev/test_vg/www_lv /mnt/www mount: permission denied [root at fiveoften mbrookov]# Fouroften has the same problem with making the volume group available, but the mount is successful. I can mount and use the file systems on fouroften only. Before the debug patch from Dave, I was able to get both hosts to mount the partition. Fencing would not work correctly because fence_tool on fiveoften was getting the connection refused. One more odd thing, fencing is only a problem on fiveoften. Fouroften has not had this problem. I tried to start fiveoften before fouroften and the fence_tool problem stayed on fiveoften. Both systems are running Fedora core 3 with the same install on both hosts, and both systems are identical 1 GHZ Pentium 3 workstations with 512MB ram sharing 2 SCSI disks on an Adaptec 2940 controller. The only difference between the 2 is the host name and IP address, and the fact that I compiled GFS and the kernel on fouroften and copied it to fiveoften. thanks for your help Matt mbrookov at mines.edu From dmorgan at gmi-mr.com Mon Dec 13 16:59:32 2004 From: dmorgan at gmi-mr.com (Duncan Morgan) Date: Mon, 13 Dec 2004 08:59:32 -0800 Subject: [Linux-cluster] A very basic question In-Reply-To: <20041213154612.GE5562@proxy-ttk.pcs-net.net> Message-ID: <004301c4e135$1e52b6b0$2200570a@DMorganMobile> Anatoly, We plan on using GFS to share a web document root (i.e.- /var/www) to a series of load-balanced web servers. The application to be run on these web servers is PHP/MySQL and is predominantly read-only for the web files (writes done mainly to the database). The biggest problem we have is the PHP session id directory which will experience heavy writes. Duncan -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Anatoly Pugachev Sent: Monday, December 13, 2004 7:46 AM To: linux-cluster at redhat.com Subject: Re: [Linux-cluster] A very basic question On Fri, Dec 10, 2004 at 10:12:29PM -0800, Duncan Morgan wrote: > Hello, hi > If all nodes in a SAN environment to only read (no writes), would GFS > actually be needed? It depends, you should say first what is needed from SAN. Maybe any other network filesystem will fit your requirements, like NFS/SMB ? Or i'm not understood question. -- Anatoly P. Pugachev From mbrookov at mines.edu Mon Dec 13 17:09:48 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Mon, 13 Dec 2004 10:09:48 -0700 Subject: [Linux-cluster] GFS and vanilla kernel In-Reply-To: <20041213155210.GF5562@proxy-ttk.pcs-net.net> References: <20041129160248.GK2721@proxy-ttk.pcs-net.net> <20041129212849.GA2609@dragonhold.org> <20041208133312.GX2721@proxy-ttk.pcs-net.net> <20041208141909.GA6269@dragonhold.org> <1102520987.23672.12.camel@merlin.Mines.EDU> <20041208163233.GA32145@redhat.com> <20041213155210.GF5562@proxy-ttk.pcs-net.net> Message-ID: <1102957788.17696.86.camel@merlin.Mines.EDU> I have had successful builds and able to bring up a cluster on Fedora core 3, GCC 3.4.2 and SUSE 9.1, GCC 3.3.3. Both distributions ship with a 2.6 kernel and I have used linux kernel 2.6.7, 2.6.8, and 2.6.9 on both distributions. I suspect that GFS can be made to run on any distribution that has the 2.6 kernel. You might try setting an include path for configure: export CFLAGS='-I//cluster/build/incdir/' I am not sure about RedHat 7.3 support for GFS. I do know that a number of packages that ship with RedHat 7.3 need updated to run a 2.6.x kernel. I doubt that these would keep the kernel from compiling. Check linux-2.6.9/documentation/changes for more info on what versions of each tool are needed for the kernel. Matt On Mon, 2004-12-13 at 08:52, Anatoly Pugachev wrote: > On Wed, Dec 08, 2004 at 10:32:33AM -0600, Michael Conrad Tadpol Tilstra wrote: > > On Wed, Dec 08, 2004 at 08:49:47AM -0700, Matthew B. Brookover wrote: > > > On Wed, 2004-12-08 at 07:19, Graham Wood wrote: > > > > On Wed, Dec 08, 2004 at 04:33:12PM +0300, Anatoly Pugachev wrote: > > > > > Hi > > > > > help. i'm stuck on the first stage compiling kernel. > > > > Ahh - my fault. I should have pointed out that patching the kernel > > > > isn't necessarily going > > > > to work at the moment - according to someone on the list a few weeks > > > > back, the patches there > > > > are not necessarily kept up to date with the code in the module form > > > > > > It seems that there is some conflicting information. Patching the > > > kernel is necessary. Without the patches you will get errors like: > > > > > > FATAL: Error inserting gfs (/lib/modules/2.6.9/kernel/fs/gfs/gfs.ko): > > > Unknown symbol in module, or unknown parameter (see dmesg) > > > > > > gfs: no version for "lm_mount" found: kernel tainted. > > > > > > gfs: Unknown symbol panic_on_oops" errors in the logs. > > > > > > The patches for the 2.6.9 kernel include the panic_on_oops symbol as > > > well as others. > > > > There is now a patch that is required. It wasn't there before. All > > this patch does is export the panic_on_oops symbol. > > cluster/gfs-kernel/patches/2.6.9/00006.patch > > > > You can either apply the patches to the kernel, or apply 00006.patch > > and build the modules from their source dirs. Both ways should work. > > > > I said a few weeks back that I forget to update the lock_gulm.patch when > > I make changes to the module's code sometimes. Ken is much better at > > keeping his patches uptodate. And in either case, you can always make > > new ones with the make files in the source dirs for the modules. (peek > > at the Makefile, you need to have the kernel source in the right place > > for the make patches command to work.) > > > > Sorry for confusing things. > > > well, patched 2.6.9 kernel successfully. But still unable to compile. > Compiling kernel modules: > > ... > CC [M] fs/gfs/glops.o > CC [M] fs/gfs/inode.o > fs/gfs/inode.c: In function `inode_init_and_link': > fs/gfs/inode.c:1277: invalid lvalue in unary `&' > fs/gfs/inode.c: In function `inode_alloc_hidden': > fs/gfs/inode.c:2092: invalid lvalue in unary `&' > make[2]: *** [fs/gfs/inode.o] Error 1 > make[1]: *** [fs/gfs] Error 2 > make: *** [fs] Error 2 > [mator at Test4 v9]$ gcc --version > 2.96 > [mator at Test4 v9]$ cat /etc/redhat-release > Red Hat Linux release 7.3 (Valhalla) > [mator at Test4 v9]$ > > > and another check on my current gentoo with same kernel sources. > > [mator at gentoo mator]$ gcc --version > gcc (GCC) 3.3.4 20040623 (Gentoo Linux 3.3.4-r1, ssp-3.3.2-2, pie-8.7.6) > Copyright (C) 2003 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > [mator at gentoo mator]$ gcc --version > CC [M] fs/gfs_locking/lock_dlm/main.o > In file included from fs/gfs_locking/lock_dlm/main.c:14: > fs/gfs_locking/lock_dlm/lock_dlm.h:28:28: cluster/cnxman.h: No such file or directory > fs/gfs_locking/lock_dlm/lock_dlm.h:29:29: cluster/service.h: No such file or directory > fs/gfs_locking/lock_dlm/lock_dlm.h:30:25: cluster/dlm.h: No such file or directory > In file included from fs/gfs_locking/lock_dlm/main.c:14: > fs/gfs_locking/lock_dlm/lock_dlm.h:89: error: syntax error before "dlm_lockspace_t" > fs/gfs_locking/lock_dlm/lock_dlm.h:89: warning: no semicolon at end of struct or union > fs/gfs_locking/lock_dlm/lock_dlm.h:96: error: `complete' redeclared as different kind of symbol > include/linux/completion.h:31: error: previous declaration of `complete' > fs/gfs_locking/lock_dlm/lock_dlm.h:125: error: syntax error before '}' token > fs/gfs_locking/lock_dlm/lock_dlm.h:175: error: field `lksb' has incomplete type > fs/gfs_locking/lock_dlm/lock_dlm.h:289: warning: `struct dlm_range' declared inside parameter list > fs/gfs_locking/lock_dlm/lock_dlm.h:289: warning: its scope is only this definition or declaration, which is pr obably not what you want > fs/gfs_locking/lock_dlm/lock_dlm.h:290: warning: `struct dlm_range' declared inside parameter list > make[3]: *** [fs/gfs_locking/lock_dlm/main.o] Error 1 > make[2]: *** [fs/gfs_locking/lock_dlm] Error 2 > make[1]: *** [fs/gfs_locking] Error 2 > make: *** [fs] Error 2 > > > where do i missed cluster/cnxman.h , cluster/service.h and cluster/dlm.h ? > > What should be installed prior to compiling patched kernel ? > > Actually i'm interested in possibility of running GFS over redhat 7.3. But if > it really hard to do, or unmaintainable, or even impossible, let's go with > more fresh distro (read compiler/glibc). > > Comments are greatly appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomc at teamics.com Mon Dec 13 17:30:53 2004 From: tomc at teamics.com (tomc at teamics.com) Date: Mon, 13 Dec 2004 11:30:53 -0600 Subject: [Linux-cluster] A very basic question In-Reply-To: <004301c4e135$1e52b6b0$2200570a@DMorganMobile> Message-ID: I have moved my php session handles into Mysql and had the Mysql clustering engine maintain the Sync among the web servers. It is not only faster, but simpler besides. tc "Duncan Morgan" Sent by: linux-cluster-bounces at redhat.com 12/13/04 10:59 AM Please respond to linux clistering To: "'linux clistering'" cc: (bcc: Tom Currie/teamics) Subject: RE: [Linux-cluster] A very basic question Anatoly, We plan on using GFS to share a web document root (i.e.- /var/www) to a series of load-balanced web servers. The application to be run on these web servers is PHP/MySQL and is predominantly read-only for the web files (writes done mainly to the database). The biggest problem we have is the PHP session id directory which will experience heavy writes. Duncan -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Anatoly Pugachev Sent: Monday, December 13, 2004 7:46 AM To: linux-cluster at redhat.com Subject: Re: [Linux-cluster] A very basic question On Fri, Dec 10, 2004 at 10:12:29PM -0800, Duncan Morgan wrote: > Hello, hi > If all nodes in a SAN environment to only read (no writes), would GFS > actually be needed? It depends, you should say first what is needed from SAN. Maybe any other network filesystem will fit your requirements, like NFS/SMB ? Or i'm not understood question. -- Anatoly P. Pugachev -- Linux-cluster mailing list Linux-cluster at redhat.com http://www.redhat.com/mailman/listinfo/linux-cluster From dmorgan at gmi-mr.com Mon Dec 13 20:56:14 2004 From: dmorgan at gmi-mr.com (Duncan Morgan) Date: Mon, 13 Dec 2004 12:56:14 -0800 Subject: [Linux-cluster] A very basic question In-Reply-To: <20041213154612.GE5562@proxy-ttk.pcs-net.net> Message-ID: <007801c4e156$2efa01f0$b508080a@DMorganMobile> Anatoly, We intend to use the SAN to share a web document root. We want to use this model because - to have one consistent data view for all web servers. We have had problems in the past with having differences on the load-balanced web servers (despite our best efforts). So with a shared web root, we can push our latest software revisions to one server only and be guaranteed a consistent view - not all web documents are required to be shared but there are some directories that must be visible across all nodes. We had considered using rsync to update these regularly but they are very large storage directories (60 GB +) and the rsync was not only slow but often raised the loads on the servers - we have tried to use NFS to share but it was far to slow and unstable - we are currently using SMB to share, but it is also slow and unstable when used in this capacity It looks like the best alternative is to use a GFS-like file system. Can you see any other alternative? Duncan -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Anatoly Pugachev Sent: Monday, December 13, 2004 7:46 AM To: linux-cluster at redhat.com Subject: Re: [Linux-cluster] A very basic question On Fri, Dec 10, 2004 at 10:12:29PM -0800, Duncan Morgan wrote: > Hello, hi > If all nodes in a SAN environment to only read (no writes), would GFS > actually be needed? It depends, you should say first what is needed from SAN. Maybe any other network filesystem will fit your requirements, like NFS/SMB ? Or i'm not understood question. -- Anatoly P. Pugachev From jbrassow at redhat.com Mon Dec 13 21:06:39 2004 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Mon, 13 Dec 2004 15:06:39 -0600 Subject: [Linux-cluster] A very basic question In-Reply-To: <000301c4df48$652b1c60$2200570a@DMorganMobile> References: <000301c4df48$652b1c60$2200570a@DMorganMobile> Message-ID: yes, you would. if not using GFS, nodes will cache the info they read and will not get updates of things that change on other nodes. I'm pretty sure it doesn't work even if you mount the file systems read-only... but I could be wrong about that. So, you would populate the storage and then mount the rest of the nodes? brassow On Dec 11, 2004, at 12:12 AM, Duncan Morgan wrote: > Hello, > > I know some might consider this a very stupid question but I will ask > anyways because its answer will help me understand GFS a little bit > better. > > If all nodes in a SAN environment to only read (no writes), would GFS > actually be needed? > > Sorry for the question in advance :)) > > Duncan > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From dmorgan at gmi-mr.com Mon Dec 13 21:11:41 2004 From: dmorgan at gmi-mr.com (Duncan Morgan) Date: Mon, 13 Dec 2004 13:11:41 -0800 Subject: [Linux-cluster] A very basic question In-Reply-To: Message-ID: <008001c4e158$5798cd60$b508080a@DMorganMobile> This even for fiber-channel SAN? -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Jonathan E Brassow Sent: Monday, December 13, 2004 1:07 PM To: linux clistering Subject: Re: [Linux-cluster] A very basic question yes, you would. if not using GFS, nodes will cache the info they read and will not get updates of things that change on other nodes. I'm pretty sure it doesn't work even if you mount the file systems read-only... but I could be wrong about that. So, you would populate the storage and then mount the rest of the nodes? brassow On Dec 11, 2004, at 12:12 AM, Duncan Morgan wrote: > Hello, > > I know some might consider this a very stupid question but I will ask > anyways because its answer will help me understand GFS a little bit > better. > > If all nodes in a SAN environment to only read (no writes), would GFS > actually be needed? > > Sorry for the question in advance :)) > > Duncan > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > -- Linux-cluster mailing list Linux-cluster at redhat.com http://www.redhat.com/mailman/listinfo/linux-cluster !DSPAM:41be0440174248364188209! From jbrassow at redhat.com Mon Dec 13 21:50:12 2004 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Mon, 13 Dec 2004 15:50:12 -0600 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: <1102955438.17696.48.camel@merlin.Mines.EDU> References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> <1102716961.9174.3.camel@merlin.Mines.EDU> <1102719921.9174.13.camel@merlin.Mines.EDU> <4D0E2418-4B06-11D9-B3C0-000A957BB1F6@redhat.com> <1102955438.17696.48.camel@merlin.Mines.EDU> Message-ID: On Dec 13, 2004, at 10:30 AM, Matthew B. Brookover wrote: > The only difference > between the 2 is the host name and IP address, and the fact that I > compiled GFS and the kernel on fouroften and copied it to fiveoften. > Is there anything listed when doing 'ls /lib/*magma*'? What do you get when you: ls `strings /usr/lib/libmagma.a | grep plugins` ? brassow From mmatus at dinha.acms.arizona.edu Mon Dec 13 22:37:56 2004 From: mmatus at dinha.acms.arizona.edu (Marcelo Matus) Date: Mon, 13 Dec 2004 15:37:56 -0700 Subject: [Linux-cluster] Heavy writing to GFS partition bringing down GFS cluster In-Reply-To: <011c01c4dfaf$23e0c360$2200570a@DMorganMobile> References: <011c01c4dfaf$23e0c360$2200570a@DMorganMobile> Message-ID: <41BE19C4.7040503@acms.arizona.edu> It seems we observed something similar in our cluster (20 nodes) when testing Bonnie in parallel ( using -s 5000, ie 5GB per node). We have Opteron CPUs, do you also use opterons? Marcelo Duncan Morgan wrote: >We are not 100% certain of this but think that very heavy reads/writes >on a GFS mount elevate the load on the individual nodes considerably and >result in crashing the entire (14-node) cluster. It appears that a >shared PHP session ID directory is the primary cause of this. > >Does anyone have any experience similar to this? > >Thanks in advance, >Duncan Morgan > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster > > From dmorgan at gmi-mr.com Mon Dec 13 22:43:58 2004 From: dmorgan at gmi-mr.com (Duncan Morgan) Date: Mon, 13 Dec 2004 14:43:58 -0800 Subject: [Linux-cluster] Heavy writing to GFS partition bringing downGFS cluster In-Reply-To: <41BE19C4.7040503@acms.arizona.edu> Message-ID: <009101c4e165$3bd5f280$7e00570a@DMorganMobile> We have dual Pentium Xeon processors. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Marcelo Matus Sent: Monday, December 13, 2004 2:38 PM To: linux clistering Subject: Re: [Linux-cluster] Heavy writing to GFS partition bringing downGFS cluster It seems we observed something similar in our cluster (20 nodes) when testing Bonnie in parallel ( using -s 5000, ie 5GB per node). We have Opteron CPUs, do you also use opterons? Marcelo Duncan Morgan wrote: >We are not 100% certain of this but think that very heavy reads/writes >on a GFS mount elevate the load on the individual nodes considerably and >result in crashing the entire (14-node) cluster. It appears that a >shared PHP session ID directory is the primary cause of this. > >Does anyone have any experience similar to this? > >Thanks in advance, >Duncan Morgan > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster > > -- Linux-cluster mailing list Linux-cluster at redhat.com http://www.redhat.com/mailman/listinfo/linux-cluster !DSPAM:41be19d1226116472770771! From mbrookov at mines.edu Mon Dec 13 23:00:26 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Mon, 13 Dec 2004 16:00:26 -0700 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> <1102716961.9174.3.camel@merlin.Mines.EDU> <1102719921.9174.13.camel@merlin.Mines.EDU> <4D0E2418-4B06-11D9-B3C0-000A957BB1F6@redhat.com> <1102955438.17696.48.camel@merlin.Mines.EDU> Message-ID: <1102978826.17696.138.camel@merlin.Mines.EDU> [root at fouroften mbrookov]# ls `strings /usr/local/GFS/usr/lib/libmagma.a | grep plugins` gulm.so* sm.so* [root at fouroften mbrookov]# strings /usr/local/GFS/usr/lib/libmagma.a | grep plugins /work/GFS-FEDORA-try5/cluster/build/lib/magma/plugins [root at fouroften mbrookov]# I have been using dlm, is gulm.so bad? Also, the directory, /work/GFS-FEDORA-try5/cluster/build/lib/magma/plugins, is the build directory and is only available on fouroften. I did not copy it over to fiveoften. I will NFS mount /work to fiveoften and retest. Could this explain why things work on fouroften and not fiveoften? FYI -- I have been using the --prefix=/usr/local/GFS option in my builds. Each time I rebuild I rm -rf /usr/local/GFS to ensure that no old copies of libraries/programs are left laying around. I did add the lib directories to /etc/ld.so.conf, the bin directories to the front of by PATH, and the man directory to /etc/man.conf. The only place I could find magma libraries was in /usr/local/GFS/usr/lib. /lib, /usr/lib, and /usr/local/GFS/lib were clean. [root at fouroften mbrookov]# ls /usr/local/GFS/usr/lib/*magma* /usr/local/GFS/usr/lib/libmagma.a /usr/local/GFS/usr/lib/libmagma_nt.so@ /usr/local/GFS/usr/lib/libmagmamsg.a /usr/local/GFS/usr/lib/libmagma_nt.so.DEVEL@ /usr/local/GFS/usr/lib/libmagmamsg.so@ /usr/local/GFS/usr/lib/libmagma_nt.so.DEVEL.1102700899 /usr/local/GFS/usr/lib/libmagmamsg.so.DEVEL@ /usr/local/GFS/usr/lib/libmagma.so@ /usr/local/GFS/usr/lib/libmagmamsg.so.DEVEL.1102700899 /usr/local/GFS/usr/lib/libmagma.so.DEVEL@ /usr/local/GFS/usr/lib/libmagma_nt.a /usr/local/GFS/usr/lib/libmagma.so.DEVEL.1102700899 /usr/local/GFS/usr/lib/magma: plugins/ I checked fiveoften, it has the same list of files as fouroften. Matt On Mon, 2004-12-13 at 14:50, Jonathan E Brassow wrote: > On Dec 13, 2004, at 10:30 AM, Matthew B. Brookover wrote: > > The only difference > > between the 2 is the host name and IP address, and the fact that I > > compiled GFS and the kernel on fouroften and copied it to fiveoften. > > > > Is there anything listed when doing 'ls /lib/*magma*'? > > What do you get when you: > ls `strings /usr/lib/libmagma.a | grep plugins` > ? > > brassow > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbrookov at mines.edu Mon Dec 13 23:23:58 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Mon, 13 Dec 2004 16:23:58 -0700 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: <1102978826.17696.138.camel@merlin.Mines.EDU> References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> <1102716961.9174.3.camel@merlin.Mines.EDU> <1102719921.9174.13.camel@merlin.Mines.EDU> <4D0E2418-4B06-11D9-B3C0-000A957BB1F6@redhat.com> <1102955438.17696.48.camel@merlin.Mines.EDU> <1102978826.17696.138.camel@merlin.Mines.EDU> Message-ID: <1102980238.17696.148.camel@merlin.Mines.EDU> Ok, starting GFS with the build directory mounted via NFS worked! fence_tool started up on both nodes fouroften and fiveoften. It looks like whatever sets the path to the plugin directory is setting the path to the build instead of the install directory. I still have to do vgchange -aly, vgchange -aln, and then vgchange -aly to get the volume groups to show up in /dev. Matt On Mon, 2004-12-13 at 16:00, Matthew B. Brookover wrote: > [root at fouroften mbrookov]# ls `strings /usr/local/GFS/usr/lib/libmagma.a | grep plugins` > gulm.so* sm.so* > [root at fouroften mbrookov]# strings /usr/local/GFS/usr/lib/libmagma.a | grep plugins > /work/GFS-FEDORA-try5/cluster/build/lib/magma/plugins > [root at fouroften mbrookov]# > > I have been using dlm, is gulm.so bad? > > Also, the directory, > /work/GFS-FEDORA-try5/cluster/build/lib/magma/plugins, is the build > directory and is only available on fouroften. I did not copy it over > to fiveoften. I will NFS mount /work to fiveoften and retest. Could > this explain why things work on fouroften and not fiveoften? > > FYI -- I have been using the --prefix=/usr/local/GFS option in my > builds. Each time I rebuild I rm -rf /usr/local/GFS to ensure that no > old copies of libraries/programs are left laying around. I did add > the lib directories to /etc/ld.so.conf, the bin directories to the > front of by PATH, and the man directory to /etc/man.conf. > > The only place I could find magma libraries was in > /usr/local/GFS/usr/lib. /lib, /usr/lib, and /usr/local/GFS/lib were > clean. > > [root at fouroften mbrookov]# ls /usr/local/GFS/usr/lib/*magma* > /usr/local/GFS/usr/lib/libmagma.a /usr/local/GFS/usr/lib/libmagma_nt.so@ > /usr/local/GFS/usr/lib/libmagmamsg.a /usr/local/GFS/usr/lib/libmagma_nt.so.DEVEL@ > /usr/local/GFS/usr/lib/libmagmamsg.so@ /usr/local/GFS/usr/lib/libmagma_nt.so.DEVEL.1102700899 > /usr/local/GFS/usr/lib/libmagmamsg.so.DEVEL@ /usr/local/GFS/usr/lib/libmagma.so@ > /usr/local/GFS/usr/lib/libmagmamsg.so.DEVEL.1102700899 /usr/local/GFS/usr/lib/libmagma.so.DEVEL@ > /usr/local/GFS/usr/lib/libmagma_nt.a /usr/local/GFS/usr/lib/libmagma.so.DEVEL.1102700899 > > /usr/local/GFS/usr/lib/magma: > plugins/ > > I checked fiveoften, it has the same list of files as fouroften. > > Matt > > On Mon, 2004-12-13 at 14:50, Jonathan E Brassow wrote: > > > On Dec 13, 2004, at 10:30 AM, Matthew B. Brookover wrote: > > > The only difference > > > between the 2 is the host name and IP address, and the fact that I > > > compiled GFS and the kernel on fouroften and copied it to fiveoften. > > > > > > > Is there anything listed when doing 'ls /lib/*magma*'? > > > > What do you get when you: > > ls `strings /usr/lib/libmagma.a | grep plugins` > > ? > > > > brassow > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > > ______________________________________________________________________ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbrassow at redhat.com Mon Dec 13 23:49:21 2004 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Mon, 13 Dec 2004 17:49:21 -0600 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: <1102980238.17696.148.camel@merlin.Mines.EDU> References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> <1102716961.9174.3.camel@merlin.Mines.EDU> <1102719921.9174.13.camel@merlin.Mines.EDU> <4D0E2418-4B06-11D9-B3C0-000A957BB1F6@redhat.com> <1102955438.17696.48.camel@merlin.Mines.EDU> <1102978826.17696.138.camel@merlin.Mines.EDU> <1102980238.17696.148.camel@merlin.Mines.EDU> Message-ID: <9CA0BE12-4D61-11D9-A86B-000A957BB1F6@redhat.com> Originally, a user had to do a ./configure; make install. This would put lib's in place for the next sub-tree, etc. Users found this annoying/non-intuitive to not have to do a regular 'make'. It appears that when the option to do just a simple 'make' was added, the problem you are seeing was over looked. The cluster/magma/lib/Makefile is setting the plugin dir, which should be the install dir, but isn't because cluster/Makefile overrides the install dir to be able to emulate the simple 'make'. Thanks for all the info, we'll get this fixed. brassow On Dec 13, 2004, at 5:23 PM, Matthew B. Brookover wrote: > Ok, starting GFS with the build directory mounted via NFS worked!? > fence_tool started up on both nodes fouroften and fiveoften.? It looks > like whatever sets the path to the plugin directory is setting the > path to the build instead of the install directory. > > I still have to do vgchange -aly, vgchange -aln, and then vgchange > -aly to get the volume groups to show up in /dev. > > Matt > > On Mon, 2004-12-13 at 16:00, Matthew B. Brookover wrote: > [root at fouroften mbrookov]# ls `strings > /usr/local/GFS/usr/lib/libmagma.a | grep plugins` > gulm.so*? sm.so* > [root at fouroften mbrookov]# strings /usr/local/GFS/usr/lib/libmagma.a | > grep plugins > /work/GFS-FEDORA-try5/cluster/build/lib/magma/plugins > [root at fouroften mbrookov]# > I have been using dlm, is gulm.so bad? > > Also, the directory, > /work/GFS-FEDORA-try5/cluster/build/lib/magma/plugins, is the build > directory and is only available on fouroften.? I did not copy it over > to fiveoften.? I will NFS mount /work to fiveoften and retest.? Could > this explain why things work on fouroften and not fiveoften? > > FYI -- I have been using the --prefix=/usr/local/GFS option in my > builds.? Each time I rebuild I rm -rf /usr/local/GFS to ensure that no > old copies of libraries/programs are left laying around.? I did add > the lib directories to /etc/ld.so.conf, the bin directories to the > front of by PATH, and the man directory to /etc/man.conf. > > The only place I could find magma libraries was in > /usr/local/GFS/usr/lib.? /lib, /usr/lib, and /usr/local/GFS/lib were > clean. > [root at fouroften mbrookov]# ls /usr/local/GFS/usr/lib/*magma* > /usr/local/GFS/usr/lib/libmagma.a?????????????????????? > /usr/local/GFS/usr/lib/libmagma_nt.so@ > /usr/local/GFS/usr/lib/libmagmamsg.a??????????????????? > /usr/local/GFS/usr/lib/libmagma_nt.so.DEVEL@ > /usr/local/GFS/usr/lib/libmagmamsg.so@????????????????? > /usr/local/GFS/usr/lib/libmagma_nt.so.DEVEL.1102700899 > /usr/local/GFS/usr/lib/libmagmamsg.so.DEVEL@??????????? > /usr/local/GFS/usr/lib/libmagma.so@ > /usr/local/GFS/usr/lib/libmagmamsg.so.DEVEL.1102700899? > /usr/local/GFS/usr/lib/libmagma.so.DEVEL@ > /usr/local/GFS/usr/lib/libmagma_nt.a??????????????????? > /usr/local/GFS/usr/lib/libmagma.so.DEVEL.1102700899 > > /usr/local/GFS/usr/lib/magma: > plugins/ > I checked fiveoften, it has the same list of files as fouroften. > > Matt > > On Mon, 2004-12-13 at 14:50, Jonathan E Brassow wrote: > On Dec 13, 2004, at 10:30 AM, Matthew B. Brookover wrote: > > The only difference > > between the 2 is the host name and IP address, and the fact that I > > compiled GFS and the kernel on fouroften and copied it to fiveoften. > > > > Is there anything listed when doing 'ls /lib/*magma*'? > > What do you get when you: > ls `strings /usr/lib/libmagma.a | grep plugins` > ? > > brassow > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 4688 bytes Desc: not available URL: From galo_4 at yahoo.com Tue Dec 14 05:46:11 2004 From: galo_4 at yahoo.com (=?iso-8859-1?q?Gabriel=20L=F3pez?=) Date: Mon, 13 Dec 2004 23:46:11 -0600 (CST) Subject: [Linux-cluster] Where to begin Message-ID: <20041214054611.65126.qmail@web10806.mail.yahoo.com> Hi list I have to configure (from scratch) a cluster for http services, so I require high availability for my apache/php applications. What do you recommend to study for this purpose? Where can I start? Thank you in advance. Galo _________________________________________________________ Do You Yahoo!? Informaci?n de Estados Unidos y Am?rica Latina, en Yahoo! Noticias. Vis?tanos en http://noticias.espanol.yahoo.com From vkarasik at nds.com Tue Dec 14 15:28:48 2004 From: vkarasik at nds.com (Karasik, Vitaly) Date: Tue, 14 Dec 2004 17:28:48 +0200 Subject: [Linux-cluster] Re: Cluster stack in RHEL 4? Message-ID: <59594F2126EF13499E4660C8E8FF189A01DB6F08@ILEX2.IL.NDS.COM> >From: Kevin Anderson >The current plan is to produce alpha level cluster RPMs against RHEL4 >beta 2 once the kernel is defined. Can anyone provide some info/whitepapers regarding new HA cluster? Date of Alpha/Beta availability? Thanks, Vitaly Karasik, RHCE *********************************************************************************** Information contained in this email message is intended only for use of the individual or entity named above. If the reader of this message is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the postmaster at nds.com and destroy the original message. *********************************************************************************** From mmiranda at americatel.com.sv Tue Dec 14 15:09:04 2004 From: mmiranda at americatel.com.sv (mmiranda at americatel.com.sv) Date: Tue, 14 Dec 2004 09:09:04 -0600 Subject: [Linux-cluster] GFS and Postgres Message-ID: <76E0DAA32C39D711B6EC0002B364A6FA03B73CF4@amsal01exc01.americatel.com.sv> Hi List, Im using linux-ha with postgres, is the traditional active/passive cluster where the services are failed over from one node to another, i mean the service is shuted down in one node and started up in the another, so far so good, but i need an active/active configuration because the next year the load in postgres with increase a lot (hundreds of additional transactions), i couldnt find any pointer on using postgres with GFS, the hardware are two HP DL380 and one HP MSA500 shared storage connected with a dual scsi controller. Is someone using a similar scenario?, if so , please share your installation/configuration or comments with us, it will be greatly appreciated, Thanks Miguel From daniel at osdl.org Tue Dec 14 16:50:48 2004 From: daniel at osdl.org (Daniel McNeil) Date: Tue, 14 Dec 2004 08:50:48 -0800 Subject: [Linux-cluster] 2 node rm hang more info - dlm hang? In-Reply-To: <20041213045642.GA4191@redhat.com> References: <1102727261.19646.39.camel@ibm-c.pdx.osdl.net> <20041213045642.GA4191@redhat.com> Message-ID: <1103043047.19781.941.camel@ibm-c.pdx.osdl.net> On Sun, 2004-12-12 at 20:56, David Teigland wrote: > > cl032.ld.decipher Glock (rgrp[3], 17) > > > Resource d6e2a5cc (parent 00000000). Name (len=24) " 3 11" > > Local Copy, Master is node 3 > > Granted Queue > > 0022031d NL Master: 001b004c > > Conversion Queue > > Waiting Queue > > 0036020c -- (EX) Master: 00330164 LQ: 0,0x8 > > > Is there an easy way to know which resource name matches with > > glock? > > These are the same, gfs prints as 3,17; dlm prints in hex 3,11 > > > > AFAIKT, the glock is waiting for the unlock to happen. > > The DLM (if this is the matching dlm lock) is NL waiting > > to grant to EX, but it not doing it. > > > > Thoughts? Is my analysis correct? > > cl030 already has a NL lock granted and is requesting a second lock (EX). > To get the full dlm picture you need to look at 3,11 on both nodes: > > cl030 (nodeid 1) > ---------------- > > Resource d6e2a5cc (parent 00000000). Name (len=24) " 3 11" > Local Copy, Master is node 3 > Granted Queue > 0022031d NL Master: 001b004c > Conversion Queue > Waiting Queue > 0036020c -- (EX) Master: 00330164 LQ: 0,0x8 > > > cl032 (nodeid 3) > ---------------- > > Resource ddac08e4 (parent 00000000). Name (len=24) " 3 11" > Master Copy > LVB: 01 16 19 70 00 00 10 28 00 00 42 1f 00 00 00 00 > 00 00 00 a1 00 00 10 8d 00 00 00 00 00 00 00 00 > Granted Queue > 002b014b EX > 001b004c NL Remote: 1 0022031d > 001d027b NL > Conversion Queue > Waiting Queue > 003300be -- (EX) Remote: 1 002d009b LQ: 0,0x8 > > > I don't see why the remote lock id's aren't correct for cl030's EX lock: > cl030_lkid 0036020c != cl032_remote_lkid 002d009b > cl032_lkid 003300be != cl030 remote_lkid 00330164 > > Compare with cl030's NL lock for which the lkid's are correct. For > something that basic there must be something really obvious I'm missing. Also, doesn't the "Granted Queue 002b014b EX" mean that someone also has the dlm lock exclusive on cl032? Daniel From daniel at osdl.org Tue Dec 14 19:18:57 2004 From: daniel at osdl.org (Daniel McNeil) Date: Tue, 14 Dec 2004 11:18:57 -0800 Subject: [Linux-cluster] manual fencing problem Message-ID: <1103051937.19781.952.camel@ibm-c.pdx.osdl.net> I was running my test last night and I got an i/o error from the disk subsystem that caused one of the nodes to panic. The other 2 nodes removed the dead node from membership, but the the fencing did not work. cl030 /var/log/messages: Dec 13 21:54:26 cl030 kernel: CMAN: no HELLO from cl032a, removing from the cluster Dec 13 21:54:27 cl030 fenced[12121]: fencing node "cl032a" Dec 13 21:54:27 cl030 fenced[12121]: fence "cl032a" failed Dec 13 21:54:28 cl030 fenced[12121]: fencing node "cl032a" Dec 13 21:54:28 cl030 fenced[12121]: fence "cl032a" failed Dec 13 21:54:29 cl030 fenced[12121]: fencing node "cl032a" This goes on all night.. cl031 /var/log/messagew: Dec 13 21:54:27 cl031 fenced[11850]: fencing deferred to 1 [root at cl030 root]# fence_ack_manual -s cl032a Warning: If the node "cl032a" has not been manually fenced (i.e. power cycled or disconnected from shared storage devices) the GFS file system may become corrupted and all its data unrecoverable! Please verify that the node shown above has been reset or disconnected from storage. Are you certain you want to continue? [yN] y can't open /tmp/fence_manual.fifo: No such file or directory I've attached my cluster.conf file. Do I have fencing set up correctly. Any ideas on why fenced is failing to fence? Thanks, Daniel -------------- next part -------------- From mbrookov at mines.edu Tue Dec 14 19:44:34 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Tue, 14 Dec 2004 12:44:34 -0700 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: <9CA0BE12-4D61-11D9-A86B-000A957BB1F6@redhat.com> References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> <1102716961.9174.3.camel@merlin.Mines.EDU> <1102719921.9174.13.camel@merlin.Mines.EDU> <4D0E2418-4B06-11D9-B3C0-000A957BB1F6@redhat.com> <1102955438.17696.48.camel@merlin.Mines.EDU> <1102978826.17696.138.camel@merlin.Mines.EDU> <1102980238.17696.148.camel@merlin.Mines.EDU> <9CA0BE12-4D61-11D9-A86B-000A957BB1F6@redhat.com> Message-ID: <1103053474.21385.0.camel@merlin.Mines.EDU> Thank you for tracking it down and working on a fix. Matt On Mon, 2004-12-13 at 16:49, Jonathan E Brassow wrote: > Originally, a user had to do a ./configure; make install. This > wouldput lib's in place for the next sub-tree, etc. Users found > thisannoying/non-intuitive to not have to do a regular 'make'. > > It appears that when the option to do just a simple 'make' was > added,the problem you are seeing was over looked. > > The cluster/magma/lib/Makefile is setting the plugin dir, which > shouldbe the install dir, but isn't because cluster/Makefile overrides > theinstall dir to be able to emulate the simple 'make'. > > Thanks for all the info, we'll get this fixed. > brassow > > On Dec 13, 2004, at 5:23 PM, Matthew B. Brookover wrote: > > > Ok, starting GFS with the build directory mounted via > NFSworked! fence_tool started up on both nodes fouroften and > fiveoften. It looks like whatever sets the path to the plugin > directory issetting the path to the build instead of the > install directory. > > I still have to do vgchange -aly, vgchange -aln, and then > vgchange-aly to get the volume groups to show up in /dev. > > Matt > > On Mon, 2004-12-13 at 16:00, Matthew B. Brookover wrote: > [root at fouroftenmbrookov]# ls `strings > /usr/local/GFS/usr/lib/libmagma.a | grepplugins` > gulm.so* sm.so* > [root at fouroften mbrookov]# strings > /usr/local/GFS/usr/lib/libmagma.a |grep plugins > /work/GFS-FEDORA-try5/cluster/build/lib/magma/plugins > [root at fouroften mbrookov]# > I havebeen using dlm, is gulm.so bad? > > Also, the > directory,/work/GFS-FEDORA-try5/cluster/build/lib/magma/plugins, is the builddirectory and is only available on fouroften. I did not copy it overto fiveoften. I will NFS mount /work to fiveoften and retest. Couldthis explain why things work on fouroften and not fiveoften? > > FYI -- I have been usingthe --prefix=/usr/local/GFS option in > my builds. Each time I rebuildI rm -rf /usr/local/GFS to > ensure that no old copies oflibraries/programs are left laying > around. I did add the libdirectories to /etc/ld.so.conf, the > bin directories to the front of byPATH, and the man directory > to /etc/man.conf. > > The only place I couldfind magma libraries was in > /usr/local/GFS/usr/lib. /lib, /usr/lib,and /usr/local/GFS/lib > were clean. > [root at fouroftenmbrookov]# ls /usr/local/GFS/usr/lib/*magma* > /usr/local/GFS/usr/lib/libmagma.a /usr/local/GFS/usr/lib/libmagma_nt.so@ > /usr/local/GFS/usr/lib/libmagmamsg.a /usr/local/GFS/usr/lib/libmagma_nt.so.DEVEL@ > /usr/local/GFS/usr/lib/libmagmamsg.so@ /usr/local/GFS/usr/lib/libmagma_nt.so.DEVEL.1102700899 > /usr/local/GFS/usr/lib/libmagmamsg.so.DEVEL@ /usr/local/GFS/usr/lib/libmagma.so@ > /usr/local/GFS/usr/lib/libmagmamsg.so.DEVEL.1102700899 /usr/local/GFS/usr/lib/libmagma.so.DEVEL@ > /usr/local/GFS/usr/lib/libmagma_nt.a /usr/local/GFS/usr/lib/libmagma.so.DEVEL.1102700899 > > /usr/local/GFS/usr/lib/magma: > plugins/ > Ichecked fiveoften, it has the same list of files as > fouroften. > > Matt > > On Mon, 2004-12-13 at14:50, Jonathan E Brassow wrote: > On Dec 13, 2004,at 10:30 AM, Matthew B. Brookover wrote: > > The only difference > > between the 2 is the host name and IP address, and the fact > that I > > compiled GFS and the kernel on fouroften and copied it to > fiveoften. > > > > Is there anything listed when doing 'ls /lib/*magma*'? > > What do you get when you: > ls `strings /usr/lib/libmagma.a | grep plugins` > ? > > brassow > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > > > > ______________________________________________________________________ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbrookov at mines.edu Tue Dec 14 19:48:14 2004 From: mbrookov at mines.edu (Matthew B. Brookover) Date: Tue, 14 Dec 2004 12:48:14 -0700 Subject: [Linux-cluster] manual fencing problem In-Reply-To: <1103051937.19781.952.camel@ibm-c.pdx.osdl.net> References: <1103051937.19781.952.camel@ibm-c.pdx.osdl.net> Message-ID: <1103053694.21385.4.camel@merlin.Mines.EDU> Try running fence_ack_manual on cl031a. I believe the fence_manual.fifo is only created on the node that succeeded in fencing the downed member. Matt mbrookov at mines.edu On Tue, 2004-12-14 at 12:18, Daniel McNeil wrote: > I was running my test last night and I got an i/o error > from the disk subsystem that caused one of the nodes to > panic. > > The other 2 nodes removed the dead node from membership, but > the the fencing did not work. > > cl030 /var/log/messages: > > Dec 13 21:54:26 cl030 kernel: CMAN: no HELLO from cl032a, removing from the cluster > Dec 13 21:54:27 cl030 fenced[12121]: fencing node "cl032a" > Dec 13 21:54:27 cl030 fenced[12121]: fence "cl032a" failed > Dec 13 21:54:28 cl030 fenced[12121]: fencing node "cl032a" > Dec 13 21:54:28 cl030 fenced[12121]: fence "cl032a" failed > Dec 13 21:54:29 cl030 fenced[12121]: fencing node "cl032a" > > This goes on all night.. > > cl031 /var/log/messagew: > Dec 13 21:54:27 cl031 fenced[11850]: fencing deferred to 1 > > [root at cl030 root]# fence_ack_manual -s cl032a > > Warning: If the node "cl032a" has not been manually fenced > (i.e. power cycled or disconnected from shared storage devices) > the GFS file system may become corrupted and all its data > unrecoverable! Please verify that the node shown above has > been reset or disconnected from storage. > > Are you certain you want to continue? [yN] y > can't open /tmp/fence_manual.fifo: No such file or directory > > I've attached my cluster.conf file. > > Do I have fencing set up correctly. Any ideas on why > fenced is failing to fence? > > Thanks, > > Daniel > > > > > ______________________________________________________________________ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbrassow at redhat.com Tue Dec 14 20:28:50 2004 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Tue, 14 Dec 2004 14:28:50 -0600 Subject: [Linux-cluster] cluster is not quorate. refusing connection In-Reply-To: <1103053474.21385.0.camel@merlin.Mines.EDU> References: <1102612121.4846.24.camel@merlin.Mines.EDU> <20041210071745.GD8758@redhat.com> <1102716961.9174.3.camel@merlin.Mines.EDU> <1102719921.9174.13.camel@merlin.Mines.EDU> <4D0E2418-4B06-11D9-B3C0-000A957BB1F6@redhat.com> <1102955438.17696.48.camel@merlin.Mines.EDU> <1102978826.17696.138.camel@merlin.Mines.EDU> <1102980238.17696.148.camel@merlin.Mines.EDU> <9CA0BE12-4D61-11D9-A86B-000A957BB1F6@redhat.com> <1103053474.21385.0.camel@merlin.Mines.EDU> Message-ID: I should have also mentioned that the short-term work-around would be to recompile with: prompt> make clean; ./configure ; make install instead of: prompt> make clean; ./configure ; make ; make install brassow On Dec 14, 2004, at 1:44 PM, Matthew B. Brookover wrote: > Thank you for tracking it down and working on a fix. > > Matt > > On Mon, 2004-12-13 at 16:49, Jonathan E Brassow wrote: > Originally, a user had to do a ./configure; make install.? This > wouldput lib's in place for the next sub-tree, etc.? Users found > thisannoying/non-intuitive to not have to do a regular 'make'. > > It appears that when the option to do just a simple 'make' was > added,the problem you are seeing was over looked. > > The cluster/magma/lib/Makefile is setting the plugin dir, which > shouldbe the install dir, but isn't because cluster/Makefile overrides > theinstall dir to be able to emulate the simple 'make'. > > Thanks for all the info, we'll get this fixed. > ? brassow > > On Dec 13, 2004, at 5:23 PM, Matthew B. Brookover wrote: > > > Ok, starting GFS with the build directory mounted via NFSworked!? > fence_tool started up on both nodes fouroften and fiveoften.?It looks > like whatever sets the path to the plugin directory issetting the path > to the build instead of the install directory. > > I still have to do vgchange -aly, vgchange -aln, and then > vgchange-aly to get the volume groups to show up in /dev. > > Matt > > On Mon, 2004-12-13 at 16:00, Matthew B. Brookover wrote: > [root at fouroftenmbrookov]# ls `strings > /usr/local/GFS/usr/lib/libmagma.a | grepplugins` > gulm.so*? sm.so* > [root at fouroften mbrookov]# strings /usr/local/GFS/usr/lib/libmagma.a > |grep plugins > /work/GFS-FEDORA-try5/cluster/build/lib/magma/plugins > [root at fouroften mbrookov]# > I havebeen using dlm, is gulm.so bad? > > Also, the > directory,/work/GFS-FEDORA-try5/cluster/build/lib/magma/plugins, is > the builddirectory and is only available on fouroften.? I did not copy > it overto fiveoften.? I will NFS mount /work to fiveoften and retest.? > Couldthis explain why things work on fouroften and not fiveoften? > > FYI -- I have been usingthe --prefix=/usr/local/GFS option in my > builds.? Each time I rebuildI rm -rf /usr/local/GFS to ensure that no > old copies oflibraries/programs are left laying around.? I did add the > libdirectories to /etc/ld.so.conf, the bin directories to the front of > byPATH, and the man directory to /etc/man.conf. > > The only place I couldfind magma libraries was in > /usr/local/GFS/usr/lib.? /lib, /usr/lib,and /usr/local/GFS/lib were > clean. > [root at fouroftenmbrookov]# ls /usr/local/GFS/usr/lib/*magma* > > /usr/local/GFS/usr/lib/libmagma.a??????????????????????/usr/local/GFS/ > usr/lib/libmagma_nt.so@ > > /usr/local/GFS/usr/lib/libmagmamsg.a???????????????????/usr/local/GFS/ > usr/lib/libmagma_nt.so.DEVEL@ > > /usr/local/GFS/usr/lib/libmagmamsg.so@?????????????????/usr/local/GFS/ > usr/lib/libmagma_nt.so.DEVEL.1102700899 > > /usr/local/GFS/usr/lib/libmagmamsg.so.DEVEL@???????????/usr/local/GFS/ > usr/lib/libmagma.so@ > > /usr/local/GFS/usr/lib/libmagmamsg.so.DEVEL.1102700899?/usr/local/GFS/ > usr/lib/libmagma.so.DEVEL@ > > /usr/local/GFS/usr/lib/libmagma_nt.a???????????????????/usr/local/GFS/ > usr/lib/libmagma.so.DEVEL.1102700899 > > /usr/local/GFS/usr/lib/magma: > plugins/ > Ichecked fiveoften, it has the same list of files as fouroften. > > Matt > > On Mon, 2004-12-13 at14:50, Jonathan E Brassow wrote: > On Dec 13, 2004,at 10:30 AM, Matthew B. Brookover wrote: > > The only difference > > between the 2 is the host name and IP address, and the fact that I > > compiled GFS and the kernel on fouroften and copied it to fiveoften. > > > > Is there anything listed when doing 'ls /lib/*magma*'? > > What do you get when you: > ls `strings /usr/lib/libmagma.a | grep plugins` > ? > > ? brassow > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 8364 bytes Desc: not available URL: From daniel at osdl.org Tue Dec 14 22:03:49 2004 From: daniel at osdl.org (Daniel McNeil) Date: Tue, 14 Dec 2004 14:03:49 -0800 Subject: [Linux-cluster] manual fencing problem In-Reply-To: <1103053694.21385.4.camel@merlin.Mines.EDU> References: <1103051937.19781.952.camel@ibm-c.pdx.osdl.net> <1103053694.21385.4.camel@merlin.Mines.EDU> Message-ID: <1103061829.19781.955.camel@ibm-c.pdx.osdl.net> It did not run there either. I did not see a /tmp/fence_manual.fifo on either node. Daniel On Tue, 2004-12-14 at 11:48, Matthew B. Brookover wrote: > Try running fence_ack_manual on cl031a. I believe the > fence_manual.fifo is only created on the node that succeeded in > fencing the downed member. > > Matt > mbrookov at mines.edu > > On Tue, 2004-12-14 at 12:18, Daniel McNeil wrote: > > I was running my test last night and I got an i/o error > > from the disk subsystem that caused one of the nodes to > > panic. > > > > The other 2 nodes removed the dead node from membership, but > > the the fencing did not work. > > > > cl030 /var/log/messages: > > > > Dec 13 21:54:26 cl030 kernel: CMAN: no HELLO from cl032a, removing from the cluster > > Dec 13 21:54:27 cl030 fenced[12121]: fencing node "cl032a" > > Dec 13 21:54:27 cl030 fenced[12121]: fence "cl032a" failed > > Dec 13 21:54:28 cl030 fenced[12121]: fencing node "cl032a" > > Dec 13 21:54:28 cl030 fenced[12121]: fence "cl032a" failed > > Dec 13 21:54:29 cl030 fenced[12121]: fencing node "cl032a" > > > > This goes on all night.. > > > > cl031 /var/log/messagew: > > Dec 13 21:54:27 cl031 fenced[11850]: fencing deferred to 1 > > > > [root at cl030 root]# fence_ack_manual -s cl032a > > > > Warning: If the node "cl032a" has not been manually fenced > > (i.e. power cycled or disconnected from shared storage devices) > > the GFS file system may become corrupted and all its data > > unrecoverable! Please verify that the node shown above has > > been reset or disconnected from storage. > > > > Are you certain you want to continue? [yN] y > > can't open /tmp/fence_manual.fifo: No such file or directory > > > > I've attached my cluster.conf file. > > > > Do I have fencing set up correctly. Any ideas on why > > fenced is failing to fence? > > > > Thanks, > > > > Daniel > > > > > > ____________________________________________________________________ > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > ______________________________________________________________________ > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From rajkum2002 at rediffmail.com Tue Dec 14 22:20:46 2004 From: rajkum2002 at rediffmail.com (Raj Kumar) Date: 14 Dec 2004 22:20:46 -0000 Subject: [Linux-cluster] fence_ilo Message-ID: <20041214222046.6946.qmail@webmail27.rediffmail.com> ? Hello: I am using GFS version 6.0.2-12 that was released on Dec 1st. It has a new fence agent fence_ilo- I/O Fencing agent for HP Integrated Lights Out card. Was anyone able to successfully use this one? I am trying simple operations like getting status using fence_ilo but it is dying with unexpected failure error message. Has anyone been able to use GFS successfully in a production environment to store critical data on a SAN storage? Please share your experience. Thank you! Raj -------------- next part -------------- An HTML attachment was scrubbed... URL: From pcaulfie at redhat.com Wed Dec 15 10:59:01 2004 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 15 Dec 2004 10:59:01 +0000 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <200412101345.45229.phillips@redhat.com> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <200412091652.42408.phillips@redhat.com> <20041210091453.GD14897@tykepenguin.com> <200412101345.45229.phillips@redhat.com> Message-ID: <20041215105901.GC27420@tykepenguin.com> On Fri, Dec 10, 2004 at 01:45:45PM -0500, Daniel Phillips wrote: > Hi Patrick, > > On Friday 10 December 2004 04:14, Patrick Caulfield wrote: > > On Thu, Dec 09, 2004 at 04:52:42PM -0500, Daniel Phillips wrote: > > If you call dlm_lock() with the same parameters twice then you will get > > two seperate locks - in that case the CR will wait for the PW to get out > > of the way. What is really confusing the issue is that both locks are > > sharing a lock status block - so it's possible that one lock is being > > marked INPROGRESS immediately after the first has completed, in that case > > the library will have a lot of difficulty in digging you out of your own > > hole I'm afraid as it can't disentangle the status of your two locks. > > For robustness, dlm_unlock(_wait) ought to return an error in that case. > > But are these locks supposed to be recursive or not? It's not a question of recursive or non-recursive locks, it's a question of the user re-using the same status block for two different locks. That is most certainly NOT supported. But if it's possible for the library code to behave better in that circumstance then I think it should. -- patrick From simon.golicnik at energetika-lj.si Wed Dec 15 06:25:10 2004 From: simon.golicnik at energetika-lj.si (Simon Golicnik) Date: Wed, 15 Dec 2004 07:25:10 +0100 Subject: [Linux-cluster] fence_ilo In-Reply-To: <20041214222046.6946.qmail@webmail27.rediffmail.com> References: <20041214222046.6946.qmail@webmail27.rediffmail.com> Message-ID: <41BFD8C6.7000908@energetika-lj.si> Raj Kumar wrote: > > Hello: > > I am using GFS version 6.0.2-12 that was released on Dec 1st. It has a > new fence agent fence_ilo- I/O Fencing agent for HP Integrated Lights > Out card. Was anyone able to successfully use this one? I am trying > simple operations like getting status using fence_ilo but it is dying > with unexpected failure error message. > Try with firmware upgrade on ILO card. I am using fence_ilo with firmware version 1.62 (08/05/2004) Simon From ikebe.takashi at lab.ntt.co.jp Wed Dec 15 07:37:43 2004 From: ikebe.takashi at lab.ntt.co.jp (Takashi Ikebe) Date: Wed, 15 Dec 2004 16:37:43 +0900 Subject: [Linux-cluster] gfs_fsck problem Message-ID: <41BFE9C7.9010209@lab.ntt.co.jp> Hi, My GFS partition recently crushed logically. The GFS partition was mounted from 2 node via FC, and there were some lock_gulmd error, then system hangs, and noway to shutdown correctly. After that I reboot systems, then try to mount GFS, however mount command is not executed at all. I also try to use gfs_fsck, however gfs_fsck can not execute with following message; [root at localhost root]# gfs_fsck /dev/pool/pool0 ATTENTION -- not doing gfs_get_meta_buffer... It seems super block is corrupt, Is there any way to recover the file system? Thanks, Takashi From jbrassow at redhat.com Wed Dec 15 16:30:09 2004 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Wed, 15 Dec 2004 10:30:09 -0600 Subject: [Linux-cluster] gfs_fsck problem In-Reply-To: <41BFE9C7.9010209@lab.ntt.co.jp> References: <41BFE9C7.9010209@lab.ntt.co.jp> Message-ID: <961CD280-4EB6-11D9-A86B-000A957BB1F6@redhat.com> Unfortunately, there are no back-ups for the superblock at present. What is the full error the fsck prints out? brassow On Dec 15, 2004, at 1:37 AM, Takashi Ikebe wrote: > Hi, > My GFS partition recently crushed logically. > The GFS partition was mounted from 2 node via FC, and there were some > lock_gulmd error, then system hangs, and noway to shutdown correctly. > > After that I reboot systems, then try to mount GFS, however mount > command is not executed at all. > I also try to use gfs_fsck, however gfs_fsck can not execute with > following message; > > [root at localhost root]# gfs_fsck /dev/pool/pool0 > ATTENTION -- not doing gfs_get_meta_buffer... > > It seems super block is corrupt, Is there any way to recover the file > system? > > Thanks, > Takashi > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From sbasto at fe.up.pt Wed Dec 15 17:25:24 2004 From: sbasto at fe.up.pt (=?ISO-8859-1?Q?S=E9rgio?= M. Basto) Date: Wed, 15 Dec 2004 17:25:24 +0000 Subject: [Linux-cluster] testing gfs on cx500 Message-ID: <1103131524.22759.34.camel@rh10.fe.up.pt> Hi, I am begging testing one cluster with 3 cpu and one storage cx 500 with GFS I put there, more than 8000 accounts and put gfs_quotas on then. After this I realize, that I have two copy of some information. that have 90G and 4000 * 1500 files about 6 000 000 files : how much time did you expect to delete it ? more than 5 hours ? What books do you advise for best admin / perform and test GFS ? thanks , -- S?rgio Monteiro Basto Tel: 22 508 1506 Ext: 1089 Unidade de Administra??o de Sistemas (UAS) Centro de Inform?tica Prof. Correia de Ara?jo (CICA) F.E.U.P. Rua Dr? Roberto Frias s/n 4200-465 Porto -------------- next part -------------- An HTML attachment was scrubbed... URL: From Vincent.Aniello at PipelineTrading.com Thu Dec 16 03:09:42 2004 From: Vincent.Aniello at PipelineTrading.com (Vincent Aniello) Date: Wed, 15 Dec 2004 22:09:42 -0500 Subject: [Linux-cluster] Failover w/GFS and IBM FAStT600 Message-ID: <834F55E6F1BE3B488AD3AFC927A0970018B003@EMAILSRV1.exad.net> I setup GFS 6.0.0-15 on two IBM xSeries 345 servers running RedHat AS 3.0 server. The plan is to add a third server in the near future. Each server has two QLogic 2312 HBAs (A and B) and are connected to two IBM SAN switches (A and B). The A HBA on each server is connected to the A SAN switch and the B HBA on each server is connected to the B SAN switch. The switches are not connected to each other. Through the SAN switches the servers connect to a IBM FAStT600 (non-turbo) storage server. The A controller on the FAStT600 connects to the A SAN switch and the B controller on the FAStT600 connects to the B SAN switch. There are three LUNs on the FAStT600. LUNs 0 and 2 prefer the A controller on the FAStT600 and LUN 1 prefers the B controller on the FAStT600. I have loaded the QLogic 7.01.01 failover driver for the HBAs on each server. Everything works great normally, but if one of the HBAs loses it's connection to the SAN (I unplug the fiber connection for testing), I start seeing ping-ponging between the A and B controllers on the FAStT600. The ping-ponging kills performance of the storage server. For example, if I unplug the fiber connection on the B HBA on one of the servers, one server is still accessing LUN 1 through the preferred path on the B HBA and the B controller on the FAStT600, but the server where the fiber was removed from the B HBA is now accessing this LUN through its A HBA and the A controller on the FAStT600. When the FAStT600 sees I/O for LUN 1 from one of the servers on its A controller it starts using the A controller for LUN 1. Then, when the FAStT600 sees I/O for LUNA 1 from the other server on its B controller it starts using the B controller for LUN 1. This results in the ping-ponging. I have been told that RDAC will solve this problem, but this isn't currently an option for my configuration because I do not have the turbo version of the FAStT600. Before I spend the considerable amount of money IBM wants for the turbo upgrade I would like to know if anyone has gotten failover to work properly in a similar configuration to what I have using a IBM FAStT600 without the turbo option. Thank you in advance for your help. --Vincent -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.m.cahill at intel.com Thu Dec 16 04:43:45 2004 From: ben.m.cahill at intel.com (Cahill, Ben M) Date: Wed, 15 Dec 2004 20:43:45 -0800 Subject: [Linux-cluster] [PATCH] Modest performance improvement Message-ID: <0604335B7764D141945E202153105960033E270D@orsmsx404.amr.corp.intel.com> Hi all, I just sent Ken a patch to modestly improve performance by storing frozen integers as needed, instead of speculatively storing entire frozen buffers, while pipelining transactions. It offers a slight performance gain (low single-digit %) when doing the tar/rm tests. I also sent him a few more comments a couple of days ago, now that I understand the transaction/logging pipeline a little better. I'm going to be "laying low" for a while, until March, while on sabbatical. Happy holidays! -- Ben -- Opinions are mine, not Intel's From amanthei at redhat.com Thu Dec 16 15:42:18 2004 From: amanthei at redhat.com (Adam Manthei) Date: Thu, 16 Dec 2004 09:42:18 -0600 Subject: [Linux-cluster] fence_ilo In-Reply-To: <20041214222046.6946.qmail@webmail27.rediffmail.com> References: <20041214222046.6946.qmail@webmail27.rediffmail.com> Message-ID: <20041216154218.GE13949@redhat.com> On Tue, Dec 14, 2004 at 10:20:46PM -0000, Raj Kumar wrote: > ? > Hello: > > I am using GFS version 6.0.2-12 that was released on Dec 1st. It has a new fence agent fence_ilo- I/O Fencing agent for HP Integrated Lights Out card. Was anyone able to successfully use this one? I am trying simple operations like getting status using fence_ilo but it is dying with unexpected failure error message. I worked for me when I wrote it. Do you have the required SSL modules installed? Perhaps if you had supplied the actual error message I could have been more helpful. -- Adam Manthei From rajkum2002 at rediffmail.com Thu Dec 16 19:14:04 2004 From: rajkum2002 at rediffmail.com (Raj Kumar) Date: 16 Dec 2004 19:14:04 -0000 Subject: [Linux-cluster] fence_ilo Message-ID: <20041216191404.22246.qmail@webmail29.rediffmail.com> Hello: Thanks for the tip! I upgraded the firmware and fence_ilo works correctly! Link for firmware download: http://h18023.www1.hp.com/support/files/lights-out/us/locate/101_5867.html Thanks again! Raj On Thu, 16 Dec 2004 Adam Manthei wrote : >On Tue, Dec 14, 2004 at 10:20:46PM -0000, Raj Kumar wrote: > > > > Hello: > > > > I am using GFS version 6.0.2-12 that was released on Dec 1st. It has a new fence agent fence_ilo- I/O Fencing agent for HP Integrated Lights Out card. Was anyone able to successfully use this one? I am trying simple operations like getting status using fence_ilo but it is dying with unexpected failure error message. > >I worked for me when I wrote it. Do you have the required SSL modules >installed? Perhaps if you had supplied the actual error message I could >have been more helpful. > >-- >Adam Manthei -------------- next part -------------- An HTML attachment was scrubbed... URL: From bujan at isqsolutions.com Thu Dec 16 20:42:21 2004 From: bujan at isqsolutions.com (Manuel Bujan) Date: Thu, 16 Dec 2004 15:42:21 -0500 Subject: [Linux-cluster] CMAN connectivity problem Message-ID: <003301c4e3af$bf4db7a0$0c9ce142@pcbujan> Hi people, We are trying to set up a two nodes cluster with the latest cvs source from RedHat and when we are going to start both nodes we get that the first node form a quorum without problems, but the second one never get connected at cman level and remain trying to connect with the following error: Dec 16 15:30:25 atmail-2 ccsd[3260]: Initial status:: Inquorate Dec 16 15:30:28 atmail-2 kernel: CMAN: sending membership request Dec 16 15:31:03 atmail-2 last message repeated 21 times Our cluster.conf file is the following: ####################################################### >From the log file in node #1 already quorated I have: Dec 16 15:29:38 atmail-1 kernel: CMAN: Waiting to join or form a Linux-cluster Dec 16 15:29:39 atmail-1 ccsd[3191]: Connected to cluster infrastruture via: CMA N/SM Plugin v1.1 Dec 16 15:29:39 atmail-1 ccsd[3191]: Initial status:: Inquorate Dec 16 15:30:10 atmail-1 kernel: CMAN: forming a new cluster Dec 16 15:30:10 atmail-1 kernel: CMAN: quorum regained, resuming activity Dec 16 15:30:10 atmail-1 ccsd[3191]: Cluster is quorate. Allowing connections. >From the second node, inquorated we have: Dec 16 15:30:22 atmail-2 ccsd[3260]: Remote copy of cluster.conf is from quorate node. Dec 16 15:30:22 atmail-2 ccsd[3260]: Local version # : 1 Dec 16 15:30:22 atmail-2 ccsd[3260]: Remote version #: 2 Dec 16 15:30:22 atmail-2 ccsd[3260]: Switching to remote copy. Dec 16 15:30:25 atmail-2 kernel: CMAN: Waiting to join or form a Linux-cluster Dec 16 15:30:25 atmail-2 ccsd[3260]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1 Dec 16 15:30:25 atmail-2 ccsd[3260]: Initial status:: Inquorate Dec 16 15:30:28 atmail-2 kernel: CMAN: sending membership request Dec 16 15:31:03 atmail-2 last message repeated 21 times Dec 16 15:32:08 atmail-2 last message repeated 39 times Dec 16 15:33:13 atmail-2 last message repeated 39 times Any hints ???? We are using kernel 2.6.9, and we have applied all the patches availables for it Regards Bujan -------------- next part -------------- An HTML attachment was scrubbed... URL: From rstevens at vitalstream.com Thu Dec 16 23:54:15 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Thu, 16 Dec 2004 15:54:15 -0800 Subject: [Linux-cluster] CMAN connectivity problem In-Reply-To: <003301c4e3af$bf4db7a0$0c9ce142@pcbujan> References: <003301c4e3af$bf4db7a0$0c9ce142@pcbujan> Message-ID: <41C22027.4050705@vitalstream.com> Manuel Bujan wrote: > Hi people, > > We are trying to set up a two nodes cluster with the latest cvs source > from RedHat and when we are going to start both nodes we get that the > first node form a quorum without problems, but the second one never get > connected at cman level and remain trying to connect with the following > error: > > Dec 16 15:30:25 atmail-2 ccsd[3260]: Initial status:: Inquorate > Dec 16 15:30:28 atmail-2 kernel: CMAN: sending membership request > Dec 16 15:31:03 atmail-2 last message repeated 21 times > Our cluster.conf file is the following: > > > > > > > > > > > > > > > ipaddr="clust > er1"/> > > > > > > > > > > ipaddr="clust > er2"/> > > > > > > > > > > > > ####################################################### > > From the log file in node #1 already quorated I have: > > Dec 16 15:29:38 atmail-1 kernel: CMAN: Waiting to join or form a > Linux-cluster > Dec 16 15:29:39 atmail-1 ccsd[3191]: Connected to cluster infrastruture > via: CMA > N/SM Plugin v1.1 > Dec 16 15:29:39 atmail-1 ccsd[3191]: Initial status:: Inquorate > Dec 16 15:30:10 atmail-1 kernel: CMAN: forming a new cluster > Dec 16 15:30:10 atmail-1 kernel: CMAN: quorum regained, resuming activity > Dec 16 15:30:10 atmail-1 ccsd[3191]: Cluster is quorate. Allowing > connections. > From the second node, inquorated we have: > > Dec 16 15:30:22 atmail-2 ccsd[3260]: Remote copy of cluster.conf is from > quorate node. > Dec 16 15:30:22 atmail-2 ccsd[3260]: Local version # : 1 > Dec 16 15:30:22 atmail-2 ccsd[3260]: Remote version #: 2 > Dec 16 15:30:22 atmail-2 ccsd[3260]: Switching to remote copy. > Dec 16 15:30:25 atmail-2 kernel: CMAN: Waiting to join or form a > Linux-cluster > Dec 16 15:30:25 atmail-2 ccsd[3260]: Connected to cluster infrastruture > via: CMAN/SM Plugin v1.1 > Dec 16 15:30:25 atmail-2 ccsd[3260]: Initial status:: Inquorate > Dec 16 15:30:28 atmail-2 kernel: CMAN: sending membership request > Dec 16 15:31:03 atmail-2 last message repeated 21 times > Dec 16 15:32:08 atmail-2 last message repeated 39 times > Dec 16 15:33:13 atmail-2 last message repeated 39 times > > Any hints ???? > > We are using kernel 2.6.9, and we have applied all the patches > availables for it Check your routes. By default, your multicast stuff will go out whatever your default route is. You may need to add a route to force the multicast out the proper NIC, e.g. route add -net 224.0.0.0 netmask 255.0.0.0 dev eth1 ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - I.R.S.: We've got what it takes to take what you've got! - ---------------------------------------------------------------------- From amanthei at redhat.com Fri Dec 17 00:01:55 2004 From: amanthei at redhat.com (Adam Manthei) Date: Thu, 16 Dec 2004 18:01:55 -0600 Subject: [Linux-cluster] fence_ilo In-Reply-To: <20041216191404.22246.qmail@webmail29.rediffmail.com> References: <20041216191404.22246.qmail@webmail29.rediffmail.com> Message-ID: <20041217000155.GH13949@redhat.com> On Thu, Dec 16, 2004 at 07:14:04PM -0000, Raj Kumar wrote: > Hello: > > Thanks for the tip! I upgraded the firmware and fence_ilo works correctly! > > Link for firmware download: > http://h18023.www1.hp.com/support/files/lights-out/us/locate/101_5867.html > Thanks again! > Raj What firmware version were you running previously? What was the error meesage that you were getting from fence_ilo? -- Adam Manthei From bujan at isqsolutions.com Fri Dec 17 00:10:47 2004 From: bujan at isqsolutions.com (Manuel Bujan) Date: Thu, 16 Dec 2004 19:10:47 -0500 Subject: [Linux-cluster] CMAN connectivity problem SOLVED References: <003301c4e3af$bf4db7a0$0c9ce142@pcbujan> <41C22027.4050705@vitalstream.com> Message-ID: <00ce01c4e3cc$dc05fcf0$0c9ce142@pcbujan> We had an IP alias for eth0, and I don't know why, but when we delete the aliased address, every thing work smoothly. Does any of you guys are working with more than one address for a physical interface ? Any experience with that configuration ? Bujan ----- Original Message ----- From: "Rick Stevens" To: "linux clistering" Sent: Thursday, December 16, 2004 6:54 PM Subject: Re: [Linux-cluster] CMAN connectivity problem > Manuel Bujan wrote: >> Hi people, >> We are trying to set up a two nodes cluster with the latest cvs source >> from RedHat and when we are going to start both nodes we get that the >> first node form a quorum without problems, but the second one never get >> connected at cman level and remain trying to connect with the following >> error: >> Dec 16 15:30:25 atmail-2 ccsd[3260]: Initial status:: Inquorate >> Dec 16 15:30:28 atmail-2 kernel: CMAN: sending membership request >> Dec 16 15:31:03 atmail-2 last message repeated 21 times >> Our cluster.conf file is the following: >> >> >> >> >> >> >> >> >> >> >> >> >> >> > ipaddr="clust >> er1"/> >> >> >> >> >> >> >> >> >> >> > ipaddr="clust >> er2"/> >> >> >> >> >> >> >> >> >> >> >> ####################################################### >> From the log file in node #1 already quorated I have: >> Dec 16 15:29:38 atmail-1 kernel: CMAN: Waiting to join or form a >> Linux-cluster >> Dec 16 15:29:39 atmail-1 ccsd[3191]: Connected to cluster infrastruture >> via: CMA >> N/SM Plugin v1.1 >> Dec 16 15:29:39 atmail-1 ccsd[3191]: Initial status:: Inquorate >> Dec 16 15:30:10 atmail-1 kernel: CMAN: forming a new cluster >> Dec 16 15:30:10 atmail-1 kernel: CMAN: quorum regained, resuming activity >> Dec 16 15:30:10 atmail-1 ccsd[3191]: Cluster is quorate. Allowing >> connections. >> From the second node, inquorated we have: >> Dec 16 15:30:22 atmail-2 ccsd[3260]: Remote copy of cluster.conf is from >> quorate node. >> Dec 16 15:30:22 atmail-2 ccsd[3260]: Local version # : 1 >> Dec 16 15:30:22 atmail-2 ccsd[3260]: Remote version #: 2 >> Dec 16 15:30:22 atmail-2 ccsd[3260]: Switching to remote copy. >> Dec 16 15:30:25 atmail-2 kernel: CMAN: Waiting to join or form a >> Linux-cluster >> Dec 16 15:30:25 atmail-2 ccsd[3260]: Connected to cluster infrastruture >> via: CMAN/SM Plugin v1.1 >> Dec 16 15:30:25 atmail-2 ccsd[3260]: Initial status:: Inquorate >> Dec 16 15:30:28 atmail-2 kernel: CMAN: sending membership request >> Dec 16 15:31:03 atmail-2 last message repeated 21 times >> Dec 16 15:32:08 atmail-2 last message repeated 39 times >> Dec 16 15:33:13 atmail-2 last message repeated 39 times >> Any hints ???? >> We are using kernel 2.6.9, and we have applied all the patches >> availables for it > > Check your routes. By default, your multicast stuff will go out > whatever your default route is. You may need to add a route to force > the multicast out the proper NIC, e.g. > > route add -net 224.0.0.0 netmask 255.0.0.0 dev eth1 > ---------------------------------------------------------------------- > - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - > - VitalStream, Inc. http://www.vitalstream.com - > - - > - I.R.S.: We've got what it takes to take what you've got! - > ---------------------------------------------------------------------- > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From daniel at osdl.org Fri Dec 17 01:18:40 2004 From: daniel at osdl.org (Daniel McNeil) Date: Thu, 16 Dec 2004 17:18:40 -0800 Subject: [Linux-cluster] DLM or SM bug after 50 hours Message-ID: <1103246319.17612.47.camel@ibm-c.pdx.osdl.net> My tests ran for 50 hours! This is a new record and is running with my up_write() before queue_ast() patch. It hit an error during a 2 node test (GFS on cl030 and cl031; cl032 was a member of the cluster, but no GFS file system mounted). On cl030 console: SM: 00000001 sm_stop: SG still joined SM: 01000410 sm_stop: SG still joined /proc/cluster/status shows cl030 is not in cluster On cl031 console: CMAN: node cl030a is not responding - removing from the cluster dlm: stripefs: recover event 6388 CMAN: node cl030a is not responding - removing from the cluster dlm: stripefs: recover event 6388 name " 5 54bdb0" flags 2 nodeid 0 ref 1 G 00240122 gr 3 rq -1 flg 0 sts 2 node 0 remid 0 lq 0,0 [60,000 lines of this] ------------[ cut here ]------------ kernel BUG at /Views/redhat-cluster/cluster/dlm-kernel/src/reccomms.c:128! invalid operand: 0000 [#1] PREEMPT SMP Modules linked in: lock_dlm dlm gfs lock_harness cman qla2200 qla2xxx dm_mod CPU: 1 EIP: 0060:[] Not tainted VLI EFLAGS: 00010286 (2.6.9) EIP is at rcom_send_message+0x193/0x250 [dlm] eax: 00000001 ebx: c27813cc ecx: c0456c0c edx: 00000286 esi: da046eb4 edi: c27812d8 ebp: da046e90 esp: da046e6c ds: 007b es: 007b ss: 0068 Process dlm_recoverd (pid: 28108, threadinfo=da046000 task=f6d656f0) Stack: f8b44904 ffffff97 f8b46c60 f8b448ed 0af345bb ffffff97 c27812d8 da046000 da046eb4 da046ee0 f8b3eff1 c27812d8 00000001 00000001 da046eb4 00000001 c181f040 00000001 00150014 00000000 01000410 00000008 01000001 c7062300 Call Trace: [] show_stack+0x7f/0xa0 [] show_registers+0x15e/0x1d0 [] die+0xfe/0x190 [] do_invalid_op+0x107/0x110 [] error_code+0x2d/0x38 [] dlm_wait_status_low+0x71/0xa0 [dlm] [] nodes_reconfig_wait+0x29/0x80 [dlm] [] ls_nodes_reconfig+0x161/0x350 [dlm] [] ls_reconfig+0x6b/0x250 [dlm] [] do_ls_recovery+0x195/0x4a0 [dlm] [] dlm_recoverd+0xf8/0x100 [dlm] [] kthread+0xba/0xc0 [] kernel_thread_helper+0x5/0x10 Code: 44 24 04 80 00 00 00 e8 dc 1c 5e c7 8b 45 f0 c7 04 24 f8 48 b4 f8 89 44 24 04 e8 c9 1c 5e c7 c7 04 24 04 49 b4 f8 e8 bd 1c 5e c7 <0f> 0b 80 00 60 6c b4 f8 c7 04 24 a0 6c b4 f8 e8 59 14 5e c7 89 <1>Unable to handle kernel paging request at virtual address 6b6b6b7b printing eip: c011967a *pde = 00000000 Oops: 0000 [#2] PREEMPT SMP Modules linked in: lock_dlm dlm gfs lock_harness cman qla2200 qla2xxx dm_mod CPU: 0 EIP: 0060:[] Not tainted VLI EFLAGS: 00010086 (2.6.9) EIP is at task_rq_lock+0x2a/0x70 eax: 6b6b6b6b ebx: c052e000 ecx: c2781350 edx: f6d656f0 esi: c0533020 edi: c052e000 ebp: eb6b8e9c esp: eb6b8e90 ds: 007b es: 007b ss: 0068 Process cman_comms (pid: 3628, threadinfo=eb6b8000 task=eb9e0910) Stack: c2781350 c27812d8 00000002 eb6b8ee4 c0119d92 f6d656f0 eb6b8ed4 0af34b37 c0456ac8 00100100 00200200 0af34b37 00000001 dead4ead 00000000 c0129790 eb9e0910 00000286 c2781350 c27812d8 00000002 eb6b8ef8 c011a02e f6d656f0 Call Trace: [] show_stack+0x7f/0xa0 [] show_registers+0x15e/0x1d0 [] die+0xfe/0x190 [] do_page_fault+0x293/0x7c1 [] error_code+0x2d/0x38 [] try_to_wake_up+0x22/0x2a0 [] wake_up_process+0x1e/0x30 [] dlm_recoverd_stop+0x48/0x6b [dlm] [] release_lockspace+0x38/0x2f0 [dlm] [] dlm_emergency_shutdown+0x4c/0x70 [dlm] [] notify_kernel_listeners+0x5a/0x90 [cman] [] node_shutdown+0x5e/0x3c0 [cman] [] cluster_kthread+0x2aa/0x350 [cman] [] kernel_thread_helper+0x5/0x10 Code: 00 55 89 e5 83 ec 0c 89 1c 24 89 74 24 04 89 7c 24 08 8b 45 0c 9c 8f 00 fa be 20 30 53 c0 bb 00 e0 52 c0 8b 55 08 89 df 8b 42 04 <8b> 40 10 8b 0c 86 01 cf 89 f8 e8 e7 2c 2c 00 8b 55 08 8b 42 04 cl032 console shows: SM: process_reply invalid id=7783 nodeid=2 CMAN: quorum lost, blocking activity The test was umounting the gfs file system on cl030 when this occurred. the gfs file system is still mounted on cl031 according to /proc/mounts. The stack trace on cl030 for umount shows: umount D 00000008 0 10862 10856 (NOTLB) e383de00 00000082 e383ddf0 00000008 00000002 e0b661e7 00000008 0000007d f71b37f8 00000001 e383dde8 c011b77b e383dde0 c0119881 eb59d8b0 e0ba257b c1716f60 00000001 00053db9 0fb9cfc1 0000a65f d678b790 d678b8f8 c1716f60 Call Trace: [] wait_for_completion+0xa4/0xe0 [] kcl_leave_service+0xfe/0x180 [cman] [] release_lockspace+0x2d6/0x2f0 [dlm] [] release_gdlm+0x1c/0x30 [lock_dlm] [] lm_dlm_unmount+0x24/0x50 [lock_dlm] [] lm_unmount+0x46/0xac [lock_harness] [] gfs_put_super+0x30f/0x3c0 [gfs] [] generic_shutdown_super+0x1b7/0x1d0 [] kill_block_super+0x1d/0x40 [] deactivate_super+0xa0/0xd0 [] sys_umount+0x3c/0xa0 [] sys_oldumount+0x19/0x20 [] sysenter_past_esp+0x52/0x71 So my guess is that the umount on cl030 caused the assert on cl031 and both nodes got kicked out of the cluster. All the data is available here: http://developer.osdl.org/daniel/GFS/panic.16dec2004/ I included /proc/cluster/dlm_debug and sm_debug (not sure what the data from those is). Thoughts? Daniel From sbasto at fe.up.pt Fri Dec 17 12:18:33 2004 From: sbasto at fe.up.pt (=?ISO-8859-1?Q?S=E9rgio?= M. Basto) Date: Fri, 17 Dec 2004 12:18:33 +0000 Subject: [Linux-cluster] =?iso-8859-1?q?Intel=AE?= Extended Memory 64 Technology (Intel =?iso-8859-1?q?=AE?= EM64T) Message-ID: <1103285913.12475.8.camel@rh10.fe.up.pt> Hi how I install my system with this arch ? what arch should I pick up to compile get full options of this tecnology ? i386, x86_64 or ia64 or none of then ? thnaks, -- S?rgio M. B. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lars.larsen at edb.com Fri Dec 17 12:46:39 2004 From: lars.larsen at edb.com (=?iso-8859-1?Q?Larsen_Lars_Asbj=F8rn?=) Date: Fri, 17 Dec 2004 13:46:39 +0100 Subject: =?iso-8859-1?Q?RE=3A_=5BLinux-cluster=5D_Intel=AEExtended_Memo?= =?iso-8859-1?Q?ry_64_Technology_=28Intel_=AE_EM64T=29?= Message-ID: Not ia64 which is Itanium2 CPUs. Probably x86_64 which should cover both AMD64 and EM64T, but sheck it to be soure. But you can of course use i686 or i386. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of S?rgio "M. Basto Sent: 17. desember 2004 13:19 To: linux clistering Subject: [Linux-cluster] Intel?Extended Memory 64 Technology (Intel ? EM64T) Hi how I install my system with this arch ? what arch should I pick up to compile get full options of this tecnology ? i386, x86_64 or ia64 or none of then ? thnaks, -- S?rgio M. B. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mtilstra at redhat.com Fri Dec 17 14:04:59 2004 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Fri, 17 Dec 2004 08:04:59 -0600 Subject: [Linux-cluster] DLM or SM bug after 50 hours In-Reply-To: <1103246319.17612.47.camel@ibm-c.pdx.osdl.net> References: <1103246319.17612.47.camel@ibm-c.pdx.osdl.net> Message-ID: <20041217140459.GB4785@redhat.com> On Thu, Dec 16, 2004 at 05:18:40PM -0800, Daniel McNeil wrote: > My tests ran for 50 hours! This is a new record and is running > with my up_write() before queue_ast() patch. > > It hit an error during a 2 node test (GFS on cl030 and cl031; > cl032 was a member of the cluster, but no GFS file system mounted). > > On cl030 console: > > SM: 00000001 sm_stop: SG still joined > SM: 01000410 sm_stop: SG still joined > > /proc/cluster/status shows cl030 is not in cluster > > On cl031 console: > > CMAN: node cl030a is not responding - removing from the cluster > dlm: stripefs: recover event 6388 > CMAN: node cl030a is not responding - removing from the cluster > dlm: stripefs: recover event 6388 > name " 5 54bdb0" flags 2 nodeid 0 ref 1 > G 00240122 gr 3 rq -1 flg 0 sts 2 node 0 remid 0 lq 0,0 > [60,000 lines of this] > ------------[ cut here ]------------ > kernel BUG at /Views/redhat-cluster/cluster/dlm-kernel/src/reccomms.c:128! > invalid operand: 0000 [#1] You should append thsi to https://bugzilla.redhat.com/beta/show_bug.cgi?id=142874 -- Michael Conrad Tadpol Tilstra Earn cash in your spare time -- blackmail your friends -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From rajkum2002 at rediffmail.com Fri Dec 17 17:51:52 2004 From: rajkum2002 at rediffmail.com (Raj Kumar) Date: 17 Dec 2004 17:51:52 -0000 Subject: [Linux-cluster] non-root access to GFS file system Message-ID: <20041217175152.27412.qmail@webmail28.rediffmail.com> Hello, By default the GFS pools mounted on to a mount point have write access only for user root. What mount option is used to enable write access to different users/group? I used chown to change the ownership permissions on a directory? If yes, then I would have to assign permissions for all mount points. Is this the recommended approach? Can someone tell how they used setfacl in their projects? Thank you, Raj -------------- next part -------------- An HTML attachment was scrubbed... URL: From yazan at ccs.com.jo Mon Dec 20 05:37:22 2004 From: yazan at ccs.com.jo (Yazan Bakheit) Date: Sun, 19 Dec 2004 21:37:22 -0800 Subject: [Linux-cluster] (no subject) Message-ID: Hi, How can I download the GFS filesystem on my redhat system can you tell me how?,and is it an rpm that i can download and begin to use with the "mount" command directly, and from where can I download it if so. Thank You. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tech.gif Type: image/gif Size: 862 bytes Desc: not available URL: From bdemulder at gmail.com Sun Dec 19 21:12:51 2004 From: bdemulder at gmail.com (Benoit de Mulder) Date: Sun, 19 Dec 2004 16:12:51 -0500 Subject: [Linux-cluster] GFS usage with MySQL Message-ID: I would like to know if there are some users of this list using GFS for mysql databases. I have some issues with the replication delay (I need a sync time less than 3s between 4 DB servers) I'm planning to use GFS (setup on multipath GNBD devices and Gigabit network ) in order to share a database among multiples MySQL server. MySQL will use external lock method with MyISAM tables and GFS will be mounted with the noatime option. I was planning to test two scenarios: - multiple write servers, all gfs volume mounted RW - one write and 3 servers with database mounted RO Does someone get an experience with a similar setup and some feedback about perfomances and reliability ? Thanks Benoit From yazan at ccs.com.jo Mon Dec 20 17:38:17 2004 From: yazan at ccs.com.jo (Yazan Bakheit) Date: Mon, 20 Dec 2004 09:38:17 -0800 Subject: [Linux-cluster] (no subject) Message-ID: hi, i really have a problem , i want to use GFS file system but i dont know from where i can start?. is it an RPM or what? , im sorry ,my question may be stupid, but im new in the field, and please just tell me from where i can start, can i download this filesystem or what? and if so how ??. Yazan Al-Sheyyab SoftWare Support Engineer Computer & Communications Systems CCS Phone # +962-6-5344088 Fax # +962-6-5346494 Mobile # +962-79-6644582 +962-77-462277 Email : yazan at ccs.com.jo -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tech.gif Type: image/gif Size: 862 bytes Desc: not available URL: From yazan at ccs.com.jo Mon Dec 20 22:19:52 2004 From: yazan at ccs.com.jo (Yazan Bakheit) Date: Mon, 20 Dec 2004 14:19:52 -0800 Subject: [Linux-cluster] (no subject) Message-ID: hiii, anyone have this kernel : kernel-smp-2.4.21-15.0.3.EL.i686.rpm i need it to complete the installation in order to save time . thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tech.gif Type: image/gif Size: 862 bytes Desc: not available URL: From laza at yu.net Mon Dec 20 12:58:27 2004 From: laza at yu.net (Lazar Obradovic) Date: Mon, 20 Dec 2004 13:58:27 +0100 Subject: [Linux-cluster] GFS usage with MySQL In-Reply-To: References: Message-ID: <1103547507.31973.21.camel@laza.eunet.yu> You should check this out: http://www.mysql.com/products/cluster/ You don't need GFS nor a shared storage for it. I'm using it for a shared Bayes database for SpamAssassin and for storing sessions for our web farm. It's still work-in-a-progress, but it works fairly good (almost no glitches :) Thinking about your idea, I'm not quite sure that it will work with MySQL after all (and, if you even do manage to make it work, I'm pretty sure it will have severe performance penalty). Note to list admin: could you please change the value of a List-Id: header field? I'm not sure that 'linux clistering' is the right name for the list ;) On Sun, 2004-12-19 at 16:12 -0500, Benoit de Mulder wrote: > I would like to know if there are some users of this list using GFS > for mysql databases. I have some issues with the replication delay (I > need a sync time less than 3s between 4 DB servers) > > I'm planning to use GFS (setup on multipath GNBD devices and Gigabit > network ) in order to share a database among multiples MySQL server. > > MySQL will use external lock method with MyISAM tables and GFS will be > mounted with the noatime option. > > I was planning to test two scenarios: > - multiple write servers, all gfs volume mounted RW > - one write and 3 servers with database mounted RO > > Does someone get an experience with a similar setup and some feedback > about perfomances and reliability ? > > Thanks > > Benoit > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -- Lazar Obradovic, System Engineer ----- laza at YU.net YUnet International http://www.EUnet.yu Dubrovacka 35/III, 11000 Belgrade Tel: +381 11 3119901; Fax: +381 11 3119901 ----- This e-mail is confidential and intended only for the recipient. Unauthorized distribution, modification or disclosure of its contents is prohibited. If you have received this e-mail in error, please notify the sender by telephone +381 11 3119901. ----- From erwan at seanodes.com Fri Dec 17 12:35:20 2004 From: erwan at seanodes.com (Velu Erwan) Date: Fri, 17 Dec 2004 13:35:20 +0100 Subject: [Linux-cluster] =?ISO-8859-1?Q?Intel=AE?= Extended Memory 64 Technology (Intel =?ISO-8859-1?Q?=AE?= EM64T) In-Reply-To: <1103285913.12475.8.camel@rh10.fe.up.pt> References: <1103285913.12475.8.camel@rh10.fe.up.pt> Message-ID: <1103286920.28255.2.camel@R1.seanodes.com> Le vendredi 17 d?cembre 2004 ? 12:18 +0000, S?rgio M. Basto a ?crit : > i386, x86_64 or ia64 or none of then ? i386 is for usual x86 systems, x86_64 for EM64T (aka noconna) or AMD64 (aka opteron, athlon 64), ia64 for Intel Itanium systems. -- Erwan Velu Consultant - Seanodes SA http://www.seanodes.com From bdemulder at gmail.com Mon Dec 20 15:25:07 2004 From: bdemulder at gmail.com (Benoit de Mulder) Date: Mon, 20 Dec 2004 10:25:07 -0500 Subject: [Linux-cluster] GFS usage with MySQL In-Reply-To: <1103547507.31973.21.camel@laza.eunet.yu> References: <1103547507.31973.21.camel@laza.eunet.yu> Message-ID: The issue I get with the mysql cluster is the size of my db and the fact that the cluster is using today a in-memory storage. I have around 10gigs of Data , so I need a cluster with 2.5 * 10 = 25 gigs of memory As the customer is using intel servers with 4 gigs of memory each, that means I will have to setup at least 8 servers. As the customer don't want to spend more money, this is not a solution. I will test and will publish the result with GFS (on debian) within the next two weeks.. Benoit On Mon, 20 Dec 2004 13:58:27 +0100, Lazar Obradovic wrote: > You should check this out: > > http://www.mysql.com/products/cluster/ > > You don't need GFS nor a shared storage for it. I'm using it for a > shared Bayes database for SpamAssassin and for storing sessions for our > web farm. It's still work-in-a-progress, but it works fairly good > (almost no glitches :) > > Thinking about your idea, I'm not quite sure that it will work with > MySQL after all (and, if you even do manage to make it work, I'm pretty > sure it will have severe performance penalty). > > Note to list admin: could you please change the value of a List-Id: > header field? I'm not sure that 'linux clistering' is the right name for > the list ;) > > > On Sun, 2004-12-19 at 16:12 -0500, Benoit de Mulder wrote: > > I would like to know if there are some users of this list using GFS > > for mysql databases. I have some issues with the replication delay (I > > need a sync time less than 3s between 4 DB servers) > > > > I'm planning to use GFS (setup on multipath GNBD devices and Gigabit > > network ) in order to share a database among multiples MySQL server. > > > > MySQL will use external lock method with MyISAM tables and GFS will be > > mounted with the noatime option. > > > > I was planning to test two scenarios: > > - multiple write servers, all gfs volume mounted RW > > - one write and 3 servers with database mounted RO > > > > Does someone get an experience with a similar setup and some feedback > > about perfomances and reliability ? > > > > Thanks > > > > Benoit > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > -- > Lazar Obradovic, System Engineer > ----- > laza at YU.net > YUnet International http://www.EUnet.yu > Dubrovacka 35/III, 11000 Belgrade > Tel: +381 11 3119901; Fax: +381 11 3119901 > ----- > This e-mail is confidential and intended only for the recipient. > Unauthorized distribution, modification or disclosure of its > contents is prohibited. If you have received this e-mail in error, > please notify the sender by telephone +381 11 3119901. > ----- > > From yazan at ccs.com.jo Tue Dec 21 22:49:25 2004 From: yazan at ccs.com.jo (Yazan Bakheit) Date: Tue, 21 Dec 2004 14:49:25 -0800 Subject: [Linux-cluster] (no subject) Message-ID: hi, how can i make a pool0.cfg file and connect it with the desired partition to use with the pool_tool utility. Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tech.gif Type: image/gif Size: 862 bytes Desc: not available URL: From rajkum2002 at rediffmail.com Mon Dec 20 16:03:00 2004 From: rajkum2002 at rediffmail.com (Raj Kumar) Date: 20 Dec 2004 16:03:00 -0000 Subject: [Linux-cluster] (no subject) Message-ID: <20041220160300.3574.qmail@webmail17.rediffmail.com> Hi, You can download the new releases of GFS at this link: http://ftp.redhat.com/pub/redhat/linux/updates/enterprise/3ES/en/RHGFS/SRPMS/ ? >kernel-smp-2.4.21-15.0.3.EL.i686.rpm When I first installed GFS I had this version of kernel but I upgraded the Kernel to the new release 2.4.21-20.0.1.ELsmp. But the src rpm GFS-6.0.2-12.src.rpm may (will) not compile with your kernel version since the kernel_patches directory has a directory called 2.4.21-20.0.1 or something similar based on kernel version. You can read the GFS admin-guide at this link: http://www.redhat.com/docs/manuals/csgfs/admin-guide/ Let us know if you have trouble compiling the src rpm and we shall help you. Raj On Tue, 21 Dec 2004 Yazan Bakheit wrote : > > >hiii, > > > anyone have this kernel : > > kernel-smp-2.4.21-15.0.3.EL.i686.rpm > > i need it to complete the installation in order to save time . > > > > thank you > > > > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel at osdl.org Tue Dec 21 18:34:41 2004 From: daniel at osdl.org (Daniel McNeil) Date: Tue, 21 Dec 2004 10:34:41 -0800 Subject: [Linux-cluster] cman bad generation number Message-ID: <1103654081.29749.17.camel@ibm-c.pdx.osdl.net> Another test run that manage 52 hours before hitting a cman bug: cl032: Dec 18 19:56:05 cl032 kernel: CMAN: bad generation number 10 in HELLO message, expected 9 Dec 18 19:56:06 cl032 kernel: CMAN: killed by STARTTRANS or NOMINATE Dec 18 19:56:06 cl032 kernel: CMAN: we are leaving the cluster. Dec 18 19:56:07 cl032 kernel: dlm: closing connection to node 2 Dec 18 19:56:07 cl032 kernel: dlm: closing connection to node 3 Dec 18 19:56:07 cl032 kernel: SM: 00000001 sm_stop: SG still joined Dec 18 19:56:07 cl032 kernel: SM: 0100081e sm_stop: SG still joined Dec 18 19:56:07 cl032 kernel: SM: 0200081f sm_stop: SG still joined cl031: Dec 18 19:56:02 cl031 kernel: CMAN: node cl032a is not responding - removing from the cluster Dec 18 19:56:06 cl031 kernel: CMAN: Being told to leave the cluster by node 1 Dec 18 19:56:06 cl031 kernel: CMAN: we are leaving the cluster. Dec 18 19:56:07 cl031 kernel: dlm: closing connection to node 2 Dec 18 19:56:07 cl031 kernel: dlm: closing connection to node 3 Dec 18 19:56:07 cl031 kernel: SM: 00000001 sm_stop: SG still joined Dec 18 19:56:07 cl031 kernel: SM: 0100081e sm_stop: SG still joined cl030: Dec 18 19:56:05 cl030 kernel: CMAN: bad generation number 10 in HELLO message, expected 9 Dec 18 19:56:06 cl030 kernel: CMAN: Node cl031a is leaving the cluster, Shutdown Dec 18 19:56:06 cl030 kernel: CMAN: quorum lost, blocking activity Looks like cl032 had the most problems. It hit a bug of asserts: $ grep BUG cl032.messages Dec 18 19:56:48 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:400! Dec 18 19:56:48 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:342! Dec 18 20:01:06 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:342! Dec 18 20:01:07 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:342! Questions: Any ideas on what is going on here? How does one know what the current "generation" number is? When CMAN gets an error, it is not shutting down all the cluster software correctly. GFS is still mounted and anything accessing it is hung. For debugging it is ok for the machine to stay up so we can figure out what is going on, but for a real operational cluster this is very bad. In normal operation, if the cluster hits a bug likes this shouldn't it just reboot, so hopefully all the other nodes can recover? Is there more debugging that can be turned on, so we can figure out what is going on? The full info is available here: http://developer.osdl.org/daniel/GFS/cman.18dec2004/ Thanks, Daniel From rajkum2002 at rediffmail.com Wed Dec 22 00:44:58 2004 From: rajkum2002 at rediffmail.com (Raj Kumar) Date: 22 Dec 2004 00:44:58 -0000 Subject: [Linux-cluster] unable to umount GFS Message-ID: <20041222004458.25343.qmail@webmail17.rediffmail.com> Hello, I was unable to mount a GFS filesystem. fuser -m mountpoint shows no processes. lsof | grep mountpoint|pool also doesn't return any pid using it. gfs_tool lockdump mountpoint however shows some locks. How do I safely unmount the GFS file system now? Please help!! Thanks, Raj ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rstevens at vitalstream.com Wed Dec 22 01:19:30 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Tue, 21 Dec 2004 17:19:30 -0800 Subject: [Linux-cluster] Bad day in writesville Message-ID: <41C8CBA2.30204@vitalstream.com> I have a test GFS cluster with two nodes using the CVS cluster code from yesterday (Dec. 20th). The nodes are using CMAN/DLM/CLVM. Experimentation has found that heavy write activity on one node causes that node to come to a screeching halt. It stops sending heartbeats, so the other node fences it. I understand that and it's the way I'd expect the _cluster_ to behave, but I'm sure unhappy that the first node crashes in the first place. The first crash occured when we decided to replicate content from a NAS to the GFS-mounted filesystem via an rsync job. After rebooting the failed node and such, I decided to wipe the data off the GFS volume via an "rm -rf *". It crashed in the same manner. An IRC conversation with one of the developers suggested that I try GULM and a lock server rather than CMAN/DLM/CLVM. The problem is that the GFS volume is an LVM volume on a SCSI-based SAN and, of course, I can't "vgchange" it because clvmd isn't available in GULM. I could use the "--ignorelockingfailure" in the vgchange command, but that doesn't sound safe to me and I'm concerned that if both nodes have to write to the filesystem, bad, evil things will happen. A second issue is that I performed the "gfs_mkfs" with the "-p lock_dlm" option. So, my main questions are: 1. Can I continue to use an LVM device under GULM safely, and if so, how? I'd like to continue with LVM as there are times where this filesystem will have to be "grown" as more content appears (in our business, it's impossible to predict how much content there will be). 2. Do I have to destroy the filesystem and reformat it using the "-p lock_gulm" option? As I said, this is a lab rat set up right now and wiping things out to start over isn't a problem. I must have reliable write activity going on here, preferably from all cluster nodes or GFS isn't going to work in our environment and I really need it or something like it. Help me, Obi Wan Kenobis! You're my only hope! ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Change is inevitable, except from a vending machine. - ---------------------------------------------------------------------- From kpreslan at redhat.com Wed Dec 22 01:31:58 2004 From: kpreslan at redhat.com (Ken Preslan) Date: Tue, 21 Dec 2004 19:31:58 -0600 Subject: [Linux-cluster] Bad day in writesville In-Reply-To: <41C8CBA2.30204@vitalstream.com> References: <41C8CBA2.30204@vitalstream.com> Message-ID: <20041222013158.GA24893@potassium.msp.redhat.com> On Tue, Dec 21, 2004 at 05:19:30PM -0800, Rick Stevens wrote: > 2. Do I have to destroy the filesystem and reformat it using the "-p > lock_gulm" option? To change the module permanently, you can unmount the filesystem on all nodes and run the commands: gfs_tool sb proto gfs_tool sb table If it's just a temporary thing, you can unmount the filesystem on all nodes and mount with options to override the defaults in the superblock: mount -o lockproto=,locktable=
is whatever you'd have for gfs_mkfs' -p and
is whatever you'd have to gfs_mkfs' -t. Note that you need to be careful when you do either of these things. Running a mixed cluster where some machines are locking a FS with one protocol/table and other machines are locking the same FS with a different protocol/table is bad. It is bound to end in great sadness. -- Ken Preslan From amanthei at redhat.com Wed Dec 22 06:39:12 2004 From: amanthei at redhat.com (Adam Manthei) Date: Wed, 22 Dec 2004 00:39:12 -0600 Subject: [Linux-cluster] unable to umount GFS In-Reply-To: <20041222004458.25343.qmail@webmail17.rediffmail.com> References: <20041222004458.25343.qmail@webmail17.rediffmail.com> Message-ID: <20041222063912.GM4310@redhat.com> On Wed, Dec 22, 2004 at 12:44:58AM -0000, Raj Kumar wrote: > Hello, > > I was unable to mount a GFS filesystem. fuser -m mountpoint shows no processes. lsof | grep mountpoint|pool also doesn't return any pid using it. gfs_tool lockdump mountpoint however shows some locks. How do I safely unmount the GFS file system now? > > Please help!! is lock_gulmd running on the client? is lock_gulmd quorate? what happens on umount? Is it telling you that the file system is busy on umount? does it just hang? did you ever restart lock_gulmd on the node mounting gfs? (see https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=129193) -- Adam Manthei From pcaulfie at redhat.com Wed Dec 22 09:05:13 2004 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 22 Dec 2004 09:05:13 +0000 Subject: [Linux-cluster] Bad day in writesville In-Reply-To: <41C8CBA2.30204@vitalstream.com> References: <41C8CBA2.30204@vitalstream.com> Message-ID: <20041222090513.GA1260@tykepenguin.com> On Tue, Dec 21, 2004 at 05:19:30PM -0800, Rick Stevens wrote: > > An IRC conversation with one of the developers suggested that I try GULM > and a lock server rather than CMAN/DLM/CLVM. The problem is that the > GFS volume is an LVM volume on a SCSI-based SAN and, of course, I can't > "vgchange" it because clvmd isn't available in GULM. ~/dev/LVM2 $ ./configure --with-clvmd=gulm It hasn't been very well tested though ... -- patrick From pcaulfie at redhat.com Wed Dec 22 09:08:32 2004 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 22 Dec 2004 09:08:32 +0000 Subject: [Linux-cluster] cman bad generation number In-Reply-To: <1103654081.29749.17.camel@ibm-c.pdx.osdl.net> References: <1103654081.29749.17.camel@ibm-c.pdx.osdl.net> Message-ID: <20041222090832.GB1260@tykepenguin.com> On Tue, Dec 21, 2004 at 10:34:41AM -0800, Daniel McNeil wrote: > Another test run that manage 52 hours before hitting a cman bug: > > cl032: > Dec 18 19:56:05 cl032 kernel: CMAN: bad generation number 10 in HELLO message, expected 9 > Dec 18 19:56:06 cl032 kernel: CMAN: killed by STARTTRANS or NOMINATE > Dec 18 19:56:06 cl032 kernel: CMAN: we are leaving the cluster. > Dec 18 19:56:07 cl032 kernel: dlm: closing connection to node 2 > Dec 18 19:56:07 cl032 kernel: dlm: closing connection to node 3 > Dec 18 19:56:07 cl032 kernel: SM: 00000001 sm_stop: SG still joined > Dec 18 19:56:07 cl032 kernel: SM: 0100081e sm_stop: SG still joined > Dec 18 19:56:07 cl032 kernel: SM: 0200081f sm_stop: SG still joined > > cl031: > Dec 18 19:56:02 cl031 kernel: CMAN: node cl032a is not responding - removing from the cluster > Dec 18 19:56:06 cl031 kernel: CMAN: Being told to leave the cluster by node 1 > Dec 18 19:56:06 cl031 kernel: CMAN: we are leaving the cluster. > Dec 18 19:56:07 cl031 kernel: dlm: closing connection to node 2 > Dec 18 19:56:07 cl031 kernel: dlm: closing connection to node 3 > Dec 18 19:56:07 cl031 kernel: SM: 00000001 sm_stop: SG still joined > Dec 18 19:56:07 cl031 kernel: SM: 0100081e sm_stop: SG still joined > > cl030: > Dec 18 19:56:05 cl030 kernel: CMAN: bad generation number 10 in HELLO message, expected 9 > Dec 18 19:56:06 cl030 kernel: CMAN: Node cl031a is leaving the cluster, Shutdown > Dec 18 19:56:06 cl030 kernel: CMAN: quorum lost, blocking activity > > Looks like cl032 had the most problems. It hit a bug of asserts: > $ grep BUG cl032.messages > Dec 18 19:56:48 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:400! > Dec 18 19:56:48 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:342! > Dec 18 20:01:06 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:342! > Dec 18 20:01:07 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:342! > > Questions: > Any ideas on what is going on here? > > How does one know what the current "generation" number is? You don't, cman does. it's the current "generation" of the cluster which is incremented for each state transition. Are you taking nodes up and down during these tests?? It does seem that cman is susceptible to heavy network traffic, despite my best efforts to increase its priority. I'm going to check in a change that will allow you to change the retry count byt it's a bit of a hack really. > When CMAN gets an error, it is not shutting down all the cluster > software correctly. GFS is still mounted and anything accessing > it is hung. For debugging it is ok for the machine to stay up > so we can figure out what is going on, but for a real operational > cluster this is very bad. In normal operation, if the cluster > hits a bug likes this shouldn't it just reboot, so hopefully > all the other nodes can recover? If you have power switch fencing and the remainder of the node is quorate then surely the failed node should be powercycled? patrick From pcaulfie at redhat.com Wed Dec 22 13:56:57 2004 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 22 Dec 2004 13:56:57 +0000 Subject: [Linux-cluster] GFS 2 node hang in rm test In-Reply-To: <200412101345.45229.phillips@redhat.com> References: <1102115280.2706.37.camel@ibm-c.pdx.osdl.net> <200412091652.42408.phillips@redhat.com> <20041210091453.GD14897@tykepenguin.com> <200412101345.45229.phillips@redhat.com> Message-ID: <20041222135657.GC667@tykepenguin.com> On a cursory test it seems OK to me. If the locks are both acquired NOQUEUE then I get AST routines delivered correctly for both (the second with EAGAIN status). With NOQUEUE, there is no second AST delivered because the second lock request is waiting because it is blocked by the first one. If you still think there's a problem, can you send me a small piece of sample code please ? patrick From danderso at redhat.com Wed Dec 22 15:30:30 2004 From: danderso at redhat.com (Derek Anderson) Date: Wed, 22 Dec 2004 09:30:30 -0600 Subject: [Linux-cluster] Bad day in writesville In-Reply-To: <20041222090513.GA1260@tykepenguin.com> References: <41C8CBA2.30204@vitalstream.com> <20041222090513.GA1260@tykepenguin.com> Message-ID: <200412220930.31015.danderso@redhat.com> On Wednesday 22 December 2004 03:05, Patrick Caulfield wrote: > On Tue, Dec 21, 2004 at 05:19:30PM -0800, Rick Stevens wrote: > > An IRC conversation with one of the developers suggested that I try GULM > > and a lock server rather than CMAN/DLM/CLVM. The problem is that the > > GFS volume is an LVM volume on a SCSI-based SAN and, of course, I can't > > "vgchange" it because clvmd isn't available in GULM. > > ~/dev/LVM2 $ ./configure --with-clvmd=gulm > > It hasn't been very well tested though ... It needs help. Bugzilla #143439, esp. comment #4. From amanthei at redhat.com Wed Dec 22 15:42:51 2004 From: amanthei at redhat.com (Adam Manthei) Date: Wed, 22 Dec 2004 09:42:51 -0600 Subject: [Linux-cluster] unable to umount GFS In-Reply-To: <20041222152133.24864.qmail@webmail29.rediffmail.com> References: <20041222152133.24864.qmail@webmail29.rediffmail.com> Message-ID: <20041222154251.GN4310@redhat.com> On Wed, Dec 22, 2004 at 03:21:33PM -0000, Raj Kumar wrote: > Typo in my posting's first line: I was unable to umount (not mount). > > >what happens on umount? Is it telling you that the file system is busy on > >umount? does it just hang? > yes, it tells mountpoint is busy. So I used lsof and fuser to see who is using it. But no pid's were returned. If it says it's busy, then the filesystem is in use. You weren't by chance trying to umount while in the directory? while another filesystem was mounted on GFS? > >is lock_gulmd running on the client? > yes, it was running. > >is lock_gulmd quorate? > I'm not running the cluster suite, just GFS. How do I check if lock_gulmd is quorate? On cluster I can use this command and check number of votes: > cat /proc/cluster/status > ? gulm_tool getstats | grep quorum This will tell you what it has and needs for quorum. > >did you ever restart lock_gulmd on the node mounting gfs? > No. > > I was experimenting different test cases to check GFS stability for our purpose. I had exported a GFS file system as NFS to node2. node3 now accesses via samba the GFS file system on node2 (node2 uses NFS). So it is GFS(node1)-->NFS(node2)-->SAMBA(node3). I was unable to save files from node3 onto GFS filesystem. This has left many smbd daemons on node2 in 'D' state (Disk Sleep). The samba daemons might be holding some locks... but it is not clear. I restarted all the nodes and cluster is fine. How do I check if there are any file corruptions? using gfs_fsck? > > Thanks, > Raj > > On Wed, 22 Dec 2004 Adam Manthei wrote : > >On Wed, Dec 22, 2004 at 12:44:58AM -0000, Raj Kumar wrote: > > > Hello, > > > > > > I was unable to mount a GFS filesystem. fuser -m mountpoint shows no processes. lsof | grep mountpoint|pool also doesn't return any pid using it. gfs_tool lockdump mountpoint however shows some locks. How do I safely unmount the GFS file system now? > > > > > > Please help!! > > > >is lock_gulmd running on the client? > > > >is lock_gulmd quorate? > > > >what happens on umount? Is it telling you that the file system is busy on > >umount? does it just hang? > > > >did you ever restart lock_gulmd on the node mounting gfs? > >(see https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=129193) > > > >-- > >Adam Manthei -- Adam Manthei From daniel at osdl.org Wed Dec 22 17:33:39 2004 From: daniel at osdl.org (Daniel McNeil) Date: Wed, 22 Dec 2004 09:33:39 -0800 Subject: [Linux-cluster] cman bad generation number In-Reply-To: <20041222090832.GB1260@tykepenguin.com> References: <1103654081.29749.17.camel@ibm-c.pdx.osdl.net> <20041222090832.GB1260@tykepenguin.com> Message-ID: <1103736819.30947.20.camel@ibm-c.pdx.osdl.net> On Wed, 2004-12-22 at 01:08, Patrick Caulfield wrote: > On Tue, Dec 21, 2004 at 10:34:41AM -0800, Daniel McNeil wrote: > > Another test run that manage 52 hours before hitting a cman bug: > > > > cl032: > > Dec 18 19:56:05 cl032 kernel: CMAN: bad generation number 10 in HELLO message, expected 9 > > Dec 18 19:56:06 cl032 kernel: CMAN: killed by STARTTRANS or NOMINATE > > Dec 18 19:56:06 cl032 kernel: CMAN: we are leaving the cluster. > > Dec 18 19:56:07 cl032 kernel: dlm: closing connection to node 2 > > Dec 18 19:56:07 cl032 kernel: dlm: closing connection to node 3 > > Dec 18 19:56:07 cl032 kernel: SM: 00000001 sm_stop: SG still joined > > Dec 18 19:56:07 cl032 kernel: SM: 0100081e sm_stop: SG still joined > > Dec 18 19:56:07 cl032 kernel: SM: 0200081f sm_stop: SG still joined > > > > cl031: > > Dec 18 19:56:02 cl031 kernel: CMAN: node cl032a is not responding - removing from the cluster > > Dec 18 19:56:06 cl031 kernel: CMAN: Being told to leave the cluster by node 1 > > Dec 18 19:56:06 cl031 kernel: CMAN: we are leaving the cluster. > > Dec 18 19:56:07 cl031 kernel: dlm: closing connection to node 2 > > Dec 18 19:56:07 cl031 kernel: dlm: closing connection to node 3 > > Dec 18 19:56:07 cl031 kernel: SM: 00000001 sm_stop: SG still joined > > Dec 18 19:56:07 cl031 kernel: SM: 0100081e sm_stop: SG still joined > > > > cl030: > > Dec 18 19:56:05 cl030 kernel: CMAN: bad generation number 10 in HELLO message, expected 9 > > Dec 18 19:56:06 cl030 kernel: CMAN: Node cl031a is leaving the cluster, Shutdown > > Dec 18 19:56:06 cl030 kernel: CMAN: quorum lost, blocking activity > > > > Looks like cl032 had the most problems. It hit a bug of asserts: > > $ grep BUG cl032.messages > > Dec 18 19:56:48 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:400! > > Dec 18 19:56:48 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:342! > > Dec 18 20:01:06 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:342! > > Dec 18 20:01:07 cl032 kernel: kernel BUG at /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c:342! > > > > Questions: > > Any ideas on what is going on here? > > > > How does one know what the current "generation" number is? > > You don't, cman does. it's the current "generation" of the cluster which is > incremented for each state transition. Are you taking nodes up and down during > these tests?? The nodes are staying up. I am mounting and umounting a lot. Any reason to not add generation /proc/cluster/status? (it would help debugging at least). > > It does seem that cman is susceptible to heavy network traffic, despite my best > efforts to increase its priority. I'm going to check in a change that will allow > you to change the retry count byt it's a bit of a hack really. > > > > When CMAN gets an error, it is not shutting down all the cluster > > software correctly. GFS is still mounted and anything accessing > > it is hung. For debugging it is ok for the machine to stay up > > so we can figure out what is going on, but for a real operational > > cluster this is very bad. In normal operation, if the cluster > > hits a bug likes this shouldn't it just reboot, so hopefully > > all the other nodes can recover? > > If you have power switch fencing and the remainder of the node is quorate then > surely the failed node should be powercycled? > I currently have it set up for manual fencing and I have yet to see that work correctly. This was a 3 node cluster. cl032 got the bad generation number and cman was "killed by STARTTRANS or NOMINATE" cl030 got a bad generation number (but stayed up) and cl031 leaves the cluster because it says cl030 told it to. So that leaves me with 1 node up without quorum. I did not see any fencing messages. Should the surviving node (cl030) have attempted fencing or does it only do that if it has quorum? I do not seem to be able to keep cman up for much past 2 days if I have my tests running. (it stays up with no load, of course). My tests are not the complicated currently either. Just tar, du and rm in separate directories from 1, 2 and then 3 nodes simultaneously. Who knows what will happen if I add tests to cause lots of dlm lock conflict. How long does cman stay up in your testing? Daniel From amanthei at redhat.com Wed Dec 22 18:57:43 2004 From: amanthei at redhat.com (Adam Manthei) Date: Wed, 22 Dec 2004 12:57:43 -0600 Subject: [Linux-cluster] unable to umount GFS In-Reply-To: <20041222182107.4601.qmail@webmail32.rediffmail.com> References: <20041222182107.4601.qmail@webmail32.rediffmail.com> Message-ID: <20041222185743.GO4310@redhat.com> On Wed, Dec 22, 2004 at 06:21:07PM -0000, Raj Kumar wrote: > >If it says it's busy, then the filesystem is in use. You weren't by chance > >trying to umount while in the directory? while another filesystem was > >mounted on GFS? > No. It looks to me that GFS-->NFS-->Samba was a problem. I was unable to write files onto GFS filesystem via Samba share. The status of smbd is "D" (disk sleep). kill -9 smbd pid didn't work either. Is there any log that I can collect when problems of this kind occur for debugging? > > Thanks for your input! > Raj If NFS was exporting a GFS filesystem, then you will get the filesystem busy error. I really don't know what to make of your "GFS-->NFS-->Samba" problem? Yikes, is Samba exporting a NFS mount that is on GFS? Seems pretty hairy to me. Your problem might lie with Samba if you are trying to run multiple Samba servers (something that does not work with Samba). -- Adam Manthei From rstevens at vitalstream.com Wed Dec 22 19:02:47 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Wed, 22 Dec 2004 11:02:47 -0800 Subject: [Linux-cluster] Bad day in writesville In-Reply-To: <20041222013158.GA24893@potassium.msp.redhat.com> References: <41C8CBA2.30204@vitalstream.com> <20041222013158.GA24893@potassium.msp.redhat.com> Message-ID: <41C9C4D7.80402@vitalstream.com> Ken Preslan wrote: > On Tue, Dec 21, 2004 at 05:19:30PM -0800, Rick Stevens wrote: > >>2. Do I have to destroy the filesystem and reformat it using the "-p >>lock_gulm" option? > > > To change the module permanently, you can unmount the filesystem on all > nodes and run the commands: > > gfs_tool sb proto > gfs_tool sb table
> > If it's just a temporary thing, you can unmount the filesystem on all > nodes and mount with options to override the defaults in the superblock: > > mount -o lockproto=,locktable=
> > > is whatever you'd have for gfs_mkfs' -p and
> is whatever you'd have to gfs_mkfs' -t. > > > Note that you need to be careful when you do either of these things. > Running a mixed cluster where some machines are locking a FS with one > protocol/table and other machines are locking the same FS with a > different protocol/table is bad. It is bound to end in great sadness. Gotcha. After looking at the bugzilla entries refered to in other replies to my question, it appears that LVM works fairly well with DLM but has major issues with GULM. However, someone mentioned that one can use DLM to lock LVM and use GULM to lock GFS. Since the LVM stuff is pretty important, I intend to try that method. I rebuilt LVM to use DLM/CMAN. I've modprobed both lock_dlm and lock_gulm and started clvmd. I've used "cman_tool join" to set up DLM and "vgchange -aly"d the LVM volume and it appeared. I've used gfs_tool to change the locking protocol on the filesystem to use GULM (the table remains the same) and it mounted fine. I'm about to start the same stuff on the second node. I'll keep you informed. ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Millihelen, adj: The amount of beauty required to launch one ship. - ---------------------------------------------------------------------- From amanthei at redhat.com Wed Dec 22 23:09:27 2004 From: amanthei at redhat.com (Adam Manthei) Date: Wed, 22 Dec 2004 17:09:27 -0600 Subject: [Linux-cluster] unable to umount GFS In-Reply-To: <20041222220229.8401.qmail@webmail32.rediffmail.com> References: <20041222220229.8401.qmail@webmail32.rediffmail.com> Message-ID: <20041222230927.GQ4310@redhat.com> On Wed, Dec 22, 2004 at 10:02:29PM -0000, Raj Kumar wrote: > >If NFS was exporting a GFS filesystem, then you will get the filesystem busy > >error. > > Does this mean I should not export GFS filesystem via NFS? Not all our clients computers are connected to storage and we intend to provide access to them via NFS exports. > > Thanks, > Raj You can't unmount GFS if the node you are trying to unmount is also exporting GFS via NFS. The filesystem busy error means just that, it's busy. If you want to unmount GFS, you will first need to stop GFS on that node. -- Adam Manthei From rstevens at vitalstream.com Wed Dec 22 23:27:30 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Wed, 22 Dec 2004 15:27:30 -0800 Subject: [Linux-cluster] Bad day in writesville - Followup In-Reply-To: <41C9C4D7.80402@vitalstream.com> References: <41C8CBA2.30204@vitalstream.com> <20041222013158.GA24893@potassium.msp.redhat.com> <41C9C4D7.80402@vitalstream.com> Message-ID: <41CA02E2.4070709@vitalstream.com> Rick Stevens wrote: > Ken Preslan wrote: > >> On Tue, Dec 21, 2004 at 05:19:30PM -0800, Rick Stevens wrote: >> >>> 2. Do I have to destroy the filesystem and reformat it using the "-p >>> lock_gulm" option? >> >> >> >> To change the module permanently, you can unmount the filesystem on all >> nodes and run the commands: >> >> gfs_tool sb proto >> gfs_tool sb table
>> >> If it's just a temporary thing, you can unmount the filesystem on all >> nodes and mount with options to override the defaults in the superblock: >> >> mount -o lockproto=,locktable=
>> >> >> is whatever you'd have for gfs_mkfs' -p and
>> is whatever you'd have to gfs_mkfs' -t. >> >> >> Note that you need to be careful when you do either of these things. >> Running a mixed cluster where some machines are locking a FS with one >> protocol/table and other machines are locking the same FS with a >> different protocol/table is bad. It is bound to end in great sadness. > > > Gotcha. After looking at the bugzilla entries refered to in other > replies to my question, it appears that LVM works fairly well with DLM > but has major issues with GULM. However, someone mentioned that one can > use DLM to lock LVM and use GULM to lock GFS. > > Since the LVM stuff is pretty important, I intend to try that method. > I rebuilt LVM to use DLM/CMAN. I've modprobed both lock_dlm and > lock_gulm and started clvmd. I've used "cman_tool join" to set up DLM > and "vgchange -aly"d the LVM volume and it appeared. I've used gfs_tool > to change the locking protocol on the filesystem to use GULM (the table > remains the same) and it mounted fine. I'm about to start the same > stuff on the second node. I'll keep you informed. Followup: Well, that seems to be the fix. I used CMAN/DLM to manage the LVM volumes via clvmd and I changed the GFS filesystem to use GULM as its locking mechanism. No hiccups so far, so it appears that heavy write activity freaks out DLM when it's also managing GFS. I wish I had traces and such to show you what was going on with CMAN/DLM only, but the machine locked up so hard only a reboot could bring it back. I'm going to keep testing this so there will be an additional follow up. I'd also like to thank Ken Preslan, Patrick Caulfield and Derek Anderson for their help on this. I may be much closer to a solution and you chaps helped a lot. Thanks! ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Reality: A crutch for those who can't handle science fiction - ---------------------------------------------------------------------- From amanthei at redhat.com Wed Dec 22 23:42:18 2004 From: amanthei at redhat.com (Adam Manthei) Date: Wed, 22 Dec 2004 17:42:18 -0600 Subject: [Linux-cluster] unable to umount GFS In-Reply-To: <20041222233218.31970.qmail@webmail27.rediffmail.com> References: <20041222233218.31970.qmail@webmail27.rediffmail.com> Message-ID: <20041222234218.GR4310@redhat.com> On Wed, Dec 22, 2004 at 11:32:18PM -0000, Raj Kumar wrote: > >You can't unmount GFS if the node you are trying to unmount is also > >exporting GFS via NFS. The filesystem busy error means just that, it's > >busy. If you want to unmount GFS, you will first need to stop GFS on that > >node. > > nodeA: connected to storage and exports GFS filesystem via NFS > No clients are using the NFS export. > > What procedure do I follow to cleanly shutdown GFS on nodeA? > > Steps: > unmount all GFS filesystems > shutdown lock server using gulm_tool shutdown localhost. > > Is the above sequence correct? If yes, I am unable to complete step 1 unmounting. > > >If you want to unmount GFS, you will first need to stop GFS on that > >node. > Were you referring to clients using NFS exports in statement "stop GFS on that node" or nodeA. No one is using NFS exports and GFS filesystem on nodeA. > > I'm sorry to bother you, but would very much appreciate your comments! > Thank you! > Raj It doesn't matter that nobody is using NFS in your above example. NFS is using GFS! and that is why you are unable to unmount GFS. So... Step 1. Stop Exporting NFS (at least stop exporting GFS) Step 2. Make sure nothing else on the system is using GFS Step 3. Umount GFS Step 4. stop the lock servers. Steps 1-3 are the same for any other filesystem. There is nothing special going on here. The init scripts will also help with the ordering. service nfs stop service gfs stop service lock_gulmd stop service ccsd stop service pool stop -- Adam Manthei From daniel at osdl.org Thu Dec 23 00:35:25 2004 From: daniel at osdl.org (Daniel McNeil) Date: Wed, 22 Dec 2004 16:35:25 -0800 Subject: [Linux-cluster] another cman/sm problem Message-ID: <1103762125.30947.64.camel@ibm-c.pdx.osdl.net> cl030: CMAN: node cl031a is not responding - removing from the cluster dlm: closing connection to node 1 dlm: closing connection to node 2 SM: 00000001 sm_stop: SG still joined SM: 01000932 sm_stop: SG still joined SM: 02000933 sm_stop: SG still joined Unable to handle kernel NULL pointer dereference at virtual address 00000004 printing eip: c0119677 *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP Modules linked in: lock_nolock lock_dlm dlm qla2200 qla2xxx gfs lock_harness cman dm_mod CPU: 1 EIP: 0060:[] Not tainted VLI EFLAGS: 00010096 (2.6.9) EIP is at task_rq_lock+0x27/0x70 eax: ea000f2c ebx: c052e000 ecx: 00000000 edx: 00000000 esi: c0533020 edi: c052e000 ebp: ea000ef4 esp: ea000ee8 ds: 007b es: 007b ss: 0068 Process cman_comms (pid: 3739, threadinfo=ea000000 task=e9de98f0) Stack: c1b037b0 e617daf4 f781243c ea000f3c c0119d92 00000000 ea000f2c c014a82f c181f040 00000020 f7810750 f8d04755 ffffff95 02000933 00000000 ea000f50 f8d047b5 00000296 c1b037b0 e617daf4 f781243c ea000f50 c011a02e 00000000 Call Trace: [] show_stack+0x7f/0xa0 [] show_registers+0x15e/0x1d0 [] die+0xfe/0x190 [] do_page_fault+0x293/0x7c1 [] error_code+0x2d/0x38 [] try_to_wake_up+0x22/0x2a0 [] wake_up_process+0x1e/0x30 [] callback_startdone_barrier+0x20/0x30 [cman] [] node_shutdown+0x291/0x3c0 [cman] [] cluster_kthread+0x2aa/0x350 [cman] [] kernel_thread_helper+0x5/0x10 SM: 00000001 sm_stop: SG still joined SM: 01000932 sm_stop: SG still joined SM: 02000933 sm_stop: SG still joined Unable to handle kernel NULL pointer dereference at virtual address 00000004 printing eip: c0119677 *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP cl031: dlm: closing connection to node 1 dlm: closing connection to node 2 SM: 00000001 sm_stop: SG still joined SM: 01000932 sm_stop: SG still joined SM: 02000933 sm_stop: SG still joined Unable to handle kernel NULL pointer dereference at virtual address 00000004 printing eip: c0119677 *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP Modules linked in: lock_dlm dlm qla2200 qla2xxx gfs lock_harness cman dm_mod CPU: 1 EIP: 0060:[] Not tainted VLI EFLAGS: 00010096 (2.6.9) EIP is at task_rq_lock+0x27/0x70 eax: eacc0f2c ebx: c052e000 ecx: 00000000 edx: 00000000 esi: c0533020 edi: c052e000 ebp: eacc0ef4 esp: eacc0ee8 ds: 007b es: 007b ss: 0068 Process cman_comms (pid: 2876, threadinfo=eacc0000 task=eae75a30) Stack: ea3a285c caeb7da4 f502e1a8 eacc0f3c c0119d92 00000000 eacc0f2c c014a82f c181f040 00000020 f7d35f38 f8d04755 ffffff95 02000933 00000000 eacc0f50 f8d047b5 00000296 ea3a285c caeb7da4 f502e1a8 eacc0f50 c011a02e 00000000 Call Trace: [] show_stack+0x7f/0xa0 [] show_registers+0x15e/0x1d0 [] die+0xfe/0x190 [] do_page_fault+0x293/0x7c1 [] error_code+0x2d/0x38 [] try_to_wake_up+0x22/0x2a0 [] wake_up_process+0x1e/0x30 [] callback_startdone_barrier_new+0x20/0x30 [cman] [] node_shutdown+0x291/0x3c0 [cman] [] cluster_kthread+0x2aa/0x350 [cman] [] kernel_thread_helper+0x5/0x10 Code: 00 00 00 00 55 89 e5 83 ec 0c 89 1c 24 89 74 24 04 89 7c 24 08 8b 45 0c 9c 8f 00 fa be 20 30 53 c0 bb 00 e0 52 c0 8b 55 08 89 df <8b> 42 04 8b 40 10 8b 0c cl032: Dec 22 05:29:19 cl032 sshd(pam_unix)[18296]: session closed for user root Dec 22 05:42:14 cl032 kernel: CMAN: bad generation number 15 in HELLO message, expected 14 Dec 22 05:42:17 cl032 kernel: CMAN: Node cl030a is leaving the cluster, ShutdownDec 22 05:42:17 cl032 kernel: CMAN: quorum lost, blocking activity My test is doing a lot of mounting and umounting. Wouldn't that stress SM code a lot. Is SM causing the problem? http://developer.osdl.org/daniel/GFS/cman.21dec2004/ Daniel From rajkum2002 at rediffmail.com Thu Dec 23 15:07:17 2004 From: rajkum2002 at rediffmail.com (Raj Kumar) Date: 23 Dec 2004 15:07:17 -0000 Subject: [Linux-cluster] unable to umount GFS Message-ID: <20041223150717.25701.qmail@webmail18.rediffmail.com> Thank you very much! ? PS: I didn't notice until now that all messages were not sent to linux-cluster. I'm sending them now with the hope that it will be useful to someone! On Thu, 23 Dec 2004 Adam Manthei wrote : >On Wed, Dec 22, 2004 at 11:32:18PM -0000, Raj Kumar wrote: > > >You can't unmount GFS if the node you are trying to unmount is also > > >exporting GFS via NFS. The filesystem busy error means just that, it's > > >busy. If you want to unmount GFS, you will first need to stop GFS on that > > >node. > > > > nodeA: connected to storage and exports GFS filesystem via NFS > > No clients are using the NFS export. > > > > What procedure do I follow to cleanly shutdown GFS on nodeA? > > > > Steps: > > unmount all GFS filesystems > > shutdown lock server using gulm_tool shutdown localhost. > > > > Is the above sequence correct? If yes, I am unable to complete step 1 unmounting. > > > > >If you want to unmount GFS, you will first need to stop GFS on that > > >node. > > Were you referring to clients using NFS exports in statement "stop GFS on that node" or nodeA. No one is using NFS exports and GFS filesystem on nodeA. > > > > I'm sorry to bother you, but would very much appreciate your comments! > > Thank you! > > Raj > >It doesn't matter that nobody is using NFS in your above example. NFS is >using GFS! and that is why you are unable to unmount GFS. > >So... > >Step 1. Stop Exporting NFS (at least stop exporting GFS) >Step 2. Make sure nothing else on the system is using GFS >Step 3. Umount GFS >Step 4. stop the lock servers. > >Steps 1-3 are the same for any other filesystem. There is nothing special >going on here. The init scripts will also help with the ordering. > >service nfs stop >service gfs stop >service lock_gulmd stop >service ccsd stop >service pool stop > >-- >Adam Manthei -------------- next part -------------- An HTML attachment was scrubbed... URL: From amanthei at redhat.com Thu Dec 23 15:14:52 2004 From: amanthei at redhat.com (Adam Manthei) Date: Thu, 23 Dec 2004 09:14:52 -0600 Subject: [Linux-cluster] unable to umount GFS In-Reply-To: <20041223150717.25701.qmail@webmail18.rediffmail.com> References: <20041223150717.25701.qmail@webmail18.rediffmail.com> Message-ID: <20041223151452.GT4310@redhat.com> On Thu, Dec 23, 2004 at 03:07:17PM -0000, Raj Kumar wrote: > Thank you very much! ? > PS: I didn't notice until now that all messages were not sent to linux-cluster. I'm sending them now with the hope that it will be useful to someone! Much appreciated. > On Thu, 23 Dec 2004 Adam Manthei wrote : > >On Wed, Dec 22, 2004 at 11:32:18PM -0000, Raj Kumar wrote: > > > >You can't unmount GFS if the node you are trying to unmount is also > > > >exporting GFS via NFS. The filesystem busy error means just that, it's > > > >busy. If you want to unmount GFS, you will first need to stop GFS on that > > > >node. > > > > > > nodeA: connected to storage and exports GFS filesystem via NFS > > > No clients are using the NFS export. > > > > > > What procedure do I follow to cleanly shutdown GFS on nodeA? > > > > > > Steps: > > > unmount all GFS filesystems > > > shutdown lock server using gulm_tool shutdown localhost. > > > > > > Is the above sequence correct? If yes, I am unable to complete step 1 unmounting. > > > > > > >If you want to unmount GFS, you will first need to stop GFS on that > > > >node. > > > Were you referring to clients using NFS exports in statement "stop GFS on that node" or nodeA. No one is using NFS exports and GFS filesystem on nodeA. > > > > > > I'm sorry to bother you, but would very much appreciate your comments! > > > Thank you! > > > Raj > > > >It doesn't matter that nobody is using NFS in your above example. NFS is > >using GFS! and that is why you are unable to unmount GFS. > > > >So... > > > >Step 1. Stop Exporting NFS (at least stop exporting GFS) > >Step 2. Make sure nothing else on the system is using GFS > >Step 3. Umount GFS > >Step 4. stop the lock servers. > > > >Steps 1-3 are the same for any other filesystem. There is nothing special > >going on here. The init scripts will also help with the ordering. > > > >service nfs stop > >service gfs stop > >service lock_gulmd stop > >service ccsd stop > >service pool stop > > > >-- > >Adam Manthei -- Adam Manthei From rstevens at vitalstream.com Thu Dec 23 22:18:07 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Thu, 23 Dec 2004 14:18:07 -0800 Subject: [Linux-cluster] Bad day in writesville - Followup #2 In-Reply-To: <41CA02E2.4070709@vitalstream.com> References: <41C8CBA2.30204@vitalstream.com> <20041222013158.GA24893@potassium.msp.redhat.com> <41C9C4D7.80402@vitalstream.com> <41CA02E2.4070709@vitalstream.com> Message-ID: <41CB441F.7040203@vitalstream.com> Rick Stevens wrote: > Rick Stevens wrote: > >> Ken Preslan wrote: >> >>> On Tue, Dec 21, 2004 at 05:19:30PM -0800, Rick Stevens wrote: >>> >>>> 2. Do I have to destroy the filesystem and reformat it using the "-p >>>> lock_gulm" option? >>> >>> >>> >>> >>> To change the module permanently, you can unmount the filesystem on all >>> nodes and run the commands: >>> >>> gfs_tool sb proto >>> gfs_tool sb table
>>> >>> If it's just a temporary thing, you can unmount the filesystem on all >>> nodes and mount with options to override the defaults in the superblock: >>> >>> mount -o lockproto=,locktable=
>>> >>> >>> is whatever you'd have for gfs_mkfs' -p and
>>> is whatever you'd have to gfs_mkfs' -t. >>> >>> >>> Note that you need to be careful when you do either of these things. >>> Running a mixed cluster where some machines are locking a FS with one >>> protocol/table and other machines are locking the same FS with a >>> different protocol/table is bad. It is bound to end in great sadness. >> >> >> >> Gotcha. After looking at the bugzilla entries refered to in other >> replies to my question, it appears that LVM works fairly well with DLM >> but has major issues with GULM. However, someone mentioned that one can >> use DLM to lock LVM and use GULM to lock GFS. >> >> Since the LVM stuff is pretty important, I intend to try that method. >> I rebuilt LVM to use DLM/CMAN. I've modprobed both lock_dlm and >> lock_gulm and started clvmd. I've used "cman_tool join" to set up DLM >> and "vgchange -aly"d the LVM volume and it appeared. I've used gfs_tool >> to change the locking protocol on the filesystem to use GULM (the table >> remains the same) and it mounted fine. I'm about to start the same >> stuff on the second node. I'll keep you informed. > > > Followup: > > Well, that seems to be the fix. I used CMAN/DLM to manage the LVM > volumes via clvmd and I changed the GFS filesystem to use GULM as its > locking mechanism. No hiccups so far, so it appears that heavy write > activity freaks out DLM when it's also managing GFS. > > I wish I had traces and such to show you what was going on with CMAN/DLM > only, but the machine locked up so hard only a reboot could bring it > back. > > I'm going to keep testing this so there will be an additional follow up. > I'd also like to thank Ken Preslan, Patrick Caulfield and Derek Anderson > for their help on this. I may be much closer to a solution and you > chaps helped a lot. Thanks! Followup #2: This is cool! Based on the 20 December 2004 CVS code, I'm using CMAN/DLM to manage the VG and LVM locking and GULM to manage the GFS locking. It works like a charm--even having both nodes write to the same file. For the final test, I added a new LU off the SAN to the VG, extended the LV by the size of the new LU and was able to grow the GFS filesystem on top of that...all without having to shut anything down. So, success is near! Next, we plan to deploy this cluster into our real environment and let our clients flog it. If it crashes, our load balancer will block access to it and I'll do a post-mortem at that point. Well, that's enough for now. It's 14:15 on 23 December here in California right now, and I'm skedattling until Monday (well, the laptop is full of stuff to do on Sunday). Thanks again for the help, gang! Have a Happy Holiday. Don't eat or drink too much! ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - First Law of Work: - - If you can't get it done in the first 24 hours, work nights. - ---------------------------------------------------------------------- From yazan at ccs.com.jo Sun Dec 26 20:28:18 2004 From: yazan at ccs.com.jo (Yazan Bakheit) Date: Sun, 26 Dec 2004 12:28:18 -0800 Subject: [Linux-cluster] problem Message-ID: i have configured the gfs and i had done everything as mentioned in the examlpe comes with the docment for the gfs with oracle9i , but at the last step when i perform the mount for the gfs from the first node it takes it , but when i mount the same frome the secaond node (as mentioned in the doc) it hanges,and it still hanges until i unmount the same from the first node. so what can i do to mount the from the second node without hanging or waiting the first node to do umount . regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tech.gif Type: image/gif Size: 862 bytes Desc: not available URL: From rajkum2002 at rediffmail.com Sun Dec 26 19:39:58 2004 From: rajkum2002 at rediffmail.com (Raj Kumar) Date: 26 Dec 2004 19:39:58 -0000 Subject: [Linux-cluster] problem Message-ID: <20041226193958.16878.qmail@webmail46.rediffmail.com> Hi, What version of GFS are you using? What are the names of two nodes? In older versions of GFS this problem appears when the first eight characters of the name are the same. (testnode1 and testnode2) Also, check if /etc/hosts contains a line like: 127.0.0.1 testnode1 testnode1.domain Give us more details and we shall try! Happy Holidays!! Raj On Mon, 27 Dec 2004 Yazan Bakheit wrote : > > i have configured the gfs and i had done everything as mentioned in the >examlpe comes with the docment for the gfs with oracle9i , but at the last >step when i perform the mount for the gfs from the first node it takes it , >but when i mount the same frome the secaond node (as mentioned in the doc) >it hanges,and it still hanges until i unmount the same from the first node. > > so what can i do to mount the from the second node without hanging or >waiting the first node to do umount . > > >regards. >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >http://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From teigland at redhat.com Thu Dec 30 10:18:29 2004 From: teigland at redhat.com (David Teigland) Date: Thu, 30 Dec 2004 18:18:29 +0800 Subject: [Linux-cluster] manual fencing problem In-Reply-To: <1103051937.19781.952.camel@ibm-c.pdx.osdl.net> References: <1103051937.19781.952.camel@ibm-c.pdx.osdl.net> Message-ID: <20041230101829.GB14171@redhat.com> On Tue, Dec 14, 2004 at 11:18:57AM -0800, Daniel McNeil wrote: > cl030 /var/log/messages: > > Dec 13 21:54:26 cl030 kernel: CMAN: no HELLO from cl032a, removing from > the cluster > Dec 13 21:54:27 cl030 fenced[12121]: fencing node "cl032a" > Dec 13 21:54:27 cl030 fenced[12121]: fence "cl032a" failed > Dec 13 21:54:28 cl030 fenced[12121]: fencing node "cl032a" > Dec 13 21:54:28 cl030 fenced[12121]: fence "cl032a" failed > Dec 13 21:54:29 cl030 fenced[12121]: fencing node "cl032a" If you're still having this problem, try running this as root on cl030: # fence_node cl032a (You need the cluster running to do this.) ps should show a fence_manual process and the standard fence_manual message should appear in syslog. fence_manual should stay there until you run the fence_ack_manual. also try: # fence_manual -n cl032a which should block waiting for you to run fence_ack_manual. -- Dave Teigland From mmiller at cruzverde.cl Thu Dec 30 14:27:50 2004 From: mmiller at cruzverde.cl (Markus Miller) Date: Thu, 30 Dec 2004 11:27:50 -0300 Subject: [Linux-cluster] GFS and Storage greater than 2 TB Message-ID: <75E9203E0F0DD511B37E00D0B789D45007E8359F@fcv-stgo.cverde.cl> Hi, researching I found a posting to this list made by Kevin Anderson (Date: Tue, 19 Oct 2004 17:56:24 -0500) where he states the following: ---snip--- Maximum size of each GFS filesystem for RHEL3 (2.4.x kernel) is 2 TB, you can have multiple filesystems of that level. So, to get access to 10TB of data requires a minimum of 5 separate filesystems/storage combinations. ---snip--- What do I have to do to achive this? Do I have to configure several GFS clusters in the cluster.ccs file (each of a m?ximum size of 2 TB)? Or do I have to configure one GFS cluster with serveral filesystems each with a maximum size of 2 TB? The GFS Admin Guide is not very precise, but what's really confusing me is the statement on page 12: "2 TB maximum, for total of all storage connected to a GFS cluster." At the moment we are evaluating to buy servers and storage, therefore I do not have any equipment to do the testing myself. Any coment is highly apreciated. Thanks and regards, Markus Markus Miller Ingeniero de Sistemas, RHCE DIFARMA Lord Cochrane 326, Santiago, Chile Tel. +56 2 6944076 mmiller at cruzverde.cl From rstevens at vitalstream.com Thu Dec 30 20:02:51 2004 From: rstevens at vitalstream.com (Rick Stevens) Date: Thu, 30 Dec 2004 12:02:51 -0800 Subject: [Linux-cluster] GFS and Storage greater than 2 TB In-Reply-To: <75E9203E0F0DD511B37E00D0B789D45007E8359F@fcv-stgo.cverde.cl> References: <75E9203E0F0DD511B37E00D0B789D45007E8359F@fcv-stgo.cverde.cl> Message-ID: <41D45EEB.8000904@vitalstream.com> Markus Miller wrote: > Hi, > > researching I found a posting to this list made by Kevin Anderson (Date: Tue, 19 Oct 2004 17:56:24 -0500) where he states the following: > > ---snip--- > Maximum size of each GFS filesystem for RHEL3 (2.4.x kernel) is 2 TB, > you can have multiple filesystems of that level. So, to get access to > 10TB of data requires a minimum of 5 separate filesystems/storage > combinations. > ---snip--- > > What do I have to do to achive this? Do I have to configure several GFS clusters in the cluster.ccs file (each of a m?ximum size of 2 TB)? Or do I have to configure one GFS cluster with serveral filesystems each with a maximum size of 2 TB? The GFS Admin Guide is not very precise, but what's really confusing me is the statement on page 12: "2 TB maximum, for total of all storage connected to a GFS cluster." > > At the moment we are evaluating to buy servers and storage, therefore I do not have any equipment to do the testing myself. > > Any coment is highly apreciated. It's the GFS filesystem that has the limit (actually, it's the 2.4 kernel). Essentially, "gfs_mkfs" can only handle a maximum of 2TB. What he means above is that you have to have five separate partitions of 2TB each and each with a GFS filesystem on them. You have to mount those five filesystems separately. If you're using VG/LVM, with a VG as "vggroup" and LVs in that group as "test1" through "test5": mount -t gfs /dev/mapper/vggroup-test1 /mnt/gfs1 mount -t gfs /dev/mapper/vggroup-test2 /mnt/gfs2 mount -t gfs /dev/mapper/vggroup-test3 /mnt/gfs3 mount -t gfs /dev/mapper/vggroup-test4 /mnt/gfs4 mount -t gfs /dev/mapper/vggroup-test5 /mnt/gfs5 How you use them after that is up to you. Just remember that a given GFS filesystem under kernel 2.4 is limited to 2TB maximum ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Brain: The organ with which we think that we think. - ---------------------------------------------------------------------- From mmiller at cruzverde.cl Thu Dec 30 21:33:55 2004 From: mmiller at cruzverde.cl (Markus Miller) Date: Thu, 30 Dec 2004 18:33:55 -0300 Subject: [Linux-cluster] GFS and Storage greater than 2 TB Message-ID: <75E9203E0F0DD511B37E00D0B789D45007E64F44@fcv-stgo.cverde.cl> Thank you for the answer. That is all I needed to know. -----Mensaje original----- De: Rick Stevens [mailto:rstevens at vitalstream.com] Enviado el: Thursday, December 30, 2004 5:03 PM Para: linux clistering Asunto: Re: [Linux-cluster] GFS and Storage greater than 2 TB Markus Miller wrote: > Hi, > > researching I found a posting to this list made by Kevin Anderson (Date: Tue, 19 Oct 2004 17:56:24 -0500) where he states the following: > > ---snip--- > Maximum size of each GFS filesystem for RHEL3 (2.4.x kernel) is 2 TB, > you can have multiple filesystems of that level. So, to get access to > 10TB of data requires a minimum of 5 separate filesystems/storage > combinations. > ---snip--- > > What do I have to do to achive this? Do I have to configure several GFS clusters in the cluster.ccs file (each of a m?ximum size of 2 TB)? Or do I have to configure one GFS cluster with serveral filesystems each with a maximum size of 2 TB? The GFS Admin Guide is not very precise, but what's really confusing me is the statement on page 12: "2 TB maximum, for total of all storage connected to a GFS cluster." > > At the moment we are evaluating to buy servers and storage, therefore I do not have any equipment to do the testing myself. > > Any coment is highly apreciated. It's the GFS filesystem that has the limit (actually, it's the 2.4 kernel). Essentially, "gfs_mkfs" can only handle a maximum of 2TB. What he means above is that you have to have five separate partitions of 2TB each and each with a GFS filesystem on them. You have to mount those five filesystems separately. If you're using VG/LVM, with a VG as "vggroup" and LVs in that group as "test1" through "test5": mount -t gfs /dev/mapper/vggroup-test1 /mnt/gfs1 mount -t gfs /dev/mapper/vggroup-test2 /mnt/gfs2 mount -t gfs /dev/mapper/vggroup-test3 /mnt/gfs3 mount -t gfs /dev/mapper/vggroup-test4 /mnt/gfs4 mount -t gfs /dev/mapper/vggroup-test5 /mnt/gfs5 How you use them after that is up to you. Just remember that a given GFS filesystem under kernel 2.4 is limited to 2TB maximum ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - Brain: The organ with which we think that we think. - ---------------------------------------------------------------------- -- Linux-cluster mailing list Linux-cluster at redhat.com http://www.redhat.com/mailman/listinfo/linux-cluster From jmjoseph at andrew.cmu.edu Fri Dec 31 06:56:31 2004 From: jmjoseph at andrew.cmu.edu (Jacob Joseph) Date: Thu, 30 Dec 2004 22:56:31 -0800 Subject: [Linux-cluster] GFS and Storage greater than 2 TB In-Reply-To: <75E9203E0F0DD511B37E00D0B789D45007E64F44@fcv-stgo.cverde.cl> References: <75E9203E0F0DD511B37E00D0B789D45007E64F44@fcv-stgo.cverde.cl> Message-ID: <41D4F81F.6000907@andrew.cmu.edu> Does this limit still exist with the cvs GFS on a 2.6 kernel? -Jacob Markus Miller wrote: > Thank you for the answer. That is all I needed to know. > > -----Mensaje original----- > De: Rick Stevens [mailto:rstevens at vitalstream.com] > Enviado el: Thursday, December 30, 2004 5:03 PM > Para: linux clistering > Asunto: Re: [Linux-cluster] GFS and Storage greater than 2 TB > > > Markus Miller wrote: > >>Hi, >> >>researching I found a posting to this list made by Kevin Anderson (Date: Tue, 19 Oct 2004 17:56:24 -0500) where he states the following: >> >>---snip--- >>Maximum size of each GFS filesystem for RHEL3 (2.4.x kernel) is 2 TB, >>you can have multiple filesystems of that level. So, to get access to >>10TB of data requires a minimum of 5 separate filesystems/storage >>combinations. >>---snip--- >> >>What do I have to do to achive this? Do I have to configure several GFS clusters in the cluster.ccs file (each of a m?ximum size of 2 TB)? Or do I have to configure one GFS cluster with serveral filesystems each with a maximum size of 2 TB? The GFS Admin Guide is not very precise, but what's really confusing me is the statement on page 12: "2 TB maximum, for total of all storage connected to a GFS cluster." >> >>At the moment we are evaluating to buy servers and storage, therefore I do not have any equipment to do the testing myself. >> >>Any coment is highly apreciated. > > > It's the GFS filesystem that has the limit (actually, it's the 2.4 > kernel). Essentially, "gfs_mkfs" can only handle a maximum of 2TB. > > What he means above is that you have to have five separate partitions > of 2TB each and each with a GFS filesystem on them. You have to mount > those five filesystems separately. If you're using VG/LVM, with a VG > as "vggroup" and LVs in that group as "test1" through "test5": > > mount -t gfs /dev/mapper/vggroup-test1 /mnt/gfs1 > mount -t gfs /dev/mapper/vggroup-test2 /mnt/gfs2 > mount -t gfs /dev/mapper/vggroup-test3 /mnt/gfs3 > mount -t gfs /dev/mapper/vggroup-test4 /mnt/gfs4 > mount -t gfs /dev/mapper/vggroup-test5 /mnt/gfs5 > > How you use them after that is up to you. Just remember that a given > GFS filesystem under kernel 2.4 is limited to 2TB maximum > ---------------------------------------------------------------------- > - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - > - VitalStream, Inc. http://www.vitalstream.com - > - - > - Brain: The organ with which we think that we think. - > ---------------------------------------------------------------------- > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From mmiller at cruzverde.cl Fri Dec 31 12:38:22 2004 From: mmiller at cruzverde.cl (Markus Miller) Date: Fri, 31 Dec 2004 09:38:22 -0300 Subject: [Linux-cluster] GFS and Storage greater than 2 TB Message-ID: <75E9203E0F0DD511B37E00D0B789D45007E64F45@fcv-stgo.cverde.cl> For what I read, the maximum device size with kernel 2.6 is 16 TB. -----Mensaje original----- De: Jacob Joseph [mailto:jmjoseph at andrew.cmu.edu] Enviado el: Friday, January 01, 2005 3:57 AM Para: linux clustering Asunto: Re: [Linux-cluster] GFS and Storage greater than 2 TB Does this limit still exist with the cvs GFS on a 2.6 kernel? -Jacob Markus Miller wrote: > Thank you for the answer. That is all I needed to know. > > -----Mensaje original----- > De: Rick Stevens [mailto:rstevens at vitalstream.com] > Enviado el: Thursday, December 30, 2004 5:03 PM > Para: linux clistering > Asunto: Re: [Linux-cluster] GFS and Storage greater than 2 TB > > > Markus Miller wrote: > >>Hi, >> >>researching I found a posting to this list made by Kevin Anderson (Date: Tue, 19 Oct 2004 17:56:24 -0500) where he states the following: >> >>---snip--- >>Maximum size of each GFS filesystem for RHEL3 (2.4.x kernel) is 2 TB, >>you can have multiple filesystems of that level. So, to get access to >>10TB of data requires a minimum of 5 separate filesystems/storage >>combinations. >>---snip--- >> >>What do I have to do to achive this? Do I have to configure several GFS clusters in the cluster.ccs file (each of a m?ximum size of 2 TB)? Or do I have to configure one GFS cluster with serveral filesystems each with a maximum size of 2 TB? The GFS Admin Guide is not very precise, but what's really confusing me is the statement on page 12: "2 TB maximum, for total of all storage connected to a GFS cluster." >> >>At the moment we are evaluating to buy servers and storage, therefore I do not have any equipment to do the testing myself. >> >>Any coment is highly apreciated. > > > It's the GFS filesystem that has the limit (actually, it's the 2.4 > kernel). Essentially, "gfs_mkfs" can only handle a maximum of 2TB. > > What he means above is that you have to have five separate partitions > of 2TB each and each with a GFS filesystem on them. You have to mount > those five filesystems separately. If you're using VG/LVM, with a VG > as "vggroup" and LVs in that group as "test1" through "test5": > > mount -t gfs /dev/mapper/vggroup-test1 /mnt/gfs1 > mount -t gfs /dev/mapper/vggroup-test2 /mnt/gfs2 > mount -t gfs /dev/mapper/vggroup-test3 /mnt/gfs3 > mount -t gfs /dev/mapper/vggroup-test4 /mnt/gfs4 > mount -t gfs /dev/mapper/vggroup-test5 /mnt/gfs5 > > How you use them after that is up to you. Just remember that a given > GFS filesystem under kernel 2.4 is limited to 2TB maximum > ---------------------------------------------------------------------- > - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - > - VitalStream, Inc. http://www.vitalstream.com - > - - > - Brain: The organ with which we think that we think. - > ---------------------------------------------------------------------- > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com http://www.redhat.com/mailman/listinfo/linux-cluster From svetljo at gmx.de Fri Dec 31 12:29:38 2004 From: svetljo at gmx.de (Svetoslav Slavtchev) Date: Fri, 31 Dec 2004 13:29:38 +0100 (MET) Subject: [Linux-cluster] sock_alloc 2.6.10 && gfs Message-ID: <22514.1104496178@www4.gmx.net> Hi guys, it seems sock_alloc became static in 2.6.10 and i'm not sure how exactly to fix gfs ( cluster/dlm/lowcomms.c:454 memset(&peeraddr, 0, sizeof(peeraddr)); newsock = sock_alloc(); if (!newsock) return -ENOMEM; ) do you think it'll be enough just to revert the change ? (see attached diff ) best, svetljo -- +++ GMX - die erste Adresse f?r Mail, Message, More +++ 1 GB Mailbox bereits in GMX FreeMail http://www.gmx.net/de/go/mail -------------- next part -------------- A non-text attachment was scrubbed... Name: sock_alloc_static_revert.diff Type: text/x-patch Size: 1159 bytes Desc: not available URL: From serge at marketplatz.ru Mon Dec 27 13:19:13 2004 From: serge at marketplatz.ru (Sergey Mikhnevich) Date: Mon, 27 Dec 2004 16:19:13 +0300 Subject: [Linux-cluster] some questions about setting up GFS Message-ID: <11710528372.20041227161913@marketplatz.ru> Hello! We bought HP ProLiant DL380 G4 Packaged Cluster-MSA500 G2 server and after installation of RHEL3 and GFS-6.0.0-15 I have some questions. Because I have no expirience in setting up such systems, please, tell me, which mistakes in configuration I made. Now system is configured this way: /dev/cciss/c0d1 - External Logical Volume, 293.6 Gbytes (RAID 5) =================== [root at hp1 root]# fdisk /dev/cciss/c0d1 Command (m for help): p Disk /dev/cciss/c0d1: 293.6 GB, 293626045440 bytes 255 heads, 63 sectors/track, 35698 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/cciss/c0d1p1 1 9 72261 fd Linux raid autodetect /dev/cciss/c0d1p2 10 35698 286671892+ fd Linux raid autodetect I've found nothing in GFS documentation about partitioning hard drive during setting up GFS. So, I used to experiment. First partition is allocated for <- CCA device ->. I'd like to know if there is enough space and right type of partition, and, at all, is it right way to allocate <- CCA device ->. Second partition was formatted as GFS, so the question is: is selected type of partition right or not? =================== [root at hp1 root]# cat pool0.cfg poolname pool0 minor 0 subpools 2 subpool 0 0 1 gfs_journal pooldevice 0 0 /dev/cciss/c0d1p1 subpool 1 0 1 gfs_data pooldevice 1 0 /dev/cciss/c0d1p2 Actually, I don't know why, but first subpool I've made as gfs_journal :-) Basically, the system works, but something may be wrong. =================== During setting up I've made this command: [root at hp1 root]# ccs_tool create /root/cluster/ /dev/cciss/c0d1p1 [root at hp1 root]# pool_tool -s Device Pool Label ====== ========== /dev/cciss/c0d0 <- partition information -> /dev/cciss/c0d0p1 <- EXT2/3 filesystem -> /dev/cciss/c0d0p2 <- swap device -> /dev/cciss/c0d0p3 <- EXT2/3 filesystem -> /dev/cciss/c0d1 <- partition information -> /dev/cciss/c0d1p1 <- CCA device -> /dev/cciss/c0d1p2 <- GFS filesystem -> I'd like to hear some comments on it. =================== Thanks. -- Sergey Mikhnevich