From mrugeshkarnik at gmail.com Fri May 1 05:15:05 2009 From: mrugeshkarnik at gmail.com (Mrugesh Karnik) Date: Fri, 1 May 2009 10:45:05 +0530 Subject: [Linux-cluster] qdiskd error `does match kernel's reported sector size' In-Reply-To: <1241116045.5206.163.camel@ayanami> References: <200904291547.44205.mrugeshkarnik@gmail.com> <1241116045.5206.163.camel@ayanami> Message-ID: <200905011045.05156.mrugeshkarnik@gmail.com> On Thursday 30 Apr 2009 23:57:25 Lon Hohberger wrote: > What tree did you build on the Debian node? This was a problem awhile > ago but (I thought) has been fixed for some time. I just used the redhat-cluster-suite-2.20081102-1 available in Lenny. Mrugesh From j.buzzard at dundee.ac.uk Fri May 1 09:07:07 2009 From: j.buzzard at dundee.ac.uk (Jonathan Buzzard) Date: Fri, 01 May 2009 10:07:07 +0100 Subject: [Linux-cluster] Hardware options In-Reply-To: References: Message-ID: <1241168827.29906.117.camel@penguin.lifesci.dundee.ac.uk> On Thu, 2009-04-30 at 18:22 +0100, Virginian wrote: > I was hit by a rather large electricity bill recently (at home). My > current cluster set up comprises 2 x HP Proliant DL380 G3s and an MSA > 500 storage array (all three are very heavy on the juice!). I decided > that if I want to continue playing with RHCS at home I needed to look > for a cheaper, greener option. I can easily get a couple of PC's (dual > or quad core cpus and plenty of RAM for running a virtualised cluster) > but the stumbling block has been a cheap low power shared storage > solution. The best that I have come up with so far is an offering from > Maxtor and from LaCie, which basically comprises an esata disk > enclosure that has two firewire 800 ports. I believe that Linux will > support these dual firewire 800 enclosures but I am a little concerned > about the speed (91MB/s) in comparison to a SCSI disk array. Ideally, > I would prefer a disk enclosure / array with dual esata ports but I > haven't been able to find anything. > > My question is, does anybody have a low cost hardware specification > that they are running xen and RHCS on with shared storage that won't > cost the earth and won't hit me in the wallet when it comes to paying > the electricity bill? The cheapest shared storage you are going to get is FireWire. You cannot do shared storage with eSATA. Thing of FireWire as a cheap mans Fibre Channel network in loop mode. However I would roll your own firewire enclosure and fit it out with some Western Digital VelociRaptor 10k RPM drives. As I would say I/O's per second is more important than actual throughput. You could also cheat a bit if you are using more than one physical drive, and use a bridge board per drive, and wire each drive back to the server. I reckon that you could do a couple of quad code nodes, with two 300GB VelociRaptor drives with a power budget under 400W easily, and less if you pay for laptop parts. There are also some nice cases that take two mini-ITX boards if you want to go small. JAB. -- Jonathan A. Buzzard Tel: +441382-386998 Storage Administrator, College of Life Sciences University of Dundee, DD1 5EH From virginian at blueyonder.co.uk Fri May 1 10:42:38 2009 From: virginian at blueyonder.co.uk (Virginian) Date: Fri, 1 May 2009 11:42:38 +0100 Subject: [Linux-cluster] Hardware options References: <1241168827.29906.117.camel@penguin.lifesci.dundee.ac.uk> Message-ID: <94A188CAF75140E0BAB7FE90AD911970@Desktop> Thanks Jonathon, that's very informative and some very good ideas. I like the idea of the 10K RPM disks, I will definitely read up on those. Also, I had been thinking of quad core CPU's too, something in a small form factor or as you say even laptop size. What I am looking for is (ideally) 2 physical machines 1 x Quad Core CPU with Intel Virtualization (for KVM) 4GB RAM External shared storage, anything from 250GB upwards (I like the idea of 2.5" disks perhaps two in RAID 1) The above would give me plenty of horse power to run quite a few guests and enable me to set up a physical and virtual cluster. If I can get the whole lot for under 400W I will be more than pleased. At present my two DL 380's run at 500W approximately as does the MSA 500 disk array. Cutting my power consumptiion by nearly 75% definitely appeals!! Anybody else got any example of a lower power set up for home use? Regards John ----- Original Message ----- From: "Jonathan Buzzard" To: "linux clustering" Sent: Friday, May 01, 2009 10:07 AM Subject: Re: [Linux-cluster] Hardware options > > On Thu, 2009-04-30 at 18:22 +0100, Virginian wrote: >> I was hit by a rather large electricity bill recently (at home). My >> current cluster set up comprises 2 x HP Proliant DL380 G3s and an MSA >> 500 storage array (all three are very heavy on the juice!). I decided >> that if I want to continue playing with RHCS at home I needed to look >> for a cheaper, greener option. I can easily get a couple of PC's (dual >> or quad core cpus and plenty of RAM for running a virtualised cluster) >> but the stumbling block has been a cheap low power shared storage >> solution. The best that I have come up with so far is an offering from >> Maxtor and from LaCie, which basically comprises an esata disk >> enclosure that has two firewire 800 ports. I believe that Linux will >> support these dual firewire 800 enclosures but I am a little concerned >> about the speed (91MB/s) in comparison to a SCSI disk array. Ideally, >> I would prefer a disk enclosure / array with dual esata ports but I >> haven't been able to find anything. >> >> My question is, does anybody have a low cost hardware specification >> that they are running xen and RHCS on with shared storage that won't >> cost the earth and won't hit me in the wallet when it comes to paying >> the electricity bill? > > The cheapest shared storage you are going to get is FireWire. You cannot > do shared storage with eSATA. Thing of FireWire as a cheap mans Fibre > Channel network in loop mode. > > However I would roll your own firewire enclosure and fit it out with > some Western Digital VelociRaptor 10k RPM drives. As I would say I/O's > per second is more important than actual throughput. > > You could also cheat a bit if you are using more than one physical > drive, and use a bridge board per drive, and wire each drive back to the > server. > > I reckon that you could do a couple of quad code nodes, with two 300GB > VelociRaptor drives with a power budget under 400W easily, and less if > you pay for laptop parts. There are also some nice cases that take two > mini-ITX boards if you want to go small. > > JAB. > > -- > Jonathan A. Buzzard Tel: +441382-386998 > Storage Administrator, College of Life Sciences > University of Dundee, DD1 5EH > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From j.buzzard at dundee.ac.uk Fri May 1 14:14:20 2009 From: j.buzzard at dundee.ac.uk (Jonathan Buzzard) Date: Fri, 01 May 2009 15:14:20 +0100 Subject: [Linux-cluster] Hardware options In-Reply-To: <94A188CAF75140E0BAB7FE90AD911970@Desktop> References: <1241168827.29906.117.camel@penguin.lifesci.dundee.ac.uk> <94A188CAF75140E0BAB7FE90AD911970@Desktop> Message-ID: <1241187260.4554.26.camel@penguin.lifesci.dundee.ac.uk> On Fri, 2009-05-01 at 11:42 +0100, Virginian wrote: > Thanks Jonathon, that's very informative and some very good ideas. > > I like the idea of the 10K RPM disks, I will definitely read up on those. Well worth it, make a big difference to running any virtualization solution. > > Also, I had been thinking of quad core CPU's too, something in a small form > factor or as you say even laptop size. What I am looking for is (ideally) > Have you looked at the ZOTAC GeForce 9300-ITX WiFi Mini-ITX board? It is a mini-ITX board that takes a up to a Core2 Extreme with 8GB of RAM, a 1Gbps Ethernet adaptor. No FireWire, but does have a PCI-E x16 slot, so you could add an adaptor in. You might even be able to squeeze two of these with two drives into the Travla C147 Dual Mini-ITX rackmount case. > 2 physical machines > > 1 x Quad Core CPU with Intel Virtualization (for KVM) > 4GB RAM More RAM, if you want to do virtualization this is what limits the number of guests more than anything. I would say 8GB is a minimum. > External shared storage, anything from 250GB upwards (I like the idea of > 2.5" disks perhaps two in RAID 1) > The VelociRaptor is a 2.5" SATA drive, under 10W a drive. I upgraded my workstation to the old 150GB Raptor drives a couple years back and it made a big difference when running lots of guests on VMware workstations. > The above would give me plenty of horse power to run quite a few guests and > enable me to set up a physical and virtual cluster. If I can get the whole > lot for under 400W I will be more than pleased. At present my two DL 380's > run at 500W approximately as does the MSA 500 disk array. Cutting my power > consumptiion by nearly 75% definitely appeals!! Take a look at the picoPSU power supplies. They are small and efficient, and pick the right one (for your application the M3) and you can do a UPS direct from a lead acid battery in the form of a battery backed power supply. Much more frugal than a normal UPS. Even if you don't want a UPS, one step down from mains to a beefy 12V, is more efficient. > Anybody else got any example of a lower power set up for home use? I have doing it for some time, but on VIA and now Atom boards. My current setup has a power draw *under* 30W at the wall plug, for which I get a 1.2GHz Via C7 with 1GB RAM, with a PCI ADSL card, 1GbE, 100GB of RAID-1 7200RPM, WiFi and with a battery backed PSU. It's role is a home file server, come ADSL gateway, come Wireless access point. I am looking at a new setup which will have an Atom N330 with 2GB of RAM, and a pair of 300GB VelociRaptors and a pair of 2TB 3.5" drives, cause I want to ditch the external firewire drives. The power budget for this will be under 50W and I will reuse the battery backed PSU. I would have thought that under 300W would easily be achievable using desktop processors. If 4GB of RAM is definitely enough, then you could go Socket P, pick one of a range of mini-ITX boards and go under 150W, possibly 100W. However this will bump the cost up because you are buying laptop parts. JAB. -- Jonathan A. Buzzard Tel: +441382-386998 Storage Administrator, College of Life Sciences University of Dundee, DD1 5EH From michael.osullivan at auckland.ac.nz Fri May 1 23:33:16 2009 From: michael.osullivan at auckland.ac.nz (Michael O'Sullivan) Date: Sat, 02 May 2009 11:33:16 +1200 Subject: [Linux-cluster] GFS/GFS2 problems with iozone Message-ID: <49FB86BC.104@auckland.ac.nz> Hi everyone, I am having some problems testing a GFS system using iozone. I am running CentOS 2.6.18-128.1.6.el5 and have a two node cluster with a GFS installed on a shared iSCSI target. The GFS sits on top of a 1.79TB clustered logical volume and can be mounted successfully on both cluster nodes. When using iozone to test performance everything goes smoothly until I get to a file size of 2GB and a record length of 2048. Then iozone exits with the error Error fwriting block 250, fd= 7 and (as far as I can tell) the GFS becomes corrupted fatal: invalid metadata block bh = 12912396 (magic) function = gfs_get_meta_buffer file = /builddir/build/BUILD/gfs-kmod-0.1.31/_kmod_build_/src/gfs/dio.c, line = 1225 Can anyone shed some light on what is happening? Kind regards, Mike O'S From theophanis_kontogiannis at yahoo.gr Sat May 2 12:15:00 2009 From: theophanis_kontogiannis at yahoo.gr (Theophanis Kontogiannis) Date: Sat, 2 May 2009 15:15:00 +0300 Subject: [Linux-cluster] Question about controlling the start of services with RIND In-Reply-To: <1241116158.5206.165.camel@ayanami> References: <006901c9c818$e0731ae0$a15950a0$@gr> <1241116158.5206.165.camel@ayanami> Message-ID: <007401c9cb1f$9ec07ed0$dc417c70$@gr> Hello Lon and All, Thank you for your answer. It helps to start getting somewhere. Is there any place I can get all the possible options for RIND and cluster.conf? Anyway, I did not made clear that in my two cluster node, the storage service for node 1 is different then the storage service for node 2. Both start it since both mount a GFS2 filesystem. Some of the other services have as preferred node 1, and some have as preffered node 2. If I put as depend="storage-t1" on a service that should start on node t1, how would the service behave if node t1 never boots, but the service is configured to start on node t2 as second choice? The service will start anyway? Is there any way to put multiple dependencies for the service in an OR way, like ... So if either node has started and if either storage service has started, the service will start? I am attaching my cluster.conf to make my point more clear, and referring to service Apache-t1-t2. Thank you All four your time. > -----Original Message----- > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster- > bounces at redhat.com] On Behalf Of Lon Hohberger > Sent: Thursday, April 30, 2009 9:29 PM > To: linux clustering > Subject: Re: [Linux-cluster] Question about controlling the start of > services with RIND > > On Tue, 2009-04-28 at 18:49 +0300, Theophanis Kontogiannis wrote: > > > Could I use RIND somehow to make the rest of the clustered services to > > start only if the filesystem service has started? > > Yeah, just set: > > > ... > > > -- Lon > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- A non-text attachment was scrubbed... Name: cluster.conf Type: application/octet-stream Size: 2570 bytes Desc: not available URL: From sunhux at gmail.com Sun May 3 03:59:38 2009 From: sunhux at gmail.com (sunhux G) Date: Sun, 3 May 2009 11:59:38 +0800 Subject: [Linux-cluster] 2 node qdiskd cluster gave "Quorum Dissolved" Message-ID: <60f08e700905022059j380a0212gbaad9b351f19b0dc@mail.gmail.com> Hi I have a 2 node qdiskd cluster (OS is RHES 5.0) with 2x heartbeat cross cables between the 2 nodes Currently we manually issue the following commands to start the cluster services in the sequence below : a) cd /etc/init.d b) ./cman start c) ./clvmd ... d) ./qdiskd ... e) ./rgmanager ... & on the primary node, issue "clusvcadm ..... Oracle_Service" to start oracle services which will also mount the SAN partition. Occasionally, we ran into the error below & cluster breaks on both nodes (ie SAN partition unmounted on both and Oracle services stopped on both) : lurgmgrd[5843]: #1: Quorum Dissolved What's wrong? Usually when this happens, I could usually make the first node rejoin the cluster + mount the SAN partition but the 2nd node usually can't rejoin the cluster/mount SAN and has to be rebooted and reissued with the commands a-e for it to rejoin the cluster. Thanks for any insights -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpeterso at redhat.com Mon May 4 15:05:20 2009 From: rpeterso at redhat.com (Bob Peterson) Date: Mon, 4 May 2009 11:05:20 -0400 (EDT) Subject: [Linux-cluster] GFS/GFS2 problems with iozone In-Reply-To: <49FB86BC.104@auckland.ac.nz> Message-ID: <494270654.5591241449520328.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> ----- "Michael O'Sullivan" wrote: | Hi everyone, | | I am having some problems testing a GFS system using iozone. I am | running CentOS 2.6.18-128.1.6.el5 and have a two node cluster with a | GFS | installed on a shared iSCSI target. The GFS sits on top of a 1.79TB | clustered logical volume and can be mounted successfully on both | cluster | nodes. | | When using iozone to test performance everything goes smoothly until I | | get to a file size of 2GB and a record length of 2048. Then iozone | exits | with the error | | Error fwriting block 250, fd= 7 | | and (as far as I can tell) the GFS becomes corrupted | | fatal: invalid metadata block | bh = 12912396 (magic) | function = gfs_get_meta_buffer | file = | /builddir/build/BUILD/gfs-kmod-0.1.31/_kmod_build_/src/gfs/dio.c, | line = 1225 | | Can anyone shed some light on what is happening? | | Kind regards, Mike O'S Hi Mike, Are you running iozone on a single node or both simultaneously? If it's running on two nodes, please make sure that both nodes have the iSCSI target mounted with lock_dlm protocol (not lock_nolock). Also, we need to make sure that they're not trying to use the same files in the file system because I think iozone is not cluster-aware. But even so, the file system should not be corrupted unless one of the nodes is using lock_nolock protocol, or if other boxes are using the iSCSI target without the knowledge of GFS. We regularly run iozone here, in single-node performance trials, and we have never seen this kind of problem. Also, you didn't specify what version of the kmod-gfs package you have installed. I've fixed at least one bug that might account for it, depending on what version of kmod-gfs you're running. I'm not aware of any other problems in the GFS kernel code that can account for this kind of corruption, except for possibly this one: https://bugzilla.redhat.com/show_bug.cgi?id=491369 (A gfs bug that really goes well beyond the nfs usage described in the bug). You can find the patch in the attachments, although I won't guarantee it'll solve your problem. There's a slight chance though. My apologies if you don't have permission to see the bug; that sometimes happens and it's out of my control. I can, however, post the patch if needed. If iozone is being run on a single node, this might be a new bug. If you can still recreate the problem with that patch in place, or if you don't want to try the patch for some reason, perhaps you should open up a bugzilla record and we'll investigate the problem. If we can reproduce it, we'll figure it out and fix it. Regards, Bob Peterson Red Hat GFS From michael.osullivan at auckland.ac.nz Mon May 4 21:37:51 2009 From: michael.osullivan at auckland.ac.nz (Michael O'Sullivan) Date: Tue, 05 May 2009 09:37:51 +1200 Subject: [Linux-cluster] Re: Re: GFS/GFS2 problems with iozone In-Reply-To: <20090504160008.07D0B618605@hormel.redhat.com> References: <20090504160008.07D0B618605@hormel.redhat.com> Message-ID: <49FF602F.2060603@auckland.ac.nz> Date: Mon, 4 May 2009 11:05:20 -0400 (EDT) > From: Bob Peterson > Subject: Re: [Linux-cluster] GFS/GFS2 problems with iozone > To: linux clustering > Message-ID: > <494270654.5591241449520328.JavaMail.root at zmail06.collab.prod.int.phx2.redhat.com> > > Content-Type: text/plain; charset=utf-8 > > ----- "Michael O'Sullivan" wrote: > | Hi everyone, > | > | I am having some problems testing a GFS system using iozone. I am > | running CentOS 2.6.18-128.1.6.el5 and have a two node cluster with a > | GFS > | installed on a shared iSCSI target. The GFS sits on top of a 1.79TB > | clustered logical volume and can be mounted successfully on both > | cluster > | nodes. > | > | When using iozone to test performance everything goes smoothly until I > | > | get to a file size of 2GB and a record length of 2048. Then iozone > | exits > | with the error > | > | Error fwriting block 250, fd= 7 > | > | and (as far as I can tell) the GFS becomes corrupted > | > | fatal: invalid metadata block > | bh = 12912396 (magic) > | function = gfs_get_meta_buffer > | file = > | /builddir/build/BUILD/gfs-kmod-0.1.31/_kmod_build_/src/gfs/dio.c, > | line = 1225 > | > | Can anyone shed some light on what is happening? > | > | Kind regards, Mike O'S > > Hi Mike, > > Are you running iozone on a single node or both simultaneously? > If it's running on two nodes, please make sure that both nodes have > the iSCSI target mounted with lock_dlm protocol (not lock_nolock). > Also, we need to make sure that they're not trying to use the same > files in the file system because I think iozone is not cluster-aware. > But even so, the file system should not be corrupted unless one of > the nodes is using lock_nolock protocol, or if other boxes are > using the iSCSI target without the knowledge of GFS. > > We regularly run iozone here, in single-node performance trials, and > we have never seen this kind of problem. > > Also, you didn't specify what version of the kmod-gfs package you have > installed. I've fixed at least one bug that might account for it, > depending on what version of kmod-gfs you're running. > > I'm not aware of any other problems in the GFS kernel code that can > account for this kind of corruption, except for possibly this one: > > https://bugzilla.redhat.com/show_bug.cgi?id=491369 > > (A gfs bug that really goes well beyond the nfs usage described in the bug). > You can find the patch in the attachments, although I won't guarantee > it'll solve your problem. There's a slight chance though. > My apologies if you don't have permission to see the bug; that sometimes > happens and it's out of my control. I can, however, post the patch > if needed. > > If iozone is being run on a single node, this might be a new bug. If you can > still recreate the problem with that patch in place, or if you don't want > to try the patch for some reason, perhaps you should open up a bugzilla > record and we'll investigate the problem. If we can reproduce it, we'll > figure it out and fix it. > > Regards, > > Bob Peterson > Red Hat GFS > Hi Bob, I have changed back to GFS2 (as I realised this is now production ready, is that correct?), but I am still having similar problems. I am running iozone on a single node and accessing the mount point of GFS2 running with lock_dlm. Note that the GFS2 is created on a multipathed device created via iSCSI/DRBD. However, I run the following commands: gfs2_fsck # which shows no errors on either node mount -t gfs2 /dev/iscsi_mirror/lvol0 /mnt/iscsi_mirror/ #mounts the file system (on top of iSCSI/DRBD) on both nodes /usr/src/ioszone3_321/src/current/iozone -Ra -g 4G -f /mnt/iscsi_mirror/test # Only on node 1 This gets to 1048576 KB and reclen 256 before giving Error reading block 1018 b6e00000 I can fix the GFS2 using gfs2_fsck (it fixes some dirty journals, but no other changes). I don't have the error messages from this latest test as I ran it over the weekend and /var/log/messages doesn't have the error messages anymore. I can recreate this test and record the error messages if necessary, but I wonder if the patch you talked about also exists for GFS2? Thanks very much for your help, Mike From esggrupos at gmail.com Tue May 5 15:37:56 2009 From: esggrupos at gmail.com (ESGLinux) Date: Tue, 5 May 2009 17:37:56 +0200 Subject: [Linux-cluster] help configuring HP ILO Message-ID: <3128ba140905050837j5654c099k4c94b3cd8a4343de@mail.gmail.com> Hello all, I?m configuring a 2 nodes cluster on 2 servers HP Proliant DL165 G5 This servers have HP ProLiant Lights-Out 100 Remote Management and I want to use it as fencing device. My first idea was to configure them with IPMI and it works almost fine but I have detected that when I have the network down, the fence devices doesn't work because the nodes can't reach the othe node to fence it. I have tried with a dedicated switch and a direct cable but it doesn't work and I begin to think I?m doing something wrong because the interface with the ipmi configured doesnt appear on the servers. I?ll try to explain: I have node1 eth0: IP1 eth1: IP2 In the bios I have configured ethMng : IP3 node2 eth0: IP4 eth1: IP5 In the bios I have configured ethMng : IP6 with the network up all works fine and I can use fence_node to fence the nodes, and the cluster works fine. But, if I disconnect IP1, IP2, IP3, IP4 (This simulate a switch fail) I expect the cluster become fencing with IP3 and IP6 but the system doesn't find these IP?s and all the cluster hungs. Looking the fence devices avaliable with conga I have seen that there is one called: ** HP iLOwith this parameters to configure: Name Hostname Login Password Password Script (optional) All are self explanatory but I don?t know what to put on Hostname (which hostname? the same machine, the other? FQDN.. IP....) So, I have 2 questions: If I use IPMI what I?m doing wrong? and If I use HP iLO, what I need to configure??? any idea, manual, doc, suggest.... is welcome thanks in advance ESG -------------- next part -------------- An HTML attachment was scrubbed... URL: From cthulhucalling at gmail.com Tue May 5 16:01:56 2009 From: cthulhucalling at gmail.com (Ian Hayes) Date: Tue, 5 May 2009 12:01:56 -0400 Subject: [Linux-cluster] help configuring HP ILO In-Reply-To: <3128ba140905050837j5654c099k4c94b3cd8a4343de@mail.gmail.com> References: <3128ba140905050837j5654c099k4c94b3cd8a4343de@mail.gmail.com> Message-ID: <36df569a0905050901y3722c456i695f89174340d2ac@mail.gmail.com> For hostname you can put the FQDN or IP address... I believe that you're a bit confused what iLO is capable of. IP3 and IP6 are for the iLO, the cluster can't use them for networking. The cluster members need to be able to reach the iLO (IP3 and 6 in this case) from eth0 or eth1. In a 2-node cluster, this can be as simple as connecting eth0 or eth1 on one node to the iLO of the other node via crossover cable. The iLO is its own device that exists outside of the operating system. Here's an example of a cluster that I've built previously that is similar to your setup Host1: eth0 192.168.0.1 (host1) eth1 10.1.1.1 (host1-management) iLO 10.1.1.2 Host 2 eth0 192.168.0.2 (host2) eth1: 10.1.1.3 (host2-management) iLO 10.1.1.4 All cluster management communication in this cluster is via eth1. I specified host1-management and host2-management as the hostnames in the cluster config to partition off cluster traffic from the interfaces that are actually doing the VIP work. The nodes provide a virtual IP on eth0, and a script service, with the daemon bound to the VIP. For the iLOs and eth1, you could either plug them into a switch on their own non-trunked VLAN, or you can connect eth1 of host1 to the iLO of host 2, and eth1 of host2 to the iLO of host1. Both eth1 and iLOs don't need a gateway since they're on the same subnet. To configure the iLO, you just set up the correct IP address, mask and create a username and password that has the appropriate privileges (power). These get put into the cluster.conf file via system-config-cluster or Luci. You would need to create two fence resources. In the above case, I would create a Fence_Host_1 and Fence_Host_2 dence devices, using fence_ilo. Fence_Host_1 would have the IP address of host1's iLO, a valid login and password for that iLO. Host2 is similar, but has the IP address of host2's iLO. Attach Fence_Host_1 to host1 and Fence_Host_2 to host2. This way, the entire cluster knows "to fence host1, I see that I need to use the Fence_Host_1 method. Fence_host_1 uses fence_ilo as its method, target ip address 10.1.1.2, username foo, password bar. To fence host2, it uses fence_ilo as its method, target address 10.1.1.4, username foobar, password barfoo". These get passed to the fence_ilo script and it handles the rest. You can play with this my manually running fence_ilo. On Tue, May 5, 2009 at 11:37 AM, ESGLinux wrote: > Hello all, > > I?m configuring a 2 nodes cluster on 2 servers HP Proliant DL165 G5 > This servers have HP ProLiant Lights-Out 100 Remote Management and I want > to use it as fencing device. > > My first idea was to configure them with IPMI and it works almost fine but > I have detected that when I have the network down, the fence devices doesn't > work because the nodes can't reach the othe node to fence it. > > I have tried with a dedicated switch and a direct cable but it doesn't work > and I begin to think I?m doing something wrong because the interface with > the ipmi configured doesnt appear on the servers. I?ll try to explain: > > I have > node1 > eth0: IP1 > eth1: IP2 > In the bios I have configured ethMng : IP3 > > node2 > eth0: IP4 > eth1: IP5 > In the bios I have configured ethMng : IP6 > > > with the network up all works fine and I can use fence_node to fence the > nodes, and the cluster works fine. > > But, if I disconnect IP1, IP2, IP3, IP4 (This simulate a switch fail) I > expect the cluster become fencing with IP3 and IP6 but the system doesn't > find these IP?s and all the cluster hungs. > > Looking the fence devices avaliable with conga I have seen that there is > one called: > ** > HP iLOwith this parameters to configure: > Name > Hostname > Login > Password > Password Script (optional) > > All are self explanatory but I don?t know what to put on Hostname (which > hostname? the same machine, the other? FQDN.. IP....) > > So, I have 2 questions: > If I use IPMI what I?m doing wrong? > and > If I use HP iLO, what I need to configure??? > > any idea, manual, doc, suggest.... is welcome > > thanks in advance > > ESG > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From esggrupos at gmail.com Tue May 5 16:54:04 2009 From: esggrupos at gmail.com (ESGLinux) Date: Tue, 5 May 2009 18:54:04 +0200 Subject: [Linux-cluster] help configuring HP ILO In-Reply-To: <36df569a0905050901y3722c456i695f89174340d2ac@mail.gmail.com> References: <3128ba140905050837j5654c099k4c94b3cd8a4343de@mail.gmail.com> <36df569a0905050901y3722c456i695f89174340d2ac@mail.gmail.com> Message-ID: <3128ba140905050954t648a283ev601e5352a8f6d9f3@mail.gmail.com> Hello, Thanks for your answer... 2009/5/5 Ian Hayes > For hostname you can put the FQDN or IP address... > > I believe that you're a bit confused what iLO is capable of. I absolutelly agree with you ;-) > IP3 and IP6 are for the iLO, the cluster can't use them for networking. I don't use for networking (I think...) I only want to use it to fence.... (i?m begin to think this is my mistake) > The cluster members need to be able to reach the iLO (IP3 and 6 in this > case) from eth0 or eth1. I think I could reach the iLO from the interfaces ILO (In my configuration ethMng, IP3 and IP6) In a 2-node cluster, this can be as simple as connecting eth0 or eth1 on one > node to the iLO of the other node via crossover cable. The iLO is its own > device that exists outside of the operating system. > > Here's an example of a cluster that I've built previously that is similar > to your setup > > Host1: > eth0 192.168.0.1 (host1) > eth1 10.1.1.1 (host1-management) > iLO 10.1.1.2 > > Host 2 > eth0 192.168.0.2 (host2) > eth1: 10.1.1.3 (host2-management) > iLO 10.1.1.4 > > All cluster management communication in this cluster is via eth1. I > specified host1-management and host2-management as the hostnames in the > cluster config to partition off cluster traffic from the interfaces that are > actually doing the VIP work. The nodes provide a virtual IP on eth0, and a > script service, with the daemon bound to the VIP. For the iLOs and eth1, you > could either plug them into a switch on their own non-trunked VLAN, or you > can connect eth1 of host1 to the iLO of host 2, and eth1 of host2 to the iLO > of host1. Both eth1 and iLOs don't need a gateway since they're on the same > subnet. > If I have understand, if I use a dedicated switch, I must to connect IP2, IP3, IP5 and IP6 to the same switche and IP1 and IP4 to the service switch, isn?t it? > > To configure the iLO, you just set up the correct IP address, mask and > create a username and password that has the appropriate privileges (power). > These get put into the cluster.conf file via system-config-cluster or Luci. > You would need to create two fence resources. In the above case, I would > create a Fence_Host_1 and Fence_Host_2 dence devices, using fence_ilo. > this is ok, is what I have done but with fence_ipmi > > Fence_Host_1 would have the IP address of host1's iLO, a valid login and > password for that iLO. Host2 is similar, but has the IP address of host2's > iLO. Attach Fence_Host_1 to host1 and Fence_Host_2 to host2. This way, the > entire cluster knows "to fence host1, I see that I need to use the > Fence_Host_1 method. Fence_host_1 uses fence_ilo as its method, target ip > address 10.1.1.2, username foo, password bar. To fence host2, it uses > fence_ilo as its method, target address 10.1.1.4, username foobar, password > barfoo". These get passed to the fence_ilo script and it handles the rest. > You can play with this my manually running fence_ilo. I think I have understood My problem was that I thought I can reach the iLO interfaces only using the iLO interfaces. I?ll try this configuration and I will post my results, thanks for your answer ESG > > > On Tue, May 5, 2009 at 11:37 AM, ESGLinux wrote: > >> Hello all, >> >> I?m configuring a 2 nodes cluster on 2 servers HP Proliant DL165 G5 >> This servers have HP ProLiant Lights-Out 100 Remote Management and I want >> to use it as fencing device. >> >> My first idea was to configure them with IPMI and it works almost fine but >> I have detected that when I have the network down, the fence devices doesn't >> work because the nodes can't reach the othe node to fence it. >> >> I have tried with a dedicated switch and a direct cable but it doesn't >> work and I begin to think I?m doing something wrong because the interface >> with the ipmi configured doesnt appear on the servers. I?ll try to explain: >> >> I have >> node1 >> eth0: IP1 >> eth1: IP2 >> In the bios I have configured ethMng : IP3 >> >> node2 >> eth0: IP4 >> eth1: IP5 >> In the bios I have configured ethMng : IP6 >> >> >> with the network up all works fine and I can use fence_node to fence the >> nodes, and the cluster works fine. >> >> But, if I disconnect IP1, IP2, IP3, IP4 (This simulate a switch fail) I >> expect the cluster become fencing with IP3 and IP6 but the system doesn't >> find these IP?s and all the cluster hungs. >> >> Looking the fence devices avaliable with conga I have seen that there is >> one called: >> ** >> HP iLOwith this parameters to configure: >> Name >> Hostname >> Login >> Password >> Password Script (optional) >> >> All are self explanatory but I don?t know what to put on Hostname (which >> hostname? the same machine, the other? FQDN.. IP....) >> >> So, I have 2 questions: >> If I use IPMI what I?m doing wrong? >> and >> If I use HP iLO, what I need to configure??? >> >> any idea, manual, doc, suggest.... is welcome >> >> thanks in advance >> >> ESG >> >> >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From EliasM at dnb.com Tue May 5 17:48:09 2009 From: EliasM at dnb.com (Elias, Michael) Date: Tue, 5 May 2009 13:48:09 -0400 Subject: [Linux-cluster] Heartbeat time outs in rhel4 understanding Message-ID: <437A36C1327D794D87D207AC80BDD8FD0B05191C@DNBMSXBH002.dnbint.net> I am trying to understand how these timers interact with each other. In a RHEL4 cluster the heartbeat defaults are; hello_timer:5 max_retries:5 deadnode_timeout:21 Meaning a heartbeat message is sent every 5 seconds, if it fails to receive a response it will start a deadnode counter @ 21 seconds. It will also try to send 5 more heartbeat requests. What is the interval of those retries? If none of those requests receive a response. 5 seconds pass.. there is 15 seconds left on the deadnode timer and we try upto 5 times to get a response.... This goes on until we hit the 4th iteration of the hellotimer it tries again upto 5 times and fails... we then hit the 21 second on the deadnode time.. fenced takes over and wham reboot. Is my understanding of this correct???? Thanks for any help.. Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From cthulhucalling at gmail.com Tue May 5 17:12:31 2009 From: cthulhucalling at gmail.com (Ian Hayes) Date: Tue, 5 May 2009 13:12:31 -0400 Subject: [Linux-cluster] help configuring HP ILO In-Reply-To: <3128ba140905050954t648a283ev601e5352a8f6d9f3@mail.gmail.com> References: <3128ba140905050837j5654c099k4c94b3cd8a4343de@mail.gmail.com> <36df569a0905050901y3722c456i695f89174340d2ac@mail.gmail.com> <3128ba140905050954t648a283ev601e5352a8f6d9f3@mail.gmail.com> Message-ID: <36df569a0905051012n59bf3dcfn340ce96cfc7b62c5@mail.gmail.com> On Tue, May 5, 2009 at 12:54 PM, ESGLinux wrote: > Hello, > > Thanks for your answer... > > 2009/5/5 Ian Hayes >> >> >> >> All cluster management communication in this cluster is via eth1. I >> specified host1-management and host2-management as the hostnames in the >> cluster config to partition off cluster traffic from the interfaces that are >> actually doing the VIP work. The nodes provide a virtual IP on eth0, and a >> script service, with the daemon bound to the VIP. For the iLOs and eth1, you >> could either plug them into a switch on their own non-trunked VLAN, or you >> can connect eth1 of host1 to the iLO of host 2, and eth1 of host2 to the iLO >> of host1. Both eth1 and iLOs don't need a gateway since they're on the same >> subnet. >> > > If I have understand, if I use a dedicated switch, I must to connect IP2, > IP3, IP5 and IP6 to the same switche and IP1 and IP4 to the service switch, > isn?t it? > If you're using a dedicated switch for cluster management, yes. Assuming that IP1 and 4 are your public facing interfaces that will be holding the service. You can use 2 and 5 for cluster management and fencing via IP3 and 6. Or you can use crossover cables IP2 to IP6, and IP5 to IP3, but you will have to run your cluster management over eth0, on the same interface that you have your services bound to. Your choice. But you can't reach the iLO interfaces using the other iLOs. Think of them as "recieve only". It's up to one of the hosts to establish a connection to them and issue commands. -------------- next part -------------- An HTML attachment was scrubbed... URL: From garromo at us.ibm.com Tue May 5 19:31:01 2009 From: garromo at us.ibm.com (Gary Romo) Date: Tue, 5 May 2009 13:31:01 -0600 Subject: [Linux-cluster] service failover Message-ID: Hello. How can I tell when a service has failed over? I'm looking for date and time stamps. Thanks. -Gary -------------- next part -------------- An HTML attachment was scrubbed... URL: From cthulhucalling at gmail.com Tue May 5 20:15:43 2009 From: cthulhucalling at gmail.com (Ian Hayes) Date: Tue, 5 May 2009 16:15:43 -0400 Subject: [Linux-cluster] service failover In-Reply-To: References: Message-ID: <36df569a0905051315x3811bb5encbac1a954b43430@mail.gmail.com> /var/log/messages Message from clurgmgrd will say " Service service:servicename started" There will be a bunch of message prior to that announcing that the cluster is in trouble, it is fencing the downed node, and that a node is taking over the service. The service will go throught he normal startup messages. The final message will be the one that says that the service is started. On Tue, May 5, 2009 at 3:31 PM, Gary Romo wrote: > Hello. > > How can I tell when a service has failed over? > I'm looking for date and time stamps. Thanks. > > -Gary > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Harri.Paivaniemi at tieto.com Wed May 6 03:56:15 2009 From: Harri.Paivaniemi at tieto.com (Harri.Paivaniemi at tieto.com) Date: Wed, 6 May 2009 06:56:15 +0300 Subject: [Linux-cluster] service failover References: Message-ID: <41E8D4F07FCE154CBEBAA60FFC92F67754D50E@apollo.eu.tieto.com> If you need a quick-n-dirty way, you can always put something in to services starting script so every time cluster says start|stop to that service, you can send mail to yourself etc and you don't have to grep messages-log ;) -hjp -----Original Message----- From: linux-cluster-bounces at redhat.com on behalf of Gary Romo Sent: Tue 5/5/2009 22:31 To: linux-cluster at redhat.com Subject: [Linux-cluster] service failover Hello. How can I tell when a service has failed over? I'm looking for date and time stamps. Thanks. -Gary -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 2681 bytes Desc: not available URL: From esggrupos at gmail.com Wed May 6 06:53:23 2009 From: esggrupos at gmail.com (ESGLinux) Date: Wed, 6 May 2009 08:53:23 +0200 Subject: [Linux-cluster] help configuring HP ILO In-Reply-To: <36df569a0905051012n59bf3dcfn340ce96cfc7b62c5@mail.gmail.com> References: <3128ba140905050837j5654c099k4c94b3cd8a4343de@mail.gmail.com> <36df569a0905050901y3722c456i695f89174340d2ac@mail.gmail.com> <3128ba140905050954t648a283ev601e5352a8f6d9f3@mail.gmail.com> <36df569a0905051012n59bf3dcfn340ce96cfc7b62c5@mail.gmail.com> Message-ID: <3128ba140905052353v58b79832j5905eea90f5b793a@mail.gmail.com> thanks again Ian, the key is what you have said "Think of them as "recieve only"" Now all is clear like water ;-) thank you very much for your help Greetings ESG 2009/5/5 Ian Hayes > On Tue, May 5, 2009 at 12:54 PM, ESGLinux wrote: > >> Hello, >> >> Thanks for your answer... >> >> 2009/5/5 Ian Hayes >> >>> >>> >>> All cluster management communication in this cluster is via eth1. I >>> specified host1-management and host2-management as the hostnames in the >>> cluster config to partition off cluster traffic from the interfaces that are >>> actually doing the VIP work. The nodes provide a virtual IP on eth0, and a >>> script service, with the daemon bound to the VIP. For the iLOs and eth1, you >>> could either plug them into a switch on their own non-trunked VLAN, or you >>> can connect eth1 of host1 to the iLO of host 2, and eth1 of host2 to the iLO >>> of host1. Both eth1 and iLOs don't need a gateway since they're on the same >>> subnet. >>> >> >> If I have understand, if I use a dedicated switch, I must to connect IP2, >> IP3, IP5 and IP6 to the same switche and IP1 and IP4 to the service switch, >> isn?t it? >> > > If you're using a dedicated switch for cluster management, yes. Assuming > that IP1 and 4 are your public facing interfaces that will be holding the > service. You can use 2 and 5 for cluster management and fencing via IP3 and > 6. > > Or you can use crossover cables IP2 to IP6, and IP5 to IP3, but you will > have to run your cluster management over eth0, on the same interface that > you have your services bound to. Your choice. > > But you can't reach the iLO interfaces using the other iLOs. Think of them > as "recieve only". It's up to one of the hosts to establish a connection to > them and issue commands. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ccaulfie at redhat.com Wed May 6 08:06:17 2009 From: ccaulfie at redhat.com (Chrissie Caulfield) Date: Wed, 06 May 2009 09:06:17 +0100 Subject: [Linux-cluster] Heartbeat time outs in rhel4 understanding In-Reply-To: <437A36C1327D794D87D207AC80BDD8FD0B05191C@DNBMSXBH002.dnbint.net> References: <437A36C1327D794D87D207AC80BDD8FD0B05191C@DNBMSXBH002.dnbint.net> Message-ID: <4A0144F9.5040701@redhat.com> Elias, Michael wrote: > I am trying to understand how these timers interact with each other. > > > > In a RHEL4 cluster the heartbeat defaults are; > > hello_timer:5 > > max_retries:5 > > deadnode_timeout:21 > > > > Meaning a heartbeat message is sent every 5 seconds, if it fails to > receive a response it will start a deadnode counter @ 21 seconds. It > will also try to send 5 more heartbeat requests. What is the interval of > those retries? If none of those requests receive a response. 5 seconds > pass.. there is 15 seconds left on the deadnode timer and we try upto 5 > times to get a response?. This goes on until we hit the 4^th iteration > of the hellotimer it tries again upto 5 times and fails? we then hit the > 21 second on the deadnode time.. fenced takes over and wham reboot. > > > > Is my understanding of this correct???? > No, I'm afraid it isn't :-) max_retries has nothing to do with the heartbeat. It is to do with cluster messages, such as service join requests, clvmd messages or the messages used in the membership protocol. So the heartbeat system is just a 5 second heartbeat and after 21 seconds the node will be evicted from the cluster and (usually) fenced. The same happens for data messages if max_retries is exceeded. The retry period here starts at 1 second and increases each time to avoid filling the ethernet buffers. I hope this helps, Chrissie From gfs2etis at ensea.fr Wed May 6 10:41:05 2009 From: gfs2etis at ensea.fr (gfs2etis) Date: Wed, 06 May 2009 12:41:05 +0200 Subject: [Linux-cluster] gsf2 Message-ID: <1241606465.8236.39.camel@tyr.ensea.fr> Hello, I am a newbie with gfs2, i have set 4 nodes on a fiber channel SAN synchronized through gfs2. When one machine crash all gfs2 crash, especially when there is a lot of IO on the gfs2 devices. I have no message on the system when it crash (???) version : gfs2-utils-2.03.07-2 thanks -- gfs2etis ETIS From ntadmin at fi.upm.es Wed May 6 10:59:45 2009 From: ntadmin at fi.upm.es (Miguel Sanchez) Date: Wed, 06 May 2009 12:59:45 +0200 Subject: [Linux-cluster] Necessary a delay to restart cman? Message-ID: <4A016DA1.5090404@fi.upm.es> Hi. I have a CentOS 5.3 cluster with two nodes. If I execute service cman restart within a node, or stop + start after few seconds, another node doesn?t recognize this membership return and its fellow stay forever offline. For example: * Before cman restart: node1# cman_tool status Version: 6.1.0 Config Version: 6 Cluster Name: CSVirtualizacion Cluster Id: 42648 Cluster Member: Yes Cluster Generation: 202600 Membership state: Cluster-Member Nodes: 2 Expected votes: 1 Total votes: 2 Quorum: 1 Active subsystems: 7 Flags: 2node Dirty Ports Bound: 0 Node name: patty Node ID: 1 Multicast addresses: 224.0.0.133 Node addresses: 138.100.8.70 * After cman stop for node2 (and before a number seconds < token parameter) node1# cman_tool status Version: 6.1.0 Config Version: 6 Cluster Name: CSVirtualizacion Cluster Id: 42648 Cluster Member: Yes Cluster Generation: 202600 Membership state: Cluster-Member Nodes: 2 Expected votes: 1 Total votes: 1 Quorum: 1 Active subsystems: 7 Flags: 2node Dirty Ports Bound: 0 Node name: patty Node ID: 1 Multicast addresses: 224.0.0.133 Node addresses: 138.100.8.70 Wed May 6 12:29:38 CEST 2009 * After cman stop for node2 (and after a number seconds > token parameter) node1# date; cman_tool status Version: 6.1.0 Config Version: 6 Cluster Name: CSVirtualizacion Cluster Id: 42648 Cluster Member: Yes Cluster Generation: 202604 Membership state: Cluster-Member Nodes: 1 Expected votes: 1 Total votes: 1 Quorum: 1 Active subsystems: 7 Flags: 2node Dirty Ports Bound: 0 Node name: patty Node ID: 1 Multicast addresses: 224.0.0.133 Node addresses: 138.100.8.70 Wed May 6 12:29:47 CEST 2009 /var/log/messages: May 6 12:35:20 node2 openais[17262]: [TOTEM] The token was lost in the OPERATIONAL state. May 6 12:35:20 node2 openais[17262]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). May 6 12:35:20 node2 openais[17262]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). May 6 12:35:20 node2 openais[17262]: [TOTEM] entering GATHER state from 2. May 6 12:35:25 node2 openais[17262]: [TOTEM] entering GATHER state from 0. May 6 12:35:25 node2 openais[17262]: [TOTEM] Creating commit token because I am the rep. May 6 12:35:25 node2 openais[17262]: [TOTEM] Saving state aru 26 high seq received 26 May 6 12:35:25 node2 openais[17262]: [TOTEM] Storing new sequence id for ring 31780 May 6 12:35:25 node2 openais[17262]: [TOTEM] entering COMMIT state. May 6 12:35:25 node2 openais[17262]: [TOTEM] entering RECOVERY state. May 6 12:35:25 node2 openais[17262]: [TOTEM] position [0] member 10.10.8.70: May 6 12:35:25 node2 openais[17262]: [TOTEM] previous ring seq 202620 rep 10.10.8.70 May 6 12:35:25 node2 openais[17262]: [TOTEM] aru 26 high delivered 26 received flag 1 May 6 12:35:25 node2 openais[17262]: [TOTEM] Did not need to originate any messages in recovery. May 6 12:35:25 node2 openais[17262]: [TOTEM] Sending initial ORF token May 6 12:35:25 node2 openais[17262]: [CLM ] CLM CONFIGURATION CHANGE May 6 12:35:25 node2 openais[17262]: [CLM ] New Configuration: May 6 12:35:25 node2 openais[17262]: [CLM ] r(0) ip(10.10.8.70) May 6 12:35:25 node2 openais[17262]: [CLM ] Members Left: May 6 12:35:25 node2 openais[17262]: [CLM ] r(0) ip(10.10.8.71) May 6 12:35:25 node2 openais[17262]: [CLM ] Members Joined: May 6 12:35:25 node2 openais[17262]: [CLM ] CLM CONFIGURATION CHANGE May 6 12:35:25 node2 openais[17262]: [CLM ] New Configuration: May 6 12:35:25 node2 openais[17262]: [CLM ] r(0) ip(10.10.8.70) May 6 12:35:25 node2 openais[17262]: [CLM ] Members Left: May 6 12:35:25 node2 openais[17262]: [CLM ] Members Joined: May 6 12:35:25 node2 openais[17262]: [SYNC ] This node is within the primary component and will provide service. May 6 12:35:25 node2 openais[17262]: [TOTEM] entering OPERATIONAL state. May 6 12:35:25 node2 kernel: dlm: closing connection to node 2 May 6 12:35:25 node2 openais[17262]: [CLM ] got nodejoin message 10.10.8.70 May 6 12:35:25 node2 openais[17262]: [CPG ] got joinlist message from node 1 if node2 doesn`t wait for run cman start to the detection the operational token's lost, node1 detect node2 like offline forever. Following attempts for cman restarts don`t change this state: node1# cman_tool nodes Node Sts Inc Joined Name 1 M 202616 2009-05-06 12:34:43 node1 2 X 202628 node2 node2# cman_tool nodes Node Sts Inc Joined Name 1 M 202644 2009-05-06 12:51:04 node1 2 M 202640 2009-05-06 12:51:04 node2 Is it necessary a delay for cman stop + start to avoid this inconsistent state or really is it a bug? Regards. From reggaestar at gmail.com Wed May 6 11:37:31 2009 From: reggaestar at gmail.com (remi doubi) Date: Wed, 6 May 2009 11:37:31 +0000 Subject: [Linux-cluster] lock_gulm Message-ID: <3c88c73a0905060437j207a4429lc52b6007330c1163@mail.gmail.com> Hi everyone, i'm trying to work with gfs to create a File System in order to be used by the cluster but i got an error when trying to mount : #gfs_mkfs -p lock_gulm -t cluster-DomU:gfs -j 2 /dev/VolGroup00/test This will destroy any data on /dev/VolGroup00/test. It appears to contain a gfs filesystem. Are you sure you want to proceed? [y/n] y Device: /dev/VolGroup00/test Blocksize: 4096 Filesystem Size: 458692 Journals: 2 Resource Groups: 8 Locking Protocol: lock_gulm Lock Table: cluster-DomU:gfs Syncing... All Done #mount /dev/VolGroup00/test /mnt/bob/ /sbin/mount.gfs: error mounting /dev/mapper/VolGroup00-test on /mnt/bob: No such file or directory please anyone can help !! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ccaulfie at redhat.com Wed May 6 12:01:21 2009 From: ccaulfie at redhat.com (Chrissie Caulfield) Date: Wed, 06 May 2009 13:01:21 +0100 Subject: [Linux-cluster] Necessary a delay to restart cman? In-Reply-To: <4A016DA1.5090404@fi.upm.es> References: <4A016DA1.5090404@fi.upm.es> Message-ID: <4A017C11.9010006@redhat.com> Miguel Sanchez wrote: > Hi. I have a CentOS 5.3 cluster with two nodes. If I execute service > cman restart within a node, or stop + start after few seconds, another > node doesn?t recognize this membership return and its fellow stay > forever offline. > > For example: > > * Before cman restart: > > node1# cman_tool status > Version: 6.1.0 > Config Version: 6 > Cluster Name: CSVirtualizacion > Cluster Id: 42648 > Cluster Member: Yes > Cluster Generation: 202600 > Membership state: Cluster-Member > Nodes: 2 > Expected votes: 1 > Total votes: 2 > Quorum: 1 > Active subsystems: 7 > Flags: 2node Dirty > Ports Bound: 0 > Node name: patty > Node ID: 1 > Multicast addresses: 224.0.0.133 > Node addresses: 138.100.8.70 > > * After cman stop for node2 (and before a number seconds < token parameter) > > node1# cman_tool status > Version: 6.1.0 > Config Version: 6 > Cluster Name: CSVirtualizacion > Cluster Id: 42648 > Cluster Member: Yes > Cluster Generation: 202600 > Membership state: Cluster-Member > Nodes: 2 > Expected votes: 1 > Total votes: 1 > Quorum: 1 > Active subsystems: 7 > Flags: 2node Dirty > Ports Bound: 0 > Node name: patty > Node ID: 1 > Multicast addresses: 224.0.0.133 > Node addresses: 138.100.8.70 > Wed May 6 12:29:38 CEST 2009 > > * After cman stop for node2 (and after a number seconds > token parameter) > > node1# date; cman_tool status > Version: 6.1.0 > Config Version: 6 > Cluster Name: CSVirtualizacion > Cluster Id: 42648 > Cluster Member: Yes > Cluster Generation: 202604 > Membership state: Cluster-Member > Nodes: 1 > Expected votes: 1 > Total votes: 1 > Quorum: 1 > Active subsystems: 7 > Flags: 2node Dirty > Ports Bound: 0 > Node name: patty > Node ID: 1 > Multicast addresses: 224.0.0.133 > Node addresses: 138.100.8.70 > Wed May 6 12:29:47 CEST 2009 > > /var/log/messages: > May 6 12:35:20 node2 openais[17262]: [TOTEM] The token was lost in the > OPERATIONAL state. > May 6 12:35:20 node2 openais[17262]: [TOTEM] Receive multicast socket > recv buffer size (288000 bytes). > May 6 12:35:20 node2 openais[17262]: [TOTEM] Transmit multicast socket > send buffer size (262142 bytes). > May 6 12:35:20 node2 openais[17262]: [TOTEM] entering GATHER state from 2. > May 6 12:35:25 node2 openais[17262]: [TOTEM] entering GATHER state from 0. > May 6 12:35:25 node2 openais[17262]: [TOTEM] Creating commit token > because I am the rep. > May 6 12:35:25 node2 openais[17262]: [TOTEM] Saving state aru 26 high > seq received 26 > May 6 12:35:25 node2 openais[17262]: [TOTEM] Storing new sequence id > for ring 31780 > May 6 12:35:25 node2 openais[17262]: [TOTEM] entering COMMIT state. > May 6 12:35:25 node2 openais[17262]: [TOTEM] entering RECOVERY state. > May 6 12:35:25 node2 openais[17262]: [TOTEM] position [0] member > 10.10.8.70: > May 6 12:35:25 node2 openais[17262]: [TOTEM] previous ring seq 202620 > rep 10.10.8.70 > May 6 12:35:25 node2 openais[17262]: [TOTEM] aru 26 high delivered 26 > received flag 1 > May 6 12:35:25 node2 openais[17262]: [TOTEM] Did not need to originate > any messages in recovery. > May 6 12:35:25 node2 openais[17262]: [TOTEM] Sending initial ORF token > May 6 12:35:25 node2 openais[17262]: [CLM ] CLM CONFIGURATION CHANGE > May 6 12:35:25 node2 openais[17262]: [CLM ] New Configuration: > May 6 12:35:25 node2 openais[17262]: [CLM ] r(0) ip(10.10.8.70) > May 6 12:35:25 node2 openais[17262]: [CLM ] Members Left: > May 6 12:35:25 node2 openais[17262]: [CLM ] r(0) ip(10.10.8.71) > May 6 12:35:25 node2 openais[17262]: [CLM ] Members Joined: > May 6 12:35:25 node2 openais[17262]: [CLM ] CLM CONFIGURATION CHANGE > May 6 12:35:25 node2 openais[17262]: [CLM ] New Configuration: > May 6 12:35:25 node2 openais[17262]: [CLM ] r(0) ip(10.10.8.70) > May 6 12:35:25 node2 openais[17262]: [CLM ] Members Left: > May 6 12:35:25 node2 openais[17262]: [CLM ] Members Joined: > May 6 12:35:25 node2 openais[17262]: [SYNC ] This node is within the > primary component and will provide service. > May 6 12:35:25 node2 openais[17262]: [TOTEM] entering OPERATIONAL state. > May 6 12:35:25 node2 kernel: dlm: closing connection to node 2 > May 6 12:35:25 node2 openais[17262]: [CLM ] got nodejoin message > 10.10.8.70 > May 6 12:35:25 node2 openais[17262]: [CPG ] got joinlist message from > node 1 > > > if node2 doesn`t wait for run cman start to the detection the > operational token's lost, node1 detect node2 like offline forever. > Following attempts for cman restarts don`t change this state: > node1# cman_tool nodes > Node Sts Inc Joined Name > 1 M 202616 2009-05-06 12:34:43 node1 > 2 X 202628 node2 > node2# cman_tool nodes > Node Sts Inc Joined Name > 1 M 202644 2009-05-06 12:51:04 node1 > 2 M 202640 2009-05-06 12:51:04 node2 > > > Is it necessary a delay for cman stop + start to avoid this inconsistent > state or really is it a bug? I suspect it's an instance of this known bug. Check that CentOS has the appropriate patch available: https://bugzilla.redhat.com/show_bug.cgi?id=485026 Chrissie From swhiteho at redhat.com Wed May 6 12:21:10 2009 From: swhiteho at redhat.com (Steven Whitehouse) Date: Wed, 06 May 2009 13:21:10 +0100 Subject: [Linux-cluster] lock_gulm In-Reply-To: <3c88c73a0905060437j207a4429lc52b6007330c1163@mail.gmail.com> References: <3c88c73a0905060437j207a4429lc52b6007330c1163@mail.gmail.com> Message-ID: <1241612470.29604.221.camel@localhost.localdomain> Hi, On Wed, 2009-05-06 at 11:37 +0000, remi doubi wrote: > Hi everyone, i'm trying to work with gfs to create a File System in > order to be used by the cluster but i got an error when trying to > mount : > > #gfs_mkfs -p lock_gulm -t cluster-DomU:gfs -j 2 /dev/VolGroup00/test > > This will destroy any data on /dev/VolGroup00/test. > It appears to contain a gfs filesystem. > > Are you sure you want to proceed? [y/n] y > > Device: /dev/VolGroup00/test > Blocksize: 4096 > Filesystem Size: 458692 > Journals: 2 > Resource Groups: 8 > Locking Protocol: lock_gulm > Lock Table: cluster-DomU:gfs > > Syncing... > All Done > > #mount /dev/VolGroup00/test /mnt/bob/ > /sbin/mount.gfs: error mounting /dev/mapper/VolGroup00-test > on /mnt/bob: No such file or directory > > please anyone can help !! > > I've no idea what version of gfs you are using, but you almost certainly want to be using lock_dlm and not lock_gulm. I'm assuming also, or course, that /mnt/bob does actually exist, is a directory and is accessible to the mount process, Steve. > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From adam at gradientzero.com Wed May 6 12:59:05 2009 From: adam at gradientzero.com (Adam Hough) Date: Wed, 6 May 2009 07:59:05 -0500 Subject: [Linux-cluster] Necessary a delay to restart cman? In-Reply-To: <4A017C11.9010006@redhat.com> References: <4A016DA1.5090404@fi.upm.es> <4A017C11.9010006@redhat.com> Message-ID: On Wed, May 6, 2009 at 7:01 AM, Chrissie Caulfield wrote: > Miguel Sanchez wrote: >> Hi. I have a CentOS 5.3 cluster with two nodes. If I execute service >> cman restart within a node, or stop + start after few seconds, another >> node doesn?t recognize this membership return and its fellow stay >> forever offline. >> >> For example: >> >> * Before cman restart: >> >> node1# cman_tool status >> Version: 6.1.0 >> Config Version: 6 >> Cluster Name: CSVirtualizacion >> Cluster Id: 42648 >> Cluster Member: Yes >> Cluster Generation: 202600 >> Membership state: Cluster-Member >> Nodes: 2 >> Expected votes: 1 >> Total votes: 2 >> Quorum: 1 >> Active subsystems: 7 >> Flags: 2node Dirty >> Ports Bound: 0 >> Node name: patty >> Node ID: 1 >> Multicast addresses: 224.0.0.133 >> Node addresses: 138.100.8.70 >> >> * After cman stop for node2 (and before a number seconds < token parameter) >> >> node1# cman_tool status >> Version: 6.1.0 >> Config Version: 6 >> Cluster Name: CSVirtualizacion >> Cluster Id: 42648 >> Cluster Member: Yes >> Cluster Generation: 202600 >> Membership state: Cluster-Member >> Nodes: 2 >> Expected votes: 1 >> Total votes: 1 >> Quorum: 1 >> Active subsystems: 7 >> Flags: 2node Dirty >> Ports Bound: 0 >> Node name: patty >> Node ID: 1 >> Multicast addresses: 224.0.0.133 >> Node addresses: 138.100.8.70 >> Wed May ?6 12:29:38 CEST 2009 >> >> * After cman stop for node2 (and after a number seconds > token parameter) >> >> node1# date; cman_tool status >> Version: 6.1.0 >> Config Version: 6 >> Cluster Name: CSVirtualizacion >> Cluster Id: 42648 >> Cluster Member: Yes >> Cluster Generation: 202604 >> Membership state: Cluster-Member >> Nodes: 1 >> Expected votes: 1 >> Total votes: 1 >> Quorum: 1 >> Active subsystems: 7 >> Flags: 2node Dirty >> Ports Bound: 0 >> Node name: patty >> Node ID: 1 >> Multicast addresses: 224.0.0.133 >> Node addresses: 138.100.8.70 >> Wed May ?6 12:29:47 CEST 2009 >> >> /var/log/messages: >> May ?6 12:35:20 node2 openais[17262]: [TOTEM] The token was lost in the >> OPERATIONAL state. >> May ?6 12:35:20 node2 openais[17262]: [TOTEM] Receive multicast socket >> recv buffer size (288000 bytes). >> May ?6 12:35:20 node2 openais[17262]: [TOTEM] Transmit multicast socket >> send buffer size (262142 bytes). >> May ?6 12:35:20 node2 openais[17262]: [TOTEM] entering GATHER state from 2. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] entering GATHER state from 0. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] Creating commit token >> because I am the rep. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] Saving state aru 26 high >> seq received 26 >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] Storing new sequence id >> for ring 31780 >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] entering COMMIT state. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] entering RECOVERY state. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] position [0] member >> 10.10.8.70: >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] previous ring seq 202620 >> rep 10.10.8.70 >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] aru 26 high delivered 26 >> received flag 1 >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] Did not need to originate >> any messages in recovery. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] Sending initial ORF token >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] CLM CONFIGURATION CHANGE >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] New Configuration: >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] ? r(0) ip(10.10.8.70) >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] Members Left: >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] ? r(0) ip(10.10.8.71) >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] Members Joined: >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] CLM CONFIGURATION CHANGE >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] New Configuration: >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] ? r(0) ip(10.10.8.70) >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] Members Left: >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] Members Joined: >> May ?6 12:35:25 node2 openais[17262]: [SYNC ] This node is within the >> primary component and will provide service. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] entering OPERATIONAL state. >> May ?6 12:35:25 node2 kernel: dlm: closing connection to node 2 >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] got nodejoin message >> 10.10.8.70 >> May ?6 12:35:25 node2 openais[17262]: [CPG ?] got joinlist message from >> node 1 >> >> >> if node2 doesn`t wait for run cman start to the detection the >> operational token's lost, node1 detect node2 like offline forever. >> Following attempts for cman restarts don`t change this state: >> node1# cman_tool nodes >> Node ?Sts ? Inc ? Joined ? ? ? ? ? ? ? Name >> ? 1 ? M ?202616 ? 2009-05-06 12:34:43 ?node1 >> ? 2 ? X ?202628 ? ? ? ? ? ? ? ? ? ? ? ?node2 >> node2# cman_tool nodes >> Node ?Sts ? Inc ? Joined ? ? ? ? ? ? ? Name >> ? 1 ? M ?202644 ? 2009-05-06 12:51:04 ?node1 >> ? 2 ? M ?202640 ? 2009-05-06 12:51:04 ?node2 >> >> >> Is it necessary a delay for cman stop + start to avoid this inconsistent >> state or really is it a bug? > > > I suspect it's an instance of this known bug. Check that CentOS has the > appropriate patch available: > > https://bugzilla.redhat.com/show_bug.cgi?id=485026 > > Chrissie > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > When restarting cman, I have always had to stop cman and then manually stop openais before trying to start cman again. If I do not follow these steps then the node would never rejoin the cluster or might fence the other node. From ccaulfie at redhat.com Wed May 6 13:05:41 2009 From: ccaulfie at redhat.com (Chrissie Caulfield) Date: Wed, 06 May 2009 14:05:41 +0100 Subject: [Linux-cluster] Necessary a delay to restart cman? In-Reply-To: References: <4A016DA1.5090404@fi.upm.es> <4A017C11.9010006@redhat.com> Message-ID: <4A018B25.1050800@redhat.com> Adam Hough wrote: > On Wed, May 6, 2009 at 7:01 AM, Chrissie Caulfield wrote: >> Miguel Sanchez wrote: >>> Hi. I have a CentOS 5.3 cluster with two nodes. If I execute service >>> cman restart within a node, or stop + start after few seconds, another >>> node doesn?t recognize this membership return and its fellow stay >>> forever offline. >>> >>> For example: >>> >>> * Before cman restart: >>> >>> node1# cman_tool status >>> Version: 6.1.0 >>> Config Version: 6 >>> Cluster Name: CSVirtualizacion >>> Cluster Id: 42648 >>> Cluster Member: Yes >>> Cluster Generation: 202600 >>> Membership state: Cluster-Member >>> Nodes: 2 >>> Expected votes: 1 >>> Total votes: 2 >>> Quorum: 1 >>> Active subsystems: 7 >>> Flags: 2node Dirty >>> Ports Bound: 0 >>> Node name: patty >>> Node ID: 1 >>> Multicast addresses: 224.0.0.133 >>> Node addresses: 138.100.8.70 >>> >>> * After cman stop for node2 (and before a number seconds < token parameter) >>> >>> node1# cman_tool status >>> Version: 6.1.0 >>> Config Version: 6 >>> Cluster Name: CSVirtualizacion >>> Cluster Id: 42648 >>> Cluster Member: Yes >>> Cluster Generation: 202600 >>> Membership state: Cluster-Member >>> Nodes: 2 >>> Expected votes: 1 >>> Total votes: 1 >>> Quorum: 1 >>> Active subsystems: 7 >>> Flags: 2node Dirty >>> Ports Bound: 0 >>> Node name: patty >>> Node ID: 1 >>> Multicast addresses: 224.0.0.133 >>> Node addresses: 138.100.8.70 >>> Wed May 6 12:29:38 CEST 2009 >>> >>> * After cman stop for node2 (and after a number seconds > token parameter) >>> >>> node1# date; cman_tool status >>> Version: 6.1.0 >>> Config Version: 6 >>> Cluster Name: CSVirtualizacion >>> Cluster Id: 42648 >>> Cluster Member: Yes >>> Cluster Generation: 202604 >>> Membership state: Cluster-Member >>> Nodes: 1 >>> Expected votes: 1 >>> Total votes: 1 >>> Quorum: 1 >>> Active subsystems: 7 >>> Flags: 2node Dirty >>> Ports Bound: 0 >>> Node name: patty >>> Node ID: 1 >>> Multicast addresses: 224.0.0.133 >>> Node addresses: 138.100.8.70 >>> Wed May 6 12:29:47 CEST 2009 >>> >>> /var/log/messages: >>> May 6 12:35:20 node2 openais[17262]: [TOTEM] The token was lost in the >>> OPERATIONAL state. >>> May 6 12:35:20 node2 openais[17262]: [TOTEM] Receive multicast socket >>> recv buffer size (288000 bytes). >>> May 6 12:35:20 node2 openais[17262]: [TOTEM] Transmit multicast socket >>> send buffer size (262142 bytes). >>> May 6 12:35:20 node2 openais[17262]: [TOTEM] entering GATHER state from 2. >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] entering GATHER state from 0. >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] Creating commit token >>> because I am the rep. >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] Saving state aru 26 high >>> seq received 26 >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] Storing new sequence id >>> for ring 31780 >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] entering COMMIT state. >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] entering RECOVERY state. >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] position [0] member >>> 10.10.8.70: >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] previous ring seq 202620 >>> rep 10.10.8.70 >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] aru 26 high delivered 26 >>> received flag 1 >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] Did not need to originate >>> any messages in recovery. >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] Sending initial ORF token >>> May 6 12:35:25 node2 openais[17262]: [CLM ] CLM CONFIGURATION CHANGE >>> May 6 12:35:25 node2 openais[17262]: [CLM ] New Configuration: >>> May 6 12:35:25 node2 openais[17262]: [CLM ] r(0) ip(10.10.8.70) >>> May 6 12:35:25 node2 openais[17262]: [CLM ] Members Left: >>> May 6 12:35:25 node2 openais[17262]: [CLM ] r(0) ip(10.10.8.71) >>> May 6 12:35:25 node2 openais[17262]: [CLM ] Members Joined: >>> May 6 12:35:25 node2 openais[17262]: [CLM ] CLM CONFIGURATION CHANGE >>> May 6 12:35:25 node2 openais[17262]: [CLM ] New Configuration: >>> May 6 12:35:25 node2 openais[17262]: [CLM ] r(0) ip(10.10.8.70) >>> May 6 12:35:25 node2 openais[17262]: [CLM ] Members Left: >>> May 6 12:35:25 node2 openais[17262]: [CLM ] Members Joined: >>> May 6 12:35:25 node2 openais[17262]: [SYNC ] This node is within the >>> primary component and will provide service. >>> May 6 12:35:25 node2 openais[17262]: [TOTEM] entering OPERATIONAL state. >>> May 6 12:35:25 node2 kernel: dlm: closing connection to node 2 >>> May 6 12:35:25 node2 openais[17262]: [CLM ] got nodejoin message >>> 10.10.8.70 >>> May 6 12:35:25 node2 openais[17262]: [CPG ] got joinlist message from >>> node 1 >>> >>> >>> if node2 doesn`t wait for run cman start to the detection the >>> operational token's lost, node1 detect node2 like offline forever. >>> Following attempts for cman restarts don`t change this state: >>> node1# cman_tool nodes >>> Node Sts Inc Joined Name >>> 1 M 202616 2009-05-06 12:34:43 node1 >>> 2 X 202628 node2 >>> node2# cman_tool nodes >>> Node Sts Inc Joined Name >>> 1 M 202644 2009-05-06 12:51:04 node1 >>> 2 M 202640 2009-05-06 12:51:04 node2 >>> >>> >>> Is it necessary a delay for cman stop + start to avoid this inconsistent >>> state or really is it a bug? >> >> I suspect it's an instance of this known bug. Check that CentOS has the >> appropriate patch available: >> >> https://bugzilla.redhat.com/show_bug.cgi?id=485026 >> >> Chrissie >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > > When restarting cman, I have always had to stop cman and then manually > stop openais before trying to start cman again. If I do not follow > these steps then the node would never rejoin the cluster or might > fence the other node. That indicates some form of configuration error. You should never have to do that. Make sure that openais is not enabled at boot time using chkconfig openais off Also, I really don't recommend stopping and starting cman without a reboot. Yes you might get away with it a few times, but one day it won't work and you'll be emailing here again ;-) Chrissie From rpeterso at redhat.com Wed May 6 13:15:27 2009 From: rpeterso at redhat.com (Bob Peterson) Date: Wed, 6 May 2009 09:15:27 -0400 (EDT) Subject: [Linux-cluster] gsf2 In-Reply-To: <1241606465.8236.39.camel@tyr.ensea.fr> Message-ID: <973733077.62501241615727925.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> ----- "gfs2etis" wrote: | Hello, | | I am a newbie with gfs2, | i have set 4 nodes on a fiber channel SAN | synchronized through gfs2. | | When one machine crash all gfs2 crash, | especially when there is a lot of IO | on the gfs2 devices. | | I have no message on the system when it crash (???) | | version : gfs2-utils-2.03.07-2 | | thanks | -- | gfs2etis | ETIS Hi, When one node crashes, gfs2 should prevent any new locks from being taken until the node it properly fenced and the journal recovered. This can appear as a freeze or lockup, and it is done intentionally to preserve data integrity. The other nodes should not crash and if they do, they should leave console messages stating what happened. Regards, Bob Peterson Red Hat GFS From EliasM at dnb.com Wed May 6 13:33:53 2009 From: EliasM at dnb.com (Elias, Michael) Date: Wed, 6 May 2009 09:33:53 -0400 Subject: [Linux-cluster] Heartbeat time outs in rhel4 understanding In-Reply-To: <4A0144F9.5040701@redhat.com> Message-ID: <437A36C1327D794D87D207AC80BDD8FD0B05191F@DNBMSXBH002.dnbint.net> Ok, so let me ask this. I did a tcpdump between nodes. Is the heartbeat the udp pack I see? I also see an xml doc. Like node1 keeps uptime and other cluster info for itself and node2. node2 keeps uptime and cluster onfo for nodes 1 and 3. Node 3 does the same for 2 and 4 and so on. I assume is a node dies then they next closest node starts watching the uptime for that node until the failed node rejoins. Thanks again -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Chrissie Caulfield Sent: Wednesday, May 06, 2009 4:06 AM To: linux clustering Subject: Re: [Linux-cluster] Heartbeat time outs in rhel4 understanding Elias, Michael wrote: > I am trying to understand how these timers interact with each other. > > > > In a RHEL4 cluster the heartbeat defaults are; > > hello_timer:5 > > max_retries:5 > > deadnode_timeout:21 > > > > Meaning a heartbeat message is sent every 5 seconds, if it fails to > receive a response it will start a deadnode counter @ 21 seconds. It > will also try to send 5 more heartbeat requests. What is the interval of > those retries? If none of those requests receive a response. 5 seconds > pass.. there is 15 seconds left on the deadnode timer and we try upto 5 > times to get a response.... This goes on until we hit the 4^th iteration > of the hellotimer it tries again upto 5 times and fails... we then hit the > 21 second on the deadnode time.. fenced takes over and wham reboot. > > > > Is my understanding of this correct???? > No, I'm afraid it isn't :-) max_retries has nothing to do with the heartbeat. It is to do with cluster messages, such as service join requests, clvmd messages or the messages used in the membership protocol. So the heartbeat system is just a 5 second heartbeat and after 21 seconds the node will be evicted from the cluster and (usually) fenced. The same happens for data messages if max_retries is exceeded. The retry period here starts at 1 second and increases each time to avoid filling the ethernet buffers. I hope this helps, Chrissie -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From Harri.Paivaniemi at tieto.com Wed May 6 14:30:45 2009 From: Harri.Paivaniemi at tieto.com (Harri.Paivaniemi at tieto.com) Date: Wed, 6 May 2009 17:30:45 +0300 Subject: [Linux-cluster] Necessary a delay to restart cman? References: <4A016DA1.5090404@fi.upm.es> <4A017C11.9010006@redhat.com> Message-ID: <41E8D4F07FCE154CBEBAA60FFC92F67754D510@apollo.eu.tieto.com> Hi, Just fyi: I had a similar problem in the past and I made a support request to RH support. They said you have to wait totem token- time after stop before starting again, or it's not giong to work... I wonder if this is correct... -hjp -----Original Message----- From: linux-cluster-bounces at redhat.com on behalf of Adam Hough Sent: Wed 5/6/2009 15:59 To: linux clustering Subject: Re: [Linux-cluster] Necessary a delay to restart cman? On Wed, May 6, 2009 at 7:01 AM, Chrissie Caulfield wrote: > Miguel Sanchez wrote: >> Hi. I have a CentOS 5.3 cluster with two nodes. If I execute service >> cman restart within a node, or stop + start after few seconds, another >> node doesn?t recognize this membership return and its fellow stay >> forever offline. >> >> For example: >> >> * Before cman restart: >> >> node1# cman_tool status >> Version: 6.1.0 >> Config Version: 6 >> Cluster Name: CSVirtualizacion >> Cluster Id: 42648 >> Cluster Member: Yes >> Cluster Generation: 202600 >> Membership state: Cluster-Member >> Nodes: 2 >> Expected votes: 1 >> Total votes: 2 >> Quorum: 1 >> Active subsystems: 7 >> Flags: 2node Dirty >> Ports Bound: 0 >> Node name: patty >> Node ID: 1 >> Multicast addresses: 224.0.0.133 >> Node addresses: 138.100.8.70 >> >> * After cman stop for node2 (and before a number seconds < token parameter) >> >> node1# cman_tool status >> Version: 6.1.0 >> Config Version: 6 >> Cluster Name: CSVirtualizacion >> Cluster Id: 42648 >> Cluster Member: Yes >> Cluster Generation: 202600 >> Membership state: Cluster-Member >> Nodes: 2 >> Expected votes: 1 >> Total votes: 1 >> Quorum: 1 >> Active subsystems: 7 >> Flags: 2node Dirty >> Ports Bound: 0 >> Node name: patty >> Node ID: 1 >> Multicast addresses: 224.0.0.133 >> Node addresses: 138.100.8.70 >> Wed May ?6 12:29:38 CEST 2009 >> >> * After cman stop for node2 (and after a number seconds > token parameter) >> >> node1# date; cman_tool status >> Version: 6.1.0 >> Config Version: 6 >> Cluster Name: CSVirtualizacion >> Cluster Id: 42648 >> Cluster Member: Yes >> Cluster Generation: 202604 >> Membership state: Cluster-Member >> Nodes: 1 >> Expected votes: 1 >> Total votes: 1 >> Quorum: 1 >> Active subsystems: 7 >> Flags: 2node Dirty >> Ports Bound: 0 >> Node name: patty >> Node ID: 1 >> Multicast addresses: 224.0.0.133 >> Node addresses: 138.100.8.70 >> Wed May ?6 12:29:47 CEST 2009 >> >> /var/log/messages: >> May ?6 12:35:20 node2 openais[17262]: [TOTEM] The token was lost in the >> OPERATIONAL state. >> May ?6 12:35:20 node2 openais[17262]: [TOTEM] Receive multicast socket >> recv buffer size (288000 bytes). >> May ?6 12:35:20 node2 openais[17262]: [TOTEM] Transmit multicast socket >> send buffer size (262142 bytes). >> May ?6 12:35:20 node2 openais[17262]: [TOTEM] entering GATHER state from 2. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] entering GATHER state from 0. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] Creating commit token >> because I am the rep. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] Saving state aru 26 high >> seq received 26 >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] Storing new sequence id >> for ring 31780 >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] entering COMMIT state. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] entering RECOVERY state. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] position [0] member >> 10.10.8.70: >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] previous ring seq 202620 >> rep 10.10.8.70 >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] aru 26 high delivered 26 >> received flag 1 >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] Did not need to originate >> any messages in recovery. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] Sending initial ORF token >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] CLM CONFIGURATION CHANGE >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] New Configuration: >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] ? r(0) ip(10.10.8.70) >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] Members Left: >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] ? r(0) ip(10.10.8.71) >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] Members Joined: >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] CLM CONFIGURATION CHANGE >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] New Configuration: >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] ? r(0) ip(10.10.8.70) >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] Members Left: >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] Members Joined: >> May ?6 12:35:25 node2 openais[17262]: [SYNC ] This node is within the >> primary component and will provide service. >> May ?6 12:35:25 node2 openais[17262]: [TOTEM] entering OPERATIONAL state. >> May ?6 12:35:25 node2 kernel: dlm: closing connection to node 2 >> May ?6 12:35:25 node2 openais[17262]: [CLM ?] got nodejoin message >> 10.10.8.70 >> May ?6 12:35:25 node2 openais[17262]: [CPG ?] got joinlist message from >> node 1 >> >> >> if node2 doesn`t wait for run cman start to the detection the >> operational token's lost, node1 detect node2 like offline forever. >> Following attempts for cman restarts don`t change this state: >> node1# cman_tool nodes >> Node ?Sts ? Inc ? Joined ? ? ? ? ? ? ? Name >> ? 1 ? M ?202616 ? 2009-05-06 12:34:43 ?node1 >> ? 2 ? X ?202628 ? ? ? ? ? ? ? ? ? ? ? ?node2 >> node2# cman_tool nodes >> Node ?Sts ? Inc ? Joined ? ? ? ? ? ? ? Name >> ? 1 ? M ?202644 ? 2009-05-06 12:51:04 ?node1 >> ? 2 ? M ?202640 ? 2009-05-06 12:51:04 ?node2 >> >> >> Is it necessary a delay for cman stop + start to avoid this inconsistent >> state or really is it a bug? > > > I suspect it's an instance of this known bug. Check that CentOS has the > appropriate patch available: > > https://bugzilla.redhat.com/show_bug.cgi?id=485026 > > Chrissie > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > When restarting cman, I have always had to stop cman and then manually stop openais before trying to start cman again. If I do not follow these steps then the node would never rejoin the cluster or might fence the other node. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 5251 bytes Desc: not available URL: From arwin.tugade at csun.edu Wed May 6 21:18:33 2009 From: arwin.tugade at csun.edu (Arwin L Tugade) Date: Wed, 6 May 2009 14:18:33 -0700 Subject: [Linux-cluster] service failover In-Reply-To: <41E8D4F07FCE154CBEBAA60FFC92F67754D50E@apollo.eu.tieto.com> References: <41E8D4F07FCE154CBEBAA60FFC92F67754D50E@apollo.eu.tieto.com> Message-ID: <6708F96BBF31F846BFA56EC0AE37D62281E94FDE68@CSUN-EX-V01.csun.edu> Yup, or the way I do it, with Swatch (http://sourceforge.net/projects/swatch/). -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Harri.Paivaniemi at tieto.com Sent: Tuesday, May 05, 2009 8:56 PM To: linux-cluster at redhat.com Subject: RE: [Linux-cluster] service failover If you need a quick-n-dirty way, you can always put something in to services starting script so every time cluster says start|stop to that service, you can send mail to yourself etc and you don't have to grep messages-log ;) -hjp -----Original Message----- From: linux-cluster-bounces at redhat.com on behalf of Gary Romo Sent: Tue 5/5/2009 22:31 To: linux-cluster at redhat.com Subject: [Linux-cluster] service failover Hello. How can I tell when a service has failed over? I'm looking for date and time stamps. Thanks. -Gary From billpp at gmail.com Wed May 6 22:44:15 2009 From: billpp at gmail.com (Flavio Junior) Date: Wed, 6 May 2009 19:44:15 -0300 Subject: [Linux-cluster] RHCS 4-node cluster: Networking/Membership issues In-Reply-To: <58aa8d780904300941l6146d881lfe102d8ebd6c8e32@mail.gmail.com> References: <58aa8d780904291321i21a8914fl956db51d61664a51@mail.gmail.com> <1DA7B2AF-E920-4701-A49B-9806478144F7@auckland.ac.nz> <58aa8d780904300941l6146d881lfe102d8ebd6c8e32@mail.gmail.com> Message-ID: <58aa8d780905061544x4bc38908n6613921ac11a756a@mail.gmail.com> Hi again folks... One update here: - I'd removed bonding for cluster heartbeat (bond0) and setup it direct on eth0 for all nodes. This solves the issue for membership. Now I can boot up all 4 nodes, join fence domain, start clvmd on them. Everything is stable and I didn't see random messages about "openais retransmit" anymore. Of course, I still have a problem :). I've 1 GFS filesystem and 16 GFS2 filesystems. I can mount all filesystems on node1 and node2 (same build/switch), but when I try to run "service gfs2 start" on node3 or node4 (another build/switch) the things becomes unstable and whole cluster fail with infinity messages about "cpg_mcast_retry RETRY_NUMBER". Log can be found here: http://pastebin.com/m2f26ab1d What apparently happened is that without bonding setup the network layer becomes more "simple" and could handle with membership but still cant handle with GFS/GFS2 heartbeat. I've set nodes to talk IGMPv2, as said at: http://archives.free.net.ph/message/20081001.223026.9cf6d7bf.de.html Well.. any hints? Thanks again. -- Fl?vio do Carmo J?nior aka waKKu On Thu, Apr 30, 2009 at 1:41 PM, Flavio Junior wrote: > Hi Abraham, thanks for your answer. > > I'd configured your suggestion to cluster.conf but still gets the same > problem. > > Here is what I did: > * Disable cman init script on boot for all nodes > * Edit config file and copy it for all nodes > * reboot all > * start cman on node1 (OK) > * start cman on node2 (OK) > * start cman on node3 (problems to become member, fence node2) > > Here is the log file with this process 'til the fence: > http://pastebin.com/f477e7114 > > PS: node1 and node2 as on the same switch at site1. node3 and node4 as > on the same switch at site2. > > Thanks again, any other suggestions ? > > I dont know if it would help but, is corosync a feasible option for > production use? > > -- > > Fl?vio do Carmo J?nior aka waKKu > > On Wed, Apr 29, 2009 at 10:19 PM, Abraham Alawi > wrote: > > If not tried already, the following settings in cluster.conf might help > > especially "clean_start" > > > > > > clean_start --> assume the cluster is in healthy state upon startup > > post_fail_delay --> seconds to wait before fencing a node that thinks it > > should be fenced (i.e. lost connection with) > > post_join_delay --> seconds to wait before fencing any node that should > be > > fenced upon startup (right after joining) > > > > On 30/04/2009, at 8:21 AM, Flavio Junior wrote: > > > >> Hi folks, > >> > >> I've been trying to set up a 4-node RHCS+GFS cluster for awhile. I've > >> another 2-node cluster using CentOS 5.3 without problem. > >> > >> Well.. My scenario is as follow: > >> > >> * System configuration and info: http://pastebin.com/f41d63624 > >> > >> * Network: > >> > http://www.uploadimagens.com/upload/2ac9074fbb10c2479c59abe419880dc8.jpg > >> * Switches on loop are 3Com 2924 (or 2948)-SFP > >> * Have STP enabled (RSTP auto) > >> * IGMP Snooping Disabled as: > >> > >> > http://magazine.redhat.com/2007/08/23/automated-failover-and-recovery-of-virtualized-guests-in-advanced-platform/ > >> comment 32 > >> * Yellow lines are a fiber link 990ft (330mts) single-mode > >> * I'm using a dedicated tagged VLAN for cluster-heartbeat > >> * I'm using 2 NIC's with bonding mode=1 (active/backup) for > >> heartbeat and 4 NIC's to "public" > >> * Every node has your public four cables plugged on same switch and > >> Link-Aggregation on it > >> * Looking to the picture, that 2 switches with below fiber link is > >> where the nodes are plugged. 2 nodes each build. > >> > >> SAN: http://img139.imageshack.us/img139/642/clusters.jpg > >> * Switches: Brocade TotalStorage 16SAN-B > >> * Storages: IBM DS4700 72A (using ERM for sync replication (storage > >> level)) > >> > >> My problem is: > >> > >> I can't get the 4 nodes up. Every time the fourth (sometimes even the > >> third) node becomes online i got one or two of them fenced. I keep > >> getting messages about openais/cman, cpg_mcast_joined very often: > >> --- snipped --- > >> Apr 29 16:08:23 athos groupd[5393]: cpg_mcast_joined retry 1098900 > >> Apr 29 16:08:23 athos groupd[5393]: cpg_mcast_joined retry 1099000 > >> --- snipped --- > >> > >> Is really seldom the times I can get a node to boot up and join on > >> fence domain, almost every time it hangs and i need to reboot and try > >> again or either reboot, enter single mode, disable cman, reboot, keep > >> trying to service cman start/stop. Sometimes another nodes can see the > >> node in domain but boot keeps hangs on "Starting fenced..." > >> > >> ######## > >> [root at athos ~]# cman_tool services > >> type level name id state > >> fence 0 default 00010001 none > >> [1 3 4] > >> dlm 1 clvmd 00020001 none > >> [1 3 4] > >> [root at athos ~]# cman_tool nodes -f > >> Node Sts Inc Joined Name > >> 0 M 0 2009-04-29 15:16:47 > >> /dev/disk/by-id/scsi-3600a0b800048834e000014fb49dcc47b > >> 1 M 7556 2009-04-29 15:16:35 athos-priv > >> Last fenced: 2009-04-29 15:13:49 by athos-ipmi > >> 2 X 7820 porthos-priv > >> Last fenced: 2009-04-29 15:31:01 by porthos-ipmi > >> Node has not been fenced since it went down > >> 3 M 7696 2009-04-29 15:27:15 aramis-priv > >> Last fenced: 2009-04-29 15:24:17 by aramis-ipmi > >> 4 M 8232 2009-04-29 16:12:34 dartagnan-priv > >> Last fenced: 2009-04-29 16:09:53 by dartagnan-ipmi > >> [root at athos ~]# ssh root at aramis-priv > >> ssh: connect to host aramis-priv port 22: Connection refused > >> [root at athos ~]# ssh root at dartagnan-priv > >> ssh: connect to host dartagnan-priv port 22: Connection refused > >> [root at athos ~]# > >> ######### > >> > >> (I know how unreliable is ssh, but I'm seeing the console screen > >> hanged.. Just trying to show it) > >> > >> > >> The BIG log file: http://pastebin.com/f453c220 > >> Every entry on this log after 16:54h is when node2 (porthos-priv > >> 172.16.1.2) was booting and hanged on "Starting fenced..." > >> > >> > >> I've no more ideias to try solve this problem, any hints is > >> appreciated. If you need any other info, just tell me how to get it > >> and I'll post just after I read. > >> > >> > >> Very thanks, in advance. > >> > >> -- > >> > >> Fl?vio do Carmo J?nior aka waKKu > >> > >> -- > >> Linux-cluster mailing list > >> Linux-cluster at redhat.com > >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > > '''''''''''''''''''''''''''''''''''''''''''''''''''''' > > Abraham Alawi > > > > Unix/Linux Systems Administrator > > Science IT > > University of Auckland > > e: a.alawi at auckland.ac.nz > > p: +64-9-373 7599, ext#: 87572 > > > > '''''''''''''''''''''''''''''''''''''''''''''''''''''' > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From CISPLengineer.hz at ril.com Thu May 7 06:13:25 2009 From: CISPLengineer.hz at ril.com (Viral .D. Ahire) Date: Thu, 07 May 2009 11:43:25 +0530 Subject: [Linux-cluster] Reg: node getting fenced during stop, restart or relocate cluster service in RHEL-5 Message-ID: <4A027C05.4010605@ril.com> Hi, I have configured two node cluster on redhat-5. now the problem is when i relocate,restart or stop, running cluster service between nodes (2 nos) ,the node get fenced and restart server . Other side, the server who obtain cluster service leave the cluster and it's cluster service (cman) stop automatically .so it is also fenced by other server. I observed that , this problem occurred while stopping cluster service (oracle). Please help me to resolve this problem. log messages and cluster.conf file are as given as below. ------------------------- /etc/cluster/cluster.conf -------------------------