From rpeterso at redhat.com Sun Apr 1 05:42:08 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Sun, 01 Apr 2007 00:42:08 -0500 Subject: [Linux-cluster] Different Distros: GFS In-Reply-To: <2007331174123.872339@leena> References: <2007331174123.872339@leena> Message-ID: <460F4630.4000909@redhat.com> isplist at logicore.net wrote: > I thought I read long ago when starting with GFS that in order to make a > cluster function, you needed to have the same versions across the nodes. > > However, does that also mean that I could use different distributions? > > I have cases where I need to use different Linux versions, RHEL and CentOS but > need them all to be in the cluster. Is that an issue? > > Mike Hi Mike, In theory, mixing distros shouldn't matter. It's also okay to mix architectures (some 32-bit, some 64-bit, etc)...of course, if you have some 32-bit nodes, your GFS would then be limited to 16TB. Regards, Bob Peterson Red Hat Cluster Suite From npf at eurotux.com Sun Apr 1 14:29:30 2007 From: npf at eurotux.com (Nuno Fernandes) Date: Sun, 1 Apr 2007 15:29:30 +0100 Subject: AW: [Linux-cluster] system-config-cluster doesn't see cluster In-Reply-To: References: <200703311250.24951.npf@eurotux.com> Message-ID: <200704011529.30729.npf@eurotux.com> Hi, thanks. It worked... Nuno Fernandes On Saturday 31 March 2007 17:38:22 Hell, Robert wrote: > Hi, > > s-c-cluster uses cman_tool to determine if the on which it is started is a > cluster member and to read out node and service state. The problem is that > s-c-cluster uses /sbin/cman_tool but cman_tool is installed in > /usr/sbin/cman_tool. > > Just try: ln -s /usr/sbin/cman_tool /sbin/cman_tool on every node and > everything will work fine. > > See also bugzilla #233621 > (https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=233621 > ) Also be > aware of this bugzilla when using the management tab of > system-config-cluster: #233633 > (https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=233633) > > Kind Regards > Robert > > Ing. Robert Hell > Fabasoft R&D Software GmbH & Co KG > Honauerstrasse 4 > 4020 Linz > Austria - Europe > > ________________________________ > > Von: linux-cluster-bounces at redhat.com im Auftrag von Nuno Fernandes > Gesendet: Sa 31.03.2007 13:50 > An: linux-cluster > Betreff: [Linux-cluster] system-config-cluster doesn't see cluster > > > > Hi, > > I've created a cluster using system-config-cluster > I'm using rhel5 with kernel 2.6.18-8.el5 > > # uname -a > Linux xen2.dc.server.pt 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:14 EST 2007 > x86_64 x86_64 x86_64 GNU/Linux > > Apearently everything seems ok: > > # clustat > Member Status: Quorate > > Member Name ID Status > ------ ---- ---- ------ > xen1.dc.server.pt 1 Online > xen2.dc.server.pt 2 Online, Local > xen3.dc.server.pt 3 Online > > But when i start system-config-cluster in one of the members it reports: > > "Because this node is not currently part of a cluster, the management tab > for this application is not available." > > The server where i'm executing system-config-cluster is xen1.dc.server.pt. > Why does it reports that its not a member of the cluster? > > [root at xen1 ~]# cman_tool status > Version: 6.0.1 > Config Version: 3 > Cluster Name: clt_server > Cluster Id: 52751 > Cluster Member: Yes > Cluster Generation: 12 > Membership state: Cluster-Member > Nodes: 3 > Expected votes: 3 > Total votes: 3 > Quorum: 2 > Active subsystems: 7 > Flags: > Ports Bound: 0 11 > Node name: xen1.dc.server.pt > Node ID: 1 > Multicast addresses: 239.192.206.221 > Node addresses: 172.16.40.107 > > I've made several clusters back in rhel4 whithout any problems. I've > reinstalled the servers twice to make sure that it wasn't any problem, so, > i'm able to reproduce the problem everytime. > > Any info? > > Thanks > Nuno Fernandes > -- > Nuno Pais Fernandes > Cisco Certified Network Associate > Oracle Certified Professional > Eurotux Informatica S.A. > Tel: +351 253257395 > Fax: +351 253257396 > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > er> -- Nuno Pais Fernandes Cisco Certified Network Associate Oracle Certified Professional Eurotux Informatica S.A. Tel: +351 253257395 Fax: +351 253257396 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From isplist at logicore.net Sun Apr 1 17:57:48 2007 From: isplist at logicore.net (isplist at logicore.net) Date: Sun, 1 Apr 2007 12:57:48 -0500 Subject: [Linux-cluster] When to use more than one cluster? Message-ID: <200741125748.597244@leena> Currently, all nodes, no matter what they are doing are all part of one single cluster. So, even though they aren't sharing services or storage, they are all part of the same cluster. I could have one cluster for web servers and their storage, one for mail servers, one cluster for sql servers, etc. Am I complicating things for myself by having one single cluster. Which would be best in my case, multiple clusters or one single cluster? I do see this as an easier method by which to grow in the future and right now, things get complex when it comes to turning certain servers on and off, changing the count of servers thereby sometimes causing the cluster not to boot until I change the config file. Any thoughts or input would be very appreciated and I'm sure be of help to some others as well. Thanks. Mike From Michael.Hagmann at hilti.com Mon Apr 2 05:58:51 2007 From: Michael.Hagmann at hilti.com (Hagmann, Michael) Date: Mon, 2 Apr 2007 07:58:51 +0200 Subject: [Linux-cluster] When to use more than one cluster? In-Reply-To: <200741125748.597244@leena> References: <200741125748.597244@leena> Message-ID: <9C203D6FD2BF9D49BFF3450201DEDA53014AC6DB@LI-OWL.hag.hilti.com> Hi Mike at the beginning of our planning we also think so , but when you have a Cluster problem ( like now, the rgmanager in RHEL4 u4 ) then the hole cluster have problem and maybe crash or freeze. I would recommend you, NOT to make only one Cluster. Mike -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of isplist at logicore.net Sent: Sonntag, 1. April 2007 19:58 To: linux-cluster Subject: [Linux-cluster] When to use more than one cluster? Currently, all nodes, no matter what they are doing are all part of one single cluster. So, even though they aren't sharing services or storage, they are all part of the same cluster. I could have one cluster for web servers and their storage, one for mail servers, one cluster for sql servers, etc. Am I complicating things for myself by having one single cluster. Which would be best in my case, multiple clusters or one single cluster? I do see this as an easier method by which to grow in the future and right now, things get complex when it comes to turning certain servers on and off, changing the count of servers thereby sometimes causing the cluster not to boot until I change the config file. Any thoughts or input would be very appreciated and I'm sure be of help to some others as well. Thanks. Mike From J.vandenHorn at xb.nl Mon Apr 2 07:35:16 2007 From: J.vandenHorn at xb.nl (Jeroen van den Horn) Date: Mon, 02 Apr 2007 09:35:16 +0200 Subject: [Linux-cluster] Fenced node never reboots properly In-Reply-To: References: <460B72F0.9020200@xb.nl> <20070329183929.GD15242@redhat.com><460CA3EE.6070102@xb.nl> <460D1994.9060601@xb.nl> Message-ID: <4610B234.1040505@xb.nl> It's ESX - it runs on HP blades (AMD). Guest OSes are all 32-bit. > > What type of vmware environment? (VI ESX 3, Server, Workstation, or > one of the older platforms?) > > > > The Vmware forums have a fair amount of help on how to handle clock > drift. Are you on AMD or Intel, 32 or 64 bit? > > > > ------------------------------------------------------------------------ > > *From:* linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] *On Behalf Of *Jeroen van > den Horn > *Sent:* Friday, March 30, 2007 10:07 AM > *To:* linux clustering > *Subject:* Re: [Linux-cluster] Fenced node never reboots properly > > > > In response to Lon's suggestion I modified the fence_vmware code and > set the type of reset to HARD - cluster node now resets properly. > Remaining issue is that under VMWare we are still experiencing > performance issues. It's as if a node in the cluster starts 'lagging > behind' (also the system clock starts drifting) and that after some > time one of the nodes declares the other dead. > > Does anybody have any pointers towards performance issues and/or clock > drifting with GFS on virtual machines? > > Regards, > Jeroen > > > I'm using fence_vmware which I downloaded from some CVS repository. > Good to hear that that is the issue - I'll take a look at the source > and see whether the VMWare API support some sort of 'hard reset'. > > Jeroen > > Lon Hohberger wrote: > > On Thu, Mar 29, 2007 at 10:04:00AM +0200, Jeroen van den Horn wrote: > >> However during shutdown node 2 executes /etc/rc6.d/S31umountnfs (it's a >> Debian system) which also attempts to unmount the GFS disk - result: >> kernel OOPS. The system continues shutdown until it says 'Will now >> restart.' but that's the end of it. I've tried setting the >> /proc/sys/kernel/panic and added 'panic=5' to the kernel boot options >> but to no avail. >> >> I'm really at a loss here - does anybody have any suggestions on how to >> solve this problem? >> > > Yes, it's supposed to be killed (immediately) when fenced, not > gracefully attempting to shut down. What fencing agent are you using? > It sounds like there's a bug. > > -- Lon > > > > > > > > > ------------------------------------------------------------------------ > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > ------------------------------------------------------------------------ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From orkcu at yahoo.com Mon Apr 2 12:14:50 2007 From: orkcu at yahoo.com (=?iso-8859-1?Q?Roger_Pe=F1a?=) Date: Mon, 2 Apr 2007 05:14:50 -0700 (PDT) Subject: [Linux-cluster] Fenced node never reboots properly In-Reply-To: <4610B234.1040505@xb.nl> Message-ID: <735036.24283.qm@web50606.mail.re2.yahoo.com> --- Jeroen van den Horn wrote: > It's ESX - it runs on HP blades (AMD). Guest OSes > are all 32-bit. > > The Vmware forums have a fair amount of help on > how to handle clock > > drift. Are you on AMD or Intel, 32 or 64 bit? just install the 'vmware-tools' in the clients and them in the host select to sincronize the clock with that particular VM cu roger __________________________________________ RedHat Certified Engineer ( RHCE ) Cisco Certified Network Associate ( CCNA ) ____________________________________________________________________________________ Looking for earth-friendly autos? Browse Top Cars by "Green Rating" at Yahoo! Autos' Green Center. http://autos.yahoo.com/green_center/ From frederik.ferner at diamond.ac.uk Mon Apr 2 13:11:38 2007 From: frederik.ferner at diamond.ac.uk (Frederik Ferner) Date: Mon, 02 Apr 2007 14:11:38 +0100 Subject: [Linux-cluster] qdiskd, multiple master, updates? In-Reply-To: <20070330163828.GB6347@redhat.com> References: <460A8FAB.5050805@diamond.ac.uk> <20070330163828.GB6347@redhat.com> Message-ID: <4611010A.80606@diamond.ac.uk> Lon Hohberger wrote: > On Wed, Mar 28, 2007 at 04:54:19PM +0100, Frederik Ferner wrote: >> Could somebody give me an estimate when we can expect that to be available? > > I think it should be beta soon, and when it does go beta, it should > appear on RHN. > >> Lon: Do the test rpms at http://people.redhat.com/lhh/packages.html >> include these fixes? Which rpms do I need to test it? This time I might >> actually get a chance to test. > > The most recent fixes are not included. Would you like me to build a > new one? That would be good, yes. Many thanks, Frederik -- Frederik Ferner Linux Systems Administrator phone: +44 1235 77 8624 Diamond Light Source Ltd. mob: +44 7917 08 5110 From hlawatschek at atix.de Mon Apr 2 14:35:27 2007 From: hlawatschek at atix.de (Mark Hlawatschek) Date: Mon, 2 Apr 2007 16:35:27 +0200 Subject: [Linux-cluster] Cluster maintenance mode ? In-Reply-To: <20070327160104.GB28370@redhat.com> References: <200703271748.25566.hlawatschek@atix.de> <20070327160104.GB28370@redhat.com> Message-ID: <200704021635.28152.hlawatschek@atix.de> On Tuesday 27 March 2007 18:01:04 you wrote: > On Tue, Mar 27, 2007 at 05:48:25PM +0200, Mark Hlawatschek wrote: > > Is there a way to put the cman into a kind of maintenance mode, where > > only one node is up and no other node is allowed to join ? If not, are > > there plans to implement something like that ? > > > > The question is adressing the following issue: I'd like to enable > > automatic GFS filesystem checks. (localfs like every Nth mount or after > > M days... ) This FS check would only be allowed if no other node has the > > filesystem mounted or no other node is in the cluster. How can this be > > assured in a general way ? > Is there a possibility to do an online passive gfs_fsck, without making changes to the filesystem ? Thanks, Mark -- Gruss / Regards, Dipl.-Ing. Mark Hlawatschek http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From isplist at logicore.net Mon Apr 2 15:53:35 2007 From: isplist at logicore.net (isplist at logicore.net) Date: Mon, 2 Apr 2007 10:53:35 -0500 Subject: [Linux-cluster] When to use more than one cluster? In-Reply-To: <9C203D6FD2BF9D49BFF3450201DEDA53014AC6DB@LI-OWL.hag.hilti.com> Message-ID: <200742105335.752418@leena> The only other point I should make is that all machines in dependant on each other since each component is required in order to make up the site. So, if MySQL servers/cluster goes down, nothing else works anyhow. Good point of your part of course. Mike > at the beginning of our planning we also think so , but when you have a > Cluster problem ( like now, the rgmanager in RHEL4 u4 ) then the hole > cluster have problem and maybe crash or freeze. I would recommend you, > NOT to make only one Cluster. > Mike > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of > isplist at logicore.net > Sent: Sonntag, 1. April 2007 19:58 > To: linux-cluster > Subject: [Linux-cluster] When to use more than one cluster? > > Currently, all nodes, no matter what they are doing are all part of one > single cluster. So, even though they aren't sharing services or storage, > they are all part of the same cluster. > > I could have one cluster for web servers and their storage, one for mail > servers, one cluster for sql servers, etc. > > Am I complicating things for myself by having one single cluster. Which > would be best in my case, multiple clusters or one single cluster? > > I do see this as an easier method by which to grow in the future and > right now, things get complex when it comes to turning certain servers > on and off, changing the count of servers thereby sometimes causing the > cluster not to boot until I change the config file. > > Any thoughts or input would be very appreciated and I'm sure be of help > to some others as well. > > Thanks. > > Mike From lhh at redhat.com Mon Apr 2 15:23:58 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 2 Apr 2007 11:23:58 -0400 Subject: [Linux-cluster] Fenced node never reboots properly In-Reply-To: <4610B234.1040505@xb.nl> References: <460B72F0.9020200@xb.nl> <460D1994.9060601@xb.nl> <4610B234.1040505@xb.nl> Message-ID: <20070402152358.GE17135@redhat.com> On Mon, Apr 02, 2007 at 09:35:16AM +0200, Jeroen van den Horn wrote: > It's ESX - it runs on HP blades (AMD). Guest OSes are all 32-bit. > > > >In response to Lon's suggestion I modified the fence_vmware code and > >set the type of reset to HARD - cluster node now resets properly. > >Remaining issue is that under VMWare we are still experiencing > >performance issues. It's as if a node in the cluster starts 'lagging > >behind' (also the system clock starts drifting) and that after some > >time one of the nodes declares the other dead. Do you have a patch or the new fence agent? I'd like the change so I can commit it to CVS. Soft-reboot is *definitely* wrong; other problems aside. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Mon Apr 2 15:22:03 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 2 Apr 2007 11:22:03 -0400 Subject: [Linux-cluster] Different Distros: GFS In-Reply-To: <2007331174123.872339@leena> References: <2007331174123.872339@leena> Message-ID: <20070402152203.GD17135@redhat.com> On Sat, Mar 31, 2007 at 05:41:23PM -0600, isplist at logicore.net wrote: > I thought I read long ago when starting with GFS that in order to make a > cluster function, you needed to have the same versions across the nodes. > > However, does that also mean that I could use different distributions? > > I have cases where I need to use different Linux versions, RHEL and CentOS but > need them all to be in the cluster. Is that an issue? Mixing RHEL and CentOS should work. -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Mon Apr 2 15:25:57 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 2 Apr 2007 11:25:57 -0400 Subject: [Linux-cluster] qdiskd, multiple master, updates? In-Reply-To: <4611010A.80606@diamond.ac.uk> References: <460A8FAB.5050805@diamond.ac.uk> <20070330163828.GB6347@redhat.com> <4611010A.80606@diamond.ac.uk> Message-ID: <20070402152557.GF17135@redhat.com> On Mon, Apr 02, 2007 at 02:11:38PM +0100, Frederik Ferner wrote: > >The most recent fixes are not included. Would you like me to build a > >new one? > > That would be good, yes. Ok, I'll do that today. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From redhat at watson-wilson.ca Mon Apr 2 15:51:38 2007 From: redhat at watson-wilson.ca (Neil Watson) Date: Mon, 2 Apr 2007 11:51:38 -0400 Subject: [Linux-cluster] Disaster recovery node Message-ID: <20070402155138.GB3140@watson-wilson.ca> We have a two node DB2 cluster, using the Red Hat Cluster Suite, in active-standby mode. We would like to add a third node located at a disaster recovery site. This node must NOT be made active automatically. Only a manual operation should activate this node. Is it possible to do this with the RHCS? -- Neil Watson | Debian Linux System Administrator | Uptime 34 days http://watson-wilson.ca From lhh at redhat.com Mon Apr 2 16:35:31 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 2 Apr 2007 12:35:31 -0400 Subject: [Linux-cluster] Disaster recovery node In-Reply-To: <20070402155138.GB3140@watson-wilson.ca> References: <20070402155138.GB3140@watson-wilson.ca> Message-ID: <20070402163531.GI17135@redhat.com> On Mon, Apr 02, 2007 at 11:51:38AM -0400, Neil Watson wrote: > We have a two node DB2 cluster, using the Red Hat Cluster Suite, in > active-standby mode. We would like to add a third node located at a > disaster recovery site. This node must NOT be made active > automatically. Only a manual operation should activate this node. Is it > possible to do this with the RHCS? How will you be replicating storage between the two sites? You can do this one of a couple of ways: * Don't run the disaster recovery node, but have it ready to go. Instead, set it up exactly like the a '1-node' cluster with the same services as the other cluster. When the other site fails, ensure the other nodes are dead and turn on the machine. * Run the cluster software on the disaster recovery node (and run the node) You'll need to manually increase the number of votes it gets when the primary site goes down. After that, you will need to fence the other two nodes (or verify they're dead). -- Lon Hohberger - Software Engineer - Red Hat, Inc. From redhat at watson-wilson.ca Mon Apr 2 16:43:17 2007 From: redhat at watson-wilson.ca (Neil Watson) Date: Mon, 2 Apr 2007 12:43:17 -0400 Subject: [Linux-cluster] Disaster recovery node In-Reply-To: <20070402163531.GI17135@redhat.com> References: <20070402155138.GB3140@watson-wilson.ca> <20070402163531.GI17135@redhat.com> Message-ID: <20070402164317.GC3140@watson-wilson.ca> On Mon, Apr 02, 2007 at 12:35:31PM -0400, Lon Hohberger wrote: >How will you be replicating storage between the two sites? SAN replication. >You can do this one of a couple of ways: > >* Don't run the disaster recovery node, but have it ready to go. I'd like to keep the node up so that it can be kept up to date. I suppose I could keep the cluster service off and turn it on manually when required. -- Neil Watson | Debian Linux System Administrator | Uptime 34 days http://watson-wilson.ca From npf at eurotux.com Mon Apr 2 18:05:48 2007 From: npf at eurotux.com (Nuno Fernandes) Date: Mon, 2 Apr 2007 19:05:48 +0100 Subject: [Linux-cluster] fenced not working Message-ID: <200704021905.49058.npf@eurotux.com> Hi, I've created an rhel5 cluster using 2 servers connected to an CX3-20. I've successfully used multipath. I'm trying to install cluster sw. I've created the cluster with this configuration (generated by system-config-cluster): Everything appears ok but in the messages i keep on geeting: Mar 30 18:19:09 xen1 fenced[3017]: fencing node "xen2.server.pt" Mar 30 18:19:09 xen1 fenced[3017]: fence "xen2.server.pt" failed Apearently xen1 can't fence xen2. Using tcpdump -i any -n port 443 i don't see any packets going to ilo of xen2. But if i try echo "action = reboot" | fence_ilo -a xen2_ilo -l user -p password it's reboots the other node. Maybe something in fenced? Any info? Thanks Nuno Fernandes -- Nuno Pais Fernandes Cisco Certified Network Associate Oracle Certified Professional Eurotux Informatica S.A. Tel: +351 253257395 Fax: +351 253257396 From lhh at redhat.com Mon Apr 2 18:12:17 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 2 Apr 2007 14:12:17 -0400 Subject: [Linux-cluster] qdiskd, multiple master, updates? In-Reply-To: <20070402152557.GF17135@redhat.com> References: <460A8FAB.5050805@diamond.ac.uk> <20070330163828.GB6347@redhat.com> <4611010A.80606@diamond.ac.uk> <20070402152557.GF17135@redhat.com> Message-ID: <20070402181217.GJ17135@redhat.com> On Mon, Apr 02, 2007 at 11:25:57AM -0400, Lon Hohberger wrote: > On Mon, Apr 02, 2007 at 02:11:38PM +0100, Frederik Ferner wrote: > > >The most recent fixes are not included. Would you like me to build a > > >new one? > > > > That would be good, yes. > > Ok, I'll do that today. > New test packages (1.0.11-0.4.1qdisk): http://people.redhat.com/lhh/packages.html -- Lon Hohberger - Software Engineer - Red Hat, Inc. From jparsons at redhat.com Mon Apr 2 19:24:11 2007 From: jparsons at redhat.com (James Parsons) Date: Mon, 02 Apr 2007 15:24:11 -0400 Subject: [Linux-cluster] fenced not working In-Reply-To: <200704021905.49058.npf@eurotux.com> References: <200704021905.49058.npf@eurotux.com> Message-ID: <4611585B.8020100@redhat.com> Nuno Fernandes wrote: >Hi, > >I've created an rhel5 cluster using 2 servers connected to an CX3-20. I've >successfully used multipath. > >I'm trying to install cluster sw. I've created the cluster with this >configuration (generated by system-config-cluster): > You do not have the fence instanced beneath the nodes telling fenced what to use for which node. Do this: > > > > > > > > > > > > > > > login="user" name="xen1" passwd="password"/> > login="user" name="xen2" passwd="password"/> > > > > > > > > > In s-c-cluster, you would do this by clicking on a node and then selecting "Manage fencing for this Node" and then adding a new fence level and then adding a fence to the new level...the dropdown will list all configured fencedevices...just choose the correct one for that node. -J From g.marshall at dalmany.co.uk Mon Apr 2 18:52:58 2007 From: g.marshall at dalmany.co.uk (G.Marshall) Date: Mon, 2 Apr 2007 19:52:58 +0100 (BST) Subject: [Linux-cluster] luci misc.lock_file problem Message-ID: <42628.82.70.154.145.1175539978.squirrel@squirrelmail.tgfslp.dalmany.co.uk> I am having a problem with luci, I get the following error 2007.04.02 19:45:07 LOG7[20352:3082353584]: Remote FD=9 initialized Traceback (most recent call last): File "/usr/lib/luci/zope/lib/python/Zope2/Startup/run.py", line 56, in ? File "/usr/lib/luci/zope/lib/python/Zope2/Startup/run.py", line 21, in run File "/usr/lib/luci/zope/lib/python/Zope2/Startup/__init__.py", line 95, in prepare File "/usr/lib/luci/zope/lib/python/Zope2/Startup/__init__.py", line 272, in makeLockFile ImportError: No module named misc.lock_file 2007.04.02 19:45:11 LOG5[20352:3082353584]: IPC reset (child died) However, it looks like it exists to me.... /usr/lib/luci/zope/lib/python/Zope2/Startup/misc/lock_file.py /usr/lib/luci/zope/lib/python/Zope2/Startup/misc/lock_file.pyc Any suggestions would be much appreciated. Many thanks, Spencer From kupcevic at redhat.com Mon Apr 2 19:52:07 2007 From: kupcevic at redhat.com (Stan Kupcevic) Date: Mon, 02 Apr 2007 15:52:07 -0400 Subject: [Linux-cluster] luci misc.lock_file problem In-Reply-To: <42628.82.70.154.145.1175539978.squirrel@squirrelmail.tgfslp.dalmany.co.uk> References: <42628.82.70.154.145.1175539978.squirrel@squirrelmail.tgfslp.dalmany.co.uk> Message-ID: <46115EE7.20101@redhat.com> Hi G, Could you give us some more information about environment: - OS and python version - how was luci installed - directly from cvs tree or rpm (where did you get it from) I'd like to reproduce this error, and fix it, since I can't stand problems of this sort... - stan G.Marshall wrote: >I am having a problem with luci, I get the following error > >2007.04.02 19:45:07 LOG7[20352:3082353584]: Remote FD=9 initialized >Traceback (most recent call last): > File "/usr/lib/luci/zope/lib/python/Zope2/Startup/run.py", line 56, in ? > File "/usr/lib/luci/zope/lib/python/Zope2/Startup/run.py", line 21, in run > File "/usr/lib/luci/zope/lib/python/Zope2/Startup/__init__.py", line 95, >in prepare > File "/usr/lib/luci/zope/lib/python/Zope2/Startup/__init__.py", line >272, in makeLockFile >ImportError: No module named misc.lock_file >2007.04.02 19:45:11 LOG5[20352:3082353584]: IPC reset (child died) > >However, it looks like it exists to me.... > >/usr/lib/luci/zope/lib/python/Zope2/Startup/misc/lock_file.py >/usr/lib/luci/zope/lib/python/Zope2/Startup/misc/lock_file.pyc > >Any suggestions would be much appreciated. > >Many thanks, > >Spencer > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > > From g.marshall at dalmany.co.uk Mon Apr 2 22:29:31 2007 From: g.marshall at dalmany.co.uk (G.Marshall) Date: Mon, 2 Apr 2007 23:29:31 +0100 (BST) Subject: [Linux-cluster] luci misc.lock_file problem In-Reply-To: <46115EE7.20101@redhat.com> References: <42628.82.70.154.145.1175539978.squirrel@squirrelmail.tgfslp.dalmany.co.uk> <46115EE7.20101@redhat.com> Message-ID: <56768.82.70.154.145.1175552971.squirrel@squirrelmail.tgfslp.dalmany.co.uk> Hello Stan, > Hi G, > > Could you give us some more information about environment: > - OS and python version Sorry, should have included that originally. Linux kernel version 2.6.20.3 (www.kernel.org) Debian Unstable Python 2.4.4 (debian install) stunnel 3.26 on i486-pc-linux-gnu PTHREAD+LIBWRAP with OpenSSL 0.9.8e 23 Feb 2007 (debian install) > - how was luci installed - directly from cvs tree or rpm (where did you > get it from) luci was direct from cvs cvs -d :pserver:cvs at sources.redhat.com:/cvs/cluster co conga zope-2.9.7-final and plone-2.5.2-1 (patched with Plone-2.5.2-1_CMFPlone.patch) are those installed with luci i.e. not debian install I could not get luci to start properly using /etc/rc.d/luci bash -x /etc/rc.d/luci start gives the following error + sed -e 's,\(^accept.*= \)\(.*\),\18084,' /var/lib/luci/etc/stunnel.conf + /usr/bin/stunnel -fd 0 2007.04.02 23:04:44 LOG3[20710:3083048640]: Invalid port: 0 (even with modifications to /etc/rc.d/luci to take into account Redhat vs Debian) so I started it by hand using stunnel -D 7 -p /var/lib/luci/var/certs/luci.pem -d 8084 -f -l /var/lib/luci/bin/runzope luci.pem contains /var/lib/luci/var/certs/https.key.pem /var/lib/luci/var/certs/https.pem The latest version of stunnel requires DH parameters too, which can be calculated using dd if=/dev/urandom count=2 | openssl dhparam -rand - 512 stunnel then listens on 8084 using firefox browse to https://localhost:8084/luci The running stunnel give the error shown below. Many thanks, Spencer > > > I'd like to reproduce this error, and fix it, since I can't stand > problems of this sort... > > > - stan > > > G.Marshall wrote: > >>I am having a problem with luci, I get the following error >> >>2007.04.02 19:45:07 LOG7[20352:3082353584]: Remote FD=9 initialized >>Traceback (most recent call last): >> File "/usr/lib/luci/zope/lib/python/Zope2/Startup/run.py", line 56, in >> ? >> File "/usr/lib/luci/zope/lib/python/Zope2/Startup/run.py", line 21, in >> run >> File "/usr/lib/luci/zope/lib/python/Zope2/Startup/__init__.py", line >> 95, >>in prepare >> File "/usr/lib/luci/zope/lib/python/Zope2/Startup/__init__.py", line >>272, in makeLockFile >>ImportError: No module named misc.lock_file >>2007.04.02 19:45:11 LOG5[20352:3082353584]: IPC reset (child died) >> >>However, it looks like it exists to me.... >> >>/usr/lib/luci/zope/lib/python/Zope2/Startup/misc/lock_file.py >>/usr/lib/luci/zope/lib/python/Zope2/Startup/misc/lock_file.pyc >> >>Any suggestions would be much appreciated. >> >>Many thanks, >> >>Spencer >> >>-- >>Linux-cluster mailing list >>Linux-cluster at redhat.com >>https://www.redhat.com/mailman/listinfo/linux-cluster >> >> > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From J.vandenHorn at xb.nl Tue Apr 3 08:04:57 2007 From: J.vandenHorn at xb.nl (Jeroen van den Horn) Date: Tue, 03 Apr 2007 10:04:57 +0200 Subject: [Linux-cluster] Fenced node never reboots properly In-Reply-To: <20070402152358.GE17135@redhat.com> References: <460B72F0.9020200@xb.nl> <460D1994.9060601@xb.nl> <4610B234.1040505@xb.nl> <20070402152358.GE17135@redhat.com> Message-ID: <46120AA9.3020802@xb.nl> Lon, The only thing I changed is the line my $powerop_mode = VM_POWEROP_MODE_HARD instead of 'soft'. The machine is now correctly power-cycled and comes back online. The last issue I am still struggling with is the fact that the heartbeat sometimes stops. In answer to a previous post: vmware-tools are installed and running. Based upon further investigation I must now conclude that the ESX-layer cannot throttle the VCPU up in time - the test-machines are mainly idle and VMWare (correctly) drops the amount of CPU-cycles allocated to this machine. When some process now suddenly needs more CPU power the machine 'chokes' - the 'virtual' load suddenly peaks and the heartbeat just doesn't execute (for example, we see fencing occur when the nightly cron jobs execute). I've tried renicing the cman_hbeat process but this also does not prove effective. I'm aware that setting a larger post-fail delay may solve the problem, but I'd prefer this value to be 0 in case of a real failure. Regards, Jeroen Lon Hohberger wrote: > On Mon, Apr 02, 2007 at 09:35:16AM +0200, Jeroen van den Horn wrote: > >> It's ESX - it runs on HP blades (AMD). Guest OSes are all 32-bit. >> >>> In response to Lon's suggestion I modified the fence_vmware code and >>> set the type of reset to HARD - cluster node now resets properly. >>> Remaining issue is that under VMWare we are still experiencing >>> performance issues. It's as if a node in the cluster starts 'lagging >>> behind' (also the system clock starts drifting) and that after some >>> time one of the nodes declares the other dead. >>> > > Do you have a patch or the new fence agent? I'd like the change so I > can commit it to CVS. Soft-reboot is *definitely* wrong; other problems > aside. > > -- Lon > > From npf at eurotux.com Tue Apr 3 09:12:20 2007 From: npf at eurotux.com (Nuno Fernandes) Date: Tue, 3 Apr 2007 10:12:20 +0100 Subject: [Linux-cluster] Problem in cluster with xen kernel Message-ID: <200704031012.27717.npf@eurotux.com> Hi, I'm using rhel5 default kernel and everything seems ok. [root at xen1 ~]# clustat Member Status: Quorate Member Name ID Status ------ ---- ---- ------ xen1.dc.server.pt 1 Online, Local xen2.dc.server.pt 2 Online xen3.dc.server.pt 3 Online Later on, i reboot xen3 to a Dom0 kernel and get in xen1 logs: Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] The token was lost in the OPERATIONAL state. Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] entering GATHER state from 2. [root at xen1 ~]# Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering GATHER state from 0. Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Creating commit token because I am the rep. Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Saving state aru 2f high seq received 2f Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering COMMIT state. Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering RECOVERY state. Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] position [0] member 172.16.40.107: Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] previous ring seq 84 rep 172.16.40.107 Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] aru 2f high delivered 2f received flag 0 Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] position [1] member 172.16.40.108: Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] previous ring seq 84 rep 172.16.40.107 Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] aru 2f high delivered 2f received flag 0 Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Did not need to originate any messages in recovery. Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Storing new sequence id for ring 58 Dec 19 23:02:52 xen1 kernel: dlm: closing connection to node 3 Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Sending initial ORF token Dec 19 23:02:52 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE Dec 19 23:02:52 xen1 openais[2747]: [CLM ] New Configuration: Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Left: Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Joined: Dec 19 23:02:52 xen1 openais[2747]: [SYNC ] This node is within the primary component and will provide service. Dec 19 23:02:52 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE Dec 19 23:02:52 xen1 openais[2747]: [CLM ] New Configuration: Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Left: Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Joined: Dec 19 23:02:52 xen1 openais[2747]: [SYNC ] This node is within the primary component and will provide service. Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. Dec 19 23:02:52 xen1 openais[2747]: [CLM ] got nodejoin message 172.16.40.107 Dec 19 23:02:53 xen1 openais[2747]: [CLM ] got nodejoin message 172.16.40.108 Dec 19 23:02:53 xen1 openais[2747]: [CPG ] got joinlist message from node 2 Dec 19 23:02:53 xen1 openais[2747]: [CPG ] got joinlist message from node 1 So far so good, xen3 is offline while it reboots... [root at xen1 ~]# clustat Member Status: Quorate Member Name ID Status ------ ---- ---- ------ xen1.dc.server.pt 1 Online, Local xen2.dc.server.pt 2 Online xen3.dc.server.pt 3 Offline After it reboots i get node join in xen1 server logs: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering GATHER state from 11. Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Creating commit token because I am the rep. Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Saving state aru 17 high seq received 17 Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering COMMIT state. Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering RECOVERY state. Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [0] member 172.16.40.107: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring seq 88 rep 172.16.40.107 Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 17 high delivered 17 received flag 0 Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [1] member 172.16.40.108: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring seq 88 rep 172.16.40.107 Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 17 high delivered 17 received flag 0 Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [2] member 172.16.40.116: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring seq 4 rep 172.16.40.116 Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 9 high delivered 9 received flag 0 Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Did not need to originate any messages in recovery. Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Storing new sequence id for ring 5c Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Sending initial ORF token Dec 19 23:05:03 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE Dec 19 23:05:03 xen1 openais[2747]: [CLM ] New Configuration: Dec 19 23:05:03 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) Dec 19 23:05:03 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Left: Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Joined: Dec 19 23:05:04 xen1 openais[2747]: [SYNC ] This node is within the primary component and will provide service. Dec 19 23:05:04 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE Dec 19 23:05:04 xen1 openais[2747]: [CLM ] New Configuration: Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Left: Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Joined: Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) Dec 19 23:05:04 xen1 openais[2747]: [SYNC ] This node is within the primary component and will provide service. Dec 19 23:05:04 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got nodejoin message 172.16.40.107 Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got nodejoin message 172.16.40.108 Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got nodejoin message 172.16.40.116 Dec 19 23:05:04 xen1 openais[2747]: [CPG ] got joinlist message from node 1 Dec 19 23:05:04 xen1 openais[2747]: [CPG ] got joinlist message from node 2 Dec 19 23:05:12 xen1 kernel: dlm: connecting to 3 Dec 19 23:05:12 xen1 kernel: dlm: got connection from 3 Clustat also reports ok status: [root at xen1 ~]# clustat Member Status: Quorate Member Name ID Status ------ ---- ---- ------ xen1.dc.server.pt 1 Online, Local xen2.dc.server.pt 2 Online xen3.dc.server.pt 3 Online Everything ok so far... Next i reboot xen2. When xen2 leaves xen1 complains that it can speak with xen3 and fences it. Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 Dec 19 23:08:55 xen1 last message repeated 47 times Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from 11. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Creating commit token because I am the rep. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Saving state aru 34 high seq received 34 Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering COMMIT state. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering RECOVERY state. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] position [0] member 172.16.40.107: Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] previous ring seq 92 rep 172.16.40.107 Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] aru 34 high delivered 34 received flag 0 Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] position [1] member 172.16.40.108: Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] previous ring seq 92 rep 172.16.40.107 Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] aru 34 high delivered 34 received flag 0 Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Did not need to originate any messages in recovery. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Storing new sequence id for ring 60 Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Sending initial ORF token Dec 19 23:08:59 xen1 kernel: dlm: closing connection to node 3 Dec 19 23:08:59 xen1 fenced[2763]: xen3.dc.aeiou.pt not a cluster member after 0 sec post_fail_delay Dec 19 23:09:00 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE Dec 19 23:09:00 xen1 fenced[2763]: xen2.dc.aeiou.pt not a cluster member after 0 sec post_fail_delay Dec 19 23:09:00 xen1 openais[2747]: [CLM ] New Configuration: Dec 19 23:09:00 xen1 fenced[2763]: fencing node "xen3.dc.aeiou.pt" Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Left: Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Joined: Dec 19 23:09:00 xen1 openais[2747]: [SYNC ] This node is within the primary component and will provide service. Dec 19 23:09:00 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE Dec 19 23:09:00 xen1 openais[2747]: [CLM ] New Configuration: Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Left: Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Joined: Dec 19 23:09:00 xen1 openais[2747]: [SYNC ] This node is within the primary component and will provide service. Dec 19 23:09:00 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. Dec 19 23:09:00 xen1 openais[2747]: [CLM ] got nodejoin message 172.16.40.107 Dec 19 23:09:00 xen1 openais[2747]: [CLM ] got nodejoin message 172.16.40.108 Dec 19 23:09:00 xen1 openais[2747]: [CPG ] got joinlist message from node 2 Dec 19 23:09:00 xen1 openais[2747]: [CPG ] got joinlist message from node 1 Dec 19 23:09:05 xen1 openais[2747]: [TOTEM] entering GATHER state from 11. Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] entering GATHER state from 0. Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Creating commit token because I am the rep. Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Saving state aru 1a high seq received 1a Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] entering COMMIT state. Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] entering RECOVERY state. Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] position [0] member 172.16.40.107: Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] previous ring seq 96 rep 172.16.40.107 Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] aru 1a high delivered 1a received flag 0 Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] position [1] member 172.16.40.116: Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] previous ring seq 92 rep 172.16.40.107 Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] aru 31 high delivered 31 received flag 0 Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Did not need to originate any messages in recovery. Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Storing new sequence id for ring 64 Dec 19 23:09:09 xen1 kernel: dlm: closing connection to node 2 Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Sending initial ORF token Dec 19 23:09:09 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE Dec 19 23:09:10 xen1 openais[2747]: [CLM ] New Configuration: Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Left: Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Joined: Dec 19 23:09:10 xen1 openais[2747]: [CMAN ] quorum lost, blocking activity Dec 19 23:09:10 xen1 openais[2747]: [SYNC ] This node is within the primary component and will provide service. Dec 19 23:09:10 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE Dec 19 23:09:10 xen1 openais[2747]: [CLM ] New Configuration: Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Left: Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Joined: Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) Dec 19 23:09:10 xen1 openais[2747]: [SYNC ] This node is within the primary component and will provide service. Dec 19 23:09:10 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. Dec 19 23:09:10 xen1 openais[2747]: [MAIN ] Node xen3.dc.aeiou.pt not joined to cman because it has rejoined an inquorate cluster Dec 19 23:09:10 xen1 openais[2747]: [CLM ] got nodejoin message 172.16.40.107 Dec 19 23:09:10 xen1 openais[2747]: [CLM ] got nodejoin message 172.16.40.116 Dec 19 23:09:10 xen1 openais[2747]: [CPG ] got joinlist message from node 3 Dec 19 23:09:10 xen1 openais[2747]: [CPG ] got joinlist message from node 1 Dec 19 23:09:14 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:09:14 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:09:19 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:09:19 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:09:24 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:09:24 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:09:29 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:09:29 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:09:34 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:09:34 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] The token was lost in the OPERATIONAL state. Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] entering GATHER state from 2. Dec 19 23:09:39 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:09:39 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering GATHER state from 0. Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Creating commit token because I am the rep. Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Saving state aru 18 high seq received 18 Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering COMMIT state. Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering RECOVERY state. Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] position [0] member 172.16.40.107: Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] previous ring seq 100 rep 172.16.40.107 Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] aru 18 high delivered 18 received flag 0 Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Did not need to originate any messages in recovery. Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Storing new sequence id for ring 68 Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Sending initial ORF token Dec 19 23:09:40 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE Dec 19 23:09:40 xen1 openais[2747]: [CLM ] New Configuration: Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Left: Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Joined: Dec 19 23:09:41 xen1 openais[2747]: [SYNC ] This node is within the primary component and will provide service. Dec 19 23:09:41 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE Dec 19 23:09:41 xen1 openais[2747]: [CLM ] New Configuration: Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Left: Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Joined: Dec 19 23:09:41 xen1 openais[2747]: [SYNC ] This node is within the primary component and will provide service. Dec 19 23:09:41 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. Dec 19 23:09:41 xen1 openais[2747]: [CLM ] got nodejoin message 172.16.40.107 Dec 19 23:09:41 xen1 openais[2747]: [CPG ] got joinlist message from node 1 Dec 19 23:09:44 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:09:44 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:09:49 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:09:49 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:09:54 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:09:54 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:09:59 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:09:59 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:10:04 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:10:04 xen1 ccsd[2740]: Error while processing connect: Connection refused Dec 19 23:10:09 xen1 ccsd[2740]: Cluster is not quorate. Refusing connection. Dec 19 23:10:09 xen1 ccsd[2740]: Error while processing connect: Connection refused The last errors are ok because the cluster isn't quorate anymore. xen2 was rebooting and xen3 was fenced, so leaving xen1 alone creates an unquorate cluster... The unusual thing is that it only happens when one of the nodes is using rhel5 xen kernel. Maybe something in the bridge-utils bug and multicast? This problem happens if i reboot xen1 server with xen kernel or xen2 server. Any intel? Thanks Nuno Fernandes -- Nuno Pais Fernandes Cisco Certified Network Associate Oracle Certified Professional Eurotux Informatica S.A. Tel: +351 253257395 Fax: +351 253257396 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From npf at eurotux.com Tue Apr 3 10:12:30 2007 From: npf at eurotux.com (Nuno Fernandes) Date: Tue, 3 Apr 2007 11:12:30 +0100 Subject: [Linux-cluster] Problem in cluster with xen kernel In-Reply-To: <200704031012.27717.npf@eurotux.com> References: <200704031012.27717.npf@eurotux.com> Message-ID: <200704031112.33501.npf@eurotux.com> Hi, Just for your information we've solved it. It was a problem in the xen bridge scripts that restarted network interfaces while the cluster is active. Changing /etc/xen/xend-config.sxp line (network-script network-bridge) to (network-script /bin/true) and creating the bridge in /etc/sysconfig/network-scripts/ifcfg-* files solved. Thanks Nuno Fernandes On Tuesday 03 April 2007 10:12:20 Nuno Fernandes wrote: > Hi, > > I'm using rhel5 default kernel and everything seems ok. > > [root at xen1 ~]# clustat > Member Status: Quorate > > Member Name ID Status > ------ ---- ---- ------ > xen1.dc.server.pt 1 Online, Local > xen2.dc.server.pt 2 Online > xen3.dc.server.pt 3 Online > > Later on, i reboot xen3 to a Dom0 kernel and get in xen1 logs: > > Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] The token was lost in the > OPERATIONAL state. > Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] Receive multicast socket recv > buffer size (262142 bytes). > Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] Transmit multicast socket send > buffer size (262142 bytes). > Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] entering GATHER state from 2. > > [root at xen1 ~]# Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering GATHER > state from 0. > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Creating commit token because I > am the rep. > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Saving state aru 2f high seq > received 2f > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering COMMIT state. > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering RECOVERY state. > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] position [0] member > 172.16.40.107: Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] previous ring > seq 84 rep 172.16.40.107 > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] aru 2f high delivered 2f > received flag 0 > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] position [1] member > 172.16.40.108: Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] previous ring > seq 84 rep 172.16.40.107 > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] aru 2f high delivered 2f > received flag 0 > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Did not need to originate any > messages in recovery. > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Storing new sequence id for > ring 58 > Dec 19 23:02:52 xen1 kernel: dlm: closing connection to node 3 > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Sending initial ORF token > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] New Configuration: > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Left: > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Joined: > Dec 19 23:02:52 xen1 openais[2747]: [SYNC ] This node is within the primary > component and will provide service. > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] New Configuration: > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Left: > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Joined: > Dec 19 23:02:52 xen1 openais[2747]: [SYNC ] This node is within the primary > component and will provide service. > Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. > Dec 19 23:02:52 xen1 openais[2747]: [CLM ] got nodejoin message > 172.16.40.107 Dec 19 23:02:53 xen1 openais[2747]: [CLM ] got nodejoin > message 172.16.40.108 Dec 19 23:02:53 xen1 openais[2747]: [CPG ] got > joinlist message from node 2 Dec 19 23:02:53 xen1 openais[2747]: [CPG ] > got joinlist message from node 1 > > So far so good, xen3 is offline while it reboots... > > [root at xen1 ~]# clustat > Member Status: Quorate > > Member Name ID Status > ------ ---- ---- ------ > xen1.dc.server.pt 1 Online, Local > xen2.dc.server.pt 2 Online > xen3.dc.server.pt 3 Offline > > After it reboots i get node join in xen1 server logs: > > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering GATHER state from 11. > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Creating commit token because I > am the rep. > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Saving state aru 17 high seq > received 17 > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering COMMIT state. > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering RECOVERY state. > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [0] member > 172.16.40.107: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring > seq 88 rep 172.16.40.107 > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 17 high delivered 17 > received flag 0 > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [1] member > 172.16.40.108: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring > seq 88 rep 172.16.40.107 > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 17 high delivered 17 > received flag 0 > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [2] member > 172.16.40.116: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring > seq 4 rep 172.16.40.116 > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 9 high delivered 9 received > flag 0 > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Did not need to originate any > messages in recovery. > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Storing new sequence id for > ring 5c > Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Sending initial ORF token > Dec 19 23:05:03 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > Dec 19 23:05:03 xen1 openais[2747]: [CLM ] New Configuration: > Dec 19 23:05:03 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > Dec 19 23:05:03 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Left: > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Joined: > Dec 19 23:05:04 xen1 openais[2747]: [SYNC ] This node is within the primary > component and will provide service. > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] New Configuration: > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Left: > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Joined: > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > Dec 19 23:05:04 xen1 openais[2747]: [SYNC ] This node is within the primary > component and will provide service. > Dec 19 23:05:04 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. > Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got nodejoin message > 172.16.40.107 Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got nodejoin > message 172.16.40.108 Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got > nodejoin message 172.16.40.116 Dec 19 23:05:04 xen1 openais[2747]: [CPG ] > got joinlist message from node 1 Dec 19 23:05:04 xen1 openais[2747]: [CPG > ] got joinlist message from node 2 Dec 19 23:05:12 xen1 kernel: dlm: > connecting to 3 > Dec 19 23:05:12 xen1 kernel: dlm: got connection from 3 > > Clustat also reports ok status: > > [root at xen1 ~]# clustat > Member Status: Quorate > > Member Name ID Status > ------ ---- ---- ------ > xen1.dc.server.pt 1 Online, Local > xen2.dc.server.pt 2 Online > xen3.dc.server.pt 3 Online > > Everything ok so far... > > Next i reboot xen2. When xen2 leaves xen1 complains that it can speak with > xen3 and fences it. > > Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 > Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 > Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > Dec 19 23:08:55 xen1 last message repeated 47 times > Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. > Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. > Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. > Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. > Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. > Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. > Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. > Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from 11. > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Creating commit token because I > am the rep. > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Saving state aru 34 high seq > received 34 > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering COMMIT state. > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering RECOVERY state. > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] position [0] member > 172.16.40.107: Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] previous ring > seq 92 rep 172.16.40.107 > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] aru 34 high delivered 34 > received flag 0 > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] position [1] member > 172.16.40.108: Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] previous ring > seq 92 rep 172.16.40.107 > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] aru 34 high delivered 34 > received flag 0 > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Did not need to originate any > messages in recovery. > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Storing new sequence id for > ring 60 > Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Sending initial ORF token > Dec 19 23:08:59 xen1 kernel: dlm: closing connection to node 3 > > > Dec 19 23:08:59 xen1 fenced[2763]: xen3.dc.aeiou.pt not a cluster member > after 0 sec post_fail_delay > > > > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > Dec 19 23:09:00 xen1 fenced[2763]: xen2.dc.aeiou.pt not a cluster member > after 0 sec post_fail_delay > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] New Configuration: > Dec 19 23:09:00 xen1 fenced[2763]: fencing node "xen3.dc.aeiou.pt" > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Left: > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Joined: > Dec 19 23:09:00 xen1 openais[2747]: [SYNC ] This node is within the primary > component and will provide service. > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] New Configuration: > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Left: > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Joined: > Dec 19 23:09:00 xen1 openais[2747]: [SYNC ] This node is within the primary > component and will provide service. > Dec 19 23:09:00 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. > Dec 19 23:09:00 xen1 openais[2747]: [CLM ] got nodejoin message > 172.16.40.107 Dec 19 23:09:00 xen1 openais[2747]: [CLM ] got nodejoin > message 172.16.40.108 Dec 19 23:09:00 xen1 openais[2747]: [CPG ] got > joinlist message from node 2 Dec 19 23:09:00 xen1 openais[2747]: [CPG ] > got joinlist message from node 1 Dec 19 23:09:05 xen1 openais[2747]: > [TOTEM] entering GATHER state from 11. Dec 19 23:09:09 xen1 openais[2747]: > [TOTEM] entering GATHER state from 0. Dec 19 23:09:09 xen1 openais[2747]: > [TOTEM] Creating commit token because I am the rep. > Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Saving state aru 1a high seq > received 1a > Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] entering COMMIT state. > Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] entering RECOVERY state. > Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] position [0] member > 172.16.40.107: Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] previous ring > seq 96 rep 172.16.40.107 > Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] aru 1a high delivered 1a > received flag 0 > Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] position [1] member > 172.16.40.116: Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] previous ring > seq 92 rep 172.16.40.107 > Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] aru 31 high delivered 31 > received flag 0 > Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Did not need to originate any > messages in recovery. > Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Storing new sequence id for > ring 64 > Dec 19 23:09:09 xen1 kernel: dlm: closing connection to node 2 > Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Sending initial ORF token > Dec 19 23:09:09 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] New Configuration: > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Left: > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Joined: > Dec 19 23:09:10 xen1 openais[2747]: [CMAN ] quorum lost, blocking activity > Dec 19 23:09:10 xen1 openais[2747]: [SYNC ] This node is within the primary > component and will provide service. > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] New Configuration: > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Left: > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Joined: > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > Dec 19 23:09:10 xen1 openais[2747]: [SYNC ] This node is within the primary > component and will provide service. > Dec 19 23:09:10 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. > Dec 19 23:09:10 xen1 openais[2747]: [MAIN ] Node xen3.dc.aeiou.pt not > joined to cman because it has rejoined an inquorate cluster > Dec 19 23:09:10 xen1 openais[2747]: [CLM ] got nodejoin message > 172.16.40.107 Dec 19 23:09:10 xen1 openais[2747]: [CLM ] got nodejoin > message 172.16.40.116 Dec 19 23:09:10 xen1 openais[2747]: [CPG ] got > joinlist message from node 3 Dec 19 23:09:10 xen1 openais[2747]: [CPG ] > got joinlist message from node 1 Dec 19 23:09:14 xen1 ccsd[2740]: Cluster > is not quorate. Refusing connection. Dec 19 23:09:14 xen1 ccsd[2740]: > Error while processing connect: Connection refused > Dec 19 23:09:19 xen1 ccsd[2740]: Cluster is not quorate. Refusing > connection. Dec 19 23:09:19 xen1 ccsd[2740]: Error while processing > connect: Connection refused > Dec 19 23:09:24 xen1 ccsd[2740]: Cluster is not quorate. Refusing > connection. Dec 19 23:09:24 xen1 ccsd[2740]: Error while processing > connect: Connection refused > Dec 19 23:09:29 xen1 ccsd[2740]: Cluster is not quorate. Refusing > connection. Dec 19 23:09:29 xen1 ccsd[2740]: Error while processing > connect: Connection refused > Dec 19 23:09:34 xen1 ccsd[2740]: Cluster is not quorate. Refusing > connection. Dec 19 23:09:34 xen1 ccsd[2740]: Error while processing > connect: Connection refused > Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] The token was lost in the > OPERATIONAL state. > Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] Receive multicast socket recv > buffer size (262142 bytes). > Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] Transmit multicast socket send > buffer size (262142 bytes). > Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] entering GATHER state from 2. > Dec 19 23:09:39 xen1 ccsd[2740]: Cluster is not quorate. Refusing > connection. Dec 19 23:09:39 xen1 ccsd[2740]: Error while processing > connect: Connection refused > Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering GATHER state from 0. > Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Creating commit token because I > am the rep. > Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Saving state aru 18 high seq > received 18 > Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering COMMIT state. > Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering RECOVERY state. > Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] position [0] member > 172.16.40.107: Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] previous ring > seq 100 rep 172.16.40.107 > Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] aru 18 high delivered 18 > received flag 0 > Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Did not need to originate any > messages in recovery. > Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Storing new sequence id for > ring 68 > Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Sending initial ORF token > Dec 19 23:09:40 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > Dec 19 23:09:40 xen1 openais[2747]: [CLM ] New Configuration: > Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Left: > Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Joined: > Dec 19 23:09:41 xen1 openais[2747]: [SYNC ] This node is within the primary > component and will provide service. > Dec 19 23:09:41 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > Dec 19 23:09:41 xen1 openais[2747]: [CLM ] New Configuration: > Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Left: > Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Joined: > Dec 19 23:09:41 xen1 openais[2747]: [SYNC ] This node is within the primary > component and will provide service. > Dec 19 23:09:41 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. > Dec 19 23:09:41 xen1 openais[2747]: [CLM ] got nodejoin message > 172.16.40.107 Dec 19 23:09:41 xen1 openais[2747]: [CPG ] got joinlist > message from node 1 Dec 19 23:09:44 xen1 ccsd[2740]: Cluster is not > quorate. Refusing connection. Dec 19 23:09:44 xen1 ccsd[2740]: Error while > processing connect: Connection refused > Dec 19 23:09:49 xen1 ccsd[2740]: Cluster is not quorate. Refusing > connection. Dec 19 23:09:49 xen1 ccsd[2740]: Error while processing > connect: Connection refused > Dec 19 23:09:54 xen1 ccsd[2740]: Cluster is not quorate. Refusing > connection. Dec 19 23:09:54 xen1 ccsd[2740]: Error while processing > connect: Connection refused > Dec 19 23:09:59 xen1 ccsd[2740]: Cluster is not quorate. Refusing > connection. Dec 19 23:09:59 xen1 ccsd[2740]: Error while processing > connect: Connection refused > Dec 19 23:10:04 xen1 ccsd[2740]: Cluster is not quorate. Refusing > connection. Dec 19 23:10:04 xen1 ccsd[2740]: Error while processing > connect: Connection refused > Dec 19 23:10:09 xen1 ccsd[2740]: Cluster is not quorate. Refusing > connection. Dec 19 23:10:09 xen1 ccsd[2740]: Error while processing > connect: Connection refused > > The last errors are ok because the cluster isn't quorate anymore. xen2 was > rebooting and xen3 was fenced, so leaving xen1 alone creates an unquorate > cluster... > > The unusual thing is that it only happens when one of the nodes is using > rhel5 xen kernel. Maybe something in the bridge-utils bug and multicast? > This problem happens if i reboot xen1 server with xen kernel or xen2 > server. > > > Any intel? > > Thanks > Nuno Fernandes -- Nuno Pais Fernandes Cisco Certified Network Associate Oracle Certified Professional Eurotux Informatica S.A. Tel: +351 253257395 Fax: +351 253257396 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From carlopmart at gmail.com Tue Apr 3 10:49:31 2007 From: carlopmart at gmail.com (carlopmart) Date: Tue, 03 Apr 2007 12:49:31 +0200 Subject: [Linux-cluster] Problem in cluster with xen kernel In-Reply-To: <200704031112.33501.npf@eurotux.com> References: <200704031012.27717.npf@eurotux.com> <200704031112.33501.npf@eurotux.com> Message-ID: <4612313B.8020408@gmail.com> Nuno Fernandes wrote: > Hi, > > Just for your information we've solved it. It was a problem in the xen bridge > scripts that restarted network interfaces while the cluster is active. > > Changing /etc/xen/xend-config.sxp line > > (network-script network-bridge) > > to > > (network-script /bin/true) > > and creating the bridge in /etc/sysconfig/network-scripts/ifcfg-* files > solved. > > Thanks > Nuno Fernandes > > On Tuesday 03 April 2007 10:12:20 Nuno Fernandes wrote: >> Hi, >> >> I'm using rhel5 default kernel and everything seems ok. >> >> [root at xen1 ~]# clustat >> Member Status: Quorate >> >> Member Name ID Status >> ------ ---- ---- ------ >> xen1.dc.server.pt 1 Online, Local >> xen2.dc.server.pt 2 Online >> xen3.dc.server.pt 3 Online >> >> Later on, i reboot xen3 to a Dom0 kernel and get in xen1 logs: >> >> Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] The token was lost in the >> OPERATIONAL state. >> Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] Receive multicast socket recv >> buffer size (262142 bytes). >> Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] Transmit multicast socket send >> buffer size (262142 bytes). >> Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] entering GATHER state from 2. >> >> [root at xen1 ~]# Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering GATHER >> state from 0. >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Creating commit token because I >> am the rep. >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Saving state aru 2f high seq >> received 2f >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering COMMIT state. >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering RECOVERY state. >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] position [0] member >> 172.16.40.107: Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] previous ring >> seq 84 rep 172.16.40.107 >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] aru 2f high delivered 2f >> received flag 0 >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] position [1] member >> 172.16.40.108: Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] previous ring >> seq 84 rep 172.16.40.107 >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] aru 2f high delivered 2f >> received flag 0 >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Did not need to originate any >> messages in recovery. >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Storing new sequence id for >> ring 58 >> Dec 19 23:02:52 xen1 kernel: dlm: closing connection to node 3 >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Sending initial ORF token >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] New Configuration: >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Left: >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Joined: >> Dec 19 23:02:52 xen1 openais[2747]: [SYNC ] This node is within the primary >> component and will provide service. >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] New Configuration: >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Left: >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Joined: >> Dec 19 23:02:52 xen1 openais[2747]: [SYNC ] This node is within the primary >> component and will provide service. >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] got nodejoin message >> 172.16.40.107 Dec 19 23:02:53 xen1 openais[2747]: [CLM ] got nodejoin >> message 172.16.40.108 Dec 19 23:02:53 xen1 openais[2747]: [CPG ] got >> joinlist message from node 2 Dec 19 23:02:53 xen1 openais[2747]: [CPG ] >> got joinlist message from node 1 >> >> So far so good, xen3 is offline while it reboots... >> >> [root at xen1 ~]# clustat >> Member Status: Quorate >> >> Member Name ID Status >> ------ ---- ---- ------ >> xen1.dc.server.pt 1 Online, Local >> xen2.dc.server.pt 2 Online >> xen3.dc.server.pt 3 Offline >> >> After it reboots i get node join in xen1 server logs: >> >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering GATHER state from 11. >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Creating commit token because I >> am the rep. >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Saving state aru 17 high seq >> received 17 >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering COMMIT state. >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering RECOVERY state. >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [0] member >> 172.16.40.107: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring >> seq 88 rep 172.16.40.107 >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 17 high delivered 17 >> received flag 0 >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [1] member >> 172.16.40.108: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring >> seq 88 rep 172.16.40.107 >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 17 high delivered 17 >> received flag 0 >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [2] member >> 172.16.40.116: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring >> seq 4 rep 172.16.40.116 >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 9 high delivered 9 received >> flag 0 >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Did not need to originate any >> messages in recovery. >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Storing new sequence id for >> ring 5c >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Sending initial ORF token >> Dec 19 23:05:03 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE >> Dec 19 23:05:03 xen1 openais[2747]: [CLM ] New Configuration: >> Dec 19 23:05:03 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) >> Dec 19 23:05:03 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Left: >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Joined: >> Dec 19 23:05:04 xen1 openais[2747]: [SYNC ] This node is within the primary >> component and will provide service. >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] New Configuration: >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Left: >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Joined: >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) >> Dec 19 23:05:04 xen1 openais[2747]: [SYNC ] This node is within the primary >> component and will provide service. >> Dec 19 23:05:04 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got nodejoin message >> 172.16.40.107 Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got nodejoin >> message 172.16.40.108 Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got >> nodejoin message 172.16.40.116 Dec 19 23:05:04 xen1 openais[2747]: [CPG ] >> got joinlist message from node 1 Dec 19 23:05:04 xen1 openais[2747]: [CPG >> ] got joinlist message from node 2 Dec 19 23:05:12 xen1 kernel: dlm: >> connecting to 3 >> Dec 19 23:05:12 xen1 kernel: dlm: got connection from 3 >> >> Clustat also reports ok status: >> >> [root at xen1 ~]# clustat >> Member Status: Quorate >> >> Member Name ID Status >> ------ ---- ---- ------ >> xen1.dc.server.pt 1 Online, Local >> xen2.dc.server.pt 2 Online >> xen3.dc.server.pt 3 Online >> >> Everything ok so far... >> >> Next i reboot xen2. When xen2 leaves xen1 complains that it can speak with >> xen3 and fences it. >> >> Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 >> Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 >> Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 >> Dec 19 23:08:55 xen1 last message repeated 47 times >> Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE >> Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. >> Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 >> Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE >> Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. >> Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 >> Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE >> Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. >> Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 >> Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE >> Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. >> Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 >> Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE >> Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. >> Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 >> Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE >> Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. >> Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 >> Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE >> Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. >> Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 >> Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE >> Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from 6. >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from 11. >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Creating commit token because I >> am the rep. >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Saving state aru 34 high seq >> received 34 >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering COMMIT state. >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering RECOVERY state. >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] position [0] member >> 172.16.40.107: Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] previous ring >> seq 92 rep 172.16.40.107 >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] aru 34 high delivered 34 >> received flag 0 >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] position [1] member >> 172.16.40.108: Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] previous ring >> seq 92 rep 172.16.40.107 >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] aru 34 high delivered 34 >> received flag 0 >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Did not need to originate any >> messages in recovery. >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Storing new sequence id for >> ring 60 >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Sending initial ORF token >> Dec 19 23:08:59 xen1 kernel: dlm: closing connection to node 3 >> >> >> Dec 19 23:08:59 xen1 fenced[2763]: xen3.dc.aeiou.pt not a cluster member >> after 0 sec post_fail_delay >> >> >> >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE >> Dec 19 23:09:00 xen1 fenced[2763]: xen2.dc.aeiou.pt not a cluster member >> after 0 sec post_fail_delay >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] New Configuration: >> Dec 19 23:09:00 xen1 fenced[2763]: fencing node "xen3.dc.aeiou.pt" >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Left: >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Joined: >> Dec 19 23:09:00 xen1 openais[2747]: [SYNC ] This node is within the primary >> component and will provide service. >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] New Configuration: >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Left: >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Joined: >> Dec 19 23:09:00 xen1 openais[2747]: [SYNC ] This node is within the primary >> component and will provide service. >> Dec 19 23:09:00 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] got nodejoin message >> 172.16.40.107 Dec 19 23:09:00 xen1 openais[2747]: [CLM ] got nodejoin >> message 172.16.40.108 Dec 19 23:09:00 xen1 openais[2747]: [CPG ] got >> joinlist message from node 2 Dec 19 23:09:00 xen1 openais[2747]: [CPG ] >> got joinlist message from node 1 Dec 19 23:09:05 xen1 openais[2747]: >> [TOTEM] entering GATHER state from 11. Dec 19 23:09:09 xen1 openais[2747]: >> [TOTEM] entering GATHER state from 0. Dec 19 23:09:09 xen1 openais[2747]: >> [TOTEM] Creating commit token because I am the rep. >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Saving state aru 1a high seq >> received 1a >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] entering COMMIT state. >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] entering RECOVERY state. >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] position [0] member >> 172.16.40.107: Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] previous ring >> seq 96 rep 172.16.40.107 >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] aru 1a high delivered 1a >> received flag 0 >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] position [1] member >> 172.16.40.116: Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] previous ring >> seq 92 rep 172.16.40.107 >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] aru 31 high delivered 31 >> received flag 0 >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Did not need to originate any >> messages in recovery. >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Storing new sequence id for >> ring 64 >> Dec 19 23:09:09 xen1 kernel: dlm: closing connection to node 2 >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Sending initial ORF token >> Dec 19 23:09:09 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] New Configuration: >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Left: >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Joined: >> Dec 19 23:09:10 xen1 openais[2747]: [CMAN ] quorum lost, blocking activity >> Dec 19 23:09:10 xen1 openais[2747]: [SYNC ] This node is within the primary >> component and will provide service. >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] New Configuration: >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Left: >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Joined: >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) >> Dec 19 23:09:10 xen1 openais[2747]: [SYNC ] This node is within the primary >> component and will provide service. >> Dec 19 23:09:10 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. >> Dec 19 23:09:10 xen1 openais[2747]: [MAIN ] Node xen3.dc.aeiou.pt not >> joined to cman because it has rejoined an inquorate cluster >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] got nodejoin message >> 172.16.40.107 Dec 19 23:09:10 xen1 openais[2747]: [CLM ] got nodejoin >> message 172.16.40.116 Dec 19 23:09:10 xen1 openais[2747]: [CPG ] got >> joinlist message from node 3 Dec 19 23:09:10 xen1 openais[2747]: [CPG ] >> got joinlist message from node 1 Dec 19 23:09:14 xen1 ccsd[2740]: Cluster >> is not quorate. Refusing connection. Dec 19 23:09:14 xen1 ccsd[2740]: >> Error while processing connect: Connection refused >> Dec 19 23:09:19 xen1 ccsd[2740]: Cluster is not quorate. Refusing >> connection. Dec 19 23:09:19 xen1 ccsd[2740]: Error while processing >> connect: Connection refused >> Dec 19 23:09:24 xen1 ccsd[2740]: Cluster is not quorate. Refusing >> connection. Dec 19 23:09:24 xen1 ccsd[2740]: Error while processing >> connect: Connection refused >> Dec 19 23:09:29 xen1 ccsd[2740]: Cluster is not quorate. Refusing >> connection. Dec 19 23:09:29 xen1 ccsd[2740]: Error while processing >> connect: Connection refused >> Dec 19 23:09:34 xen1 ccsd[2740]: Cluster is not quorate. Refusing >> connection. Dec 19 23:09:34 xen1 ccsd[2740]: Error while processing >> connect: Connection refused >> Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] The token was lost in the >> OPERATIONAL state. >> Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] Receive multicast socket recv >> buffer size (262142 bytes). >> Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] Transmit multicast socket send >> buffer size (262142 bytes). >> Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] entering GATHER state from 2. >> Dec 19 23:09:39 xen1 ccsd[2740]: Cluster is not quorate. Refusing >> connection. Dec 19 23:09:39 xen1 ccsd[2740]: Error while processing >> connect: Connection refused >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering GATHER state from 0. >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Creating commit token because I >> am the rep. >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Saving state aru 18 high seq >> received 18 >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering COMMIT state. >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering RECOVERY state. >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] position [0] member >> 172.16.40.107: Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] previous ring >> seq 100 rep 172.16.40.107 >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] aru 18 high delivered 18 >> received flag 0 >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Did not need to originate any >> messages in recovery. >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Storing new sequence id for >> ring 68 >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Sending initial ORF token >> Dec 19 23:09:40 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE >> Dec 19 23:09:40 xen1 openais[2747]: [CLM ] New Configuration: >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Left: >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Joined: >> Dec 19 23:09:41 xen1 openais[2747]: [SYNC ] This node is within the primary >> component and will provide service. >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] New Configuration: >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Left: >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Joined: >> Dec 19 23:09:41 xen1 openais[2747]: [SYNC ] This node is within the primary >> component and will provide service. >> Dec 19 23:09:41 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] got nodejoin message >> 172.16.40.107 Dec 19 23:09:41 xen1 openais[2747]: [CPG ] got joinlist >> message from node 1 Dec 19 23:09:44 xen1 ccsd[2740]: Cluster is not >> quorate. Refusing connection. Dec 19 23:09:44 xen1 ccsd[2740]: Error while >> processing connect: Connection refused >> Dec 19 23:09:49 xen1 ccsd[2740]: Cluster is not quorate. Refusing >> connection. Dec 19 23:09:49 xen1 ccsd[2740]: Error while processing >> connect: Connection refused >> Dec 19 23:09:54 xen1 ccsd[2740]: Cluster is not quorate. Refusing >> connection. Dec 19 23:09:54 xen1 ccsd[2740]: Error while processing >> connect: Connection refused >> Dec 19 23:09:59 xen1 ccsd[2740]: Cluster is not quorate. Refusing >> connection. Dec 19 23:09:59 xen1 ccsd[2740]: Error while processing >> connect: Connection refused >> Dec 19 23:10:04 xen1 ccsd[2740]: Cluster is not quorate. Refusing >> connection. Dec 19 23:10:04 xen1 ccsd[2740]: Error while processing >> connect: Connection refused >> Dec 19 23:10:09 xen1 ccsd[2740]: Cluster is not quorate. Refusing >> connection. Dec 19 23:10:09 xen1 ccsd[2740]: Error while processing >> connect: Connection refused >> >> The last errors are ok because the cluster isn't quorate anymore. xen2 was >> rebooting and xen3 was fenced, so leaving xen1 alone creates an unquorate >> cluster... >> >> The unusual thing is that it only happens when one of the nodes is using >> rhel5 xen kernel. Maybe something in the bridge-utils bug and multicast? >> This problem happens if i reboot xen1 server with xen kernel or xen2 >> server. >> >> >> Any intel? >> >> Thanks >> Nuno Fernandes Hi Nuno, Sorry for this question: how do you have created xen bridges with ifcfg-files?? I am trying to do the same, but it doesn't works for me ... > > > > > ------------------------------------------------------------------------ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- CL Martinez carlopmart {at} gmail {d0t} com From npf at eurotux.com Tue Apr 3 14:02:39 2007 From: npf at eurotux.com (Nuno Fernandes) Date: Tue, 3 Apr 2007 15:02:39 +0100 Subject: [Linux-cluster] Problem in cluster with xen kernel In-Reply-To: <4612313B.8020408@gmail.com> References: <200704031012.27717.npf@eurotux.com> <200704031112.33501.npf@eurotux.com> <4612313B.8020408@gmail.com> Message-ID: <200704031502.48009.npf@eurotux.com> On Tuesday 03 April 2007 11:49:31 carlopmart wrote: > Nuno Fernandes wrote: > > Hi, > > > > Just for your information we've solved it. It was a problem in the xen > > bridge scripts that restarted network interfaces while the cluster is > > active. > > > > Changing /etc/xen/xend-config.sxp line > > > > (network-script network-bridge) > > > > to > > > > (network-script /bin/true) > > > > and creating the bridge in /etc/sysconfig/network-scripts/ifcfg-* files > > solved. > > > > Thanks > > Nuno Fernandes > > > > On Tuesday 03 April 2007 10:12:20 Nuno Fernandes wrote: > >> Hi, > >> > >> I'm using rhel5 default kernel and everything seems ok. > >> > >> [root at xen1 ~]# clustat > >> Member Status: Quorate > >> > >> Member Name ID Status > >> ------ ---- ---- ------ > >> xen1.dc.server.pt 1 Online, Local > >> xen2.dc.server.pt 2 Online > >> xen3.dc.server.pt 3 Online > >> > >> Later on, i reboot xen3 to a Dom0 kernel and get in xen1 logs: > >> > >> Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] The token was lost in the > >> OPERATIONAL state. > >> Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] Receive multicast socket > >> recv buffer size (262142 bytes). > >> Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] Transmit multicast socket > >> send buffer size (262142 bytes). > >> Dec 19 23:02:47 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 2. > >> > >> [root at xen1 ~]# Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering > >> GATHER state from 0. > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Creating commit token > >> because I am the rep. > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Saving state aru 2f high seq > >> received 2f > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering COMMIT state. > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering RECOVERY state. > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] position [0] member > >> 172.16.40.107: Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] previous ring > >> seq 84 rep 172.16.40.107 > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] aru 2f high delivered 2f > >> received flag 0 > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] position [1] member > >> 172.16.40.108: Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] previous ring > >> seq 84 rep 172.16.40.107 > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] aru 2f high delivered 2f > >> received flag 0 > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Did not need to originate > >> any messages in recovery. > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Storing new sequence id for > >> ring 58 > >> Dec 19 23:02:52 xen1 kernel: dlm: closing connection to node 3 > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] Sending initial ORF token > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] New Configuration: > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Left: > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Joined: > >> Dec 19 23:02:52 xen1 openais[2747]: [SYNC ] This node is within the > >> primary component and will provide service. > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] New Configuration: > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Left: > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] Members Joined: > >> Dec 19 23:02:52 xen1 openais[2747]: [SYNC ] This node is within the > >> primary component and will provide service. > >> Dec 19 23:02:52 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. > >> Dec 19 23:02:52 xen1 openais[2747]: [CLM ] got nodejoin message > >> 172.16.40.107 Dec 19 23:02:53 xen1 openais[2747]: [CLM ] got nodejoin > >> message 172.16.40.108 Dec 19 23:02:53 xen1 openais[2747]: [CPG ] got > >> joinlist message from node 2 Dec 19 23:02:53 xen1 openais[2747]: [CPG ] > >> got joinlist message from node 1 > >> > >> So far so good, xen3 is offline while it reboots... > >> > >> [root at xen1 ~]# clustat > >> Member Status: Quorate > >> > >> Member Name ID Status > >> ------ ---- ---- ------ > >> xen1.dc.server.pt 1 Online, Local > >> xen2.dc.server.pt 2 Online > >> xen3.dc.server.pt 3 Offline > >> > >> After it reboots i get node join in xen1 server logs: > >> > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 11. Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Creating commit token > >> because I am the rep. > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Saving state aru 17 high seq > >> received 17 > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering COMMIT state. > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] entering RECOVERY state. > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [0] member > >> 172.16.40.107: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring > >> seq 88 rep 172.16.40.107 > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 17 high delivered 17 > >> received flag 0 > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [1] member > >> 172.16.40.108: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring > >> seq 88 rep 172.16.40.107 > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 17 high delivered 17 > >> received flag 0 > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] position [2] member > >> 172.16.40.116: Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] previous ring > >> seq 4 rep 172.16.40.116 > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] aru 9 high delivered 9 > >> received flag 0 > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Did not need to originate > >> any messages in recovery. > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Storing new sequence id for > >> ring 5c > >> Dec 19 23:05:03 xen1 openais[2747]: [TOTEM] Sending initial ORF token > >> Dec 19 23:05:03 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > >> Dec 19 23:05:03 xen1 openais[2747]: [CLM ] New Configuration: > >> Dec 19 23:05:03 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > >> Dec 19 23:05:03 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Left: > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Joined: > >> Dec 19 23:05:04 xen1 openais[2747]: [SYNC ] This node is within the > >> primary component and will provide service. > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] New Configuration: > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Left: > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] Members Joined: > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > >> Dec 19 23:05:04 xen1 openais[2747]: [SYNC ] This node is within the > >> primary component and will provide service. > >> Dec 19 23:05:04 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. > >> Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got nodejoin message > >> 172.16.40.107 Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got nodejoin > >> message 172.16.40.108 Dec 19 23:05:04 xen1 openais[2747]: [CLM ] got > >> nodejoin message 172.16.40.116 Dec 19 23:05:04 xen1 openais[2747]: [CPG > >> ] got joinlist message from node 1 Dec 19 23:05:04 xen1 openais[2747]: > >> [CPG ] got joinlist message from node 2 Dec 19 23:05:12 xen1 kernel: > >> dlm: connecting to 3 > >> Dec 19 23:05:12 xen1 kernel: dlm: got connection from 3 > >> > >> Clustat also reports ok status: > >> > >> [root at xen1 ~]# clustat > >> Member Status: Quorate > >> > >> Member Name ID Status > >> ------ ---- ---- ------ > >> xen1.dc.server.pt 1 Online, Local > >> xen2.dc.server.pt 2 Online > >> xen3.dc.server.pt 3 Online > >> > >> Everything ok so far... > >> > >> Next i reboot xen2. When xen2 leaves xen1 complains that it can speak > >> with xen3 and fences it. > >> > >> Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 > >> Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 > >> Dec 19 23:08:48 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > >> Dec 19 23:08:55 xen1 last message repeated 47 times > >> Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > >> Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 6. Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > >> Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > >> Dec 19 23:08:55 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 6. Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > >> Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > >> Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 6. Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > >> Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > >> Dec 19 23:08:56 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 6. Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > >> Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > >> Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 6. Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > >> Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > >> Dec 19 23:08:57 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 6. Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > >> Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > >> Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 6. Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > >> Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > >> Dec 19 23:08:58 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 6. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 6. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Retransmit List: 32 33 34 > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] FAILED TO RECEIVE > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 6. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering GATHER state > >> from 11. Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Creating commit > >> token because I am the rep. > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Saving state aru 34 high seq > >> received 34 > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering COMMIT state. > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] entering RECOVERY state. > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] position [0] member > >> 172.16.40.107: Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] previous ring > >> seq 92 rep 172.16.40.107 > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] aru 34 high delivered 34 > >> received flag 0 > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] position [1] member > >> 172.16.40.108: Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] previous ring > >> seq 92 rep 172.16.40.107 > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] aru 34 high delivered 34 > >> received flag 0 > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Did not need to originate > >> any messages in recovery. > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Storing new sequence id for > >> ring 60 > >> Dec 19 23:08:59 xen1 openais[2747]: [TOTEM] Sending initial ORF token > >> Dec 19 23:08:59 xen1 kernel: dlm: closing connection to node 3 > >> > >> > >> Dec 19 23:08:59 xen1 fenced[2763]: xen3.dc.aeiou.pt not a cluster member > >> after 0 sec post_fail_delay > >> > >> > >> > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > >> Dec 19 23:09:00 xen1 fenced[2763]: xen2.dc.aeiou.pt not a cluster member > >> after 0 sec post_fail_delay > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] New Configuration: > >> Dec 19 23:09:00 xen1 fenced[2763]: fencing node "xen3.dc.aeiou.pt" > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Left: > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Joined: > >> Dec 19 23:09:00 xen1 openais[2747]: [SYNC ] This node is within the > >> primary component and will provide service. > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] New Configuration: > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Left: > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] Members Joined: > >> Dec 19 23:09:00 xen1 openais[2747]: [SYNC ] This node is within the > >> primary component and will provide service. > >> Dec 19 23:09:00 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. > >> Dec 19 23:09:00 xen1 openais[2747]: [CLM ] got nodejoin message > >> 172.16.40.107 Dec 19 23:09:00 xen1 openais[2747]: [CLM ] got nodejoin > >> message 172.16.40.108 Dec 19 23:09:00 xen1 openais[2747]: [CPG ] got > >> joinlist message from node 2 Dec 19 23:09:00 xen1 openais[2747]: [CPG ] > >> got joinlist message from node 1 Dec 19 23:09:05 xen1 openais[2747]: > >> [TOTEM] entering GATHER state from 11. Dec 19 23:09:09 xen1 > >> openais[2747]: [TOTEM] entering GATHER state from 0. Dec 19 23:09:09 > >> xen1 openais[2747]: [TOTEM] Creating commit token because I am the rep. > >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Saving state aru 1a high seq > >> received 1a > >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] entering COMMIT state. > >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] entering RECOVERY state. > >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] position [0] member > >> 172.16.40.107: Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] previous ring > >> seq 96 rep 172.16.40.107 > >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] aru 1a high delivered 1a > >> received flag 0 > >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] position [1] member > >> 172.16.40.116: Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] previous ring > >> seq 92 rep 172.16.40.107 > >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] aru 31 high delivered 31 > >> received flag 0 > >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Did not need to originate > >> any messages in recovery. > >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Storing new sequence id for > >> ring 64 > >> Dec 19 23:09:09 xen1 kernel: dlm: closing connection to node 2 > >> Dec 19 23:09:09 xen1 openais[2747]: [TOTEM] Sending initial ORF token > >> Dec 19 23:09:09 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] New Configuration: > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Left: > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.108) > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Joined: > >> Dec 19 23:09:10 xen1 openais[2747]: [CMAN ] quorum lost, blocking > >> activity Dec 19 23:09:10 xen1 openais[2747]: [SYNC ] This node is within > >> the primary component and will provide service. > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] New Configuration: > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Left: > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] Members Joined: > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > >> Dec 19 23:09:10 xen1 openais[2747]: [SYNC ] This node is within the > >> primary component and will provide service. > >> Dec 19 23:09:10 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. > >> Dec 19 23:09:10 xen1 openais[2747]: [MAIN ] Node xen3.dc.aeiou.pt not > >> joined to cman because it has rejoined an inquorate cluster > >> Dec 19 23:09:10 xen1 openais[2747]: [CLM ] got nodejoin message > >> 172.16.40.107 Dec 19 23:09:10 xen1 openais[2747]: [CLM ] got nodejoin > >> message 172.16.40.116 Dec 19 23:09:10 xen1 openais[2747]: [CPG ] got > >> joinlist message from node 3 Dec 19 23:09:10 xen1 openais[2747]: [CPG ] > >> got joinlist message from node 1 Dec 19 23:09:14 xen1 ccsd[2740]: > >> Cluster is not quorate. Refusing connection. Dec 19 23:09:14 xen1 > >> ccsd[2740]: Error while processing connect: Connection refused > >> Dec 19 23:09:19 xen1 ccsd[2740]: Cluster is not quorate. Refusing > >> connection. Dec 19 23:09:19 xen1 ccsd[2740]: Error while processing > >> connect: Connection refused > >> Dec 19 23:09:24 xen1 ccsd[2740]: Cluster is not quorate. Refusing > >> connection. Dec 19 23:09:24 xen1 ccsd[2740]: Error while processing > >> connect: Connection refused > >> Dec 19 23:09:29 xen1 ccsd[2740]: Cluster is not quorate. Refusing > >> connection. Dec 19 23:09:29 xen1 ccsd[2740]: Error while processing > >> connect: Connection refused > >> Dec 19 23:09:34 xen1 ccsd[2740]: Cluster is not quorate. Refusing > >> connection. Dec 19 23:09:34 xen1 ccsd[2740]: Error while processing > >> connect: Connection refused > >> Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] The token was lost in the > >> OPERATIONAL state. > >> Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] Receive multicast socket > >> recv buffer size (262142 bytes). > >> Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] Transmit multicast socket > >> send buffer size (262142 bytes). > >> Dec 19 23:09:36 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 2. Dec 19 23:09:39 xen1 ccsd[2740]: Cluster is not quorate. Refusing > >> connection. Dec 19 23:09:39 xen1 ccsd[2740]: Error while processing > >> connect: Connection refused > >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering GATHER state from > >> 0. Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Creating commit token > >> because I am the rep. > >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Saving state aru 18 high seq > >> received 18 > >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering COMMIT state. > >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] entering RECOVERY state. > >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] position [0] member > >> 172.16.40.107: Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] previous ring > >> seq 100 rep 172.16.40.107 > >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] aru 18 high delivered 18 > >> received flag 0 > >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Did not need to originate > >> any messages in recovery. > >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Storing new sequence id for > >> ring 68 > >> Dec 19 23:09:40 xen1 openais[2747]: [TOTEM] Sending initial ORF token > >> Dec 19 23:09:40 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > >> Dec 19 23:09:40 xen1 openais[2747]: [CLM ] New Configuration: > >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Left: > >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.116) > >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Joined: > >> Dec 19 23:09:41 xen1 openais[2747]: [SYNC ] This node is within the > >> primary component and will provide service. > >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] CLM CONFIGURATION CHANGE > >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] New Configuration: > >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] r(0) ip(172.16.40.107) > >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Left: > >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] Members Joined: > >> Dec 19 23:09:41 xen1 openais[2747]: [SYNC ] This node is within the > >> primary component and will provide service. > >> Dec 19 23:09:41 xen1 openais[2747]: [TOTEM] entering OPERATIONAL state. > >> Dec 19 23:09:41 xen1 openais[2747]: [CLM ] got nodejoin message > >> 172.16.40.107 Dec 19 23:09:41 xen1 openais[2747]: [CPG ] got joinlist > >> message from node 1 Dec 19 23:09:44 xen1 ccsd[2740]: Cluster is not > >> quorate. Refusing connection. Dec 19 23:09:44 xen1 ccsd[2740]: Error > >> while processing connect: Connection refused > >> Dec 19 23:09:49 xen1 ccsd[2740]: Cluster is not quorate. Refusing > >> connection. Dec 19 23:09:49 xen1 ccsd[2740]: Error while processing > >> connect: Connection refused > >> Dec 19 23:09:54 xen1 ccsd[2740]: Cluster is not quorate. Refusing > >> connection. Dec 19 23:09:54 xen1 ccsd[2740]: Error while processing > >> connect: Connection refused > >> Dec 19 23:09:59 xen1 ccsd[2740]: Cluster is not quorate. Refusing > >> connection. Dec 19 23:09:59 xen1 ccsd[2740]: Error while processing > >> connect: Connection refused > >> Dec 19 23:10:04 xen1 ccsd[2740]: Cluster is not quorate. Refusing > >> connection. Dec 19 23:10:04 xen1 ccsd[2740]: Error while processing > >> connect: Connection refused > >> Dec 19 23:10:09 xen1 ccsd[2740]: Cluster is not quorate. Refusing > >> connection. Dec 19 23:10:09 xen1 ccsd[2740]: Error while processing > >> connect: Connection refused > >> > >> The last errors are ok because the cluster isn't quorate anymore. xen2 > >> was rebooting and xen3 was fenced, so leaving xen1 alone creates an > >> unquorate cluster... > >> > >> The unusual thing is that it only happens when one of the nodes is using > >> rhel5 xen kernel. Maybe something in the bridge-utils bug and multicast? > >> This problem happens if i reboot xen1 server with xen kernel or xen2 > >> server. > >> > >> > >> Any intel? > >> > >> Thanks > >> Nuno Fernandes > > Hi Nuno, > > Sorry for this question: how do you have created xen bridges with > ifcfg-files?? I am trying to do the same, but it doesn't works for me ... > > > ------------------------------------------------------------------------ > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster Create/Edit the files: :::::::::::::: File: /etc/sysconfig/network-scripts/ifcfg-eth0 :::::::::::::: DEVICE=eth0 ONBOOT=yes BRIDGE=xenbr0 HWADDR=XX:XX:XX:XX:XX:XX where XX:XX:XX:XX:XX:XX is the mac address of your network card. :::::::::::::: File: /etc/sysconfig/network-scripts/ifcfg-xenbr0 :::::::::::::: DEVICE=xenbr0 ONBOOT=yes BOOTPROTO=static IPADDR=172.16.40.116 NETMASK=255.255.255.0 GATEWAY=172.16.40.254 TYPE=Bridge DELAY=0 The ips are for my network. You have to change them to match yours. And then: service network restart Rgds Nuno Fernandes -- Nuno Pais Fernandes Cisco Certified Network Associate Oracle Certified Professional Eurotux Informatica S.A. Tel: +351 253257395 Fax: +351 253257396 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From rhurst at bidmc.harvard.edu Tue Apr 3 16:52:06 2007 From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu) Date: Tue, 3 Apr 2007 12:52:06 -0400 Subject: [Linux-cluster] Making a cloned clustered VG mountable on a backup media server? Message-ID: <1175619126.3437.23.camel@WSBID06223> I am having problems making a clustered VG visible on a backup media server. The originating RHEL cluster issues the CLARiiON cloning process, and it completes successfully, to another volume. That clone volume LUN is visible to the backup media server (outside of the cluster) only. Q1: What do I need to do on the backup media server's LVM to activate this newly-made VG? vgscan --mknodes is not helping any, but I do see some SCSI references, i.e., /dev/sdz to its 420GB sized disk. Is there a safe way to vgexport / vgimport the original VG map to this server? Q2: I loaded GFS software on the backup media server, so do I need to modify the VG's lvols superblock to a changed PROTO value of nolock to make it mountable for backup? Thanks. Robert Hurst, Sr. Cach? Administrator Beth Israel Deaconess Medical Center 1135 Tremont Street, REN-7 Boston, Massachusetts 02120-2140 617-754-8754 ? Fax: 617-754-8730 ? Cell: 401-787-3154 Any technology distinguishable from magic is insufficiently advanced. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2178 bytes Desc: not available URL: From Peter_Sopko at tempest.sk Wed Apr 4 12:17:32 2007 From: Peter_Sopko at tempest.sk (Peter Sopko) Date: Wed, 04 Apr 2007 14:17:32 +0200 Subject: [Linux-cluster] problem with deadlocked processes (D) Message-ID: <001601c776b3$39017990$ab046cb0$@sk> Hi, today a strange thing occurred - on both of our cluster nodes a lot of processes suddenly started to become locked in the D state (i/o lock). This thing has already happened once before (six months ago), but a simple reboot helped to solve this issue. But as it appeared again, I don't want to solve it this way again, I would like to find the reason why this is happening, but have no idea where to start. In /var/log/messages there is nothing unusual, the only thing is that some directories are unremoveable and a lot of processes locked. Some infos about our configuration : The storage array is an MSA 1000. There are two cluster nodes, that are connected with each other and both are connected to the storage array. The only use of the cluster is web/mail server - apache, courier-imap, postfix, spamassassin, clamav, mysql, ... The last time this thing happened only courier-imap processes (imapd, pop3d) were locked, today it is also apaches httpd. The current state is 261 processes out of 472 are locked in the D state. Here are some examples taken using ps afx (the XXXXX are just filtred e-mail addresses) : 3024 ? Ss 7:14 /usr/libexec/postfix/master 4544 ? S 0:17 \_ tlsmgr -l -t unix -u 15922 ? S 0:00 \_ proxymap -t unix -u 15943 ? D 0:00 \_ virtual -t unix 16129 ? D 0:00 \_ virtual -t unix 16153 ? D 0:00 \_ virtual -t unix 16261 ? S 0:00 \_ proxymap -t unix -u 16262 ? D 0:00 \_ virtual -t unix 16269 ? D 0:00 \_ virtual -t unix 16271 ? D 0:00 \_ virtual -t unix 16424 ? D 0:00 \_ virtual -t unix 19138 ? D 0:00 \_ virtual -t unix 19147 ? D 0:00 \_ virtual -t unix 19153 ? D 0:00 \_ virtual -t unix 19205 ? D 0:00 \_ virtual -t unix 19835 ? S 0:00 \_ pickup -l -t fifo -u 19919 ? S 0:00 \_ qmgr -l -t fifo -u 19920 ? S 0:00 \_ pipe -n filter -t unix flags=Rq user=filter argv=/data/spam/spamfilter.sh -f ${sender} --{recipient} 20346 ? Ss 0:00 | \_ /bin/sh /data/spam/spamfilter.sh -f XXXXX -- XXXXX 20353 ? S 0:00 | \_ cat 20354 ? D 0:00 | \_ /bin/sh /data/spam/spamfilter.sh -f XXXXX -- XXXXX 3039 ? Ss 0:03 /usr/sbin/httpd 15674 ? D 0:37 \_ /usr/sbin/httpd 15675 ? D 0:32 \_ /usr/sbin/httpd 15676 ? D 0:30 \_ /usr/sbin/httpd 15677 ? D 0:34 \_ /usr/sbin/httpd 15678 ? D 0:31 \_ /usr/sbin/httpd 15679 ? S 0:33 \_ /usr/sbin/httpd 15680 ? D 0:34 \_ /usr/sbin/httpd 15681 ? D 0:34 \_ /usr/sbin/httpd 30808 ? D 0:15 \_ /usr/sbin/httpd 30809 ? D 0:15 \_ /usr/sbin/httpd 30810 ? D 0:13 \_ /usr/sbin/httpd 30825 ? D 0:16 \_ /usr/sbin/httpd 30827 ? D 0:15 \_ /usr/sbin/httpd 30828 ? D 0:17 \_ /usr/sbin/httpd 30829 ? D 0:14 \_ /usr/sbin/httpd 30830 ? D 0:14 \_ /usr/sbin/httpd 30831 ? D 0:17 \_ /usr/sbin/httpd 30832 ? D 0:12 \_ /usr/sbin/httpd 30835 ? D 0:15 \_ /usr/sbin/httpd 30840 ? D 0:12 \_ /usr/sbin/httpd 20441 ? D 0:00 \_ /usr/sbin/httpd 20500 ? D 0:00 \_ /usr/sbin/httpd 20501 ? S 0:00 \_ /usr/sbin/httpd any idea where to start with debuging and looking for the reason this is happening ? I find it quit weird, that for more than 6 month it is ok a now in a sudden it starts doing this..... Any help would be greatly appreciated. Thanks Peter Sopko, IT Security Consultant Tempest a.s. Slovak Republic From breeves at redhat.com Wed Apr 4 12:44:38 2007 From: breeves at redhat.com (Bryn M. Reeves) Date: Wed, 04 Apr 2007 13:44:38 +0100 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <001601c776b3$39017990$ab046cb0$@sk> References: <001601c776b3$39017990$ab046cb0$@sk> Message-ID: <46139DB6.4070501@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Peter Sopko wrote: > Hi, > > today a strange thing occurred - on both of our cluster nodes a lot of > processes suddenly started to become locked in the D state (i/o lock). This > thing has already happened once before (six months ago), but a simple reboot > helped to solve this issue. But as it appeared again, I don't want to solve > it this way again, I would like to find the reason why this is happening, > but have no idea where to start. In /var/log/messages there is nothing > unusual, the only thing is that some directories are unremoveable and a lot > of processes locked. For problems where processes are getting stuck in D state it's usually helpful to get sysrq-t data to see where the threads are stuck. Grab two sets of data a few seconds apart so that you can see if things are really stuck or just making slow progress. You can also get some information from the wchan data exposed in /proc - it's easiest to view with ps: $ ps ax -ocomm,pid,state,wchan COMMAND PID S WCHAN vim 22322 S - bash 22471 S - man 22817 S wait sh 22820 S wait sh 22821 S wait less 22826 S - bash 22839 S wait screen 23435 S pause [...] Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFGE5226YSQoMYUY94RAgm0AKDdPg/mcTHilSwMpd6+Meno2zBLtACgt+/j TT3MsBrg6/gpdBdPDYMEp5Q= =ADyt -----END PGP SIGNATURE----- From Peter_Sopko at tempest.sk Wed Apr 4 13:18:18 2007 From: Peter_Sopko at tempest.sk (Peter Sopko) Date: Wed, 04 Apr 2007 15:18:18 +0200 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <46139DB6.4070501@redhat.com> References: <001601c776b3$39017990$ab046cb0$@sk> <46139DB6.4070501@redhat.com> Message-ID: <001e01c776bb$b60b1510$22213f30$@sk> Hi, thanks for your reply Bryn. The output of the ps command you suggested (i've ommited the standard system processes) : [root at mail1 subsys]# ps ax -ocomm,pid,state,wchan |more COMMAND PID S WCHAN ccsd 2258 S - cman_comms 2310 S cluster_kthread cman_serviced 2312 S serviced cman_memb 2311 S membership_kthread cman_hbeat 2315 S hello_kthread fenced 2336 S rt_sigsuspend dlm_astd 2354 S dlm_astd dlm_recvd 2355 S dlm_recvd dlm_sendd 2356 S dlm_sendd lock_dlm1 2358 S dlm_async lock_dlm2 2359 S dlm_async gfs_scand 2360 S - gfs_glockd 2361 S gfs_glockd gfs_recoverd 2362 S - gfs_logd 2363 S - gfs_quotad 2364 D glock_wait_internal gfs_inoded 2365 D dlm_lock_sync syslogd 2374 S - klogd 2394 S syslog heartbeat 2503 S - courierlogger 2526 S pipe_wait authdaemond 2527 S - authdaemond 2551 S - authdaemond 2552 S - authdaemond 2553 S - authdaemond 2554 S - authdaemond 2555 S - heartbeat 2586 S pipe_wait heartbeat 2587 S - heartbeat 2588 S - heartbeat 2589 S - heartbeat 2590 S - acpid 2595 S - ipfail 2608 S - nod32d 2609 S - nod32smtp 2618 S - sshd 2627 S - ntpd 2642 S - courierlogger 2654 S pipe_wait couriertcpd 2655 S - courierlogger 2661 S pipe_wait couriertcpd 2662 S - courierlogger 2667 S pipe_wait couriertcpd 2668 S wait courierlogger 2673 S pipe_wait couriertcpd 2674 S - master 2815 S - master 3024 S - httpd 3039 S - crond 3048 S - rhnsd 3067 S - mingetty 3074 S - mingetty 3075 S - mingetty 3076 S - mingetty 3077 S - mingetty 3078 S - mingetty 3079 S - ntpd 3888 S rt_sigsuspend tlsmgr 4544 S - tlsmgr 1585 S - anvil 1699 S - spamd 29941 S - httpd 15674 D glock_wait_internal httpd 15675 D glock_wait_internal httpd 15676 D glock_wait_internal httpd 15677 D glock_wait_internal httpd 15678 D glock_wait_internal httpd 15679 D glock_wait_internal httpd 15680 D glock_wait_internal httpd 15681 D glock_wait_internal httpd 30808 D glock_wait_internal httpd 30809 D glock_wait_internal httpd 30810 D glock_wait_internal httpd 30825 D glock_wait_internal httpd 30827 D glock_wait_internal httpd 30828 D glock_wait_internal httpd 30829 D glock_wait_internal httpd 30830 D glock_wait_internal httpd 30831 D glock_wait_internal httpd 30832 D glock_wait_internal httpd 30835 D glock_wait_internal httpd 30840 D glock_wait_internal spamd 17341 S - proxymap 24868 S - proxymap 27542 S - mysqld_safe 30617 S wait mysqld 30650 S - trivial-rewrite 30735 S - proxymap 30742 S - sshd 517 S - sshd 519 S - bash 520 S wait su 740 S wait bash 741 S - imapd 15018 D lock_on_glock virtual 15699 D lock_on_glock trivial-rewrite 15918 S - proxymap 15922 S - virtual 15943 D lock_on_glock virtual 15952 D lock_on_glock virtual 15965 D lock_on_glock pop3d 15966 D lock_on_glock pop3d 15967 D lock_on_glock virtual 15968 D lock_on_glock pop3d 15971 D lock_on_glock pop3d 15983 D lock_on_glock virtual 16046 D lock_on_glock pop3d 16049 D lock_on_glock pop3d 16053 D lock_on_glock pop3d 16068 D glock_wait_internal pop3d 16074 D glock_wait_internal virtual 16077 D lock_on_glock spamd 16112 S - virtual 16129 D lock_on_glock virtual 16133 D lock_on_glock pop3d 16143 D glock_wait_internal virtual 16153 D lock_on_glock virtual 16160 D glock_wait_internal virtual 16163 D lock_on_glock pop3d 16164 D glock_wait_internal virtual 16179 D lock_on_glock pop3d 16183 D glock_wait_internal pop3d 16186 D glock_wait_internal pop3d 16187 D glock_wait_internal virtual 16191 D lock_on_glock pop3d 16192 D lock_on_glock virtual 16194 D lock_on_glock pop3d 16202 D glock_wait_internal virtual 16207 D lock_on_glock virtual 16217 D lock_on_glock virtual 16222 D lock_on_glock .... smtp 21150 S - smtp 21162 S flock_lock_file_wait cleanup 21181 S flock_lock_file_wait smtpd 21213 S - spamfilter.sh 21224 S wait cat 21225 S pipe_wait spamfilter.sh 21226 D - spamfilter.sh 21229 S wait pipe 21230 S - cat 21231 S pipe_wait spamfilter.sh 21232 D - spamfilter.sh 21235 S wait cat 21236 S pipe_wait spamfilter.sh 21237 D - spamfilter.sh 21239 S wait spamfilter.sh 21240 S wait cat 21242 S pipe_wait spamfilter.sh 21243 D - virtual 21244 D lock_on_glock cat 21245 S pipe_wait spamfilter.sh 21246 D - spamfilter.sh 21249 S wait cat 21250 S pipe_wait spamfilter.sh 21251 D - spamfilter.sh 21252 S wait cat 21253 S pipe_wait spamfilter.sh 21254 D - spamfilter.sh 21257 S wait cat 21258 S pipe_wait spamfilter.sh 21259 D - spamfilter.sh 21261 S wait spamfilter.sh 21262 S wait spamfilter.sh 21263 S wait cat 21264 S pipe_wait spamfilter.sh 21265 D - spamfilter.sh 21267 D - cat 21268 S pipe_wait spamfilter.sh 21269 D - spamfilter.sh 21273 S wait ... etc.... The sysrq-t output is to be found on this url - http://www.backbone.sk/sysrq.tar. It's 400k in size, so I have chosen not to attach it as in here. There are two files in this .tar - one was taken 15:04 and the other one on 15:08. Again I will be very thankful for any help. Peter Sopko, IT Security Consultant Tempest a.s. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Bryn M. Reeves Sent: Wednesday, April 04, 2007 2:45 PM To: linux clustering Subject: Re: [Linux-cluster] problem with deadlocked processes (D) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Peter Sopko wrote: > Hi, > > today a strange thing occurred - on both of our cluster nodes a lot of > processes suddenly started to become locked in the D state (i/o lock). This > thing has already happened once before (six months ago), but a simple reboot > helped to solve this issue. But as it appeared again, I don't want to solve > it this way again, I would like to find the reason why this is happening, > but have no idea where to start. In /var/log/messages there is nothing > unusual, the only thing is that some directories are unremoveable and a lot > of processes locked. For problems where processes are getting stuck in D state it's usually helpful to get sysrq-t data to see where the threads are stuck. Grab two sets of data a few seconds apart so that you can see if things are really stuck or just making slow progress. You can also get some information from the wchan data exposed in /proc - it's easiest to view with ps: $ ps ax -ocomm,pid,state,wchan COMMAND PID S WCHAN vim 22322 S - bash 22471 S - man 22817 S wait sh 22820 S wait sh 22821 S wait less 22826 S - bash 22839 S wait screen 23435 S pause [...] Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFGE5226YSQoMYUY94RAgm0AKDdPg/mcTHilSwMpd6+Meno2zBLtACgt+/j TT3MsBrg6/gpdBdPDYMEp5Q= =ADyt -----END PGP SIGNATURE----- -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From lhh at redhat.com Wed Apr 4 13:38:39 2007 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 4 Apr 2007 09:38:39 -0400 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <001601c776b3$39017990$ab046cb0$@sk> References: <001601c776b3$39017990$ab046cb0$@sk> Message-ID: <20070404133839.GA25589@redhat.com> On Wed, Apr 04, 2007 at 02:17:32PM +0200, Peter Sopko wrote: > > any idea where to start with debuging and looking for the reason this is > happening ? I find it quit weird, that for more than 6 month it is ok a now > in a sudden it starts doing this..... sysreq-t output would be wonderful, along with the kernel-version you have, cman-kernel, dlm-kernel, and gfs-kernel versions :) -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Wed Apr 4 13:41:28 2007 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 4 Apr 2007 09:41:28 -0400 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <20070404133839.GA25589@redhat.com> References: <001601c776b3$39017990$ab046cb0$@sk> <20070404133839.GA25589@redhat.com> Message-ID: <20070404134128.GB25589@redhat.com> On Wed, Apr 04, 2007 at 09:38:39AM -0400, Lon Hohberger wrote: > On Wed, Apr 04, 2007 at 02:17:32PM +0200, Peter Sopko wrote: > > > > any idea where to start with debuging and looking for the reason this is > > happening ? I find it quit weird, that for more than 6 month it is ok a now > > in a sudden it starts doing this..... > > sysreq-t output would be wonderful, along with the kernel-version you > have, cman-kernel, dlm-kernel, and gfs-kernel versions :) echo 1 > /proc/sys/kernel/sysrq echo t > /proc/sysrq-trigger If logging is set up, syslog should capture it and put it in /var/log/messages -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From breeves at redhat.com Wed Apr 4 13:48:45 2007 From: breeves at redhat.com (Bryn M. Reeves) Date: Wed, 04 Apr 2007 14:48:45 +0100 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <20070404134128.GB25589@redhat.com> References: <001601c776b3$39017990$ab046cb0$@sk> <20070404133839.GA25589@redhat.com> <20070404134128.GB25589@redhat.com> Message-ID: <4613ACBD.1080806@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lon Hohberger wrote: >> sysreq-t output would be wonderful, along with the kernel-version you >> have, cman-kernel, dlm-kernel, and gfs-kernel versions :) > > echo 1 > /proc/sys/kernel/sysrq > echo t > /proc/sysrq-trigger > > If logging is set up, syslog should capture it and put it in > /var/log/messages > > -- Lon > Peter's already posted a tarball of sysrq-t data here: http://www.backbone.sk/sysrq.tar Peter, can you post the versions of the above packages you're using? Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFGE6y96YSQoMYUY94RAm3uAJ4hTJwepc7ayfGxX5DPx0GCTcOl1wCeMaN9 UOOFRlddBaAWFwSPSe/JA8o= =36K7 -----END PGP SIGNATURE----- From rhurst at bidmc.harvard.edu Wed Apr 4 13:49:52 2007 From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu) Date: Wed, 4 Apr 2007 09:49:52 -0400 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <20070404134128.GB25589@redhat.com> References: <001601c776b3$39017990$ab046cb0$@sk> <20070404133839.GA25589@redhat.com> <20070404134128.GB25589@redhat.com> Message-ID: <1175694592.5476.2.camel@WSBID06223> Are these two kernel sysctl parameters recommended to be enabled all the time, particularly with production GFS/clustered systems, or only for dev/test systems for debugging purposes? Thanks. On Wed, 2007-04-04 at 09:41 -0400, Lon Hohberger wrote: > On Wed, Apr 04, 2007 at 09:38:39AM -0400, Lon Hohberger wrote: > > On Wed, Apr 04, 2007 at 02:17:32PM +0200, Peter Sopko wrote: > > > > > > any idea where to start with debuging and looking for the reason this is > > > happening ? I find it quit weird, that for more than 6 month it is ok a now > > > in a sudden it starts doing this..... > > > > sysreq-t output would be wonderful, along with the kernel-version you > > have, cman-kernel, dlm-kernel, and gfs-kernel versions :) > > echo 1 > /proc/sys/kernel/sysrq > echo t > /proc/sysrq-trigger > > If logging is set up, syslog should capture it and put it in > /var/log/messages > > -- Lon > Robert Hurst, Sr. Cach? Administrator Beth Israel Deaconess Medical Center 1135 Tremont Street, REN-7 Boston, Massachusetts 02120-2140 617-754-8754 ? Fax: 617-754-8730 ? Cell: 401-787-3154 Any technology distinguishable from magic is insufficiently advanced. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2178 bytes Desc: not available URL: From teigland at redhat.com Wed Apr 4 13:56:27 2007 From: teigland at redhat.com (David Teigland) Date: Wed, 4 Apr 2007 08:56:27 -0500 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <001e01c776bb$b60b1510$22213f30$@sk> References: <001601c776b3$39017990$ab046cb0$@sk> <46139DB6.4070501@redhat.com> <001e01c776bb$b60b1510$22213f30$@sk> Message-ID: <20070404135627.GA27558@redhat.com> On Wed, Apr 04, 2007 at 03:18:18PM +0200, Peter Sopko wrote: > The output of the ps command you suggested (i've ommited the standard system > processes) : > > [root at mail1 subsys]# ps ax -ocomm,pid,state,wchan |more Did you happen to capture this on the other machine, too? Dave From lhh at redhat.com Wed Apr 4 13:54:55 2007 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 4 Apr 2007 09:54:55 -0400 Subject: [Linux-cluster] Problem in cluster with xen kernel In-Reply-To: <200704031502.48009.npf@eurotux.com> References: <200704031012.27717.npf@eurotux.com> <200704031112.33501.npf@eurotux.com> <4612313B.8020408@gmail.com> <200704031502.48009.npf@eurotux.com> Message-ID: <20070404135455.GC25589@redhat.com> On Tue, Apr 03, 2007 at 03:02:39PM +0100, Nuno Fernandes wrote: > > Hi Nuno, > > > > Sorry for this question: how do you have created xen bridges with > > ifcfg-files?? I am trying to do the same, but it doesn't works for me ... > > > > > ------------------------------------------------------------------------ > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > Create/Edit the files: > > :::::::::::::: > File: /etc/sysconfig/network-scripts/ifcfg-eth0 > :::::::::::::: > DEVICE=eth0 > ONBOOT=yes > BRIDGE=xenbr0 > HWADDR=XX:XX:XX:XX:XX:XX > > where XX:XX:XX:XX:XX:XX is the mac address of your network card. > > :::::::::::::: > File: /etc/sysconfig/network-scripts/ifcfg-xenbr0 > :::::::::::::: > > DEVICE=xenbr0 > ONBOOT=yes > BOOTPROTO=static > IPADDR=172.16.40.116 > NETMASK=255.255.255.0 > GATEWAY=172.16.40.254 > TYPE=Bridge > DELAY=0 > > The ips are for my network. You have to change them to match yours. > > And then: > > service network restart Nuno, A most excellent solution; we should add this to the FAQ. :) -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From lhh at redhat.com Wed Apr 4 13:57:05 2007 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 4 Apr 2007 09:57:05 -0400 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <4613ACBD.1080806@redhat.com> References: <001601c776b3$39017990$ab046cb0$@sk> <20070404133839.GA25589@redhat.com> <20070404134128.GB25589@redhat.com> <4613ACBD.1080806@redhat.com> Message-ID: <20070404135705.GE25589@redhat.com> On Wed, Apr 04, 2007 at 02:48:45PM +0100, Bryn M. Reeves wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Lon Hohberger wrote: > >> sysreq-t output would be wonderful, along with the kernel-version you > >> have, cman-kernel, dlm-kernel, and gfs-kernel versions :) > > > > echo 1 > /proc/sys/kernel/sysrq > > echo t > /proc/sysrq-trigger > > > > If logging is set up, syslog should capture it and put it in > > /var/log/messages > > > > -- Lon > > > > Peter's already posted a tarball of sysrq-t data here: > > http://www.backbone.sk/sysrq.tar Oh, I missed that. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From hlawatschek at atix.de Wed Apr 4 13:58:41 2007 From: hlawatschek at atix.de (Mark Hlawatschek) Date: Wed, 4 Apr 2007 15:58:41 +0200 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <001e01c776bb$b60b1510$22213f30$@sk> References: <001601c776b3$39017990$ab046cb0$@sk> <46139DB6.4070501@redhat.com> <001e01c776bb$b60b1510$22213f30$@sk> Message-ID: <200704041558.42233.hlawatschek@atix.de> Hi, I observed quite the same problem at some time. There's the bugzilla entry I opened: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=228916 Mark On Wednesday 04 April 2007 15:18:18 Peter Sopko wrote: > Hi, > > thanks for your reply Bryn. > > The output of the ps command you suggested (i've ommited the standard > system processes) : > > [root at mail1 subsys]# ps ax -ocomm,pid,state,wchan |more > COMMAND PID S WCHAN > ccsd 2258 S - > cman_comms 2310 S cluster_kthread > cman_serviced 2312 S serviced > cman_memb 2311 S membership_kthread > cman_hbeat 2315 S hello_kthread > fenced 2336 S rt_sigsuspend > dlm_astd 2354 S dlm_astd > dlm_recvd 2355 S dlm_recvd > dlm_sendd 2356 S dlm_sendd > lock_dlm1 2358 S dlm_async > lock_dlm2 2359 S dlm_async > gfs_scand 2360 S - > gfs_glockd 2361 S gfs_glockd > gfs_recoverd 2362 S - > gfs_logd 2363 S - > gfs_quotad 2364 D glock_wait_internal > gfs_inoded 2365 D dlm_lock_sync > syslogd 2374 S - > klogd 2394 S syslog > heartbeat 2503 S - > courierlogger 2526 S pipe_wait > authdaemond 2527 S - > authdaemond 2551 S - > authdaemond 2552 S - > authdaemond 2553 S - > authdaemond 2554 S - > authdaemond 2555 S - > heartbeat 2586 S pipe_wait > heartbeat 2587 S - > heartbeat 2588 S - > heartbeat 2589 S - > heartbeat 2590 S - > acpid 2595 S - > ipfail 2608 S - > nod32d 2609 S - > nod32smtp 2618 S - > sshd 2627 S - > ntpd 2642 S - > courierlogger 2654 S pipe_wait > couriertcpd 2655 S - > courierlogger 2661 S pipe_wait > couriertcpd 2662 S - > courierlogger 2667 S pipe_wait > couriertcpd 2668 S wait > courierlogger 2673 S pipe_wait > couriertcpd 2674 S - > master 2815 S - > master 3024 S - > httpd 3039 S - > crond 3048 S - > rhnsd 3067 S - > mingetty 3074 S - > mingetty 3075 S - > mingetty 3076 S - > mingetty 3077 S - > mingetty 3078 S - > mingetty 3079 S - > ntpd 3888 S rt_sigsuspend > tlsmgr 4544 S - > tlsmgr 1585 S - > anvil 1699 S - > spamd 29941 S - > httpd 15674 D glock_wait_internal > httpd 15675 D glock_wait_internal > httpd 15676 D glock_wait_internal > httpd 15677 D glock_wait_internal > httpd 15678 D glock_wait_internal > httpd 15679 D glock_wait_internal > httpd 15680 D glock_wait_internal > httpd 15681 D glock_wait_internal > httpd 30808 D glock_wait_internal > httpd 30809 D glock_wait_internal > httpd 30810 D glock_wait_internal > httpd 30825 D glock_wait_internal > httpd 30827 D glock_wait_internal > httpd 30828 D glock_wait_internal > httpd 30829 D glock_wait_internal > httpd 30830 D glock_wait_internal > httpd 30831 D glock_wait_internal > httpd 30832 D glock_wait_internal > httpd 30835 D glock_wait_internal > httpd 30840 D glock_wait_internal > spamd 17341 S - > proxymap 24868 S - > proxymap 27542 S - > mysqld_safe 30617 S wait > mysqld 30650 S - > trivial-rewrite 30735 S - > proxymap 30742 S - > sshd 517 S - > sshd 519 S - > bash 520 S wait > su 740 S wait > bash 741 S - > imapd 15018 D lock_on_glock > virtual 15699 D lock_on_glock > trivial-rewrite 15918 S - > proxymap 15922 S - > virtual 15943 D lock_on_glock > virtual 15952 D lock_on_glock > virtual 15965 D lock_on_glock > pop3d 15966 D lock_on_glock > pop3d 15967 D lock_on_glock > virtual 15968 D lock_on_glock > pop3d 15971 D lock_on_glock > pop3d 15983 D lock_on_glock > virtual 16046 D lock_on_glock > pop3d 16049 D lock_on_glock > pop3d 16053 D lock_on_glock > pop3d 16068 D glock_wait_internal > pop3d 16074 D glock_wait_internal > virtual 16077 D lock_on_glock > spamd 16112 S - > virtual 16129 D lock_on_glock > virtual 16133 D lock_on_glock > pop3d 16143 D glock_wait_internal > virtual 16153 D lock_on_glock > virtual 16160 D glock_wait_internal > virtual 16163 D lock_on_glock > pop3d 16164 D glock_wait_internal > virtual 16179 D lock_on_glock > pop3d 16183 D glock_wait_internal > pop3d 16186 D glock_wait_internal > pop3d 16187 D glock_wait_internal > virtual 16191 D lock_on_glock > pop3d 16192 D lock_on_glock > virtual 16194 D lock_on_glock > pop3d 16202 D glock_wait_internal > virtual 16207 D lock_on_glock > virtual 16217 D lock_on_glock > virtual 16222 D lock_on_glock > .... > smtp 21150 S - > smtp 21162 S flock_lock_file_wait > cleanup 21181 S flock_lock_file_wait > smtpd 21213 S - > spamfilter.sh 21224 S wait > cat 21225 S pipe_wait > spamfilter.sh 21226 D - > spamfilter.sh 21229 S wait > pipe 21230 S - > cat 21231 S pipe_wait > spamfilter.sh 21232 D - > spamfilter.sh 21235 S wait > cat 21236 S pipe_wait > spamfilter.sh 21237 D - > spamfilter.sh 21239 S wait > spamfilter.sh 21240 S wait > cat 21242 S pipe_wait > spamfilter.sh 21243 D - > virtual 21244 D lock_on_glock > cat 21245 S pipe_wait > spamfilter.sh 21246 D - > spamfilter.sh 21249 S wait > cat 21250 S pipe_wait > spamfilter.sh 21251 D - > spamfilter.sh 21252 S wait > cat 21253 S pipe_wait > spamfilter.sh 21254 D - > spamfilter.sh 21257 S wait > cat 21258 S pipe_wait > spamfilter.sh 21259 D - > spamfilter.sh 21261 S wait > spamfilter.sh 21262 S wait > spamfilter.sh 21263 S wait > cat 21264 S pipe_wait > spamfilter.sh 21265 D - > spamfilter.sh 21267 D - > cat 21268 S pipe_wait > spamfilter.sh 21269 D - > spamfilter.sh 21273 S wait > ... > etc.... > > > The sysrq-t output is to be found on this url - > http://www.backbone.sk/sysrq.tar. It's 400k in size, so I have chosen not > to attach it as in here. There are two files in this .tar - one was taken > 15:04 and the other one on 15:08. > > Again I will be very thankful for any help. > > Peter Sopko, IT Security Consultant > Tempest a.s. > > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Bryn M. Reeves > Sent: Wednesday, April 04, 2007 2:45 PM > To: linux clustering > Subject: Re: [Linux-cluster] problem with deadlocked processes (D) > > Peter Sopko wrote: > > Hi, > > > > today a strange thing occurred - on both of our cluster nodes a lot of > > processes suddenly started to become locked in the D state (i/o lock). > > This > > > thing has already happened once before (six months ago), but a simple > > reboot > > > helped to solve this issue. But as it appeared again, I don't want to > > solve > > > it this way again, I would like to find the reason why this is happening, > > but have no idea where to start. In /var/log/messages there is nothing > > unusual, the only thing is that some directories are unremoveable and a > > lot > > > of processes locked. > > For problems where processes are getting stuck in D state it's usually > helpful to get sysrq-t data to see where the threads are stuck. Grab two > sets of data a few seconds apart so that you can see if things are > really stuck or just making slow progress. > > You can also get some information from the wchan data exposed in /proc - > it's easiest to view with ps: > > $ ps ax -ocomm,pid,state,wchan > COMMAND PID S WCHAN > vim 22322 S - > bash 22471 S - > man 22817 S wait > sh 22820 S wait > sh 22821 S wait > less 22826 S - > bash 22839 S wait > screen 23435 S pause > [...] > > Regards, > Bryn. > > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Gruss / Regards, Dipl.-Ing. Mark Hlawatschek http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From lhh at redhat.com Wed Apr 4 13:59:20 2007 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 4 Apr 2007 09:59:20 -0400 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <1175694592.5476.2.camel@WSBID06223> References: <001601c776b3$39017990$ab046cb0$@sk> <20070404133839.GA25589@redhat.com> <20070404134128.GB25589@redhat.com> <1175694592.5476.2.camel@WSBID06223> Message-ID: <20070404135918.GF25589@redhat.com> On Wed, Apr 04, 2007 at 09:49:52AM -0400, rhurst at bidmc.harvard.edu wrote: > Are these two kernel sysctl parameters recommended to be enabled all the > time, particularly with production GFS/clustered systems, or only for > dev/test systems for debugging purposes? Thanks. > > echo 1 > /proc/sys/kernel/sysrq * This enables sysrq > > echo t > /proc/sysrq-trigger * This is the same as pressing Alt+SysRq+t at the console, but can be done from a remote session Sometimes it's helpful to leave sysrq enabled, but it's really up to you. I usually just enable it when I need to. If you want to permanently enable it (which shouldn't hurt anything at all), edit /etc/sysctl.conf -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From breeves at redhat.com Wed Apr 4 14:02:28 2007 From: breeves at redhat.com (Bryn M. Reeves) Date: Wed, 04 Apr 2007 15:02:28 +0100 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <1175694592.5476.2.camel@WSBID06223> References: <001601c776b3$39017990$ab046cb0$@sk> <20070404133839.GA25589@redhat.com> <20070404134128.GB25589@redhat.com> <1175694592.5476.2.camel@WSBID06223> Message-ID: <4613AFF4.50103@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 rhurst at bidmc.harvard.edu wrote: > Are these two kernel sysctl parameters recommended to be enabled all the > time, particularly with production GFS/clustered systems, or only for > dev/test systems for debugging purposes? Thanks. > Only the first of these is a kernel parameter (kernel.sysrq). It enables keyboard sysrqs (holding down alt-sysrq-$KEY to trigger a sysrq command). This is off by default as it means anyone with console access can do "bad stuff" (reboot, DoS with thread dumps, SAKs etc). If physical access is sufficiently secure you can leave them on all the time - I do this on all my lab/test boxes. For production it all depends on your situation. The second is just a trigger (/proc/sysrq-trigger). Echoing a char into there just triggers the corresponding sysrq action once. This does not need /proc/sys/kernel/sysrq enabled since only root has write access to the file. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFGE6/06YSQoMYUY94RAviUAJ4pRO6UVaZCujXSzZ0zIrbqV1qcNgCgsaUM /Q8U7J65Bt4pMxnxGVD9pbc= =j198 -----END PGP SIGNATURE----- From lhh at redhat.com Wed Apr 4 14:04:25 2007 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 4 Apr 2007 10:04:25 -0400 Subject: [Linux-cluster] Updated test packages (cman-kernel/qdisk) Message-ID: <20070404140425.GG25589@redhat.com> For those of you running my qdisk test packages on RHEL4U4, I've rebuilt cman-kernel for the 42.0.10 kernel. Also, if you are not running the 0.4.1qdisk package, please upgrade. There is code which resolves multi-master (should the condition ever be reached) now, as well as fully fixes the original cause of the multi-master condition in bugzilla 220211. http://people.redhat.com/lhh/packages.html -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From beres.laszlo at sys-admin.hu Wed Apr 4 14:04:40 2007 From: beres.laszlo at sys-admin.hu (BERES Laszlo) Date: Wed, 04 Apr 2007 16:04:40 +0200 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <1175694592.5476.2.camel@WSBID06223> References: <001601c776b3$39017990$ab046cb0$@sk> <20070404133839.GA25589@redhat.com> <20070404134128.GB25589@redhat.com> <1175694592.5476.2.camel@WSBID06223> Message-ID: <4613B078.4030501@sys-admin.hu> rhurst at bidmc.harvard.edu wrote: > Are these two kernel sysctl parameters recommended to be enabled all the > time, particularly with production GFS/clustered systems, or only for > dev/test systems for debugging purposes? Thanks. Basically enabling the sysrq facility IMHO decreases the local (console) security, because anybody can trigger a full reboot via the keyboard. But if you maintain high console security (restricted server area, etc) then it's recommended, -- B?RES L?szl? RHCE, RHCX senior IT engineer, trainer From redhat at watson-wilson.ca Wed Apr 4 14:06:36 2007 From: redhat at watson-wilson.ca (Neil Watson) Date: Wed, 4 Apr 2007 10:06:36 -0400 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <4613B078.4030501@sys-admin.hu> References: <001601c776b3$39017990$ab046cb0$@sk> <20070404133839.GA25589@redhat.com> <20070404134128.GB25589@redhat.com> <1175694592.5476.2.camel@WSBID06223> <4613B078.4030501@sys-admin.hu> Message-ID: <20070404140636.GA18652@watson-wilson.ca> On Wed, Apr 04, 2007 at 04:04:40PM +0200, BERES Laszlo wrote: >Basically enabling the sysrq facility IMHO decreases the local (console) >security, because anybody can trigger a full reboot via the keyboard. >But if you maintain high console security (restricted server area, etc) >then it's recommended, Red Hat enables ctrl-alt-del reboots by default (grumble). -- Neil Watson | Debian Linux System Administrator | Uptime 36 days http://watson-wilson.ca From beres.laszlo at sys-admin.hu Wed Apr 4 14:09:28 2007 From: beres.laszlo at sys-admin.hu (BERES Laszlo) Date: Wed, 04 Apr 2007 16:09:28 +0200 Subject: [Linux-cluster] problem with deadlocked processes (D) In-Reply-To: <20070404140636.GA18652@watson-wilson.ca> References: <001601c776b3$39017990$ab046cb0$@sk> <20070404133839.GA25589@redhat.com> <20070404134128.GB25589@redhat.com> <1175694592.5476.2.camel@WSBID06223> <4613B078.4030501@sys-admin.hu> <20070404140636.GA18652@watson-wilson.ca> Message-ID: <4613B198.10306@sys-admin.hu> Neil Watson wrote: > Red Hat enables ctrl-alt-del reboots by default (grumble). It does a regular reboot. But sysrq-b reboots without syncing, umounting, etc. (BTW, you're right.) -- B?RES L?szl? RHCE, RHCX senior IT engineer, trainer From rhurst at bidmc.harvard.edu Wed Apr 4 14:31:46 2007 From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu) Date: Wed, 4 Apr 2007 10:31:46 -0400 Subject: [Linux-cluster] Making a cloned clustered VG mountable on abackup media server? In-Reply-To: <1175619126.3437.23.camel@WSBID06223> References: <1175619126.3437.23.camel@WSBID06223> Message-ID: <1175697106.5476.16.camel@WSBID06223> I believe I found my answer to question #1, although I would welcome any suggestions on improving this process: First, scp /etc/lvm/backup/VGWATSON (the origin VG that was cloned) to the backup media server (intermezzo). Also get its PV uuid: # pvscan -u | grep VGWATSON PV /dev/sdae1 with UUID ZRcfam-06Ay-Ddby-f9lM-Pdkt-Pzqn-f0AB2T VG VGWATSON lvm2 [420.00 GB / 92.50 GB free] Then execute the following on the backup media server (intermezzo): # parted /dev/sdz set 1 lvm on This activates its partition table and /dev/sdz1 becomes available. I will need to add to my script the whereabouts of this device, as it may 'walk' from a reboot, i.e., use powermt display dev=all and search for its CLARiiON ID. # pvcreate --restorefile ./VGWATSON --uuid ZRcfam-06Ay-Ddby-f9lM-Pdkt-Pzqn-f0AB2T /dev/sdz1 # vgrename VGWATSON VGWATSON-CLONE # vgchange -a y /dev/VGWATSON-CLONE On Tue, 2007-04-03 at 12:52 -0400, rhurst at bidmc.harvard.edu wrote: > I am having problems making a clustered VG visible on a backup media > server. > > The originating RHEL cluster issues the CLARiiON cloning process, and > it completes successfully, to another volume. That clone volume LUN > is visible to the backup media server (outside of the cluster) only. > > Q1: What do I need to do on the backup media server's LVM to activate > this newly-made VG? vgscan --mknodes is not helping any, but I do see > some SCSI references, i.e., /dev/sdz to its 420GB sized disk. Is > there a safe way to vgexport / vgimport the original VG map to this > server? > > Q2: I loaded GFS software on the backup media server, so do I need to > modify the VG's lvols superblock to a changed PROTO value of nolock to > make it mountable for backup? > > Thanks. Robert Hurst, Sr. Cach? Administrator Beth Israel Deaconess Medical Center 1135 Tremont Street, REN-7 Boston, Massachusetts 02120-2140 617-754-8754 ? Fax: 617-754-8730 ? Cell: 401-787-3154 Any technology distinguishable from magic is insufficiently advanced. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2178 bytes Desc: not available URL: From djo at dotis.us Wed Apr 4 20:40:43 2007 From: djo at dotis.us (David J. Otis) Date: Wed, 4 Apr 2007 16:40:43 -0400 Subject: [Linux-cluster] multiple clusters on same net Message-ID: <000c01c776f9$84549b20$87dea8c0@cabcds.org> Is it possible to have the cluster communications happen on an internal network so it will be contained? now eht0 = outside net eth1 = san eth2 = fence could cluster communication be configured to happen on eht1? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpeterso at redhat.com Wed Apr 4 21:40:56 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Wed, 04 Apr 2007 16:40:56 -0500 Subject: [Linux-cluster] multiple clusters on same net In-Reply-To: <000c01c776f9$84549b20$87dea8c0@cabcds.org> References: <000c01c776f9$84549b20$87dea8c0@cabcds.org> Message-ID: <46141B68.6010604@redhat.com> David J. Otis wrote: > Is it possible to have the cluster communications happen on an internal network so it will be contained? > > now > eht0 = outside net > eth1 = san > eth2 = fence > > could cluster communication be configured to happen on eht1? Hi David, Yes. http://sources.redhat.com/cluster/faq.html#cman_heartbeat_nic Regards, Bob Peterson Red Hat Cluster Suite From rhurst at bidmc.harvard.edu Thu Apr 5 11:38:59 2007 From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu) Date: Thu, 5 Apr 2007 07:38:59 -0400 Subject: [Linux-cluster] multiple clusters on same net In-Reply-To: <46141B68.6010604@redhat.com> References: <000c01c776f9$84549b20$87dea8c0@cabcds.org> <46141B68.6010604@redhat.com> Message-ID: <1175773139.3384.13.camel@WSBID06223> > Yes. > http://sources.redhat.com/cluster/faq.html#cman_heartbeat_nic Is that FAQ entry still valid that you cannot override cman options in the /etc/init.d/cman script? I thought it would use the file /etc/sysconfig/cluster for such a thing, i.e., cman_join_opts="-v 5" Robert Hurst, Sr. Cach? Administrator Beth Israel Deaconess Medical Center 1135 Tremont Street, REN-7 Boston, Massachusetts 02120-2140 617-754-8754 ? Fax: 617-754-8730 ? Cell: 401-787-3154 Any technology distinguishable from magic is insufficiently advanced. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2178 bytes Desc: not available URL: From bfilipek at crscold.com Thu Apr 5 15:35:19 2007 From: bfilipek at crscold.com (Brad Filipek) Date: Thu, 5 Apr 2007 10:35:19 -0500 Subject: [Linux-cluster] Active/Passive - GFS needed? Message-ID: <9C01E18EF3BC2448A3B1A4812EB87D024D95DB@SRVEDI.upark.crscold.com> I will have a two node cluster connecting to a fibre channel SAN (AX150). There is only one application on the server that our users will access by using telnet or SSH with a terminal emulator. If I setup this two node cluster up as active/passive, is there a need for GFS? Brad Filipek Network Administrator Continental Refrigerated Services, LLC. P- (708) 587-3640 F- (708) 534-4699 bfilipek at crscold.com http://www.crscold.com Confidentiality Notice: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by email reply or by telephone and immediately delete this message and any attachments. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbrassow at redhat.com Thu Apr 5 16:18:46 2007 From: jbrassow at redhat.com (Jonathan Brassow) Date: Thu, 5 Apr 2007 11:18:46 -0500 Subject: [Linux-cluster] Active/Passive - GFS needed? In-Reply-To: <9C01E18EF3BC2448A3B1A4812EB87D024D95DB@SRVEDI.upark.crscold.com> References: <9C01E18EF3BC2448A3B1A4812EB87D024D95DB@SRVEDI.upark.crscold.com> Message-ID: <6570F544-49A7-487F-9FAC-FFE8668D9139@redhat.com> not really. You could use cluster suite (rgmanager) for failover and use ext3. brassow On Apr 5, 2007, at 10:35 AM, Brad Filipek wrote: > I will have a two node cluster connecting to a fibre channel SAN > (AX150). There is only one application on the server that our users > will access by using telnet or SSH with a terminal emulator. If I > setup this two node cluster up as active/passive, is there a need > for GFS? > > > > > > Brad Filipek > Network Administrator > Continental Refrigerated Services, LLC. > P- (708) 587-3640 > F- (708) 534-4699 > bfilipek at crscold.com > http://www.crscold.com > > > > > > Confidentiality Notice: This message is intended for the use of the > individual or entity to which it is addressed and may contain > information that is privileged, confidential and exempt from > disclosure under applicable law. If the reader of this message is > not the intended recipient or the employee or agent responsible for > delivering this message to the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. > > If you have received this communication in error, please notify us > immediately by email reply or by telephone and immediately delete > this message and any attachments. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From Lin_ChiiShing at emc.com Thu Apr 5 17:35:51 2007 From: Lin_ChiiShing at emc.com (Lin_ChiiShing at emc.com) Date: Thu, 5 Apr 2007 13:35:51 -0400 Subject: [Linux-cluster] Different sub-network Message-ID: <97982DAF1EA1494DB6C65C2EEAA6296105B5344A@CORPUSMX40B.corp.emc.com> In a business continuance configuration, nodes in a cluster will be running on different networks. Is it possible to configure a cluster with nodes running on different networks ? Thanks, -Mark From c_triantafillou at hotmail.com Thu Apr 5 19:37:12 2007 From: c_triantafillou at hotmail.com (Christos Triantafillou) Date: Thu, 05 Apr 2007 21:37:12 +0200 Subject: [Linux-cluster] Starting with DLM Message-ID: Hello, What are the necessary steps to configure DLM on RHEL4 and use it from an application? Regards, Christos Triantafillou _________________________________________________________________ Don't just search. Find. Check out the new MSN Search! http://search.msn.com/ From srramasw at cisco.com Thu Apr 5 21:19:11 2007 From: srramasw at cisco.com (Sridhar Ramaswamy (srramasw)) Date: Thu, 5 Apr 2007 14:19:11 -0700 Subject: [Linux-cluster] GFS2 hangs with dlm messages In-Reply-To: Message-ID: FYI. DLM patch mentioned below solved the problem. David - Thanks again! Think I should follow one of your earlier advise to grab the latest gfs2/dlm sources often. - Sridhar > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of > Sridhar Ramaswamy (srramasw) > Sent: Friday, March 30, 2007 11:55 AM > To: David Teigland > Cc: linux clustering > Subject: RE: [Linux-cluster] GFS2 hangs with dlm messages > > Thanks David. I'll try the DLM patch. > > I guess 'cman_tool nodes' shows things are okay over there, > > [root at cfs1 cluster-2.00.00]# cman_tool nodes > Node Sts Inc Joined Name > 1 M 4 2007-03-29 15:15:01 cfs1 > 2 M 8 2007-03-29 15:15:03 cfs5 > > Thanks, > Sridhar > > > -----Original Message----- > > From: David Teigland [mailto:teigland at redhat.com] > > Sent: Friday, March 30, 2007 11:07 AM > > To: Sridhar Ramaswamy (srramasw) > > Cc: linux clustering > > Subject: Re: [Linux-cluster] GFS2 hangs with dlm messages > > > > On Fri, Mar 30, 2007 at 10:46:34AM -0700, Sridhar Ramaswamy > > (srramasw) wrote: > > > Mar 30 09:58:34 cfs1 kernel: dlm: message size 5457 from 2 > > too big, buf > > > len 4632 > > > > I'm not sure, but this patch might be the fix for the message sizes. > > > > http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/steve/gfs2 > > -2.6-nmw.git;a=commitdiff;h=74a3949ec42455f400a5bb4b5c9edaeda1 > > 92b42e;hp=ed65d5376177cc04de39d6410a5c2d2d0883f616 > > > > > see tons of activity in 'aisexec' process with lots of > > sendmsg/recvmsg > > > > 'cman_tool nodes' and /var/log/messages should give some idea about > > cluster membership problems. > > > > Dave > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From benjhenrion.cluster at gmail.com Fri Apr 6 10:33:20 2007 From: benjhenrion.cluster at gmail.com (ben henrion) Date: Fri, 6 Apr 2007 12:33:20 +0200 Subject: [Linux-cluster] Active/Passive - GFS needed? In-Reply-To: <6570F544-49A7-487F-9FAC-FFE8668D9139@redhat.com> References: <9C01E18EF3BC2448A3B1A4812EB87D024D95DB@SRVEDI.upark.crscold.com> <6570F544-49A7-487F-9FAC-FFE8668D9139@redhat.com> Message-ID: <72f4f1c50704060333w24c92923n1caf1cd5e2814605@mail.gmail.com> No at all .... GFS is able to support FOS and multi write/read in the same FS. If you needn't use ext3. 2007/4/5, Jonathan Brassow : > > not really. You could use cluster suite (rgmanager) for failover and use > ext3. > brassow > > On Apr 5, 2007, at 10:35 AM, Brad Filipek wrote: > > I will have a two node cluster connecting to a fibre channel SAN (AX150). > There is only one application on the server that our users will access by > using telnet or SSH with a terminal emulator. If I setup this two node > cluster up as active/passive, is there a need for GFS? > > > > > > Brad Filipek > Network Administrator > Continental Refrigerated Services, LLC. > P- (708) 587-3640 > F- (708) 534-4699 > bfilipek at crscold.com > http://www.crscold.com > > > > > > *Confidentiality Notice: *This message is intended for the use of the > individual or entity to which it is addressed and may contain information > that is privileged, confidential and exempt from disclosure under applicable > law. If the reader of this message is not the intended recipient or the > employee or agent responsible for delivering this message to the intended > recipient, you are hereby notified that any dissemination, distribution or > copying of this communication is strictly prohibited. > > If you have received this communication in error, please notify us > immediately by email reply or by telephone and immediately delete this > message and any attachments. > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From djo at dotis.us Fri Apr 6 13:06:55 2007 From: djo at dotis.us (David J. Otis) Date: Fri, 6 Apr 2007 09:06:55 -0400 Subject: [Linux-cluster] 2 node GFS with DLM Message-ID: <002b01c7784c$73891600$87dea8c0@cabcds.org> I have a 2 node GFS with DLM is there a way to allow one node to come without the other on initial boot? -------------- next part -------------- An HTML attachment was scrubbed... URL: From basv at sara.nl Sat Apr 7 09:47:15 2007 From: basv at sara.nl (Bas van der Vlies) Date: Sat, 7 Apr 2007 11:47:15 +0200 Subject: [Linux-cluster] mvapich Message-ID: <5B06982C-B57A-420B-8F26-E3C0AF1DCB96@sara.nl> Willem, Ben wat aan het spelen met BLACS/SCALAPACK. Wij gebruiken voor BLACS -O2 voor de c-compilers heb ik nu de default gelaten en dat is - O4 en dan werkt de xcbrd test wel ;-) ( :-( ) Dit is voor gnu-mpich1-openib-beta Ga nog wat meer testen! Na het weekend -- Bas van der Vlies basv at sara.nl From teigland at redhat.com Mon Apr 9 13:35:09 2007 From: teigland at redhat.com (David Teigland) Date: Mon, 9 Apr 2007 08:35:09 -0500 Subject: [Linux-cluster] 2 node GFS with DLM In-Reply-To: <002b01c7784c$73891600$87dea8c0@cabcds.org> References: <002b01c7784c$73891600$87dea8c0@cabcds.org> Message-ID: <20070409133508.GA28404@redhat.com> On Fri, Apr 06, 2007 at 09:06:55AM -0400, David J. Otis wrote: > I have a 2 node GFS with DLM is there a way to allow one node to come > without the other on initial boot? See the part about the "two_node" option: http://sources.redhat.com/cluster/doc/usage.txt Dave From rpeterso at redhat.com Mon Apr 9 13:39:28 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Mon, 09 Apr 2007 08:39:28 -0500 Subject: [Linux-cluster] configuring GFS2 In-Reply-To: References: Message-ID: <461A4210.8000204@redhat.com> Hi Nathan, I don't remember if anyone replied yet, so I apologize if this is redundant. Nathan J Dragun wrote: > 1. What -exactly- is the process necessary for setting up gfs2. I'm confused because there are so many 'pieces' to this, I feel like I've lost my way? > - To give a little background, I'm using an iSCSI target which allows any machine to make the SAN device a local scsi device on the system. Ex: /dev/sdb (This in turn allows direct block access, and I can do whatever with it; as if it were literally a system disk.) I've read something about needing to set up fencing, cluster information, and .... ?? I'm lost. lol > - I set up a partition with dlm compiled into the kernel (2.6.20.1), and apparently I can only mount when using no_lock, dlm gives me a resource error (I'm assuming this has to do with no cluster information being set, or anything else configured). Perhaps you should take a look at my "Unofficial NFS/GFS Cookbook" that goes through every step: http://sources.redhat.com/cluster/doc/nfscookbook.pdf > 2. Is there a way to grow a gfs2 partition at the moment? I see something for gfs1, yet no binaries, source, or references to a command except via this web page which outlines the command: http://www.die.net/doc/linux/man/man8/gfs2_grow.8.html My bad. This was an oversight. I'm putting the final touches on a new gfs2_grow command right now. I'm tracking it here: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=234844 > 3. Quota. Fortunately this is one of the last things I have to worry about on the chain of events that need to unfold. I have not done much research on this last part as a result. Does gfs's quota work like the quota system that .... well, I don't know if you want to call it a default, or generic, quota system .. or what? I think the GFS quota stuff is pretty much the same as any, but like you, I haven't dug into it enough to know for sure. Regards, Bob Peterson Red Hat Cluster Suite From teigland at redhat.com Mon Apr 9 13:53:47 2007 From: teigland at redhat.com (David Teigland) Date: Mon, 9 Apr 2007 08:53:47 -0500 Subject: [Linux-cluster] configuring GFS2 In-Reply-To: <461A4210.8000204@redhat.com> References: <461A4210.8000204@redhat.com> Message-ID: <20070409135347.GC28404@redhat.com> On Mon, Apr 09, 2007 at 08:39:28AM -0500, Robert Peterson wrote: > >3. Quota. Fortunately this is one of the last things I have to worry > >about on the chain of events that need to unfold. I have not done much > >research on this last part as a result. Does gfs's quota work like the > >quota system that .... well, I don't know if you want to call it a > >default, or generic, quota system .. or what? > > I think the GFS quota stuff is pretty much the same as any, but like you, I > haven't dug into it enough to know for sure. The notable thing about GFS quotas is that they are not done like quotas on other linux fs's. GFS has it's own, internal, quota file and its own command (gfs_quota) for managing them. Dave From scott.mcclanahan at trnswrks.com Mon Apr 9 14:24:06 2007 From: scott.mcclanahan at trnswrks.com (Scott McClanahan) Date: Mon, 09 Apr 2007 10:24:06 -0400 Subject: [Linux-cluster] delete node Message-ID: <1176128646.4694.0.camel@jarjar.trnswrks.com> Hello, we are running CentOS 4.3 and the latest cluster suite packages from the csgfs yum repository for this release and need to delete a cluster member. According to the documentation we need to restart all cluster related services on all remaining nodes in the cluster after the node has been removed. This is a four node cluster so removing one node obviously degrades the cluster to three nodes, but we can have the remaining cluster members recalculate the number of votes to remain quorate. It still surprises me that we have to bring down the entire cluster just because we are deleting a node. Any feedback as to why this might be? From diggercheer at gmail.com Mon Apr 9 17:22:26 2007 From: diggercheer at gmail.com (David M) Date: Mon, 9 Apr 2007 12:22:26 -0500 Subject: [Linux-cluster] rgmanager or clustat problem Message-ID: I am running a four node GFS cluster with about 20 services per node. All four nodes belong to the same failover domain, and they each have a priority of 1. My shared storage is an iSCSI SAN. After rgmanager has been running for a couple of days, clustat produces the following result on all four nodes: Timed out waiting for a response from Resource Group Manager Member Status: Quorate Member Name Status ------ ---- ------ node01 Online, rgmanager node02 Online, Local, rgmanager node03 Online, rgmanager node04 Online, rgmanager I also get a time out when I try to determine the status of a particular service with "clustat -s servicename". All of the services seem to be up and running, but clustat does not work. Is there something wrong? Is there a way for me to increase the time out? clurgmgrd and dlm_recvd seem to be using a lot of CPU cycles on Node02, 40 and 60 percent, respectively. Thank you for your help. cman_tool services: NODE01: Service Name GID LID State Code Fence Domain: "default" 4 2 run - [1 3 2 4] DLM Lock Space: "clvmd" 1 3 run - [1 3 2 4] DLM Lock Space: "Magma" 3 5 run - [1 3 2 4] DLM Lock Space: "gfslv" 5 6 run - [2 1 3 4] GFS Mount Group: "gfslv" 6 7 run - [2 1 3 4] User: "usrm::manager" 2 4 run - [1 3 2 4] NODE02: Service Name GID LID State Code Fence Domain: "default" 4 5 run - [1 3 2 4] DLM Lock Space: "clvmd" 1 1 run - [1 3 2 4] DLM Lock Space: "Magma" 3 3 run - [1 3 2 4] DLM Lock Space: "gfslv" 5 6 run - [1 4 2 3] GFS Mount Group: "gfslv" 6 7 run - [1 4 2 3] User: "usrm::manager" 2 2 run - [1 3 2 4] NODE03: Service Name GID LID State Code Fence Domain: "default" 4 2 run - [1 2 3 4] DLM Lock Space: "clvmd" 1 3 run - [1 2 3 4] DLM Lock Space: "Magma" 3 5 run - [1 2 3 4] DLM Lock Space: "gfslv" 5 6 run - [1 2 4 3] GFS Mount Group: "gfslv" 6 7 run - [1 2 4 3] User: "usrm::manager" 2 4 run - [1 2 3 4] NODE04: Service Name GID LID State Code Fence Domain: "default" 4 2 run - [1 2 3 4] DLM Lock Space: "clvmd" 1 3 run - [1 2 3 4] DLM Lock Space: "Magma" 3 5 run - [1 2 3 4] DLM Lock Space: "gfslv" 5 6 run - [1 4 2 3] GFS Mount Group: "gfslv" 6 7 run - [1 4 2 3] User: "usrm::manager" 2 4 run - [1 2 3 4] -------------- next part -------------- An HTML attachment was scrubbed... URL: From diggercheer at gmail.com Mon Apr 9 17:39:33 2007 From: diggercheer at gmail.com (David M) Date: Mon, 9 Apr 2007 12:39:33 -0500 Subject: [Linux-cluster] generic error on gfs status Message-ID: I am running a four node GFS cluster with about 20 services per node. All four nodes belong to the same failover domain, and they each have a priority of 1. My shared storage is an iSCSI SAN (on a dedicated switch). Over the last 24 hours, /gfsdata has logged 90 "notices" in the daemon log stating that the gfs "status" check returned a generic error: clurgmgrd[6074]: status on clusterfs "/gfsdata" returned 1 (generic error) clurgmgrd[6074]: Stopping service service1_03 clurgmgrd[6074]: Service service1_03 is recovering clurgmgrd[6074]: Recovering failed service service1_03 clurgmgrd[6074]: Service service1_03 started So, everytime /gfsdata returns a generic error, the rgmanager restarts a service. Can anyone shed any light on why I might be losing my mount point or why gfs is returning a 1? Thank you for your help. cat /proc/mounts /dev/vg0/gfslv /gfsdata gfs rw,noatime,nodiratime 0 0 cman_tool services: NODE01: Service Name GID LID State Code Fence Domain: "default" 4 2 run - [1 3 2 4] DLM Lock Space: "clvmd" 1 3 run - [1 3 2 4] DLM Lock Space: "Magma" 3 5 run - [1 3 2 4] DLM Lock Space: "gfslv" 5 6 run - [2 1 3 4] GFS Mount Group: "gfslv" 6 7 run - [2 1 3 4] User: "usrm::manager" 2 4 run - [1 3 2 4] Node02: Service Name GID LID State Code Fence Domain: "default" 4 5 run - [1 3 2 4] DLM Lock Space: "clvmd" 1 1 run - [1 3 2 4] DLM Lock Space: "Magma" 3 3 run - [1 3 2 4] DLM Lock Space: "gfslv" 5 6 run - [1 4 2 3] GFS Mount Group: "gfslv" 6 7 run - [1 4 2 3] User: "usrm::manager" 2 2 run - [1 3 2 4] NODE03: Service Name GID LID State Code Fence Domain: "default" 4 2 run - [1 2 3 4] DLM Lock Space: "clvmd" 1 3 run - [1 2 3 4] DLM Lock Space: "Magma" 3 5 run - [1 2 3 4] DLM Lock Space: "gfslv" 5 6 run - [1 2 4 3] GFS Mount Group: "gfslv" 6 7 run - [1 2 4 3] User: "usrm::manager" 2 4 run - [1 2 3 4] NODE04: Service Name GID LID State Code Fence Domain: "default" 4 2 run - [1 2 3 4] DLM Lock Space: "clvmd" 1 3 run - [1 2 3 4] DLM Lock Space: "Magma" 3 5 run - [1 2 3 4] DLM Lock Space: "gfslv" 5 6 run - [1 4 2 3] GFS Mount Group: "gfslv" 6 7 run - [1 4 2 3] User: "usrm::manager" 2 4 run - [1 2 3 4] cat /proc/slabinfo slabinfo - version: 2.0 # name : tunables : slabdata gfs_meta_header_cache 127 147 80 49 1 : tunables 120 60 8 : slabdata 3 3 0 gfs_bufdata 16 50 160 25 1 : tunables 120 60 8 : slabdata 2 2 0 gfs_inode 284 287 536 7 1 : tunables 54 27 8 : slabdata 41 41 0 gfs_glock 2156 2196 312 12 1 : tunables 54 27 8 : slabdata 183 183 0 gfs_bio_wrapper 520 675 16 225 1 : tunables 120 60 8 : slabdata 3 3 0 dlm_conn 6 20 192 20 1 : tunables 120 60 8 : slabdata 1 1 0 dlm_lvb/range 6446 6545 32 119 1 : tunables 120 60 8 : slabdata 55 55 0 dlm_resdir(s) 553 1932 56 69 1 : tunables 120 60 8 : slabdata 28 28 0 dlm_resdir(l) 0 0 88 45 1 : tunables 120 60 8 : slabdata 0 0 0 dlm_lkb 3318298 3318298 232 17 1 : tunables 120 60 8 : slabdata 195194 195194 0 dlm_rsb(large) 1 13 304 13 1 : tunables 54 27 8 : slabdata 1 1 0 dlm_rsb(small) 1994 2254 272 14 1 : tunables 54 27 8 : slabdata 161 161 0 cluster_sock 6 22 704 11 2 : tunables 54 27 8 : slabdata 2 2 0 rpc_buffers 8 8 2048 2 1 : tunables 24 12 8 : slabdata 4 4 0 rpc_tasks 8 12 320 12 1 : tunables 54 27 8 : slabdata 1 1 0 rpc_inode_cache 6 8 832 4 1 : tunables 54 27 8 : slabdata 2 2 0 iscsi_task_cache 287 287 96 41 1 : tunables 120 60 8 : slabdata 7 7 60 msi_cache 2 2 5760 1 2 : tunables 8 4 0 : slabdata 2 2 0 fib6_nodes 9 61 64 61 1 : tunables 120 60 8 : slabdata 1 1 0 ip6_dst_cache 8 24 320 12 1 : tunables 54 27 8 : slabdata 2 2 0 ndisc_cache 2 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0 rawv6_sock 6 8 1024 4 1 : tunables 54 27 8 : slabdata 2 2 0 udpv6_sock 2 4 1024 4 1 : tunables 54 27 8 : slabdata 1 1 0 tcpv6_sock 7 8 1728 4 2 : tunables 24 12 8 : slabdata 2 2 0 ip_fib_alias 38 119 32 119 1 : tunables 120 60 8 : slabdata 1 1 0 ip_fib_hash 38 61 64 61 1 : tunables 120 60 8 : slabdata 1 1 0 uhci_urb_priv 0 0 88 45 1 : tunables 120 60 8 : slabdata 0 0 0 dm-snapshot-in 128 140 112 35 1 : tunables 120 60 8 : slabdata 4 4 0 dm-snapshot-ex 0 0 32 119 1 : tunables 120 60 8 : slabdata 0 0 0 ext3_inode_cache 34129 34144 840 4 1 : tunables 54 27 8 : slabdata 8536 8536 0 ext3_xattr 0 0 88 45 1 : tunables 120 60 8 : slabdata 0 0 0 journal_handle 24 162 48 81 1 : tunables 120 60 8 : slabdata 2 2 0 journal_head 125 270 88 45 1 : tunables 120 60 8 : slabdata 6 6 0 revoke_table 4 225 16 225 1 : tunables 120 60 8 : slabdata 1 1 0 revoke_record 0 0 32 119 1 : tunables 120 60 8 : slabdata 0 0 0 dm_tio 1345 1560 24 156 1 : tunables 120 60 8 : slabdata 10 10 0 dm_io 1345 1536 40 96 1 : tunables 120 60 8 : slabdata 16 16 0 scsi_cmd_cache 22 35 512 7 1 : tunables 54 27 8 : slabdata 4 5 0 sgpool-128 32 32 4096 1 1 : tunables 24 12 8 : slabdata 32 32 0 sgpool-64 32 32 2048 2 1 : tunables 24 12 8 : slabdata 16 16 0 sgpool-32 32 36 1024 4 1 : tunables 54 27 8 : slabdata 8 9 0 sgpool-16 32 32 512 8 1 : tunables 54 27 8 : slabdata 4 4 0 sgpool-8 60 75 256 15 1 : tunables 120 60 8 : slabdata 5 5 0 unix_sock 104 125 768 5 1 : tunables 54 27 8 : slabdata 25 25 0 ip_mrt_cache 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0 tcp_tw_bucket 106 180 192 20 1 : tunables 120 60 8 : slabdata 9 9 0 tcp_bind_bucket 155 357 32 119 1 : tunables 120 60 8 : slabdata 3 3 0 tcp_open_request 18 31 128 31 1 : tunables 120 60 8 : slabdata 1 1 0 inet_peer_cache 25 62 128 31 1 : tunables 120 60 8 : slabdata 2 2 0 secpath_cache 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 8 : slabdata 0 0 0 ip_dst_cache 148 170 384 10 1 : tunables 54 27 8 : slabdata 17 17 0 arp_cache 10 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0 raw_sock 8 9 832 9 2 : tunables 54 27 8 : slabdata 1 1 0 udp_sock 29 36 832 9 2 : tunables 54 27 8 : slabdata 4 4 0 tcp_sock 75 75 1536 5 2 : tunables 24 12 8 : slabdata 15 15 0 flow_cache 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0 mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 8 : slabdata 1 1 0 relayfs_inode_cache 0 0 576 7 1 : tunables 54 27 8 : slabdata 0 0 0 isofs_inode_cache 0 0 616 6 1 : tunables 54 27 8 : slabdata 0 0 0 hugetlbfs_inode_cache 1 6 608 6 1 : tunables 54 27 8 : slabdata 1 1 0 ext2_inode_cache 0 0 736 5 1 : tunables 54 27 8 : slabdata 0 0 0 ext2_xattr 0 0 88 45 1 : tunables 120 60 8 : slabdata 0 0 0 dquot 0 0 224 17 1 : tunables 120 60 8 : slabdata 0 0 0 eventpoll_pwq 1 54 72 54 1 : tunables 120 60 8 : slabdata 1 1 0 eventpoll_epi 1 20 192 20 1 : tunables 120 60 8 : slabdata 1 1 0 kioctx 0 0 384 10 1 : tunables 54 27 8 : slabdata 0 0 0 kiocb 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 dnotify_cache 2 96 40 96 1 : tunables 120 60 8 : slabdata 1 1 0 fasync_cache 1 156 24 156 1 : tunables 120 60 8 : slabdata 1 1 0 shmem_inode_cache 281 290 800 5 1 : tunables 54 27 8 : slabdata 58 58 0 posix_timers_cache 0 0 184 21 1 : tunables 120 60 8 : slabdata 0 0 0 uid_cache 4 31 128 31 1 : tunables 120 60 8 : slabdata 1 1 0 cfq_pool 86 138 56 69 1 : tunables 120 60 8 : slabdata 2 2 0 crq_pool 34 216 72 54 1 : tunables 120 60 8 : slabdata 4 4 0 deadline_drq 0 0 96 41 1 : tunables 120 60 8 : slabdata 0 0 0 as_arq 0 0 112 35 1 : tunables 120 60 8 : slabdata 0 0 0 blkdev_ioc 31 119 32 119 1 : tunables 120 60 8 : slabdata 1 1 0 blkdev_queue 27 54 848 9 2 : tunables 54 27 8 : slabdata 6 6 0 blkdev_requests 21 105 264 15 1 : tunables 54 27 8 : slabdata 7 7 0 biovec-(256) 256 256 4096 1 1 : tunables 24 12 8 : slabdata 256 256 0 biovec-128 256 256 2048 2 1 : tunables 24 12 8 : slabdata 128 128 0 biovec-64 256 256 1024 4 1 : tunables 54 27 8 : slabdata 64 64 0 biovec-16 256 270 256 15 1 : tunables 120 60 8 : slabdata 18 18 0 biovec-4 256 305 64 61 1 : tunables 120 60 8 : slabdata 5 5 0 biovec-1 388 675 16 225 1 : tunables 120 60 8 : slabdata 3 3 60 bio 408 465 128 31 1 : tunables 120 60 8 : slabdata 15 15 60 file_lock_cache 151 175 160 25 1 : tunables 120 60 8 : slabdata 7 7 0 sock_inode_cache 224 235 704 5 1 : tunables 54 27 8 : slabdata 47 47 0 skbuff_head_cache 681 852 320 12 1 : tunables 54 27 8 : slabdata 71 71 54 sock 24 24 640 6 1 : tunables 54 27 8 : slabdata 4 4 0 proc_inode_cache 812 816 600 6 1 : tunables 54 27 8 : slabdata 136 136 0 sigqueue 188 253 168 23 1 : tunables 120 60 8 : slabdata 11 11 0 radix_tree_node 20031 20069 536 7 1 : tunables 54 27 8 : slabdata 2867 2867 0 bdev_cache 46 50 768 5 1 : tunables 54 27 8 : slabdata 10 10 0 mnt_cache 28 60 192 20 1 : tunables 120 60 8 : slabdata 3 3 0 audit_watch_cache 0 0 88 45 1 : tunables 120 60 8 : slabdata 0 0 0 inode_cache 3969 3969 568 7 1 : tunables 54 27 8 : slabdata 567 567 27 dentry_cache 1617792 1617792 240 16 1 : tunables 120 60 8 : slabdata 101112 101112 0 filp 1747 1815 256 15 1 : tunables 120 60 8 : slabdata 121 121 60 names_cache 51 51 4096 1 1 : tunables 24 12 8 : slabdata 51 51 12 avc_node 13 378 72 54 1 : tunables 120 60 8 : slabdata 7 7 0 key_jar 8 20 192 20 1 : tunables 120 60 8 : slabdata 1 1 0 idr_layer_cache 101 105 528 7 1 : tunables 54 27 8 : slabdata 15 15 0 buffer_head 53280 53415 88 45 1 : tunables 120 60 8 : slabdata 1187 1187 0 mm_struct 140 147 1152 7 2 : tunables 24 12 8 : slabdata 21 21 36 vm_area_struct 5098 5478 176 22 1 : tunables 120 60 8 : slabdata 249 249 240 fs_cache 316 427 64 61 1 : tunables 120 60 8 : slabdata 7 7 60 files_cache 194 216 832 9 2 : tunables 54 27 8 : slabdata 24 24 27 signal_cache 297 360 256 15 1 : tunables 120 60 8 : slabdata 24 24 0 sighand_cache 178 183 2112 3 2 : tunables 24 12 8 : slabdata 61 61 0 task_struct 213 218 1984 2 1 : tunables 24 12 8 : slabdata 109 109 12 anon_vma 1960 2184 24 156 1 : tunables 120 60 8 : slabdata 14 14 60 shared_policy_node 0 0 56 69 1 : tunables 120 60 8 : slabdata 0 0 0 numa_policy 44 450 16 225 1 : tunables 120 60 8 : slabdata 2 2 0 size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0 size-131072 1 1 131072 1 32 : tunables 8 4 0 : slabdata 1 1 0 size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0 size-65536 1 1 65536 1 16 : tunables 8 4 0 : slabdata 1 1 0 size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0 size-32768 4 4 32768 1 8 : tunables 8 4 0 : slabdata 4 4 0 size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0 size-16384 3 3 16384 1 4 : tunables 8 4 0 : slabdata 3 3 0 size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0 size-8192 56 56 8192 1 2 : tunables 8 4 0 : slabdata 56 56 0 size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 : slabdata 0 0 0 size-4096 172 172 4096 1 1 : tunables 24 12 8 : slabdata 172 172 0 size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 : slabdata 0 0 0 size-2048 113 114 2048 2 1 : tunables 24 12 8 : slabdata 57 57 0 size-1620(DMA) 0 0 1664 4 2 : tunables 24 12 8 : slabdata 0 0 0 size-1620 53 60 1664 4 2 : tunables 24 12 8 : slabdata 15 15 0 size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0 size-1024 1126 1240 1024 4 1 : tunables 54 27 8 : slabdata 310 310 81 size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 : slabdata 0 0 0 size-512 2060 2064 512 8 1 : tunables 54 27 8 : slabdata 258 258 65 size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 size-256 1137540 1137540 256 15 1 : tunables 120 60 8 : slabdata 75836 75836 0 size-128(DMA) 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0 size-128 2335 2790 128 31 1 : tunables 120 60 8 : slabdata 90 90 30 size-64(DMA) 0 0 64 61 1 : tunables 120 60 8 : slabdata 0 0 0 size-64 8619 11163 64 61 1 : tunables 120 60 8 : slabdata 183 183 90 size-32(DMA) 0 0 32 119 1 : tunables 120 60 8 : slabdata 0 0 0 size-32 2697 3570 32 119 1 : tunables 120 60 8 : slabdata 30 30 60 kmem_cache 180 180 256 15 1 : tunables 120 60 8 : slabdata 12 12 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pcaulfie at redhat.com Tue Apr 10 07:37:44 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 10 Apr 2007 08:37:44 +0100 Subject: [Linux-cluster] delete node In-Reply-To: <1176128646.4694.0.camel@jarjar.trnswrks.com> References: <1176128646.4694.0.camel@jarjar.trnswrks.com> Message-ID: <461B3EC8.80100@redhat.com> Scott McClanahan wrote: > Hello, we are running CentOS 4.3 and the latest cluster suite packages > from the csgfs yum repository for this release and need to delete a > cluster member. According to the documentation we need to restart all > cluster related services on all remaining nodes in the cluster after the > node has been removed. This is a four node cluster so removing one node > obviously degrades the cluster to three nodes, but we can have the > remaining cluster members recalculate the number of votes to remain > quorate. It still surprises me that we have to bring down the entire > cluster just because we are deleting a node. Any feedback as to why > this might be? > You only need to reboot the whole cluster if you want to remove all trace of the node from /proc/cluster/nodes. If you're not bothered about that them just remove with "cman_tool leave remove" and forget about it. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From pcaulfie at redhat.com Tue Apr 10 07:39:28 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 10 Apr 2007 08:39:28 +0100 Subject: [Linux-cluster] Different sub-network In-Reply-To: <97982DAF1EA1494DB6C65C2EEAA6296105B5344A@CORPUSMX40B.corp.emc.com> References: <97982DAF1EA1494DB6C65C2EEAA6296105B5344A@CORPUSMX40B.corp.emc.com> Message-ID: <461B3F30.2020407@redhat.com> Lin_ChiiShing at emc.com wrote: > In a business continuance configuration, nodes in a cluster will be > running on different networks. > > Is it possible to configure a cluster with nodes running on different > networks ? > Yes, it is. If you configure the cluster to use multicast rather than broadcast (there is an option for this in system-config-cluster) then the nodes can be on different subnets. Be careful that any switches and/or routers between the nodes are of good specification and are set to pass multicast traffic though. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From pcaulfie at redhat.com Tue Apr 10 07:40:43 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 10 Apr 2007 08:40:43 +0100 Subject: [Linux-cluster] Starting with DLM In-Reply-To: References: Message-ID: <461B3F7B.4080101@redhat.com> Christos Triantafillou wrote: > Hello, > > What are the necessary steps to configure DLM on RHEL4 and use it from > an application? > The DLM doesn't need any configuration, just load the dlm.ko kernel module and install the library. To use it in an application, download the cluster suite sources and have a look at the example programs in there and also the documentation in the dlm/docs directory. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From janne.peltonen at helsinki.fi Tue Apr 10 10:54:42 2007 From: janne.peltonen at helsinki.fi (Janne Peltonen) Date: Tue, 10 Apr 2007 13:54:42 +0300 Subject: [Linux-cluster] 4 node cluster even split Message-ID: <20070410105442.GB26722@helsinki.fi> Hi! I've been wondering... If I were to build a 4 node cluster, what would happen (with default quorum settings) if I lost two nodes at once (say, by losing one of my two blade racks)? Would the remaining two nodes be able to continue, or would they consider quorum dissolved and shut down the remaining services? I only have three nodes for testing purposes, so I haven't been able to look at this yet. --Janne -- Janne Peltonen From pcaulfie at redhat.com Tue Apr 10 11:43:43 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 10 Apr 2007 12:43:43 +0100 Subject: [Linux-cluster] 4 node cluster even split In-Reply-To: <20070410105442.GB26722@helsinki.fi> References: <20070410105442.GB26722@helsinki.fi> Message-ID: <461B786F.1070204@redhat.com> Janne Peltonen wrote: > Hi! > > I've been wondering... > > If I were to build a 4 node cluster, what would happen (with default > quorum settings) if I lost two nodes at once (say, by losing one of my > two blade racks)? Would the remaining two nodes be able to continue, or > would they consider quorum dissolved and shut down the remaining > services? I only have three nodes for testing purposes, so I haven't > been able to look at this yet. Under normal circumstances you need (n/2)+1 nodes to keep quorum. So if you lose two nodes out of four then the services will stop. To prevent this you can use the qdisk program to keep the cluster quorate in such circumstances. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From janne.peltonen at helsinki.fi Tue Apr 10 12:20:01 2007 From: janne.peltonen at helsinki.fi (Janne Peltonen) Date: Tue, 10 Apr 2007 15:20:01 +0300 Subject: [Linux-cluster] 4 node cluster even split In-Reply-To: <461B786F.1070204@redhat.com> References: <20070410105442.GB26722@helsinki.fi> <461B786F.1070204@redhat.com> Message-ID: <20070410122000.GC26722@helsinki.fi> On Tue, Apr 10, 2007 at 12:43:43PM +0100, Patrick Caulfield wrote: > Under normal circumstances you need (n/2)+1 nodes to keep quorum. So if you lose > two nodes out of four then the services will stop. To prevent this you can use > the qdisk program to keep the cluster quorate in such circumstances. OK, that's what I thought... this FAQ entry just confused me: http://sources.redhat.com/cluster/faq.html#cman_oddnodes . But I think you mean floor((n/2) + 1) (to account for clusters with an odd number of nodes, too) ;) The planned setup is as follows: -two blade racks -four blade servers to serve as cluster nodes, two in each rack We'll have two racks anyway, for other projects, so I thought splitting the cluster nodes evenly among the racks would give us some redundancy in case we lose one of the racks. But if we lose one of the racks, this'll cause the cluster to lose quorum... One solution would be to have one non-blade server in the cluster as well, so if we lose either rack, we'll still have 3 (=floor((5/2) + 1)) nodes available, and keep quorum. But this would mean that we'd have to add nodes in pairs (otherwise, we could still have a rack-ful of fully operable nodes that would refuse to keep quorum - unless, of course, the node outside the racks has enough votes - but that would create trouble if we lost that one node...). Now I wonder. Would it be simpler to set up a qdisk? That'd require some consideration of the heuristics to use. And the documentation appears sparse. Is this correct: -whichever partition of a partitioned cluster contains the qdisk master, has all the qdisk votes, AND -qdisk master will always be in the partition that contains the lowest-numbered cluster node that considers itself feasible to be in the cluster (regardless of the opinion of the other nodes in that partition)? In my scenario (failing rack), the qdisk master and the qdisk votes would find their way to alive partition and the partition would keep quorum. I think. If I were to use a simple heuristic (such as pinging a router upstream), that wouldn't break anything in the failing-rack scenario. But what if there was a real split in the heartbeat network, with both halves of the cluster still seeing the upstream router (because there was no split in the 'outer' network)? If SAN access is kept, and both halves still see the quorum device, would the cluster halves be able to agree on the master? --Janne -- Janne Peltonen From scott.mcclanahan at trnswrks.com Tue Apr 10 12:44:27 2007 From: scott.mcclanahan at trnswrks.com (Scott McClanahan) Date: Tue, 10 Apr 2007 08:44:27 -0400 Subject: [Linux-cluster] delete node In-Reply-To: <461B3EC8.80100@redhat.com> References: <1176128646.4694.0.camel@jarjar.trnswrks.com> <461B3EC8.80100@redhat.com> Message-ID: <1176209067.10294.51.camel@jarjar.trnswrks.com> On Tue, 2007-04-10 at 03:37 -0400, Patrick Caulfield wrote: > Scott McClanahan wrote: > > Hello, we are running CentOS 4.3 and the latest cluster suite > packages > > from the csgfs yum repository for this release and need to delete a > > cluster member. According to the documentation we need to restart > all > > cluster related services on all remaining nodes in the cluster after > the > > node has been removed. This is a four node cluster so removing one > node > > obviously degrades the cluster to three nodes, but we can have the > > remaining cluster members recalculate the number of votes to remain > > quorate. It still surprises me that we have to bring down the > entire > > cluster just because we are deleting a node. Any feedback as to why > > this might be? > > > > > You only need to reboot the whole cluster if you want to remove all > trace of the > node from /proc/cluster/nodes. If you're not bothered about that them > just > remove with "cman_tool leave remove" and forget about it. > > -- > Patrick > > Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod > Street, > Windsor, Berkshire, SL4 ITE, UK. > Registered in England and Wales under Company Registration No. 3798903 > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > Ok, thanks for the answer. To further elaborate my concern, if I want to rebuild an existing cluster member and reintroduce it to the cluster how do you suspect the cluster would handle that. What would be the best way to rebuild an existing cluster member. Other than the traces of node details in /proc/cluster/nodes is there any kind of state kept locally on these cluster nodes that would cause issues if I rebuild the node and attempt to transparently add it back to the cluster. Would I need to delete the cluster member from the cluster config using the proper utilities and then add it again? Could I avoid having to delete the cluster member at all from the cluster and simply rebuild/reintroduce? Why? :) Thanks so much. > > -- Scott McClanahan From pcaulfie at redhat.com Tue Apr 10 12:54:41 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 10 Apr 2007 13:54:41 +0100 Subject: [Linux-cluster] delete node In-Reply-To: <1176209067.10294.51.camel@jarjar.trnswrks.com> References: <1176128646.4694.0.camel@jarjar.trnswrks.com> <461B3EC8.80100@redhat.com> <1176209067.10294.51.camel@jarjar.trnswrks.com> Message-ID: <461B8911.10709@redhat.com> Scott McClanahan wrote: > On Tue, 2007-04-10 at 03:37 -0400, Patrick Caulfield wrote: >> Scott McClanahan wrote: >>> Hello, we are running CentOS 4.3 and the latest cluster suite >> packages >>> from the csgfs yum repository for this release and need to delete a >>> cluster member. According to the documentation we need to restart >> all >>> cluster related services on all remaining nodes in the cluster after >> the >>> node has been removed. This is a four node cluster so removing one >> node >>> obviously degrades the cluster to three nodes, but we can have the >>> remaining cluster members recalculate the number of votes to remain >>> quorate. It still surprises me that we have to bring down the >> entire >>> cluster just because we are deleting a node. Any feedback as to why >>> this might be? >>> >> >> You only need to reboot the whole cluster if you want to remove all >> trace of the >> node from /proc/cluster/nodes. If you're not bothered about that them >> just >> remove with "cman_tool leave remove" and forget about it. >> >> -- >> Patrick >> >> Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod >> Street, >> Windsor, Berkshire, SL4 ITE, UK. >> Registered in England and Wales under Company Registration No. 3798903 >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > Ok, thanks for the answer. To further elaborate my concern, if I want > to rebuild an existing cluster member and reintroduce it to the cluster > how do you suspect the cluster would handle that. What would be the > best way to rebuild an existing cluster member. Other than the traces > of node details in /proc/cluster/nodes is there any kind of state kept > locally on these cluster nodes that would cause issues if I rebuild the > node and attempt to transparently add it back to the cluster. > > Would I need to delete the cluster member from the cluster config using > the proper utilities and then add it again? > > Could I avoid having to delete the cluster member at all from the > cluster and simply rebuild/reintroduce? If the rebuilt node has the same node name and IP address (and cluster sofware, obviously) then there will be no problems. The other members of the cluster will welcome it with open arms as an old friend. If you give it a different name/ip address then it will appear in the cluster as a new node and its old incarnation will remain in the list as 'dead'. > Why? :) All the existing members care about the old node is that it has the same name, IP address and node ID. Anything else is nothing to do with them really, or can be sorted out at run time. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From scott.mcclanahan at trnswrks.com Tue Apr 10 13:10:29 2007 From: scott.mcclanahan at trnswrks.com (Scott McClanahan) Date: Tue, 10 Apr 2007 09:10:29 -0400 Subject: [Linux-cluster] delete node In-Reply-To: <461B8911.10709@redhat.com> References: <1176128646.4694.0.camel@jarjar.trnswrks.com> <461B3EC8.80100@redhat.com> <1176209067.10294.51.camel@jarjar.trnswrks.com> <461B8911.10709@redhat.com> Message-ID: <1176210629.10294.62.camel@jarjar.trnswrks.com> On Tue, 2007-04-10 at 13:54 +0100, Patrick Caulfield wrote: > Scott McClanahan wrote: > > On Tue, 2007-04-10 at 03:37 -0400, Patrick Caulfield wrote: > >> Scott McClanahan wrote: > >>> Hello, we are running CentOS 4.3 and the latest cluster suite > >> packages > >>> from the csgfs yum repository for this release and need to delete a > >>> cluster member. According to the documentation we need to restart > >> all > >>> cluster related services on all remaining nodes in the cluster after > >> the > >>> node has been removed. This is a four node cluster so removing one > >> node > >>> obviously degrades the cluster to three nodes, but we can have the > >>> remaining cluster members recalculate the number of votes to remain > >>> quorate. It still surprises me that we have to bring down the > >> entire > >>> cluster just because we are deleting a node. Any feedback as to why > >>> this might be? > >>> > >> > >> You only need to reboot the whole cluster if you want to remove all > >> trace of the > >> node from /proc/cluster/nodes. If you're not bothered about that them > >> just > >> remove with "cman_tool leave remove" and forget about it. > >> > >> -- > >> Patrick > >> > >> Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod > >> Street, > >> Windsor, Berkshire, SL4 ITE, UK. > >> Registered in England and Wales under Company Registration No. 3798903 > >> > >> -- > >> Linux-cluster mailing list > >> Linux-cluster at redhat.com > >> https://www.redhat.com/mailman/listinfo/linux-cluster > >> > > Ok, thanks for the answer. To further elaborate my concern, if I want > > to rebuild an existing cluster member and reintroduce it to the cluster > > how do you suspect the cluster would handle that. What would be the > > best way to rebuild an existing cluster member. Other than the traces > > of node details in /proc/cluster/nodes is there any kind of state kept > > locally on these cluster nodes that would cause issues if I rebuild the > > node and attempt to transparently add it back to the cluster. > > > > Would I need to delete the cluster member from the cluster config using > > the proper utilities and then add it again? > > > > Could I avoid having to delete the cluster member at all from the > > cluster and simply rebuild/reintroduce? > > If the rebuilt node has the same node name and IP address (and cluster sofware, > obviously) then there will be no problems. The other members of the cluster will > welcome it with open arms as an old friend. > > If you give it a different name/ip address then it will appear in the cluster as > a new node and its old incarnation will remain in the list as 'dead'. > > > Why? :) > > All the existing members care about the old node is that it has the same name, > IP address and node ID. Anything else is nothing to do with them really, or can > be sorted out at run time. > Sorry to be so chatty but just to be sure (this is production), I can just rebuild the node without deleting it from the config? I will obviously need to copy the cluster.conf file to the newly built node but I thought that the node ID was "assigned" during the joining process of cluster membership. If that is the case and as long as I don't manually override the node ID, do I really have to worry about anything other than the hostname/IP? Not sure if it helps to reiterate that this is the cluster suite software built for RHEL version 4.3. > > From pcaulfie at redhat.com Tue Apr 10 13:26:13 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 10 Apr 2007 14:26:13 +0100 Subject: [Linux-cluster] delete node In-Reply-To: <1176210629.10294.62.camel@jarjar.trnswrks.com> References: <1176128646.4694.0.camel@jarjar.trnswrks.com> <461B3EC8.80100@redhat.com> <1176209067.10294.51.camel@jarjar.trnswrks.com> <461B8911.10709@redhat.com> <1176210629.10294.62.camel@jarjar.trnswrks.com> Message-ID: <461B9075.1060203@redhat.com> Scott McClanahan wrote: > On Tue, 2007-04-10 at 13:54 +0100, Patrick Caulfield wrote: >> Scott McClanahan wrote: >>> On Tue, 2007-04-10 at 03:37 -0400, Patrick Caulfield wrote: >>>> Scott McClanahan wrote: >>>>> Hello, we are running CentOS 4.3 and the latest cluster suite >>>> packages >>>>> from the csgfs yum repository for this release and need to delete a >>>>> cluster member. According to the documentation we need to restart >>>> all >>>>> cluster related services on all remaining nodes in the cluster after >>>> the >>>>> node has been removed. This is a four node cluster so removing one >>>> node >>>>> obviously degrades the cluster to three nodes, but we can have the >>>>> remaining cluster members recalculate the number of votes to remain >>>>> quorate. It still surprises me that we have to bring down the >>>> entire >>>>> cluster just because we are deleting a node. Any feedback as to why >>>>> this might be? >>>>> >>>> You only need to reboot the whole cluster if you want to remove all >>>> trace of the >>>> node from /proc/cluster/nodes. If you're not bothered about that them >>>> just >>>> remove with "cman_tool leave remove" and forget about it. >>>> >>>> -- >>>> Patrick >>>> >>>> Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod >>>> Street, >>>> Windsor, Berkshire, SL4 ITE, UK. >>>> Registered in England and Wales under Company Registration No. 3798903 >>>> >>>> -- >>>> Linux-cluster mailing list >>>> Linux-cluster at redhat.com >>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>> >>> Ok, thanks for the answer. To further elaborate my concern, if I want >>> to rebuild an existing cluster member and reintroduce it to the cluster >>> how do you suspect the cluster would handle that. What would be the >>> best way to rebuild an existing cluster member. Other than the traces >>> of node details in /proc/cluster/nodes is there any kind of state kept >>> locally on these cluster nodes that would cause issues if I rebuild the >>> node and attempt to transparently add it back to the cluster. >>> >>> Would I need to delete the cluster member from the cluster config using >>> the proper utilities and then add it again? >>> >>> Could I avoid having to delete the cluster member at all from the >>> cluster and simply rebuild/reintroduce? >> If the rebuilt node has the same node name and IP address (and cluster sofware, >> obviously) then there will be no problems. The other members of the cluster will >> welcome it with open arms as an old friend. >> >> If you give it a different name/ip address then it will appear in the cluster as >> a new node and its old incarnation will remain in the list as 'dead'. >> >>> Why? :) >> All the existing members care about the old node is that it has the same name, >> IP address and node ID. Anything else is nothing to do with them really, or can >> be sorted out at run time. >> > Sorry to be so chatty but just to be sure (this is production), I can > just rebuild the node without deleting it from the config? I will > obviously need to copy the cluster.conf file to the newly built node but > I thought that the node ID was "assigned" during the joining process of > cluster membership. If that is the case and as long as I don't manually > override the node ID, do I really have to worry about anything other > than the hostname/IP? Not sure if it helps to reiterate that this is > the cluster suite software built for RHEL version 4.3. Sounds like it'll be fine. Yes, the nodeid is normally assigned by cman (you can override it but few people seem to). So if the node joins with the same name/IP address it will get its old node ID back. As far as cman is concerned it will be the same node as it always was. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From Robert.Hell at fabasoft.com Tue Apr 10 17:02:52 2007 From: Robert.Hell at fabasoft.com (Hell, Robert) Date: Tue, 10 Apr 2007 19:02:52 +0200 Subject: [Linux-cluster] RHEL 5: Problems with rgmanager and qdiskd Message-ID: Hi! I got some problems using RHEL 5 Cluster Suite for a two node cluster with a quorum disk. The quorum disk is registered by qdiskd in cman (cman_tool shows quorum as a cluster member) correctly. When starting rgmanager the services don't come up and cman_tool hangs when executed. The following entries are found in /var/log/messages: Mar 30 14:09:27 pg-ba-001 clurgmgrd[20629]: #34: Cannot get status for service service:pg-ba-vts1 Mar 30 14:09:43 pg-ba-001 clurgmgrd[20629]: #34: Cannot get status for service service:pg-ba-vts2 Here are my settings for qdiskd in cluster.conf: I'm using: RHEL 5: cman-2.0.60-1.el5, rgmanager-2.0.23-1 (x86_64) Any ideas? Regards, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From dex.chen at crosswalkinc.com Tue Apr 10 18:27:29 2007 From: dex.chen at crosswalkinc.com (Dex Chen) Date: Tue, 10 Apr 2007 12:27:29 -0600 Subject: [Linux-cluster] RHEL 4: quorate or inquorate In-Reply-To: Message-ID: <2E02749DAF5338479606A056219BE10901787582@smail.crosswalkinc.com> Hi, all: We saw a rare situation that a 3 node cluster has 2 nodes offline, but it maintains quorum. Is there anyone has any idea how it could happen and what/where I should look into. Here is the output from clustat on one of the nodes: Member Status: Quorate Member Name Status ------ ---- ------ lizard01 Offline lizard02 Online, Local, rgmanager lizard03 Offline Service Name Owner (Last) State ------- ---- ----- ------ ----- 192.168.210.121 lizard02 started 192.168.210.122 lizard02 started 192.168.210.123 lizard02 started 192.168.210.124 lizard02 started We are on REHL 2.6.9-34 (update 4). Thanks, Dex -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhurst at bidmc.harvard.edu Tue Apr 10 19:21:25 2007 From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu) Date: Tue, 10 Apr 2007 15:21:25 -0400 Subject: [Linux-cluster] 4 node cluster even split In-Reply-To: <461B786F.1070204@redhat.com> References: <20070410105442.GB26722@helsinki.fi> <461B786F.1070204@redhat.com> Message-ID: <1176232885.3375.18.camel@WSBID06223> I don't know if this could help you, but let me share our configuration as an example. I have overridden the default votes on 2 nodes that are vital to application implementation of an 11-node cluster overall. If I cold start the cluster, those 2 nodes running alone are enough to quorate, because they each carry 5 votes each for a total of 10 votes. The remaining 9 nodes carry the default 1 vote each, so the cluster expected votes are 19. 10 = (19 / 2) + 1 If I lose 1 of those 2 network director nodes, you lose 5 votes but you remain quorate, unless you lose 5 more regular nodes along with it. If I lose BOTH network director nodes (10 votes), I don't care about quorum, because my application is dead anyways (no network directors managing client connections!). But, we have a contingency plan to "promote" one of the failover nodes as a network director by running the correct services and adjusting its vote count to 5 for extra redundancy. It would be nice to see other implementations that vary from the typical 1 vote per node cluster. On Tue, 2007-04-10 at 12:43 +0100, Patrick Caulfield wrote: > Janne Peltonen wrote: > > Hi! > > > > I've been wondering... > > > > If I were to build a 4 node cluster, what would happen (with default > > quorum settings) if I lost two nodes at once (say, by losing one of my > > two blade racks)? Would the remaining two nodes be able to continue, or > > would they consider quorum dissolved and shut down the remaining > > services? I only have three nodes for testing purposes, so I haven't > > been able to look at this yet. > > Under normal circumstances you need (n/2)+1 nodes to keep quorum. So if you lose > two nodes out of four then the services will stop. To prevent this you can use > the qdisk program to keep the cluster quorate in such circumstances. > Robert Hurst, Sr. Cach? Administrator Beth Israel Deaconess Medical Center 1135 Tremont Street, REN-7 Boston, Massachusetts 02120-2140 617-754-8754 ? Fax: 617-754-8730 ? Cell: 401-787-3154 Any technology distinguishable from magic is insufficiently advanced. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2178 bytes Desc: not available URL: From Alain.Moulle at bull.net Wed Apr 11 07:11:29 2007 From: Alain.Moulle at bull.net (Alain Moulle) Date: Wed, 11 Apr 2007 09:11:29 +0200 Subject: [Linux-cluster] CS4 Update4 / pb with a cluster.conf built with GUI Message-ID: <461C8A21.2060200@bull.net> Hi cluster.conf has been built via the system-config-cluster GUI (system-config-cluster-1.0.27-1.0) : /etc/cluster/cluster.conf:20: element cman: Relax-NG validity error : Expecting element gulm, got cman Relax-NG validity error : Extra element fencedevices in interleave /etc/cluster/cluster.conf:2: element cluster: Relax-NG validity error : Element cluster failed to validate content /etc/cluster/cluster.conf:15: element device: validity error : IDREF attribute name references an unknown ID "hwmtv12" /etc/cluster/cluster.conf fails to validate Thanks for your help Alain Moull? -------------- next part -------------- A non-text attachment was scrubbed... Name: cluster.conf Type: text/xml Size: 1209 bytes Desc: not available URL: From jos at xos.nl Wed Apr 11 06:39:19 2007 From: jos at xos.nl (Jos Vos) Date: Wed, 11 Apr 2007 08:39:19 +0200 Subject: [Linux-cluster] CS4 Update4 / pb with a cluster.conf built with GUI In-Reply-To: <461C8A21.2060200@bull.net>; from Alain.Moulle@bull.net on Wed, Apr 11, 2007 at 09:11:29AM +0200 References: <461C8A21.2060200@bull.net> Message-ID: <20070411083919.A9155@xos037.xos.nl> On Wed, Apr 11, 2007 at 09:11:29AM +0200, Alain Moulle wrote: ` > cluster.conf has been built via the system-config-cluster GUI > (system-config-cluster-1.0.27-1.0) : > > /etc/cluster/cluster.conf:20: element cman: Relax-NG validity error : Expecting > element gulm, got cman > Relax-NG validity error : Extra element fencedevices in interleave > /etc/cluster/cluster.conf:2: element cluster: Relax-NG validity error : Element > cluster failed to validate content > /etc/cluster/cluster.conf:15: element device: validity error : IDREF attribute > name references an unknown > ID "hwmtv12" > /etc/cluster/cluster.conf fails to validate This is probably the same problem as the bug I reported last year October: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=210611 I don't know why RH does not distribute bug fixes for real problems like this, annoying for many people (since then, I always patch the Python file as I described in the bug report). Anyway, RHEL4 U5 is around the corner I think... -- -- Jos Vos -- X/OS Experts in Open Systems BV | Phone: +31 20 6938364 -- Amsterdam, The Netherlands | Fax: +31 20 6948204 From laule75 at yahoo.fr Wed Apr 11 07:24:48 2007 From: laule75 at yahoo.fr (laurent) Date: Wed, 11 Apr 2007 09:24:48 +0200 (CEST) Subject: [Linux-cluster] Veritas Cluster Suite VS Redhat CLuster Suite Message-ID: <182044.18633.qm@web26504.mail.ukl.yahoo.com> Hello, I'm about to start an exhaustive study on the Veritas Cluster Suite and the Redhat Cluster Suite in order to compare their functionalities and their performances The objective of this comparison is to determine our future production Cluster environment. As I don"t want to reinvent the wheel, are you aware of any existing study about this subject ? Any help would be greatly appreciated. Kind regards Laurent --------------------------------- D?couvrez une nouvelle fa?on d'obtenir des r?ponses ? toutes vos questions ! Profitez des connaissances, des opinions et des exp?riences des internautes sur Yahoo! Questions/R?ponses. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pcaulfie at redhat.com Wed Apr 11 08:17:41 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 11 Apr 2007 09:17:41 +0100 Subject: [Linux-cluster] 4 node cluster even split In-Reply-To: <1176232885.3375.18.camel@WSBID06223> References: <20070410105442.GB26722@helsinki.fi> <461B786F.1070204@redhat.com> <1176232885.3375.18.camel@WSBID06223> Message-ID: <461C99A5.3050906@redhat.com> rhurst at bidmc.harvard.edu wrote: > I don't know if this could help you, but let me share our configuration > as an example. I have overridden the default votes on 2 nodes that are > vital to application implementation of an 11-node cluster overall. If I > cold start the cluster, those 2 nodes running alone are enough to > quorate, because they each carry 5 votes each for a total of 10 votes. > The remaining 9 nodes carry the default 1 vote each, so the cluster > expected votes are 19. > > 10 = (19 / 2) + 1 > > If I lose 1 of those 2 network director nodes, you lose 5 votes but you > remain quorate, unless you lose 5 more regular nodes along with it. If > I lose BOTH network director nodes (10 votes), I don't care about > quorum, because my application is dead anyways (no network directors > managing client connections!). But, we have a contingency plan to > "promote" one of the failover nodes as a network director by running the > correct services and adjusting its vote count to 5 for extra redundancy. > > It would be nice to see other implementations that vary from the typical > 1 vote per node cluster. > Oh, it's not uncommon. I used to do this with a 96 node VAX cluster way-back-when! The main reason it's not a the front of the documentation is that you really need to know what you are doing with that sort of system. If people start arbitrarily increasing the votes of nodes just to keep quorum they could get themselves into some horrible data-corrupting scenarios. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From janne.peltonen at helsinki.fi Wed Apr 11 09:54:21 2007 From: janne.peltonen at helsinki.fi (Janne Peltonen) Date: Wed, 11 Apr 2007 12:54:21 +0300 Subject: On documentation (Re: [Linux-cluster] 4 node cluster even split) In-Reply-To: <461C99A5.3050906@redhat.com> References: <20070410105442.GB26722@helsinki.fi> <461B786F.1070204@redhat.com> <1176232885.3375.18.camel@WSBID06223> <461C99A5.3050906@redhat.com> Message-ID: <20070411095420.GH26722@helsinki.fi> On Wed, Apr 11, 2007 at 09:17:41AM +0100, Patrick Caulfield wrote: > > It would be nice to see other implementations that vary from the typical > > 1 vote per node cluster. > Oh, it's not uncommon. I used to do this with a 96 node VAX cluster way-back-when! > The main reason it's not a the front of the documentation is that you really > need to know what you are doing with that sort of system. If people start > arbitrarily increasing the votes of nodes just to keep quorum they could get > themselves into some horrible data-corrupting scenarios. That's probably very true. But on the other hand, managing a cluster /is/ a bit more challenging task overall than everyday system administration. Reading the csgfs documentation, I got the feeling that the writer is trying to explain an inherently complicated task to a complete newbie by leaving relevant information out. For example, there is hardly any mention about the command line tools that are much more versatile than the GUI. It seemed to me even that there weren't any (or, at least, many) pointers to more in-depth information... How to put it? People start doing stupid things if they need to do something and aren't told how to do it right. About the question at hand, it wouldn't probably hurt if there was an example or two in the QDisk man page. Or on a web page, with a pointer in the documentation (or FAQ). Or, yes, in this list. :) --Janne -- Janne Peltonen From Lin_ChiiShing at emc.com Wed Apr 11 14:52:11 2007 From: Lin_ChiiShing at emc.com (Lin_ChiiShing at emc.com) Date: Wed, 11 Apr 2007 10:52:11 -0400 Subject: [Linux-cluster] RE: Veritas Cluster Suite VS Redhat CLuster Suite In-Reply-To: <20070411095435.25E1073336@hormel.redhat.com> References: <20070411095435.25E1073336@hormel.redhat.com> Message-ID: <97982DAF1EA1494DB6C65C2EEAA6296105C765FE@CORPUSMX40B.corp.emc.com> I am also interested to see any existing comparison between these two clusters. I am specifically interested in the performance of their cluster mirrored volumes. Here below is some performance test data I collected. (take it at your own risk) The VERITAS test configuration include two nodes cluster, a VG (Volume Group) which consists of two physical disks, and two LV (logical volumes) from this VG. One LV with cluster-striped attributes, the other LV with cluster-mirrored attributes. The IOMETER (sequential IO) IOPS data follows: IOsize 512B 4K 16K 32K Striped-read 20383 33845 12381 6226 Mirrored-read 22711 33934 11769 6080 Striped-write 10820 16711 8299 4817 Mirrored-write 9626 9283 4428 2600 The VERITAS test data looks as expected to me, for example, some expected overhead for mirrored write, and in general the IOPS number looks normal to me. The RedHat cluster also include two nodes cluster. Two VGs, one VG is local and the other VG with cluster property. Each VG has three disks, and two LVs, the striped LV and the mirrored LV. Its IOMETER (sequential IO) data follows: IOsize 512B 4K 16K 32K Local-Striped-read 9070 7717 3795 3009 cluster-Striped-read 9180 7751 4686 3257 Local-mirrored-read 8724 7506 4922 3398 cluster-Mirrored-read 331 332 332 332 <<<< why ? Local-Striped-write 3711 3404 2626 1989 Local-Mirrored-write 331 331 331 329 <<< why ? Does anyone know why the significant "read IO" performance degradation on cluster-mirrored LV, and the write IO drop on local-mirrored LV ? The RedHat component versions are: [root at linux-cluster1 sbin]# ccs_tool -V ccs_tool DEVEL.1170370189 (built Feb 21 2007 15:11:13) Copyright (C) Red Hat, Inc. 2004 All rights reserved. [root at linux-cluster1 sbin]# cman_tool version 5.0.1 config 20 [root at linux-cluster1 sbin]# lvm version Logging initialised at Wed Apr 11 10:46:29 2007 Set umask to 0077 WARNING: Ignoring duplicate config node: library_dir (seeking library_dir) Loaded external locking library liblvm2clusterlock.so LVM version: 2.02.18-cvs (2007-01-11) Library version: 1.02.14-cvs (2007-01-07) Driver version: 4.5.0 Wiping internal VG cache Is there a fix to the mirrored-LV performance in newer releases ? Thanks, -ChiiShing -----Original Message----- Date: Wed, 11 Apr 2007 09:24:48 +0200 (CEST) From: laurent Subject: [Linux-cluster] Veritas Cluster Suite VS Redhat CLuster Suite To: linux-cluster at redhat.com Message-ID: <182044.18633.qm at web26504.mail.ukl.yahoo.com> Content-Type: text/plain; charset="iso-8859-1" Hello, I'm about to start an exhaustive study on the Veritas Cluster Suite and the Redhat Cluster Suite in order to compare their functionalities and their performances The objective of this comparison is to determine our future production Cluster environment. As I don"t want to reinvent the wheel, are you aware of any existing study about this subject ? Any help would be greatly appreciated. Kind regards Laurent From diggercheer at gmail.com Wed Apr 11 16:00:40 2007 From: diggercheer at gmail.com (David M) Date: Wed, 11 Apr 2007 11:00:40 -0500 Subject: [Linux-cluster] Re: generic error on gfs status In-Reply-To: References: Message-ID: The problem seems to be that the status check is failing on /usr/share/cluster/clusterfs.sh. Either "isMounted" or "isAlive" (in /usr/share/cluster/clusterfs.sh) is failing to return a "0". It seems that whenever the "status" polling on a particular init script returns a "1", the rgmanager checks the file system. Every time, the clusterfs.sh returns a "1" (rather than a "0"). Then, the service is disabled and re-enabled. I will try to find out why clusterfs.sh is not returning a "0". On 4/9/07, David M wrote: > > I am running a four node GFS cluster with about 20 services per node. All > four nodes belong to the same failover domain, and they each have a priority > of 1. My shared storage is an iSCSI SAN (on a dedicated switch). > > Over the last 24 hours, /gfsdata has logged 90 "notices" in the daemon log > stating that the gfs "status" check returned a generic error: > clurgmgrd[6074]: status on clusterfs "/gfsdata" returned 1 > (generic error) > clurgmgrd[6074]: Stopping service service1_03 > clurgmgrd[6074]: Service service1_03 is recovering > clurgmgrd[6074]: Recovering failed service service1_03 > clurgmgrd[6074]: Service service1_03 started > > So, everytime /gfsdata returns a generic error, the rgmanager restarts a > service. > > Can anyone shed any light on why I might be losing my mount point or why > gfs is returning a 1? > > Thank you for your help. > > cat /proc/mounts > > /dev/vg0/gfslv /gfsdata gfs rw,noatime,nodiratime 0 0 > > cman_tool services: > > NODE01: > Service Name GID LID State Code > Fence Domain: "default" 4 2 run - > [1 3 2 4] > > DLM Lock Space: "clvmd" 1 3 run - > [1 3 2 4] > > DLM Lock Space: "Magma" 3 5 run - > [1 3 2 4] > > DLM Lock Space: "gfslv" 5 6 run - > [2 1 3 4] > > GFS Mount Group: "gfslv" 6 7 run - > [2 1 3 4] > > User: "usrm::manager" 2 4 run - > [1 3 2 4] > > > Node02: > Service Name GID LID State Code > Fence Domain: "default" 4 5 run - > [1 3 2 4] > > DLM Lock Space: "clvmd" 1 1 run - > [1 3 2 4] > > DLM Lock Space: "Magma" 3 3 run - > [1 3 2 4] > > DLM Lock Space: "gfslv" 5 6 run - > [1 4 2 3] > > GFS Mount Group: "gfslv" 6 7 run - > [1 4 2 3] > > User: "usrm::manager" 2 2 run - > [1 3 2 4] > > > NODE03: > Service Name GID LID State Code > Fence Domain: "default" 4 2 run - > [1 2 3 4] > > DLM Lock Space: "clvmd" 1 3 run - > [1 2 3 4] > > DLM Lock Space: "Magma" 3 5 run - > [1 2 3 4] > > DLM Lock Space: "gfslv" 5 6 run - > [1 2 4 3] > > GFS Mount Group: "gfslv" 6 7 run - > [1 2 4 3] > > User: "usrm::manager" 2 4 run - > [1 2 3 4] > > > NODE04: > Service Name GID LID State Code > Fence Domain: "default" 4 2 run - > [1 2 3 4] > > DLM Lock Space: "clvmd" 1 3 run - > [1 2 3 4] > > DLM Lock Space: "Magma" 3 5 run - > [1 2 3 4] > > DLM Lock Space: "gfslv" 5 6 run - > [1 4 2 3] > > GFS Mount Group: "gfslv" 6 7 run - > [1 4 2 3] > > User: "usrm::manager" 2 4 run - > [1 2 3 4] > > > cat /proc/slabinfo > slabinfo - version: 2.0 > # name > : tunables : slabdata > > gfs_meta_header_cache 127 147 80 49 1 : tunables 120 > 60 8 : slabdata 3 3 0 > gfs_bufdata 16 50 160 25 1 : tunables 120 60 8 > : slabdata 2 2 0 > gfs_inode 284 287 536 7 1 : tunables 54 27 8 > : slabdata 41 41 0 > gfs_glock 2156 2196 312 12 1 : tunables 54 27 8 > : slabdata 183 183 0 > gfs_bio_wrapper 520 675 16 225 1 : tunables 120 60 8 > : slabdata 3 3 0 > dlm_conn 6 20 192 20 1 : tunables 120 60 8 > : slabdata 1 1 0 > dlm_lvb/range 6446 6545 32 119 1 : tunables 120 60 8 > : slabdata 55 55 0 > dlm_resdir(s) 553 1932 56 69 1 : tunables 120 60 8 > : slabdata 28 28 0 > dlm_resdir(l) 0 0 88 45 1 : tunables 120 60 8 > : slabdata 0 0 0 > dlm_lkb 3318298 3318298 232 17 1 : tunables 120 60 > 8 : slabdata 195194 195194 0 > dlm_rsb(large) 1 13 304 13 1 : tunables 54 27 8 > : slabdata 1 1 0 > dlm_rsb(small) 1994 2254 272 14 1 : tunables 54 27 8 > : slabdata 161 161 0 > cluster_sock 6 22 704 11 2 : tunables 54 27 8 > : slabdata 2 2 0 > rpc_buffers 8 8 2048 2 1 : tunables 24 12 8 > : slabdata 4 4 0 > rpc_tasks 8 12 320 12 1 : tunables 54 27 8 > : slabdata 1 1 0 > rpc_inode_cache 6 8 832 4 1 : tunables 54 27 8 > : slabdata 2 2 0 > iscsi_task_cache 287 287 96 41 1 : tunables 120 60 8 > : slabdata 7 7 60 > msi_cache 2 2 5760 1 2 : tunables 8 4 0 > : slabdata 2 2 0 > fib6_nodes 9 61 64 61 1 : tunables 120 60 8 > : slabdata 1 1 0 > ip6_dst_cache 8 24 320 12 1 : tunables 54 27 8 > : slabdata 2 2 0 > ndisc_cache 2 15 256 15 1 : tunables 120 60 8 > : slabdata 1 1 0 > rawv6_sock 6 8 1024 4 1 : tunables 54 27 8 > : slabdata 2 2 0 > udpv6_sock 2 4 1024 4 1 : tunables 54 27 8 > : slabdata 1 1 0 > tcpv6_sock 7 8 1728 4 2 : tunables 24 12 8 > : slabdata 2 2 0 > ip_fib_alias 38 119 32 119 1 : tunables 120 60 8 > : slabdata 1 1 0 > ip_fib_hash 38 61 64 61 1 : tunables 120 60 8 > : slabdata 1 1 0 > uhci_urb_priv 0 0 88 45 1 : tunables 120 60 8 > : slabdata 0 0 0 > dm-snapshot-in 128 140 112 35 1 : tunables 120 60 8 > : slabdata 4 4 0 > dm-snapshot-ex 0 0 32 119 1 : tunables 120 60 8 > : slabdata 0 0 0 > ext3_inode_cache 34129 34144 840 4 1 : tunables 54 27 8 > : slabdata 8536 8536 0 > ext3_xattr 0 0 88 45 1 : tunables 120 60 8 > : slabdata 0 0 0 > journal_handle 24 162 48 81 1 : tunables 120 60 8 > : slabdata 2 2 0 > journal_head 125 270 88 45 1 : tunables 120 60 8 > : slabdata 6 6 0 > revoke_table 4 225 16 225 1 : tunables 120 60 8 > : slabdata 1 1 0 > revoke_record 0 0 32 119 1 : tunables 120 60 8 > : slabdata 0 0 0 > dm_tio 1345 1560 24 156 1 : tunables 120 60 8 > : slabdata 10 10 0 > dm_io 1345 1536 40 96 1 : tunables 120 60 8 > : slabdata 16 16 0 > scsi_cmd_cache 22 35 512 7 1 : tunables 54 27 8 > : slabdata 4 5 0 > sgpool-128 32 32 4096 1 1 : tunables 24 12 8 > : slabdata 32 32 0 > sgpool-64 32 32 2048 2 1 : tunables 24 12 8 > : slabdata 16 16 0 > sgpool-32 32 36 1024 4 1 : tunables 54 27 8 > : slabdata 8 9 0 > sgpool-16 32 32 512 8 1 : tunables 54 27 8 > : slabdata 4 4 0 > sgpool-8 60 75 256 15 1 : tunables 120 60 8 > : slabdata 5 5 0 > unix_sock 104 125 768 5 1 : tunables 54 27 8 > : slabdata 25 25 0 > ip_mrt_cache 0 0 128 31 1 : tunables 120 60 8 > : slabdata 0 0 0 > tcp_tw_bucket 106 180 192 20 1 : tunables 120 60 8 > : slabdata 9 9 0 > tcp_bind_bucket 155 357 32 119 1 : tunables 120 60 8 > : slabdata 3 3 0 > tcp_open_request 18 31 128 31 1 : tunables 120 60 8 > : slabdata 1 1 0 > inet_peer_cache 25 62 128 31 1 : tunables 120 60 8 > : slabdata 2 2 0 > secpath_cache 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 8 > : slabdata 0 0 0 > ip_dst_cache 148 170 384 10 1 : tunables 54 27 8 > : slabdata 17 17 0 > arp_cache 10 15 256 15 1 : tunables 120 60 8 > : slabdata 1 1 0 > raw_sock 8 9 832 9 2 : tunables 54 27 8 > : slabdata 1 1 0 > udp_sock 29 36 832 9 2 : tunables 54 27 8 > : slabdata 4 4 0 > tcp_sock 75 75 1536 5 2 : tunables 24 12 8 > : slabdata 15 15 0 > flow_cache 0 0 128 31 1 : tunables 120 60 8 > : slabdata 0 0 0 > mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 > 8 : slabdata 1 1 0 > relayfs_inode_cache 0 0 576 7 1 : tunables 54 27 > 8 : slabdata 0 0 0 > isofs_inode_cache 0 0 616 6 1 : tunables 54 27 8 > : slabdata 0 0 0 > hugetlbfs_inode_cache 1 6 608 6 1 : tunables 54 > 27 8 : slabdata 1 1 0 > ext2_inode_cache 0 0 736 5 1 : tunables 54 27 8 > : slabdata 0 0 0 > ext2_xattr 0 0 88 45 1 : tunables 120 60 8 > : slabdata 0 0 0 > dquot 0 0 224 17 1 : tunables 120 60 8 > : slabdata 0 0 0 > eventpoll_pwq 1 54 72 54 1 : tunables 120 60 8 > : slabdata 1 1 0 > eventpoll_epi 1 20 192 20 1 : tunables 120 60 8 > : slabdata 1 1 0 > kioctx 0 0 384 10 1 : tunables 54 27 8 > : slabdata 0 0 0 > kiocb 0 0 256 15 1 : tunables 120 60 8 > : slabdata 0 0 0 > dnotify_cache 2 96 40 96 1 : tunables 120 60 8 > : slabdata 1 1 0 > fasync_cache 1 156 24 156 1 : tunables 120 60 8 > : slabdata 1 1 0 > shmem_inode_cache 281 290 800 5 1 : tunables 54 27 8 > : slabdata 58 58 0 > posix_timers_cache 0 0 184 21 1 : tunables 120 60 > 8 : slabdata 0 0 0 > uid_cache 4 31 128 31 1 : tunables 120 60 8 > : slabdata 1 1 0 > cfq_pool 86 138 56 69 1 : tunables 120 60 8 > : slabdata 2 2 0 > crq_pool 34 216 72 54 1 : tunables 120 60 8 > : slabdata 4 4 0 > deadline_drq 0 0 96 41 1 : tunables 120 60 8 > : slabdata 0 0 0 > as_arq 0 0 112 35 1 : tunables 120 60 8 > : slabdata 0 0 0 > blkdev_ioc 31 119 32 119 1 : tunables 120 60 8 > : slabdata 1 1 0 > blkdev_queue 27 54 848 9 2 : tunables 54 27 8 > : slabdata 6 6 0 > blkdev_requests 21 105 264 15 1 : tunables 54 27 8 > : slabdata 7 7 0 > biovec-(256) 256 256 4096 1 1 : tunables 24 12 8 > : slabdata 256 256 0 > biovec-128 256 256 2048 2 1 : tunables 24 12 8 > : slabdata 128 128 0 > biovec-64 256 256 1024 4 1 : tunables 54 27 8 > : slabdata 64 64 0 > biovec-16 256 270 256 15 1 : tunables 120 60 8 > : slabdata 18 18 0 > biovec-4 256 305 64 61 1 : tunables 120 60 8 > : slabdata 5 5 0 > biovec-1 388 675 16 225 1 : tunables 120 60 8 > : slabdata 3 3 60 > bio 408 465 128 31 1 : tunables 120 60 8 > : slabdata 15 15 60 > file_lock_cache 151 175 160 25 1 : tunables 120 60 8 > : slabdata 7 7 0 > sock_inode_cache 224 235 704 5 1 : tunables 54 27 8 > : slabdata 47 47 0 > skbuff_head_cache 681 852 320 12 1 : tunables 54 27 8 > : slabdata 71 71 54 > sock 24 24 640 6 1 : tunables 54 27 8 > : slabdata 4 4 0 > proc_inode_cache 812 816 600 6 1 : tunables 54 27 8 > : slabdata 136 136 0 > sigqueue 188 253 168 23 1 : tunables 120 60 8 > : slabdata 11 11 0 > radix_tree_node 20031 20069 536 7 1 : tunables 54 27 8 > : slabdata 2867 2867 0 > bdev_cache 46 50 768 5 1 : tunables 54 27 8 > : slabdata 10 10 0 > mnt_cache 28 60 192 20 1 : tunables 120 60 8 > : slabdata 3 3 0 > audit_watch_cache 0 0 88 45 1 : tunables 120 60 8 > : slabdata 0 0 0 > inode_cache 3969 3969 568 7 1 : tunables 54 27 8 > : slabdata 567 567 27 > dentry_cache 1617792 1617792 240 16 1 : tunables 120 60 > 8 : slabdata 101112 101112 0 > filp 1747 1815 256 15 1 : tunables 120 60 8 > : slabdata 121 121 60 > names_cache 51 51 4096 1 1 : tunables 24 12 8 > : slabdata 51 51 12 > avc_node 13 378 72 54 1 : tunables 120 60 8 > : slabdata 7 7 0 > key_jar 8 20 192 20 1 : tunables 120 60 8 > : slabdata 1 1 0 > idr_layer_cache 101 105 528 7 1 : tunables 54 27 8 > : slabdata 15 15 0 > buffer_head 53280 53415 88 45 1 : tunables 120 60 8 > : slabdata 1187 1187 0 > mm_struct 140 147 1152 7 2 : tunables 24 12 8 > : slabdata 21 21 36 > vm_area_struct 5098 5478 176 22 1 : tunables 120 60 8 > : slabdata 249 249 240 > fs_cache 316 427 64 61 1 : tunables 120 60 8 > : slabdata 7 7 60 > files_cache 194 216 832 9 2 : tunables 54 27 8 > : slabdata 24 24 27 > signal_cache 297 360 256 15 1 : tunables 120 60 8 > : slabdata 24 24 0 > sighand_cache 178 183 2112 3 2 : tunables 24 12 8 > : slabdata 61 61 0 > task_struct 213 218 1984 2 1 : tunables 24 12 8 > : slabdata 109 109 12 > anon_vma 1960 2184 24 156 1 : tunables 120 60 8 > : slabdata 14 14 60 > shared_policy_node 0 0 56 69 1 : tunables 120 60 > 8 : slabdata 0 0 0 > numa_policy 44 450 16 225 1 : tunables 120 60 8 > : slabdata 2 2 0 > size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 > : slabdata 0 0 0 > size-131072 1 1 131072 1 32 : tunables 8 4 0 > : slabdata 1 1 0 > size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 > : slabdata 0 0 0 > size-65536 1 1 65536 1 16 : tunables 8 4 0 > : slabdata 1 1 0 > size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 > : slabdata 0 0 0 > size-32768 4 4 32768 1 8 : tunables 8 4 0 > : slabdata 4 4 0 > size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 > : slabdata 0 0 0 > size-16384 3 3 16384 1 4 : tunables 8 4 0 > : slabdata 3 3 0 > size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 > : slabdata 0 0 0 > size-8192 56 56 8192 1 2 : tunables 8 4 0 > : slabdata 56 56 0 > size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 > : slabdata 0 0 0 > size-4096 172 172 4096 1 1 : tunables 24 12 8 > : slabdata 172 172 0 > size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 > : slabdata 0 0 0 > size-2048 113 114 2048 2 1 : tunables 24 12 8 > : slabdata 57 57 0 > size-1620(DMA) 0 0 1664 4 2 : tunables 24 12 8 > : slabdata 0 0 0 > size-1620 53 60 1664 4 2 : tunables 24 12 8 > : slabdata 15 15 0 > size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 > : slabdata 0 0 0 > size-1024 1126 1240 1024 4 1 : tunables 54 27 8 > : slabdata 310 310 81 > size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 > : slabdata 0 0 0 > size-512 2060 2064 512 8 1 : tunables 54 27 8 > : slabdata 258 258 65 > size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-256 1137540 1137540 256 15 1 : tunables 120 60 > 8 : slabdata 75836 75836 0 > size-128(DMA) 0 0 128 31 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-128 2335 2790 128 31 1 : tunables 120 60 8 > : slabdata 90 90 30 > size-64(DMA) 0 0 64 61 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-64 8619 11163 64 61 1 : tunables 120 60 8 > : slabdata 183 183 90 > size-32(DMA) 0 0 32 119 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-32 2697 3570 32 119 1 : tunables 120 60 8 > : slabdata 30 30 60 > kmem_cache 180 180 256 15 1 : tunables 120 60 8 > : slabdata 12 12 0 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From diggercheer at gmail.com Wed Apr 11 16:24:57 2007 From: diggercheer at gmail.com (David M) Date: Wed, 11 Apr 2007 11:24:57 -0500 Subject: [Linux-cluster] Re: rgmanager or clustat problem In-Reply-To: References: Message-ID: It seems that rgmanager is failing to report because it is busy. Since I am running 20 to 25 services on each node, perhaps I can increase the poll interval from 30 seconds to one minute in /usr/share/cluster/script.sh. Maybe the cluster suite is not properly configured for so many services on each node. Do you think that this might help clustat to report? Thank you for your help. On 4/9/07, David M wrote: > > > I am running a four node GFS cluster with about 20 services per node. All > four nodes belong to the same failover domain, and they each have a priority > of 1. My shared storage is an iSCSI SAN. > > After rgmanager has been running for a couple of days, clustat produces > the following result on all four nodes: > > Timed out waiting for a response from Resource Group Manager > Member Status: Quorate > > Member Name Status > ------ ---- ------ > node01 Online, rgmanager > node02 Online, Local, rgmanager > node03 Online, rgmanager > node04 Online, rgmanager > > I also get a time out when I try to determine the status of a particular > service with "clustat -s servicename". > > All of the services seem to be up and running, but clustat does not work. > Is there something wrong? Is there a way for me to increase the time out? > > clurgmgrd and dlm_recvd seem to be using a lot of CPU cycles on Node02, 40 > and 60 percent, respectively. > > Thank you for your help. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhh at redhat.com Thu Apr 5 21:03:18 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 5 Apr 2007 17:03:18 -0400 Subject: [Linux-cluster] Starting with DLM In-Reply-To: References: Message-ID: <20070405210317.GA14667@redhat.com> On Thu, Apr 05, 2007 at 09:37:12PM +0200, Christos Triantafillou wrote: > Hello, > > What are the necessary steps to configure DLM on RHEL4 and use it from an > application? (a) set up a cluster the normal way (b) write code You'll have to create/open a lockspace first (APIs are provided to do that). Here's some example code with synchronous wrappers around the DLM APIs: http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/rgmanager/src/clulib/lock.c?rev=1.3.4.1&content-type=text/x-cvsweb-markup&cvsroot=cluster And the original DLM test code: http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/dlm/tests/usertest/?cvsroot=cluster -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Mon Apr 9 20:20:56 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 9 Apr 2007 16:20:56 -0400 Subject: [Linux-cluster] generic error on gfs status In-Reply-To: References: Message-ID: <20070409202056.GB23134@redhat.com> On Mon, Apr 09, 2007 at 12:39:33PM -0500, David M wrote: > I am running a four node GFS cluster with about 20 services per node. All > four nodes belong to the same failover domain, and they each have a priority > of 1. My shared storage is an iSCSI SAN (on a dedicated switch). > > Over the last 24 hours, /gfsdata has logged 90 "notices" in the daemon log > stating that the gfs "status" check returned a generic error: > clurgmgrd[6074]: status on clusterfs "/gfsdata" returned 1 (generic > error) > clurgmgrd[6074]: Stopping service service1_03 > clurgmgrd[6074]: Service service1_03 is recovering > clurgmgrd[6074]: Recovering failed service service1_03 > clurgmgrd[6074]: Service service1_03 started > > So, everytime /gfsdata returns a generic error, the rgmanager restarts a > service. > > Can anyone shed any light on why I might be losing my mount point or why gfs > is returning a 1? > dlm_lkb 3318298 3318298 232 17 1 : tunables 120 60 8 > : slabdata 195194 195194 0 ^^ That is definitely 212644; use aforementioned packages to fix. It is actually likely that in your case, the status check is failing because of it because you're using GFS as part of a service. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Mon Apr 9 20:16:16 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 9 Apr 2007 16:16:16 -0400 Subject: [Linux-cluster] rgmanager or clustat problem In-Reply-To: References: Message-ID: <20070409201616.GA23134@redhat.com> On Mon, Apr 09, 2007 at 12:22:26PM -0500, David M wrote: > I am running a four node GFS cluster with about 20 services per node. All > four nodes belong to the same failover domain, and they each have a priority > of 1. My shared storage is an iSCSI SAN. > > After rgmanager has been running for a couple of days, clustat produces the > following result on all four nodes: > > Timed out waiting for a response from Resource Group Manager > Member Status: Quorate > > Member Name Status > ------ ---- ------ > node01 Online, rgmanager > node02 Online, Local, rgmanager > node03 Online, rgmanager > node04 Online, rgmanager > > I also get a time out when I try to determine the status of a particular > service with "clustat -s servicename". > > All of the services seem to be up and running, but clustat does not work. > Is there something wrong? Is there a way for me to increase the time out? > > clurgmgrd and dlm_recvd seem to be using a lot of CPU cycles on Node02, 40 > and 60 percent, respectively. What version of rgmanager do you have installed? It sounds like you're hitting #212644, which is fixed with packages available here: http://people.redhat.com/lhh/packages.html (It will also be fixed in the next Red Hat update, which will then trickle down to CentOS, I suspect) -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From diggercheer at gmail.com Wed Apr 11 19:40:00 2007 From: diggercheer at gmail.com (David M) Date: Wed, 11 Apr 2007 14:40:00 -0500 Subject: [Linux-cluster] generic error on gfs status In-Reply-To: <20070409202056.GB23134@redhat.com> References: <20070409202056.GB23134@redhat.com> Message-ID: I will upgrade the rgmanager from rgmanager-1.9.54-1 to rgmanager-1.9.54-3.228823test. Do you think that it is the service script that is failing the status check? Or do you think that clusterfs.sh is failing the status check? David On 4/9/07, Lon Hohberger wrote: > > > > > dlm_lkb 3318298 3318298 232 17 1 : tunables 120 > 60 8 > > : slabdata 195194 195194 0 > > ^^ That is definitely 212644; use aforementioned packages to fix. > > It is actually likely that in your case, the status check is failing > because of it because you're using GFS as part of a service. > > -- Lon > > -- > Lon Hohberger - Software Engineer - Red Hat, Inc. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From diggercheer at gmail.com Wed Apr 11 19:51:19 2007 From: diggercheer at gmail.com (David M) Date: Wed, 11 Apr 2007 14:51:19 -0500 Subject: [Linux-cluster] rgmanager or clustat problem In-Reply-To: <20070409201616.GA23134@redhat.com> References: <20070409201616.GA23134@redhat.com> Message-ID: I am using rgmanager-1.9.54-1. I will upgrade the rgmanager from rgmanager-1.9.54-1 to rgmanager-1.9.54-3.228823test. Don't you mean 212634? https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=212634 This is a dlm lock leak created by rgmanager (the clu_lock_verbose() function). David On 4/9/07, Lon Hohberger wrote: > > On Mon, Apr 09, 2007 at 12:22:26PM -0500, David M wrote: > > clurgmgrd and dlm_recvd seem to be using a lot of CPU cycles on Node02, > 40 > > and 60 percent, respectively. > > What version of rgmanager do you have installed? It sounds like you're > hitting #212644, which is fixed with packages available here: > > http://people.redhat.com/lhh/packages.html > > (It will also be fixed in the next Red Hat update, which will then > trickle down to CentOS, I suspect) > > -- Lon > > -- > Lon Hohberger - Software Engineer - Red Hat, Inc. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alain.Moulle at bull.net Thu Apr 12 09:03:40 2007 From: Alain.Moulle at bull.net (Alain Moulle) Date: Thu, 12 Apr 2007 11:03:40 +0200 Subject: [Linux-cluster] Re: CS4 Update4 / pb with a cluster.conf built with GUI (Jos Vos) Message-ID: <461DF5EC.2070303@bull.net> Thanks Jos The workaround indicated in bugzilla is : """"" Fix the Relax-NG scheme or put back the "return" as first statement in the check_xml() function in CommandHandler.py, as it existed in previous versions. """"" but I've checked in some "previous" versions, I can't see any "return" as first line of check_xml function to bypass it ... But I'll try this workaround Alain > This is probably the same problem as the bug I reported last year > October: > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=210611 > I don't know why RH does not distribute bug fixes for real problems > like this, annoying for many people (since then, I always patch the > Python file as I described in the bug report). > Anyway, RHEL4 U5 is around the corner I think... From mcurry at redhat.com Thu Apr 12 12:25:21 2007 From: mcurry at redhat.com (Marc Curry) Date: Thu, 12 Apr 2007 08:25:21 -0400 Subject: [Linux-cluster] /proc/cluster gone? Message-ID: <461E2531.9030501@redhat.com> I see in RHEL5.0 that /proc/cluster is now gone. Were the tunables that were in there moved into the XML, other, or are they gone? (e.g. hello_timer, deadnode_timer, ...). Thanks, -Marc -- Marc Curry, RHCA, RHCX Instructor, GLS Mobile: +1 919 280 3039 gpg: 1024D/25AFA662 029F 542C 081B 9CB1 3220 E15F 9F76 C7E1 25AF A662 From pcaulfie at redhat.com Thu Apr 12 12:40:11 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Thu, 12 Apr 2007 13:40:11 +0100 Subject: [Linux-cluster] /proc/cluster gone? In-Reply-To: <461E2531.9030501@redhat.com> References: <461E2531.9030501@redhat.com> Message-ID: <461E28AB.4020100@redhat.com> Marc Curry wrote: > I see in RHEL5.0 that /proc/cluster is now gone. Were the tunables that > were in there moved into the XML, other, or are they gone? (e.g. > hello_timer, deadnode_timer, ...). > They are all gone. Partly because the cluster manager is in user space, but mainly because they don't make any sense in the RHEL5 cluster manager. RHEL5 cluster is based on openais. You can tweak the openais configuration variables in cluster.conf (as you can with RHEL4 actually). see http://people.redhat.com/pcaulfie/docs/aiscman.pdf and the man page for openais.conf -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From scott.mcclanahan at trnswrks.com Thu Apr 12 12:45:35 2007 From: scott.mcclanahan at trnswrks.com (Scott McClanahan) Date: Thu, 12 Apr 2007 08:45:35 -0400 Subject: [Linux-cluster] bonding Message-ID: <1176381935.31564.19.camel@jarjar.trnswrks.com> I have every node in my four node cluster setup to do active-backup bonding and the drivers loaded for the bonded network interfaces vary between tg3 and e100. All interfaces with the e100 driver loaded report errors much like what you see here: bonding: bond0: link status definitely down for interface eth2, disabling it e100: eth2: e100_watchdog: link up, 100Mbps, full-duplex bonding: bond0: link status definitely up for interface eth2. This happens all day on every node. I have configured the bonding module to do MII link monitoring at a frequency of 100 milliseconds and it is using basic carrier link detection to test if the interface is alive or not. There was no custom building of any modules on these nodes and the o/s is CentOS 4.3. Some more relevant information is below (this display is consistent across all nodes): [smccl at tf35 ~]$uname -srvmpio Linux 2.6.9-34.ELhugemem #1 SMP Wed Mar 8 00:47:12 CST 2006 i686 i686 i386 GNU/Linux [smccl at tf35 ~]$head -5 /etc/modprobe.conf alias bond0 bonding options bonding miimon=100 mode=1 alias eth0 tg3 alias eth1 tg3 alias eth2 e100 [smccl at tf35 ~]$cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v2.6.1 (October 29, 2004) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:10:18:0c:86:a4 Slave Interface: eth2 MII Status: up Link Failure Count: 12 Permanent HW addr: 00:02:55:ac:a2:ea Any idea why these e100 links report failures so often? They are directly plugged into a Cisco Catalyst 4506. Thanks. From rhurst at bidmc.harvard.edu Thu Apr 12 13:52:47 2007 From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu) Date: Thu, 12 Apr 2007 09:52:47 -0400 Subject: [Linux-cluster] bonding In-Reply-To: <1176381935.31564.19.camel@jarjar.trnswrks.com> References: <1176381935.31564.19.camel@jarjar.trnswrks.com> Message-ID: <1176385967.3751.19.camel@WSBID06223> I have the same hardware configuration for 11 nodes, but without any of the spurious failover events. The main thing different I had to do was to increase the bond device count to 2 (the driver defaults to only 1), as I have mine teamed between dual tg3/e1000 ports from the mobo and PCI card. bond0 is on a gigabit switch, while bond1 is on 100mb. In /etc/modprobe.conf: alias bond0 bonding alias bond1 bonding options bonding max_bonds=2 mode=1 miimon=100 updelay=200 alias eth0 e1000 alias eth1 e1000 alias eth2 tg3 alias eth3 tg3 So eth0/eth2 are teamed, and eth1/eth3 are teamed. In dmesg: e1000: eth0: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex bonding: bond0: making interface eth0 the new active one 0 ms earlier. bonding: bond0: enslaving eth0 as an active interface with an up link. bonding: bond0: enslaving eth2 as a backup interface with a down link. tg3: eth2: Link is up at 1000 Mbps, full duplex. tg3: eth2: Flow control is on for TX and on for RX. bonding: bond0: link status up for interface eth2, enabling it in 200 ms. bonding: bond0: link status definitely up for interface eth2. e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex bonding: bond1: making interface eth1 the new active one 0 ms earlier. bonding: bond1: enslaving eth1 as an active interface with an up link. bonding: bond1: enslaving eth3 as a backup interface with a down link. bond0: duplicate address detected! tg3: eth3: Link is up at 100 Mbps, full duplex. tg3: eth3: Flow control is off for TX and off for RX. bonding: bond1: link status up for interface eth3, enabling it in 200 ms. bonding: bond1: link status definitely up for interface eth3. $ uname -srvmpio Linux 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux $ cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v2.6.3 (June 8, 2005) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 200 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:11:0a:5f:1e:0a Slave Interface: eth2 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:a7:9a:54 $ cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v2.6.3 (June 8, 2005) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth1 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 200 Down Delay (ms): 0 Slave Interface: eth1 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:11:0a:5f:1e:0b Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:a7:9a:53 On Thu, 2007-04-12 at 08:45 -0400, Scott McClanahan wrote: > I have every node in my four node cluster setup to do active-backup > bonding and the drivers loaded for the bonded network interfaces vary > between tg3 and e100. All interfaces with the e100 driver loaded report > errors much like what you see here: > > bonding: bond0: link status definitely down for interface eth2, > disabling it > e100: eth2: e100_watchdog: link up, 100Mbps, full-duplex > bonding: bond0: link status definitely up for interface eth2. > > This happens all day on every node. I have configured the bonding > module to do MII link monitoring at a frequency of 100 milliseconds and > it is using basic carrier link detection to test if the interface is > alive or not. There was no custom building of any modules on these > nodes and the o/s is CentOS 4.3. > > Some more relevant information is below (this display is consistent > across all nodes): > > [smccl at tf35 ~]$uname -srvmpio > Linux 2.6.9-34.ELhugemem #1 SMP Wed Mar 8 00:47:12 CST 2006 i686 i686 > i386 GNU/Linux > > [smccl at tf35 ~]$head -5 /etc/modprobe.conf > alias bond0 bonding > options bonding miimon=100 mode=1 > alias eth0 tg3 > alias eth1 tg3 > alias eth2 e100 > > [smccl at tf35 ~]$cat /proc/net/bonding/bond0 > Ethernet Channel Bonding Driver: v2.6.1 (October 29, 2004) > > Bonding Mode: fault-tolerance (active-backup) > Primary Slave: None > Currently Active Slave: eth0 > MII Status: up > MII Polling Interval (ms): 100 > Up Delay (ms): 0 > Down Delay (ms): 0 > > Slave Interface: eth0 > MII Status: up > Link Failure Count: 0 > Permanent HW addr: 00:10:18:0c:86:a4 > > Slave Interface: eth2 > MII Status: up > Link Failure Count: 12 > Permanent HW addr: 00:02:55:ac:a2:ea > > Any idea why these e100 links report failures so often? They are > directly plugged into a Cisco Catalyst 4506. Thanks. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > Robert Hurst, Sr. Cach? Administrator Beth Israel Deaconess Medical Center 1135 Tremont Street, REN-7 Boston, Massachusetts 02120-2140 617-754-8754 ? Fax: 617-754-8730 ? Cell: 401-787-3154 Any technology distinguishable from magic is insufficiently advanced. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2178 bytes Desc: not available URL: From scott.mcclanahan at trnswrks.com Thu Apr 12 14:19:43 2007 From: scott.mcclanahan at trnswrks.com (Scott McClanahan) Date: Thu, 12 Apr 2007 10:19:43 -0400 Subject: [Linux-cluster] bonding In-Reply-To: <1176385967.3751.19.camel@WSBID06223> References: <1176381935.31564.19.camel@jarjar.trnswrks.com> <1176385967.3751.19.camel@WSBID06223> Message-ID: <1176387583.31564.38.camel@jarjar.trnswrks.com> I don't know that I'd need to increase max_bonds since I only have one bond on each node but I have considered resorting to the old MII or ETHTOOL ioctl method to determine link state. You are running a newer kernel and I haven't checked the changelog to see what differences might be pertinent but mainly you are using e1000 drivers compared to my e100 driver. I just can't seem to associate the link status failures with any other events on the box, it's really strange. On Thu, 2007-04-12 at 09:52 -0400, rhurst at bidmc.harvard.edu wrote: > I have the same hardware configuration for 11 nodes, but without any > of the spurious failover events. The main thing different I had to do > was to increase the bond device count to 2 (the driver defaults to > only 1), as I have mine teamed between dual tg3/e1000 ports from the > mobo and PCI card. bond0 is on a gigabit switch, while bond1 is on > 100mb. In /etc/modprobe.conf: > > alias bond0 bonding > alias bond1 bonding > options bonding max_bonds=2 mode=1 miimon=100 updelay=200 > alias eth0 e1000 > alias eth1 e1000 > alias eth2 tg3 > alias eth3 tg3 > > So eth0/eth2 are teamed, and eth1/eth3 are teamed. In dmesg: > > e1000: eth0: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex > bonding: bond0: making interface eth0 the new active one 0 ms earlier. > bonding: bond0: enslaving eth0 as an active interface with an up link. > bonding: bond0: enslaving eth2 as a backup interface with a down link. > tg3: eth2: Link is up at 1000 Mbps, full duplex. > tg3: eth2: Flow control is on for TX and on for RX. > bonding: bond0: link status up for interface eth2, enabling it in 200 > ms. > bonding: bond0: link status definitely up for interface eth2. > e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex > bonding: bond1: making interface eth1 the new active one 0 ms earlier. > bonding: bond1: enslaving eth1 as an active interface with an up link. > bonding: bond1: enslaving eth3 as a backup interface with a down link. > bond0: duplicate address detected! > tg3: eth3: Link is up at 100 Mbps, full duplex. > tg3: eth3: Flow control is off for TX and off for RX. > bonding: bond1: link status up for interface eth3, enabling it in 200 > ms. > bonding: bond1: link status definitely up for interface eth3. > > $ uname -srvmpio > Linux 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 > x86_64 x86_64 GNU/Linux > > $ cat /proc/net/bonding/bond0 > Ethernet Channel Bonding Driver: v2.6.3 (June 8, 2005) > > Bonding Mode: fault-tolerance (active-backup) > Primary Slave: None > Currently Active Slave: eth0 > MII Status: up > MII Polling Interval (ms): 100 > Up Delay (ms): 200 > Down Delay (ms): 0 > > Slave Interface: eth0 > MII Status: up > Link Failure Count: 0 > Permanent HW addr: 00:11:0a:5f:1e:0a > > Slave Interface: eth2 > MII Status: up > Link Failure Count: 0 > Permanent HW addr: 00:17:a4:a7:9a:54 > > $ cat /proc/net/bonding/bond1 > Ethernet Channel Bonding Driver: v2.6.3 (June 8, 2005) > > Bonding Mode: fault-tolerance (active-backup) > Primary Slave: None > Currently Active Slave: eth1 > MII Status: up > MII Polling Interval (ms): 100 > Up Delay (ms): 200 > Down Delay (ms): 0 > > Slave Interface: eth1 > MII Status: up > Link Failure Count: 0 > Permanent HW addr: 00:11:0a:5f:1e:0b > > Slave Interface: eth3 > MII Status: up > Link Failure Count: 0 > Permanent HW addr: 00:17:a4:a7:9a:53 > > > On Thu, 2007-04-12 at 08:45 -0400, Scott McClanahan wrote: > > I have every node in my four node cluster setup to do active-backup > > bonding and the drivers loaded for the bonded network interfaces vary > > between tg3 and e100. All interfaces with the e100 driver loaded report > > errors much like what you see here: > > > > bonding: bond0: link status definitely down for interface eth2, > > disabling it > > e100: eth2: e100_watchdog: link up, 100Mbps, full-duplex > > bonding: bond0: link status definitely up for interface eth2. > > > > This happens all day on every node. I have configured the bonding > > module to do MII link monitoring at a frequency of 100 milliseconds and > > it is using basic carrier link detection to test if the interface is > > alive or not. There was no custom building of any modules on these > > nodes and the o/s is CentOS 4.3. > > > > Some more relevant information is below (this display is consistent > > across all nodes): > > > > [smccl at tf35 ~]$uname -srvmpio > > Linux 2.6.9-34.ELhugemem #1 SMP Wed Mar 8 00:47:12 CST 2006 i686 i686 > > i386 GNU/Linux > > > > [smccl at tf35 ~]$head -5 /etc/modprobe.conf > > alias bond0 bonding > > options bonding miimon=100 mode=1 > > alias eth0 tg3 > > alias eth1 tg3 > > alias eth2 e100 > > > > [smccl at tf35 ~]$cat /proc/net/bonding/bond0 > > Ethernet Channel Bonding Driver: v2.6.1 (October 29, 2004) > > > > Bonding Mode: fault-tolerance (active-backup) > > Primary Slave: None > > Currently Active Slave: eth0 > > MII Status: up > > MII Polling Interval (ms): 100 > > Up Delay (ms): 0 > > Down Delay (ms): 0 > > > > Slave Interface: eth0 > > MII Status: up > > Link Failure Count: 0 > > Permanent HW addr: 00:10:18:0c:86:a4 > > > > Slave Interface: eth2 > > MII Status: up > > Link Failure Count: 12 > > Permanent HW addr: 00:02:55:ac:a2:ea > > > > Any idea why these e100 links report failures so often? They are > > directly plugged into a Cisco Catalyst 4506. Thanks. > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > Robert Hurst, Sr. Cach? Administrator > Beth Israel Deaconess Medical Center > 1135 Tremont Street, REN-7 > Boston, Massachusetts 02120-2140 > 617-754-8754 ? Fax: 617-754-8730 ? Cell: 401-787-3154 > Any technology distinguishable from magic is insufficiently advanced. > plain text document attachment (ATT362682.txt), "ATT362682.txt" > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From redhat at watson-wilson.ca Thu Apr 12 14:39:07 2007 From: redhat at watson-wilson.ca (Neil Watson) Date: Thu, 12 Apr 2007 10:39:07 -0400 Subject: [Linux-cluster] bonding In-Reply-To: <1176385967.3751.19.camel@WSBID06223> References: <1176381935.31564.19.camel@jarjar.trnswrks.com> <1176385967.3751.19.camel@WSBID06223> Message-ID: <20070412143907.GB1053@watson-wilson.ca> Bonding can be tricky. I missed the parent request. What I've learned about bonding is that how the module is loaded makes all the difference: # Loading module: # for Red Hat AS4, may work with other 2.6 Linuxes. install bond0 /sbin/modprobe bonding -o bond0 mode=0 miimon=100 # for Red Hat AS3, may work with other 2.4 Linuxes. alias bond0 bonding options bond0 -o bonding mode=0 miimon=100 # For Redhat distributions # ifcfg-ethx DEVICE=ethx USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none # ifcfg-bond0 DEVICE=bond0 USERCTL=no ONBOOT=yes IPADDR=172.16.48.66 NETMASK=255.255.255.0 GATEWAY=172.16.48.1 http://linux-net.osdl.org/index.php/Bonding -- Neil Watson | Debian Linux System Administrator | Uptime 44 days http://watson-wilson.ca From ghudson at MIT.EDU Thu Apr 12 18:25:08 2007 From: ghudson at MIT.EDU (Greg Hudson) Date: Thu, 12 Apr 2007 14:25:08 -0400 Subject: [Linux-cluster] Cluster software availability for FC6? Message-ID: <200704121825.l3CIP8HR017472@equal-rites.mit.edu> I'm interested in setting up a cluster using GFS 1 and GNBD. MIT has a site license for RHEL 5 Advanced Platform, but I would like at least some of my cluster nodes to be running FC6 instead of RHEL 5. The Linux Cluster FAQ implies that the cluster software is available for FC6 (e.g. "Yes, but it's new and only available for RHEL5 and Fedora Core 6. It's called Conga.") but FC6 only appears to contain the basic cluster management framework (cman) and gfs2-utils. No GFS 1, no gnbd, no luci or ricci. Do FC6 users need to install from source, or am I missing something? Thanks. From lhh at redhat.com Thu Apr 12 18:26:32 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 12 Apr 2007 14:26:32 -0400 Subject: [Linux-cluster] rgmanager or clustat problem In-Reply-To: References: <20070409201616.GA23134@redhat.com> Message-ID: <20070412182632.GM1804@redhat.com> On Wed, Apr 11, 2007 at 02:51:19PM -0500, David M wrote: > I am using rgmanager-1.9.54-1. I will upgrade the rgmanager from > rgmanager-1.9.54-1 to rgmanager-1.9.54-3.228823test. > > Don't you mean 212634? > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=212634 > This is a dlm lock leak created by rgmanager (the clu_lock_verbose() > function). Yes. Oops. -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Thu Apr 12 18:31:29 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 12 Apr 2007 14:31:29 -0400 Subject: [Linux-cluster] 4 node cluster even split In-Reply-To: <20070410122000.GC26722@helsinki.fi> References: <20070410105442.GB26722@helsinki.fi> <461B786F.1070204@redhat.com> <20070410122000.GC26722@helsinki.fi> Message-ID: <20070412183129.GN1804@redhat.com> On Tue, Apr 10, 2007 at 03:20:01PM +0300, Janne Peltonen wrote: > > Now I wonder. Would it be simpler to set up a qdisk? That'd require > some consideration of the heuristics to use. And the documentation > appears sparse. Is this correct: > > -whichever partition of a partitioned cluster contains the qdisk > master, has all the qdisk votes, > > AND > > -qdisk master will always be in the partition that contains the > lowest-numbered cluster node that considers itself feasible to be in > the cluster (regardless of the opinion of the other nodes in that > partition)? Actually, no... if that's in the FAQ, it's wrong. Qdisk will elect a master, which is generally the lowest node - however, that doesn't mean that the master status is removed simply because of an even split. If you have an even split with qdisk with incorrect heuristics (or no heuristics at all with the updated test packages), the partition which currently has the master will remain quorate. > > In my scenario (failing rack), the qdisk master and the qdisk votes > would find their way to alive partition and the partition would keep > quorum. I think. > > If I were to use a simple heuristic (such as pinging a router upstream), > that wouldn't break anything in the failing-rack scenario. But what if > there was a real split in the heartbeat network, with both halves of the > cluster still seeing the upstream router (because there was no split in > the 'outer' network)? If SAN access is kept, and both halves still see > the quorum device, would the cluster halves be able to agree on the > master? Worst case is that they fence race. With the current test packages, the partition which contains the QDiskd master will (is supposed to?) fence the other partition(s). -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Thu Apr 12 18:34:21 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 12 Apr 2007 14:34:21 -0400 Subject: [Linux-cluster] RHEL 4: quorate or inquorate In-Reply-To: <2E02749DAF5338479606A056219BE10901787582@smail.crosswalkinc.com> References: <2E02749DAF5338479606A056219BE10901787582@smail.crosswalkinc.com> Message-ID: <20070412183421.GO1804@redhat.com> On Tue, Apr 10, 2007 at 12:27:29PM -0600, Dex Chen wrote: > Hi, all: > > We saw a rare situation that a 3 node cluster has 2 nodes offline, but > it maintains quorum. Is there anyone has any idea how it could happen > and what/where I should look into. > > Here is the output from clustat on one of the nodes: > > Member Status: Quorate > > Member Name Status > ------ ---- ------ > lizard01 Offline > lizard02 Online, Local, rgmanager > lizard03 Offline > > Service Name Owner (Last) State > ------- ---- ----- ------ ----- > 192.168.210.121 lizard02 started > 192.168.210.122 lizard02 started > 192.168.210.123 lizard02 started > 192.168.210.124 lizard02 started > > > We are on REHL 2.6.9-34 (update 4). Could you include: cman_tool status cman_tool nodes You can do this if you are using qdisk, but it's not generally supposed to happen. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From jparsons at redhat.com Thu Apr 12 18:29:25 2007 From: jparsons at redhat.com (Jim Parsons) Date: Thu, 12 Apr 2007 14:29:25 -0400 Subject: [Linux-cluster] Cluster software availability for FC6? References: <200704121825.l3CIP8HR017472@equal-rites.mit.edu> Message-ID: <461E7A85.3010101@redhat.com> Greg Hudson wrote: >I'm interested in setting up a cluster using GFS 1 and GNBD. MIT has >a site license for RHEL 5 Advanced Platform, but I would like at least >some of my cluster nodes to be running FC6 instead of RHEL 5. > Hi Greg, thanks for your interest. Mixing OS versions in a cluster can be a tricky business. Granted that RHEL5 is an immediate descendent of fc6 and all, everything should be OK - but there is still a broader possibility for an issue to arise when doing a hybrid thing, imho. > > >The Linux Cluster FAQ implies that the cluster software is available >for FC6 (e.g. "Yes, but it's new and only available for RHEL5 and >Fedora Core 6. It's called Conga.") but FC6 only appears to contain >the basic cluster management framework (cman) and gfs2-utils. No GFS >1, no gnbd, no luci or ricci. > All base cluster suite packages plus GFS and CLVM are built and available with fc6. The conga component you are referring to is a management interface. Its development cycle prevented its inclusion in fc6, and it won't be in fc7 for awhile, as there are issues with zope and python versioning in fc7 (conga uses pieces of zope). This very week the Conga team is working on packages compiled for fc6 that we will distribute off of sources.redhat.com/cluster/conga - there will be stable releases there as well as weeklies. I am trying to have that up and available by Monday of next week. I think it is worth checking out - if you are interested in trying out virtualization, the conga interface makes it simple to start VMs as cluster services, and even create clusters from VMs. In the meanwhile, you can use the pygtk app system-config-cluster to set up your configuration. It is in fc6. I will send announce mail to this list when the new conga repo is available. -Jim From dex.chen at crosswalkinc.com Thu Apr 12 21:08:27 2007 From: dex.chen at crosswalkinc.com (Dex Chen) Date: Thu, 12 Apr 2007 15:08:27 -0600 Subject: [Linux-cluster] RHEL 4: quorate or inquorate In-Reply-To: <20070412183421.GO1804@redhat.com> Message-ID: <2E02749DAF5338479606A056219BE109017879AD@smail.crosswalkinc.com> Thanks for the response. I figured out the cause. I used "cman_tool leave remove" when I shutdown the other 2 nodes. In this way, the remaining one/cluster had "quorum: 1, and expected vote 1". I was surprised that the "2 node cluster" special case did not prevent this. My remaining question is: Do you see any potential risks such as "split brain" when the stopped nodes come back, especially in a more-node cluster (say 5 node cluster)? Thanks, Dex -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Lon Hohberger Sent: Thursday, April 12, 2007 12:34 PM To: linux clustering Subject: Re: [Linux-cluster] RHEL 4: quorate or inquorate On Tue, Apr 10, 2007 at 12:27:29PM -0600, Dex Chen wrote: > Hi, all: > > We saw a rare situation that a 3 node cluster has 2 nodes offline, but > it maintains quorum. Is there anyone has any idea how it could happen > and what/where I should look into. > > Here is the output from clustat on one of the nodes: > > Member Status: Quorate > > Member Name Status > ------ ---- ------ > lizard01 Offline > lizard02 Online, Local, rgmanager > lizard03 Offline > > Service Name Owner (Last) State > ------- ---- ----- ------ ----- > 192.168.210.121 lizard02 started > 192.168.210.122 lizard02 started > 192.168.210.123 lizard02 started > 192.168.210.124 lizard02 started > > > We are on REHL 2.6.9-34 (update 4). Could you include: cman_tool status cman_tool nodes You can do this if you are using qdisk, but it's not generally supposed to happen. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From garylua at singnet.com.sg Fri Apr 13 02:29:14 2007 From: garylua at singnet.com.sg (garylua at singnet.com.sg) Date: Fri, 13 Apr 2007 10:29:14 +0800 (SGT) Subject: [Linux-cluster] disable cluster startup after a reboot Message-ID: <1176431354.461eeafa7eda2@discus.singnet.com.sg> Hi,i've a question which may or may not berelated to redhat clustering. Assuming I have 4 machines, cluster A consists of 2 machines and Cluster B the other 2 machines. In this scenario, cluster A is up and running and has a virtual ip assigned to the cluster. Is there any way i can prevent cluster B from starting up its cluster service when i reboot or power on the machines at cluster B when it detects that cluster A is up and running? I assume i need to insert some sort of "ping" script at linux startup to achieve my aim. As in if a machine n cluster B is starting up and when it can ping the virtual ip of cluster A, it should not start its own cluster service. But i'm really poor at scripting. Can somebody help me out on this? Thanks in advance. From pcaulfie at redhat.com Fri Apr 13 07:56:15 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Fri, 13 Apr 2007 08:56:15 +0100 Subject: [Linux-cluster] RHEL 4: quorate or inquorate In-Reply-To: <2E02749DAF5338479606A056219BE109017879AD@smail.crosswalkinc.com> References: <2E02749DAF5338479606A056219BE109017879AD@smail.crosswalkinc.com> Message-ID: <461F379F.5030100@redhat.com> Dex Chen wrote: > Thanks for the response. > > I figured out the cause. I used "cman_tool leave remove" when I shutdown > the other 2 nodes. In this way, the remaining one/cluster had "quorum: > 1, and expected vote 1". I was surprised that the "2 node cluster" > special case did not prevent this. 'cman_tool leave remove' is one of those "I know what I'm doing" commands. > My remaining question is: > Do you see any potential risks such as "split brain" when the stopped > nodes come back, especially in a more-node cluster (say 5 node cluster)? As soon as you add a node back in, it will impose its version of expected votes on the cluster and it will go inquorate again. once you have (in your example) 3 nodes in the cluster all will start working normally. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From marco at fuertux.com Fri Apr 13 10:22:53 2007 From: marco at fuertux.com (Marco Minato) Date: Fri, 13 Apr 2007 11:22:53 +0100 Subject: [Linux-cluster] fence option problem Message-ID: <461F59FD.90204@fuertux.com> Hi all guys, I've some problem to change the default action of a fence action. I want the cluster to shutdown the not-responding node instead of reboot it. I've tried to change the default action of the script "fence_apc" but it seems not to be used from the cluster (as I've renemed it and still working). I'm interested to change the action on the plugin.. How is it possible? Thanks. From janne.peltonen at helsinki.fi Fri Apr 13 10:25:19 2007 From: janne.peltonen at helsinki.fi (Janne Peltonen) Date: Fri, 13 Apr 2007 13:25:19 +0300 Subject: [Linux-cluster] 'Run exclusive' semantics Message-ID: <20070413102519.GT26722@helsinki.fi> Hi. I've been playing around with the 'Run exclusive' option of a cluster service. Seems to me that you can't /relocate/ a service to a node that already runs an exclusive service, neither can you /relocate/ the exclusive service to a node that already runs some other service. But nothing seems to prevent you from /enabling/ a service on a node that already has an exclusive service running, or /enabling/ an exclusive service on a node that already runs some other service. Is this by design? --Janne -- Janne Peltonen From Alain.Moulle at bull.net Fri Apr 13 10:58:18 2007 From: Alain.Moulle at bull.net (Alain Moulle) Date: Fri, 13 Apr 2007 12:58:18 +0200 Subject: [Linux-cluster] Agents / # Default fence action Message-ID: <461F624A.30402@bull.net> Hi Question about agents : in several agents, we can see that a "Default fence action" is defined at begining of script , such as in fence_bullpap. But for agents such as ipmilan, is there a source elsewhere where I could change the "requested operation" ? Thanks Alain Moull? From olaktionov at gmail.com Fri Apr 13 11:01:01 2007 From: olaktionov at gmail.com (Oleg Laktionov) Date: Fri, 13 Apr 2007 15:01:01 +0400 Subject: [Linux-cluster] NFS4 and RHCS Message-ID: <9289c4f30704130401y7d72520cq50e36869cf94deda@mail.gmail.com> Hello, all. Can anybody explain me how to configure a clustered NFS4 service with the Red Hat Cluster Suite? Does RHCS have a support of using NFS via tcp protocol? I've looked for examples in Internet and at Redhat.com, but there was nothing useful. There are no explanation about fields, like Name, Export, Options and so on in "NFS Mount", "NFS Client" and "NFS Export" tabs in the Official RedHat cluster suite Documentation. Which resources does I have to configure for the properly functionality of cluster? Your assistance will be greatly appreciated. Thanks. best regards, Oleg Laktionov -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From hlawatschek at atix.de Fri Apr 13 12:01:51 2007 From: hlawatschek at atix.de (Mark Hlawatschek) Date: Fri, 13 Apr 2007 14:01:51 +0200 Subject: [Linux-cluster] NFS sharedroot cluster Message-ID: <200704131401.51696.hlawatschek@atix.de> Hi, I wrote a mini howto for building up a NFS based diskless sharedroot cluster with RHEL4. If you are interested in, please have a look at http://www.open-sharedroot.org/documentation/nfs-sharedroot-mini-howto/ Any comments welcome, Mark -- Gruss / Regards, Dipl.-Ing. Mark Hlawatschek http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From jparsons at redhat.com Fri Apr 13 13:36:01 2007 From: jparsons at redhat.com (James Parsons) Date: Fri, 13 Apr 2007 09:36:01 -0400 Subject: [Linux-cluster] fence option problem In-Reply-To: <461F59FD.90204@fuertux.com> References: <461F59FD.90204@fuertux.com> Message-ID: <461F8741.5090506@redhat.com> Marco Minato wrote: > Hi all guys, > > I've some problem to change the default action of a fence action. > I want the cluster to shutdown the not-responding node instead of > reboot it. > > I've tried to change the default action of the script "fence_apc" but > it seems not to be used from the cluster (as I've renemed it and still > working). > > I'm interested to change the action on the plugin.. How is it possible? > > Thanks. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster In the conf file, in the section, under the clusternode, where you are specifying the port (outlet) # on your apc switch, include an attribute stating option="Off", and the node should stay off when fenced. If it does not, then you need to look towards the bios settings on the system. If you want, XXX out your passwords for your devices and send along your conf file. I will help you add the attribute. -Jim From jparsons at redhat.com Fri Apr 13 13:40:50 2007 From: jparsons at redhat.com (James Parsons) Date: Fri, 13 Apr 2007 09:40:50 -0400 Subject: [Linux-cluster] Agents / # Default fence action In-Reply-To: <461F624A.30402@bull.net> References: <461F624A.30402@bull.net> Message-ID: <461F8862.9040001@redhat.com> Alain Moulle wrote: >Hi > >Question about agents : > >in several agents, we can see that a >"Default fence action" >is defined at begining of script , such >as in fence_bullpap. > >But for agents such as ipmilan, is there a source >elsewhere where I could change the "requested operation" ? > >Thanks >Alain Moull? > > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > > In the section of he conf file, under the clusternode, add an attribute that says option="off". This is detailed in the man page for every agent. Dfault action is reboot. -J From rpeterso at redhat.com Fri Apr 13 13:54:05 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Fri, 13 Apr 2007 08:54:05 -0500 Subject: [Linux-cluster] NFS4 and RHCS In-Reply-To: <9289c4f30704130401y7d72520cq50e36869cf94deda@mail.gmail.com> References: <9289c4f30704130401y7d72520cq50e36869cf94deda@mail.gmail.com> Message-ID: <461F8B7D.20109@redhat.com> Oleg Laktionov wrote: > Hello, all. > > Can anybody explain me how to configure a clustered NFS4 service with > the Red Hat Cluster Suite? Does RHCS have a support of using NFS via tcp > protocol? > I've looked for examples in Internet and at Redhat.com, but there was > nothing useful. There are no explanation about fields, like Name, Export, > Options and so on in "NFS Mount", "NFS Client" and "NFS Export" tabs in the > Official RedHat cluster suite Documentation. Which resources does I have to > configure for the properly functionality of cluster? > Your assistance will be greatly appreciated. > Thanks. > > > > > best regards, > Oleg Laktionov Hi Oleg, Have you seen my "Unofficial NFS/GFS Cookbook"? http://sources.redhat.com/cluster/doc/nfscookbook.pdf Regards, Bob Peterson Red Hat Cluster Suite From marco at fuertux.com Fri Apr 13 15:12:16 2007 From: marco at fuertux.com (Marco Minato) Date: Fri, 13 Apr 2007 16:12:16 +0100 Subject: [Linux-cluster] fence option problem In-Reply-To: <461F8741.5090506@redhat.com> References: <461F59FD.90204@fuertux.com> <461F8741.5090506@redhat.com> Message-ID: <461F9DD0.60109@fuertux.com> James Parsons ha scritto: > Marco Minato wrote: > >> Hi all guys, >> >> I've some problem to change the default action of a fence action. >> I want the cluster to shutdown the not-responding node instead of >> reboot it. >> >> I've tried to change the default action of the script "fence_apc" but >> it seems not to be used from the cluster (as I've renemed it and >> still working). >> >> I'm interested to change the action on the plugin.. How is it possible? >> >> Thanks. >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > In the conf file, in the section, under the clusternode, > where you are specifying the port (outlet) # on your apc switch, > include an attribute stating option="Off", and the node should stay > off when fenced. If it does not, then you need to look towards the > bios settings on the system. > > If you want, XXX out your passwords for your devices and send along > your conf file. I will help you add the attribute. > > -Jim > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > yes.. but all the times that I modify d configuration through system-config-cluster the system change the "option" statement.. I was searching another way to do it.. Now I've changed from bios the conf and it's working. it's good for me. From jparsons at redhat.com Fri Apr 13 15:29:49 2007 From: jparsons at redhat.com (James Parsons) Date: Fri, 13 Apr 2007 11:29:49 -0400 Subject: [Linux-cluster] fence option problem In-Reply-To: <461F9DD0.60109@fuertux.com> References: <461F59FD.90204@fuertux.com> <461F8741.5090506@redhat.com> <461F9DD0.60109@fuertux.com> Message-ID: <461FA1ED.3030606@redhat.com> Marco Minato wrote: > James Parsons ha scritto: > >> Marco Minato wrote: >> >>> Hi all guys, >>> >>> I've some problem to change the default action of a fence action. >>> I want the cluster to shutdown the not-responding node instead of >>> reboot it. >>> >>> I've tried to change the default action of the script "fence_apc" >>> but it seems not to be used from the cluster (as I've renemed it and >>> still working). >>> >>> I'm interested to change the action on the plugin.. How is it possible? >>> >>> Thanks. >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> In the conf file, in the section, under the clusternode, >> where you are specifying the port (outlet) # on your apc switch, >> include an attribute stating option="Off", and the node should stay >> off when fenced. If it does not, then you need to look towards the >> bios settings on the system. >> >> If you want, XXX out your passwords for your devices and send along >> your conf file. I will help you add the attribute. >> >> -Jim >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> > yes.. but all the times that I modify d configuration through > system-config-cluster the system change the "option" statement.. I was > searching another way to do it.. Now I've changed from bios the conf > and it's working. it's good for me. Marco - are you saying that when you added the option="Off" to the conf file by hand and then started system-config-cluster, that the option="Off" attribute was removed? That would be a bug :) Which version of cluster suite and s-c-cluster are you running?? Thanks, -J > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From marco at fuertux.com Fri Apr 13 16:39:58 2007 From: marco at fuertux.com (Marco Minato) Date: Fri, 13 Apr 2007 17:39:58 +0100 Subject: [Linux-cluster] fence option problem In-Reply-To: <461FA1ED.3030606@redhat.com> References: <461F59FD.90204@fuertux.com> <461F8741.5090506@redhat.com> <461F9DD0.60109@fuertux.com> <461FA1ED.3030606@redhat.com> Message-ID: <461FB25E.8000006@fuertux.com> James Parsons ha scritto: > Marco Minato wrote: > >> James Parsons ha scritto: >> >>> Marco Minato wrote: >>> >>>> Hi all guys, >>>> >>>> I've some problem to change the default action of a fence action. >>>> I want the cluster to shutdown the not-responding node instead of >>>> reboot it. >>>> >>>> I've tried to change the default action of the script "fence_apc" >>>> but it seems not to be used from the cluster (as I've renemed it >>>> and still working). >>>> >>>> I'm interested to change the action on the plugin.. How is it >>>> possible? >>>> >>>> Thanks. >>>> >>>> -- >>>> Linux-cluster mailing list >>>> Linux-cluster at redhat.com >>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >>> >>> In the conf file, in the section, under the clusternode, >>> where you are specifying the port (outlet) # on your apc switch, >>> include an attribute stating option="Off", and the node should stay >>> off when fenced. If it does not, then you need to look towards the >>> bios settings on the system. >>> >>> If you want, XXX out your passwords for your devices and send along >>> your conf file. I will help you add the attribute. >>> >>> -Jim >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >>> >> yes.. but all the times that I modify d configuration through >> system-config-cluster the system change the "option" statement.. I >> was searching another way to do it.. Now I've changed from bios the >> conf and it's working. it's good for me. > > Marco - are you saying that when you added the option="Off" to the > conf file by hand and then started system-config-cluster, that the > option="Off" attribute was removed? That would be a bug :) > > Which version of cluster suite and s-c-cluster are you running?? > > Thanks, > > -J > >> >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > Look. I've modified now cluster.conf as down wondering option statement is ok.. I've modified part as you said, upgrade the config_version and copied the conf file in nodes and restarted cluster. now in s-c-c i've no per-node fence device configured.. if I click on "send to cluster" icon the system update the configuration as down: No fence device!! I've tried to use ccs_tool for propagating the config file with the same result.. What I'm doing bad?? :) Cluster Suite ver 4 and s-c-c ver 1.0.25. Thak you. From jparsons at redhat.com Fri Apr 13 18:38:15 2007 From: jparsons at redhat.com (James Parsons) Date: Fri, 13 Apr 2007 14:38:15 -0400 Subject: [Linux-cluster] fence option problem In-Reply-To: <461FB25E.8000006@fuertux.com> References: <461F59FD.90204@fuertux.com> <461F8741.5090506@redhat.com> <461F9DD0.60109@fuertux.com> <461FA1ED.3030606@redhat.com> <461FB25E.8000006@fuertux.com> Message-ID: <461FCE17.7040203@redhat.com> Marco Minato wrote: > James Parsons ha scritto: > >> Marco Minato wrote: >> >>> James Parsons ha scritto: >>> >>>> Marco Minato wrote: >>>> >>>>> Hi all guys, >>>>> >>>>> I've some problem to change the default action of a fence action. >>>>> I want the cluster to shutdown the not-responding node instead of >>>>> reboot it. >>>>> >>>>> I've tried to change the default action of the script "fence_apc" >>>>> but it seems not to be used from the cluster (as I've renemed it >>>>> and still working). >>>>> >>>>> I'm interested to change the action on the plugin.. How is it >>>>> possible? >>>>> >>>>> Thanks. >>>>> >>>>> -- >>>>> Linux-cluster mailing list >>>>> Linux-cluster at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>> >>>> >>>> >>>> In the conf file, in the section, under the clusternode, >>>> where you are specifying the port (outlet) # on your apc switch, >>>> include an attribute stating option="Off", and the node should stay >>>> off when fenced. If it does not, then you need to look towards the >>>> bios settings on the system. >>>> >>>> If you want, XXX out your passwords for your devices and send along >>>> your conf file. I will help you add the attribute. >>>> >>>> -Jim >>>> >>>> -- >>>> Linux-cluster mailing list >>>> Linux-cluster at redhat.com >>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>> >>>> >>> yes.. but all the times that I modify d configuration through >>> system-config-cluster the system change the "option" statement.. I >>> was searching another way to do it.. Now I've changed from bios the >>> conf and it's working. it's good for me. >> >> >> Marco - are you saying that when you added the option="Off" to the >> conf file by hand and then started system-config-cluster, that the >> option="Off" attribute was removed? That would be a bug :) >> >> Which version of cluster suite and s-c-cluster are you running?? >> >> Thanks, >> >> -J >> >>> >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> > Look. I've modified now cluster.conf as down > > > > > port="1" switch="10.0.0.10" option="Off"/> > > > > > > > port="2" switch="10.0.0.10" option="Off"/> > > > > > wondering option statement is ok.. > > I've modified part as you said, upgrade the config_version > and copied the conf file in nodes and restarted cluster. > now in s-c-c i've no per-node fence device configured.. if I click on > "send to cluster" icon the system update > the configuration as down: > > > > > > > > > > > > > No fence device!! I've tried to use ccs_tool for propagating the > config file with the same result.. > What I'm doing bad?? :) 1) The 'switch' parameter (which you have set to "10.0.0.10") is used only when you are using ganged apc switches, that is, multiple daisy-chained switches. Leave that parameter off...and it is def not supposed to be a dotted quad :) 2) Do you have a section in your conf file? There should be something that looks like this: It is possible that if the fencedevices block is missing, then the ui might be saying to itself, 'these fence instances are useless' and not including them. I will have to look at the code...but first, I want to get your cluster running - and you will need to specify fencedevices in order for that to happen. -J From ghudson at MIT.EDU Fri Apr 13 19:43:31 2007 From: ghudson at MIT.EDU (Greg Hudson) Date: Fri, 13 Apr 2007 15:43:31 -0400 Subject: [Linux-cluster] Cluster software availability for FC6? In-Reply-To: <461E7A85.3010101@redhat.com> References: <200704121825.l3CIP8HR017472@equal-rites.mit.edu> <461E7A85.3010101@redhat.com> Message-ID: <1176493411.3881.96.camel@error-messages.mit.edu> On Thu, 2007-04-12 at 14:29 -0400, Jim Parsons wrote: > Hi Greg, thanks for your interest. Mixing OS versions in a cluster can > be a tricky business. Granted that RHEL5 is an immediate descendent of > fc6 and all, everything should be OK - but there is still a broader > possibility for an issue to arise when doing a hybrid thing, imho. Thanks for your answer. I'm planning to run FC6 on all the cluster nodes (although they will begin as paravirtual guest images on an RHEL5 host), but that plan is driven by the requirements of a few nodes in particular. > All base cluster suite packages plus GFS and CLVM are built and > available with fc6. The conga component you are referring to is a > management interface. I can certainly live without Conga, but of the cluster components I see packaged with RHEL5, some other key ones appear to be missing on FC6, particularly: gndb and kmod-gnbd gfs-utils and kmod-gfs I do see cman, rgmanager, system-config-cluster, and ipvsadm. Apologies in advance if I'm just not looking in the correct places. From olaktionov at gmail.com Fri Apr 13 20:09:46 2007 From: olaktionov at gmail.com (Oleg Laktionov) Date: Sat, 14 Apr 2007 00:09:46 +0400 Subject: [Linux-cluster] NFS4 and RHCS In-Reply-To: <461F8B7D.20109@redhat.com> References: <9289c4f30704130401y7d72520cq50e36869cf94deda@mail.gmail.com> <461F8B7D.20109@redhat.com> Message-ID: <9289c4f30704131309u2a4cbed7y144daea069085f70@mail.gmail.com> Thank you, Bob. 2007/4/13, Robert Peterson : > > Oleg Laktionov wrote: > > Hello, all. > > > > Can anybody explain me how to configure a clustered NFS4 service with > > the Red Hat Cluster Suite? Does RHCS have a support of using NFS > via tcp > > protocol? > > I've looked for examples in Internet and at Redhat.com, but there was > > nothing useful. There are no explanation about fields, like Name, > Export, > > Options and so on in "NFS Mount", "NFS Client" and "NFS Export" tabs in > the > > Official RedHat cluster suite Documentation. Which resources does I have > to > > configure for the properly functionality of cluster? > > Your assistance will be greatly appreciated. > > Thanks. > > > > > > > > > > best regards, > > Oleg Laktionov > > Hi Oleg, > > Have you seen my "Unofficial NFS/GFS Cookbook"? > http://sources.redhat.com/cluster/doc/nfscookbook.pdf > > Regards, > > Bob Peterson > Red Hat Cluster Suite > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- best regards, Oleg Laktionov -------------- next part -------------- An HTML attachment was scrubbed... URL: From nirmaltom at hotmail.com Fri Apr 13 21:21:35 2007 From: nirmaltom at hotmail.com (nirmal tom) Date: Sat, 14 Apr 2007 02:51:35 +0530 Subject: [Linux-cluster] Cluster software availability for FC6? In-Reply-To: <1176493411.3881.96.camel@error-messages.mit.edu> Message-ID: hi, u can use yum -y install gfs2-utils GNBD is only needed ,only when u r in need of a shared storage. My suggestion is update ur kernel to 2.6.20 and dowload the cluster source tar ball and install it on fc6 regards, Nirmal Tom. >From: Greg Hudson >Reply-To: linux clustering >To: linux clustering >Subject: Re: [Linux-cluster] Cluster software availability for FC6? >Date: Fri, 13 Apr 2007 15:43:31 -0400 > >On Thu, 2007-04-12 at 14:29 -0400, Jim Parsons wrote: > > Hi Greg, thanks for your interest. Mixing OS versions in a cluster can > > be a tricky business. Granted that RHEL5 is an immediate descendent of > > fc6 and all, everything should be OK - but there is still a broader > > possibility for an issue to arise when doing a hybrid thing, imho. > >Thanks for your answer. I'm planning to run FC6 on all the cluster >nodes (although they will begin as paravirtual guest images on an RHEL5 >host), but that plan is driven by the requirements of a few nodes in >particular. > > > All base cluster suite packages plus GFS and CLVM are built and > > available with fc6. The conga component you are referring to is a > > management interface. > >I can certainly live without Conga, but of the cluster components I see >packaged with RHEL5, some other key ones appear to be missing on FC6, >particularly: > >gndb and kmod-gnbd >gfs-utils and kmod-gfs > >I do see cman, rgmanager, system-config-cluster, and ipvsadm. > >Apologies in advance if I'm just not looking in the correct places. > > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster From srramasw at cisco.com Sat Apr 14 00:04:18 2007 From: srramasw at cisco.com (Sridhar Ramaswamy (srramasw)) Date: Fri, 13 Apr 2007 17:04:18 -0700 Subject: [Linux-cluster] Correct GFS source for kernel 2.6.11 Message-ID: Hi folks, I know this is an old kernel version. But I've a need to build GFS against 2.6.11.11 version(from kernel.org). I tried various tarballs like cluster-1.00.00 and CVS source bundles using STABLE and RHEL4U4 tag. I'm still hitting compile errors one after another like SOCK_ZAPPED and LOCK_USE_CLNT. Appreciate any pointers. thanks, Sridhar -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlopmart at gmail.com Sat Apr 14 21:39:49 2007 From: carlopmart at gmail.com (carlopmart) Date: Sat, 14 Apr 2007 23:39:49 +0200 Subject: [Linux-cluster] gnbd module for 2.6.18-8.1.1.el5xen kernel Message-ID: <46214A25.9090802@gmail.com> hi all, Somebody knows whereis gnbd module for this kernel version?? On rhn.redhat.com only appears for 2.6.18-8.el5xen version ... Thanks. -- CL Martinez carlopmart {at} gmail {d0t} com From marco at fuertux.com Mon Apr 16 10:18:46 2007 From: marco at fuertux.com (Marco Minato) Date: Mon, 16 Apr 2007 11:18:46 +0100 Subject: [Linux-cluster] fence option problem In-Reply-To: <461FCE17.7040203@redhat.com> References: <461F59FD.90204@fuertux.com> <461F8741.5090506@redhat.com> <461F9DD0.60109@fuertux.com> <461FA1ED.3030606@redhat.com> <461FB25E.8000006@fuertux.com> <461FCE17.7040203@redhat.com> Message-ID: <46234D86.8070100@fuertux.com> James Parsons ha scritto: > Marco Minato wrote: > >> James Parsons ha scritto: >> >>> Marco Minato wrote: >>> >>>> James Parsons ha scritto: >>>> >>>>> Marco Minato wrote: >>>>> >>>>>> Hi all guys, >>>>>> >>>>>> I've some problem to change the default action of a fence action. >>>>>> I want the cluster to shutdown the not-responding node instead of >>>>>> reboot it. >>>>>> >>>>>> I've tried to change the default action of the script "fence_apc" >>>>>> but it seems not to be used from the cluster (as I've renemed it >>>>>> and still working). >>>>>> >>>>>> I'm interested to change the action on the plugin.. How is it >>>>>> possible? >>>>>> >>>>>> Thanks. >>>>>> >>>>>> -- >>>>>> Linux-cluster mailing list >>>>>> Linux-cluster at redhat.com >>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>>> >>>>> >>>>> >>>>> In the conf file, in the section, under the clusternode, >>>>> where you are specifying the port (outlet) # on your apc switch, >>>>> include an attribute stating option="Off", and the node should >>>>> stay off when fenced. If it does not, then you need to look >>>>> towards the bios settings on the system. >>>>> >>>>> If you want, XXX out your passwords for your devices and send >>>>> along your conf file. I will help you add the attribute. >>>>> >>>>> -Jim >>>>> >>>>> -- >>>>> Linux-cluster mailing list >>>>> Linux-cluster at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>>> >>>>> >>>> yes.. but all the times that I modify d configuration through >>>> system-config-cluster the system change the "option" statement.. I >>>> was searching another way to do it.. Now I've changed from bios the >>>> conf and it's working. it's good for me. >>> >>> >>> Marco - are you saying that when you added the option="Off" to the >>> conf file by hand and then started system-config-cluster, that the >>> option="Off" attribute was removed? That would be a bug :) >>> >>> Which version of cluster suite and s-c-cluster are you running?? >>> >>> Thanks, >>> >>> -J >>> >>>> >>>> >>>> -- >>>> Linux-cluster mailing list >>>> Linux-cluster at redhat.com >>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >>> >>> >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >>> >> Look. I've modified now cluster.conf as down >> >> >> >> >> > port="1" switch="10.0.0.10" option="Off"/> >> >> >> >> >> >> >> > port="2" switch="10.0.0.10" option="Off"/> >> >> >> >> >> wondering option statement is ok.. >> >> I've modified part as you said, upgrade the config_version >> and copied the conf file in nodes and restarted cluster. >> now in s-c-c i've no per-node fence device configured.. if I click on >> "send to cluster" icon the system update >> the configuration as down: >> >> >> >> >> >> >> >> >> >> >> >> >> No fence device!! I've tried to use ccs_tool for propagating the >> config file with the same result.. >> What I'm doing bad?? :) > > 1) The 'switch' parameter (which you have set to "10.0.0.10") is used > only when you are using ganged apc switches, that is, multiple > daisy-chained switches. Leave that parameter off...and it is def not > supposed to be a dotted quad :) > 2) Do you have a section in your conf file? There > should be something that looks like this: > > name="APC_fence"/> > > > It is possible that if the fencedevices block is missing, then the ui > might be saying to itself, 'these fence instances are useless' and not > including them. I will have to look at the code...but first, I want to > get your cluster running - and you will need to specify fencedevices > in order for that to happen. > > -J > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > Hi, good morning, 1) If I don't set the switch parameter the system-config-cluster program says: "A switch address must be provided for this Fence"... for me a common address is a dotted quad expression.. what can I put instead of it? 2) For sure I've the fencedevice section configured.. I said the fencing IS working.. the only thing is that I don't want the cluster restarts nodes but switch them off! If the fencedevice section is missing, fencing not works... The problem actually is that I can't setup the fence system with a power off of the nodes, even changing the fence_apc script. All the cluster is working good in my test scenario apart that little thing! Tnx From jparsons at redhat.com Mon Apr 16 12:38:10 2007 From: jparsons at redhat.com (Jim Parsons) Date: Mon, 16 Apr 2007 08:38:10 -0400 Subject: [Linux-cluster] fence option problem References: <461F59FD.90204@fuertux.com> <461F8741.5090506@redhat.com> <461F9DD0.60109@fuertux.com> <461FA1ED.3030606@redhat.com> <461FB25E.8000006@fuertux.com> <461FCE17.7040203@redhat.com> <46234D86.8070100@fuertux.com> Message-ID: <46236E32.5050801@redhat.com> Marco Minato wrote: > James Parsons ha scritto: > >> Marco Minato wrote: >> >>> James Parsons ha scritto: >>> >>>> Marco Minato wrote: >>>> >>>>> James Parsons ha scritto: >>>>> >>>>>> Marco Minato wrote: >>>>>> >>>>>>> Hi all guys, >>>>>>> >>>>>>> I've some problem to change the default action of a fence action. >>>>>>> I want the cluster to shutdown the not-responding node instead >>>>>>> of reboot it. >>>>>>> >>>>>>> I've tried to change the default action of the script >>>>>>> "fence_apc" but it seems not to be used from the cluster (as >>>>>>> I've renemed it and still working). >>>>>>> >>>>>>> I'm interested to change the action on the plugin.. How is it >>>>>>> possible? >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> -- >>>>>>> Linux-cluster mailing list >>>>>>> Linux-cluster at redhat.com >>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> In the conf file, in the section, under the clusternode, >>>>>> where you are specifying the port (outlet) # on your apc switch, >>>>>> include an attribute stating option="Off", and the node should >>>>>> stay off when fenced. If it does not, then you need to look >>>>>> towards the bios settings on the system. >>>>>> >>>>>> If you want, XXX out your passwords for your devices and send >>>>>> along your conf file. I will help you add the attribute. >>>>>> >>>>>> -Jim >>>>>> >>>>>> -- >>>>>> Linux-cluster mailing list >>>>>> Linux-cluster at redhat.com >>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>>>> >>>>>> >>>>> yes.. but all the times that I modify d configuration through >>>>> system-config-cluster the system change the "option" statement.. I >>>>> was searching another way to do it.. Now I've changed from bios >>>>> the conf and it's working. it's good for me. >>>> >>>> >>>> >>>> Marco - are you saying that when you added the option="Off" to the >>>> conf file by hand and then started system-config-cluster, that the >>>> option="Off" attribute was removed? That would be a bug :) >>>> >>>> Which version of cluster suite and s-c-cluster are you running?? >>>> >>>> Thanks, >>>> >>>> -J >>>> >>>>> >>>>> >>>>> -- >>>>> Linux-cluster mailing list >>>>> Linux-cluster at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Linux-cluster mailing list >>>> Linux-cluster at redhat.com >>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>> >>>> >>> Look. I've modified now cluster.conf as down >>> >>> >>> >>> >>> >> port="1" switch="10.0.0.10" option="Off"/> >>> >>> >>> >>> >>> >>> >>> >> port="2" switch="10.0.0.10" option="Off"/> >>> >>> >>> >>> >>> wondering option statement is ok.. >>> >>> I've modified part as you said, upgrade the config_version >>> and copied the conf file in nodes and restarted cluster. >>> now in s-c-c i've no per-node fence device configured.. if I click >>> on "send to cluster" icon the system update >>> the configuration as down: >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> No fence device!! I've tried to use ccs_tool for propagating the >>> config file with the same result.. >>> What I'm doing bad?? :) >> >> >> 1) The 'switch' parameter (which you have set to "10.0.0.10") is >> used only when you are using ganged apc switches, that is, multiple >> daisy-chained switches. Leave that parameter off...and it is def not >> supposed to be a dotted quad :) >> 2) Do you have a section in your conf file? There >> should be something that looks like this: >> >> > name="APC_fence"/> >> >> >> It is possible that if the fencedevices block is missing, then the ui >> might be saying to itself, 'these fence instances are useless' and >> not including them. I will have to look at the code...but first, I >> want to get your cluster running - and you will need to specify >> fencedevices in order for that to happen. >> >> -J >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> > Hi, good morning, > > 1) If I don't set the switch parameter the system-config-cluster > program says: "A switch address must be provided for this Fence"... > for me a common address is a dotted quad expression.. what can I put > instead of it? > 2) For sure I've the fencedevice section configured.. I said the > fencing IS working.. the only thing is that I don't want the cluster > restarts nodes but switch them off! If the fencedevice section is > missing, fencing not works... > > The problem actually is that I can't setup the fence system with a > power off of the nodes, even changing the fence_apc script. All the > cluster is working good in my test scenario apart that little thing! > > Tnx > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Put '0' in the switch field. That requirement has been fixed in a more recent version of the UI. You really should not have to change the fence_apc script at all. It is a pretty stable agent. If you wish to have the agent power the node off when fenced, rather thaan reboot it, in the conf file include 'option="Off"' attribute in the device section under the node (on the same line that says 'port="x"'). -J From lhh at redhat.com Mon Apr 16 14:35:20 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 16 Apr 2007 10:35:20 -0400 Subject: [Linux-cluster] [PATCH] Fix rgmanager + qdisk behavior problems Message-ID: <20070416143520.GA28846@redhat.com> Thanks to Simone Gotti for pointing me at this. Rgmanager thinks qdisk is a node (with node ID 0), so it tries to send VF information to node 0 - which doesn't exist, causing rgmanger to not work when qdisk is running :( -- Lon Hohberger - Software Engineer - Red Hat, Inc. -------------- next part -------------- Index: src/clulib/vft.c =================================================================== RCS file: /cvs/cluster/cluster/rgmanager/src/clulib/vft.c,v retrieving revision 1.17.2.1 diff -u -r1.17.2.1 vft.c --- src/clulib/vft.c 28 Mar 2007 14:54:47 -0000 1.17.2.1 +++ src/clulib/vft.c 16 Apr 2007 14:31:05 -0000 @@ -1152,7 +1152,8 @@ remain = 0; for (x = 0, y = 0; x < membership->cml_count; x++) { - if (membership->cml_members[x].cn_member) { + if (membership->cml_members[x].cn_nodeid && + membership->cml_members[x].cn_member) { remain++; } } Index: src/daemons/fo_domain.c =================================================================== RCS file: /cvs/cluster/cluster/rgmanager/src/daemons/fo_domain.c,v retrieving revision 1.11 diff -u -r1.11 fo_domain.c --- src/daemons/fo_domain.c 27 Sep 2006 16:28:41 -0000 1.11 +++ src/daemons/fo_domain.c 16 Apr 2007 14:31:05 -0000 @@ -340,6 +340,10 @@ ENTER(); + if (nodeid <= 0) { + RETURN(FOD_ILLEGAL); + } + /* * Um, if the node isn't online... */ Index: src/daemons/groups.c =================================================================== RCS file: /cvs/cluster/cluster/rgmanager/src/daemons/groups.c,v retrieving revision 1.25.2.3 diff -u -r1.25.2.3 groups.c --- src/daemons/groups.c 20 Mar 2007 17:09:11 -0000 1.25.2.3 +++ src/daemons/groups.c 16 Apr 2007 14:31:05 -0000 @@ -164,6 +164,8 @@ pthread_rwlock_unlock(&resource_lock); for (x=0; x < allowed->cml_count; x++) { + if (allowed->cml_members[x].cn_nodeid == 0) + continue; if (!allowed->cml_members[x].cn_member) continue; From lhh at redhat.com Mon Apr 16 14:42:18 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 16 Apr 2007 10:42:18 -0400 Subject: [Linux-cluster] [PATCH] Fix rgmanager + qdisk behavior problems In-Reply-To: <20070416143520.GA28846@redhat.com> References: <20070416143520.GA28846@redhat.com> Message-ID: <20070416144218.GB28846@redhat.com> On Mon, Apr 16, 2007 at 10:35:20AM -0400, Lon Hohberger wrote: > Thanks to Simone Gotti for pointing me at this. > > Rgmanager thinks qdisk is a node (with node ID 0), so it tries to send > VF information to node 0 - which doesn't exist, causing rgmanger to not > work when qdisk is running :( one-liner which eliminates the problem in one shot: mark qdisk as 'down' so rgmanager doesn't try to talk to it. -------------- next part -------------- Index: main.c =================================================================== RCS file: /cvs/cluster/cluster/rgmanager/src/daemons/main.c,v retrieving revision 1.34.2.2 diff -u -r1.34.2.2 main.c --- main.c 12 Apr 2007 17:23:05 -0000 1.34.2.2 +++ main.c 16 Apr 2007 14:41:54 -0000 @@ -199,6 +199,7 @@ cman_finish(h); member_list_update(new_ml); + member_set_state(0, 0); /* Mark qdisk as dead */ /* * Handle nodes lost. Do our local node event first. From christian.brandes at forschungsgruppe.de Mon Apr 16 15:05:59 2007 From: christian.brandes at forschungsgruppe.de (Christian Brandes) Date: Mon, 16 Apr 2007 17:05:59 +0200 Subject: [Linux-cluster] NFSv4 with ACL-Support on GFS In-Reply-To: <20070329160006.5850F733E2@hormel.redhat.com> References: <20070329160006.5850F733E2@hormel.redhat.com> Message-ID: <462390D7.9070808@forschungsgruppe.de> Hi Marc and Bob, >>> I would like to set up an active/active cluster of redundant NFSv4 >>> servers with ACLs that have the same GFS file system exported at the >>> same time. >>> >>> At the moment I have a two node cluster of test servers with a GFS >>> exported over NFSv4, but I can not get or set ACLs with getfacl or >>> setfacl, which is the same behavior I had with samba. >>> >>> Some weeks ago I tried two SAMBA servers on GFS, though it was said not >>> to work in the Cluster FAQ. >>> It seemed to work -- but no ACLs. >> 1. Are you mounting gfs on the nfs server with the "-o acl" option? >> 2. Can you do setfacl and getfacl from the gfs host (i.e. not through nfs)? >> 3. On what version of the cluster software and gfs are you seeing this? Sorry for the late answers: 1. Yes, /gfs is mounted with "-o acl" but not the / filesystem and that is the point. I was not aware that it is not sufficient to mount the filesystem itself "-o acl" but also the root fs. But if you do it works with NFSv4 as well as with Samba. Now that I found out I took my Samba setup from some weeks ago and setup a Samba PDC and two BDCs that offer the same GFS at a time and I am very happy that it works. Finally I configured an IP-Cluster-Resource that is failed over and use that to mount SMB-shares e.g. home directories on SMB clients. 2. Yes, I can now and I could before / was "-o acl", too. > Hi Christian, > Yes but samba - without having "shared shares" in a multiple writer > configuration - works since the beginning even with acl. You need the -o acl > and everything should be fine. OK, but what is the problem with "shared shares"? Do 2 instances of Samba on 1 server, serving the same shares cause problems? Why do they do on GFS with different machines? E.g. I nerver had problems accessing a combined Samba/NFS share by SMB an NFS at the same time. I am still going to try out different scenarios. Thanks and best regards Christian -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4348 bytes Desc: S/MIME Cryptographic Signature URL: From olaktionov at gmail.com Mon Apr 16 18:03:50 2007 From: olaktionov at gmail.com (Oleg Laktionov) Date: Mon, 16 Apr 2007 22:03:50 +0400 Subject: [Linux-cluster] NFSv4 pseudofilesystem Message-ID: <9289c4f30704161103l3ec91ed5qa9fa6afb497f0b8b@mail.gmail.com> Hello all! A little question. It is possible to make a clustered NFSv4 service equal non clustered configuration like this: /dev/sda1 ---> shared storage (GFS is not available) /dev/sdb1 ---> shared storage (GFS is not available) mkdir -p /fs/{software,video} mkdir /{software.video} mount /dev/sda1 /video mount /dev/sdb1 /software mount --bind /video /fs/video mount --bind /software /fs/software cat /etc/exports ----> /fs 192.168.1.0/24(rw,nohide,insecure,no_subtree_check,fsid=0) /fs/video 192.168.1.0/24(rw,nohide,insecure,no_subtree_check) /fs/software 192.168.1.0/24(rw,nohide,insecure,no_subtree_check) ---> Attempt to export with RHCS only one directory with fsid=0 was succesfull. This configuration ( with only one directory export ) works fine: This is my current cluster.conf: This configuration does not work. Your assistance will be greatly appreciated. Thanks. -- best regards, Oleg Laktionov -------------- next part -------------- An HTML attachment was scrubbed... URL: From grimme at atix.de Mon Apr 16 18:09:42 2007 From: grimme at atix.de (Marc Grimme) Date: Mon, 16 Apr 2007 20:09:42 +0200 Subject: [Linux-cluster] NFSv4 with ACL-Support on GFS In-Reply-To: <462390D7.9070808@forschungsgruppe.de> References: <20070329160006.5850F733E2@hormel.redhat.com> <462390D7.9070808@forschungsgruppe.de> Message-ID: <200704162009.42866.grimme@atix.de> On Monday 16 April 2007 17:05:59 Christian Brandes wrote: > Hi Marc and Bob, > > >>> I would like to set up an active/active cluster of redundant NFSv4 > >>> servers with ACLs that have the same GFS file system exported at the > >>> same time. > >>> > >>> At the moment I have a two node cluster of test servers with a GFS > >>> exported over NFSv4, but I can not get or set ACLs with getfacl or > >>> setfacl, which is the same behavior I had with samba. > >>> > >>> Some weeks ago I tried two SAMBA servers on GFS, though it was said not > >>> to work in the Cluster FAQ. > >>> It seemed to work -- but no ACLs. > >> > >> 1. Are you mounting gfs on the nfs server with the "-o acl" option? > >> 2. Can you do setfacl and getfacl from the gfs host (i.e. not through > >> nfs)? 3. On what version of the cluster software and gfs are you seeing > >> this? > > Sorry for the late answers: > > 1. Yes, /gfs is mounted with "-o acl" but not the / filesystem and that > is the point. > I was not aware that it is not sufficient to mount the filesystem itself > "-o acl" but also the root fs. > But if you do it works with NFSv4 as well as with Samba. > Now that I found out I took my Samba setup from some weeks ago and setup > a Samba PDC and two BDCs that offer the same GFS at a time and I am very > happy that it works. Finally I configured an IP-Cluster-Resource that is > failed over and use that to mount SMB-shares e.g. home directories on > SMB clients. > > 2. Yes, I can now and I could before / was "-o acl", too. > > > Hi Christian, > > Yes but samba - without having "shared shares" in a multiple writer > > configuration - works since the beginning even with acl. You need the -o > > acl and everything should be fine. > > OK, but what is the problem with "shared shares"? My understanding of "shared shares" is as follows: If both, server A and B "export" share C on path D. Then if client Z accesses share C on server A and client Y accesses share C on server A and both write to the same file you'll run into problems with dataconsistency for the data in the file. That is because samba on server A and samba on server B don't have cache coherrency or some kind of locking. IMHO that is also a problem for NFS v3/4. > Do 2 instances of Samba on 1 server, serving the same shares cause > problems? Why do they do on GFS with different machines > E.g. I nerver had problems accessing a combined Samba/NFS share by SMB > an NFS at the same time. If you have 2 instances of samba on 1 server you might end up with the same problem as state above. The same for NFS?! Only a fully disabled cache and synchronous writing might change things or even make concurrent access more stable. Regards Marc. > > I am still going to try out different scenarios. > > Thanks and best regards > Christian -- Gruss / Regards, Marc Grimme Phone: +49-89 452 3538-14 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany Registergericht: Amtsgericht M?nchen Registernummer: HRB 131682 USt.-Id.: DE209485962 Gesch?ftsf?hrung: Marc Grimme, Mark Hlawatschek, Thomas Merz From Nick.Couchman at seakr.com Mon Apr 16 20:37:25 2007 From: Nick.Couchman at seakr.com (Nick Couchman) Date: Mon, 16 Apr 2007 14:37:25 -0600 Subject: [Linux-cluster] gfs2 mount issue Message-ID: <46238A25.87A6.0099.1@seakr.com> I know this has been mentioned a few times in the list, but I haven't seen anything too recent on this issue. I'm attempting to use GFS2 and am getting some kernel bug messages when I mount the filesystems. This seems to happen with kernels 2.6.19-2.6.21-rc6-mm1 (the one I'm currently using). The first message is this: ------------[ cut here ]------------ kernel BUG at fs/gfs2/glock.c:656! invalid opcode: 0000 [#1] last sysfs file: fs/gfs2/fstest:testfs/lock_module/block Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs crc32c libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi af_packet button battery ac loop pcnet32 mii ext3 jbd dm_snapshot edd dm_mod fan thermal processor ide_generic sg BusLogic piix sd_mod scsi_mod ide_disk ide_core CPU: 0 EIP: 0060:[] Not tainted VLI EFLAGS: 00010296 (2.6.21-rc6-mm1-default #1) EIP is at gfs2_glmutex_unlock+0x1b/0x1f [gfs2] eax: c223bec8 ebx: c34cc000 ecx: 00000000 edx: c23833c0 esi: c223be84 edi: c14dbf8c ebp: 00000000 esp: c14dbf58 ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 Process gfs2_glockd (pid: 3804, ti=c14da000 task=c22e8a90 task.ti=c14da000) Stack: c0369794 c23833c0 c34cc000 d0a30e7f c34cc000 c34cc000 c26f3c94 d0a29477 00000000 00000000 00000000 00000000 00000000 c26f3c98 00000001 00000282 23ed8d84 00001337 c14dbfc0 00000000 c22e8a90 c0125c44 c14dbfb0 c14dbfb0 Call Trace: [] gfs2_reclaim_glock+0x72/0x80 [gfs2] [] gfs2_glockd+0x13/0xc0 [gfs2] [] autoremove_wake_function+0x0/0x35 [] gfs2_glockd+0x0/0xc0 [gfs2] [] kthread+0xa3/0xcc [] kthread+0x0/0xcc [] kernel_thread_helper+0x7/0x10 ======================= Code: 5e 5f 5d e9 0a ef ff ff 83 c4 0c 5b 5e 5f 5d c3 83 ec 0c 0f ba 70 08 01 c7 40 2c 00 00 00 00 c7 40 30 00 00 00 00 e8 50 f7 ff ff <0f> 0b eb fe 56 53 89 c3 83 ec 04 8d 80 44 03 00 00 39 83 44 03 EIP: [] gfs2_glmutex_unlock+0x1b/0x1f [gfs2] SS:ESP 0068:c14dbf58 followed shortly by this: ------------[ cut here ]------------ kernel BUG at fs/gfs2/glock.c:656! invalid opcode: 0000 [#2] last sysfs file: fs/gfs2/fstest:testfs/lock_module/block Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs crc32c libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi af_packet button battery ac loop pcnet32 mii ext3 jbd dm_snapshot edd dm_mod fan thermal processor ide_generic sg BusLogic piix sd_mod scsi_mod ide_disk ide_core CPU: 0 EIP: 0060:[] Not tainted VLI EFLAGS: 00010292 (2.6.21-rc6-mm1-default #1) EIP is at gfs2_glmutex_unlock+0x1b/0x1f [gfs2] eax: c223bf64 ebx: c223bf20 ecx: 00000001 edx: c223bc14 esi: 00000001 edi: c34cc000 ebp: d0a3125c esp: c14d9f78 ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 Process gfs2_scand (pid: 3803, ti=c14d8000 task=c22ea030 task.ti=c14d8000) Stack: c26f3c20 c223bc14 c223bf20 d0a30068 00000003 c26f3c98 00000001 00001078 c34cc000 d0a29524 00000000 d0a3018d c038ad60 c34cc000 c26f3c94 d0a29533 c26f3c94 d0a29524 c34cc000 c0125ae3 00000000 00000000 ffffffff ffffffff Call Trace: [] examine_bucket+0x38/0x59 [gfs2] [] gfs2_scand+0x0/0x2d [gfs2] [] gfs2_scand_internal+0x18/0x24 [gfs2] [] gfs2_scand+0xf/0x2d [gfs2] [] gfs2_scand+0x0/0x2d [gfs2] [] kthread+0xa3/0xcc [] kthread+0x0/0xcc [] kernel_thread_helper+0x7/0x10 ======================= Code: 5e 5f 5d e9 0a ef ff ff 83 c4 0c 5b 5e 5f 5d c3 83 ec 0c 0f ba 70 08 01 c7 40 2c 00 00 00 00 c7 40 30 00 00 00 00 e8 50 f7 ff ff <0f> 0b eb fe 56 53 89 c3 83 ec 04 8d 80 44 03 00 00 39 83 44 03 EIP: [] gfs2_glmutex_unlock+0x1b/0x1f [gfs2] SS:ESP 0068:c14d9f78 After I get those messages, I can list files, create files, and delete files. I run into problems if I try to use quotas or ACLs on the filesystem, and I can't unmount the filesystem - I have to hard reset the machine. Also, it doesn't seem to matter whether I use the lock_dlm or lock_nolock protocols - both seem to generate these messages. Nick Couchman Systems Integrator SEAKR Engineering, Inc. 6221 South Racine Circle Centennial, CO 80111 Main: (303) 790-8499 Fax: (303) 790-8720 Web: http://www.seakr.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kadlec at sunserv.kfki.hu Tue Apr 17 11:50:16 2007 From: kadlec at sunserv.kfki.hu (Kadlecsik Jozsi) Date: Tue, 17 Apr 2007 13:50:16 +0200 (MEST) Subject: [Linux-cluster] GFS over AOE without fencing? Message-ID: Hello, Assuming a dedicated VLAN which servers only AOE and GFS traffic among the Coraid boxes and the GFS hosts, is there any need for fencing at all? What are the hidden traps behind such setups? Best regards, Jozsef -- E-mail : kadlec at sunserv.kfki.hu, kadlec at blackhole.kfki.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt Address: KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary From breeves at redhat.com Tue Apr 17 12:28:51 2007 From: breeves at redhat.com (Bryn M. Reeves) Date: Tue, 17 Apr 2007 13:28:51 +0100 Subject: [Linux-cluster] GFS over AOE without fencing? In-Reply-To: References: Message-ID: <4624BD83.4070603@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Kadlecsik Jozsi wrote: > Hello, > > Assuming a dedicated VLAN which servers only AOE and GFS traffic among > the Coraid boxes and the GFS hosts, is there any need for fencing at all? > > What are the hidden traps behind such setups? > > Best regards, > Jozsef If you are using GFS on shared storage then you need fencing. Period. The only way you can guarantee data integrity in this scenario is by completely cutting a failed or misbehaving node off from the storage; either by power cycling it or having the storage reject its access. Otherwise, imagine a situation where a node hangs for some reason and is ejected from the cluster. At this point none of its locks for the shared data are valid anymore. Some time later, the node recovers from the hang and begins flushing writes to the storage -> corruption. Regardless of the storage type or networking, you need some way for the cluster to isolate failing nodes to prevent harm to the GFS volumes. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFGJL2D6YSQoMYUY94RAopXAKCP7eh0qzFLUY2H+H6kYmjXaKsYegCdHHAp 8DzUGonsXdLs9wYgNCkmt3k= =xM/U -----END PGP SIGNATURE----- From kadlec at sunserv.kfki.hu Tue Apr 17 13:05:26 2007 From: kadlec at sunserv.kfki.hu (Kadlecsik Jozsi) Date: Tue, 17 Apr 2007 15:05:26 +0200 (MEST) Subject: [Linux-cluster] GFS over AOE without fencing? In-Reply-To: <4624BD83.4070603@redhat.com> References: <4624BD83.4070603@redhat.com> Message-ID: Hi Bryn, On Tue, 17 Apr 2007, Bryn M. Reeves wrote: > > Assuming a dedicated VLAN which servers only AOE and GFS traffic among > > the Coraid boxes and the GFS hosts, is there any need for fencing at all? > > > > What are the hidden traps behind such setups? > > > > Best regards, > > Jozsef > > If you are using GFS on shared storage then you need fencing. Period. > > The only way you can guarantee data integrity in this scenario is by > completely cutting a failed or misbehaving node off from the storage; > either by power cycling it or having the storage reject its access. > > Otherwise, imagine a situation where a node hangs for some reason and is > ejected from the cluster. At this point none of its locks for the shared > data are valid anymore. Some time later, the node recovers from the hang > and begins flushing writes to the storage -> corruption. I see - the sentences above make much more clearer why fencing is needed. Thank you the explanation - and the hint for the possibility to reject the access at the storage itself! Best regards, Jozsef -- E-mail : kadlec at sunserv.kfki.hu, kadlec at blackhole.kfki.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt Address: KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary From breeves at redhat.com Tue Apr 17 14:23:04 2007 From: breeves at redhat.com (Bryn M. Reeves) Date: Tue, 17 Apr 2007 15:23:04 +0100 Subject: [Linux-cluster] GFS over AOE without fencing? In-Reply-To: References: <4624BD83.4070603@redhat.com> Message-ID: <4624D848.10203@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Kadlecsik Jozsi wrote: >> If you are using GFS on shared storage then you need fencing. Period. >> >> The only way you can guarantee data integrity in this scenario is by >> completely cutting a failed or misbehaving node off from the storage; >> either by power cycling it or having the storage reject its access. >> >> Otherwise, imagine a situation where a node hangs for some reason and is >> ejected from the cluster. At this point none of its locks for the shared >> data are valid anymore. Some time later, the node recovers from the hang >> and begins flushing writes to the storage -> corruption. > > I see - the sentences above make much more clearer why fencing is needed. > Thank you the explanation - and the hint for the possibility to reject > the access at the storage itself! > > Best regards, > Jozsef Hi Jozsef, No problem! For cutting the access off at the storage there is a new fence agent in the cluster CVS called fence_scsi - you can use that if the storage supports SCSI3 reservations. I don't know if that is the case for AOE though. Otherwise you could use a regular network power switch, or maybe cook something up with a custom fencing script (I've used hacks with iptables before for Linux based iscsi devices). Kind regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFGJNhI6YSQoMYUY94RAsYKAJ42ozE78Nb4mRo/zide7PNsGTTG0ACgiHYi C2n4BK9zij/IzbnmEIFTcZw= =FGan -----END PGP SIGNATURE----- From berthiaume_wayne at emc.com Tue Apr 17 14:27:59 2007 From: berthiaume_wayne at emc.com (berthiaume_wayne at emc.com) Date: Tue, 17 Apr 2007 10:27:59 -0400 Subject: [Linux-cluster] GFS and Simultaneous Access In-Reply-To: <4624D848.10203@redhat.com> References: <4624BD83.4070603@redhat.com> <4624D848.10203@redhat.com> Message-ID: Hi Bryn. I'm curious about the fencing used with iSCSI - (I've used hacks with iptables before for Linux based iscsi devices). Also, I've been trying to figure out a way to allow all my nodes access to the same filesystem/LUN with separate directories for each one within the same filesystem for simultaneous access. Is this possible or would this be the place to use the SCSI reservations? This is being done strictly for testing. Regards, Wayne. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Bryn M. Reeves Sent: Tuesday, April 17, 2007 10:23 AM To: linux clustering Subject: Re: [Linux-cluster] GFS over AOE without fencing? -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Kadlecsik Jozsi wrote: >> If you are using GFS on shared storage then you need fencing. Period. >> >> The only way you can guarantee data integrity in this scenario is by >> completely cutting a failed or misbehaving node off from the storage; >> either by power cycling it or having the storage reject its access. >> >> Otherwise, imagine a situation where a node hangs for some reason and is >> ejected from the cluster. At this point none of its locks for the shared >> data are valid anymore. Some time later, the node recovers from the hang >> and begins flushing writes to the storage -> corruption. > > I see - the sentences above make much more clearer why fencing is needed. > Thank you the explanation - and the hint for the possibility to reject > the access at the storage itself! > > Best regards, > Jozsef Hi Jozsef, No problem! For cutting the access off at the storage there is a new fence agent in the cluster CVS called fence_scsi - you can use that if the storage supports SCSI3 reservations. I don't know if that is the case for AOE though. Otherwise you could use a regular network power switch, or maybe cook something up with a custom fencing script (I've used hacks with iptables before for Linux based iscsi devices). Kind regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFGJNhI6YSQoMYUY94RAsYKAJ42ozE78Nb4mRo/zide7PNsGTTG0ACgiHYi C2n4BK9zij/IzbnmEIFTcZw= =FGan -----END PGP SIGNATURE----- -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From jbe at webde.de Tue Apr 17 14:36:06 2007 From: jbe at webde.de (Jens Beyer) Date: Tue, 17 Apr 2007 16:36:06 +0200 Subject: [Linux-cluster] dlm spinlock BUG Message-ID: <20070417143606.GA8621@pcnocjb.dlan.cinetic.de> Hi, I am (trying) t run a GFS testcluster based on cluster-2.00.00 and 2.6.20.6/32bit vs cluster-2.00.00 and 2.6.20.6/64bit. When mounting the shared device on both nodes I get on the 32bit node: [82301.201322] BUG: spinlock already unlocked on CPU#2, dlm_recoverd/21239 [82301.201335] lock: eaf638e4, .magic: dead4ead, .owner: /-1, .owner_cpu: -1 [82301.201356] [] _raw_spin_unlock+0x70/0x72 [82301.201375] [] dlm_lowcomms_get_buffer+0x53/0xed [dlm] [82301.201408] [] create_rcom+0x2d/0xb3 [dlm] [82301.201435] [] dlm_rcom_status+0x45/0x110 [dlm] [82301.201461] [] ping_members+0x38/0x85 [dlm] [82301.201486] [] dlm_recover_members+0x19e/0x1d9 [dlm] [82301.201512] [] ls_recover+0x6b/0x2dd [dlm] [82301.201537] [] do_ls_recovery+0x62/0x80 [dlm] [82301.201558] [] dlm_recoverd+0x0/0x85 [dlm] [82301.201580] [] dlm_recoverd+0x4d/0x85 [dlm] [82301.201602] [] kthread+0x8c/0xb1 [82301.201614] [] kthread+0x0/0xb1 [82301.201625] [] kernel_thread_helper+0x7/0x14 [82301.201636] ======================= [82301.201664] dlm: connecting to 2 [82301.210513] dlm: clustervol1: config mismatch: 32,0 nodeid 2: 0,0 [82301.210526] dlm: clustervol1: ping_members aborted -22 last nodeid 2 [82301.210535] dlm: clustervol1: total members 2 error -22 [82301.210543] dlm: clustervol1: recover_members failed -22 [82301.210564] dlm: clustervol1: recover 3 error -22 Afterwards I have to reset the server (locked mount), while the second (64bit) server is still useable. Any ideas ? Regards, Jens From tmornini at engineyard.com Tue Apr 17 14:36:13 2007 From: tmornini at engineyard.com (Tom Mornini) Date: Tue, 17 Apr 2007 07:36:13 -0700 Subject: [Linux-cluster] GFS and Simultaneous Access In-Reply-To: References: <4624BD83.4070603@redhat.com> <4624D848.10203@redhat.com> Message-ID: On Apr 17, 2007, at 7:27 AM, berthiaume_wayne at emc.com wrote: > Also, I've been trying to figure out a way to allow all my nodes > access to the same filesystem/LUN with separate directories for > each one > within the same filesystem for simultaneous access. Is this > possible or > would this be the place to use the SCSI reservations? This is being > done > strictly for testing. Take a look at Context Sensitive Symlinks. If you: ln -s @host common_directory Then when each host accesses common_directory it will resolve to that system's hostname instead. -- -- Tom Mornini, CTO -- Engine Yard, Ruby on Rails Hosting -- Support, Scalability, Reliability -- (866) 518-YARD (9273) From pcaulfie at redhat.com Tue Apr 17 14:51:53 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 17 Apr 2007 15:51:53 +0100 Subject: [Linux-cluster] dlm spinlock BUG In-Reply-To: <20070417143606.GA8621@pcnocjb.dlan.cinetic.de> References: <20070417143606.GA8621@pcnocjb.dlan.cinetic.de> Message-ID: <4624DF09.9050606@redhat.com> Jens Beyer wrote: > Hi, > > I am (trying) t run a GFS testcluster based on cluster-2.00.00 and 2.6.20.6/32bit > vs cluster-2.00.00 and 2.6.20.6/64bit. > > When mounting the shared device on both nodes I get on the 32bit node: > > [82301.201322] BUG: spinlock already unlocked on CPU#2, dlm_recoverd/21239 > [82301.201335] lock: eaf638e4, .magic: dead4ead, .owner: /-1, .owner_cpu: -1 > [82301.201356] [] _raw_spin_unlock+0x70/0x72 > [82301.201375] [] dlm_lowcomms_get_buffer+0x53/0xed [dlm] > [82301.201408] [] create_rcom+0x2d/0xb3 [dlm] > [82301.201435] [] dlm_rcom_status+0x45/0x110 [dlm] > [82301.201461] [] ping_members+0x38/0x85 [dlm] > [82301.201486] [] dlm_recover_members+0x19e/0x1d9 [dlm] > [82301.201512] [] ls_recover+0x6b/0x2dd [dlm] > [82301.201537] [] do_ls_recovery+0x62/0x80 [dlm] > [82301.201558] [] dlm_recoverd+0x0/0x85 [dlm] > [82301.201580] [] dlm_recoverd+0x4d/0x85 [dlm] > [82301.201602] [] kthread+0x8c/0xb1 > [82301.201614] [] kthread+0x0/0xb1 > [82301.201625] [] kernel_thread_helper+0x7/0x14 > [82301.201636] ======================= Which kernel are you using ? -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From swhiteho at redhat.com Tue Apr 17 15:06:28 2007 From: swhiteho at redhat.com (Steven Whitehouse) Date: Tue, 17 Apr 2007 16:06:28 +0100 Subject: [Linux-cluster] GFS and Simultaneous Access In-Reply-To: References: <4624BD83.4070603@redhat.com> <4624D848.10203@redhat.com> Message-ID: <1176822389.1636.273.camel@quoit.chygwyn.com> Hi, On Tue, 2007-04-17 at 07:36 -0700, Tom Mornini wrote: > On Apr 17, 2007, at 7:27 AM, berthiaume_wayne at emc.com wrote: > > > Also, I've been trying to figure out a way to allow all my nodes > > access to the same filesystem/LUN with separate directories for > > each one > > within the same filesystem for simultaneous access. Is this > > possible or > > would this be the place to use the SCSI reservations? This is being > > done > > strictly for testing. > > Take a look at Context Sensitive Symlinks. > > If you: > > ln -s @host common_directory > > Then when each host accesses common_directory it will resolve to that > system's hostname instead. > A better plan, if possible, is to use bind mounts which are available on all filesystems and more powerful than the context sensitive symlinks. It depends on which kernel version you are using as to whether this is available to you and how comprehensive the features are, Steve. From berthiaume_wayne at emc.com Tue Apr 17 15:17:34 2007 From: berthiaume_wayne at emc.com (berthiaume_wayne at emc.com) Date: Tue, 17 Apr 2007 11:17:34 -0400 Subject: [Linux-cluster] GFS and Simultaneous Access In-Reply-To: <1176822389.1636.273.camel@quoit.chygwyn.com> References: <4624BD83.4070603@redhat.com><4624D848.10203@redhat.com> <1176822389.1636.273.camel@quoit.chygwyn.com> Message-ID: This is RHEL 4.4. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse Sent: Tuesday, April 17, 2007 11:06 AM To: linux clustering Subject: Re: [Linux-cluster] GFS and Simultaneous Access Hi, On Tue, 2007-04-17 at 07:36 -0700, Tom Mornini wrote: > On Apr 17, 2007, at 7:27 AM, berthiaume_wayne at emc.com wrote: > > > Also, I've been trying to figure out a way to allow all my nodes > > access to the same filesystem/LUN with separate directories for > > each one > > within the same filesystem for simultaneous access. Is this > > possible or > > would this be the place to use the SCSI reservations? This is being > > done > > strictly for testing. > > Take a look at Context Sensitive Symlinks. > > If you: > > ln -s @host common_directory > > Then when each host accesses common_directory it will resolve to that > system's hostname instead. > A better plan, if possible, is to use bind mounts which are available on all filesystems and more powerful than the context sensitive symlinks. It depends on which kernel version you are using as to whether this is available to you and how comprehensive the features are, Steve. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From jmfaidy at aic.fr Tue Apr 17 15:40:19 2007 From: jmfaidy at aic.fr (jmfaidy at aic.fr) Date: Tue, 17 Apr 2007 17:40:19 +0200 Subject: [Linux-cluster] clvmd / cman / fence, in order to make xen cluster Message-ID: <20070417174019.z6lzrp2hes0csg0o@webmail.aic.fr> Hi everybody, first, i'll explain my goal, and the way i think i'm suppose to do this, then i'll explain my probleme, but please, tell me if i'm wrong with the lead i follow. I want to make a xen architecture stable, redoundant with no single point of failure. So in order to make this, i think i've to have 2 dom0 linked by heartbeat, accessing to a storage fileserver (where the domU are stored) by ISCSI (software sicsi, not hardware), and i should make clustered logical volume (with clvmd-lvm2) on the exported iscsi partition. And i will also make drbd between 2 filesservers. So my problem, is at the begining of my quest (well, i allready made work xen with iscsi), when i want to use clvm on the exported iscsi partition. Since I cant start clvmd, (it tells me "clvmd could not connect to cluster manager") i work on configuring cman, i must do that, no? (no cman = no clvmd, right?) So here I have a lot of problem, right now, I have the cluster.conf on primary node (xentest.aic.fr) : then i do : on primary node : ccsd service start cman Starting cluster: Loading modules... done Mounting configfs... done Starting ccsd... done Starting cman... done Starting daemons... done Starting fencing... yes, i've a problem with fencing devide, i dont understand well how it work, even by reading :http://sources.redhat.com/cluster/doc/usage.txt http://people.mandriva.com/~aginies/doc/procdure_lvmgnbd http://www.centos.org/docs/4/html/rh-cs-en-4/s1-config-powercontroller.html http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Cluster_Administration/s2-create-fence-devices-conga-CA.html im still lost with fence devices. Well, then i do cman_tool join -c XenCluster with cman_tool nodes i see : Node Sts Inc Joined Name 1 M 20 2007-04-17 17:07:49 xentest.aic.fr 2 X 0 xentest2.aic.fr so, my second node, is not allready there, normal. On 2nd node: i launch ccsd, if i try "cman_tool join -c XenCluster" -> cman_tool: ccsd is not running or service cman start Starting cluster: Loading modules... done Mounting configfs... done Starting ccsd... done Starting cman... failed /usr/sbin/cman_tool: ccsd is not running so i'm sad, and i tried with a copy of the cluster.conf file on 2nd node, i relaunch ccsd then cman_tool join -c XenCluster seems to work, bu actually, when i do "cman_tool nodes" i have : Node Sts Inc Joined Name 1 X 0 xentest.aic.fr 2 M 36 2007-04-17 17:44:24 xentest2.aic.fr As if there were 2 cluster named XenCluster, right? So i'm really sad, an d i dont know what to do anymore, if someone have just a hint to me, it would be really great. btw : uname -r = 2.6.18-1.2798.fc6xen have a nice evenning, tx for reading me, i know, my explanation is long, and i'm sur it don't worth it. ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. From carlopmart at gmail.com Tue Apr 17 16:52:29 2007 From: carlopmart at gmail.com (carlopmart) Date: Tue, 17 Apr 2007 18:52:29 +0200 Subject: [Linux-cluster] About quorum disk and GFS2 on rhel5 Message-ID: <4624FB4D.5080605@gmail.com> hi all, I have a question about quorum disk and gfs2-utils on rhel5: a) Do i need to use a raw partition for quorum disks or formatted partition ? If I need to use a raw partition, where is rawdevices script present on rhel 4.x ?? b) about gfs2: which param do i need to put on fstab to mount gfs2 filesystem on every boot ?? I have inserted _netdev param but gfs2 startup script returns this error: GFS2: can't parse mount arguments. /sbin/mount.gfs2: error 22 mounting /dev/sda2 on /data. If I change _netdev for defaults, works ok when system is up, but when I reboot, filesystem doesn't mounts ... many thanks .. -- CL Martinez carlopmart {at} gmail {d0t} com From rpeterso at redhat.com Tue Apr 17 17:16:35 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Tue, 17 Apr 2007 12:16:35 -0500 Subject: [Linux-cluster] About quorum disk and GFS2 on rhel5 In-Reply-To: <4624FB4D.5080605@gmail.com> References: <4624FB4D.5080605@gmail.com> Message-ID: <462500F3.8060701@redhat.com> carlopmart wrote: > hi all, > > I have a question about quorum disk and gfs2-utils on rhel5: > > a) Do i need to use a raw partition for quorum disks or formatted > partition ? If I need to use a raw partition, where is rawdevices script > present on rhel 4.x ?? > b) about gfs2: which param do i need to put on fstab to mount gfs2 > filesystem on every boot ?? I have inserted _netdev param but gfs2 > startup script returns this error: GFS2: can't parse mount arguments. > /sbin/mount.gfs2: error 22 mounting /dev/sda2 on /data. > > If I change _netdev for defaults, works ok when system is up, but when > I reboot, filesystem doesn't mounts ... > > many thanks .. > Hi, Regarding (a): I'm not a qdisk expert, but I know the man page has lots of good information about this. Regarding (b): This is a known problem. There's a bugzilla here that contains a patch: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=235204 In theory, you should also be able to do: chkconfig gfs2 on to get the gfs2 startup script to run at init time, and that should take care of mounting gfs2 partitions for you. Regards, Bob Peterson Red Hat Cluster Suite From Nick.Couchman at seakr.com Tue Apr 17 17:23:06 2007 From: Nick.Couchman at seakr.com (Nick Couchman) Date: Tue, 17 Apr 2007 11:23:06 -0600 Subject: [Linux-cluster] Re: gfs2 mount issue Message-ID: <4624AE1A.87A6.0099.1@seakr.com> By the way - I found this thread on the linux-kernel mailing list that references the same sort of bug: http://lkml.org/lkml/2007/1/25/8 There was a suggestion made that this has to do with kernel preemption - I have preemption completely disabled and still get the same bug. From my very limited kernel knowledge (that is, reading the output of the bug message) it seems to have to do with spinlocks in the kernel. I've enabled spinlock debugging and I'll see if I can get any more information, but I'm just not a kernel developer. There don't seem to be any patches out in the 2.6.21-rc or the -mm branches of the kernel to fix this issue. I know this has been mentioned a few times in the list, but I haven't seen anything too recent on this issue. I'm attempting to use GFS2 and am getting some kernel bug messages when I mount the filesystems. This seems to happen with kernels 2.6.19-2.6.21-rc6-mm1 (the one I'm currently using). The first message is this: ------------[ cut here ]------------ kernel BUG at fs/gfs2/glock.c:656! invalid opcode: 0000 [#1] last sysfs file: fs/gfs2/fstest:testfs/lock_module/block Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs crc32c libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi af_packet button battery ac loop pcnet32 mii ext3 jbd dm_snapshot edd dm_mod fan thermal processor ide_generic sg BusLogic piix sd_mod scsi_mod ide_disk ide_core CPU: 0 EIP: 0060:[] Not tainted VLI EFLAGS: 00010296 (2.6.21-rc6-mm1-default #1) EIP is at gfs2_glmutex_unlock+0x1b/0x1f [gfs2] eax: c223bec8 ebx: c34cc000 ecx: 00000000 edx: c23833c0 esi: c223be84 edi: c14dbf8c ebp: 00000000 esp: c14dbf58 ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 Process gfs2_glockd (pid: 3804, ti=c14da000 task=c22e8a90 task.ti=c14da000) Stack: c0369794 c23833c0 c34cc000 d0a30e7f c34cc000 c34cc000 c26f3c94 d0a29477 00000000 00000000 00000000 00000000 00000000 c26f3c98 00000001 00000282 23ed8d84 00001337 c14dbfc0 00000000 c22e8a90 c0125c44 c14dbfb0 c14dbfb0 Call Trace: [] gfs2_reclaim_glock+0x72/0x80 [gfs2] [] gfs2_glockd+0x13/0xc0 [gfs2] [] autoremove_wake_function+0x0/0x35 [] gfs2_glockd+0x0/0xc0 [gfs2] [] kthread+0xa3/0xcc [] kthread+0x0/0xcc [] kernel_thread_helper+0x7/0x10 ======================= Code: 5e 5f 5d e9 0a ef ff ff 83 c4 0c 5b 5e 5f 5d c3 83 ec 0c 0f ba 70 08 01 c7 40 2c 00 00 00 00 c7 40 30 00 00 00 00 e8 50 f7 ff ff <0f> 0b eb fe 56 53 89 c3 83 ec 04 8d 80 44 03 00 00 39 83 44 03 EIP: [] gfs2_glmutex_unlock+0x1b/0x1f [gfs2] SS:ESP 0068:c14dbf58 followed shortly by this: ------------[ cut here ]------------ kernel BUG at fs/gfs2/glock.c:656! invalid opcode: 0000 [#2] last sysfs file: fs/gfs2/fstest:testfs/lock_module/block Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs crc32c libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi af_packet button battery ac loop pcnet32 mii ext3 jbd dm_snapshot edd dm_mod fan thermal processor ide_generic sg BusLogic piix sd_mod scsi_mod ide_disk ide_core CPU: 0 EIP: 0060:[] Not tainted VLI EFLAGS: 00010292 (2.6.21-rc6-mm1-default #1) EIP is at gfs2_glmutex_unlock+0x1b/0x1f [gfs2] eax: c223bf64 ebx: c223bf20 ecx: 00000001 edx: c223bc14 esi: 00000001 edi: c34cc000 ebp: d0a3125c esp: c14d9f78 ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 Process gfs2_scand (pid: 3803, ti=c14d8000 task=c22ea030 task.ti=c14d8000) Stack: c26f3c20 c223bc14 c223bf20 d0a30068 00000003 c26f3c98 00000001 00001078 c34cc000 d0a29524 00000000 d0a3018d c038ad60 c34cc000 c26f3c94 d0a29533 c26f3c94 d0a29524 c34cc000 c0125ae3 00000000 00000000 ffffffff ffffffff Call Trace: [] examine_bucket+0x38/0x59 [gfs2] [] gfs2_scand+0x0/0x2d [gfs2] [] gfs2_scand_internal+0x18/0x24 [gfs2] [] gfs2_scand+0xf/0x2d [gfs2] [] gfs2_scand+0x0/0x2d [gfs2] [] kthread+0xa3/0xcc [] kthread+0x0/0xcc [] kernel_thread_helper+0x7/0x10 ======================= Code: 5e 5f 5d e9 0a ef ff ff 83 c4 0c 5b 5e 5f 5d c3 83 ec 0c 0f ba 70 08 01 c7 40 2c 00 00 00 00 c7 40 30 00 00 00 00 e8 50 f7 ff ff <0f> 0b eb fe 56 53 89 c3 83 ec 04 8d 80 44 03 00 00 39 83 44 03 EIP: [] gfs2_glmutex_unlock+0x1b/0x1f [gfs2] SS:ESP 0068:c14d9f78 After I get those messages, I can list files, create files, and delete files. I run into problems if I try to use quotas or ACLs on the filesystem, and I can't unmount the filesystem - I have to hard reset the machine. Also, it doesn't seem to matter whether I use the lock_dlm or lock_nolock protocols - both seem to generate these messages. Nick Couchman Systems Integrator SEAKR Engineering, Inc. 6221 South Racine Circle Centennial, CO 80111 Main: (303) 790-8499 Fax: (303) 790-8720 Web: http://www.seakr.com Nick Couchman Systems Integrator SEAKR Engineering, Inc. 6221 South Racine Circle Centennial, CO 80111 Main: (303) 790-8499 Fax: (303) 790-8720 Web: http://www.seakr.com From bpontz at ll.mit.edu Tue Apr 17 20:24:42 2007 From: bpontz at ll.mit.edu (Brian Pontz) Date: Tue, 17 Apr 2007 16:24:42 -0400 Subject: [Linux-cluster] kernel panic In-Reply-To: <77900.45969.qm@web33210.mail.mud.yahoo.com> References: <77900.45969.qm@web33210.mail.mud.yahoo.com> Message-ID: <1176841482.5095.89.camel@dhcp86.sst.ll.mit.edu> Hi, I keep getting the below crash every few weeks. Is this a known issue? I wasnt able to find anything in bugzilla. Thanks, Brian On Thu, 2007-02-08 at 07:26 -0800, Brian Pontz wrote: > I got the following kernel panic last night on one of > my cluster nodes. Does this look like a known bug? > I'll try and give some other useful info and let me > know if you need any other info. > > OS: CentOS release 4.4 (Final) > > uname -a > Linux scylla1 2.6.9-42.0.3.ELsmp #1 SMP Fri Oct 6 > 06:21:39 CDT 2006 i686 i686 i386 GNU/Linux > > --------------------------------------------------- > Feb 7 18:28:59 scylla1 clurgmgrd: [4346]: > Executing /etc/init.d/httpd status > Feb 7 18:28:59 scylla1 clurgmgrd: [4346]: > Executing /etc/init.d/mysqld status > Feb 7 18:29:00 scylla1 kernel: Unable to handle > kernel NULL pointer dereference at virtual address > 00000000 > Feb 7 18:29:00 scylla1 kernel: printing eip: > Feb 7 18:29:00 scylla1 kernel: f8c743a6 > Feb 7 18:29:00 scylla1 kernel: *pde = 00004001 > Feb 7 18:29:00 scylla1 kernel: Oops: 0000 [#1] > Feb 7 18:29:00 scylla1 kernel: SMP > Feb 7 18:29:00 scylla1 kernel: Modules linked in: > iptable_filter ip_tables nfs nfsd exportfs lockd > nfs_acl parport_pc lp parport autofs4 i2c_dev i2c_core > lock_dlm(U) gfs > (U) lock_harness(U) dlm(U) cman(U) sunrpc dm_mirror > dm_multipath dm_mod button battery ac md5 ipv6 > uhci_hcd ehci_hcd hw_random tg3 floppy ext3 jbd cciss > sd_mod scsi_mod > Feb 7 18:29:00 scylla1 kernel: CPU: 1 > Feb 7 18:29:00 scylla1 kernel: EIP: > 0060:[] Not tainted VLI > Feb 7 18:29:00 scylla1 kernel: EFLAGS: 00010206 > (2.6.9-42.0.3.ELsmp) > Feb 7 18:29:00 scylla1 kernel: EIP is at > gfs_glock_dq+0xaf/0x16e [gfs] > Feb 7 18:29:00 scylla1 kernel: eax: d3758b30 ebx: > d3758b24 ecx: f7f46400 edx: 00000000 > Feb 7 18:29:00 scylla1 kernel: esi: 00000000 edi: > d3758b08 ebp: f6a8b41c esp: e9adae98 > Feb 7 18:29:00 scylla1 kernel: ds: 007b es: 007b > ss: 0068 > Feb 7 18:29:00 scylla1 kernel: Process ypserv (pid: > 31510, threadinfo=e9ada000 task=f387c130) > Feb 7 18:29:00 scylla1 kernel: Stack: 0219c74e > ee98019c f8ca96a0 f8945000 f6a8b41c f6a8b41c f6a8b404 > f6a8b400 > Feb 7 18:29:00 scylla1 kernel: f8c747aa > f1cb7280 f8c8945c e9adaeec f1cb7280 00000000 00000007 > f1cb7280 > Feb 7 18:29:00 scylla1 kernel: f8c894d0 > f1cb7280 f8ca98e0 efeb2e20 c016e8ac 00000000 00000000 > 00000000 > Feb 7 18:29:00 scylla1 kernel: Call Trace: > Feb 7 18:29:00 scylla1 kernel: [] > gfs_glock_dq_uninit+0x8/0x10 [gfs] > Feb 7 18:29:00 scylla1 kernel: [] > do_unflock+0x4f/0x61 [gfs] > Feb 7 18:29:00 scylla1 kernel: [] > gfs_flock+0x62/0x76 [gfs] > Feb 7 18:29:00 scylla1 kernel: [] > locks_remove_flock+0x49/0xe1 > Feb 7 18:29:00 scylla1 kernel: [] > __fput+0x41/0x100 > Feb 7 18:29:00 scylla1 kernel: [] > filp_close+0x59/0x5f > Feb 7 18:29:00 scylla1 kernel: [] > put_files_struct+0x57/0xc0 > Feb 7 18:29:00 scylla1 kernel: [] > do_exit+0x245/0x404 > Feb 7 18:29:00 scylla1 kernel: [] > sys_exit_group+0x0/0xd > Feb 7 18:29:00 scylla1 kernel: [] > syscall_call+0x7/0xb > Feb 7 18:29:00 scylla1 kernel: Code: f8 ba 57 85 c9 > f8 68 2d 82 c9 f8 8b 44 24 14 e8 e0 1e 02 00 59 5b f6 > 45 15 08 74 06 f0 0f ba 6f 08 04 f6 45 15 04 74 38 8b > 57 28 <8b > > 02 0f 18 00 90 8d 47 28 39 c2 74 0b ff 04 24 89 54 > 24 04 8b > Feb 7 18:29:00 scylla1 kernel: <0>Fatal exception: > panic in 5 seconds > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From carlopmart at gmail.com Wed Apr 18 05:44:23 2007 From: carlopmart at gmail.com (carlopmart) Date: Wed, 18 Apr 2007 07:44:23 +0200 Subject: [Linux-cluster] About quorum disk and GFS2 on rhel5 In-Reply-To: <462500F3.8060701@redhat.com> References: <4624FB4D.5080605@gmail.com> <462500F3.8060701@redhat.com> Message-ID: <4625B037.5080402@gmail.com> Robert Peterson wrote: > carlopmart wrote: >> hi all, >> >> I have a question about quorum disk and gfs2-utils on rhel5: >> >> a) Do i need to use a raw partition for quorum disks or formatted >> partition ? If I need to use a raw partition, where is rawdevices >> script present on rhel 4.x ?? >> b) about gfs2: which param do i need to put on fstab to mount gfs2 >> filesystem on every boot ?? I have inserted _netdev param but gfs2 >> startup script returns this error: GFS2: can't parse mount arguments. >> /sbin/mount.gfs2: error 22 mounting /dev/sda2 on /data. >> >> If I change _netdev for defaults, works ok when system is up, but >> when I reboot, filesystem doesn't mounts ... >> >> many thanks .. >> > > Hi, > > Regarding (a): I'm not a qdisk expert, but I know the man page has > lots of good information about this. > Regarding (b): This is a known problem. There's a bugzilla here > that contains a patch: > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=235204 > > In theory, you should also be able to do: chkconfig gfs2 on to get > the gfs2 startup script to run at init time, and that should take > care of mounting gfs2 partitions for you. > > Regards, > > Bob Peterson > Red Hat Cluster Suite > > > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > Thanks robert ... I find a solution to my problem about quorum disk, mkqdisk command... But for gfs2, will Redhat release a patch to resolve this issue?? Many thanks ... -- CL Martinez carlopmart {at} gmail {d0t} com From kadlec at sunserv.kfki.hu Wed Apr 18 07:22:42 2007 From: kadlec at sunserv.kfki.hu (Kadlecsik Jozsi) Date: Wed, 18 Apr 2007 09:22:42 +0200 (MEST) Subject: [Linux-cluster] GFS over AOE without fencing? In-Reply-To: <4624D848.10203@redhat.com> References: <4624BD83.4070603@redhat.com> <4624D848.10203@redhat.com> Message-ID: Hi, On Tue, 17 Apr 2007, Bryn M. Reeves wrote: > For cutting the access off at the storage there is a new > fence agent in the cluster CVS called fence_scsi - you can use that if > the storage supports SCSI3 reservations. I don't know if that is the > case for AOE though. The Coraid box supports MAC based access control at the level of the logical blades so it is not hard to write a fence agent which disables the access right of the failing node. Best regards, Jozsef -- E-mail : kadlec at sunserv.kfki.hu, kadlec at blackhole.kfki.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt Address: KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary From swhiteho at redhat.com Wed Apr 18 07:37:58 2007 From: swhiteho at redhat.com (Steven Whitehouse) Date: Wed, 18 Apr 2007 08:37:58 +0100 Subject: [Linux-cluster] GFS and Simultaneous Access In-Reply-To: References: <4624BD83.4070603@redhat.com> <4624D848.10203@redhat.com> <1176822389.1636.273.camel@quoit.chygwyn.com> Message-ID: <1176881878.1636.278.camel@quoit.chygwyn.com> Hi, Then it should have the basic bind mount stuff, but I don't think it has the more advanced shared subtree features. Also note that the "ro" flag will be silently ignored for bind mounts, as they take their cue from the original mount. Some other flags, e.g. nosuid, do work as you'd expect though, Steve. On Tue, 2007-04-17 at 11:17 -0400, berthiaume_wayne at emc.com wrote: > This is RHEL 4.4. > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse > Sent: Tuesday, April 17, 2007 11:06 AM > To: linux clustering > Subject: Re: [Linux-cluster] GFS and Simultaneous Access > > Hi, > > On Tue, 2007-04-17 at 07:36 -0700, Tom Mornini wrote: > > On Apr 17, 2007, at 7:27 AM, berthiaume_wayne at emc.com wrote: > > > > > Also, I've been trying to figure out a way to allow all my nodes > > > access to the same filesystem/LUN with separate directories for > > > each one > > > within the same filesystem for simultaneous access. Is this > > > possible or > > > would this be the place to use the SCSI reservations? This is being > > > > done > > > strictly for testing. > > > > Take a look at Context Sensitive Symlinks. > > > > If you: > > > > ln -s @host common_directory > > > > Then when each host accesses common_directory it will resolve to that > > system's hostname instead. > > > > A better plan, if possible, is to use bind mounts which are available on > all filesystems and more powerful than the context sensitive symlinks. > It depends on which kernel version you are using as to whether this is > available to you and how comprehensive the features are, > > Steve. > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From swhiteho at redhat.com Wed Apr 18 07:46:42 2007 From: swhiteho at redhat.com (Steven Whitehouse) Date: Wed, 18 Apr 2007 08:46:42 +0100 Subject: [Linux-cluster] Re: gfs2 mount issue In-Reply-To: <4624AE1A.87A6.0099.1@seakr.com> References: <4624AE1A.87A6.0099.1@seakr.com> Message-ID: <1176882402.1636.286.camel@quoit.chygwyn.com> Hi, I suspect its got missed since there is no bugzilla entry. It looks like a glock has been used after its been freed. I suspect that the second oops is just a consequence of the first. So the question here is really, how did the glock get freed, and yet still apparently be in the reclaim list (and it looks like also in the hash table). I've not seen this bug locally, so I guess that it might be related to the relative speeds of various operations on different hardware. Quotas on gfs2 will need some work and thats a known bug, but what you have run into seems to be unconnected with that, Steve. On Tue, 2007-04-17 at 11:23 -0600, Nick Couchman wrote: > By the way - I found this thread on the linux-kernel mailing list that references the same sort of bug: > http://lkml.org/lkml/2007/1/25/8 > > There was a suggestion made that this has to do with kernel preemption - I have preemption completely disabled and still get the same bug. From my very limited kernel knowledge (that is, reading the output of the bug message) it seems to have to do with spinlocks in the kernel. I've enabled spinlock debugging and I'll see if I can get any more information, but I'm just not a kernel developer. There don't seem to be any patches out in the 2.6.21-rc or the -mm branches of the kernel to fix this issue. > > I know this has been mentioned a few times in the list, but I haven't seen anything too recent on this issue. I'm attempting to use GFS2 and am getting some kernel bug messages when I mount the filesystems. This seems to happen with kernels 2.6.19-2.6.21-rc6-mm1 (the one I'm currently using). The first message is this: > ------------[ cut here ]------------ > kernel BUG at fs/gfs2/glock.c:656! > invalid opcode: 0000 [#1] > last sysfs file: fs/gfs2/fstest:testfs/lock_module/block > Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs crc32c libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi af_packet button battery ac loop pcnet32 mii ext3 jbd dm_snapshot edd dm_mod fan thermal processor ide_generic sg BusLogic piix sd_mod scsi_mod ide_disk ide_core > CPU: 0 > EIP: 0060:[] Not tainted VLI > EFLAGS: 00010296 (2.6.21-rc6-mm1-default #1) > EIP is at gfs2_glmutex_unlock+0x1b/0x1f [gfs2] > eax: c223bec8 ebx: c34cc000 ecx: 00000000 edx: c23833c0 > esi: c223be84 edi: c14dbf8c ebp: 00000000 esp: c14dbf58 > ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 > Process gfs2_glockd (pid: 3804, ti=c14da000 task=c22e8a90 task.ti=c14da000) > Stack: c0369794 c23833c0 c34cc000 d0a30e7f c34cc000 c34cc000 c26f3c94 d0a29477 > 00000000 00000000 00000000 00000000 00000000 c26f3c98 00000001 00000282 > 23ed8d84 00001337 c14dbfc0 00000000 c22e8a90 c0125c44 c14dbfb0 c14dbfb0 > Call Trace: > [] gfs2_reclaim_glock+0x72/0x80 [gfs2] > [] gfs2_glockd+0x13/0xc0 [gfs2] > [] autoremove_wake_function+0x0/0x35 > [] gfs2_glockd+0x0/0xc0 [gfs2] > [] kthread+0xa3/0xcc > [] kthread+0x0/0xcc > [] kernel_thread_helper+0x7/0x10 > ======================= > Code: 5e 5f 5d e9 0a ef ff ff 83 c4 0c 5b 5e 5f 5d c3 83 ec 0c 0f ba 70 08 01 c7 40 2c 00 00 00 00 c7 40 30 00 00 00 00 e8 50 f7 ff ff <0f> 0b eb fe 56 53 89 c3 83 ec 04 8d 80 44 03 00 00 39 83 44 03 > EIP: [] gfs2_glmutex_unlock+0x1b/0x1f [gfs2] SS:ESP 0068:c14dbf58 > > > followed shortly by this: > ------------[ cut here ]------------ > kernel BUG at fs/gfs2/glock.c:656! > invalid opcode: 0000 [#2] > last sysfs file: fs/gfs2/fstest:testfs/lock_module/block > Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs crc32c libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi af_packet button battery ac loop pcnet32 mii ext3 jbd dm_snapshot edd dm_mod fan thermal processor ide_generic sg BusLogic piix sd_mod scsi_mod ide_disk ide_core > CPU: 0 > EIP: 0060:[] Not tainted VLI > EFLAGS: 00010292 (2.6.21-rc6-mm1-default #1) > EIP is at gfs2_glmutex_unlock+0x1b/0x1f [gfs2] > eax: c223bf64 ebx: c223bf20 ecx: 00000001 edx: c223bc14 > esi: 00000001 edi: c34cc000 ebp: d0a3125c esp: c14d9f78 > ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 > Process gfs2_scand (pid: 3803, ti=c14d8000 task=c22ea030 task.ti=c14d8000) > Stack: c26f3c20 c223bc14 c223bf20 d0a30068 00000003 c26f3c98 00000001 00001078 > c34cc000 d0a29524 00000000 d0a3018d c038ad60 c34cc000 c26f3c94 d0a29533 > c26f3c94 d0a29524 c34cc000 c0125ae3 00000000 00000000 ffffffff ffffffff > Call Trace: > [] examine_bucket+0x38/0x59 [gfs2] > [] gfs2_scand+0x0/0x2d [gfs2] > [] gfs2_scand_internal+0x18/0x24 [gfs2] > [] gfs2_scand+0xf/0x2d [gfs2] > [] gfs2_scand+0x0/0x2d [gfs2] > [] kthread+0xa3/0xcc > [] kthread+0x0/0xcc > [] kernel_thread_helper+0x7/0x10 > ======================= > Code: 5e 5f 5d e9 0a ef ff ff 83 c4 0c 5b 5e 5f 5d c3 83 ec 0c 0f ba 70 08 01 c7 40 2c 00 00 00 00 c7 40 30 00 00 00 00 e8 50 f7 ff ff <0f> 0b eb fe 56 53 89 c3 83 ec 04 8d 80 44 03 00 00 39 83 44 03 > EIP: [] gfs2_glmutex_unlock+0x1b/0x1f [gfs2] SS:ESP 0068:c14d9f78 > > > After I get those messages, I can list files, create files, and delete files. I run into problems if I try to use quotas or ACLs on the filesystem, and I can't unmount the filesystem - I have to hard reset the machine. Also, it doesn't seem to matter whether I use the lock_dlm or lock_nolock protocols - both seem to generate these messages. > > Nick Couchman > Systems Integrator > SEAKR Engineering, Inc. > 6221 South Racine Circle > Centennial, CO 80111 > Main: (303) 790-8499 > Fax: (303) 790-8720 > Web: http://www.seakr.com > > > > > > > Nick Couchman > Systems Integrator > SEAKR Engineering, Inc. > 6221 South Racine Circle > Centennial, CO 80111 > Main: (303) 790-8499 > Fax: (303) 790-8720 > Web: http://www.seakr.com > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From jbe at webde.de Wed Apr 18 07:47:19 2007 From: jbe at webde.de (Jens Beyer) Date: Wed, 18 Apr 2007 09:47:19 +0200 Subject: [Linux-cluster] dlm spinlock BUG In-Reply-To: <4624DF09.9050606@redhat.com> References: <20070417143606.GA8621@pcnocjb.dlan.cinetic.de> <4624DF09.9050606@redhat.com> Message-ID: <20070418074719.GA19343@pcnocjb.dlan.cinetic.de> On Tue, Apr 17, 2007 at 03:51:53PM +0100, Patrick Caulfield wrote: > Jens Beyer wrote: > > Hi, > > > > I am (trying) t run a GFS testcluster based on cluster-2.00.00 and 2.6.20.6/32bit > > vs cluster-2.00.00 and 2.6.20.6/64bit. > > > > When mounting the shared device on both nodes I get on the 32bit node: > > > > [82301.201322] BUG: spinlock already unlocked on CPU#2, dlm_recoverd/21239 > > [82301.201335] lock: eaf638e4, .magic: dead4ead, .owner: /-1, .owner_cpu: -1 > > [82301.201356] [] _raw_spin_unlock+0x70/0x72 > > [82301.201375] [] dlm_lowcomms_get_buffer+0x53/0xed [dlm] > > [82301.201408] [] create_rcom+0x2d/0xb3 [dlm] > > [82301.201435] [] dlm_rcom_status+0x45/0x110 [dlm] > > [82301.201461] [] ping_members+0x38/0x85 [dlm] > > [82301.201486] [] dlm_recover_members+0x19e/0x1d9 [dlm] > > [82301.201512] [] ls_recover+0x6b/0x2dd [dlm] > > [82301.201537] [] do_ls_recovery+0x62/0x80 [dlm] > > [82301.201558] [] dlm_recoverd+0x0/0x85 [dlm] > > [82301.201580] [] dlm_recoverd+0x4d/0x85 [dlm] > > [82301.201602] [] kthread+0x8c/0xb1 > > [82301.201614] [] kthread+0x0/0xb1 > > [82301.201625] [] kernel_thread_helper+0x7/0x14 > > [82301.201636] ======================= > > Which kernel are you using ? > I am using a vanilla 2.6.20.6 (same with 2.6.20.x). Jens From raj4linux at gmail.com Wed Apr 18 09:26:39 2007 From: raj4linux at gmail.com (rajesh mishra) Date: Wed, 18 Apr 2007 14:56:39 +0530 Subject: [Linux-cluster] gfs on gnbd Message-ID: <5a8d914c0704180226g3089880arbbabd48dadb2167e@mail.gmail.com> Hi all, I have RHEL 5 operating system. I want to use the gfs2 on gnbd using bundled package in RHEL 5. The basic two node cluster and one gnbd server. Does any body know how to accomplish this..? Thanking u all in advance. With Regards Rajesh. From christian.brandes at forschungsgruppe.de Wed Apr 18 11:43:50 2007 From: christian.brandes at forschungsgruppe.de (Christian Brandes) Date: Wed, 18 Apr 2007 13:43:50 +0200 Subject: [Linux-cluster] Samba on GFS problems changing directory ACLs Message-ID: <46260476.6000008@forschungsgruppe.de> Hi, while testing Samba on a GFS, I found out some thing new: When changing properties of a Samba on GFS directory in Windows Explorer an error message is displayed. properties-->security and then any change When you click on "Accept" it is shown something like: "Permissions for could not be saved. Access denied." When you then click on "Cancel" and reopen properties-->security, you see all changes have been committed as desired. So it is not a real error, but an ugly false error message, that just causes confusion. This does not happen at all with ext3 and with GFS, only when changing directory attributes, not file attributes. Did anyone see that before? Best regards Christian From grimme at atix.de Wed Apr 18 12:07:23 2007 From: grimme at atix.de (Marc Grimme) Date: Wed, 18 Apr 2007 14:07:23 +0200 Subject: [Linux-cluster] Samba on GFS problems changing directory ACLs In-Reply-To: <46260476.6000008@forschungsgruppe.de> References: <46260476.6000008@forschungsgruppe.de> Message-ID: <200704181407.24823.grimme@atix.de> Hi, what GFS version are you using? That was a bug in old GFS versions when setting acls (IMHO kernel > 2.6.9-43.x.y shoud have this fixed). Marc. On Wednesday 18 April 2007 13:43:50 Christian Brandes wrote: > Hi, > > while testing Samba on a GFS, I found out some thing new: > > When changing properties of a Samba on GFS directory in Windows Explorer > an error message is displayed. > properties-->security and then any change > When you click on "Accept" it is shown something like: > > "Permissions for could not be saved. Access denied." > > When you then click on "Cancel" and reopen properties-->security, you > see all changes have been committed as desired. > So it is not a real error, but an ugly false error message, that just > causes confusion. > > This does not happen at all with ext3 and with GFS, only when changing > directory attributes, not file attributes. > > Did anyone see that before? > > Best regards > Christian > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Gruss / Regards, Marc Grimme Phone: +49-89 452 3538-14 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany Registergericht: Amtsgericht M?nchen Registernummer: HRB 131682 USt.-Id.: DE209485962 Gesch?ftsf?hrung: Marc Grimme, Mark Hlawatschek, Thomas Merz From rpeterso at redhat.com Wed Apr 18 13:34:07 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Wed, 18 Apr 2007 08:34:07 -0500 Subject: [Linux-cluster] About quorum disk and GFS2 on rhel5 In-Reply-To: <4625B037.5080402@gmail.com> References: <4624FB4D.5080605@gmail.com> <462500F3.8060701@redhat.com> <4625B037.5080402@gmail.com> Message-ID: <46261E4F.7020605@redhat.com> carlopmart wrote: > Robert Peterson wrote: >> Regarding (b): This is a known problem. There's a bugzilla here >> that contains a patch: >> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=235204 > But for gfs2, will Redhat release a patch to resolve this issue?? I'm not sure what you mean by "release". The patch is attached to the bugzilla, so it's already released. If you're talking about putting it into the CVS tree, yes, that is my plan. If you're talking about a hotfix for RHEL5, I don't have much control over that. I can't speak for Red Hat or influence the decisions made regarding GFS2, but at the very least, it should be in RHEL5.1. Regards, Bob Peterson Red Hat Cluster Suite From teigland at redhat.com Wed Apr 18 14:25:23 2007 From: teigland at redhat.com (David Teigland) Date: Wed, 18 Apr 2007 09:25:23 -0500 Subject: [Linux-cluster] kernel panic In-Reply-To: <1176841482.5095.89.camel@dhcp86.sst.ll.mit.edu> References: <77900.45969.qm@web33210.mail.mud.yahoo.com> <1176841482.5095.89.camel@dhcp86.sst.ll.mit.edu> Message-ID: <20070418142523.GB11556@redhat.com> On Tue, Apr 17, 2007 at 04:24:42PM -0400, Brian Pontz wrote: > Hi, > > I keep getting the below crash every few weeks. Is this a known issue? I > wasnt able to find anything in bugzilla. Could you add a new bugzilla with the info below? > On Thu, 2007-02-08 at 07:26 -0800, Brian Pontz wrote: > > I got the following kernel panic last night on one of > > my cluster nodes. Does this look like a known bug? > > I'll try and give some other useful info and let me > > know if you need any other info. > > > > OS: CentOS release 4.4 (Final) > > > > uname -a > > Linux scylla1 2.6.9-42.0.3.ELsmp #1 SMP Fri Oct 6 > > 06:21:39 CDT 2006 i686 i686 i386 GNU/Linux > > > > --------------------------------------------------- > > Feb 7 18:28:59 scylla1 clurgmgrd: [4346]: > > Executing /etc/init.d/httpd status > > Feb 7 18:28:59 scylla1 clurgmgrd: [4346]: > > Executing /etc/init.d/mysqld status > > Feb 7 18:29:00 scylla1 kernel: Unable to handle > > kernel NULL pointer dereference at virtual address > > 00000000 > > Feb 7 18:29:00 scylla1 kernel: printing eip: > > Feb 7 18:29:00 scylla1 kernel: f8c743a6 > > Feb 7 18:29:00 scylla1 kernel: *pde = 00004001 > > Feb 7 18:29:00 scylla1 kernel: Oops: 0000 [#1] > > Feb 7 18:29:00 scylla1 kernel: SMP > > Feb 7 18:29:00 scylla1 kernel: Modules linked in: > > iptable_filter ip_tables nfs nfsd exportfs lockd > > nfs_acl parport_pc lp parport autofs4 i2c_dev i2c_core > > lock_dlm(U) gfs > > (U) lock_harness(U) dlm(U) cman(U) sunrpc dm_mirror > > dm_multipath dm_mod button battery ac md5 ipv6 > > uhci_hcd ehci_hcd hw_random tg3 floppy ext3 jbd cciss > > sd_mod scsi_mod > > Feb 7 18:29:00 scylla1 kernel: CPU: 1 > > Feb 7 18:29:00 scylla1 kernel: EIP: > > 0060:[] Not tainted VLI > > Feb 7 18:29:00 scylla1 kernel: EFLAGS: 00010206 > > (2.6.9-42.0.3.ELsmp) > > Feb 7 18:29:00 scylla1 kernel: EIP is at > > gfs_glock_dq+0xaf/0x16e [gfs] > > Feb 7 18:29:00 scylla1 kernel: eax: d3758b30 ebx: > > d3758b24 ecx: f7f46400 edx: 00000000 > > Feb 7 18:29:00 scylla1 kernel: esi: 00000000 edi: > > d3758b08 ebp: f6a8b41c esp: e9adae98 > > Feb 7 18:29:00 scylla1 kernel: ds: 007b es: 007b > > ss: 0068 > > Feb 7 18:29:00 scylla1 kernel: Process ypserv (pid: > > 31510, threadinfo=e9ada000 task=f387c130) > > Feb 7 18:29:00 scylla1 kernel: Stack: 0219c74e > > ee98019c f8ca96a0 f8945000 f6a8b41c f6a8b41c f6a8b404 > > f6a8b400 > > Feb 7 18:29:00 scylla1 kernel: f8c747aa > > f1cb7280 f8c8945c e9adaeec f1cb7280 00000000 00000007 > > f1cb7280 > > Feb 7 18:29:00 scylla1 kernel: f8c894d0 > > f1cb7280 f8ca98e0 efeb2e20 c016e8ac 00000000 00000000 > > 00000000 > > Feb 7 18:29:00 scylla1 kernel: Call Trace: > > Feb 7 18:29:00 scylla1 kernel: [] > > gfs_glock_dq_uninit+0x8/0x10 [gfs] > > Feb 7 18:29:00 scylla1 kernel: [] > > do_unflock+0x4f/0x61 [gfs] > > Feb 7 18:29:00 scylla1 kernel: [] > > gfs_flock+0x62/0x76 [gfs] > > Feb 7 18:29:00 scylla1 kernel: [] > > locks_remove_flock+0x49/0xe1 > > Feb 7 18:29:00 scylla1 kernel: [] > > __fput+0x41/0x100 > > Feb 7 18:29:00 scylla1 kernel: [] > > filp_close+0x59/0x5f > > Feb 7 18:29:00 scylla1 kernel: [] > > put_files_struct+0x57/0xc0 > > Feb 7 18:29:00 scylla1 kernel: [] > > do_exit+0x245/0x404 > > Feb 7 18:29:00 scylla1 kernel: [] > > sys_exit_group+0x0/0xd > > Feb 7 18:29:00 scylla1 kernel: [] > > syscall_call+0x7/0xb > > Feb 7 18:29:00 scylla1 kernel: Code: f8 ba 57 85 c9 > > f8 68 2d 82 c9 f8 8b 44 24 14 e8 e0 1e 02 00 59 5b f6 > > 45 15 08 74 06 f0 0f ba 6f 08 04 f6 45 15 04 74 38 8b > > 57 28 <8b > > > 02 0f 18 00 90 8d 47 28 39 c2 74 0b ff 04 24 89 54 > > 24 04 8b > > Feb 7 18:29:00 scylla1 kernel: <0>Fatal exception: > > panic in 5 seconds > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From Nick.Couchman at seakr.com Wed Apr 18 14:33:40 2007 From: Nick.Couchman at seakr.com (Nick Couchman) Date: Wed, 18 Apr 2007 08:33:40 -0600 Subject: [Linux-cluster] Re: gfs2 mount issue Message-ID: <4625D7E3.87A6.0099.1@seakr.com> Well, first of all, I'm using this with an iSCSI device, so perhaps the relatively high latency of iSCSI (as compared to direct attached devices) is causing something? Also, I've turned on a bunch of debugging options (mainly having to do with spin locks and whatever else I could find about locking) and the bug messages at mount time seem to have gone away. Instead I get the following warning when I connect to an iSCSI device, followed by a very long string of kernel debug output that talks about where the hard-safe switched to hard-unsafe, etc.: [ INFO: hard-safe -> hard-unsafe lock order detected ] I'm not going to post the output, since this isn't the open-iscsi forum. I still have trouble with locks on the GFS2 volume, though - if a process has a file locked (for example, editing a file with vi or starting Samba with configuration information and shares on the GFS2 volume) I can't perform any other operations on the volume (like an ls or removing a directory) - the processes lock waiting on I/O and never come back (and I can't shut down the original process that took out the first lock. I'm not really sure how to debug this one, since I don't get any other error messages and the only other symptom is that my CPU wait time goes through the roof. I haven't tried this with anything other than iSCSI devices, so it's quite possible that some bug in the iSCSI code is causing locking problems in the GFS2 code, but I don't really know. If you need any more info from me I'm happy to provide whatever you like. Thanks for the feedback! Nick Couchman Systems Integrator SEAKR Engineering, Inc. 6221 South Racine Circle Centennial, CO 80111 Main: (303) 790-8499 Fax: (303) 790-8720 Web: http://www.seakr.com >>> On Wed, Apr 18, 2007 at 1:46 AM, Steven Whitehouse wrote: Hi, I suspect its got missed since there is no bugzilla entry. It looks like a glock has been used after its been freed. I suspect that the second oops is just a consequence of the first. So the question here is really, how did the glock get freed, and yet still apparently be in the reclaim list (and it looks like also in the hash table). I've not seen this bug locally, so I guess that it might be related to the relative speeds of various operations on different hardware. Quotas on gfs2 will need some work and thats a known bug, but what you have run into seems to be unconnected with that, Steve. On Tue, 2007-04-17 at 11:23 -0600, Nick Couchman wrote: > By the way - I found this thread on the linux-kernel mailing list that references the same sort of bug: > http://lkml.org/lkml/2007/1/25/8 > > There was a suggestion made that this has to do with kernel preemption - I have preemption completely disabled and still get the same bug. From my very limited kernel knowledge (that is, reading the output of the bug message) it seems to have to do with spinlocks in the kernel. I've enabled spinlock debugging and I'll see if I can get any more information, but I'm just not a kernel developer. There don't seem to be any patches out in the 2.6.21-rc or the -mm branches of the kernel to fix this issue. > > I know this has been mentioned a few times in the list, but I haven't seen anything too recent on this issue. I'm attempting to use GFS2 and am getting some kernel bug messages when I mount the filesystems. This seems to happen with kernels 2.6.19-2.6.21-rc6-mm1 (the one I'm currently using). The first message is this: > ------------[ cut here ]------------ > kernel BUG at fs/gfs2/glock.c:656! > invalid opcode: 0000 [#1] > last sysfs file: fs/gfs2/fstest:testfs/lock_module/block > Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs crc32c libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi af_packet button battery ac loop pcnet32 mii ext3 jbd dm_snapshot edd dm_mod fan thermal processor ide_generic sg BusLogic piix sd_mod scsi_mod ide_disk ide_core > CPU: 0 > EIP: 0060:[] Not tainted VLI > EFLAGS: 00010296 (2.6.21-rc6-mm1-default #1) > EIP is at gfs2_glmutex_unlock+0x1b/0x1f [gfs2] > eax: c223bec8 ebx: c34cc000 ecx: 00000000 edx: c23833c0 > esi: c223be84 edi: c14dbf8c ebp: 00000000 esp: c14dbf58 > ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 > Process gfs2_glockd (pid: 3804, ti=c14da000 task=c22e8a90 task.ti=c14da000) > Stack: c0369794 c23833c0 c34cc000 d0a30e7f c34cc000 c34cc000 c26f3c94 d0a29477 > 00000000 00000000 00000000 00000000 00000000 c26f3c98 00000001 00000282 > 23ed8d84 00001337 c14dbfc0 00000000 c22e8a90 c0125c44 c14dbfb0 c14dbfb0 > Call Trace: > [] gfs2_reclaim_glock+0x72/0x80 [gfs2] > [] gfs2_glockd+0x13/0xc0 [gfs2] > [] autoremove_wake_function+0x0/0x35 > [] gfs2_glockd+0x0/0xc0 [gfs2] > [] kthread+0xa3/0xcc > [] kthread+0x0/0xcc > [] kernel_thread_helper+0x7/0x10 > ======================= > Code: 5e 5f 5d e9 0a ef ff ff 83 c4 0c 5b 5e 5f 5d c3 83 ec 0c 0f ba 70 08 01 c7 40 2c 00 00 00 00 c7 40 30 00 00 00 00 e8 50 f7 ff ff <0f> 0b eb fe 56 53 89 c3 83 ec 04 8d 80 44 03 00 00 39 83 44 03 > EIP: [] gfs2_glmutex_unlock+0x1b/0x1f [gfs2] SS:ESP 0068:c14dbf58 > > > followed shortly by this: > ------------[ cut here ]------------ > kernel BUG at fs/gfs2/glock.c:656! > invalid opcode: 0000 [#2] > last sysfs file: fs/gfs2/fstest:testfs/lock_module/block > Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs crc32c libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi af_packet button battery ac loop pcnet32 mii ext3 jbd dm_snapshot edd dm_mod fan thermal processor ide_generic sg BusLogic piix sd_mod scsi_mod ide_disk ide_core > CPU: 0 > EIP: 0060:[] Not tainted VLI > EFLAGS: 00010292 (2.6.21-rc6-mm1-default #1) > EIP is at gfs2_glmutex_unlock+0x1b/0x1f [gfs2] > eax: c223bf64 ebx: c223bf20 ecx: 00000001 edx: c223bc14 > esi: 00000001 edi: c34cc000 ebp: d0a3125c esp: c14d9f78 > ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 > Process gfs2_scand (pid: 3803, ti=c14d8000 task=c22ea030 task.ti=c14d8000) > Stack: c26f3c20 c223bc14 c223bf20 d0a30068 00000003 c26f3c98 00000001 00001078 > c34cc000 d0a29524 00000000 d0a3018d c038ad60 c34cc000 c26f3c94 d0a29533 > c26f3c94 d0a29524 c34cc000 c0125ae3 00000000 00000000 ffffffff ffffffff > Call Trace: > [] examine_bucket+0x38/0x59 [gfs2] > [] gfs2_scand+0x0/0x2d [gfs2] > [] gfs2_scand_internal+0x18/0x24 [gfs2] > [] gfs2_scand+0xf/0x2d [gfs2] > [] gfs2_scand+0x0/0x2d [gfs2] > [] kthread+0xa3/0xcc > [] kthread+0x0/0xcc > [] kernel_thread_helper+0x7/0x10 > ======================= > Code: 5e 5f 5d e9 0a ef ff ff 83 c4 0c 5b 5e 5f 5d c3 83 ec 0c 0f ba 70 08 01 c7 40 2c 00 00 00 00 c7 40 30 00 00 00 00 e8 50 f7 ff ff <0f> 0b eb fe 56 53 89 c3 83 ec 04 8d 80 44 03 00 00 39 83 44 03 > EIP: [] gfs2_glmutex_unlock+0x1b/0x1f [gfs2] SS:ESP 0068:c14d9f78 > > > After I get those messages, I can list files, create files, and delete files. I run into problems if I try to use quotas or ACLs on the filesystem, and I can't unmount the filesystem - I have to hard reset the machine. Also, it doesn't seem to matter whether I use the lock_dlm or lock_nolock protocols - both seem to generate these messages. > > Nick Couchman > Systems Integrator > SEAKR Engineering, Inc. > 6221 South Racine Circle > Centennial, CO 80111 > Main: (303) 790-8499 > Fax: (303) 790-8720 > Web: http://www.seakr.com > > > > > > > Nick Couchman > Systems Integrator > SEAKR Engineering, Inc. > 6221 South Racine Circle > Centennial, CO 80111 > Main: (303) 790-8499 > Fax: (303) 790-8720 > Web: http://www.seakr.com > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian.brandes at forschungsgruppe.de Wed Apr 18 14:42:25 2007 From: christian.brandes at forschungsgruppe.de (Christian Brandes) Date: Wed, 18 Apr 2007 16:42:25 +0200 Subject: [Linux-cluster] Re: Samba on GFS problems changing directory ACLs Message-ID: <46262E51.80305@forschungsgruppe.de> Hi Marc, > what GFS version are you using? > That was a bug in old GFS versions when setting acls (IMHO kernel > > 2.6.9-43.x.y shoud have this fixed). I am using a linux kernel compiled from Ubuntu linux sources 2.6.15. How can I see what gfs version it is actually? From pcaulfie at redhat.com Wed Apr 18 15:04:13 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 18 Apr 2007 16:04:13 +0100 Subject: [Linux-cluster] dlm spinlock BUG In-Reply-To: <20070418074719.GA19343@pcnocjb.dlan.cinetic.de> References: <20070417143606.GA8621@pcnocjb.dlan.cinetic.de> <4624DF09.9050606@redhat.com> <20070418074719.GA19343@pcnocjb.dlan.cinetic.de> Message-ID: <4626336D.8030506@redhat.com> Jens Beyer wrote: > On Tue, Apr 17, 2007 at 03:51:53PM +0100, Patrick Caulfield wrote: > > I am using a vanilla 2.6.20.6 (same with 2.6.20.x). > Hmm, I'm not sure how that got left unfixed upstream Here's the patch: -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 -------------- next part -------------- A non-text attachment was scrubbed... Name: lowcomms-upstream.diff Type: text/x-patch Size: 407 bytes Desc: not available URL: From laule75 at yahoo.fr Wed Apr 18 16:04:11 2007 From: laule75 at yahoo.fr (laurent) Date: Wed, 18 Apr 2007 18:04:11 +0200 (CEST) Subject: [Linux-cluster] Boot Cluster Suite 4 from SAN Message-ID: <20070418160411.62407.qmail@web26501.mail.ukl.yahoo.com> Hello, I would like to know if it's possible to boot directly the cluster (RH4.4) from a SAN ? Thanks in advance Laurent --------------------------------- D?couvrez une nouvelle fa?on d'obtenir des r?ponses ? toutes vos questions ! Profitez des connaissances, des opinions et des exp?riences des internautes sur Yahoo! Questions/R?ponses. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Michael.Hagmann at hilti.com Wed Apr 18 17:05:07 2007 From: Michael.Hagmann at hilti.com (Hagmann, Michael) Date: Wed, 18 Apr 2007 19:05:07 +0200 Subject: [Linux-cluster] Boot Cluster Suite 4 from SAN References: <20070418160411.62407.qmail@web26501.mail.ukl.yahoo.com> Message-ID: <9C203D6FD2BF9D49BFF3450201DEDA530D0F4E@LI-OWL.hag.hilti.com> Hi Laurent yes it's possible we have a lot of such clusters. What you need is a selfmade initrd to put all the necessary stuff for the SAN boot in. ( take a lock at opensharedroot.org ) Mike -----Original Message----- From: linux-cluster-bounces at redhat.com on behalf of laurent Sent: Wed 18.04.2007 18:04 To: linux-cluster at redhat.com Subject: [Linux-cluster] Boot Cluster Suite 4 from SAN Hello, I would like to know if it's possible to boot directly the cluster (RH4.4) from a SAN ? Thanks in advance Laurent -------------- next part -------------- An HTML attachment was scrubbed... URL: From jparsons at redhat.com Wed Apr 18 17:13:15 2007 From: jparsons at redhat.com (James Parsons) Date: Wed, 18 Apr 2007 13:13:15 -0400 Subject: [Linux-cluster] Conga page makeover Message-ID: <462651AB.1010209@redhat.com> A new conga page is available that has download links for conga compiled for fc6. It also has a cookbook of a couple dozen common use case recipes...the list of pre-reqs for setting up VMs as services is pretty extensive and hopefully helpful. http://sources.redhat.com/cluster/conga/index.html We are still completing the FAQ section and some of the recipes are not complete yet. Comments, as always, are very welcome. -J From galopin at hotmail.com Wed Apr 18 14:23:56 2007 From: galopin at hotmail.com (Benoit Galopin) Date: Wed, 18 Apr 2007 14:23:56 +0000 Subject: [Linux-cluster] multiple network interfaces Message-ID: We're using two network Nic's bonded for production in a 3 node RH4EL update 2 cluster. There are 5 clustered services, 3 of them are Oracle 9i databases on ext3 filesystems (no GFS here). However, we encounter short network failures, and we does'nt know why, for instance. The problem is, when the network is down, the cluster tries to restart clustered services where it expects that the node is healthy ... bad idea My question is : is it possible to check the cluster health through two NIC (private first, production second) in the cluster instead of one, as this was true in RH2.1AS ? Thank you in advance Benoit Galopin IT Manager EFSAM France _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/ From treddy at rallydev.com Wed Apr 18 23:41:03 2007 From: treddy at rallydev.com (Tarun Reddy) Date: Wed, 18 Apr 2007 17:41:03 -0600 Subject: [Linux-cluster] killproc annoyance Message-ID: So just started working with RH4's clustering services and have run into a bit of a "deadlock" problem that I'm trying to see if anyone else has seen/fixed. 1) Start off with working config, add httpd as a clustered service, and every thing is great. Fails over to other machines great. 2) Mess up the apache config (like adding a virtual IP that doesn't exist on the system). Even though configtest works, we have a broken config. 3) So you restart apache without knowing the config is bad, while the clustering service is running. Apache doesn't come back up. Okay, cool, well go fix the problem and try to tell clustering to restart the service. Here is where things get annoying. 4) Now clustering says the service is failed. So it attempts to "service httpd stop" which killproc in /etc/init.d/functions returns a 1 since it wasn't running before. This causes the clustering software to fail the stop, and hence leave the service in a failed state. I can't get httpd up without the virtual IPs that are associated to the service, so I can't get killproc to ever return a 0 when stopping the service. Shouldn't killproc return a 0 if none of the httpd daemons are still running? I guess for now, I'll try and force some aliases for the IPs, get httpd up and running, disable the service, remove the aliases, and then enable the service. Lots of stuff to do if I was in a crisis mode in production. Anyone have an opinion on killproc return codes? Thanks, Tarun From rpeterso at redhat.com Thu Apr 19 02:06:50 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Wed, 18 Apr 2007 21:06:50 -0500 Subject: [Linux-cluster] Cluster maintenance mode ? In-Reply-To: <200704021635.28152.hlawatschek@atix.de> References: <200703271748.25566.hlawatschek@atix.de> <20070327160104.GB28370@redhat.com> <200704021635.28152.hlawatschek@atix.de> Message-ID: <4626CEBA.3090709@redhat.com> Mark Hlawatschek wrote: > On Tuesday 27 March 2007 18:01:04 you wrote: >> On Tue, Mar 27, 2007 at 05:48:25PM +0200, Mark Hlawatschek wrote: >>> Is there a way to put the cman into a kind of maintenance mode, where >>> only one node is up and no other node is allowed to join ? If not, are >>> there plans to implement something like that ? >>> >>> The question is adressing the following issue: I'd like to enable >>> automatic GFS filesystem checks. (localfs like every Nth mount or after >>> M days... ) This FS check would only be allowed if no other node has the >>> filesystem mounted or no other node is in the cluster. How can this be >>> assured in a general way ? > > Is there a possibility to do an online passive gfs_fsck, without making > changes to the filesystem ? > > Thanks, > > Mark Hi Mark, Until recently, I would have said yes: just do gfs_fsck -n and the -n would answer all the questions "no" making it passive. However, I recently discovered that gfs_fsck apparently can silently "fix" a few minor things under the covers without even asking the user for permission to changes things. Of course, if the file system is mounted by any node, the gfs kernel module won't know about the changes, which is a problem: it's likely to produce "consistency" problems. It would be a good exercise to go through the code and figure out all the places where it can do this and make some hard decisions about this. Right now it's not a high priority, but maybe someday I'll find the time to do it. Regards, Bob Peterson Red Hat Cluster Suite From shailesh at verismonetworks.com Thu Apr 19 05:49:41 2007 From: shailesh at verismonetworks.com (Shailesh) Date: Thu, 19 Apr 2007 11:19:41 +0530 Subject: [Linux-cluster] gfs on gnbd In-Reply-To: <5a8d914c0704180226g3089880arbbabd48dadb2167e@mail.gmail.com> References: <5a8d914c0704180226g3089880arbbabd48dadb2167e@mail.gmail.com> Message-ID: <1176961781.6713.27.camel@linux.site> If it is about setting up the servers, may be this link will help you https://open.datacore.ch/DCwiki.open/Wiki.jsp?page=GFS.GNBD.Usage#section-GFS.GNBD.Usage-Prerequisites Follow the man page on for setting up the 'cluster.conf'. - Shailesh On Wed, 2007-04-18 at 14:56 +0530, rajesh mishra wrote: > Hi all, > > I have RHEL 5 operating system. > I want to use the gfs2 on gnbd using bundled package in RHEL 5. The > basic two node cluster and one gnbd server. > Does any body know how to accomplish this..? > > > Thanking u all in advance. > With Regards > Rajesh. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > From Alain.Moulle at bull.net Thu Apr 19 07:23:53 2007 From: Alain.Moulle at bull.net (Alain Moulle) Date: Thu, 19 Apr 2007 09:23:53 +0200 Subject: [Linux-cluster] Agents / # Default fence action Message-ID: <46271909.60306@bull.net> Hi OK. Thanks. Is there a way to add this option field via system-config-cluster or is it mandatory to edit the file ? Thanks Alain Moull? >Hi >> >>Question about agents : >> >>in several agents, we can see that a >>"Default fence action" >>is defined at begining of script , such >>as in fence_bullpap. >> >>But for agents such as ipmilan, is there a source >>elsewhere where I could change the "requested operation" ? >In the section of he conf file, under the clusternode, add an >attribute that says option="off". This is detailed in the man page for >every agent. Dfault action is reboot. -J From jbe at webde.de Thu Apr 19 08:04:17 2007 From: jbe at webde.de (Jens Beyer) Date: Thu, 19 Apr 2007 10:04:17 +0200 Subject: [Linux-cluster] dlm spinlock BUG In-Reply-To: <4626336D.8030506@redhat.com> References: <20070417143606.GA8621@pcnocjb.dlan.cinetic.de> <4624DF09.9050606@redhat.com> <20070418074719.GA19343@pcnocjb.dlan.cinetic.de> <4626336D.8030506@redhat.com> Message-ID: <20070419080417.GA4533@pcnocjb.dlan.cinetic.de> Hi, On Wed, Apr 18, 2007 at 04:04:13PM +0100, Patrick Caulfield wrote: > Jens Beyer wrote: > > > > I am using a vanilla 2.6.20.6 (same with 2.6.20.x). > > > > Hmm, I'm not sure how that got left unfixed upstream > > Here's the patch: > the Patch did fix one spinlock BUG; now I get an otherone: [ 315.040936] BUG: spinlock already unlocked on CPU#1, dlm_recvd/14593 [ 315.040949] lock: ee108f64, .magic: dead4ead, .owner: /-1, .owner_cpu: -1 [ 315.040964] [] _raw_spin_unlock+0x70/0x72 [ 315.040976] [] dlm_lowcomms_commit_buffer+0x2f/0x9a [dlm] [ 315.040998] [] send_rcom+0xa/0x12 [dlm] ... which seems to be fixed in 2.6.21-rc6 from where I got --- fs/dlm/lowcomms-tcp.c.orig 2007-04-19 09:42:53.000000000 +0200 +++ fs/dlm/lowcomms-tcp.c 2007-04-19 09:43:23.000000000 +0200 @@ -748,6 +748,7 @@ struct connection *con = e->con; int users; + spin_lock(&con->writequeue_lock); users = --e->users; if (users) goto out; But now it hangs during mount: boxfe01:/home/jbe # mount -t gfs2 -v /dev/sdd1 /export/vol1 /sbin/mount.gfs2: mount /dev/sdd1 /export/vol1 /sbin/mount.gfs2: parse_opts: opts = "rw" /sbin/mount.gfs2: clear flag 1 for "rw", flags = 0 /sbin/mount.gfs2: parse_opts: flags = 0 /sbin/mount.gfs2: parse_opts: extra = "" /sbin/mount.gfs2: parse_opts: hostdata = "" /sbin/mount.gfs2: parse_opts: lockproto = "" /sbin/mount.gfs2: parse_opts: locktable = "" /sbin/mount.gfs2: message to gfs_controld: asking to join mountgroup: /sbin/mount.gfs2: write "join /export/vol1 gfs2 lock_dlm boxfe:clustervol1 rw /dev/sdd1" /sbin/mount.gfs2: message from gfs_controld: response to join request: /sbin/mount.gfs2: lock_dlm_join: read "0" /sbin/mount.gfs2: message from gfs_controld: mount options: /sbin/mount.gfs2: lock_dlm_join: read "hostdata=jid=1:id=262146:first=0" /sbin/mount.gfs2: lock_dlm_join: hostdata: "hostdata=jid=1:id=262146:first=0" /sbin/mount.gfs2: lock_dlm_join: extra_plus: "hostdata=jid=1:id=262146:first=0" boxfe01:/home/jbe # dmesg | tail -15 [ 137.276428] GFS2 (built Apr 19 2007 09:15:21) installed [ 137.285199] Lock_DLM (built Apr 19 2007 09:15:33) installed [ 149.628806] drbd1: role( Secondary -> Primary ) [ 149.628827] drbd1: Writing meta data super block now. [ 156.324500] GFS2: fsid=: Trying to join cluster "lock_dlm", "boxfe:clustervol1" [ 156.397920] dlm: got connection from 2 [ 156.399738] dlm: clustervol1: recover 1 [ 156.399792] dlm: clustervol1: add member 2 [ 156.399796] dlm: clustervol1: add member 1 [ 156.400514] dlm: clustervol1: config mismatch: 32,0 nodeid 2: 11,0 [ 156.400519] dlm: clustervol1: ping_members aborted -22 last nodeid 2 [ 156.400523] dlm: clustervol1: total members 2 error -22 [ 156.400526] dlm: clustervol1: recover_members failed -22 [ 156.400529] dlm: clustervol1: recover 1 error -22 [ 156.404760] GFS2: fsid=boxfe:clustervol1.1: Joined cluster. Now mounting FS... Regards, Jens From jvantuyl at engineyard.com Thu Apr 19 10:24:45 2007 From: jvantuyl at engineyard.com (Jayson Vantuyl) Date: Thu, 19 Apr 2007 05:24:45 -0500 Subject: [Linux-cluster] GFS over AOE without fencing? In-Reply-To: References: <4624BD83.4070603@redhat.com> Message-ID: <7FB45AE0-9823-4AF9-8DA8-2FD1A7CA941F@engineyard.com> It is supposedly possible to script using the mask command to block servers on individual MAC address to the AoE storage. While this is often offered by Coraid support as an option, I've not seen anyone implement it. To be sure, I have my doubts about it anyways, give that the utility that you automate (cec) does not exactly provide an API (you actually are writing directly into the unit's console!). On Apr 17, 2007, at 8:05 AM, Kadlecsik Jozsi wrote: > Hi Bryn, > > On Tue, 17 Apr 2007, Bryn M. Reeves wrote: > >>> Assuming a dedicated VLAN which servers only AOE and GFS traffic >>> among >>> the Coraid boxes and the GFS hosts, is there any need for fencing >>> at all? >>> >>> What are the hidden traps behind such setups? >>> >>> Best regards, >>> Jozsef >> >> If you are using GFS on shared storage then you need fencing. Period. >> >> The only way you can guarantee data integrity in this scenario is by >> completely cutting a failed or misbehaving node off from the storage; >> either by power cycling it or having the storage reject its access. >> >> Otherwise, imagine a situation where a node hangs for some reason >> and is >> ejected from the cluster. At this point none of its locks for the >> shared >> data are valid anymore. Some time later, the node recovers from >> the hang >> and begins flushing writes to the storage -> corruption. > > I see - the sentences above make much more clearer why fencing is > needed. > Thank you the explanation - and the hint for the possibility to reject > the access at the storage itself! > > Best regards, > Jozsef > -- > E-mail : kadlec at sunserv.kfki.hu, kadlec at blackhole.kfki.hu > PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt > Address: KFKI Research Institute for Particle and Nuclear Physics > H-1525 Budapest 114, POB. 49, Hungary > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Jayson Vantuyl Systems Architect Engine Yard jvantuyl at engineyard.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kadlec at sunserv.kfki.hu Thu Apr 19 10:33:33 2007 From: kadlec at sunserv.kfki.hu (Kadlecsik Jozsi) Date: Thu, 19 Apr 2007 12:33:33 +0200 (MEST) Subject: [Linux-cluster] GFS over AOE without fencing? In-Reply-To: <7FB45AE0-9823-4AF9-8DA8-2FD1A7CA941F@engineyard.com> References: <4624BD83.4070603@redhat.com> <7FB45AE0-9823-4AF9-8DA8-2FD1A7CA941F@engineyard.com> Message-ID: On Thu, 19 Apr 2007, Jayson Vantuyl wrote: > It is supposedly possible to script using the mask command to block servers on > individual MAC address to the AoE storage. While this is often offered by > Coraid support as an option, I've not seen anyone implement it. To be sure, I > have my doubts about it anyways, give that the utility that you automate (cec) > does not exactly provide an API (you actually are writing directly into the > unit's console!). I have already written the script which does exactly that: i.e it calls 'cec' and issues the proper mask command to disable/enable access to the logical blades. Now I only have to test it in our first testbed GFS setup ;-). Best regards, Jozsef -- E-mail : kadlec at sunserv.kfki.hu, kadlec at blackhole.kfki.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt Address: KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary From breeves at redhat.com Thu Apr 19 10:47:36 2007 From: breeves at redhat.com (Bryn M. Reeves) Date: Thu, 19 Apr 2007 11:47:36 +0100 Subject: [Linux-cluster] GFS over AOE without fencing? In-Reply-To: References: <4624BD83.4070603@redhat.com> <7FB45AE0-9823-4AF9-8DA8-2FD1A7CA941F@engineyard.com> Message-ID: <462748C8.6010802@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Kadlecsik Jozsi wrote: > On Thu, 19 Apr 2007, Jayson Vantuyl wrote: > >> It is supposedly possible to script using the mask command to block servers on >> individual MAC address to the AoE storage. While this is often offered by >> Coraid support as an option, I've not seen anyone implement it. To be sure, I >> have my doubts about it anyways, give that the utility that you automate (cec) >> does not exactly provide an API (you actually are writing directly into the >> unit's console!). > > I have already written the script which does exactly that: i.e it calls > 'cec' and issues the proper mask command to disable/enable access to the > logical blades. Now I only have to test it in our first testbed GFS setup > ;-). > This doesn't seem so unreasonable - the fence_apc script does a similar thing, connecting to the console over telnet and squirting commands into the device's menu system. Kind regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFGJ0jI6YSQoMYUY94RAkswAJ9k4Yym/PHs/Bwj9AXz0dXTgPCoJACgnoLQ MEJ63uWPdBTdGMo+GYuJtyo= =HL8a -----END PGP SIGNATURE----- From scott.mcclanahan at trnswrks.com Thu Apr 19 12:31:01 2007 From: scott.mcclanahan at trnswrks.com (Scott McClanahan) Date: Thu, 19 Apr 2007 08:31:01 -0400 Subject: [Linux-cluster] killproc annoyance In-Reply-To: References: Message-ID: <1176985862.3709.15.camel@jarjar.trnswrks.com> According to the link at the bottom, a stop argument to an already stopped service should return a success just like a start on an already started service should be interpreted as a success. So I have rewritten most of my init scripts which are managed by clurgd to be spec compliant. Unfortunately, how do you really know if the service is stopped? How can you be certain the pid file got written or is even correct (race conditions)? Anyway, I have written init scripts for apache, tomcat, and IBM MQ to name a few and have tested the hell out of them so let me know if you want a working example. http://refspecs.freestandards.org/LSB_2.0.1/LSB-Core/LSB-Core/iniscrptact.html On Wed, 2007-04-18 at 19:41 -0400, Tarun Reddy wrote: > So just started working with RH4's clustering services and have run > into a bit of a "deadlock" problem that I'm trying to see if anyone > else has seen/fixed. > > 1) Start off with working config, add httpd as a clustered service, > and every thing is great. Fails over to other machines great. > > 2) Mess up the apache config (like adding a virtual IP that doesn't > exist on the system). Even though configtest works, we have a broken > config. > > 3) So you restart apache without knowing the config is bad, while the > clustering service is running. Apache doesn't come back up. Okay, > cool, well go fix the problem and try to tell clustering to restart > the service. > > Here is where things get annoying. > 4) Now clustering says the service is failed. So it attempts to > "service httpd stop" which killproc in /etc/init.d/functions returns > a 1 since it wasn't running before. This causes the clustering > software to fail the stop, and hence leave the service in a failed > state. I can't get httpd up without the virtual IPs that are > associated to the service, so I can't get killproc to ever return a 0 > when stopping the service. Shouldn't killproc return a 0 if none of > the httpd daemons are still running? > > I guess for now, I'll try and force some aliases for the IPs, get > httpd up and running, disable the service, remove the aliases, and > then enable the service. Lots of stuff to do if I was in a crisis > mode in production. > > Anyone have an opinion on killproc return codes? > > Thanks, > Tarun > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > From jparsons at redhat.com Thu Apr 19 16:55:18 2007 From: jparsons at redhat.com (jim parsons) Date: Thu, 19 Apr 2007 12:55:18 -0400 Subject: [Linux-cluster] Agents / # Default fence action In-Reply-To: <46271909.60306@bull.net> References: <46271909.60306@bull.net> Message-ID: <46279EF6.7040001@redhat.com> Alain Moulle wrote: > Hi > OK. Thanks. Is there a way to add this option field via > system-config-cluster or is it mandatory to edit the file ? > Thanks > Alain Moull? > You must edit the conf file to add these attributes. -j From treddy at rallydev.com Thu Apr 19 17:37:26 2007 From: treddy at rallydev.com (Tarun Reddy) Date: Thu, 19 Apr 2007 11:37:26 -0600 Subject: [Linux-cluster] killproc annoyance In-Reply-To: <1176985862.3709.15.camel@jarjar.trnswrks.com> References: <1176985862.3709.15.camel@jarjar.trnswrks.com> Message-ID: <0B5167FE-B88A-4F12-A0C2-BC44703A903D@rallydev.com> On Apr 19, 2007, at 6:31 AM, Scott McClanahan wrote: > According to the link at the bottom, a stop argument to an already > stopped service should return a success just like a start on an > already > started service should be interpreted as a success. So I have > rewritten > most of my init scripts which are managed by clurgd to be spec > compliant. Unfortunately, how do you really know if the service is > stopped? How can you be certain the pid file got written or is even > correct (race conditions)? Anyway, I have written init scripts for > apache, tomcat, and IBM MQ to name a few and have tested the hell > out of > them so let me know if you want a working example. > > http://refspecs.freestandards.org/LSB_2.0.1/LSB-Core/LSB-Core/ > iniscrptact.html > > On Wed, 2007-04-18 at 19:41 -0400, Tarun Reddy wrote: >> So just started working with RH4's clustering services and have run >> into a bit of a "deadlock" problem that I'm trying to see if anyone >> else has seen/fixed. >> >> 1) Start off with working config, add httpd as a clustered service, >> and every thing is great. Fails over to other machines great. >> >> 2) Mess up the apache config (like adding a virtual IP that doesn't >> exist on the system). Even though configtest works, we have a broken >> config. >> >> 3) So you restart apache without knowing the config is bad, while the >> clustering service is running. Apache doesn't come back up. Okay, >> cool, well go fix the problem and try to tell clustering to restart >> the service. >> >> Here is where things get annoying. >> 4) Now clustering says the service is failed. So it attempts to >> "service httpd stop" which killproc in /etc/init.d/functions returns >> a 1 since it wasn't running before. This causes the clustering >> software to fail the stop, and hence leave the service in a failed >> state. I can't get httpd up without the virtual IPs that are >> associated to the service, so I can't get killproc to ever return a 0 >> when stopping the service. Shouldn't killproc return a 0 if none of >> the httpd daemons are still running? >> >> I guess for now, I'll try and force some aliases for the IPs, get >> httpd up and running, disable the service, remove the aliases, and >> then enable the service. Lots of stuff to do if I was in a crisis >> mode in production. >> >> Anyone have an opinion on killproc return codes? Scott, Thank you very much for the link! I thought I wasn't crazy. So some more testing shows that on RHEL5, /etc/init.d/httpd stop when apache is stopped, does the right thing and has RETVAL of 0, while RHEL4 is "broken" in this respect. I think I'll look at where the differences are and possibly integrate the change back. Thanks, Tarun From teigland at redhat.com Thu Apr 19 18:00:12 2007 From: teigland at redhat.com (David Teigland) Date: Thu, 19 Apr 2007 13:00:12 -0500 Subject: [Linux-cluster] dlm spinlock BUG In-Reply-To: <20070419080417.GA4533@pcnocjb.dlan.cinetic.de> References: <20070417143606.GA8621@pcnocjb.dlan.cinetic.de> <4624DF09.9050606@redhat.com> <20070418074719.GA19343@pcnocjb.dlan.cinetic.de> <4626336D.8030506@redhat.com> <20070419080417.GA4533@pcnocjb.dlan.cinetic.de> Message-ID: <20070419180012.GI23079@redhat.com> On Thu, Apr 19, 2007 at 10:04:17AM +0200, Jens Beyer wrote: > Hi, > > On Wed, Apr 18, 2007 at 04:04:13PM +0100, Patrick Caulfield wrote: > > Jens Beyer wrote: > > > > > > I am using a vanilla 2.6.20.6 (same with 2.6.20.x). > > > > > > > Hmm, I'm not sure how that got left unfixed upstream > > > > Here's the patch: > > > > the Patch did fix one spinlock BUG; now I get an otherone: Right, there are a lot of bugs fixed since 2.6.20, including all these. Use 2.6.21-rc, or at least copy the dlm/gfs2 sources from there into 2.6.20. Dave From treddy at rallydev.com Thu Apr 19 17:54:21 2007 From: treddy at rallydev.com (Tarun Reddy) Date: Thu, 19 Apr 2007 11:54:21 -0600 Subject: [Linux-cluster] killproc annoyance In-Reply-To: <0B5167FE-B88A-4F12-A0C2-BC44703A903D@rallydev.com> References: <1176985862.3709.15.camel@jarjar.trnswrks.com> <0B5167FE-B88A-4F12-A0C2-BC44703A903D@rallydev.com> Message-ID: On Apr 19, 2007, at 11:37 AM, Tarun Reddy wrote: > > On Apr 19, 2007, at 6:31 AM, Scott McClanahan wrote: > >> According to the link at the bottom, a stop argument to an already >> stopped service should return a success just like a start on an >> already >> started service should be interpreted as a success. So I have >> rewritten >> most of my init scripts which are managed by clurgd to be spec >> compliant. Unfortunately, how do you really know if the service is >> stopped? How can you be certain the pid file got written or is even >> correct (race conditions)? Anyway, I have written init scripts for >> apache, tomcat, and IBM MQ to name a few and have tested the hell >> out of >> them so let me know if you want a working example. >> >> http://refspecs.freestandards.org/LSB_2.0.1/LSB-Core/LSB-Core/ >> iniscrptact.html >> >> On Wed, 2007-04-18 at 19:41 -0400, Tarun Reddy wrote: >>> So just started working with RH4's clustering services and have run >>> into a bit of a "deadlock" problem that I'm trying to see if anyone >>> else has seen/fixed. >>> >>> 1) Start off with working config, add httpd as a clustered service, >>> and every thing is great. Fails over to other machines great. >>> >>> 2) Mess up the apache config (like adding a virtual IP that doesn't >>> exist on the system). Even though configtest works, we have a broken >>> config. >>> >>> 3) So you restart apache without knowing the config is bad, while >>> the >>> clustering service is running. Apache doesn't come back up. Okay, >>> cool, well go fix the problem and try to tell clustering to restart >>> the service. >>> >>> Here is where things get annoying. >>> 4) Now clustering says the service is failed. So it attempts to >>> "service httpd stop" which killproc in /etc/init.d/functions returns >>> a 1 since it wasn't running before. This causes the clustering >>> software to fail the stop, and hence leave the service in a failed >>> state. I can't get httpd up without the virtual IPs that are >>> associated to the service, so I can't get killproc to ever return >>> a 0 >>> when stopping the service. Shouldn't killproc return a 0 if none of >>> the httpd daemons are still running? >>> >>> I guess for now, I'll try and force some aliases for the IPs, get >>> httpd up and running, disable the service, remove the aliases, and >>> then enable the service. Lots of stuff to do if I was in a crisis >>> mode in production. >>> >>> Anyone have an opinion on killproc return codes? > > Scott, > > Thank you very much for the link! I thought I wasn't crazy. So some > more testing shows that on RHEL5, /etc/init.d/httpd stop when > apache is stopped, does the right thing and has RETVAL of 0, while > RHEL4 is "broken" in this respect. > > I think I'll look at where the differences are and possibly > integrate the change back. > > Thanks, > Tarun > For future reference, RH clearly saw the change is a violation of LSB and change the following in killproc between RHEL4 and RHEL5 else failure $"$base shutdown" RC=1 fi changed to: else if [ -n "${LSB:-}" -a -n "$killlevel" ]; then RC=7 # Program is not running else failure $"$base shutdown" RC=0 fi fi I think I will change it to return RC=0 and hope nothing else breaks :-) Tarun From lhh at redhat.com Thu Apr 19 19:32:45 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 19 Apr 2007 15:32:45 -0400 Subject: [Linux-cluster] RHEL 4: quorate or inquorate In-Reply-To: <2E02749DAF5338479606A056219BE109017879AD@smail.crosswalkinc.com> References: <20070412183421.GO1804@redhat.com> <2E02749DAF5338479606A056219BE109017879AD@smail.crosswalkinc.com> Message-ID: <20070419193245.GH19391@redhat.com> On Thu, Apr 12, 2007 at 03:08:27PM -0600, Dex Chen wrote: > Thanks for the response. > > I figured out the cause. I used "cman_tool leave remove" when I shutdown > the other 2 nodes. In this way, the remaining one/cluster had "quorum: > 1, and expected vote 1". I was surprised that the "2 node cluster" > special case did not prevent this. > > My remaining question is: > Do you see any potential risks such as "split brain" when the stopped > nodes come back, especially in a more-node cluster (say 5 node cluster)? You can have a split brain if you try to boot the other nodes during a network partition... -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Thu Apr 19 19:36:00 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 19 Apr 2007 15:36:00 -0400 Subject: [Linux-cluster] 'Run exclusive' semantics In-Reply-To: <20070413102519.GT26722@helsinki.fi> References: <20070413102519.GT26722@helsinki.fi> Message-ID: <20070419193600.GI19391@redhat.com> On Fri, Apr 13, 2007 at 01:25:19PM +0300, Janne Peltonen wrote: > Hi. > > I've been playing around with the 'Run exclusive' option of a cluster > service. Seems to me that you can't /relocate/ a service to a node that > already runs an exclusive service, neither can you /relocate/ the > exclusive service to a node that already runs some other service. But > nothing seems to prevent you from /enabling/ a service on a node that > already has an exclusive service running, or /enabling/ an exclusive > service on a node that already runs some other service. Is this by > design? It's supposed to not let an excl service run on any other node running a service. There's a BZ open about it, and a patch in the bz. -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Thu Apr 19 19:39:10 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 19 Apr 2007 15:39:10 -0400 Subject: [Linux-cluster] clvmd / cman / fence, in order to make xen cluster In-Reply-To: <20070417174019.z6lzrp2hes0csg0o@webmail.aic.fr> References: <20070417174019.z6lzrp2hes0csg0o@webmail.aic.fr> Message-ID: <20070419193910.GJ19391@redhat.com> On Tue, Apr 17, 2007 at 05:40:19PM +0200, jmfaidy at aic.fr wrote: > > so i'm sad, and i tried with a copy of the cluster.conf file on 2nd node, > i relaunch ccsd > then cman_tool join -c XenCluster > seems to work, bu actually, when i do "cman_tool nodes" i have : > Node Sts Inc Joined Name > 1 X 0 xentest.aic.fr > 2 M 36 2007-04-17 17:44:24 xentest2.aic.fr This could be a couple of things: * Try disabling iptables * set selinux to permissive * id you run fence_ack_manual on one of the nodes? This doesn't sound like a kernel problem. -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Thu Apr 19 19:40:19 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 19 Apr 2007 15:40:19 -0400 Subject: [Linux-cluster] About quorum disk and GFS2 on rhel5 In-Reply-To: <462500F3.8060701@redhat.com> References: <4624FB4D.5080605@gmail.com> <462500F3.8060701@redhat.com> Message-ID: <20070419194019.GK19391@redhat.com> On Tue, Apr 17, 2007 at 12:16:35PM -0500, Robert Peterson wrote: > carlopmart wrote: > >hi all, > > > > I have a question about quorum disk and gfs2-utils on rhel5: > > > > a) Do i need to use a raw partition for quorum disks or formatted > >partition ? If I need to use a raw partition, where is rawdevices script > >present on rhel 4.x ?? > > Regarding (a): I'm not a qdisk expert, but I know the man page has > lots of good information about this. Qdisk stuff should be on an unformatted partition. However, you should initialize it with 'mkqdisk'; see qdisk(5) and mkqdisk man pages for more info. -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Thu Apr 19 19:43:05 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 19 Apr 2007 15:43:05 -0400 Subject: [Linux-cluster] killproc annoyance In-Reply-To: References: Message-ID: <20070419194304.GL19391@redhat.com> On Wed, Apr 18, 2007 at 05:41:03PM -0600, Tarun Reddy wrote: > Here is where things get annoying. > 4) Now clustering says the service is failed. So it attempts to > "service httpd stop" which killproc in /etc/init.d/functions returns > a 1 since it wasn't running before. This causes the clustering > software to fail the stop, and hence leave the service in a failed > state. I can't get httpd up without the virtual IPs that are > associated to the service, so I can't get killproc to ever return a 0 > when stopping the service. Shouldn't killproc return a 0 if none of > the httpd daemons are still running? This is a bug in the rhel4 initscripts package. There's a question in the FAQ about it: http://sources.redhat.com/cluster/faq.html#rgm_wontrestart -- Lon Hohberger - Software Engineer - Red Hat, Inc. From jvantuyl at engineyard.com Fri Apr 20 08:28:35 2007 From: jvantuyl at engineyard.com (Jayson Vantuyl) Date: Fri, 20 Apr 2007 03:28:35 -0500 Subject: [Linux-cluster] GFS over AOE without fencing? In-Reply-To: <462748C8.6010802@redhat.com> References: <4624BD83.4070603@redhat.com> <7FB45AE0-9823-4AF9-8DA8-2FD1A7CA941F@engineyard.com> <462748C8.6010802@redhat.com> Message-ID: <86FA514A-EF0C-49EE-80AD-D302A701EF8E@engineyard.com> In principal this is true. However, cec is not so reliable of a connection. It is NOT TCP. I have little information about how resilient the protocol is, however, in a unit we have with a bad disk, I've had the cec connection spontaneously drop mid-command. I'm sure they're working to fix this, but it doesn't bode well for something as critical as fencing. I'm also unclear on whether a dropped connection generates a non-zero exit code (i.e. is even detectable). Also, on APCs, the fence_apc script has the benefit that the APC switches do not allow more than one concurrent telnet connection, which effectively serializes fence requests. With the cec, not so much. Also, this fences the entire Coraid device in a way that must be manually cleared if it gets left masked. This is a real possibility where multiple nodes are racing to fence each other--especially on multiple Coraid shelfs (as it must be done per shelf). Since we use our Coraids for non-GFS boot volumes as well, this is also problematic for us, since a stale mask entry keeps us from booting. It's really not so simple. I'd almost recommend shutting off the ports at the switch rather than the Coraid, assuming you have good enough switches to do this reliably. On Apr 19, 2007, at 5:47 AM, Bryn M. Reeves wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Kadlecsik Jozsi wrote: >> On Thu, 19 Apr 2007, Jayson Vantuyl wrote: >> >>> It is supposedly possible to script using the mask command to >>> block servers on >>> individual MAC address to the AoE storage. While this is often >>> offered by >>> Coraid support as an option, I've not seen anyone implement it. >>> To be sure, I >>> have my doubts about it anyways, give that the utility that you >>> automate (cec) >>> does not exactly provide an API (you actually are writing >>> directly into the >>> unit's console!). >> >> I have already written the script which does exactly that: i.e it >> calls >> 'cec' and issues the proper mask command to disable/enable access >> to the >> logical blades. Now I only have to test it in our first testbed >> GFS setup >> ;-). >> > > This doesn't seem so unreasonable - the fence_apc script does a > similar > thing, connecting to the console over telnet and squirting commands > into > the device's menu system. > > Kind regards, > > Bryn. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.7 (GNU/Linux) > Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org > > iD8DBQFGJ0jI6YSQoMYUY94RAkswAJ9k4Yym/PHs/Bwj9AXz0dXTgPCoJACgnoLQ > MEJ63uWPdBTdGMo+GYuJtyo= > =HL8a > -----END PGP SIGNATURE----- > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Jayson Vantuyl Systems Architect Engine Yard jvantuyl at engineyard.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kadlec at sunserv.kfki.hu Fri Apr 20 08:49:12 2007 From: kadlec at sunserv.kfki.hu (Kadlecsik Jozsi) Date: Fri, 20 Apr 2007 10:49:12 +0200 (MEST) Subject: [Linux-cluster] GFS over AOE without fencing? In-Reply-To: <86FA514A-EF0C-49EE-80AD-D302A701EF8E@engineyard.com> References: <4624BD83.4070603@redhat.com> <7FB45AE0-9823-4AF9-8DA8-2FD1A7CA941F@engineyard.com> <462748C8.6010802@redhat.com> <86FA514A-EF0C-49EE-80AD-D302A701EF8E@engineyard.com> Message-ID: On Fri, 20 Apr 2007, Jayson Vantuyl wrote: > However, cec is not so reliable of a connection. It is NOT TCP. Yes, it's true. > I have little information about how resilient the protocol is, however, > in a unit we have with a bad disk, I've had the cec connection > spontaneously drop mid-command. I'm sure they're working to fix this, > but it doesn't bode well for something as critical as fencing. I'm also > unclear on whether a dropped connection generates a non-zero exit code > (i.e. is even detectable). The fence_coraid script I wrote uses expect in perl. So if the cec connection fails (at any point) it is detected and reported by the script. > Also, on APCs, the fence_apc script has the benefit that the APC > switches do not allow more than one concurrent telnet connection, which > effectively serializes fence requests.? With the cec, not so much. This is problematic: the requests are not serialized at all, two concurrent cec sessions are totally mixed: command issued in one cec appears in the other (letter by letter). Yes, this is a real issue. > Also, this fences the entire Coraid device in a way that must be manually > cleared if it gets left masked. This is a real possibility where multiple > nodes are racing to fence each other--especially on multiple Coraid shelfs (as > it must be done per shelf). > > Since we use our Coraids for non-GFS boot volumes as well, this is also > problematic for us, since a stale mask entry keeps us from booting. The masking disallows the access to the logical blades only. The host still able to connect to the Coraid box over cec and re-enable it's access rights to the lblades. Best regards, Jozsef -- E-mail : kadlec at sunserv.kfki.hu, kadlec at blackhole.kfki.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt Address: KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary From jvantuyl at engineyard.com Fri Apr 20 09:18:25 2007 From: jvantuyl at engineyard.com (Jayson Vantuyl) Date: Fri, 20 Apr 2007 04:18:25 -0500 Subject: [Linux-cluster] GFS over AOE without fencing? In-Reply-To: References: <4624BD83.4070603@redhat.com> <7FB45AE0-9823-4AF9-8DA8-2FD1A7CA941F@engineyard.com> <462748C8.6010802@redhat.com> <86FA514A-EF0C-49EE-80AD-D302A701EF8E@engineyard.com> Message-ID: <8FD6E23E-41C8-47AA-9F4B-08C54F5E4590@engineyard.com> On Apr 20, 2007, at 3:49 AM, Kadlecsik Jozsi wrote: >> I have little information about how resilient the protocol is, >> however, >> in a unit we have with a bad disk, I've had the cec connection >> spontaneously drop mid-command. I'm sure they're working to fix >> this, >> but it doesn't bode well for something as critical as fencing. >> I'm also >> unclear on whether a dropped connection generates a non-zero exit >> code >> (i.e. is even detectable). > > The fence_coraid script I wrote uses expect in perl. So if the cec > connection fails (at any point) it is detected and reported by the > script. Also of interest is whether these masks are saved over reboot. I think they are, but its probably worth checking. >> Also, on APCs, the fence_apc script has the benefit that the APC >> switches do not allow more than one concurrent telnet connection, >> which >> effectively serializes fence requests. With the cec, not so much. > > This is problematic: the requests are not serialized at all, two > concurrent cec sessions are totally mixed: command issued in one cec > appears in the other (letter by letter). Yes, this is a real issue. In our current setup, we utilize over 25 lblades on four shelves. We are adding a fifth this weekend, with an additional six lblades. As you can imaging, fencing in this situation becomes complex. Additionally, if you desire to dynamically detect which lblades to fence, this becomes fairly complex quickly. I will leave to the reader envisioning the alternative of manually updating fencing scripts across the cluster for each lblade addition. >> Also, this fences the entire Coraid device in a way that must be >> manually >> cleared if it gets left masked. This is a real possibility where >> multiple >> nodes are racing to fence each other--especially on multiple >> Coraid shelfs (as >> it must be done per shelf). >> >> Since we use our Coraids for non-GFS boot volumes as well, this is >> also >> problematic for us, since a stale mask entry keeps us from booting. > > The masking disallows the access to the logical blades only. The host > still able to connect to the Coraid box over cec and re-enable it's > access rights to the lblades. This certainly complicates setups that attempt to use the Coraid as a root device. I don't like the idea of having to include cec and expect in an initrd. -- Jayson Vantuyl Systems Architect Engine Yard jvantuyl at engineyard.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From orkcu at yahoo.com Fri Apr 20 13:59:10 2007 From: orkcu at yahoo.com (=?iso-8859-1?Q?Roger_Pe=F1a?=) Date: Fri, 20 Apr 2007 06:59:10 -0700 (PDT) Subject: [Linux-cluster] killproc annoyance In-Reply-To: <0B5167FE-B88A-4F12-A0C2-BC44703A903D@rallydev.com> Message-ID: <209247.70499.qm@web50602.mail.re2.yahoo.com> --- Tarun Reddy wrote: > > On Apr 19, 2007, at 6:31 AM, Scott McClanahan wrote: > > > According to the link at the bottom, a stop > argument to an already > > stopped service should return a success just like > a start on an > > already > > started service should be interpreted as a > success. So I have > > rewritten > > most of my init scripts which are managed by > clurgd to be spec > > compliant. Unfortunately, how do you really know > if the service is > > stopped? How can you be certain the pid file got > written or is even > > correct (race conditions)? Anyway, I have written > init scripts for > > apache, tomcat, and IBM MQ to name a few and have > tested the hell > > out of > > them so let me know if you want a working example. > > > > > http://refspecs.freestandards.org/LSB_2.0.1/LSB-Core/LSB-Core/ > > > iniscrptact.html > > > > On Wed, 2007-04-18 at 19:41 -0400, Tarun Reddy > wrote: > >> So just started working with RH4's clustering > services and have run > >> into a bit of a "deadlock" problem that I'm > trying to see if anyone > >> else has seen/fixed. > >> > >> 1) Start off with working config, add httpd as a > clustered service, > >> and every thing is great. Fails over to other > machines great. > >> > >> 2) Mess up the apache config (like adding a > virtual IP that doesn't > >> exist on the system). Even though configtest > works, we have a broken > >> config. > >> > >> 3) So you restart apache without knowing the > config is bad, while the > >> clustering service is running. Apache doesn't > come back up. Okay, > >> cool, well go fix the problem and try to tell > clustering to restart > >> the service. > >> > >> Here is where things get annoying. > >> 4) Now clustering says the service is failed. So > it attempts to > >> "service httpd stop" which killproc in > /etc/init.d/functions returns > >> a 1 since it wasn't running before. This causes > the clustering > >> software to fail the stop, and hence leave the > service in a failed > >> state. I can't get httpd up without the virtual > IPs that are > >> associated to the service, so I can't get > killproc to ever return a 0 > >> when stopping the service. Shouldn't killproc > return a 0 if none of > >> the httpd daemons are still running? > >> > >> I guess for now, I'll try and force some aliases > for the IPs, get > >> httpd up and running, disable the service, remove > the aliases, and > >> then enable the service. Lots of stuff to do if I > was in a crisis > >> mode in production. > >> > >> Anyone have an opinion on killproc return codes? > > Scott, > > Thank you very much for the link! I thought I wasn't > crazy. So some > more testing shows that on RHEL5, /etc/init.d/httpd > stop when apache > is stopped, does the right thing and has RETVAL of > 0, while RHEL4 is > "broken" in this respect. > > I think I'll look at where the differences are and > possibly integrate > the change back. someone already send you the link in cluster-faq where it is explained the problem and how to fixed (in the function killproc of initrd functions file) it is a very simple fix, and it works for every initrd script that use the killproc function, the ones that not use it will still get you problems (like mysql initrd script) cu roger __________________________________________ RedHat Certified Engineer ( RHCE ) Cisco Certified Network Associate ( CCNA ) __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From cluster at defuturo.co.uk Fri Apr 20 15:31:17 2007 From: cluster at defuturo.co.uk (Robert Clark) Date: Fri, 20 Apr 2007 16:31:17 +0100 Subject: [Linux-cluster] lm_dlm_cancel during quota operations Message-ID: <1177083077.2849.42.camel@localhost.localdomain> I have a script which runs gfs_quota to set quotas for all the users on my GFS filesystem. When it's run simultaneously on two nodes, errors like the following begin to appear: lock_dlm: lm_dlm_cancel 2,34 flags 80 lock_dlm: lm_dlm_cancel rv 0 2,34 flags 40080 lock_dlm: complete dlm cancel 2,34 flags 40000 ... lock_dlm: lm_dlm_cancel 2,34 flags 80 lock_dlm: complete dlm cancel 2,34 flags 40000 lock_dlm: lm_dlm_cancel rv 0 2,34 flags 80 ... lock_dlm: lm_dlm_cancel 2,34 flags 84 lock_dlm: lm_dlm_cancel skip 2,34 flags 84 ... lock_dlm: lm_dlm_cancel 2,34 flags 80 dlm: cancel granted 1350055 lock_dlm: lm_dlm_cancel rv 0 2,34 flags 40000 lock_dlm: extra completion 2,34 5,5 id 1350055 flags 40000 and, more rarely: lock_dlm: lm_dlm_cancel 2,34 flags 80 lock_dlm: lm_dlm_cancel rv 0 2,34 flags 40080 dlm: desktop-home-1: cancel reply ret -22 lock_dlm: ast sb_status -22 2,34 flags 40000 ... lock_dlm: lm_dlm_cancel 2,34 flags 80 lock_dlm: lm_dlm_cancel rv -16 2,34 flags 40080 At the same time, I/O to the GFS partition hangs. Rebooting one of the two nodes allows the cluster to recover. On my smaller test cluster, I've been able to reproduce some of the errors: lock_dlm: lm_dlm_cancel 2,18 flags 84 lock_dlm: lm_dlm_cancel rv 0 2,18 flags 40080 lock_dlm: complete dlm cancel 2,18 flags 40000 ... lock_dlm: lm_dlm_cancel 2,18 flags 80 lock_dlm: lm_dlm_cancel skip 2,18 flags 0 though not the I/O hangs. My shared storage is over AoE and I'm using the following packages: GFS-6.1.6-1 dlm-1.0.1-1 cman-1.0.11-0 GFS-kernel-hugemem-2.6.9-60.9 dlm-kernel-hugemem-2.6.9-44.9 cman-kernel-hugemem-2.6.9-45.15 kernel-hugemem-2.6.9-42.0.10.EL I must admit, I've not been able to find out much about what dlm cancels are or what triggers them. Can anyone shed some light on this? Robert From teigland at redhat.com Fri Apr 20 16:13:18 2007 From: teigland at redhat.com (David Teigland) Date: Fri, 20 Apr 2007 11:13:18 -0500 Subject: [Linux-cluster] lm_dlm_cancel during quota operations In-Reply-To: <1177083077.2849.42.camel@localhost.localdomain> References: <1177083077.2849.42.camel@localhost.localdomain> Message-ID: <20070420161318.GB32310@redhat.com> On Fri, Apr 20, 2007 at 04:31:17PM +0100, Robert Clark wrote: > I have a script which runs gfs_quota to set quotas for all the users > on my GFS filesystem. When it's run simultaneously on two nodes, errors > like the following begin to appear: > > lock_dlm: lm_dlm_cancel 2,34 flags 80 > lock_dlm: lm_dlm_cancel rv 0 2,34 flags 40080 > lock_dlm: complete dlm cancel 2,34 flags 40000 Writes to a hidden/internal file (like the quota file) set the PRIORITY flag in the glock request, which cause the cancels. I don't understand why gfs would want to do cancels in this case, though, I was very surprised to see that. So, first job is to figure out why these writes need to do cancels (or if that's the bug right there), and the second job is to figure out why those cancels, if legitimate, are causing problems. Dave From dingxincai at 126.com Sat Apr 21 02:40:08 2007 From: dingxincai at 126.com (dingxincai at 126.com) Date: Sat, 21 Apr 2007 10:40:08 +0800 (CST) Subject: [Linux-cluster] Agents / # Default fence action (jim pars ons) Message-ID: <46297988.000061.16081@bj126app6.126.com> -------------- next part -------------- An HTML attachment was scrubbed... URL: From wferi at niif.hu Sat Apr 21 11:42:45 2007 From: wferi at niif.hu (Ferenc Wagner) Date: Sat, 21 Apr 2007 13:42:45 +0200 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <20070328162726.GF22230@redhat.com> (David Teigland's message of "Wed, 28 Mar 2007 11:27:27 -0500") References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> Message-ID: <871wieeyx6.fsf@tac.ki.iif.hu> David Teigland writes: > On Wed, Mar 28, 2007 at 05:48:10PM +0200, Wagner Ferenc wrote: >> Ferenc Wagner writes: >> >> > There's a good bunch of RRDs in a directory. A script scans them for >> > their last modification times, and then updates each in turn for a >> > couple of times. The number of files scanned and the length of the >> > update rounds are printed. The results are much different for 500 and >> > 501 files: >> > >> > filecount=501 >> > iteration=0 elapsed time=10.425568 s >> > iteration=1 elapsed time= 9.766178 s >> > iteration=2 elapsed time=20.14514 s >> > iteration=3 elapsed time= 2.991397 s >> > iteration=4 elapsed time=20.496422 s >> > total elapsed time=63.824705 s >> > >> > filecount=500 >> > iteration=0 elapsed time=6.560811 s >> > iteration=1 elapsed time=0.229375 s >> > iteration=2 elapsed time=0.202973 s >> > iteration=3 elapsed time=0.203439 s >> > iteration=4 elapsed time=0.203095 s >> > total elapsed time=7.399693 s >> >> Following up to myself with one more data point: raising >> SHRINK_CACHE_MAX from 1000 to 20000 in gfs/dlm/lock_dlm.h helps >> significantly, but still isn't enough. Besides, I don't know what I'm >> doing. Should I tweak the surrounding #defines, too? > > SHRINK_CACHE_MAX is related to fcntl posix locks, did you intend to change > the app to use flock (which is much faster that fcntl)? >> > SHRINK_CACHE_MAX from 1000 to 20000 in gfs/dlm/lock_dlm.h helps > >> SHRINK_CACHE_MAX is related to fcntl posix locks, did you intend to change >> the app to use flock (which is much faster that fcntl)? > >> > significantly, but still isn't enough. Besides, I don't know what I'm >> > doing. Should I tweak the surrounding #defines, too? > > lock_dlm caches dlm locks for old plocks for a while in an attempt to > improve performance and reduce thrashing the dlm -- SHRINK_CACHE_MAX is > the max level of caching, it's fine to change it as you've done. The fact > that you're hitting it, though, indicates that your app is using plocks > more heavily than gfs/dlm are suited to handle. Switching to flock will > obviate all of this. (Or switching to the new dlm and cluster > infrastructure which has a completely different and far better approach to > plocks). > > Dave From wferi at niif.hu Sat Apr 21 11:59:52 2007 From: wferi at niif.hu (Ferenc Wagner) Date: Sat, 21 Apr 2007 13:59:52 +0200 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <20070328163850.GG22230@redhat.com> (David Teigland's message of "Wed, 28 Mar 2007 11:38:50 -0500") References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> Message-ID: <87wt06djk7.fsf@tac.ki.iif.hu> David Teigland writes: > On Wed, Mar 28, 2007 at 11:27:27AM -0500, David Teigland wrote: >> On Wed, Mar 28, 2007 at 05:48:10PM +0200, Wagner Ferenc wrote: >>> Ferenc Wagner writes: >>> >>>> There's a good bunch of RRDs in a directory. A script scans them for >>>> their last modification times, and then updates each in turn for a >>>> couple of times. The number of files scanned and the length of the >>>> update rounds are printed. The results are much different for 500 and >>>> 501 files: >>>> >>>> filecount=501 >>>> iteration=0 elapsed time=10.425568 s >>>> iteration=1 elapsed time= 9.766178 s >>>> iteration=2 elapsed time=20.14514 s >>>> iteration=3 elapsed time= 2.991397 s >>>> iteration=4 elapsed time=20.496422 s >>>> total elapsed time=63.824705 s >>>> >>>> filecount=500 >>>> iteration=0 elapsed time=6.560811 s >>>> iteration=1 elapsed time=0.229375 s >>>> iteration=2 elapsed time=0.202973 s >>>> iteration=3 elapsed time=0.203439 s >>>> iteration=4 elapsed time=0.203095 s >>>> total elapsed time=7.399693 s >>> >>> Following up to myself with one more data point: raising >>> SHRINK_CACHE_MAX from 1000 to 20000 in gfs/dlm/lock_dlm.h helps > >> SHRINK_CACHE_MAX is related to fcntl posix locks, did you intend to change >> the app to use flock (which is much faster that fcntl)? I modified librrd to use flock instead if the LIBRRD_USE_FLOCK environmental variable is defined. It makes this case evenly slow (with SHRINK_CACHE_MAX et al. left at their defaults): filecount=500 iteration=0 elapsed time=10.335737 s iteration=1 elapsed time=10.426273 s iteration=2 elapsed time=10.634286 s iteration=3 elapsed time=10.342333 s iteration=4 elapsed time=10.458371 s total elapsed time=52.197 s (The same for 501 files). strace -T reveals an irregular pattern here: filecount=5 flock(3, LOCK_EX) = 0 <0.000116> flock(3, LOCK_EX) = 0 <0.000261> flock(3, LOCK_EX) = 0 <0.000113> flock(3, LOCK_EX) = 0 <0.037657> flock(3, LOCK_EX) = 0 <0.000093> iteration=0 elapsed time=0.04244 s flock(3, LOCK_EX) = 0 <0.000094> flock(3, LOCK_EX) = 0 <0.037314> flock(3, LOCK_EX) = 0 <0.000105> flock(3, LOCK_EX) = 0 <0.038323> flock(3, LOCK_EX) = 0 <0.000087> iteration=1 elapsed time=0.079957 s [...] So I've got the impression that flock is slower than fcntl locking. The difference probably erodes with greater number of files, both being equally slow. I must be missing something, it's just the opposite of what you suggest. >> > significantly, but still isn't enough. Besides, I don't know what I'm >> > doing. Should I tweak the surrounding #defines, too? > > lock_dlm caches dlm locks for old plocks for a while in an attempt to > improve performance and reduce thrashing the dlm -- SHRINK_CACHE_MAX is > the max level of caching, it's fine to change it as you've done. The fact > that you're hitting it, though, indicates that your app is using plocks > more heavily than gfs/dlm are suited to handle. Switching to flock will > obviate all of this. (Or switching to the new dlm and cluster > infrastructure which has a completely different and far better approach to > plocks). The lock usage of my application is very simple, at least in this testing phase, like I described at the start of the thread. I'd be glad to go into more detail if you are willing to assess my use case. Since flock doesn't seem to help at all, I guess something is missing from the picture. Also, what's that new infrastructure? Do you mean GFS2? I read it was not production quality yet, so I didn't mean to try it. But again you may have got something else in your head... -- Thanks, Feri. From wferi at niif.hu Sat Apr 21 15:01:38 2007 From: wferi at niif.hu (Ferenc Wagner) Date: Sat, 21 Apr 2007 17:01:38 +0200 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <87wt06djk7.fsf@tac.ki.iif.hu> (Ferenc Wagner's message of "Sat, 21 Apr 2007 13:59:52 +0200") References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> <87wt06djk7.fsf@tac.ki.iif.hu> Message-ID: <87r6qdpy99.fsf@tac.ki.iif.hu> Ferenc Wagner writes: > So I've got the impression that flock is slower than fcntl locking. > The difference probably erodes with greater number of files, both > being equally slow. I must be missing something, it's just the > opposite of what you suggest. For bigger file lists flock seems indeed faster, although not that much. I'd like to see the 500-file fcntl behaviour in the 10000-file regime. Is that possible with proper tuning of the DLM #defines for example? I'll make some random changes, but would appreciate some guidance. Regards, Feri. FLOCK: filecount=500 iteration=0 elapsed time=10.326359 s iteration=1 elapsed time=10.470205 s iteration=2 elapsed time=10.302251 s iteration=3 elapsed time=10.302252 s iteration=4 elapsed time=10.562203 s total elapsed time=51.96327 s filecount=1000 iteration=0 elapsed time=21.540327 s iteration=1 elapsed time=21.522852 s iteration=2 elapsed time=21.860283 s iteration=3 elapsed time=21.196407 s iteration=4 elapsed time=21.420367 s total elapsed time=107.540236 s filecount=2000 iteration=0 elapsed time=46.920414 s iteration=1 elapsed time=42.492109 s iteration=2 elapsed time=42.864716 s iteration=3 elapsed time=43.092679 s iteration=4 elapsed time=43.556623 s total elapsed time=218.926541 s filecount=5000 iteration=0 elapsed time=115.27529 s iteration=1 elapsed time=102.525867 s iteration=2 elapsed time=102.666411 s iteration=3 elapsed time=104.190295 s iteration=4 elapsed time=105.114153 s total elapsed time=529.772016 s FCNTL: filecount=500 iteration=0 elapsed time=9.634164 s iteration=1 elapsed time=0.203278 s iteration=2 elapsed time=0.204565 s iteration=3 elapsed time=0.203895 s iteration=4 elapsed time=0.203649 s total elapsed time=10.449551 s filecount=1000 iteration=0 elapsed time=16.111548 s iteration=1 elapsed time=27.165518 s iteration=2 elapsed time=25.927519 s iteration=3 elapsed time=22.904105 s iteration=4 elapsed time=23.176309 s total elapsed time=115.284999 s filecount=2000 iteration=0 elapsed time=46.482252 s iteration=1 elapsed time=50.130122 s iteration=2 elapsed time=45.76822 s iteration=3 elapsed time=52.462871 s iteration=4 elapsed time=56.386634 s total elapsed time=251.230099 s filecount=5000 iteration=0 elapsed time=125.121509 s iteration=1 elapsed time=131.344402 s iteration=2 elapsed time=120.272978 s iteration=3 elapsed time=126.29043 s iteration=4 elapsed time=131.561647 s total elapsed time=634.590966 s From grimme at atix.de Sun Apr 22 08:17:33 2007 From: grimme at atix.de (Marc Grimme) Date: Sun, 22 Apr 2007 10:17:33 +0200 Subject: [Linux-cluster] Re: Samba on GFS problems changing directory ACLs In-Reply-To: <46262E51.80305@forschungsgruppe.de> References: <46262E51.80305@forschungsgruppe.de> Message-ID: <200704221017.33550.grimme@atix.de> On Wednesday 18 April 2007 16:42:25 Christian Brandes wrote: > Hi Marc, > > > what GFS version are you using? > > That was a bug in old GFS versions when setting acls (IMHO kernel > > > 2.6.9-43.x.y shoud have this fixed). > > I am using a linux kernel compiled from Ubuntu linux sources 2.6.15. > > How can I see what gfs version it is actually? > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Hmm, how about modinfo gfs? I don't know but e.g. modinfo on RHEL4 shows something that looks like a version. filename: /lib/modules/2.6.9-42.0.3.ELsmp/kernel/fs/gfs/gfs.ko description: Global File System 2.6.9-60.3 author: Red Hat, Inc. license: GPL vermagic: 2.6.9-42.0.3.ELsmp SMP 686 REGPARM 4KSTACKS gcc-3.4 depends: lock_harness,cman Isn't there a way to show the version and compiletime or some more informations of debs (dpkg --something). Are not your gfs-mods installed via apt or did you compile them right away? -- Gruss / Regards, Marc Grimme Phone: +49-89 452 3538-14 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany Registergericht: Amtsgericht M?nchen Registernummer: HRB 131682 USt.-Id.: DE209485962 Gesch?ftsf?hrung: Marc Grimme, Mark Hlawatschek, Thomas Merz From simone.gotti at email.it Mon Apr 23 09:15:22 2007 From: simone.gotti at email.it (Simone Gotti) Date: Mon, 23 Apr 2007 11:15:22 +0200 Subject: [Linux-cluster] [RFC][PATCH] Add ability to freeze e service. Message-ID: <1177319722.3870.29.camel@localhost> Hi, like discussed with lon on IRC I'm trying to add to rgmanager the ability to freeze a service. I worked on it in these days and did an example patch. Here is how I think what a "freeze" can be and, of course, it can be implemented in many other ways so it's only an example. == What freeze means? == All actions on the service are blocked (start, stop, status) so you can work by hand on the various resources. When you unfreeze the service everything returns as before (so if you manually stopped a resource then the status will fail and the rg recovery is done). == When does a service can be freezed? == You can freeze only if the service status is DISABLED, STOPPED, or STARTED. It doesn't have sense to freeze a service that is in a transictional state. == How is it implemented? == *) As I don't want to lose the previous state and I don't think it's a service state, "freezed" is implemented like a service flag. As a "service flag" didn't existed before, this patch adds it to rg_state_t, so it will be transmitted around the cluster. *) Two options are added to clusvcasm (-F to freeze, -U to unfreeze), obviously these options names can be changed (perhaps they can be only a long option like --freeze, --unfreeze?). So you can freeze with: #clusvcadm -F $SERVICE and unfreeze with: #clusvcadm -U $SERVICE *) clustat reports these new flags in 2 ways: on normal mode the flags are between () and in long mode e new line "Flags:" is added. The functions added in rg_strings.c aren't well tested but should work with multiple flags. *) !!!! In the patch I haven't changed the function handle_start_remote_req because looking at the code I cannot find when it can be called. Maybe I'm missing something... :D Thanks! Bye! -- Simone Gotti -- Email.it, the professional e-mail, gratis per te: http://www.email.it/f Sponsor: Lo sai che hai un tesoro in soffitta? Quello che non serve pi? a te, pu? servire agli altri. * Vendi GRATIS ci? che vuoi con AdBoom.it Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=6418&d=23-4 -------------- next part -------------- A non-text attachment was scrubbed... Name: rgmanager-cvsHEAD-add_service_freezing-try01.patch Type: text/x-patch Size: 13527 bytes Desc: not available URL: From sebastian.walter at fu-berlin.de Mon Apr 23 09:34:27 2007 From: sebastian.walter at fu-berlin.de (Sebastian Walter) Date: Mon, 23 Apr 2007 11:34:27 +0200 Subject: [Linux-cluster] strange ccsd behavior Message-ID: <462C7DA3.1020309@fu-berlin.de> Dear list, my cluster suite doesn't start up. Some nodes work, but some give me ccsd -n Starting ccsd 1.0.7: Built: Aug 26 2006 15:01:49 Copyright (C) Red Hat, Inc. 2004 All rights reserved. No Daemon:: SET Unable to connect to cluster infrastructure after 30 seconds. Unable to connect to cluster infrastructure after 60 seconds. Unable to connect to cluster infrastructure after 90 seconds. Unable to connect to cluster infrastructure after 120 seconds. Unable to connect to cluster infrastructure after 150 seconds. Cluster.conf is the some on all nodes. Hostnames are correct, the firewalls open for the appropriate subnet. The strange thing is that sometimes it works. On the other cluster nodes I don't see any messages. How can I debug this? System: CentOS 4.4 Version: ccsd 1.0.7 (built Aug 26 2006 15:01:49) Thanks in advance, Sebastian From hlawatschek at atix.de Mon Apr 23 11:06:56 2007 From: hlawatschek at atix.de (Mark Hlawatschek) Date: Mon, 23 Apr 2007 13:06:56 +0200 Subject: [Linux-cluster] strange ccsd behavior In-Reply-To: <462C7DA3.1020309@fu-berlin.de> References: <462C7DA3.1020309@fu-berlin.de> Message-ID: <200704231306.56258.hlawatschek@atix.de> Hi, I think you need to start cman. cman_tool join -w would do that. Mark On Monday 23 April 2007 11:34:27 Sebastian Walter wrote: > Dear list, > > my cluster suite doesn't start up. Some nodes work, but some give me > > ccsd -n > Starting ccsd 1.0.7: > Built: Aug 26 2006 15:01:49 > Copyright (C) Red Hat, Inc. 2004 All rights reserved. > No Daemon:: SET > > Unable to connect to cluster infrastructure after 30 seconds. > Unable to connect to cluster infrastructure after 60 seconds. > Unable to connect to cluster infrastructure after 90 seconds. > Unable to connect to cluster infrastructure after 120 seconds. > Unable to connect to cluster infrastructure after 150 seconds. > > > Cluster.conf is the some on all nodes. Hostnames are correct, the > firewalls open for the appropriate subnet. The strange thing is that > sometimes it works. On the other cluster nodes I don't see any messages. > How can I debug this? > > System: CentOS 4.4 > Version: ccsd 1.0.7 (built Aug 26 2006 15:01:49) > > Thanks in advance, > Sebastian > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Gruss / Regards, Dipl.-Ing. Mark Hlawatschek http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From christian.brandes at forschungsgruppe.de Mon Apr 23 16:29:10 2007 From: christian.brandes at forschungsgruppe.de (Christian Brandes) Date: Mon, 23 Apr 2007 18:29:10 +0200 Subject: [Linux-cluster] Samba on GFS problems changing, directory ACLs In-Reply-To: <20070422160006.2676972FA0@hormel.redhat.com> References: <20070422160006.2676972FA0@hormel.redhat.com> Message-ID: <462CDED6.6030400@forschungsgruppe.de> >> How can I see what gfs version it is actually? > Hmm, how about modinfo gfs? filename: /lib/modules/2.6.15.7-ubuntu1-xxx-xxx-firepig-2.4/kernel/fs/gfs/gfs.ko description: Global File System author: Red Hat, Inc. license: GPL vermagic: 2.6.15.7-ubuntu1-xxx-xxx-firepig-2.4 preempt PENTIUM4 gcc-4.0 depends: lock_harness That shows only my kernel version number and that GFS was taken from CVS. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4348 bytes Desc: S/MIME Cryptographic Signature URL: From isplist at logicore.net Mon Apr 23 16:12:47 2007 From: isplist at logicore.net (isplist at logicore.net) Date: Mon, 23 Apr 2007 11:12:47 -0500 Subject: [Linux-cluster] Kernel Crashes on all nodes when one dies Message-ID: <2007423111247.390108@leena> Someone accidentally messed up iptables on a node so that it could no longer communicate with the cluster. That should have been the end of the problem, one node down but instead, all nodes died with a kernel crash. Here is a paste from one of the log's. I think this is the right section which shows the dying nodes; Mike Apr 22 11:55:26 qm250 kernel: qm move flags 0,1,0 ids 0,3,0 Apr 22 11:55:26 qm250 kernel: qm move use event 3 Apr 22 11:55:26 qm250 kernel: qm recover event 3 (first) Apr 22 11:55:26 qm250 kernel: qm add nodes Apr 22 11:55:26 qm250 kernel: qm total nodes 2 Apr 22 11:55:26 qm250 kernel: qm rebuild resource directory Apr 22 11:55:26 qm250 kernel: qm rebuilt 8 resources Apr 22 11:55:26 qm250 kernel: qm recover event 3 done Apr 22 11:55:26 qm250 kernel: qm move flags 0,0,1 ids 0,3,3 Apr 22 11:55:26 qm250 kernel: qm process held requests Apr 22 11:55:26 qm250 kernel: qm processed 0 requests Apr 22 11:55:26 qm250 kernel: qm recover event 3 finished Apr 22 11:55:26 qm250 kernel: clvmd move flags 1,0,0 ids 2,2,2 Apr 22 11:55:26 qm250 kernel: qm move flags 1,0,0 ids 3,3,3 Apr 22 11:55:26 qm250 kernel: 2640 pr_start last_stop 0 last_start 4 last_finish 0 Apr 22 11:55:26 qm250 kernel: 2640 pr_start count 2 type 2 event 4 flags 250 Apr 22 11:55:26 qm250 kernel: 2640 claim_jid 1 Apr 22 11:55:26 qm250 kernel: 2640 pr_start 4 done 1 Apr 22 11:55:26 qm250 kernel: 2640 pr_finish flags 5a Apr 22 11:55:27 qm250 kernel: 2566 recovery_done jid 1 msg 309 a Apr 22 11:55:27 qm250 kernel: 2566 recovery_done nodeid 250 flg 18 Apr 22 11:55:27 qm250 kernel: Apr 22 11:55:27 qm250 kernel: lock_dlm: Assertion failed on line 357 of file /home/buildcentos/rpmbuild/BUILD/gf s-kernel-2.6.9-60/up/src/dlm/lock.c Apr 22 11:55:27 qm250 kernel: lock_dlm: assertion: "!error" Apr 22 11:55:27 qm250 kernel: lock_dlm: time = 14525882 Apr 22 11:55:27 qm250 kernel: qm: error=-22 num=2,1a lkf=10000 flags=84 Apr 22 11:55:27 qm250 kernel: Apr 22 11:55:27 qm250 kernel: ------------[ cut here ]------------ Apr 22 11:55:27 qm250 kernel: kernel BUG at /home/buildcentos/rpmbuild/BUILD/gfs-kernel-2.6.9-60/up/src/dlm/lock. c:357! Apr 22 11:55:27 qm250 kernel: invalid operand: 0000 [#1] Apr 22 11:55:27 qm250 kernel: Modules linked in: lock_dlm(U) gfs(U) lock_harness(U) parport_pc lp parport autofs4 dlm(U) cman(U) md5 ipv6 sunrpc dm_mirror dm_mod uhci_hcd e100 mii floppy ext3 jbd qla2200 qla2xxx scsi_transport _fc sd_mod scsi_mod Apr 22 11:55:27 qm250 kernel: CPU: 0 Apr 22 11:55:27 qm250 kernel: EIP: 0060:[] Not tainted VLI Apr 22 11:55:27 qm250 kernel: EFLAGS: 00010246 (2.6.9-42.0.3.EL) Apr 22 11:55:27 qm250 kernel: EIP is at do_dlm_unlock+0x89/0x9e [lock_dlm] Apr 22 11:55:27 qm250 kernel: eax: 00000001 ebx: dfd552e0 ecx: e09b089f edx: dafe9f44 Apr 22 11:55:27 qm250 kernel: esi: ffffffea edi: dfd552e0 ebp: e0a62000 esp: dafe9f40 Apr 22 11:55:27 qm250 kernel: ds: 007b es: 007b ss: 0068 Apr 22 11:55:27 qm250 kernel: Process gfs_glockd (pid: 2647, threadinfo=dafe9000 task=de442c50) Apr 22 11:55:27 qm250 kernel: Stack: e09b089f e0a62000 00000003 e09aafff e0ae3e51 dfd7d4ac e0a62000 e0b156c0 Apr 22 11:55:27 qm250 kernel: e0ad6bd4 dfd7d4ac e0b156c0 dafe9fb4 e0ad5683 dfd7d4ac 00000001 e0ad5840 Apr 22 11:55:27 qm250 kernel: dfd7d4ac dfd7d4ac e0ad5af9 dfd7d550 e0ad9182 dafe9000 dafe9fc0 e0ac8e9a Apr 22 11:55:27 qm250 kernel: Call Trace: Apr 22 11:55:27 qm250 kernel: [] lm_dlm_unlock+0x13/0x1b [lock_dlm] Apr 22 11:55:27 qm250 kernel: [] gfs_lm_unlock+0x2b/0x40 [gfs] Apr 22 11:55:27 qm250 kernel: [] gfs_glock_drop_th+0x17a/0x1b0 [gfs] Apr 22 11:55:27 qm250 kernel: [] rq_demote+0x15c/0x1da [gfs] Apr 22 11:55:27 qm250 kernel: [] run_queue+0x5a/0xc1 [gfs] Apr 22 11:55:27 qm250 kernel: [] unlock_on_glock+0x6e/0xc8 [gfs] Apr 22 11:55:27 qm250 kernel: [] gfs_reclaim_glock+0x257/0x2ae [gfs] Apr 22 11:55:27 qm250 kernel: [] gfs_glockd+0x38/0xde [gfs] Apr 22 11:55:27 qm250 kernel: [] default_wake_function+0x0/0xc Apr 22 11:55:27 qm250 kernel: [] ret_from_fork+0x6/0x14 Apr 22 11:55:27 qm250 kernel: [] default_wake_function+0x0/0xc Apr 22 11:55:28 qm250 kernel: [] gfs_glockd+0x0/0xde [gfs] Apr 22 11:55:28 qm250 kernel: [] kernel_thread_helper+0x5/0xb Apr 22 11:55:28 qm250 kernel: Code: 73 34 8b 03 ff 73 2c ff 73 08 ff 73 04 ff 73 0c 56 ff 70 18 68 ac 09 9b e0 e8 10 9c 77 df 83 c4 34 68 9f 08 9b e0 e8 03 9c 77 df <0f> 0b 65 01 2e 07 9b e0 68 a1 08 9b e0 e8 5b 90 77 df 5b 5e c3 Apr 22 11:55:28 qm250 kernel: <0>Fatal exception: panic in 5 seconds From rmccabe at redhat.com Mon Apr 23 18:04:42 2007 From: rmccabe at redhat.com (Ryan McCabe) Date: Mon, 23 Apr 2007 14:04:42 -0400 Subject: [Linux-cluster] Workaround for breakage caused by xend bridged networking Message-ID: <20070423180442.GA19224@redhat.com> Hi, Attached (and inline for review) is a proposed patch to work around the problems xend's bridged networking startup causes for RHCS as a result of the init script ordering and the network disruption caused by the code that creates the bridge interfaces used by xend. If xend will start in the current runlevel and is and configured to use bridged networking, the cman init script will run '/etc/xen/scripts/network-bridge start' before doing anything else. This will create the network bridges early--a process that entails briging down and renaming interfaces, which causes all kinds of breakage currently--before openais starts. xend doesn't mind if the network bridges are setup before it starts, and the bridges need not be removed by the cman init script when cman stops: xend will remove them when it stops. Ryan Index: cman =================================================================== RCS file: /cvs/cluster/cluster/cman/init.d/cman,v retrieving revision 1.26.4.2 diff -u -r1.26.4.2 cman --- cman 15 Nov 2006 16:55:16 -0000 1.26.4.2 +++ cman 23 Apr 2007 17:46:03 -0000 @@ -132,6 +132,47 @@ return 0 } +xend_bridged_net_enabled() { + current_runlevel=$(/sbin/runlevel 2>/dev/null | awk '{ print $2 }' 2>/dev/null) + if [ -z "$current_runlevel" ]; then + errmsg='Unable to determine the current runlevel' + return 1 + fi + + /sbin/chkconfig --levels "$current_runlevel" xend 2>/dev/null + if [ $? -ne 0 ]; then + # xend doesn't start at this runlevel. + return 1 + fi + + if [ ! -f /etc/xen/xend-config.sxp ]; then + # xend isn't configured to use bridged networking. + return 1 + fi + + egrep "^[[:blank:]]*\([[:blank:]]*network-script[[:blank:]]+network-bridge([[:blank:]]*\)|[[:blank:]]+)" /etc/xen/xend-config.sxp 2>/dev/null + if [ $? -ne 0 ]; then + # xend isn't configured to use bridged networking. + return 1 + fi + return 0 +} + +xend_bridged_net_start() { + if [ ! -x /etc/xen/scripts/network-bridge ]; then + if [ -f /etc/xen/scripts/network-bridge ]; then + errmsg='The xend bridged network script cannot be run' + else + errmsg='The xend bridged network script is missing' + fi + return 1 + fi + + /sbin/modprobe netbk >& /dev/null + /sbin/modprobe netloop >& /dev/null + errmsg=$(/etc/xen/scripts/network-bridge start 2>&1) || return 1 + return 0 +} fence_xvmd_enabled() { @@ -163,6 +204,20 @@ start() { echo "Starting cluster: " + + xend_bridged_net_enabled + if [ $? -eq 0 ] + then + echo -n " Enabling workaround for Xend bridged networking... " + xend_bridged_net_start + if [ $? -eq 0 ] + then + echo "done" + else + echo "failed: $errmsg" + return 1 + fi + fi echo -n " Loading modules... " ulimit -c unlimited load_modules -------------- next part -------------- ? cman-init.diff Index: cman =================================================================== RCS file: /cvs/cluster/cluster/cman/init.d/cman,v retrieving revision 1.26.4.2 diff -u -r1.26.4.2 cman --- cman 15 Nov 2006 16:55:16 -0000 1.26.4.2 +++ cman 23 Apr 2007 17:46:03 -0000 @@ -132,6 +132,47 @@ return 0 } +xend_bridged_net_enabled() { + current_runlevel=$(/sbin/runlevel 2>/dev/null | awk '{ print $2 }' 2>/dev/null) + if [ -z "$current_runlevel" ]; then + errmsg='Unable to determine the current runlevel' + return 1 + fi + + /sbin/chkconfig --levels "$current_runlevel" xend 2>/dev/null + if [ $? -ne 0 ]; then + # xend doesn't start at this runlevel. + return 1 + fi + + if [ ! -f /etc/xen/xend-config.sxp ]; then + # xend isn't configured to use bridged networking. + return 1 + fi + + egrep "^[[:blank:]]*\([[:blank:]]*network-script[[:blank:]]+network-bridge([[:blank:]]*\)|[[:blank:]]+)" /etc/xen/xend-config.sxp 2>/dev/null + if [ $? -ne 0 ]; then + # xend isn't configured to use bridged networking. + return 1 + fi + return 0 +} + +xend_bridged_net_start() { + if [ ! -x /etc/xen/scripts/network-bridge ]; then + if [ -f /etc/xen/scripts/network-bridge ]; then + errmsg='The xend bridged network script cannot be run' + else + errmsg='The xend bridged network script is missing' + fi + return 1 + fi + + /sbin/modprobe netbk >& /dev/null + /sbin/modprobe netloop >& /dev/null + errmsg=$(/etc/xen/scripts/network-bridge start 2>&1) || return 1 + return 0 +} fence_xvmd_enabled() { @@ -163,6 +204,20 @@ start() { echo "Starting cluster: " + + xend_bridged_net_enabled + if [ $? -eq 0 ] + then + echo -n " Enabling workaround for Xend bridged networking... " + xend_bridged_net_start + if [ $? -eq 0 ] + then + echo "done" + else + echo "failed: $errmsg" + return 1 + fi + fi echo -n " Loading modules... " ulimit -c unlimited load_modules From lhh at redhat.com Mon Apr 23 20:02:51 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 23 Apr 2007 16:02:51 -0400 Subject: [Linux-cluster] [RFC][PATCH] Add ability to freeze e service. In-Reply-To: <1177319722.3870.29.camel@localhost> References: <1177319722.3870.29.camel@localhost> Message-ID: <20070423200249.GB4897@redhat.com> On Mon, Apr 23, 2007 at 11:15:22AM +0200, Simone Gotti wrote: > Hi, > > like discussed with lon on IRC I'm trying to add to rgmanager the > ability to freeze a service. I worked on it in these days and did an > example patch. Here is how I think what a "freeze" can be and, of > course, it can be implemented in many other ways so it's only an > example. Hi, couple of comments - (1) s/freezed/frozen/ig :) (2) svc_status_inquiry shouldn't call rg_lock(); you should use get_rg_state_local() instead of rg_lock/get_rg_state/rg_unlock Note that svc_status_inquiry isn't even used yet, so the behavior might change at a later date ;) (3.a) Warning: The addition of rs_flags to rg_state_t will break compatibility with existing software (which is fine in -HEAD). It would be easier to cope with this change (that is- make upgrades from old->new work) if you added the rs_flags field after the rs_transition field. (3.b) If you don't want to do that, then you need to add another 32-bits worth of data (could be just a "pad" field) before rs_transition because the kernel on ia64 machines will complain if they read a 64-bit int and it's not aligned on a 64-bit boundary. Aside from that, it looks like you got it right for what you wanted it to do. I can fix (1) (language stuff) if you fix (2) and (3). -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From teigland at redhat.com Mon Apr 23 21:17:18 2007 From: teigland at redhat.com (David Teigland) Date: Mon, 23 Apr 2007 16:17:18 -0500 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <87wt06djk7.fsf@tac.ki.iif.hu> References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> <87wt06djk7.fsf@tac.ki.iif.hu> Message-ID: <20070423211717.GA22147@redhat.com> On Sat, Apr 21, 2007 at 01:59:52PM +0200, Ferenc Wagner wrote: > I modified librrd to use flock instead if the LIBRRD_USE_FLOCK > environmental variable is defined. It makes this case evenly slow > (with SHRINK_CACHE_MAX et al. left at their defaults): > > filecount=500 > iteration=0 elapsed time=10.335737 s > iteration=1 elapsed time=10.426273 s > iteration=2 elapsed time=10.634286 s > iteration=3 elapsed time=10.342333 s > iteration=4 elapsed time=10.458371 s > total elapsed time=52.197 s > > (The same for 501 files). strace -T reveals an irregular pattern here: > > filecount=5 > flock(3, LOCK_EX) = 0 <0.000116> > flock(3, LOCK_EX) = 0 <0.000261> > flock(3, LOCK_EX) = 0 <0.000113> > flock(3, LOCK_EX) = 0 <0.037657> > flock(3, LOCK_EX) = 0 <0.000093> > iteration=0 elapsed time=0.04244 s > flock(3, LOCK_EX) = 0 <0.000094> > flock(3, LOCK_EX) = 0 <0.037314> > flock(3, LOCK_EX) = 0 <0.000105> > flock(3, LOCK_EX) = 0 <0.038323> > flock(3, LOCK_EX) = 0 <0.000087> > iteration=1 elapsed time=0.079957 s > [...] I suspect what's happening here is that another node is mounting the fs, causing some of the dlm resource-directory lookups to be remote -- those are the ones taking the longer time. To verify, you could run this same test with just the one node mounting the fs, then all should be uniformly quick. No way to change this. > So I've got the impression that flock is slower than fcntl locking. > The difference probably erodes with greater number of files, both > being equally slow. I must be missing something, it's just the > opposite of what you suggest. I suspect that the faster fcntl results below 1000 (shown in your other email) are a result of the cache you were adjusting of recently used dlm locks. Once the number of files/locks exceeds SHRINK_CACHE_MAX, then the cache is useless and you see plocks a bit slower than the flocks (although I'm surprised, I expected them to be slower than you've shown.) I'm going to write a quick test to simulate what you're doing here to verify these suspicions. > Also, what's that new infrastructure? Do you mean GFS2? I read it > was not production quality yet, so I didn't mean to try it. But again > you may have got something else in your head... GFS1 and GFS2 both run on the new openais-based cluster infrastructure. (in the cluster-2.00.00 release, and the RHEL5 and HEAD cvs branches). Dave From pgharios at gmail.com Tue Apr 24 09:43:03 2007 From: pgharios at gmail.com (Patrick Gharios) Date: Tue, 24 Apr 2007 12:43:03 +0300 Subject: [Linux-cluster] Resource Script. Message-ID: <000001c78655$0ca1a210$3b6410ac@snoopy> Hello, Question: In order to run a script (bash script on a screen for example) by the cluster service, do I need to include a start argument when calling the execution of that script? I am trying to execute a simple script but I noticed in /var/log/messages that the cluster is trying to send a 'start' argument to the script , the script is not being fired up on the node and then a 'status' call is executed on the simple script. Seems like it's a monitoring feature. In in all, can someone tell me how can I execute a simple bash script on the cluster service? Thank you, Pat. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian.walter at fu-berlin.de Tue Apr 24 10:49:10 2007 From: sebastian.walter at fu-berlin.de (Sebastian Walter) Date: Tue, 24 Apr 2007 12:49:10 +0200 Subject: [Linux-cluster] running cs on hosts with two NIC's In-Reply-To: <000001c78655$0ca1a210$3b6410ac@snoopy> References: <000001c78655$0ca1a210$3b6410ac@snoopy> Message-ID: <462DE0A6.5060505@fu-berlin.de> Hello List, I'd like to run the cluster suite on computers with two NIC's. One of them is configured as 10.2.1.1/16, tho other one is our public ip range. I set up both the public addresses in /etc/hosts. Unfortunately, /proc/cluster/status states the private ip addresses instead of the public ones. How can I force cman to use the public addresses and therefore get the cluster being accessible for other hosts? Thanks in advance! Sebastian From james.lapthorn at lapthornconsulting.com Tue Apr 24 11:31:18 2007 From: james.lapthorn at lapthornconsulting.com (James Lapthorn) Date: Tue, 24 Apr 2007 12:31:18 +0100 Subject: [Linux-cluster] Fencing using APC7921 Message-ID: I use RSA II cards as my primary fencing device on my IBM 3850's. After some modifications to the 'fence_rsa' script it now recognises 'WMN170370264' as the default card name. My secondary fencing device is and APC Power Switch APC7921. This is not a Masterswitch plus so there are two APC7921's, one for each POwer supply. When testing the the device using the following command: fence_apc -a 10.179.20.52 -l apc -p apc -n 4 -v I get: failed: unrecognised menu response Here;s is the output from the /tmp/apclog.txt ***************************************************************************************************************** User Name : apc Password : *** American Power Conversion Network Management Card AOS v2.7.0 (c) Copyright 2004 All Rights Reserved Rack PDU APP v2.7.3 ------------------------------------------------------------------------------- Name : RackPDU Date : 10/24/2000 Contact : Unknown Time : 11:45:43 Location : Unknown User : Administrator Up Time : 64 Days 0 Hours 51 Minutes Stat : P+ N+ A+ Switched Rack PDU: Communication Established ------- Control Console ------------------------------------------------------- 1- Device Manager 2- Network 3- System 4- Logout - Main Menu, - Refresh, - Event Log > ------- Control Console ------------------------------------------------------- 1- Device Manager 2- Network 3- System 4- Logout - Main Menu, - Refresh, - Event Log > American Power Conversion Network Management Card AOS v2.7.0 (c) Copyright 2004 All Rights Reserved Rack PDU APP v2.7.3 ------------------------------------------------------------------------------- Name : RackPDU Date : 10/24/2000 Contact : Unknown Time : 11:45:43 Location : Unknown User : Administrator Up Time : 64 Days 0 Hours 51 Minutes Stat : P+ N+ A+ Switched Rack PDU: Communication Established ------- Control Console ------------------------------------------------------- 1- Device Manager 2- Network 3- System 4- Logout - Main Menu, - Refresh, - Event Log > ------- Control Console ------------------------------------------------------- 1- Device Manager 2- Network 3- System 4- Logout - Main Menu, - Refresh, - Event Log > 1 ------- Device Manager -------------------------------------------------------- 1- Phase Monitor/Configuration 2- Outlet Restriction Configuration 3- Outlet Control/Configuration 4- Power Supply Status - Back, - Refresh, - Event Log > 3 ------- Outlet Control/Configuration ------------------------------------------ 1- leoukpdb4 ON 2- leoukldb4 ON 3- leoukpdb3 ON 4- leoukldb3 ON 5- leoukpdb2 ON 6- leoukldb2 ON 7- leoukpdb1 ON 8- leoukldb1 ON 9- Master Control/Configuration - Back, - Refresh, - Event Log > ------- Outlet Control/Configuration ------------------------------------------ 1- leoukpdb4 ON 2- leoukldb4 ON 3- leoukpdb3 ON 4- leoukldb3 ON 5- leoukpdb2 ON 6- leoukldb2 ON 7- leoukpdb1 ON 8- leoukldb1 ON 9- Master Control/Configuration - Back, - Refresh, - Event Log > ------- Device Manager -------------------------------------------------------- 1- Phase Monitor/Configuration 2- Outlet Restriction Configuration 3- Outlet Control/Configuration 4- Power Supply Status - Back, - Refresh, - Event Log > ------- Control Console ------------------------------------------------------- 1- Device Manager 2- Network 3- System 4- Logout - Main Menu, - Refresh, - Event Log ***************************************************************************************************************** Has anybody else had problem with the APC7921. I am going to assume it's because the APC want the telnet session to type 'YES' to confirm. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian.walter at fu-berlin.de Tue Apr 24 12:20:48 2007 From: sebastian.walter at fu-berlin.de (Sebastian Walter) Date: Tue, 24 Apr 2007 14:20:48 +0200 Subject: [Linux-cluster] running cs on hosts with two NIC's In-Reply-To: <462DE0A6.5060505@fu-berlin.de> References: <000001c78655$0ca1a210$3b6410ac@snoopy> <462DE0A6.5060505@fu-berlin.de> Message-ID: <462DF620.3070104@fu-berlin.de> I solved it by using the same hostnames with different domain names e.g. node1.local node1.public.domain and inserting the complete public FQHN in custer.conf Sebastian Walter wrote: > How can I force cman to use the public addresses and therefore get the > cluster being accessible for other hosts? From berthiaume_wayne at emc.com Tue Apr 24 13:19:17 2007 From: berthiaume_wayne at emc.com (berthiaume_wayne at emc.com) Date: Tue, 24 Apr 2007 09:19:17 -0400 Subject: [Linux-cluster] running cs on hosts with two NIC's In-Reply-To: <462DE0A6.5060505@fu-berlin.de> References: <000001c78655$0ca1a210$3b6410ac@snoopy> <462DE0A6.5060505@fu-berlin.de> Message-ID: Here's how I did it in a three node iSCSI cluster... [root at l82bi121 ~]# cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 172.23.189.121 l82bi121.lss.emc.com l82bi121 172.23.189.212 l82bi212.lss.emc.com l82bi212 172.23.189.36 l82bi036.lss.emc.com l82bi036 10.10.10.36 node036.gfs node036 10.10.10.121 node121.gfs node121 10.10.10.212 node212.gfs node212 This host also contains two other NICs being used for iSCSI connectivity. [root at l82bi121 ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:04:23:AC:84:E6 inet addr:10.10.10.121 Bcast:10.10.10.255 Mask:255.255.255.0 inet6 addr: fe80::204:23ff:feac:84e6/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1410177 errors:0 dropped:0 overruns:0 frame:0 TX packets:698571 errors:0 dropped:0 overruns:0 carrier:0 collisions:329 txqueuelen:100 RX bytes:104886622 (100.0 MiB) TX bytes:52137668 (49.7 MiB) Base address:0xa000 Memory:fc7c0000-fc7e0000 eth1 Link encap:Ethernet HWaddr 00:04:23:AC:84:E7 inet addr:51.51.51.121 Bcast:51.51.51.255 Mask:255.255.255.0 inet6 addr: fe80::204:23ff:feac:84e7/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3254248 errors:0 dropped:0 overruns:0 frame:0 TX packets:567156 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:274268921 (261.5 MiB) TX bytes:70167976 (66.9 MiB) Base address:0xac00 Memory:fc7e0000-fc800000 eth2 Link encap:Ethernet HWaddr 00:02:B3:EA:75:90 inet addr:51.50.51.121 Bcast:51.50.51.255 Mask:255.255.255.0 inet6 addr: fe80::202:b3ff:feea:7590/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1966589 errors:0 dropped:0 overruns:0 frame:0 TX packets:1969235 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:319932260 (305.1 MiB) TX bytes:249523544 (237.9 MiB) Base address:0xb800 Memory:fc8c0000-fc8e0000 eth3 Link encap:Ethernet HWaddr 00:02:B3:EA:75:91 inet addr:172.23.189.121 Bcast:172.23.189.255 Mask:255.255.255.0 inet6 addr: fe80::202:b3ff:feea:7591/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:60138714 errors:0 dropped:0 overruns:0 frame:0 TX packets:69530472 errors:213 dropped:0 overruns:0 carrier:213 collisions:2809411 txqueuelen:10 RX bytes:5321507661 (4.9 GiB) TX bytes:103854304295 (96.7 GiB) Base address:0xbc00 Memory:fc8e0000-fc900000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:8456866 errors:0 dropped:0 overruns:0 frame:0 TX packets:8456866 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:581951659 (554.9 MiB) TX bytes:581951659 (554.9 MiB) Note the cluster node names... [root at l82bi121 ~]# cat /etc/cluster/cluster.conf [root at l82bi121 ~]# Regards, Wayne. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Sebastian Walter Sent: Tuesday, April 24, 2007 6:49 AM To: linux clustering Subject: [Linux-cluster] running cs on hosts with two NIC's Hello List, I'd like to run the cluster suite on computers with two NIC's. One of them is configured as 10.2.1.1/16, tho other one is our public ip range. I set up both the public addresses in /etc/hosts. Unfortunately, /proc/cluster/status states the private ip addresses instead of the public ones. How can I force cman to use the public addresses and therefore get the cluster being accessible for other hosts? Thanks in advance! Sebastian -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From beres.laszlo at sys-admin.hu Tue Apr 24 13:40:22 2007 From: beres.laszlo at sys-admin.hu (BERES Laszlo) Date: Tue, 24 Apr 2007 15:40:22 +0200 Subject: [Linux-cluster] HP iLO fencing issues Message-ID: <462E08C6.1030707@sys-admin.hu> Hi all, I know this list is not for HP support, but do you have any kind of (negative) experience with iLO2 fencing? Our customer has sleepless nights, because his new HP Blade's management is failing: ccsd[5659]: process_get: Invalid connection descriptor received. ccsd[5659]: Error while processing get: Invalid request descriptor fenced[5761]: fence "host1" failed -- B?RES L?szl? RHCE, RHCX senior IT engineer, trainer From rpeterso at redhat.com Tue Apr 24 13:58:53 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Tue, 24 Apr 2007 08:58:53 -0500 Subject: [Linux-cluster] Resource Script. In-Reply-To: <000001c78655$0ca1a210$3b6410ac@snoopy> References: <000001c78655$0ca1a210$3b6410ac@snoopy> Message-ID: <462E0D1D.1000608@redhat.com> Patrick Gharios wrote: > Hello, > > > > Question: > > > > In order to run a script (bash script on a screen for example) by the > cluster service, do I need to include a start argument when calling the > execution of that script? I am trying to execute a simple script but I > noticed in /var/log/messages that the cluster is trying to send a 'start' > argument to the script , the script is not being fired up on the node and > then a 'status' call is executed on the simple script. Seems like it's a > monitoring feature. > > > > In in all, can someone tell me how can I execute a simple bash script on the > cluster service? > > > > Thank you, Pat. Hi Pat, The rgmanager uses start/stop/status similar to when init calls the init scripts, i.e., service start. I recommend making your script look like the other resource agents in /usr/share/cluster. Regards, Bob Peterson Red Hat Cluster Suite From jparsons at redhat.com Tue Apr 24 14:13:28 2007 From: jparsons at redhat.com (James Parsons) Date: Tue, 24 Apr 2007 10:13:28 -0400 Subject: [Linux-cluster] Fencing using APC7921 In-Reply-To: References: Message-ID: <462E1088.6060505@redhat.com> James Lapthorn wrote: > I use RSA II cards as my primary fencing device on my IBM 3850's. > After some modifications to the 'fence_rsa' script it now recognises > 'WMN170370264' as the default card name. Hi James - which version of cluster suite are you running? fence_rsa was recently modified to handle prompts other than the default...I wonder if this is what you encountered. The fix went out in the rhel5 distribution, and will be in rhel4 update 5. If there is something else amiss with the rsa agent, please consider filing a bugzilla for it - or maybe you could send me the patch you applied? I would like to keep that agent as current as possible. > > > My secondary fencing device is and APC Power Switch APC7921. This is > not a Masterswitch plus so there are two APC7921's, one for each POwer > supply. When testing the the device using the following command: > > fence_apc -a 10.179.20.52 -l apc -p apc -n 4 -v > > I get: > > failed: unrecognised menu response In your apc fence log, the problem appears to be the agent not recognizing the custom names you have set up for the outlets. The latest version of the fence_apc agent (which was heavily refactored - completely re-written from the ground up, actually) should handle custom outlet names...the only problem is - once you change the default name from 'Outlet 4' to 'leoukldb3 ', you have to use that name. If you are running the latest agent (it is written in python instead of perl, like the original agent), could you please try re-running the command above with the following args? fence_apc -a 10.179.20.52 -l apc -p apc -n leoukldb3 -v ? If you have the older perl agent installed in /sbin with the distribution that you are running, then I am sorry, but the agent will be confused by the custom names - one way to fix it fast and test the fix is to telnet directly into your apc switch and change the name of outlet 4 to 'Outlet 4'. -Jim From redhat at watson-wilson.ca Tue Apr 24 14:20:44 2007 From: redhat at watson-wilson.ca (Neil Watson) Date: Tue, 24 Apr 2007 10:20:44 -0400 Subject: [Linux-cluster] HP iLO fencing issues In-Reply-To: <462E08C6.1030707@sys-admin.hu> References: <462E08C6.1030707@sys-admin.hu> Message-ID: <20070424142044.GA6978@watson-wilson.ca> On Tue, Apr 24, 2007 at 03:40:22PM +0200, BERES Laszlo wrote: >Hi all, > >I know this list is not for HP support, but do you have any kind of >(negative) experience with iLO2 fencing? Our customer has sleepless >nights, because his new HP Blade's management is failing: I use ILO fences and ILOs for general console access on most servers. There have been no problems. Make sure the ILO are running the latest firmware. -- Neil Watson | Debian Linux System Administrator | Uptime 10:19:31 up 17:16, 1 user, load average: 0.00, 0.00, 0.00 http://watson-wilson.ca From james.lapthorn at lapthornconsulting.com Tue Apr 24 14:28:25 2007 From: james.lapthorn at lapthornconsulting.com (James Lapthorn) Date: Tue, 24 Apr 2007 15:28:25 +0100 Subject: [Linux-cluster] Fencing using APC7921 In-Reply-To: <462E1088.6060505@redhat.com> References: <462E1088.6060505@redhat.com> Message-ID: Thanks for your reply James, I believe the problem with the fence_rsa was fixed in a recent update, I originally filed the bugzilla report for this. My problem on the fence_apc was similar. As a tidy up I had named all the outlets according to node they controlled. The script Appears to accept 'Outlet 1' as an acceptable menu option. # "Device Manager", "1- Cluster Node 0 ON" /--\s*Outlet Control.*\s+?(\d+)\s*-\s+[^\n\r]*\s*Outlet\s+$opt_n\D[^\n]*\s(?-i:ON|OFF)\*?\s/ism || If you think that this is still not fixed i'll file a report with Bugzilla James Lapthorn On 24/04/07, James Parsons wrote: > > James Lapthorn wrote: > > > I use RSA II cards as my primary fencing device on my IBM 3850's. > > After some modifications to the 'fence_rsa' script it now recognises > > 'WMN170370264' as the default card name. > > Hi James - which version of cluster suite are you running? fence_rsa was > recently modified to handle prompts other than the default...I wonder if > this is what you encountered. The fix went out in the rhel5 > distribution, and will be in rhel4 update 5. If there is something else > amiss with the rsa agent, please consider filing a bugzilla for it - or > maybe you could send me the patch you applied? I would like to keep that > agent as current as possible. > > > > > > > My secondary fencing device is and APC Power Switch APC7921. This is > > not a Masterswitch plus so there are two APC7921's, one for each POwer > > supply. When testing the the device using the following command: > > > > fence_apc -a 10.179.20.52 -l apc -p apc -n 4 -v > > > > I get: > > > > failed: unrecognised menu response > > In your apc fence log, the problem appears to be the agent not > recognizing the custom names you have set up for the outlets. The latest > version of the fence_apc agent (which was heavily refactored - > completely re-written from the ground up, actually) should handle custom > outlet names...the only problem is - once you change the default name > from 'Outlet 4' to 'leoukldb3 ', you have to use that name. If you are > running the latest agent (it is written in python instead of perl, like > the original agent), could you please try re-running the command above > with the following args? > > fence_apc -a 10.179.20.52 -l apc -p apc -n > leoukldb3 -v ? > > If you have the older perl agent installed in /sbin with the > distribution that you are running, then I am sorry, but the agent will > be confused by the custom names - one way to fix it fast and test the > fix is to telnet directly into your apc switch and change the name of > outlet 4 to 'Outlet 4'. > > -Jim > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jparsons at redhat.com Tue Apr 24 14:39:55 2007 From: jparsons at redhat.com (James Parsons) Date: Tue, 24 Apr 2007 10:39:55 -0400 Subject: [Linux-cluster] Fencing using APC7921 In-Reply-To: References: <462E1088.6060505@redhat.com> Message-ID: <462E16BB.4060604@redhat.com> James Lapthorn wrote: > Thanks for your reply James, I believe the problem with the fence_rsa > was fixed in a recent update, I originally filed the bugzilla report > for this. > > My problem on the fence_apc was similar. As a tidy up I had named all > the outlets according to node they controlled. The script Appears to > accept 'Outlet 1' as an acceptable menu option. > > # "Device Manager", "1- Cluster Node 0 ON" > /--\s*Outlet > Control.*\s+?(\d+)\s*-\s+[^\n\r]*\s*Outlet\s+$opt_n\D[^\n]*\s(?-i:ON|OFF)\*?\s/ism > || > > If you think that this is still not fixed i'll file a report with > Bugzilla I am very certain it is fixed - when the new agent finally wends its way into the distribution ;) The above perl line is frightening, which is why the agent was re-written in python...I don't wish to offend anyones religion, however! The new apc agent also handles huge powerstrips and outlet grouping as well as aliased outlet names. -J > > > > James Lapthorn > > On 24/04/07, *James Parsons* > wrote: > > James Lapthorn wrote: > > > I use RSA II cards as my primary fencing device on my IBM 3850's. > > After some modifications to the 'fence_rsa' script it now recognises > > 'WMN170370264' as the default card name. > > Hi James - which version of cluster suite are you running? > fence_rsa was > recently modified to handle prompts other than the default...I > wonder if > this is what you encountered. The fix went out in the rhel5 > distribution, and will be in rhel4 update 5. If there is something > else > amiss with the rsa agent, please consider filing a bugzilla for it > - or > maybe you could send me the patch you applied? I would like to > keep that > agent as current as possible. > > > > > > > My secondary fencing device is and APC Power Switch > APC7921. This is > > not a Masterswitch plus so there are two APC7921's, one for each > POwer > > supply. When testing the the device using the following command: > > > > fence_apc -a 10.179.20.52 > -l apc -p apc -n 4 -v > > > > I get: > > > > failed: unrecognised menu response > > In your apc fence log, the problem appears to be the agent not > recognizing the custom names you have set up for the outlets. The > latest > version of the fence_apc agent (which was heavily refactored - > completely re-written from the ground up, actually) should handle > custom > outlet names...the only problem is - once you change the default name > from 'Outlet 4' to 'leoukldb3 ', you have to use that name. If you are > running the latest agent (it is written in python instead of perl, > like > the original agent), could you please try re-running the command above > with the following args? > > fence_apc -a 10.179.20.52 > > -l apc -p apc -n > leoukldb3 -v ? > > If you have the older perl agent installed in /sbin with the > distribution that you are running, then I am sorry, but the agent will > be confused by the custom names - one way to fix it fast and test the > fix is to telnet directly into your apc switch and change the name of > outlet 4 to 'Outlet 4'. > > -Jim > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > >------------------------------------------------------------------------ > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > From sebastian.walter at fu-berlin.de Tue Apr 24 14:42:41 2007 From: sebastian.walter at fu-berlin.de (Sebastian Walter) Date: Tue, 24 Apr 2007 16:42:41 +0200 Subject: [Linux-cluster] HP iLO fencing issues In-Reply-To: <462E08C6.1030707@sys-admin.hu> References: <462E08C6.1030707@sys-admin.hu> Message-ID: <462E1761.8010900@fu-berlin.de> Hi Beres, I had problems with the new DRAC5 from dell, so what I did was downloading the new cluster tgz from ftp://sources.redhat.com/pub/cluster/releases/cluster-2.00.00.tar.gz extracted the apropriate fence_* script and substitute it with the older agent. You will find the fence_ilo.pl from the 2.00.00 release attached. regards, sebastian BERES Laszlo wrote: > Hi all, > > I know this list is not for HP support, but do you have any kind of > (negative) experience with iLO2 fencing? Our customer has sleepless > nights, because his new HP Blade's management is failing: > > ccsd[5659]: process_get: Invalid connection descriptor received. > ccsd[5659]: Error while processing get: Invalid request descriptor > fenced[5761]: fence "host1" failed > > -------------- next part -------------- A non-text attachment was scrubbed... Name: fence_ilo.pl Type: application/x-perl Size: 13441 bytes Desc: not available URL: From npf at eurotux.com Tue Apr 24 15:04:12 2007 From: npf at eurotux.com (Nuno Fernandes) Date: Tue, 24 Apr 2007 16:04:12 +0100 Subject: [Linux-cluster] HP iLO fencing issues In-Reply-To: <462E08C6.1030707@sys-admin.hu> References: <462E08C6.1030707@sys-admin.hu> Message-ID: <200704241604.15718.npf@eurotux.com> On Tuesday 24 April 2007 14:40:22 BERES Laszlo wrote: > Hi all, > > I know this list is not for HP support, but do you have any kind of > (negative) experience with iLO2 fencing? Our customer has sleepless > nights, because his new HP Blade's management is failing: > > ccsd[5659]: process_get: Invalid connection descriptor received. > ccsd[5659]: Error while processing get: Invalid request descriptor > fenced[5761]: fence "host1" failed Hi, We had same issues... but installing perl-Crypt-SSLeay solved the problem. Fenced rpm should have that dependency... but it doesn't.. Rgds Nuno Fernandes -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From grimme at atix.de Tue Apr 24 15:04:27 2007 From: grimme at atix.de (Marc Grimme) Date: Tue, 24 Apr 2007 17:04:27 +0200 Subject: [Linux-cluster] Samba on GFS problems changing, =?utf-8?q?=09directory?= ACLs In-Reply-To: <462CDED6.6030400@forschungsgruppe.de> References: <20070422160006.2676972FA0@hormel.redhat.com> <462CDED6.6030400@forschungsgruppe.de> Message-ID: <200704241704.27862.grimme@atix.de> On Monday 23 April 2007 18:29:10 Christian Brandes wrote: > >> How can I see what gfs version it is actually? > > > > Hmm, how about modinfo gfs? > > filename: > /lib/modules/2.6.15.7-ubuntu1-xxx-xxx-firepig-2.4/kernel/fs/gfs/gfs.ko > description: Global File System > author: Red Hat, Inc. > license: GPL > vermagic: 2.6.15.7-ubuntu1-xxx-xxx-firepig-2.4 preempt PENTIUM4 > gcc-4.0 > depends: lock_harness > > That shows only my kernel version number and that GFS was taken from CVS. How about update the cvs and compile everything new ;-) ? -- Gruss / Regards, Marc Grimme Phone: +49-89 452 3538-14 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany Registergericht: Amtsgericht M?nchen Registernummer: HRB 131682 USt.-Id.: DE209485962 Gesch?ftsf?hrung: Marc Grimme, Mark Hlawatschek, Thomas Merz From beres.laszlo at sys-admin.hu Tue Apr 24 15:23:04 2007 From: beres.laszlo at sys-admin.hu (BERES Laszlo) Date: Tue, 24 Apr 2007 17:23:04 +0200 Subject: [Linux-cluster] HP iLO fencing issues In-Reply-To: <20070424142044.GA6978@watson-wilson.ca> References: <462E08C6.1030707@sys-admin.hu> <20070424142044.GA6978@watson-wilson.ca> Message-ID: <462E20D8.1080301@sys-admin.hu> Neil Watson wrote: > I use ILO fences and ILOs for general console access on most servers. > There have been no problems. Make sure the ILO are running the latest > firmware. Well, one of the blades had an earlier version installed. We upgraded, let's see what will happen. -- B?RES L?szl? RHCE, RHCX senior IT engineer, trainer From rmccabe at redhat.com Tue Apr 24 16:40:35 2007 From: rmccabe at redhat.com (Ryan McCabe) Date: Tue, 24 Apr 2007 12:40:35 -0400 Subject: [Linux-cluster] HP iLO fencing issues In-Reply-To: <462E1761.8010900@fu-berlin.de> References: <462E08C6.1030707@sys-admin.hu> <462E1761.8010900@fu-berlin.de> Message-ID: <20070424164034.GA24701@redhat.com> On Tue, Apr 24, 2007 at 04:42:41PM +0200, Sebastian Walter wrote: > Hi Beres, > > I had problems with the new DRAC5 from dell, so what I did was > downloading the new cluster tgz from > ftp://sources.redhat.com/pub/cluster/releases/cluster-2.00.00.tar.gz > extracted the apropriate fence_* script and substitute it with the older > agent. You will find the fence_ilo.pl from the 2.00.00 release attached. There's a newer version of the agent available now that addresses some iLO2 issues. Please give it a shot if you're still having issues, and let us know how it goes. The script is attached. Ryan -------------- next part -------------- #!/usr/bin/perl ############################################################################### ############################################################################### ## ## Copyright (C) 2006-2007 Red Hat, Inc. All rights reserved. ## ## This copyrighted material is made available to anyone wishing to use, ## modify, copy, or redistribute it subject to the terms and conditions ## of the GNU General Public License v.2. ## ############################################################################### ############################################################################### $|=1; eval { $ssl_mod="Net::SSL" if require Net::SSL} || eval { $ssl_mod="Net::SSLeay::Handle" if require Net::SSLeay::Handle } || die "Net::SSL.pm or Net::SSLeay::Handle.pm not found.\n". "Please install the perl-Crypt-SSLeay package from RHN (http://rhn.redhat.com)\n". "or Net::SSLeay from CPAN (http://www.cpan.org)\n"; use IO::Socket; use Getopt::Std; # WARNING!! Do not add code bewteen "#BEGIN_VERSION_GENERATION" and # "#END_VERSION_GENERATION" It is generated by the Makefile #BEGIN_VERSION_GENERATION $FENCE_RELEASE_NAME=""; $REDHAT_COPYRIGHT=""; $BUILD_DATE=""; #END_VERSION_GENERATION # Get the program name from $0 and strip directory names $_=$0; s/.*\///; my $pname = $_; ################################################################################ sub usage { print "Usage:\n"; print "\n"; print "$pname [options]\n"; print "\n"; print "Options:\n"; print " -a IP address or hostname of iLO card\n"; print " -h usage\n"; print " -l Login name\n"; print " -o Action: reboot (default), off, on or status\n"; print " -p Login password\n"; print " -S Script to run to retrieve login password\n"; print " -q quiet mode\n"; print " -V version\n"; print " -v verbose\n"; exit 0; } sub fail { ($msg)=@_; print $msg unless defined $quiet; exit 1; } sub fail_usage { ($msg)=@_; print STDERR $msg if $msg; print STDERR "Please use '-h' for usage.\n"; exit 1; } sub version { print "$pname $FENCE_RELEASE_NAME $BUILD_DATE\n"; print "$REDHAT_COPYRIGHT\n" if ( $REDHAT_COPYRIGHT ); exit 0; } sub sendsock { my ($sock, $msg, $junk) = @_; $sock->print($msg); if ($verbose) { chomp $msg; print "SEND: $msg\n" } } # This will slurp up all the data on the socket. The SSL connection # automatically times out after 10 seconds. If $mode is defined, the # function will return immediately following the initial packet # terminator "" or "" # RIBCL VERSION=1.2 -> # RIBCL VERSION=2.0 -> sub receive_response { my ($sock,$mode) = @_; $mode = 0 unless defined $mode; my $count=0; my $buf; my $buffer = ""; ### sock should automatically be closed by iLO after 10 seconds ### XXX read buf length of 256. Is this enough? Do I need to buffer? while ($ssl_mod eq "Net::SSLeay::Handle" ? $buf=<$sock> : $sock->read($buf,256) ) { $rd = length($buf); last unless ($rd); $buffer="$buffer$buf"; if ($verbose) { chomp $buf; print "READ:($rd,$mode) $buf\n"; } # RIBCL VERSION=1.2 last if ( $buffer =~ /$/mi && $mode==1 ); # RIBCL VERSION=2.0 last if ( $buffer =~ /<\/RIBCL>$/mi && $mode==1 ); } ### Determine version of RIBCL if not already defined if (!defined $ribcl_vers) { if ($buffer =~ /new( PeerAddr => $hostname, PeerPort => $port ); $ssl->configure(); $ssl->connect($port,inet_aton($hostname)) or fail "unable to connect to $hostname:$port $@\n"; } return $ssl; } sub close_socket { my ($sock, $junk) = @_; #shutdown ($sock,1); close $sock; } # power_off() -- power off the node # return 0 on success, non-zero on failure sub power_off { my $response = set_power_state ("N"); my @response = split /\n/,$response; my $no_err=0; my $agent_status = -1; foreach my $line (@response) { if ($line =~ /MESSAGE='(.*)'/) { my $msg = $1; if ($msg eq "No error") { $no_err++; next; } elsif ($msg eq "Host power is already OFF.") { $agent_status = 0; print "warning: $msg\n" unless defined $quiet; } else { $agent_status = 1; print STDERR "error: $msg\n"; } } } # There should be about 6 or more response packets on a successful # power off command. if ($agent_status<0) { $agent_status = ($no_err<5) ? 1 : 0; } return $agent_status; } # power_on() -- power on the node # return 0 on success, non-zero on failure sub power_on { my $response = set_power_state ("Y"); my @response = split /\n/,$response; my $no_err=0; my $agent_status = -1; foreach my $line (@response) { if ($line =~ /MESSAGE='(.*)'/) { my $msg = $1; if ($msg eq "No error") { $no_err++; next; } elsif ($msg eq "Host power is already ON.") { $agent_status = 0; print "warning: $msg\n" unless defined $quiet; } else { $agent_status = 1; print STDERR "error: $msg\n"; } } } # There should be about 6 or more response packets on a successful # power on command. if ($agent_status<0) { $agent_status = ($no_err<5) ? 1 : 0; } return $agent_status; } # power_status() -- print the power status of the node # return 0 on success, non-zero on failure # set $? to power status from ilo ("ON" or "OFF") sub power_status { my $response = get_power_state (); my @response = split /\n/,$response; my $agent_status = -1; my $power_status = "UNKNOWN"; foreach my $line (@response) { if ($line =~ /MANAGEMENT_PROCESSOR\s*=\s*\"(.*)\"/) { if ($1 eq "iLO2") { $ilo_vers = 2; print "power_status: reporting iLO2\n" if ($verbose); } } if ($line =~ /MESSAGE='(.*)'/) { my $msg = $1; if ($msg eq "No error") { next; } else { $agent_status = 1; print STDERR "error: $msg\n"; } } # RIBCL VERSION=1.2 elsif ($line =~ /HOST POWER=\"(.*)\"/) { $agent_status = 0; $power_status = $1; } # RIBCL VERSION=2.0 elsif ($line =~ /HOST_POWER=\"(.*)\"/) { $agent_status = 0; $power_status = $1; } } $_ = $power_status; print "power_status: reporting power is $_\n" if ($verbose); return $agent_status; } sub set_power_state { my $state = shift; my $response = ""; if (!defined $state || ( $state ne "Y" && $state ne "N") ) { fail "illegal state\n"; } $socket = open_socket; sendsock $socket, "\r\n"; $response = receive_response($socket,1); print "Sending power-o".(($state eq "Y")?"n":"ff")."\n" if ($verbose); if ($ribcl_vers < 2 ) { sendsock $socket, "\n"; } else { # It seems the firmware can't handle the tag # RIBCL VERSION=2.0 #> sendsock $socket, "\n"; sendsock $socket, "\n"; } sendsock $socket, "\n"; sendsock $socket, "\n"; if ($ilo_vers == 2) { # iLO2 with RIBCL v2.22 behaves differently from # iLO with RIBCL v2.22. For the former, HOLD_PWR_BTN is # used to both power the machine on and off; when the power # is off, PRESS_PWR_BUTTON has no effect. For the latter, # HOLD_PWR_BUTTON is used to power the machine off, and # PRESS_PWR_BUTTON is used to power the machine on; # when the power is off, HOLD_PWR_BUTTON has no effect. sendsock $socket, "\n"; } # As of firmware version 1.71 (RIBCL 2.21) The SET_HOST_POWER command # is no longer available. HOLD_PWR_BTN and PRESS_PWR_BTN are used # instead now :( elsif ($ribcl_vers < 2.21) { sendsock $socket, "\n"; } else { if ($state eq "Y" ) { sendsock $socket, "\n"; } else { sendsock $socket, "\n"; } } sendsock $socket, "\n"; sendsock $socket, "\n"; sendsock $socket, "\n"; # It seems the firmware can't handle the tag # RIBCL VERSION=2.0 #> sendsock $socket, "\n" if ($ribcl_vers >= 2) ; $response = receive_response($socket); print "Closing connection\n" if ($verbose); close_socket($socket); return $response; } sub get_power_state { my $response = ""; $socket = open_socket; sendsock $socket, "\r\n"; $response = receive_response($socket,1); print "Sending get-status\n" if ($verbose); if ($ribcl_vers < 2 ) { sendsock $socket, "\n"; } else { # It seems the firmware can't handle the tag # RIBCL VERSION=2.0 #> sendsock $socket, "\n"; sendsock $socket, "\n"; } sendsock $socket, "\n"; if ($ribcl_vers >= 2) { sendsock $socket, "\n"; } sendsock $socket, "\n"; sendsock $socket, "\n"; sendsock $socket, "\n"; sendsock $socket, "\n"; sendsock $socket, "\n"; # It seems the firmware can't handle the tag # RIBCL VERSION=2.0 #> sendsock $socket, "\r\n" if ($ribcl_vers >= 2) ; $response = receive_response($socket); print "Closing connection\n" if ($verbose); close_socket($socket); return $response; } sub get_options_stdin { my $opt; my $line = 0; while( defined($in = <>) ) { $_ = $in; chomp; # strip leading and trailing whitespace s/^\s*//; s/\s*$//; # skip comments next if /^#/; $line+=1; $opt=$_; next unless $opt; ($name,$val)=split /\s*=\s*/, $opt; if ( $name eq "" ) { print STDERR "parse error: illegal name in option $line\n"; exit 2; } elsif ($name eq "action" ) { $action = $val; } # DO NOTHING -- this field is used by fenced or stomithd elsif ($name eq "agent" ) { } elsif ($name eq "hostname" ) { $hostname = $val; } elsif ($name eq "login" ) { $username = $val; } elsif ($name eq "passwd" ) { $passwd = $val; } elsif ($name eq "passwd_script" ) { $passwd_script = $val; } elsif ($name eq "ribcl" ) { $ribcl_vers = $val; } elsif ($name eq "verbose" ) { $verbose = $val; } } } ################################################################################ # MAIN $action = "reboot"; $ribcl_vers = undef; # undef = autodetect $ilo_vers = 1; if (@ARGV > 0) { getopts("a:hl:n:o:p:S:r:qvV") || fail_usage ; usage if defined $opt_h; version if defined $opt_V; fail_usage "Unkown parameter." if (@ARGV > 0); fail_usage "No '-a' flag specified." unless defined $opt_a; $hostname = $opt_a; fail_usage "No '-l' flag specified." unless defined $opt_l; $username = $opt_l; if (defined $opt_S) { $pwd_script_out = `$opt_S`; chomp($pwd_script_out); if ($pwd_script_out) { $opt_p = $pwd_script_out; } } fail_usage "No '-p' or '-S' flag specified." unless defined $opt_p; $passwd = $opt_p; $action = $opt_o if defined $opt_o; fail_usage "Unrecognised action '$action' for '-o' flag" unless $action=~ /^(off|on|reboot|status)$/; $localport = $opt_n if defined $opt_n; $quiet = 1 if defined $opt_q; $verbose = 1 if defined $opt_v; $ribcl_vers = $opt_r if defined $opt_r; } else { get_options_stdin(); fail "no host\n" unless defined $hostname; fail "no login name\n" unless defined $username; if (defined $passwd_script) { $pwd_script_out = `$passwd_script`; chomp($pwd_script_out); if ($pwd_script_out) { $passwd = $pwd_script_out; } } fail "no password\n" unless defined $passwd; fail "unrecognised action: $action\n" unless $action=~ /^(off|on|reboot|status)$/; } # Parse user specified port from apaddr parameter ($hostname_tmp,$port,$junk) = split(/:/,$hostname); fail "bad hostname/ipaddr format: $hostname\n" if defined $junk; $hostname = $hostname_tmp; $port = 443 unless defined $port; print "ssl module: $ssl_mod\n" if $verbose; $_=$action; if (/on/) { fail "power_status: unexpected error\n" if power_status; if (! /^on$/i) { fail "power_on: unexpected error\n" if power_on; fail "power_status: unexpected error\n" if power_status; fail "failed to turn on\n" unless (/^on$/i); } else { print "power is already on\n" unless defined $quiet; } } elsif (/off/) { fail "power_status: unexpected error\n" if power_status; if (! /^off$/i) { fail "power_off: unexpected error\n" if power_off; fail "power_status: unexpected error\n" if power_status; fail "failed to turn off\n" unless (/^off$/i); } else { print "power is already off\n" unless defined $quiet; } } elsif (/reboot/) { fail "power_status: unexpected error\n" if power_status; if (! /^off$/i) { fail "power_off: unexpected error\n" if power_off; fail "power_status: unexpected error\n" if power_status; fail "failed to turn off\n" unless (/^off$/i); } if (/^off$/i) { fail "power_on: unexpected error\n" if power_on; fail "power_status: unexpected error\n" if power_status; fail "failed to turn on\n" unless (/^on$/i); } else { fail "unexpected power state: '$_'\n"; } } elsif (/status/) { fail "power_status: unexpected error\n" if power_status; print "power is $_\n"; } else { fail "illegal action: '$_'\n"; } print "success\n" unless defined $quiet; exit 0 ################################################################################ From venilton.junior at sercompe.com.br Tue Apr 24 16:52:35 2007 From: venilton.junior at sercompe.com.br (Venilton Junior) Date: Tue, 24 Apr 2007 13:52:35 -0300 Subject: [Linux-cluster] HP iLO fencing issues In-Reply-To: <462E1761.8010900@fu-berlin.de> References: <462E08C6.1030707@sys-admin.hu> <462E1761.8010900@fu-berlin.de> Message-ID: If they're blades, make sure they're using the latest Proliant Support Pack, which u can download at hp.com and upgrade the firmware to the 1.89 version. Guess that is the latest one. I have no problems to run RHCS+GFS on proliant servers, including blades. Hope this solves your problems. Best regards, Venilton C. Junior HP Certified Professional Sercompe Computadores Ltda. Office:???+55 47?3431-9700 Fax:???????+55 47?3431-9747 Mobile: +55 47 9653-5872 www.sercompe.com.br -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Sebastian Walter Sent: ter?a-feira, 24 de abril de 2007 11:43 To: beres.laszlo at sys-admin.hu Cc: linux clustering Subject: Re: [Linux-cluster] HP iLO fencing issues Hi Beres, I had problems with the new DRAC5 from dell, so what I did was downloading the new cluster tgz from ftp://sources.redhat.com/pub/cluster/releases/cluster-2.00.00.tar.gz extracted the apropriate fence_* script and substitute it with the older agent. You will find the fence_ilo.pl from the 2.00.00 release attached. regards, sebastian BERES Laszlo wrote: > Hi all, > > I know this list is not for HP support, but do you have any kind of > (negative) experience with iLO2 fencing? Our customer has sleepless > nights, because his new HP Blade's management is failing: > > ccsd[5659]: process_get: Invalid connection descriptor received. > ccsd[5659]: Error while processing get: Invalid request descriptor > fenced[5761]: fence "host1" failed > > From teigland at redhat.com Tue Apr 24 19:36:00 2007 From: teigland at redhat.com (David Teigland) Date: Tue, 24 Apr 2007 14:36:00 -0500 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <20070423211717.GA22147@redhat.com> References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> <87wt06djk7.fsf@tac.ki.iif.hu> <20070423211717.GA22147@redhat.com> Message-ID: <20070424193600.GB11156@redhat.com> On Mon, Apr 23, 2007 at 04:17:18PM -0500, David Teigland wrote: > > Also, what's that new infrastructure? Do you mean GFS2? I read it > > was not production quality yet, so I didn't mean to try it. But again > > you may have got something else in your head... > > GFS1 and GFS2 both run on the new openais-based cluster infrastructure. > (in the cluster-2.00.00 release, and the RHEL5 and HEAD cvs branches). I've attached a little flock/plock performance test that emulates what you're doing; could you run it on your cluster and send the results? The only cluster I have for testing right now is based on the new code which includes the new dlm (would effect flock performance) and the new gfs_controld plock implementation. Here are the numbers I got, I had two nodes mounting the fs, and one node running the test. gfs_controld running with default settings: # fplockperf flock: filecount=100 iteration=0 elapsed time=0.098 s flock: filecount=100 iteration=1 elapsed time=0.007 s flock: filecount=100 iteration=2 elapsed time=0.007 s flock: filecount=100 iteration=3 elapsed time=0.008 s flock: filecount=100 iteration=4 elapsed time=0.007 s total elapsed time=0.129 s flock: filecount=500 iteration=0 elapsed time=0.483 s flock: filecount=500 iteration=1 elapsed time=0.037 s flock: filecount=500 iteration=2 elapsed time=0.039 s flock: filecount=500 iteration=3 elapsed time=0.037 s flock: filecount=500 iteration=4 elapsed time=0.037 s total elapsed time=0.634 s flock: filecount=1000 iteration=0 elapsed time=0.523 s flock: filecount=1000 iteration=1 elapsed time=0.077 s flock: filecount=1000 iteration=2 elapsed time=0.076 s flock: filecount=1000 iteration=3 elapsed time=0.076 s flock: filecount=1000 iteration=4 elapsed time=0.076 s total elapsed time=0.830 s flock: filecount=2000 iteration=0 elapsed time=1.064 s flock: filecount=2000 iteration=1 elapsed time=0.151 s flock: filecount=2000 iteration=2 elapsed time=0.151 s flock: filecount=2000 iteration=3 elapsed time=0.146 s flock: filecount=2000 iteration=4 elapsed time=0.147 s total elapsed time=1.661 s flock: filecount=5000 iteration=0 elapsed time=3.505 s flock: filecount=5000 iteration=1 elapsed time=0.405 s flock: filecount=5000 iteration=2 elapsed time=0.407 s flock: filecount=5000 iteration=3 elapsed time=0.405 s flock: filecount=5000 iteration=4 elapsed time=0.405 s total elapsed time=5.128 s plock: filecount=100 iteration=0 elapsed time=0.621 s plock: filecount=100 iteration=1 elapsed time=2.108 s plock: filecount=100 iteration=2 elapsed time=2.099 s plock: filecount=100 iteration=3 elapsed time=2.099 s plock: filecount=100 iteration=4 elapsed time=2.011 s total elapsed time=8.939 s plock: filecount=500 iteration=0 elapsed time=10.519 s plock: filecount=500 iteration=1 elapsed time=10.350 s plock: filecount=500 iteration=2 elapsed time=10.457 s plock: filecount=500 iteration=3 elapsed time=10.178 s plock: filecount=500 iteration=4 elapsed time=10.164 s total elapsed time=51.669 s plock: filecount=1000 iteration=0 elapsed time=20.533 s plock: filecount=1000 iteration=1 elapsed time=20.379 s plock: filecount=1000 iteration=2 elapsed time=20.656 s plock: filecount=1000 iteration=3 elapsed time=20.494 s plock: filecount=1000 iteration=4 elapsed time=20.810 s total elapsed time=102.872 s plock: filecount=2000 iteration=0 elapsed time=41.037 s plock: filecount=2000 iteration=1 elapsed time=40.951 s plock: filecount=2000 iteration=2 elapsed time=41.034 s plock: filecount=2000 iteration=3 elapsed time=41.077 s plock: filecount=2000 iteration=4 elapsed time=41.052 s total elapsed time=205.152 s plock: filecount=5000 iteration=0 elapsed time=103.086 s plock: filecount=5000 iteration=1 elapsed time=102.845 s plock: filecount=5000 iteration=2 elapsed time=102.391 s plock: filecount=5000 iteration=3 elapsed time=103.586 s plock: filecount=5000 iteration=4 elapsed time=102.104 s total elapsed time=514.012 s The new dlm caches resource lookups which is probably responsible for the big difference between the first flock iteration and the following four. These cached rsbs are dropped after about 10 sec, so I've added a delay between each filecount setting so the caching effects are separated. We see the expected slight rise in the initial iteration time for flocks after adding the delay. (In the new infrastructure, flocks are still implemented with dlm locks, but plocks aren't, so the dlm has no effect on plock performance). # fplockperf 20 flock: filecount=100 iteration=0 elapsed time=0.100 s flock: filecount=100 iteration=1 elapsed time=0.008 s flock: filecount=100 iteration=2 elapsed time=0.008 s flock: filecount=100 iteration=3 elapsed time=0.007 s flock: filecount=100 iteration=4 elapsed time=0.007 s total elapsed time=0.131 s 20 sec delay... flock: filecount=500 iteration=0 elapsed time=0.559 s flock: filecount=500 iteration=1 elapsed time=0.038 s flock: filecount=500 iteration=2 elapsed time=0.037 s flock: filecount=500 iteration=3 elapsed time=0.036 s flock: filecount=500 iteration=4 elapsed time=0.036 s total elapsed time=0.707 s 20 sec delay... flock: filecount=1000 iteration=0 elapsed time=1.057 s flock: filecount=1000 iteration=1 elapsed time=0.076 s flock: filecount=1000 iteration=2 elapsed time=0.076 s flock: filecount=1000 iteration=3 elapsed time=0.078 s flock: filecount=1000 iteration=4 elapsed time=0.072 s total elapsed time=1.359 s 20 sec delay... flock: filecount=2000 iteration=0 elapsed time=2.042 s flock: filecount=2000 iteration=1 elapsed time=0.156 s flock: filecount=2000 iteration=2 elapsed time=0.154 s flock: filecount=2000 iteration=3 elapsed time=0.155 s flock: filecount=2000 iteration=4 elapsed time=0.155 s total elapsed time=2.663 s 20 sec delay... flock: filecount=5000 iteration=0 elapsed time=5.169 s flock: filecount=5000 iteration=1 elapsed time=0.397 s flock: filecount=5000 iteration=2 elapsed time=0.399 s flock: filecount=5000 iteration=3 elapsed time=0.392 s flock: filecount=5000 iteration=4 elapsed time=0.386 s total elapsed time=6.743 s By default, gfs_controld limits the rate at which plocks can be sent/acquired so that the network isn't accidentally flooded. We can disable this rate limiting to speed up the plocks with 'gfs_controld -l0' which I've done next. # fplockperf 20 flock: filecount=100 iteration=0 elapsed time=0.094 s flock: filecount=100 iteration=1 elapsed time=0.007 s flock: filecount=100 iteration=2 elapsed time=0.007 s flock: filecount=100 iteration=3 elapsed time=0.007 s flock: filecount=100 iteration=4 elapsed time=0.007 s total elapsed time=0.125 s 20 sec delay... flock: filecount=500 iteration=0 elapsed time=0.551 s flock: filecount=500 iteration=1 elapsed time=0.036 s flock: filecount=500 iteration=2 elapsed time=0.035 s flock: filecount=500 iteration=3 elapsed time=0.036 s flock: filecount=500 iteration=4 elapsed time=0.035 s total elapsed time=0.693 s 20 sec delay... flock: filecount=1000 iteration=0 elapsed time=1.031 s flock: filecount=1000 iteration=1 elapsed time=0.075 s flock: filecount=1000 iteration=2 elapsed time=0.074 s flock: filecount=1000 iteration=3 elapsed time=0.074 s flock: filecount=1000 iteration=4 elapsed time=0.075 s total elapsed time=1.329 s 20 sec delay... flock: filecount=2000 iteration=0 elapsed time=1.985 s flock: filecount=2000 iteration=1 elapsed time=0.146 s flock: filecount=2000 iteration=2 elapsed time=0.156 s flock: filecount=2000 iteration=3 elapsed time=0.145 s flock: filecount=2000 iteration=4 elapsed time=0.150 s total elapsed time=2.583 s 20 sec delay... flock: filecount=5000 iteration=0 elapsed time=5.064 s flock: filecount=5000 iteration=1 elapsed time=0.378 s flock: filecount=5000 iteration=2 elapsed time=0.378 s flock: filecount=5000 iteration=3 elapsed time=0.378 s flock: filecount=5000 iteration=4 elapsed time=0.379 s total elapsed time=6.577 s 20 sec delay... plock: filecount=100 iteration=0 elapsed time=0.604 s plock: filecount=100 iteration=1 elapsed time=0.601 s plock: filecount=100 iteration=2 elapsed time=0.599 s plock: filecount=100 iteration=3 elapsed time=0.600 s plock: filecount=100 iteration=4 elapsed time=0.600 s total elapsed time=3.005 s 20 sec delay... plock: filecount=500 iteration=0 elapsed time=3.010 s plock: filecount=500 iteration=1 elapsed time=3.008 s plock: filecount=500 iteration=2 elapsed time=2.993 s plock: filecount=500 iteration=3 elapsed time=3.013 s plock: filecount=500 iteration=4 elapsed time=3.006 s total elapsed time=15.032 s 20 sec delay... plock: filecount=1000 iteration=0 elapsed time=6.030 s plock: filecount=1000 iteration=1 elapsed time=6.075 s plock: filecount=1000 iteration=2 elapsed time=6.023 s plock: filecount=1000 iteration=3 elapsed time=6.074 s plock: filecount=1000 iteration=4 elapsed time=6.088 s total elapsed time=30.291 s 20 sec delay... plock: filecount=2000 iteration=0 elapsed time=12.064 s plock: filecount=2000 iteration=1 elapsed time=12.028 s plock: filecount=2000 iteration=2 elapsed time=12.056 s plock: filecount=2000 iteration=3 elapsed time=12.055 s plock: filecount=2000 iteration=4 elapsed time=12.009 s total elapsed time=60.212 s 20 sec delay... plock: filecount=5000 iteration=0 elapsed time=30.128 s plock: filecount=5000 iteration=1 elapsed time=30.126 s plock: filecount=5000 iteration=2 elapsed time=30.102 s plock: filecount=5000 iteration=3 elapsed time=30.129 s plock: filecount=5000 iteration=4 elapsed time=30.400 s total elapsed time=150.887 s From jody.brooks at nasa.gov Tue Apr 24 19:32:52 2007 From: jody.brooks at nasa.gov (Brooks, Jody) Date: Tue, 24 Apr 2007 14:32:52 -0500 Subject: [Linux-cluster] Piranha question: Running client on a real server hangs Message-ID: I have a direct-routed Piranha/LVS cluster that I would like to load balance between real servers svr1 and svr2. As a test I'm using the telnet service. For my setup, I'm using WRR, no load monitoring right now, direct-route, weights of 1 on both svr1 and svr2, using virtual ip address of test_vip. Also have I have ip_forward turned on on my router machine (say it's on host rtr1). Kernel: "2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:17:21 EST 2007". I also have an iptables rule in the nat table, PREROUTING chain for my telnet service and test_vip on both real servers saying to "REDIRECT". If I get on another machine, say client1, and telnet to test_vip, everything works fine ... meaning each subsequent connection goes to the next server in the cluster: svr1, svr2, svr1, svr2, ... However, if I run my telnet client from one of my real servers, say svr2 connecting to my service's virtual IP, test_vip, I will get connected to the other real server (svr1) fine when the round-robin schedule says I should go there, but when the schedule points me back to my host (svr2), my connection hangs. ipvsadm -l reports that a connection did go through the ipvs tables, but it does not connect. Same thing will happen if I start from svr1.... Works to svr2, but hangs trying to come back to svr1, the source machine. Bottom line: it works if client is run from non-real-server machines, and half the time if client is run from a real server machine but the other half of the time (when redirected to itself), it hangs. (I recognize "half" is because I just have 2 real servers.) Any clues as to why it won't work consistently when balanced back to itself? (Extra note: I say it works from non-real-server machines... not completely true: doesn't work from the router machine, rtr1, as that always goes back to itself, rtr1, ignoring the ipvs tables apparently as ipvsadm -l never shows the connection in its statistics... Apparently the virtual IP address being local short-circuits the ipvs table operations). TIA. -------------- next part -------------- An HTML attachment was scrubbed... URL: From teigland at redhat.com Tue Apr 24 19:46:13 2007 From: teigland at redhat.com (David Teigland) Date: Tue, 24 Apr 2007 14:46:13 -0500 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <20070424193600.GB11156@redhat.com> References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> <87wt06djk7.fsf@tac.ki.iif.hu> <20070423211717.GA22147@redhat.com> <20070424193600.GB11156@redhat.com> Message-ID: <20070424194612.GC11156@redhat.com> On Tue, Apr 24, 2007 at 02:36:00PM -0500, David Teigland wrote: > On Mon, Apr 23, 2007 at 04:17:18PM -0500, David Teigland wrote: > > > Also, what's that new infrastructure? Do you mean GFS2? I read it > > > was not production quality yet, so I didn't mean to try it. But again > > > you may have got something else in your head... > > > > GFS1 and GFS2 both run on the new openais-based cluster infrastructure. > > (in the cluster-2.00.00 release, and the RHEL5 and HEAD cvs branches). > > I've attached a little flock/plock performance test that emulates what > you're doing; could you run it on your cluster and send the results? Attached now -------------- next part -------------- #include #include #include #include #include #include #include #include #include #include #include #define die(fmt, args...) \ { \ fprintf(stderr, "%s: ", prog_name); \ fprintf(stderr, fmt, ##args); \ exit(EXIT_FAILURE); \ } char *prog_name; unsigned int wait = 1; int delay; int pid; int files[5000]; int do_plock(int fd, unsigned long long offset, int len, int type) { struct flock lock; int action; lock.l_type = type; lock.l_start = offset; lock.l_whence = SEEK_SET; lock.l_len = len; if (wait) action = F_SETLKW; else action = F_SETLK; return fcntl(fd, action, &lock); } void read_plock(int fd, unsigned long long offset, int len) { int error; error = do_plock(fd, offset, len, F_RDLCK); if (error < 0) { printf("%d read lock failed %d errno %d\n", pid, error, errno); exit(EXIT_FAILURE); } } void write_plock(int fd, unsigned long long offset, int len) { int error; error = do_plock(fd, offset, len, F_WRLCK); if (error < 0) { printf("%d write lock failed %d errno %d\n", pid, error, errno); exit(EXIT_FAILURE); } } void unlock_plock(int fd, unsigned long long offset, int len) { int error; error = do_plock(fd, offset, len, F_UNLCK); if (error < 0) { printf("%d unlock failed %d errno %d\n", pid, error, errno); exit(EXIT_FAILURE); } } void read_flock(int fd) { int error; int mode = LOCK_SH; if (!wait) mode |= LOCK_NB; error = flock(fd, mode); if (error < 0) { printf("%d read lock failed %d errno %d\n", pid, error, errno); exit(EXIT_FAILURE); } } void write_flock(int fd) { int error; int mode = LOCK_EX; if (!wait) mode |= LOCK_NB; error = flock(fd, mode); if (error < 0) { printf("%d write lock failed %d errno %d\n", pid, error, errno); exit(EXIT_FAILURE); } } void unlock_flock(int fd) { int error; error = flock(fd, LOCK_UN); if (error < 0) { printf("%d unlock failed %d errno %d\n", pid, error, errno); exit(EXIT_FAILURE); } } uint64_t dt_usec(struct timeval *start, struct timeval *stop) { uint64_t dt; dt = stop->tv_sec - start->tv_sec; dt *= 1000000; dt += stop->tv_usec - start->tv_usec; return dt; } void file_loop(int filecount, int flock) { char path[64]; struct timeval enter, exit, begin, end; uint64_t et_usec; int x, i, fd; gettimeofday(&enter, NULL); for (x = 0; x < 5; x++) { gettimeofday(&begin, NULL); for (i = 0; i < filecount; i++) { memset(path, 0, sizeof(path)); sprintf(path, "file%.10u", i); fd = open(path, O_RDWR, 0644); if (flock) { read_flock(fd); unlock_flock(fd); } else { read_plock(fd, 0, 0); unlock_plock(fd, 0, 0); } close(fd); } gettimeofday(&end, NULL); et_usec = dt_usec(&begin, &end); printf("%s: filecount=%d iteration=%d elapsed time=%.3f s\n", flock ? "flock" : "plock", filecount, x, (1.e-6 * et_usec)); } gettimeofday(&exit, NULL); et_usec = dt_usec(&enter, &exit); printf("total elapsed time=%.3f s\n", (1.e-6 * et_usec)); if (delay) { printf("%d sec delay...\n", delay); sleep(delay); } } void create_all(int filecount) { int i, fd; char path[64]; for (i = 0; i < filecount; i++) { memset(path, 0, sizeof(path)); sprintf(path, "file%.10u", i); fd = open(path, O_RDWR | O_CREAT, 0644); if (fd < 0) die("can't create file %s\n", strerror(errno)); close(fd); } } void open_all(int filecount) { int i, fd; char path[64]; for (i = 0; i < filecount; i++) { memset(path, 0, sizeof(path)); sprintf(path, "file%.10u", i); files[i] = fd = open(path, O_RDWR | O_CREAT, 0644); if (fd < 0) die("can't create file %s\n", strerror(errno)); } } void close_all(int filecount) { int i; for (i = 0; i < filecount; i++) close(files[i]); } int main(int argc, char *argv[]) { pid = getpid(); prog_name = argv[0]; if (argc > 1) delay = atoi(argv[1]); create_all(5000); file_loop(100, 1); file_loop(500, 1); file_loop(1000, 1); file_loop(2000, 1); file_loop(5000, 1); file_loop(100, 0); file_loop(500, 0); file_loop(1000, 0); file_loop(2000, 0); file_loop(5000, 0); exit(EXIT_SUCCESS); } From wferi at niif.hu Tue Apr 24 19:40:56 2007 From: wferi at niif.hu (Ferenc Wagner) Date: Tue, 24 Apr 2007 21:40:56 +0200 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <20070424193600.GB11156@redhat.com> (David Teigland's message of "Tue, 24 Apr 2007 14:36:00 -0500") References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> <87wt06djk7.fsf@tac.ki.iif.hu> <20070423211717.GA22147@redhat.com> <20070424193600.GB11156@redhat.com> Message-ID: <87irblimrb.fsf@tac.ki.iif.hu> David Teigland writes: > I've attached a little flock/plock performance test that emulates what > you're doing; could you run it on your cluster and send the results? Hi David, I can't find the test program in your mail. Are you sure you attached it? Thanks to the shape of the Earth, I'm going to bed now, but would run it tomorrow. Your support is very much appreciated, thanks for looking into this! -- Regards, Feri. From wferi at niif.hu Tue Apr 24 19:42:03 2007 From: wferi at niif.hu (Ferenc Wagner) Date: Tue, 24 Apr 2007 21:42:03 +0200 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <20070424194612.GC11156@redhat.com> (David Teigland's message of "Tue, 24 Apr 2007 14:46:13 -0500") References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> <87wt06djk7.fsf@tac.ki.iif.hu> <20070423211717.GA22147@redhat.com> <20070424193600.GB11156@redhat.com> <20070424194612.GC11156@redhat.com> Message-ID: <87ejm9impg.fsf@tac.ki.iif.hu> David Teigland writes: >> I've attached a little flock/plock performance test that emulates what >> you're doing; could you run it on your cluster and send the results? > > Attached now Fine, got it. -- Feri. From pgharios at gmail.com Wed Apr 25 05:55:18 2007 From: pgharios at gmail.com (Patrick Gharios) Date: Wed, 25 Apr 2007 08:55:18 +0300 Subject: [Linux-cluster] Resource Script. In-Reply-To: <462E0D1D.1000608@redhat.com> References: <000001c78655$0ca1a210$3b6410ac@snoopy> <462E0D1D.1000608@redhat.com> Message-ID: <002d01c786fe$5343b5b0$3b6410ac@snoopy> Thank you, its working! I made my scripts look like System V init scripts and its working fine, Thank you, Pat. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Robert Peterson Sent: Tuesday, April 24, 2007 4:59 PM To: linux clustering Subject: Re: [Linux-cluster] Resource Script. Patrick Gharios wrote: > Hello, > > > > Question: > > > > In order to run a script (bash script on a screen for example) by the > cluster service, do I need to include a start argument when calling the > execution of that script? I am trying to execute a simple script but I > noticed in /var/log/messages that the cluster is trying to send a 'start' > argument to the script , the script is not being fired up on the node and > then a 'status' call is executed on the simple script. Seems like it's a > monitoring feature. > > > > In in all, can someone tell me how can I execute a simple bash script on the > cluster service? > > > > Thank you, Pat. Hi Pat, The rgmanager uses start/stop/status similar to when init calls the init scripts, i.e., service start. I recommend making your script look like the other resource agents in /usr/share/cluster. Regards, Bob Peterson Red Hat Cluster Suite -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From sksamy27 at gmail.com Wed Apr 25 05:57:33 2007 From: sksamy27 at gmail.com (Sam K) Date: Wed, 25 Apr 2007 11:27:33 +0530 Subject: [Linux-cluster] iscsi initiator Message-ID: <839a99c80704242257h673c23bvf7c96cd1f82e7c85@mail.gmail.com> hi Thanks. I want iscsi initiator for RHEL 4. How can i download it? What is my login name and password. Please give the needful information. Thanks & Regards S.Kuppusamy -------------- next part -------------- An HTML attachment was scrubbed... URL: From jos at xos.nl Wed Apr 25 06:22:56 2007 From: jos at xos.nl (Jos Vos) Date: Wed, 25 Apr 2007 08:22:56 +0200 Subject: [Linux-cluster] iscsi initiator In-Reply-To: <839a99c80704242257h673c23bvf7c96cd1f82e7c85@mail.gmail.com>; from sksamy27@gmail.com on Wed, Apr 25, 2007 at 11:27:33AM +0530 References: <839a99c80704242257h673c23bvf7c96cd1f82e7c85@mail.gmail.com> Message-ID: <20070425082256.A1968@xos037.xos.nl> On Wed, Apr 25, 2007 at 11:27:33AM +0530, Sam K wrote: > I want iscsi initiator for RHEL 4. How can i download it? > > What is my login name and password. Please give the needful information. Just do "up2date -i iscsi-initiator-utils". -- -- Jos Vos -- X/OS Experts in Open Systems BV | Phone: +31 20 6938364 -- Amsterdam, The Netherlands | Fax: +31 20 6948204 From Alain.Moulle at bull.net Wed Apr 25 07:04:35 2007 From: Alain.Moulle at bull.net (Alain Moulle) Date: Wed, 25 Apr 2007 09:04:35 +0200 Subject: [Linux-cluster] CS4 U4 / crash when "less /proc/cluster/status" Message-ID: <462EFD83.7080107@bull.net> Hi I sent this problem on CS4 U2 some months ago, and it seems that CS4 U4 does not solve this little problem. I don't remember the bugzilla number ... so can't check the responses on the the bug, do you have some news about this ? Thanks Alain > > Strange, when you do : > less /proc/cluster/status > on one node of a HA pair, the system crashes ! > If you use cat /proc/cluster/status, no crash. > Any explanation ? > Is it always true with CS4 U4 ? > > Thanks > Alain Moull? -- mailto:Alain.Moulle at bull.net +------------------------------+--------------------------------+ | Alain Moull? | from France : 04 76 29 75 99 | | | FAX number : 04 76 29 72 49 | | Bull SA | | | 1, Rue de Provence | Adr : FREC B1-041 | | B.P. 208 | | | 38432 Echirolles - CEDEX | Email: Alain.Moulle at bull.net | | France | BCOM : 229 7599 | +-------------------------------+-------------------------------+ From pcaulfie at redhat.com Wed Apr 25 07:45:12 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 25 Apr 2007 08:45:12 +0100 Subject: [Linux-cluster] CS4 U4 / crash when "less /proc/cluster/status" In-Reply-To: <462EFD83.7080107@bull.net> References: <462EFD83.7080107@bull.net> Message-ID: <462F0708.6090803@redhat.com> Alain Moulle wrote: > Hi > > I sent this problem on CS4 U2 some months ago, > and it seems that CS4 U4 does not solve this > little problem. I don't remember the bugzilla > number ... so can't check the responses on the > the bug, do you have some news about this ? > > Thanks > Alain >> Strange, when you do : >> less /proc/cluster/status >> on one node of a HA pair, the system crashes ! >> If you use cat /proc/cluster/status, no crash. >> Any explanation ? >> Is it always true with CS4 U4 ? >> >> Thanks >> Alain Moull? I very much doubt that it is "always" true, as I do this several times a day and have yet to see it crash. Do you have any more information - such as a traceback ? -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From tomas.hoger at gmail.com Wed Apr 25 11:43:22 2007 From: tomas.hoger at gmail.com (Tomas Hoger) Date: Wed, 25 Apr 2007 13:43:22 +0200 Subject: [Linux-cluster] Maximum restarts and maximum false restarts Message-ID: <6cfbd1b40704250443j78f92dbep697a0b397a53a2a4@mail.gmail.com> Hi! Can anyone provide some additional information regarding following FAQ entry? http://sources.redhat.com/cluster/faq.html#rgm_restarts It says: "If either of those values are exceeded, the service is relocated rather than restarted locally." What are "those values" and how can they be tweaked? Thanks in advance! th. From hlawatschek at atix.de Wed Apr 25 11:58:30 2007 From: hlawatschek at atix.de (Mark Hlawatschek) Date: Wed, 25 Apr 2007 13:58:30 +0200 Subject: [Linux-cluster] GFS assertion failed Message-ID: <200704251358.30234.hlawatschek@atix.de> Hi, I'm trying to compile a module ontop of a gfs filesystem and I receive the follwoing error: Note, that there is an Input/output Error. # ./build make: Entering directory `/usr/src/kernels/2.6.9-42.0.3.EL-smp-x86_64' LD /scratch/qim-v1.0.02b3-src/built-in.o CC [M] /scratch/qim-v1.0.02b3-src/qim_os.o CC [M] /scratch/qim-v1.0.02b3-src/qim_xioct.o CC [M] /scratch/qim-v1.0.02b3-src/qim_mbx.o CC [M] /scratch/qim-v1.0.02b3-src/qim_sup.o CC [M] /scratch/qim-v1.0.02b3-src/qim_inioct.o CC [M] /scratch/qim-v1.0.02b3-src/qim_fo.o CC [M] /scratch/qim-v1.0.02b3-src/qim_32ioctl.o LD [M] /scratch/qim-v1.0.02b3-src/qioctlmod.o Building modules, stage 2. MODPOST Warning: writing sum in /scratch/qim-v1.0.02b3-src/qioctlmod.o failed: Input/output error CC /scratch/qim-v1.0.02b3-src/qioctlmod.mod.o LD [M] /scratch/qim-v1.0.02b3-src/qioctlmod.ko make: Leaving directory `/usr/src/kernels/2.6.9-42.0.3.EL-smp-x86_64' dmesg shows: GFS: fsid=lilr202:lt_scratch.0: warning: assertion "(tmp_gh->gh_flags & GL_LOCAL_EXCL) || !(gh->gh_flags & GL_LOCAL_EXCL)" failed GFS: fsid=lilr202:lt_scratch.0: function = add_to_queue GFS: fsid=lilr202:lt_scratch.0: file = /builddir/build/BUILD/gfs-kernel-2.6.9-60/smp/src/gfs/glock.c, line = 1410 GFS: fsid=lilr202:lt_scratch.0: time = 1177501509 What does that mean ? Mark -- Gruss / Regards, Dipl.-Ing. Mark Hlawatschek http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From Nick.Couchman at seakr.com Wed Apr 25 13:12:46 2007 From: Nick.Couchman at seakr.com (Nick Couchman) Date: Wed, 25 Apr 2007 07:12:46 -0600 Subject: [Linux-cluster] iscsi initiator Message-ID: <462EFF6D.87A6.0099.1@seakr.com> First of all, this isn't the place to ask for that information. This is the linux-cluster mailing list - for questions about the RedHat Cluster Suite - iSCSI initiators are not part of the RH Cluster Suite.. Second, you haven't said which site you're trying to log in to. Even if this was the right place to ask, we couldn't help you, because you haven't given enough information. You could be talking about any site in the world - we don't know. Third, if you want an iSCSI initiator for RHEL4, there are several places you can get it. You might be able to download it from the RedHat Network site, however this requires an active support contract with RedHat. You can also try rpmfind.net - it has many, many RPMs for many distributions and may have an iSCSI initiator for RHEL4. Also try http://linux-iscsi.sourceforge.net/ - this is the site for the Cisco iSCSI initiator for Linux, which works on kernel 2.6 up to 2.6.10 or 2.6.11, I think. If that one doesn't work, go to www.open-iscsi.org - that's the latest iSCSI initiator for Linux. --Nick >>> On 2007/04/24 at 23:57:33, "Sam K" wrote: hi Thanks. I want iscsi initiator for RHEL 4. How can i download it? What is my login name and password. Please give the needful information. Thanks & Regards S.Kuppusamy -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpeterso at redhat.com Wed Apr 25 13:43:30 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Wed, 25 Apr 2007 08:43:30 -0500 Subject: [Linux-cluster] CS4 U4 / crash when "less /proc/cluster/status" In-Reply-To: <462EFD83.7080107@bull.net> References: <462EFD83.7080107@bull.net> Message-ID: <462F5B02.8040104@redhat.com> Alain Moulle wrote: > Hi > > I sent this problem on CS4 U2 some months ago, > and it seems that CS4 U4 does not solve this > little problem. I don't remember the bugzilla > number ... so can't check the responses on the > the bug, do you have some news about this ? > > Thanks > Alain >> Strange, when you do : >> less /proc/cluster/status >> on one node of a HA pair, the system crashes ! >> If you use cat /proc/cluster/status, no crash. >> Any explanation ? >> Is it always true with CS4 U4 ? >> >> Thanks >> Alain Moull? Hi Alain, I wrote the fix for this bug. The bugzilla tracking it is: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=213723 I submitted the code change in Early November of last year, but I don't know which RHEL4 update it went out at. Regards, Bob Peterson Red Hat Cluster Suite From pcaulfie at redhat.com Wed Apr 25 14:11:11 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 25 Apr 2007 15:11:11 +0100 Subject: [Linux-cluster] CS4 U4 / crash when "less /proc/cluster/status" In-Reply-To: <462F5B02.8040104@redhat.com> References: <462EFD83.7080107@bull.net> <462F5B02.8040104@redhat.com> Message-ID: <462F617F.7060706@redhat.com> Robert Peterson wrote: > Alain Moulle wrote: >> Hi >> >> I sent this problem on CS4 U2 some months ago, >> and it seems that CS4 U4 does not solve this >> little problem. I don't remember the bugzilla >> number ... so can't check the responses on the >> the bug, do you have some news about this ? >> >> Thanks >> Alain >>> Strange, when you do : >>> less /proc/cluster/status >>> on one node of a HA pair, the system crashes ! >>> If you use cat /proc/cluster/status, no crash. >>> Any explanation ? >>> Is it always true with CS4 U4 ? >>> >>> Thanks >>> Alain Moull? > Hi Alain, > > I wrote the fix for this bug. The bugzilla tracking it is: > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=213723 > > I submitted the code change in Early November of last year, > but I don't know which RHEL4 update it went out at. It looks like you might have just missed U4. It'll be in U5 though. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From lhh at redhat.com Wed Apr 25 14:46:42 2007 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 25 Apr 2007 10:46:42 -0400 Subject: [Linux-cluster] strange ccsd behavior In-Reply-To: <462C7DA3.1020309@fu-berlin.de> References: <462C7DA3.1020309@fu-berlin.de> Message-ID: <20070425144642.GL4897@redhat.com> On Mon, Apr 23, 2007 at 11:34:27AM +0200, Sebastian Walter wrote: > Dear list, > > my cluster suite doesn't start up. Some nodes work, but some give me > > ccsd -n > Starting ccsd 1.0.7: > Built: Aug 26 2006 15:01:49 > Copyright (C) Red Hat, Inc. 2004 All rights reserved. > No Daemon:: SET > > Unable to connect to cluster infrastructure after 30 seconds. > Unable to connect to cluster infrastructure after 60 seconds. > Unable to connect to cluster infrastructure after 90 seconds. > Unable to connect to cluster infrastructure after 120 seconds. > Unable to connect to cluster infrastructure after 150 seconds. > > > Cluster.conf is the some on all nodes. Hostnames are correct, the > firewalls open for the appropriate subnet. The strange thing is that > sometimes it works. On the other cluster nodes I don't see any messages. > How can I debug this? > > System: CentOS 4.4 > Version: ccsd 1.0.7 (built Aug 26 2006 15:01:49) Check /proc/cluster/nodes and /proc/cluster/status -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Wed Apr 25 14:48:43 2007 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 25 Apr 2007 10:48:43 -0400 Subject: [Linux-cluster] Resource Script. In-Reply-To: <000001c78655$0ca1a210$3b6410ac@snoopy> References: <000001c78655$0ca1a210$3b6410ac@snoopy> Message-ID: <20070425144843.GM4897@redhat.com> On Tue, Apr 24, 2007 at 12:43:03PM +0300, Patrick Gharios wrote: > Hello, > Question: > > In order to run a script (bash script on a screen for example) by the > cluster service, do I need to include a start argument when calling the > execution of that script? I am trying to execute a simple script but I > noticed in /var/log/messages that the cluster is trying to send a 'start' > argument to the script , the script is not being fired up on the node and > then a 'status' call is executed on the simple script. Seems like it's a > monitoring feature. > > In in all, can someone tell me how can I execute a simple bash script on the > cluster service? Like, a one-time thing? Just add an argument handler at the beginning: case $1 in start) ;; *) exit 0 esac -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Wed Apr 25 15:03:17 2007 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 25 Apr 2007 11:03:17 -0400 Subject: [Linux-cluster] Piranha question: Running client on a real server hangs In-Reply-To: References: Message-ID: <20070425150317.GN4897@redhat.com> On Tue, Apr 24, 2007 at 02:32:52PM -0500, Brooks, Jody wrote: > Bottom line: it works if client is run from non-real-server machines, > and half the time if client is run from a real server machine but the > other half of the time (when redirected to itself), it hangs. (I > recognize "half" is because I just have 2 real servers.) > Any clues as to why it won't work consistently when balanced back to > itself? http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-DR.html#realserver_as_client_in_LVS-DR I'm not sure whether or not it can be done without any problems. Are you using the redirect bits like in "method #2", here: http://people.redhat.com/lhh/piranha-direct-routing-howto.txt ? If so, if you used the other method, connections from the LVS real servers would simply connect locally. It would always work - but not be load-balanced via the director. > (Extra note: I say it works from non-real-server machines... not > completely true: doesn't work from the router machine, rtr1, as that > always goes back to itself, rtr1, ignoring the ipvs tables apparently as > ipvsadm -l never shows the connection in its statistics... Apparently > the virtual IP address being local short-circuits the ipvs table > operations). Correct. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From wferi at niif.hu Wed Apr 25 15:47:53 2007 From: wferi at niif.hu (Ferenc Wagner) Date: Wed, 25 Apr 2007 17:47:53 +0200 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <20070424193600.GB11156@redhat.com> (David Teigland's message of "Tue, 24 Apr 2007 14:36:00 -0500") References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> <87wt06djk7.fsf@tac.ki.iif.hu> <20070423211717.GA22147@redhat.com> <20070424193600.GB11156@redhat.com> Message-ID: <87irbktpzq.fsf@tac.ki.iif.hu> David Teigland writes: > On Mon, Apr 23, 2007 at 04:17:18PM -0500, David Teigland wrote: >> > Also, what's that new infrastructure? Do you mean GFS2? I read it >> > was not production quality yet, so I didn't mean to try it. But again >> > you may have got something else in your head... >> >> GFS1 and GFS2 both run on the new openais-based cluster infrastructure. >> (in the cluster-2.00.00 release, and the RHEL5 and HEAD cvs branches). > > I've attached a little flock/plock performance test that emulates what > you're doing; could you run it on your cluster and send the results? Here you go. This is with three nodes mounting the FS, one running the test. /proc/cluster/lock_dlm/drop_count was set to 0 before the mounts. wferi at rs20:/mnt/rrdtest/david/test$ ../fplockperf flock: filecount=100 iteration=0 elapsed time=1.163 s flock: filecount=100 iteration=1 elapsed time=1.200 s flock: filecount=100 iteration=2 elapsed time=1.200 s flock: filecount=100 iteration=3 elapsed time=1.200 s flock: filecount=100 iteration=4 elapsed time=1.200 s total elapsed time=5.963 s flock: filecount=500 iteration=0 elapsed time=6.997 s flock: filecount=500 iteration=1 elapsed time=6.998 s flock: filecount=500 iteration=2 elapsed time=7.235 s flock: filecount=500 iteration=3 elapsed time=6.999 s flock: filecount=500 iteration=4 elapsed time=6.999 s total elapsed time=35.228 s flock: filecount=1000 iteration=0 elapsed time=13.797 s flock: filecount=1000 iteration=1 elapsed time=14.006 s flock: filecount=1000 iteration=2 elapsed time=13.798 s flock: filecount=1000 iteration=3 elapsed time=14.010 s flock: filecount=1000 iteration=4 elapsed time=13.798 s total elapsed time=69.408 s flock: filecount=2000 iteration=0 elapsed time=28.888 s flock: filecount=2000 iteration=1 elapsed time=28.883 s flock: filecount=2000 iteration=2 elapsed time=28.675 s flock: filecount=2000 iteration=3 elapsed time=28.879 s flock: filecount=2000 iteration=4 elapsed time=28.879 s total elapsed time=144.205 s flock: filecount=5000 iteration=0 elapsed time=71.272 s flock: filecount=5000 iteration=1 elapsed time=68.620 s flock: filecount=5000 iteration=2 elapsed time=68.668 s flock: filecount=5000 iteration=3 elapsed time=68.664 s flock: filecount=5000 iteration=4 elapsed time=68.676 s total elapsed time=345.901 s plock: filecount=100 iteration=0 elapsed time=1.515 s plock: filecount=100 iteration=1 elapsed time=1.480 s plock: filecount=100 iteration=2 elapsed time=1.480 s plock: filecount=100 iteration=3 elapsed time=1.480 s plock: filecount=100 iteration=4 elapsed time=1.480 s total elapsed time=7.434 s plock: filecount=500 iteration=0 elapsed time=6.156 s plock: filecount=500 iteration=1 elapsed time=6.011 s plock: filecount=500 iteration=2 elapsed time=6.279 s plock: filecount=500 iteration=3 elapsed time=6.043 s plock: filecount=500 iteration=4 elapsed time=6.039 s total elapsed time=30.528 s plock: filecount=1000 iteration=0 elapsed time=12.382 s plock: filecount=1000 iteration=1 elapsed time=12.709 s plock: filecount=1000 iteration=2 elapsed time=12.318 s plock: filecount=1000 iteration=3 elapsed time=12.471 s plock: filecount=1000 iteration=4 elapsed time=12.609 s total elapsed time=62.488 s plock: filecount=2000 iteration=0 elapsed time=26.784 s plock: filecount=2000 iteration=1 elapsed time=30.671 s plock: filecount=2000 iteration=2 elapsed time=30.563 s plock: filecount=2000 iteration=3 elapsed time=30.407 s plock: filecount=2000 iteration=4 elapsed time=30.443 s total elapsed time=148.867 s plock: filecount=5000 iteration=0 elapsed time=81.813 s plock: filecount=5000 iteration=1 elapsed time=79.037 s plock: filecount=5000 iteration=2 elapsed time=77.407 s plock: filecount=5000 iteration=3 elapsed time=77.771 s plock: filecount=5000 iteration=4 elapsed time=78.243 s total elapsed time=394.271 s ======================================================= After umount on all nodes, rmmod gfs lock_dlm, mount on single node, rm file000000*: wferi at rs20:/mnt/rrdtest/david/test$ ../fplockperf flock: filecount=100 iteration=0 elapsed time=0.008 s flock: filecount=100 iteration=1 elapsed time=0.008 s flock: filecount=100 iteration=2 elapsed time=0.008 s flock: filecount=100 iteration=3 elapsed time=0.008 s flock: filecount=100 iteration=4 elapsed time=0.008 s total elapsed time=0.039 s flock: filecount=500 iteration=0 elapsed time=0.039 s flock: filecount=500 iteration=1 elapsed time=0.038 s flock: filecount=500 iteration=2 elapsed time=0.038 s flock: filecount=500 iteration=3 elapsed time=0.038 s flock: filecount=500 iteration=4 elapsed time=0.038 s total elapsed time=0.192 s flock: filecount=1000 iteration=0 elapsed time=0.077 s flock: filecount=1000 iteration=1 elapsed time=0.076 s flock: filecount=1000 iteration=2 elapsed time=0.076 s flock: filecount=1000 iteration=3 elapsed time=0.077 s flock: filecount=1000 iteration=4 elapsed time=0.077 s total elapsed time=0.383 s flock: filecount=2000 iteration=0 elapsed time=0.153 s flock: filecount=2000 iteration=1 elapsed time=0.153 s flock: filecount=2000 iteration=2 elapsed time=0.154 s flock: filecount=2000 iteration=3 elapsed time=0.153 s flock: filecount=2000 iteration=4 elapsed time=0.150 s total elapsed time=0.763 s flock: filecount=5000 iteration=0 elapsed time=0.377 s flock: filecount=5000 iteration=1 elapsed time=0.373 s flock: filecount=5000 iteration=2 elapsed time=0.378 s flock: filecount=5000 iteration=3 elapsed time=0.381 s flock: filecount=5000 iteration=4 elapsed time=0.385 s total elapsed time=1.895 s plock: filecount=100 iteration=0 elapsed time=0.017 s plock: filecount=100 iteration=1 elapsed time=0.015 s plock: filecount=100 iteration=2 elapsed time=0.015 s plock: filecount=100 iteration=3 elapsed time=0.016 s plock: filecount=100 iteration=4 elapsed time=0.015 s total elapsed time=0.079 s plock: filecount=500 iteration=0 elapsed time=0.089 s plock: filecount=500 iteration=1 elapsed time=0.081 s plock: filecount=500 iteration=2 elapsed time=0.081 s plock: filecount=500 iteration=3 elapsed time=0.080 s plock: filecount=500 iteration=4 elapsed time=0.081 s total elapsed time=0.412 s plock: filecount=1000 iteration=0 elapsed time=0.182 s plock: filecount=1000 iteration=1 elapsed time=0.179 s plock: filecount=1000 iteration=2 elapsed time=0.178 s plock: filecount=1000 iteration=3 elapsed time=0.178 s plock: filecount=1000 iteration=4 elapsed time=0.177 s total elapsed time=0.894 s plock: filecount=2000 iteration=0 elapsed time=0.437 s plock: filecount=2000 iteration=1 elapsed time=0.446 s plock: filecount=2000 iteration=2 elapsed time=0.455 s plock: filecount=2000 iteration=3 elapsed time=0.457 s plock: filecount=2000 iteration=4 elapsed time=0.460 s total elapsed time=2.255 s plock: filecount=5000 iteration=0 elapsed time=1.136 s plock: filecount=5000 iteration=1 elapsed time=1.159 s plock: filecount=5000 iteration=2 elapsed time=1.151 s plock: filecount=5000 iteration=3 elapsed time=1.153 s plock: filecount=5000 iteration=4 elapsed time=1.171 s total elapsed time=5.770 s ====================================================== After mount on another node, rm file000000*: wferi at rs20:/mnt/rrdtest/david/test$ ../fplockperf flock: filecount=100 iteration=0 elapsed time=0.013 s flock: filecount=100 iteration=1 elapsed time=0.013 s flock: filecount=100 iteration=2 elapsed time=0.013 s flock: filecount=100 iteration=3 elapsed time=0.013 s flock: filecount=100 iteration=4 elapsed time=0.013 s total elapsed time=0.066 s flock: filecount=500 iteration=0 elapsed time=0.067 s flock: filecount=500 iteration=1 elapsed time=0.067 s flock: filecount=500 iteration=2 elapsed time=0.067 s flock: filecount=500 iteration=3 elapsed time=0.067 s flock: filecount=500 iteration=4 elapsed time=0.067 s total elapsed time=0.333 s flock: filecount=1000 iteration=0 elapsed time=0.134 s flock: filecount=1000 iteration=1 elapsed time=0.132 s flock: filecount=1000 iteration=2 elapsed time=0.137 s flock: filecount=1000 iteration=3 elapsed time=0.134 s flock: filecount=1000 iteration=4 elapsed time=0.133 s total elapsed time=0.670 s flock: filecount=2000 iteration=0 elapsed time=0.274 s flock: filecount=2000 iteration=1 elapsed time=0.282 s flock: filecount=2000 iteration=2 elapsed time=0.284 s flock: filecount=2000 iteration=3 elapsed time=0.285 s flock: filecount=2000 iteration=4 elapsed time=0.284 s total elapsed time=1.408 s flock: filecount=5000 iteration=0 elapsed time=0.716 s flock: filecount=5000 iteration=1 elapsed time=0.716 s flock: filecount=5000 iteration=2 elapsed time=0.694 s flock: filecount=5000 iteration=3 elapsed time=0.705 s flock: filecount=5000 iteration=4 elapsed time=0.839 s total elapsed time=3.671 s plock: filecount=100 iteration=0 elapsed time=0.029 s plock: filecount=100 iteration=1 elapsed time=0.021 s plock: filecount=100 iteration=2 elapsed time=0.021 s plock: filecount=100 iteration=3 elapsed time=0.021 s plock: filecount=100 iteration=4 elapsed time=0.021 s total elapsed time=0.114 s plock: filecount=500 iteration=0 elapsed time=0.144 s plock: filecount=500 iteration=1 elapsed time=0.114 s plock: filecount=500 iteration=2 elapsed time=0.111 s plock: filecount=500 iteration=3 elapsed time=0.111 s plock: filecount=500 iteration=4 elapsed time=0.111 s total elapsed time=0.591 s plock: filecount=1000 iteration=0 elapsed time=0.271 s plock: filecount=1000 iteration=1 elapsed time=0.235 s plock: filecount=1000 iteration=2 elapsed time=0.236 s plock: filecount=1000 iteration=3 elapsed time=0.235 s plock: filecount=1000 iteration=4 elapsed time=0.234 s total elapsed time=1.212 s plock: filecount=2000 iteration=0 elapsed time=0.657 s plock: filecount=2000 iteration=1 elapsed time=0.843 s plock: filecount=2000 iteration=2 elapsed time=0.794 s plock: filecount=2000 iteration=3 elapsed time=0.701 s plock: filecount=2000 iteration=4 elapsed time=0.794 s total elapsed time=3.789 s plock: filecount=5000 iteration=0 elapsed time=1.941 s plock: filecount=5000 iteration=1 elapsed time=1.946 s plock: filecount=5000 iteration=2 elapsed time=2.062 s plock: filecount=5000 iteration=3 elapsed time=1.972 s plock: filecount=5000 iteration=4 elapsed time=1.964 s total elapsed time=9.886 s ====================================================== After mount on the third node, rm file000000* again: wferi at rs20:/mnt/rrdtest/david/test$ ../fplockperf flock: filecount=100 iteration=0 elapsed time=1.273 s flock: filecount=100 iteration=1 elapsed time=1.360 s flock: filecount=100 iteration=2 elapsed time=1.360 s flock: filecount=100 iteration=3 elapsed time=1.360 s flock: filecount=100 iteration=4 elapsed time=1.360 s total elapsed time=6.712 s flock: filecount=500 iteration=0 elapsed time=6.479 s flock: filecount=500 iteration=1 elapsed time=6.479 s flock: filecount=500 iteration=2 elapsed time=6.671 s flock: filecount=500 iteration=3 elapsed time=6.479 s flock: filecount=500 iteration=4 elapsed time=6.479 s total elapsed time=32.586 s flock: filecount=1000 iteration=0 elapsed time=13.546 s flock: filecount=1000 iteration=1 elapsed time=13.790 s flock: filecount=1000 iteration=2 elapsed time=13.598 s flock: filecount=1000 iteration=3 elapsed time=13.598 s flock: filecount=1000 iteration=4 elapsed time=13.790 s total elapsed time=68.321 s flock: filecount=2000 iteration=0 elapsed time=27.852 s flock: filecount=2000 iteration=1 elapsed time=27.906 s flock: filecount=2000 iteration=2 elapsed time=28.159 s flock: filecount=2000 iteration=3 elapsed time=28.147 s flock: filecount=2000 iteration=4 elapsed time=28.099 s total elapsed time=140.164 s flock: filecount=5000 iteration=0 elapsed time=66.564 s flock: filecount=5000 iteration=1 elapsed time=66.401 s flock: filecount=5000 iteration=2 elapsed time=66.217 s flock: filecount=5000 iteration=3 elapsed time=66.401 s flock: filecount=5000 iteration=4 elapsed time=66.413 s total elapsed time=331.996 s plock: filecount=100 iteration=0 elapsed time=1.520 s plock: filecount=100 iteration=1 elapsed time=1.520 s plock: filecount=100 iteration=2 elapsed time=1.520 s plock: filecount=100 iteration=3 elapsed time=1.520 s plock: filecount=100 iteration=4 elapsed time=1.520 s total elapsed time=7.599 s plock: filecount=500 iteration=0 elapsed time=6.911 s plock: filecount=500 iteration=1 elapsed time=6.615 s plock: filecount=500 iteration=2 elapsed time=6.491 s plock: filecount=500 iteration=3 elapsed time=6.519 s plock: filecount=500 iteration=4 elapsed time=6.519 s total elapsed time=33.055 s plock: filecount=1000 iteration=0 elapsed time=13.906 s plock: filecount=1000 iteration=1 elapsed time=13.458 s plock: filecount=1000 iteration=2 elapsed time=13.554 s plock: filecount=1000 iteration=3 elapsed time=13.774 s plock: filecount=1000 iteration=4 elapsed time=13.405 s total elapsed time=68.096 s plock: filecount=2000 iteration=0 elapsed time=31.127 s plock: filecount=2000 iteration=1 elapsed time=31.475 s plock: filecount=2000 iteration=2 elapsed time=31.883 s plock: filecount=2000 iteration=3 elapsed time=31.799 s plock: filecount=2000 iteration=4 elapsed time=32.067 s total elapsed time=158.349 s plock: filecount=5000 iteration=0 elapsed time=86.233 s plock: filecount=5000 iteration=1 elapsed time=79.686 s plock: filecount=5000 iteration=2 elapsed time=78.507 s plock: filecount=5000 iteration=3 elapsed time=80.542 s plock: filecount=5000 iteration=4 elapsed time=79.151 s total elapsed time=404.119 s That is, we are basically back at the first result. The third node mounting the FS causes much more slowdown than I expected. It's possible that I can live with only two nodes for this task, but more will be needed for the next, so I'm absolutely willing to investigate further or hear possbile ways out of this issue. -- Thanks, Feri. From jos at xos.nl Wed Apr 25 15:52:26 2007 From: jos at xos.nl (Jos Vos) Date: Wed, 25 Apr 2007 17:52:26 +0200 Subject: [Linux-cluster] iscsi initiator In-Reply-To: <462EFF6D.87A6.0099.1@seakr.com>; from Nick.Couchman@seakr.com on Wed, Apr 25, 2007 at 07:12:46AM -0600 References: <462EFF6D.87A6.0099.1@seakr.com> Message-ID: <20070425175226.A5937@xos037.xos.nl> On Wed, Apr 25, 2007 at 07:12:46AM -0600, Nick Couchman wrote: > Third, if you want an iSCSI initiator for RHEL4, there are several > places you can get it. You might be able to download it from the > RedHat Network site, however this requires an active support contract > with RedHat. [...] Formally, when someone says he's running RHEL4 (assuming he is really meaning RHEL4, and not some clone), he (AFAIK) should have an active support contract with RH. -- -- Jos Vos -- X/OS Experts in Open Systems BV | Phone: +31 20 6938364 -- Amsterdam, The Netherlands | Fax: +31 20 6948204 From jody.brooks at nasa.gov Wed Apr 25 15:56:24 2007 From: jody.brooks at nasa.gov (Brooks, Jody) Date: Wed, 25 Apr 2007 10:56:24 -0500 Subject: [Linux-cluster] Piranha question: Running client on a real serverhangs In-Reply-To: <20070425150317.GN4897@redhat.com> References: <20070425150317.GN4897@redhat.com> Message-ID: Thanks for the links... working my way through them now ... For info: I am using the redirect bits like in "method #2". In fact, that's all I'm doing on the real servers as I have not assigned the VIP to the lo interface on the real servers as seems to be discussed in LVS-DR.html (haven't read all that yet but that appears to be implied as I've read that in other places as well). This VIP on lo doesn't appear to be critical as my service works fine when the client is on a non-real-server/non-director host. Am I missing something here? Seems like the VIP on lo would be used if you weren't doing the iptables redirect (i.e., method 2)... right? wrong? I will try to incorporate some of the pieces from the LVS-DR.html ideas and see if that gets me any further. For the record, our need of this is that one of our services is also a consumer of other of our services, all of which should be load-balanced across the same cluster as they're related. Thanks again. -- Jody -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Lon Hohberger Sent: Wednesday, April 25, 2007 10:03 AM To: linux clustering Subject: Re: [Linux-cluster] Piranha question: Running client on a real serverhangs On Tue, Apr 24, 2007 at 02:32:52PM -0500, Brooks, Jody wrote: > Bottom line: it works if client is run from non-real-server machines, > and half the time if client is run from a real server machine but the > other half of the time (when redirected to itself), it hangs. (I > recognize "half" is because I just have 2 real servers.) > Any clues as to why it won't work consistently when balanced back to > itself? http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-DR.html#reals erver_as_client_in_LVS-DR I'm not sure whether or not it can be done without any problems. Are you using the redirect bits like in "method #2", here: http://people.redhat.com/lhh/piranha-direct-routing-howto.txt ? If so, if you used the other method, connections from the LVS real servers would simply connect locally. It would always work - but not be load-balanced via the director. > (Extra note: I say it works from non-real-server machines... not > completely true: doesn't work from the router machine, rtr1, as that > always goes back to itself, rtr1, ignoring the ipvs tables apparently as > ipvsadm -l never shows the connection in its statistics... Apparently > the virtual IP address being local short-circuits the ipvs table > operations). Correct. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From Arne.Brieseneck at vodafone.com Wed Apr 25 16:26:29 2007 From: Arne.Brieseneck at vodafone.com (Brieseneck, Arne, VF-Group) Date: Wed, 25 Apr 2007 18:26:29 +0200 Subject: [Linux-cluster] startup problem: "cman not started: Can't bind to local cman socket /usr/sbin/cman_tool: aisexec daemon didn't start " Message-ID: it is high likely that anybody already knows this error and knows how to fix it: ---snip--- [root at box1 ~]# /etc/init.d/cman start Starting cluster: Loading modules... done Mounting configfs... done Starting ccsd... done Starting cman... failed cman not started: Can't bind to local cman socket /usr/sbin/cman_tool: aisexec daemon didn't start [FAILED] [root at box1 ~]# ---snap--- the cluster does not start even what is written in the logfileif looks OK for me: ---snip--- Apr 24 19:59:06 box1 ccsd[5129]: Starting ccsd 2.0.60: Apr 24 19:59:06 box1 ccsd[5129]: Built: Jan 24 2007 15:31:03 Apr 24 19:59:06 box1 ccsd[5129]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Apr 24 19:59:06 box1 ccsd[5129]: cluster.conf (cluster name = alpha_cluster, version = 6) found. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] AIS Executive Service RELEASE 'subrev 1204 version 0.80.1' Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Using default multicast address of 239.192.196.121 Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component openais_cpg loaded. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component openais_cfg loaded. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service handler 'openais configuration service' Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component openais_msg loaded. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service handler 'openais message service B.01.01' Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component openais_lck loaded. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component openais_evt loaded. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service handler 'openais event service B.01.01' Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component openais_ckpt loaded. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component openais_amf loaded. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component openais_clm loaded. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component openais_evs loaded. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component openais_cman loaded. Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' Apr 24 19:59:08 box1 openais[5070]: [TOTEM] entering GATHER state from 12. Apr 24 19:59:08 box1 openais[5070]: [TOTEM] Creating commit token because I am the rep. Apr 24 19:59:08 box1 openais[5070]: [TOTEM] Saving state aru 1e high seq received 1e Apr 24 19:59:08 box1 openais[5070]: [TOTEM] Storing new sequence id for ring 304 Apr 24 19:59:08 box1 openais[5070]: [TOTEM] entering COMMIT state. Apr 24 19:59:09 box1 openais[5070]: [TOTEM] entering RECOVERY state. Apr 24 19:59:09 box1 openais[5070]: [TOTEM] position [0] member 192.168.50.194: Apr 24 19:59:09 box1 openais[5070]: [TOTEM] previous ring seq 768 rep 192.168.50.194 Apr 24 19:59:09 box1 openais[5070]: [TOTEM] aru 1e high delivered 1e received flag 0 Apr 24 19:59:09 box1 openais[5070]: [TOTEM] position [1] member 192.168.50.195: Apr 24 19:59:09 box1 openais[5070]: [TOTEM] previous ring seq 768 rep 192.168.50.194 Apr 24 19:59:09 box1 openais[5070]: [TOTEM] aru 1e high delivered 1e received flag 0 Apr 24 19:59:09 box1 openais[5070]: [TOTEM] position [2] member 192.168.50.196: Apr 24 19:59:09 box1 openais[5070]: [TOTEM] previous ring seq 768 rep 192.168.50.194 Apr 24 19:59:09 box1 openais[5070]: [TOTEM] aru 1e high delivered 1e received flag 0 Apr 24 19:59:09 box1 openais[5070]: [TOTEM] Did not need to originate any messages in recovery. Apr 24 19:59:09 box1 openais[5070]: [CLM ] CLM CONFIGURATION CHANGE Apr 24 19:59:09 box1 openais[5070]: [CLM ] New Configuration: Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.194) Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.195) Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.196) Apr 24 19:59:09 box1 openais[5070]: [CLM ] Members Left: Apr 24 19:59:09 box1 openais[5070]: [CLM ] Members Joined: Apr 24 19:59:09 box1 openais[5070]: [SYNC ] This node is within the primary component and will provide service. Apr 24 19:59:09 box1 openais[5070]: [CLM ] CLM CONFIGURATION CHANGE Apr 24 19:59:09 box1 openais[5070]: [CLM ] New Configuration: Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.194) Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.195) Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.196) Apr 24 19:59:09 box1 openais[5070]: [CLM ] Members Left: Apr 24 19:59:09 box1 openais[5070]: [CLM ] Members Joined: Apr 24 19:59:09 box1 openais[5070]: [SYNC ] This node is within the primary component and will provide service. Apr 24 19:59:09 box1 openais[5070]: [TOTEM] entering OPERATIONAL state. Apr 24 19:59:09 box1 openais[5070]: [CLM ] got nodejoin message 192.168.50.195 Apr 24 19:59:09 box1 openais[5070]: [CLM ] got nodejoin message 192.168.50.196 Apr 24 19:59:09 box1 openais[5070]: [CLM ] got nodejoin message 192.168.50.194 Apr 24 19:59:35 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 30 seconds. Apr 24 20:00:05 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 60 seconds. Apr 24 20:00:35 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 90 seconds. Apr 24 20:01:05 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 120 seconds. Apr 24 20:01:35 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 150 seconds. Apr 24 20:02:05 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 180 seconds. Apr 24 20:02:35 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 210 seconds. Apr 24 20:03:05 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 240 seconds. Apr 24 20:03:35 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 270 seconds. Apr 24 20:04:05 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 300 seconds. Apr 24 20:04:35 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 330 seconds. Apr 24 20:05:05 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 360 seconds. Apr 24 20:05:35 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 390 seconds. Apr 24 20:06:05 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 420 seconds. Apr 24 20:06:35 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 450 seconds. Apr 24 20:07:05 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 480 seconds. Apr 24 20:07:35 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 510 seconds. Apr 24 20:08:05 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 540 seconds. Apr 24 20:08:35 box1 ccsd[5129]: Unable to connect to cluster infrastructure after 570 seconds. Apr 24 20:09:01 box1 ccsd[5129]: Stopping ccsd, SIGTERM received. ---snap--- thats the /etc/cluster/cluster.conf: ---snip--- [root at box1 ~]# cat /etc/cluster/cluster.conf ---snap--- -------------- next part -------------- An HTML attachment was scrubbed... URL: From teigland at redhat.com Wed Apr 25 16:55:53 2007 From: teigland at redhat.com (David Teigland) Date: Wed, 25 Apr 2007 11:55:53 -0500 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <87irbktpzq.fsf@tac.ki.iif.hu> References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> <87wt06djk7.fsf@tac.ki.iif.hu> <20070423211717.GA22147@redhat.com> <20070424193600.GB11156@redhat.com> <87irbktpzq.fsf@tac.ki.iif.hu> Message-ID: <20070425165553.GC9891@redhat.com> On Wed, Apr 25, 2007 at 05:47:53PM +0200, Ferenc Wagner wrote: > David Teigland writes: > > > On Mon, Apr 23, 2007 at 04:17:18PM -0500, David Teigland wrote: > >> > Also, what's that new infrastructure? Do you mean GFS2? I read it > >> > was not production quality yet, so I didn't mean to try it. But again > >> > you may have got something else in your head... > >> > >> GFS1 and GFS2 both run on the new openais-based cluster infrastructure. > >> (in the cluster-2.00.00 release, and the RHEL5 and HEAD cvs branches). > > > > I've attached a little flock/plock performance test that emulates what > > you're doing; could you run it on your cluster and send the results? > > Here you go. This is with three nodes mounting the FS, one running > the test. /proc/cluster/lock_dlm/drop_count was set to 0 before > the mounts. Thanks, some of these numbers look odd to me; I'll need run the test on my own rhel4 cluster to understand them better. In the end, though, I think you'll get the best performance by using flocks under the new cluster infrastructure and kernel. Dave From simone.gotti at email.it Wed Apr 25 18:52:08 2007 From: simone.gotti at email.it (Simone Gotti) Date: Wed, 25 Apr 2007 20:52:08 +0200 Subject: [Linux-cluster] [RFC][PATCH] Add ability to freeze a service. In-Reply-To: <20070423200249.GB4897@redhat.com> References: <1177319722.3870.29.camel@localhost> <20070423200249.GB4897@redhat.com> Message-ID: <1177527128.14525.15.camel@localhost> On Mon, 2007-04-23 at 16:02 -0400, Lon Hohberger wrote: > On Mon, Apr 23, 2007 at 11:15:22AM +0200, Simone Gotti wrote: > > Hi, > > > > like discussed with lon on IRC I'm trying to add to rgmanager the > > ability to freeze a service. I worked on it in these days and did an > > example patch. Here is how I think what a "freeze" can be and, of > > course, it can be implemented in many other ways so it's only an > > example. > > Hi, > > couple of comments - > Hi Lon, > (1) s/freezed/frozen/ig :) Uh... :( I hope I fixed them all now without adding other language errors. > (2) svc_status_inquiry shouldn't call rg_lock(); you should use > get_rg_state_local() instead of rg_lock/get_rg_state/rg_unlock done. I didn't knew what do if its return value was != 0 so I logged end exited. Is it right? > Note that svc_status_inquiry isn't even used yet, so the behavior > might change at a later date ;) Ok. > (3.a) Warning: The addition of rs_flags to rg_state_t will break > compatibility with existing software (which is fine in -HEAD). It > would be easier to cope with this change (that is- make upgrades from > old->new work) if you added the rs_flags field after the > rs_transition field. > > (3.b) If you don't want to do that, then you need to add another 32-bits > worth of data (could be just a "pad" field) before rs_transition because > the kernel on ia64 machines will complain if they read a 64-bit int and > it's not aligned on a 64-bit boundary. I implemented 3.a. So it should be easily backportable. I wasn't aware of the ia64 alignment problems so thanks for the explanation. > Aside from that, it looks like you got it right for what you wanted it > to do. I can fix (1) (language stuff) if you fix (2) and (3). I tried also to fix (1) (I hope). In the meantime I fixed (like you reported on IRC) the missing changes to clustat.c:xml_rg_state and enhanced a little the default clustat output adding a "Flags" column. I added a parameter "separator" to rg_flags_str so it can be defined. For example in the default clustat I'm using a comma separated list while in the xml output I'm using a space as separator (dunno if this is right to do with xml). Thanks! Bye! > -- Lon > -- Simone Gotti -- Email.it, the professional e-mail, gratis per te: http://www.email.it/f Sponsor: Vuoi diventare un vero esperto sul Controllo di Gestione? Scopri come nella tua azienda puoi migliorare gli utili e ridurre le spese Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=6197&d=25-4 -------------- next part -------------- A non-text attachment was scrubbed... Name: rgmanager-cvsHEAD-add_service_freezing-try02.patch Type: text/x-patch Size: 14859 bytes Desc: not available URL: From chad.slater at clickfox.com Tue Apr 24 03:08:24 2007 From: chad.slater at clickfox.com (Chad Slater) Date: Mon, 23 Apr 2007 23:08:24 -0400 Subject: [Linux-cluster] GFS: fh2dentry misses Message-ID: While troubleshooting some GFS performance issues, I came across this when viewing the counters: fh2dentry misses 292928 What is fh2dentry misses? Here are my counters after the server had been up about 30 minutes. Server1 is primary, server2 is secondary, and they export a 1.5TB GFS file system and a 1TB GFS file system via NFS. server1# gfs_tool counters / locks 1381450 locks held 717600 incore inodes 687837 metadata buffers 0 unlinked inodes 0 quota IDs 0 incore log buffers 0 log space used 0.05% meta header cache entries 0 glock dependencies 0 glocks on reclaim list 0 log wraps 1 outstanding LM calls 0 outstanding BIO calls 3 fh2dentry misses 292928 glocks reclaimed 641232 glock nq calls 17182855 glock dq calls 16460254 glock prefetch calls 1003991 lm_lock calls 2384352 lm_unlock calls 1615283 lm callbacks 4009688 address operations 835770 dentry operations 121681 export operations 3157423 file operations 715530 inode operations 11415856 super operations 3533953 vm operations 0 block I/O reads 1263007 block I/O writes 168759 server2# gfs_tool counters / locks 5787 locks held 32 incore inodes 5 metadata buffers 0 unlinked inodes 0 quota IDs 0 incore log buffers 0 log space used 0.00% meta header cache entries 0 glock dependencies 0 glocks on reclaim list 0 log wraps 0 outstanding LM calls 0 outstanding BIO calls 0 fh2dentry misses 0 glocks reclaimed 1217379 glock nq calls 1230292 glock dq calls 1230285 glock prefetch calls 0 lm_lock calls 1223696 lm_unlock calls 1221535 lm callbacks 2449408 address operations 0 dentry operations 0 export operations 0 file operations 3 inode operations 430 super operations 5052 vm operations 0 block I/O reads 10811 block I/O writes 4 *Pardon the non-plaintext format. -------------- next part -------------- An HTML attachment was scrubbed... URL: From costi at cdvultur.com Tue Apr 24 11:49:51 2007 From: costi at cdvultur.com (Costi) Date: Tue, 24 Apr 2007 14:49:51 +0300 Subject: [Linux-cluster] Fencing using APC7921 In-Reply-To: References: Message-ID: <462DEEDF.6010804@cdvultur.com> I had this type of problem and I had to modify the fence_apc script. There is another thread regarding issues with APC 7921. This were on Fedora Core 4. Hope this helps. James Lapthorn wrote: > I use RSA II cards as my primary fencing device on my IBM 3850's. > After some modifications to the 'fence_rsa' script it now recognises > 'WMN170370264' as the default card name. > > My secondary fencing device is and APC Power Switch APC7921. This is > not a Masterswitch plus so there are two APC7921's, one for each POwer > supply. When testing the the device using the following command: > > fence_apc -a 10.179.20.52 -l apc -p apc -n 4 -v > > I get: > > failed: unrecognised menu response > > Here;s is the output from the /tmp/apclog.txt > ***************************************************************************************************************** > > User Name : apc > Password : *** > > > American Power Conversion Network Management Card > AOS v2.7.0 > (c) Copyright 2004 All Rights Reserved Rack PDU > APP v2.7.3 > ------------------------------------------------------------------------------- > > Name : RackPDU Date : 10/24/2000 > Contact : Unknown Time : 11:45:43 > Location : Unknown User : > Administrator > Up Time : 64 Days 0 Hours 51 Minutes Stat : P+ N+ A+ > > Switched Rack PDU: Communication Established > > ------- Control Console > ------------------------------------------------------- > > 1- Device Manager > 2- Network > 3- System > 4- Logout > > - Main Menu, - Refresh, - Event Log > > > > ------- Control Console > ------------------------------------------------------- > > 1- Device Manager > 2- Network > 3- System > 4- Logout > > - Main Menu, - Refresh, - Event Log > > > > > American Power Conversion Network Management Card > AOS v2.7.0 > (c) Copyright 2004 All Rights Reserved Rack PDU > APP v2.7.3 > ------------------------------------------------------------------------------- > Name : RackPDU Date : 10/24/2000 > Contact : Unknown Time : 11:45:43 > Location : Unknown User : Administrator > Up Time : 64 Days 0 Hours 51 Minutes Stat : P+ N+ A+ > > Switched Rack PDU: Communication Established > > ------- Control Console > ------------------------------------------------------- > > 1- Device Manager > 2- Network > 3- System > 4- Logout > > - Main Menu, - Refresh, - Event Log > > > > ------- Control Console > ------------------------------------------------------- > > 1- Device Manager > 2- Network > 3- System > 4- Logout > > - Main Menu, - Refresh, - Event Log > > 1 > > ------- Device Manager > -------------------------------------------------------- > > 1- Phase Monitor/Configuration > 2- Outlet Restriction Configuration > 3- Outlet Control/Configuration > 4- Power Supply Status > > - Back, - Refresh, - Event Log > > 3 > > ------- Outlet Control/Configuration > ------------------------------------------ > > 1- leoukpdb4 ON > 2- leoukldb4 ON > 3- leoukpdb3 ON > 4- leoukldb3 ON > 5- leoukpdb2 ON > 6- leoukldb2 ON > 7- leoukpdb1 ON > 8- leoukldb1 ON > 9- Master Control/Configuration > > - Back, - Refresh, - Event Log > > > > ------- Outlet Control/Configuration > ------------------------------------------ > > 1- leoukpdb4 ON > 2- leoukldb4 ON > 3- leoukpdb3 ON > 4- leoukldb3 ON > 5- leoukpdb2 ON > 6- leoukldb2 ON > 7- leoukpdb1 ON > 8- leoukldb1 ON > 9- Master Control/Configuration > > - Back, - Refresh, - Event Log > > > > ------- Device Manager > -------------------------------------------------------- > > 1- Phase Monitor/Configuration > 2- Outlet Restriction Configuration > 3- Outlet Control/Configuration > 4- Power Supply Status > > - Back, - Refresh, - Event Log > > > > ------- Control Console > ------------------------------------------------------- > > 1- Device Manager > 2- Network > 3- System > 4- Logout > > - Main Menu, - Refresh, - Event Log > > > > ***************************************************************************************************************** > > Has anybody else had problem with the APC7921. I am going to assume > it's because the APC want the telnet session to type 'YES' to confirm. > > > > ------------------------------------------------------------------------ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From wcheng at redhat.com Wed Apr 25 19:33:22 2007 From: wcheng at redhat.com (Wendy Cheng) Date: Wed, 25 Apr 2007 15:33:22 -0400 Subject: [Linux-cluster] GFS: fh2dentry misses In-Reply-To: References: Message-ID: <462FAD02.2000409@redhat.com> Chad Slater wrote: > While troubleshooting some GFS performance issues, I came across this > when viewing the counters: > > fh2dentry misses 292928 > > What is fh2dentry misses? This symbol name is kind of mis-leading. It is actually a counter of "inode miss". With Linux NFS implementation, each nfs file handle has file inode embedded in it. When client sends file handle to the server, nfsd passes the file handle to GFS (or any filesystem). GFS will check to see whether the subject inode is still in the memory cache. If not, it increments this counter and re-do lookup. As any filesystem, lookup is normally expensive since it involves disk read. The more cache misses, the slower your performance will be. -- Wendy From sdake at redhat.com Wed Apr 25 20:53:24 2007 From: sdake at redhat.com (Steven Dake) Date: Wed, 25 Apr 2007 13:53:24 -0700 Subject: [Linux-cluster] startup problem: "cman not started: Can't bind to local cman socket /usr/sbin/cman_tool: aisexec daemon didn't start " In-Reply-To: References: Message-ID: <1177534404.8725.6.camel@shih.broked.org> After cman returns the "aisexec daemon didn't start" is the process "aisexec" running? If it isn't, there should be a core file in /var/run/openais I believe. Install the openais-debug package if using RPMS, or build a debug version of all of the infrastructure and use gdb to get a backtrace. gdb /usr/sbin/aisexec /var/run/openais/core.xxxx run the where command regards -steve On Wed, 2007-04-25 at 18:26 +0200, Brieseneck, Arne, VF-Group wrote: > it is high likely that anybody already knows this error and knows how > to fix it: > > ---snip--- > [root at box1 ~]# /etc/init.d/cman start > Starting cluster: > Loading modules... done > Mounting configfs... done > Starting ccsd... done > Starting cman... failed > cman not started: Can't bind to local cman socket /usr/sbin/cman_tool: > aisexec daemon didn't start > [FAILED] > [root at box1 ~]# > > ---snap--- > > the cluster does not start even what is written in the logfileif looks > OK for me: > > ---snip--- > > Apr 24 19:59:06 box1 ccsd[5129]: Starting ccsd 2.0.60: > Apr 24 19:59:06 box1 ccsd[5129]: Built: Jan 24 2007 15:31:03 > Apr 24 19:59:06 box1 ccsd[5129]: Copyright (C) Red Hat, Inc. 2004 All > rights reserved. > Apr 24 19:59:06 box1 ccsd[5129]: cluster.conf (cluster name = > alpha_cluster, version = 6) found. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] AIS Executive Service > RELEASE 'subrev 1204 version 0.80.1' > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Copyright (C) 2002-2006 > MontaVista Software, Inc and contributors. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Copyright (C) 2006 Red > Hat, Inc. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Using default multicast > address of 239.192.196.121 > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component > openais_cpg loaded. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service > handler 'openais cluster closed process group service v1.01' > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component > openais_cfg loaded. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service > handler 'openais configuration service' > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component > openais_msg loaded. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service > handler 'openais message service B.01.01' > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component > openais_lck loaded. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service > handler 'openais distributed locking service B.01.01' > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component > openais_evt loaded. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service > handler 'openais event service B.01.01' > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component > openais_ckpt loaded. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service > handler 'openais checkpoint service B.01.01' > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component > openais_amf loaded. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service > handler 'openais availability management framework B.01.01' > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component > openais_clm loaded. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service > handler 'openais cluster membership service B.01.01' > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component > openais_evs loaded. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service > handler 'openais extended virtual synchrony service' > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] openais component > openais_cman loaded. > Apr 24 19:59:08 box1 openais[5135]: [MAIN ] Registering service > handler 'openais CMAN membership service 2.01' > Apr 24 19:59:08 box1 openais[5070]: [TOTEM] entering GATHER state from > 12. > Apr 24 19:59:08 box1 openais[5070]: [TOTEM] Creating commit token > because I am the rep. > Apr 24 19:59:08 box1 openais[5070]: [TOTEM] Saving state aru 1e high > seq received 1e > Apr 24 19:59:08 box1 openais[5070]: [TOTEM] Storing new sequence id > for ring 304 > Apr 24 19:59:08 box1 openais[5070]: [TOTEM] entering COMMIT state. > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] entering RECOVERY state. > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] position [0] member > 192.168.50.194: > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] previous ring seq 768 rep > 192.168.50.194 > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] aru 1e high delivered 1e > received flag 0 > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] position [1] member > 192.168.50.195: > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] previous ring seq 768 rep > 192.168.50.194 > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] aru 1e high delivered 1e > received flag 0 > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] position [2] member > 192.168.50.196: > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] previous ring seq 768 rep > 192.168.50.194 > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] aru 1e high delivered 1e > received flag 0 > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] Did not need to originate > any messages in recovery. > Apr 24 19:59:09 box1 openais[5070]: [CLM ] CLM CONFIGURATION CHANGE > Apr 24 19:59:09 box1 openais[5070]: [CLM ] New Configuration: > Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.194) > Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.195) > Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.196) > Apr 24 19:59:09 box1 openais[5070]: [CLM ] Members Left: > Apr 24 19:59:09 box1 openais[5070]: [CLM ] Members Joined: > Apr 24 19:59:09 box1 openais[5070]: [SYNC ] This node is within the > primary component and will provide service. > Apr 24 19:59:09 box1 openais[5070]: [CLM ] CLM CONFIGURATION CHANGE > Apr 24 19:59:09 box1 openais[5070]: [CLM ] New Configuration: > Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.194) > Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.195) > Apr 24 19:59:09 box1 openais[5070]: [CLM ] r(0) ip(192.168.50.196) > Apr 24 19:59:09 box1 openais[5070]: [CLM ] Members Left: > Apr 24 19:59:09 box1 openais[5070]: [CLM ] Members Joined: > Apr 24 19:59:09 box1 openais[5070]: [SYNC ] This node is within the > primary component and will provide service. > Apr 24 19:59:09 box1 openais[5070]: [TOTEM] entering OPERATIONAL > state. > Apr 24 19:59:09 box1 openais[5070]: [CLM ] got nodejoin message > 192.168.50.195 > Apr 24 19:59:09 box1 openais[5070]: [CLM ] got nodejoin message > 192.168.50.196 > Apr 24 19:59:09 box1 openais[5070]: [CLM ] got nodejoin message > 192.168.50.194 > Apr 24 19:59:35 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 30 seconds. > Apr 24 20:00:05 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 60 seconds. > Apr 24 20:00:35 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 90 seconds. > Apr 24 20:01:05 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 120 seconds. > Apr 24 20:01:35 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 150 seconds. > Apr 24 20:02:05 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 180 seconds. > Apr 24 20:02:35 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 210 seconds. > Apr 24 20:03:05 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 240 seconds. > Apr 24 20:03:35 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 270 seconds. > Apr 24 20:04:05 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 300 seconds. > Apr 24 20:04:35 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 330 seconds. > Apr 24 20:05:05 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 360 seconds. > Apr 24 20:05:35 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 390 seconds. > Apr 24 20:06:05 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 420 seconds. > Apr 24 20:06:35 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 450 seconds. > Apr 24 20:07:05 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 480 seconds. > Apr 24 20:07:35 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 510 seconds. > Apr 24 20:08:05 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 540 seconds. > Apr 24 20:08:35 box1 ccsd[5129]: Unable to connect to cluster > infrastructure after 570 seconds. > Apr 24 20:09:01 box1 ccsd[5129]: Stopping ccsd, SIGTERM received. > > ---snap--- > > thats the /etc/cluster/cluster.conf: > > ---snip--- > [root at box1 ~]# cat /etc/cluster/cluster.conf > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---snap--- > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From Alain.Moulle at bull.net Thu Apr 26 06:34:52 2007 From: Alain.Moulle at bull.net (Alain Moulle) Date: Thu, 26 Apr 2007 08:34:52 +0200 Subject: [Linux-cluster] CS4 U4 / question about service monitoring stalled Message-ID: <4630480C.2090700@bull.net> Hi A question about the behavior of CS4 : suppose that the service configured in the cluster.conf does not return on the periodic status launch by CS4, for whatever reason. What is the behavior of CS4 , and eventually, is there smthg to do in CS4 configuration to avoid problems in this case ? Thanks Alain From pcaulfie at redhat.com Thu Apr 26 08:27:03 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Thu, 26 Apr 2007 09:27:03 +0100 Subject: [Linux-cluster] startup problem: "cman not started: Can't bind to local cman socket /usr/sbin/cman_tool: aisexec daemon didn't start " In-Reply-To: References: Message-ID: <46306257.1050509@redhat.com> Brieseneck, Arne, VF-Group wrote: > it is high likely that anybody already knows this error and knows how to > fix it: > > ---snip--- > [root at box1 ~]# /etc/init.d/cman start > Starting cluster: > Loading modules... done > Mounting configfs... done > Starting ccsd... done > Starting cman... failed > cman not started: Can't bind to local cman socket /usr/sbin/cman_tool: > aisexec daemon didn't start > [FAILED] > [root at box1 ~]# It means that cman could not create the local socket in /var/run for communication with the cluster clients. cman tries to create /var/run/cman_client & /var/run/cman_admin - things like cman_tool/groupd/ccsd etc talk to cman over this link. If it can't be created then its not much use. Check /var/run is writable and able to hold Unix domain sockets. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From jos at xos.nl Thu Apr 26 08:32:08 2007 From: jos at xos.nl (Jos Vos) Date: Thu, 26 Apr 2007 10:32:08 +0200 Subject: [Linux-cluster] CS4 U4 / question about service monitoring stalled In-Reply-To: <4630480C.2090700@bull.net>; from Alain.Moulle@bull.net on Thu, Apr 26, 2007 at 08:34:52AM +0200 References: <4630480C.2090700@bull.net> Message-ID: <20070426103208.A14253@xos037.xos.nl> On Thu, Apr 26, 2007 at 08:34:52AM +0200, Alain Moulle wrote: > suppose that the service configured in the cluster.conf > does not return on the periodic status launch by CS4, > for whatever reason. What is the behavior of CS4 , and > eventually, is there smthg to do in CS4 configuration > to avoid problems in this case ? The behavior in case of a failure can be configured (service attribute "recovery", value choices are "restart", "relocate" and "disable"). In case you don't care for the status or want some custom begavior, the best way is to write a custom service check script (or write a wrapper around an existing one), e.g. one that always succeeds when asked for the "status". -- -- Jos Vos -- X/OS Experts in Open Systems BV | Phone: +31 20 6938364 -- Amsterdam, The Netherlands | Fax: +31 20 6948204 From Arne.Brieseneck at vodafone.com Thu Apr 26 09:11:16 2007 From: Arne.Brieseneck at vodafone.com (Brieseneck, Arne, VF-Group) Date: Thu, 26 Apr 2007 11:11:16 +0200 Subject: [Linux-cluster] startup problem: "cman not started: Can't bindto local cman socket /usr/sbin/cman_tool: aisexec daemon didn't start " In-Reply-To: <46306257.1050509@redhat.com> Message-ID: SOLVED! This was the problem. Thanks a lot. The directories and links where there from time when it was running perfect I think. Anyhow, there is always space for improvement. Do you think we should change the error message to make it more clear? May be we can update the FAQ list as well. Thanks a lot, Arne -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Patrick Caulfield Sent: Donnerstag, 26. April 2007 10:27 To: linux clustering Subject: Re: [Linux-cluster] startup problem: "cman not started: Can't bindto local cman socket /usr/sbin/cman_tool: aisexec daemon didn't start " Brieseneck, Arne, VF-Group wrote: > it is high likely that anybody already knows this error and knows how > to fix it: > > ---snip--- > [root at box1 ~]# /etc/init.d/cman start > Starting cluster: > Loading modules... done > Mounting configfs... done > Starting ccsd... done > Starting cman... failed > cman not started: Can't bind to local cman socket /usr/sbin/cman_tool: > aisexec daemon didn't start > [FAILED] > [root at box1 ~]# It means that cman could not create the local socket in /var/run for communication with the cluster clients. cman tries to create /var/run/cman_client & /var/run/cman_admin - things like cman_tool/groupd/ccsd etc talk to cman over this link. If it can't be created then its not much use. Check /var/run is writable and able to hold Unix domain sockets. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From ramon at vanalteren.nl Thu Apr 26 10:58:59 2007 From: ramon at vanalteren.nl (Ramon van Alteren) Date: Thu, 26 Apr 2007 12:58:59 +0200 Subject: [Linux-cluster] clvm problems with multiple clusters in same vlan Message-ID: <463085F3.2050901@vanalteren.nl> Hi list, We're having problems with running multiple clusters in the same vlan. For some reason clvmd refuses to start properly. We think (but cannot completely confirm) that clvm is informed by the clvmd on the other cluster about volumegroups which it can't reach because it can't reach the underlying storage / physical volumes. The sympotoms are: node fails on reboot with hung clvmd daemon no output from clvmd even when started with debug (clvmd -d) versions: Cluster LVM daemon version: 2.02.09 (2006-08-17) Protocol version: 0.2.1 cman_tool 1.03.00 (built Oct 19 2006 10:58:21) Copyright (C) Red Hat, Inc. 2004 All rights reserved. ccsd 1.03.00 (built Oct 19 2006 10:58:17) Copyright (C) Red Hat, Inc. 2004 All rights reserved. The node does join the cman cluster group and the defined fence group. If the second cluster is shutdown nodes reboot without problems. We're considering to force ccs to use different ports [1] for communication as a workaround, any other options or suggestions would be highly appreciated. Best regards, Ramon van Alteren [1] man ccsd: -P : You have the option of specifying the port numbers used by ccsd. The port identifier is either: b, c, or f. "b" is the port which ccsd attempts to communicate with ccsd processes on other machines, via broadcast/multicast, to obtain or validate its config file (cluster.conf). This is known as the backend port. "c" is the base port number of two consecutive ports used by ccsd processes to communicate cluster membership information. This is known as the cluster base port. "f" is the port number that listens for information requests from the CCS library (or programs using it). This is known as the frontend port. So, to change the frontend port one might specify -P f:60000. From swplotner at amherst.edu Thu Apr 26 12:50:56 2007 From: swplotner at amherst.edu (Steffen Plotner) Date: Thu, 26 Apr 2007 08:50:56 -0400 Subject: [Linux-cluster] clvm problems with multiple clusters in same vlan In-Reply-To: <463085F3.2050901@vanalteren.nl> Message-ID: <0456A130F613AD459887FF012652963F0194CA8E@mail7.amherst.edu> > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ramon > van Alteren > Sent: Thursday, April 26, 2007 6:59 AM > To: linux clustering > Cc: System Administration > Subject: [Linux-cluster] clvm problems with multiple clusters > in same vlan > > Hi list, > > We're having problems with running multiple clusters in the same vlan. > For some reason clvmd refuses to start properly. I am currently running 2 redhat clusters in the same VLAN using the same IP range with different netmasks however (i.e. each cluster in its own subnet, both on the same VLAN). So far, I have not seen any clvmd issues. Steffen > > We think (but cannot completely confirm) that clvm is > informed by the clvmd on the other cluster about volumegroups > which it can't reach because it can't reach the underlying > storage / physical volumes. > > The sympotoms are: > > node fails on reboot with hung clvmd daemon no output from > clvmd even when started with debug (clvmd -d) > > versions: > Cluster LVM daemon version: 2.02.09 (2006-08-17) > Protocol version: 0.2.1 > > cman_tool 1.03.00 (built Oct 19 2006 10:58:21) Copyright (C) > Red Hat, Inc. 2004 All rights reserved. > > ccsd 1.03.00 (built Oct 19 2006 10:58:17) Copyright (C) Red > Hat, Inc. 2004 All rights reserved. > > The node does join the cman cluster group and the defined fence group. > If the second cluster is shutdown nodes reboot without problems. > > We're considering to force ccs to use different ports [1] for > communication as a workaround, any other options or > suggestions would be highly appreciated. > > Best regards, > > Ramon van Alteren > > > [1] man ccsd: > > -P : > You have the option of specifying the port > numbers used by ccsd. The port identifier is either: > b, c, or f. "b" is the port which ccsd > attempts to > communicate with ccsd processes on other > machines, via broadcast/multicast, to obtain or > validate its config file (cluster.conf). This is > known as the backend port. "c" is the base > port number of two consecutive ports used by ccsd > processes to communicate cluster membership > information. > This is known as the cluster base port. > "f" is the port number that listens for > information requests from the CCS library (or programs > using it). This is known as the frontend port. > > So, to change the frontend port one might > specify -P f:60000. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From pcaulfie at redhat.com Thu Apr 26 14:17:51 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Thu, 26 Apr 2007 15:17:51 +0100 Subject: [Linux-cluster] clvm problems with multiple clusters in same vlan In-Reply-To: <463085F3.2050901@vanalteren.nl> References: <463085F3.2050901@vanalteren.nl> Message-ID: <4630B48F.8090202@redhat.com> Ramon van Alteren wrote: > Hi list, > > We're having problems with running multiple clusters in the same vlan. > For some reason clvmd refuses to start properly. > > We think (but cannot completely confirm) that clvm is informed by the > clvmd on the other cluster about volumegroups > which it can't reach because it can't reach the underlying storage / > physical volumes. This sounds unlikely. clvmd can only talk to other clvmds in the same cluster - it has no way of communicating outside the cluster on a cman system. > The sympotoms are: > > node fails on reboot with hung clvmd daemon > no output from clvmd even when started with debug (clvmd -d) > > versions: > Cluster LVM daemon version: 2.02.09 (2006-08-17) > Protocol version: 0.2.1 > > cman_tool 1.03.00 (built Oct 19 2006 10:58:21) > Copyright (C) Red Hat, Inc. 2004 All rights reserved. > > ccsd 1.03.00 (built Oct 19 2006 10:58:17) > Copyright (C) Red Hat, Inc. 2004 All rights reserved. > > The node does join the cman cluster group and the defined fence group. > If the second cluster is shutdown nodes reboot without problems. Is there a clvmd lockspace? if not it might be something simple like the cluster not having quorum. > We're considering to force ccs to use different ports [1] for > communication as a workaround, any other options or suggestions would be > highly appreciated. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From ramon at vanalteren.nl Thu Apr 26 14:52:07 2007 From: ramon at vanalteren.nl (Ramon van Alteren) Date: Thu, 26 Apr 2007 16:52:07 +0200 Subject: [Linux-cluster] clvm problems with multiple clusters in same vlan In-Reply-To: <4630B48F.8090202@redhat.com> References: <463085F3.2050901@vanalteren.nl> <4630B48F.8090202@redhat.com> Message-ID: <4630BC97.8070008@vanalteren.nl> Hi Patrick, Patrick Caulfield wrote: > This sounds unlikely. clvmd can only talk to other clvmds in the same cluster - > it has no way of communicating outside the cluster on a cman system. > clvmd picks up on volumegroups belonging to the other cluster. I was under the understanding that clvmd communicated through ccsd libraries with other cluster members and should never pick up any clustered volume groups outside the cluster. Yet this is what we are seeing. >> The sympotoms are: >> >> node fails on reboot with hung clvmd daemon >> no output from clvmd even when started with debug (clvmd -d) >> >> versions: >> Cluster LVM daemon version: 2.02.09 (2006-08-17) >> Protocol version: 0.2.1 >> >> cman_tool 1.03.00 (built Oct 19 2006 10:58:21) >> Copyright (C) Red Hat, Inc. 2004 All rights reserved. >> >> ccsd 1.03.00 (built Oct 19 2006 10:58:17) >> Copyright (C) Red Hat, Inc. 2004 All rights reserved. >> >> The node does join the cman cluster group and the defined fence group. >> If the second cluster is shutdown nodes reboot without problems. >> > > Is there a clvmd lockspace? if not it might be something simple like the cluster > not having quorum. > Nope, both clusters have quorum, one is a 5 node cluster with one node down. The other is a 3-node cluster with no nodes down. How do I check for clvmd lockspace ? Is there a utility or proc / sys entry where I can find out the clvm lockspace ? Regards, Ramon From pcaulfie at redhat.com Thu Apr 26 15:04:20 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Thu, 26 Apr 2007 16:04:20 +0100 Subject: [Linux-cluster] clvm problems with multiple clusters in same vlan In-Reply-To: <4630BC97.8070008@vanalteren.nl> References: <463085F3.2050901@vanalteren.nl> <4630B48F.8090202@redhat.com> <4630BC97.8070008@vanalteren.nl> Message-ID: <4630BF74.80103@redhat.com> Ramon van Alterer wrote: > Nope, both clusters have quorum, one is a 5 node cluster with one node > down. > The other is a 3-node cluster with no nodes down. > > How do I check for clvmd lockspace ? Is there a utility or proc / sys > entry where I can find out the clvm lockspace ? > cman_tool services will show you the lockspaces (amongst other things) If there isn't one then try stracing "clvmd -d" to see where it hangs. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From ramon at vanalteren.nl Thu Apr 26 16:23:08 2007 From: ramon at vanalteren.nl (Ramon van Alteren) Date: Thu, 26 Apr 2007 18:23:08 +0200 Subject: [Linux-cluster] clvm problems with multiple clusters in same vlan In-Reply-To: <4630BF74.80103@redhat.com> References: <463085F3.2050901@vanalteren.nl> <4630B48F.8090202@redhat.com> <4630BC97.8070008@vanalteren.nl> <4630BF74.80103@redhat.com> Message-ID: <4630D1EC.5030101@vanalteren.nl> Patrick Caulfield wrote: > cman_tool services will show you the lockspaces (amongst other things) > > It's showing multiple lockspaces: DLM Lock Space: "clvmd" 2 3 run - [2 3 5 1 4] This would be the clvmd lockspace then ? The once below correspond to the 4 gfs logical volumes we mount from this particular shared storage DLM Lock Space: "e1.1-lv" 3 4 run - [2 3 5 1 4] DLM Lock Space: "e1.2" 5 6 run - [2 3 5 1 4] DLM Lock Space: "e2.1-lv" 7 8 run - [2 3 5 1 4] DLM Lock Space: "e2.2" 9 10 run - [2 3 5 1 4] I suspect that the other cluster uses the same clvmd lockspace "clvmd" How do we set a different clvm lockspace on the other, currently disabled cluster ? Regards, Ramon From wferi at niif.hu Thu Apr 26 16:36:02 2007 From: wferi at niif.hu (Ferenc Wagner) Date: Thu, 26 Apr 2007 18:36:02 +0200 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <20070425165553.GC9891@redhat.com> (David Teigland's message of "Wed, 25 Apr 2007 11:55:53 -0500") References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> <87wt06djk7.fsf@tac.ki.iif.hu> <20070423211717.GA22147@redhat.com> <20070424193600.GB11156@redhat.com> <87irbktpzq.fsf@tac.ki.iif.hu> <20070425165553.GC9891@redhat.com> Message-ID: <87bqhbdrf1.fsf@tac.ki.iif.hu> David Teigland writes: > Thanks, some of these numbers look odd to me; I'll need run the test on my > own rhel4 cluster to understand them better. I'm working with three nodes: 1, 2 and 3. Looks like the mount by 3 makes a big difference. When the filesystem is mounted by 1 and 2 only, my test runs much faster. Filesystem mounted by 3 alone is also fast. But 3 doesn't seem to cooperate with anyone else with reasonable performance. If I mount the filesystem on all three nodes and run the test on 1, the network traffic of 2 and 3 is rather unbalanced: tcpdump receives 19566 packets on 2 and 29181 on 3. It's all 21064/tcp traffic, I can provide detailed data if that seems useful. > In the end, though, I think you'll get the best performance by using > flocks under the new cluster infrastructure and kernel. Can you point me to the relevant documentation and version numbers for upgrading to the new cluster infrastructure? Unfortunately it's not prepackaged, which means much more maintenance work, but I'm willing to test if it helps enough to justify that. -- Regards, Feri. From teigland at redhat.com Thu Apr 26 17:08:44 2007 From: teigland at redhat.com (David Teigland) Date: Thu, 26 Apr 2007 12:08:44 -0500 Subject: [Linux-cluster] Slowness above 500 RRDs In-Reply-To: <87bqhbdrf1.fsf@tac.ki.iif.hu> References: <87648r6hdi.fsf@tac.ki.iif.hu> <87ps6tl685.fsf@szonett.ki.iif.hu> <20070328162726.GF22230@redhat.com> <20070328163850.GG22230@redhat.com> <87wt06djk7.fsf@tac.ki.iif.hu> <20070423211717.GA22147@redhat.com> <20070424193600.GB11156@redhat.com> <87irbktpzq.fsf@tac.ki.iif.hu> <20070425165553.GC9891@redhat.com> <87bqhbdrf1.fsf@tac.ki.iif.hu> Message-ID: <20070426170844.GA14224@redhat.com> On Thu, Apr 26, 2007 at 06:36:02PM +0200, Ferenc Wagner wrote: > David Teigland writes: > > > Thanks, some of these numbers look odd to me; I'll need run the test on my > > own rhel4 cluster to understand them better. > > I'm working with three nodes: 1, 2 and 3. Looks like the mount by 3 > makes a big difference. When the filesystem is mounted by 1 and 2 > only, my test runs much faster. Filesystem mounted by 3 alone is also > fast. But 3 doesn't seem to cooperate with anyone else with > reasonable performance. > > If I mount the filesystem on all three nodes and run the test on 1, > the network traffic of 2 and 3 is rather unbalanced: tcpdump receives > 19566 packets on 2 and 29181 on 3. It's all 21064/tcp traffic, I can > provide detailed data if that seems useful. It sounds like your tests are mixing the effects of the flocks/plocks with the effects of gfs's own internal file locking. If you want to test and compare flock/plock performance you need to make sure that gfs's internal dlm locks are always mastered on the same node (either locally, in which case it'll be fast, or remotely in which case it'll be slow). The first node to use a lock will be the master of it. > > In the end, though, I think you'll get the best performance by using > > flocks under the new cluster infrastructure and kernel. > > Can you point me to the relevant documentation and version numbers for > upgrading to the new cluster infrastructure? Unfortunately it's not > prepackaged, which means much more maintenance work, but I'm willing > to test if it helps enough to justify that. It depends on what kind of packaging you want... there's RHEL5 and its clones, of course. There's the cluster-2.00.00.tar.gz release. You can look at this for usage: http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/usage.txt?cvsroot=cluster Dave From lhh at redhat.com Thu Apr 26 19:41:55 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 26 Apr 2007 15:41:55 -0400 Subject: [Linux-cluster] CS4 U4 / question about service monitoring stalled In-Reply-To: <4630480C.2090700@bull.net> References: <4630480C.2090700@bull.net> Message-ID: <20070426194154.GD20650@redhat.com> On Thu, Apr 26, 2007 at 08:34:52AM +0200, Alain Moulle wrote: > Hi > > A question about the behavior of CS4 : > > suppose that the service configured in the cluster.conf > does not return on the periodic status launch by CS4, > for whatever reason. What is the behavior of CS4 , and > eventually, is there smthg to do in CS4 configuration > to avoid problems in this case ? There's nothing it does to recover that. It's not a "hard thing" to do, but as you're doing status queries - if your status queries are hanging, something is *wrong* -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Thu Apr 26 20:06:10 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 26 Apr 2007 16:06:10 -0400 Subject: [Linux-cluster] Piranha question: Running client on a real serverhangs In-Reply-To: References: <20070425150317.GN4897@redhat.com> Message-ID: <20070426200610.GE20650@redhat.com> On Wed, Apr 25, 2007 at 10:56:24AM -0500, Brooks, Jody wrote: > Thanks for the links... working my way through them now ... > > For info: I am using the redirect bits like in "method #2". > > In fact, that's all I'm doing on the real servers as I have not assigned > the VIP to the lo interface on the real servers as seems to be discussed > in LVS-DR.html (haven't read all that yet but that appears to be implied > as I've read that in other places as well). This VIP on lo doesn't > appear to be critical as my service works fine when the client is on a > non-real-server/non-director host. Am I missing something here? Seems > like the VIP on lo would be used if you weren't doing the iptables > redirect (i.e., method 2)... right? wrong? That's right; the redirect is a completely different solution to the "ARP Problem" that LVS-DR.html talks about. You need one or the other, but not both. What I was saying is that if you did #1 in the doc (or configured the IP on lo as per LVS-DR.html), the connection would always work - it just wouldn't be load balanced. That would be a way to get rid of part of your failure cases, if elimination of the failure was more important than being optimally load balanced. > I will try to incorporate some of the pieces from the LVS-DR.html ideas > and see if that gets me any further. > For the record, our need of this is that one of our services is also a > consumer of other of our services, all of which should be load-balanced > across the same cluster as they're related. If you can't get it working with LVS, you could always try something like RRDNS to do something similar. Unfortunately, I have not tried this configuration at all - so my insight will naturally be rather limited. :( -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Thu Apr 26 20:07:23 2007 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 26 Apr 2007 16:07:23 -0400 Subject: [Linux-cluster] Maximum restarts and maximum false restarts In-Reply-To: <6cfbd1b40704250443j78f92dbep697a0b397a53a2a4@mail.gmail.com> References: <6cfbd1b40704250443j78f92dbep697a0b397a53a2a4@mail.gmail.com> Message-ID: <20070426200723.GF20650@redhat.com> On Wed, Apr 25, 2007 at 01:43:22PM +0200, Tomas Hoger wrote: > Hi! > > Can anyone provide some additional information regarding following FAQ > entry? > > http://sources.redhat.com/cluster/faq.html#rgm_restarts Hmm... > It says: "If either of those values are exceeded, the service is > relocated rather than restarted locally." What are "those values" and > how can they be tweaked? Bug in faq... there's no max restarts/max_false_starts in linux-cluster; it was a clumanager thing. One of two things can happen: * reimplement them ;) * fix the faq. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From garylua at singnet.com.sg Fri Apr 27 05:21:29 2007 From: garylua at singnet.com.sg (garylua at singnet.com.sg) Date: Fri, 27 Apr 2007 13:21:29 +0800 (SGT) Subject: [Linux-cluster] shared storage configuration on a 2 node-cluster Message-ID: <1177651289.46318859d3c24@discus.singnet.com.sg> Hi, I'm currently running a cluster (RHEL4, Cluster Suite Update 3) with 2 nodes. There is a virtual ip and a shared storage resources defined, together with other scripts that are supposed to be executed. During failover, I discovered that the time taken is a little long (around 25 seconds). I realised that the "bottleneck" of this long failover time is contributed by the unmounting/mounting of the shared storage filesystem and the failover of the virtual ip to the other node. I'm just wondering how can i make the shared storage in such a way that BOTH nodes can access the shared storage filesystem, but only the active node has the write privilege? So that there is no risk of data corruption, and yet the failover time can be reduced (hopefully?). All in all, I'm trying to reduce the time of the failover to as short a period as possible. I've already change the status monitoring (heartbeat) interval to 5 seconds. And on top of the shared storage and virtual ip, i have 4 scripts that need to be failed over. I configured the cluster in such a way that when any of the scripts fails, the rest of the scripts will fail over too. As such, there are a lot of dependencies among the resources and I'm trying to reduce the time of the failover to maybe about 10 seconds, if possible. Thanks for any help in advance. From pcaulfie at redhat.com Fri Apr 27 08:01:14 2007 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Fri, 27 Apr 2007 09:01:14 +0100 Subject: [Linux-cluster] clvm problems with multiple clusters in same vlan In-Reply-To: <4630D1EC.5030101@vanalteren.nl> References: <463085F3.2050901@vanalteren.nl> <4630B48F.8090202@redhat.com> <4630BC97.8070008@vanalteren.nl> <4630BF74.80103@redhat.com> <4630D1EC.5030101@vanalteren.nl> Message-ID: <4631ADCA.9080009@redhat.com> Ramon van Alteren wrote: > Patrick Caulfield wrote: >> cman_tool services will show you the lockspaces (amongst other things) >> >> > It's showing multiple lockspaces: > > DLM Lock Space: "clvmd" 2 3 run - > [2 3 5 1 4] > > This would be the clvmd lockspace then ? yes it is. > The once below correspond to the 4 gfs logical volumes we mount from > this particular shared storage > > DLM Lock Space: "e1.1-lv" 3 4 run - > [2 3 5 1 4] > > DLM Lock Space: "e1.2" 5 6 run - > [2 3 5 1 4] > > DLM Lock Space: "e2.1-lv" 7 8 run - > [2 3 5 1 4] > > DLM Lock Space: "e2.2" 9 10 run - > [2 3 5 1 4] > > I suspect that the other cluster uses the same clvmd lockspace "clvmd" > How do we set a different clvm lockspace on the other, currently > disabled cluster ? It will be a different clvmd lockspace. Different cluster share nothing between them because they don't intercommunicate. so a lockspace called 'clvmd' on one cluster is completely independent of every other lockspace called 'clvmd' on other clusters. -- Patrick Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 ITE, UK. Registered in England and Wales under Company Registration No. 3798903 From pgharios at gmail.com Fri Apr 27 08:50:31 2007 From: pgharios at gmail.com (Patrick Gharios) Date: Fri, 27 Apr 2007 11:50:31 +0300 Subject: [Linux-cluster] Serial Heartbeat. Message-ID: <000001c788a9$36205e50$3b6410ac@snoopy> Hello, Does anyone have any experience on how to configure serial heartbeat between multiple machines using a terminal server? The scenario is that I have 3 backup servers to serve 7 live servers and I need to connect each of these backup servers to the live systems, so my optimal is to use a terminal server to do so. Thank you, Pat. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mathieu.avila at seanodes.com Fri Apr 27 09:00:41 2007 From: mathieu.avila at seanodes.com (Mathieu Avila) Date: Fri, 27 Apr 2007 11:00:41 +0200 Subject: [Linux-cluster] I/O Error management in GFS Message-ID: <20070427110041.0cd462cd@mathieu.toulouse> Hello all, >From what i understand of the GFS1 source code, I/O error are not managed : when an I/O error happens, either it exits the locking protocol's cluster (Gulm or CMAN), or sometimes it asserts/panics. Anyway, most of the time, the node that got an I/O error must be rebooted (file system layer is instable) and the device must be checked and the file system must be fsck'ed. Are there any plans for a cleaner management of I/O errors in GFS1, like, say, remount in R/O mode with -EIO returned to apps, or even better, advanced features like relocation mechanisms ? Is it planned in GFS2 ? Thanks, -- Mathieu From tomas.hoger at gmail.com Fri Apr 27 09:58:05 2007 From: tomas.hoger at gmail.com (Tomas Hoger) Date: Fri, 27 Apr 2007 11:58:05 +0200 Subject: [Linux-cluster] Maximum restarts and maximum false restarts In-Reply-To: <20070426200723.GF20650@redhat.com> References: <6cfbd1b40704250443j78f92dbep697a0b397a53a2a4@mail.gmail.com> <20070426200723.GF20650@redhat.com> Message-ID: <6cfbd1b40704270258s7c45c0eeyb6833b2cd1aa707b@mail.gmail.com> Hi Lon! On 4/26/07, Lon Hohberger wrote: > > It says: "If either of those values are exceeded, the service is > > relocated rather than restarted locally." What are "those values" and > > how can they be tweaked? > > Bug in faq... there's no max restarts/max_false_starts in linux-cluster; > it was a clumanager thing. > > One of two things can happen: > * reimplement them ;) > * fix the faq. Thanks for clarification! That pretty much explains, why I was not able to find any traces of that in documentation, rgmanager sources and cluster.conf schema reference ;). th. From lhh at redhat.com Fri Apr 27 16:43:06 2007 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 27 Apr 2007 12:43:06 -0400 Subject: [Linux-cluster] shared storage configuration on a 2 node-cluster In-Reply-To: <1177651289.46318859d3c24@discus.singnet.com.sg> References: <1177651289.46318859d3c24@discus.singnet.com.sg> Message-ID: <20070427164306.GG20650@redhat.com> On Fri, Apr 27, 2007 at 01:21:29PM +0800, garylua at singnet.com.sg wrote: > Hi, > > I'm currently running a cluster (RHEL4, Cluster Suite Update 3) with 2 nodes. There is a virtual ip and a shared storage resources defined, together with other scripts that are supposed to be executed. > > During failover, I discovered that the time taken is a little long (around 25 seconds). I realised that the "bottleneck" of this long failover time is contributed by the unmounting/mounting of the shared storage filesystem and the failover of the virtual ip to the other node. Each IP address teardown adds 10 seconds. > I'm just wondering how can i make the shared storage in such a way that BOTH nodes can access the shared storage filesystem, but only the active node has the write privilege? So that there is no risk of data corruption, and yet the failover time can be reduced (hopefully?). I don't think you can do this safely. You could use GFS if you wanted to. > All in all, I'm trying to reduce the time of the failover to as short a period as possible. I've already change the status monitoring (heartbeat) interval to 5 seconds. And on top of the shared storage and virtual ip, i have 4 scripts that need to be failed over. I configured the cluster in such a way that when any of the scripts fails, the rest of the scripts will fail over too. If you're not doing NFS as part of your service (e.g. with the "nfsexport/nfsclient setup"), kill the 'sleep 10' in the /usr/share/cluster/ip.sh script. That will speed things up a bit. > As such, there are a lot of dependencies among the resources and I'm trying to reduce the time of the failover to maybe about 10 seconds, if possible. -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Fri Apr 27 16:44:43 2007 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 27 Apr 2007 12:44:43 -0400 Subject: [Linux-cluster] Serial Heartbeat. In-Reply-To: <000001c788a9$36205e50$3b6410ac@snoopy> References: <000001c788a9$36205e50$3b6410ac@snoopy> Message-ID: <20070427164442.GH20650@redhat.com> On Fri, Apr 27, 2007 at 11:50:31AM +0300, Patrick Gharios wrote: > Hello, > > Does anyone have any experience on how to configure serial heartbeat between > multiple machines using a terminal server? Terminal servers aren't network hubs...? > The scenario is that I have 3 backup servers to serve 7 live servers and I > need to connect each of these backup servers to the live systems, so my > optimal is to use a terminal server to do so. Why is that more optimal than using maybe a private network switch? Maybe I don't understand your configuration well enough... Could you expand on it some? -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Fri Apr 27 16:47:52 2007 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 27 Apr 2007 12:47:52 -0400 Subject: [Linux-cluster] Maximum restarts and maximum false restarts In-Reply-To: <6cfbd1b40704270258s7c45c0eeyb6833b2cd1aa707b@mail.gmail.com> References: <6cfbd1b40704250443j78f92dbep697a0b397a53a2a4@mail.gmail.com> <20070426200723.GF20650@redhat.com> <6cfbd1b40704270258s7c45c0eeyb6833b2cd1aa707b@mail.gmail.com> Message-ID: <20070427164751.GI20650@redhat.com> On Fri, Apr 27, 2007 at 11:58:05AM +0200, Tomas Hoger wrote: > Hi Lon! > > On 4/26/07, Lon Hohberger wrote: > >> It says: "If either of those values are exceeded, the service is > >> relocated rather than restarted locally." What are "those values" and > >> how can they be tweaked? > > > >Bug in faq... there's no max restarts/max_false_starts in linux-cluster; > >it was a clumanager thing. > > > >One of two things can happen: > >* reimplement them ;) > >* fix the faq. > > Thanks for clarification! That pretty much explains, why I was not > able to find any traces of that in documentation, rgmanager sources > and cluster.conf schema reference ;). If you file a bugzilla about them, we can see how much it'll be worth it to reintroduce them. max restarts used to mean: * If the service is restarted X times, fail over A false start used to mean: * A service 'start' phase succeeds, but the first status check fails. Max false starts was then: * If the service false-started X times, fail over It might be difficult to do a service-wide max-false-starts because individual resources are checked at different times. In RHCS3, everything was checked at the same time all the time. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From teigland at redhat.com Fri Apr 27 17:03:47 2007 From: teigland at redhat.com (David Teigland) Date: Fri, 27 Apr 2007 12:03:47 -0500 Subject: [Linux-cluster] I/O Error management in GFS In-Reply-To: <20070427110041.0cd462cd@mathieu.toulouse> References: <20070427110041.0cd462cd@mathieu.toulouse> Message-ID: <20070427170347.GB31211@redhat.com> On Fri, Apr 27, 2007 at 11:00:41AM +0200, Mathieu Avila wrote: > Hello all, > > >From what i understand of the GFS1 source code, I/O error are not > managed : when an I/O error happens, either it exits the locking > protocol's cluster (Gulm or CMAN), or sometimes it asserts/panics. > > Anyway, most of the time, the node that got an I/O error must be > rebooted (file system layer is instable) and the device must be checked > and the file system must be fsck'ed. > > Are there any plans for a cleaner management of I/O errors in GFS1, > like, say, remount in R/O mode with -EIO returned to apps, or even > better, advanced features like relocation mechanisms ? Is it planned in > GFS2 ? You've got very close to what you're asking for with the "withdraw" feature which has existed in gfs1 since rhel4. When gfs detects an io error, it does a "withdraw" on that fs, which means shutting it down: returning EIO to anything accessing it, telling other nodes to do journal recovery for it, dropping all global locks that were held, then you can unmount the withdrawn fs. It's mainly about getting the node with the errors out of the way of other nodes so the others can continue. It also allows you to shut down and reboot the node experiencing errors in a controlled fashion. Dave From lhh at redhat.com Fri Apr 27 18:10:54 2007 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 27 Apr 2007 14:10:54 -0400 Subject: [Linux-cluster] [RFC][PATCH] Add ability to freeze a service. In-Reply-To: <1177527128.14525.15.camel@localhost> References: <1177319722.3870.29.camel@localhost> <20070423200249.GB4897@redhat.com> <1177527128.14525.15.camel@localhost> Message-ID: <20070427181054.GA32119@redhat.com> On Wed, Apr 25, 2007 at 08:52:08PM +0200, Simone Gotti wrote: > > (3.a) Warning: The addition of rs_flags to rg_state_t will break > > compatibility with existing software (which is fine in -HEAD). It > > would be easier to cope with this change (that is- make upgrades from > > old->new work) if you added the rs_flags field after the > > rs_transition field. > > > I tried also to fix (1) (I hope). > In the meantime I fixed (like you reported on IRC) the missing changes > to clustat.c:xml_rg_state and enhanced a little the default clustat > output adding a "Flags" column. > I added a parameter "separator" to rg_flags_str so it can be defined. > For example in the default clustat I'm using a comma separated list > while in the xml output I'm using a space as separator (dunno if this is > right to do with xml). Patch applied. -- Lon Hohberger - Software Engineer - Red Hat, Inc. From jvantuyl at engineyard.com Sun Apr 29 14:16:23 2007 From: jvantuyl at engineyard.com (Jayson Vantuyl) Date: Sun, 29 Apr 2007 09:16:23 -0500 Subject: [Linux-cluster] CMan Build Issue Message-ID: <40E03DB0-735D-4368-96AC-A72C101B51A4@engineyard.com> Trying to build cman from cluster relase 1.03.00. Also has same error with 1.04.00. Building against Xen-3.0.5 patched 2.6.18. I get this: CC [M] /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/ cluster-1.03.00/cman-kernel/src/proc.o /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/cluster-1.03.00/ cman-kernel/src/cnxman.c: In function 'do_ioctl_join_cluster': /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/cluster-1.03.00/ cman-kernel/src/cnxman.c:1751: warning: implicit declaration of function 'init_utsname' /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/cluster-1.03.00/ cman-kernel/src/cnxman.c:1751: error: invalid type argument of '->' make[3]: *** [/var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/ cluster-1.03.00/cman-kernel/src/cnxman.o] Error 1 make[3]: *** Waiting for unfinished jobs.... make[2]: *** [_module_/var/tmp/portage/sys-cluster/cman- kernel-1.03.00/work/cluster-1.03.00/cman-kernel/src] Error 2 make[2]: Leaving directory `/usr/src/linux-2.6.18-xenU' make[1]: *** [all] Error 2 make[1]: Leaving directory `/var/tmp/portage/sys-cluster/cman- kernel-1.03.00/work/cluster-1.03.00/cman-kernel/src' make: *** [all] Error 2 Any ideas? -- Jayson Vantuyl Systems Architect Engine Yard jvantuyl at engineyard.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpeterso at redhat.com Mon Apr 30 02:59:09 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Sun, 29 Apr 2007 21:59:09 -0500 Subject: [Linux-cluster] CMan Build Issue In-Reply-To: <40E03DB0-735D-4368-96AC-A72C101B51A4@engineyard.com> References: <40E03DB0-735D-4368-96AC-A72C101B51A4@engineyard.com> Message-ID: <46355B7D.3080700@redhat.com> Jayson Vantuyl wrote: > Trying to build cman from cluster relase 1.03.00. Also has same error > with 1.04.00. Building against Xen-3.0.5 patched 2.6.18. I get this: > > CC [M] > /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/cluster-1.03.00/cman-kernel/src/proc.o > > /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/cluster-1.03.00/cman-kernel/src/cnxman.c: > In function 'do_ioctl_join_cluster': > /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/cluster-1.03.00/cman-kernel/src/cnxman.c:1751: > warning: implicit declaration of function 'init_utsname' > /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/cluster-1.03.00/cman-kernel/src/cnxman.c:1751: > error: invalid type argument of '->' > > Any ideas? > > --Jayson Vantuyl > Systems Architect > Engine Yard > jvantuyl at engineyard.com Hi Jayson, Back in February, I updated the STABLE branch of cvs from the older kernels up to the newer ones. That's probably why it doesn't compile in 2.6.18, but should in the 2.6.20 and 21 series. This particular change to cnxman.c can be seen here: http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/cman-kernel/src/Attic/cnxman.c.diff?r1=1.42.2.12.4.1.2.14&r2=1.42.2.12.4.1.2.15&cvsroot=cluster&f=h You can always revert that change if you need to stick with the older kernel. However, there were a bunch of things changed, and they might also give you problems with the older kernel. For a complete list of changes I made, please see: https://www.redhat.com/archives/cluster-devel/2007-February/msg00115.html Regards, Bob Peterson Red Hat Cluster Suite From cnguyen.ext at orange-ftgroup.com Mon Apr 30 09:11:29 2007 From: cnguyen.ext at orange-ftgroup.com (NGUYEN Can Ext ROSI/DPS) Date: Mon, 30 Apr 2007 11:11:29 +0200 Subject: [Linux-cluster] service starts up problem Message-ID: <1177924289.2858.15.camel@ngua> Dear community, Could someone help me please ? I get a starting service problem that seems the same problem than the other one ... I tried to resolve the problem by following the experts advices in the forum ... but still not ok here you are my problem : When i start /etc/init.d/crond for example (manually), it works fine, but when i try to start it by clurgmgrd, it's all the time in recovery (disabled, relocate or restart the service). Accordingly to the experts advices, i have to return 0 for starting script even after a failed service stop .... But still not start my script by clurgmgrd Could someone help me please ? for example : give me a starting script that works or show me where i can find patch for ... /etc/init.d/functions ?? I used : RHEL4, kernel-smp-2.6.9-42.0.3 rgmanager-1.9.54-1 cman-1.0.11-0 ccs-1.0.7-0 etc ... thank you very much guys Can Nguyen ********************************* This message and any attachments (the "message") are confidential and intended solely for the addressees. Any unauthorised use or dissemination is prohibited. Messages are susceptible to alteration. France Telecom Group shall not be liable for the message if altered, changed or falsified. If you are not the intended addressee of this message, please cancel it immediately and inform the sender. ******************************** From jbe at webde.de Mon Apr 30 09:52:03 2007 From: jbe at webde.de (Jens Beyer) Date: Mon, 30 Apr 2007 11:52:03 +0200 Subject: [Linux-cluster] BUG: "NULL pointer dereference" in dlm_recv Message-ID: <20070430095203.GA6717@pcnocjb.dlan.cinetic.de> Hi, I am currently testing an GFS testcluster (2 Nodes) based on cluster-2.00.00 and a Vanilla 2.6.21-rc7 Kernel (As far as I saw in changes there where no gfs/related patches from 2.6.21-rc7 to 2.6.21 ?). While testing access via NFS on both nodes I got the following bug on one Node while writing on the other one: 428481.314146] BUG: unable to handle kernel NULL pointer dereference at virtual address 00000000 [428481.314152] printing eip: [428481.314154] 00000000 [428481.314155] *pde = 00000000 [428481.314158] Oops: 0000 [#1] [428481.314164] SMP [428481.314167] Modules linked in: nfs nfsd exportfs lockd nfs_acl sunrpc lock_dlm gfs2 dlm configfs drbd cn floppy nvram autofs4 e1000 piix tg3 dm_mod md_mod ide_cd cdrom ide_core sg sd_mod qla2300 [428481.314197] CPU: 1 [428481.314198] EIP: 0060:[<00000000>] Not tainted VLI [428481.314199] EFLAGS: 00010097 (2.6.21-rc7-QLw #1) [428481.314208] EIP is at rest_init+0x3feff000/0x25 [428481.314211] eax: ef7601f4 ebx: ef7601f4 ecx: 00000000 edx: 00000003 [428481.314215] esi: 00001f68 edi: 00000000 ebp: edf9df14 esp: edf9def4 [428481.314219] ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 [428481.314223] Process dlm_recv/1 (pid: 14727, ti=edf9c000 task=ee3e3570 task.ti=edf9c000) [428481.314226] Stack: c0117515 00000000 ef760288 00000001 00000003 ef760278 00000000 00000001 [428481.314235] edf9df38 c0117567 00000000 00000000 00000082 00000003 ef6cf830 ef760240 [428481.314244] ef6cf82c 00000246 c012af03 00000000 c38296e0 ee3e3690 0000006e ef760278 [428481.314253] Call Trace: [428481.314257] [] __wake_up_common+0x35/0x55 [428481.314264] [] __wake_up+0x32/0x43 [428481.314270] [] run_workqueue+0xd0/0x146 [428481.314277] [] process_recv_sockets+0x0/0x15 [dlm] [428481.314293] [] worker_thread+0x12b/0x156 [428481.314299] [] default_wake_function+0x0/0xc [428481.314305] [] default_wake_function+0x0/0xc [428481.314310] [] worker_thread+0x0/0x156 [428481.314315] [] kthread+0x8c/0xb1 [428481.314320] [] kthread+0x0/0xb1 [428481.314325] [] kernel_thread_helper+0x7/0x14 [428481.314331] ======================= [428481.314333] Code: Bad EIP value. [428481.314338] EIP: [<00000000>] rest_init+0x3feff000/0x25 SS:ESP 0068:edf9def4 Regards, Jens From jakub.suchy at enlogit.cz Mon Apr 30 10:56:07 2007 From: jakub.suchy at enlogit.cz (Jakub Suchy) Date: Mon, 30 Apr 2007 12:56:07 +0200 Subject: [Linux-cluster] APC 7920 as fencing device Message-ID: <20070430105606.GB31184@localhost.localdomain> Hi, did anybody tried Rack PDU, Switched, 1U, 12A/208V, 10A/230V, (8)C13: http://www.apcc.com/resource/include/techspec_index.cfm?base_sku=AP7920 as fencing device for cluster? is it supported? Thanks, Jakub From sebastian.walter at fu-berlin.de Mon Apr 30 12:42:28 2007 From: sebastian.walter at fu-berlin.de (Sebastian Walter) Date: Mon, 30 Apr 2007 14:42:28 +0200 Subject: [Linux-cluster] clvmd hangs Message-ID: <4635E434.3090607@fu-berlin.de> Dear list, on my testcluster (which I narrowed down to 3 hosts), I have problems starting clvmd and accessing the vg* tools. ccsd and cman are started on all nodes, but when it's about to start cvlmd, I only get: [root at master ~]# service clvmd start Starting clvmd: How could this be solved? [root at master ~]# cat /proc/cluster/services Service Name GID LID State Code Fence Domain: "default" 2 3 run - [1 3 2] DLM Lock Space: "clvmd" 1 1 join S-6,20,2 [1 3] User: "usrm::manager" 3 4 run - [1 3] [root at master~]# clustat Timed out waiting for a response from Resource Group Manager Member Status: Quorate Member Name Status ------ ---- ------ master.omain.com Online, Local, rgmanager node1.domain.com Online, rgmanager node2.domain.com Online How to debug? Thanks in advance! Sebastian From lhh at redhat.com Mon Apr 30 17:01:44 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 30 Apr 2007 13:01:44 -0400 Subject: [Linux-cluster] Comment on IRC Message-ID: <20070430170144.GC4012@redhat.com> Take what we can get, you know? [13:00:24] lon: and props to whoever wrote the fencing modules for qlogic swithes and the dell DRAC ;) -- Lon Hohberger - Software Engineer - Red Hat, Inc. From teigland at redhat.com Mon Apr 30 17:17:11 2007 From: teigland at redhat.com (David Teigland) Date: Mon, 30 Apr 2007 13:17:11 -0400 Subject: [Linux-cluster] clvmd hangs In-Reply-To: <4635E434.3090607@fu-berlin.de> References: <4635E434.3090607@fu-berlin.de> Message-ID: <20070430171711.GA7539@redhat.com> On Mon, Apr 30, 2007 at 02:42:28PM +0200, Sebastian Walter wrote: > DLM Lock Space: "clvmd" 1 1 join > S-6,20,2 > [1 3] The dlm is stuck trying to form a lockspace. It would be useful to see this info from each node, and any cman/dlm/fenced lines from /var/log/messages. Dave From jvantuyl at engineyard.com Mon Apr 30 19:18:14 2007 From: jvantuyl at engineyard.com (Jayson Vantuyl) Date: Mon, 30 Apr 2007 14:18:14 -0500 Subject: [Linux-cluster] CMan Build Issue In-Reply-To: <46355B7D.3080700@redhat.com> References: <40E03DB0-735D-4368-96AC-A72C101B51A4@engineyard.com> <46355B7D.3080700@redhat.com> Message-ID: <12D83D7C-B312-41D8-89ED-A7501E2BC258@engineyard.com> Eek. This is probably worth investigating. I'll let you know what I come up with. This is particularly important for people who virtualize their clusters, as Xen doesn't run on anything newer than 2.6.18 yet (and that includes the forthcoming 3.0.5 release). Out of curiosity, how does this figure into the kernel merge of DLM and GFS? I've seen them in newer kernels but been kind of afraid to turn anything on. Can I build the old cluster suite against a kernel containing DLM and still expect it to work? On Apr 29, 2007, at 9:59 PM, Robert Peterson wrote: > Jayson Vantuyl wrote: >> Trying to build cman from cluster relase 1.03.00. Also has same >> error with 1.04.00. Building against Xen-3.0.5 patched 2.6.18. I >> get this: >> CC [M] /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/ >> cluster-1.03.00/cman-kernel/src/proc.o /var/tmp/portage/sys- >> cluster/cman-kernel-1.03.00/work/cluster-1.03.00/cman-kernel/src/ >> cnxman.c: In function 'do_ioctl_join_cluster': >> /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/ >> cluster-1.03.00/cman-kernel/src/cnxman.c:1751: warning: implicit >> declaration of function 'init_utsname' >> /var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/ >> cluster-1.03.00/cman-kernel/src/cnxman.c:1751: error: invalid type >> argument of '->' >> Any ideas? >> --Jayson Vantuyl >> Systems Architect >> Engine Yard >> jvantuyl at engineyard.com > > Hi Jayson, > > Back in February, I updated the STABLE branch of cvs from the older > kernels up to the newer ones. That's probably why it doesn't > compile in > 2.6.18, but should in the 2.6.20 and 21 series. This particular > change to cnxman.c can be seen here: > > http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/cman-kernel/ > src/Attic/cnxman.c.diff? > r1=1.42.2.12.4.1.2.14&r2=1.42.2.12.4.1.2.15&cvsroot=cluster&f=h > > You can always revert that change if you need to stick with the older > kernel. However, there were a bunch of things changed, and they might > also give you problems with the older kernel. > > For a complete list of changes I made, please see: > > https://www.redhat.com/archives/cluster-devel/2007-February/ > msg00115.html > > Regards, > > Bob Peterson > Red Hat Cluster Suite > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Jayson Vantuyl Systems Architect Engine Yard jvantuyl at engineyard.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhh at redhat.com Mon Apr 30 19:26:49 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 30 Apr 2007 15:26:49 -0400 Subject: [Linux-cluster] APC 7920 as fencing device In-Reply-To: <20070430105606.GB31184@localhost.localdomain> References: <20070430105606.GB31184@localhost.localdomain> Message-ID: <20070430192649.GE4012@redhat.com> On Mon, Apr 30, 2007 at 12:56:07PM +0200, Jakub Suchy wrote: > Hi, > did anybody tried Rack PDU, Switched, 1U, 12A/208V, 10A/230V, (8)C13: > http://www.apcc.com/resource/include/techspec_index.cfm?base_sku=AP7920 > > as fencing device for cluster? is it supported? It should work and be supported, IIRC. -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From lhh at redhat.com Mon Apr 30 19:29:35 2007 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 30 Apr 2007 15:29:35 -0400 Subject: [Linux-cluster] service starts up problem In-Reply-To: <1177924289.2858.15.camel@ngua> References: <1177924289.2858.15.camel@ngua> Message-ID: <20070430192934.GF4012@redhat.com> On Mon, Apr 30, 2007 at 11:11:29AM +0200, NGUYEN Can Ext ROSI/DPS wrote: > Could someone help me please ? for example : give me a starting script > that works or show me where i can find patch > for ... /etc/init.d/functions ?? http://sources.redhat.com/cluster/faq.html#rgm_wontrestart More information ^^^ Patch vvv https://bugzilla.redhat.com/bugzilla/attachment.cgi?id=111998 -- Lon -- Lon Hohberger - Software Engineer - Red Hat, Inc. From carlopmart at gmail.com Mon Apr 30 19:29:28 2007 From: carlopmart at gmail.com (carlopmart) Date: Mon, 30 Apr 2007 21:29:28 +0200 Subject: [Linux-cluster] Using ext3 on shared storge (GFS and GFS2 hangs) Message-ID: <46364398.7080401@gmail.com> Hi al, I have serious problems to setup shared storge under RHEL5 with GFS and GFS2 filesystems. My first problem is _netdev param: GFS and GFS2 doesn't support this ( i need to use iscsi disks). Second problem: GFS and GFS2 hangs under xen environment (I am using XenExpress as a host domU). Now my question: which can be pros and cons about using ext3 filesystem under a shared storage for RHEL5 CS?? Many thanks. P.D: I post gfs crash if somebody is interested (kernel 2.6.18-8.1.1.el5xen) Apr 30 20:41:44 rivendel kernel: GFS2: fsid=XenRhelCluster:gfsvol01.1: fatal: filesystem consistency error Apr 30 20:41:44 rivendel kernel: GFS2: fsid=XenRhelCluster:gfsvol01.1: inode = 5267 10434360 Apr 30 20:41:44 rivendel kernel: GFS2: fsid=XenRhelCluster:gfsvol01.1: function = gfs2_drevalidate, file = fs/gfs2/ops_dentry.c, line = 80 Apr 30 20:41:44 rivendel kernel: GFS2: fsid=XenRhelCluster:gfsvol01.1: about to withdraw this file system Apr 30 20:41:44 rivendel kernel: GFS2: fsid=XenRhelCluster:gfsvol01.1: telling LM to withdraw Apr 30 20:41:45 rivendel dlm_controld[1229]: open "/sys/kernel/dlm/gfsvol01/event_done" error -1 2 Apr 30 20:41:45 rivendel kernel: GFS2: fsid=XenRhelCluster:gfsvol01.1: withdrawn Apr 30 20:41:45 rivendel kernel: [] gfs2_lm_withdraw+0x73/0x7f [gfs2] Apr 30 20:41:45 rivendel kernel: [] gfs2_consist_inode_i+0x42/0x47 [gfs2] Apr 30 20:41:45 rivendel kernel: [] gfs2_drevalidate+0x12b/0x1db [gfs2] Apr 30 20:41:45 rivendel kernel: [] gfs2_drevalidate+0x7f/0x1db [gfs2] Apr 30 20:41:45 rivendel kernel: [] do_lookup+0x111/0x157 Apr 30 20:41:45 rivendel kernel: [] __link_path_walk+0x87a/0xd33 Apr 30 20:41:45 rivendel kernel: [] __handle_mm_fault+0x616/0x1070 Apr 30 20:41:45 rivendel kernel: [] link_path_walk+0x49/0xbd Apr 30 20:41:45 rivendel kernel: [] do_page_fault+0x6e8/0xbeb Apr 30 20:41:45 rivendel kernel: [] do_page_fault+0x761/0xbeb Apr 30 20:41:45 rivendel kernel: [] open_exec+0x92/0x9a Apr 30 20:41:45 rivendel kernel: [] _write_lock_irq+0x8/0x1a Apr 30 20:41:45 rivendel kernel: [] do_path_lookup+0x20e/0x25f Apr 30 20:41:45 rivendel kernel: [] __user_walk_fd+0x29/0x3a Apr 30 20:41:45 rivendel kernel: [] vfs_stat_fd+0x15/0x3c Apr 30 20:41:45 rivendel kernel: [] do_page_fault+0x6e8/0xbeb Apr 30 20:41:45 rivendel kernel: [] do_page_fault+0x761/0xbeb Apr 30 20:41:45 rivendel kernel: [] open_exec+0x92/0x9a Apr 30 20:41:45 rivendel kernel: [] _write_lock_irq+0x8/0x1a Apr 30 20:41:45 rivendel kernel: [] sys_stat64+0xf/0x23 Apr 30 20:41:45 rivendel kernel: [] copy_to_user+0x31/0x48 Apr 30 20:41:45 rivendel kernel: [] kfree+0xe/0x77 Apr 30 20:41:45 rivendel kernel: [] do_execve+0x1ec/0x1f5 Apr 30 20:41:45 rivendel kernel: [] do_page_fault+0x0/0xbeb Apr 30 20:41:45 rivendel kernel: [] syscall_call+0x7/0xb Apr 30 20:41:45 rivendel kernel: ======================= -- CL Martinez carlopmart {at} gmail {d0t} com From rpeterso at redhat.com Mon Apr 30 19:51:27 2007 From: rpeterso at redhat.com (Robert Peterson) Date: Mon, 30 Apr 2007 14:51:27 -0500 Subject: [Linux-cluster] CMan Build Issue In-Reply-To: <12D83D7C-B312-41D8-89ED-A7501E2BC258@engineyard.com> References: <40E03DB0-735D-4368-96AC-A72C101B51A4@engineyard.com> <46355B7D.3080700@redhat.com> <12D83D7C-B312-41D8-89ED-A7501E2BC258@engineyard.com> Message-ID: <463648BF.8080908@redhat.com> Jayson Vantuyl wrote: > Eek. This is probably worth investigating. I'll let you know what I > come up with. > > This is particularly important for people who virtualize their clusters, > as Xen doesn't run on anything newer than 2.6.18 yet (and that includes > the forthcoming 3.0.5 release). > > Out of curiosity, how does this figure into the kernel merge of DLM and > GFS? I've seen them in newer kernels but been kind of afraid to turn > anything on. Can I build the old cluster suite against a kernel > containing DLM and still expect it to work? Hi Jayson, There aren't that many "big" differences. I think in most cases it should "just work." We've always tried to keep up with the latest upstream kernel for the STABLE and HEAD branches. Regarding the kernel merge of dlm and gfs2: There are two separate and distinct cluster infrastructures: (1) The "tried and true" that people are using (RHEL4 and STABLE branches of CVS) and (2) The "new and improved" infrastructure (HEAD, RHEL5, branches). The "kernel merge" is for (2). Mixing and matching won't work. However, the old "tried and true" stuff may be built and used on an upstream kernel. I even wrote a document that describes exactly how to accomplish this (although I haven't read it in a while, so it might be slightly out of date by now) and put it on my 108 page: https://rpeterso.108.redhat.com/files/documents/98/247/STABLE.txt RHEL5 is built with the new infrastructure on top of 2.6.18, and has xen, so it's not all that different from upstream. Regards, Bob Peterson Red Hat Cluster Suite