From pasik at iki.fi Mon Mar 1 13:53:16 2010 From: pasik at iki.fi (Pasi =?iso-8859-1?Q?K=E4rkk=E4inen?=) Date: Mon, 1 Mar 2010 15:53:16 +0200 Subject: [Linux-cluster] Xen on RHEL cluster In-Reply-To: <4B8AFB97.90201@srce.hr> References: <4B880508.40808@srce.hr> <4B8AFB97.90201@srce.hr> Message-ID: <20100301135316.GP2761@reaktio.net> On Mon, Mar 01, 2010 at 12:26:15AM +0100, Jakov Sosic wrote: > On 02/26/2010 06:29 PM, Jakov Sosic wrote: > > Hi. > > > > I'm using Xen on RHEL cluster, and I have strange problems. I gave raw > > volumes from storage to Xen virtual machines. With windows, I have a > > problem that nodes don't see the volume as same one.... for example: > > > > clusternode1# clusvcadm -d vm:winxp > > clusternode1# dd if=/dev/mapper/winxp of=/node1winxp > > clusternode2# dd if=/dev/mapper/winxp of=/node2winxp > > clusternode3# dd if=/dev/mapper/winxp of=/node3winxp > > > > When I download these files and diff them, they all three differ. > > > > Also, sometimes very strange things happen. For example I download some > > file into winxp, shut it down, then start it on another node, and file > > is missing?!?!?!?! > > > > Should I use CLVM and not raw volumes from storage? Why is this happening? > > It seems that CLVM is solution (migrated whole cluster to clvm today), > and that the bug is double caching (both domU and dom0 caching), and > that bugs like this occur when using raw LUNs: > https://bugzilla.redhat.com/show_bug.cgi?id=466681 > Oh yeah, this bug can definitely affect you if you're accessing the same disk images from both domU and from dom0. -- Pasi From lhh at redhat.com Mon Mar 1 18:15:24 2010 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 01 Mar 2010 13:15:24 -0500 Subject: [Linux-cluster] Service not migrating In-Reply-To: References: Message-ID: <1267467324.12347.3.camel@localhost.localdomain> On Fri, 2010-02-26 at 00:09 +0000, Joseph L. Casale wrote: > Hi, > Still testing around and learning the ins and outs of rhcs, I have an apache > service with a mount etc configured that starts fine, migrates as well. If > I drop the Ethernet interface on the node with the service active, the remaining > node never starts the service, but if I gracefully take the running node out by > rebooting it etc, the service migrates. > > Any ideas what I am missing? ? -- Lon From lhh at redhat.com Mon Mar 1 18:19:30 2010 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 01 Mar 2010 13:19:30 -0500 Subject: [Linux-cluster] Virtual machine fence fail question In-Reply-To: <95994e3c1002231803y7b08b74cq39202646983431e5@mail.gmail.com> References: <95994e3c1002221905t7a6ccd8dg3eee1439979c62d3@mail.gmail.com> <1266955511.23520.25.camel@localhost.localdomain> <95994e3c1002231803y7b08b74cq39202646983431e5@mail.gmail.com> Message-ID: <1267467570.12347.7.camel@localhost.localdomain> On Wed, 2010-02-24 at 10:03 +0800, Bernard Chew wrote: > > On Wed, Feb 24, 2010 at 4:05 AM, Lon Hohberger wrote: > > On Tue, 2010-02-23 at 11:05 +0800, Bernard Chew wrote: > >> Hi, > >> > >> Given I have 2 Red Hat Clusters; 1 cluster consisting of physical > >> hosts and another consisting of virtual guests which are hosted in the > >> physical hosts. The physical host cluster uses DRAC fencing while the > >> virtual guest cluster uses virtual machine fencing. > >> > >> If a physical host goes down, I saw that DRAC fencing takes place > >> successfully but fencing fail for the virtual guests on the physical > >> host which go down (together). Does the virtual machine fencing fails > >> because the virtual guests are no longer available? How can I > >> configure fencing so that both physical hosts and virtual guests are > >> fenced correctly? > > > > Are you using fence_xvm/fence_xvmd or fence_virsh ? > > > > -- Lon > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > Hi Lon, > > Thank you for looking into this. I am currently using fence_xvm/fence_xvmd. So, fence_xvmd should be making decisions in this case. If a host is not available (dead), fence_xvmd checks the location of the VM we're trying to fence. If the VM was on a host the physical/bare-metal cluster has previously fenced, then fence_xvmd tells the requesting VM that the VM is indeed fenced. This of course requires fence_xvmd to be running in cluster mode and so forth. Fencing the VM works if just the VM dies, right? -- Lon From jcasale at activenetwerx.com Mon Mar 1 18:30:28 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Mon, 1 Mar 2010 18:30:28 +0000 Subject: [Linux-cluster] Service not migrating In-Reply-To: <1267467324.12347.3.camel@localhost.localdomain> References: <1267467324.12347.3.camel@localhost.localdomain> Message-ID: > > >? Hey Lon, It was a fencing error I was missing, its working as expected now! jlc From jcasale at activenetwerx.com Mon Mar 1 18:34:08 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Mon, 1 Mar 2010 18:34:08 +0000 Subject: [Linux-cluster] fence_ifmib problem In-Reply-To: <4B895C0D.6070909@quah.ro> References: <4B895C0D.6070909@quah.ro> Message-ID: >Hi > >I didn't have the same error message but I made fence_ifmib to work by >installing dev-python/pysnmp-2.0.9 on the OS. >You could check which python modules it uses.Maybe this hint could help you. There's an srpm available that I couldn't get built for that, but I installed the egg anyway given I am on a test box. That's not the issue anyway, its complaining about the exit handler, which exists in the stock rh python provided. Something awry with the script itself. Would you mind pastebin'ing your script or posting it here? Thanks! jlc From bernardchew at gmail.com Tue Mar 2 02:37:55 2010 From: bernardchew at gmail.com (Bernard Chew) Date: Tue, 2 Mar 2010 10:37:55 +0800 Subject: [Linux-cluster] Virtual machine fence fail question In-Reply-To: <1267467570.12347.7.camel@localhost.localdomain> References: <95994e3c1002221905t7a6ccd8dg3eee1439979c62d3@mail.gmail.com> <1266955511.23520.25.camel@localhost.localdomain> <95994e3c1002231803y7b08b74cq39202646983431e5@mail.gmail.com> <1267467570.12347.7.camel@localhost.localdomain> Message-ID: <95994e3c1003011837j5805ad5bmd37e7ed2d5c2a73f@mail.gmail.com> > On Tue, Mar 2, 2010 at 2:19 AM, Lon Hohberger wrote: > On Wed, 2010-02-24 at 10:03 +0800, Bernard Chew wrote: >> > On Wed, Feb 24, 2010 at 4:05 AM, Lon Hohberger wrote: >> > On Tue, 2010-02-23 at 11:05 +0800, Bernard Chew wrote: >> >> Hi, >> >> >> >> Given I have 2 Red Hat Clusters; 1 cluster consisting of physical >> >> hosts and another consisting of virtual guests which are hosted in the >> >> physical hosts. The physical host cluster uses DRAC fencing while the >> >> virtual guest cluster uses virtual machine fencing. >> >> >> >> If a physical host goes down, I saw that DRAC fencing takes place >> >> successfully but fencing fail for the virtual guests on the physical >> >> host which go down (together). Does the virtual machine fencing fails >> >> because the virtual guests are no longer available? How can I >> >> configure fencing so that both physical hosts and virtual guests are >> >> fenced correctly? >> > >> > Are you using fence_xvm/fence_xvmd or fence_virsh ? >> > >> > -- Lon >> > >> > -- >> > Linux-cluster mailing list >> > Linux-cluster at redhat.com >> > https://www.redhat.com/mailman/listinfo/linux-cluster >> > >> >> Hi Lon, >> >> Thank you for looking into this. I am currently using fence_xvm/fence_xvmd. > > So, fence_xvmd should be making decisions in this case. > > If a host is not available (dead), fence_xvmd checks the location of the > VM we're trying to fence. ?If the VM was on a host the > physical/bare-metal cluster has previously fenced, then fence_xvmd tells > the requesting VM that the VM is indeed fenced. > > This of course requires fence_xvmd to be running in cluster mode and so > forth. > > Fencing the VM works if just the VM dies, right? > > -- Lon > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > Hi Lon, Fencing works perfectly if just the VM dies. Thanks, Bernard From linux at alteeve.com Tue Mar 2 04:17:20 2010 From: linux at alteeve.com (Digimer) Date: Mon, 01 Mar 2010 23:17:20 -0500 Subject: [Linux-cluster] cluster.conf arguments sent to fence agents Message-ID: <4B8C9150.9080705@alteeve.com> Hi all, I've built a new fence device and am now trying to sort out how and what arguments CMAN (?) sends to the fence agent. Is there a document explaining this interaction? From looking at other fence agents, it seems like CMAN sends either a series of single-dashed arguments followed by their values or a list of arguments, one argument per new line. What I don't know is how CMAN interprets the arguments in the ... statement to arguments sent to the actual fence agent. Thanks for any help! Digi From jcasale at activenetwerx.com Tue Mar 2 04:40:50 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Tue, 2 Mar 2010 04:40:50 +0000 Subject: [Linux-cluster] cluster.conf arguments sent to fence agents In-Reply-To: <4B8C9150.9080705@alteeve.com> References: <4B8C9150.9080705@alteeve.com> Message-ID: > I've built a new fence device and am now trying to sort out how and >what arguments CMAN (?) sends to the fence agent. Is there a document >explaining this interaction? > > From looking at other fence agents, it seems like CMAN sends either a >series of single-dashed arguments followed by their values or a list of >arguments, one argument per new line. What I don't know is how CMAN >interprets the arguments in the ... statement to >arguments sent to the actual fence agent. > >Thanks for any help! Funny, I am hammering away on this now trying to write a Perl fence_ifmib replacement:) http://sources.redhat.com/cluster/wiki/FenceAgentAPI I also made a fencing agent that simply takes stdin and write it to a file to make sure I know what's being passed. It will push through whatever you put in there! jlc From linux at alteeve.com Tue Mar 2 05:18:22 2010 From: linux at alteeve.com (Digimer) Date: Tue, 02 Mar 2010 00:18:22 -0500 Subject: [Linux-cluster] cluster.conf arguments sent to fence agents In-Reply-To: References: <4B8C9150.9080705@alteeve.com> Message-ID: <4B8C9F9E.9040007@alteeve.com> On 10-03-01 11:40 PM, Joseph L. Casale wrote: >> I've built a new fence device and am now trying to sort out how and >> what arguments CMAN (?) sends to the fence agent. Is there a document >> explaining this interaction? >> >> From looking at other fence agents, it seems like CMAN sends either a >> series of single-dashed arguments followed by their values or a list of >> arguments, one argument per new line. What I don't know is how CMAN >> interprets the arguments in the... statement to >> arguments sent to the actual fence agent. >> >> Thanks for any help! > > Funny, I am hammering away on this now trying to write a Perl fence_ifmib > replacement:) > > http://sources.redhat.com/cluster/wiki/FenceAgentAPI > > I also made a fencing agent that simply takes stdin and write it to a file > to make sure I know what's being passed. > > It will push through whatever you put in there! > > jlc Awesome, thanks! Queued that up for light reading in the morning. :) Digi From swhiteho at redhat.com Tue Mar 2 09:06:54 2010 From: swhiteho at redhat.com (Steven Whitehouse) Date: Tue, 02 Mar 2010 09:06:54 +0000 Subject: [Linux-cluster] GFS2 and D state HTTPD processes In-Reply-To: References: Message-ID: <1267520814.3405.2.camel@localhost> Hi, On Fri, 2010-02-26 at 16:52 +0100, Emilio Arjona wrote: > Hi, > > we are experiencing some problems commented in an old thread: > > http://www.mail-archive.com/linux-cluster at redhat.com/msg07091.html > > We have 3 clustered servers under Red Hat 5.4 accessing a GFS2 resource. > > fstab options: > /dev/vg_cluster/lv_cluster /opt/datacluster gfs2 > defaults,noatime,nodiratime,noquota 0 0 > > GFS options: > plock_rate_limit="0" > plock_ownership=1 > > httpd processes run into D status sometimes and the only solution is > hard reset the affected server. > > Can anyone give me some hints to diagnose the problem? > > Thanks :) > Can you give me a rough idea of what the actual workload is and how it is distributed amoung the director(y/ies) ? This is often down to contention on glocks (one per inode) and maybe because there is a process of processes writing a file or directory which is in use (either read-only or writable) by other processes. If you are using php, then you might have to strace it to find out what it is really doing, Steve. > -- > > Emilio Arjona. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From fdinitto at redhat.com Tue Mar 2 09:20:18 2010 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Tue, 02 Mar 2010 10:20:18 +0100 Subject: [Linux-cluster] Cluster 3.0.9 stable release Message-ID: <4B8CD852.4090502@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The cluster team and its community are proud to announce the 3.0.9 stable release from the STABLE3 branch. This release contains a few major bug fixes and several enhancements to qdiskd autocofiguration. We strongly recommend people to update their clusters. In order to build/run the 3.0.9 release you will need: - - corosync 1.2.0 - - openais 1.1.2 - - linux kernel 2.6.31 (only for GFS1 users) The new source tarball can be downloaded here: https://fedorahosted.org/releases/c/l/cluster/cluster-3.0.9.tar.bz2 To report bugs or issues: https://bugzilla.redhat.com/ Would you like to meet the cluster team or members of its community? Join us on IRC (irc.freenode.net #linux-cluster) and share your experience with other sysadministrators or power users. Thanks/congratulations to all people that contributed to achieve this great milestone. Happy clustering, Fabio Under the hood (from 3.0.8): Abhijith Das (2): gfs2_quota: fix uninitialized fiemap flags Merge branch 'STABLE3' of ssh://git.fedoraproject.org/git/cluster into mySTABLE3 Bob Peterson (2): GFS2: libgfs2: build_rgrps was not attaching bh's properly gfs2: fix regressions from performance fixes Fabio M. Di Nitto (2): build: fix publishing of tags to git qdiskd: Enable master_wins intelligently Lon Hohberger (4): qdiskd: Dynamic votes qdisk: Make configuration errors fatal during startup qdiskd: Reject invalid vote counts resource-agents: Clear vm.sh default Steven Whitehouse (1): gfs2: man page updates cman/man/qdisk.5 | 14 +++- cman/qdisk/disk.h | 4 +- cman/qdisk/main.c | 201 ++++++++++++++++++++++++++++++----------- gfs2/edit/hexedit.c | 2 +- gfs2/edit/savemeta.c | 3 +- gfs2/libgfs2/fs_geometry.c | 21 ++--- gfs2/libgfs2/fs_ops.c | 12 +-- gfs2/libgfs2/structures.c | 16 ++-- gfs2/man/gfs2_tool.8 | 12 ++- gfs2/man/mount.gfs2.8 | 108 +++++++++++++++------- gfs2/mkfs/main_grow.c | 2 + gfs2/mkfs/main_mkfs.c | 1 + gfs2/quota/check.c | 2 + gfs2/quota/main.c | 6 +- make/release.mk | 2 +- rgmanager/src/resources/vm.sh | 2 +- 16 files changed, 283 insertions(+), 125 deletions(-) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJLjNhMAAoJEFA6oBJjVJ+ONE0P+gJGdVsShZzAbAGDyoOT9O5x v89WAxBs2HzGe7qeg24EHkQeIQXbnv0/FFdFTjDbTygMEU/2MoszPBLt+io6ji+a hBrYzOk9/e6lPW1q5Qy2l8bI8mhMVtYo1Oh85cugwrRxUEYcYTe8CQrVAgUN3Uv6 i8rUguxKEigWUqkVESF7V3NEIFlNM7nvYuEPO1mJpCotaPxObWqafHhucp2dsx1z Eq9J3uKOacrVsdg3jDn9v9GFPXDVwQDAU+rFFJIQd6M9csbHKgVBiQwhLvz9jVEQ 0E87eBFVfqhBhBmZPkmPWn85MU/hqJuUXFJ7I7E7dEz17Mf7TqFhn5LOLHv77+Q5 jMof4mvEZJUqQ+qo4z2V2ieNAo72VM+vdK27udJsoA0XfXbSSPyyE46EDhx2ujBI 9Vn+8Khha4iSHOrz3W8UW7G3rqHljVl6jneB4E89N7yOIi97wVIj8BOZ5FCgpPE6 yD2O2rDNAzSg6EXCY26KU/VyIAA0WlyqSsVXCAO2uDGJKPQCbCXgkw2dCVPXYTfK VFrSYxyjZ7pxksnKxFd1h0o8R7KunmPxtkOc0HS+NE5bLOILSI+24DBsvv2uJ0rx d+jL/HQe6e+G1wyOEqOm+LpdWoIWC6acsUFtK6K9mD1Z0C+VtnfI5wDrfe45nAmx fdlacHOdvX8CV/O3hcW5 =18Wl -----END PGP SIGNATURE----- From jfriesse at redhat.com Tue Mar 2 09:22:57 2010 From: jfriesse at redhat.com (Jan Friesse) Date: Tue, 02 Mar 2010 10:22:57 +0100 Subject: [Linux-cluster] fence_ifmib problem In-Reply-To: References: Message-ID: <4B8CD8F1.9010209@redhat.com> Hi, it looks you are using new FA (from STABLE3/master). Can you please send me what Python version you are using (python -V] and maybe what distribution/version? Regards, Honza Joseph L. Casale wrote: > Trying to run this script gives me the following error: > > Traceback (most recent call last): > File "/usr/lib/fence/fence_ifmib.py", line 139, in ? > main() > File "/usr/lib/fence/fence_ifmib.py", line 112, in main > atexit.register(atexit_handler) > NameError: global name 'atexit' is not defined > > Anyone know what I am missing? I copied it from git and edited the > sys.path.append("/usr/lib/fence") > > Thanks, > jlc > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From dan at quah.ro Tue Mar 2 10:22:49 2010 From: dan at quah.ro (Dan Candea) Date: Tue, 02 Mar 2010 12:22:49 +0200 Subject: [Linux-cluster] fence_ifmib problem In-Reply-To: References: <4B895C0D.6070909@quah.ro> Message-ID: <4B8CE6F9.10504@quah.ro> On 03/01/2010 08:34 PM, Joseph L. Casale wrote: >> Hi >> >> I didn't have the same error message but I made fence_ifmib to work by >> installing dev-python/pysnmp-2.0.9 on the OS. >> You could check which python modules it uses.Maybe this hint could help you. >> > There's an srpm available that I couldn't get built for that, but I installed > the egg anyway given I am on a test box. > > That's not the issue anyway, its complaining about the exit handler, which > exists in the stock rh python provided. Something awry with the script itself. > > Would you mind pastebin'ing your script or posting it here? > > Thanks! > jlc > > Jan Friesse made a point in the next post. I'm using the one from STABLE2, version 2.3.10. I don't think is your case. -- Dan C?ndea Does God Play Dice? From emilio.ah at gmail.com Tue Mar 2 11:10:43 2010 From: emilio.ah at gmail.com (Emilio Arjona) Date: Tue, 2 Mar 2010 12:10:43 +0100 Subject: [Linux-cluster] GFS2 and D state HTTPD processes In-Reply-To: <1267520814.3405.2.camel@localhost> References: <1267520814.3405.2.camel@localhost> Message-ID: Thanks for your response, Steve. 2010/3/2 Steven Whitehouse : > Hi, > > On Fri, 2010-02-26 at 16:52 +0100, Emilio Arjona wrote: >> Hi, >> >> we are experiencing some problems commented in an old thread: >> >> http://www.mail-archive.com/linux-cluster at redhat.com/msg07091.html >> >> We have 3 clustered servers under Red Hat 5.4 accessing a GFS2 resource. >> >> fstab options: >> /dev/vg_cluster/lv_cluster /opt/datacluster gfs2 >> defaults,noatime,nodiratime,noquota 0 0 >> >> GFS options: >> plock_rate_limit="0" >> plock_ownership=1 >> >> httpd processes run into D status sometimes and the only solution is >> hard reset the affected server. >> >> Can anyone give me some hints to diagnose the problem? >> >> Thanks :) >> > Can you give me a rough idea of what the actual workload is and how it > is distributed amoung the director(y/ies) ? We had problems with php sessions in the past but we fixed it by configuring php to store the sessions in the database instead of in the GFS filesystem. Now, we're having problems with files and directories in the "data" folder of Moodle LMS. "lsof -p" returned a i/o operation over the same folder in 2/3 nodes, we did a hard reset of these nodes but some hours after the CPU load grew up again, specially in the node that wasn't rebooted. We decided to reboot (v?a ssh) this node, then the CPU load went down to normal values in all nodes. I don't think the system's load is high enough to produce concurrent access problems. It's more likely to be some misconfiguration, in fact, we changed some GFS2 options to non default values to increase performance (http://www.linuxdynasty.org/howto-increase-gfs2-performance-in-a-cluster.html). > > This is often down to contention on glocks (one per inode) and maybe > because there is a process of processes writing a file or directory > which is in use (either read-only or writable) by other processes. > > If you are using php, then you might have to strace it to find out what > it is really doing, Ok, we will try to strace the D processes and post the results. Hope we find something!! > > Steve. > >> -- >> >> Emilio Arjona. >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Emilio Arjona. From jcasale at activenetwerx.com Tue Mar 2 14:49:09 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Tue, 2 Mar 2010 14:49:09 +0000 Subject: [Linux-cluster] fence_ifmib problem In-Reply-To: <4B8CD8F1.9010209@redhat.com> References: <4B8CD8F1.9010209@redhat.com> Message-ID: >Hi, >it looks you are using new FA (from STABLE3/master). Can you please send >me what Python version you are using (python -V] and maybe what >distribution/version? Honza, I am using CentOS 5.4 for my tests: # uname -a Linux node1.domain.local 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010 x86_64 x86_64 x86_64 GNU/Linux # python -V Python 2.4.3 Thanks for the help! jlc From jfriesse at redhat.com Tue Mar 2 16:15:51 2010 From: jfriesse at redhat.com (Jan Friesse) Date: Tue, 02 Mar 2010 17:15:51 +0100 Subject: [Linux-cluster] fence_ifmib problem In-Reply-To: References: <4B8CD8F1.9010209@redhat.com> Message-ID: <4B8D39B7.2010309@redhat.com> Hi, I think I found where main problem is. You downloaded very new fencing agent, but your system has OLD fencing.py library. This is main reason, why it doesn't work. You must also download: - lib/fencing.py.py - lib/fencing_snmp.py.py Both of this files must be renamed to contain only one .py (this means lib/fencing.py and fencing_snmp.py). There is also need to replace some of @XXX@ strings (like @SNMPBIN@). Much faster choice is to download git, run autogen and make. Script will build lib/*.py for you (correctly). This all apply, if you really want newest coolest one from FA master GIT. BUT you can also use cluster.git, with RHEL55 branch. Here, just go to fence directory, type ./configure and in fence/agents/lib, type make + make install, and you should be ready to go. Last possibility (from my point the best one) is to wait for CentOS 5.5, this will include new ifmib and correct python lib. Regards, Honza Joseph L. Casale wrote: >> Hi, >> it looks you are using new FA (from STABLE3/master). Can you please send >> me what Python version you are using (python -V] and maybe what >> distribution/version? > > Honza, > I am using CentOS 5.4 for my tests: > # uname -a > Linux node1.domain.local 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010 x86_64 x86_64 x86_64 GNU/Linux > # python -V > Python 2.4.3 > > Thanks for the help! > jlc > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From sklemer at gmail.com Tue Mar 2 17:10:12 2010 From: sklemer at gmail.com (=?UTF-8?B?16nXnNeV150g16fXnNee16g=?=) Date: Tue, 2 Mar 2010 19:10:12 +0200 Subject: [Linux-cluster] fencing issue. Message-ID: <2746211a1003020910p12b21c3ds27cbb38baf84bd00@mail.gmail.com> Hello. I build 2 members redhat 5.3 cluster with latest cman rpm. Everything is working fine. We did some failover tests, all tests passed ( nic , fiber , reboot ). The only test which failed was - pulling out power cable from one node. The other system try in loop fencing the failed node & services not relocated. Can someone advise ?? *Best Regards, Shalom Klemer* -------------- next part -------------- An HTML attachment was scrubbed... URL: From sklemer at gmail.com Tue Mar 2 17:17:44 2010 From: sklemer at gmail.com (=?UTF-8?B?16nXnNeV150g16fXnNee16g=?=) Date: Tue, 2 Mar 2010 19:17:44 +0200 Subject: [Linux-cluster] fencing issue - with attach logs&conf Message-ID: <2746211a1003020917q6154faeese3f95e50c785264f@mail.gmail.com> Hello. I build 2 members redhat 5.3 cluster with latest cman rpm. Everything is working fine. We did some failover tests, all tests passed ( nic , fiber , reboot ). The only test which failed was - pulling out power cable from one node. The other system try in loop fencing the failed node & services not relocated. Can someone advise ?? *Best Regards, Shalom Klemer* -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cluster.conf Type: application/octet-stream Size: 3262 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: messages.gz Type: application/x-gzip Size: 72089 bytes Desc: not available URL: From esggrupos at gmail.com Tue Mar 2 17:40:08 2010 From: esggrupos at gmail.com (ESGLinux) Date: Tue, 2 Mar 2010 18:40:08 +0100 Subject: [Linux-cluster] nfsclient: nfsclient1 is missing Message-ID: <3128ba141003020940t38d8f6dcgd4c9d08e9064247@mail.gmail.com> Hi All, I?m configuring a cluster with a nfs sevice. I have followd this documentation: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/pdf/Configuration_Example_-_NFS_Over_GFS.pdf now I have only one node but I can?t make it works, The nfs service starts but after few seconds I get this error: Mar 2 18:32:11 server1 clurgmgrd: [25700]: nfsclient:nfsclient1 is missing! Mar 2 18:32:11 server1 clurgmgrd[25700]: status on nfsclient "nfsclient1" returned 1 (generic error) Mar 2 18:32:21 server1 clurgmgrd[25700]: Stopping service service:nfssvc Mar 2 18:32:21 server1 avahi-daemon[4075]: Withdrawing address record for 172.16.178.82 on eth0. Mar 2 18:32:31 server1 clurgmgrd[25700]: Service service:nfssvc is recovering Mar 2 18:32:31 server1 clurgmgrd[25700]: Service service:nfssvc is stopped And the services stops, I suposse the problem is with the configuration of the nfsclient, but I don?t see the problem. The configuration is here: I have checked this link http://lists.centos.org/pipermail/centos/2008-January/049697.html but I don?t see the error, any idea? Thanks in advance ESG -------------- next part -------------- An HTML attachment was scrubbed... URL: From esggrupos at gmail.com Tue Mar 2 18:28:59 2010 From: esggrupos at gmail.com (ESGLinux) Date: Tue, 2 Mar 2010 19:28:59 +0100 Subject: [Linux-cluster] nfsclient: nfsclient1 is missing In-Reply-To: <3128ba141003020940t38d8f6dcgd4c9d08e9064247@mail.gmail.com> References: <3128ba141003020940t38d8f6dcgd4c9d08e9064247@mail.gmail.com> Message-ID: <3128ba141003021028m1abb24d1x1223f31d7c4e35a5@mail.gmail.com> Hello, I have commented and changed two lines of the script /usr/share/cluster/nfsclient.sh declare OCF_RESKEY_target_tmp=$(clufindhostname -i "$OCF_RESKEY_target") if [ $? -ne 0 ]; then [ "$OCF_RESKEY_use_cache" = "1" ] || rm -f $tmpfn # ocf_log err "nfsclient:$OCF_RESKEY_name is missing!" # exit 1 exit 0 ..... # ocf_log err "nfsclient:$OCF_RESKEY_name is missing!" # exit 1 exit 0 and now it works, the question is: what hell does this code do???? and how can I debug it to try to find my problem? Greetings, ESG > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlopmart at gmail.com Tue Mar 2 20:00:33 2010 From: carlopmart at gmail.com (carlopmart) Date: Tue, 02 Mar 2010 21:00:33 +0100 Subject: [Linux-cluster] Strange error returned by openais Message-ID: <4B8D6E61.50308@gmail.com> Hi all, I am trying to setup a rh5.4 cluster with only two nodes, but I can't. Under /var/log/messages I can see a lot of errors like these: Mar 2 20:52:48 asfaloth ccsd[2488]: Error while processing connect: Connection refused Mar 2 20:52:48 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. Mar 2 20:52:48 asfaloth ccsd[2488]: Error while processing connect: Connection refused Mar 2 20:52:48 asfaloth openais[2496]: [CMAN ] Can't read CCS to get updated config version 10. Activity suspended on this node Mar 2 20:52:49 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. Mar 2 20:52:49 asfaloth ccsd[2488]: Error while processing connect: Connection refused Mar 2 20:52:49 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. Mar 2 20:52:49 asfaloth ccsd[2488]: Error while processing connect: Connection refused Mar 2 20:52:49 asfaloth openais[2496]: [CMAN ] Can't read CCS to get updated config version 10. Activity suspended on this node Mar 2 20:52:50 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. Mar 2 20:52:50 asfaloth ccsd[2488]: Error while processing connect: Connection refused Mar 2 20:52:50 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. Mar 2 20:52:50 asfaloth ccsd[2488]: Error while processing connect: Connection refused Mar 2 20:52:51 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. Mar 2 20:52:51 asfaloth ccsd[2488]: Error while processing connect: Connection refused Mar 2 20:52:51 asfaloth openais[2496]: [CMAN ] Can't read CCS to get updated config version 10. Activity suspended on this node Mar 2 20:52:51 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. Mar 2 20:52:51 asfaloth ccsd[2488]: Error while processing connect: Connection refused These nodes have two network interfaces, one on the same network for cluster operation and another on different subnet. Like this: Node01: 172.16.1.1 (eth0) and 192.168.35.1 (eth1) Node02: 172.16.1.2 (eth0) and 172.26.50.1 (eth1) Default gateways point to 192.168.35.20 in node01 and on node02 to 172.26.50.30 ... maybe this is the problem?? I have put ip routing rules on both nodes but problem continues ... How can I fix this?? Thanks. -- CL Martinez carlopmart {at} gmail {d0t} com From a.alawi at auckland.ac.nz Tue Mar 2 20:22:53 2010 From: a.alawi at auckland.ac.nz (Abraham Alawi) Date: Wed, 3 Mar 2010 09:22:53 +1300 Subject: [Linux-cluster] Strange error returned by openais In-Reply-To: <4B8D6E61.50308@gmail.com> References: <4B8D6E61.50308@gmail.com> Message-ID: <49D77859-899B-40F4-A6DA-72E2A9AEE6BA@auckland.ac.nz> You can easily tell if they are communicating through the dedicated network "172.16" by having them pinging each other, or run 'cman_tool status' on the two nodes if 'Node addresses' not the same subnet then that's could be why. You should define static records for them in /etc/hosts: > 172.16.1.1 node01 > 172.16.1.2 node02 But as far as I know they don't have to be on the same subnet as long as the multicasting work, tcpdump would be your best friend if you need to debug it further .. Cheers, -- Abraham On 3/03/2010, at 9:00 AM, carlopmart wrote: > Hi all, > > I am trying to setup a rh5.4 cluster with only two nodes, but I can't. Under /var/log/messages I can see a lot of errors like these: > > Mar 2 20:52:48 asfaloth ccsd[2488]: Error while processing connect: Connection refused > Mar 2 20:52:48 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. > Mar 2 20:52:48 asfaloth ccsd[2488]: Error while processing connect: Connection refused > Mar 2 20:52:48 asfaloth openais[2496]: [CMAN ] Can't read CCS to get updated config version 10. Activity suspended on this node > Mar 2 20:52:49 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. > Mar 2 20:52:49 asfaloth ccsd[2488]: Error while processing connect: Connection refused > Mar 2 20:52:49 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. > Mar 2 20:52:49 asfaloth ccsd[2488]: Error while processing connect: Connection refused > Mar 2 20:52:49 asfaloth openais[2496]: [CMAN ] Can't read CCS to get updated config version 10. Activity suspended on this node > Mar 2 20:52:50 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. > Mar 2 20:52:50 asfaloth ccsd[2488]: Error while processing connect: Connection refused > Mar 2 20:52:50 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. > Mar 2 20:52:50 asfaloth ccsd[2488]: Error while processing connect: Connection refused > Mar 2 20:52:51 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. > Mar 2 20:52:51 asfaloth ccsd[2488]: Error while processing connect: Connection refused > Mar 2 20:52:51 asfaloth openais[2496]: [CMAN ] Can't read CCS to get updated config version 10. Activity suspended on this node > Mar 2 20:52:51 asfaloth ccsd[2488]: Cluster is not quorate. Refusing connection. > Mar 2 20:52:51 asfaloth ccsd[2488]: Error while processing connect: Connection refused > > These nodes have two network interfaces, one on the same network for cluster operation and another on different subnet. Like this: > > Node01: 172.16.1.1 (eth0) and 192.168.35.1 (eth1) > Node02: 172.16.1.2 (eth0) and 172.26.50.1 (eth1) > > Default gateways point to 192.168.35.20 in node01 and on node02 to 172.26.50.30 ... maybe this is the problem?? > > I have put ip routing rules on both nodes but problem continues ... How can I fix this?? > > Thanks. > -- > CL Martinez > carlopmart {at} gmail {d0t} com > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster '''''''''''''''''''''''''''''''''''''''''''''''''''''' Abraham Alawi Unix/Linux Systems Administrator Science IT University of Auckland e: a.alawi at auckland.ac.nz p: +64-9-373 7599, ext#: 87572 '''''''''''''''''''''''''''''''''''''''''''''''''''''' From quickshiftin at gmail.com Tue Mar 2 22:14:38 2010 From: quickshiftin at gmail.com (Nathan Nobbe) Date: Tue, 2 Mar 2010 14:14:38 -0800 (PST) Subject: [Linux-cluster] Invitation to connect on LinkedIn Message-ID: <1631048129.6382767.1267568078582.JavaMail.app@ech3-cdn11.prod> LinkedIn ------------Nathan Nobbe requested to add you as a connection on LinkedIn: ------------------------------------------ Marian, I'd like to add you to my professional network on LinkedIn. - Nathan Accept invitation from Nathan Nobbe http://www.linkedin.com/e/ulDuieLaAX544oVCOYcgj_GaXIys4TuLMXGmOx/blk/I1855266268_2/1BpC5vrmRLoRZcjkkZt5YCpnlOt3RApnhMpmdzgmhxrSNBszYOnPwSczoSczkRe359bPcOsmpTcQRVbPwRc3APd3kNcj4LrCBxbOYWrSlI/EML_comm_afe/ View invitation from Nathan Nobbe http://www.linkedin.com/e/ulDuieLaAX544oVCOYcgj_GaXIys4TuLMXGmOx/blk/I1855266268_2/39ve3oOdzoOdjkUckALqnpPbOYWrSlI/svi/ ------------------------------------------ DID YOU KNOW you can be the first to know when a trusted member of your network changes jobs? With Network Updates on your LinkedIn home page, you'll be notified as members of your network change their current position. Be the first to know and reach out! http://www.linkedin.com/ ------ (c) 2010, LinkedIn Corporation -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeff at 3d0g.net Tue Mar 2 22:50:26 2010 From: jeff at 3d0g.net (Jeff Karpinski) Date: Tue, 02 Mar 2010 15:50:26 -0700 Subject: [Linux-cluster] Shared storage across clustered VMs? Message-ID: <4B8D9632.80206@3d0g.net> I've got a 4 node cluster back-ended with iSCSI storage that's happily running 20 or so VMs. I now have a request to present some shared storage across several of the VMs and am wondering what's the best way to accomplish this. GFS2? Can VMs even communicate back with the cluster for lock_dlm to work? Interested in how others have skinned this cat... From glisha at gmail.com Tue Mar 2 23:15:12 2010 From: glisha at gmail.com (Georgi Stanojevski) Date: Wed, 3 Mar 2010 00:15:12 +0100 Subject: [Linux-cluster] fencing issue - with attach logs&conf In-Reply-To: <2746211a1003020917q6154faeese3f95e50c785264f@mail.gmail.com> References: <2746211a1003020917q6154faeese3f95e50c785264f@mail.gmail.com> Message-ID: On Tue, Mar 2, 2010 at 6:17 PM, ???? ???? wrote: > > The only test which failed was - pulling out power cable from one node. > > The other system try in loop fencing the failed node & services not > relocated. > > > As I understand this is by design. When you pull out the power cables you shutdown the fencing device on that node and the remaining node can't successfully fence it. The nodes should have redundant power supplies, to avoid single point of failure. -- Glisha http:// -------------- next part -------------- An HTML attachment was scrubbed... URL: From brem.belguebli at gmail.com Wed Mar 3 02:11:50 2010 From: brem.belguebli at gmail.com (brem belguebli) Date: Wed, 03 Mar 2010 03:11:50 +0100 Subject: [Linux-cluster] strange cluster behavior Message-ID: <1267582310.2226.104.camel@localhost> Hi, I experienced a strange cluster behavior that I couldn't explain. I have a 4 nodes Rhel 5.4 cluster (node1, node2, node3 and node4). Node1 and node2 are connected to an ethernet switch (sw1), node3 and node4 are connected to another switch (sw2). The 4 nodes are on the same Vlan. sw1 and sw2 are connected thru a couple of core switches, and the nodes Vlan is well propagated across the network that I just described. Latency between node1 and node4 (on 2 different switches) doesn't exceed 0.3 ms. The cluster is normally configured with a iscsi quorum device located on another switch. I wanted to check how it would behave when quorum disk is not active (removed from cluster.conf) if a member node came to get isolated (link up but not on the right vlan). Node3 is the one I played with. The fence_device for this node is intentionally misconfigured to be able to follow on this node console what happens. When changing the vlan membership of node3, results are as expected, the 3 remaining nodes see it come offline after totem timer expiry, and node1 (lowest node id) starts trying to fence node3 (without success as intentionally misconfigured). Node3 sees itself the only member of the cluster which is inquorate. Coherent as it became a single node parition. When putting back node3 vlan conf to the right value, things go bad. Node1, 2 and 4 instruct node3 cman to kill itself as it did re appear with an already existing status. Why not. Node1 and node2 then say then the quorum is dissolved and see themselves offline (????), node3 offline and node4 online. Node4 sees itself online but cluster inquorate as we also lost node1 and node2. I thought about potential multicast problems, but it behaves the same way when cman is configured to broadcast. The same test run with qdisk enabled is behaving normally, when node3 gets back to network it gets automatically rebooted (thx to qdisk), the cluster remains stable. Any idea why node1 and node2 go bad when node3 is back ? Thanks Brem From michaelm at plumbersstock.com Wed Mar 3 02:02:40 2010 From: michaelm at plumbersstock.com (Michael McGlothlin) Date: Tue, 2 Mar 2010 19:02:40 -0700 Subject: [Linux-cluster] Is Global FS the way to go? Message-ID: <5790d5b71003021802x3e7b92d2ia5ad95168a30a76a@mail.gmail.com> I'm setting up two servers to share the load of running several virtual machines (currently running ESXi) that are currently running on just one server (which will later be upgraded and added back to the cluster). I am trying to decide what is an affordable way to provide a shared file system so that in case of a server going down another can pick up where it left off. I want data to be mirrored live if possible and for data access to be fast and reliable. Is Global FS a good fit for this use? From looking at the docs it appears I could set up a local virtual sever on each machine and have it keep a local copy that keeps itself in sync with the other servers using the filesystem. Then all other VMs could access their local server via iSCSI or NFS. Or is this all completely the wrong way to go and there is a better way to meet my goals? My budget is really tight so I can't spend tens of thousands of dollars on equipment or software. Thanks, Michael McGlothlin -------------- next part -------------- An HTML attachment was scrubbed... URL: From zaeem.arshad at gmail.com Wed Mar 3 04:03:53 2010 From: zaeem.arshad at gmail.com (Zaeem Arshad) Date: Wed, 3 Mar 2010 09:03:53 +0500 Subject: [Linux-cluster] Is Global FS the way to go? In-Reply-To: <5790d5b71003021802x3e7b92d2ia5ad95168a30a76a@mail.gmail.com> References: <5790d5b71003021802x3e7b92d2ia5ad95168a30a76a@mail.gmail.com> Message-ID: <3e1809421003022003r586adea5qfeef6fcdfa0e75bf@mail.gmail.com> On Wed, Mar 3, 2010 at 7:02 AM, Michael McGlothlin wrote: > I'm setting up two servers to share the load of running several virtual > machines (currently running ESXi) that are currently running on just one > server (which will later be upgraded and added back to the cluster). I am > trying to decide what is an affordable way to provide a shared file system > so that in case of a server going down another can pick up where it left > off. I want data to be mirrored live if possible and for data access to be > fast and reliable. Is Global FS a good fit for this use? From looking at the > docs it appears I could set up a local virtual sever on each machine and > have it keep a local copy that keeps itself in sync with the other servers > using the filesystem. Then all other VMs could access their local server via > iSCSI or NFS. > > Or is this all completely the wrong way to go and there is a better way to > meet my goals? My budget is really tight so I can't spend tens of thousands > of dollars on equipment or software. > I will be watching this thread with great interest as I am in a similar situation. Right now, my plan is to share the disks on the two servers using DRBD providing a RAID-1 and then share the DRBD volume using GFS. As per my little understanding, you will have to use CLVM to make sure you don't run into other issues. I have yet to test this configuration though. Regards -- Zaeem From raju.rajsand at gmail.com Wed Mar 3 04:22:42 2010 From: raju.rajsand at gmail.com (Rajagopal Swaminathan) Date: Wed, 3 Mar 2010 09:52:42 +0530 Subject: [Linux-cluster] fencing issue. In-Reply-To: <2746211a1003020910p12b21c3ds27cbb38baf84bd00@mail.gmail.com> References: <2746211a1003020910p12b21c3ds27cbb38baf84bd00@mail.gmail.com> Message-ID: <8786b91c1003022022i6035e3f2gdab67460a5a5615a@mail.gmail.com> Greetings, On Tue, Mar 2, 2010 at 10:40 PM, ???? ???? wrote: > Hello. > > Everything is working fine. We did some failover tests, all tests passed ( > nic , fiber , reboot ). > > The only test which failed was - pulling out power cable from one node. > How do you do the fencing? in-band (ILO, RSA, ALOM, DRAC) or out-of-band (Power fencing) Regards, Rajagopal From raju.rajsand at gmail.com Wed Mar 3 04:32:41 2010 From: raju.rajsand at gmail.com (Rajagopal Swaminathan) Date: Wed, 3 Mar 2010 10:02:41 +0530 Subject: [Linux-cluster] Strange error returned by openais In-Reply-To: <4B8D6E61.50308@gmail.com> References: <4B8D6E61.50308@gmail.com> Message-ID: <8786b91c1003022032j48369ea1nd398111499e1781f@mail.gmail.com> Greetings, On Wed, Mar 3, 2010 at 1:30 AM, carlopmart wrote: Have you put in the two_node=1 directive in cluster config file? Regards Rajagopal From xavier.montagutelli at unilim.fr Wed Mar 3 07:16:45 2010 From: xavier.montagutelli at unilim.fr (Xavier Montagutelli) Date: Wed, 3 Mar 2010 08:16:45 +0100 Subject: [Linux-cluster] strange cluster behavior In-Reply-To: <1267582310.2226.104.camel@localhost> References: <1267582310.2226.104.camel@localhost> Message-ID: <201003030816.45302.xavier.montagutelli@unilim.fr> On Wednesday 03 March 2010 03:11:50 brem belguebli wrote: > Hi, > > I experienced a strange cluster behavior that I couldn't explain. > > I have a 4 nodes Rhel 5.4 cluster (node1, node2, node3 and node4). > > Node1 and node2 are connected to an ethernet switch (sw1), node3 and > node4 are connected to another switch (sw2). The 4 nodes are on the same > Vlan. > > sw1 and sw2 are connected thru a couple of core switches, and the nodes > Vlan is well propagated across the network that I just described. > > Latency between node1 and node4 (on 2 different switches) doesn't exceed > 0.3 ms. > > The cluster is normally configured with a iscsi quorum device located on > another switch. > > I wanted to check how it would behave when quorum disk is not active > (removed from cluster.conf) if a member node came to get isolated (link > up but not on the right vlan). > > Node3 is the one I played with. > > The fence_device for this node is intentionally misconfigured to be able > to follow on this node console what happens. > > When changing the vlan membership of node3, results are as expected, the > 3 remaining nodes see it come offline after totem timer expiry, and > node1 (lowest node id) starts trying to fence node3 (without success as > intentionally misconfigured). > > Node3 sees itself the only member of the cluster which is inquorate. > Coherent as it became a single node parition. > > When putting back node3 vlan conf to the right value, things go bad. (My two cents) You just put it back in the good VLAN, without restarting the host ? I did this kind of test (under RH 5.3), and things always get bad if a node supposed to be fenced is not really fenced and comes back. Perhaps this is an intended behaviour to prevent "split brain" cases (even at the cost of the whole cluster going down) ? Or perhaps it depends how your misconfigured fence device behaves (does it give an exit status ? What exit status does it send ?). > > Node1, 2 and 4 instruct node3 cman to kill itself as it did re appear > with an already existing status. Why not. > > Node1 and node2 then say then the quorum is dissolved and see themselves > offline (????), node3 offline and node4 online. > > Node4 sees itself online but cluster inquorate as we also lost node1 and > node2. > > I thought about potential multicast problems, but it behaves the same > way when cman is configured to broadcast. > > The same test run with qdisk enabled is behaving normally, when node3 > gets back to network it gets automatically rebooted (thx to qdisk), the > cluster remains stable. > > Any idea why node1 and node2 go bad when node3 is back ? > > Thanks > > Brem > > > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Xavier Montagutelli Tel : +33 (0)5 55 45 77 20 Service Commun Informatique Fax : +33 (0)5 55 45 75 95 Universite de Limoges 123, avenue Albert Thomas 87060 Limoges cedex From xavier.montagutelli at unilim.fr Wed Mar 3 07:28:21 2010 From: xavier.montagutelli at unilim.fr (Xavier Montagutelli) Date: Wed, 3 Mar 2010 08:28:21 +0100 Subject: [Linux-cluster] Shared storage across clustered VMs? In-Reply-To: <4B8D9632.80206@3d0g.net> References: <4B8D9632.80206@3d0g.net> Message-ID: <201003030828.21320.xavier.montagutelli@unilim.fr> On Tuesday 02 March 2010 23:50:26 Jeff Karpinski wrote: > I've got a 4 node cluster back-ended with iSCSI storage that's happily > running 20 or so VMs. I now have a request to present some shared > storage across several of the VMs and am wondering what's the best way > to accomplish this. GFS2? Can VMs even communicate back with the cluster > for lock_dlm to work? > > Interested in how others have skinned this cat... I don't know if GFS2 is the best way to share data among your VMs. But if you go for GFS2, I suppose the clusters should be different : one cluster for the hosts, and one cluster for the VMs having a shared disk with GFS2. They have different purposes, you should not mix them. -- Xavier Montagutelli Tel : +33 (0)5 55 45 77 20 Service Commun Informatique Fax : +33 (0)5 55 45 75 95 Universite de Limoges 123, avenue Albert Thomas 87060 Limoges cedex From kitgerrits at gmail.com Wed Mar 3 08:09:28 2010 From: kitgerrits at gmail.com (Kit Gerrits) Date: Wed, 3 Mar 2010 09:09:28 +0100 Subject: [Linux-cluster] Shared storage across clustered VMs? In-Reply-To: <201003030828.21320.xavier.montagutelli@unilim.fr> Message-ID: <4b8e192c.9298cc0a.61b6.0a24@mx.google.com> Might it be a good idea to stick them in the same cluster, but with different failure domains? That way, chances are higher of staying quorate. Just a thought... Kit -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Xavier Montagutelli Sent: woensdag 3 maart 2010 8:28 To: linux clustering Subject: Re: [Linux-cluster] Shared storage across clustered VMs? On Tuesday 02 March 2010 23:50:26 Jeff Karpinski wrote: > I've got a 4 node cluster back-ended with iSCSI storage that's happily > running 20 or so VMs. I now have a request to present some shared > storage across several of the VMs and am wondering what's the best way > to accomplish this. GFS2? Can VMs even communicate back with the > cluster for lock_dlm to work? > > Interested in how others have skinned this cat... I don't know if GFS2 is the best way to share data among your VMs. But if you go for GFS2, I suppose the clusters should be different : one cluster for the hosts, and one cluster for the VMs having a shared disk with GFS2. They have different purposes, you should not mix them. From martijn at tenheuvel.net Wed Mar 3 08:35:20 2010 From: martijn at tenheuvel.net (martijn at tenheuvel.net) Date: Wed, 3 Mar 2010 09:35:20 +0100 (CET) Subject: [Linux-cluster] Strange error returned by openais In-Reply-To: <4B8D6E61.50308@gmail.com> References: <4B8D6E61.50308@gmail.com> Message-ID: <502e5cced35476572445b660278ce9fc.squirrel@flerp.net> > Hi all, > > I am trying to setup a rh5.4 cluster with only two nodes, but I can't. > Under > /var/log/messages I can see a lot of errors like these: > > These nodes have two network interfaces, one on the same network for > cluster > operation and another on different subnet. Like this: > > Node01: 172.16.1.1 (eth0) and 192.168.35.1 (eth1) > Node02: 172.16.1.2 (eth0) and 172.26.50.1 (eth1) > > Default gateways point to 192.168.35.20 in node01 and on node02 to > 172.26.50.30 > ... maybe this is the problem?? > > I have put ip routing rules on both nodes but problem continues ... How > can I fix > this?? I've had exactly the same errors, and eventually found what was wrong. The problem seems to be the vlans, switches which block the multicast traffic. For now I'm using a crosscable. So, check with the network engineers, they should be able to assist you, but you can convince them they're blocking you using the crosscable. regards, Martijn From carlopmart at gmail.com Wed Mar 3 09:02:24 2010 From: carlopmart at gmail.com (carlopmart) Date: Wed, 03 Mar 2010 10:02:24 +0100 Subject: [Linux-cluster] Strange error returned by openais In-Reply-To: <502e5cced35476572445b660278ce9fc.squirrel@flerp.net> References: <4B8D6E61.50308@gmail.com> <502e5cced35476572445b660278ce9fc.squirrel@flerp.net> Message-ID: <4B8E25A0.1080005@gmail.com> martijn at tenheuvel.net wrote: >> Hi all, >> >> I am trying to setup a rh5.4 cluster with only two nodes, but I can't. >> Under >> /var/log/messages I can see a lot of errors like these: >> >> These nodes have two network interfaces, one on the same network for >> cluster >> operation and another on different subnet. Like this: >> >> Node01: 172.16.1.1 (eth0) and 192.168.35.1 (eth1) >> Node02: 172.16.1.2 (eth0) and 172.26.50.1 (eth1) >> >> Default gateways point to 192.168.35.20 in node01 and on node02 to >> 172.26.50.30 >> ... maybe this is the problem?? >> >> I have put ip routing rules on both nodes but problem continues ... How >> can I fix >> this?? > > I've had exactly the same errors, and eventually found what was wrong. > The problem seems to be the vlans, switches which block the multicast > traffic. For now I'm using a crosscable. > > So, check with the network engineers, they should be able to assist you, > but you can convince them they're blocking you using the crosscable. > > regards, > Martijn > > > Maybe you are right Martijn. I have copied manually cluster.conf from node02 to node01 and all works ok (node01 joins to cluster). But If mutlicast is the problem, why node01 joins to cluster if cluster.conf it is at same version than on node02?? My problem only occurs when cluster.conf version is different between nodes ... > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- CL Martinez carlopmart {at} gmail {d0t} com From ccaulfie at redhat.com Wed Mar 3 09:28:23 2010 From: ccaulfie at redhat.com (Christine Caulfield) Date: Wed, 03 Mar 2010 09:28:23 +0000 Subject: [Linux-cluster] Strange error returned by openais In-Reply-To: <4B8E25A0.1080005@gmail.com> References: <4B8D6E61.50308@gmail.com> <502e5cced35476572445b660278ce9fc.squirrel@flerp.net> <4B8E25A0.1080005@gmail.com> Message-ID: <4B8E2BB7.1090807@redhat.com> On 03/03/10 09:02, carlopmart wrote: > martijn at tenheuvel.net wrote: >>> Hi all, >>> >>> I am trying to setup a rh5.4 cluster with only two nodes, but I can't. >>> Under >>> /var/log/messages I can see a lot of errors like these: >>> >>> These nodes have two network interfaces, one on the same network for >>> cluster >>> operation and another on different subnet. Like this: >>> >>> Node01: 172.16.1.1 (eth0) and 192.168.35.1 (eth1) >>> Node02: 172.16.1.2 (eth0) and 172.26.50.1 (eth1) >>> >>> Default gateways point to 192.168.35.20 in node01 and on node02 to >>> 172.26.50.30 >>> ... maybe this is the problem?? >>> >>> I have put ip routing rules on both nodes but problem continues ... How >>> can I fix >>> this?? >> >> I've had exactly the same errors, and eventually found what was wrong. >> The problem seems to be the vlans, switches which block the multicast >> traffic. For now I'm using a crosscable. >> >> So, check with the network engineers, they should be able to assist you, >> but you can convince them they're blocking you using the crosscable. >> >> regards, >> Martijn >> >> >> > > Maybe you are right Martijn. I have copied manually cluster.conf from > node02 to node01 and all works ok (node01 joins to cluster). But If > mutlicast is the problem, why node01 joins to cluster if cluster.conf it > is at same version than on node02?? > > My problem only occurs when cluster.conf version is different between > nodes ... Well, that's exactly your problem! cman expects the cluster.conf to be the same version on all nodes. ccsd is meant to synchronise these in RHEL5 but it has problems with a two node cluster where quorum cannot be established. What you need to do is either use two_node="1" mode in cluster.conf or use a quorum disk to maintain quorum while a single node is up. Chrissie From carlopmart at gmail.com Wed Mar 3 09:33:49 2010 From: carlopmart at gmail.com (carlopmart) Date: Wed, 03 Mar 2010 10:33:49 +0100 Subject: [Linux-cluster] Strange error returned by openais In-Reply-To: <4B8E2BB7.1090807@redhat.com> References: <4B8D6E61.50308@gmail.com> <502e5cced35476572445b660278ce9fc.squirrel@flerp.net> <4B8E25A0.1080005@gmail.com> <4B8E2BB7.1090807@redhat.com> Message-ID: <4B8E2CFD.6070707@gmail.com> Christine Caulfield wrote: > On 03/03/10 09:02, carlopmart wrote: >> martijn at tenheuvel.net wrote: >>>> Hi all, >>>> >>>> I am trying to setup a rh5.4 cluster with only two nodes, but I can't. >>>> Under >>>> /var/log/messages I can see a lot of errors like these: >>>> >>>> These nodes have two network interfaces, one on the same network for >>>> cluster >>>> operation and another on different subnet. Like this: >>>> >>>> Node01: 172.16.1.1 (eth0) and 192.168.35.1 (eth1) >>>> Node02: 172.16.1.2 (eth0) and 172.26.50.1 (eth1) >>>> >>>> Default gateways point to 192.168.35.20 in node01 and on node02 to >>>> 172.26.50.30 >>>> ... maybe this is the problem?? >>>> >>>> I have put ip routing rules on both nodes but problem continues ... How >>>> can I fix >>>> this?? >>> >>> I've had exactly the same errors, and eventually found what was wrong. >>> The problem seems to be the vlans, switches which block the multicast >>> traffic. For now I'm using a crosscable. >>> >>> So, check with the network engineers, they should be able to assist you, >>> but you can convince them they're blocking you using the crosscable. >>> >>> regards, >>> Martijn >>> >>> >>> >> >> Maybe you are right Martijn. I have copied manually cluster.conf from >> node02 to node01 and all works ok (node01 joins to cluster). But If >> mutlicast is the problem, why node01 joins to cluster if cluster.conf it >> is at same version than on node02?? >> >> My problem only occurs when cluster.conf version is different between >> nodes ... > > > Well, that's exactly your problem! cman expects the cluster.conf to be > the same version on all nodes. ccsd is meant to synchronise these in > RHEL5 but it has problems with a two node cluster where quorum cannot be > established. > > What you need to do is either use two_node="1" mode in cluster.conf or > use a quorum disk to maintain quorum while a single node is up. > > Chrissie > But I am using two_node=1 on my cluster.conf. Here it is: I have another two-node cluster configured like this (except on these nodes they have only one interface) and all works ok. When I make changes in cluster.conf on one node is replicated automatically on the other ... Why doesn't occurs the same on this two-node cluster?? Thanks. -- CL Martinez carlopmart {at} gmail {d0t} com From bernardchew at gmail.com Wed Mar 3 09:53:42 2010 From: bernardchew at gmail.com (Bernard Chew) Date: Wed, 3 Mar 2010 17:53:42 +0800 Subject: [Linux-cluster] Shared storage across clustered VMs? In-Reply-To: <4b8e192c.9298cc0a.61b6.0a24@mx.google.com> References: <201003030828.21320.xavier.montagutelli@unilim.fr> <4b8e192c.9298cc0a.61b6.0a24@mx.google.com> Message-ID: <95994e3c1003030153p74433903te9c00a94edb83e3d@mail.gmail.com> > On Wed, Mar 3, 2010 at 4:09 PM, Kit Gerrits wrote: > > Might it be a good idea to stick them in the same cluster, but with > different failure domains? > That way, chances are higher of staying quorate. > > Just a thought... > > Kit > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Xavier Montagutelli > Sent: woensdag 3 maart 2010 8:28 > To: linux clustering > Subject: Re: [Linux-cluster] Shared storage across clustered VMs? > > On Tuesday 02 March 2010 23:50:26 Jeff Karpinski wrote: >> I've got a 4 node cluster back-ended with iSCSI storage that's happily >> running 20 or so VMs. I now have a request to present some shared >> storage across several of the VMs and am wondering what's the best way >> to accomplish this. GFS2? Can VMs even communicate back with the >> cluster for lock_dlm to work? >> >> Interested in how others have skinned this cat... > > I don't know if GFS2 is the best way to share data among your VMs. > > But if you go for GFS2, I suppose the clusters should be different : one > cluster for the hosts, and one cluster for the VMs having a shared disk with > GFS2. They have different purposes, you should not mix them. > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > Hi, I read that you should never mix domU and dom0 nodes in the same cluster due to quorum problems. Regards, Bernard From sklemer at gmail.com Wed Mar 3 11:05:49 2010 From: sklemer at gmail.com (=?UTF-8?B?16nXnNeV150g16fXnNee16g=?=) Date: Wed, 3 Mar 2010 13:05:49 +0200 Subject: [Linux-cluster] fencing issue. In-Reply-To: <8786b91c1003022022i6035e3f2gdab67460a5a5615a@mail.gmail.com> References: <2746211a1003020910p12b21c3ds27cbb38baf84bd00@mail.gmail.com> <8786b91c1003022022i6035e3f2gdab67460a5a5615a@mail.gmail.com> Message-ID: <2746211a1003030305w51836ecy8f64e611bfacf610@mail.gmail.com> Hi. I am runnig fence_ipmilan on iDRAC6. Regards Shalom On Wed, Mar 3, 2010 at 6:22 AM, Rajagopal Swaminathan < raju.rajsand at gmail.com> wrote: > Greetings, > > On Tue, Mar 2, 2010 at 10:40 PM, ???? ???? wrote: > > Hello. > > > > Everything is working fine. We did some failover tests, all tests passed > ( > > nic , fiber , reboot ). > > > > The only test which failed was - pulling out power cable from one node. > > > > > How do you do the fencing? in-band (ILO, RSA, ALOM, DRAC) or > out-of-band (Power fencing) > > Regards, > > Rajagopal > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sklemer at gmail.com Wed Mar 3 11:10:29 2010 From: sklemer at gmail.com (=?UTF-8?B?16nXnNeV150g16fXnNee16g=?=) Date: Wed, 3 Mar 2010 13:10:29 +0200 Subject: [Linux-cluster] fencing issue - with attach logs&conf In-Reply-To: References: <2746211a1003020917q6154faeese3f95e50c785264f@mail.gmail.com> Message-ID: <2746211a1003030310q2534b8a4la1a82266e33f8a07@mail.gmail.com> Hi. I got 2 power supplies. But if someone by mistake pull the power cables , is that mean That the services will not failover ?? Regards Shalom. On Wed, Mar 3, 2010 at 1:15 AM, Georgi Stanojevski wrote: > On Tue, Mar 2, 2010 at 6:17 PM, ???? ???? wrote: > >> >> The only test which failed was - pulling out power cable from one node. >> >> The other system try in loop fencing the failed node & services not >> relocated. >> >> >> > As I understand this is by design. > > When you pull out the power cables you shutdown the fencing device on that > node and the remaining node can't successfully fence it. > > The nodes should have redundant power supplies, to avoid single point of > failure. > > -- > Glisha > http:// > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From corey.kovacs at gmail.com Wed Mar 3 11:19:20 2010 From: corey.kovacs at gmail.com (Corey Kovacs) Date: Wed, 3 Mar 2010 11:19:20 +0000 Subject: [Linux-cluster] fencing issue. In-Reply-To: <2746211a1003030305w51836ecy8f64e611bfacf610@mail.gmail.com> References: <2746211a1003020910p12b21c3ds27cbb38baf84bd00@mail.gmail.com> <8786b91c1003022022i6035e3f2gdab67460a5a5615a@mail.gmail.com> <2746211a1003030305w51836ecy8f64e611bfacf610@mail.gmail.com> Message-ID: <7d6e8da41003030319r655b25bav4d511978d5644fdd@mail.gmail.com> That won't work then since pulling the power cable effectively disables the drac port. You need some out of band controller for that type of fencing to work. I use APC units, others will prefer different units. It's the same problem as with an iLO. Actually, you can get into the same problem if your power controllers and nodes are in different racks and you have a rack level failure. Regards Corey On Wed, Mar 3, 2010 at 11:05 AM, ???? ???? wrote: > Hi. > > I am runnig fence_ipmilan on iDRAC6. > > Regards > > Shalom > > On Wed, Mar 3, 2010 at 6:22 AM, Rajagopal Swaminathan < > raju.rajsand at gmail.com> wrote: > >> Greetings, >> >> On Tue, Mar 2, 2010 at 10:40 PM, ???? ???? wrote: >> > Hello. >> > >> > Everything is working fine. We did some failover tests, all tests passed >> ( >> > nic , fiber , reboot ). >> > >> > The only test which failed was - pulling out power cable from one node. >> > >> >> >> How do you do the fencing? in-band (ILO, RSA, ALOM, DRAC) or >> out-of-band (Power fencing) >> >> Regards, >> >> Rajagopal >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brem.belguebli at gmail.com Wed Mar 3 13:23:53 2010 From: brem.belguebli at gmail.com (brem belguebli) Date: Wed, 3 Mar 2010 14:23:53 +0100 Subject: [Linux-cluster] strange cluster behavior In-Reply-To: <201003030816.45302.xavier.montagutelli@unilim.fr> References: <1267582310.2226.104.camel@localhost> <201003030816.45302.xavier.montagutelli@unilim.fr> Message-ID: <29ae894c1003030523q2631d85i1d2459a0cba2827d@mail.gmail.com> Hi Xavier, 2010/3/3 Xavier Montagutelli : > On Wednesday 03 March 2010 03:11:50 brem belguebli wrote: >> Hi, >> >> I experienced a strange cluster behavior that I couldn't explain. >> >> I have a 4 nodes Rhel 5.4 cluster (node1, node2, node3 and node4). >> >> Node1 and node2 are connected to an ethernet switch (sw1), node3 and >> node4 are connected to another switch (sw2). The 4 nodes are on the same >> Vlan. >> >> sw1 and sw2 are connected thru a couple of core switches, and the nodes >> Vlan is well propagated across the network that I just described. >> >> Latency between node1 and node4 (on 2 different switches) doesn't exceed >> 0.3 ms. >> >> The cluster is normally configured with a iscsi quorum device located on >> another switch. >> >> I wanted to check how it would behave when quorum disk is not active >> (removed from cluster.conf) if a member node came to get isolated (link >> up but not on the right vlan). >> >> Node3 is the one I played with. >> >> The fence_device for this node is intentionally misconfigured to be able >> to follow on this node console what happens. >> >> When changing the vlan membership of node3, results are as expected, the >> 3 remaining nodes see it come offline after totem timer expiry, and >> node1 (lowest node id) starts trying to fence node3 (without success as >> intentionally misconfigured). >> >> Node3 sees itself the only member of the cluster which is inquorate. >> Coherent as it became a single node parition. >> >> When putting back node3 vlan conf to the right value, things go bad. > > (My two cents) > > You just put it back in the good VLAN, without restarting the host ? Yeap, this it what I wanted to test. > > I did this kind of test (under RH 5.3), and things always get bad if a node > supposed to be fenced is not really fenced and comes back. Perhaps this is an > intended behaviour to prevent "split brain" cases (even at the cost of the > whole cluster going down) ? Or perhaps it depends how your misconfigured fence > device behaves (does it give an exit status ? What exit status does it send > ?). When node3 comes back with the same membership state as previously, node1 (2 and 4) kill node3 (instruct cman to exit) because of this previous state being the same as the new one. The problem is that, in the log, node1 and node2 at the very same time loose the quorum ( clurgmgrd[10469]: #1: Quorum Dissolved) and go offline. This is what I cannot explain. There is no split brain thing involved here as I expected node1 (and why not all the other nodes) to instruct node3 cman to exit and things could continue to run (may be without relocating node3 services as I couldn't get fenced). Concerning the fencing, it may return a non zero value as I can see in node1 logs that it is looping trying to fence node3. > >> >> Node1, 2 and 4 instruct node3 cman to kill itself as it did re appear >> with an already existing status. Why not. >> >> Node1 and node2 then say then the quorum is dissolved and see themselves >> offline (????), node3 offline and node4 online. >> >> Node4 sees itself online but cluster inquorate as we also lost node1 and >> node2. >> >> I thought about potential multicast problems, but it behaves the same >> way when cman is configured to broadcast. >> >> The same test run with qdisk enabled is behaving normally, when node3 >> gets back to network it gets automatically rebooted (thx to qdisk), the >> cluster remains stable. Concerning the fact that it works when qdisk is enabled may be a "side effect" as I use a iscsi LUN accessed through the LAN interface, qdisk being a "heartbeat vector" node3 not being able to write to the LUN may make things more stable. I should give a try with a SAN LUN used as qdisk and see how it behaves. >> >> Any idea why node1 and node2 go bad when node3 is back ? >> >> Thanks >> >> Brem >> >> >> >> >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > -- > Xavier Montagutelli ? ? ? ? ? ? ? ? ? ? ?Tel : +33 (0)5 55 45 77 20 > Service Commun Informatique ? ? ? ? ? ? ?Fax : +33 (0)5 55 45 75 95 > Universite de Limoges > 123, avenue Albert Thomas > 87060 Limoges cedex > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From brem.belguebli at gmail.com Wed Mar 3 13:25:44 2010 From: brem.belguebli at gmail.com (brem belguebli) Date: Wed, 3 Mar 2010 14:25:44 +0100 Subject: [Linux-cluster] fencing issue - with attach logs&conf In-Reply-To: <2746211a1003030310q2534b8a4la1a82266e33f8a07@mail.gmail.com> References: <2746211a1003020917q6154faeese3f95e50c785264f@mail.gmail.com> <2746211a1003030310q2534b8a4la1a82266e33f8a07@mail.gmail.com> Message-ID: <29ae894c1003030525x7b28ef28tb0ed7f9fa3e49df7@mail.gmail.com> Exactly. 2010/3/3 ???? ???? : > Hi. > > I got 2 power supplies. But if someone by mistake pull the power cables , is > that mean > > That the services will not failover ?? > > Regards > > Shalom. > > On Wed, Mar 3, 2010 at 1:15 AM, Georgi Stanojevski wrote: >> >> On Tue, Mar 2, 2010 at 6:17 PM, ???? ???? wrote: >> >>> >>> The only test which failed was - pulling out power cable from one node. >>> >>> The other system try in loop fencing the failed node & services not >>> relocated. >>> >> >> As I understand this is by design. >> >> When you pull out the power cables you shutdown the fencing device on that >> node and the remaining node can't successfully fence it. >> >> The nodes should have redundant power supplies, to avoid single point of >> failure. >> >> -- >> Glisha >> http:// >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From gianluca.cecchi at gmail.com Wed Mar 3 17:01:21 2010 From: gianluca.cecchi at gmail.com (Gianluca Cecchi) Date: Wed, 3 Mar 2010 18:01:21 +0100 Subject: [Linux-cluster] Possible bug in rhel5 for nested HA-LVM resources? Message-ID: <561c252c1003030901j4fe6274eicf3e0a8295410a6a@mail.gmail.com> Hello, my problem begins from this need: - having a rh el 5.4 cluster with 2 nodes where I have HA-LVM in place and some lvm/fs pairs resources componing one service I want to add a new lvm/fs to the cluster, without disrupting the running service. My already configured and running lvm/mountpoints are: /dev/mapper/VG_TEST_APPL-LV_TEST_APPL 5.0G 139M 4.6G 3% /appl_db1 /dev/mapper/VG_TEST_DATA-LV_TEST_DATA 5.0G 139M 4.6G 3% /oradata/TEST The new desired mount point is to be put under /oradata/TEST/newtemp Current extract of cluster.conf is But this service is checked every 30 seconds ... Do I need to configure something more?? Thanks. -- CL Martinez carlopmart {at} gmail {d0t} com From sklemer at gmail.com Mon Mar 29 05:48:59 2010 From: sklemer at gmail.com (=?UTF-8?B?16nXnNeV150g16fXnNee16g=?=) Date: Mon, 29 Mar 2010 08:48:59 +0300 Subject: [Linux-cluster] Action status doesn't works on Rhel5.4?? In-Reply-To: <4BAFB705.9030008@gmail.com> References: <4BAFB705.9030008@gmail.com> Message-ID: <2746211a1003282248x357c2ba0v8ebd4108c7e1300b@mail.gmail.com> Hi. look at /usr/share/cluster/script.sh each resource got his own status interval check. Shalom. On Sun, Mar 28, 2010 at 11:07 PM, carlopmart wrote: > Hi all, > > I have configured some cluster resources assigning a custom time for > action status, but doesn't works. For example: > > recovery="relocate"> > > > > > > > > But this service is checked every 30 seconds ... Do I need to configure > something more?? > > Thanks. > > -- > CL Martinez > carlopmart {at} gmail {d0t} com > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlopmart at gmail.com Mon Mar 29 08:14:49 2010 From: carlopmart at gmail.com (carlopmart) Date: Mon, 29 Mar 2010 10:14:49 +0200 Subject: [Linux-cluster] Action status doesn't works on Rhel5.4?? In-Reply-To: <2746211a1003282248x357c2ba0v8ebd4108c7e1300b@mail.gmail.com> References: <4BAFB705.9030008@gmail.com> <2746211a1003282248x357c2ba0v8ebd4108c7e1300b@mail.gmail.com> Message-ID: <4BB06179.5090201@gmail.com> oops. Maybe the problem is that I need to put seconds like this: interval="1s" ... i will try it. wrote: > Hi. > > look at /usr/share/cluster/script.sh > > each resource got his own status interval check. > > Shalom. > > On Sun, Mar 28, 2010 at 11:07 PM, carlopmart > wrote: > > Hi all, > > I have configured some cluster resources assigning a custom time > for action status, but doesn't works. For example: > > recovery="relocate"> > > > > > > > > But this service is checked every 30 seconds ... Do I need to > configure something more?? > > Thanks. > > -- > CL Martinez > carlopmart {at} gmail {d0t} com > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > ------------------------------------------------------------------------ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- CL Martinez carlopmart {at} gmail {d0t} com From carlopmart at gmail.com Mon Mar 29 08:20:40 2010 From: carlopmart at gmail.com (carlopmart) Date: Mon, 29 Mar 2010 10:20:40 +0200 Subject: [Linux-cluster] Action status doesn't works on Rhel5.4?? In-Reply-To: <4BB06179.5090201@gmail.com> References: <4BAFB705.9030008@gmail.com> <2746211a1003282248x357c2ba0v8ebd4108c7e1300b@mail.gmail.com> <4BB06179.5090201@gmail.com> Message-ID: <4BB062D8.7010104@gmail.com> Ok, I have tried it. I have changed this: by: But my script is checked every 30 seconds ... Any ideas? carlopmart wrote: > oops. Maybe the problem is that I need to put seconds like this: > interval="1s" ... i will try it. > > wrote: >> Hi. >> >> look at /usr/share/cluster/script.sh >> >> each resource got his own status interval check. >> >> Shalom. >> >> On Sun, Mar 28, 2010 at 11:07 PM, carlopmart > > wrote: >> >> Hi all, >> >> I have configured some cluster resources assigning a custom time >> for action status, but doesn't works. For example: >> >> > recovery="relocate"> >> >> >> >> >> >> >> >> But this service is checked every 30 seconds ... Do I need to >> configure something more?? >> >> Thanks. >> >> -- CL Martinez >> carlopmart {at} gmail {d0t} com >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> >> ------------------------------------------------------------------------ >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- CL Martinez carlopmart {at} gmail {d0t} com From swhiteho at redhat.com Mon Mar 29 08:36:30 2010 From: swhiteho at redhat.com (Steven Whitehouse) Date: Mon, 29 Mar 2010 09:36:30 +0100 Subject: [Linux-cluster] dump(8) for GFS2 In-Reply-To: <036B68E61A28CA49AC2767596576CD596905CFF650@GVW1113EXC.americas.hpqcorp.net> References: <036B68E61A28CA49AC2767596576CD596905CFF177@GVW1113EXC.americas.hpqcorp.net> <1269609637.2379.1.camel@localhost> <036B68E61A28CA49AC2767596576CD596905CFF650@GVW1113EXC.americas.hpqcorp.net> Message-ID: <1269851790.2504.12.camel@localhost> Hi, On Sun, 2010-03-28 at 02:58 +0000, Jankowski, Chris wrote: > Steve, > > dump(8) cetrainly does (not* work on RHEL 5.4 and GFS2. It complains about wrong information in the superblock. > > It is a very old code. I very much doubt that it would use FIEMAP that is a realtively recent development. > > Thanks and regards, > > Chris > Ah, so it seems that it might be trying to read the sb directly. That is generally a bad idea for GFS2 since there is unlikely to be any useful information in the sb, and reading any other disk block on the fs requires locking. Either patching dump to use FIEMAP to map sparse files, or writing a replacement sounds like the right way to go. I don't know enough about the current dump code to comment on which would be easier, Steve. > -----Original Message----- > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse > Sent: Saturday, 27 March 2010 00:21 > To: linux clustering > Subject: Re: [Linux-cluster] dump(8) for GFS2 > > Hi, > > On Fri, 2010-03-26 at 02:48 +0000, Jankowski, Chris wrote: > > Hi, > > > > Question: > > --------- > > Are there any plans to develop a backup utility working on the same principles as dump(8) does for ext3fs? This means getting the backup done by walking the block structure contained in the inodes instead of just reading the file the way tar(1), cpio(1) and others do it. > > > Does dump use FIEMAP? If so it should "just work" on recent GFS2, > > Steve. > > > > I need dump(8) to deal with a specific problem created by the customer's application. A library used by the application has a bug which demonstrates itself in an upredictable creation of huge sparse files. For example the application may create a sparse file of 5TB with only a few kB of data in it. There may be tens of those files created in the database. GNU tar handles sparse files correctly and will recreate them as sparse files too. This is fine. But it still needs to read all of those nulls and that is done at a rate of about 1.5TB per hour on this system. With 100+ TB of the apparent size of all of those sparse filesmy bakup would take about 3 days to complete. > > > > By comparison, dump(8) would deal with this situation perfectly well. It know the inodes of the file, will follow the few that exist in a sparse file and back up the data. It does not have to read through the tons of nulls happily delivered by the OS, as tar(1) does. > > > > Regards, > > > > Chris > > > > > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From swhiteho at redhat.com Mon Mar 29 08:41:29 2010 From: swhiteho at redhat.com (Steven Whitehouse) Date: Mon, 29 Mar 2010 09:41:29 +0100 Subject: [Linux-cluster] GFS2 - monitoring the rate of Posix lock operations In-Reply-To: <036B68E61A28CA49AC2767596576CD596905CFF653@GVW1113EXC.americas.hpqcorp.net> References: <036B68E61A28CA49AC2767596576CD596905CFF147@GVW1113EXC.americas.hpqcorp.net> <1269609947.2379.5.camel@localhost> <036B68E61A28CA49AC2767596576CD596905CFF653@GVW1113EXC.americas.hpqcorp.net> Message-ID: <1269852089.2504.17.camel@localhost> On Sun, 2010-03-28 at 02:32 +0000, Jankowski, Chris wrote: > Steve, > > Q2: > >>> Are you sure that the workload isn't causing too many cache invalidations due to sharing files/directories between nodes? This is the most usual cause of poor performance. > > The other node is completely idle and kept that way by design. Users are connecting through an IP alias managed by the appplication service. Application administrators also log in through the alias to do their maintenance work. In the case of this particular test I manually listed what is running where. I am very concious of the fact that accesses from multiple nodes invalidate local in-memory caching. > > Q3: > >>> Have you used the noatime mount option? If you can use it, its highly recommended. Also turn off selinux if that is running on the GFS2 filesystem. > > The filesystem is mounted with noatime and no nodiratime options. SELinux is disabled. > nodiratime isn't supported, noatime is enough. > Q4: > >>>Potentially there might be. I don't know enough about the application to say, but it depends on how the workload can be arranged, > > The application runs on one node at a time. It has to, as it uses shared memory. The application uses a database of indexed files. There are thousands of them. Also, it uses standard UNIX flile locking and range locking. > > What else can I do to minimise the GFS2 locking overhead in this asymetrical configuration. > You can use localflocks on each node provided you never access any of the locked files from more than once node at once (which may be true depending on how the failover is designed). Then you will get local fcntl lock performance at the expense of cluster fcntl locks. > Q5: > Is this the case that when gfs_controld gets to 100% of one coe DPU usage then this is a hard limit on the number of Posix locks taken. Is there only one gfs_lockd daemon servicng all GFS2 filesystems or are they run on a per filesystems basis? In the latter case I would have thought that breaking the one filesystem that I have into several may help. Would it not? > > Thanks and regards, > > Chris > Assuming that you have a version in which gfs_controld takes care of the locking (newer GFS2 send the locks via dlm_controld) then yes, that will provide a hard limit on the rate at which locks can be acquired/dropped, Steve. From kkovachev at varna.net Mon Mar 29 09:56:23 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Mon, 29 Mar 2010 12:56:23 +0300 Subject: [Linux-cluster] =?utf-8?q?Action_status_doesn=27t_works_on_Rhel5?= =?utf-8?b?LjQ/Pw==?= In-Reply-To: <4BB06179.5090201@gmail.com> References: <4BAFB705.9030008@gmail.com> <2746211a1003282248x357c2ba0v8ebd4108c7e1300b@mail.gmail.com> <4BB06179.5090201@gmail.com> Message-ID: On Mon, 29 Mar 2010 10:14:49 +0200, carlopmart wrote: > oops. Maybe the problem is that I need to put seconds like this: > interval="1s" ... i > will try it. > the (missing) seconds are not a problem, but you need to restart rgmanager for it to reread the values > wrote: >> Hi. >> >> look at /usr/share/cluster/script.sh >> >> each resource got his own status interval check. >> >> Shalom. >> >> On Sun, Mar 28, 2010 at 11:07 PM, carlopmart > > wrote: >> >> Hi all, >> >> I have configured some cluster resources assigning a custom time >> for action status, but doesn't works. For example: >> >> > recovery="relocate"> >> >> >> >> >> >> >> >> But this service is checked every 30 seconds ... Do I need to >> configure something more?? >> >> Thanks. >> >> -- >> CL Martinez >> carlopmart {at} gmail {d0t} com >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> >> ------------------------------------------------------------------------ >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster From sklemer at gmail.com Mon Mar 29 09:51:29 2010 From: sklemer at gmail.com (=?UTF-8?B?16nXnNeV150g16fXnNee16g=?=) Date: Mon, 29 Mar 2010 12:51:29 +0300 Subject: [Linux-cluster] Action status doesn't works on Rhel5.4?? In-Reply-To: <4BB062D8.7010104@gmail.com> References: <4BAFB705.9030008@gmail.com> <2746211a1003282248x357c2ba0v8ebd4108c7e1300b@mail.gmail.com> <4BB06179.5090201@gmail.com> <4BB062D8.7010104@gmail.com> Message-ID: <2746211a1003290251g76c48c1fu293f792d5f93b0c@mail.gmail.com> Hi. I think that if you want to overwrite the shells interval you need to add "depth": Customizing Actions with Multiple Depths Some agents implement multiple monitoring or status depths with different timeouts in order to perform different levels of checking. For example, a '0' depth for the status check of an IP address may be a simple existence check, a '10' depth might include ethernet link checking, and a '20' depth may include pinging an upstream router. If you wish, you may override all depths of a given action by using an asterisk as the depth in cluster.conf. For example: Note that doing this causes only the *highest* or *most invasive* checks to be performed. On Mon, Mar 29, 2010 at 11:20 AM, carlopmart wrote: > Ok, I have tried it. I have changed this: > > > > > by: > > > > But my script is checked every 30 seconds ... > > Any ideas? > > > > carlopmart wrote: > >> oops. Maybe the problem is that I need to put seconds like this: >> interval="1s" ... i will try it. >> >> wrote: >> >>> Hi. >>> look at /usr/share/cluster/script.sh >>> each resource got his own status interval check. >>> Shalom. >>> >>> On Sun, Mar 28, 2010 at 11:07 PM, carlopmart >> carlopmart at gmail.com>> wrote: >>> >>> Hi all, >>> >>> I have configured some cluster resources assigning a custom time >>> for action status, but doesn't works. For example: >>> >>> >> recovery="relocate"> >>> >>> >>> >>> >>> >>> >>> >>> But this service is checked every 30 seconds ... Do I need to >>> configure something more?? >>> >>> Thanks. >>> >>> -- CL Martinez >>> carlopmart {at} gmail {d0t} com >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >> >> >> > > -- > CL Martinez > carlopmart {at} gmail {d0t} com > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markus.wolfgart at dlr.de Mon Mar 29 10:49:19 2010 From: markus.wolfgart at dlr.de (Markus Wolfgart) Date: Mon, 29 Mar 2010 12:49:19 +0200 Subject: [Linux-cluster] gfs2-utils source for recovery purpose of a corrupt gfs2 partition Message-ID: <4BB085AF.90400@dlr.de> Hallo Cluster and GFS Experts, I'm a new subscriber of this mailing list and appologise in the case my posting is offtopic. I'm looking for help concerning a corrupt gfs2 file system which could not be recovered by me by fsck.gfs2 (Ver. 3.0.9) due to to less less physical memory (4GB) eaven if increasing it by a additional swap space (now about 35GB). I would like to parse a image created of the lost fs (the first 6TB) with the code provided in the new gfs2-utils release. Due to this circumstance I hope to find in this mailing list some hints concerning an automated step by step recovery of lost data. Many Thanks in advance for your help Markus PS.: some infos concerning the damaged fs: ******************************************************* Markus Wolfgart DLR Oberpfaffenhofen German Remote Sensing Data Center . . . e-mail: markus.wolfgart at dlr.de ********************************************************** gfs2_edit -p sb /dev/oa_vg_storage1/oa_lv_storage1 ========================================= Block #16 (0x10) of 3097745408 (0xB8A3D800) (superblock) Superblock: mh_magic 0x01161970(hex) mh_type 1 0x1 mh_format 100 0x64 sb_fs_format 1801 0x709 sb_multihost_format 1900 0x76c sb_bsize 4096 0x1000 sb_bsize_shift 12 0xc master dir: 2 0x2 addr: 51 0x33 root dir : 1 0x1 addr: 50 0x32 sb_lockproto fsck_dlm sb_locktable oa-dp:oa_gfs1 sb_uuid 00000000-0000-0000-0000-000000000000 The superblock has 2 directories 1. (1). 50 (0x32): Dir root 2. (2). 51 (0x33): Dir master ------------------------------------------------------ gfs2_edit -p root /dev/oa_vg_storage1/oa_lv_storage1 ========================================== Block #50 (0x32) of 3097745408 (0xB8A3D800) (disk inode) --------------- Root directory ------------------ Dinode: mh_magic 0x01161970(hex) mh_type 4 0x4 mh_format 400 0x190 no_formal_ino 1 0x1 no_addr 50 0x32 di_mode 040777(decimal) di_uid 10069 0x2755 di_gid 10060 0x274c di_nlink 14 0xe di_size 3864 0xf18 di_blocks 1 0x1 di_atime 1269603308 0x4bac9bec di_mtime 1269429185 0x4ba9f3c1 di_ctime 1269429185 0x4ba9f3c1 di_major 0 0x0 di_minor 0 0x0 di_goal_meta 50 0x32 di_goal_data 50 0x32 di_flags 0x00000001(hex) di_payload_format 1200 0x4b0 di_height 0 0x0 di_depth 0 0x0 di_entries 19 0x13 di_eattr 0 0x0 This directory block contains 19 directory entries. 1. (1). 50 (0x32): Dir . 2. (2). 50 (0x32): Dir .. 3. (3). 881038445 (0x3483946d): Dir x <== masked by me ;-) 4. (4). 929563055 (0x376801af): Dir x 5. (5). 1623975855 (0x60cbe7af): File x 6. (6). 881038473 (0x34839489): Dir x 7. (7). 881042981 (0x3483a625): File x 8. (8). 881043493 (0x3483a825): File x 9. (9). 1028392434 (0x3d4c05f2): Dir lost+found 10. (10). 1028392437 (0x3d4c05f5): Dir x 11. (11). 1028394418 (0x3d4c0db2): Dir x 12. (12). 1028518858 (0x3d4df3ca): Dir x 13. (13). 1028518874 (0x3d4df3da): Dir x 14. (14). 1028518886 (0x3d4df3e6): Dir x 15. (15). 1058257256 (0x3f13b968): Symlink x 16. (16). 1072794841 (0x3ff18cd9): Dir x 17. (17). 367165514 (0x15e2804a): Dir x 18. (18). 1623975860 (0x60cbe7b4): File x 19. (19). 1667192704 (0x635f5780): Dir markus ------------------------------------------------------ gfs2_edit -p master /dev/oa_vg_storage1/oa_lv_storage1 ============================================ Block #51 (0x33) of 3097745408 (0xB8A3D800) (disk inode) -------------- Master directory ----------------- Dinode: mh_magic 0x01161970(hex) mh_type 4 0x4 mh_format 400 0x190 no_formal_ino 2 0x2 no_addr 51 0x33 di_mode 040755(decimal) di_uid 0 0x0 di_gid 0 0x0 di_nlink 4 0x4 di_size 3864 0xf18 di_blocks 1 0x1 di_atime 1221815862 0x48d36e36 di_mtime 1221815862 0x48d36e36 di_ctime 1221815862 0x48d36e36 di_major 0 0x0 di_minor 0 0x0 di_goal_meta 51 0x33 di_goal_data 51 0x33 di_flags 0x00000201(hex) di_payload_format 1200 0x4b0 di_height 0 0x0 di_depth 0 0x0 di_entries 8 0x8 di_eattr 0 0x0 This directory block contains 8 directory entries. 1. (1). 51 (0x33): Dir . 2. (2). 51 (0x33): Dir .. 3. (3). 52 (0x34): Dir jindex 4. (4). 262725 (0x40245): Dir per_node 5. (5). 264798 (0x40a5e): File inum 6. (6). 264799 (0x40a5f): File statfs 7. (7). 264800 (0x40a60): File rindex 8. (8). 264940 (0x40aec): File quota ------------------------------------------------------ gfs2_edit -p statfs /dev/oa_vg_storage1/oa_lv_storage1 =========================================== Block #264799 (0x40a5f) of 3097745408 (0xB8A3D800) (disk inode) ---------------- statfs file ------------------- Dinode: mh_magic 0x01161970(hex) mh_type 4 0x4 mh_format 400 0x190 no_formal_ino 38 0x26 no_addr 264799 0x40a5f di_mode 0100600(decimal) di_uid 0 0x0 di_gid 0 0x0 di_nlink 1 0x1 di_size 24 0x18 di_blocks 1 0x1 di_atime 1269421208 0x4ba9d498 di_mtime 1221815862 0x48d36e36 di_ctime 1221815862 0x48d36e36 di_major 0 0x0 di_minor 0 0x0 di_goal_meta 264799 0x40a5f di_goal_data 264799 0x40a5f di_flags 0x00000201(hex) di_payload_format 0 0x0 di_height 0 0x0 di_depth 0 0x0 di_entries 0 0x0 di_eattr 0 0x0 statfs file contents: sc_total 6049365640 0x16891fe88 sc_free 4561998802 0x10fea93d2 sc_dinodes 920663 0xe0c57 ------------------------------------------------------ gfs2_edit -p per_node /dev/oa_vg_storage1/oa_lv_storage1 ============================================== Block #262725 (0x40245) of 3097745408 (0xB8A3D800) (disk inode) --------------- Per-node Dir ------------------- Dinode: mh_magic 0x01161970(hex) mh_type 4 0x4 mh_format 400 0x190 no_formal_ino 12 0xc no_addr 262725 0x40245 di_mode 040700(decimal) di_uid 0 0x0 di_gid 0 0x0 di_nlink 2 0x2 di_size 3864 0xf18 di_blocks 1 0x1 di_atime 1221815862 0x48d36e36 di_mtime 1221815862 0x48d36e36 di_ctime 1221815862 0x48d36e36 di_major 0 0x0 di_minor 0 0x0 di_goal_meta 262725 0x40245 di_goal_data 262725 0x40245 di_flags 0x00000201(hex) di_payload_format 1200 0x4b0 di_height 0 0x0 di_depth 0 0x0 di_entries 26 0x1a di_eattr 0 0x0 This directory block contains 26 directory entries. 1. (1). 262725 (0x40245): Dir . 2. (2). 51 (0x33): Dir .. 3. (3). 262726 (0x40246): File inum_range0 4. (4). 262727 (0x40247): File statfs_change0 5. (5). 262728 (0x40248): File quota_change0 6. (6). 262985 (0x40349): File inum_range1 7. (7). 262986 (0x4034a): File statfs_change1 8. (8). 262987 (0x4034b): File quota_change1 9. (9). 263244 (0x4044c): File inum_range2 10. (10). 263245 (0x4044d): File statfs_change2 11. (11). 263246 (0x4044e): File quota_change2 12. (12). 263503 (0x4054f): File inum_range3 13. (13). 263504 (0x40550): File statfs_change3 14. (14). 263505 (0x40551): File quota_change3 15. (15). 263762 (0x40652): File inum_range4 16. (16). 263763 (0x40653): File statfs_change4 17. (17). 263764 (0x40654): File quota_change4 18. (18). 264021 (0x40755): File inum_range5 19. (19). 264022 (0x40756): File statfs_change5 20. (20). 264023 (0x40757): File quota_change5 21. (21). 264280 (0x40858): File inum_range6 22. (22). 264281 (0x40859): File statfs_change6 23. (23). 264282 (0x4085a): File quota_change6 24. (24). 264539 (0x4095b): File inum_range7 25. (25). 264540 (0x4095c): File statfs_change7 26. (26). 264541 (0x4095d): File quota_change7 ------------------------------------------------------ gfs2_edit -p inum /dev/oa_vg_storage1/oa_lv_storage1 =========================================== Block #264798 (0x40a5e) of 3097745408 (0xB8A3D800) (disk inode) ---------------- Inum file --------------------- Dinode: mh_magic 0x01161970(hex) mh_type 4 0x4 mh_format 400 0x190 no_formal_ino 37 0x25 no_addr 264798 0x40a5e di_mode 0100600(decimal) di_uid 0 0x0 di_gid 0 0x0 di_nlink 1 0x1 di_size 8 0x8 di_blocks 1 0x1 di_atime 1221815862 0x48d36e36 di_mtime 1221815862 0x48d36e36 di_ctime 1221815862 0x48d36e36 di_major 0 0x0 di_minor 0 0x0 di_goal_meta 264798 0x40a5e di_goal_data 264798 0x40a5e di_flags 0x00000201(hex) di_payload_format 0 0x0 di_height 0 0x0 di_depth 0 0x0 di_entries 0 0x0 di_eattr 0 0x0 Next inode num = 4194345 (0x400029) ------------------------------------------------------ gfs2_edit -p rgs /dev/oa_vg_storage1/oa_lv_storage1 > /tmp/dev_oa_vg_storage1_oa_lv_storage1.rgs =============================================================================== bad seek: Invalid argument from parse_rindex:991: block 3098269649 (0xb8abd7d1) Block #RG List of 3097745408 (0xB8A3D800) ------------------ RG List ---------------------- Dinode: mh_magic 0x01161970(hex) mh_type 4 0x4 mh_format 400 0x190 no_formal_ino 2 0x2 no_addr 51 0x33 di_mode 040755(decimal) di_uid 0 0x0 di_gid 0 0x0 di_nlink 4 0x4 di_size 3864 0xf18 di_blocks 1 0x1 di_atime 1221815862 0x48d36e36 di_mtime 1221815862 0x48d36e36 di_ctime 1221815862 0x48d36e36 di_major 0 0x0 di_minor 0 0x0 di_goal_meta 51 0x33 di_goal_data 51 0x33 di_flags 0x00000201(hex) di_payload_format 1200 0x4b0 di_height 0 0x0 di_depth 0 0x0 di_entries 8 0x8 di_eattr 0 0x0 RG index entries found: 11541. RG #0 located at: 17 (0x11) mh_magic 0x01161970(hex) mh_type 2 0x2 mh_format 200 0xc8 rg_flags 0 0x0 rg_free 13 0xd rg_dinodes 891 0x37b RG #1 located at: 529580 (0x814ac) mh_magic 0x01161970(hex) mh_type 2 0x2 mh_format 200 0xc8 rg_flags 0 0x0 rg_free 17 0x11 rg_dinodes 1068 0x42c ... trancated by me ( could be sent on demand by mail if necessary!) RG #5908 located at: 3097221167 (0xb89bd82f) mh_magic 0x01161970(hex) mh_type 2 0x2 mh_format 200 0xc8 rg_flags 0 0x0 rg_free 524208 0x7ffb0 rg_dinodes 0 0x0 RG #5909 located at: 3097745408 (0xb8a3d800) mh_magic 0x00000000(hex) mh_type 0 0x0 mh_format 0 0x0 rg_flags 0 0x0 rg_free 0 0x0 rg_dinodes 0 0x0 RG #5910 located at: 3098269649 (0xb8abd7d1) -- From Chris.Jankowski at hp.com Mon Mar 29 12:15:56 2010 From: Chris.Jankowski at hp.com (Jankowski, Chris) Date: Mon, 29 Mar 2010 12:15:56 +0000 Subject: [Linux-cluster] GFS2 - monitoring the rate of Posix lock operations In-Reply-To: <1269852089.2504.17.camel@localhost> References: <036B68E61A28CA49AC2767596576CD596905CFF147@GVW1113EXC.americas.hpqcorp.net> <1269609947.2379.5.camel@localhost> <036B68E61A28CA49AC2767596576CD596905CFF653@GVW1113EXC.americas.hpqcorp.net> <1269852089.2504.17.camel@localhost> Message-ID: <036B68E61A28CA49AC2767596576CD596906B1622F@GVW1113EXC.americas.hpqcorp.net> Steven, >>>You can use localflocks on each node provided you never access any of the locked files from more than once node at once (which may be true depending on how the failover is designed). Then you will get local fcntl lock performance at the expense of cluster fcntl locks. I could guarantee that only one node will use the filesystem by putting mount/unmount into the start/stop script for the application service. This is the easy part. What I would like to understand is how the GFS2 recovery would look like after failure of the node that had the filesystem mounted. I'd guess that the local locks will be gone with the failed system and there is nothing to recover. The only thing to do would be to replay to transaction log from the failed system. Is this correct? This would work essentially like having a non-cluster filesystem such as ext3fs, but in case of recovery from a node failure doing only transaction log replay instaed of full fsck? Or would fsck still be triggered on the attempt to mount the filesystem on the other node.? Thanks and regards, Chris -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse Sent: Monday, 29 March 2010 19:41 To: linux clustering Subject: Re: [Linux-cluster] GFS2 - monitoring the rate of Posix lock operations On Sun, 2010-03-28 at 02:32 +0000, Jankowski, Chris wrote: > Steve, > > Q2: > >>> Are you sure that the workload isn't causing too many cache invalidations due to sharing files/directories between nodes? This is the most usual cause of poor performance. > > The other node is completely idle and kept that way by design. Users are connecting through an IP alias managed by the appplication service. Application administrators also log in through the alias to do their maintenance work. In the case of this particular test I manually listed what is running where. I am very concious of the fact that accesses from multiple nodes invalidate local in-memory caching. > > Q3: > >>> Have you used the noatime mount option? If you can use it, its highly recommended. Also turn off selinux if that is running on the GFS2 filesystem. > > The filesystem is mounted with noatime and no nodiratime options. SELinux is disabled. > nodiratime isn't supported, noatime is enough. > Q4: > >>>Potentially there might be. I don't know enough about the > >>>application to say, but it depends on how the workload can be > >>>arranged, > > The application runs on one node at a time. It has to, as it uses shared memory. The application uses a database of indexed files. There are thousands of them. Also, it uses standard UNIX flile locking and range locking. > > What else can I do to minimise the GFS2 locking overhead in this asymetrical configuration. > You can use localflocks on each node provided you never access any of the locked files from more than once node at once (which may be true depending on how the failover is designed). Then you will get local fcntl lock performance at the expense of cluster fcntl locks. > Q5: > Is this the case that when gfs_controld gets to 100% of one coe DPU usage then this is a hard limit on the number of Posix locks taken. Is there only one gfs_lockd daemon servicng all GFS2 filesystems or are they run on a per filesystems basis? In the latter case I would have thought that breaking the one filesystem that I have into several may help. Would it not? > > Thanks and regards, > > Chris > Assuming that you have a version in which gfs_controld takes care of the locking (newer GFS2 send the locks via dlm_controld) then yes, that will provide a hard limit on the rate at which locks can be acquired/dropped, Steve. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From swhiteho at redhat.com Mon Mar 29 12:35:46 2010 From: swhiteho at redhat.com (Steven Whitehouse) Date: Mon, 29 Mar 2010 13:35:46 +0100 Subject: [Linux-cluster] GFS2 - monitoring the rate of Posix lock operations In-Reply-To: <036B68E61A28CA49AC2767596576CD596906B1622F@GVW1113EXC.americas.hpqcorp.net> References: <036B68E61A28CA49AC2767596576CD596905CFF147@GVW1113EXC.americas.hpqcorp.net> <1269609947.2379.5.camel@localhost> <036B68E61A28CA49AC2767596576CD596905CFF653@GVW1113EXC.americas.hpqcorp.net> <1269852089.2504.17.camel@localhost> <036B68E61A28CA49AC2767596576CD596906B1622F@GVW1113EXC.americas.hpqcorp.net> Message-ID: <1269866146.727.180.camel@localhost.localdomain> Hi, On Mon, 2010-03-29 at 12:15 +0000, Jankowski, Chris wrote: > Steven, > > >>>You can use localflocks on each node provided you never access any of the locked files from more than once node at once (which may be true depending on how the failover is designed). Then you will get local fcntl lock performance at the expense of cluster fcntl locks. > > I could guarantee that only one node will use the filesystem by putting mount/unmount into the start/stop script for the application service. This is the easy part. > > What I would like to understand is how the GFS2 recovery would look like after failure of the node that had the filesystem mounted. I'd guess that the local locks will be gone with the failed system and there is nothing to recover. The only thing to do would be to replay to transaction log from the failed system. Is this correct? > Yes, that is correct. > This would work essentially like having a non-cluster filesystem such as ext3fs, but in case of recovery from a node failure doing only transaction log replay instaed of full fsck? Or would fsck still be triggered on the attempt to mount the filesystem on the other node.? > > Thanks and regards, > > Chris > Fsck would not be triggered on the mount attempt. Its not possible to run fsck while the fs is mounted on any node, Steve. > > -----Original Message----- > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse > Sent: Monday, 29 March 2010 19:41 > To: linux clustering > Subject: Re: [Linux-cluster] GFS2 - monitoring the rate of Posix lock operations > > On Sun, 2010-03-28 at 02:32 +0000, Jankowski, Chris wrote: > > Steve, > > > > Q2: > > >>> Are you sure that the workload isn't causing too many cache invalidations due to sharing files/directories between nodes? This is the most usual cause of poor performance. > > > > The other node is completely idle and kept that way by design. Users are connecting through an IP alias managed by the appplication service. Application administrators also log in through the alias to do their maintenance work. In the case of this particular test I manually listed what is running where. I am very concious of the fact that accesses from multiple nodes invalidate local in-memory caching. > > > > Q3: > > >>> Have you used the noatime mount option? If you can use it, its highly recommended. Also turn off selinux if that is running on the GFS2 filesystem. > > > > The filesystem is mounted with noatime and no nodiratime options. SELinux is disabled. > > > nodiratime isn't supported, noatime is enough. > > > Q4: > > >>>Potentially there might be. I don't know enough about the > > >>>application to say, but it depends on how the workload can be > > >>>arranged, > > > > The application runs on one node at a time. It has to, as it uses shared memory. The application uses a database of indexed files. There are thousands of them. Also, it uses standard UNIX flile locking and range locking. > > > > What else can I do to minimise the GFS2 locking overhead in this asymetrical configuration. > > > You can use localflocks on each node provided you never access any of the locked files from more than once node at once (which may be true depending on how the failover is designed). Then you will get local fcntl lock performance at the expense of cluster fcntl locks. > > > Q5: > > Is this the case that when gfs_controld gets to 100% of one coe DPU usage then this is a hard limit on the number of Posix locks taken. Is there only one gfs_lockd daemon servicng all GFS2 filesystems or are they run on a per filesystems basis? In the latter case I would have thought that breaking the one filesystem that I have into several may help. Would it not? > > > > Thanks and regards, > > > > Chris > > > Assuming that you have a version in which gfs_controld takes care of the locking (newer GFS2 send the locks via dlm_controld) then yes, that will provide a hard limit on the rate at which locks can be acquired/dropped, > > Steve. > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From rpeterso at redhat.com Mon Mar 29 13:05:05 2010 From: rpeterso at redhat.com (Bob Peterson) Date: Mon, 29 Mar 2010 09:05:05 -0400 (EDT) Subject: [Linux-cluster] gfs2-utils source for recovery purpose of a corrupt gfs2 partition In-Reply-To: <76883282.134081269867288635.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> Message-ID: <406336262.135511269867905178.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> ----- "Markus Wolfgart" wrote: | Hallo Cluster and GFS Experts, | | I'm a new subscriber of this mailing list and appologise | in the case my posting is offtopic. | | I'm looking for help concerning a corrupt gfs2 file system | which could not be recovered by me by fsck.gfs2 (Ver. 3.0.9) | due to to less less physical memory (4GB) eaven if increasing it | by a additional swap space (now about 35GB). | | I would like to parse a image created of the lost fs (the first 6TB) | with the code provided in the new gfs2-utils release. | | Due to this circumstance I hope to find in this mailing list some | hints | concerning an automated step by step recovery of lost data. | | Many Thanks in advance for your help | | Markus Hi Markus, You said that fsck.gfs2 is not working but you did not say what messages it gives you when you try. This must be a very big file system. How big is it? Was it converted from gfs1? Regards, Bob Peterson Red Hat File Systems From Chris.Jankowski at hp.com Mon Mar 29 13:10:34 2010 From: Chris.Jankowski at hp.com (Jankowski, Chris) Date: Mon, 29 Mar 2010 13:10:34 +0000 Subject: [Linux-cluster] GFS2 - monitoring the rate of Posix lock operations In-Reply-To: <1269866146.727.180.camel@localhost.localdomain> References: <036B68E61A28CA49AC2767596576CD596905CFF147@GVW1113EXC.americas.hpqcorp.net> <1269609947.2379.5.camel@localhost> <036B68E61A28CA49AC2767596576CD596905CFF653@GVW1113EXC.americas.hpqcorp.net> <1269852089.2504.17.camel@localhost> <036B68E61A28CA49AC2767596576CD596906B1622F@GVW1113EXC.americas.hpqcorp.net> <1269866146.727.180.camel@localhost.localdomain> Message-ID: <036B68E61A28CA49AC2767596576CD596906B16255@GVW1113EXC.americas.hpqcorp.net> Steve, Thank you. This looks quite promissing. I can have maximum performance and fast failover. Regards, Chris -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse Sent: Monday, 29 March 2010 23:36 To: linux clustering Subject: Re: [Linux-cluster] GFS2 - monitoring the rate of Posix lock operations Hi, On Mon, 2010-03-29 at 12:15 +0000, Jankowski, Chris wrote: > Steven, > > >>>You can use localflocks on each node provided you never access any of the locked files from more than once node at once (which may be true depending on how the failover is designed). Then you will get local fcntl lock performance at the expense of cluster fcntl locks. > > I could guarantee that only one node will use the filesystem by putting mount/unmount into the start/stop script for the application service. This is the easy part. > > What I would like to understand is how the GFS2 recovery would look like after failure of the node that had the filesystem mounted. I'd guess that the local locks will be gone with the failed system and there is nothing to recover. The only thing to do would be to replay to transaction log from the failed system. Is this correct? > Yes, that is correct. > This would work essentially like having a non-cluster filesystem such as ext3fs, but in case of recovery from a node failure doing only transaction log replay instaed of full fsck? Or would fsck still be triggered on the attempt to mount the filesystem on the other node.? > > Thanks and regards, > > Chris > Fsck would not be triggered on the mount attempt. Its not possible to run fsck while the fs is mounted on any node, Steve. > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven > Whitehouse > Sent: Monday, 29 March 2010 19:41 > To: linux clustering > Subject: Re: [Linux-cluster] GFS2 - monitoring the rate of Posix lock > operations > > On Sun, 2010-03-28 at 02:32 +0000, Jankowski, Chris wrote: > > Steve, > > > > Q2: > > >>> Are you sure that the workload isn't causing too many cache invalidations due to sharing files/directories between nodes? This is the most usual cause of poor performance. > > > > The other node is completely idle and kept that way by design. Users are connecting through an IP alias managed by the appplication service. Application administrators also log in through the alias to do their maintenance work. In the case of this particular test I manually listed what is running where. I am very concious of the fact that accesses from multiple nodes invalidate local in-memory caching. > > > > Q3: > > >>> Have you used the noatime mount option? If you can use it, its highly recommended. Also turn off selinux if that is running on the GFS2 filesystem. > > > > The filesystem is mounted with noatime and no nodiratime options. SELinux is disabled. > > > nodiratime isn't supported, noatime is enough. > > > Q4: > > >>>Potentially there might be. I don't know enough about the > > >>>application to say, but it depends on how the workload can be > > >>>arranged, > > > > The application runs on one node at a time. It has to, as it uses shared memory. The application uses a database of indexed files. There are thousands of them. Also, it uses standard UNIX flile locking and range locking. > > > > What else can I do to minimise the GFS2 locking overhead in this asymetrical configuration. > > > You can use localflocks on each node provided you never access any of the locked files from more than once node at once (which may be true depending on how the failover is designed). Then you will get local fcntl lock performance at the expense of cluster fcntl locks. > > > Q5: > > Is this the case that when gfs_controld gets to 100% of one coe DPU usage then this is a hard limit on the number of Posix locks taken. Is there only one gfs_lockd daemon servicng all GFS2 filesystems or are they run on a per filesystems basis? In the latter case I would have thought that breaking the one filesystem that I have into several may help. Would it not? > > > > Thanks and regards, > > > > Chris > > > Assuming that you have a version in which gfs_controld takes care of > the locking (newer GFS2 send the locks via dlm_controld) then yes, > that will provide a hard limit on the rate at which locks can be > acquired/dropped, > > Steve. > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From markus.wolfgart at dlr.de Mon Mar 29 15:23:57 2010 From: markus.wolfgart at dlr.de (Markus Wolfgart) Date: Mon, 29 Mar 2010 17:23:57 +0200 Subject: [Linux-cluster] gfs2-utils source for recovery purpose of a corrupt gfs2 partition Message-ID: <4BB0C60D.8040509@dlr.de> Hi Bob, thanks for prompt reply! the fs originally was 12.4TB (6TB used) big. After a resize attempt to 25TB by gfs2_grow (very very old version gfs2-utils 1.62) The fs was expand and the first impression looks good as df reported the size of 25TB. But looking from the second node to the fs (two nod system) ls -r and ls -R throws IO errors and gfs2 mount get frozen (reboot of machine was performed). As no shrinking of gfs2 was possible to rollback, the additional physical volume was removed from the logical volume (lvresize to org. size & pvremove). This hard cut of the gsf2 unfenced partition should be hopefully repaired by the fsck.gfs2 (newest version), this was my thought. Even if this will not be the case, I could not run the fsck.gfs2 due to a "of memory in compute_rgrp_layout" message. see strace output: write(1, "9098813: start: 4769970307031 (0"..., 739098813: start: 4769970307031 (0x4569862bfd7), length = 524241 (0x7ffd1) ) = 73 write(1, "9098814: start: 4769970831272 (0"..., 739098814: start: 4769970831272 (0x456986abfa8), length = 524241 (0x7ffd1) ) = 73 write(1, "9098815: start: 4769971355513 (0"..., 739098815: start: 4769971355513 (0x4569872bf79), length = 524241 (0x7ffd1) ) = 73 write(1, "9098816: start: 4769971879754 (0"..., 739098816: start: 4769971879754 (0x456987abf4a), length = 524241 (0x7ffd1) ) = 73 write(1, "9098817: start: 4769972403995 (0"..., 739098817: start: 4769972403995 (0x4569882bf1b), length = 524241 (0x7ffd1) ) = 73 brk(0xb7dea000) = 0xb7dc9000 mmap2(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory) mmap2(NULL, 1048576, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory) mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory) mmap2(NULL, 1048576, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory) write(2, "Out of memory in compute_rgrp_la"..., 37Out of memory in compute_rgrp_layout ) = 37 exit_group(-1) = ? As I had already increased my swapspace swapon -s Filename Type Size Used Priority /dev/sda3 partition 8385920 0 -3 /var/swapfile.bin file 33554424 144 1 and run again the same situation as before I decide to start to extract the lost files by a c prog. Now I have create a big Image (7TB) on a xfs partition and would like to recover my files of interest by a program using libgfs2 or part of the source from gfs2-utils, as mentioned in my previous posting. As I see nearly all of the files located in the dir structure and get the position in the image by a simple string command, I hope to extract them in a simpler way. The RG size was set to the Max value of 2GB end each file I'm looking for is about 250BM big. The amount of files to be recovered is more then 16k. Every file have a header with his file name ant the total size, so it should be easy to check if the recovery of it is successful. So thats my theory, but this could be a easter vacation project without the right knowledge of gfs2. As I'm lucky to have the gfs2-utils source I hope it could be done. But if there is a simpler way to do a recovery by the installed gfs2 progs like gfs2_edit or gfs2_tool or other tools it would be nice if someone could show my the proper way. Many Thanks in advance Markus -- ******************************************************* Markus Wolfgart DLR Oberpfaffenhofen German Remote Sensing Data Center . . . e-mail: markus.wolfgart at dlr.de ********************************************************** ----- "Markus Wolfgart" wrote: | Hallo Cluster and GFS Experts, | | I'm a new subscriber of this mailing list and appologise | in the case my posting is offtopic. | | I'm looking for help concerning a corrupt gfs2 file system | which could not be recovered by me by fsck.gfs2 (Ver. 3.0.9) | due to to less less physical memory (4GB) eaven if increasing it | by a additional swap space (now about 35GB). | | I would like to parse a image created of the lost fs (the first 6TB) | with the code provided in the new gfs2-utils release. | | Due to this circumstance I hope to find in this mailing list some | hints | concerning an automated step by step recovery of lost data. | | Many Thanks in advance for your help | | Markus Hi Markus, You said that fsck.gfs2 is not working but you did not say what messages it gives you when you try. This must be a very big file system. How big is it? Was it converted from gfs1? Regards, Bob Peterson Red Hat File Systems From james.hofmeister at hp.com Mon Mar 29 18:11:45 2010 From: james.hofmeister at hp.com (Hofmeister, James (WTEC Linux)) Date: Mon, 29 Mar 2010 18:11:45 +0000 Subject: [Linux-cluster] RHEL5.4: conga luci - Runtime Error: maximum recursion depth exceeded Message-ID: Hello All, RE: RHEL5.4: conga luci - Runtime Error: maximum recursion depth exceeded Has anybody seen this? RHEL5.4 with ricci-0.12.2-6.el5_4.1-x86_64 and luci-0.12.1-7.el5.x86_64: Runtime Error Sorry, a site error occurred. Traceback (innermost last): * Module ZPublisher.Publish, line 196, in publish_module_standard * Module Products.PlacelessTranslationService.PatchStringIO, line 34, in new_publish * Module ZPublisher.Publish, line 146, in publish * Module Zope2.App.startup, line 222, in zpublisher_exception_hook * Module ZPublisher.Publish, line 121, in publish * Module Zope2.App.startup, line 240, in commit * Module transaction._manager, line 96, in commit * Module transaction._transaction, line 380, in commit * Module transaction._transaction, line 378, in commit * Module transaction._transaction, line 433, in _commitResources * Module ZODB.Connection, line 484, in commit * Module ZODB.Connection, line 526, in _commit * Module ZODB.Connection, line 553, in _store_objects * Module ZODB.serialize, line 407, in serialize * Module ZODB.serialize, line 416, in _dump Runtime Error: maximum recursion depth exceeded (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: An operation previously failed, with traceback: File "/usr/lib64/luci/zope/lib/python/ZServer/PubCore/ZServerPubl isher.py", line 23, in __init__ response=response) File "/usr/lib64/luci/zope/lib/python/ZPublisher/Publish.py&q uot;, line 395, in publish_module environ, debug, request, response) File "/usr/lib64/luci/zope/lib/python/ZPublisher/Publish.py&q uot;, line 196, in publish_module_standard response = publish(request, module_name, after list, debug=debug) File "/usr/lib64/luci/zope/lib/python/Products/PlacelessTranslati onService/PatchStringIO.py", line 34, in new_publish x = Publish.old_publish(request, module_name, after_list, debug) File "/usr/lib64/luci/zope/lib/python/ZPublisher/Publish.py&q uot;, line 121, in publish transactions_manager.commit() File "/usr/lib64/luci/zope/lib/python/Zope2/App/startup.py&qu ot;, line 240, in commit transaction.commit() File "/usr/lib64/luci/zope/lib/python/transaction/_manager.py& ;quot;, line 96, in commit return self.get().commit(sub, deprecation_wng=False) File "/usr/lib64/luci/zope/lib/python/transaction/_transaction.py ", line 380, in commit self._saveCommitishError() # This raises! File "/usr/lib64/luci/zope/lib/python/transaction/_transaction.py ", line 378, in commit self._commitResources() File "/usr/lib64/luci/zope/lib/python/transaction/_transaction.py ", line 433, in _commitResources rm.commit(self) File "/usr/lib64/luci/zope/lib/python/ZODB/Connection.py" ;, line 484, in commit self._commit(transaction) File "/usr/lib64/luci/zope/lib/python/ZODB/Connection.py" ;, line 526, in _commit self._store_objects(ObjectWriter(obj), transaction) File "/usr/lib64/luci/zope/lib/python/ZODB/Connection.py" ;, line 553, in _store_objects p = writer.serialize(obj) # This calls __getstate__ of obj File "/usr/lib64/luci/zope/lib/python/ZODB/serialize.py" , line 407, in serialize return self._dump(meta, obj.__getstate__()) File "/usr/lib64/luci/zope/lib/python/ZODB/serialize.py" , line 416, in _dump self._p.dump(state) RuntimeError: maximum recursion depth exceeded ) Regards, James Hofmeister From Martin.Waite at datacash.com Tue Mar 30 04:11:51 2010 From: Martin.Waite at datacash.com (Martin Waite) Date: Tue, 30 Mar 2010 05:11:51 +0100 Subject: [Linux-cluster] why does ip.sh launch rdisc ? Message-ID: Hi, I have noticed that rdisc - apparently a router discovery protocol daemon - has started running on nodes that take possession of a VIP using ip.sh. I am not familiar with rdisc. It is currently installed on all my RHEL hosts, but is not running. Do I need to run rdisc ? Also, the man page says that rdisc uses 224.0.0.1 as a multicast address. So does my current cman configuration. Should I configure cman to avoid this address ? regards, Martin From jcasale at activenetwerx.com Tue Mar 30 21:28:46 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Tue, 30 Mar 2010 21:28:46 +0000 Subject: [Linux-cluster] Cron Jobs Message-ID: Anyone know how I might accomplish keeping cron jobs on the active node? I realize I can create the job on all nodes such that it quietly checks for status, if it's the active node, it runs but it's much easier to maintain my config on the moving fs as I do with httpd/MySQL etc so there exists only one copy. Thanks for any ideas! jlc From Chris.Jankowski at hp.com Tue Mar 30 22:12:53 2010 From: Chris.Jankowski at hp.com (Jankowski, Chris) Date: Tue, 30 Mar 2010 22:12:53 +0000 Subject: [Linux-cluster] Cron Jobs In-Reply-To: References: Message-ID: <036B68E61A28CA49AC2767596576CD596906B16940@GVW1113EXC.americas.hpqcorp.net> A few ideas. 1. What about replacing the directory containing the cron job descriptions in /var with a symbolic link to a directory on the sahred filesystem. 2. You application service start/stop script may modify the cron job description files. This is more complex, as it has to deal with remote nodes that may be down. Regards, Chris Jankowski -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Joseph L. Casale Sent: Wednesday, 31 March 2010 08:29 To: 'Linux-cluster at redhat.com' Subject: [Linux-cluster] Cron Jobs Anyone know how I might accomplish keeping cron jobs on the active node? I realize I can create the job on all nodes such that it quietly checks for status, if it's the active node, it runs but it's much easier to maintain my config on the moving fs as I do with httpd/MySQL etc so there exists only one copy. Thanks for any ideas! jlc -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From jcasale at activenetwerx.com Tue Mar 30 22:42:22 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Tue, 30 Mar 2010 22:42:22 +0000 Subject: [Linux-cluster] Cron Jobs In-Reply-To: <036B68E61A28CA49AC2767596576CD596906B16940@GVW1113EXC.americas.hpqcorp.net> References: <036B68E61A28CA49AC2767596576CD596906B16940@GVW1113EXC.americas.hpqcorp.net> Message-ID: >1. >What about replacing the directory containing the cron job descriptions in /var with a symbolic link to a directory on the sahred filesystem. yeah, my first inkling was to symlink /etc/cron.daily but that breaks so much existing functionality. >2. >You application service start/stop script may modify the cron job description files. This is more complex, as it has to deal with remote nodes that may be >down. I followed you until you spoke of remote nodes? What exactly do you have in mind? Thanks! jlc From robejrm at gmail.com Tue Mar 30 23:13:24 2010 From: robejrm at gmail.com (Juan Ramon Martin Blanco) Date: Wed, 31 Mar 2010 01:13:24 +0200 Subject: [Linux-cluster] Cron Jobs In-Reply-To: References: <036B68E61A28CA49AC2767596576CD596906B16940@GVW1113EXC.americas.hpqcorp.net> Message-ID: <8a5668961003301613n45ffb50et648aa85500d20eb0@mail.gmail.com> On Wed, Mar 31, 2010 at 12:42 AM, Joseph L. Casale wrote: >>1. >>What about replacing the directory containing the cron job descriptions in /var with a symbolic link to a directory on the sahred filesystem. > You could use a shared filesystem mounted on /etc/cron.d to keep cron scripts, and a certain control inside these scripts for not executing the job on both nodes. Greetings, Juanra > yeah, my first inkling was to symlink /etc/cron.daily but that breaks so much existing functionality. > >>2. >>You application service start/stop script may modify the cron job description files. ?This is more complex, as it has to deal with remote nodes that may be >down. > > I followed you until you spoke of remote nodes? What exactly do you have in mind? > Thanks! > jlc > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From Chris.Jankowski at hp.com Tue Mar 30 23:48:51 2010 From: Chris.Jankowski at hp.com (Jankowski, Chris) Date: Tue, 30 Mar 2010 23:48:51 +0000 Subject: [Linux-cluster] Cron Jobs In-Reply-To: References: <036B68E61A28CA49AC2767596576CD596906B16940@GVW1113EXC.americas.hpqcorp.net> Message-ID: <036B68E61A28CA49AC2767596576CD596906B1694E@GVW1113EXC.americas.hpqcorp.net> Hi, 1. >>>yeah, my first inkling was to symlink /etc/cron.daily but that breaks so much existing functionality. I was actually thinking about /var/spool/cron/crontabs directory. You can put your cron definitions there in the old UNIX style. It works perfectly well and is more general and flexible then the /etc/cron.* files, I believe. 2. >>>I followed you until you spoke of remote nodes? What exactly do you have in mind? I implemented that approach in my old failover scripts for Digital/Compaq/HP TruCluster. Attached is a README file for this functionality. This will give you the concepts, although there are bits there that are TruCluster specific like CDSLs. If you are interested I am happy to share the scripts from which you can extract the relevant code and modify it for your needs. Regards, Chris -------------- # # # Crontab file management. # ------------------------ # # There is a need for having a schedule of commands for cron # that is active on a node only when the service is running # on this node. # In other words, certain commands must be scheduled only # when the service is running and only on the node on which # the service is running. # # One way to implement it would be to modify every such command # to check for the presence of the service on the node on which # the command is run. This will be quite cumbersome if there # is large number of such commands. # # Another way to achieve execution of commands dependent # on presence of a service would be by writing a jacket script # taking as arguments the name of service in question and the # pathname of the script to be executed and its arguments. # # The implementation here takes advantage of the fact that service # specific cron processing is commonly done by a certain user or users # and that crontab(1) maintains a separate configuration file for each # user. Thus, it is relatively easy to manipulate the crontab file # of such a user. # # A directory is chosen eg. /usr/local/crontabs. # # This directory contains templates of crontab files for users that # are associated with certain services in a sense that the cron jobs # for such a user are to be run only on the node on which this service # is running. # # The script starting the service will install the template as the # crontab file for such a user on startup of the service. # # The template of the crontab file should be named by the username # with the extension service_on. # # Eg. for a user "fred" and chosen extension ".service_on" the template # should be named: # # fred.service_on # # Typically, by convention, the name of the CAA application resource # will be used as the "service" string in the extension. # # The contents of the template will be active on the member running # the service for the lifetime of the service. # # On a graceful shutdown of the service the script will install # another template of the crontab file for the user. # # This template of the crontab file should be named by the username # with a predefined extension. # # Eg. for a user "fred" and chosen extension ".service_off" the template # should be named: # # fred.service_off # # Typically, by convention, the name of the CAA application resource # will be used as the "service" string in the extension. # # The contents of the template will be active on every member not running # the service at the time. # # This template specifies periodically scheduled processing for a user # on members that do not run the service at the time. # The file may of course contain no commands, but it should exist. # # Of course both of those templates should be in the standard crontab(1) # format. # # Notes and assumptions: # # 1. # Please note that the above mechanism of crontab file management # assumes that a user is associated with only one service. # More state would need to be kept if a user would need different # processing depending on whether 0, 1, 2 or more services were # running on a node. # # 2. # Please note that /var/spool/cron is a CDSL in the TCS cluster and thus # all crontab files in /var/spool/cron/crontabs are node specific. # # 3. # If a node dies suddenly and then reboots, then it will reboot # with a set of crontabs that may not reflect the current state # of services on the node after reboot. # In fact the node will have all the crontabs from the moment it # crashed augmented by changes caused by any sevices restarted # on it after its reboot. # # What is really needed is another script - run on boot from # /sbin/rc2.d directory that will install correct initial, inactive # (*.service_off) versions of the crontabs on boot. # # 4. # The crontab templates must be readable by the user for whom # they are to be installed. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Joseph L. Casale Sent: Wednesday, 31 March 2010 09:42 To: 'linux clustering' Subject: Re: [Linux-cluster] Cron Jobs >1. >What about replacing the directory containing the cron job descriptions in /var with a symbolic link to a directory on the sahred filesystem. yeah, my first inkling was to symlink /etc/cron.daily but that breaks so much existing functionality. >2. >You application service start/stop script may modify the cron job description files. This is more complex, as it has to deal with remote nodes that may be >down. I followed you until you spoke of remote nodes? What exactly do you have in mind? Thanks! jlc -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From martin.fuerstenau at oce.com Wed Mar 31 06:06:14 2010 From: martin.fuerstenau at oce.com (Martin Fuerstenau) Date: Wed, 31 Mar 2010 08:06:14 +0200 Subject: [Linux-cluster] Cron Jobs In-Reply-To: <036B68E61A28CA49AC2767596576CD596906B1694E@GVW1113EXC.americas.hpqcorp.net> References: <036B68E61A28CA49AC2767596576CD596906B16940@GVW1113EXC.americas.hpqcorp.net> <036B68E61A28CA49AC2767596576CD596906B1694E@GVW1113EXC.americas.hpqcorp.net> Message-ID: <1270015574.26381.37.camel@lx002140.ops.de> Hi I think it is relatively easy to make cronjobs "cluster safe" . For shell scripts I do it the following way (the job should run on the same node the service mysqld is running): # Who am I? THIS_NODE=$(/usr/sbin/clustat | grep Local | awk '{print $1}') # On which node is the service running the cronjob depends? RUN_NODE=$(/usr/sbin/clustat | grep mysqld | awk '{print $2}') if [ $THIS_NODE != $RUN_NODE ] then echo "ERROR! Wrong cluster node." echo "This shall run from the same node where mysql is running." exit 1 fi This works fine for me for a lot of cronjobs. Martin F?rstenau Senior System Engineer ? Oc? Printing Systems GmbH On Tue, 2010-03-30 at 23:48 +0000, Jankowski, Chris wrote: > Hi, > > 1. > >>>yeah, my first inkling was to symlink /etc/cron.daily but that breaks so much existing functionality. > > I was actually thinking about /var/spool/cron/crontabs directory. You can put your cron definitions there in the old UNIX style. It works perfectly well and is more general and flexible then the /etc/cron.* files, I believe. > > 2. > >>>I followed you until you spoke of remote nodes? What exactly do you have in mind? > > I implemented that approach in my old failover scripts for Digital/Compaq/HP TruCluster. Attached is a README file for this functionality. This will give you the concepts, although there are bits there that are TruCluster specific like CDSLs. If you are interested I am happy to share the scripts from which you can extract the relevant code and modify it for your needs. > > Regards, > > Chris > > -------------- > > # > # > # Crontab file management. > # ------------------------ > # > # There is a need for having a schedule of commands for cron > # that is active on a node only when the service is running > # on this node. > # In other words, certain commands must be scheduled only > # when the service is running and only on the node on which > # the service is running. > # > # One way to implement it would be to modify every such command > # to check for the presence of the service on the node on which > # the command is run. This will be quite cumbersome if there > # is large number of such commands. > # > # Another way to achieve execution of commands dependent > # on presence of a service would be by writing a jacket script > # taking as arguments the name of service in question and the > # pathname of the script to be executed and its arguments. > # > # The implementation here takes advantage of the fact that service > # specific cron processing is commonly done by a certain user or users > # and that crontab(1) maintains a separate configuration file for each > # user. Thus, it is relatively easy to manipulate the crontab file > # of such a user. > # > # A directory is chosen eg. /usr/local/crontabs. > # > # This directory contains templates of crontab files for users that > # are associated with certain services in a sense that the cron jobs > # for such a user are to be run only on the node on which this service > # is running. > # > # The script starting the service will install the template as the > # crontab file for such a user on startup of the service. > # > # The template of the crontab file should be named by the username > # with the extension service_on. > # > # Eg. for a user "fred" and chosen extension ".service_on" the template > # should be named: > # > # fred.service_on > # > # Typically, by convention, the name of the CAA application resource > # will be used as the "service" string in the extension. > # > # The contents of the template will be active on the member running > # the service for the lifetime of the service. > # > # On a graceful shutdown of the service the script will install > # another template of the crontab file for the user. > # > # This template of the crontab file should be named by the username > # with a predefined extension. > # > # Eg. for a user "fred" and chosen extension ".service_off" the template > # should be named: > # > # fred.service_off > # > # Typically, by convention, the name of the CAA application resource > # will be used as the "service" string in the extension. > # > # The contents of the template will be active on every member not running > # the service at the time. > # > # This template specifies periodically scheduled processing for a user > # on members that do not run the service at the time. > # The file may of course contain no commands, but it should exist. > # > # Of course both of those templates should be in the standard crontab(1) > # format. > # > # Notes and assumptions: > # > # 1. > # Please note that the above mechanism of crontab file management > # assumes that a user is associated with only one service. > # More state would need to be kept if a user would need different > # processing depending on whether 0, 1, 2 or more services were > # running on a node. > # > # 2. > # Please note that /var/spool/cron is a CDSL in the TCS cluster and thus > # all crontab files in /var/spool/cron/crontabs are node specific. > # > # 3. > # If a node dies suddenly and then reboots, then it will reboot > # with a set of crontabs that may not reflect the current state > # of services on the node after reboot. > # In fact the node will have all the crontabs from the moment it > # crashed augmented by changes caused by any sevices restarted > # on it after its reboot. > # > # What is really needed is another script - run on boot from > # /sbin/rc2.d directory that will install correct initial, inactive > # (*.service_off) versions of the crontabs on boot. > # > # 4. > # The crontab templates must be readable by the user for whom > # they are to be installed. > > > > -----Original Message----- > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Joseph L. Casale > Sent: Wednesday, 31 March 2010 09:42 > To: 'linux clustering' > Subject: Re: [Linux-cluster] Cron Jobs > > >1. > >What about replacing the directory containing the cron job descriptions in /var with a symbolic link to a directory on the sahred filesystem. > > yeah, my first inkling was to symlink /etc/cron.daily but that breaks so much existing functionality. > > >2. > >You application service start/stop script may modify the cron job description files. This is more complex, as it has to deal with remote nodes that may be >down. > > I followed you until you spoke of remote nodes? What exactly do you have in mind? > Thanks! > jlc > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > This message and attachment(s) are intended solely for use by the addressee and may contain information that is privileged, confidential or otherwise exempt from disclosure under applicable law. If you are not the intended recipient or agent thereof responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by telephone and with a 'reply' message. Thank you for your co-operation. From kkovachev at varna.net Wed Mar 31 09:16:52 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Wed, 31 Mar 2010 12:16:52 +0300 Subject: [Linux-cluster] Cron Jobs In-Reply-To: References: Message-ID: <8e41372283b9bf32af06fb9c6daba9c8@mx.varna.net> On Tue, 30 Mar 2010 21:28:46 +0000, "Joseph L. Casale" wrote: > Anyone know how I might accomplish keeping cron jobs on the active node? > I realize I can create the job on all nodes such that it quietly checks > for status, if it's the active node, it runs but it's much easier to > maintain my config on the moving fs as I do with httpd/MySQL etc so > there exists only one copy. > What about adding a 'script' resource on the same service and use status checks to run your jobs. You may need to increase the status check to 60s or some other logic (lock file) to avoid running the scripts twice per minute. > Thanks for any ideas! > jlc > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From christoph at macht-blau.org Wed Mar 31 09:32:15 2010 From: christoph at macht-blau.org (C. Handel) Date: Wed, 31 Mar 2010 11:32:15 +0200 Subject: [Linux-cluster] ocf_log Message-ID: Hi List, i'm writing a custom resource agent. In the resource agent i try to use the ocf_log funtions but they don't work as expected. When i run the rgmanager in the foreground (clurgmgrd -df) i get all the message i want. When running as a normal daemon i can't find my log entries. The agent goes something like this #!/bin/bash PATH=/bin:/sbin:/usr/bin:/usr/sbin . $(dirname $0)/ocf-shellfuncs status() { ocf_log debug "status has been called (ocf_log)" logger -p local4.debug "status has been called (logger)" } the cluster.conf part ... running in foreground i get both (ocf and logger) debug messages. Running as a daemon i can find debug messages send by logger but no messages send via ocf_log. The IP Resource agent (part of the same service) shows up in the logfile. Any Hints? Greetings Christoph From brem.belguebli at gmail.com Wed Mar 31 10:18:06 2010 From: brem.belguebli at gmail.com (brem belguebli) Date: Wed, 31 Mar 2010 11:18:06 +0100 Subject: [Linux-cluster] ocf_log In-Reply-To: References: Message-ID: Have you defined a syslog.conf entry for your local4 facility ? example: local4.* /var/log/rgmanager Brem 2010/3/31 C. Handel : > ?Hi List, > > i'm writing a custom resource agent. In the resource agent i try to > use the ocf_log funtions but they don't work as expected. When i run > the rgmanager in the foreground (clurgmgrd -df) i get all the message > i want. When running as a normal daemon i can't find my log entries. > > The agent goes something like this > > #!/bin/bash > PATH=/bin:/sbin:/usr/bin:/usr/sbin > . $(dirname $0)/ocf-shellfuncs > > status() { > ? ocf_log debug "status has been called (ocf_log)" > ? logger -p local4.debug "status has been called (logger)" > } > > > the cluster.conf part > > > ... > > > running in foreground i get both (ocf and logger) debug messages. > Running as a daemon i can find debug messages send by logger but no > messages send via ocf_log. The IP Resource agent (part of the same > service) shows up in the logfile. > > Any Hints? > > Greetings > ? Christoph > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From jeff.sturm at eprize.com Wed Mar 31 13:23:39 2010 From: jeff.sturm at eprize.com (Jeff Sturm) Date: Wed, 31 Mar 2010 09:23:39 -0400 Subject: [Linux-cluster] Cron Jobs In-Reply-To: References: Message-ID: <64D0546C5EBBD147B75DE133D798665F055D8D7D@hugo.eprize.local> > -----Original Message----- > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] > On Behalf Of Joseph L. Casale > Sent: Tuesday, March 30, 2010 5:29 PM > To: 'Linux-cluster at redhat.com' > Subject: [Linux-cluster] Cron Jobs > > Anyone know how I might accomplish keeping cron jobs on the active node? > I realize I can create the job on all nodes such that it quietly checks > for status, if it's the active node, it runs but it's much easier to > maintain my config on the moving fs as I do with httpd/MySQL etc so > there exists only one copy. It's probably overkill for most people, but we've implemented Quartz for scheduled tasks so that we'd no longer have to worry about migrating cron jobs. Quartz is cluster-aware when coupled to a RDBMS, though it's unrelated to the RHCS product suite. Implement a high-availability RDBMS with cluster tools, run Quartz on each cluster node and it "just works". It's also an active-active configuration so we can exercise all cluster nodes at once. Someone else already mentioned it, but another good idea is creating lock files on a GFS filesystem. We have a "semaphore" filesystem implemented on GFS for certain processes that must be serialized. The high-availability scheduler doesn't completely replace it because we may have to worry about overlapping scheduled tasks with different start times. Works great. -Jeff From christoph at macht-blau.org Wed Mar 31 20:03:14 2010 From: christoph at macht-blau.org (C. Handel) Date: Wed, 31 Mar 2010 22:03:14 +0200 Subject: [Linux-cluster] ocf_log Message-ID: yes. Messages from logger (which uses the same facility as rm) and debug messages from the ip resource agent show up. Greetings Christoph > > Have you defined a syslog.conf entry for your local4 facility ? > > example: > > local4.* ? ? ? ? ? ? ? ?/var/log/rgmanager > > Brem > > 2010/3/31 C. Handel : >> ?Hi List, >> >> i'm writing a custom resource agent. In the resource agent i try to >> use the ocf_log funtions but they don't work as expected. When i run >> the rgmanager in the foreground (clurgmgrd -df) i get all the message >> i want. When running as a normal daemon i can't find my log entries. >> >> The agent goes something like this >> >> #!/bin/bash >> PATH=/bin:/sbin:/usr/bin:/usr/sbin >> . $(dirname $0)/ocf-shellfuncs >> >> status() { >> ? ocf_log debug "status has been called (ocf_log)" >> ? logger -p local4.debug "status has been called (logger)" >> } >> >> >> the cluster.conf part >> >> >> ... >>