From fdinitto at redhat.com Sun Nov 2 08:49:00 2008 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Sun, 2 Nov 2008 09:49:00 +0100 (CET) Subject: [Linux-cluster] Fw: Building error in Cluster 2.03.09 In-Reply-To: <0d9d01c93b49$a5a81fc0$a401a8c0@mainoffice.nodex.ru> References: <0d9d01c93b49$a5a81fc0$a401a8c0@mainoffice.nodex.ru> Message-ID: On Fri, 31 Oct 2008, Pavel Kuzin wrote: > Hello! > > I`m trying to build cluster 2.03.09 against linux 2.6.27.4. > > When building have a error: > > upgrade.o: In function `upgrade_device_archive': > /root/newcluster/cluster-2.03.09/ccs/ccs_tool/upgrade.c:226: undefined > reference to `mkostemp' > collect2: ld returned 1 exit status > make[2]: *** [ccs_tool] Error 1 > make[2]: Leaving directory `/root/newcluster/cluster-2.03.09/ccs/ccs_tool' > make[1]: *** [all] Error 2 > make[1]: Leaving directory `/root/newcluster/cluster-2.03.09/ccs' > make: *** [ccs] Error 2 > > node2:~/newcluster/cluster-2.03.09# uname -a > Linux node2 2.6.27.4 #2 SMP Fri Oct 31 13:42:09 MSK 2008 i686 GNU/Linux > > Distro - Debian Etch > Seems mkostemp is available since glibc 2.7. > I have 2.6. > Can "mkostemp" be changed to another similar function? Probably, I'll have a look on monday. Fabio -- I'm going to make him an offer he can't refuse. From fdinitto at redhat.com Mon Nov 3 04:55:45 2008 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Mon, 3 Nov 2008 05:55:45 +0100 (CET) Subject: [Linux-cluster] Fw: Building error in Cluster 2.03.09 In-Reply-To: <0d9d01c93b49$a5a81fc0$a401a8c0@mainoffice.nodex.ru> References: <0d9d01c93b49$a5a81fc0$a401a8c0@mainoffice.nodex.ru> Message-ID: On Fri, 31 Oct 2008, Pavel Kuzin wrote: > Hello! > > I`m trying to build cluster 2.03.09 against linux 2.6.27.4. > > When building have a error: > > upgrade.o: In function `upgrade_device_archive': > /root/newcluster/cluster-2.03.09/ccs/ccs_tool/upgrade.c:226: undefined > reference to `mkostemp' > collect2: ld returned 1 exit status > make[2]: *** [ccs_tool] Error 1 > make[2]: Leaving directory `/root/newcluster/cluster-2.03.09/ccs/ccs_tool' > make[1]: *** [all] Error 2 > make[1]: Leaving directory `/root/newcluster/cluster-2.03.09/ccs' > make: *** [ccs] Error 2 > > node2:~/newcluster/cluster-2.03.09# uname -a > Linux node2 2.6.27.4 #2 SMP Fri Oct 31 13:42:09 MSK 2008 i686 GNU/Linux > > Distro - Debian Etch > Seems mkostemp is available since glibc 2.7. > I have 2.6. > Can "mkostemp" be changed to another similar function? You can apply this patch or wait for the next relese: commit 6e8c492f8e8233bc5e295ae12322c40936279178 Author: Fabio M. Di Nitto Date: Mon Nov 3 05:52:39 2008 +0100 ccs: fix build with older glibc mkostemp has been introduced only in glibc 2.7. Switch to mkstemp. Signed-off-by: Fabio M. Di Nitto diff --git a/ccs/ccs_tool/upgrade.c b/ccs/ccs_tool/upgrade.c index b7cecf0..6a0e150 100644 --- a/ccs/ccs_tool/upgrade.c +++ b/ccs/ccs_tool/upgrade.c @@ -223,7 +223,7 @@ static int upgrade_device_archive(char *location){ memset(tmp_file, 0, 128); sprintf(tmp_file, "/tmp/ccs_tool_tmp_XXXXXX"); - tmp_fd = mkostemp(tmp_file, O_RDWR | O_CREAT |O_TRUNC); + tmp_fd = mkstemp(tmp_file); if(tmp_fd < 0){ fprintf(stderr, "Unable to create temporary archive: %s\n", strerror(errno)); error = -errno; -- I'm going to make him an offer he can't refuse. From Harri.Paivaniemi at tietoenator.com Mon Nov 3 06:26:30 2008 From: Harri.Paivaniemi at tietoenator.com (Harri.Paivaniemi at tietoenator.com) Date: Mon, 3 Nov 2008 08:26:30 +0200 Subject: [Linux-cluster] Can clustered RHEL 5 use a SAN with differentaccess rights for different nodes in the cluster? References: <639ce0480806171423u4503665ewd7426080145309ea@mail.gmail.com> Message-ID: <41E8D4F07FCE154CBEBAA60FFC92F67709FE33@apollo.eu.tieto.com> Hi, If you have ability (in your storage system) to export that disk also via NFS, it's the simpliest way... you could restrict access just via mount/export perms and no gfs needed... works perfectly in ftp-usage. -hjp 2008/6/18 Richard Williams - IoDynamix : > Please advise and/or redirect this posting if this is not the correct forum > for my question - thanks. > > A company wants to use clustered rhel5 systems as inside/outside ftp > servers. Users on the inside (LAN) cluster nodes can read and write to the > SAN, while users on the outside (DMZ) cluster can only read. > > Is this application possible without GFS? > > If one node in the cluster fails, can the other node be provisioned to > provide all services until recovery? > > Can a SAN be used as the "single" ftp location for both services (inside > FTP > & outside FTP?) > > Does the customer need more than four systems (i.e. 2 inside - 2 outside) - > is a separate "command" system required? > > > Have Dell's m1000e & 600 series blades been certified for this operating > system? > > Is there any documentation available regarding separate access rights for > multiple nodes in a cluster available? > > Thanks for your constructive reply. > > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3799 bytes Desc: not available URL: From tuckerd at engr.smu.edu Mon Nov 3 15:56:27 2008 From: tuckerd at engr.smu.edu (Doug Tucker) Date: Mon, 03 Nov 2008 09:56:27 -0600 Subject: [Linux-cluster] Data Loss / Files and Folders "2-Node_GFS-Cluster" In-Reply-To: <490A01C1.5030003@gmail.com> References: <2fd157df0810301037jf985e3bne5ca25e91dd74872@mail.gmail.com> <490A01C1.5030003@gmail.com> Message-ID: <1225727787.8639.3.camel@thor.seas.smu.edu> > I don't (or "didn't") have adequate involvements with RHEL5 GFS. I may > not know enough to response. However, users should be aware of ... > > Before RHEL 5.1 and community version 2.6.22 kernels, NFS locks (i.e. > flock, posix lock, etc) is not populated into filesystem layer. It only > reaches Linux VFS layer (local to one particular server). If your file > access needs to get synchronized via either flock or posix locks > *between multiple hosts (i.e. NFS servers)*, data loss could occur. > Newer versions of RHEL and 2.6.22-and-above kernels should have the code > to support this new feature. > > There was an old write-up in section 4.1 of > "http://people.redhat.com/wcheng/Project/nfs.htm" about this issue. > > -- Wendy > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Wendy, To be clear, does this include RHEL 4.7, or is it specific to 5.x? From s.wendy.cheng at gmail.com Mon Nov 3 16:31:30 2008 From: s.wendy.cheng at gmail.com (Wendy Cheng) Date: Mon, 03 Nov 2008 11:31:30 -0500 Subject: [Linux-cluster] Data Loss / Files and Folders "2-Node_GFS-Cluster" In-Reply-To: <1225727787.8639.3.camel@thor.seas.smu.edu> References: <2fd157df0810301037jf985e3bne5ca25e91dd74872@mail.gmail.com> <490A01C1.5030003@gmail.com> <1225727787.8639.3.camel@thor.seas.smu.edu> Message-ID: <490F2762.80902@gmail.com> Doug Tucker wrote: > >> I don't (or "didn't") have adequate involvements with RHEL5 GFS. I may >> not know enough to response. However, users should be aware of ... >> >> Before RHEL 5.1 and community version 2.6.22 kernels, NFS locks (i.e. >> flock, posix lock, etc) is not populated into filesystem layer. It only >> reaches Linux VFS layer (local to one particular server). If your file >> access needs to get synchronized via either flock or posix locks >> *between multiple hosts (i.e. NFS servers)*, data loss could occur. >> Newer versions of RHEL and 2.6.22-and-above kernels should have the code >> to support this new feature. >> >> There was an old write-up in section 4.1 of >> "http://people.redhat.com/wcheng/Project/nfs.htm" about this issue. >> >> -- Wendy >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > Wendy, > > To be clear, does this include RHEL 4.7, or is it specific to 5.x? > The changes were made on 2.6.22 kernel. I would think RHEL 4.7 has the same issue - but I'm not sure as I left Red Hat before 4.7 was released. Better to open a service ticket to Red Hat if you need the fix. If applications are directly run on GFS nodes, instead of going thru NFS servers, posix locks and flocks should work *fine* across different nodes. The problem had existed in Linux NFS servers for years - no one seemed to complain about it until clusters started to get deployed more commonly. -- Wendy From a.holstvoogd at nedforce.nl Tue Nov 4 09:45:36 2008 From: a.holstvoogd at nedforce.nl (Arthur Holstvoogd) Date: Tue, 04 Nov 2008 10:45:36 +0100 Subject: [Linux-cluster] gfs2-utils 0.1.17 and kernel-2.6.18-120> Message-ID: <491019C0.4000700@nedforce.nl> Hi, I'm using a beta kernel from http://people.redhat.com/dzickus/el5 because of trouble with dlm which is solved in this version. This has broken some of the gfs2-utils tools, specifically gfs2_quota which still uses the old metafs. I have two questions: - is the utils 0.1.49 version available anywhere as a rpm orso? Or only from source? - will other tools, like fsck, from the gfs2-utils 0.1.17 ( current version with centos 5.2) still work? Regards, Arthur From jeder at invision.net Tue Nov 4 16:56:47 2008 From: jeder at invision.net (Jeremy Eder) Date: Tue, 4 Nov 2008 11:56:47 -0500 Subject: [Linux-cluster] cluster-snmp setup question Message-ID: Hello, I have a newly installed rhel4u7 32bit (2node) cluster. I have cluster-snmp-0.11.1-2.el4 installed, and I have added this to snmpd.conf per instructions found in /usr/share/doc/cluster-snmp-0.11.1/README.snmpd -------- dlmod RedHatCluster /usr/lib/cluster-snmp/libClusterMonitorSnmp.so view systemview included REDHAT-CLUSTER-MIB:RedHatCluster -------- The problem is that the system only responds to snmpwalk on this: root at db1: /etc/snmp # snmpwalk -v1 -c public localhost .1.3.6.1.4.1.2312.8 SNMPv2-SMI::enterprises.2312.8.1.1.0 = INTEGER: 1 Any other OID silently completes with no output... root at db1: /etc/snmp # snmpwalk -v1 -c public localhost .1.3.6.1.4.1.2312.8.2.1.0 root at db1: /etc/snmp # The 2nd snmpwalk should output the rhcClusterName as a STRING. Note that REDHAT-MIB and REDHAT-CLUSTER-MIB are installed properly: -rw-r--r-- 1 root root 7957 Apr 14 2008 /usr/share/snmp/mibs/REDHAT-CLUSTER-MIB -rw-r--r-- 1 root root 772 Apr 14 2008 /usr/share/snmp/mibs/REDHAT-MIB Did I miss some step in the setup (are there some services that need to be started ?) There are no hits for "snmp" on the cluster-wiki... -- jer From ffv at tjpr.jus.br Tue Nov 4 19:12:38 2008 From: ffv at tjpr.jus.br (ffv at tjpr.jus.br) Date: Tue, 04 Nov 2008 17:12:38 -0200 Subject: [Linux-cluster] GFS2 poor performance Message-ID: <49109ea6.+nLiDwujAEL50VEe%ffv@tjpr.jus.br> Hi all, I?m getting a very poor performance using GFS2. I have two qmail (mail) servers and one gfs2 filesystem shared by them. In this case, each directory in GFS2 filesystem may have upon to 10000 files (mails) The problem is in performance of some operations like ls, du, rm, etc for example, # time du -sh /dados/teste 40M /dados/teste real 7m14.919s user 0m0.008s sys 0m0.129s this is unacceptable Some attributes i already set using gfs2_tool: gfs2_tool settune /dados demote_secs 100 gfs2_tool setflag jdata /dados gfs2_tool setflag sync /dados gfs2_tool setflag directio /dados but the performance is still very bad Anybody know how to tune the filesystem for a acceptable performance working with directory with 10000 files? thanks for any help From rpeterso at redhat.com Tue Nov 4 19:16:53 2008 From: rpeterso at redhat.com (Bob Peterson) Date: Tue, 4 Nov 2008 14:16:53 -0500 (EST) Subject: [Linux-cluster] GFS2 poor performance In-Reply-To: <49109ea6.+nLiDwujAEL50VEe%ffv@tjpr.jus.br> Message-ID: <1819458317.178811225826213442.JavaMail.root@zmail02.collab.prod.int.phx2.redhat.com> ----- ffv at tjpr.jus.br wrote: | Hi all, | | I?m getting a very poor performance using GFS2. | I have two qmail (mail) servers and one gfs2 filesystem shared by | them. | In this case, each directory in GFS2 filesystem may have upon to 10000 | files (mails) | | The problem is in performance of some operations like ls, du, rm, etc | for example, | | # time du -sh /dados/teste | 40M /dados/teste | | real 7m14.919s | user 0m0.008s | sys 0m0.129s | | this is unacceptable What version of GFS2 and what kernel are you using? Regards, Bob Peterson Red Hat Clustering & GFS From jeff.sturm at eprize.com Tue Nov 4 19:41:56 2008 From: jeff.sturm at eprize.com (Jeff Sturm) Date: Tue, 4 Nov 2008 14:41:56 -0500 Subject: [Linux-cluster] GFS2 poor performance In-Reply-To: <49109ea6.+nLiDwujAEL50VEe%ffv@tjpr.jus.br> References: <49109ea6.+nLiDwujAEL50VEe%ffv@tjpr.jus.br> Message-ID: <64D0546C5EBBD147B75DE133D798665F01806B7C@hugo.eprize.local> What sort of network and storage device are you using? Also, why set demote_secs so low? -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of ffv at tjpr.jus.br Sent: Tuesday, November 04, 2008 2:13 PM To: linux-cluster at redhat.com Subject: [Linux-cluster] GFS2 poor performance Hi all, I?m getting a very poor performance using GFS2. I have two qmail (mail) servers and one gfs2 filesystem shared by them. In this case, each directory in GFS2 filesystem may have upon to 10000 files (mails) The problem is in performance of some operations like ls, du, rm, etc for example, # time du -sh /dados/teste 40M /dados/teste real 7m14.919s user 0m0.008s sys 0m0.129s this is unacceptable Some attributes i already set using gfs2_tool: gfs2_tool settune /dados demote_secs 100 gfs2_tool setflag jdata /dados gfs2_tool setflag sync /dados gfs2_tool setflag directio /dados but the performance is still very bad Anybody know how to tune the filesystem for a acceptable performance working with directory with 10000 files? thanks for any help -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From ffv at tjpr.jus.br Tue Nov 4 20:00:42 2008 From: ffv at tjpr.jus.br (Fabiano F. Vitale) Date: Tue, 4 Nov 2008 18:00:42 -0200 Subject: [Linux-cluster] GFS2 poor performance References: <1819458317.178811225826213442.JavaMail.root@zmail02.collab.prod.int.phx2.redhat.com> Message-ID: <007301c93eb8$04926f60$3e0a10ac@tjpr.net> Hi all, I use CentOS 5.2 and the kernel and gfs2 versions are: [root at smtp01 ~]# uname -r 2.6.18-92.1.13.el5 [root at smtp01 ~]# gfs2_tool version gfs2_tool 0.1.44 (built Jul 6 2008 10:58:08) Copyright (C) Red Hat, Inc. 2004-2006 All rights reserved. thanks for any help > ----- ffv at tjpr.jus.br wrote: > | Hi all, > | > | I?m getting a very poor performance using GFS2. > | I have two qmail (mail) servers and one gfs2 filesystem shared by > | them. > | In this case, each directory in GFS2 filesystem may have upon to 10000 > | files (mails) > | > | The problem is in performance of some operations like ls, du, rm, etc > | for example, > | > | # time du -sh /dados/teste > | 40M /dados/teste > | > | real 7m14.919s > | user 0m0.008s > | sys 0m0.129s > | > | this is unacceptable > What version of GFS2 and what kernel are you using? From ffv at tjpr.jus.br Tue Nov 4 20:18:51 2008 From: ffv at tjpr.jus.br (Fabiano F. Vitale) Date: Tue, 4 Nov 2008 18:18:51 -0200 Subject: [Linux-cluster] GFS2 poor performance References: <49109ea6.+nLiDwujAEL50VEe%ffv@tjpr.jus.br> <64D0546C5EBBD147B75DE133D798665F01806B7C@hugo.eprize.local> Message-ID: <007f01c93eba$8d731b20$3e0a10ac@tjpr.net> Hi, for cluster purpose the two nodes are linked by a patch cord cat6 and the lan interfaces are gigabit. All nodes have a Fibre Channel Emulex Corporation Zephyr-X LightPulse and the Storage is a HP EVA8100 I read the document http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Summit08presentation_GFSBestPractices_Final.pdf which show some parameters to tune and one of them is demote_secs, to adjust to 100sec thanks > What sort of network and storage device are you using? > > Also, why set demote_secs so low? > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of ffv at tjpr.jus.br > Sent: Tuesday, November 04, 2008 2:13 PM > To: linux-cluster at redhat.com > Subject: [Linux-cluster] GFS2 poor performance > > Hi all, > > I?m getting a very poor performance using GFS2. > I have two qmail (mail) servers and one gfs2 filesystem shared by them. > In this case, each directory in GFS2 filesystem may have upon to 10000 > files (mails) > > The problem is in performance of some operations like ls, du, rm, etc for > example, > > # time du -sh /dados/teste > 40M /dados/teste > > real 7m14.919s > user 0m0.008s > sys 0m0.129s > > this is unacceptable > > Some attributes i already set using gfs2_tool: > > gfs2_tool settune /dados demote_secs 100 gfs2_tool setflag jdata /dados > gfs2_tool setflag sync /dados gfs2_tool setflag directio /dados > > but the performance is still very bad > > > Anybody know how to tune the filesystem for a acceptable performance > working with directory with 10000 files? > thanks for any help > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From fdinitto at redhat.com Wed Nov 5 13:36:56 2008 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Wed, 5 Nov 2008 14:36:56 +0100 (CET) Subject: [Linux-cluster] making init scripts distro agnostic Message-ID: One of the goals agreed at the Cluster Summit 2008 was to ship init scripts that did work on all distribution. I just landed a first cut of this work into the initscripts branch in git: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=shortlog;h=refs/heads/initscripts I tested those changes on 3 distributions and they work for me. Unless I receive any major complain, those changes will land in master friday. There is still some work that needs to be done to cleanup the scripts. I am perfectly aware of that, but this is one step forward. Thanks Fabio -- I'm going to make him an offer he can't refuse. From afahounko at gmail.com Wed Nov 5 15:00:30 2008 From: afahounko at gmail.com (AFAHOUNKO Danny) Date: Wed, 05 Nov 2008 15:00:30 +0000 Subject: [Linux-Cluster] RHCS + DRBD Message-ID: <4911B50E.1080809@gmail.com> Hi, i'm deploying a cluster (two nodes without a share storage) with redhat cluster suite. I want to synchronise data between two partitions. I've started with rsync, but it's complicated. I've read that i can do it with DRBD. I've started drbd installation and it work fine. But i don't know how to integrate it to my cluster ressources. I mean, before mounting the device (/dev/drbd0 for example) on one node, the partition must be promote as master and as secondary on the second node. Anyone have expierences in configuring drbd with RHCS ?! thx. -- Cordialement AFAHOUNKO Danny Administrateur R?seaux & Syst?me d'Information - CICA-RE Gsm: +228 914.55.89 Tel: +228 223.62.62 From tuckerd at engr.smu.edu Wed Nov 5 15:51:52 2008 From: tuckerd at engr.smu.edu (Doug Tucker) Date: Wed, 05 Nov 2008 09:51:52 -0600 Subject: [Linux-cluster] Data Loss / Files and Folders "2-Node_GFS-Cluster" In-Reply-To: <490F2762.80902@gmail.com> References: <2fd157df0810301037jf985e3bne5ca25e91dd74872@mail.gmail.com> <490A01C1.5030003@gmail.com> <1225727787.8639.3.camel@thor.seas.smu.edu> <490F2762.80902@gmail.com> Message-ID: <1225900312.26904.6.camel@thor.seas.smu.edu> > > The changes were made on 2.6.22 kernel. I would think RHEL 4.7 has the > same issue - but I'm not sure as I left Red Hat before 4.7 was released. > Better to open a service ticket to Red Hat if you need the fix. > > If applications are directly run on GFS nodes, instead of going thru NFS > servers, posix locks and flocks should work *fine* across different > nodes. The problem had existed in Linux NFS servers for years - no one > seemed to complain about it until clusters started to get deployed more > commonly. > > -- Wendy > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster That's always been tough for me to discern, as they stay with the same base kernel "name" while actually moving the code forward. 4.7 has kernel: 2.6.9-78.0.1.ELsmp . Now how that translates to the "actual" kernel number as 2.6.21, 22, etc, I never can figure out. From chris at cmiware.com Wed Nov 5 16:15:04 2008 From: chris at cmiware.com (Chris Harms) Date: Wed, 05 Nov 2008 10:15:04 -0600 Subject: [Linux-Cluster] RHCS + DRBD In-Reply-To: <4911B50E.1080809@gmail.com> References: <4911B50E.1080809@gmail.com> Message-ID: <4911C688.4060708@cmiware.com> You need a script that does it and you add that to the service in RHCS. I cobbled one together from the linux-ha project with some customizations. AFAHOUNKO Danny wrote: > Hi, > i'm deploying a cluster (two nodes without a share storage) with > redhat cluster suite. > I want to synchronise data between two partitions. > I've started with rsync, but it's complicated. > I've read that i can do it with DRBD. > I've started drbd installation and it work fine. But i don't know how > to integrate it to my cluster ressources. > I mean, before mounting the device (/dev/drbd0 for example) on one > node, the partition must be promote as master and as secondary on the > second node. > Anyone have expierences in configuring drbd with RHCS ?! > thx. > From afahounko at gmail.com Wed Nov 5 17:29:37 2008 From: afahounko at gmail.com (AFAHOUNKO Danny) Date: Wed, 05 Nov 2008 17:29:37 +0000 Subject: [Linux-cluster] Re: RHCS and DRBD In-Reply-To: References: Message-ID: <4911D801.7080101@gmail.com> Hi ! thx ! i'll try it and will let you know ! ;) Dani Filth a ?crit : > Hi- > > I've had good luck with the script shown on this page: > http://www.redhat.com/archives/linux-cluster/2006-July/msg00109.html > > Just replace NUM=0 with the number of your drbd resource. > > In my cluster.conf, I have a resource that calls that script, and > another resource to mount the filesystem. > > Good luck! -- Cordialement AFAHOUNKO Danny Administrateur R?seaux & Syst?me d'Information - CICA-RE Gsm: +228 914.55.89 Tel: +228 223.62.62 From s.wendy.cheng at gmail.com Wed Nov 5 18:06:38 2008 From: s.wendy.cheng at gmail.com (Wendy Cheng) Date: Wed, 05 Nov 2008 13:06:38 -0500 Subject: [Linux-cluster] Data Loss / Files and Folders "2-Node_GFS-Cluster" In-Reply-To: <1225900312.26904.6.camel@thor.seas.smu.edu> References: <2fd157df0810301037jf985e3bne5ca25e91dd74872@mail.gmail.com> <490A01C1.5030003@gmail.com> <1225727787.8639.3.camel@thor.seas.smu.edu> <490F2762.80902@gmail.com> <1225900312.26904.6.camel@thor.seas.smu.edu> Message-ID: <4911E0AE.20502@gmail.com> Doug Tucker wrote: >> The changes were made on 2.6.22 kernel. I would think RHEL 4.7 has the >> same issue - but I'm not sure as I left Red Hat before 4.7 was released. >> Better to open a service ticket to Red Hat if you need the fix. >> >> If applications are directly run on GFS nodes, instead of going thru NFS >> servers, posix locks and flocks should work *fine* across different >> nodes. The problem had existed in Linux NFS servers for years - no one >> seemed to complain about it until clusters started to get deployed more >> commonly. >> >> -- Wendy >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > That's always been tough for me to discern, as they stay with the same > base kernel "name" while actually moving the code forward. 4.7 has > kernel: 2.6.9-78.0.1.ELsmp . Now how that translates to the "actual" > kernel number as 2.6.21, 22, etc, I never can figure out. > You seem to assume, if the service ticket is approved, the fix would have to move the whole kernel from 2.6.9 into 2.6.22 ? That is a (surprising) mis-understanding. As any bug fix with any operating system distribution, it could get done across different kernels, if it passes certain types of risk and resource review process(es). The code change has to be tailored into its own release framework - the actual implementation may look different but it should accomplish similar logic(s) to fix the identical problem. Hopefully I interpret your comment right. -- Wendy From jerlyon at gmail.com Wed Nov 5 21:00:40 2008 From: jerlyon at gmail.com (Jeremy Lyon) Date: Wed, 5 Nov 2008 14:00:40 -0700 Subject: [Linux-cluster] Oracle locking on GFS Message-ID: <779919740811051300j3ae82ccav98e129d3a824b3c8@mail.gmail.com> Hi, I was curious if anyone has run into a similar issue. We have a 2 node cluster with a GFS file system running Oracle 10g (not RAC). The DB crashed and RHCS failed over the service as expected, but Oracle couldn't start correctly because of an exiting lock on one of the files. We had to unmount the GFS from both nodes, then mount. At that point, Oracle start up correctly. I'm assuming that the lock was in place during the crash and DLM was honoring it and that's what cause Oracle to not start correctly. Then by unmounting DLM dropped all locks. Any recommendations on how to avoid this? Is it a good idea to set up the GFS resource with a force unmount to avoid this scenario? TIA -Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.osullivan at auckland.ac.nz Thu Nov 6 04:43:26 2008 From: michael.osullivan at auckland.ac.nz (Michael O'Sullivan) Date: Wed, 05 Nov 2008 21:43:26 -0700 Subject: [Linux-cluster] Distributed RAID Message-ID: <491275EE.8050508@auckland.ac.nz> Hi everyone, I have just read that GFS on mdadm does not work because mdadm is not cluster aware. I really hoped to build a n + 1 RAID of the disks I have presented to the RHCS nodes via iSCSI. I had a look at DDRAID which is old and looks like it only supports 3, 5 and 9 disks in the distributed RAID. I currently only have two (multipathed) devices, but I want them to be active-active. If I put them into a mirrored logical volume in CLVM will this do the trick? Or will I have to install DRDB? Is there any more up-to-date distributed RAID options available for when I want to make a 2 + 1, 3 +1, etc storage array? There are some posts that say this may be available in CLVM soon or that mdadm may be cluster aware soon. Any progress on either of these options? Any help on this would be great. Thanks, Mike From fdinitto at redhat.com Thu Nov 6 07:31:01 2008 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Thu, 6 Nov 2008 08:31:01 +0100 (CET) Subject: [Linux-Cluster] RHCS + DRBD In-Reply-To: <4911C688.4060708@cmiware.com> References: <4911B50E.1080809@gmail.com> <4911C688.4060708@cmiware.com> Message-ID: Hi, all those kind of scripts can be really useful to a wider audience and could be easily added to the normal release process. Would it be possible for you to share the script with a GPLv2+ licence and proper copyright attached to it? Thanks Fabio On Wed, 5 Nov 2008, Chris Harms wrote: > You need a script that does it and you add that to the service in RHCS. I > cobbled one together from the linux-ha project with some customizations. > > > AFAHOUNKO Danny wrote: >> Hi, >> i'm deploying a cluster (two nodes without a share storage) with redhat >> cluster suite. >> I want to synchronise data between two partitions. >> I've started with rsync, but it's complicated. >> I've read that i can do it with DRBD. >> I've started drbd installation and it work fine. But i don't know how to >> integrate it to my cluster ressources. >> I mean, before mounting the device (/dev/drbd0 for example) on one node, >> the partition must be promote as master and as secondary on the second >> node. >> Anyone have expierences in configuring drbd with RHCS ?! >> thx. >> > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- I'm going to make him an offer he can't refuse. From gordan at bobich.net Thu Nov 6 09:38:02 2008 From: gordan at bobich.net (Gordan Bobic) Date: Thu, 06 Nov 2008 09:38:02 +0000 Subject: [Linux-cluster] Distributed RAID In-Reply-To: <491275EE.8050508@auckland.ac.nz> References: <491275EE.8050508@auckland.ac.nz> Message-ID: <4912BAFA.2080901@bobich.net> Michael O'Sullivan wrote: > I have just read that GFS on mdadm does not work because mdadm is not > cluster aware. I really hoped to build a n + 1 RAID of the disks I have > presented to the RHCS nodes via iSCSI. I had a look at DDRAID which is > old and looks like it only supports 3, 5 and 9 disks in the distributed > RAID. I currently only have two (multipathed) devices, but I want them > to be active-active. If I put them into a mirrored logical volume in > CLVM will this do the trick? Or will I have to install DRDB? Is there > any more up-to-date distributed RAID options available for when I want > to make a 2 + 1, 3 +1, etc storage array? There are some posts that say > this may be available in CLVM soon or that mdadm may be cluster aware > soon. Any progress on either of these options? You probably saw me asking these very same questions in the archives, without any response. DDRAID is unmaintained, and IIRC the code was removed from the current development tree a while back. So don't count on it ever getting resurrected. I rather doubt md will become cluster aware any time soon. CLVM doesn't yet support even more important features like snapshotting, so I wouldn't count on it supporting anything more advanced. For straight mirroring (which is all you could sensibly do with 2 nodes anyway), I can highly recommend DRBD. It "just works" and works well. I have a number of 2-node clusters deployed with it with shared-root. Of you really want to look into larger scale clustering with n+m redundancy, look into cleversafe.org. There's a thread I started on the forum there looking into exactly this sort of thing. I'll be testing it in the next month or so, when I get the hardware together, but it's looking plausible. It also provides proper n+m redundancy. Another thing to note is that RAID5 is not really usable on today's big disks in arrays of more than 6. Think about the expected read failure rates on modern disks: 10^-14. That's about one uncorrectable error every 10TB. So if you have a 6x1TB disk array, and you lose a disk, you have to read 5TB of data to reconstruct onto a fresh disk. Since you get one uncorrectable error every 10TB, that means you have a 50/50 chance of another disk encountering an error and dropping out of the array, and losing all your data. These days higher RAID levels are really a necessity, not an optional extra, and at this rate, considering the read error rates have stayed constant while sizes have exploded, even RAID6 won't last long. Gordan From hicheerup at gmail.com Thu Nov 6 12:46:28 2008 From: hicheerup at gmail.com (lingu) Date: Thu, 6 Nov 2008 18:16:28 +0530 Subject: [Linux-cluster] RHEL3 Cluster Heart Beat Using Cross Over Cable Message-ID: <29e045b80811060446q5b868dfdxd3281608b04dca62@mail.gmail.com> Hi, I am running two node active/passive cluster running RHEL3 update 8 64 bit OS on Hp Box with external hp storage connected via scsi. My cluster was running fine for last 3 years.But all of a sudden cluster service keep on shifting (atleast one time in a day )form one node to another. After analysed the syslog i found that due to some network fluctuation service was getting shifted.Both the nodes has two NIC bonded together and configured with below ip. My network details: 192.168.1.2 --node 1 physical ip with class c subnet (bond0 ) 192.168.1.3 --node 2 physical ip with class c subnet (bond0 ) 192.168.1.4 --- floating ip ( cluster ) Since it is a very critical and busy server may be due to heavy network load some hear beat signal is getting missed resulting in shifting of service from one node to another. So i planned to connect crossover cable for heart beat messages, can any one guide me or provide me the link that best explains how to do the same and the changes i have to made in cluster configuration file after connecting the crossover cable. Regards, Lingu From federico.simoncelli at gmail.com Thu Nov 6 15:40:55 2008 From: federico.simoncelli at gmail.com (Federico Simoncelli) Date: Thu, 6 Nov 2008 16:40:55 +0100 Subject: [Linux-cluster] Distributed RAID In-Reply-To: <4912BAFA.2080901@bobich.net> References: <491275EE.8050508@auckland.ac.nz> <4912BAFA.2080901@bobich.net> Message-ID: On Thu, Nov 6, 2008 at 10:38 AM, Gordan Bobic wrote: > I rather doubt md will become cluster aware any time soon. CLVM doesn't yet > support even more important features like snapshotting, so I wouldn't count > on it supporting anything more advanced. I worked a little on clvm snapshots: https://www.redhat.com/archives/linux-lvm/2008-October/msg00027.html Review and testing is required. -- Federico. From ricks at nerd.com Thu Nov 6 17:38:11 2008 From: ricks at nerd.com (Rick Stevens) Date: Thu, 06 Nov 2008 09:38:11 -0800 Subject: [Linux-cluster] RHEL3 Cluster Heart Beat Using Cross Over Cable In-Reply-To: <29e045b80811060446q5b868dfdxd3281608b04dca62@mail.gmail.com> References: <29e045b80811060446q5b868dfdxd3281608b04dca62@mail.gmail.com> Message-ID: <49132B83.1010402@nerd.com> lingu wrote: > Hi, > > I am running two node active/passive cluster running RHEL3 update > 8 64 bit OS on Hp Box with external hp storage connected via scsi. My > cluster was running fine for last 3 years.But all of a sudden cluster > service keep on shifting (atleast one time in a day )form one node to > another. > > After analysed the syslog i found that due to some network > fluctuation service was getting shifted.Both the nodes has two NIC > bonded together and configured with below ip. > > My network details: > > 192.168.1.2 --node 1 physical ip with class c subnet (bond0 ) > 192.168.1.3 --node 2 physical ip with class c subnet (bond0 ) > 192.168.1.4 --- floating ip ( cluster ) > > Since it is a very critical and busy server may be due to heavy > network load some hear beat signal is getting missed resulting in > shifting of service from one node to another. > > So i planned to connect crossover cable for heart beat messages, can > any one guide me or provide me the link that best explains how to do > the same and the changes i have to made in cluster configuration file > after connecting the crossover cable. The crossover cable is pretty easy to make and a lot of places have ones prebuilt. If you want to make one yourself, you're interested in the orange pair of wires (normally pins 1 and 2) and the green pair of wires (normally pins 3 and 6). The blue and brown pairs don't do anyting in standard TIA-56B cables. The wiring diagram is: End "A" (std) End "B" (crossover) pin 1 Orange/White pin 3 pin 2 Orange pin 6 pin 3 Green/White pin 1 pin 4 Blue pin 4 pin 5 Blue/White pin 5 pin 6 Green pin 2 pin 7 Brown/White pin 7 pin 8 Brown pin 8 Remember that the pins are numbered from the left, looking at the hole the cable goes into with the latch on the bottom. I generally put some sort of rather blatant mark on any such cable such as a big piece of tape or coloring the ends with a red marker so it's obvious that the cable is "special". To use it, just plug one end of the cable into the cluster NIC of the first system and the other end into the cluster NIC of the second system. You should get link lights at both ends. As far as any other changes, the only thing that may go a bit weird is the ARP tables on the systems since you've removed the hub/switch from the signal path and the ARP table may retain the old HW addresses. I don't think that'll be a problem. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer ricks at nerd.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - Tempt not the dragons of fate, since thou art crunchy and taste - - good with ketchup. - ---------------------------------------------------------------------- From jbrassow at redhat.com Thu Nov 6 19:49:45 2008 From: jbrassow at redhat.com (Jonathan Brassow) Date: Thu, 6 Nov 2008 13:49:45 -0600 Subject: [Linux-cluster] Distributed RAID In-Reply-To: References: <491275EE.8050508@auckland.ac.nz> <4912BAFA.2080901@bobich.net> Message-ID: Cluster mirror (RAID1) will be available in rhel5.3 for LVM. brassow On Nov 6, 2008, at 9:40 AM, Federico Simoncelli wrote: > On Thu, Nov 6, 2008 at 10:38 AM, Gordan Bobic > wrote: >> I rather doubt md will become cluster aware any time soon. CLVM >> doesn't yet >> support even more important features like snapshotting, so I >> wouldn't count >> on it supporting anything more advanced. > > I worked a little on clvm snapshots: > https://www.redhat.com/archives/linux-lvm/2008-October/msg00027.html > > Review and testing is required. > -- > Federico. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From jeff.sturm at eprize.com Thu Nov 6 19:53:30 2008 From: jeff.sturm at eprize.com (Jeff Sturm) Date: Thu, 6 Nov 2008 14:53:30 -0500 Subject: [Linux-cluster] GFS2 poor performance In-Reply-To: <007f01c93eba$8d731b20$3e0a10ac@tjpr.net> References: <49109ea6.+nLiDwujAEL50VEe%ffv@tjpr.jus.br><64D0546C5EBBD147B75DE133D798665F01806B7C@hugo.eprize.local> <007f01c93eba$8d731b20$3e0a10ac@tjpr.net> Message-ID: <64D0546C5EBBD147B75DE133D798665F01806C15@hugo.eprize.local> I looked over the summit document you referenced below. The value of demote_secs mentioned is an example setting, and unfortunately no recommendations or rationale accompany this. For some access patterns you can get better performance by actually increasing demote_secs. For example, we have a node that we routinely rsync a file tree onto using a GFS partition. Increasing demote_secs from 300 to 86400 reduced the average rsync time by a factor of about 4. The reason is that this node has little lock contention and needs to lock each file every time we start an rsync process. With demote_secs=300, it was doing much more work to reacquire locks on each run. Whereas demote_secs=86400 allowed the locks to persist up to a day, since the overall number of files in our application is bounded such that they will fit in buffer cache, together with locks. At another extreme, we have an application that creates a lot of files but seldom opens them on the same node. In this case there is no value in holding onto the locks, so we set demote_secs to a small value and glock_purge as high as 70 to ensure locks are quickly released in memory. The best advice I can give in general is to experiment with different settings for demote_secs and glock_purge while watching the output of "gfs_tool counters" to see how they behave. Jeff -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Fabiano F. Vitale Sent: Tuesday, November 04, 2008 3:19 PM To: linux clustering Subject: Re: [Linux-cluster] GFS2 poor performance Hi, for cluster purpose the two nodes are linked by a patch cord cat6 and the lan interfaces are gigabit. All nodes have a Fibre Channel Emulex Corporation Zephyr-X LightPulse and the Storage is a HP EVA8100 I read the document http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Summit08presentation_GFSBestPractices_Final.pdf which show some parameters to tune and one of them is demote_secs, to adjust to 100sec thanks > What sort of network and storage device are you using? > > Also, why set demote_secs so low? > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of ffv at tjpr.jus.br > Sent: Tuesday, November 04, 2008 2:13 PM > To: linux-cluster at redhat.com > Subject: [Linux-cluster] GFS2 poor performance > > Hi all, > > I?m getting a very poor performance using GFS2. > I have two qmail (mail) servers and one gfs2 filesystem shared by them. > In this case, each directory in GFS2 filesystem may have upon to 10000 > files (mails) > > The problem is in performance of some operations like ls, du, rm, etc > for example, > > # time du -sh /dados/teste > 40M /dados/teste > > real 7m14.919s > user 0m0.008s > sys 0m0.129s > > this is unacceptable > > Some attributes i already set using gfs2_tool: > > gfs2_tool settune /dados demote_secs 100 gfs2_tool setflag jdata > /dados gfs2_tool setflag sync /dados gfs2_tool setflag directio /dados > > but the performance is still very bad > > > Anybody know how to tune the filesystem for a acceptable performance > working with directory with 10000 files? > thanks for any help > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From gordan at bobich.net Thu Nov 6 20:40:39 2008 From: gordan at bobich.net (Gordan Bobic) Date: Thu, 06 Nov 2008 20:40:39 +0000 Subject: [Linux-cluster] Distributed RAID In-Reply-To: References: <491275EE.8050508@auckland.ac.nz> <4912BAFA.2080901@bobich.net> Message-ID: <49135647.3020701@bobich.net> What about CLVM based striping (RAID0)? Does that work already or is it planned for the near future? Gordan Jonathan Brassow wrote: > Cluster mirror (RAID1) will be available in rhel5.3 for LVM. > > brassow > > On Nov 6, 2008, at 9:40 AM, Federico Simoncelli wrote: > >> On Thu, Nov 6, 2008 at 10:38 AM, Gordan Bobic wrote: >>> I rather doubt md will become cluster aware any time soon. CLVM >>> doesn't yet >>> support even more important features like snapshotting, so I wouldn't >>> count >>> on it supporting anything more advanced. >> >> I worked a little on clvm snapshots: >> https://www.redhat.com/archives/linux-lvm/2008-October/msg00027.html >> >> Review and testing is required. >> -- >> Federico. >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From quickshiftin at gmail.com Thu Nov 6 21:30:35 2008 From: quickshiftin at gmail.com (Nathan Nobbe) Date: Thu, 6 Nov 2008 14:30:35 -0700 Subject: [Linux-cluster] RHEL3 Cluster Heart Beat Using Cross Over Cable In-Reply-To: <49132B83.1010402@nerd.com> References: <29e045b80811060446q5b868dfdxd3281608b04dca62@mail.gmail.com> <49132B83.1010402@nerd.com> Message-ID: <7dd2dc0b0811061330x693bdd9eifc092bf14ed0f959@mail.gmail.com> On Thu, Nov 6, 2008 at 10:38 AM, Rick Stevens wrote: > lingu wrote: > >> Hi, >> >> I am running two node active/passive cluster running RHEL3 update >> 8 64 bit OS on Hp Box with external hp storage connected via scsi. My >> cluster was running fine for last 3 years.But all of a sudden cluster >> service keep on shifting (atleast one time in a day )form one node to >> another. >> >> After analysed the syslog i found that due to some network >> fluctuation service was getting shifted.Both the nodes has two NIC >> bonded together and configured with below ip. >> >> My network details: >> >> 192.168.1.2 --node 1 physical ip with class c subnet (bond0 ) >> 192.168.1.3 --node 2 physical ip with class c subnet (bond0 ) >> 192.168.1.4 --- floating ip ( cluster ) >> >> Since it is a very critical and busy server may be due to heavy >> network load some hear beat signal is getting missed resulting in >> shifting of service from one node to another. >> >> So i planned to connect crossover cable for heart beat messages, can >> any one guide me or provide me the link that best explains how to do >> the same and the changes i have to made in cluster configuration file >> after connecting the crossover cable. >> > > The crossover cable is pretty easy to make and a lot of places have > ones prebuilt. If you want to make one yourself, you're interested in > the orange pair of wires (normally pins 1 and 2) and the green pair of > wires (normally pins 3 and 6). The blue and brown pairs don't do > anyting in standard TIA-56B cables. The wiring diagram is: > > End "A" (std) End "B" (crossover) > pin 1 Orange/White pin 3 > pin 2 Orange pin 6 > pin 3 Green/White pin 1 > pin 4 Blue pin 4 > pin 5 Blue/White pin 5 > pin 6 Green pin 2 > pin 7 Brown/White pin 7 > pin 8 Brown pin 8 > > Remember that the pins are numbered from the left, looking at the hole > the cable goes into with the latch on the bottom. I generally put some > sort of rather blatant mark on any such cable such as a big piece of > tape or coloring the ends with a red marker so it's obvious that the > cable is "special". > > To use it, just plug one end of the cable into the cluster NIC of the > first system and the other end into the cluster NIC of the second > system. You should get link lights at both ends. many modern machines will work w/o a crossover cable. ive got 2 dell 1650s running heartbeat / drbd over a direct connection for heartbeat communication. i dont need to use a crossover on the 1650s for the direct connection to work, and those boxes are pretty old by now. so long story short, probly worth saving a little hassle and just trying a regular cat-5 cable for the direct connection. or if its a requirement for you hardware you can pick up a 3 foot crossover at radio shack, bust buy etc, for less than 10 bucks. -nathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ricks at nerd.com Thu Nov 6 22:03:53 2008 From: ricks at nerd.com (Rick Stevens) Date: Thu, 06 Nov 2008 14:03:53 -0800 Subject: [Linux-cluster] RHEL3 Cluster Heart Beat Using Cross Over Cable In-Reply-To: <7dd2dc0b0811061330x693bdd9eifc092bf14ed0f959@mail.gmail.com> References: <29e045b80811060446q5b868dfdxd3281608b04dca62@mail.gmail.com> <49132B83.1010402@nerd.com> <7dd2dc0b0811061330x693bdd9eifc092bf14ed0f959@mail.gmail.com> Message-ID: <491369C9.5060602@nerd.com> Nathan Nobbe wrote: > On Thu, Nov 6, 2008 at 10:38 AM, Rick Stevens wrote: > >> lingu wrote: >> >>> Hi, >>> >>> I am running two node active/passive cluster running RHEL3 update >>> 8 64 bit OS on Hp Box with external hp storage connected via scsi. My >>> cluster was running fine for last 3 years.But all of a sudden cluster >>> service keep on shifting (atleast one time in a day )form one node to >>> another. >>> >>> After analysed the syslog i found that due to some network >>> fluctuation service was getting shifted.Both the nodes has two NIC >>> bonded together and configured with below ip. >>> >>> My network details: >>> >>> 192.168.1.2 --node 1 physical ip with class c subnet (bond0 ) >>> 192.168.1.3 --node 2 physical ip with class c subnet (bond0 ) >>> 192.168.1.4 --- floating ip ( cluster ) >>> >>> Since it is a very critical and busy server may be due to heavy >>> network load some hear beat signal is getting missed resulting in >>> shifting of service from one node to another. >>> >>> So i planned to connect crossover cable for heart beat messages, can >>> any one guide me or provide me the link that best explains how to do >>> the same and the changes i have to made in cluster configuration file >>> after connecting the crossover cable. >>> >> The crossover cable is pretty easy to make and a lot of places have >> ones prebuilt. If you want to make one yourself, you're interested in >> the orange pair of wires (normally pins 1 and 2) and the green pair of >> wires (normally pins 3 and 6). The blue and brown pairs don't do >> anyting in standard TIA-56B cables. The wiring diagram is: >> >> End "A" (std) End "B" (crossover) >> pin 1 Orange/White pin 3 >> pin 2 Orange pin 6 >> pin 3 Green/White pin 1 >> pin 4 Blue pin 4 >> pin 5 Blue/White pin 5 >> pin 6 Green pin 2 >> pin 7 Brown/White pin 7 >> pin 8 Brown pin 8 >> >> Remember that the pins are numbered from the left, looking at the hole >> the cable goes into with the latch on the bottom. I generally put some >> sort of rather blatant mark on any such cable such as a big piece of >> tape or coloring the ends with a red marker so it's obvious that the >> cable is "special". >> >> To use it, just plug one end of the cable into the cluster NIC of the >> first system and the other end into the cluster NIC of the second >> system. You should get link lights at both ends. > > > many modern machines will work w/o a crossover cable. ive got 2 dell 1650s > running heartbeat / drbd over a direct connection for heartbeat > communication. i dont need to use a crossover on the 1650s for the direct > connection to work, and those boxes are pretty old by now. so long story > short, probly worth saving a little hassle and just trying a regular cat-5 > cable for the direct connection. > > or if its a requirement for you hardware you can pick up a 3 foot crossover > at radio shack, bust buy etc, for less than 10 bucks. True. Some NICs have autosense for MDI and MDIX cables (the technical terms for straight and crossover, respectively), but a lot of them don't. Nathan's right, try a regular cable first. If it doesn't work, crossovers are available at lots of places quite cheaply. They often use red cable (the ones I've bought are red), but there are a lot of straight cables that use red as well, so I'd still mark MDIX cables very conspicuously. A big tag that says "I'M A CROSSOVER" can't hurt! My diagram above is valid if you really want to "roll your own". I've made so damned many CAT5/5e/6 cables in my life (MDI and MDIX both), that I can do it almost in my sleep. Ditto with thinnet (10Base-2) and I'm a past master at putting parasite taps on thicknet (10Base-5) cables, sticking on the transceivers and snaking that gawdawful AUI cable down cable stud pockets. I'm an original DECnet geek! ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer ricks at nerd.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - On a scale of 1 to 10 I'd say... oh, somewhere in there. - ---------------------------------------------------------------------- From jbrassow at redhat.com Thu Nov 6 22:34:49 2008 From: jbrassow at redhat.com (Jonathan Brassow) Date: Thu, 6 Nov 2008 16:34:49 -0600 Subject: [Linux-cluster] Distributed RAID In-Reply-To: <49135647.3020701@bobich.net> References: <491275EE.8050508@auckland.ac.nz> <4912BAFA.2080901@bobich.net> <49135647.3020701@bobich.net> Message-ID: <940E48BD-8C04-4525-86AD-B4ED1998883E@redhat.com> that works already. single machine: linear, stripe, mirror, snapshot cluster-aware: linear, stripe, mirror (5.3) brassow On Nov 6, 2008, at 2:40 PM, Gordan Bobic wrote: > What about CLVM based striping (RAID0)? Does that work already or is > it planned for the near future? > > Gordan > > Jonathan Brassow wrote: >> Cluster mirror (RAID1) will be available in rhel5.3 for LVM. >> brassow >> On Nov 6, 2008, at 9:40 AM, Federico Simoncelli wrote: >>> On Thu, Nov 6, 2008 at 10:38 AM, Gordan Bobic >>> wrote: >>>> I rather doubt md will become cluster aware any time soon. CLVM >>>> doesn't yet >>>> support even more important features like snapshotting, so I >>>> wouldn't count >>>> on it supporting anything more advanced. >>> >>> I worked a little on clvm snapshots: >>> https://www.redhat.com/archives/linux-lvm/2008-October/msg00027.html >>> >>> Review and testing is required. >>> -- >>> Federico. >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From hicheerup at gmail.com Fri Nov 7 10:44:08 2008 From: hicheerup at gmail.com (lingu) Date: Fri, 7 Nov 2008 16:14:08 +0530 Subject: [Linux-cluster] Cluster Broken pipe Error & node Reboot Message-ID: <29e045b80811070244w142f0ceey62ea0a8ec65db714@mail.gmail.com> Hi all, I am running two node RHEL3U8 cluster of below cluster version on HP servers connected via scsi channel to HP Storage (SAN) for oracle database server. Kernel & Cluster Version Kernel-2.4.21-47.EL #1 SMP redhat-config-cluster-1.0.7-1-noarch clumanager-1.2.26.1-1-x86_64 Suddenly my active node got rebooted after analysed the logs it is throwing below errors on syslog.I want to know what might cause this type of error and also after analysed the sar output indicates there was no load on the server at the time system get rebooted as well as on the time i am getting I/O Hang error. Nov 3 14:23:00 cluster1 clulockd[1996]: Denied 20.1.2.162: Broken pipe Nov 3 14:23:00 cluster1 clulockd[1996]: select error: Broken pipe Nov 3 14:23:06 cluster1 clulockd[1996]: Denied 20.1.2.162: Broken pipe Nov 3 14:23:06 cluster1 clulockd[1996]: select error: Broken pipe Nov 3 14:23:13 cluster1 cluquorumd[1921]: Disk-TB: Detected I/O Hang! Nov 3 14:23:15 cluster1 clulockd[1996]: Denied 20.1.2.161: Broken pipe Nov 3 14:23:15 cluster1 clulockd[1996]: select error: Broken pipe Nov 3 14:23:12 cluster1 clusvcmgrd[2011]: Unable to obtain cluster lock: Connection timed out Nov 5 17:18:00 cluster1 cluquorumd[1921]: Disk-TB: Detected I/O Hang! Nov 5 17:18:00 cluster1 clulockd[1996]: Denied 20.1.2.162: Broken pipe Nov 5 17:18:00 cluster1 clulockd[1996]: select error: Broken pipe Nov 5 17:18:17 cluster1 clulockd[1996]: Denied 20.1.2.162: Broken pipe Nov 5 17:18:17 cluster1 clulockd[1996]: select error: Broken pipe Nov 5 17:18:17 cluster1 clulockd[1996]: Potential recursive lock #0 grant to member #1, PID1962 I need some one help in guiding how to fix out this error and also the real cause for such above errors . Attached my cluster.xml file. Regards, Lingu From mad at wol.de Fri Nov 7 12:22:36 2008 From: mad at wol.de (Marc - A. Dahlhaus [ Administration | Westermann GmbH ]) Date: Fri, 07 Nov 2008 13:22:36 +0100 Subject: [Linux-cluster] Distributed RAID In-Reply-To: <940E48BD-8C04-4525-86AD-B4ED1998883E@redhat.com> References: <491275EE.8050508@auckland.ac.nz> <4912BAFA.2080901@bobich.net> <49135647.3020701@bobich.net> <940E48BD-8C04-4525-86AD-B4ED1998883E@redhat.com> Message-ID: <1226060556.12833.4.camel@marc> Hello, will the changes to mirroring get merged into stable2 and head after RHEL-5.3 release? Marc Am Donnerstag, den 06.11.2008, 16:34 -0600 schrieb Jonathan Brassow: > that works already. > > single machine: linear, stripe, mirror, snapshot > cluster-aware: linear, stripe, mirror (5.3) > > brassow > > On Nov 6, 2008, at 2:40 PM, Gordan Bobic wrote: > > > What about CLVM based striping (RAID0)? Does that work already or is > > it planned for the near future? > > > > Gordan > > > > Jonathan Brassow wrote: > >> Cluster mirror (RAID1) will be available in rhel5.3 for LVM. > >> brassow > >> On Nov 6, 2008, at 9:40 AM, Federico Simoncelli wrote: > >>> On Thu, Nov 6, 2008 at 10:38 AM, Gordan Bobic > >>> wrote: > >>>> I rather doubt md will become cluster aware any time soon. CLVM > >>>> doesn't yet > >>>> support even more important features like snapshotting, so I > >>>> wouldn't count > >>>> on it supporting anything more advanced. > >>> > >>> I worked a little on clvm snapshots: > >>> https://www.redhat.com/archives/linux-lvm/2008-October/msg00027.html > >>> > >>> Review and testing is required. > >>> -- > >>> Federico. > >>> > >>> -- > >>> Linux-cluster mailing list > >>> Linux-cluster at redhat.com > >>> https://www.redhat.com/mailman/listinfo/linux-cluster > >> -- > >> Linux-cluster mailing list > >> Linux-cluster at redhat.com > >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From pronix.service at gmail.com Fri Nov 7 12:59:09 2008 From: pronix.service at gmail.com (pronix pronix) Date: Fri, 7 Nov 2008 15:59:09 +0300 Subject: [Linux-cluster] Distributed RAID In-Reply-To: <1226060556.12833.4.camel@marc> References: <491275EE.8050508@auckland.ac.nz> <4912BAFA.2080901@bobich.net> <49135647.3020701@bobich.net> <940E48BD-8C04-4525-86AD-B4ED1998883E@redhat.com> <1226060556.12833.4.camel@marc> Message-ID: <639ce0480811070459l18a0071bia889bb56402f6ab9@mail.gmail.com> can i use cluster raid1 if i get development release from sources.redhat.com/cluster ? 2008/11/7 Marc - A. Dahlhaus [ Administration | Westermann GmbH ] < mad at wol.de> > Hello, > > > will the changes to mirroring get merged into stable2 and head after > RHEL-5.3 release? > > > Marc > > Am Donnerstag, den 06.11.2008, 16:34 -0600 schrieb Jonathan Brassow: > > that works already. > > > > single machine: linear, stripe, mirror, snapshot > > cluster-aware: linear, stripe, mirror (5.3) > > > > brassow > > > > On Nov 6, 2008, at 2:40 PM, Gordan Bobic wrote: > > > > > What about CLVM based striping (RAID0)? Does that work already or is > > > it planned for the near future? > > > > > > Gordan > > > > > > Jonathan Brassow wrote: > > >> Cluster mirror (RAID1) will be available in rhel5.3 for LVM. > > >> brassow > > >> On Nov 6, 2008, at 9:40 AM, Federico Simoncelli wrote: > > >>> On Thu, Nov 6, 2008 at 10:38 AM, Gordan Bobic > > >>> wrote: > > >>>> I rather doubt md will become cluster aware any time soon. CLVM > > >>>> doesn't yet > > >>>> support even more important features like snapshotting, so I > > >>>> wouldn't count > > >>>> on it supporting anything more advanced. > > >>> > > >>> I worked a little on clvm snapshots: > > >>> https://www.redhat.com/archives/linux-lvm/2008-October/msg00027.html > > >>> > > >>> Review and testing is required. > > >>> -- > > >>> Federico. > > >>> > > >>> -- > > >>> Linux-cluster mailing list > > >>> Linux-cluster at redhat.com > > >>> https://www.redhat.com/mailman/listinfo/linux-cluster > > >> -- > > >> Linux-cluster mailing list > > >> Linux-cluster at redhat.com > > >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.costakos at gmail.com Sat Nov 8 03:53:23 2008 From: david.costakos at gmail.com (Dave Costakos) Date: Fri, 7 Nov 2008 19:53:23 -0800 Subject: [Linux-cluster] gfs2 convert hosed all VMs Message-ID: <6b6836c60811071953m52d068cfpd0a204f8f4d1b99d@mail.gmail.com> I just converted a shared file-backed Xen VM GFS filesystem to a GFS2 filesystem. The conversion was successfully and all my files appear intact. I followed the GFS instructions by unmounting the filesystem on all machines, running gfs_fsck, and gfs2_covert. Since I converted the filesystem, all my file-backed Xen VMs can no longer boot. pygrub reports errors that the boot loader isn't returning any data. If I edit the Xen config to boot of a kernel on the DomU, VMs still can't start up because LVM cannot identify any volume groups. If I try to access the VM device files locally via losetup and kpartx, I get "read errors". So what's the deal I know GFS2 is a preview, but I have to assume I've missed some crucial step here. -- Dave Costakos mailto:david.costakos at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From fs at debian.org Mon Nov 10 09:27:41 2008 From: fs at debian.org (Frederik =?iso-8859-1?q?Sch=FCler?=) Date: Mon, 10 Nov 2008 10:27:41 +0100 Subject: [Linux-cluster] soft lockups on 2.6.27/2.03.09 with gfs1 Message-ID: <200811101027.47158.fs@debian.org> Hello, I am experiencing a rather big problem with deadlocks on a 9 nodes GFS1 cluster, with vanilla 2.6.27 and both rhcs 2.03.09 and latest git stable2. Fencing is done via fabric, the node keeps throwing these errors after it got fenced. This is a rather busy webserver cluster, with usually some dozens to hundreds of apache processes running concurrently, and 4 gfs1 shares with lots of small writes on the "template cache" volume from all 9 nodes. Lockups look like this: [44955.425003] BUG: soft lockup - CPU#2 stuck for 61s! [apache:12639] [44955.425007] Modules linked in: gfs ac battery ipv6 iptable_filter xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables x_tables lock_dlm gfs2 dlm configfs snd_pcm snd_timer snd soundcore snd_page_alloc rtc_cmos rtc_core i2c_nforce2 k8temp shpchp rtc_lib pcspkr pci_hotplug i2c_core button evdev ext3 jbd mbcache dm_mirror dm_log dm_snapshot dm_mod ide_cd_mod cdrom amd74xx sd_mod ide_pci_generic ide_core floppy qla2xxx scsi_transport_fc 3w_9xxx e1000e scsi_tgt ata_generic sata_nv forcedeth libata ehci_hcd scsi_mod dock ohci_hcd thermal processor fan thermal_sys [44955.425007] irq event stamp: 0 [44955.425007] hardirqs last enabled at (0): [<0000000000000000>] 0x0 [44955.425007] hardirqs last disabled at (0): [] copy_process+0x543/0x12b4 [44955.425007] softirqs last enabled at (0): [] copy_process+0x543/0x12b4 [44955.425007] softirqs last disabled at (0): [<0000000000000000>] 0x0 [44955.425007] CPU 2: [44955.425007] Modules linked in: gfs ac battery ipv6 iptable_filter xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables x_tables lock_dlm gfs2 dlm configfs snd_pcm snd_timer snd soundcore snd_page_alloc rtc_cmos rtc_core i2c_nforce2 k8temp shpchp rtc_lib pcspkr pci_hotplug i2c_core button evdev ext3 jbd mbcache dm_mirror dm_log dm_snapshot dm_mod ide_cd_mod cdrom amd74xx sd_mod ide_pci_generic ide_core floppy qla2xxx scsi_transport_fc 3w_9xxx e1000e scsi_tgt ata_generic sata_nv forcedeth libata ehci_hcd scsi_mod dock ohci_hcd thermal processor fan thermal_sys [44955.425007] Pid: 12639, comm: apache Not tainted 2.6.27-2-amd64 #1 [44955.425007] RIP: 0010:[] [] native_read_tsc+0x6/0x18 [44955.425007] RSP: 0018:ffff880214af9d80 EFLAGS: 00000202 [44955.425007] RAX: 0000000000000000 RBX: 00000000498fb129 RCX: ffffffff8085d300 [44955.425007] RDX: 000062bb00000000 RSI: 0000000001062560 RDI: 0000000000000001 [44955.425007] RBP: 0000000000000002 R08: 0000000000000002 R09: 0000000000000000 [44955.425007] R10: 0000000000000000 R11: ffffffff8033dd3e R12: ffff88041f0b0000 [44955.425007] R13: ffff8802abb76000 R14: ffff880214af8000 R15: ffffffff8085a890 [44955.425007] FS: 00007f3e8ea7d6d0(0000) GS:ffff88041f0c9940(0000) knlGS:0000000000000000 [44955.425007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [44955.425007] CR2: 00007f3e8e9fc000 CR3: 0000000214adf000 CR4: 00000000000006e0 [44955.425007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [44955.425007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [44955.425007] [44955.425007] Call Trace: [44955.425007] [] ? delay_tsc+0x15/0x45 [44955.425007] [] ? _raw_spin_lock+0x98/0x100 [44955.425007] [] ? _spin_lock+0x4e/0x5a [44955.425007] [] ? igrab+0x10/0x36 [44955.425007] [] ? igrab+0x10/0x36 [44955.425007] [] ? gfs_getattr+0x83/0xb7 [gfs] [44955.425007] [] ? vfs_getattr+0x1a/0x5e [44955.425007] [] ? vfs_stat_fd+0x2f/0x43 [44955.425007] [] ? sys_newstat+0x19/0x31 [44955.425007] [] ? system_call_fastpath+0x16/0x1b Best regards Frederik Sch?ler -- ENOSIG -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From swhiteho at redhat.com Mon Nov 10 09:24:13 2008 From: swhiteho at redhat.com (Steven Whitehouse) Date: Mon, 10 Nov 2008 09:24:13 +0000 Subject: [Linux-cluster] gfs2 convert hosed all VMs In-Reply-To: <6b6836c60811071953m52d068cfpd0a204f8f4d1b99d@mail.gmail.com> References: <6b6836c60811071953m52d068cfpd0a204f8f4d1b99d@mail.gmail.com> Message-ID: <1226309053.25004.196.camel@quoit> Hi, You don't say what kernel version you are using. I'd suspect that maybe its too old. Do you get any messages in syslog at all? Steve. On Fri, 2008-11-07 at 19:53 -0800, Dave Costakos wrote: > > I just converted a shared file-backed Xen VM GFS filesystem to a GFS2 > filesystem. The conversion was successfully and all my files appear > intact. I followed the GFS instructions by unmounting the filesystem > on all machines, running gfs_fsck, and gfs2_covert. > > Since I converted the filesystem, all my file-backed Xen VMs can no > longer boot. pygrub reports errors that the boot loader isn't > returning any data. If I edit the Xen config to boot of a kernel on > the DomU, VMs still can't start up because LVM cannot identify any > volume groups. > > If I try to access the VM device files locally via losetup and kpartx, > I get "read errors". > > So what's the deal I know GFS2 is a preview, but I have to assume > I've missed some crucial step here. > > -- > Dave Costakos > mailto:david.costakos at gmail.com > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From david.costakos at gmail.com Mon Nov 10 17:00:08 2008 From: david.costakos at gmail.com (Dave Costakos) Date: Mon, 10 Nov 2008 09:00:08 -0800 Subject: [Linux-cluster] gfs2 convert hosed all VMs In-Reply-To: <1226309053.25004.196.camel@quoit> References: <6b6836c60811071953m52d068cfpd0a204f8f4d1b99d@mail.gmail.com> <1226309053.25004.196.camel@quoit> Message-ID: <6b6836c60811100900r1779220ft3d18c1ac9576dbf@mail.gmail.com> I'm using the kernel from RHEL 5.2 release. $ uname -rm 2.6.18-92.el5xen x86_64 I got no syslog messages, but when I ran gfs_fsck, it complained quite a bit. I presume it destroyed my files. Sadly, I don't have too much time to reproduce the error since I need to restore 30 virtual machines from backup. Luckily, it's just our lab environment. On Mon, Nov 10, 2008 at 1:24 AM, Steven Whitehouse wrote: > Hi, > > You don't say what kernel version you are using. I'd suspect that maybe > its too old. Do you get any messages in syslog at all? > > Steve. > > On Fri, 2008-11-07 at 19:53 -0800, Dave Costakos wrote: > > > > I just converted a shared file-backed Xen VM GFS filesystem to a GFS2 > > filesystem. The conversion was successfully and all my files appear > > intact. I followed the GFS instructions by unmounting the filesystem > > on all machines, running gfs_fsck, and gfs2_covert. > > > > Since I converted the filesystem, all my file-backed Xen VMs can no > > longer boot. pygrub reports errors that the boot loader isn't > > returning any data. If I edit the Xen config to boot of a kernel on > > the DomU, VMs still can't start up because LVM cannot identify any > > volume groups. > > > > If I try to access the VM device files locally via losetup and kpartx, > > I get "read errors". > > > > So what's the deal I know GFS2 is a preview, but I have to assume > > I've missed some crucial step here. > > > > -- > > Dave Costakos > > mailto:david.costakos at gmail.com > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Dave Costakos mailto:david.costakos at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From swhiteho at redhat.com Mon Nov 10 17:05:05 2008 From: swhiteho at redhat.com (Steven Whitehouse) Date: Mon, 10 Nov 2008 17:05:05 +0000 Subject: [Linux-cluster] gfs2 convert hosed all VMs In-Reply-To: <6b6836c60811100900r1779220ft3d18c1ac9576dbf@mail.gmail.com> References: <6b6836c60811071953m52d068cfpd0a204f8f4d1b99d@mail.gmail.com> <1226309053.25004.196.camel@quoit> <6b6836c60811100900r1779220ft3d18c1ac9576dbf@mail.gmail.com> Message-ID: <1226336705.25004.220.camel@quoit> Hi, On Mon, 2008-11-10 at 09:00 -0800, Dave Costakos wrote: > I'm using the kernel from RHEL 5.2 release. > > $ uname -rm > 2.6.18-92.el5xen x86_64 > Well that is quite old, but even so I'm surprised that you had a problem like that. What kind of messages did fsck spit out? I can't really debug this without any information on what is going wrong, Steve. > I got no syslog messages, but when I ran gfs_fsck, it complained quite > a bit. I presume it destroyed my files. Sadly, I don't have too much > time to reproduce the error since I need to restore 30 virtual > machines from backup. Luckily, it's just our lab environment. > > On Mon, Nov 10, 2008 at 1:24 AM, Steven Whitehouse > wrote: > Hi, > > You don't say what kernel version you are using. I'd suspect > that maybe > its too old. Do you get any messages in syslog at all? > > Steve. > > > On Fri, 2008-11-07 at 19:53 -0800, Dave Costakos wrote: > > > > I just converted a shared file-backed Xen VM GFS filesystem > to a GFS2 > > filesystem. The conversion was successfully and all my > files appear > > intact. I followed the GFS instructions by unmounting the > filesystem > > on all machines, running gfs_fsck, and gfs2_covert. > > > > Since I converted the filesystem, all my file-backed Xen VMs > can no > > longer boot. pygrub reports errors that the boot loader > isn't > > returning any data. If I edit the Xen config to boot of a > kernel on > > the DomU, VMs still can't start up because LVM cannot > identify any > > volume groups. > > > > If I try to access the VM device files locally via losetup > and kpartx, > > I get "read errors". > > > > So what's the deal I know GFS2 is a preview, but I have to > assume > > I've missed some crucial step here. > > > > -- > > Dave Costakos > > mailto:david.costakos at gmail.com > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Dave Costakos > mailto:david.costakos at gmail.com > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From ffv at tjpr.jus.br Mon Nov 10 18:03:43 2008 From: ffv at tjpr.jus.br (Fabiano F. Vitale) Date: Mon, 10 Nov 2008 16:03:43 -0200 Subject: [Linux-cluster] GFS2 poor performance References: <49109ea6.+nLiDwujAEL50VEe%ffv@tjpr.jus.br><64D0546C5EBBD147B75DE133D798665F01806B7C@hugo.eprize.local><007f01c93eba$8d731b20$3e0a10ac@tjpr.net> <64D0546C5EBBD147B75DE133D798665F01806C15@hugo.eprize.local> Message-ID: <000d01c9435e$abc63410$3e0a10ac@tjpr.net> Setting demote_secs to 30 and glock_purge to 70 in a gfs filesystem increased frightfully performance of commands like ls, df, in a directory that has many files. But the gfs2 filesystem doesn't have the attribute glock_purge to tune. Exists any attribute in gfs2 in place of glock_purge which exists only in gfs1 thanks Fabiano ----- Original Message ----- From: "Jeff Sturm" To: "linux clustering" Sent: Thursday, November 06, 2008 5:53 PM Subject: RE: [Linux-cluster] GFS2 poor performance >I looked over the summit document you referenced below. The value of >demote_secs mentioned is an example setting, and unfortunately no >recommendations or rationale accompany this. > > For some access patterns you can get better performance by actually > increasing demote_secs. For example, we have a node that we routinely > rsync a file tree onto using a GFS partition. Increasing demote_secs from > 300 to 86400 reduced the average rsync time by a factor of about 4. The > reason is that this node has little lock contention and needs to lock each > file every time we start an rsync process. With demote_secs=300, it was > doing much more work to reacquire locks on each run. Whereas > demote_secs=86400 allowed the locks to persist up to a day, since the > overall number of files in our application is bounded such that they will > fit in buffer cache, together with locks. > > At another extreme, we have an application that creates a lot of files but > seldom opens them on the same node. In this case there is no value in > holding onto the locks, so we set demote_secs to a small value and > glock_purge as high as 70 to ensure locks are quickly released in memory. > > The best advice I can give in general is to experiment with different > settings for demote_secs and glock_purge while watching the output of > "gfs_tool counters" to see how they behave. > > Jeff > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Fabiano F. Vitale > Sent: Tuesday, November 04, 2008 3:19 PM > To: linux clustering > Subject: Re: [Linux-cluster] GFS2 poor performance > > Hi, > > for cluster purpose the two nodes are linked by a patch cord cat6 and the > lan interfaces are gigabit. > > All nodes have a Fibre Channel Emulex Corporation Zephyr-X LightPulse and > the Storage is a HP EVA8100 > > I read the document > http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Summit08presentation_GFSBestPractices_Final.pdf > which show some parameters to tune and one of them is demote_secs, to > adjust to 100sec > > thanks > >> What sort of network and storage device are you using? >> >> Also, why set demote_secs so low? >> >> -----Original Message----- >> From: linux-cluster-bounces at redhat.com >> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of ffv at tjpr.jus.br >> Sent: Tuesday, November 04, 2008 2:13 PM >> To: linux-cluster at redhat.com >> Subject: [Linux-cluster] GFS2 poor performance >> >> Hi all, >> >> I?m getting a very poor performance using GFS2. >> I have two qmail (mail) servers and one gfs2 filesystem shared by them. >> In this case, each directory in GFS2 filesystem may have upon to 10000 >> files (mails) >> >> The problem is in performance of some operations like ls, du, rm, etc >> for example, >> >> # time du -sh /dados/teste >> 40M /dados/teste >> >> real 7m14.919s >> user 0m0.008s >> sys 0m0.129s >> >> this is unacceptable >> >> Some attributes i already set using gfs2_tool: >> >> gfs2_tool settune /dados demote_secs 100 gfs2_tool setflag jdata >> /dados gfs2_tool setflag sync /dados gfs2_tool setflag directio /dados >> >> but the performance is still very bad >> >> >> Anybody know how to tune the filesystem for a acceptable performance >> working with directory with 10000 files? >> thanks for any help >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From fdinitto at redhat.com Mon Nov 10 19:09:10 2008 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Mon, 10 Nov 2008 20:09:10 +0100 Subject: [Linux-cluster] logging: final call on configuration, output and implementation Message-ID: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> Hi all, those logging threads have been going on for way too long. It's time to close them and make a final decision. This is a long email, so please take time to read it all. This is a recap of what I believe a user would like to see: - a consistent, easy and quick way to configure logging. - a reasonable default if nothing is specified. - a consistent, easy to read, output. In order to avoid misinterpretation, "debug" priority means DEBUGLO as defined here by David: http://www.redhat.com/archives/cluster-devel/2008-November/msg00002.html == Configuration == XML version: corosync/openais equivalent: logging { debug="on|off"; (read note below about debugging) to_syslog="yes|no"; syslog_facility="local4"; syslog_priority="debug|info|notice|etc.."; to_file="yes|no"; logfile="/var/log/cluster/foo.log"; log_priority="debug|info|notice|etc.."; logger_subsys { subsys="NAME"; debug=.. } } (the two configuration are equivalent) The global configuration (/logging {) contains all the keywords required to configure logging as discussed so far and would affects every daemon or subsystems. The common user would generally stop here. The logger_subsys can be used to do more detailed control over logging. This is where more advance users or developers work. Equivalent entries in logger_subsys override configurations in the global section. Use of environment variables and settings from the command line can override any configuration. This is used by some people to enable debugging via init scripts (for example /etc/sysconfig/cman or equivalent for other distros). == Defaults == - debug = off. Switch to "on" to enable execution of debugging code (read note below) and set log_priority to debug. - to_syslog = yes. - syslog_facility = local4 (default to local4 to respect old behaviour). - syslog_priority = info (default to respect old behaviour and it seems to be a recent compromise at this point of development, doesn't flood logs and has enough info). - to_file = yes - logfile = LOGDIR/[daemon|subsystem].log. By default each daemon/subsystem should log in its own file (mostly valid for daemons as corosync/openais and plugins all share the same file and logging system). - logpriority= info (same reason as above) to_stderr will disappear from the config options. Set to off by default as most daemons will fork in background, we can set it "ON" automatically when not running in background (it's the only case where it actually makes sense). == Output == to_file: echo $(date "+%b %d %T") subsystem_or_daemon: entry_to_log Nov 10 19:46:40 subsystem_or_daemon: entry_to_log It is important to note 2 things. First we need to have a time stamp to be able to compare logs if daemons are logging to different files, then we need to know what subsystem is logging if we are logging to the same file for everything. Since we don't want to create N combinations that will add/remove date/daemon_name etc, this is one format can fit them all and it is the same that would appear in syslog (go for consistency!). to_syslog: openlog("subsystem_or_daemon", options, facility); syslog(priority, "%s", entry_to_log); (consistent with to_file). to_stderr: fprintf(stderr, "%s\n", entry); <- this is really free format. == Implementations requirements == - (strict requirement) must support the configuration options agreed by everybody. - (high priority) logging should never fail. For example, if we can't open a log file we will not fail, we will try to warn by other meanings, but nothing should block our operations if logging is not working. - (wish list) logging should be non-blocking. Best if we can delegate the logging work to something else than ourselves. == Note about debugging == Debugging is an art. Every developer on this planet has a different view of what debugging is. What we want with the debug="on|off" option is an absolute quick way to set all the different log_priority="debug" on and have a flag that can be used in the code to follow different paths. I often find myself setting and be done with everything. No need to remember fancy keywords or understand the whole implementation details on what overrides what and how. At some point in future we should probably talk about debugging in general and what means for all of us but it's outside the scope of this email. Fabio From teigland at redhat.com Mon Nov 10 19:27:30 2008 From: teigland at redhat.com (David Teigland) Date: Mon, 10 Nov 2008 13:27:30 -0600 Subject: [Linux-cluster] Re: [Cluster-devel] logging: final call on configuration, output and implementation In-Reply-To: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> References: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> Message-ID: <20081110192730.GB17894@redhat.com> On Mon, Nov 10, 2008 at 08:09:10PM +0100, Fabio M. Di Nitto wrote: > Hi all, > > those logging threads have been going on for way too long. It's time to > close them and make a final decision. This is a long email, so please > take time to read it all. > > This is a recap of what I believe a user would like to see: > > - a consistent, easy and quick way to configure logging. > - a reasonable default if nothing is specified. > - a consistent, easy to read, output. I like this. Two minor points regarding the actual terminology; I'd like to be a little more consistent and identify some "keywords". Right now the word "log" is combined with other words in a bunch of ways (logging, logger, logfile, syslog, log_foo). How about: . logging, logging_subsys (common keyword "logging") for the config file section tags - to_syslog, syslog_facility, syslog_priority (common keyword "syslog") for every parameter related to syslog - to_logfile, logfile, logfile_priority (common keyword "logfile") for every parameter related to logfile And then we have some values that are "on/off" and others that are "yes/no"; let's pick one. Dave From fdinitto at redhat.com Mon Nov 10 19:48:05 2008 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Mon, 10 Nov 2008 20:48:05 +0100 Subject: [Linux-cluster] Re: [Cluster-devel] logging: final call on configuration, output and implementation In-Reply-To: <20081110192730.GB17894@redhat.com> References: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> <20081110192730.GB17894@redhat.com> Message-ID: <1226346485.2445.68.camel@daitarn-fedora.int.fabbione.net> On Mon, 2008-11-10 at 13:27 -0600, David Teigland wrote: > On Mon, Nov 10, 2008 at 08:09:10PM +0100, Fabio M. Di Nitto wrote: > > Hi all, > > > > those logging threads have been going on for way too long. It's time to > > close them and make a final decision. This is a long email, so please > > take time to read it all. > > > > This is a recap of what I believe a user would like to see: > > > > - a consistent, easy and quick way to configure logging. > > - a reasonable default if nothing is specified. > > - a consistent, easy to read, output. > > I like this. Two minor points regarding the actual terminology; I'd like > to be a little more consistent and identify some "keywords". Right now > the word "log" is combined with other words in a bunch of ways (logging, > logger, logfile, syslog, log_foo). How about: > > . logging, logging_subsys (common keyword "logging") > for the config file section tags > > - to_syslog, syslog_facility, syslog_priority (common keyword "syslog") > for every parameter related to syslog > > - to_logfile, logfile, logfile_priority (common keyword "logfile") > for every parameter related to logfile +1 on all 3 of them. The only reason I did try to avoid keyword changes was to minimize the impact in compat layers since most of those are already in use by us. > > And then we have some values that are "on/off" and others that are > "yes/no"; let's pick one. +1. Given the boolean value, "on/off" sounds more nerdy. Fabio From Joel.Becker at oracle.com Mon Nov 10 20:49:48 2008 From: Joel.Becker at oracle.com (Joel Becker) Date: Mon, 10 Nov 2008 12:49:48 -0800 Subject: [Linux-cluster] Re: [Cluster-devel] logging: final call on configuration, output and implementation In-Reply-To: <20081110192730.GB17894@redhat.com> References: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> <20081110192730.GB17894@redhat.com> Message-ID: <20081110204948.GB12445@mail.oracle.com> On Mon, Nov 10, 2008 at 01:27:30PM -0600, David Teigland wrote: > And then we have some values that are "on/off" and others that are > "yes/no"; let's pick one. I always liked the X resource route: /true|yes|on/i -> "True" /false|no|off/i -> "False" Joel -- "In the beginning, the universe was created. This has made a lot of people very angry, and is generally considered to have been a bad move." - Douglas Adams Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127 From merhar at arlut.utexas.edu Mon Nov 10 21:04:17 2008 From: merhar at arlut.utexas.edu (David Merhar) Date: Mon, 10 Nov 2008 15:04:17 -0600 Subject: [Linux-cluster] GFS2 poor performance (gfs2_tool counters) In-Reply-To: <64D0546C5EBBD147B75DE133D798665F01806C15@hugo.eprize.local> References: <49109ea6.+nLiDwujAEL50VEe%ffv@tjpr.jus.br><64D0546C5EBBD147B75DE133D798665F01806B7C@hugo.eprize.local> <007f01c93eba$8d731b20$3e0a10ac@tjpr.net> <64D0546C5EBBD147B75DE133D798665F01806C15@hugo.eprize.local> Message-ID: <6F4B5DBB-3217-42DA-8BAA-E73AB2E7C5B9@arlut.utexas.edu> Is "gfs2_tool counters" supported? Doesn't work for us, and I found reference to correcting the man page so it's no longer included. Thanks. djm On Nov 6, 2008, at 1:53 PM, Jeff Sturm wrote: > I looked over the summit document you referenced below. The value > of demote_secs mentioned is an example setting, and unfortunately no > recommendations or rationale accompany this. > > For some access patterns you can get better performance by actually > increasing demote_secs. For example, we have a node that we > routinely rsync a file tree onto using a GFS partition. Increasing > demote_secs from 300 to 86400 reduced the average rsync time by a > factor of about 4. The reason is that this node has little lock > contention and needs to lock each file every time we start an rsync > process. With demote_secs=300, it was doing much more work to > reacquire locks on each run. Whereas demote_secs=86400 allowed the > locks to persist up to a day, since the overall number of files in > our application is bounded such that they will fit in buffer cache, > together with locks. > > At another extreme, we have an application that creates a lot of > files but seldom opens them on the same node. In this case there is > no value in holding onto the locks, so we set demote_secs to a small > value and glock_purge as high as 70 to ensure locks are quickly > released in memory. > > The best advice I can give in general is to experiment with > different settings for demote_secs and glock_purge while watching > the output of "gfs_tool counters" to see how they behave. > > Jeff > > -----Original Message----- > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com > ] On Behalf Of Fabiano F. Vitale > Sent: Tuesday, November 04, 2008 3:19 PM > To: linux clustering > Subject: Re: [Linux-cluster] GFS2 poor performance > > Hi, > > for cluster purpose the two nodes are linked by a patch cord cat6 > and the lan interfaces are gigabit. > > All nodes have a Fibre Channel Emulex Corporation Zephyr-X > LightPulse and the Storage is a HP EVA8100 > > I read the document > http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Summit08presentation_GFSBestPractices_Final.pdf > which show some parameters to tune and one of them is demote_secs, > to adjust to 100sec > > thanks > >> What sort of network and storage device are you using? >> >> Also, why set demote_secs so low? >> >> -----Original Message----- >> From: linux-cluster-bounces at redhat.com >> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of >> ffv at tjpr.jus.br >> Sent: Tuesday, November 04, 2008 2:13 PM >> To: linux-cluster at redhat.com >> Subject: [Linux-cluster] GFS2 poor performance >> >> Hi all, >> >> I?m getting a very poor performance using GFS2. >> I have two qmail (mail) servers and one gfs2 filesystem shared by >> them. >> In this case, each directory in GFS2 filesystem may have upon to >> 10000 >> files (mails) >> >> The problem is in performance of some operations like ls, du, rm, etc >> for example, >> >> # time du -sh /dados/teste >> 40M /dados/teste >> >> real 7m14.919s >> user 0m0.008s >> sys 0m0.129s >> >> this is unacceptable >> >> Some attributes i already set using gfs2_tool: >> >> gfs2_tool settune /dados demote_secs 100 gfs2_tool setflag jdata >> /dados gfs2_tool setflag sync /dados gfs2_tool setflag directio / >> dados >> >> but the performance is still very bad >> >> >> Anybody know how to tune the filesystem for a acceptable performance >> working with directory with 10000 files? >> thanks for any help >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From sdake at redhat.com Tue Nov 11 00:46:28 2008 From: sdake at redhat.com (Steven Dake) Date: Mon, 10 Nov 2008 17:46:28 -0700 Subject: [Linux-cluster] Re: [Cluster-devel] logging: final call on configuration, output and implementation In-Reply-To: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> References: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> Message-ID: <1226364388.14398.7.camel@balance> I disagree with a global debug keyword. At one time I thought it was a good idea but that time has long since passed. The idea of turning debug to on and then having all debug output go to syslog is frightening and will result in lost messages. While it appears this proposal includes the selectable log output filtering per output medium as was discussed already, it is unclear how the debug keyword affects this. It would simply make sense to change the file's log priority or the syslog's log priority if that is the behavior desired and then no need for any extra keyword. Regards -steve On Mon, 2008-11-10 at 20:09 +0100, Fabio M. Di Nitto wrote: > Hi all, > > those logging threads have been going on for way too long. It's time to > close them and make a final decision. This is a long email, so please > take time to read it all. > > This is a recap of what I believe a user would like to see: > > - a consistent, easy and quick way to configure logging. > - a reasonable default if nothing is specified. > - a consistent, easy to read, output. > > In order to avoid misinterpretation, "debug" priority means DEBUGLO as > defined here by David: > http://www.redhat.com/archives/cluster-devel/2008-November/msg00002.html > > == Configuration == > > XML version: > > to_syslog="yes|no" > syslog_facility="local4" > syslog_priority="debug|info|notice|etc.." > to_file="yes|no" > logfile="/var/log/cluster/foo.log" > log_priority="debug|info|notice|etc.."> > > > > > corosync/openais equivalent: > > logging { > debug="on|off"; (read note below about debugging) > > to_syslog="yes|no"; > syslog_facility="local4"; > syslog_priority="debug|info|notice|etc.."; > > to_file="yes|no"; > logfile="/var/log/cluster/foo.log"; > log_priority="debug|info|notice|etc.."; > > logger_subsys { > subsys="NAME"; > debug=.. > } > } > > (the two configuration are equivalent) > > The global configuration (/logging {) contains all the keywords > required to configure logging as discussed so far and would affects > every daemon or subsystems. The common user would generally stop here. > > The logger_subsys can be used to do more detailed control over logging. > This is where more advance users or developers work. > > Equivalent entries in logger_subsys override configurations in the > global section. > > Use of environment variables and settings from the command line can > override any configuration. This is used by some people to enable > debugging via init scripts (for example /etc/sysconfig/cman or > equivalent for other distros). > > > == Defaults == > > - debug = off. Switch to "on" to enable execution of debugging code > (read note below) and set log_priority to debug. > > - to_syslog = yes. > - syslog_facility = local4 (default to local4 to respect old behaviour). > - syslog_priority = info (default to respect old behaviour and it seems > to be a recent compromise at this point of development, doesn't flood > logs and has enough info). > > - to_file = yes > - logfile = LOGDIR/[daemon|subsystem].log. By default each > daemon/subsystem should log in its own file (mostly valid for daemons as > corosync/openais and plugins all share the same file and logging > system). > - logpriority= info (same reason as above) > > to_stderr will disappear from the config options. Set to off by default > as most daemons will fork in background, we can set it "ON" > automatically when not running in background (it's the only case where > it actually makes sense). > > > == Output == > > to_file: > > echo $(date "+%b %d %T") subsystem_or_daemon: entry_to_log > Nov 10 19:46:40 subsystem_or_daemon: entry_to_log > > It is important to note 2 things. First we need to have a time stamp to > be able to compare logs if daemons are logging to different files, then > we need to know what subsystem is logging if we are logging to the same > file for everything. > > Since we don't want to create N combinations that will add/remove > date/daemon_name etc, this is one format can fit them all and it is the > same that would appear in syslog (go for consistency!). > > to_syslog: > > openlog("subsystem_or_daemon", options, facility); > syslog(priority, "%s", entry_to_log); > > (consistent with to_file). > > to_stderr: > > fprintf(stderr, "%s\n", entry); <- this is really free format. > > > == Implementations requirements == > > - (strict requirement) must support the configuration options agreed by > everybody. > > - (high priority) logging should never fail. For example, if we can't > open a log file we will not fail, we will try to warn by other meanings, > but nothing should block our operations if logging is not working. > > - (wish list) logging should be non-blocking. Best if we can delegate > the logging work to something else than ourselves. > > > == Note about debugging == > > Debugging is an art. Every developer on this planet has a different view > of what debugging is. What we want with the debug="on|off" option is an > absolute quick way to set all the different log_priority="debug" on and > have a flag that can be used in the code to follow different paths. > > I often find myself setting and be done with > everything. No need to remember fancy keywords or understand the whole > implementation details on what overrides what and how. > > At some point in future we should probably talk about debugging in > general and what means for all of us but it's outside the scope of this > email. > > Fabio > From fdinitto at redhat.com Tue Nov 11 04:55:12 2008 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Tue, 11 Nov 2008 05:55:12 +0100 Subject: [Linux-cluster] Re: [Cluster-devel] logging: final call on configuration, output and implementation In-Reply-To: <1226364388.14398.7.camel@balance> References: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> <1226364388.14398.7.camel@balance> Message-ID: <1226379312.2445.73.camel@daitarn-fedora.int.fabbione.net> On Mon, 2008-11-10 at 17:46 -0700, Steven Dake wrote: > I disagree with a global debug keyword. > At one time I thought it was a > good idea but that time has long since passed. The idea of turning > debug to on and then having all debug output go to syslog is frightening > and will result in lost messages. While it appears this proposal > includes the selectable log output filtering per output medium as was > discussed already, it is unclear how the debug keyword affects this. It > would simply make sense to change the file's log priority or the > syslog's log priority if that is the behavior desired and then no need > for any extra keyword. You have these two situations: print_log(LOG_DEBUG, "doing this and that....\n"); if (debug) { /* gather_some_data_that_is_very_expensive_operation_to_do_all_the_time(); print_log(LOG_DEBUG, "print those extra data\n"); } as it is now, it would basically be an alias to set logpriority to DEBUG but enables people to execute debugging code conditionally and as I wrote it is an easy keyword to remember compared to syslog_priority/logpriority. Fabio From sdake at redhat.com Tue Nov 11 05:47:23 2008 From: sdake at redhat.com (Steven Dake) Date: Mon, 10 Nov 2008 22:47:23 -0700 Subject: [Linux-cluster] Re: [Cluster-devel] logging: final call on configuration, output and implementation In-Reply-To: <1226379312.2445.73.camel@daitarn-fedora.int.fabbione.net> References: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> <1226364388.14398.7.camel@balance> <1226379312.2445.73.camel@daitarn-fedora.int.fabbione.net> Message-ID: <1226382444.14398.18.camel@balance> On Tue, 2008-11-11 at 05:55 +0100, Fabio M. Di Nitto wrote: > On Mon, 2008-11-10 at 17:46 -0700, Steven Dake wrote: > > I disagree with a global debug keyword. > > At one time I thought it was a > > good idea but that time has long since passed. The idea of turning > > debug to on and then having all debug output go to syslog is frightening > > and will result in lost messages. While it appears this proposal > > includes the selectable log output filtering per output medium as was > > discussed already, it is unclear how the debug keyword affects this. It > > would simply make sense to change the file's log priority or the > > syslog's log priority if that is the behavior desired and then no need > > for any extra keyword. > > You have these two situations: > > print_log(LOG_DEBUG, "doing this and that....\n"); > > if (debug) { /* > gather_some_data_that_is_very_expensive_operation_to_do_all_the_time(); > print_log(LOG_DEBUG, "print those extra data\n"); > } > > as it is now, it would basically be an alias to set logpriority to DEBUG > but enables people to execute debugging code conditionally and as I > wrote it is an easy keyword to remember compared to > syslog_priority/logpriority. > > Fabio > The second situation doesn't exist in any code I have written and never would. Having any conditional debug output is asking for trouble. Been down that road, done that, and discarded that idea... The "debughi" or high volume debug messages do not go through log_printf nor would they be committed to any persistent log (only memory). The output of the logging message is significantly more expensive then that of gathering logging data. Turning debug on for all of the entire stack to be output to syslog is not satisfactory because messages would be lost in overload conditions. Logging to file is only a slight bit better solution but if you really must have debug output in a persistent store that doesn't occur as a result of a failure, logging to file is the only suitable answer. A global debug option without selecting log output is not a workable solution because of overload of syslog, even overload of the filesystem, or other issues. What makes sense is to have a mechanism to set the priority for each specific log output mechanism and forget about any global debug option nonsense. Regards -steve From fdinitto at redhat.com Tue Nov 11 05:54:00 2008 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Tue, 11 Nov 2008 06:54:00 +0100 Subject: [Linux-cluster] Re: [Cluster-devel] logging: final call on configuration, output and implementation In-Reply-To: <1226382444.14398.18.camel@balance> References: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> <1226364388.14398.7.camel@balance> <1226379312.2445.73.camel@daitarn-fedora.int.fabbione.net> <1226382444.14398.18.camel@balance> Message-ID: <1226382840.2445.78.camel@daitarn-fedora.int.fabbione.net> On Mon, 2008-11-10 at 22:47 -0700, Steven Dake wrote: > On Tue, 2008-11-11 at 05:55 +0100, Fabio M. Di Nitto wrote: > > On Mon, 2008-11-10 at 17:46 -0700, Steven Dake wrote: > > > I disagree with a global debug keyword. > > > At one time I thought it was a > > > good idea but that time has long since passed. The idea of turning > > > debug to on and then having all debug output go to syslog is frightening > > > and will result in lost messages. While it appears this proposal > > > includes the selectable log output filtering per output medium as was > > > discussed already, it is unclear how the debug keyword affects this. It > > > would simply make sense to change the file's log priority or the > > > syslog's log priority if that is the behavior desired and then no need > > > for any extra keyword. > > > > You have these two situations: > > > > print_log(LOG_DEBUG, "doing this and that....\n"); > > > > if (debug) { /* > > gather_some_data_that_is_very_expensive_operation_to_do_all_the_time(); > > print_log(LOG_DEBUG, "print those extra data\n"); > > } > > > > as it is now, it would basically be an alias to set logpriority to DEBUG > > but enables people to execute debugging code conditionally and as I > > wrote it is an easy keyword to remember compared to > > syslog_priority/logpriority. > > > > Fabio > > > > The second situation doesn't exist in any code I have written and never > would. Clearly you haven't read what I wrote in the debugging note. > Turning debug on for all of the entire stack to be output to syslog is > not satisfactory because messages would be lost in overload conditions. > Logging to file is only a slight bit better solution but if you really > must have debug output in a persistent store that doesn't occur as a > result of a failure, logging to file is the only suitable answer. Please point me to where I wrote that it should go to syslog as I only mentioned logfile_priority so far. Fabio From sdake at redhat.com Tue Nov 11 06:00:58 2008 From: sdake at redhat.com (Steven Dake) Date: Mon, 10 Nov 2008 23:00:58 -0700 Subject: [Linux-cluster] Re: [Cluster-devel] logging: final call on configuration, output and implementation In-Reply-To: <1226382840.2445.78.camel@daitarn-fedora.int.fabbione.net> References: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> <1226364388.14398.7.camel@balance> <1226379312.2445.73.camel@daitarn-fedora.int.fabbione.net> <1226382444.14398.18.camel@balance> <1226382840.2445.78.camel@daitarn-fedora.int.fabbione.net> Message-ID: <1226383258.14398.31.camel@balance> On Tue, 2008-11-11 at 06:54 +0100, Fabio M. Di Nitto wrote: > On Mon, 2008-11-10 at 22:47 -0700, Steven Dake wrote: > > On Tue, 2008-11-11 at 05:55 +0100, Fabio M. Di Nitto wrote: > > > On Mon, 2008-11-10 at 17:46 -0700, Steven Dake wrote: > > > > I disagree with a global debug keyword. > > > > At one time I thought it was a > > > > good idea but that time has long since passed. The idea of turning > > > > debug to on and then having all debug output go to syslog is frightening > > > > and will result in lost messages. While it appears this proposal > > > > includes the selectable log output filtering per output medium as was > > > > discussed already, it is unclear how the debug keyword affects this. It > > > > would simply make sense to change the file's log priority or the > > > > syslog's log priority if that is the behavior desired and then no need > > > > for any extra keyword. > > > > > > You have these two situations: > > > > > > print_log(LOG_DEBUG, "doing this and that....\n"); > > > > > > if (debug) { /* > > > gather_some_data_that_is_very_expensive_operation_to_do_all_the_time(); > > > print_log(LOG_DEBUG, "print those extra data\n"); > > > } > > > > > > as it is now, it would basically be an alias to set logpriority to DEBUG > > > but enables people to execute debugging code conditionally and as I > > > wrote it is an easy keyword to remember compared to > > > syslog_priority/logpriority. > > > > > > Fabio > > > > > > > The second situation doesn't exist in any code I have written and never > > would. > > Clearly you haven't read what I wrote in the debugging note. > I read it but don't agree you can have a discussion about logging and flight recording without discussing how debugging fits into the log system. > > Turning debug on for all of the entire stack to be output to syslog is > > not satisfactory because messages would be lost in overload conditions. > > Logging to file is only a slight bit better solution but if you really > > must have debug output in a persistent store that doesn't occur as a > > result of a failure, logging to file is the only suitable answer. > > Please point me to where I wrote that it should go to syslog as I only > mentioned logfile_priority so far. > If syslog is configured it will go to syslog by default in your scheme. Regards -steve > Fabio > From maurizio.rottin at gmail.com Tue Nov 11 08:26:05 2008 From: maurizio.rottin at gmail.com (Maurizio Rottin) Date: Tue, 11 Nov 2008 09:26:05 +0100 Subject: [Linux-cluster] Fence VirtualIron - i have the script but... Message-ID: Hello everyone, I need to fence VirtualIron VM in order to GFS to work when a node is not responding. Actually, i wrote a simple python script that fences the node, but...i don't understand how to integrate it in the cluster suite! What files should i touch in order to have this fence method in luci? -- mr From swhiteho at redhat.com Tue Nov 11 09:23:43 2008 From: swhiteho at redhat.com (Steven Whitehouse) Date: Tue, 11 Nov 2008 09:23:43 +0000 Subject: [Linux-cluster] GFS2 poor performance In-Reply-To: <000d01c9435e$abc63410$3e0a10ac@tjpr.net> References: <49109ea6.+nLiDwujAEL50VEe%ffv@tjpr.jus.br> <64D0546C5EBBD147B75DE133D798665F01806B7C@hugo.eprize.local> <007f01c93eba$8d731b20$3e0a10ac@tjpr.net> <64D0546C5EBBD147B75DE133D798665F01806C15@hugo.eprize.local> <000d01c9435e$abc63410$3e0a10ac@tjpr.net> Message-ID: <1226395423.9046.3.camel@quoit> Hi, On Mon, 2008-11-10 at 16:03 -0200, Fabiano F. Vitale wrote: > Setting demote_secs to 30 and glock_purge to 70 in a gfs filesystem > increased frightfully performance of commands like ls, df, in a directory > that has many files. > But the gfs2 filesystem doesn't have the attribute glock_purge to tune. > Exists any attribute in gfs2 in place of glock_purge which exists only in > gfs1 > > thanks > > Fabiano > > That is entirely deliberate. GFS2 is self-tuning so far as glocks goes, so such settings are not needed. The demote time setting for glocks in GFS2 only applies to non-inode glocks and it might well go away in the future when we have an automatic way to deal with them too. One of the goals of GFS2 is to reduce the need for users to have to change obscure settings in order to get the best performance in any particular situation, Steve. > ----- Original Message ----- > From: "Jeff Sturm" > To: "linux clustering" > Sent: Thursday, November 06, 2008 5:53 PM > Subject: RE: [Linux-cluster] GFS2 poor performance > > > >I looked over the summit document you referenced below. The value of > >demote_secs mentioned is an example setting, and unfortunately no > >recommendations or rationale accompany this. > > > > For some access patterns you can get better performance by actually > > increasing demote_secs. For example, we have a node that we routinely > > rsync a file tree onto using a GFS partition. Increasing demote_secs from > > 300 to 86400 reduced the average rsync time by a factor of about 4. The > > reason is that this node has little lock contention and needs to lock each > > file every time we start an rsync process. With demote_secs=300, it was > > doing much more work to reacquire locks on each run. Whereas > > demote_secs=86400 allowed the locks to persist up to a day, since the > > overall number of files in our application is bounded such that they will > > fit in buffer cache, together with locks. > > > > At another extreme, we have an application that creates a lot of files but > > seldom opens them on the same node. In this case there is no value in > > holding onto the locks, so we set demote_secs to a small value and > > glock_purge as high as 70 to ensure locks are quickly released in memory. > > > > The best advice I can give in general is to experiment with different > > settings for demote_secs and glock_purge while watching the output of > > "gfs_tool counters" to see how they behave. > > > > Jeff > > > > -----Original Message----- > > From: linux-cluster-bounces at redhat.com > > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Fabiano F. Vitale > > Sent: Tuesday, November 04, 2008 3:19 PM > > To: linux clustering > > Subject: Re: [Linux-cluster] GFS2 poor performance > > > > Hi, > > > > for cluster purpose the two nodes are linked by a patch cord cat6 and the > > lan interfaces are gigabit. > > > > All nodes have a Fibre Channel Emulex Corporation Zephyr-X LightPulse and > > the Storage is a HP EVA8100 > > > > I read the document > > http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Summit08presentation_GFSBestPractices_Final.pdf > > which show some parameters to tune and one of them is demote_secs, to > > adjust to 100sec > > > > thanks > > > >> What sort of network and storage device are you using? > >> > >> Also, why set demote_secs so low? > >> > >> -----Original Message----- > >> From: linux-cluster-bounces at redhat.com > >> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of ffv at tjpr.jus.br > >> Sent: Tuesday, November 04, 2008 2:13 PM > >> To: linux-cluster at redhat.com > >> Subject: [Linux-cluster] GFS2 poor performance > >> > >> Hi all, > >> > >> I?m getting a very poor performance using GFS2. > >> I have two qmail (mail) servers and one gfs2 filesystem shared by them. > >> In this case, each directory in GFS2 filesystem may have upon to 10000 > >> files (mails) > >> > >> The problem is in performance of some operations like ls, du, rm, etc > >> for example, > >> > >> # time du -sh /dados/teste > >> 40M /dados/teste > >> > >> real 7m14.919s > >> user 0m0.008s > >> sys 0m0.129s > >> > >> this is unacceptable > >> > >> Some attributes i already set using gfs2_tool: > >> > >> gfs2_tool settune /dados demote_secs 100 gfs2_tool setflag jdata > >> /dados gfs2_tool setflag sync /dados gfs2_tool setflag directio /dados > >> > >> but the performance is still very bad > >> > >> > >> Anybody know how to tune the filesystem for a acceptable performance > >> working with directory with 10000 files? > >> thanks for any help > >> > >> -- > >> Linux-cluster mailing list > >> Linux-cluster at redhat.com > >> https://www.redhat.com/mailman/listinfo/linux-cluster > >> > >> > >> > >> -- > >> Linux-cluster mailing list > >> Linux-cluster at redhat.com > >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From swhiteho at redhat.com Tue Nov 11 09:27:09 2008 From: swhiteho at redhat.com (Steven Whitehouse) Date: Tue, 11 Nov 2008 09:27:09 +0000 Subject: [Linux-cluster] GFS2 poor performance (gfs2_tool counters) In-Reply-To: <6F4B5DBB-3217-42DA-8BAA-E73AB2E7C5B9@arlut.utexas.edu> References: <49109ea6.+nLiDwujAEL50VEe%ffv@tjpr.jus.br> <64D0546C5EBBD147B75DE133D798665F01806B7C@hugo.eprize.local> <007f01c93eba$8d731b20$3e0a10ac@tjpr.net> <64D0546C5EBBD147B75DE133D798665F01806C15@hugo.eprize.local> <6F4B5DBB-3217-42DA-8BAA-E73AB2E7C5B9@arlut.utexas.edu> Message-ID: <1226395629.9046.7.camel@quoit> Hi, On Mon, 2008-11-10 at 15:04 -0600, David Merhar wrote: > Is "gfs2_tool counters" supported? > > Doesn't work for us, and I found reference to correcting the man page > so it's no longer included. > > Thanks. > > djm > > No, it isn't supported any more. There are plenty of existing methods of tracing the actions of the filesystem, such as strace, blktrace, and more recently FIEMAP so that the counters are no longer needed, Steve. > > On Nov 6, 2008, at 1:53 PM, Jeff Sturm wrote: > > > I looked over the summit document you referenced below. The value > > of demote_secs mentioned is an example setting, and unfortunately no > > recommendations or rationale accompany this. > > > > For some access patterns you can get better performance by actually > > increasing demote_secs. For example, we have a node that we > > routinely rsync a file tree onto using a GFS partition. Increasing > > demote_secs from 300 to 86400 reduced the average rsync time by a > > factor of about 4. The reason is that this node has little lock > > contention and needs to lock each file every time we start an rsync > > process. With demote_secs=300, it was doing much more work to > > reacquire locks on each run. Whereas demote_secs=86400 allowed the > > locks to persist up to a day, since the overall number of files in > > our application is bounded such that they will fit in buffer cache, > > together with locks. > > > > At another extreme, we have an application that creates a lot of > > files but seldom opens them on the same node. In this case there is > > no value in holding onto the locks, so we set demote_secs to a small > > value and glock_purge as high as 70 to ensure locks are quickly > > released in memory. > > > > The best advice I can give in general is to experiment with > > different settings for demote_secs and glock_purge while watching > > the output of "gfs_tool counters" to see how they behave. > > > > Jeff > > > > -----Original Message----- > > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com > > ] On Behalf Of Fabiano F. Vitale > > Sent: Tuesday, November 04, 2008 3:19 PM > > To: linux clustering > > Subject: Re: [Linux-cluster] GFS2 poor performance > > > > Hi, > > > > for cluster purpose the two nodes are linked by a patch cord cat6 > > and the lan interfaces are gigabit. > > > > All nodes have a Fibre Channel Emulex Corporation Zephyr-X > > LightPulse and the Storage is a HP EVA8100 > > > > I read the document > > http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Summit08presentation_GFSBestPractices_Final.pdf > > which show some parameters to tune and one of them is demote_secs, > > to adjust to 100sec > > > > thanks > > > >> What sort of network and storage device are you using? > >> > >> Also, why set demote_secs so low? > >> > >> -----Original Message----- > >> From: linux-cluster-bounces at redhat.com > >> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of > >> ffv at tjpr.jus.br > >> Sent: Tuesday, November 04, 2008 2:13 PM > >> To: linux-cluster at redhat.com > >> Subject: [Linux-cluster] GFS2 poor performance > >> > >> Hi all, > >> > >> I?m getting a very poor performance using GFS2. > >> I have two qmail (mail) servers and one gfs2 filesystem shared by > >> them. > >> In this case, each directory in GFS2 filesystem may have upon to > >> 10000 > >> files (mails) > >> > >> The problem is in performance of some operations like ls, du, rm, etc > >> for example, > >> > >> # time du -sh /dados/teste > >> 40M /dados/teste > >> > >> real 7m14.919s > >> user 0m0.008s > >> sys 0m0.129s > >> > >> this is unacceptable > >> > >> Some attributes i already set using gfs2_tool: > >> > >> gfs2_tool settune /dados demote_secs 100 gfs2_tool setflag jdata > >> /dados gfs2_tool setflag sync /dados gfs2_tool setflag directio / > >> dados > >> > >> but the performance is still very bad > >> > >> > >> Anybody know how to tune the filesystem for a acceptable performance > >> working with directory with 10000 files? > >> thanks for any help > >> > >> -- > >> Linux-cluster mailing list > >> Linux-cluster at redhat.com > >> https://www.redhat.com/mailman/listinfo/linux-cluster > >> > >> > >> > >> -- > >> Linux-cluster mailing list > >> Linux-cluster at redhat.com > >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From finnzi at finnzi.com Tue Nov 11 11:25:32 2008 From: finnzi at finnzi.com (Finnur =?iso-8859-1?Q?=D6rn_Gu=F0mundsson?=) Date: Tue, 11 Nov 2008 11:25:32 -0000 (GMT) Subject: [Linux-cluster] Multiple oracle databases with RHCS Message-ID: <60659.217.28.182.1.1226402732.squirrel@webmail.finnzi.com> Hi, I'm running 3 RHCS clusters that have one Oracle database. Now i need to configure a RHCS cluster that will be running 4 databases, but from my initial testing i can see i need to have a special user for each database. However, in other cluster software we are used to use only single oracle user (ie: ServiceGuard) and we've had no issues there. So i am wondering, is there any reason why Red Hat has choosen to do this like this or would it be ok if i would just modify the oracledb.sh script so i can have a single user....would it still be supported ? There is hardly any documentation (that i could find) regarding running multiple databases on a RHCS cluster. If someone can point me to a good document regarding this please do! :) Thanks in advanced, Finnur From fdinitto at redhat.com Tue Nov 11 18:11:06 2008 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Tue, 11 Nov 2008 19:11:06 +0100 Subject: [Linux-cluster] Re: [Cluster-devel] logging: final call on configuration, output and implementation In-Reply-To: <1226383258.14398.31.camel@balance> References: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> <1226364388.14398.7.camel@balance> <1226379312.2445.73.camel@daitarn-fedora.int.fabbione.net> <1226382444.14398.18.camel@balance> <1226382840.2445.78.camel@daitarn-fedora.int.fabbione.net> <1226383258.14398.31.camel@balance> Message-ID: <1226427066.2445.82.camel@daitarn-fedora.int.fabbione.net> Just for the record, Steven and I had a chat on IRC. This is the transcript: 07:51 < riley_dt> yes i read your original message 07:52 < riley_dt> i was reswponding to your email not dave's 07:52 < riley_dt> your right i had not read dave's message at all 07:53 < fabbione> riley_dt: well.. nothing I can do about that, but the thread is evolving in a positive direction IMHO. There are only few bits that needs smoothing 07:53 < riley_dt> i see the correction in dave's email 07:53 < fabbione> and I am simply talking from the end of the thread. I can't possible know what have you read or not 07:54 < riley_dt> well you can put debug in the top level config 07:54 < riley_dt> but i dont intend to do anything about it :) 07:55 < fabbione> riley_dt: and that _IS_ fine :) 07:55 < fabbione> riley_dt: allow others to use it if they want 07:55 < riley_dt> also the oring has to go 07:55 < fabbione> oring? 07:55 < riley_dt> DEBUG|WARN 07:55 < riley_dt> should be DEBUG then debug+ are logged 07:55 < fabbione> there is no oring... 07:55 < fabbione> it's an example to show what values can be there 07:55 < fabbione> you can't ORING priorities 07:56 < riley_dt> you could or select priorities 07:56 < riley_dt> i thought that is what you proposed 07:56 < riley_dt> then we are good to go 07:56 < fabbione> no.. simply a list of options 07:56 < fabbione> i received some comments from other people in my inbox that I need to sort out 07:56 < fabbione> they didn't go to mailing lists 07:56 < riley_dt> well for trace there is oring 07:56 < riley_dt> ie: TRACE1|TRACE2 07:57 < riley_dt> it isn't "trace 1 or trace2" it is an oring 07:57 < fabbione> some are really good points.. so before everybody dives into implementing, let's wait for the "final email" 07:57 < fabbione> gotcha... 07:57 < riley_dt> when I see | I think "or" 07:57 < riley_dt> overloaded term i guess 08:49 < fabbione> riley_dt: just for the record, can you please reply to thread and tell that you understood what we have been talking about? I strongly want to avoid the situation where in 2 months from now we will be fighting again:" I said this, not this, understood that etc." 10:11 < riley_dt> sure On Mon, 2008-11-10 at 23:00 -0700, Steven Dake wrote: > On Tue, 2008-11-11 at 06:54 +0100, Fabio M. Di Nitto wrote: > > On Mon, 2008-11-10 at 22:47 -0700, Steven Dake wrote: > > > On Tue, 2008-11-11 at 05:55 +0100, Fabio M. Di Nitto wrote: > > > > On Mon, 2008-11-10 at 17:46 -0700, Steven Dake wrote: > > > > > I disagree with a global debug keyword. > > > > > At one time I thought it was a > > > > > good idea but that time has long since passed. The idea of turning > > > > > debug to on and then having all debug output go to syslog is frightening > > > > > and will result in lost messages. While it appears this proposal > > > > > includes the selectable log output filtering per output medium as was > > > > > discussed already, it is unclear how the debug keyword affects this. It > > > > > would simply make sense to change the file's log priority or the > > > > > syslog's log priority if that is the behavior desired and then no need > > > > > for any extra keyword. > > > > > > > > You have these two situations: > > > > > > > > print_log(LOG_DEBUG, "doing this and that....\n"); > > > > > > > > if (debug) { /* > > > > gather_some_data_that_is_very_expensive_operation_to_do_all_the_time(); > > > > print_log(LOG_DEBUG, "print those extra data\n"); > > > > } > > > > > > > > as it is now, it would basically be an alias to set logpriority to DEBUG > > > > but enables people to execute debugging code conditionally and as I > > > > wrote it is an easy keyword to remember compared to > > > > syslog_priority/logpriority. > > > > > > > > Fabio > > > > > > > > > > The second situation doesn't exist in any code I have written and never > > > would. > > > > Clearly you haven't read what I wrote in the debugging note. > > > > I read it but don't agree you can have a discussion about logging and > flight recording without discussing how debugging fits into the log > system. > > > > > Turning debug on for all of the entire stack to be output to syslog is > > > not satisfactory because messages would be lost in overload conditions. > > > Logging to file is only a slight bit better solution but if you really > > > must have debug output in a persistent store that doesn't occur as a > > > result of a failure, logging to file is the only suitable answer. > > > > Please point me to where I wrote that it should go to syslog as I only > > mentioned logfile_priority so far. > > > > If syslog is configured it will go to syslog by default in your scheme. > > Regards > -steve > > > Fabio > > > From garromo at us.ibm.com Tue Nov 11 18:22:18 2008 From: garromo at us.ibm.com (Gary Romo) Date: Tue, 11 Nov 2008 11:22:18 -0700 Subject: [Linux-cluster] Multiple oracle databases with RHCS In-Reply-To: <60659.217.28.182.1.1226402732.squirrel@webmail.finnzi.com> Message-ID: We run multiple oracle databases on our clusters, and they all use the one oracle account. I'm not clear as to what testing you did tha determined you needed one account per database. Gary Romo Finnur ?rn Gu?mundsson linux-cluster at redhat.com Sent by: cc linux-cluster-bou nces at redhat.com Subject [Linux-cluster] Multiple oracle databases with RHCS 11/11/2008 04:25 AM Please respond to linux clustering Hi, I'm running 3 RHCS clusters that have one Oracle database. Now i need to configure a RHCS cluster that will be running 4 databases, but from my initial testing i can see i need to have a special user for each database. However, in other cluster software we are used to use only single oracle user (ie: ServiceGuard) and we've had no issues there. So i am wondering, is there any reason why Red Hat has choosen to do this like this or would it be ok if i would just modify the oracledb.sh script so i can have a single user....would it still be supported ? There is hardly any documentation (that i could find) regarding running multiple databases on a RHCS cluster. If someone can point me to a good document regarding this please do! :) Thanks in advanced, Finnur -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pic08461.gif Type: image/gif Size: 1255 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: From lhh at redhat.com Tue Nov 11 19:21:31 2008 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 11 Nov 2008 14:21:31 -0500 Subject: [Linux-cluster] Multiple oracle databases with RHCS In-Reply-To: <60659.217.28.182.1.1226402732.squirrel@webmail.finnzi.com> References: <60659.217.28.182.1.1226402732.squirrel@webmail.finnzi.com> Message-ID: <1226431291.16686.55.camel@ayanami> On Tue, 2008-11-11 at 11:25 +0000, Finnur ?rn Gu?mundsson wrote: > Hi, > > I'm running 3 RHCS clusters that have one Oracle database. > > Now i need to configure a RHCS cluster that will be running 4 databases, > but from my initial testing i can see i need to have a special user for > each database. > However, in other cluster software we are used to use only single oracle > user (ie: ServiceGuard) and we've had no issues there. So i am wondering, > is there any reason why Red Hat has choosen to do this like this or would > it be ok if i would just modify the oracledb.sh script so i can have a > single user....would it still be supported ? You can edit it, but you need to modify the oracledb.sh script to not kill all processes. There's a bugzilla open about this one: https://bugzilla.redhat.com/show_bug.cgi?id=458481 In the bugzilla is a better approximation of a multiple instance Oracledb.sh with a bug in it (and possible fix). We don't have a lot of reports on people running this in larger environments, so any feedback you provide will be helpful. -- Lon From lhh at redhat.com Tue Nov 11 22:26:20 2008 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 11 Nov 2008 17:26:20 -0500 Subject: [Linux-cluster] Fence VirtualIron - i have the script but... In-Reply-To: References: Message-ID: <1226442380.7132.65.camel@ayanami> On Tue, 2008-11-11 at 09:26 +0100, Maurizio Rottin wrote: > Hello everyone, > I need to fence VirtualIron VM in order to GFS to work when a node is > not responding. > > Actually, i wrote a simple python script that fences the node, but...i > don't understand how to integrate it in the cluster suite! > > What files should i touch in order to have this fence method in luci? You'd have to write a UI for it; I am not sure how to do it. However... http://sources.redhat.com/cluster/wiki/FenceAgentAPI If your agent follows those guidelines, you can just stick the agent directives and configuration in cluster.conf. -- Lon From maurizio.rottin at gmail.com Tue Nov 11 23:36:35 2008 From: maurizio.rottin at gmail.com (Maurizio Rottin) Date: Wed, 12 Nov 2008 00:36:35 +0100 Subject: [Linux-cluster] Fence VirtualIron - i have the script but... In-Reply-To: <1226442380.7132.65.camel@ayanami> References: <1226442380.7132.65.camel@ayanami> Message-ID: 2008/11/11 Lon Hohberger : > http://sources.redhat.com/cluster/wiki/FenceAgentAPI > > If your agent follows those guidelines, you can just stick the agent > directives and configuration in cluster.conf. > This is enought! thanks you, Lon! -- mr From jbrassow at redhat.com Wed Nov 12 02:00:28 2008 From: jbrassow at redhat.com (Jonathan Brassow) Date: Tue, 11 Nov 2008 20:00:28 -0600 Subject: [Linux-cluster] Distributed RAID In-Reply-To: <1226060556.12833.4.camel@marc> References: <491275EE.8050508@auckland.ac.nz> <4912BAFA.2080901@bobich.net> <49135647.3020701@bobich.net> <940E48BD-8C04-4525-86AD-B4ED1998883E@redhat.com> <1226060556.12833.4.camel@marc> Message-ID: cmirror will progress on the RHEL5 branch, but for upstream placement, it will probably move to the LVM repository. That will be happening soon. brassow On Nov 7, 2008, at 6:22 AM, Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote: > Hello, > > > will the changes to mirroring get merged into stable2 and head after > RHEL-5.3 release? > > > Marc > > Am Donnerstag, den 06.11.2008, 16:34 -0600 schrieb Jonathan Brassow: >> that works already. >> >> single machine: linear, stripe, mirror, snapshot >> cluster-aware: linear, stripe, mirror (5.3) >> >> brassow >> >> On Nov 6, 2008, at 2:40 PM, Gordan Bobic wrote: >> >>> What about CLVM based striping (RAID0)? Does that work already or is >>> it planned for the near future? >>> >>> Gordan >>> >>> Jonathan Brassow wrote: >>>> Cluster mirror (RAID1) will be available in rhel5.3 for LVM. >>>> brassow >>>> On Nov 6, 2008, at 9:40 AM, Federico Simoncelli wrote: >>>>> On Thu, Nov 6, 2008 at 10:38 AM, Gordan Bobic >>>>> wrote: >>>>>> I rather doubt md will become cluster aware any time soon. CLVM >>>>>> doesn't yet >>>>>> support even more important features like snapshotting, so I >>>>>> wouldn't count >>>>>> on it supporting anything more advanced. >>>>> >>>>> I worked a little on clvm snapshots: >>>>> https://www.redhat.com/archives/linux-lvm/2008-October/msg00027.html >>>>> >>>>> Review and testing is required. >>>>> -- >>>>> Federico. >>>>> >>>>> -- >>>>> Linux-cluster mailing list >>>>> Linux-cluster at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>> -- >>>> Linux-cluster mailing list >>>> Linux-cluster at redhat.com >>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From jbrassow at redhat.com Wed Nov 12 02:01:44 2008 From: jbrassow at redhat.com (Jonathan Brassow) Date: Tue, 11 Nov 2008 20:01:44 -0600 Subject: [Linux-cluster] Distributed RAID In-Reply-To: <639ce0480811070459l18a0071bia889bb56402f6ab9@mail.gmail.com> References: <491275EE.8050508@auckland.ac.nz> <4912BAFA.2080901@bobich.net> <49135647.3020701@bobich.net> <940E48BD-8C04-4525-86AD-B4ED1998883E@redhat.com> <1226060556.12833.4.camel@marc> <639ce0480811070459l18a0071bia889bb56402f6ab9@mail.gmail.com> Message-ID: <322B81E0-5500-475F-B00B-4FE81C9D15B7@redhat.com> Sure. In fact, if you have access to the red hat 5.3 beta, it is ready there. brassow On Nov 7, 2008, at 6:59 AM, pronix pronix wrote: > can i use cluster raid1 if i get development release from > sources.redhat.com/cluster ? > > > 2008/11/7 Marc - A. Dahlhaus [ Administration | Westermann GmbH ] > > Hello, > > > will the changes to mirroring get merged into stable2 and head after > RHEL-5.3 release? > > > Marc > > Am Donnerstag, den 06.11.2008, 16:34 -0600 schrieb Jonathan Brassow: > > that works already. > > > > single machine: linear, stripe, mirror, snapshot > > cluster-aware: linear, stripe, mirror (5.3) > > > > brassow > > > > On Nov 6, 2008, at 2:40 PM, Gordan Bobic wrote: > > > > > What about CLVM based striping (RAID0)? Does that work already > or is > > > it planned for the near future? > > > > > > Gordan > > > > > > Jonathan Brassow wrote: > > >> Cluster mirror (RAID1) will be available in rhel5.3 for LVM. > > >> brassow > > >> On Nov 6, 2008, at 9:40 AM, Federico Simoncelli wrote: > > >>> On Thu, Nov 6, 2008 at 10:38 AM, Gordan Bobic > > > >>> wrote: > > >>>> I rather doubt md will become cluster aware any time soon. CLVM > > >>>> doesn't yet > > >>>> support even more important features like snapshotting, so I > > >>>> wouldn't count > > >>>> on it supporting anything more advanced. > > >>> > > >>> I worked a little on clvm snapshots: > > >>> https://www.redhat.com/archives/linux-lvm/2008-October/msg00027.html > > >>> > > >>> Review and testing is required. > > >>> -- > > >>> Federico. > > >>> > > >>> -- > > >>> Linux-cluster mailing list > > >>> Linux-cluster at redhat.com > > >>> https://www.redhat.com/mailman/listinfo/linux-cluster > > >> -- > > >> Linux-cluster mailing list > > >> Linux-cluster at redhat.com > > >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From achievement.hk at gmail.com Wed Nov 12 09:44:59 2008 From: achievement.hk at gmail.com (Achievement Chan) Date: Wed, 12 Nov 2008 17:44:59 +0800 Subject: [Linux-cluster] GFS performance of imap service (Maildir) Message-ID: Dear All, I've setup a courier-imap server which store the email data in Maildir format. The mailbox are saved under a LUN in ISCSI SAN. For handling a mailbox with 10000 email, it takes 6-8 seconds for waiting response from first "SELECT" command. The response time is also unstable too, sometimes it takes 10-20 seconds for the same mailbox. Based some online material, i've tried to tune the gfs. But there are seems no improvement. e.g. gfs_tool setflag inherit_jdata /home/domains gfs_tool settune /home/domains recoverd_secs 60 gfs_tool settune /home/domains glock_purge 50 gfs_tool settune /home/domains demote_secs 100 gfs_tool settune /home/domains scand_secs 3 gfs_tool settune /home/domains max_readahead 262144 gfs_tool settune /home/domains statfs_fast 1 I've tested the mailbox under ext3 and gfs2, both under the LUN in the same SAN. The response time can be within 1 second. Has anyone tried to provide imap service in GFS? or I need to go for GFS2? Is GFS2 still unstable for production system? Regards, Achievement Chan From kiss.zoltan at bardiauto.hu Wed Nov 12 10:57:04 2008 From: kiss.zoltan at bardiauto.hu (=?iso-8859-2?B?S2lzcyBab2x04W4=?=) Date: Wed, 12 Nov 2008 11:57:04 +0100 Subject: [Linux-cluster] slow lock performance Message-ID: <91632af9d61397469c647a809455d908@mail.bardiauto.hu> Hello, I have a little problem with my GFS2 installation. My Company have a little bit old business software (written in clipper), but our programmers can compile itt o linux platform with the xHarbour compiler. The sw uses DBF databases to store data. Here is the problem: We have 100-150 client computers, that mean we must run 100-150 (minimum) application on our servers. The problem is the GFS or GFS2 locking. If i mount the storage GFS/GFS2 partition with the localflocks option, then the application is running very fast (slower like ext3, but fast). Without the localflocks option the applications is slow down. If i enable cluster wide flocks, then my apps running approx. 100-140X slower. Can anybody help me? Any useful mount option, gfs tune option, etc.etc.. Thank you in anticipation! Best regards, Zoltan Kiss system administrator B?rdi Aut? Zrt. zoltan.kiss at bardiauto.hu +36204300386 -------------- next part -------------- An HTML attachment was scrubbed... URL: From veliogluh at itu.edu.tr Wed Nov 12 11:17:00 2008 From: veliogluh at itu.edu.tr (Hakan VELIOGLU) Date: Wed, 12 Nov 2008 13:17:00 +0200 Subject: [Linux-cluster] Cman doesn't realize the failed node In-Reply-To: <91632af9d61397469c647a809455d908@mail.bardiauto.hu> References: <91632af9d61397469c647a809455d908@mail.bardiauto.hu> Message-ID: <20081112131700.16801d3do17enme8@webmail.beta.itu.edu.tr> Hi, I am testing and trying to understand the cluster environment. I ve built a two node cluster system without any service (Red Hat EL 5.2 x64). I run the cman and rgmanager services succesfully and then poweroff one node suddenly. After thsi I excpect that the other node realize this failure and take up all the resources however running node doesn't realize this failure. I use "cman_tool nodes" and "clustat" commands and they say the failed node is active and online. What am i missing? Why cman doesn't realize the failure? [root at cl1 ~]# cat /etc/cluster/cluster.conf [root at cl1 ~]# When the node gows down, the TOTEM repeastedly logs messages like this. Nov 12 13:12:57 cl1 openais[5809]: [TOTEM] The consensus timeout expired. Nov 12 13:12:57 cl1 openais[5809]: [TOTEM] entering GATHER state from 3. Nov 12 13:13:03 cl1 openais[5809]: [TOTEM] The consensus timeout expired. Nov 12 13:13:03 cl1 openais[5809]: [TOTEM] entering GATHER state from 3. Nov 12 13:13:09 cl1 openais[5809]: [TOTEM] The consensus timeout expired. Nov 12 13:13:09 cl1 openais[5809]: [TOTEM] entering GATHER state from 3. Nov 12 13:13:14 cl1 openais[5809]: [TOTEM] The consensus timeout expired. Nov 12 13:13:14 cl1 openais[5809]: [TOTEM] entering GATHER state from 3. Hakan From hicheerup at gmail.com Wed Nov 12 13:44:08 2008 From: hicheerup at gmail.com (lingu) Date: Wed, 12 Nov 2008 19:14:08 +0530 Subject: [Linux-cluster] RHEL3 Cluster Broken Pipe error and Heartbeat configuration Message-ID: <29e045b80811120544j1a85eeay237b72daf8de3e16@mail.gmail.com> Hi, I am running two node active/passive cluster on RHEL3U8-64 bit operating system for my oracle database,both the nodes are connected to HP MSA-500 storage(scsi not Fibre channel) . Below are my hardware and clumanager version details. It was running fine and stable for last two years but all of a sudden for the past one month i am getting below errors on syslog and cluster restarting locally. Server Hardware: HP ProLiant DL580 G4 OS: RHEL3U8-64BIT INTEL EMT Kernel : 2.4.21-47.EL Storage : HP MSA-500 storage (scsci channel) Cluster Version: clumanager-1.2.26.1-1 redhat-config-cluster-1.0.7-1 NODE1 ip: 20.2.135.161 (network bonding configured) NODE2 ip: 20.2.135.162 (network bonding configured) VIP : 20.2.135.35 Syslog errors cluquorumd[1921]: Disk-TB: Detected I/O Hang! clulockd[1996]: Potential recursive lock #0 grant to member #1, PID1962 clulockd[1996]: Denied 20.1.135.162: Broken pipe clulockd[1996]: select error: Broken pipe clulockd[1996]: Denied 20.1.135.162: Broken pipe clulockd[1996]: select error: Broken pipe cluquorumd[1921]: Disk-TB: Detected I/O Hang! clulockd[1996]: Denied 20.1.135.161: Broken pipe clulockd[1996]: select error: Broken pipe clusvcmgrd[2011]: Unable to obtain cluster lock: Connection timed out cluquorumd[2100]: VF: Abort: Invalid header in reply from member #0 cluquorumd[1934]: __msg_send: Incomplete write to 13. Error: Connection reset by peer Can any one guide me what is this above error indicates and how to troubleshoot.After a long google search i found the below link from redhat that is matching my scenario.Can i follow the same because it is my very critical production server. https://bugzilla.redhat.com/show_bug.cgi?id=185484 Also anyone help me to configure a dedicated LAN (for example eth3) as heartbeat(private point to point cross over cable network for cluster communications),I don't wish heartbeat over public LAN , because of heavy Network saturation. Fot the above heartbeat configuration i didnot found any suitable document for rhel. Can any one provide me the suitable link or guide me what are all the changes i have to made in my existing cluster.xml file for this private heartbeat configuration to work. Waiting for some one reply its urgent for me Regards, Lingu From treed at ultraviolet.org Wed Nov 12 18:43:52 2008 From: treed at ultraviolet.org (Tracy Reed) Date: Wed, 12 Nov 2008 10:43:52 -0800 Subject: [Linux-cluster] HA cluster cache coherency Message-ID: <20081112184351.GC21564@tracyreed.org> Hello all, I have been wondering what most people do about cache coherency issues when doing high availability failover between two or more Linux servers (RHEL, if it matters) with shared storage? Consider a typical master-slave arrangement managed by heartbeat and a fibrechannel HBA in each node each connected to a switch with a bunch of storage also connected to the switch. The master and the slave are sharing this disk to other clients on the network via NFS. When the master fails during heavy writes with a gig of data in its cache all of that data will be lost. What do most people do about this? Is there any way to tell the kernel to only do write-through and no caching? This might not be infeasible if one has a lot of cache in the disk storage connected to the fibrechannel switch which is the case for me. I have read http://www.westnet.com/~gsmith/content/linux-pdflush.htm which seems to be an excellent treatment of pdflush related issues. However, it does not seem to address this specific issue. It mentions four tunables in /proc/sys/vm which when set to zero seem like they might accomplish what I'm looking for: dirty_background_ratio dirty_ratio dirty_expire_centisecs dirty_writeback_centisecs but I set them all to zero on a test system and the Dirty field of /proc/meminfo still routinely shows dirty pages. Your comments are appreciated. -- Tracy Reed http://tracyreed.org From kadlec at sunserv.kfki.hu Wed Nov 12 18:51:40 2008 From: kadlec at sunserv.kfki.hu (Kadlecsik Jozsef) Date: Wed, 12 Nov 2008 19:51:40 +0100 (CET) Subject: [Linux-cluster] GFS performance of imap service (Maildir) In-Reply-To: References: Message-ID: Hello, On Wed, 12 Nov 2008, Achievement Chan wrote: > For handling a mailbox with 10000 email, it takes 6-8 seconds for > waiting response from first "SELECT" command. > The response time is also unstable too, sometimes it takes 10-20 > seconds for the same mailbox. > > Based some online material, i've tried to tune the gfs. But there are > seems no improvement. > e.g. > gfs_tool setflag inherit_jdata /home/domains > gfs_tool settune /home/domains recoverd_secs 60 > gfs_tool settune /home/domains glock_purge 50 > gfs_tool settune /home/domains demote_secs 100 > gfs_tool settune /home/domains scand_secs 3 > gfs_tool settune /home/domains max_readahead 262144 > gfs_tool settune /home/domains statfs_fast 1 > > Has anyone tried to provide imap service in GFS? We have had exactly the same problems with maildir over GFS. There was no tuning whatsoever which helped: the fighting for the locks for every file in the maildir costs so much that you cannot expect better performance. The best is to avoid maildir and use simple mailbox format instead. We went (back) to mailbox and since then our users have not complained about performance. Best regards, Jozsef -- E-mail : kadlec at mail.kfki.hu, kadlec at blackhole.kfki.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt Address: KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary From fdinitto at redhat.com Thu Nov 13 05:51:29 2008 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Thu, 13 Nov 2008 06:51:29 +0100 Subject: [Linux-cluster] Re: logging: more input In-Reply-To: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> References: <1226344150.2445.61.camel@daitarn-fedora.int.fabbione.net> Message-ID: <1226555489.4022.15.camel@daitarn-fedora.int.fabbione.net> Hi guys, I have received a bunch of comments and suggestions in my personal inbox from people that didn't want to participate in the thread directly. Some of them are absolutely valid points IMHO. So here they are: On Mon, 2008-11-10 at 20:09 +0100, Fabio M. Di Nitto wrote: > == Output == > > to_file: > > echo $(date "+%b %d %T") subsystem_or_daemon: entry_to_log > Nov 10 19:46:40 subsystem_or_daemon: entry_to_log Quoting: -- I've not read any of the threads but did you consider a date format that sorts easily i.e. YYYY MM DD-based? And for investigating races across files/nodes using sub-second timestamps (e.g. 19:46:40.123 ) and adding the node name. So you can just extract relevant sections from several files, cat them all together, sort them and then review the sequence of happenings. If you're generalising, make the log format string a customisable option similar to apache. -- I agree that a more precise time and sortable format is a major winner. He also has a good point to add the node name to the log. My reply to the "customisable format" request was based on the discussion we had at the Summit (that we want to avoid parsing lots of different logging etc.) and his reply was: Quoting: -- 1. Most people will use the default or one of the example lines you supply. 2. If it is a problem, you've got the corresponding config line so it should be straightforward to have a utility to convert it back to your preferred canonical output format. (And yes, "Nov 14" format can also be parsed and converted back into a sortable format without much difficulty, so it's no big deal.) 3. If lots of people change the log format, then you probably chose a poor default. 4. There's no reason why log-to-file should exactly match syslog format. If you want syslog format, use syslog. You want the best format to assist you in debugging problems etc. -- Fabio From robejrm at gmail.com Thu Nov 13 09:56:28 2008 From: robejrm at gmail.com (Juan Ramon Martin Blanco) Date: Thu, 13 Nov 2008 10:56:28 +0100 Subject: [Linux-cluster] Limit service restarting times Message-ID: <8a5668960811130156i2ef6f533s1f39332b5dd72195@mail.gmail.com> First of all, hello and many thanks everyone, this list has helped me a lot in the cluster world ;) I have configured a 2 node cluster with RHEL 5.2, shared storage and GFS2. I have configured several services with our company own software. This software evolves fast because we are in active development, so sometimes cores are dumped. When this happens, the cluster tries to restart the failing service again and again...filling the service's filesystem with cores. Is there any way to limit the number of retries for a certain service? Thanks in advance, Juan Ram?n Mart?n Blanco -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcasiraghi73 at gmail.com Thu Nov 13 14:14:11 2008 From: mcasiraghi73 at gmail.com (Mauro Casiraghi) Date: Thu, 13 Nov 2008 15:14:11 +0100 Subject: [Linux-cluster] RedHat Cluster Suite cluster resource management Message-ID: <18c35c650811130614i2a58e035w3bb9f574075389ca@mail.gmail.com> I had created this cluster configuration with Redhat Cluster Suite I Have one Service Group with the follow resources Service Group Name : WEB Resources of the service group: 1) IP_ADRESS 2) APACHE The resource dependency are: The Apache resource is dependent of ip_adress How can i stop only apache resource wiyhout stopping all service group ??? What is the cluster.conf to do this ?? Best Regards Mauro Casiraghi -------------- next part -------------- An HTML attachment was scrubbed... URL: From Harri.Paivaniemi at tietoenator.com Thu Nov 13 14:22:05 2008 From: Harri.Paivaniemi at tietoenator.com (Harri.Paivaniemi at tietoenator.com) Date: Thu, 13 Nov 2008 16:22:05 +0200 Subject: [Linux-cluster] RedHat Cluster Suite cluster resource management References: <18c35c650811130614i2a58e035w3bb9f574075389ca@mail.gmail.com> Message-ID: <41E8D4F07FCE154CBEBAA60FFC92F67709FE46@apollo.eu.tieto.com> Add "exit 0" to the beginning of your apache status-check portion ;) -hjp -----Original Message----- From: linux-cluster-bounces at redhat.com on behalf of Mauro Casiraghi Sent: Thu 11/13/2008 16:14 To: linux-cluster at redhat.com Subject: [Linux-cluster] RedHat Cluster Suite cluster resource management I had created this cluster configuration with Redhat Cluster Suite I Have one Service Group with the follow resources Service Group Name : WEB Resources of the service group: 1) IP_ADRESS 2) APACHE The resource dependency are: The Apache resource is dependent of ip_adress How can i stop only apache resource wiyhout stopping all service group ??? What is the cluster.conf to do this ?? Best Regards Mauro Casiraghi -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3009 bytes Desc: not available URL: From mcasiraghi73 at gmail.com Thu Nov 13 14:28:17 2008 From: mcasiraghi73 at gmail.com (Mauro Casiraghi) Date: Thu, 13 Nov 2008 15:28:17 +0100 Subject: [Linux-cluster] Service Group Dependency Message-ID: <18c35c650811130628r613fddbfi42bf4d0b2a83e1fe@mail.gmail.com> If i have two or more services how can set dependency for the services using Redhat Cluster Suite?? For example if i have service A B end C how can set those dependency: B depend from A and C depend from B In this case to stop service group A i must stop before C and B What is the cluster.conf setting ?? Best Regards Mauro Casiraghi -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcasiraghi73 at gmail.com Thu Nov 13 14:34:34 2008 From: mcasiraghi73 at gmail.com (Mauro Casiraghi) Date: Thu, 13 Nov 2008 15:34:34 +0100 Subject: [Linux-cluster] RedHat Cluster Suite cluster resource management In-Reply-To: <41E8D4F07FCE154CBEBAA60FFC92F67709FE46@apollo.eu.tieto.com> References: <18c35c650811130614i2a58e035w3bb9f574075389ca@mail.gmail.com> <41E8D4F07FCE154CBEBAA60FFC92F67709FE46@apollo.eu.tieto.com> Message-ID: <18c35c650811130634y3539ac24waf6649dde280db4d@mail.gmail.com> Ok this it works but is a workaround. Normaly in a cluster (veritas or Sun Cluster) you can enable,disable stop and start a resource in a service group without change monitoring script. Is it possible do the same in RHCS ??? Thank you for your help Mauro Casiraghi 2008/11/13 > > Add "exit 0" to the beginning of your apache status-check portion ;) > > -hjp > > > > -----Original Message----- > From: linux-cluster-bounces at redhat.com on behalf of Mauro Casiraghi > Sent: Thu 11/13/2008 16:14 > To: linux-cluster at redhat.com > Subject: [Linux-cluster] RedHat Cluster Suite cluster resource management > > I had created this cluster configuration with Redhat Cluster Suite > > I Have one Service Group with the follow resources > > Service Group Name : WEB > > Resources of the service group: > > 1) IP_ADRESS > 2) APACHE > > The resource dependency are: > > The Apache resource is dependent of ip_adress > > How can i stop only apache resource wiyhout stopping all service group ??? > What is the cluster.conf to do this ?? > > Best Regards > > Mauro Casiraghi > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcasiraghi73 at gmail.com Thu Nov 13 14:51:00 2008 From: mcasiraghi73 at gmail.com (Mauro Casiraghi) Date: Thu, 13 Nov 2008 15:51:00 +0100 Subject: [Linux-cluster] Manual Fencing problem Message-ID: <18c35c650811130651i2931bd0h211a37192390f9e4@mail.gmail.com> I have two cluster nodes with the follow configuration For each node i had setup manual fencing On my messages (node0) i had recived this message Nov 13 12:06:34 lxxxxxxx fenced[2002]: fencing node "node1" Nov 13 12:06:34 lxxxxxxx fenced[2002]: agent "fence_manual" reports: failed: fence_manual no node name How can i fix this problem Best Regards Mauro Casiraghi -------------- next part -------------- An HTML attachment was scrubbed... URL: From jruemker at redhat.com Thu Nov 13 14:58:08 2008 From: jruemker at redhat.com (John Ruemker) Date: Thu, 13 Nov 2008 09:58:08 -0500 Subject: [Linux-cluster] Manual Fencing problem In-Reply-To: <18c35c650811130651i2931bd0h211a37192390f9e4@mail.gmail.com> References: <18c35c650811130651i2931bd0h211a37192390f9e4@mail.gmail.com> Message-ID: <491C4080.3050400@redhat.com> Try adding the nodename attribute to each device as seen here: Mauro Casiraghi wrote: > I have two cluster nodes with the follow configuration > > For each node i had setup manual fencing > > > > post_join_delay="3"/> > > > > > > > > > > > > > > > > > > > > > > restricted="1"> > priority="1"/> > priority="1"/> > > > > > > exclusive="0" name="rhcs-web" recovery="relocate"> > > > > > > On my messages (node0) i had recived this message > > Nov 13 12:06:34 lxxxxxxx fenced[2002]: fencing node "node1" > Nov 13 12:06:34 lxxxxxxx fenced[2002]: agent "fence_manual" reports: > failed: fence_manual no node name > > How can i fix this problem > -John From mcasiraghi73 at gmail.com Thu Nov 13 15:05:53 2008 From: mcasiraghi73 at gmail.com (Mauro Casiraghi) Date: Thu, 13 Nov 2008 16:05:53 +0100 Subject: [Linux-cluster] Resource Restart Message-ID: <18c35c650811130705r6c6628fbmbbef0e9e07be9f64@mail.gmail.com> If i have one resource in a service cluster, how can set the max number of time it can be restarted from the cluster before is considered fault ??? And in this case is it possible stop the service group relocation on other nodes ?? Some time if the restart problem of the resource is an application problem, the best choice is to live the resource faulted on the node and investigate about the problem. In Veritas Cluster this is very easy to implement. Is it possible implement the same configuration using RedHat Cluster suite ??? Best Regards Mauro Casiraghi -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcasiraghi73 at gmail.com Thu Nov 13 15:25:03 2008 From: mcasiraghi73 at gmail.com (Mauro Casiraghi) Date: Thu, 13 Nov 2008 16:25:03 +0100 Subject: [Linux-cluster] Manual Fencing problem In-Reply-To: <491C4080.3050400@redhat.com> References: <18c35c650811130651i2931bd0h211a37192390f9e4@mail.gmail.com> <491C4080.3050400@redhat.com> Message-ID: <18c35c650811130725q7384fb57o414a3743bd9b4d73@mail.gmail.com> Ok i think that it works but now i have another problem On the node0 messages i can see the follow message fence_manual: Node node1 needs to be reset before recovery can procede. Waiting for node1 to rejoin the cluster or for manual acknowledgement that it has been reset (i.e. fence_ack_manual -n node1) so i try to fence_ack_manual -n node1 and i recived this message fence_ack_manual -n node1 Warning: If the node "node1" has not been manually fenced (i.e. power cycled or disconnected from shared storage devices) the GFS file system may become corrupted and all its data unrecoverable! Please verify that the node shown above has been reset or disconnected from storage. Are you certain you want to continue? [yN] y can't open /tmp/fence_manual.fifo: No such file or directory Thank you for your help Best Regards Mauro Casiraghi On Thu, Nov 13, 2008 at 3:58 PM, John Ruemker wrote: > Try adding the nodename attribute to each device as seen here: > > Mauro Casiraghi wrote: > >> I have two cluster nodes with the follow configuration >> For each node i had setup manual fencing >> >> >> > post_join_delay="3"/> >> >> >> >> >> > nodename="node0"/> > >> >> >> >> >> >> >> > nodename="node1"/> > > >> >> >> >> >> >> >> >> >> >> >> > restricted="1"> >> > priority="1"/> >> > priority="1"/> >> >> >> >> >> >> > name="rhcs-web" recovery="relocate"> >> >> >> >> >> On my messages (node0) i had recived this message >> Nov 13 12:06:34 lxxxxxxx fenced[2002]: fencing node "node1" >> Nov 13 12:06:34 lxxxxxxx fenced[2002]: agent "fence_manual" reports: >> failed: fence_manual no node name >> How can i fix this problem >> >> > > -John > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.osullivan at auckland.ac.nz Thu Nov 13 16:21:22 2008 From: michael.osullivan at auckland.ac.nz (Michael O'Sullivan) Date: Thu, 13 Nov 2008 09:21:22 -0700 Subject: [Linux-cluster] Clusters with multihomed hosts Message-ID: <491C5402.8060706@auckland.ac.nz> Hi all, I need to know more about using redundant NICs in clusters. I have a 2-node cluster with 2 NICs in each node. The first NICs on each node are connected to one switch, the second NICs on each node are connected to another switch. This is an experimental arrangement so I am using /etc/hosts instead of DNS. It appears that the cluster software becomes confused if I put both NICs for the hosts in the /etc/hosts file, even if they are in different subnets. Here is the /etc/hosts file I would like to use: # localhost line 192.168.10.1 node1 192.168.10.2 node2 192.168.20.1 node1 # Second NIC on node 1 192.168.20.2 node2 # Second NIC on node 2 but this seems to cause the cluster to hang (confused about which NIC to use?), so I have removed the last 2 lines and everything works fine. However, this means if the switch on the 192.168.10.x subnet fails the cluster will "break". If the cluster would recognise that node1 and node2 are available via the second NICs then I wouldn't have to worry about this single point-of-failure. I have thought about bonding the NICs which (I think) would take care of the problem, but I have heard that boding two NICs usually does not give double the bandwidth. I have read a little about high-availability and failing over IP addresses, but this seems to be between different nodes, not different NICs in the same host. Would anyone please be able to give me some direction about the best way to set up my cluster and NICs to make the cluster reliable in the event of switch failure? And keep the full bandwidth of the NICs intact? Thanks in advance for any help you can give. Kind regards, Mike From billpp at gmail.com Thu Nov 13 16:25:52 2008 From: billpp at gmail.com (Flavio Junior) Date: Thu, 13 Nov 2008 14:25:52 -0200 Subject: [Linux-cluster] Clusters with multihomed hosts In-Reply-To: <491C5402.8060706@auckland.ac.nz> References: <491C5402.8060706@auckland.ac.nz> Message-ID: <58aa8d780811130825w64c4476crb64c2704d4e0b2fd@mail.gmail.com> You can use bonding NICs with active-backup mode. Only one NIC is used at a time, the second will only come up if the primary (active) fails. Dont forget to configure miimon value for link monitor. -- Fl?vio do Carmo J?nior aka waKKu On Thu, Nov 13, 2008 at 2:21 PM, Michael O'Sullivan < michael.osullivan at auckland.ac.nz> wrote: > Hi all, > > I need to know more about using redundant NICs in clusters. > > I have a 2-node cluster with 2 NICs in each node. The first NICs on each > node are connected to one switch, the second NICs on each node are connected > to another switch. This is an experimental arrangement so I am using > /etc/hosts instead of DNS. It appears that the cluster software becomes > confused if I put both NICs for the hosts in the /etc/hosts file, even if > they are in different subnets. Here is the /etc/hosts file I would like to > use: > > # localhost line > 192.168.10.1 node1 > 192.168.10.2 node2 > 192.168.20.1 node1 # Second NIC on node 1 > 192.168.20.2 node2 # Second NIC on node 2 > > but this seems to cause the cluster to hang (confused about which NIC to > use?), so I have removed the last 2 lines and everything works fine. > However, this means if the switch on the 192.168.10.x subnet fails the > cluster will "break". If the cluster would recognise that node1 and node2 > are available via the second NICs then I wouldn't have to worry about this > single point-of-failure. > > I have thought about bonding the NICs which (I think) would take care of > the problem, but I have heard that boding two NICs usually does not give > double the bandwidth. I have read a little about high-availability and > failing over IP addresses, but this seems to be between different nodes, not > different NICs in the same host. > > Would anyone please be able to give me some direction about the best way to > set up my cluster and NICs to make the cluster reliable in the event of > switch failure? And keep the full bandwidth of the NICs intact? > > Thanks in advance for any help you can give. Kind regards, Mike > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raju.rajsand at gmail.com Thu Nov 13 16:34:52 2008 From: raju.rajsand at gmail.com (Rajagopal Swaminathan) Date: Thu, 13 Nov 2008 22:04:52 +0530 Subject: [Linux-cluster] Manual Fencing problem In-Reply-To: <18c35c650811130725q7384fb57o414a3743bd9b4d73@mail.gmail.com> References: <18c35c650811130651i2931bd0h211a37192390f9e4@mail.gmail.com> <491C4080.3050400@redhat.com> <18c35c650811130725q7384fb57o414a3743bd9b4d73@mail.gmail.com> Message-ID: <8786b91c0811130834h4f9cd5f2hce49028b4df42c97@mail.gmail.com> Greetings, 2008/11/13 Mauro Casiraghi : > Ok i think that it works but now i have another problem > > On the node0 messages i can see the follow message > Apologies to remind of a basic fact of life in network. Do you have both node's name in the /etc/hosts file or DNS server oin your environment Regards, Rajagopal From Harri.Paivaniemi at tietoenator.com Thu Nov 13 16:34:31 2008 From: Harri.Paivaniemi at tietoenator.com (Harri.Paivaniemi at tietoenator.com) Date: Thu, 13 Nov 2008 18:34:31 +0200 Subject: [Linux-cluster] Clusters with multihomed hosts References: <491C5402.8060706@auckland.ac.nz> <58aa8d780811130825w64c4476crb64c2704d4e0b2fd@mail.gmail.com> Message-ID: <41E8D4F07FCE154CBEBAA60FFC92F67709FE49@apollo.eu.tieto.com> ... and you could also use mode 0 in bonding to get roundrobin-lb. -hpj -----Original Message----- From: linux-cluster-bounces at redhat.com on behalf of Flavio Junior Sent: Thu 11/13/2008 18:25 To: linux clustering Subject: Re: [Linux-cluster] Clusters with multihomed hosts You can use bonding NICs with active-backup mode. Only one NIC is used at a time, the second will only come up if the primary (active) fails. Dont forget to configure miimon value for link monitor. -- Fl?vio do Carmo J?nior aka waKKu On Thu, Nov 13, 2008 at 2:21 PM, Michael O'Sullivan < michael.osullivan at auckland.ac.nz> wrote: > Hi all, > > I need to know more about using redundant NICs in clusters. > > I have a 2-node cluster with 2 NICs in each node. The first NICs on each > node are connected to one switch, the second NICs on each node are connected > to another switch. This is an experimental arrangement so I am using > /etc/hosts instead of DNS. It appears that the cluster software becomes > confused if I put both NICs for the hosts in the /etc/hosts file, even if > they are in different subnets. Here is the /etc/hosts file I would like to > use: > > # localhost line > 192.168.10.1 node1 > 192.168.10.2 node2 > 192.168.20.1 node1 # Second NIC on node 1 > 192.168.20.2 node2 # Second NIC on node 2 > > but this seems to cause the cluster to hang (confused about which NIC to > use?), so I have removed the last 2 lines and everything works fine. > However, this means if the switch on the 192.168.10.x subnet fails the > cluster will "break". If the cluster would recognise that node1 and node2 > are available via the second NICs then I wouldn't have to worry about this > single point-of-failure. > > I have thought about bonding the NICs which (I think) would take care of > the problem, but I have heard that boding two NICs usually does not give > double the bandwidth. I have read a little about high-availability and > failing over IP addresses, but this seems to be between different nodes, not > different NICs in the same host. > > Would anyone please be able to give me some direction about the best way to > set up my cluster and NICs to make the cluster reliable in the event of > switch failure? And keep the full bandwidth of the NICs intact? > > Thanks in advance for any help you can give. Kind regards, Mike > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 4011 bytes Desc: not available URL: From raju.rajsand at gmail.com Thu Nov 13 16:38:04 2008 From: raju.rajsand at gmail.com (Rajagopal Swaminathan) Date: Thu, 13 Nov 2008 22:08:04 +0530 Subject: [Linux-cluster] Clusters with multihomed hosts In-Reply-To: <491C5402.8060706@auckland.ac.nz> References: <491C5402.8060706@auckland.ac.nz> Message-ID: <8786b91c0811130838gd857579s962212dfc378efb7@mail.gmail.com> Greetings, On Thu, Nov 13, 2008 at 9:51 PM, Michael O'Sullivan wrote: > Hi all, > > I need to know more about using redundant NICs in clusters. 1. Have you enabled the multicast feature in the switch carrying the Cluster heartbeat traffic? 2. HAve you tried 802.3ad in the bonding mode? Regards Rajagopal From swhiteho at redhat.com Thu Nov 13 16:54:24 2008 From: swhiteho at redhat.com (Steven Whitehouse) Date: Thu, 13 Nov 2008 16:54:24 +0000 Subject: [Linux-cluster] Clusters with multihomed hosts In-Reply-To: <491C5402.8060706@auckland.ac.nz> References: <491C5402.8060706@auckland.ac.nz> Message-ID: <1226595264.9571.75.camel@quoit> Hi, On Thu, 2008-11-13 at 09:21 -0700, Michael O'Sullivan wrote: > Hi all, > > I need to know more about using redundant NICs in clusters. > > I have a 2-node cluster with 2 NICs in each node. The first NICs on each > node are connected to one switch, the second NICs on each node are > connected to another switch. This is an experimental arrangement so I am > using /etc/hosts instead of DNS. It appears that the cluster software > becomes confused if I put both NICs for the hosts in the /etc/hosts > file, even if they are in different subnets. Here is the /etc/hosts file > I would like to use: > > # localhost line > 192.168.10.1 node1 > 192.168.10.2 node2 > 192.168.20.1 node1 # Second NIC on node 1 > 192.168.20.2 node2 # Second NIC on node 2 > > but this seems to cause the cluster to hang (confused about which NIC to > use?), so I have removed the last 2 lines and everything works fine. > However, this means if the switch on the 192.168.10.x subnet fails the > cluster will "break". If the cluster would recognise that node1 and > node2 are available via the second NICs then I wouldn't have to worry > about this single point-of-failure. > The trouble with this kind of thing is that you find that its not easy to control which external IP address a particular application uses as you have discovered. It can be done though, with the aid of iproute2. The kernel will look at the routing table to work out where to send a particular packet, and once its found a suitable destination interface it will then look at the various possible source IPs on that interface in order to work out which one to use. It tries to use the source address which has most bits matching with the destination (counting from the network end to the host end of the IP address) so that if the destination address is on a particular subnet, it will try to use an IP from the same subnet as the source address if one is available. You can alter this quite easily though, you just set up a second routing table and use routing rules in order to select the correct table according to your network. Thats where iproute2 comes in and there is a set of docs here: http://lartc.org/ Also, just because you have two NIC's connected to different switches doesn't mean that you need to give them different IP addresses/subnets. The Linux IP stack can easily cope with them being the same, which would also simplify the situation that you have, where, I suspect the cluster stack has replied via a different NIC and thus received a different source address. > I have thought about bonding the NICs which (I think) would take care of > the problem, but I have heard that boding two NICs usually does not > give double the bandwidth. I have read a little about high-availability > and failing over IP addresses, but this seems to be between different > nodes, not different NICs in the same host. > > Would anyone please be able to give me some direction about the best way > to set up my cluster and NICs to make the cluster reliable in the event > of switch failure? And keep the full bandwidth of the NICs intact? > > Thanks in advance for any help you can give. Kind regards, Mike > The problem with bonding is that a single packet can only use a single one of the parallel links. Also, by using multiple links on a single stream (i.e. a TCP connection) you run the risk, if you are not careful, of reordering the packets and that can cause slow downs at the receiving end, and possibly generation of out of order ACK packets which might cause retransmissions at the sending end, further slowing things down. The Linux bonding driver has various modes to try and avoid that, and in addition it also has 802.3ad mode which allows it to automatically negotiate settings with a switch. Thats ideal if all the bonded links for a particular node go to the same switch, but won't work across switches as in your situation. I suspect that the choice will come down to one of the following: 1. something easy to set up & not very efficient in terms of bandwidth, but probably not too bad either. -> choose bonding (just be sure to select the right mode) 2. something more complex to set up, but which can be made to make full use all of the available bandwidth, and be extended into more complicated setups (dynamic routing, etc), given enough application support & tweeking. -> choose the IP based solution Steve. From achievement.hk at gmail.com Thu Nov 13 18:31:32 2008 From: achievement.hk at gmail.com (Achievement Chan) Date: Fri, 14 Nov 2008 02:31:32 +0800 Subject: [Linux-cluster] Is GFS2 stable for production system? Message-ID: Dear All, Is GFS2 stable for production system? Is it still not defined as stable by redhat? I would like to use it with apache, and courier-imap (Maildir format mailbox) reagrds, Achievement Chan From a.holstvoogd at nedforce.nl Thu Nov 13 19:09:51 2008 From: a.holstvoogd at nedforce.nl (Arthur Holstvoogd) Date: Thu, 13 Nov 2008 20:09:51 +0100 Subject: [Linux-cluster] Is GFS2 stable for production system? In-Reply-To: References: Message-ID: Hi, In my experience, it is not really. We have been using the 'stable' version that comes with centos 5.2 and had to upgrade to beta releases to get it running properly. Now we have some fs corruption we can't solve because fsck keeps segfaulting. Some of the other tools don't work either, especially if your mixing versions of the different parts. (which is to be expected of course) I guess if your using the latest beta versions it runs stable, but some tooling just doesn't work properly yet. Might be out specific case of bad luck, but still... I'm considering moving back to gfs1 for production. Cheers Arthur On Nov 13, 2008, at 19:31 , Achievement Chan wrote: > Dear All, > Is GFS2 stable for production system? Is it still not defined as > stable by redhat? > > I would like to use it with apache, and courier-imap (Maildir format > mailbox) > > reagrds, > Achievement Chan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From kanderso at redhat.com Thu Nov 13 19:54:24 2008 From: kanderso at redhat.com (Kevin Anderson) Date: Thu, 13 Nov 2008 13:54:24 -0600 Subject: [Linux-cluster] Is GFS2 stable for production system? In-Reply-To: References: Message-ID: <1226606064.4108.60.camel@dhcp80-204.msp.redhat.com> For anything prior to the latest Fedora 9/10 kernels and RHEL 5.3, GFS2 is not considered stable and has known blatant issues with cluster coherent operations. RHEL 5.3 beta has been released with a working GFS2. This will move from our tech preview status to supported when RHEL 5.3 GA version. No one should be running GFS2 in a cluster production environment prior to these versions. Kevin On Fri, 2008-11-14 at 02:31 +0800, Achievement Chan wrote: > Dear All, > Is GFS2 stable for production system? Is it still not defined as > stable by redhat? > > I would like to use it with apache, and courier-imap (Maildir format mailbox) > > reagrds, > Achievement Chan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From andrew at ntsg.umt.edu Thu Nov 13 20:01:18 2008 From: andrew at ntsg.umt.edu (Andrew Neuschwander) Date: Thu, 13 Nov 2008 13:01:18 -0700 Subject: [Linux-cluster] Is GFS2 stable for production system? In-Reply-To: <1226606064.4108.60.camel@dhcp80-204.msp.redhat.com> References: <1226606064.4108.60.camel@dhcp80-204.msp.redhat.com> Message-ID: <491C878E.70607@ntsg.umt.edu> Does this mean that it should work fine (i.e. no known issues) as a local only file system (one node, with lock_nolock)? -A -- Andrew A. Neuschwander, RHCE Linux Systems/Software Engineer College of Forestry and Conservation The University of Montana http://www.ntsg.umt.edu andrew at ntsg.umt.edu - 406.243.6310 Kevin Anderson wrote: > For anything prior to the latest Fedora 9/10 kernels and RHEL 5.3, GFS2 > is not considered stable and has known blatant issues with cluster > coherent operations. RHEL 5.3 beta has been released with a working > GFS2. This will move from our tech preview status to supported when RHEL > 5.3 GA version. No one should be running GFS2 in a cluster production > environment prior to these versions. > > Kevin > > On Fri, 2008-11-14 at 02:31 +0800, Achievement Chan wrote: >> Dear All, >> Is GFS2 stable for production system? Is it still not defined as >> stable by redhat? >> >> I would like to use it with apache, and courier-imap (Maildir format mailbox) >> >> reagrds, >> Achievement Chan >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From kanderso at redhat.com Thu Nov 13 20:07:40 2008 From: kanderso at redhat.com (Kevin Anderson) Date: Thu, 13 Nov 2008 14:07:40 -0600 Subject: [Linux-cluster] Is GFS2 stable for production system? In-Reply-To: <491C878E.70607@ntsg.umt.edu> References: <1226606064.4108.60.camel@dhcp80-204.msp.redhat.com> <491C878E.70607@ntsg.umt.edu> Message-ID: <1226606860.4108.64.camel@dhcp80-204.msp.redhat.com> Single node GFS2 has been pretty stable since May/June timeframe of this year, not sure which kernel version this mapped into. We did an errata release post RHEL 5.2 with a couple of fixes in a special gfs2-kmod rpm, but still is considered tech preview from a support perspective. Kevin On Thu, 2008-11-13 at 13:01 -0700, Andrew Neuschwander wrote: > Does this mean that it should work fine (i.e. no known issues) as a > local only file system (one node, with lock_nolock)? > > -A > -- > Andrew A. Neuschwander, RHCE > Linux Systems/Software Engineer > College of Forestry and Conservation > The University of Montana > http://www.ntsg.umt.edu > andrew at ntsg.umt.edu - 406.243.6310 > > Kevin Anderson wrote: > > For anything prior to the latest Fedora 9/10 kernels and RHEL 5.3, GFS2 > > is not considered stable and has known blatant issues with cluster > > coherent operations. RHEL 5.3 beta has been released with a working > > GFS2. This will move from our tech preview status to supported when RHEL > > 5.3 GA version. No one should be running GFS2 in a cluster production > > environment prior to these versions. > > > > Kevin > > > > On Fri, 2008-11-14 at 02:31 +0800, Achievement Chan wrote: > >> Dear All, > >> Is GFS2 stable for production system? Is it still not defined as > >> stable by redhat? > >> > >> I would like to use it with apache, and courier-imap (Maildir format mailbox) > >> > >> reagrds, > >> Achievement Chan > >> > >> -- > >> Linux-cluster mailing list > >> Linux-cluster at redhat.com > >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From treed at ultraviolet.org Thu Nov 13 20:16:34 2008 From: treed at ultraviolet.org (Tracy Reed) Date: Thu, 13 Nov 2008 12:16:34 -0800 Subject: [Linux-cluster] Is GFS2 stable for production system? In-Reply-To: References: Message-ID: <20081113201634.GA10575@tracyreed.org> On Fri, Nov 14, 2008 at 02:31:32AM +0800, Achievement Chan spake thusly: > I would like to use it with apache, and courier-imap (Maildir format mailbox) See the thread: [Linux-cluster] GFS performance of imap service (Maildir) from yesterday. It seems that GFS performance may not be so good with lots of small files due to locking issues. Not sure if Kevin Anderson 's comments about using RHEL5.3 will have any bearing on this issue or not. I've been following the GFS project since it was first created at the University of Minnesota, went to Sistina, DotHill and other SAN vendors looked into it and built special disk firmware which implemented locking, went closed source, OpenGFS forked it, Sistina got bought by Redhat, OpenGFS sorta died, went open-source by RedHat again...and through all of this I'm still waiting for a stable GFS with decent performance. :) Hopefully RHEL5.3 finally provides it. -- Tracy Reed http://tracyreed.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From kanderso at redhat.com Thu Nov 13 20:50:27 2008 From: kanderso at redhat.com (Kevin Anderson) Date: Thu, 13 Nov 2008 14:50:27 -0600 Subject: [Linux-cluster] Is GFS2 stable for production system? In-Reply-To: <20081113201634.GA10575@tracyreed.org> References: <20081113201634.GA10575@tracyreed.org> Message-ID: <1226609427.4108.77.camel@dhcp80-204.msp.redhat.com> On Thu, 2008-11-13 at 12:16 -0800, Tracy Reed wrote: > On Fri, Nov 14, 2008 at 02:31:32AM +0800, Achievement Chan spake thusly: > > I would like to use it with apache, and courier-imap (Maildir format mailbox) > > See the thread: [Linux-cluster] GFS performance of imap service > (Maildir) from yesterday. It seems that GFS performance may not be so > good with lots of small files due to locking issues. Not sure if Kevin > Anderson 's comments about using RHEL5.3 will > have any bearing on this issue or not. > > I've been following the GFS project since it was first created at the > University of Minnesota, went to Sistina, DotHill and other SAN > vendors looked into it and built special disk firmware which > implemented locking, went closed source, OpenGFS forked it, Sistina > got bought by Redhat, OpenGFS sorta died, went open-source by RedHat > again...and through all of this I'm still waiting for a stable GFS > with decent performance. :) Hopefully RHEL5.3 finally provides it. > We have done what we can from an internal testing standpoint. Would really love some real life feedback on the beta bits or the latest Fedora releases to see if we have been successful. So, how about it, participate by pulling the RHEL 5.3 beta version, configure courier-imap with real data and provide feedback. This is an opensource community effort, am sure Steve Whitehouse would love to have feedback, analysis, patches, etc..... :) Thanks Kevin From mwill at penguincomputing.com Thu Nov 13 21:31:54 2008 From: mwill at penguincomputing.com (Michael Will) Date: Thu, 13 Nov 2008 13:31:54 -0800 Subject: [Linux-cluster] Clusters with multihomed hosts In-Reply-To: <1226595264.9571.75.camel@quoit> References: <491C5402.8060706@auckland.ac.nz> <1226595264.9571.75.camel@quoit> Message-ID: <20081113213154.GI17008@miwi.penguincomputing.com> Double check that the netmask on the two interfaces when you inspect it with ifconfig is 255.255.255.0 and not 255.255.0.0 so that you really are on two separate networks. Michael On Thu, Nov 13, 2008 at 04:54:24PM +0000, Steven Whitehouse wrote: > Hi, > > On Thu, 2008-11-13 at 09:21 -0700, Michael O'Sullivan wrote: > > Hi all, > > > > I need to know more about using redundant NICs in clusters. > > > > I have a 2-node cluster with 2 NICs in each node. The first NICs on each > > node are connected to one switch, the second NICs on each node are > > connected to another switch. This is an experimental arrangement so I am > > using /etc/hosts instead of DNS. It appears that the cluster software > > becomes confused if I put both NICs for the hosts in the /etc/hosts > > file, even if they are in different subnets. Here is the /etc/hosts file > > I would like to use: > > > > # localhost line > > 192.168.10.1 node1 > > 192.168.10.2 node2 > > 192.168.20.1 node1 # Second NIC on node 1 > > 192.168.20.2 node2 # Second NIC on node 2 > > > > but this seems to cause the cluster to hang (confused about which NIC to > > use?), so I have removed the last 2 lines and everything works fine. > > However, this means if the switch on the 192.168.10.x subnet fails the > > cluster will "break". If the cluster would recognise that node1 and > > node2 are available via the second NICs then I wouldn't have to worry > > about this single point-of-failure. > > > The trouble with this kind of thing is that you find that its not easy > to control which external IP address a particular application uses as > you have discovered. It can be done though, with the aid of iproute2. > > The kernel will look at the routing table to work out where to send a > particular packet, and once its found a suitable destination interface > it will then look at the various possible source IPs on that interface > in order to work out which one to use. It tries to use the source > address which has most bits matching with the destination (counting from > the network end to the host end of the IP address) so that if the > destination address is on a particular subnet, it will try to use an IP > from the same subnet as the source address if one is available. > > You can alter this quite easily though, you just set up a second routing > table and use routing rules in order to select the correct table > according to your network. Thats where iproute2 comes in and there is a > set of docs here: http://lartc.org/ > > Also, just because you have two NIC's connected to different switches > doesn't mean that you need to give them different IP addresses/subnets. > The Linux IP stack can easily cope with them being the same, which would > also simplify the situation that you have, where, I suspect the cluster > stack has replied via a different NIC and thus received a different > source address. > > > I have thought about bonding the NICs which (I think) would take care of > > the problem, but I have heard that boding two NICs usually does not > > give double the bandwidth. I have read a little about high-availability > > and failing over IP addresses, but this seems to be between different > > nodes, not different NICs in the same host. > > > > Would anyone please be able to give me some direction about the best way > > to set up my cluster and NICs to make the cluster reliable in the event > > of switch failure? And keep the full bandwidth of the NICs intact? > > > > Thanks in advance for any help you can give. Kind regards, Mike > > > The problem with bonding is that a single packet can only use a single > one of the parallel links. Also, by using multiple links on a single > stream (i.e. a TCP connection) you run the risk, if you are not careful, > of reordering the packets and that can cause slow downs at the receiving > end, and possibly generation of out of order ACK packets which might > cause retransmissions at the sending end, further slowing things down. > > The Linux bonding driver has various modes to try and avoid that, and in > addition it also has 802.3ad mode which allows it to automatically > negotiate settings with a switch. Thats ideal if all the bonded links > for a particular node go to the same switch, but won't work across > switches as in your situation. > > I suspect that the choice will come down to one of the following: > > 1. something easy to set up & not very efficient in terms of bandwidth, > but probably not too bad either. > -> choose bonding (just be sure to select the right mode) > > 2. something more complex to set up, but which can be made to make full > use all of the available bandwidth, and be extended into more > complicated setups (dynamic routing, etc), given enough application > support & tweeking. > -> choose the IP based solution > > Steve. > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From diegoliz at gmail.com Thu Nov 13 23:03:54 2008 From: diegoliz at gmail.com (Diego Liziero) Date: Fri, 14 Nov 2008 00:03:54 +0100 Subject: [Linux-cluster] Is GFS2 stable for production system? In-Reply-To: References: Message-ID: <68fe87e60811131503m25acf15ei7175c93d6ff95453@mail.gmail.com> On Thu, Nov 13, 2008 at 7:31 PM, Achievement Chan wrote: > Dear All, > Is GFS2 stable for production system? Not as regards the one in current 5.2. I had this issues: - locking with samba running on a single node that caused continuous freezes of the shares, even when exported read-only after the latest kernel+cman update (this could have been cased by the fact that not all nodes have been rebooted after the update). After a reboot of all nodes and a switch to the old stable gfs this hasn't happened any longer. - the last modification time of a file is not always updated on all nodes (doing an ls of the same file on different nodes may show different modification time after it has been edited on a node). - sometimes the space used by deleted files is not freed. Launching gfs2_fsck -y on unmounted filesystem detect lots of "Ondisk status is 1 (Data) but FSCK thinks it should be 0 (Free)" messages, but, despite that, the space is still not freed. BTW I've still a corrupted gfs2 empty filesystem that shows incorrect free space if someone feels like to debug it. Regards, Diego. From npf-mlists at eurotux.com Fri Nov 14 09:48:20 2008 From: npf-mlists at eurotux.com (Nuno Fernandes) Date: Fri, 14 Nov 2008 09:48:20 +0000 Subject: [Linux-cluster] Fence VirtualIron - i have the script but... In-Reply-To: References: Message-ID: <200811140948.20717.npf-mlists@eurotux.com> On Tuesday 11 November 2008 08:26:05 Maurizio Rottin wrote: > Hello everyone, > I need to fence VirtualIron VM in order to GFS to work when a node is > not responding. > > Actually, i wrote a simple python script that fences the node, but...i > don't understand how to integrate it in the cluster suite! > > What files should i touch in order to have this fence method in luci? Hi, We also use virtual iron. Could you please post your script? Thanks, Nuno Fernandes -------------- next part -------------- An HTML attachment was scrubbed... URL: From npf-mlists at eurotux.com Fri Nov 14 10:00:13 2008 From: npf-mlists at eurotux.com (Nuno Fernandes) Date: Fri, 14 Nov 2008 10:00:13 +0000 Subject: [Linux-cluster] Problem in clvmd/dlm_recoverd Message-ID: <200811141000.13623.npf-mlists@eurotux.com> Hi, we have an cluster with 7 machines with a SAN. We are using them to provide virtual machines, so we are using clvmd. At some point we are unable to access any of the pv/lv/vg tools. They are all stuck. From stracing them i've come to the conclusion that they are waiting for clvmd. Has anyone been in this situation? Thanks for any help, Nuno Fernandes in host xen1: Linux blade01.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 14:13:09 EST 2008 x86_64 x86_64 x86_64 GNU/Linux lvm2-cluster-2.02.32-4.el5 cman-2.0.84-2.el5_2.1 PID TTY STAT TIME COMMAND 20874 ? D< 0:00 \_ [dlm_recoverd] 20854 pts/1 S+ 0:00 \_ /bin/sh /sbin/service clvmd start 20861 pts/1 S+ 0:00 \_ /bin/bash /etc/init.d/clvmd start 20931 pts/1 S+ 0:00 \_ /usr/sbin/vgscan -d 20869 ? Ssl 0:00 clvmd -T40 ps ax -o pid,cmd,wchan 20874 [dlm_recoverd] - ------------------------------ Connection to xen1 closed. in host xen2: Linux blade02.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 WEST 2007 x86_64 x86_64 x86_64 GNU/Linux lvm2-cluster-2.02.16-3.el5 cman-2.0.64-1.0.1.el5 PID TTY STAT TIME COMMAND 22662 ? D< 0:00 \_ [dlm_recoverd] 22613 ? Ssl 0:02 clvmd -T40 ps ax -o pid,cmd,wchan 22662 [dlm_recoverd] - ------------------------------ Connection to xen2 closed. in host xen3: Linux blade03.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 WEST 2007 x86_64 x86_64 x86_64 GNU/Linux lvm2-cluster-2.02.16-3.el5 cman-2.0.64-1.0.1.el5 PID TTY STAT TIME COMMAND 22236 ? D< 0:00 \_ [dlm_recoverd] 22231 ? Ssl 0:02 clvmd -T40 ps ax -o pid,cmd,wchan Connection to xen3 closed. 22236 [dlm_recoverd] dlm_wait_function ------------------------------ in host xen4: Linux blade04.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 WEST 2007 x86_64 x86_64 x86_64 GNU/Linux lvm2-cluster-2.02.16-3.el5 cman-2.0.64-1.0.1.el5 PID TTY STAT TIME COMMAND 25097 ? D< 0:00 \_ [dlm_recoverd] 25092 ? Ssl 0:02 clvmd -T40 ps ax -o pid,cmd,wchan 25097 [dlm_recoverd] dlm_wait_function ------------------------------ Connection to xen4 closed. in host xen5: Linux blade05.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 14:13:09 EST 2008 x86_64 x86_64 x86_64 GNU/Linux lvm2-cluster-2.02.32-4.el5 cman-2.0.84-2.el5_2.1 PID TTY STAT TIME COMMAND 22333 ? D< 0:00 \_ [dlm_recoverd] 22328 ? Ssl 0:02 clvmd -T40 ps ax -o pid,cmd,wchan 22333 [dlm_recoverd] - ------------------------------ Connection to xen5 closed. in host xen6: Linux blade06.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 14:13:09 EST 2008 x86_64 x86_64 x86_64 GNU/Linux lvm2-cluster-2.02.32-4.el5 cman-2.0.84-2.el5_2.1 PID TTY STAT TIME COMMAND ps ax -o pid,cmd,wchan ------------------------------ Connection to xen6 closed. in host xen7: Linux blade07.dc.xpto.com 2.6.18-92.1.13.el5xen #1 SMP Wed Sep 24 20:01:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux lvm2-cluster-2.02.32-4.el5 cman-2.0.84-2.el5 cman-2.0.84-2.el5_2.1 PID TTY STAT TIME COMMAND 19793 ? D< 0:00 \_ [dlm_recoverd] 19788 ? Ssl 0:01 clvmd -T40 ps ax -o pid,cmd,wchan 19793 [dlm_recoverd] - ------------------------------ Connection to xen7 closed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ccaulfie at redhat.com Fri Nov 14 10:29:50 2008 From: ccaulfie at redhat.com (Christine Caulfield) Date: Fri, 14 Nov 2008 10:29:50 +0000 Subject: [Linux-cluster] Problem in clvmd/dlm_recoverd In-Reply-To: <200811141000.13623.npf-mlists@eurotux.com> References: <200811141000.13623.npf-mlists@eurotux.com> Message-ID: <491D531E.9010104@redhat.com> Nuno Fernandes wrote: > Hi, > > we have an cluster with 7 machines with a SAN. We are using them to > provide virtual machines, so we are using clvmd. > > At some point we are unable to access any of the pv/lv/vg tools. They > are all stuck. From stracing them i've come to the conclusion that they > are waiting for clvmd. > They could be waiting for fencing to complete. Have a look at the output from group_tool, that will tell you which services have recovered after a node has joined or left the cluster Chrissie > Nuno Fernandes > > in host xen1: > > Linux blade01.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 > 14:13:09 EST 2008 x86_64 x86_64 x86_64 GNU/Linux > > lvm2-cluster-2.02.32-4.el5 > > cman-2.0.84-2.el5_2.1 > > PID TTY STAT TIME COMMAND > > 20874 ? D< 0:00 \_ [dlm_recoverd] > > 20854 pts/1 S+ 0:00 \_ /bin/sh /sbin/service clvmd start > > 20861 pts/1 S+ 0:00 \_ /bin/bash /etc/init.d/clvmd start > > 20931 pts/1 S+ 0:00 \_ /usr/sbin/vgscan -d > > 20869 ? Ssl 0:00 clvmd -T40 > > ps ax -o pid,cmd,wchan > > 20874 [dlm_recoverd] - > > ------------------------------ > > Connection to xen1 closed. > > in host xen2: > > Linux blade02.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 > WEST 2007 x86_64 x86_64 x86_64 GNU/Linux > > lvm2-cluster-2.02.16-3.el5 > > cman-2.0.64-1.0.1.el5 > > PID TTY STAT TIME COMMAND > > 22662 ? D< 0:00 \_ [dlm_recoverd] > > 22613 ? Ssl 0:02 clvmd -T40 > > ps ax -o pid,cmd,wchan > > 22662 [dlm_recoverd] - > > ------------------------------ > > Connection to xen2 closed. > > in host xen3: > > Linux blade03.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 > WEST 2007 x86_64 x86_64 x86_64 GNU/Linux > > lvm2-cluster-2.02.16-3.el5 > > cman-2.0.64-1.0.1.el5 > > PID TTY STAT TIME COMMAND > > 22236 ? D< 0:00 \_ [dlm_recoverd] > > 22231 ? Ssl 0:02 clvmd -T40 > > ps ax -o pid,cmd,wchan > > Connection to xen3 closed. > > 22236 [dlm_recoverd] dlm_wait_function > > ------------------------------ > > in host xen4: > > Linux blade04.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 > WEST 2007 x86_64 x86_64 x86_64 GNU/Linux > > lvm2-cluster-2.02.16-3.el5 > > cman-2.0.64-1.0.1.el5 > > PID TTY STAT TIME COMMAND > > 25097 ? D< 0:00 \_ [dlm_recoverd] > > 25092 ? Ssl 0:02 clvmd -T40 > > ps ax -o pid,cmd,wchan > > 25097 [dlm_recoverd] dlm_wait_function > > ------------------------------ > > Connection to xen4 closed. > > in host xen5: > > Linux blade05.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 > 14:13:09 EST 2008 x86_64 x86_64 x86_64 GNU/Linux > > lvm2-cluster-2.02.32-4.el5 > > cman-2.0.84-2.el5_2.1 > > PID TTY STAT TIME COMMAND > > 22333 ? D< 0:00 \_ [dlm_recoverd] > > 22328 ? Ssl 0:02 clvmd -T40 > > ps ax -o pid,cmd,wchan > > 22333 [dlm_recoverd] - > > ------------------------------ > > Connection to xen5 closed. > > in host xen6: > > Linux blade06.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 > 14:13:09 EST 2008 x86_64 x86_64 x86_64 GNU/Linux > > lvm2-cluster-2.02.32-4.el5 > > cman-2.0.84-2.el5_2.1 > > PID TTY STAT TIME COMMAND > > ps ax -o pid,cmd,wchan > > ------------------------------ > > Connection to xen6 closed. > > in host xen7: > > Linux blade07.dc.xpto.com 2.6.18-92.1.13.el5xen #1 SMP Wed Sep 24 > 20:01:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux > > lvm2-cluster-2.02.32-4.el5 > > cman-2.0.84-2.el5 > > cman-2.0.84-2.el5_2.1 > > PID TTY STAT TIME COMMAND > > 19793 ? D< 0:00 \_ [dlm_recoverd] > > 19788 ? Ssl 0:01 clvmd -T40 > > ps ax -o pid,cmd,wchan > > 19793 [dlm_recoverd] - > > ------------------------------ > > Connection to xen7 closed. > From npf-mlists at eurotux.com Fri Nov 14 11:02:36 2008 From: npf-mlists at eurotux.com (Nuno Fernandes) Date: Fri, 14 Nov 2008 11:02:36 +0000 Subject: [Linux-cluster] Problem in clvmd/dlm_recoverd In-Reply-To: <491D531E.9010104@redhat.com> References: <200811141000.13623.npf-mlists@eurotux.com> <491D531E.9010104@redhat.com> Message-ID: <200811141102.37139.npf-mlists@eurotux.com> On Friday 14 November 2008 10:29:50 Christine Caulfield wrote: > Nuno Fernandes wrote: > > Hi, > > > > we have an cluster with 7 machines with a SAN. We are using them to > > provide virtual machines, so we are using clvmd. > > > > At some point we are unable to access any of the pv/lv/vg tools. They > > are all stuck. From stracing them i've come to the conclusion that they > > are waiting for clvmd. > > They could be waiting for fencing to complete. > > Have a look at the output from group_tool, that will tell you which > services have recovered after a node has joined or left the cluster I don't think that is the reason.. # group_tool type level name id state fence 0 default 00010002 none [1 2 3 4 5 7] dlm 1 clvmd 00010004 none [1 2 3 4 5 7] Any other ideas? Best regards, Nuno Fernandes > > Chrissie > > > Nuno Fernandes > > > > in host xen1: > > > > Linux blade01.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 > > 14:13:09 EST 2008 x86_64 x86_64 x86_64 GNU/Linux > > > > lvm2-cluster-2.02.32-4.el5 > > > > cman-2.0.84-2.el5_2.1 > > > > PID TTY STAT TIME COMMAND > > > > 20874 ? D< 0:00 \_ [dlm_recoverd] > > > > 20854 pts/1 S+ 0:00 \_ /bin/sh /sbin/service clvmd start > > > > 20861 pts/1 S+ 0:00 \_ /bin/bash /etc/init.d/clvmd start > > > > 20931 pts/1 S+ 0:00 \_ /usr/sbin/vgscan -d > > > > 20869 ? Ssl 0:00 clvmd -T40 > > > > ps ax -o pid,cmd,wchan > > > > 20874 [dlm_recoverd] - > > > > ------------------------------ > > > > Connection to xen1 closed. > > > > in host xen2: > > > > Linux blade02.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 > > WEST 2007 x86_64 x86_64 x86_64 GNU/Linux > > > > lvm2-cluster-2.02.16-3.el5 > > > > cman-2.0.64-1.0.1.el5 > > > > PID TTY STAT TIME COMMAND > > > > 22662 ? D< 0:00 \_ [dlm_recoverd] > > > > 22613 ? Ssl 0:02 clvmd -T40 > > > > ps ax -o pid,cmd,wchan > > > > 22662 [dlm_recoverd] - > > > > ------------------------------ > > > > Connection to xen2 closed. > > > > in host xen3: > > > > Linux blade03.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 > > WEST 2007 x86_64 x86_64 x86_64 GNU/Linux > > > > lvm2-cluster-2.02.16-3.el5 > > > > cman-2.0.64-1.0.1.el5 > > > > PID TTY STAT TIME COMMAND > > > > 22236 ? D< 0:00 \_ [dlm_recoverd] > > > > 22231 ? Ssl 0:02 clvmd -T40 > > > > ps ax -o pid,cmd,wchan > > > > Connection to xen3 closed. > > > > 22236 [dlm_recoverd] dlm_wait_function > > > > ------------------------------ > > > > in host xen4: > > > > Linux blade04.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 > > WEST 2007 x86_64 x86_64 x86_64 GNU/Linux > > > > lvm2-cluster-2.02.16-3.el5 > > > > cman-2.0.64-1.0.1.el5 > > > > PID TTY STAT TIME COMMAND > > > > 25097 ? D< 0:00 \_ [dlm_recoverd] > > > > 25092 ? Ssl 0:02 clvmd -T40 > > > > ps ax -o pid,cmd,wchan > > > > 25097 [dlm_recoverd] dlm_wait_function > > > > ------------------------------ > > > > Connection to xen4 closed. > > > > in host xen5: > > > > Linux blade05.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 > > 14:13:09 EST 2008 x86_64 x86_64 x86_64 GNU/Linux > > > > lvm2-cluster-2.02.32-4.el5 > > > > cman-2.0.84-2.el5_2.1 > > > > PID TTY STAT TIME COMMAND > > > > 22333 ? D< 0:00 \_ [dlm_recoverd] > > > > 22328 ? Ssl 0:02 clvmd -T40 > > > > ps ax -o pid,cmd,wchan > > > > 22333 [dlm_recoverd] - > > > > ------------------------------ > > > > Connection to xen5 closed. > > > > in host xen6: > > > > Linux blade06.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 > > 14:13:09 EST 2008 x86_64 x86_64 x86_64 GNU/Linux > > > > lvm2-cluster-2.02.32-4.el5 > > > > cman-2.0.84-2.el5_2.1 > > > > PID TTY STAT TIME COMMAND > > > > ps ax -o pid,cmd,wchan > > > > ------------------------------ > > > > Connection to xen6 closed. > > > > in host xen7: > > > > Linux blade07.dc.xpto.com 2.6.18-92.1.13.el5xen #1 SMP Wed Sep 24 > > 20:01:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux > > > > lvm2-cluster-2.02.32-4.el5 > > > > cman-2.0.84-2.el5 > > > > cman-2.0.84-2.el5_2.1 > > > > PID TTY STAT TIME COMMAND > > > > 19793 ? D< 0:00 \_ [dlm_recoverd] > > > > 19788 ? Ssl 0:01 clvmd -T40 > > > > ps ax -o pid,cmd,wchan > > > > 19793 [dlm_recoverd] - > > > > ------------------------------ > > > > Connection to xen7 closed. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcasiraghi73 at gmail.com Fri Nov 14 13:45:46 2008 From: mcasiraghi73 at gmail.com (Mauro Casiraghi) Date: Fri, 14 Nov 2008 14:45:46 +0100 Subject: [Linux-cluster] Service Dependency Message-ID: <18c35c650811140545v4d8b8b0cob050e2964965ef6c@mail.gmail.com> How can i set service dependency ?? I need to set service dependency for to services service A depend from service B How is the file cluster.conf to do this ?? Best Regards Mauro Casiraghi -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcasiraghi73 at gmail.com Fri Nov 14 13:46:03 2008 From: mcasiraghi73 at gmail.com (Mauro Casiraghi) Date: Fri, 14 Nov 2008 14:46:03 +0100 Subject: [Linux-cluster] Service Dependency Message-ID: <18c35c650811140546m14e40adcuc4b37fcc9b739a6b@mail.gmail.com> How can i set service dependency ?? I need to set service dependency for to services service A depend from service B How is the file cluster.conf to do this ?? Best Regards Mauro Casiraghi -------------- next part -------------- An HTML attachment was scrubbed... URL: From veliogluh at itu.edu.tr Fri Nov 14 14:31:49 2008 From: veliogluh at itu.edu.tr (Hakan VELIOGLU) Date: Fri, 14 Nov 2008 16:31:49 +0200 Subject: [Linux-cluster] Cman doesn't realize the failed node In-Reply-To: <20081112131700.16801d3do17enme8@webmail.beta.itu.edu.tr> References: <91632af9d61397469c647a809455d908@mail.bardiauto.hu> <20081112131700.16801d3do17enme8@webmail.beta.itu.edu.tr> Message-ID: <20081114163149.27785czyk7jxdfbs@webmail.beta.itu.edu.tr> Hi, I solved my problem. When the kernel IP forwarding feature (/proc/sys/net/ipv4/ip_forward) is 0, then cluster nodes don't realize the failure. I write this solution to help others. However, I am curious about that is all of your RedHat 5 OS default ip_forward settting is enabled? Are all your failover clusters working as expected ? Have a nice day list. PS: This change is included just in Red Hat 4 Cluster Suite documentation not in Red Hat 5 cluster suite. Interesting!!! ----- veliogluh at itu.edu.tr den ileti --------- Tarih: Wed, 12 Nov 2008 13:17:00 +0200 Kimden: Hakan VELIOGLU Yan?t Adresi:linux clustering Konu: [Linux-cluster] Cman doesn't realize the failed node Kime: linux clustering > Hi, > > I am testing and trying to understand the cluster environment. I ve > built a two node cluster system without any service (Red Hat EL 5.2 > x64). I run the cman and rgmanager services succesfully and then > poweroff one node suddenly. After thsi I excpect that the other node > realize this failure and take up all the resources however running > node doesn't realize this failure. I use "cman_tool nodes" and > "clustat" commands and they say the failed node is active and > online. What am i missing? Why cman doesn't realize the failure? > > [root at cl1 ~]# cat /etc/cluster/cluster.conf > > > > > > > > > > > > > > > > > restricted="1"> > name="cl2.cc.itu.edu.tr" priority="1"/> > name="cl1.cc.itu.edu.tr" priority="2"/> > > > > name="veritabani" recovery="restart"/> > > > [root at cl1 ~]# > > > When the node gows down, the TOTEM repeastedly logs messages like this. > Nov 12 13:12:57 cl1 openais[5809]: [TOTEM] The consensus timeout expired. > Nov 12 13:12:57 cl1 openais[5809]: [TOTEM] entering GATHER state from 3. > Nov 12 13:13:03 cl1 openais[5809]: [TOTEM] The consensus timeout expired. > Nov 12 13:13:03 cl1 openais[5809]: [TOTEM] entering GATHER state from 3. > Nov 12 13:13:09 cl1 openais[5809]: [TOTEM] The consensus timeout expired. > Nov 12 13:13:09 cl1 openais[5809]: [TOTEM] entering GATHER state from 3. > Nov 12 13:13:14 cl1 openais[5809]: [TOTEM] The consensus timeout expired. > Nov 12 13:13:14 cl1 openais[5809]: [TOTEM] entering GATHER state from 3. > > > > Hakan > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > ----- veliogluh at itu.edu.tr den iletiyi bitir ----- From maurizio.rottin at gmail.com Fri Nov 14 14:31:37 2008 From: maurizio.rottin at gmail.com (Maurizio Rottin) Date: Fri, 14 Nov 2008 15:31:37 +0100 Subject: Fwd: [Linux-cluster] Fence VirtualIron - i have the script but... In-Reply-To: References: <200811140948.20717.npf-mlists@eurotux.com> Message-ID: it seems that i lost the mailing list address...then i forward the answer to Nuno Fernandes. And Nuno, please, write after this thread if you make some improvments, i'll be glad to discuss bugs/improvements/ideas. Bye! ---------- Forwarded message ---------- From: Maurizio Rottin Date: 2008/11/14 Subject: Re: [Linux-cluster] Fence VirtualIron - i have the script but... To: Nuno Fernandes 2008/11/14 Nuno Fernandes : > We also use virtual iron. Could you please post your script? > > Thanks, > > Nuno Fernandes first of all you must have a working java>1.5.0 than (mind that my scripts are little "temporary") vim /sbin/fence_vivm #!/bin/bash # Maurizio Rottin 2008-11-11 # # fence a VirtualIron VirtualServer # ############################################################################### # # Copyright (C) 2008 Maurizio Rottin. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License, version 2, as # published by the Free Software Foundation. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # or visit http://www.gnu.org/licenses/gpl.txt # ############################################################################### #must parse arguments passed from stdin #in format name=param #must ignore #name=param #there should not be two name=param string passed, as the xml should be considered bad! name="" vm="" while read param;do if [ `expr index $param \# ` -ne 0 ];then continue elif expr match $param 'ip=.*' >/dev/null ;then ip=${param#ip=} elif expr match $param 'name=.*' >/dev/null ;then name=${param#name=} elif expr match $param 'vm=.*' >/dev/null ;then vm=${param#vm=} fi done #end parsing if [ "$vm" == "" ];then echo "$$: No name provided!" exit 1 fi basedir="/root/vso" if [ -d $basedir ];then cd /root/vso sh ./runner --vivmgr=http://youripaddress:80 --username=admin --password='yourpassword' --inputfile=vsOperations.py --action=fence --vs="${vm}" retval=$? else retval=1 fi exit $retval ->>>>end of script of course you can pass also password, ip and so on, depending on your needs. then you must copy this files from VirtualIron directory into /root/vso (if you change /root/vso, then change it also in the fence_vivm) ./system ./system/resources ./system/resources/lib ./system/resources/lib/myoocomClient.jar ./system/resources/lib/MgmtAPI.jar ./system/resources/lib/MgmtUtil.jar ./system/resources/lib/runner.jar ./system/resources/lib/jython.jar ./system/resources/lib/log4j.jar ./system/resources/lib/MgmtControl.jar ./system/resources/lib/jline.jar ./system/resources/lib/MgmtTools.jar ./system/resources/lib/myoodbClient.jar ./etc ./etc/runner.properties ok, now we need two more scripts This is taken from VirtualIron so i don't know if some licences are involved. vim /root/vso/runner #!/bin/bash export INS_PATH=`pwd` # # run in the context of the install # cd "$INS_PATH" # # usage: runner.sh --mode= # # --mode : default is # --vivmgr : default is # --username : default is # --password : default is # --inputfile : default is # # --help : print usage # java -Xmx512m -Dpython.inclusive.packages="java,javax,org.python" -jar "system/resources/lib/runner.jar" $* exit $? ->>> end script chmod 700 /root/vso/runner AND taken from VirtualIron scripts and modified for our needs: vim /root/vso/vsOperations.py # # vsOperations: perform start/stop/shutdown/restart action on VirtualServer # from com.virtualiron.vce.mgmt.api import VirtualizationManager from com.virtualiron.vce.mgmt.api.virtual import VirtualServer import java.lang import java.util import os import string import sys import traceback def Usage(): if os.name == 'nt': command = 'runner.bat' else: command = 'runner.sh' print 'Usage: %s --inputfile=vsOperations.py --vs="Virtual Server" --action=[start,stop,shutdown,restart]' % (command) print 'or Usage: %s --vivmgr=http://192.168.0.48:80 --username=admin --password=\'dba at iron\' --inputfile=vsOperations.py --action=[start,stop,shutdown,restart,reboot]' % (command) sys.exit(1) # # parse command line arguments # vsName = None action = None for arg in sys.argv[1:]: if string.find(arg, "--vs=") != -1: tokens = string.splitfields(arg, "=") vsName = tokens[1] elif string.find(arg, "--action=") != -1: tokens = string.splitfields(arg, "=") action = tokens[1] # # check if required arguments were specified # if vsName is None or action is None: Usage() # # get connection to Database # configurationManager = VirtualizationManager.getConfigurationManager() # # find virtual server object # vs = configurationManager.findObject(VirtualServer, vsName) if vs is None: print 'FAIL to find VirtualServer %s' % (vsName) sys.exit(1) # # wrap VS action in job control # error = 0 try: jobName = java.lang.Long.toString(configurationManager.getLocalTime()) job = configurationManager.createJob(jobName) job.begin() if action == 'start': vs.start() job.addOperationDescription("Start VirtualServer", vs, vs, vs) elif action == 'stop': vs.stop() job.addOperationDescription("Stop VirtualServer", vs, vs, vs) elif action == 'shutdown': vs.shutdown() job.addOperationDescription("Shutdown VirtualServer", vs, vs, vs) elif action == 'restart': vs.restart() job.addOperationDescription("Restart VirtualServer", vs, vs, vs) elif action == 'reboot': vs.reboot() job.addOperationDescription("Hard reset and boot VirtualServer", vs, vs, vs) elif action == 'fence': statusEvent = vs.getStatusEvent().toString() if string.find(statusEvent, 'VirtualServerStoppedEvent') == 0: print "fence: starting" vs.start() job.addOperationDescription("Fence Start VirtualServer", vs, vs, vs) elif string.find(statusEvent, 'VirtualServerRunningEvent') == 0: print "fence: rebooting" vs.reboot() job.addOperationDescription("Fence reboot VirtualServer", vs, vs, vs) elif string.find(statusEvent, 'VirtualServerStartingEvent') == 0: print "wait 60 sec reboot()? or do nothing?" else: print "Unknown status %s " % (statusEvent) else: error = 1 print 'Unknown action:', action job.abort() if not error: job.commit() # commit job except java.lang.Throwable, throw: job.abort() # if job fails, rollback throw.printStackTrace() except: job.abort() # if job fails, rollback traceback.print_exc() if error: Usage() ->>> end script chmod 700 /root/vso/vsOperations.py Last thing is to modify the cluster.conf copy your cluster.conf somewhere add plus 1 to "config_version=number" the beginning should look like this: <------begin --------->cut<------- where "bend_02" and "bend_01" are the real names in the VirtualIron manager! update the cluste.conf while the cluster is running: ccs_tool update cluster.conf enjoy it! -- mr -- mr From teigland at redhat.com Fri Nov 14 16:26:49 2008 From: teigland at redhat.com (David Teigland) Date: Fri, 14 Nov 2008 10:26:49 -0600 Subject: [Linux-cluster] Problem in clvmd/dlm_recoverd In-Reply-To: <200811141000.13623.npf-mlists@eurotux.com> References: <200811141000.13623.npf-mlists@eurotux.com> Message-ID: <20081114162649.GA4054@redhat.com> On Fri, Nov 14, 2008 at 10:00:13AM +0000, Nuno Fernandes wrote: > 22236 [dlm_recoverd] dlm_wait_function > 25097 [dlm_recoverd] dlm_wait_function dlm recovery appears to be stuck; this is usually due to a problem at the network level. The recovery seems to be caused by a node starting clvmd. sysrq-t backtraces from all the nodes could confirm some of this, and adding to cluster.conf would give us more information the next time it happens. Dave From curtis at athabascau.ca Fri Nov 14 17:06:51 2008 From: curtis at athabascau.ca (Curtis Collicutt) Date: Fri, 14 Nov 2008 10:06:51 -0700 Subject: [Linux-cluster] Problem in clvmd/dlm_recoverd In-Reply-To: <200811141000.13623.npf-mlists@eurotux.com> References: <200811141000.13623.npf-mlists@eurotux.com> Message-ID: <1226682383-sup-5263@beaker.cs.athabascau.ca> Excerpts from Nuno Fernandes's message of Fri Nov 14 03:00:13 -0700 2008: > Hi, > > we have an cluster with 7 machines with a SAN. We are using them to provide > virtual machines, so we are using clvmd. > > At some point we are unable to access any of the pv/lv/vg tools. They are all > stuck. From stracing them i've come to the conclusion that they are waiting > for clvmd. > > Has anyone been in this situation? This happens to me as well every once and a while. Haven't figure it out yet either. Thanks, Curtis. > > Thanks for any help, > Nuno Fernandes > > in host xen1: > > Linux blade01.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 14:13:09 EST > 2008 x86_64 x86_64 x86_64 GNU/Linux > lvm2-cluster-2.02.32-4.el5 > > cman-2.0.84-2.el5_2.1 > > PID TTY STAT TIME COMMAND > > 20874 ? D< 0:00 \_ [dlm_recoverd] > > 20854 pts/1 S+ 0:00 \_ /bin/sh /sbin/service clvmd start > > 20861 pts/1 S+ 0:00 \_ /bin/bash /etc/init.d/clvmd start > > 20931 pts/1 S+ 0:00 \_ /usr/sbin/vgscan -d > > 20869 ? Ssl 0:00 clvmd -T40 > > ps ax -o pid,cmd,wchan > > 20874 [dlm_recoverd] - > > ------------------------------ > > Connection to xen1 closed. > > in host xen2: > > Linux blade02.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 WEST > 2007 x86_64 x86_64 x86_64 GNU/Linux > lvm2-cluster-2.02.16-3.el5 > > cman-2.0.64-1.0.1.el5 > > PID TTY STAT TIME COMMAND > > 22662 ? D< 0:00 \_ [dlm_recoverd] > > 22613 ? Ssl 0:02 clvmd -T40 > > ps ax -o pid,cmd,wchan > > 22662 [dlm_recoverd] - > > ------------------------------ > > Connection to xen2 closed. > > in host xen3: > > Linux blade03.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 WEST > 2007 x86_64 x86_64 x86_64 GNU/Linux > lvm2-cluster-2.02.16-3.el5 > > cman-2.0.64-1.0.1.el5 > > PID TTY STAT TIME COMMAND > > 22236 ? D< 0:00 \_ [dlm_recoverd] > > 22231 ? Ssl 0:02 clvmd -T40 > > ps ax -o pid,cmd,wchan > > Connection to xen3 closed. > > 22236 [dlm_recoverd] dlm_wait_function > > ------------------------------ > > in host xen4: > > Linux blade04.dc.xpto.com 2.6.18-8.1.14.el5xen #1 SMP Thu Oct 4 11:38:56 WEST > 2007 x86_64 x86_64 x86_64 GNU/Linux > lvm2-cluster-2.02.16-3.el5 > > cman-2.0.64-1.0.1.el5 > > PID TTY STAT TIME COMMAND > > 25097 ? D< 0:00 \_ [dlm_recoverd] > > 25092 ? Ssl 0:02 clvmd -T40 > > ps ax -o pid,cmd,wchan > > 25097 [dlm_recoverd] dlm_wait_function > > ------------------------------ > > Connection to xen4 closed. > > in host xen5: > Linux blade05.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 14:13:09 EST > 2008 x86_64 x86_64 x86_64 GNU/Linux > lvm2-cluster-2.02.32-4.el5 > cman-2.0.84-2.el5_2.1 > PID TTY STAT TIME COMMAND > 22333 ? D< 0:00 \_ [dlm_recoverd] > 22328 ? Ssl 0:02 clvmd -T40 > ps ax -o pid,cmd,wchan > 22333 [dlm_recoverd] - > ------------------------------ > Connection to xen5 closed. > in host xen6: > Linux blade06.dc.xpto.com 2.6.18-92.1.17.el5xen #1 SMP Tue Nov 4 14:13:09 EST > 2008 x86_64 x86_64 x86_64 GNU/Linux > lvm2-cluster-2.02.32-4.el5 > cman-2.0.84-2.el5_2.1 > PID TTY STAT TIME COMMAND > ps ax -o pid,cmd,wchan > ------------------------------ > Connection to xen6 closed. > in host xen7: > Linux blade07.dc.xpto.com 2.6.18-92.1.13.el5xen #1 SMP Wed Sep 24 20:01:15 EDT > 2008 x86_64 x86_64 x86_64 GNU/Linux > lvm2-cluster-2.02.32-4.el5 > cman-2.0.84-2.el5 > cman-2.0.84-2.el5_2.1 > PID TTY STAT TIME COMMAND > 19793 ? D< 0:00 \_ [dlm_recoverd] > 19788 ? Ssl 0:01 clvmd -T40 > ps ax -o pid,cmd,wchan > 19793 [dlm_recoverd] - __ This communication is intended for the use of the recipient to whom it is addressed, and may contain confidential, personal, and or privileged information. Please contact us immediately if you are not the intended recipient of this communication, and do not copy, distribute, or take action relying on it. Any communications received in error, or subsequent reply, should be deleted or destroyed. --- From lhh at redhat.com Fri Nov 14 21:49:43 2008 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 14 Nov 2008 16:49:43 -0500 Subject: [Linux-cluster] Limit service restarting times In-Reply-To: <8a5668960811130156i2ef6f533s1f39332b5dd72195@mail.gmail.com> References: <8a5668960811130156i2ef6f533s1f39332b5dd72195@mail.gmail.com> Message-ID: <1226699383.25751.66.camel@ayanami> On Thu, 2008-11-13 at 10:56 +0100, Juan Ramon Martin Blanco wrote: > First of all, hello and many thanks everyone, this list has helped me > a lot in the cluster world ;) > > I have configured a 2 node cluster with RHEL 5.2, shared storage and > GFS2. > I have configured several services with our company own software. This > software evolves fast because we are in active development, so > sometimes cores are dumped. When this happens, the cluster tries to > restart the failing service again and again...filling the service's > filesystem with cores. > Is there any way to limit the number of retries for a certain service? max_restarts="x" * Maximum tolerated. Ex: 3 means the *4th* restart will fail restart_expire="y" * After this # of seconds time, a restart is forgotten. -- Lon From npf-mlists at eurotux.com Fri Nov 14 21:53:13 2008 From: npf-mlists at eurotux.com (Nuno Fernandes) Date: Fri, 14 Nov 2008 21:53:13 +0000 Subject: [Linux-cluster] Problem in clvmd/dlm_recoverd In-Reply-To: <20081114162649.GA4054@redhat.com> References: <200811141000.13623.npf-mlists@eurotux.com> <20081114162649.GA4054@redhat.com> Message-ID: <200811142153.13752.npf-mlists@eurotux.com> On Friday 14 November 2008 16:26:49 David Teigland wrote: > On Fri, Nov 14, 2008 at 10:00:13AM +0000, Nuno Fernandes wrote: > > 22236 [dlm_recoverd] dlm_wait_function > > 25097 [dlm_recoverd] dlm_wait_function > > dlm recovery appears to be stuck; this is usually due to a problem at the > network level. The recovery seems to be caused by a node starting clvmd. Hi, I don't know if it helps, but groupd is using all available CPU, but only in 2 of the nodes. I don't know if it's required to be up.. but we've disabled IPV6.. snip of modprobe.conf: alias net-pf-10 off Best regards, ./npf > > sysrq-t backtraces from all the nodes could confirm some of this, and > adding to cluster.conf would give us more information > the next time it happens. > > Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhh at redhat.com Fri Nov 14 21:54:45 2008 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 14 Nov 2008 16:54:45 -0500 Subject: [Linux-cluster] RHEL3 Cluster Broken Pipe error and Heartbeat configuration In-Reply-To: <29e045b80811120544j1a85eeay237b72daf8de3e16@mail.gmail.com> References: <29e045b80811120544j1a85eeay237b72daf8de3e16@mail.gmail.com> Message-ID: <1226699685.25751.72.camel@ayanami> On Wed, 2008-11-12 at 19:14 +0530, lingu wrote: > cluquorumd[1921]: Disk-TB: Detected I/O Hang! Eep. This means that I/O to shared storage has gotten slow. Strange. I heard reports of this on another cluster (after going from U3->U8), but I don't know what the cause is. With this cluster, we straced the cluquorumd process and found that it was slowing down *a lot* in the write() call when writing to shared storage. You can try the current U9+erratum clumanager or the test release if you want to (it makes unlock more robust when I/O performance is slow for some reason). However, someone really needs to profile the kernel if you're seeing slow write times while stracing cluquorumd... -- Lon From lhh at redhat.com Fri Nov 14 22:02:03 2008 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 14 Nov 2008 17:02:03 -0500 Subject: [Linux-cluster] RedHat Cluster Suite cluster resource management In-Reply-To: <18c35c650811130614i2a58e035w3bb9f574075389ca@mail.gmail.com> References: <18c35c650811130614i2a58e035w3bb9f574075389ca@mail.gmail.com> Message-ID: <1226700123.25751.80.camel@ayanami> On Thu, 2008-11-13 at 15:14 +0100, Mauro Casiraghi wrote: > I had created this cluster configuration with Redhat Cluster Suite > > I Have one Service Group with the follow resources > > Service Group Name : WEB > > Resources of the service group: > > 1) IP_ADRESS > 2) APACHE > > The resource dependency are: > > The Apache resource is dependent of ip_adress Assuming you are running RHEL 5.3 beta or RHEL5.2 (or using stable2 branch of linux-cluster):