From cluster at squiz.net Wed Mar 1 03:02:11 2006 From: cluster at squiz.net (oly) Date: Wed, 01 Mar 2006 14:02:11 +1100 Subject: [Linux-cluster] GFS = filesystem consistency error Message-ID: <1141182131.29086.46.camel@sunrise.squiz.net> Hi there I've got a 4nodes RHEL4 cluster with GFS version 6.1.0 (built Jun 7 2005 12:46:04). The shared disk is a NAS detected by aoe as /dev/etherd/e0.0. ANd i have problem on few files on teh file system : if i tried to modify the inodes o this files (delete the file, or unlink the inode) the cluster nodes where i launch the command lost the GFS and the GFS modules stay busy and cannot be remove from the kernel. my nodes is so stuck and the only solution is only to hardware restart this nodes. All the GFS journal seems to work fine ...i can even get stat of the DEAD file. Is GFS got problem to manipulate file in a 'more than 1 million files' folder ? IS anyone got a solution to remove this dead files or delete teh fodler that content all these dead files ? Is a gfs.fsck can resolv my problem ? Is there any later version that fix this problem ? Thanks in advance. PS = see below all the details The error i get when i try to unlink the file inode: ===========ERROR============ GFS: fsid=entcluster:sataide.2: fatal: filesystem consistency error GFS: fsid=entcluster:sataide.2: inode = 8516674/8516674 GFS: fsid=entcluster:sataide.2: function = gfs_change_nlink GFS: fsid=entcluster:sataide.2: file = /usr/src/build/574067-i686/BUILD/smp/src/gfs/inode.c, line = 843 GFS: fsid=entcluster:sataide.2: time = 1141080134 GFS: fsid=entcluster:sataide.2: about to withdraw from the cluster GFS: fsid=entcluster:sataide.2: waiting for outstanding I/O GFS: fsid=entcluster:sataide.2: telling LM to withdraw lock_dlm: withdraw abandoned memory GFS: fsid=entcluster:sataide.2: withdrawn mh_magic = 0x01161970 mh_type = 4 mh_generation = 68 mh_format = 400 mh_incarn = 6 no_formal_ino = 8516674 no_addr = 8516674 di_mode = 0664 di_uid = 500 di_gid = 500 di_nlink = 0 di_size = 0 di_blocks = 1 di_atime = 1141042636 di_mtime = 1140001370 di_ctime = 1140001370 di_major = 0 di_minor = 0 di_rgrp = 8513987 di_goal_rgrp = 8513987 di_goal_dblk = 2682 di_goal_mblk = 2682 di_flags = 0x00000004 di_payload_format = 0 di_type = 1 di_height = 0 di_incarn = 0 di_pad = 0 di_depth = 0 di_entries = 0 no_formal_ino = 0 no_addr = 0 di_eattr = 0 di_reserved = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ========END OF ERROR========== My cman status: ==========STATUS============ Protocol version: 5.0.1 Config version: 4 Cluster name: entcluster Cluster ID: 42548 Cluster Member: Yes Membership state: Cluster-Member Nodes: 4 Expected_votes: 1 Total_votes: 4 Quorum: 3 Active subsystems: 5 Node name: XXX.domainX.tld Node addresses: x.x.x.x ========END CMAN========= My gfs_tool df : ============DF========= /home: SB lock proto = "lock_dlm" SB lock table = "entcluster:sataide" SB ondisk format = 1309 SB multihost format = 1401 Block size = 4096 Journals = 4 Resource Groups = 274 Mounted lock proto = "lock_dlm" Mounted lock table = "entcluster:sataide" Mounted host data = "" Journal number = 0 Lock module flags = Local flocks = FALSE Local caching = FALSE Oopses OK = FALSE Type Total Used Free use% ------------------------------------------------------------------------ inodes 100642 100642 0 100% metadata 3842538 8527 3834011 0% data 13999476 2760327 11239149 20% =============END DF ========= Version of my modules : ========modules======== CMAN 2.6.9-36.0 (built May 31 2005 12:15:02) installed DLM 2.6.9-34.0 (built Jun 2 2005 15:17:56) installed Lock_Harness 2.6.9-35.5 (built Jun 7 2005 12:42:30) installed GFS 2.6.9-35.5 (built Jun 7 2005 12:42:49) installed aoe: aoe_init: AoE v2.6-11 initialised. Lock_DLM (built Jun 7 2005 12:42:32) installed ========end modules======== -- Aurelien Lemaire (oly) http://www.squiz.net Sydney | Canberra | London 92 Jarrett St Leichhardt, Sydney, NSW 2040 T:+61 2 9568 6866 F:+61 2 9568 6733 From cjk at techma.com Wed Mar 1 19:06:30 2006 From: cjk at techma.com (Kovacs, Corey J.) Date: Wed, 1 Mar 2006 14:06:30 -0500 Subject: [Linux-cluster] NFS exports, RHCS3 and Autofs. Message-ID: Folks, question regarding the subject line... We have a 3 node cluster running RHEL3u6+RHCS3u6. Each system has a nfs service with two exports, one system has an additional nfs service with two exports. This last service starts and mounts locally to the cluster, the problem is the two exports never get listed in to exportfs list. We can issue clusvcadm -R clu_srv2 and the exports will appear. Any thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From pcaulfie at redhat.com Thu Mar 2 08:25:33 2006 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Thu, 02 Mar 2006 08:25:33 +0000 Subject: [Linux-cluster] where to find user documentation for DLM? In-Reply-To: References: Message-ID: <4406ABFD.5050502@redhat.com> jalmeter_99 at yahoo.com wrote: > Hello, > > I have googled everything I can think of, but haven't found any > documentation for using DLM as a developer. Would someone please > point me at a tutorial of some kind? > > Background: > > My employer has recently (last week) set up a GFS cluster for > evaluation. I am trying to set up a locking test that mimics a system > that we use on VMS. > If you download or checout the sources there is documentation in cluster/dlm/doc as well as several example programs in cluster/dlm/test/usertest. -- patrick From Frank.Weyns at ordina.nl Thu Mar 2 10:19:02 2006 From: Frank.Weyns at ordina.nl (Weyns, Frank) Date: Thu, 2 Mar 2006 11:19:02 +0100 Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? Message-ID: <4D30FCF33FE1FC4DB79C18A73D46C6730859A3@BA12-0013.work.local> I'm designing a very simple oracle cluster with a NetApp filer. Just two nodes, one oracle production instance falling over to the second node if needed. Second node running the "test-acceptance" oracle instance, which is brought down if needed. The Oracle filesystems ( binary, database and archive logs) will be nfs mounted. (I worked with Fiber SANs before, not a NetApp I have my doubts but you can take them away ;-) Any caveats ? Any best practices. Why should I avoid nfs or why is it good ? Versions to have or avoid.) If you don't want to fill the mailing list with unneeded materials: Frank*Weyns.net (*=@) Regards, Frank Disclaimer Dit bericht met eventuele bijlagen is vertrouwelijk en uitsluitend bestemd voor de geadresseerde. Indien u niet de bedoelde ontvanger bent, wordt u verzocht de afzender te waarschuwen en dit bericht met eventuele bijlagen direct te verwijderen en/of te vernietigen. Het is niet toegestaan dit bericht en eventuele bijlagen te vermenigvuldigen, door te sturen, openbaar te maken, op te slaan of op andere wijze te gebruiken. Ordina N.V. en/of haar groepsmaatschappijen accepteren geen verantwoordelijkheid of aansprakelijkheid voor schade die voortvloeit uit de inhoud en/of de verzending van dit bericht. This e-mail and any attachments are confidential and is solely intended for the addressee only. If you are not the intended recipient, please notify the sender and delete and/or destroy this message and any attachments immediately. It is prohibited to copy, to distribute, to disclose or to use this e-mail and any attachments in any other way. Ordina N.V. and/or its group companies do not accept any responsibility nor liability for any damage resulting from the content of and/or the transmission of this message. -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3544 bytes Desc: not available URL: From deval.kulshrestha at progression.com Thu Mar 2 12:47:36 2006 From: deval.kulshrestha at progression.com (Deval kulshrestha) Date: Thu, 2 Mar 2006 18:17:36 +0530 Subject: [Linux-cluster] Buffer I/O error on dev cciss c0d0p9, lost pagewrite due to io error Message-ID: <001401c63df7$7b660da0$cf00a8c0@PROGRESSION> Hi Whenever I write some large file on SAN Logical Volume, after writing some data it gives me error in /var/log/message. as follows: Feb 28 11:08:53 s1_new kernel: cciss: cmd f7400000 timedout Feb 28 11:08:54 s1_new kernel: cciss: cmd f7436fb4 timedout Feb 28 11:08:54 s1_new kernel: printk: 232 messages suppressed. Feb 28 11:08:54 s1_new kernel: Buffer I/O error on device cciss/c0d0p9, logical block 2162845 Feb 28 11:08:54 s1_new kernel: lost page write due to I/O error on cciss/c0d0p9 Also it is noticed that when started with clean reboot of systems servers including SAN device) I/O on SAN works very fast for the first few seconds then it starts timeout and page lost errors This is HP MSA 500G2 connected with HP DL 360 G4 , RHEL 4 ES U1(2.6.9-11) RHCS4 With Regard Deval K. Progression Infonet Pvt. Ltd. 55, Independent Electronic Modules, Sector - 18, Electronic City, Gurgaon - 122015 Tel : - 0124 - 2455070, Ext. 215, Fax: 91-124-2398647 Mobile : - 98186 -82509 URL : - www.progression.com =========================================================== Privileged or confidential information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), please delete this message and kindly notify the sender by an emailed reply. Opinions, conclusions and other information in this message that do not relate to the official business of Progression and its associate entities shall be understood as neither given nor endorsed by them. ------------------------------------------------------------- Progression Infonet Private Limited, Gurgaon (Haryana), India -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fabrizio.Lippolis at AurigaInformatica.it Thu Mar 2 16:37:04 2006 From: Fabrizio.Lippolis at AurigaInformatica.it (Fabrizio Lippolis) Date: Thu, 02 Mar 2006 17:37:04 +0100 Subject: [Linux-cluster] cluster.conf reference Message-ID: <44071F30.9060500@aurigainformatica.it> I am configuring a cluster of two Linux machines and I would like to configure a service so that, should a machine fail, the over can start the service and go on. I have read I have to configure a failover domain containing the two machines and the service so that the task can be accomplished. Unfortunately the cluster.conf man page just documents a very basic configuration and shows nothing about failover domains and so on. Can anybody point me to better resources because even googling I wasn't able to find anything around and with the graphical tool (system-config-cluster) I am unable to save the configuration. Thanks in advance. -- Fabrizio Lippolis fabrizio.lippolis at aurigainformatica.it Auriga Informatica s.r.l. Via Don Guanella 15/B - 70124 Bari Tel.: 080/5025414 Fax: 080/5027448 From devrim at gunduz.org Thu Mar 2 16:50:03 2006 From: devrim at gunduz.org (Devrim GUNDUZ) Date: Thu, 2 Mar 2006 18:50:03 +0200 (EET) Subject: [Linux-cluster] GFS: "transport endpoint is not connected" error In-Reply-To: <200602281014.15703.hlawatschek@atix.de> References: <200602281014.15703.hlawatschek@atix.de> Message-ID: Hi, On Tue, 28 Feb 2006, Mark Hlawatschek wrote: > have you already started your cluster environment (CMAN, fenced, DLM/Gulm) ? > What are the exact steps you have done ? Sorry for the delay in response. After double,triple checking the configuration, I make CMAN and DLM working. Thanks, and now I can mount the filesystems. It was a misconfiguration in the cluster.conf And also thanks to Michael Will who helped me off-list for the LVM thing. Regards, -- Devrim GUNDUZ Kivi Bili?im Teknolojileri - http://www.kivi.com.tr devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org From devrim at gunduz.org Thu Mar 2 17:14:44 2006 From: devrim at gunduz.org (Devrim GUNDUZ) Date: Thu, 2 Mar 2006 19:14:44 +0200 (EET) Subject: [Linux-cluster] Using GFS on a hybrid system Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, We have a RHEL hybrid system, which has 8 servers in it. First of all, let me draw a picture of the system: Due to a binary driver problem (IBM!), we had to install RHEL ES 4 U1 to 4 servers (Let's call them S1, S2, S3 and S4) . The other ones have RHEL AS 4 U2 (S5,S6,S7,S8) . The ESU1 ones have GFS 6.0 and the other ones have 6.1. They are connected to a SAN. 2 of the ASU2 ones are using a seperate partition in SAN, and I had no problem in clustering and mounting the systems. S3 and S4 will work as a Cluster. S1, S2, S5 and S6 are standalone servers. S1,S2,S5 and S6 needs shared access to the LVM#1. S1,S2,S3 and S4 needs shared access to another partition in SAN. S1,S2,S5 and S6 needs shared access to the LVM#2. The problem arose when we wanted to share LVM#1. We mkfs'ed LVM#1 using GFS 6.1 from S6. It is ok when we mount the LVM from S5 and S6. As we want to access data from S1 and S2, S5 and S6 ooopes and we need to reboot the servers, even if we mount with -o oopses_ok. Now the questions: * What should be the cluster.conf files for S1...S6? Should they have the same cluster name? * Is using GFS 6.0 and 6.1 dangerous? I have to use 6.0 in ESU1 servers. Should I rollback to RHEL AS 4 U1 on the U2 systems? I wanted to ask the list before getting help from Red Hat, for Google to catch the answer and possibly help other people who may need it. Any help/comment is appreciated. Regards, - -- Devrim GUNDUZ Kivi Bili?im Teknolojileri - http://www.kivi.com.tr devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFEBygJ4zE8DGqpiZARAgdoAKCKaTtq1RsaRUee6rI6rQzdlroChQCePomf 0A5B4mQP1Zdw84dearDMdQw= =H/v9 -----END PGP SIGNATURE----- From devrim at gunduz.org Thu Mar 2 17:41:19 2006 From: devrim at gunduz.org (Devrim GUNDUZ) Date: Thu, 2 Mar 2006 19:41:19 +0200 (EET) Subject: [Linux-cluster] Using GFS on a hybrid system In-Reply-To: References: Message-ID: Hi, On Thu, 2 Mar 2006, Devrim GUNDUZ wrote: > Due to a binary driver problem (IBM!), we had to install RHEL ES 4 U1 to 4 > servers (Let's call them S1, S2, S3 and S4) . The other ones have RHEL AS 4 > U2 (S5,S6,S7,S8) . The ESU1 ones have GFS 6.0 and the other ones have 6.1. Oops, there is a typo: The ESU1 ones have GFS 6.1 and the other ones have 6.1.3. > * Is using GFS 6.0 and 6.1 dangerous? I have to use 6.0 in ESU1 servers. > Should I rollback to RHEL AS 4 U1 on the U2 systems? This should also be " Is using GFS 6.1.0 and 6.1.3 dangerous?" Regards, -- Devrim GUNDUZ Kivi Bili?im Teknolojileri - http://www.kivi.com.tr devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org From lhh at redhat.com Thu Mar 2 19:29:43 2006 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 02 Mar 2006 14:29:43 -0500 Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? In-Reply-To: <4D30FCF33FE1FC4DB79C18A73D46C6730859A3@BA12-0013.work.local> References: <4D30FCF33FE1FC4DB79C18A73D46C6730859A3@BA12-0013.work.local> Message-ID: <1141327783.13130.163.camel@ayanami.boston.redhat.com> On Thu, 2006-03-02 at 11:19 +0100, Weyns, Frank wrote: > I'm designing a very simple oracle cluster with a NetApp filer. > Just two nodes, one oracle production instance falling over to the second node if needed. > Second node running the "test-acceptance" oracle instance, which is brought down if needed. > > The Oracle filesystems ( binary, database and archive logs) will be nfs mounted. > (I worked with Fiber SANs before, not a NetApp I have my doubts but you can take them away ;-) > > Any caveats ? Any best practices. Why should I avoid nfs or why is it good ? Versions to have or avoid.) I wrote a howto on how to do it with SAN storage for 10g Release 2. It's fairly similar, I suspect, to how one might do it with NFS; it's attached to Bugzilla 182423 if you want to give it a peek and/or make comments. -- Lon From clusterbuilder at gmail.com Thu Mar 2 21:59:44 2006 From: clusterbuilder at gmail.com (Nick I) Date: Thu, 2 Mar 2006 14:59:44 -0700 Subject: [Linux-cluster] GFS Message-ID: Hi, I help maintain a Web site called www.clusterbuilder.org. We have a question and answer section to help those involved in clustering. We are developing a knowledgebase of cluster questions and responses so people with similar problems might be able to find answers to their question. I received a question concerning Red Hat and wanted to see what the opinions are of everyone here. "How to configure a two node GFS Cluster?" I have found some documentation on Red Hats' site, but wanted to see if anyone has any advice for this user. You can respond to this email or submit a response at www.clusterbuilder.org/FAQ Any response is greatly appreciated. Thanks, Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From cluster at squiz.net Thu Mar 2 22:13:41 2006 From: cluster at squiz.net (oly) Date: Fri, 03 Mar 2006 09:13:41 +1100 Subject: [Linux-cluster] GFS = filesystem consistency error In-Reply-To: <1141182131.29086.46.camel@sunrise.squiz.net> References: <1141182131.29086.46.camel@sunrise.squiz.net> Message-ID: <1141337621.26844.4.camel@sunrise.squiz.net> Hi there, I would like to give an update to my ticket. That will maybe help people who've got similar trouble : I resolved my problem by doing: - gfs_tool shrink /home (supposed to reclaim but did not) - gfs_tool reclaim /home (still not enough ) unmount the /home on all my nodes -gfs_fsck -y /dev/etherd/e0.0 -remount my /home VICTORY = i lost all the broken inode files ADVICE= avoid 1 million file folder in the future Cheers, Oly On Wed, 2006-03-01 at 14:02 +1100, oly wrote: > Hi there > I've got a 4nodes RHEL4 cluster with GFS version 6.1.0 (built > Jun 7 > 2005 12:46:04). > The shared disk is a NAS detected by aoe as /dev/etherd/e0.0. > ANd i have problem on few files on teh file system : if i tried > to > modify the inodes o this files (delete the file, or unlink the > inode) > the cluster nodes where i launch the command lost the GFS and > the GFS > modules stay busy and cannot be remove from the kernel. my nodes > is so > stuck and the only solution is only to hardware restart this > nodes. > All the GFS journal seems to work fine ...i can even get stat > of the > DEAD file. > Is GFS got problem to manipulate file in a 'more than 1 million > files' > folder ? > IS anyone got a solution to remove this dead files or delete > teh fodler > that content all these dead files ? > Is a gfs.fsck can resolv my problem ? > Is there any later version that fix this problem ? > > Thanks in advance. > PS = see below all the details > > The error i get when i try to unlink the file inode: > ===========ERROR============ > GFS: fsid=entcluster:sataide.2: fatal: filesystem consistency > error > GFS: fsid=entcluster:sataide.2: inode = 8516674/8516674 > GFS: fsid=entcluster:sataide.2: function = gfs_change_nlink > GFS: fsid=entcluster:sataide.2: file > = /usr/src/build/574067-i686/BUILD/smp/src/gfs/inode.c, line = > 843 > GFS: fsid=entcluster:sataide.2: time = 1141080134 > GFS: fsid=entcluster:sataide.2: about to withdraw from the > cluster > GFS: fsid=entcluster:sataide.2: waiting for outstanding I/O > GFS: fsid=entcluster:sataide.2: telling LM to withdraw > lock_dlm: withdraw abandoned memory > GFS: fsid=entcluster:sataide.2: withdrawn > mh_magic = 0x01161970 > mh_type = 4 > mh_generation = 68 > mh_format = 400 > mh_incarn = 6 > no_formal_ino = 8516674 > no_addr = 8516674 > di_mode = 0664 > di_uid = 500 > di_gid = 500 > di_nlink = 0 > di_size = 0 > di_blocks = 1 > di_atime = 1141042636 > di_mtime = 1140001370 > di_ctime = 1140001370 > di_major = 0 > di_minor = 0 > di_rgrp = 8513987 > di_goal_rgrp = 8513987 > di_goal_dblk = 2682 > di_goal_mblk = 2682 > di_flags = 0x00000004 > di_payload_format = 0 > di_type = 1 > di_height = 0 > di_incarn = 0 > di_pad = 0 > di_depth = 0 > di_entries = 0 > no_formal_ino = 0 > no_addr = 0 > di_eattr = 0 > di_reserved = > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00 > ========END OF ERROR========== > > My cman status: > ==========STATUS============ > Protocol version: 5.0.1 > Config version: 4 > Cluster name: entcluster > Cluster ID: 42548 > Cluster Member: Yes > Membership state: Cluster-Member > Nodes: 4 > Expected_votes: 1 > Total_votes: 4 > Quorum: 3 > Active subsystems: 5 > Node name: XXX.domainX.tld > Node addresses: x.x.x.x > ========END CMAN========= > > My gfs_tool df : > ============DF========= > /home: > SB lock proto = "lock_dlm" > SB lock table = "entcluster:sataide" > SB ondisk format = 1309 > SB multihost format = 1401 > Block size = 4096 > Journals = 4 > Resource Groups = 274 > Mounted lock proto = "lock_dlm" > Mounted lock table = "entcluster:sataide" > Mounted host data = "" > Journal number = 0 > Lock module flags = > Local flocks = FALSE > Local caching = FALSE > Oopses OK = FALSE > > Type Total Used Free > use% > > ------------------------------------------------------------------------ > inodes 100642 100642 0 > 100% > metadata 3842538 8527 3834011 0% > data 13999476 2760327 11239149 > 20% > =============END DF ========= > Version of my modules : > ========modules======== > CMAN 2.6.9-36.0 (built May 31 2005 12:15:02) installed > DLM 2.6.9-34.0 (built Jun 2 2005 15:17:56) installed > Lock_Harness 2.6.9-35.5 (built Jun 7 2005 12:42:30) installed > GFS 2.6.9-35.5 (built Jun 7 2005 12:42:49) installed > aoe: aoe_init: AoE v2.6-11 initialised. > Lock_DLM (built Jun 7 2005 12:42:32) installed > ========end modules======== > > > > -- > Aurelien Lemaire (oly) > http://www.squiz.net > Sydney | Canberra | London > 92 Jarrett St Leichhardt, Sydney, NSW 2040 > T:+61 2 9568 6866 > F:+61 2 9568 6733 > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From wcheng at redhat.com Thu Mar 2 20:15:58 2006 From: wcheng at redhat.com (Wendy Cheng) Date: Thu, 02 Mar 2006 15:15:58 -0500 Subject: [Linux-cluster] GFS = filesystem consistency error In-Reply-To: <1141337621.26844.4.camel@sunrise.squiz.net> References: <1141182131.29086.46.camel@sunrise.squiz.net> <1141337621.26844.4.camel@sunrise.squiz.net> Message-ID: <1141330558.6362.5.camel@localhost.localdomain> On Fri, 2006-03-03 at 09:13 +1100, oly wrote: > Hi there, > I would like to give an update to my ticket. That will maybe help people > who've got similar trouble : > I resolved my problem by doing: > - gfs_tool shrink /home (supposed to reclaim but did not) > - gfs_tool reclaim /home (still not enough ) > unmount the /home on all my nodes > -gfs_fsck -y /dev/etherd/e0.0 > -remount my /home > VICTORY = i lost all the broken inode files > ADVICE= avoid 1 million file folder in the future Sorry, this is probably a late reply but out of curiosity .. Look to me that the filesystem had been corrupted before you unlinked the file. Is there any other errors *before* the filesystem consistency errors ? Out of memory warning ? How much memory had you put on this machine ? And could I assume that was an i686 machine ? -- Wendy > > > On Wed, 2006-03-01 at 14:02 +1100, oly wrote: > > Hi there > > I've got a 4nodes RHEL4 cluster with GFS version 6.1.0 (built > > Jun 7 > > 2005 12:46:04). > > The shared disk is a NAS detected by aoe as /dev/etherd/e0.0. > > ANd i have problem on few files on teh file system : if i tried > > to > > modify the inodes o this files (delete the file, or unlink the > > inode) > > the cluster nodes where i launch the command lost the GFS and > > the GFS > > modules stay busy and cannot be remove from the kernel. my nodes > > is so > > stuck and the only solution is only to hardware restart this > > nodes. > > All the GFS journal seems to work fine ...i can even get stat > > of the > > DEAD file. > > Is GFS got problem to manipulate file in a 'more than 1 million > > files' > > folder ? > > IS anyone got a solution to remove this dead files or delete > > teh fodler > > that content all these dead files ? > > Is a gfs.fsck can resolv my problem ? > > Is there any later version that fix this problem ? > > > > Thanks in advance. > > PS = see below all the details > > > > The error i get when i try to unlink the file inode: > > ===========ERROR============ > > GFS: fsid=entcluster:sataide.2: fatal: filesystem consistency > > error > > GFS: fsid=entcluster:sataide.2: inode = 8516674/8516674 > > GFS: fsid=entcluster:sataide.2: function = gfs_change_nlink > > GFS: fsid=entcluster:sataide.2: file > > = /usr/src/build/574067-i686/BUILD/smp/src/gfs/inode.c, line = > > 843 > > GFS: fsid=entcluster:sataide.2: time = 1141080134 > > GFS: fsid=entcluster:sataide.2: about to withdraw from the > > cluster > > GFS: fsid=entcluster:sataide.2: waiting for outstanding I/O > > GFS: fsid=entcluster:sataide.2: telling LM to withdraw > > lock_dlm: withdraw abandoned memory > > GFS: fsid=entcluster:sataide.2: withdrawn > > mh_magic = 0x01161970 > > mh_type = 4 > > mh_generation = 68 > > mh_format = 400 > > mh_incarn = 6 > > no_formal_ino = 8516674 > > no_addr = 8516674 > > di_mode = 0664 > > di_uid = 500 > > di_gid = 500 > > di_nlink = 0 > > di_size = 0 > > di_blocks = 1 > > di_atime = 1141042636 > > di_mtime = 1140001370 > > di_ctime = 1140001370 > > di_major = 0 > > di_minor = 0 > > di_rgrp = 8513987 > > di_goal_rgrp = 8513987 > > di_goal_dblk = 2682 > > di_goal_mblk = 2682 > > di_flags = 0x00000004 > > di_payload_format = 0 > > di_type = 1 > > di_height = 0 > > di_incarn = 0 > > di_pad = 0 > > di_depth = 0 > > di_entries = 0 > > no_formal_ino = 0 > > no_addr = 0 > > di_eattr = 0 > > di_reserved = > > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > 00 00 00 00 00 00 00 00 > > ========END OF ERROR========== > > > > My cman status: > > ==========STATUS============ > > Protocol version: 5.0.1 > > Config version: 4 > > Cluster name: entcluster > > Cluster ID: 42548 > > Cluster Member: Yes > > Membership state: Cluster-Member > > Nodes: 4 > > Expected_votes: 1 > > Total_votes: 4 > > Quorum: 3 > > Active subsystems: 5 > > Node name: XXX.domainX.tld > > Node addresses: x.x.x.x > > ========END CMAN========= > > > > My gfs_tool df : > > ============DF========= > > /home: > > SB lock proto = "lock_dlm" > > SB lock table = "entcluster:sataide" > > SB ondisk format = 1309 > > SB multihost format = 1401 > > Block size = 4096 > > Journals = 4 > > Resource Groups = 274 > > Mounted lock proto = "lock_dlm" > > Mounted lock table = "entcluster:sataide" > > Mounted host data = "" > > Journal number = 0 > > Lock module flags = > > Local flocks = FALSE > > Local caching = FALSE > > Oopses OK = FALSE > > > > Type Total Used Free > > use% > > > > ------------------------------------------------------------------------ > > inodes 100642 100642 0 > > 100% > > metadata 3842538 8527 3834011 0% > > data 13999476 2760327 11239149 > > 20% > > =============END DF ========= > > Version of my modules : > > ========modules======== > > CMAN 2.6.9-36.0 (built May 31 2005 12:15:02) installed > > DLM 2.6.9-34.0 (built Jun 2 2005 15:17:56) installed > > Lock_Harness 2.6.9-35.5 (built Jun 7 2005 12:42:30) installed > > GFS 2.6.9-35.5 (built Jun 7 2005 12:42:49) installed > > aoe: aoe_init: AoE v2.6-11 initialised. > > Lock_DLM (built Jun 7 2005 12:42:32) installed > > ========end modules======== > > > > > > > > -- > > Aurelien Lemaire (oly) > > http://www.squiz.net > > Sydney | Canberra | London > > 92 Jarrett St Leichhardt, Sydney, NSW 2040 > > T:+61 2 9568 6866 > > F:+61 2 9568 6733 > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From suvankar_moitra at yahoo.com Fri Mar 3 05:14:09 2006 From: suvankar_moitra at yahoo.com (SUVANKAR MOITRA) Date: Thu, 2 Mar 2006 21:14:09 -0800 (PST) Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? In-Reply-To: <1141327783.13130.163.camel@ayanami.boston.redhat.com> Message-ID: <20060303051409.90698.qmail@web52302.mail.yahoo.com> dear lon, On 6th i will be in customer place and apply the whole thing again which u have put.If i am facing any problem i will mail u again and pl help me at that time. with warm regsrds Suvankar kolkata, india --- Lon Hohberger wrote: > On Thu, 2006-03-02 at 11:19 +0100, Weyns, Frank > wrote: > > I'm designing a very simple oracle cluster with a > NetApp filer. > > Just two nodes, one oracle production instance > falling over to the second node if needed. > > Second node running the "test-acceptance" oracle > instance, which is brought down if needed. > > > > The Oracle filesystems ( binary, database and > archive logs) will be nfs mounted. > > (I worked with Fiber SANs before, not a NetApp I > have my doubts but you can take them away ;-) > > > > Any caveats ? Any best practices. Why should I > avoid nfs or why is it good ? Versions to have or > avoid.) > > I wrote a howto on how to do it with SAN storage for > 10g Release 2. > It's fairly similar, I suspect, to how one might do > it with NFS; it's > attached to Bugzilla 182423 if you want to give it a > peek and/or make > comments. > > -- Lon > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From omer at faruk.net Fri Mar 3 07:35:56 2006 From: omer at faruk.net (Omer Faruk Sen) Date: Fri, 3 Mar 2006 09:35:56 +0200 (EET) Subject: [Linux-cluster] RH 4.3 release date? Message-ID: <52405.193.140.74.2.1141371356.squirrel@193.140.74.2> A few weeks ago it has been stated in this list that there are problems with rgmanager (clurgmgrd) that causes clurgmgrd to die suddenly. I hope this will be fixed in RH 4.3 so when will the 4.3 come out? There is also a bug in initscripts that causes service relocation fail (stop-after-stop problem) this one will also be fixed in 4.3 right? -- Omer Faruk Sen http://www.faruk.net From deval.kulshrestha at progression.com Fri Mar 3 11:13:17 2006 From: deval.kulshrestha at progression.com (Deval kulshrestha) Date: Fri, 3 Mar 2006 16:43:17 +0530 Subject: [Linux-cluster] Is anybody using MSA 500 G2 with HP Server's Message-ID: <000601c63eb3$790fbb20$cf00a8c0@PROGRESSION> Hi Now it s another problem have started coming up. Whenever I try to create a partition using mke2fs -j /dev/cciss/c0d0p5 Screen shows that its creating file system on SAN partitions, but in var/log/messages it continuously keeps on showing Feb 28 11:08:53 s1_new kernel: cciss: cmd f7400000 timedout Feb 28 11:08:54 s1_new kernel: cciss: cmd f7436fb4 timedout Feb 28 11:08:54 s1_new kernel: printk: 232 messages suppressed. Feb 28 11:08:54 s1_new kernel: Buffer I/O error on device cciss/c0d0p9, logical block 2162845 Feb 28 11:08:54 s1_new kernel: lost page write due to I/O error on cciss/c0d0p9 If partitions size is small i.e 5 GB it anyway get created using some 20-25 minutes, but if I create large file system than it used to give error , it simply stuck up at "Writing Superblock.." and above messages keeps on coming in /var/log/messages. If I write any data in partitions, than first few bytes get stored on partitions but later it also shows Buffer I/O error. I am using one MSA 500 G2 , two no. of HP DL360 G4P server with HP's HBA 642, Server installed with RHEL 4 ES U1 and RHCS4 Any Help Would be highly appreciable. With regard Deval K. =========================================================== Privileged or confidential information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), please delete this message and kindly notify the sender by an emailed reply. Opinions, conclusions and other information in this message that do not relate to the official business of Progression and its associate entities shall be understood as neither given nor endorsed by them. ------------------------------------------------------------- Progression Infonet Private Limited, Gurgaon (Haryana), India -------------- next part -------------- An HTML attachment was scrubbed... URL: From E.H.Beekman at amc.nl Fri Mar 3 12:57:28 2006 From: E.H.Beekman at amc.nl (Ewald Beekman) Date: Fri, 3 Mar 2006 13:57:28 +0100 Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? In-Reply-To: <1141327783.13130.163.camel@ayanami.boston.redhat.com> References: <4D30FCF33FE1FC4DB79C18A73D46C6730859A3@BA12-0013.work.local> <1141327783.13130.163.camel@ayanami.boston.redhat.com> Message-ID: <20060303125728.GU6090@core.amc.uva.nl> Hi Lon, Is the howto available on the net? I would like to play around with GFS and i suspect a NFS shared storage is the simplest way to try it out. best regards, Ewald... On Thu, Mar 02, 2006 at 02:29:43PM -0500, Lon Hohberger wrote: > On Thu, 2006-03-02 at 11:19 +0100, Weyns, Frank wrote: > > I'm designing a very simple oracle cluster with a NetApp filer. > > Just two nodes, one oracle production instance falling over to the second node if needed. > > Second node running the "test-acceptance" oracle instance, which is brought down if needed. > > > > The Oracle filesystems ( binary, database and archive logs) will be nfs mounted. > > (I worked with Fiber SANs before, not a NetApp I have my doubts but you can take them away ;-) > > > > Any caveats ? Any best practices. Why should I avoid nfs or why is it good ? Versions to have or avoid.) > > I wrote a howto on how to do it with SAN storage for 10g Release 2. > It's fairly similar, I suspect, to how one might do it with NFS; it's > attached to Bugzilla 182423 if you want to give it a peek and/or make > comments. > > -- Lon > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Ewald Beekman, Security Engineer, Academic Medical Center, dept. ADB/ICT Computer & Network Services, The Netherlands ## Your mind-mint is: The IRS spends God knows how much of your tax money on these toll-free information hot lines staffed by IRS employees, whose idea of a dynamite tax tip is that you should print neatly. If you ask them a real tax question, such as how you can cheat, they're useless. So, for guidance, you want to look to big business. Big business never pays a nickel in taxes, according to Ralph Nader, who represents a big consumer organization that never pays a nickel in taxes... -- Dave Barry, "Sweating Out Taxes" From lhh at redhat.com Fri Mar 3 15:13:19 2006 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 03 Mar 2006 10:13:19 -0500 Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? In-Reply-To: <20060303125728.GU6090@core.amc.uva.nl> References: <4D30FCF33FE1FC4DB79C18A73D46C6730859A3@BA12-0013.work.local> <1141327783.13130.163.camel@ayanami.boston.redhat.com> <20060303125728.GU6090@core.amc.uva.nl> Message-ID: <1141398799.13130.184.camel@ayanami.boston.redhat.com> On Fri, 2006-03-03 at 13:57 +0100, Ewald Beekman wrote: > Is the howto available on the net? I would like to play around with > GFS and i suspect a NFS shared storage is the simplest way to try > it out. It is only in bugzilla, because there is not enough feedback to even make the claim that it works ;) Here is a link to the tar.gz though (howto, some screen captures, and an agent), though: https://bugzilla.redhat.com/bugzilla/attachment.cgi?id=125371 Once there is enough feedback to get it working (read: send comments to the bugzilla as to what worked and what did not work for you, please), I will add it to CVS. -- Lon From lhh at redhat.com Fri Mar 3 15:23:20 2006 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 03 Mar 2006 10:23:20 -0500 Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? In-Reply-To: <20060303051409.90698.qmail@web52302.mail.yahoo.com> References: <20060303051409.90698.qmail@web52302.mail.yahoo.com> Message-ID: <1141399400.13130.195.camel@ayanami.boston.redhat.com> On Thu, 2006-03-02 at 21:14 -0800, SUVANKAR MOITRA wrote: > dear lon, > > On 6th i will be in customer place and apply the whole > thing again which u have put.If i am facing any > problem i will mail u again and pl help me at that > time. Suvankar, You should really test the HOWTO in your lab *before* trying to use it to deploy anything remotely close to a production environment. It is a beta-quality HOWTO right now. -- Lon From magobin at gmail.com Fri Mar 3 16:23:09 2006 From: magobin at gmail.com (Alessandro Binarelli) Date: Fri, 3 Mar 2006 17:23:09 +0100 Subject: [Linux-cluster] HELP: Newbe on service cluster configuration! Message-ID: <44086d7e.2afce7ef.6883.7291@mx.gmail.com> Hi, I'm a Newbe about cluster configuration and I've some problem understannding how to set up a service in cluster scenario.... For my test I use an nfs partition as shared storage for 2 server in cluster mode, so this is scenario: ServerA: 192.168.1.10 ServerB: 192.168.2.20 ServerC(nfs export) : 192.168.1.50 (hostname: san) Initially I tried to set up a dns in cluster, so I mounted nfs partition as /var/named/ If I try to run named normally on serverA it works...but when I try to start named in cluster it failed I set up a resources as NFS mount with this parameters: Name: Dns Mount point: /var/named Host: san Export Path: /SAN/DNS ...then I congigured a service with this resource and I attached an Ip address (10.23.5.240) and a script (etc/init.d/named/) as Private Resource. When I boot the server always I have an Service Failed, but when I check the logs I find only "clurgmgrd: #43: Service DNS has failed; can not start." and a "#13: Service DNS failed to stop cleanly" I suppose that thi is a problem how I configured the service... Any suggestion?? Thanks in advance! Alex From wcheng at redhat.com Fri Mar 3 14:07:15 2006 From: wcheng at redhat.com (Wendy Cheng) Date: Fri, 03 Mar 2006 09:07:15 -0500 Subject: [Linux-cluster] GFS = filesystem consistency error In-Reply-To: <1141330558.6362.5.camel@localhost.localdomain> References: <1141182131.29086.46.camel@sunrise.squiz.net> <1141337621.26844.4.camel@sunrise.squiz.net> <1141330558.6362.5.camel@localhost.localdomain> Message-ID: <1141394836.3705.11.camel@localhost.localdomain> On Thu, 2006-03-02 at 15:15 -0500, Wendy Cheng wrote: > On Fri, 2006-03-03 at 09:13 +1100, oly wrote: > > Hi there, > > I would like to give an update to my ticket. That will maybe help people > > who've got similar trouble : > > I resolved my problem by doing: > > - gfs_tool shrink /home (supposed to reclaim but did not) > > - gfs_tool reclaim /home (still not enough ) > > unmount the /home on all my nodes > > -gfs_fsck -y /dev/etherd/e0.0 > > -remount my /home > > VICTORY = i lost all the broken inode files > > ADVICE= avoid 1 million file folder in the future > One more question since this "file folder" confuses me. What's the max file count you have within one directory (excluding files in any subdirectory) ? This could be a bug from our end so the input is highly appreciated. -- Wendy From baesso at ksolutions.it Fri Mar 3 17:48:32 2006 From: baesso at ksolutions.it (Baesso Mirko) Date: Fri, 3 Mar 2006 18:48:32 +0100 Subject: [Linux-cluster] sun cluster ccp for redhat Message-ID: <984C9DBB29704B47B7AAD308F2C95A3B04DE71@kmail.ksolutions.it> Hi, i would like to known if there is a tool like cluster console panel to manage cluster node as sun cluster do Thanks in advance Baesso Mirko - System Engineer KSolutions.S.p.A. Via Lenin 132/26 56017 S.Martino Ulmiano (PI) - Italy tel.+ 39 0 50 898369 fax. + 39 0 50 861200 baesso at ksolutions.it http//www.ksolutions.it -------------- next part -------------- An HTML attachment was scrubbed... URL: From suvankar_moitra at yahoo.com Sat Mar 4 06:35:04 2006 From: suvankar_moitra at yahoo.com (SUVANKAR MOITRA) Date: Fri, 3 Mar 2006 22:35:04 -0800 (PST) Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? In-Reply-To: <1141399400.13130.195.camel@ayanami.boston.redhat.com> Message-ID: <20060304063504.30937.qmail@web52308.mail.yahoo.com> dear lon, I have some question about the script and the installation :-- 1> Can i install RHCS4 after install the Oracle 10g? 2> The /mnt/oracle mount point is temporary for the oracle installation or should i write on /etc/fstab ? 3> Can i mention ORACLE_HOME,ORACLE_BASE,ORACLE_SID etc on .bash_profile of every node or leave it as it is only create oracle user and group? 4> Where should i place oracledb.sh file? I think its required in every node, am i write ? 5>What is the exact use of oracledb.sh file? 6> How can i shutdown the oracle, should i write script for that, like orastop and orastart for up the oracle? thanks and warm regards Suvankar --- Lon Hohberger wrote: > On Thu, 2006-03-02 at 21:14 -0800, SUVANKAR MOITRA > wrote: > > dear lon, > > > > On 6th i will be in customer place and apply the > whole > > thing again which u have put.If i am facing any > > problem i will mail u again and pl help me at that > > time. > > Suvankar, > > You should really test the HOWTO in your lab > *before* trying to use it > to deploy anything remotely close to a production > environment. It is a > beta-quality HOWTO right now. > > -- Lon > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From tekion at gmail.com Sun Mar 5 16:11:08 2006 From: tekion at gmail.com (Screaming Eagle) Date: Sun, 5 Mar 2006 11:11:08 -0500 Subject: [Linux-cluster] FC4 and LVM GFS ... Message-ID: All, I got this to work with FC4. However, rebooting the server does not mount the gfs file system (lvm + gfs). Here's what I have for modprobe.conf: alias eth0 e1000 alias eth1 e1000 alias usb-controller ohci-hcd alias block-major-152 aoe alias char-major-152 aoe any idea? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tekion at gmail.com Sun Mar 5 16:12:11 2006 From: tekion at gmail.com (Screaming Eagle) Date: Sun, 5 Mar 2006 11:12:11 -0500 Subject: [Linux-cluster] limits on fs size for GFS ... Message-ID: Does any one know what is the limit of file system size on GFS, in particular LVM + GFS. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.bridgwater at sinergy.it Mon Mar 6 09:24:40 2006 From: s.bridgwater at sinergy.it (Simon Bridgwater) Date: Mon, 06 Mar 2006 10:24:40 +0100 Subject: [Linux-cluster] vsftpd clusterized in virtual IP + iscsi initiator question Message-ID: <3d16136b7b5caf4d9be1d00bd1a36bc0@sinergy.it> Hi I am new to the red hat cluster suite and have recently set up a two node HA cluster (Red Hat Cluster Suite 4) with iscsi inititiatior software (iscsid) and a netapp as the iscsi storage target. I have a problem with clusterizing vsftpd on a virtual IP that belongs to the cluster (not the real node's IP) . If I bind vsftpd standalone on the real IP of the node it works fine (listen_address ip_real). If I bind vsftpd standalone on a virtual IP defined as a cluster resource it starts to give me problems. Ftp clients (even from the localhost) connect but after a few "ls" commands it starts to get a "passive mode refused". Has anybody any suggestions ? Also I have the following question. I have two network cards in each node in an active-failover ethernet channel bond. The two cards are connected to two seperate switches for fault tollerance. If RHCS 4 doesn't use a quorum partition and the heartbeat is exclusively via network, how can the cluster tell when the iscsi initiator connection is down (while the network is up) and failover to the other node ? Simon Bridgwater -------------- next part -------------- An HTML attachment was scrubbed... URL: From jan.kudjak at snt.sk Sun Mar 5 21:47:40 2006 From: jan.kudjak at snt.sk (Kudjak Jan) Date: Sun, 5 Mar 2006 22:47:40 +0100 Subject: [Linux-cluster] 4 node gfs cluster, quorum needs 3 Message-ID: <139607FAAB7E0F46AD1A9BA381EC815A012C2516@KLEIO.snt.sk> Hello, I have at this time 4 node gfs cluster using RLM. Two nodes (node1, node2) have mounted gfs filesystem and other two (node3, node4) are working as loadballancers and as redundant lock servers (no gfs fs mounted on node3 or node4). (i am using GFS-6.0.2.20-2, GFS-modules-smp-6.0.2.20-2, kernel-smp-2.4.21-32.0.1.EL) So when all nodes are up there is: quorum_has = 4 quorum_needs = 3 I tried to stop lock_gulm on node3 and node4. Although the cluster was in state quorum_has = 2 quorum_needs = 3 the gfs filesystem on node1 or node2 still remained read/write accessible. Is this behaviour correct ? ---- nodes quorum_needs quorum_has filesystem 3 >=2 2 r/w 4 >=3 2 r/w ????? 5 >=3 3 r/w Can anybody help me out to correct or even extend the table above? Where is the truth ? :) Thanks a lot for your answers. -- J?n Kudj?k UNIX/Linux Consultant From saju8 at rediffmail.com Mon Mar 6 06:47:49 2006 From: saju8 at rediffmail.com (saju john) Date: 6 Mar 2006 06:47:49 -0000 Subject: [Linux-cluster] Cluster service restarting Locally Message-ID: <20060306064749.22036.qmail@webmail50.rediffmail.com> Dear All, I have a 2 node cluster with RHAS3 update 3. Kernel : 2.4.21-20.Elsmp Clumanager : clumanager-1.2.16-1 For more than a year everyting had been fine. Suddenly it started showing the follwing and restarted the service locally clusvcmgrd[1388]: Unable to obtain cluster lock: Connection timed out clulockd[1378]: Denied A.B.C.D: Broken pipe clulockd[1378]: select error: Broken pipe clusvcmgrd: [1625]: service notice: Stopping service postgresql ... clusvcmgrd: [1625]: service notice: Running user script '/etc/init.d/postgresql stop' clusvcmgrd: [1625]: service notice: Stopped service postgresql clusvcmgrd: [1625]: service notice: Starting service postgresql ... clusvcmgrd: [1625]: service notice: Running user script '/etc/init.d/postgresql start' clusvcmgrd: [1625]: service notice: Started service postgresql ... I saw the same problem already reported by Mr. Anu Matthew Is there any solution to this reported problem Thanks in advance Saju John -------------- next part -------------- An HTML attachment was scrubbed... URL: From kabobofpug at yahoo.com Mon Mar 6 06:22:13 2006 From: kabobofpug at yahoo.com (paul raymond) Date: Sun, 5 Mar 2006 22:22:13 -0800 (PST) Subject: [Linux-cluster] Trouble with RHCS 3.0 Message-ID: <20060306062213.85577.qmail@web36107.mail.mud.yahoo.com> Greetings Lon, I am trying to setup a simple two cluster system using RedHat Cluster Manager! The problem is that I can not get Quorum to start unless I run the command "cluforce"! But after viewing clustat commands on systems c11 and c12, it looks like c11 and c12 cant see each other status due to some issue with the raw partitions I believe? I am using a Mylex Fiber Channel Box with QLogic 2300 interface card! The raw devices are setup on a 2 mirror drives, Raid 1. Can you please shed any good ideas what might be wrong here? The vidals are below! Thanks! Paul R Linux System Admin c11***************************************** [root at c11 root] # clustat -i 5 Cluster Status - TESTVNSHA 21:42:16 Cluster Quorum Incarnation #2 Shared State: Shared Raw Device Driver v1.2 Member Status ------------------ ---------- c11 Active <-- You are here c12 Inactive Service Status Owner (Last) Last Transition Chk Restarts -------------- -------- ---------------- --------------- --- -------- vns started c12 19:25:12 Mar 05 2 0 [root at c11 root]# shutil -p /cluster/header /cluster/header is 144 bytes long SharedStateHeader { ss_magic = 0x39119fcd ss_timestamp = 0x000000004408c23c (14:25:00 Mar 03 2006) ss_updateHost = c11.wf.ibm.com } [root at c11 root]# fdisk -l Disk /dev/sda: 146.8 GB, 146814976000 bytes 255 heads, 63 sectors/track, 17849 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 16 128488+ 83 Linux /dev/sda2 17 526 4096575 83 Linux /dev/sda3 527 1036 4096575 83 Linux /dev/sda4 1037 17849 135050422+ f Win95 Ext'd (LBA) /dev/sda5 1037 1546 4096543+ 83 Linux /dev/sda6 1547 1801 2048256 82 Linux swap /dev/sda7 1802 1930 1036161 83 Linux /dev/sda8 1931 1993 506016 83 Linux Disk /dev/sdb: 72.7 GB, 72796340224 bytes 255 heads, 63 sectors/track, 8850 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 * 1 8850 71087593+ 83 Linux Disk /dev/sdc: 72.7 GB, 72796340224 bytes 255 heads, 63 sectors/track, 8850 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 * 1 5 40131 83 Linux /dev/sdc2 6 10 40162+ 83 Linux [root at c11 root]# raw -qa /dev/raw/raw1: bound to major 8, minor 33 /dev/raw/raw2: bound to major 8, minor 34 [root at c11 root]# rpm -qa |grep clu redhat-config-cluster-1.0.2-2.0 clumanager-1.2.22-2 [root at c11 root]# uname -r 2.4.21-32.0.1.ELsmp [root at c11 root]# lsmod Module Size Used by Not tainted soundcore 7012 0 (autoclean) ide-cd 34016 0 (autoclean) cdrom 32864 0 (autoclean) [ide-cd] iptable_filter 2412 0 (autoclean) (unused) ip_tables 16544 1 [iptable_filter] softdog 2972 1 lp 9124 0 (autoclean) parport 38816 0 (autoclean) [lp] autofs 13620 0 (autoclean) (unused) tg3 69768 2 floppy 57520 0 (autoclean) microcode 6848 0 (autoclean) keybdev 2976 0 (unused) mousedev 5624 1 hid 22500 0 (unused) input 6144 0 [keybdev mousedev hid] usb-ohci 23208 0 (unused) usbcore 81120 1 [hid usb-ohci] ext3 89928 7 jbd 55124 7 [ext3] qla2300 696284 5 mptscsih 42384 7 mptbase 42816 3 [mptscsih] diskdumplib 5228 0 [mptscsih mptbase] sd_mod 14096 24 scsi_mod 115368 3 [qla2300 mptscsih sd_mod] c12***************************************** [root at c12 root]# clustat -i 5 Cluster Status - TESTVNSHA 21:56:30 Cluster Quorum Incarnation #4 Shared State: Shared Raw Device Driver v1.2 Member Status ------------------ ---------- c11 Inactive c12 Active <-- You are here Service Status Owner (Last) Last Transition Chk Restarts -------------- -------- ---------------- --------------- --- -------- vns started c12 19:25:12 Mar 05 2 0 [root at c12 root]# shutil -p /cluster/header /cluster/header is 144 bytes long SharedStateHeader { ss_magic = 0x39119fcd ss_timestamp = 0x000000004408c23c (14:25:00 Mar 03 2006) ss_updateHost = c11.wf.ibm.com } [root at c12 root]# shutil -p /cluster/header /cluster/header is 144 bytes long SharedStateHeader { ss_magic = 0x39119fcd ss_timestamp = 0x000000004408c23c (14:25:00 Mar 03 2006) ss_updateHost = c11.wf.ibm.com } [root at c12 root]# fdisk -l Disk /dev/sda: 146.8 GB, 146814976000 bytes 255 heads, 63 sectors/track, 17849 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 16 128488+ 83 Linux /dev/sda2 17 526 4096575 83 Linux /dev/sda3 527 1036 4096575 83 Linux /dev/sda4 1037 17849 135050422+ f Win95 Ext'd (LBA) /dev/sda5 1037 1546 4096543+ 83 Linux /dev/sda6 1547 1801 2048256 82 Linux swap /dev/sda7 1802 1930 1036161 83 Linux /dev/sda8 1931 1993 506016 83 Linux Disk /dev/sdb: 72.7 GB, 72796340224 bytes 255 heads, 63 sectors/track, 8850 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 * 1 8850 71087593+ 83 Linux Disk /dev/sdc: 72.7 GB, 72796340224 bytes 255 heads, 63 sectors/track, 8850 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 * 1 5 40131 83 Linux /dev/sdc2 6 10 40162+ 83 Linux [root at c12 root]# raw -qa /dev/raw/raw1: bound to major 8, minor 33 /dev/raw/raw2: bound to major 8, minor 34 [root at c12 root]# rpm -qa |grep clu redhat-config-cluster-1.0.2-2.0 clumanager-1.2.22-2 [root at c12 root]# uname -r 2.4.21-32.0.1.ELsmp [root at c12 root]# lsmod Module Size Used by Not tainted soundcore 7012 0 (autoclean) ide-cd 34016 0 (autoclean) cdrom 32864 0 (autoclean) [ide-cd] iptable_filter 2412 0 (autoclean) (unused) ip_tables 16544 1 [iptable_filter] softdog 2972 1 lp 9124 0 (autoclean) parport 38816 0 (autoclean) [lp] autofs 13620 0 (autoclean) (unused) tg3 69768 2 floppy 57520 0 (autoclean) microcode 6848 0 (autoclean) keybdev 2976 0 (unused) mousedev 5624 1 hid 22500 0 (unused) input 6144 0 [keybdev mousedev hid] usb-ohci 23208 0 (unused) usbcore 81120 1 [hid usb-ohci] ext3 89928 7 jbd 55124 7 [ext3] qla2300 696284 5 mptscsih 42384 7 mptbase 42816 3 [mptscsih] diskdumplib 5228 0 [mptscsih mptbase] sd_mod 14096 24 scsi_mod 115368 3 [qla2300 mptscsih sd_mod] --------------------------------- Yahoo! Mail Use Photomail to share photos without annoying attachments. -------------- next part -------------- An HTML attachment was scrubbed... URL: From basv at sara.nl Mon Mar 6 19:59:26 2006 From: basv at sara.nl (Bas van der Vlies) Date: Mon, 6 Mar 2006 20:59:26 +0100 Subject: [Linux-cluster] (no subject) Message-ID: <05AF3B6E-B88A-45CC-83E4-1353291E6347@sara.nl> Out setup is: * We are using GFS from cvs stable branch on our 2.6.14.7 cluster. Just updated today to the newest CVS version. Only had to change the mutex() calls. * The 4 nodes are running debian sarge; * The 4 nodes act as NFS-servers for +/- 640 client-nodes * brocade switch with SGI TP9300 4 controllers (15 TB) We did a lot of testing an we could not crash the cluster, bonnie/ iozone and other tools/jobs. Now the cluster is in production we get a lot of nfsd crashed with EIP is at fda_create. We had it with our previous kernel 2.16.4.4 and with this one and "latest" CVS stable version. The server still runs ++ the load is high and it does not respond any more. If we are luckly only one NFS thread is gone and rest is still up. The rest of the nodes still work. Have users experienced this kind of problems and maybe have a solution for this problem? Regards, Here is a oops message: Unable to handle kernel NULL pointer dereference at virtual address 00000038 printing eip: f89bf999 *pde = 37bff001 *pte = 00000000 Oops: 0000 [#1] SMP Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg ide_floppy ide_cd cdrom qla2300 qla2xxx_conf qla2xxx firmware_class siimage piix e1000 gfs lock_harness dm_mod CPU: 0 EIP: 0060:[] Tainted: GF VLI EFLAGS: 00010246 (2.6.14.7-sara1) EIP is at gfs_create+0xa9/0x1e0 [gfs] eax: ffffffef ebx: ffffffef ecx: 00000001 edx: 00000000 esi: f296e24c edi: ebf01e18 ebp: ebf01e84 esp: ebf01df8 ds: 007b es: 007b ss: 0068 Process nfsd (pid: 16924, threadinfo=ebf00000 task=ebe84540) Stack: ebf01e48 f296e24c 00000001 00008180 ebf01e18 00000001 f8cb9000 dd042254 ebf01e18 ebf01e18 00000000 ebe84540 00000001 00000120 00000000 000000c2 00000000 00000001 ebf01e40 ebf01e40 ebf01e48 ebf01e48 df0bd858 ebe84540 Call Trace: [] show_stack+0x7f/0xa0 [] show_registers+0x162/0x1d0 [] die+0xf4/0x180 [] do_page_fault+0x2e7/0x6b2 [] error_code+0x4f/0x54 [] vfs_create+0x83/0xf0 [] nfsd_create_v3+0x40e/0x550 [] nfsd3_proc_create+0x11d/0x180 [] nfsd_dispatch+0xd7/0x200 [] svc_process+0x536/0x670 [] nfsd+0x1bd/0x350 [] kernel_thread_helper+0x5/0x18 Code: 24 08 8d 45 c4 89 54 24 0c 89 74 24 04 89 04 24 e8 1d c3 fe ff 85 c0 89 c3 0f 84 2e 01 00 00 83 f8 ef 0f 85 13 01 00 00 8b 55 14 <80> 7a 38 00 0f 88 06 01 00 00 89 7c 24 0c 31 c0 8d 55 c4 89 44 -- Bas van der Vlies basv at sara.nl From basv at sara.nl Mon Mar 6 20:10:12 2006 From: basv at sara.nl (Bas van der Vlies) Date: Mon, 6 Mar 2006 21:10:12 +0100 Subject: [Linux-cluster] gfs + nfsd crash In-Reply-To: <05AF3B6E-B88A-45CC-83E4-1353291E6347@sara.nl> References: <05AF3B6E-B88A-45CC-83E4-1353291E6347@sara.nl> Message-ID: Sorry no subject ;-( On Mar 6, 2006, at 8:59 PM, Bas van der Vlies wrote: > Out setup is: > * We are using GFS from cvs stable branch on our 2.6.14.7 > cluster. Just updated today to the > newest CVS version. Only had to change the mutex() calls. > * The 4 nodes are running debian sarge; > * The 4 nodes act as NFS-servers for +/- 640 client-nodes > * brocade switch with SGI TP9300 4 controllers (15 TB) > > We did a lot of testing an we could not crash the cluster, bonnie/ > iozone and other tools/jobs. Now the cluster is in production we > get a lot of nfsd crashed with EIP is at fda_create. We had it with > our previous kernel 2.16.4.4 and with this one and "latest" > CVS stable version. The server still runs ++ the load is high and > it does not respond any more. If we are luckly only one NFS > thread is gone and rest is still up. The rest of the nodes still work. > > Have users experienced this kind of problems and maybe have a > solution for this problem? > > > Regards, > > > Here is a oops message: > Unable to handle kernel NULL pointer dereference at virtual address > 00000038 > printing eip: > f89bf999 > *pde = 37bff001 > *pte = 00000000 > Oops: 0000 [#1] > SMP > Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg > ide_floppy ide_cd cdrom qla2300 qla2xxx_conf qla2xxx firmware_class > siimage piix e1000 gfs lock_harness dm_mod > CPU: 0 > EIP: 0060:[] Tainted: GF VLI > EFLAGS: 00010246 (2.6.14.7-sara1) > EIP is at gfs_create+0xa9/0x1e0 [gfs] > eax: ffffffef ebx: ffffffef ecx: 00000001 edx: 00000000 > esi: f296e24c edi: ebf01e18 ebp: ebf01e84 esp: ebf01df8 > ds: 007b es: 007b ss: 0068 > Process nfsd (pid: 16924, threadinfo=ebf00000 task=ebe84540) > Stack: ebf01e48 f296e24c 00000001 00008180 ebf01e18 00000001 > f8cb9000 dd042254 > ebf01e18 ebf01e18 00000000 ebe84540 00000001 00000120 > 00000000 000000c2 > 00000000 00000001 ebf01e40 ebf01e40 ebf01e48 ebf01e48 > df0bd858 ebe84540 > Call Trace: > [] show_stack+0x7f/0xa0 > [] show_registers+0x162/0x1d0 > [] die+0xf4/0x180 > [] do_page_fault+0x2e7/0x6b2 > [] error_code+0x4f/0x54 > [] vfs_create+0x83/0xf0 > [] nfsd_create_v3+0x40e/0x550 > [] nfsd3_proc_create+0x11d/0x180 > [] nfsd_dispatch+0xd7/0x200 > [] svc_process+0x536/0x670 > [] nfsd+0x1bd/0x350 > [] kernel_thread_helper+0x5/0x18 > Code: 24 08 8d 45 c4 89 54 24 0c 89 74 24 04 89 04 24 e8 1d c3 fe > ff 85 c0 89 c3 0f 84 2e 01 00 00 83 f8 ef 0f 85 13 01 00 00 8b 55 > 14 <80> 7a 38 00 0f 88 06 01 00 00 89 7c 24 0c 31 c0 8d 55 c4 89 44 > > > > > > > -- > Bas van der Vlies > basv at sara.nl > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Bas van der Vlies basv at sara.nl From hong.zheng at wsdtx.org Mon Mar 6 20:02:52 2006 From: hong.zheng at wsdtx.org (Hong Zheng) Date: Mon, 6 Mar 2006 14:02:52 -0600 Subject: [Linux-cluster] Cluster service restarting Locally Message-ID: I'm having the same problem. My system configuration is as follows: 2-node cluster: RH ES3, GFS6.0, clumanager-1.2.28-1 and redhat-config-cluster-1.0.8-1 Kernel: 2.4.21-37.EL Linux-iscsi-3.6.3 initiator: connections to iSCSI shared storage server I just noticed that whenever I have a heavy IO access this problem happens. Any suggestion I would really appreciate. Hong ________________________________ From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of saju john Sent: Monday, March 06, 2006 12:48 AM To: linux-cluster at redhat.com Subject: [Linux-cluster] Cluster service restarting Locally Dear All, I have a 2 node cluster with RHAS3 update 3. Kernel : 2.4.21-20.Elsmp Clumanager : clumanager-1.2.16-1 For more than a year everyting had been fine. Suddenly it started showing the follwing and restarted the service locally clusvcmgrd[1388]: Unable to obtain cluster lock: Connection timed out clulockd[1378]: Denied A.B.C.D: Broken pipe clulockd[1378]: select error: Broken pipe clusvcmgrd: [1625]: service notice: Stopping service postgresql ... clusvcmgrd: [1625]: service notice: Running user script '/etc/init.d/postgresql stop' clusvcmgrd: [1625]: service notice: Stopped service postgresql clusvcmgrd: [1625]: service notice: Starting service postgresql ... clusvcmgrd: [1625]: service notice: Running user script '/etc/init.d/postgresql start' clusvcmgrd: [1625]: service notice: Started service postgresql ... I saw the same problem already reported by Mr. Anu Matthew Is there any solution to this reported problem Thanks in advance Saju John -------------- next part -------------- An HTML attachment was scrubbed... URL: From jan.kudjak at snt.sk Tue Mar 7 09:15:50 2006 From: jan.kudjak at snt.sk (Kudjak Jan) Date: Tue, 7 Mar 2006 10:15:50 +0100 Subject: [Linux-cluster] 4 node gfs cluster, quorum needs 3 Message-ID: <139607FAAB7E0F46AD1A9BA381EC815A012C2668@KLEIO.snt.sk> Hello, I have at this time 4 node gfs cluster using RLM. Two nodes (node1, node2) have mounted gfs filesystem and other two (node3, node4) are working as loadballancers and as redundant lock servers (no gfs fs mounted on node3 or node4). (i am using GFS-6.0.2.20-2, GFS-modules-smp-6.0.2.20-2, kernel-smp-2.4.21-32.0.1.EL) So when all nodes are up there is: quorum_has = 4 quorum_needs = 3 I tried to stop lock_gulm on node3 and node4. Although the cluster was in state quorum_has = 2 quorum_needs = 3 the gfs filesystem on node1 or node2 still remained read/write accessible. Is this behaviour correct ? ---- nodes quorum_needs quorum_has filesystem 3 >=2 2 r/w 4 >=3 2 r/w ????? 5 >=3 3 r/w Can anybody help me out to correct or even extend the table above? Where is the truth ? :) or have I misunderstood something ? Thanks a lot for your answers. -- J?n Kudj?k UNIX/Linux Consultant From sebastien.didier at gmail.com Tue Mar 7 10:50:25 2006 From: sebastien.didier at gmail.com (=?ISO-8859-1?Q?S=E9bastien_DIDIER?=) Date: Tue, 7 Mar 2006 11:50:25 +0100 Subject: [Linux-cluster] Httpd Process io blocked Message-ID: Hi, I'm running a two-nodes GFS cluster which hosts web sites. The GFS partition is over a Iscsi device and by now, i'm using manual fencing. Today, I got 5 httpd process on both nodes which got stuck in IO blocking state. I suspected a GFS filesystem corruption but I haven't got any output from the kernel. I ran a fsck two days ago after a power chute. Here's the wait state of the process. (idem for the other node) # ps -o pid,tt,user,fname,wchan -C apache PID TT USER COMMAND WCHAN 4426 ? root apache - 14970 ? www-data apache glock_wait_internal 15103 ? www-data apache glock_wait_internal 16780 ? www-data apache glock_wait_internal 16959 ? www-data apache glock_wait_internal 14936 ? www-data apache finish_stop 12859 ? www-data apache - 13005 ? www-data apache - 13311 ? www-data apache semtimedop 13390 ? www-data apache semtimedop How can I debug further this problem ? And how can I bring back home my httpd processes without a reboot ? Many thanks for your help. Regards, S?bastien DIDIER From grimme at atix.de Tue Mar 7 11:12:13 2006 From: grimme at atix.de (Marc Grimme) Date: Tue, 7 Mar 2006 12:12:13 +0100 Subject: [Linux-cluster] Httpd Process io blocked In-Reply-To: References: Message-ID: <200603071212.13600.grimme@atix.de> Hi, to debug you could use strace. E.g. executing strace -p 14970 will probably show you that the process is waiting for a lock. As the ps already does. My first guess would be, that you use apache with php and sessions. If so, the phplib uses flocks for locking the session-ids. Normally it happens that one process locks a session. If another process comes along to get an flock on that session it has to wait until the further flock is closed. It very often happens that the other process gets that flock when the client and session are not available any more. Then the flock is held until the apache process timesout. We have made a patch for a better locking with php which you can find on http:/www.open-sharedroot.org in the downloads section. Hope that helps Regards Marc. On Tuesday 07 March 2006 11:50, S?bastien DIDIER wrote: > Hi, > > I'm running a two-nodes GFS cluster which hosts web sites. The GFS > partition is over a Iscsi device and by now, i'm using manual fencing. > > Today, I got 5 httpd process on both nodes which got stuck in IO > blocking state. I suspected a GFS filesystem corruption but I haven't > got any output from the kernel. I ran a fsck two days ago after a > power chute. > > Here's the wait state of the process. (idem for the other node) > > # ps -o pid,tt,user,fname,wchan -C apache > PID TT USER COMMAND WCHAN > 4426 ? root apache - > 14970 ? www-data apache glock_wait_internal > 15103 ? www-data apache glock_wait_internal > 16780 ? www-data apache glock_wait_internal > 16959 ? www-data apache glock_wait_internal > 14936 ? www-data apache finish_stop > 12859 ? www-data apache - > 13005 ? www-data apache - > 13311 ? www-data apache semtimedop > 13390 ? www-data apache semtimedop > > How can I debug further this problem ? And how can I bring back home > my httpd processes without a reboot ? > > Many thanks for your help. > > Regards, > S?bastien DIDIER > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Gruss / Regards, Marc Grimme Phone: +49-89 121 409-54 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From sebastien.didier at gmail.com Tue Mar 7 11:35:09 2006 From: sebastien.didier at gmail.com (=?ISO-8859-1?Q?S=E9bastien_DIDIER?=) Date: Tue, 7 Mar 2006 12:35:09 +0100 Subject: [Linux-cluster] Httpd Process io blocked In-Reply-To: <200603071212.13600.grimme@atix.de> References: <200603071212.13600.grimme@atix.de> Message-ID: 2006/3/7, Marc Grimme : > Hi, > to debug you could use strace. E.g. executing strace -p 14970 will probably > show you that the process is waiting for a lock. As the ps already does. My > first guess would be, that you use apache with php and sessions. Thanks. But strace doesnt output anything and became Ctrl-C imune. It needs a sigkill to exit and the traced process stays in T state. I seems that it doesnt manage to get last system call where the process is in D state. > > If so, the phplib uses flocks for locking the session-ids. Normally it happens > that one process locks a session. If another process comes along to get an > flock on that session it has to wait until the further flock is closed. It > very often happens that the other process gets that flock when the client and > session are not available any more. Then the flock is held until the apache > process timesout. > I don't think it is session related because I store sessions file outside the GFS mount point (/tmp) and I run a load balancer based upon the source adress (to always send requests to the same server and then keep sessions) But, we are using mysql query caching (with some libraries like AdoDb) inside the GFS mount point. Do you think it could be the cache files which are dead-locked ? > We have made a patch for a better locking with php which you can find on > http:/www.open-sharedroot.org in the downloads section. > Hope that helps > Regards Marc. > > On Tuesday 07 March 2006 11:50, S?bastien DIDIER wrote: > > Hi, > > > > I'm running a two-nodes GFS cluster which hosts web sites. The GFS > > partition is over a Iscsi device and by now, i'm using manual fencing. > > > > Today, I got 5 httpd process on both nodes which got stuck in IO > > blocking state. I suspected a GFS filesystem corruption but I haven't > > got any output from the kernel. I ran a fsck two days ago after a > > power chute. > > > > Here's the wait state of the process. (idem for the other node) > > > > # ps -o pid,tt,user,fname,wchan -C apache > > PID TT USER COMMAND WCHAN > > 4426 ? root apache - > > 14970 ? www-data apache glock_wait_internal > > 15103 ? www-data apache glock_wait_internal > > 16780 ? www-data apache glock_wait_internal > > 16959 ? www-data apache glock_wait_internal > > 14936 ? www-data apache finish_stop > > 12859 ? www-data apache - > > 13005 ? www-data apache - > > 13311 ? www-data apache semtimedop > > 13390 ? www-data apache semtimedop > > > > How can I debug further this problem ? And how can I bring back home > > my httpd processes without a reboot ? > > > > Many thanks for your help. > > > > Regards, > > S?bastien DIDIER > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Gruss / Regards, > > Marc Grimme > Phone: +49-89 121 409-54 > http://www.atix.de/ http://www.open-sharedroot.org/ > > ** > ATIX - Ges. fuer Informationstechnologie und Consulting mbH > Einsteinstr. 10 - 85716 Unterschleissheim - Germany > > From grimme at atix.de Tue Mar 7 12:43:20 2006 From: grimme at atix.de (Marc Grimme) Date: Tue, 7 Mar 2006 13:43:20 +0100 Subject: [Linux-cluster] Httpd Process io blocked In-Reply-To: References: <200603071212.13600.grimme@atix.de> Message-ID: <200603071343.21212.grimme@atix.de> Sebastien, On Tuesday 07 March 2006 12:35, S?bastien DIDIER wrote: > 2006/3/7, Marc Grimme : > > Hi, > > to debug you could use strace. E.g. executing strace -p 14970 will > > probably show you that the process is waiting for a lock. As the ps > > already does. My first guess would be, that you use apache with php and > > sessions. > > Thanks. But strace doesnt output anything and became Ctrl-C imune. It > needs a sigkill to exit and the traced process stays in T state. I > seems that it doesnt manage to get last system call where the process > is in D state. Hmm, sounds like I've heard that already. If you trace the root httpd with -f and -t and lookout for great timeslices you'll propably find processes waiting for locks. The D state is a good indicator (ps ax | grep " D " and look at the pids). Do the pids of the D processes change from time to time or do they stay the same pids? > > > If so, the phplib uses flocks for locking the session-ids. Normally it > > happens that one process locks a session. If another process comes along > > to get an flock on that session it has to wait until the further flock is > > closed. It very often happens that the other process gets that flock when > > the client and session are not available any more. Then the flock is held > > until the apache process timesout. > > I don't think it is session related because I store sessions file > outside the GFS mount point (/tmp) and I run a load balancer based > upon the source adress (to always send requests to the same server and > then keep sessions) Yes, I agree. Sessions get lost if the the node fails, right? > > But, we are using mysql query caching (with some libraries like AdoDb) > inside the GFS mount point. Do you think it could be the cache files > which are dead-locked ? It depends on how those files are locked and how and when the locks are set and released. If a lock is set at apache-child forktime and released at process terminate time, then yes that could happen. If only accesses to data of those files are protected with flocks then it should perform quite well. Is that query caching part of perl-adodb or is it implemented by yourselves? Have a look and play with strace and watch out for great times and the syscalls concerned with that. I would expect you ending up with flock-timeouts. Hope that helps, regards Marc. > > > We have made a patch for a better locking with php which you can find on > > http:/www.open-sharedroot.org in the downloads section. > > Hope that helps > > Regards Marc. > > > > On Tuesday 07 March 2006 11:50, S?bastien DIDIER wrote: > > > Hi, > > > > > > I'm running a two-nodes GFS cluster which hosts web sites. The GFS > > > partition is over a Iscsi device and by now, i'm using manual fencing. > > > > > > Today, I got 5 httpd process on both nodes which got stuck in IO > > > blocking state. I suspected a GFS filesystem corruption but I haven't > > > got any output from the kernel. I ran a fsck two days ago after a > > > power chute. > > > > > > Here's the wait state of the process. (idem for the other node) > > > > > > # ps -o pid,tt,user,fname,wchan -C apache > > > PID TT USER COMMAND WCHAN > > > 4426 ? root apache - > > > 14970 ? www-data apache glock_wait_internal > > > 15103 ? www-data apache glock_wait_internal > > > 16780 ? www-data apache glock_wait_internal > > > 16959 ? www-data apache glock_wait_internal > > > 14936 ? www-data apache finish_stop > > > 12859 ? www-data apache - > > > 13005 ? www-data apache - > > > 13311 ? www-data apache semtimedop > > > 13390 ? www-data apache semtimedop > > > > > > How can I debug further this problem ? And how can I bring back home > > > my httpd processes without a reboot ? > > > > > > Many thanks for your help. > > > > > > Regards, > > > S?bastien DIDIER > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Gruss / Regards, > > > > Marc Grimme > > Phone: +49-89 121 409-54 > > http://www.atix.de/ http://www.open-sharedroot.org/ > > > > ** > > ATIX - Ges. fuer Informationstechnologie und Consulting mbH > > Einsteinstr. 10 - 85716 Unterschleissheim - Germany > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Gruss / Regards, Marc Grimme Phone: +49-89 121 409-54 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From sebastien.didier at gmail.com Tue Mar 7 14:12:36 2006 From: sebastien.didier at gmail.com (=?ISO-8859-1?Q?S=E9bastien_DIDIER?=) Date: Tue, 7 Mar 2006 15:12:36 +0100 Subject: [Linux-cluster] Httpd Process io blocked In-Reply-To: <200603071343.21212.grimme@atix.de> References: <200603071212.13600.grimme@atix.de> <200603071343.21212.grimme@atix.de> Message-ID: 2006/3/7, Marc Grimme : > Sebastien, > On Tuesday 07 March 2006 12:35, S?bastien DIDIER wrote: > > 2006/3/7, Marc Grimme : > > > Hi, > > > to debug you could use strace. E.g. executing strace -p 14970 will > > > probably show you that the process is waiting for a lock. As the ps > > > already does. My first guess would be, that you use apache with php and > > > sessions. > > > > Thanks. But strace doesnt output anything and became Ctrl-C imune. It > > needs a sigkill to exit and the traced process stays in T state. I > > seems that it doesnt manage to get last system call where the process > > is in D state. > Hmm, sounds like I've heard that already. If you trace the root httpd with -f > and -t and lookout for great timeslices you'll propably find processes > waiting for locks. The D state is a good indicator (ps ax | grep " D " and > look at the pids). Do the pids of the D processes change from time to time or > do they stay the same pids? Marc, All the blocked processes have the same pid since the beginning of this issue. (22 hours by now) > > > > > If so, the phplib uses flocks for locking the session-ids. Normally it > > > happens that one process locks a session. If another process comes along > > > to get an flock on that session it has to wait until the further flock is > > > closed. It very often happens that the other process gets that flock when > > > the client and session are not available any more. Then the flock is held > > > until the apache process timesout. > > > > I don't think it is session related because I store sessions file > > outside the GFS mount point (/tmp) and I run a load balancer based > > upon the source adress (to always send requests to the same server and > > then keep sessions) > Yes, I agree. Sessions get lost if the the node fails, right? Yes. That may be a problem for some apps... But it is easier (and more efficient) than storing session data into SQL. > > > > But, we are using mysql query caching (with some libraries like AdoDb) > > inside the GFS mount point. Do you think it could be the cache files > > which are dead-locked ? > It depends on how those files are locked and how and when the locks are set > and released. If a lock is set at apache-child forktime and released at > process terminate time, then yes that could happen. If only accesses to data > of those files are protected with flocks then it should perform quite well. > > Is that query caching part of perl-adodb or is it implemented by yourselves? It appears that we are using a very common PHP AdoDB abstact class without any change in the code. When I run a "lsof -p" on each blocked process on the two nodes, each one has exactly the same file open : apache 23327 www-data 10r REG 253,0 2128 5053927 /home/sites/website/web/queryCache/ca/adodb_cad1702c2e5d18a71d765e95bf55ea3b.cache (deleted) > > Have a look and play with strace and watch out for great times and the > syscalls concerned with that. I would expect you ending up with > flock-timeouts. > > Hope that helps, > regards Marc. > > > > > We have made a patch for a better locking with php which you can find on > > > http:/www.open-sharedroot.org in the downloads section. > > > Hope that helps > > > Regards Marc. > > > > > > On Tuesday 07 March 2006 11:50, S?bastien DIDIER wrote: > > > > Hi, > > > > > > > > I'm running a two-nodes GFS cluster which hosts web sites. The GFS > > > > partition is over a Iscsi device and by now, i'm using manual fencing. > > > > > > > > Today, I got 5 httpd process on both nodes which got stuck in IO > > > > blocking state. I suspected a GFS filesystem corruption but I haven't > > > > got any output from the kernel. I ran a fsck two days ago after a > > > > power chute. > > > > > > > > Here's the wait state of the process. (idem for the other node) > > > > > > > > # ps -o pid,tt,user,fname,wchan -C apache > > > > PID TT USER COMMAND WCHAN > > > > 4426 ? root apache - > > > > 14970 ? www-data apache glock_wait_internal > > > > 15103 ? www-data apache glock_wait_internal > > > > 16780 ? www-data apache glock_wait_internal > > > > 16959 ? www-data apache glock_wait_internal > > > > 14936 ? www-data apache finish_stop > > > > 12859 ? www-data apache - > > > > 13005 ? www-data apache - > > > > 13311 ? www-data apache semtimedop > > > > 13390 ? www-data apache semtimedop > > > > > > > > How can I debug further this problem ? And how can I bring back home > > > > my httpd processes without a reboot ? > > > > > > > > Many thanks for your help. > > > > > > > > Regards, > > > > S?bastien DIDIER > > > > > > > > -- > > > > Linux-cluster mailing list > > > > Linux-cluster at redhat.com > > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > -- > > > Gruss / Regards, > > > > > > Marc Grimme > > > Phone: +49-89 121 409-54 > > > http://www.atix.de/ http://www.open-sharedroot.org/ > > > > > > ** > > > ATIX - Ges. fuer Informationstechnologie und Consulting mbH > > > Einsteinstr. 10 - 85716 Unterschleissheim - Germany > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Gruss / Regards, > > Marc Grimme > Phone: +49-89 121 409-54 > http://www.atix.de/ http://www.open-sharedroot.org/ > > ** > ATIX - Ges. fuer Informationstechnologie und Consulting mbH > Einsteinstr. 10 - 85716 Unterschleissheim - Germany > > From cjk at techma.com Tue Mar 7 15:52:57 2006 From: cjk at techma.com (Kovacs, Corey J.) Date: Tue, 7 Mar 2006 10:52:57 -0500 Subject: [Linux-cluster] Trouble with RHCS 3.0 Message-ID: Do your host files have entries for the node names that are NOT loopback? The default for RedHat is to have the nodename in the loopback line. Correct that if it is the case and you might get better results. Cheers Corey ________________________________ From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of paul raymond Sent: Monday, March 06, 2006 1:22 AM To: lhh at redhat.com; linux-cluster at redhat.com Subject: [Linux-cluster] Trouble with RHCS 3.0 Greetings Lon, I am trying to setup a simple two cluster system using RedHat Cluster Manager! The problem is that I can not get Quorum to start unless I run the command "cluforce"! But after viewing clustat commands on systems c11 and c12, it looks like c11 and c12 cant see each other status due to some issue with the raw partitions I believe? I am using a Mylex Fiber Channel Box with QLogic 2300 interface card! The raw devices are setup on a 2 mirror drives, Raid 1. Can you please shed any good ideas what might be wrong here? The vidals are below! Thanks! Paul R Linux System Admin c11***************************************** [root at c11 root] # clustat -i 5 Cluster Status - TESTVNSHA 21:42:16 Cluster Quorum Incarnation #2 Shared State: Shared Raw Device Driver v1.2 Member Status ------------------ ---------- c11 Active <-- You are here c12 Inactive Service Status Owner (Last) Last Transition Chk Restarts -------------- -------- ---------! ------- --------------- --- -------- vns started c12 19:25:12 Mar 05 2 0 [root at c11 root]# shutil -p /cluster/header /cluster/header is 144 bytes long SharedStateHeader { ss_magic = 0x39119fcd ss_timestamp = 0x000000004408c23c (14:25:00 Mar 03 2006) ss_updateHost = c11.wf.ibm.com } [root at c11 root]# fdisk -l Disk /dev/sda: 146.8 GB, 146814976000 bytes 255 heads, 63 sectors/track, 17849 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks ! Id System /dev/sda1 * 1 16 128488+ 83 Linux /dev/sda2 17 526 4096575 83 Linux /dev/sda3 527 1036 4096575 83 Linux /dev/sda4 1037 17849 135050422+ f Win95 Ext'd (LBA) /dev/sda5 1037 1546 4096543+ 83 Linux /dev/sda6 1547 1801 2048256 82 Linux swap /dev/sda7 1802 1930 1036161 83 Linux /dev/sda8 1931 1993 506016 83 Linux Disk /dev/sdb: 72.7 GB, 72796340224 bytes 255 heads, 63 sectors/track, 8850 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 * 1 8850 71087593+ 83 Linux Disk /dev/sdc: 72.7 GB, 72796340224 bytes 255 heads, 63 sectors/track, 8850 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot ! ; Start End Blocks Id System /dev/sdc1 * 1 5 40131 83 Linux /dev/sdc2 6 10 40162+ 83 Linux [root at c11 root]# raw -qa /dev/raw/raw1: bound to major 8, minor 33 /dev/raw/raw2: bound to major 8, minor 34 [root at c11 root]# rpm -qa |grep clu redhat-config-cluster-1.0.2-2.0 clumanager-1.2.22-2 [root at c11 root]# uname -r 2.4.21-32.0.1.ELsmp [root at c11 root]# lsmod Module Size Used by Not tainted soundcore 7012 0 (autoclean) ide-cd 34016 0 (autoclean) cdrom 32864 0 (autoclean) [ide-cd] iptable_filter 2412 0 (autoclean) (unused) ip_tables 16544 1 [iptable_filter] softdog 2972 1 lp 9124 0 (autoclean) parport 38816 0 (autoclean) [lp] autofs 13620 0 (autoclean) (unused) tg3 69768 2 floppy 57520 0 (autoclean) microcode 6848 0 (autoclean) keybdev 2976 0 (unused) mousedev &nbs! p; 5624 1 hid 22500 0 (unused) input 6144 0 [keybdev mousedev hid] usb-ohci 23208 0 (unused) usbcore 81120 1 [hid usb-ohci] ext3 89928 7 jbd 55124 7 [ext3] qla2300 696284 5 mptscsih 42384 7 mptbase 42816 3 [mptscsih] diskdumplib 5228 0 [mptscsih mptbase] sd_mod 14096 24 scsi_mod 115368 3 [qla2300 mptscsih sd_mod] c12***************************************** [root at c12 root]# clustat -i 5 Cluster Status - TESTVNSHA 21:56:30 Cluster Quorum Incarnation #4 Shared State: Shared Raw Device Driver v1.2 Member Status ------------------ ---------- c11 Inactive c12 Active <-- You are here Service Status Owner (Last) Last Transition Chk Restarts -------------- -------- ---------! ------- --------------- --- -------- vns started c12 19:25:12 Mar 05 2 0 [root at c12 root]# shutil -p /cluster/header /cluster/header is 144 bytes long SharedStateHeader { ss_magic = 0x39119fcd ss_timestamp = 0x000000004408c23c (14:25:00 Mar 03 2006) ss_updateHost = c11.wf.ibm.com } [root at c12 root]# shutil -p /cluster/header /cluster/header is 144 bytes long SharedStateHeader { ss_magic = 0x39119fcd ss_timestamp = 0x000000004408c23c (14:25:00 Mar 03 2006) ss_updateHost = c11.wf.ibm.com } [root at c12 root]# fdisk -l Disk /dev/sda: 146.8 GB, 146814976000 bytes 255 heads, 63 sectors/track, 17849 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 16 128488+ 83 Linux /dev/sda2 17 526 4096575 83 Linux /dev/sda3 527 1036 4096575 83 Linux /dev/sda4 1037 17849 135050422+ f Win95 Ext'd (LBA) /dev/sda5 1037 1546 4096543+ 83 Linux /dev/sda6 1547 1801 2048256 82 Linux swap /dev/sda7 1802 1930 1036161 83 Linux /dev/sda8 1931 1993 506016 83 Linux Disk /dev/sdb: 72.7 GB, 72796340224 bytes 255 heads, 63 sectors/track, 8850 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot &nbs! p; Start End Blocks Id System /dev/sdb1 * 1 8850 71087593+ 83 Linux Disk /dev/sdc: 72.7 GB, 72796340224 bytes 255 heads, 63 sectors/track, 8850 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 * 1 5 40131 83 Linux /dev/sdc2 6 10 40162+ 83 Linux [root at c12 root]# raw -qa /dev/raw/raw1: bound to major 8, minor 33 /dev/raw/raw2: bound to major 8, minor 34 [root at c12 root]# rpm -qa |grep clu redhat-config-cluster-1.0.2-2.0 clumanager-1.2.22-2 [root at c12 root]# uname -r 2.4.21-32.0.1.ELsmp [root at c12 root]# lsmod Module Size Used by Not tainted soundcore 7012 0 (autoclean) ide-cd 34016 0 (autoclean) cdrom 32864 0 (autoclean) [ide-cd] iptable_filter &! nbsp; 2412 0 (autoclean) (unused) ip_tables 16544 1 [iptable_filter] softdog 2972 1 lp 9124 0 (autoclean) parport 38816 0 (autoclean) [lp] autofs 13620 0 (autoclean) (unused) tg3 69768 2 floppy 57520 0 (autoclean) microcode 6848 0 (autoclean) keybdev 2976 0 (unused) mousedev 5624 1 hid 22500 0 (unused) input 6144 0 [keybdev mousedev hid] usb-ohci 23208 ! 0 (unused) usbcore 81120 1 [hid usb-ohci] ext3 89928 7 jbd 55124 7 [ext3] qla2300 696284 5 mptscsih 42384 7 mptbase 42816 3 [mptscsih] diskdumplib 5228 0 [mptscsih mptbase] sd_mod 14096 24 scsi_mod 115368 3 [qla2300 mptscsih sd_mod] ________________________________ Yahoo! Mail Use Photomail to share photos without annoying attachments. -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.catanese at kinetikon.com Tue Mar 7 16:04:06 2006 From: m.catanese at kinetikon.com (Matteo Catanese) Date: Tue, 7 Mar 2006 17:04:06 +0100 Subject: [Linux-cluster] More CS4 fencing fun Message-ID: <590F0181-7330-408F-B693-E182780DE8A7@kinetikon.com> Hi, im doing failover tests on a CS4 cluster. I have 2 HP dl380 + HP msa1000 (aka dl380 packaged cluster). I already read this post https://www.redhat.com/archives/linux-cluster/2006-January/msg00195.html Im clustering a single oracle instance using active/passive. I don't use GFS. I use fence_ilo I have a fully working clustered oracle, i tried to migrate oracle instance from a node to another using system-config-cluster and everything works perfectly. I tried some more "rude" failover tests with this setup: node1 = active node node2 = passive node and those are the results: Situation 1: I rudely disconnect the powercable(s) from node1, so that node1 is _completely_ turned off, no current flows in it. ILO is down. I have redundant powerunits but i wanted to simulate short circuit or motherboard failure Node2, using fence, tries to poweroff node1 Fence_ilo tries to connect to node1_ilo_ip_address, but ilo is down because of power failure so fencing fails and starts retrying forever. Result: One node perfectly up but cluster service stalled Situation2: I push the on/off button on node1. It stops in 4 seconds, but power is still on, so ILO is up and working. node2, using fence, tries to poweroff the node1. ilo is working so fence_ilo correctly connects to node1_ilo_ip_address, it tries for some time to poweroff the already poweroff'd server, then it finally decides that server is off. Oracle is STILL down, no virtual ip, no storage mounted bla bla bla Now node2 tries to wake up the turned_off_but_still_powered_ node1. Node1 wakes up, then it does bootstrap (cluster is still stalled) then joins fence_domain. Fence on node2 completes succesfully and unlocks cluster and everything is up again Switch time: 55 seconds (+ oracle startup time). Situation 3: This is not a real failover test. Everything is off. I turn on the msa1000 and wait for its bootstrap. Then i turn on node1 but i still have node2 electrically disconnected. Node1 tries to turn on node2 to complete the fence_domain, node2 is disconnected from power current so it will never wake up. Cluster is stalled Can you change fence behaviour to be less "radical" ? If ILO is unreachable means that machine is already off and could not be powered on so fence shold spit out a warning and let the failover happen If ILO is reachable then check its status to avoid pointless poweroff/ poweron As of today fence is really dangerous in a production environment, for now i will turn it off Matteo From alfeijoo at cesga.es Tue Mar 7 18:40:48 2006 From: alfeijoo at cesga.es (Alejandro Feijoo) Date: Tue, 7 Mar 2006 19:40:48 +0100 (CET) Subject: [Linux-cluster] cman for CS Message-ID: <46998.193.144.44.59.1141756848.squirrel@webmail.cesga.es> hi i have a linux kernel version 2.6.9.22.0.2 (the lastest!) buttt the cman for dowload is cman-kernel-2.6.9-39.8.src.rpm.... there are any problem if i install that cman ????? and where is rpm for kernel 2.6.9-39.8??? Tanks! ++-------------------------++ Alejandro Feij?o Fraga Tecnico de Sistemas. Centro de supercomputaci?n de Galicia Avda. de Vigo s/n. Campus Sur. 15705 - Santiago de Compostela. Spain Tlfn.: 981 56 98 10 Extension: 216 Fax: 981 59 46 16 From lhh at redhat.com Tue Mar 7 19:12:39 2006 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 07 Mar 2006 14:12:39 -0500 Subject: [Linux-cluster] More CS4 fencing fun In-Reply-To: <590F0181-7330-408F-B693-E182780DE8A7@kinetikon.com> References: <590F0181-7330-408F-B693-E182780DE8A7@kinetikon.com> Message-ID: <1141758759.25169.120.camel@ayanami.boston.redhat.com> On Tue, 2006-03-07 at 17:04 +0100, Matteo Catanese wrote: > Result: One node perfectly up but cluster service stalled Fencing never completes because iLO does not have power. This an architectural limitation to using iLO (or IPMI, actually) in a cluster environment as the sole fencing method. Compare to RSA - which can have its own external power supply - even though it is an integrated solution like iLO. With redundant power supplies, the expectation is that different circuits (or preferably - different power sources entirely) are used, which should make the tested case significantly less likely to occur. > Switch time: 55 seconds (+ oracle startup time). Hrm, the backup node should take over the service after the primary node is confirmed 'dead', i.e. after fencing is complete. It should certainly not be waiting around for the other node to come back to life. What does your fence + service configuration look like, and were there any obvious log messages which might explain the odd behavior? > Cluster is stalled > > Can you change fence behaviour to be less "radical" ? > > If ILO is unreachable means that machine is already off and could not > be powered on so fence shold spit out a warning and let the failover > happen iLO being unreachable means iLO is unreachable, and assumptions as to why should probably not be limited to lack of power. Routing problems, bad network cable, disconnected cable, and the occasional infinite iLO-DHCP loop will all make iLO unreachable, but in no way confirm that the node is dead. More to the point, though, you can get around this particular behavior (fencing on startup -> hang because fencing fails) by starting fenced with the clean start parameter. In a two node cluster, this is useful to start things up in a controlled way when you know you won't be able to fence the other node. I think it's: fence_tool join -c If you (the administrator) are sure that the node is dead and does not have any services running, it will cause fenced to not fence the other node on startup, thereby avoiding the hang entirely. However, automatically doing this is unsafe if both nodes are booting while a network partition exists between the nodes, the cluster will end up with a split brain. -- Lon From milis at ogs-id.com Wed Mar 8 04:13:57 2006 From: milis at ogs-id.com (Milis) Date: Wed, 8 Mar 2006 11:13:57 +0700 Subject: [Linux-cluster] Clustering RHEL 4 with EXP400 Message-ID: <751353696.20060308111357@sur.ogs-id.com> Dear All, does any one here have experince to cluster RHEL 4 with 2 IBM x346 and 1 EXP 400, I need to do this for build Oracle10g On Rac, whether I need driver for build and startup of this device. may I know what I need for this requirement? and what should I do to get driver of IBM EXP400 ? I've been success install IBM X346 with Raid 1 option on RHEL 4, but I really confuse what next to do to show up cluster on EXP400 (whether I have no driver for this device to share storage) Thanks for your share knowledge. -- Tks & Best regards, Andi EP IT Engineer mailto:milis at ogs-id.com From francisco_javier.pena at roche.com Wed Mar 8 08:49:27 2006 From: francisco_javier.pena at roche.com (Pena, Francisco Javier) Date: Wed, 8 Mar 2006 09:49:27 +0100 Subject: [Linux-cluster] cman for CS Message-ID: Hi Alejandro, The cman version after the dash sign does not have to be the same as the kernel version. Just do a "rpm -qp --requires cman-kernel-2.6.9-39.8.src.rpm", and it should tell you which kernel version is required. Cheers, Javier > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of > Alejandro Feijoo > Sent: Tuesday, March 07, 2006 7:41 PM > To: linux-cluster at redhat.com > Subject: [Linux-cluster] cman for CS > > > > > hi i have a linux kernel version 2.6.9.22.0.2 (the lastest!) > buttt the cman for dowload is cman-kernel-2.6.9-39.8.src.rpm.... > > there are any problem if i install that cman ????? and where > is rpm for kernel 2.6.9-39.8??? > > > Tanks! > > ++-------------------------++ > Alejandro Feij?o Fraga > Tecnico de Sistemas. > Centro de supercomputaci?n de Galicia > Avda. de Vigo s/n. Campus Sur. > 15705 - Santiago de Compostela. Spain > Tlfn.: 981 56 98 10 Extension: 216 > Fax: 981 59 46 16 > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From mykleb at no.ibm.com Wed Mar 8 09:11:57 2006 From: mykleb at no.ibm.com (Jan-Frode Myklebust) Date: Wed, 8 Mar 2006 10:11:57 +0100 Subject: [Linux-cluster] Re: Clustering RHEL 4 with EXP400 References: <751353696.20060308111357@sur.ogs-id.com> Message-ID: On 2006-03-08, Milis wrote: > > does any one here have experince to cluster RHEL 4 with 2 IBM x346 > and 1 EXP 400, I've clustered RHEL3 with 2x Dell PowerEdge 2650/ServeRAID 6M and 1 EXP400. This was an active/passive cluster, using the ServeRAID's hardware fencing to prevent more than one node from accessing the volumes on the EXP400. x346+RHEL4 shouldn't be any difference. Now, I'm not much familiar with Oracle10g On Rac, but is it really an active/passive solution you want there? AFAIK you can't have both nodes active against the volumes on the EXP400 at the same time. > I've been success install IBM X346 with Raid 1 option on RHEL 4, but I > really confuse what next to do to show up cluster on EXP400 (whether I > have no driver for this device to share storage) -jf From carlopmart at gmail.com Wed Mar 8 12:35:17 2006 From: carlopmart at gmail.com (carlopmart) Date: Wed, 08 Mar 2006 13:35:17 +0100 Subject: [Linux-cluster] Postfix under cluster suite Message-ID: <440ECF85.2050807@gmail.com> Hi all, Somebody have tried to setup a postfix cluster service under RHCS 4? Is it possible to mantain two postfix instances (one for node and another to the cluster)? Thanks. -- CL Martinez carlopmart {at} gmail {d0t} com From basv at sara.nl Wed Mar 8 15:35:25 2006 From: basv at sara.nl (Bas van der Vlies) Date: Wed, 08 Mar 2006 16:35:25 +0100 Subject: [Linux-cluster] gfs + nfsd crash In-Reply-To: References: <05AF3B6E-B88A-45CC-83E4-1353291E6347@sara.nl> Message-ID: <440EF9BD.4030108@sara.nl> We just upgraded to 2.6.16-rc5 and cvs stable gfs. We still have gfs_create crashes. === Ooops ===== Unable to handle kernel NULL pointer dereference at virtual address 00000038 printing eip: f89a4be3 *pde = 37809001 *pte = 00000000 Oops: 0000 [#1] SMP Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness dm_mod CPU: 0 EIP: 0060:[] Tainted: GF VLI EFLAGS: 00010246 (2.6.16-rc5-sara3 #1) EIP is at gfs_create+0x6f/0x153 [gfs] eax: 00000000 ebx: ffffffef ecx: f27d0d98 edx: ffffffef esi: f2f84690 edi: f8b93000 ebp: f34a5e98 esp: f34a5e20 ds: 007b es: 007b ss: 0068 Process nfsd (pid: 8973, threadinfo=f34a4000 task=f3462a70) Stack: <0>f092a530 00000001 f34a5e48 00000000 f34a5e84 f89a6628 f34a5e48 ee1fc324 00000003 00000000 f34a5e48 f34a5e48 00000000 f3462a70 00000003 f34a5e5c f34a5e5c f27d0d98 f3462a70 00000001 00000020 00000000 000000c2 00000000 Call Trace: [] show_stack_log_lvl+0xad/0xb5 [] show_registers+0x10d/0x176 [] die+0xf2/0x16d [] do_page_fault+0x3dd/0x57a [] error_code+0x4f/0x54 [] vfs_create+0x6a/0xa7 [] nfsd_create_v3+0x2b1/0x48a [] nfsd3_proc_create+0x116/0x123 [] nfsd_dispatch+0xbe/0x17f [] svc_process+0x381/0x5c7 [] nfsd+0x18d/0x2e2 [] kernel_thread_helper+0x5/0xb Code: 94 50 8b 45 0c ff 75 10 83 c0 1c 6a 01 89 45 88 50 8d 45 c4 50 e8 70 08 ff ff 83 c4 14 89 c3 85 c0 74 4883 f8 ef 75 33 8b 45 14 <80> 78 38 00 78 2a 8d 45 94 50 8d 45 c4 6a 00 ff 75 88 50 e8 3c BUG: nfsd/8973, lock held at task exit time! [ee1fc398] {inode_init_once} .. held by: nfsd: 8973 [f3462a70, 115] ... acquired at: nfsd_create_v3+0x127/0x48a -- -- ******************************************************************** * * * Bas van der Vlies e-mail: basv at sara.nl * * SARA - Academic Computing Services phone: +31 20 592 8012 * * Kruislaan 415 fax: +31 20 6683167 * * 1098 SJ Amsterdam * * * ******************************************************************** From lhh at redhat.com Wed Mar 8 15:48:30 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 08 Mar 2006 10:48:30 -0500 Subject: [Linux-cluster] Postfix under cluster suite In-Reply-To: <440ECF85.2050807@gmail.com> References: <440ECF85.2050807@gmail.com> Message-ID: <1141832910.25169.151.camel@ayanami.boston.redhat.com> On Wed, 2006-03-08 at 13:35 +0100, carlopmart wrote: > Hi all, > > Somebody have tried to setup a postfix cluster service under RHCS 4? > Is it possible to mantain two postfix instances (one for node and > another to the cluster)? I have not tried it, and I am not familiar with configuring Postfix in the least, but here are some hints and gotchas which might exist ;) * Both instances will want to (by default) bind to INADDR_ANY. The two instances must be binding to specific IP addresses - i.e., one instance must bind to the host's IP, the other to the cluster floating IP - in order for two instances to start in the first place. * Both instances will want to deliver stuff in /var/mail ... and email readers want to read from /var/mail. Maybe the "RightThing(tm)" to do is something weird like the following. This is a complete shot in the dark... ;) - Cluster-mounted /var/mail (either GFS or not, it shouldn't matter) - Node-specific Postfix instances never deliver mail directly, but rather, they both forward to the cluster-instance Postfix IP. - All postfix instances may accept mail for sending off-site. This way, you don't have two instances of postfix both trying to manage the contents of /var/mail (do they play nicely together?), all instances of postfix can send mail, but only one does the ultimate receiving of mail. Also, if you use GFS for /var/mail, you may be able to run imapd on multiple cluster nodes, but I've never tried this either. As long as multiple people aren't accessing the same imap mailbox, I am guessing it would "just work" (famous last words, I know ;) ). -- Lon From gstaltari at arnet.net.ar Wed Mar 8 18:05:27 2006 From: gstaltari at arnet.net.ar (German Staltari) Date: Wed, 08 Mar 2006 15:05:27 -0300 Subject: [Linux-cluster] missing services Message-ID: <440F1CE7.2010104@arnet.net.ar> Hi, we have a 6 node cluster, each one mounts 6 GFS partitions. When I ask for the services to cman, there is always a mount point missing. Is this correct? FC 4 kernel-smp-2.6.15-1.1831_FC4 dlm-kernel-smp-2.6.11.5-20050601.152643.FC4.21 GFS-kernel-smp-2.6.11.8-20050601.152643.FC4.24 cman-kernel-smp-2.6.11.5-20050601.152643.FC4.22 TIA German Staltari # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 59G 2.4G 54G 5% / /dev/shm 2.0G 0 2.0G 0% /dev/shm /dev/mapper/vg1-store1 399G 184K 399G 1% /store/1 /dev/mapper/vg2-store2 399G 2.8M 399G 1% /store/2 /dev/mapper/vg3-store3 399G 180K 399G 1% /store/3 /dev/mapper/vg4-store4 399G 180K 399G 1% /store/4 /dev/mapper/vg5-store5 399G 180K 399G 1% /store/5 /dev/mapper/vg6-store6 399G 180K 399G 1% /store/6 # cman_tool services Service Name GID LID State Code Fence Domain: "default" 1 2 run - [1 3] DLM Lock Space: "clvmd" 7 3 run - [1 4 3] DLM Lock Space: "mailstore01" 20 4 run - [1 3] DLM Lock Space: "mailstore02" 22 6 run - [1 3] DLM Lock Space: "mailstore03" 24 8 run - [1 3] DLM Lock Space: "mailstore04" 26 10 run - [1 3] DLM Lock Space: "mailstore05" 28 12 run - [1 3] DLM Lock Space: "mailstore06" 30 14 run - [1 3] GFS Mount Group: "mailstore01" 21 5 run - [1 3] GFS Mount Group: "mailstore02" 23 7 run - [1 3] GFS Mount Group: "mailstore03" 25 9 run - [1 3] GFS Mount Group: "mailstore04" 27 11 run - [1 3] GFS Mount Group: "mailstore05" 29 13 run - [1 3] From bobby.m.dalton at nasa.gov Wed Mar 8 18:43:07 2006 From: bobby.m.dalton at nasa.gov (Dalton, Maurice) Date: Wed, 8 Mar 2006 12:43:07 -0600 Subject: [Linux-cluster] RHEL4.0 CS and Ldap Message-ID: Is there a way to create an Ldap cluster that can do replication with RHEL 4.0 CS? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jon.Stanley at savvis.net Wed Mar 8 18:54:22 2006 From: Jon.Stanley at savvis.net (Stanley, Jon) Date: Wed, 8 Mar 2006 12:54:22 -0600 Subject: [Linux-cluster] GFS load average and locking Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B90143604F@s228130hz1ew08.apptix-01.savvis.net> I have a 7 node GFS cluster, plus 3 lock servers (RH AS3U5, GULM locking) that do not mount the filesystem. I have a problem whereby the load average on the system is extremely high (occasionally astronomical), eventually leading to a complete site outage, via inability to access the shared filesystem. I have a couple questions about the innards of GFS that I would be most grateful for someone to answer: The application is written in PHP, and the PHP sessioning is handled via the GFS filesystem as well, if that's important. 1) I notice that I have a lot of processes in uninterruptible sleep. When I attached strace to one of these processes, I obviously found it doing nothing for a period of ~30-60 seconds. An excerpt of the strace (using -r) follows: 0.001224 stat64("/media/files/global/2/6/26c4f61c69117d55b352ce328babbff4.jpg", {st_mode=S_IFREG|0644, st_size=9072, ...}) = 0 0.000251 open("/media/files/global/2/6/26c4f61c69117d55b352ce328babbff4.jpg", O_RDONLY) = 5 0.000108 mmap2(NULL, 9072, PROT_READ, MAP_PRIVATE, 5, 0) = 0xaf381000 0.000069 writev(4, [{"HTTP/1.1 200 OK\r\nDate: Wed, 08 M"..., 318}, {"\377\330\377\340\0\20JFIF\0\1\2\0\0d\0d\0\0\377\354\0\21"..., 9072}], 2) = 9390 0.000630 close(5) = 0 0.000049 munmap(0xaf381000, 9072) = 0 0.000052 rt_sigaction(SIGUSR1, {0x81ef474, [], SA_RESTORER|SA_INTERRUPT, 0x1b2eb8}, {SIG_IGN}, 8) = 0 0.000068 read(4, 0xa239b3c, 4096) = ? ERESTARTSYS (To be restarted) 6.546891 --- SIGALRM (Alarm clock) @ 0 (0) --- 0.000119 close(4) = 0 What it looks like is it hangs out in read() for a period of time, thus leading to the uninterruptible sleep. This particular example was 6 seconds, however it seems that the time is variable. The particular file in this instance is not large, only 9k. I've never seen ERESTARTSYS before, and some googling tells me that it's basically telling the kernel to interrupt the current syscall in order to handle a signal (SIGALRM in this case, which I'm not sure the function of). I could be *way* off base here - I'm not a programmer by any stretch of the imagination. 2) The locking statistics seems to be a huge mystery. The lock total doesn't seem to correspond to the number of open files that I have (I hope!). Here's the output of a 'cat /proc/gulm/lockspace - I can't imagine that I have 300,000+ files open on this system at this point - when are the locks released, or is this even an indication of how many locks that are active at the current time? What does the 'pending' number mean? [svadmin at s259830hz1sl01 gulm]$ cat lockspace lock counts: total: 369822 unl: 176518 exl: 1555 shd: 191501 dfr: 0 pending: 5 lvbs: 2000 lops: 21467433 [svadmin at s259830hz1sl01 gulm]$ Thanks for any help that anyone can provide on this! Thanks! -Jon From cjk at techma.com Wed Mar 8 19:06:10 2006 From: cjk at techma.com (Kovacs, Corey J.) Date: Wed, 8 Mar 2006 14:06:10 -0500 Subject: [Linux-cluster] GFS load average and locking Message-ID: There is a condition (known) where locks are not being released as they should be. In a forthcoming patch, there is a tunable parameter which allows the purging of unused, yet retained locks by a percentage. I've tested this under conditions which affect my ststem and it was rock solid afterwards. At the time I tested it, you had to make the change after the system was up and running (ie, not a config setting). Hopefully this will make it into update 7. Regards, Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Stanley, Jon Sent: Wednesday, March 08, 2006 1:54 PM To: linux-cluster at redhat.com Subject: [Linux-cluster] GFS load average and locking I have a 7 node GFS cluster, plus 3 lock servers (RH AS3U5, GULM locking) that do not mount the filesystem. I have a problem whereby the load average on the system is extremely high (occasionally astronomical), eventually leading to a complete site outage, via inability to access the shared filesystem. I have a couple questions about the innards of GFS that I would be most grateful for someone to answer: The application is written in PHP, and the PHP sessioning is handled via the GFS filesystem as well, if that's important. 1) I notice that I have a lot of processes in uninterruptible sleep. When I attached strace to one of these processes, I obviously found it doing nothing for a period of ~30-60 seconds. An excerpt of the strace (using -r) follows: 0.001224 stat64("/media/files/global/2/6/26c4f61c69117d55b352ce328babbff4.jpg", {st_mode=S_IFREG|0644, st_size=9072, ...}) = 0 0.000251 open("/media/files/global/2/6/26c4f61c69117d55b352ce328babbff4.jpg", O_RDONLY) = 5 0.000108 mmap2(NULL, 9072, PROT_READ, MAP_PRIVATE, 5, 0) = 0xaf381000 0.000069 writev(4, [{"HTTP/1.1 200 OK\r\nDate: Wed, 08 M"..., 318}, {"\377\330\377\340\0\20JFIF\0\1\2\0\0d\0d\0\0\377\354\0\21"..., 9072}], 2) = 9390 0.000630 close(5) = 0 0.000049 munmap(0xaf381000, 9072) = 0 0.000052 rt_sigaction(SIGUSR1, {0x81ef474, [], SA_RESTORER|SA_INTERRUPT, 0x1b2eb8}, {SIG_IGN}, 8) = 0 0.000068 read(4, 0xa239b3c, 4096) = ? ERESTARTSYS (To be restarted) 6.546891 --- SIGALRM (Alarm clock) @ 0 (0) --- 0.000119 close(4) = 0 What it looks like is it hangs out in read() for a period of time, thus leading to the uninterruptible sleep. This particular example was 6 seconds, however it seems that the time is variable. The particular file in this instance is not large, only 9k. I've never seen ERESTARTSYS before, and some googling tells me that it's basically telling the kernel to interrupt the current syscall in order to handle a signal (SIGALRM in this case, which I'm not sure the function of). I could be *way* off base here - I'm not a programmer by any stretch of the imagination. 2) The locking statistics seems to be a huge mystery. The lock total doesn't seem to correspond to the number of open files that I have (I hope!). Here's the output of a 'cat /proc/gulm/lockspace - I can't imagine that I have 300,000+ files open on this system at this point - when are the locks released, or is this even an indication of how many locks that are active at the current time? What does the 'pending' number mean? [svadmin at s259830hz1sl01 gulm]$ cat lockspace lock counts: total: 369822 unl: 176518 exl: 1555 shd: 191501 dfr: 0 pending: 5 lvbs: 2000 lops: 21467433 [svadmin at s259830hz1sl01 gulm]$ Thanks for any help that anyone can provide on this! Thanks! -Jon -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From wcheng at redhat.com Wed Mar 8 19:20:32 2006 From: wcheng at redhat.com (Wendy Cheng) Date: Wed, 08 Mar 2006 14:20:32 -0500 Subject: [Linux-cluster] GFS load average and locking In-Reply-To: <9A6FE0FCC2B29846824C5CD81C6647B90143604F@s228130hz1ew08.apptix-01.savvis.net> References: <9A6FE0FCC2B29846824C5CD81C6647B90143604F@s228130hz1ew08.apptix-01.savvis.net> Message-ID: <440F2E80.9030507@redhat.com> Stanley, Jon wrote: >2) The locking statistics seems to be a huge mystery. The lock total >doesn't seem to correspond to the number of open files that I have (I >hope!). Here's the output of a 'cat /proc/gulm/lockspace - I can't >imagine that I have 300,000+ files open on this system at this point - >when are the locks released, or is this even an indication of how many >locks that are active at the current time? What does the 'pending' >number mean? > > GFS caches locks and normally won't release them (for performance reason). However, we do find this could cause latency issue, particularly after back up and/or tar command where lots of locks are accumulated into one single node that previously issued the backup command. Judging by your description of read latency and number of "shared" locks in your lockspace output, we do have a new tunable in to-be-released-soon RHEL3 Update 7 that allows admin to purge the locks. This seems to help several of (beta) customers to resolve their latency issues. Other than this, do you find any error messages in your /var/log/messages file ? -- Wendy From Jon.Stanley at savvis.net Wed Mar 8 19:36:00 2006 From: Jon.Stanley at savvis.net (Stanley, Jon) Date: Wed, 8 Mar 2006 13:36:00 -0600 Subject: [Linux-cluster] GFS load average and locking Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B901436104@s228130hz1ew08.apptix-01.savvis.net> We're not doing any tar/backup of the filesystem, so I don't think that this is the issue. There are a *large* number of small files (but the files per directory are kept small). I'm not sure if that has anything to do with this. There are no abnormal messages in /var/log/messages. The lockspace output that I gave you is from a client, not the lock master. Let me know if there is any more information that I might be able to provide. We have a GFS service request open, but it doesn't seem to be getting very far :-( -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Wendy Cheng Sent: Wednesday, March 08, 2006 1:21 PM To: linux clustering Subject: Re: [Linux-cluster] GFS load average and locking Stanley, Jon wrote: >2) The locking statistics seems to be a huge mystery. The lock total >doesn't seem to correspond to the number of open files that I have (I >hope!). Here's the output of a 'cat /proc/gulm/lockspace - I can't >imagine that I have 300,000+ files open on this system at this point - >when are the locks released, or is this even an indication of how many >locks that are active at the current time? What does the 'pending' >number mean? > > GFS caches locks and normally won't release them (for performance reason). However, we do find this could cause latency issue, particularly after back up and/or tar command where lots of locks are accumulated into one single node that previously issued the backup command. Judging by your description of read latency and number of "shared" locks in your lockspace output, we do have a new tunable in to-be-released-soon RHEL3 Update 7 that allows admin to purge the locks. This seems to help several of (beta) customers to resolve their latency issues. Other than this, do you find any error messages in your /var/log/messages file ? -- Wendy -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From wcheng at redhat.com Wed Mar 8 19:51:59 2006 From: wcheng at redhat.com (Wendy Cheng) Date: Wed, 08 Mar 2006 14:51:59 -0500 Subject: [Linux-cluster] GFS load average and locking In-Reply-To: <9A6FE0FCC2B29846824C5CD81C6647B901436104@s228130hz1ew08.apptix-01.savvis.net> References: <9A6FE0FCC2B29846824C5CD81C6647B901436104@s228130hz1ew08.apptix-01.savvis.net> Message-ID: <440F35DF.1070604@redhat.com> Stanley, Jon wrote: >We have a GFS service request open, > Could you pass your ticket number so we can check into this ? -- Wendy From Britt.Treece at savvis.net Wed Mar 8 19:58:23 2006 From: Britt.Treece at savvis.net (Treece, Britt) Date: Wed, 8 Mar 2006 13:58:23 -0600 Subject: [Linux-cluster] GFS load average and locking Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B901436169@s228130hz1ew08.apptix-01.savvis.net> Wendy, The ticket number is 836516. We've been told that it has been escalated to the GFS engineers. Here is lockdump information that we've pulled on an inode whose httpd process is in the "D" state... $ sudo lsof /data02 | grep 14060 httpd 14060 nobody cwd DIR 254,66 3864 93725927 /data02/resources/htdocs/SVVS-2006-03-02-06-17-20 httpd 14060 nobody 5u REG 254,66 15624 52992012 /data02/sessions/6/c/2/4/sess_6c249351c42e2c19c669b068433db9a8 $ ps -auxwww | grep 14060 root 24901 0.0 0.0 1700 432 pts/2 S 12:57 0:00 strace -rp 14060 nobody 14060 0.1 0.5 150564 42348 ? D 09:47 0:19 /usr/local/apache/bin/httpd -DSSL following parsed from a lockdump of /data02... Glock (7, 52992012) gl_flags = gl_count = 2 gl_state = 3 lvb_count = 0 object = yes dependencies = no reclaim = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 5 6 Glock (8, 52992012) gl_flags = gl_count = 4 gl_state = 1 lvb_count = 0 object = no dependencies = no reclaim = no Holder owner = -1 gh_state = 1 gh_flags = 5 7 error = 0 gh_iflags = 1 5 6 Waiter2 owner = -1 gh_state = 0 gh_flags = 0 error = 0 gh_iflags = 2 3 4 Waiter2 owner = -1 gh_state = 1 gh_flags = 5 7 error = 0 gh_iflags = 1 Glock (4, 52992012) gl_flags = gl_count = 3 gl_state = 3 lvb_count = 0 object = yes dependencies = no reclaim = no Inode: num = 52992012/52992012 type = 1 i_count = 1 i_flags = vnode = yes -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Wendy Cheng Sent: Wednesday, March 08, 2006 1:52 PM To: linux clustering Subject: Re: [Linux-cluster] GFS load average and locking Stanley, Jon wrote: >We have a GFS service request open, > Could you pass your ticket number so we can check into this ? -- Wendy -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From rstevens at vitalstream.com Wed Mar 8 21:05:55 2006 From: rstevens at vitalstream.com (Rick Stevens) Date: Wed, 08 Mar 2006 13:05:55 -0800 Subject: [Linux-cluster] Clustering RHEL 4 with EXP400 In-Reply-To: <751353696.20060308111357@sur.ogs-id.com> References: <751353696.20060308111357@sur.ogs-id.com> Message-ID: <1141851956.890.268.camel@prophead.corp.publichost.com> On Wed, 2006-03-08 at 11:13 +0700, Milis wrote: > Dear All, > > does any one here have experince to cluster RHEL 4 with 2 IBM x346 > and 1 EXP 400, > I need to do this for build Oracle10g On Rac, whether I need driver > for build and startup of this device. > may I know what I need for this requirement? > and what should I do to get driver of IBM EXP400 ? > I've been success install IBM X346 with Raid 1 option on RHEL 4, but I > really confuse what next to do to show up cluster on EXP400 (whether I > have no driver for this device to share storage) It's rather too much to go into on the mailing list. I first recommend you google "linux +cluster" for some background information. You probably should also join the linux-cluster mailing list for details on this. If you purchased the RHEL HA package or GFS system, check your manuals. Generally, you need a SAN of some sort to provide the storage (fiberchannel disk array, iSCSI array, something). You then need to install the cluster software and kernel patches, decide on what kind of device management you need (gulm, dlm, etc.) and fire it up. As I said, it's too complicated to give you a tutorial on a mailing list. ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - I never drink water because of the disgusting things that fish do - - in it. - - -- WC. Fields - ---------------------------------------------------------------------- From lhh at redhat.com Wed Mar 8 22:32:37 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 08 Mar 2006 17:32:37 -0500 Subject: [Linux-cluster] RHEL4.0 CS and Ldap In-Reply-To: References: Message-ID: <1141857157.25169.153.camel@ayanami.boston.redhat.com> On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: > Is there a way to create an Ldap cluster that can do replication with > RHEL 4.0 CS? For some reason, I though OpenLDAP had built-in replication? -- Lon From rainer at ultra-secure.de Wed Mar 8 22:43:29 2006 From: rainer at ultra-secure.de (Rainer Duffner) Date: Wed, 08 Mar 2006 23:43:29 +0100 Subject: [Linux-cluster] RHEL4.0 CS and Ldap In-Reply-To: <1141857157.25169.153.camel@ayanami.boston.redhat.com> References: <1141857157.25169.153.camel@ayanami.boston.redhat.com> Message-ID: <440F5E11.4040406@ultra-secure.de> Lon Hohberger wrote: >On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: > > >>Is there a way to create an Ldap cluster that can do replication with >>RHEL 4.0 CS? >> >> > >For some reason, I though OpenLDAP had built-in replication? > > Not multi-master (which I assume is what the original poster wants). http://www.redhat.com/en_us/USA/home/solutions/directoryserver/ exists for a reason... cheers, Rainer From lhh at redhat.com Wed Mar 8 22:48:10 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 08 Mar 2006 17:48:10 -0500 Subject: [Linux-cluster] Re: Trouble with RHCS 3.0 In-Reply-To: <20060306062213.85577.qmail@web36107.mail.mud.yahoo.com> References: <20060306062213.85577.qmail@web36107.mail.mud.yahoo.com> Message-ID: <1141858090.25169.166.camel@ayanami.boston.redhat.com> On Sun, 2006-03-05 at 22:22 -0800, paul raymond wrote: > Greetings Lon, Hi, sorry I'm late responding to this. > The problem is that I can not get Quorum to start unless I run the > command "cluforce"! But after viewing clustat commands on systems c11 > and c12, it looks like c11 and c12 cant see each other status due to > some issue with the raw partitions I believe? If you're using an IP tiebreaker, they won't be looking for each other on the shared partitions. The nodes communicate with each other primarily over the network - if they don't see each other, they will not form a quorum. You can try this if you want more detailed information: # service clumanager stop (on both nodes) # clumembd -fd (on both nodes) It will give you all sorts of information, but the most important one you should be looking for is: [PID] info: Membership View #1:0x00000001 If you see both nodes, it will show 0x00000003 (it's a bitmap). If the nodes can't see each other over the network, they will show 1 or 2. If this happens, you should check your network configuration and clumanager's settings - you might want to try using broadcast instead of multicast, etc. > I am using a Mylex Fiber Channel Box with QLogic 2300 interface > card! The raw devices are setup on a 2 mirror drives, Raid 1. Can you > please shed any good ideas what might be wrong here? The vidals are > below! Note that for one node to *start* without the other when using an IP tiebreaker, having to run 'cluforce' is the default behavior. If you wish to change this, please check the man page for the 'cluforce' command and the 'cludb' command. The IP tiebreaker is typically used to *maintain* a quorum after a node failure, because there are certain network faults in which two nodes may see the tiebreaker - but not each other. -- Lon From lhh at redhat.com Wed Mar 8 22:48:41 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 08 Mar 2006 17:48:41 -0500 Subject: [Linux-cluster] sun cluster ccp for redhat In-Reply-To: <984C9DBB29704B47B7AAD308F2C95A3B04DE71@kmail.ksolutions.it> References: <984C9DBB29704B47B7AAD308F2C95A3B04DE71@kmail.ksolutions.it> Message-ID: <1141858121.25169.168.camel@ayanami.boston.redhat.com> On Fri, 2006-03-03 at 18:48 +0100, Baesso Mirko wrote: > Hi, > > i would like to known if there is a tool like cluster console panel to > manage cluster node as sun cluster do I don't know what CCP is, but there's system-config-cluster -- Lon From lhh at redhat.com Wed Mar 8 22:50:52 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 08 Mar 2006 17:50:52 -0500 Subject: [Linux-cluster] Cluster service restarting Locally In-Reply-To: <20060306064749.22036.qmail@webmail50.rediffmail.com> References: <20060306064749.22036.qmail@webmail50.rediffmail.com> Message-ID: <1141858252.25169.170.camel@ayanami.boston.redhat.com> On Mon, 2006-03-06 at 06:47 +0000, saju john wrote: > > > Dear All, > > I have a 2 node cluster with RHAS3 update 3. > Kernel : 2.4.21-20.Elsmp > Clumanager : clumanager-1.2.16-1 > > For more than a year everyting had been fine. Suddenly it started > showing the follwing and restarted the service locally > > clusvcmgrd[1388]: Unable to obtain cluster lock: Connection > timed out > clulockd[1378]: Denied A.B.C.D: Broken pipe > clulockd[1378]: select error: Broken pipe > clusvcmgrd: [1625]: service notice: Stopping service > postgresql ... > clusvcmgrd: [1625]: service notice: Running user script > '/etc/init.d/postgresql stop' > clusvcmgrd: [1625]: service notice: Stopped service > postgresql > clusvcmgrd: [1625]: service notice: Starting service > postgresql ... > clusvcmgrd: [1625]: service notice: Running user script > '/etc/init.d/postgresql start' > clusvcmgrd: [1625]: service notice: Started service > postgresql ... It should be fixed in RHCS3U7 -- Lon From lhh at redhat.com Wed Mar 8 22:51:58 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 08 Mar 2006 17:51:58 -0500 Subject: [Linux-cluster] Cluster service restarting Locally In-Reply-To: References: Message-ID: <1141858318.25169.172.camel@ayanami.boston.redhat.com> On Mon, 2006-03-06 at 14:02 -0600, Hong Zheng wrote: > I?m having the same problem. My system configuration is as follows: > > 2-node cluster: RH ES3, GFS6.0, clumanager-1.2.28-1 and > redhat-config-cluster-1.0.8-1 > > Kernel: 2.4.21-37.EL > > Linux-iscsi-3.6.3 initiator: connections to iSCSI shared storage > server If it's not fixed in U7 (which I think it should be), please file a bugzilla... It sounds like the lock traffic is getting network-starved. -- Lon From lhh at redhat.com Wed Mar 8 22:53:46 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 08 Mar 2006 17:53:46 -0500 Subject: [Linux-cluster] RHEL4.0 CS and Ldap In-Reply-To: <440F5E11.4040406@ultra-secure.de> References: <1141857157.25169.153.camel@ayanami.boston.redhat.com> <440F5E11.4040406@ultra-secure.de> Message-ID: <1141858426.25169.175.camel@ayanami.boston.redhat.com> On Wed, 2006-03-08 at 23:43 +0100, Rainer Duffner wrote: > Lon Hohberger wrote: > > >On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: > > > > > >>Is there a way to create an Ldap cluster that can do replication with > >>RHEL 4.0 CS? > >> > >> > > > >For some reason, I though OpenLDAP had built-in replication? > > > > > > > Not multi-master (which I assume is what the original poster wants). Indeed. -- Lon From lhh at redhat.com Wed Mar 8 23:22:29 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 08 Mar 2006 18:22:29 -0500 Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? In-Reply-To: <20060304063504.30937.qmail@web52308.mail.yahoo.com> References: <20060304063504.30937.qmail@web52308.mail.yahoo.com> Message-ID: <1141860149.25169.204.camel@ayanami.boston.redhat.com> On Fri, 2006-03-03 at 22:35 -0800, SUVANKAR MOITRA wrote: > dear lon, > > I have some question about the script and the > installation :-- > 1> Can i install RHCS4 after install the Oracle 10g? Yes, but that was not how the document was written. The major difference is that you have to manually test everything rather than using the cluster tools to help you. e.g. to stop (being consistent with the presented example in the howto, and assuming all environment vars are set correctly): /usr/share/cluster/oracledb.sh stop umount /mnt/oracle ip addr del 192.168.1.20/22 dev eth0 or to start the service: ip addr add 192.168.1.20/22 dev eth0 mount -t ext3 /dev/sdb7 /mnt/oracle /usr/share/cluster/oracledb.sh start If the cluster is set up correctly, this will work on both nodes (note that you must stop it on one before starting on another). > 2> The /mnt/oracle mount point is temporary for the > oracle installation or should i write on /etc/fstab ? It is mounted by the cluster when you start the service. Do not place it in /etc/fstab, as mounting an ext3 file system on multiple nodes will cause you to have a corrupt file system *very* quickly! > 3> Can i mention ORACLE_HOME,ORACLE_BASE,ORACLE_SID > etc on .bash_profile of every node or leave it as it > is only create oracle user and group? You can, and it will help the testing/debugging phase. However, it is not used by the cluster software when starting/stopping Oracle; everything must be in the cluster configuration. Don't forget to set ORACLE_HOSTNAME (which is used by the script to trick Oracle in to using the service IP address/hostname that you set in Part 2 - step 2), since apparently OUI_HOSTNAME does not seem to work the way I expected it should. > 4> Where should i place oracledb.sh file? I think its > required in every node, am i write ? Part 1, step 6 (steps to take on all nodes): Install the oracledb.sh resource agent in to /usr/share/cluster > 5>What is the exact use of oracledb.sh file? It is called by the cluster software to start/stop/check status of the Oracle instance. Additionally, if you have the environment variables set up correctly, it will start/stop Oracle outside of the cluster environment, too (just like a normal initscript...). > 6> How can i shutdown the oracle, should i write > script for that, like orastop and orastart for up the > oracle? If your environment variables are set correctly and the cluster is not running: /usr/share/cluster/oracledb.sh stop Once the instance is managed by RHCS (and RHCS is running!), you can use 'clusvcadm' to disable and enable the now failover-capable Oracle instance, and move it around (see the clusvcadm man page for more details). -- Lon From bobby.m.dalton at nasa.gov Wed Mar 8 23:40:21 2006 From: bobby.m.dalton at nasa.gov (Dalton, Maurice) Date: Wed, 8 Mar 2006 17:40:21 -0600 Subject: [Linux-cluster] RHEL4.0 CS and Ldap References: <1141857157.25169.153.camel@ayanami.boston.redhat.com> Message-ID: Yes multi-master is what I am looking for. ________________________________ From: linux-cluster-bounces at redhat.com on behalf of Lon Hohberger Sent: Wed 3/8/2006 4:32 PM To: linux clustering Subject: Re: [Linux-cluster] RHEL4.0 CS and Ldap On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: > Is there a way to create an Ldap cluster that can do replication with > RHEL 4.0 CS? For some reason, I though OpenLDAP had built-in replication? -- Lon -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3676 bytes Desc: not available URL: From suvankar_moitra at yahoo.com Thu Mar 9 05:38:24 2006 From: suvankar_moitra at yahoo.com (SUVANKAR MOITRA) Date: Wed, 8 Mar 2006 21:38:24 -0800 (PST) Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? In-Reply-To: <1141860149.25169.204.camel@ayanami.boston.redhat.com> Message-ID: <20060309053824.82455.qmail@web52301.mail.yahoo.com> dear lon, I am thankful to u .Its a great opertunity for me to take your guidence .I was sucessfully load the oracle in virtual ip and the script is runing .But one problem is there the script is not runing from cluster if i mension the path /usr/share/cluster/oracledb.sh start.When i want to run the oracle i am using the following command :- ./oracledb.sh start or stop or status etc.... How can i put the thing in cluster suite? Lon can i install oracle rac on rhcs4?Is it possible? Because oracle rac using ocfs. Thanks & regards Suvankar Moitra Kolkata , India --- Lon Hohberger wrote: > On Fri, 2006-03-03 at 22:35 -0800, SUVANKAR MOITRA > wrote: > > dear lon, > > > > I have some question about the script and the > > installation :-- > > 1> Can i install RHCS4 after install the Oracle > 10g? > > Yes, but that was not how the document was written. > The major > difference is that you have to manually test > everything rather than > using the cluster tools to help you. > > e.g. to stop (being consistent with the presented > example in the howto, > and assuming all environment vars are set > correctly): > > /usr/share/cluster/oracledb.sh stop > umount /mnt/oracle > ip addr del 192.168.1.20/22 dev eth0 > > or to start the service: > > ip addr add 192.168.1.20/22 dev eth0 > mount -t ext3 /dev/sdb7 /mnt/oracle > /usr/share/cluster/oracledb.sh start > > If the cluster is set up correctly, this will work > on both nodes (note > that you must stop it on one before starting on > another). > > > > > > 2> The /mnt/oracle mount point is temporary for > the > > oracle installation or should i write on > /etc/fstab ? > > It is mounted by the cluster when you start the > service. Do not place > it in /etc/fstab, as mounting an ext3 file system on > multiple nodes will > cause you to have a corrupt file system *very* > quickly! > > > > 3> Can i mention > ORACLE_HOME,ORACLE_BASE,ORACLE_SID > > etc on .bash_profile of every node or leave it as > it > > is only create oracle user and group? > > You can, and it will help the testing/debugging > phase. However, it is > not used by the cluster software when > starting/stopping Oracle; > everything must be in the cluster configuration. > > Don't forget to set ORACLE_HOSTNAME (which is used > by the script to > trick Oracle in to using the service IP > address/hostname that you set in > Part 2 - step 2), since apparently OUI_HOSTNAME does > not seem to work > the way I expected it should. > > > > 4> Where should i place oracledb.sh file? I think > its > > required in every node, am i write ? > > Part 1, step 6 (steps to take on all nodes): > Install the oracledb.sh resource agent in to > /usr/share/cluster > > > > 5>What is the exact use of oracledb.sh file? > > It is called by the cluster software to > start/stop/check status of the > Oracle instance. > > Additionally, if you have the environment variables > set up correctly, it > will start/stop Oracle outside of the cluster > environment, too (just > like a normal initscript...). > > > > 6> How can i shutdown the oracle, should i write > > script for that, like orastop and orastart for up > the > > oracle? > > If your environment variables are set correctly and > the cluster is not > running: > > /usr/share/cluster/oracledb.sh stop > > Once the instance is managed by RHCS (and RHCS is > running!), you can use > 'clusvcadm' to disable and enable the now > failover-capable Oracle > instance, and move it around (see the clusvcadm man > page for more > details). > > -- Lon > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From basv at sara.nl Thu Mar 9 06:34:50 2006 From: basv at sara.nl (Bas van der Vlies) Date: Thu, 09 Mar 2006 07:34:50 +0100 Subject: [Linux-cluster] gfs + nfsd crash In-Reply-To: <440EF9BD.4030108@sara.nl> References: <05AF3B6E-B88A-45CC-83E4-1353291E6347@sara.nl> <440EF9BD.4030108@sara.nl> Message-ID: <440FCC8A.7040803@sara.nl> I have checked the CVS source and found this Changelog below. Does this fix my GFS crashes with NFS? It describes my kind of problems. Thanks ====== Changelog CVSROOT: /cvs/cluster Module name: cluster Branch: STABLE Changes by: bmarzins at sourceware.org 2006-03-08 20:47:09 Modified files: gfs-kernel/src/gfs: ops_inode.c Log message: Really gross hack!!! This is a workaround for one of the bugs the got lumped into 166701. It Breaks POSIX behavior in a corner case to avoid crashing... It's icky. when NFS opens a file with O_CREAT, the kernel nfs daemon checks to see if the file exists. If it does, nfsd does the *right thing* (either opens the file, or if the file was opened with O_EXCL, returns an error). If the file doesn't exist, it passes the request down to the underlying file system. Unfortunately, since nfs *knows* that the file doesn't exist, it doesn't bother to pass a nameidata structure, which would include the intent information. However since gfs is a cluster file system, the file could have been created on another node after nfs checks for it. If this is the case, gfs needs the intent information to do the *right thing*. It panics when it finds a NULL pointer, instead of the nameidata. Now, instead of panicing, if gfs finds a NULL nameidata pointer. It assumes that the file was not created with O_EXCL. This assumption could be wrong, with the result that an application could thing that it has created a new file, when in fact, it has opened an existing one. === End Changelog === Bas van der Vlies wrote: > We just upgraded to 2.6.16-rc5 and cvs stable gfs. We still have > gfs_create crashes. > > === Ooops ===== > Unable to handle kernel NULL pointer dereference at virtual address > 00000038 > printing eip: > f89a4be3 > *pde = 37809001 > *pte = 00000000 > Oops: 0000 [#1] > SMP > Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg > ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness dm_mod > CPU: 0 > EIP: 0060:[] Tainted: GF VLI > EFLAGS: 00010246 (2.6.16-rc5-sara3 #1) > EIP is at gfs_create+0x6f/0x153 [gfs] > eax: 00000000 ebx: ffffffef ecx: f27d0d98 edx: ffffffef > esi: f2f84690 edi: f8b93000 ebp: f34a5e98 esp: f34a5e20 > ds: 007b es: 007b ss: 0068 > Process nfsd (pid: 8973, threadinfo=f34a4000 task=f3462a70) > Stack: <0>f092a530 00000001 f34a5e48 00000000 f34a5e84 f89a6628 f34a5e48 > ee1fc324 > 00000003 00000000 f34a5e48 f34a5e48 00000000 f3462a70 00000003 > f34a5e5c > f34a5e5c f27d0d98 f3462a70 00000001 00000020 00000000 000000c2 > 00000000 > Call Trace: > [] show_stack_log_lvl+0xad/0xb5 > [] show_registers+0x10d/0x176 > [] die+0xf2/0x16d > [] do_page_fault+0x3dd/0x57a > [] error_code+0x4f/0x54 > [] vfs_create+0x6a/0xa7 > [] nfsd_create_v3+0x2b1/0x48a > [] nfsd3_proc_create+0x116/0x123 > [] nfsd_dispatch+0xbe/0x17f > [] svc_process+0x381/0x5c7 > [] nfsd+0x18d/0x2e2 > [] kernel_thread_helper+0x5/0xb > Code: 94 50 8b 45 0c ff 75 10 83 c0 1c 6a 01 89 45 88 50 8d 45 c4 50 e8 > 70 08 ff ff 83 c4 14 89 c3 85 c0 74 4883 f8 ef 75 33 8b 45 14 <80> 78 38 > 00 78 2a 8d 45 94 50 8d 45 c4 6a 00 ff 75 88 50 e8 3c > BUG: nfsd/8973, lock held at task exit time! > [ee1fc398] {inode_init_once} > .. held by: nfsd: 8973 [f3462a70, 115] > ... acquired at: nfsd_create_v3+0x127/0x48a > > > -- -- ******************************************************************** * * * Bas van der Vlies e-mail: basv at sara.nl * * SARA - Academic Computing Services phone: +31 20 592 8012 * * Kruislaan 415 fax: +31 20 6683167 * * 1098 SJ Amsterdam * * * ******************************************************************** From grimme at atix.de Thu Mar 9 08:48:01 2006 From: grimme at atix.de (Marc Grimme) Date: Thu, 9 Mar 2006 09:48:01 +0100 Subject: [Linux-cluster] GFS load average and locking In-Reply-To: <9A6FE0FCC2B29846824C5CD81C6647B90143604F@s228130hz1ew08.apptix-01.savvis.net> References: <9A6FE0FCC2B29846824C5CD81C6647B90143604F@s228130hz1ew08.apptix-01.savvis.net> Message-ID: <200603090948.03134.grimme@atix.de> Hi, On Wednesday 08 March 2006 19:54, Stanley, Jon wrote: > I have a 7 node GFS cluster, plus 3 lock servers (RH AS3U5, GULM > locking) that do not mount the filesystem. I have a problem whereby the > load average on the system is extremely high (occasionally > astronomical), eventually leading to a complete site outage, via > inability to access the shared filesystem. I have a couple questions > about the innards of GFS that I would be most grateful for someone to > answer: > > The application is written in PHP, and the PHP sessioning is handled via > the GFS filesystem as well, if that's important. > > 1) I notice that I have a lot of processes in uninterruptible sleep. > When I attached strace to one of these processes, I obviously found it > doing nothing for a period of ~30-60 seconds. An excerpt of the strace > (using -r) follows: > > 0.001224 > stat64("/media/files/global/2/6/26c4f61c69117d55b352ce328babbff4.jpg", > {st_mode=S_IFREG|0644, st_size=9072, ...}) = 0 > 0.000251 > open("/media/files/global/2/6/26c4f61c69117d55b352ce328babbff4.jpg", > O_RDONLY) = 5 > 0.000108 mmap2(NULL, 9072, PROT_READ, MAP_PRIVATE, 5, 0) = > 0xaf381000 > 0.000069 writev(4, [{"HTTP/1.1 200 OK\r\nDate: Wed, 08 M"..., 318}, > {"\377\330\377\340\0\20JFIF\0\1\2\0\0d\0d\0\0\377\354\0\21"..., 9072}], > 2) = 9390 > 0.000630 close(5) = 0 > 0.000049 munmap(0xaf381000, 9072) = 0 > 0.000052 rt_sigaction(SIGUSR1, {0x81ef474, [], > SA_RESTORER|SA_INTERRUPT, 0x1b2eb8}, {SIG_IGN}, 8) = 0 > 0.000068 read(4, 0xa239b3c, 4096) = ? ERESTARTSYS (To be > restarted) > 6.546891 --- SIGALRM (Alarm clock) @ 0 (0) --- > 0.000119 close(4) = 0 > > What it looks like is it hangs out in read() for a period of time, thus > leading to the uninterruptible sleep. This particular example was 6 > seconds, however it seems that the time is variable. The particular > file in this instance is not large, only 9k. Although the strace does not show the output I know of the problem description sounds like a deja vu. We had loads of problems with having sessions on GFS and httpd s ending up with "D" state for some time (at high load times we had ServerLimit httpd in D per node which ended up in the service not being available). As I posted already we think it is because of the "bad" locking of sessions with php (as php sessions are on gfs and strace showed those timeouts with the session files). When you issue a "session_start" or what ever that function is called, the session_file is locked via an flock syscall. That lock is held until you end the session which is implicitly done when the tcp connection to the client is ended. Now comes another http process (on whatever node) and calls a "session start" and trys an flock on that session while another process already holds that lock. The process might end up in the seen timeouts (30-60secs) which (as far as I remember relates to the timeout of the tcp connection defined in the httpd.conf or some timeout in the php.ini) - there is an explanation on this but I cannot rember ;-) ). Nevertheless in our scenario the problems were the "bad" session handling by php. We have made a patch for the phplib where you can disable the locking, or just implicitly do locking and therefore keep consitency while session data is read or written. We could make apache work as expected and now we don't see any "D" process anymore since a year. Oh yes the patch can be found at www.opensharedroot.org in the download section. Besides: You will never encounter this on a localfilesystem or nfs (as nfs ignores flocks). As nfs does not support flocks and silently ignores them. Hope that helps and let us know about problems. Regards Marc. > > I've never seen ERESTARTSYS before, and some googling tells me that it's > basically telling the kernel to interrupt the current syscall in order > to handle a signal (SIGALRM in this case, which I'm not sure the > function of). I could be *way* off base here - I'm not a programmer by > any stretch of the imagination. > > 2) The locking statistics seems to be a huge mystery. The lock total > doesn't seem to correspond to the number of open files that I have (I > hope!). Here's the output of a 'cat /proc/gulm/lockspace - I can't > imagine that I have 300,000+ files open on this system at this point - > when are the locks released, or is this even an indication of how many > locks that are active at the current time? What does the 'pending' > number mean? > > [svadmin at s259830hz1sl01 gulm]$ cat lockspace > > lock counts: > total: 369822 > unl: 176518 > exl: 1555 > shd: 191501 > dfr: 0 > pending: 5 > lvbs: 2000 > lops: 21467433 > > [svadmin at s259830hz1sl01 gulm]$ > > Thanks for any help that anyone can provide on this! > > Thanks! > -Jon > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Gruss / Regards, Marc Grimme Phone: +49-89 121 409-54 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From nick at sqrt.co.uk Thu Mar 9 09:48:39 2006 From: nick at sqrt.co.uk (Nick Burrett) Date: Thu, 09 Mar 2006 09:48:39 +0000 Subject: [Linux-cluster] RHEL4.0 CS and Ldap In-Reply-To: <440F5E11.4040406@ultra-secure.de> References: <1141857157.25169.153.camel@ayanami.boston.redhat.com> <440F5E11.4040406@ultra-secure.de> Message-ID: <440FF9F7.2060500@sqrt.co.uk> Rainer Duffner wrote: > Lon Hohberger wrote: > >> On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: >> >> >>> Is there a way to create an Ldap cluster that can do replication with >>> RHEL 4.0 CS? >>> >> >> >> For some reason, I though OpenLDAP had built-in replication? >> >> > > > Not multi-master (which I assume is what the original poster wants). > > http://www.redhat.com/en_us/USA/home/solutions/directoryserver/ > > exists for a reason... As I understand it, multi-master can be accomplished in OpenLDAP 2.3 using sync-replication between the two nodes. This is not true multi-master, but the effect is near-enough. Regards, Nick. From cjk at techma.com Thu Mar 9 13:43:03 2006 From: cjk at techma.com (Kovacs, Corey J.) Date: Thu, 9 Mar 2006 08:43:03 -0500 Subject: [Linux-cluster] RHEL4.0 CS and Ldap Message-ID: Multi master LDAP is not all that it's cracked up to be. There are few benefits in being able to write to different servers. The problem is that like many things, the rule is "Last write wins". Writing in LDAP dirs is not something that is "typically" done in high volumes anyway, but reading is. If you just need to load balance traffic, then you might look into an LVS implementation. That way you can set up several replicas and spread the load across them. If your Master gets blown away, you can promote of the replicas to a master.. Just takes some manual intervention. Just my two cents... Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Nick Burrett Sent: Thursday, March 09, 2006 4:49 AM To: linux clustering Subject: Re: [Linux-cluster] RHEL4.0 CS and Ldap Rainer Duffner wrote: > Lon Hohberger wrote: > >> On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: >> >> >>> Is there a way to create an Ldap cluster that can do replication >>> with RHEL 4.0 CS? >>> >> >> >> For some reason, I though OpenLDAP had built-in replication? >> >> > > > Not multi-master (which I assume is what the original poster wants). > > http://www.redhat.com/en_us/USA/home/solutions/directoryserver/ > > exists for a reason... As I understand it, multi-master can be accomplished in OpenLDAP 2.3 using sync-replication between the two nodes. This is not true multi-master, but the effect is near-enough. Regards, Nick. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From baesso at ksolutions.it Thu Mar 9 13:59:36 2006 From: baesso at ksolutions.it (Baesso Mirko) Date: Thu, 9 Mar 2006 14:59:36 +0100 Subject: R: [Linux-cluster] RHEL4.0 CS and Ldap Message-ID: <984C9DBB29704B47B7AAD308F2C95A3B04DEF4@kmail.ksolutions.it> Sorry but i try to use sync-replica either then slurpd on openldap 2.3, can i setup a multi-master environment? Baesso Mirko - System Engineer KSolutions.S.p.A. Via Lenin 132/26 56017 S.Martino Ulmiano (PI) - Italy tel.+ 39 0 50 898369 fax. + 39 0 50 861200 baesso at ksolutions.it http//www.ksolutions.it -----Messaggio originale----- Da: Kovacs, Corey J. [mailto:cjk at techma.com] Inviato: gioved? 9 marzo 2006 14.43 A: linux clustering Oggetto: RE: [Linux-cluster] RHEL4.0 CS and Ldap Multi master LDAP is not all that it's cracked up to be. There are few benefits in being able to write to different servers. The problem is that like many things, the rule is "Last write wins". Writing in LDAP dirs is not something that is "typically" done in high volumes anyway, but reading is. If you just need to load balance traffic, then you might look into an LVS implementation. That way you can set up several replicas and spread the load across them. If your Master gets blown away, you can promote of the replicas to a master.. Just takes some manual intervention. Just my two cents... Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Nick Burrett Sent: Thursday, March 09, 2006 4:49 AM To: linux clustering Subject: Re: [Linux-cluster] RHEL4.0 CS and Ldap Rainer Duffner wrote: > Lon Hohberger wrote: > >> On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: >> >> >>> Is there a way to create an Ldap cluster that can do replication >>> with RHEL 4.0 CS? >>> >> >> >> For some reason, I though OpenLDAP had built-in replication? >> >> > > > Not multi-master (which I assume is what the original poster wants). > > http://www.redhat.com/en_us/USA/home/solutions/directoryserver/ > > exists for a reason... As I understand it, multi-master can be accomplished in OpenLDAP 2.3 using sync-replication between the two nodes. This is not true multi-master, but the effect is near-enough. Regards, Nick. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From cjk at techma.com Thu Mar 9 14:07:34 2006 From: cjk at techma.com (Kovacs, Corey J.) Date: Thu, 9 Mar 2006 09:07:34 -0500 Subject: [Linux-cluster] RHEL4.0 CS and Ldap Message-ID: You might be able to, but it's not really cluster related. You'll get better information on the openldap lists. Cheers. Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Baesso Mirko Sent: Thursday, March 09, 2006 9:00 AM To: linux clustering Subject: R: [Linux-cluster] RHEL4.0 CS and Ldap Sorry but i try to use sync-replica either then slurpd on openldap 2.3, can i setup a multi-master environment? Baesso Mirko - System Engineer KSolutions.S.p.A. Via Lenin 132/26 56017 S.Martino Ulmiano (PI) - Italy tel.+ 39 0 50 898369 fax. + 39 0 50 861200 baesso at ksolutions.it http//www.ksolutions.it -----Messaggio originale----- Da: Kovacs, Corey J. [mailto:cjk at techma.com] Inviato: gioved? 9 marzo 2006 14.43 A: linux clustering Oggetto: RE: [Linux-cluster] RHEL4.0 CS and Ldap Multi master LDAP is not all that it's cracked up to be. There are few benefits in being able to write to different servers. The problem is that like many things, the rule is "Last write wins". Writing in LDAP dirs is not something that is "typically" done in high volumes anyway, but reading is. If you just need to load balance traffic, then you might look into an LVS implementation. That way you can set up several replicas and spread the load across them. If your Master gets blown away, you can promote of the replicas to a master.. Just takes some manual intervention. Just my two cents... Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Nick Burrett Sent: Thursday, March 09, 2006 4:49 AM To: linux clustering Subject: Re: [Linux-cluster] RHEL4.0 CS and Ldap Rainer Duffner wrote: > Lon Hohberger wrote: > >> On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: >> >> >>> Is there a way to create an Ldap cluster that can do replication >>> with RHEL 4.0 CS? >>> >> >> >> For some reason, I though OpenLDAP had built-in replication? >> >> > > > Not multi-master (which I assume is what the original poster wants). > > http://www.redhat.com/en_us/USA/home/solutions/directoryserver/ > > exists for a reason... As I understand it, multi-master can be accomplished in OpenLDAP 2.3 using sync-replication between the two nodes. This is not true multi-master, but the effect is near-enough. Regards, Nick. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From bobby.m.dalton at nasa.gov Thu Mar 9 14:17:05 2006 From: bobby.m.dalton at nasa.gov (Dalton, Maurice) Date: Thu, 9 Mar 2006 08:17:05 -0600 Subject: [Linux-cluster] RHEL4.0 CS and Ldap Message-ID: I am currently using Heartbeat to solve my need for Highly Available Ldap servers. I was just trying to figure another way to provide HA, replication with RHEL4.0 CS. It would be nice to build a RHEL4.0 CS service for ldap that would cover all of my requirements. Just a thought. Thanks for the replies... -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Kovacs, Corey J. Sent: Thursday, March 09, 2006 8:08 AM To: linux clustering Subject: RE: [Linux-cluster] RHEL4.0 CS and Ldap You might be able to, but it's not really cluster related. You'll get better information on the openldap lists. Cheers. Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Baesso Mirko Sent: Thursday, March 09, 2006 9:00 AM To: linux clustering Subject: R: [Linux-cluster] RHEL4.0 CS and Ldap Sorry but i try to use sync-replica either then slurpd on openldap 2.3, can i setup a multi-master environment? Baesso Mirko - System Engineer KSolutions.S.p.A. Via Lenin 132/26 56017 S.Martino Ulmiano (PI) - Italy tel.+ 39 0 50 898369 fax. + 39 0 50 861200 baesso at ksolutions.it http//www.ksolutions.it -----Messaggio originale----- Da: Kovacs, Corey J. [mailto:cjk at techma.com] Inviato: gioved? 9 marzo 2006 14.43 A: linux clustering Oggetto: RE: [Linux-cluster] RHEL4.0 CS and Ldap Multi master LDAP is not all that it's cracked up to be. There are few benefits in being able to write to different servers. The problem is that like many things, the rule is "Last write wins". Writing in LDAP dirs is not something that is "typically" done in high volumes anyway, but reading is. If you just need to load balance traffic, then you might look into an LVS implementation. That way you can set up several replicas and spread the load across them. If your Master gets blown away, you can promote of the replicas to a master.. Just takes some manual intervention. Just my two cents... Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Nick Burrett Sent: Thursday, March 09, 2006 4:49 AM To: linux clustering Subject: Re: [Linux-cluster] RHEL4.0 CS and Ldap Rainer Duffner wrote: > Lon Hohberger wrote: > >> On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: >> >> >>> Is there a way to create an Ldap cluster that can do replication >>> with RHEL 4.0 CS? >>> >> >> >> For some reason, I though OpenLDAP had built-in replication? >> >> > > > Not multi-master (which I assume is what the original poster wants). > > http://www.redhat.com/en_us/USA/home/solutions/directoryserver/ > > exists for a reason... As I understand it, multi-master can be accomplished in OpenLDAP 2.3 using sync-replication between the two nodes. This is not true multi-master, but the effect is near-enough. Regards, Nick. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From cjk at techma.com Thu Mar 9 14:24:55 2006 From: cjk at techma.com (Kovacs, Corey J.) Date: Thu, 9 Mar 2006 09:24:55 -0500 Subject: [Linux-cluster] RHEL4.0 CS and Ldap Message-ID: Dalton, I figured as much and you could indeed use the CS portion to do what Heartbeat does. I do that same thing on RHEL3CS. It's the replication part that doesn't fall into the clusterring category, that's all. The cluster serverices (resource manager) does, in effect, what hearbeat does tho so you can either have a single instence bounce aroung from node to node as needed, or you could have a master + replicas all running at the same time (with differnet data stores) and not have to worry about the service failing over, just flip IP address, which is what it sounds like you are doing already. For better performance tho, if that's needed, you'll prolly want the LVS (piranha) solution. Good luck Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Dalton, Maurice Sent: Thursday, March 09, 2006 9:17 AM To: linux clustering Subject: RE: [Linux-cluster] RHEL4.0 CS and Ldap I am currently using Heartbeat to solve my need for Highly Available Ldap servers. I was just trying to figure another way to provide HA, replication with RHEL4.0 CS. It would be nice to build a RHEL4.0 CS service for ldap that would cover all of my requirements. Just a thought. Thanks for the replies... -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Kovacs, Corey J. Sent: Thursday, March 09, 2006 8:08 AM To: linux clustering Subject: RE: [Linux-cluster] RHEL4.0 CS and Ldap You might be able to, but it's not really cluster related. You'll get better information on the openldap lists. Cheers. Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Baesso Mirko Sent: Thursday, March 09, 2006 9:00 AM To: linux clustering Subject: R: [Linux-cluster] RHEL4.0 CS and Ldap Sorry but i try to use sync-replica either then slurpd on openldap 2.3, can i setup a multi-master environment? Baesso Mirko - System Engineer KSolutions.S.p.A. Via Lenin 132/26 56017 S.Martino Ulmiano (PI) - Italy tel.+ 39 0 50 898369 fax. + 39 0 50 861200 baesso at ksolutions.it http//www.ksolutions.it -----Messaggio originale----- Da: Kovacs, Corey J. [mailto:cjk at techma.com] Inviato: gioved? 9 marzo 2006 14.43 A: linux clustering Oggetto: RE: [Linux-cluster] RHEL4.0 CS and Ldap Multi master LDAP is not all that it's cracked up to be. There are few benefits in being able to write to different servers. The problem is that like many things, the rule is "Last write wins". Writing in LDAP dirs is not something that is "typically" done in high volumes anyway, but reading is. If you just need to load balance traffic, then you might look into an LVS implementation. That way you can set up several replicas and spread the load across them. If your Master gets blown away, you can promote of the replicas to a master.. Just takes some manual intervention. Just my two cents... Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Nick Burrett Sent: Thursday, March 09, 2006 4:49 AM To: linux clustering Subject: Re: [Linux-cluster] RHEL4.0 CS and Ldap Rainer Duffner wrote: > Lon Hohberger wrote: > >> On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: >> >> >>> Is there a way to create an Ldap cluster that can do replication >>> with RHEL 4.0 CS? >>> >> >> >> For some reason, I though OpenLDAP had built-in replication? >> >> > > > Not multi-master (which I assume is what the original poster wants). > > http://www.redhat.com/en_us/USA/home/solutions/directoryserver/ > > exists for a reason... As I understand it, multi-master can be accomplished in OpenLDAP 2.3 using sync-replication between the two nodes. This is not true multi-master, but the effect is near-enough. Regards, Nick. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From bobby.m.dalton at nasa.gov Thu Mar 9 14:33:18 2006 From: bobby.m.dalton at nasa.gov (Dalton, Maurice) Date: Thu, 9 Mar 2006 08:33:18 -0600 Subject: [Linux-cluster] RHEL4.0 CS and Ldap Message-ID: Thanks Corey. That's exactly what I want. At least 2 servers doing replication and the failover part is just moving the virtual IP to the slave server.. Thanks.. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Kovacs, Corey J. Sent: Thursday, March 09, 2006 8:25 AM To: linux clustering Subject: RE: [Linux-cluster] RHEL4.0 CS and Ldap Dalton, I figured as much and you could indeed use the CS portion to do what Heartbeat does. I do that same thing on RHEL3CS. It's the replication part that doesn't fall into the clusterring category, that's all. The cluster serverices (resource manager) does, in effect, what hearbeat does tho so you can either have a single instence bounce aroung from node to node as needed, or you could have a master + replicas all running at the same time (with differnet data stores) and not have to worry about the service failing over, just flip IP address, which is what it sounds like you are doing already. For better performance tho, if that's needed, you'll prolly want the LVS (piranha) solution. Good luck Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Dalton, Maurice Sent: Thursday, March 09, 2006 9:17 AM To: linux clustering Subject: RE: [Linux-cluster] RHEL4.0 CS and Ldap I am currently using Heartbeat to solve my need for Highly Available Ldap servers. I was just trying to figure another way to provide HA, replication with RHEL4.0 CS. It would be nice to build a RHEL4.0 CS service for ldap that would cover all of my requirements. Just a thought. Thanks for the replies... -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Kovacs, Corey J. Sent: Thursday, March 09, 2006 8:08 AM To: linux clustering Subject: RE: [Linux-cluster] RHEL4.0 CS and Ldap You might be able to, but it's not really cluster related. You'll get better information on the openldap lists. Cheers. Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Baesso Mirko Sent: Thursday, March 09, 2006 9:00 AM To: linux clustering Subject: R: [Linux-cluster] RHEL4.0 CS and Ldap Sorry but i try to use sync-replica either then slurpd on openldap 2.3, can i setup a multi-master environment? Baesso Mirko - System Engineer KSolutions.S.p.A. Via Lenin 132/26 56017 S.Martino Ulmiano (PI) - Italy tel.+ 39 0 50 898369 fax. + 39 0 50 861200 baesso at ksolutions.it http//www.ksolutions.it -----Messaggio originale----- Da: Kovacs, Corey J. [mailto:cjk at techma.com] Inviato: gioved? 9 marzo 2006 14.43 A: linux clustering Oggetto: RE: [Linux-cluster] RHEL4.0 CS and Ldap Multi master LDAP is not all that it's cracked up to be. There are few benefits in being able to write to different servers. The problem is that like many things, the rule is "Last write wins". Writing in LDAP dirs is not something that is "typically" done in high volumes anyway, but reading is. If you just need to load balance traffic, then you might look into an LVS implementation. That way you can set up several replicas and spread the load across them. If your Master gets blown away, you can promote of the replicas to a master.. Just takes some manual intervention. Just my two cents... Corey -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Nick Burrett Sent: Thursday, March 09, 2006 4:49 AM To: linux clustering Subject: Re: [Linux-cluster] RHEL4.0 CS and Ldap Rainer Duffner wrote: > Lon Hohberger wrote: > >> On Wed, 2006-03-08 at 12:43 -0600, Dalton, Maurice wrote: >> >> >>> Is there a way to create an Ldap cluster that can do replication >>> with RHEL 4.0 CS? >>> >> >> >> For some reason, I though OpenLDAP had built-in replication? >> >> > > > Not multi-master (which I assume is what the original poster wants). > > http://www.redhat.com/en_us/USA/home/solutions/directoryserver/ > > exists for a reason... As I understand it, multi-master can be accomplished in OpenLDAP 2.3 using sync-replication between the two nodes. This is not true multi-master, but the effect is near-enough. Regards, Nick. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From lhh at redhat.com Thu Mar 9 14:35:37 2006 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 09 Mar 2006 09:35:37 -0500 Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? In-Reply-To: <20060309053824.82455.qmail@web52301.mail.yahoo.com> References: <20060309053824.82455.qmail@web52301.mail.yahoo.com> Message-ID: <1141914937.25169.220.camel@ayanami.boston.redhat.com> On Wed, 2006-03-08 at 21:38 -0800, SUVANKAR MOITRA wrote: > dear lon, > > I am thankful to u .Its a great opertunity for me to > take your guidence .I was sucessfully load the oracle > in virtual ip and the script is runing .But one > problem is there the script is not runing from cluster > if i mension the path /usr/share/cluster/oracledb.sh > start.When i want to run the oracle i am using the > following command :- ./oracledb.sh start or stop or > status etc.... > How can i put the thing in cluster suite? > Lon can i install oracle rac on rhcs4?Is it possible? > Because oracle rac using ocfs. The HOWTO was written to show how to do a single instance Oracle 10g R2 database failover configuration using RHCS4. RAC is a *very* different operational model. It has its own notion of membership and quorum, which is redundant with RHCS. Oracle 10g RAC should not need RHCS to work properly or fail over - RAC should handle all of this for you if you have set it up correctly. Basically, if you have both instances of RAC running, you're done. You don't need to make it work with RHCS at all...! On a side note, I am surprised the oracledb.sh script started RAC correctly, since it doesn't start ocssd or any of the other Oracle Clusterware components... -- Lon From teigland at redhat.com Thu Mar 9 14:43:51 2006 From: teigland at redhat.com (David Teigland) Date: Thu, 9 Mar 2006 08:43:51 -0600 Subject: [Linux-cluster] gfs + nfsd crash In-Reply-To: <440FCC8A.7040803@sara.nl> References: <05AF3B6E-B88A-45CC-83E4-1353291E6347@sara.nl> <440EF9BD.4030108@sara.nl> <440FCC8A.7040803@sara.nl> Message-ID: <20060309144351.GA22258@redhat.com> On Thu, Mar 09, 2006 at 07:34:50AM +0100, Bas van der Vlies wrote: > I have checked the CVS source and found this Changelog below. > Does this fix my GFS crashes with NFS? It describes my kind of > problems. Yes, we hope so, your problem looks very similar. Dave From basv at sara.nl Thu Mar 9 15:00:48 2006 From: basv at sara.nl (Bas van der Vlies) Date: Thu, 09 Mar 2006 16:00:48 +0100 Subject: [Linux-cluster] gfs + nfsd crash In-Reply-To: <20060309144351.GA22258@redhat.com> References: <05AF3B6E-B88A-45CC-83E4-1353291E6347@sara.nl> <440EF9BD.4030108@sara.nl> <440FCC8A.7040803@sara.nl> <20060309144351.GA22258@redhat.com> Message-ID: <44104320.7060404@sara.nl> David Teigland wrote: > On Thu, Mar 09, 2006 at 07:34:50AM +0100, Bas van der Vlies wrote: >> I have checked the CVS source and found this Changelog below. >> Does this fix my GFS crashes with NFS? It describes my kind of >> problems. > > Yes, we hope so, your problem looks very similar. > I have installed the newest GFS version from cvs STABLE and did not encounter any nfsd crashes ;-). Just to inform the progress. -- -- ******************************************************************** * * * Bas van der Vlies e-mail: basv at sara.nl * * SARA - Academic Computing Services phone: +31 20 592 8012 * * Kruislaan 415 fax: +31 20 6683167 * * 1098 SJ Amsterdam * * * ******************************************************************** From hong.zheng at wsdtx.org Thu Mar 9 17:02:54 2006 From: hong.zheng at wsdtx.org (Hong Zheng) Date: Thu, 9 Mar 2006 11:02:54 -0600 Subject: [Linux-cluster] Cluster service restarting Locally Message-ID: Lon, Thanks for your reply. In my system I don't use any lock system like lock_gulm or lock_dlm, I use no_lock because our applications' limitation. Do you think no_lock will also bring some lock traffic or not? When I tried lock_gulm before, our application had very bad performance, so I choose no_lock. And I'm not sure which update we have right now. Do you know the versions for clumanager and redhat-config-cluster of RHCS3U7? Hong -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Lon Hohberger Sent: Wednesday, March 08, 2006 4:52 PM To: linux clustering Subject: RE: [Linux-cluster] Cluster service restarting Locally On Mon, 2006-03-06 at 14:02 -0600, Hong Zheng wrote: > I'm having the same problem. My system configuration is as follows: > > 2-node cluster: RH ES3, GFS6.0, clumanager-1.2.28-1 and > redhat-config-cluster-1.0.8-1 > > Kernel: 2.4.21-37.EL > > Linux-iscsi-3.6.3 initiator: connections to iSCSI shared storage > server If it's not fixed in U7 (which I think it should be), please file a bugzilla... It sounds like the lock traffic is getting network-starved. -- Lon -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From kabobofpug at yahoo.com Thu Mar 9 18:49:31 2006 From: kabobofpug at yahoo.com (paul raymond) Date: Thu, 9 Mar 2006 10:49:31 -0800 (PST) Subject: [Linux-cluster] Thank You! Trouble with RHCS 3.0 In-Reply-To: <1141858090.25169.166.camel@ayanami.boston.redhat.com> Message-ID: <20060309184931.73576.qmail@web36108.mail.mud.yahoo.com> Greetings Lon, Thank you very much in pointing me in the correct direct on this! I thought it was a rawdevice issue, but it was network issue with several wrong network settings! Yikes! We are system testing the cluster now but still have one issue: Sometimes when I stop Cluster Manager service on the member #0, with this command: "service clumanager stop", the system reboots it's self automatically! Do you have any clues why this happens? Thanks! Warm Regards, Paul Linux/AIX/Windows System Admin Lon Hohberger wrote: On Sun, 2006-03-05 at 22:22 -0800, paul raymond wrote: > Greetings Lon, Hi, sorry I'm late responding to this. > The problem is that I can not get Quorum to start unless I run the > command "cluforce"! But after viewing clustat commands on systems c11 > and c12, it looks like c11 and c12 cant see each other status due to > some issue with the raw partitions I believe? If you're using an IP tiebreaker, they won't be looking for each other on the shared partitions. The nodes communicate with each other primarily over the network - if they don't see each other, they will not form a quorum. You can try this if you want more detailed information: # service clumanager stop (on both nodes) # clumembd -fd (on both nodes) It will give you all sorts of information, but the most important one you should be looking for is: [PID] info: Membership View #1:0x00000001 If you see both nodes, it will show 0x00000003 (it's a bitmap). If the nodes can't see each other over the network, they will show 1 or 2. If this happens, you should check your network configuration and clumanager's settings - you might want to try using broadcast instead of multicast, etc. > I am using a Mylex Fiber Channel Box with QLogic 2300 interface > card! The raw devices are setup on a 2 mirror drives, Raid 1. Can you > please shed any good ideas what might be wrong here? The vidals are > below! Note that for one node to *start* without the other when using an IP tiebreaker, having to run 'cluforce' is the default behavior. If you wish to change this, please check the man page for the 'cluforce' command and the 'cludb' command. The IP tiebreaker is typically used to *maintain* a quorum after a node failure, because there are certain network faults in which two nodes may see the tiebreaker - but not each other. -- Lon --------------------------------- Yahoo! Mail Use Photomail to share photos without annoying attachments. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wcheng at redhat.com Thu Mar 9 20:32:56 2006 From: wcheng at redhat.com (Wendy Cheng) Date: Thu, 09 Mar 2006 15:32:56 -0500 Subject: [Linux-cluster] GFS load average and locking In-Reply-To: <200603090948.03134.grimme@atix.de> References: <9A6FE0FCC2B29846824C5CD81C6647B90143604F@s228130hz1ew08.apptix-01.savvis.net> <200603090948.03134.grimme@atix.de> Message-ID: <441090F8.4050308@redhat.com> Marc Grimme wrote: >Although the strace does not show the output I know of the problem description >sounds like a deja vu. >We had loads of problems with having sessions on GFS and httpd s ending up >with "D" state for some time (at high load times we had ServerLimit httpd in >D per node which ended up in the service not being available). >As I posted already we think it is because of the "bad" locking of sessions >with php (as php sessions are on gfs and strace showed those timeouts with >the session files). When you issue a "session_start" or what ever that >function is called, the session_file is locked via an flock syscall. That >lock is held until you end the session which is implicitly done when the tcp >connection to the client is ended. Now comes another http process (on >whatever node) and calls a "session start" and trys an flock on that session >while another process already holds that lock. The process might end up in >the seen timeouts (30-60secs) which (as far as I remember relates to the >timeout of the tcp connection defined in the httpd.conf or some timeout in >the php.ini) - there is an explanation on this but I cannot rember ;-) ). >Nevertheless in our scenario the problems were the "bad" session handling by >php. We have made a patch for the phplib where you can disable the locking, >or just implicitly do locking and therefore keep consitency while session >data is read or written. We could make apache work as expected and now we >don't see any "D" process anymore since a year. >Oh yes the patch can be found at >www.opensharedroot.org in the download section. > >Besides: You will never encounter this on a localfilesystem or nfs (as nfs >ignores flocks). As nfs does not support flocks and silently ignores them. > > > Hi, This does look like the problem description sent out by savvis.net folks during our off-list email exchanges. However, without actually looking at the thread traces (when they are in D state), it is difficult to be sure. One way to obtain the exact thread trace is using "crash" tool to do a back trace (e.g. "bt ", you need kernel debuginfo RPM though). Britt, do let us know whether this php patch helps and/or using crash command to obtain the thread trace output. On the other hand, I don't understand how a local (non-cluster) filesystem can be immune from this problem ? -- Wendy From tekion at gmail.com Thu Mar 9 20:40:28 2006 From: tekion at gmail.com (Screaming Eagle) Date: Thu, 9 Mar 2006 15:40:28 -0500 Subject: [Linux-cluster] GFS and extend attribute ... Message-ID: Hi, I am running GFS with Coraid. I tried using extended attribute on GFS, but it err out with (using setfacl )message: "Operation not supported". Does anyone know for sure that GFS does not support extended attribute options? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From erling.nygaard at gmail.com Thu Mar 9 21:52:14 2006 From: erling.nygaard at gmail.com (Erling Nygaard) Date: Thu, 9 Mar 2006 22:52:14 +0100 Subject: [Linux-cluster] Cluster service restarting Locally In-Reply-To: References: Message-ID: I am sorry if this sounds a little harsh, but I'm not sure if laughing or crying is the correct reaction to this email. Let us get one thing straight. You are currently mounting a GFS filesystem _concurrently_ on multiple nodes using lock_nolock? If this is the case I can tell you that this will _not_ work. You _will_ corrupt your filesystem. Mounting a GFS filesystem with lock_nolock for all practical purposes turns the GFS filesystem into a local filesystem. There is _no_ locking done anymore. With this setup there is no longer any coordination done among the nodes to control the filesystem access, so they are all going to step on each others toes. You might as well use ext3, the end result will be the same ;-) The purpose of lock_nolock is to (temporarily) be able to mount a GFS filesystem on a single node in such cases where the entire locking infrastructure is unavailable. (Something like a massive cluster failure) So you should really look into setting up one of the lock services :-) E. On 3/9/06, Hong Zheng wrote: > Lon, > > Thanks for your reply. In my system I don't use any lock system like > lock_gulm or lock_dlm, I use no_lock because our applications' > limitation. Do you think no_lock will also bring some lock traffic or > not? When I tried lock_gulm before, our application had very bad > performance, so I choose no_lock. > > And I'm not sure which update we have right now. Do you know the > versions for clumanager and redhat-config-cluster of RHCS3U7? > > Hong > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Lon Hohberger > Sent: Wednesday, March 08, 2006 4:52 PM > To: linux clustering > Subject: RE: [Linux-cluster] Cluster service restarting Locally > > On Mon, 2006-03-06 at 14:02 -0600, Hong Zheng wrote: > > I'm having the same problem. My system configuration is as follows: > > > > 2-node cluster: RH ES3, GFS6.0, clumanager-1.2.28-1 and > > redhat-config-cluster-1.0.8-1 > > > > Kernel: 2.4.21-37.EL > > > > Linux-iscsi-3.6.3 initiator: connections to iSCSI shared storage > > server > > If it's not fixed in U7 (which I think it should be), please file a > bugzilla... It sounds like the lock traffic is getting network-starved. > > -- Lon > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- - Mac OS X. Because making Unix user-friendly is easier than debugging Windows From hong.zheng at wsdtx.org Thu Mar 9 22:26:07 2006 From: hong.zheng at wsdtx.org (Hong Zheng) Date: Thu, 9 Mar 2006 16:26:07 -0600 Subject: [Linux-cluster] Cluster service restarting Locally Message-ID: I understand no_lock won't work for multiple nodes, so I never mount GFS w/ no_lock to multiple nodes, our cluster is two-node active-passive cluster. So every time only active node has GFS mount. I could use iSCSI disk only, but just want to test if GFS has better performance than iSCSI. Hong -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Erling Nygaard Sent: Thursday, March 09, 2006 3:52 PM To: linux clustering Subject: Re: [Linux-cluster] Cluster service restarting Locally I am sorry if this sounds a little harsh, but I'm not sure if laughing or crying is the correct reaction to this email. Let us get one thing straight. You are currently mounting a GFS filesystem _concurrently_ on multiple nodes using lock_nolock? If this is the case I can tell you that this will _not_ work. You _will_ corrupt your filesystem. Mounting a GFS filesystem with lock_nolock for all practical purposes turns the GFS filesystem into a local filesystem. There is _no_ locking done anymore. With this setup there is no longer any coordination done among the nodes to control the filesystem access, so they are all going to step on each others toes. You might as well use ext3, the end result will be the same ;-) The purpose of lock_nolock is to (temporarily) be able to mount a GFS filesystem on a single node in such cases where the entire locking infrastructure is unavailable. (Something like a massive cluster failure) So you should really look into setting up one of the lock services :-) E. On 3/9/06, Hong Zheng wrote: > Lon, > > Thanks for your reply. In my system I don't use any lock system like > lock_gulm or lock_dlm, I use no_lock because our applications' > limitation. Do you think no_lock will also bring some lock traffic or > not? When I tried lock_gulm before, our application had very bad > performance, so I choose no_lock. > > And I'm not sure which update we have right now. Do you know the > versions for clumanager and redhat-config-cluster of RHCS3U7? > > Hong > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Lon Hohberger > Sent: Wednesday, March 08, 2006 4:52 PM > To: linux clustering > Subject: RE: [Linux-cluster] Cluster service restarting Locally > > On Mon, 2006-03-06 at 14:02 -0600, Hong Zheng wrote: > > I'm having the same problem. My system configuration is as follows: > > > > 2-node cluster: RH ES3, GFS6.0, clumanager-1.2.28-1 and > > redhat-config-cluster-1.0.8-1 > > > > Kernel: 2.4.21-37.EL > > > > Linux-iscsi-3.6.3 initiator: connections to iSCSI shared storage > > server > > If it's not fixed in U7 (which I think it should be), please file a > bugzilla... It sounds like the lock traffic is getting network-starved. > > -- Lon > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- - Mac OS X. Because making Unix user-friendly is easier than debugging Windows -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From erling.nygaard at gmail.com Thu Mar 9 22:34:29 2006 From: erling.nygaard at gmail.com (Erling Nygaard) Date: Thu, 9 Mar 2006 23:34:29 +0100 Subject: [Linux-cluster] Cluster service restarting Locally In-Reply-To: References: Message-ID: oh, thats good to hear :-) Multiple lock_nolock nodes would be... interesting... However, you are saying you want to compare the performance of GFS with the performance of iSCSI. GFS is a filesystem, iSCSI is a block level device. May I ask how you intend to "compare" the performance of the two? Erling On 3/9/06, Hong Zheng wrote: > I understand no_lock won't work for multiple nodes, so I never mount GFS > w/ no_lock to multiple nodes, our cluster is two-node active-passive > cluster. So every time only active node has GFS mount. I could use iSCSI > disk only, but just want to test if GFS has better performance than > iSCSI. > > Hong > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Erling Nygaard > Sent: Thursday, March 09, 2006 3:52 PM > To: linux clustering > Subject: Re: [Linux-cluster] Cluster service restarting Locally > > I am sorry if this sounds a little harsh, but I'm not sure if laughing > or crying is the correct reaction to this email. > > Let us get one thing straight. > You are currently mounting a GFS filesystem _concurrently_ on multiple > nodes using lock_nolock? > > If this is the case I can tell you that this will _not_ work. You > _will_ corrupt your filesystem. > > Mounting a GFS filesystem with lock_nolock for all practical purposes > turns the GFS filesystem into a local filesystem. There is _no_ > locking done anymore. > With this setup there is no longer any coordination done among the > nodes to control the filesystem access, so they are all going to step > on each others toes. > You might as well use ext3, the end result will be the same ;-) > > The purpose of lock_nolock is to (temporarily) be able to mount a GFS > filesystem on a single node in such cases where the entire locking > infrastructure is unavailable. (Something like a massive cluster > failure) > > So you should really look into setting up one of the lock services :-) > > E. > > > > > > > On 3/9/06, Hong Zheng wrote: > > Lon, > > > > Thanks for your reply. In my system I don't use any lock system like > > lock_gulm or lock_dlm, I use no_lock because our applications' > > limitation. Do you think no_lock will also bring some lock traffic or > > not? When I tried lock_gulm before, our application had very bad > > performance, so I choose no_lock. > > > > And I'm not sure which update we have right now. Do you know the > > versions for clumanager and redhat-config-cluster of RHCS3U7? > > > > Hong > > > > -----Original Message----- > > From: linux-cluster-bounces at redhat.com > > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Lon Hohberger > > Sent: Wednesday, March 08, 2006 4:52 PM > > To: linux clustering > > Subject: RE: [Linux-cluster] Cluster service restarting Locally > > > > On Mon, 2006-03-06 at 14:02 -0600, Hong Zheng wrote: > > > I'm having the same problem. My system configuration is as follows: > > > > > > 2-node cluster: RH ES3, GFS6.0, clumanager-1.2.28-1 and > > > redhat-config-cluster-1.0.8-1 > > > > > > Kernel: 2.4.21-37.EL > > > > > > Linux-iscsi-3.6.3 initiator: connections to iSCSI shared storage > > > server > > > > If it's not fixed in U7 (which I think it should be), please file a > > bugzilla... It sounds like the lock traffic is getting > network-starved. > > > > -- Lon > > > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > -- > - > Mac OS X. Because making Unix user-friendly is easier than debugging > Windows > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- - Mac OS X. Because making Unix user-friendly is easier than debugging Windows From hong.zheng at wsdtx.org Thu Mar 9 22:45:26 2006 From: hong.zheng at wsdtx.org (Hong Zheng) Date: Thu, 9 Mar 2006 16:45:26 -0600 Subject: [Linux-cluster] Cluster service restarting Locally Message-ID: We have iSCSI external storage server and on the cluster node we use software initiator connect to iSCSI target. One way is to format that iSCSI disk to ext3, another test is to format it to GFS filesystem. I thought ext3 should be better than GFS, but the benchmark result shows GFS is better. That's what we are testing for. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Erling Nygaard Sent: Thursday, March 09, 2006 4:34 PM To: linux clustering Subject: Re: [Linux-cluster] Cluster service restarting Locally oh, thats good to hear :-) Multiple lock_nolock nodes would be... interesting... However, you are saying you want to compare the performance of GFS with the performance of iSCSI. GFS is a filesystem, iSCSI is a block level device. May I ask how you intend to "compare" the performance of the two? Erling On 3/9/06, Hong Zheng wrote: > I understand no_lock won't work for multiple nodes, so I never mount GFS > w/ no_lock to multiple nodes, our cluster is two-node active-passive > cluster. So every time only active node has GFS mount. I could use iSCSI > disk only, but just want to test if GFS has better performance than > iSCSI. > > Hong > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Erling Nygaard > Sent: Thursday, March 09, 2006 3:52 PM > To: linux clustering > Subject: Re: [Linux-cluster] Cluster service restarting Locally > > I am sorry if this sounds a little harsh, but I'm not sure if laughing > or crying is the correct reaction to this email. > > Let us get one thing straight. > You are currently mounting a GFS filesystem _concurrently_ on multiple > nodes using lock_nolock? > > If this is the case I can tell you that this will _not_ work. You > _will_ corrupt your filesystem. > > Mounting a GFS filesystem with lock_nolock for all practical purposes > turns the GFS filesystem into a local filesystem. There is _no_ > locking done anymore. > With this setup there is no longer any coordination done among the > nodes to control the filesystem access, so they are all going to step > on each others toes. > You might as well use ext3, the end result will be the same ;-) > > The purpose of lock_nolock is to (temporarily) be able to mount a GFS > filesystem on a single node in such cases where the entire locking > infrastructure is unavailable. (Something like a massive cluster > failure) > > So you should really look into setting up one of the lock services :-) > > E. > > > > > > > On 3/9/06, Hong Zheng wrote: > > Lon, > > > > Thanks for your reply. In my system I don't use any lock system like > > lock_gulm or lock_dlm, I use no_lock because our applications' > > limitation. Do you think no_lock will also bring some lock traffic or > > not? When I tried lock_gulm before, our application had very bad > > performance, so I choose no_lock. > > > > And I'm not sure which update we have right now. Do you know the > > versions for clumanager and redhat-config-cluster of RHCS3U7? > > > > Hong > > > > -----Original Message----- > > From: linux-cluster-bounces at redhat.com > > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Lon Hohberger > > Sent: Wednesday, March 08, 2006 4:52 PM > > To: linux clustering > > Subject: RE: [Linux-cluster] Cluster service restarting Locally > > > > On Mon, 2006-03-06 at 14:02 -0600, Hong Zheng wrote: > > > I'm having the same problem. My system configuration is as follows: > > > > > > 2-node cluster: RH ES3, GFS6.0, clumanager-1.2.28-1 and > > > redhat-config-cluster-1.0.8-1 > > > > > > Kernel: 2.4.21-37.EL > > > > > > Linux-iscsi-3.6.3 initiator: connections to iSCSI shared storage > > > server > > > > If it's not fixed in U7 (which I think it should be), please file a > > bugzilla... It sounds like the lock traffic is getting > network-starved. > > > > -- Lon > > > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > -- > - > Mac OS X. Because making Unix user-friendly is easier than debugging > Windows > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- - Mac OS X. Because making Unix user-friendly is easier than debugging Windows -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From Britt.Treece at savvis.net Thu Mar 9 23:04:59 2006 From: Britt.Treece at savvis.net (Treece, Britt) Date: Thu, 9 Mar 2006 17:04:59 -0600 Subject: [Linux-cluster] GFS load average and locking Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B9014836E9@s228130hz1ew08.apptix-01.savvis.net> Wendy, Unfortunately our customer has (for the time being) moved their PHP sessions off of the GFS filesystem because of the instability. Our GFS performance has returned to normal, but our customer expects us to fix GFS so that they can have the PHP sessions on GFS. I'm *attempting* to reproduce the behavior on a lab GFS cluster. Assuming I can successfully do this I will send strace's of the issue as it occurs. Is Redhat aware of any issues with GFS and flock syscalls? Regarding the U7 kernel suggestion you made previously, is this going to help with the flock issue or is it strictly for keeping the number of cached locks down? Britt -----Original Message----- From: Wendy Cheng [mailto:wcheng at redhat.com] Sent: Thursday, March 09, 2006 2:33 PM To: linux clustering Cc: Stanley, Jon; Treece, Britt Subject: Re: [Linux-cluster] GFS load average and locking Marc Grimme wrote: >Although the strace does not show the output I know of the problem description >sounds like a deja vu. >We had loads of problems with having sessions on GFS and httpd s ending up >with "D" state for some time (at high load times we had ServerLimit httpd in >D per node which ended up in the service not being available). >As I posted already we think it is because of the "bad" locking of sessions >with php (as php sessions are on gfs and strace showed those timeouts with >the session files). When you issue a "session_start" or what ever that >function is called, the session_file is locked via an flock syscall. That >lock is held until you end the session which is implicitly done when the tcp >connection to the client is ended. Now comes another http process (on >whatever node) and calls a "session start" and trys an flock on that session >while another process already holds that lock. The process might end up in >the seen timeouts (30-60secs) which (as far as I remember relates to the >timeout of the tcp connection defined in the httpd.conf or some timeout in >the php.ini) - there is an explanation on this but I cannot rember ;-) ). >Nevertheless in our scenario the problems were the "bad" session handling by >php. We have made a patch for the phplib where you can disable the locking, >or just implicitly do locking and therefore keep consitency while session >data is read or written. We could make apache work as expected and now we >don't see any "D" process anymore since a year. >Oh yes the patch can be found at >www.opensharedroot.org in the download section. > >Besides: You will never encounter this on a localfilesystem or nfs (as nfs >ignores flocks). As nfs does not support flocks and silently ignores them. > > > Hi, This does look like the problem description sent out by savvis.net folks during our off-list email exchanges. However, without actually looking at the thread traces (when they are in D state), it is difficult to be sure. One way to obtain the exact thread trace is using "crash" tool to do a back trace (e.g. "bt ", you need kernel debuginfo RPM though). Britt, do let us know whether this php patch helps and/or using crash command to obtain the thread trace output. On the other hand, I don't understand how a local (non-cluster) filesystem can be immune from this problem ? -- Wendy From lhh at redhat.com Thu Mar 9 23:09:16 2006 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 09 Mar 2006 18:09:16 -0500 Subject: [Linux-cluster] Cluster service restarting Locally In-Reply-To: References: Message-ID: <1141945756.25169.292.camel@ayanami.boston.redhat.com> On Thu, 2006-03-09 at 11:02 -0600, Hong Zheng wrote: > Lon, > > Thanks for your reply. In my system I don't use any lock system like > lock_gulm or lock_dlm, I use no_lock because our applications' > limitation. Do you think no_lock will also bring some lock traffic or > not? No, but if you mount the file system on more than one node, say "good bye" to your data. > When I tried lock_gulm before, our application had very bad > performance, so I choose no_lock. > > And I'm not sure which update we have right now. Do you know the > versions for clumanager and redhat-config-cluster of RHCS3U7? 1.2.28-1 is U6. U7 will be out soon. You can contact Red Hat Support if you want an earlier version. Another way to make things work a little better for you is to separate the cluster communication path from the iSCSI path so they're not contending for the same network. -- Lon From wcheng at redhat.com Thu Mar 9 23:21:39 2006 From: wcheng at redhat.com (Wendy Cheng) Date: Thu, 09 Mar 2006 18:21:39 -0500 Subject: [Linux-cluster] GFS load average and locking In-Reply-To: <9A6FE0FCC2B29846824C5CD81C6647B9014836E9@s228130hz1ew08.apptix-01.savvis.net> References: <9A6FE0FCC2B29846824C5CD81C6647B9014836E9@s228130hz1ew08.apptix-01.savvis.net> Message-ID: <4410B883.30608@redhat.com> Treece, Britt wrote: >Wendy, > >Unfortunately our customer has (for the time being) moved their PHP >sessions off of the GFS filesystem because of the instability. Our GFS >performance has returned to normal, but our customer expects us to fix >GFS so that they can have the PHP sessions on GFS. I'm *attempting* to >reproduce the behavior on a lab GFS cluster. Assuming I can >successfully do this I will send strace's of the issue as it occurs. > > So this problem doesn't show up in local filesystem ? Is it ext3 ? Also I prefer thread back trace in kernel mode (sysrq-t and/or crash output) to strace - since thread kernel back trace can really show where it gets stuck. If you plan to recreate this in your lab, turn the fencing off (make heart beat interval very long) so we can get a decent sysrq-t output. >Is Redhat aware of any issues with GFS and flock syscalls? > > Will check but I don't recall such issues from top of my head. >Regarding the U7 kernel suggestion you made previously, is this going to >help with the flock issue or is it strictly for keeping the number of >cached locks down? > > > The new tuning parameters added into U7 do help with several lock latency issues. Based on your lockspace output, I strongly believe they can help. However, they can't do much if the bottleneck of your customer's application is in flock as described in previous post. -- Wendy From wcheng at redhat.com Fri Mar 10 03:30:02 2006 From: wcheng at redhat.com (Wendy Cheng) Date: Thu, 09 Mar 2006 22:30:02 -0500 Subject: [Linux-cluster] GFS load average and locking In-Reply-To: <9A6FE0FCC2B29846824C5CD81C6647B9014836E9@s228130hz1ew08.apptix-01.savvis.net> References: <9A6FE0FCC2B29846824C5CD81C6647B9014836E9@s228130hz1ew08.apptix-01.savvis.net> Message-ID: <1141961403.3409.18.camel@localhost.localdomain> On Thu, 2006-03-09 at 17:04 -0600, Treece, Britt wrote: > Is Redhat aware of any issues with GFS and flock syscalls? Just checked kernel source and got a rough idea what could go wrong. In RHEL 3 (linux 2.4 based) kernel, flock has the following logic: 1. lock_kernel (Big Kernel Lock - BKL) 2. call filesystem-specific supplemental lock 3. handle linux vfs flock 4. unlock_kernel There are two issues here: * performance Step 2 is a noop for most of the local filesystems (e.g. ext3) and the code path of step 3 is relatively short. So you won't see much impacts of BKL. For GFS, if step 2 is run concurrently (as in other cases such as read, write, etc), it is reasonably "fast" unless you need the lock for the very same file and/or the lock network traffic is congested. However, adding BKL on top of that would have a big impact - it virtually serializes *every* flock attempt. * deadlock I'm a little bit fuzzy how Linux's BKL is implemented. In theory, the above sequence would get into deadlock (unless when process goes to sleep, it'll drop BKL), regardless whether step 2 is a noop or not. Will ask our base kernel folks about this. In any case, I think we need to remove that BKL if we can. At the mean time, to work around this issue, you have to either: * use previous mentioned PHP patch to turn off flock if you can; or * get GFS U7 RPMs where we have two tuning parameters that could speed up the lock process. However, I don't have quantitative data at this moment to know how effective they'll be in this kind of situation. -- Wendy From pcaulfie at redhat.com Fri Mar 10 08:51:32 2006 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Fri, 10 Mar 2006 08:51:32 +0000 Subject: [Linux-cluster] missing services In-Reply-To: <440F1CE7.2010104@arnet.net.ar> References: <440F1CE7.2010104@arnet.net.ar> Message-ID: <44113E14.9030802@redhat.com> German Staltari wrote: > Hi, we have a 6 node cluster, each one mounts 6 GFS partitions. When I > ask for the services to cman, there is always a mount point missing. Is > this correct? > FC 4 > kernel-smp-2.6.15-1.1831_FC4 > dlm-kernel-smp-2.6.11.5-20050601.152643.FC4.21 > GFS-kernel-smp-2.6.11.8-20050601.152643.FC4.24 > cman-kernel-smp-2.6.11.5-20050601.152643.FC4.22 > > TIA > German Staltari > > # df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sda1 59G 2.4G 54G 5% / > /dev/shm 2.0G 0 2.0G 0% /dev/shm > /dev/mapper/vg1-store1 399G 184K 399G 1% /store/1 > /dev/mapper/vg2-store2 399G 2.8M 399G 1% /store/2 > /dev/mapper/vg3-store3 399G 180K 399G 1% /store/3 > /dev/mapper/vg4-store4 399G 180K 399G 1% /store/4 > /dev/mapper/vg5-store5 399G 180K 399G 1% /store/5 > /dev/mapper/vg6-store6 399G 180K 399G 1% /store/6 > > # cman_tool services > Service Name GID LID State Code > Fence Domain: "default" 1 2 run - > [1 3] > DLM Lock Space: "clvmd" 7 3 run - > [1 4 3] > DLM Lock Space: "mailstore01" 20 4 run - > [1 3] > DLM Lock Space: "mailstore02" 22 6 run - > [1 3] > DLM Lock Space: "mailstore03" 24 8 run - > [1 3] > DLM Lock Space: "mailstore04" 26 10 run - > [1 3] > DLM Lock Space: "mailstore05" 28 12 run - > [1 3] > DLM Lock Space: "mailstore06" 30 14 run - > [1 3] > GFS Mount Group: "mailstore01" 21 5 run - > [1 3] > GFS Mount Group: "mailstore02" 23 7 run - > [1 3] > GFS Mount Group: "mailstore03" 25 9 run - > [1 3] > GFS Mount Group: "mailstore04" 27 11 run - > [1 3] > GFS Mount Group: "mailstore05" 29 13 run - > [1 3] > It's possible this is a (now fixed) bug in the /proc code. See https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=175372 Does "cat /proc/cluster/services" show the same thing ? -- patrick From magobin at gmail.com Fri Mar 10 09:38:22 2006 From: magobin at gmail.com (Alex aka Magobin) Date: Fri, 10 Mar 2006 10:38:22 +0100 Subject: [Linux-cluster] Strange behaviour of the services in cluster!! In-Reply-To: <1141961403.3409.18.camel@localhost.localdomain> References: <9A6FE0FCC2B29846824C5CD81C6647B9014836E9@s228130hz1ew08.apptix-01.savvis.net> <1141961403.3409.18.camel@localhost.localdomain> Message-ID: <1141983502.9580.19.camel@localhost.localdomain> Hi, I configured as first Service in cluster a DNS....it works fine and from cluster console I can move service from serverA to ServerB without problem. According with documentation I've configured Apache...exactly ! The problem is that with 2 services in cluster I'm not able to switch they from server to another server anymore...Console say thats this is an error...but there isn't any error in /var/log/messages..only a warning #70 NOTICE that if I disabled a service and then restart to other server, services run correctly...(both services), but if I want to switch on the fly...doesn't work!...It seems that Dns hung up ! any help is greatly appreciated! Alex below...tail -40 /var/log/messages trying to start DNS from serverB to ServerA while apache is running on serverA: Mar 10 10:24:57 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named status Mar 10 10:25:14 nodo2 clurgmgrd[2569]: Stopping service dns Mar 10 10:25:14 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named stop Mar 10 10:25:14 nodo2 named[13926]: shutting down: flushing changes Mar 10 10:25:14 nodo2 named: succeeded Mar 10 10:25:14 nodo2 named[13926]: stopping command channel on 127.0.0.1#953 Mar 10 10:25:14 nodo2 named[13926]: no longer listening on 127.0.0.1#53 Mar 10 10:25:14 nodo2 named[13926]: no longer listening on 10.23.5.253#53 Mar 10 10:25:14 nodo2 named[13926]: no longer listening on 10.23.5.240#53 Mar 10 10:25:14 nodo2 named[13926]: no longer listening on 192.168.254.3#53 Mar 10 10:25:14 nodo2 named[13926]: exiting Mar 10 10:25:14 nodo2 clurgmgrd: [2569]: Removing IPv4 address 10.23.5.240 from eth0 Mar 10 10:25:24 nodo2 clurgmgrd: [2569]: unmounting san:/SAN/DNS (/var/named) Mar 10 10:25:24 nodo2 clurgmgrd[2569]: Service dns is stopped Mar 10 10:25:24 nodo2 clurgmgrd[2569]: #70: Attempting to restart service dns locally. Mar 10 10:25:24 nodo2 clurgmgrd[2569]: Starting stopped service dns Mar 10 10:25:24 nodo2 clurgmgrd: [2569]: Adding IPv4 address 10.23.5.240 to eth0 Mar 10 10:25:25 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named start Mar 10 10:25:25 nodo2 named: Avvio named succeeded Mar 10 10:25:25 nodo2 named[14145]: starting BIND 9.2.4 -u named -t /var/named/chroot Mar 10 10:25:25 nodo2 named[14145]: using 1 CPU Mar 10 10:25:25 nodo2 named[14145]: loading configuration from '/etc/named.conf' Mar 10 10:25:25 nodo2 named[14145]: listening on IPv4 interface lo, 127.0.0.1#53 Mar 10 10:25:25 nodo2 named[14145]: listening on IPv4 interface eth0, 10.23.5.253#53 Mar 10 10:25:25 nodo2 named[14145]: listening on IPv4 interface eth0, 10.23.5.240#53 Mar 10 10:25:25 nodo2 named[14145]: listening on IPv4 interface eth1, 192.168.254.3#53 Mar 10 10:25:25 nodo2 clurgmgrd[2569]: Service dns started Mar 10 10:25:25 nodo2 named[14145]: command channel listening on 127.0.0.1#953 Mar 10 10:25:25 nodo2 named[14145]: zone 5.23.10.in-addr.arpa/IN: loaded serial 199609206 Mar 10 10:25:25 nodo2 named[14145]: zone 0.0.127.in-addr.arpa/IN: loaded serial 199609206 Mar 10 10:25:25 nodo2 named[14145]: zone linux.testing/IN: loaded serial 199609206 Mar 10 10:25:25 nodo2 named[14145]: running Mar 10 10:25:56 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named status Mar 10 10:26:01 nodo2 crond(pam_unix)[14196]: session opened for user root by (uid=0) Mar 10 10:26:03 nodo2 crond(pam_unix)[14196]: session closed for user root Mar 10 10:26:26 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named status Mar 10 10:26:57 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named status Mar 10 10:27:58 nodo2 last message repeated 2 times Mar 10 10:28:01 nodo2 crond(pam_unix)[14526]: session opened for user root by (uid=0) Mar 10 10:28:03 nodo2 crond(pam_unix)[14526]: session closed for user root From magobin at gmail.com Fri Mar 10 09:45:21 2006 From: magobin at gmail.com (Alex aka Magobin) Date: Fri, 10 Mar 2006 10:45:21 +0100 Subject: [Linux-cluster] Strange behaviour of the services in cluster!! Message-ID: <1141983922.9580.21.camel@localhost.localdomain> Hi, I configured as first Service in cluster a DNS....it works fine and from cluster console I can move service from serverA to ServerB without problem. According with documentation I've configured Apache...exactly ! The problem is that with 2 services in cluster I'm not able to switch they from server to another server anymore...Console say thats this is an error...but there isn't any error in /var/log/messages..only a warning #70 NOTICE that if I disabled a service and then restart to other server, services run correctly...(both services), but if I want to switch on the fly...doesn't work!...It seems that Dns hung up ! any help is greatly appreciated! Alex below...tail -40 /var/log/messages trying to start DNS from serverB to ServerA while apache is running on serverA: Mar 10 10:24:57 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named status Mar 10 10:25:14 nodo2 clurgmgrd[2569]: Stopping service dns Mar 10 10:25:14 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named stop Mar 10 10:25:14 nodo2 named[13926]: shutting down: flushing changes Mar 10 10:25:14 nodo2 named: succeeded Mar 10 10:25:14 nodo2 named[13926]: stopping command channel on 127.0.0.1#953 Mar 10 10:25:14 nodo2 named[13926]: no longer listening on 127.0.0.1#53 Mar 10 10:25:14 nodo2 named[13926]: no longer listening on 10.23.5.253#53 Mar 10 10:25:14 nodo2 named[13926]: no longer listening on 10.23.5.240#53 Mar 10 10:25:14 nodo2 named[13926]: no longer listening on 192.168.254.3#53 Mar 10 10:25:14 nodo2 named[13926]: exiting Mar 10 10:25:14 nodo2 clurgmgrd: [2569]: Removing IPv4 address 10.23.5.240 from eth0 Mar 10 10:25:24 nodo2 clurgmgrd: [2569]: unmounting san:/SAN/DNS (/var/named) Mar 10 10:25:24 nodo2 clurgmgrd[2569]: Service dns is stopped Mar 10 10:25:24 nodo2 clurgmgrd[2569]: #70: Attempting to restart service dns locally. Mar 10 10:25:24 nodo2 clurgmgrd[2569]: Starting stopped service dns Mar 10 10:25:24 nodo2 clurgmgrd: [2569]: Adding IPv4 address 10.23.5.240 to eth0 Mar 10 10:25:25 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named start Mar 10 10:25:25 nodo2 named: Avvio named succeeded Mar 10 10:25:25 nodo2 named[14145]: starting BIND 9.2.4 -u named -t /var/named/chroot Mar 10 10:25:25 nodo2 named[14145]: using 1 CPU Mar 10 10:25:25 nodo2 named[14145]: loading configuration from '/etc/named.conf' Mar 10 10:25:25 nodo2 named[14145]: listening on IPv4 interface lo, 127.0.0.1#53 Mar 10 10:25:25 nodo2 named[14145]: listening on IPv4 interface eth0, 10.23.5.253#53 Mar 10 10:25:25 nodo2 named[14145]: listening on IPv4 interface eth0, 10.23.5.240#53 Mar 10 10:25:25 nodo2 named[14145]: listening on IPv4 interface eth1, 192.168.254.3#53 Mar 10 10:25:25 nodo2 clurgmgrd[2569]: Service dns started Mar 10 10:25:25 nodo2 named[14145]: command channel listening on 127.0.0.1#953 Mar 10 10:25:25 nodo2 named[14145]: zone 5.23.10.in-addr.arpa/IN: loaded serial 199609206 Mar 10 10:25:25 nodo2 named[14145]: zone 0.0.127.in-addr.arpa/IN: loaded serial 199609206 Mar 10 10:25:25 nodo2 named[14145]: zone linux.testing/IN: loaded serial 199609206 Mar 10 10:25:25 nodo2 named[14145]: running Mar 10 10:25:56 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named status Mar 10 10:26:01 nodo2 crond(pam_unix)[14196]: session opened for user root by (uid=0) Mar 10 10:26:03 nodo2 crond(pam_unix)[14196]: session closed for user root Mar 10 10:26:26 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named status Mar 10 10:26:57 nodo2 clurgmgrd: [2569]: Executing /etc/init.d/named status Mar 10 10:27:58 nodo2 last message repeated 2 times Mar 10 10:28:01 nodo2 crond(pam_unix)[14526]: session opened for user root by (uid=0) Mar 10 10:28:03 nodo2 crond(pam_unix)[14526]: session closed for user root From adingman at cookgroup.com Fri Mar 10 13:04:57 2006 From: adingman at cookgroup.com (Andrew C. Dingman) Date: Fri, 10 Mar 2006 08:04:57 -0500 Subject: [Linux-cluster] Any recommentdations for Oracle on a Netapp filer ? In-Reply-To: <1141914937.25169.220.camel@ayanami.boston.redhat.com> References: <20060309053824.82455.qmail@web52301.mail.yahoo.com> <1141914937.25169.220.camel@ayanami.boston.redhat.com> Message-ID: <1141995897.23733.2.camel@adingman.cin.cook> On Thu, 2006-03-09 at 09:35 -0500, Lon Hohberger wrote: > On a side note, I am surprised the oracledb.sh script started RAC > correctly, since it doesn't start ocssd or any of the other Oracle > Clusterware components... Oracle's RAC installation puts the cluster system into /etc/inittab. They're probably already running when the shell script is called. -- Andrew C. Dingman Unix Administrator Cook Incorporated (812)339-2235 x2131 adingman at cookgroup.com From hong.zheng at wsdtx.org Fri Mar 10 13:37:40 2006 From: hong.zheng at wsdtx.org (Hong Zheng) Date: Fri, 10 Mar 2006 07:37:40 -0600 Subject: [Linux-cluster] Cluster service restarting Locally Message-ID: We didn't mount multiple nodes to that file system and also the iSCSI channel is in a separate subnet. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Lon Hohberger Sent: Thursday, March 09, 2006 5:09 PM To: linux clustering Subject: RE: [Linux-cluster] Cluster service restarting Locally On Thu, 2006-03-09 at 11:02 -0600, Hong Zheng wrote: > Lon, > > Thanks for your reply. In my system I don't use any lock system like > lock_gulm or lock_dlm, I use no_lock because our applications' > limitation. Do you think no_lock will also bring some lock traffic or > not? No, but if you mount the file system on more than one node, say "good bye" to your data. > When I tried lock_gulm before, our application had very bad > performance, so I choose no_lock. > > And I'm not sure which update we have right now. Do you know the > versions for clumanager and redhat-config-cluster of RHCS3U7? 1.2.28-1 is U6. U7 will be out soon. You can contact Red Hat Support if you want an earlier version. Another way to make things work a little better for you is to separate the cluster communication path from the iSCSI path so they're not contending for the same network. -- Lon -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From Britt.Treece at savvis.net Fri Mar 10 14:25:48 2006 From: Britt.Treece at savvis.net (Treece, Britt) Date: Fri, 10 Mar 2006 08:25:48 -0600 Subject: [Linux-cluster] GFS load average and locking Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B901483935@s228130hz1ew08.apptix-01.savvis.net> Wendy, Did the sysrq-t's that I sent illustrate this problem further? I'm hoping that they corroborate the situation that you described below. Britt -----Original Message----- From: Wendy Cheng [mailto:wcheng at redhat.com] Sent: Thursday, March 09, 2006 9:30 PM To: Treece, Britt Cc: linux clustering; Stanley, Jon Subject: RE: [Linux-cluster] GFS load average and locking On Thu, 2006-03-09 at 17:04 -0600, Treece, Britt wrote: > Is Redhat aware of any issues with GFS and flock syscalls? Just checked kernel source and got a rough idea what could go wrong. In RHEL 3 (linux 2.4 based) kernel, flock has the following logic: 1. lock_kernel (Big Kernel Lock - BKL) 2. call filesystem-specific supplemental lock 3. handle linux vfs flock 4. unlock_kernel There are two issues here: * performance Step 2 is a noop for most of the local filesystems (e.g. ext3) and the code path of step 3 is relatively short. So you won't see much impacts of BKL. For GFS, if step 2 is run concurrently (as in other cases such as read, write, etc), it is reasonably "fast" unless you need the lock for the very same file and/or the lock network traffic is congested. However, adding BKL on top of that would have a big impact - it virtually serializes *every* flock attempt. * deadlock I'm a little bit fuzzy how Linux's BKL is implemented. In theory, the above sequence would get into deadlock (unless when process goes to sleep, it'll drop BKL), regardless whether step 2 is a noop or not. Will ask our base kernel folks about this. In any case, I think we need to remove that BKL if we can. At the mean time, to work around this issue, you have to either: * use previous mentioned PHP patch to turn off flock if you can; or * get GFS U7 RPMs where we have two tuning parameters that could speed up the lock process. However, I don't have quantitative data at this moment to know how effective they'll be in this kind of situation. -- Wendy From wcheng at redhat.com Fri Mar 10 15:18:10 2006 From: wcheng at redhat.com (Wendy Cheng) Date: Fri, 10 Mar 2006 10:18:10 -0500 Subject: [Linux-cluster] GFS load average and locking In-Reply-To: <9A6FE0FCC2B29846824C5CD81C6647B901483935@s228130hz1ew08.apptix-01.savvis.net> References: <9A6FE0FCC2B29846824C5CD81C6647B901483935@s228130hz1ew08.apptix-01.savvis.net> Message-ID: <441198B2.1000209@redhat.com> Treece, Britt wrote: >Wendy, > >Did the sysrq-t's that I sent illustrate this problem further? I'm >hoping that they corroborate the situation that you described below. > > > Funny thing is that the sysrq-t shows the symptom we addressed in RHEL3 U7 - so look to me like a combination of serveral issues. I believe a conf. call via support has been scheduled. Let's discuss this off-list. -- Wendy From gstaltari at arnet.net.ar Fri Mar 10 15:50:14 2006 From: gstaltari at arnet.net.ar (German Staltari) Date: Fri, 10 Mar 2006 12:50:14 -0300 Subject: [Linux-cluster] missing services In-Reply-To: <44113E14.9030802@redhat.com> References: <440F1CE7.2010104@arnet.net.ar> <44113E14.9030802@redhat.com> Message-ID: <4411A036.3040200@arnet.net.ar> Patrick Caulfield wrote: > German Staltari wrote: > >> Hi, we have a 6 node cluster, each one mounts 6 GFS partitions. When I >> ask for the services to cman, there is always a mount point missing. Is >> this correct? >> FC 4 >> kernel-smp-2.6.15-1.1831_FC4 >> dlm-kernel-smp-2.6.11.5-20050601.152643.FC4.21 >> GFS-kernel-smp-2.6.11.8-20050601.152643.FC4.24 >> cman-kernel-smp-2.6.11.5-20050601.152643.FC4.22 >> >> TIA >> German Staltari >> >> # df -h >> Filesystem Size Used Avail Use% Mounted on >> /dev/sda1 59G 2.4G 54G 5% / >> /dev/shm 2.0G 0 2.0G 0% /dev/shm >> /dev/mapper/vg1-store1 399G 184K 399G 1% /store/1 >> /dev/mapper/vg2-store2 399G 2.8M 399G 1% /store/2 >> /dev/mapper/vg3-store3 399G 180K 399G 1% /store/3 >> /dev/mapper/vg4-store4 399G 180K 399G 1% /store/4 >> /dev/mapper/vg5-store5 399G 180K 399G 1% /store/5 >> /dev/mapper/vg6-store6 399G 180K 399G 1% /store/6 >> >> # cman_tool services >> Service Name GID LID State Code >> Fence Domain: "default" 1 2 run - >> [1 3] >> DLM Lock Space: "clvmd" 7 3 run - >> [1 4 3] >> DLM Lock Space: "mailstore01" 20 4 run - >> [1 3] >> DLM Lock Space: "mailstore02" 22 6 run - >> [1 3] >> DLM Lock Space: "mailstore03" 24 8 run - >> [1 3] >> DLM Lock Space: "mailstore04" 26 10 run - >> [1 3] >> DLM Lock Space: "mailstore05" 28 12 run - >> [1 3] >> DLM Lock Space: "mailstore06" 30 14 run - >> [1 3] >> GFS Mount Group: "mailstore01" 21 5 run - >> [1 3] >> GFS Mount Group: "mailstore02" 23 7 run - >> [1 3] >> GFS Mount Group: "mailstore03" 25 9 run - >> [1 3] >> GFS Mount Group: "mailstore04" 27 11 run - >> [1 3] >> GFS Mount Group: "mailstore05" 29 13 run - >> [1 3] >> >> > > It's possible this is a (now fixed) bug in the /proc code. See > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=175372 > > Does "cat /proc/cluster/services" show the same thing ? > > Yes it does. Is there a way to make updated packages for the FC4 cluster/gfs rpm's based on the RHEL4 rpm's? I'm asking this because I see some bugs fixes, that definitely need to be in the FC4 rpm's. Thanks German From kanderso at redhat.com Fri Mar 10 18:14:30 2006 From: kanderso at redhat.com (Kevin Anderson) Date: Fri, 10 Mar 2006 12:14:30 -0600 Subject: [Linux-cluster] missing services In-Reply-To: <4411A036.3040200@arnet.net.ar> References: <440F1CE7.2010104@arnet.net.ar> <44113E14.9030802@redhat.com> <4411A036.3040200@arnet.net.ar> Message-ID: <1142014470.2932.28.camel@localhost.localdomain> On Fri, 2006-03-10 at 12:50 -0300, German Staltari wrote: > > > > > Yes it does. > Is there a way to make updated packages for the FC4 cluster/gfs rpm's > based on the RHEL4 rpm's? > I'm asking this because I see some bugs fixes, that definitely need to > be in the FC4 rpm's. Which fixes? We try to make sure that everything that gets checked into RHEL4 also goes under the STABLE tag which is what FC4 gets built. If we are missing something, then we need to get the source base fixed. Kevin From epeelea at gmail.com Fri Mar 10 19:01:34 2006 From: epeelea at gmail.com (Daniel EPEE LEA) Date: Fri, 10 Mar 2006 11:01:34 -0800 Subject: [Linux-cluster] Help: Cannot mount GFS partition in cluster Message-ID: Hello, After installing application on one of my 2 node cluster, the system won't mount the GFS partition automatically as usual. - clustat shows both nodes are still in the cluster, - clvmd started correctly on both nodes, but no luch in mounting the partition - lvscan gives this error lvscan: symbol lookup error: /usr/lib/liblvm2clusterlock.so: undefined symbol: malloc_aux How can I get out of this ? Waiting for answers. Best Regards -- -------------------------- Daniel Epee Lea From e.tano at palazzochigi.it Fri Mar 10 19:12:48 2006 From: e.tano at palazzochigi.it (Tano Enzo) Date: Fri, 10 Mar 2006 20:12:48 +0100 Subject: [Linux-cluster] Resource Shared Message-ID: <6C5FB8EA05488B44B62544668AB4EE9C1BD870@PCM-EXCH-VIRT2.pcm.it> Hi, I have a cluster with 2 nodes RHEL 4 U3, my cluster not use GFS, I have a ECM2 shared storage. I have some shared resource: ip address and file system. Can I use the shared resource in my service more then one times? For example I have a file system /mnt/san/web where more then one services access to it. I have created 2 service Test and Test2 which use the only file system as resource, but only Test use it, Test2 ignore it. When Test2 become enable, it not mount the file system. Why? Thanks for help Enzo _______________________________________________________ Messaggio analizzato e protetto da tecnologia antivirus Servizio erogato dal sistema informativo della Presidenza del Consiglio dei Ministri From cjkovacs at verizon.net Fri Mar 10 22:28:47 2006 From: cjkovacs at verizon.net (Corey Kovacs) Date: Fri, 10 Mar 2006 17:28:47 -0500 Subject: [Linux-cluster] GFS load average and locking In-Reply-To: <441198B2.1000209@redhat.com> References: <9A6FE0FCC2B29846824C5CD81C6647B901483935@s228130hz1ew08.apptix-01.savvis.net> <441198B2.1000209@redhat.com> Message-ID: <1142029727.28510.0.camel@ronin.home.net> Is there a target date for the release of RHELu7? I'm expecting it will be soon since RHEL4u3 was just released. Any solid date? Regards Corey On Fri, 2006-03-10 at 10:18 -0500, Wendy Cheng wrote: > Treece, Britt wrote: > > >Wendy, > > > >Did the sysrq-t's that I sent illustrate this problem further? I'm > >hoping that they corroborate the situation that you described below. > > > > > > > Funny thing is that the sysrq-t shows the symptom we addressed in RHEL3 > U7 - so look to me like a combination of serveral issues. > > I believe a conf. call via support has been scheduled. Let's discuss > this off-list. > > -- Wendy > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From brilong at cisco.com Sat Mar 11 01:55:30 2006 From: brilong at cisco.com (Brian Long) Date: Fri, 10 Mar 2006 20:55:30 -0500 Subject: [Linux-cluster] GFS load average and locking In-Reply-To: <1142029727.28510.0.camel@ronin.home.net> References: <9A6FE0FCC2B29846824C5CD81C6647B901483935@s228130hz1ew08.apptix-01.savvis.net> <441198B2.1000209@redhat.com> <1142029727.28510.0.camel@ronin.home.net> Message-ID: <44122E12.2030704@cisco.com> Corey Kovacs wrote: >Is there a target date for the release of RHELu7? I'm expecting >it will be soon since RHEL4u3 was just released. Any solid date? > > > I was told next week. It's been pushed out a few times since it was originally due around 3/1. /Brian/ From saju8 at rediffmail.com Sat Mar 11 10:50:07 2006 From: saju8 at rediffmail.com (saju john) Date: 11 Mar 2006 10:50:07 -0000 Subject: [Linux-cluster] Cluster service restarting Locally Message-ID: <20060311105007.27577.qmail@webmail50.rediffmail.com> Dear Mr. Hohberger, Thanx for the replay. I saw your comments for the problem I reported. ie lock traffic is getting network-starved. But I think differently. Because when I stop clumanager on one of the node, the frequency of service restart is very very less compared to that was earlier when clumanager is running on both nodes .My assumption is that, the problem is due to some curruption of meta data information writing to the quroum partition ,as both nodes writing to quroum cuncurrently. May be due to bug in the rawdeivce driver.I am not sure.Then interesting question is ,how the cluster worked all these days(for me around one year with out any major problem). Could you pelase consider this also when releasing the RHCS3U7. Thank You, Saju John Linux System Administrator, Thuraya Satellite Telicommunications Company UAE,Sharjah On Thu, 09 Mar 2006 Lon Hohberger wrote : >On Mon, 2006-03-06 at 06:47 +0000, saju john wrote: > > > > > > Dear All, > > > > I have a 2 node cluster with RHAS3 update 3. > > Kernel : 2.4.21-20.Elsmp > > Clumanager : clumanager-1.2.16-1 > > > > For more than a year everyting had been fine. Suddenly it started > > showing the follwing and restarted the service locally > > > > clusvcmgrd[1388]: Unable to obtain cluster lock: Connection > > timed out > > clulockd[1378]: Denied A.B.C.D: Broken pipe > > clulockd[1378]: select error: Broken pipe > > clusvcmgrd: [1625]: service notice: Stopping service > > postgresql ... > > clusvcmgrd: [1625]: service notice: Running user script > > '/etc/init.d/postgresql stop' > > clusvcmgrd: [1625]: service notice: Stopped service > > postgresql > > clusvcmgrd: [1625]: service notice: Starting service > > postgresql ... > > clusvcmgrd: [1625]: service notice: Running user script > > '/etc/init.d/postgresql start' > > clusvcmgrd: [1625]: service notice: Started service > > postgresql ... > >It should be fixed in RHCS3U7 > >-- Lon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alban.crequy at seanodes.com Mon Mar 13 09:46:40 2006 From: alban.crequy at seanodes.com (Alban Crequy) Date: Mon, 13 Mar 2006 10:46:40 +0100 Subject: [Linux-cluster] GFS locks granularity (DLM or GULM) Message-ID: <44153F80.2060302@seanodes.com> Hello, What is the locking granularity in GFS? Can GFS do range locks? Is the granularity of DLM different than the GULM one? The only explanation I found are: ?Locking in GFS is closely tied to physical storage. Earlier versions of GFS [21] required locking to be implemented at the disk device via extensions to the SCSI protocol. Newer versions allow the use of an external distributed lock manager, but still lock individual disk blocks of 4kB or 8kB size. Therefore, accessing large files in GFS entails significantly more locking overhead than the byte-range locks used in GPFS.? http://www.broadcastpapers.com/asset/IBMGPFS07.htm But maybe this is outdated? Other doc: ?GFS has a couple pf locks for each file. (one for data, one for meta data, one for iopen counts. maybe others, don't recall off the top of my head.) Directories get a lock, as well as most of the interal structures. So more-or-less gfs locks at the file level. (note that this is not the same or similar to fcntl locking, nor is it compatible.)? http://www.redhat.com/archives/linux-cluster/2005-June/msg00016.html -- Alban From Alain.Moulle at bull.net Mon Mar 13 10:38:36 2006 From: Alain.Moulle at bull.net (Alain Moulle) Date: Mon, 13 Mar 2006 11:38:36 +0100 Subject: [Linux-cluster] CS4 behavior on killall -9 Message-ID: <44154BAC.1070007@bull.net> Hi On a HA pair in mutual takeover, it seems that if we do a "killall -9" on one node, there is no failover, the CS4 seems to be stalled . Any reason ? idea ? Thanks Alain -- mailto:Alain.Moulle at bull.net +------------------------------+--------------------------------+ | Alain Moull? | from France : 04 76 29 75 99 | | | FAX number : 04 76 29 72 49 | | Bull SA | | | 1, Rue de Provence | Adr : FREC B1-041 | | B.P. 208 | | | 38432 Echirolles - CEDEX | Email: Alain.Moulle at bull.net | | France | BCOM : 229 7599 | +-------------------------------+-------------------------------+ From magobin at gmail.com Mon Mar 13 12:12:18 2006 From: magobin at gmail.com (Alessandro Binarelli) Date: Mon, 13 Mar 2006 13:12:18 +0100 Subject: [Linux-cluster] Where to study Cluster suite ?? Message-ID: <108b923c0603130412q610e2908q@mail.gmail.com> Hi, I 've some problem (basic problem) with cluster suite that is not cover from documentation...so I would to know if there are some site that explain step by step a basic installation and configuration . As I said in my previous message I configured dns and http service but once installed I'm not able to move services from serverA to serverB...only if I disabled it and restart on other server....plus....I try to disable ethernet card on serverB and I thought that service moves to other server automatically when it see server died...but doesnt' work for me.. So I think that there is some steps that I have to know before trying to configure cluster in HA....is there some site that explain Redhat Cluster Suite? Thanks in advance Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From updatemyself at gmail.com Mon Mar 13 13:00:23 2006 From: updatemyself at gmail.com (updatemyself .) Date: Mon, 13 Mar 2006 18:30:23 +0530 Subject: [Linux-cluster] Few Doubts About "GFS + ISCSI with Multipathing And NIC Bonding" Message-ID: Hai All, I have few questions to ask.. i already have a setup of GPFS Cluster on SAN with mulipathing (total 12 TB Volumes) And planning to go for a Another One With GFS + ISCSI with Multipathing And NIC Bonding So my doubt are about 1, Multipathing 2, NIC Bonding 3, Whats the Option for ISCSI Multipath same as RDAC in SAN 4, Comparison of GPFS on SAN and GFS with ISCSI (which is better ) 5, Is it needed to go for Redhat AS or Fedora Core 4 is enough? Thank You In Advance, jerrynikky -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.bridgwater at sinergy.it Mon Mar 13 13:27:48 2006 From: s.bridgwater at sinergy.it (Simon Bridgwater) Date: Mon, 13 Mar 2006 14:27:48 +0100 Subject: [Linux-cluster] problem with vip addresses Message-ID: I seem to have a problem with the vip's in a bonding interface disappearing. At first I thought it was a problem with the vsftpd service but when I closely monitored the vip with the command "ip addr show bond0" I sometimes see that the virtual ip's disappear and then reappear after a few seconds. What could be causing this problem ? Is it an operating system problem or a cluster problem. Could it be caused by a malconfigured service ? (I have to put two scripts which depend on each other into a single service). I have tried upgrading the cluster suite (with corresponding kernel-smp- 2.6.9-22.0.2) to the most updated version but it is still giving me this error. I have two e1000 NICS configured in bonding in active-backup with miimon=100. Could it be a problem with bonding or the miimon parameter ? Simon Bridgwater Sinergy Srl -------------- next part -------------- An HTML attachment was scrubbed... URL: From brilong at cisco.com Mon Mar 13 13:36:33 2006 From: brilong at cisco.com (Brian Long) Date: Mon, 13 Mar 2006 08:36:33 -0500 Subject: [Linux-cluster] Few Doubts About "GFS + ISCSI with Multipathing And NIC Bonding" In-Reply-To: References: Message-ID: <1142256993.4566.6.camel@brilong-lnx> On Mon, 2006-03-13 at 18:30 +0530, updatemyself . wrote: > Hai All, > > I have few questions to ask.. > i already have a setup of GPFS Cluster on SAN with mulipathing (total > 12 TB Volumes) > And planning to go for a Another One With GFS + ISCSI with > Multipathing And NIC Bonding > > So my doubt are about > 1, Multipathing > 2, NIC Bonding > 3, Whats the Option for ISCSI Multipath same as RDAC in SAN > 4, Comparison of GPFS on SAN and GFS with ISCSI (which is better ) > 5, Is it needed to go for Redhat AS or Fedora Core 4 is enough? Jerrynikky, I can answer #5 easily. Do you require Enterprise-level support for your implementation or are you just setting this up to play around? If you require Enterprise support with a vendor's throat to choke when something dies, you absolutely need to pursue Red Hat AS or ES. /Brian/ -- Brian Long | | | IT Data Center Systems | .|||. .|||. Cisco Linux Developer | ..:|||||||:...:|||||||:.. Phone: (919) 392-7363 | C i s c o S y s t e m s From updatemyself at gmail.com Mon Mar 13 14:05:05 2006 From: updatemyself at gmail.com (updatemyself .) Date: Mon, 13 Mar 2006 19:35:05 +0530 Subject: [Linux-cluster] Few Doubts About "GFS + ISCSI with Multipathing And NIC Bonding" In-Reply-To: <1142256993.4566.6.camel@brilong-lnx> References: <1142256993.4566.6.camel@brilong-lnx> Message-ID: Thank You Brian, That i know i already having 5 Enterprise Licence i mean only about modules.. and stability... who can help me.. to get all other information... Yhanks a lot.. Jerrynikky. On 3/13/06, Brian Long wrote: > > On Mon, 2006-03-13 at 18:30 +0530, updatemyself . wrote: > > Hai All, > > > > I have few questions to ask.. > > i already have a setup of GPFS Cluster on SAN with mulipathing (total > > 12 TB Volumes) > > And planning to go for a Another One With GFS + ISCSI with > > Multipathing And NIC Bonding > > > > So my doubt are about > > 1, Multipathing > > 2, NIC Bonding > > 3, Whats the Option for ISCSI Multipath same as RDAC in SAN > > 4, Comparison of GPFS on SAN and GFS with ISCSI (which is better ) > > 5, Is it needed to go for Redhat AS or Fedora Core 4 is enough? > > Jerrynikky, > > I can answer #5 easily. Do you require Enterprise-level support for > your implementation or are you just setting this up to play around? If > you require Enterprise support with a vendor's throat to choke when > something dies, you absolutely need to pursue Red Hat AS or ES. > > /Brian/ > -- > Brian Long | | | > IT Data Center Systems | .|||. .|||. > Cisco Linux Developer | ..:|||||||:...:|||||||:.. > Phone: (919) 392-7363 | C i s c o S y s t e m s > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhh at redhat.com Mon Mar 13 16:38:07 2006 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 13 Mar 2006 11:38:07 -0500 Subject: [Linux-cluster] Resource Shared In-Reply-To: <6C5FB8EA05488B44B62544668AB4EE9C1BD870@PCM-EXCH-VIRT2.pcm.it> References: <6C5FB8EA05488B44B62544668AB4EE9C1BD870@PCM-EXCH-VIRT2.pcm.it> Message-ID: <1142267887.15119.28.camel@ayanami.boston.redhat.com> On Fri, 2006-03-10 at 20:12 +0100, Tano Enzo wrote: > Hi, > > I have a cluster with 2 nodes RHEL 4 U3, my cluster not use GFS, I have > a ECM2 shared storage. I have some shared resource: ip address and file > system. Can I use the shared resource in my service more then one times? > For example I have a file system /mnt/san/web where more then one > services access to it. I have created 2 service Test and Test2 which use > the only file system as resource, but only Test use it, Test2 ignore it. You can not reference an IP or a regular "fs" (ext3, ext2, reiserfs, etc) multiple times, because mounting those types of file systems on two systems is an invitation for a corrupt file system. Similarly, bringing up a single IP on two separate systems ... well, generally does not work well ;) Currently, you can reuse: - clusterfs (i.e. GFS; could be extended for other cluster file systems though) - netfs (mounting a file system from an NFS server - locks + file system consistency is handled server-side) - nfsexport (Meta-resource which is a child of fs or clusterfs to help with creation of an NFS failover service) - nfsclient (Resource which describes a target of a failover NFS service... hostname, wildcard, etc.) - script (though this should be done with caution!) Reusing non-shareable resources causes some of the references to be ignored. If you want to see what rgmanager thinks about the resource tree, run: rg_test test /etc/cluster/cluster.conf 2>&1 | less It will tell you of problems it finds in the resource tree (like exceeding max reference counts for resources, etc.) -- Lon From dex.chen at crosswalkinc.com Mon Mar 13 17:31:18 2006 From: dex.chen at crosswalkinc.com (Dex Chen) Date: Mon, 13 Mar 2006 10:31:18 -0700 Subject: [Linux-cluster] lost quorum, but the cluster services and GFS are still up Message-ID: <2E02749DAF5338479606A056219BE109E0DB42@smail.crosswalkinc.com> Hi, I believe that I saw something unusual here. I have a 3 node cluster (with GFS) using CMAN. After I shutdown 2 nodes in short time span, the cluster shows it lost quorum, but I run the clustat on the third node, and clustat shows the cluster has 3 nodes (2 are offline) and the other services are up. I was able to access/read the share storage. CMAN_TOOL shows cluster lost quorum and the activity is blocked. What I expected is that I should not allow accessing the shared storage and other services at all when the cluster lost the quorum. Anyone has seen the similar things? What/where should I look into? Thanks, Dex -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhh at redhat.com Mon Mar 13 18:35:59 2006 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 13 Mar 2006 13:35:59 -0500 Subject: [Linux-cluster] lost quorum, but the cluster services and GFS are still up In-Reply-To: <2E02749DAF5338479606A056219BE109E0DB42@smail.crosswalkinc.com> References: <2E02749DAF5338479606A056219BE109E0DB42@smail.crosswalkinc.com> Message-ID: <1142274959.15119.114.camel@ayanami.boston.redhat.com> On Mon, 2006-03-13 at 10:31 -0700, Dex Chen wrote: > Hi, > > > > I believe that I saw something unusual here. > > > > I have a 3 node cluster (with GFS) using CMAN. After I shutdown 2 > nodes in short time span, the cluster shows it lost quorum, but I run > the clustat on the third node, and clustat shows the cluster has 3 > nodes (2 are offline) and the other services are up. I was able to > access/read the share storage. CMAN_TOOL shows cluster lost quorum and > the activity is blocked. What I expected is that I should not allow > accessing the shared storage and other services at all when the > cluster lost the quorum. Anyone has seen the similar things? > What/where should I look into? CMAN is supposed to deliver (more or less) a STATECHANGE event to clients. At that point, quorum is checked by rgmanager, and if the cluster is no longer quorate, it halts all services immediately. Are there anything in the logs which would indicate this? It would look like: #1 Quorum Dissolved Given that you can still access service data (e.g. clustat reports something), that means that rgmanager can still acquire locks for some reason (it takes DLM locks before giving out service data...). Does clustat report that the cluster is quorate or not? -- Lon From lhh at redhat.com Mon Mar 13 18:38:18 2006 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 13 Mar 2006 13:38:18 -0500 Subject: [Linux-cluster] problem with vip addresses In-Reply-To: References: Message-ID: <1142275098.15119.117.camel@ayanami.boston.redhat.com> On Mon, 2006-03-13 at 14:27 +0100, Simon Bridgwater wrote: > I seem to have a problem with the vip's in a bonding interface > disappearing. At first I thought it was a problem with the vsftpd > service but when I closely monitored the vip with the command "ip addr > show bond0" I sometimes see that the virtual ip's disappear and then > reappear after a few seconds. What could be causing this problem ? Is > it an operating system problem or a cluster problem. Could it be > caused by a malconfigured service ? (I have to put two scripts which > depend on each other into a single service). How often does this happen, and is rgmanager the thing tearing down / restarting the IPs? > I have tried upgrading the cluster suite (with corresponding > kernel-smp- 2.6.9-22.0.2) to the most updated version but it is still > giving me this error. I have two e1000 NICS configured in bonding in > active-backup with miimon=100. Could it be a problem with bonding or > the miimon parameter ? There are some odd problems with e1000 in bonding configuration and the SIOCGIFCONF ioctls where sometimes, the ioctl() just returns nothing, but I was under the impression this kind of problem did not occur with the newer Netlink interfaces (e.g. what /sbin/ip uses). -- Lon From dex.chen at crosswalkinc.com Mon Mar 13 18:46:34 2006 From: dex.chen at crosswalkinc.com (Dex Chen) Date: Mon, 13 Mar 2006 11:46:34 -0700 Subject: [Linux-cluster] lost quorum, but the cluster services and GFSare still up Message-ID: <2E02749DAF5338479606A056219BE109E0DBA7@smail.crosswalkinc.com> Odd enough! Clustat still reports "Inquorate". See the screen capture: Member Status: Inquorate Member Name Status ------ ---- ------ c01 Offline c02 Offline c03 Online, Local, rgmanager Service Name Owner (Last) State ------- ---- ----- ------ ----- c-mgmt c03 started snapshot c03 started email_notifier c03 started Thanks, Dex -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Lon Hohberger Sent: Monday, March 13, 2006 11:36 AM To: linux clustering Subject: Re: [Linux-cluster] lost quorum, but the cluster services and GFSare still up On Mon, 2006-03-13 at 10:31 -0700, Dex Chen wrote: > Hi, > > > > I believe that I saw something unusual here. > > > > I have a 3 node cluster (with GFS) using CMAN. After I shutdown 2 > nodes in short time span, the cluster shows it lost quorum, but I run > the clustat on the third node, and clustat shows the cluster has 3 > nodes (2 are offline) and the other services are up. I was able to > access/read the share storage. CMAN_TOOL shows cluster lost quorum and > the activity is blocked. What I expected is that I should not allow > accessing the shared storage and other services at all when the > cluster lost the quorum. Anyone has seen the similar things? > What/where should I look into? CMAN is supposed to deliver (more or less) a STATECHANGE event to clients. At that point, quorum is checked by rgmanager, and if the cluster is no longer quorate, it halts all services immediately. Are there anything in the logs which would indicate this? It would look like: #1 Quorum Dissolved Given that you can still access service data (e.g. clustat reports something), that means that rgmanager can still acquire locks for some reason (it takes DLM locks before giving out service data...). Does clustat report that the cluster is quorate or not? -- Lon -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From lhh at redhat.com Mon Mar 13 19:44:40 2006 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 13 Mar 2006 14:44:40 -0500 Subject: [Linux-cluster] CS4 behavior on killall -9 In-Reply-To: <44154BAC.1070007@bull.net> References: <44154BAC.1070007@bull.net> Message-ID: <1142279080.15119.119.camel@ayanami.boston.redhat.com> On Mon, 2006-03-13 at 11:38 +0100, Alain Moulle wrote: > Hi > On a HA pair in mutual takeover, it seems that if we do a "killall -9" on one > node, there is no failover, the CS4 seems to be stalled . > Any reason ? idea ? Killall -9 on what specifically...? It sounds like a bug. -- Lon From teigland at redhat.com Mon Mar 13 22:24:03 2006 From: teigland at redhat.com (David Teigland) Date: Mon, 13 Mar 2006 16:24:03 -0600 Subject: [Linux-cluster] lost quorum, but the cluster services and GFS are still up In-Reply-To: <2E02749DAF5338479606A056219BE109E0DB42@smail.crosswalkinc.com> References: <2E02749DAF5338479606A056219BE109E0DB42@smail.crosswalkinc.com> Message-ID: <20060313222403.GA17640@redhat.com> On Mon, Mar 13, 2006 at 10:31:18AM -0700, Dex Chen wrote: > Hi, > > I believe that I saw something unusual here. > > I have a 3 node cluster (with GFS) using CMAN. After I shutdown 2 nodes > in short time span, the cluster shows it lost quorum, but I run the > clustat on the third node, and clustat shows the cluster has 3 nodes (2 > are offline) and the other services are up. I was able to access/read > the share storage. CMAN_TOOL shows cluster lost quorum and the activity > is blocked. What I expected is that I should not allow accessing the > shared storage and other services at all when the cluster lost the > quorum. Anyone has seen the similar things? What/where should I look > into? Quorum is the normal method of preventing an instance of some cluster subsystem or application (a gfs mount-group, a dlm lock-space, an rgmanager service/app/resource, etc) from being enabled on both sides of a partitioned cluster. It does this by preventing the creation of new instances in inquorate clusters and by preventing recovery (re-enabling) of existing instances in inquorate clusters. There's one special case where we also rely on fencing to prevent an instance from being enabled on both sides of a split at once. It's where all the nodes using the instance before the failure/partition, also exist on the inquorate side of the split afterward. If a quorate partition then forms, the first thing it does is fence all nodes it can't talk with, which are the nodes on the inquorate side. The quorate side then enables instances of dlm/gfs/etc, the fencing having guaranteed there are none elsewhere. Apart from this, each service/instance/system responds internally to the loss of quorum in its own way. In the special case I described where all the nodes using the instance remain after the event, dlm and gfs both continue to run normally on the inquorate nodes; there's been no reason to do otherwise. I suspect what you saw is that nodes A and B failed/shutdown but weren't using any of the dlm/gfs instances that C was. C was then this special case and dlm/gfs continued to run normally. If A and B had come back and formed a partitioned, quorate cluster, they would have fenced C before enabling any dlm or gfs instances. Dave From michaelc at cs.wisc.edu Tue Mar 14 00:11:54 2006 From: michaelc at cs.wisc.edu (Mike Christie) Date: Mon, 13 Mar 2006 18:11:54 -0600 Subject: [Linux-cluster] Few Doubts About "GFS + ISCSI with Multipathing And NIC Bonding" In-Reply-To: References: Message-ID: <44160A4A.6060609@cs.wisc.edu> updatemyself . wrote: > Hai All, > > I have few questions to ask.. > i already have a setup of GPFS Cluster on SAN with mulipathing (total 12 > TB Volumes) > And planning to go for a Another One With GFS + ISCSI with Multipathing > And NIC Bonding > > So my doubt are about > 1, Multipathing > 2, NIC Bonding For iscsi in linux you can use network bonding or dm-multipath or maybe even both :) For example, you can use bonding on the initiator over multiple host NICs, and use dm-multipath to multpath over multiple target portals. > 3, Whats the Option for ISCSI Multipath same as RDAC in SAN Not exactly sure what you mean by this. Are you thinking about RDAC as in Engenio's RDAC where we might need to do some sort of manual failover? From afletdinov at mail.dc.baikal.ru Tue Mar 14 00:20:04 2006 From: afletdinov at mail.dc.baikal.ru (Afletdinov A.R.) Date: Tue, 14 Mar 2006 08:20:04 +0800 Subject: [Linux-cluster] GFS and extend attribute ... In-Reply-To: References: Message-ID: <44160C34.2060808@mail.dc.baikal.ru> Screaming Eagle wrote: > Hi, > I am running GFS with Coraid. I tried using extended attribute on GFS, > but it err out with (using setfacl )message: "Operation not > supported". Does anyone know for sure that GFS does not support > extended attribute options? Thanks. > >------------------------------------------------------------------------ > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=182066 From saju8 at rediffmail.com Tue Mar 14 03:58:41 2006 From: saju8 at rediffmail.com (saju john) Date: 14 Mar 2006 03:58:41 -0000 Subject: [Linux-cluster] Cluster service restarting Locally Message-ID: <20060314035841.21034.qmail@webmail8.rediffmail.com> Dear Mr. Hohberger Thanks for the replay. When running only one node the frequency of restart is very less, but it happens with the same symtoms The machines are HP DL380G3 (2 node) with MSA SAN 1000 storage. The load average is around 4. The cluster is primarly for running postgresql database of around 116 GB size Saju John On Mon, 13 Mar 2006 Lon Hohberger wrote : >On Sat, 2006-03-11 at 10:50 +0000, saju john wrote: > > > > Dear Mr. Hohberger, > > > > Thanx for the replay. > > > > I saw your comments for the problem I reported. ie lock traffic is > > getting network-starved. > >It could be getting I/O starved too, which might explain more given that >this seems to happen on one node. When running just one node and the >service restarts, are the symptoms the same? Does it report these kinds >of errors, or are they different? > >[quote from your previous mail] >clusvcmgrd[1388]: Unable to obtain cluster lock: Connection >timed out >clulockd[1378]: Denied A.B.C.D: Broken pipe >clulockd[1378]: select error: Broken pipe >[/quote] > >If they're different in the one-node case, what are the errors? Also, >are there any other errors in the logs? > > > > My assumption is that, the problem is due to some curruption of meta > > data information writing to the quroum partition ,as both nodes > > writing to quroum cuncurrently. > >I really doubt that. In the case of lock information, only one node >writes at a time anyway... > > > May be due to bug in the rawdeivce driver.I am not sure.Then > > interesting question is ,how the cluster worked all these days(for me > > around one year with out any major problem). > >The odds of random, block-level corruption going undetected when reading > from the raw partitions is low - between (2^32):1 and (2^96):1 against >per block, based on internal consistency checks that clumanager >performs. My math might be a little off, but it requires two randomly >correct 32-bit magic numbers and one randomly valid 32-bit CRC, with >other data incorrect to cause a problem. > >Specifically in the lock case, a lock block which passed all of the >consistency checks but was *actually* corrupt would almost always cause >clulockd to crash. > >Timeout errors mean that clulockd didn't respond to a request in a given >amount of time, and can be caused by either network saturation or poor >raw I/O performance to shared storage. It looks like it's getting to an >incoming request too late... > > > > Could you pelase consider this also when releasing the RHCS3U7. > >If this is a critical issue for you, then you should file a ticket with >Red Hat Support if you have not already done so: > > http://www.redhat.com/apps/support/ > >If you think this is a bug, you can also file a Bugzilla, and we will >get to it when we can: > > http://bugzilla.redhat.com/bugzilla/ > >-- Lon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Birger.Wathne at ift.uib.no Tue Mar 14 08:31:42 2006 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Tue, 14 Mar 2006 09:31:42 +0100 Subject: [Linux-cluster] samba on gfs Message-ID: <44167F6E.4000401@ift.uib.no> What is the problem with running samba on GFS, and when will it be resolved? I have seen a hint from lon here that running samba on GFS isn't possible right now. I have a 2-node cluster running NFS services from GFS, and would like to dedicate one node for NFS, the other for samba (running from the same filesystems). I guess I could do it by NFS mounting from the NFS node, but that kind of defeats the purpose of moving the samba services into the cluster... Btw: These nodes currently have Gb interfaces for the public networks, but only a 10Mb private network. Is that enough, or should I upgrade the private network when I start using both nodes actively? -- birger From zeebala at yahoo.com Tue Mar 14 09:49:47 2006 From: zeebala at yahoo.com (bala) Date: Tue, 14 Mar 2006 01:49:47 -0800 (PST) Subject: [Linux-cluster] samba on gfs In-Reply-To: <44167F6E.4000401@ift.uib.no> Message-ID: <20060314094947.64549.qmail@web36510.mail.mud.yahoo.com> hi guys iam new to linux cluster suite and gfs iam very eager to learn at present iam having celeron 400 mhz processor and i think within month i will get new system with good configuration how can i implement rhcs and gfs Birger Wathne wrote: What is the problem with running samba on GFS, and when will it be resolved? I have seen a hint from lon here that running samba on GFS isn't possible right now. I have a 2-node cluster running NFS services from GFS, and would like to dedicate one node for NFS, the other for samba (running from the same filesystems). I guess I could do it by NFS mounting from the NFS node, but that kind of defeats the purpose of moving the samba services into the cluster... Btw: These nodes currently have Gb interfaces for the public networks, but only a 10Mb private network. Is that enough, or should I upgrade the private network when I start using both nodes actively? -- birger -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster --------------------------------- Relax. Yahoo! Mail virus scanning helps detect nasty viruses! -------------- next part -------------- An HTML attachment was scrubbed... URL: From erling.nygaard at gmail.com Tue Mar 14 10:05:52 2006 From: erling.nygaard at gmail.com (Erling Nygaard) Date: Tue, 14 Mar 2006 11:05:52 +0100 Subject: [Linux-cluster] samba on gfs In-Reply-To: <44167F6E.4000401@ift.uib.no> References: <44167F6E.4000401@ift.uib.no> Message-ID: Birger The short story is that Samba keeps some state information internally. So there are issues with keeping multiple Samba serves in sync. The information in question is not synced to the underlying filesystem, so GFS can't really do the job of keeping this info in sync between the nodes. I am sure other people on the list can provide more details of the problem and status of any progress :-) Erling On 3/14/06, Birger Wathne wrote: > What is the problem with running samba on GFS, and when will it be resolved? > I have seen a hint from lon here that running samba on GFS isn't > possible right now. > I have a 2-node cluster running NFS services from GFS, and would like to > dedicate one node for NFS, the other for samba (running from the same > filesystems). > > I guess I could do it by NFS mounting from the NFS node, but that kind > of defeats the purpose of moving the samba services into the cluster... > > Btw: These nodes currently have Gb interfaces for the public networks, > but only a 10Mb private network. Is that enough, or should I upgrade the > private network when I start using both nodes actively? > > -- > birger > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- - Mac OS X. Because making Unix user-friendly is easier than debugging Windows From l.dardini at comune.prato.it Tue Mar 14 12:53:06 2006 From: l.dardini at comune.prato.it (Leandro Dardini) Date: Tue, 14 Mar 2006 13:53:06 +0100 Subject: [Linux-cluster] Cluster @hostname link utilities Message-ID: <404AA6666D14D14CA0D410C1BC6CC4C53FC49C@exchange3.comune.prato.local> Hi, We all know the use of programs like "cat" or "tail" to show file contents. I have a cluster of apache and each httpd logs to /var/log/httpd/access.log where /var/log is a link to a gfs filesystem [root at apache1]# ls -la /var/log/httpd lrwxrwxrwx 1 root root 20 27 feb 15:44 /var/log/httpd -> /gfsvolume/log/httpd_local/@hostname For each apache I have a directory, like /gfsvolume/log/httpd_local/apache1 /gfsvolume/log/httpd_local/apache2 I'd like to show on screen, like with a cat program, all access.log files reading them from each directory in cronological order. Is there an already made program to do this? I need something as follow: If /gfsvolume/log/httpd_local/apache1/access.log contains something like the following: 127.0.0.1 - - [14/Mar/2006:13:27:26 +0100] "GET /ocsinventory/deploy/label HTTP/1.0" 500 616 "-" "NSISDL/1.2" 192.168.26.163 - - [14/Mar/2006:13:27:41 +0100] "POST /ocsinventory HTTP/1.1" 200 83 "-" "OCS-NG_windows_client_v4014" And /gfsvolume/log/httpd_local/apache2.comune.prato.it/access.log contains something like the following: 192.168.26.163 - - [14/Mar/2006:13:27:23 +0100] "GET /ocsinventory/deploy/label HTTP/1.0" 500 616 "-" "NSISDL/1.2" 192.168.1.110 - - [14/Mar/2006:13:27:42 +0100] "GET /ocsinventory/deploy/ocsagent.exe HTTP/1.0" 500 616 "-" "NSISDL/1.2" With "multicat" (the program I am looking for, just invented a name to refer to it) multicat /var/log/httpd/acces.log I obtain the following: 192.168.26.163 - - [14/Mar/2006:13:27:23 +0100] "GET /ocsinventory/deploy/label HTTP/1.0" 500 616 "-" "NSISDL/1.2" 127.0.0.1 - - [14/Mar/2006:13:27:26 +0100] "GET /ocsinventory/deploy/label HTTP/1.0" 500 616 "-" "NSISDL/1.2" 192.168.26.163 - - [14/Mar/2006:13:27:41 +0100] "POST /ocsinventory HTTP/1.1" 200 83 "-" "OCS-NG_windows_client_v4014" 192.168.1.110 - - [14/Mar/2006:13:27:42 +0100] "GET /ocsinventory/deploy/ocsagent.exe HTTP/1.0" 500 616 "-" "NSISDL/1.2" I understand there be some configuration file that map /var/log/httpd/access.log to /gfsvolume/log/httpd_local/apache1/access.log /gfsvolume/log/httpd_local/apache2/access.log And inform the program about the format and the position of the sorting key. Leandro From sgray at bluestarinc.com Tue Mar 14 14:31:05 2006 From: sgray at bluestarinc.com (Sean Gray) Date: Tue, 14 Mar 2006 09:31:05 -0500 Subject: [Linux-cluster] RHEL+RAC+GFS Message-ID: <4416D3A9.5080908@bluestarinc.com> All: Does anyone have a working combination of Oracle RAC (9i for E-Business Suite) on RHEL (4 preferably) using GFS for the DB nodes? RedHat's whitepaper suggests this combination as optimal. However, in my experience I have not yet found a magic combination that works well. I keep have hi load issues with dlmsendd. Sean -- Sean N. Gray Director of Information Technology United Radio Incorporated, DBA BlueStar 24 Spiral Drive Florence, Kentucky 41042 office: 859.371.4423 x3263 toll free: 800.371.4423 x3263 fax: 859.371.4425 mobile: 513.616.3379 From orcl.listas at gmail.com Tue Mar 14 18:27:17 2006 From: orcl.listas at gmail.com (Allyson - Listas) Date: Tue, 14 Mar 2006 15:27:17 -0300 Subject: [Linux-cluster] rhcs doubts. Message-ID: <44170B05.20805@gmail.com> Hi guys, I'm new at redhat cluster suite. Could Anybody help me in some questions? 1st) I installed rhcs on 2 virtual machines and create a new cluster, setup a manual fence, a failvoer domain, create a IP resource and a service that uses just that IP for tests. Well, I'd like to know how can I force a failover of the service between nodes. This option is not available at system-config-cluster that allow just disable and enable the service. I noticed that the ip service created is not a virtual interface like eth0:1, but it was working because I could ping it, Is it Normal? 2nd) What is the real fuction of a fence device? 3rd) How can I setup a quorum device, and isn't necessary for a failover service? I read that it was needed at rhel3 but at rhel4 is not anymore, could you explain me that. Any help is welcome :) tks, -- Allyson A. Brito From gforte at leopard.us.udel.edu Tue Mar 14 19:23:18 2006 From: gforte at leopard.us.udel.edu (Greg Forte) Date: Tue, 14 Mar 2006 14:23:18 -0500 Subject: [Linux-cluster] dependencies between services In-Reply-To: <1138981187.5992.62.camel@ayanami.boston.redhat.com> References: <43E2948D.3060108@leopard.us.udel.edu> <1138981187.5992.62.camel@ayanami.boston.redhat.com> Message-ID: <44171826.2040202@leopard.us.udel.edu> heh, you asked me to file a bug about this a month-and-a-half ago and I got sidetracked fighting with Oracle and various other components, but now I'm back on this. What heading should I file it under? Cluster Suite v4, obviously, but what component? -g Lon Hohberger wrote: > On Thu, 2006-02-02 at 18:23 -0500, Greg Forte wrote: >> Is it possible to set up dependencies between cluster services? That >> is, I have services A, B, C, and D. B, C, D can't run unless A is >> running, but B, C, and D are all independent of each other and I want to >> be able to control them individually, i.e. be able to start/stop (or >> rather, enable/disable) each without affecting the others. I know I >> could define them as dependent resources all in the same service, but >> then I can't have that independence between B, C, and D ... unless I'm >> missing something. > > Not at the moment, but it should not be a difficult thing to add. > > Could you file a bugzilla about it? > > -- Lon > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Greg Forte gforte at udel.edu IT - User Services University of Delaware 302-831-1982 Newark, DE From treed-cluster at ultraviolet.org Tue Mar 14 19:55:02 2006 From: treed-cluster at ultraviolet.org (treed-cluster at ultraviolet.org) Date: Tue, 14 Mar 2006 11:55:02 -0800 Subject: [Linux-cluster] Cluster aware RAID/LVM Message-ID: <20060314195502.GA25640@ultraviolet.org> I would like to build a highly reliable SAN by having 3 storage nodes exporting their disk as a block device using AoE and then have the 3 compute nodes each RAID 5 the 3 block devices exported by the storage nodes so that we end up with one block device, the same block device, seen by all 3 compute nodes. Then I would like to initialize this as a physical volume and then create different logical volumes within it with each compute node mounting a different set of logical volumes. But my understanding is that this will not currently work because LVM is not cluster aware. It seems that Linux software RAID is not cluster aware either. Perhaps I could use EVMS (which does seem to be cluster aware) on each of the compute nodes to manage the disk on the storage nodes and then export a specific volume from each storage node to just one compute node which would then do RAID 5. This way we have a cluster aware volume manager exporting volumes to be RAID'd which would only be mounted by one host each. Does this sound reasonable? I would like to avoid the use of GFS anywhere in this particular system but I might have occasion to use GFS on a different project in the future. It's been a few years since I have seriously looked into GFS but it seems to have come a long way towards being usable in a production environment. I remember it used to have its own volume management. Are most people doing GFS volume management with EVMS also? Thanks! -- Tracy Reed http://ultraviolet.org From filipe.miranda at gmail.com Tue Mar 14 20:14:28 2006 From: filipe.miranda at gmail.com (Filipe Miranda) Date: Tue, 14 Mar 2006 17:14:28 -0300 Subject: [Linux-cluster] RHCS/RHEL3 power switches options Message-ID: Hello, I'm having a really hard time trying to figure out what power switches do work with RHCS/RHEL3. Did anyone implement RHCS/RHEL3 with power switches ? What options do I have when using power switches with this solution? I appreciate any help Att. --- Filipe Miranda -------------- next part -------------- An HTML attachment was scrubbed... URL: From adingman at cookgroup.com Tue Mar 14 20:58:00 2006 From: adingman at cookgroup.com (Andrew C. Dingman) Date: Tue, 14 Mar 2006 15:58:00 -0500 Subject: [Linux-cluster] RHCS/RHEL3 power switches options In-Reply-To: References: Message-ID: <1142369880.2489.39.camel@adingman.cin.cook> I'm using APC AP7901 power switches in two different REL3 clusters. If you don't alter the default configuration beyond setting your own password, I believe the provided fencing agent works quite well. If you try to use a more restricted user, it may or may not work. (Probably not, in my experience.) The problem is that the APC telnet menus change depending on the privileges of the connected user, so a restricted user on the switch will not get the menus that the fencing agent expects. I am also using GFS on those clusters, so I set up the fencing in GFS, and then used the gulm bridge fencing agent in RHCS, which causes it to pass the fencing work off to GFS. I have no experience using cluster suite without GFS, on either RHEL3 or RHEL4. Hope that helps. On Tue, 2006-03-14 at 17:14 -0300, Filipe Miranda wrote: > Hello, > > I'm having a really hard time trying to figure out what power switches > do work with RHCS/RHEL3. > Did anyone implement RHCS/RHEL3 with power switches ? > What options do I have when using power switches with this solution? > I appreciate any help > > > Att. > --- > Filipe Miranda > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Andrew C. Dingman Unix Administrator Cook Incorporated (812)339-2235 x2131 adingman at cookgroup.com From filipe.miranda at gmail.com Tue Mar 14 21:11:31 2006 From: filipe.miranda at gmail.com (Filipe Miranda) Date: Tue, 14 Mar 2006 18:11:31 -0300 Subject: [Linux-cluster] RHCS/RHEL3 power switches options In-Reply-To: <1142369880.2489.39.camel@adingman.cin.cook> References: <1142369880.2489.39.camel@adingman.cin.cook> Message-ID: Andrew, It definatelly helps, I was researching and I found this type of fence device that would work with RHEL3/RHCS: http://www.wti.com/rps10-ec.htm That's why I posted this message so I could contact people that actually use fence devices/power switches on RHEL/RHCS solutions Thanks a lot Att. Filipe Miranda On 3/14/06, Andrew C. Dingman wrote: > > I'm using APC AP7901 power switches in two different REL3 clusters. If > you don't alter the default configuration beyond setting your own > password, I believe the provided fencing agent works quite well. If you > try to use a more restricted user, it may or may not work. (Probably > not, in my experience.) The problem is that the APC telnet menus change > depending on the privileges of the connected user, so a restricted user > on the switch will not get the menus that the fencing agent expects. > > I am also using GFS on those clusters, so I set up the fencing in GFS, > and then used the gulm bridge fencing agent in RHCS, which causes it to > pass the fencing work off to GFS. I have no experience using cluster > suite without GFS, on either RHEL3 or RHEL4. > > Hope that helps. > > On Tue, 2006-03-14 at 17:14 -0300, Filipe Miranda wrote: > > Hello, > > > > I'm having a really hard time trying to figure out what power switches > > do work with RHCS/RHEL3. > > Did anyone implement RHCS/RHEL3 with power switches ? > > What options do I have when using power switches with this solution? > > I appreciate any help > > > > > > Att. > > --- > > Filipe Miranda > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > -- > Andrew C. Dingman > Unix Administrator > Cook Incorporated > (812)339-2235 x2131 > adingman at cookgroup.com > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Att. --- Filipe T Miranda RHCE - Red Hat Certified Engineer OCP8i - Oracle Certified Professional -------------- next part -------------- An HTML attachment was scrubbed... URL: From Birger.Wathne at ift.uib.no Wed Mar 15 06:17:01 2006 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Wed, 15 Mar 2006 07:17:01 +0100 Subject: [Linux-cluster] samba on gfs In-Reply-To: References: <44167F6E.4000401@ift.uib.no> Message-ID: <4417B15D.7050807@uib.no> Erling Nygaard wrote: > Birger > > The short story is that Samba keeps some state information internally. > So there are issues with keeping multiple Samba serves in sync. > The information in question is not synced to the underlying > filesystem, so GFS can't really do the job of keeping this info in > sync between the nodes. > > I am sure other people on the list can provide more details of the > problem and status of any progress :-) So... This means there is a problem only when you want to run multiple samba servers in a cluster? There should be no problem sharing the same GFS disk for one samba instance and one NFS instance running on separate nodes (or even on the same node during maintenance)? -- birger From robert at deakin.edu.au Wed Mar 15 06:30:16 2006 From: robert at deakin.edu.au (Robert Ruge) Date: Wed, 15 Mar 2006 17:30:16 +1100 Subject: [Linux-cluster] samba on gfs In-Reply-To: <4417B15D.7050807@uib.no> Message-ID: <000f01c647f9$ec572240$0132a8c0@eit.deakin.edu.au> I have had problems with samba running on just one node. It works for a while and then samba just starts locking up. Not a reccomended path if you ask me. Robert > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Birger Wathne > Sent: Wednesday, 15 March 2006 5:17 > To: linux clustering > Subject: Re: [Linux-cluster] samba on gfs > > Erling Nygaard wrote: > > > Birger > > > > The short story is that Samba keeps some state information > internally. > > So there are issues with keeping multiple Samba serves in sync. > > The information in question is not synced to the underlying > > filesystem, so GFS can't really do the job of keeping this info in > > sync between the nodes. > > > > I am sure other people on the list can provide more details of the > > problem and status of any progress :-) > > So... This means there is a problem only when you want to run > multiple samba > servers in a cluster? There should be no problem sharing the > same GFS disk > for one samba instance and one NFS instance running on > separate nodes (or > even on the same node during maintenance)? > > -- > birger > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From Alain.Moulle at bull.net Wed Mar 15 07:33:24 2006 From: Alain.Moulle at bull.net (Alain Moulle) Date: Wed, 15 Mar 2006 08:33:24 +0100 Subject: [Linux-cluster] Re: CS4 behavior on killall -9 (Lon Hohberger) Message-ID: <4417C344.4040907@bull.net> On Mon, 2006-03-13 at 11:38 +0100, Alain Moulle wrote: >>>> Hi >>>> On a HA pair in mutual takeover, it seems that if we do a "killall -9" on >one >>>> node, there is no failover, the CS4 seems to be stalled . >>>> Any reason ? idea ? >>Killall -9 on what specifically...? That's a killall so ... nothing specifically, but all ... Just to simulate sort of system hang ... Did someone has give it a try ? Alain >>It sounds like a bug. >>-- Lon mailto:Alain.Moulle at bull.net +------------------------------+--------------------------------+ | Alain Moull? | from France : 04 76 29 75 99 | | | FAX number : 04 76 29 72 49 | | Bull SA | | | 1, Rue de Provence | Adr : FREC B1-041 | | B.P. 208 | | | 38432 Echirolles - CEDEX | Email: Alain.Moulle at bull.net | | France | BCOM : 229 7599 | +-------------------------------+-------------------------------+ From grimme at atix.de Wed Mar 15 07:44:38 2006 From: grimme at atix.de (Marc Grimme) Date: Wed, 15 Mar 2006 08:44:38 +0100 Subject: [Linux-cluster] samba on gfs In-Reply-To: <000f01c647f9$ec572240$0132a8c0@eit.deakin.edu.au> References: <000f01c647f9$ec572240$0132a8c0@eit.deakin.edu.au> Message-ID: <200603150844.38490.grimme@atix.de> Hello, we have some customers that have samba clusters with GFS running for a long time without problems. There are some things you need to take into account but nevertheless samba on GFS runs very well even when you export the same data via NFS. E.G. one customer runs a "active/active" Samba/NFS Cluster on GFS as ADS Member for about a year (up to 600 Users) without problems. I would say samba and GFS is a very nice combination. Regards Marc. On Wednesday 15 March 2006 07:30, Robert Ruge wrote: > I have had problems with samba running on just one node. It works for > a while and then samba just starts locking up. > > Not a reccomended path if you ask me. > > Robert > > > -----Original Message----- > > From: linux-cluster-bounces at redhat.com > > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Birger Wathne > > Sent: Wednesday, 15 March 2006 5:17 > > To: linux clustering > > Subject: Re: [Linux-cluster] samba on gfs > > > > Erling Nygaard wrote: > > > Birger > > > > > > The short story is that Samba keeps some state information > > > > internally. > > > > > So there are issues with keeping multiple Samba serves in sync. > > > The information in question is not synced to the underlying > > > filesystem, so GFS can't really do the job of keeping this info in > > > sync between the nodes. > > > > > > I am sure other people on the list can provide more details of the > > > problem and status of any progress :-) > > > > So... This means there is a problem only when you want to run > > multiple samba > > servers in a cluster? There should be no problem sharing the > > same GFS disk > > for one samba instance and one NFS instance running on > > separate nodes (or > > even on the same node during maintenance)? > > > > -- > > birger > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Gruss / Regards, Marc Grimme Phone: +49-89 121 409-54 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From l.dardini at comune.prato.it Wed Mar 15 08:39:48 2006 From: l.dardini at comune.prato.it (Leandro Dardini) Date: Wed, 15 Mar 2006 09:39:48 +0100 Subject: R: [Linux-cluster] samba on gfs Message-ID: <404AA6666D14D14CA0D410C1BC6CC4C53FC4F1@exchange3.comune.prato.local> > -----Messaggio originale----- > Da: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] Per conto di Marc Grimme > Inviato: mercoled? 15 marzo 2006 8.45 > A: linux clustering > Oggetto: Re: [Linux-cluster] samba on gfs > > Hello, > we have some customers that have samba clusters with GFS > running for a long time without problems. There are some > things you need to take into account but nevertheless samba > on GFS runs very well even when you export the same data via NFS. > E.G. one customer runs a "active/active" Samba/NFS Cluster on > GFS as ADS Member for about a year (up to 600 Users) without problems. > I would say samba and GFS is a very nice combination. > Regards Marc. Maybe it can be interesting if you can post the smb.conf used. Leandro > > On Wednesday 15 March 2006 07:30, Robert Ruge wrote: > > I have had problems with samba running on just one node. It > works for > > a while and then samba just starts locking up. > > > > Not a reccomended path if you ask me. > > > > Robert > > > > > -----Original Message----- > > > From: linux-cluster-bounces at redhat.com > > > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of > Birger Wathne > > > Sent: Wednesday, 15 March 2006 5:17 > > > To: linux clustering > > > Subject: Re: [Linux-cluster] samba on gfs > > > > > > Erling Nygaard wrote: > > > > Birger > > > > > > > > The short story is that Samba keeps some state information > > > > > > internally. > > > > > > > So there are issues with keeping multiple Samba serves in sync. > > > > The information in question is not synced to the underlying > > > > filesystem, so GFS can't really do the job of keeping > this info in > > > > sync between the nodes. > > > > > > > > I am sure other people on the list can provide more > details of the > > > > problem and status of any progress :-) > > > > > > So... This means there is a problem only when you want to run > > > multiple samba servers in a cluster? There should be no problem > > > sharing the same GFS disk for one samba instance and one NFS > > > instance running on separate nodes (or even on the same > node during > > > maintenance)? > > > > > > -- > > > birger > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Gruss / Regards, > > Marc Grimme > Phone: +49-89 121 409-54 > http://www.atix.de/ http://www.open-sharedroot.org/ > > ** > ATIX - Ges. fuer Informationstechnologie und Consulting mbH > Einsteinstr. 10 - 85716 Unterschleissheim - Germany > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From Matthew.Patton.ctr at osd.mil Wed Mar 15 13:40:52 2006 From: Matthew.Patton.ctr at osd.mil (Patton, Matthew F, CTR, OSD-PA&E) Date: Wed, 15 Mar 2006 08:40:52 -0500 Subject: [Linux-cluster] samba on gfs Message-ID: Classification: UNCLASSIFIED I'm setting up 40 blade servers each with direct access to a shared FC SAN volume and mounted GFS and serving it up via both SAMBA and NFS to ~10 virtual machines (XEN and VMWARE) local to that blade. I don't expect the unix VM's to use SAMBA (exception might be HOMEDIRs) nor the Windows VM's to use NFS, but otherwise it's active/active. NFS will also be used to rootless boot the XEN/VMWARE virtual machines that run Linux. So that's 40 independent SAMBA servers all potentially serving the same files located on the common GFS volume. Think I'm headed for trouble? That config would come in handy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From erwan at seanodes.com Wed Mar 15 13:45:11 2006 From: erwan at seanodes.com (Velu Erwan) Date: Wed, 15 Mar 2006 14:45:11 +0100 Subject: [Linux-cluster] samba on gfs In-Reply-To: <200603150844.38490.grimme@atix.de> References: <000f01c647f9$ec572240$0132a8c0@eit.deakin.edu.au> <200603150844.38490.grimme@atix.de> Message-ID: <44181A67.6070504@seanodes.com> Marc Grimme a ?crit : >Hello, >we have some customers that have samba clusters with GFS running for a long >time without problems. There are some things you need to take into account >but nevertheless samba on GFS runs very well even when you export the same >data via NFS. >E.G. one customer runs a "active/active" Samba/NFS Cluster on GFS as ADS >Member for about a year (up to 600 Users) without problems. >I would say samba and GFS is a very nice combination. >Regards Marc. > > This sounds to be done in a homedir approach but this doesn't sounds to work when you share a directory on several samba servers. I mean, there will be troubles if two users want to access to the same file by using 2 different samba servers. The samba team sounds to address this issue in the future samba 4.0. From filipe.miranda at gmail.com Wed Mar 15 13:41:39 2006 From: filipe.miranda at gmail.com (Filipe Miranda) Date: Wed, 15 Mar 2006 10:41:39 -0300 Subject: [Linux-cluster] samba on gfs In-Reply-To: <200603150844.38490.grimme@atix.de> References: <000f01c647f9$ec572240$0132a8c0@eit.deakin.edu.au> <200603150844.38490.grimme@atix.de> Message-ID: Just one question about running multiple samba servers using GFS.... You can have the same dada to be shared on multiple servers, but each samba server will need a differente NETBIOS name on the network right? So if I have a WindowsXP client machine, it will have to map 2(or more) network paths to reach each samba server. Am I correct? If so, how to address this issue so the user can have just one network mapped drive accessing multiple samba servers(if necessary, if one machine fails for example?) Att. Filipe Miranda On 3/15/06, Marc Grimme wrote: > > Hello, > we have some customers that have samba clusters with GFS running for a > long > time without problems. There are some things you need to take into account > but nevertheless samba on GFS runs very well even when you export the same > data via NFS. > E.G. one customer runs a "active/active" Samba/NFS Cluster on GFS as ADS > Member for about a year (up to 600 Users) without problems. > I would say samba and GFS is a very nice combination. > Regards Marc. > > On Wednesday 15 March 2006 07:30, Robert Ruge wrote: > > I have had problems with samba running on just one node. It works for > > a while and then samba just starts locking up. > > > > Not a reccomended path if you ask me. > > > > Robert > > > > > -----Original Message----- > > > From: linux-cluster-bounces at redhat.com > > > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Birger Wathne > > > Sent: Wednesday, 15 March 2006 5:17 > > > To: linux clustering > > > Subject: Re: [Linux-cluster] samba on gfs > > > > > > Erling Nygaard wrote: > > > > Birger > > > > > > > > The short story is that Samba keeps some state information > > > > > > internally. > > > > > > > So there are issues with keeping multiple Samba serves in sync. > > > > The information in question is not synced to the underlying > > > > filesystem, so GFS can't really do the job of keeping this info in > > > > sync between the nodes. > > > > > > > > I am sure other people on the list can provide more details of the > > > > problem and status of any progress :-) > > > > > > So... This means there is a problem only when you want to run > > > multiple samba > > > servers in a cluster? There should be no problem sharing the > > > same GFS disk > > > for one samba instance and one NFS instance running on > > > separate nodes (or > > > even on the same node during maintenance)? > > > > > > -- > > > birger > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Gruss / Regards, > > Marc Grimme > Phone: +49-89 121 409-54 > http://www.atix.de/ http://www.open-sharedroot.org/ > > ** > ATIX - Ges. fuer Informationstechnologie und Consulting mbH > Einsteinstr. 10 - 85716 Unterschleissheim - Germany > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Att. --- Filipe T Miranda RHCE - Red Hat Certified Engineer OCP8i - Oracle Certified Professional -------------- next part -------------- An HTML attachment was scrubbed... URL: From grimme at atix.de Wed Mar 15 13:52:38 2006 From: grimme at atix.de (Marc Grimme) Date: Wed, 15 Mar 2006 14:52:38 +0100 Subject: [Linux-cluster] samba on gfs In-Reply-To: <44181A67.6070504@seanodes.com> References: <000f01c647f9$ec572240$0132a8c0@eit.deakin.edu.au> <200603150844.38490.grimme@atix.de> <44181A67.6070504@seanodes.com> Message-ID: <200603151452.38992.grimme@atix.de> On Wednesday 15 March 2006 14:45, Velu Erwan wrote: > Marc Grimme a ?crit : > >Hello, > >we have some customers that have samba clusters with GFS running for a > > long time without problems. There are some things you need to take into > > account but nevertheless samba on GFS runs very well even when you export > > the same data via NFS. > >E.G. one customer runs a "active/active" Samba/NFS Cluster on GFS as ADS > >Member for about a year (up to 600 Users) without problems. > >I would say samba and GFS is a very nice combination. > >Regards Marc. > > This sounds to be done in a homedir approach but this doesn't sounds to > work when you share a directory on several samba servers. > I mean, there will be troubles if two users want to access to the same > file by using 2 different samba servers. > The samba team sounds to address this issue in the future samba 4.0. Right, because of that i wrote "active/active". It might work but I would accept the same problems to happen in special cases. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Gruss / Regards, Marc Grimme Phone: +49-89 121 409-54 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From orcl.listas at gmail.com Wed Mar 15 13:56:58 2006 From: orcl.listas at gmail.com (Allyson - Listas) Date: Wed, 15 Mar 2006 10:56:58 -0300 Subject: [Linux-cluster] samba on gfs In-Reply-To: References: <000f01c647f9$ec572240$0132a8c0@eit.deakin.edu.au> <200603150844.38490.grimme@atix.de> Message-ID: <44181D2A.6010000@gmail.com> Filipe, I think, you can use LVS to load balance requests into servers and map just one virtual ip at windows. tks, -- Allyson A. Brito MSN: allysonbrito78 at hotmail.com Filipe Miranda wrote: > Just one question about running multiple samba servers using GFS.... > You can have the same dada to be shared on multiple servers, but each > samba server will need a differente NETBIOS name on the network right? > > So if I have a WindowsXP client machine, it will have to map 2(or > more) network paths to reach each samba server. > > Am I correct? If so, how to address this issue so the user can have > just one network mapped drive accessing multiple samba servers(if > necessary, if one machine fails for example?) > > Att. > Filipe Miranda > > On 3/15/06, *Marc Grimme* > wrote: > > Hello, > we have some customers that have samba clusters with GFS running > for a long > time without problems. There are some things you need to take into > account > but nevertheless samba on GFS runs very well even when you export > the same > data via NFS. > E.G. one customer runs a "active/active" Samba/NFS Cluster on GFS > as ADS > Member for about a year (up to 600 Users) without problems. > I would say samba and GFS is a very nice combination. > Regards Marc. > > On Wednesday 15 March 2006 07:30, Robert Ruge wrote: > > I have had problems with samba running on just one node. It > works for > > a while and then samba just starts locking up. > > > > Not a reccomended path if you ask me. > > > > Robert > > > > > -----Original Message----- > > > From: linux-cluster-bounces at redhat.com > > > > [mailto:linux-cluster-bounces at redhat.com > ] On Behalf Of Birger Wathne > > > Sent: Wednesday, 15 March 2006 5:17 > > > To: linux clustering > > > Subject: Re: [Linux-cluster] samba on gfs > > > > > > Erling Nygaard wrote: > > > > Birger > > > > > > > > The short story is that Samba keeps some state information > > > > > > internally. > > > > > > > So there are issues with keeping multiple Samba serves in sync. > > > > The information in question is not synced to the underlying > > > > filesystem, so GFS can't really do the job of keeping this > info in > > > > sync between the nodes. > > > > > > > > I am sure other people on the list can provide more details > of the > > > > problem and status of any progress :-) > > > > > > So... This means there is a problem only when you want to run > > > multiple samba > > > servers in a cluster? There should be no problem sharing the > > > same GFS disk > > > for one samba instance and one NFS instance running on > > > separate nodes (or > > > even on the same node during maintenance)? > > > > > > -- > > > birger > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Gruss / Regards, > > Marc Grimme > Phone: +49-89 121 409-54 > http://www.atix.de/ http://www.open-sharedroot.org/ > > ** > ATIX - Ges. fuer Informationstechnologie und Consulting mbH > Einsteinstr. 10 - 85716 Unterschleissheim - Germany > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > -- > Att. > --- > Filipe T Miranda > RHCE - Red Hat Certified Engineer > OCP8i - Oracle Certified Professional > >------------------------------------------------------------------------ > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > From grimme at atix.de Wed Mar 15 13:58:06 2006 From: grimme at atix.de (Marc Grimme) Date: Wed, 15 Mar 2006 14:58:06 +0100 Subject: [Linux-cluster] samba on gfs In-Reply-To: References: <000f01c647f9$ec572240$0132a8c0@eit.deakin.edu.au> <200603150844.38490.grimme@atix.de> Message-ID: <200603151458.07066.grimme@atix.de> On Wednesday 15 March 2006 14:41, Filipe Miranda wrote: > Just one question about running multiple samba servers using GFS.... > You can have the same dada to be shared on multiple servers, but each samba > server will need a differente NETBIOS name on the network right? Yes. Every server also needs two IPs ond VIP and one IP and every Server needs to be registered in the Windows domain. Just like the "active/active" Microsoft Cluster Services work. > > So if I have a WindowsXP client machine, it will have to map 2(or more) > network paths to reach each samba server. No. Basically every server serves exactly its shares exclusively. HA-services take over the availability. But the active/active granularity is the share. If you would like to have real loadblancing you would need a samba which is capable of that (I also heard samba4 will solve that issue). > > Am I correct? If so, how to address this issue so the user can have just > one network mapped drive accessing multiple samba servers(if necessary, if > one machine fails for example?) That is not possible. Sorry for the missunderstanding Marc. > > Att. > Filipe Miranda > > On 3/15/06, Marc Grimme wrote: > > Hello, > > we have some customers that have samba clusters with GFS running for a > > long > > time without problems. There are some things you need to take into > > account but nevertheless samba on GFS runs very well even when you export > > the same data via NFS. > > E.G. one customer runs a "active/active" Samba/NFS Cluster on GFS as ADS > > Member for about a year (up to 600 Users) without problems. > > I would say samba and GFS is a very nice combination. > > Regards Marc. > > > > On Wednesday 15 March 2006 07:30, Robert Ruge wrote: > > > I have had problems with samba running on just one node. It works for > > > a while and then samba just starts locking up. > > > > > > Not a reccomended path if you ask me. > > > > > > Robert > > > > > > > -----Original Message----- > > > > From: linux-cluster-bounces at redhat.com > > > > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Birger Wathne > > > > Sent: Wednesday, 15 March 2006 5:17 > > > > To: linux clustering > > > > Subject: Re: [Linux-cluster] samba on gfs > > > > > > > > Erling Nygaard wrote: > > > > > Birger > > > > > > > > > > The short story is that Samba keeps some state information > > > > > > > > internally. > > > > > > > > > So there are issues with keeping multiple Samba serves in sync. > > > > > The information in question is not synced to the underlying > > > > > filesystem, so GFS can't really do the job of keeping this info in > > > > > sync between the nodes. > > > > > > > > > > I am sure other people on the list can provide more details of the > > > > > problem and status of any progress :-) > > > > > > > > So... This means there is a problem only when you want to run > > > > multiple samba > > > > servers in a cluster? There should be no problem sharing the > > > > same GFS disk > > > > for one samba instance and one NFS instance running on > > > > separate nodes (or > > > > even on the same node during maintenance)? > > > > > > > > -- > > > > birger > > > > > > > > -- > > > > Linux-cluster mailing list > > > > Linux-cluster at redhat.com > > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Gruss / Regards, > > > > Marc Grimme > > Phone: +49-89 121 409-54 > > http://www.atix.de/ http://www.open-sharedroot.org/ > > > > ** > > ATIX - Ges. fuer Informationstechnologie und Consulting mbH > > Einsteinstr. 10 - 85716 Unterschleissheim - Germany > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Att. > --- > Filipe T Miranda > RHCE - Red Hat Certified Engineer > OCP8i - Oracle Certified Professional -- Gruss / Regards, Marc Grimme Phone: +49-89 121 409-54 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From lhh at redhat.com Wed Mar 15 16:04:58 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 15 Mar 2006 11:04:58 -0500 Subject: [Linux-cluster] RHCS/RHEL3 power switches options In-Reply-To: References: Message-ID: <1142438698.19535.17.camel@ayanami.boston.redhat.com> On Tue, 2006-03-14 at 17:14 -0300, Filipe Miranda wrote: > Hello, > > I'm having a really hard time trying to figure out what power switches > do work with RHCS/RHEL3. > Did anyone implement RHCS/RHEL3 with power switches ? > What options do I have when using power switches with this solution? > I appreciate any help These have been known to work: * WTI NPS, IPS, or TPS series. * WTI RPS10 (serial; two node only) * APC 9211, 9212. * APC 9225 with 9606 management card. * APC 7900 and 7921 have been tried by some people with 1.2.22 and later. -- Lon From lhh at redhat.com Wed Mar 15 16:05:21 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 15 Mar 2006 11:05:21 -0500 Subject: [Linux-cluster] RHCS/RHEL3 power switches options In-Reply-To: <1142369880.2489.39.camel@adingman.cin.cook> References: <1142369880.2489.39.camel@adingman.cin.cook> Message-ID: <1142438721.19535.19.camel@ayanami.boston.redhat.com> On Tue, 2006-03-14 at 15:58 -0500, Andrew C. Dingman wrote: > I'm using APC AP7901 power switches in two different REL3 clusters. Woot. The 7901 works too ;) -- Lon From lhh at redhat.com Wed Mar 15 16:09:49 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 15 Mar 2006 11:09:49 -0500 Subject: [Linux-cluster] RHCS/RHEL3 power switches options In-Reply-To: References: <1142369880.2489.39.camel@adingman.cin.cook> Message-ID: <1142438989.19535.25.camel@ayanami.boston.redhat.com> On Tue, 2006-03-14 at 18:11 -0300, Filipe Miranda wrote: > Andrew, > > It definatelly helps, I was researching and I found this type of fence > device that would work with RHEL3/RHCS: > http://www.wti.com/rps10-ec.htm Personally, I prefer IPS800 over dual RPS10 - they provide two power sources, and the ability to turn off machines from remote for testing, or other maintenance conditions. The IPS800 has two power rails, allowing NSPF configurations (if done intelligently...), and can control more devices for only a little bit more money than two RPS-10s. -- Lon From lhh at redhat.com Wed Mar 15 16:15:25 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 15 Mar 2006 11:15:25 -0500 Subject: [Linux-cluster] rhcs doubts. In-Reply-To: <44170B05.20805@gmail.com> References: <44170B05.20805@gmail.com> Message-ID: <1142439325.19535.31.camel@ayanami.boston.redhat.com> On Tue, 2006-03-14 at 15:27 -0300, Allyson - Listas wrote: > Hi guys, > > I'm new at redhat cluster suite. Could Anybody help me in some questions? > > 1st) I installed rhcs on 2 virtual machines and create a new cluster, > setup a manual fence, a failvoer domain, create a IP resource and a > service that uses just that IP for tests. Well, I'd like to know how can > I force a failover of the service between nodes. This option is not > available at system-config-cluster that allow just disable and enable > the service. Drag it to the other node in the gui. > I noticed that the ip service created is not a virtual > interface like eth0:1, but it was working because I could ping it, Is it > Normal? Yes, try "/sbin/ip addr list", which is noted in the documentation. > 2nd) What is the real fuction of a fence device? Prevent data corruption in the event of a live-hang of a node with outstanding dirty buffers. > 3rd) How can I setup a quorum device, and isn't necessary for a failover > service? I read that it was needed at rhel3 but at rhel4 is not > anymore, could you explain me that. What do you need to know? It's not needed because of the way CMAN recovers - see http://people.redhat.com/teigland/sca.pdf -- Lon From filipe.miranda at gmail.com Wed Mar 15 17:41:18 2006 From: filipe.miranda at gmail.com (Filipe Miranda) Date: Wed, 15 Mar 2006 14:41:18 -0300 Subject: [Linux-cluster] RHCS/RHEL3 power switches options In-Reply-To: <1142438989.19535.25.camel@ayanami.boston.redhat.com> References: <1142369880.2489.39.camel@adingman.cin.cook> <1142438989.19535.25.camel@ayanami.boston.redhat.com> Message-ID: But will all those models work with RHEL3/RHCS ? Or RHEL4/RHCS? Thanks a lot in advance Att. Filipe Miranda On 3/15/06, Lon Hohberger wrote: > > On Tue, 2006-03-14 at 18:11 -0300, Filipe Miranda wrote: > > Andrew, > > > > It definatelly helps, I was researching and I found this type of fence > > device that would work with RHEL3/RHCS: > > http://www.wti.com/rps10-ec.htm > > Personally, I prefer IPS800 over dual RPS10 - they provide two power > sources, and the ability to turn off machines from remote for testing, > or other maintenance conditions. > > The IPS800 has two power rails, allowing NSPF configurations (if done > intelligently...), and can control more devices for only a little bit > more money than two RPS-10s. > > -- Lon > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Att. --- Filipe T Miranda RHCE - Red Hat Certified Engineer OCP8i - Oracle Certified Professional -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhh at redhat.com Wed Mar 15 18:20:25 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 15 Mar 2006 13:20:25 -0500 Subject: [Linux-cluster] RHCS/RHEL3 power switches options In-Reply-To: References: <1142369880.2489.39.camel@adingman.cin.cook> <1142438989.19535.25.camel@ayanami.boston.redhat.com> Message-ID: <1142446826.19535.40.camel@ayanami.boston.redhat.com> On Wed, 2006-03-15 at 14:41 -0300, Filipe Miranda wrote: > But will all those models work with RHEL3/RHCS ? > Or RHEL4/RHCS? With the exception of the APC 9211, 9212, and 9225 (which only work on RHCS3, and I think are out of their support lifetime anyway), all of the models (or series) I noted should work on both RHCS3 and RHCS4. If you have a specific question about a specific switch, ask away. -- Lon From lhh at redhat.com Wed Mar 15 18:23:23 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 15 Mar 2006 13:23:23 -0500 Subject: [Linux-cluster] Re: CS4 behavior on killall -9 (Lon Hohberger) In-Reply-To: <4417C344.4040907@bull.net> References: <4417C344.4040907@bull.net> Message-ID: <1142447003.19535.44.camel@ayanami.boston.redhat.com> On Wed, 2006-03-15 at 08:33 +0100, Alain Moulle wrote: > On Mon, 2006-03-13 at 11:38 +0100, Alain Moulle wrote: > > >>>> Hi > >>>> On a HA pair in mutual takeover, it seems that if we do a "killall -9" on >one > >>>> node, there is no failover, the CS4 seems to be stalled . > >>>> Any reason ? idea ? > > >>Killall -9 on what specifically...? > That's a killall so ... nothing specifically, but all ... > Just to simulate sort of system hang ... > Did someone has give it a try ? On RHCS4, you probably won't get much of the desired result here. CMAN (which manages membership transitions, among other things) runs in the kernel. I'm not sure (off the top of my head) that sending one of the kernel threads a SIGKILL actually has any guaranteed effect... -- Lon From jparsons at redhat.com Wed Mar 15 19:23:43 2006 From: jparsons at redhat.com (James Parsons) Date: Wed, 15 Mar 2006 14:23:43 -0500 Subject: [Linux-cluster] RHCS/RHEL3 power switches options In-Reply-To: <1142446826.19535.40.camel@ayanami.boston.redhat.com> References: <1142369880.2489.39.camel@adingman.cin.cook> <1142438989.19535.25.camel@ayanami.boston.redhat.com> <1142446826.19535.40.camel@ayanami.boston.redhat.com> Message-ID: <441869BF.30008@redhat.com> Lon Hohberger wrote: >On Wed, 2006-03-15 at 14:41 -0300, Filipe Miranda wrote: > > >>But will all those models work with RHEL3/RHCS ? >> >> > > > >>Or RHEL4/RHCS? >> >> > >With the exception of the APC 9211, 9212, and 9225 (which only work on >RHCS3, and I think are out of their support lifetime anyway), all of the >models (or series) I noted should work on both RHCS3 and RHCS4. > >If you have a specific question about a specific switch, ask away. > Yes! Now is the time to ask about APC switches...I will be down in their facility in less than two weeks testing our scripts against all of their latest products and firmware. If you have a particular switch in mind for a project, I will try and test that exact model for you. -J From brilong at cisco.com Wed Mar 15 19:51:25 2006 From: brilong at cisco.com (Brian Long) Date: Wed, 15 Mar 2006 14:51:25 -0500 Subject: [Linux-cluster] RHCS/RHEL3 power switches options In-Reply-To: <1142438698.19535.17.camel@ayanami.boston.redhat.com> References: <1142438698.19535.17.camel@ayanami.boston.redhat.com> Message-ID: <1142452285.4416.46.camel@brilong-lnx> On Wed, 2006-03-15 at 11:04 -0500, Lon Hohberger wrote: > On Tue, 2006-03-14 at 17:14 -0300, Filipe Miranda wrote: > > Hello, > > > > I'm having a really hard time trying to figure out what power switches > > do work with RHCS/RHEL3. > > Did anyone implement RHCS/RHEL3 with power switches ? > > What options do I have when using power switches with this solution? > > I appreciate any help > > These have been known to work: > > * WTI NPS, IPS, or TPS series. > * WTI RPS10 (serial; two node only) > * APC 9211, 9212. > * APC 9225 with 9606 management card. > * APC 7900 and 7921 have been tried by some people with 1.2.22 and > later. Given all the talk about remotely-managed power switches, are they still needed when you can fence at the ILO level on an HP Proliant, for example? You can also fence most tier 1 servers using ipmitool, right? /Brian/ -- Brian Long | | | IT Data Center Systems | .|||. .|||. Cisco Linux Developer | ..:|||||||:...:|||||||:.. Phone: (919) 392-7363 | C i s c o S y s t e m s From Birger.Wathne at ift.uib.no Wed Mar 15 21:48:56 2006 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Wed, 15 Mar 2006 22:48:56 +0100 Subject: [Linux-cluster] stress-testing GFS ? Message-ID: <44188BC8.2070800@ift.uib.no> I would like to put my cluster through a little controlled hell before declaring it ready for production. Is there any kind of stress-test/verification procedure to 'certify' shared storage with GFS? Ideally there would be some distributed software that could be run in a cluster to check that the shared storage behaves as expected under all kinds of load. Throughput, concurrent writing, GFS locking, file system locking, etc... Something that could interface with GFS internals to see that everything was 'right' at every step. Since I have seen nothing about the issue, I assume something like that doesn't exist, so... Any ideas on how to stress test GFS? Homegrown scripts? Known problems with hardware that a test should look for? -- birger From mwill at penguincomputing.com Wed Mar 15 22:20:28 2006 From: mwill at penguincomputing.com (Michael Will) Date: Wed, 15 Mar 2006 14:20:28 -0800 Subject: [Linux-cluster] stress-testing GFS ? In-Reply-To: <44188BC8.2070800@ift.uib.no> References: <44188BC8.2070800@ift.uib.no> Message-ID: <4418932C.9080001@jellyfish.highlyscyld.com> iozone does test for a lot of different access patterns, and can create nice spreadsheets including graphs from the point of view of a single node. It also has a multiple node flag for running it across a cluster. See -+m and -t options. It knows how to use 'rsh' and can also be configured for any other remote execution command by setting the enviroment variable RSH to say ssh or bpsh. Don't forget to post your benchmark results to this mailinglist ;-) Michael Birger Wathne wrote: > I would like to put my cluster through a little controlled hell before > declaring it ready for production. > > Is there any kind of stress-test/verification procedure to 'certify' > shared storage with GFS? > Ideally there would be some distributed software that could be run in > a cluster to check that the shared storage behaves as expected under > all kinds of load. Throughput, concurrent writing, GFS locking, file > system locking, etc... > Something that could interface with GFS internals to see that > everything was 'right' at every step. > > Since I have seen nothing about the issue, I assume something like > that doesn't exist, so... Any ideas on how to stress test GFS? > Homegrown scripts? Known problems with hardware that a test should > look for? > > From toxictux at gmail.com Wed Mar 15 22:44:14 2006 From: toxictux at gmail.com (toxictux) Date: Wed, 15 Mar 2006 16:44:14 -0600 Subject: [Linux-cluster] Cluster Newbie Questions........ Message-ID: <17df45710603151444s51d1d34dpbdcad003f421a625@mail.gmail.com> hi all, i am setting up a two node cluster with San based shared storage. i have following questions regarding my setup. 1. i am planning to use this cluster to setup a web based application. i saw the example in redhat documentation on how to setup http service in cluster. is it possible to have http and mysql services served by same cluster?? 2. can i set them up on the same LUN after i created 2 separate partitions?? 3. which lock manager is recommended DLM or GuLM? Thanks, -F From Matthew.Patton.ctr at osd.mil Wed Mar 15 23:14:05 2006 From: Matthew.Patton.ctr at osd.mil (Patton, Matthew F, CTR, OSD-PA&E) Date: Wed, 15 Mar 2006 18:14:05 -0500 Subject: [Linux-cluster] stress-testing GFS ? Message-ID: Classification: UNCLASSIFIED on a related note, should I anticipate an major gotcha's with respect to a 30 nodes on a GFS volume? I intend to run GFS 6.1 with DLM. Using RHEL4u2 at the moment. (I posted a query earlier today but received no responses so I'm wondering if this is getting thru.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Birger.Wathne at ift.uib.no Thu Mar 16 00:13:29 2006 From: Birger.Wathne at ift.uib.no (Birger Wathne) Date: Thu, 16 Mar 2006 01:13:29 +0100 Subject: [Linux-cluster] stress-testing GFS ? In-Reply-To: <4418932C.9080001@jellyfish.highlyscyld.com> References: <44188BC8.2070800@ift.uib.no> <4418932C.9080001@jellyfish.highlyscyld.com> Message-ID: <4418ADA9.8060609@ift.uib.no> Michael Will wrote: > iozone does test for a lot of different access patterns, and can > create nice spreadsheets including graphs > from the point of view of a single node. It also has a multiple node > flag for running it across a cluster. See -+m and -t > options. It knows how to use 'rsh' and can also be configured for any > other remote execution command by setting the > enviroment variable RSH to say ssh or bpsh. > > Don't forget to post your benchmark results to this mailinglist ;-) > I used iozone and some homegrown scripts some years ago to test performance of various raid controllers as well as software raid on Sun systems. Always in single-node configurations. The easiest way to communicate the performance of a raid controller to other people was a series of 3d surface plots. Sadly, OpenOffice doesn't have those, so I had to switch to that commercial office package. I tried gnuplot, but frankly.... compare the readability of the final plot with excel and there was no comparison :-/ Perhaps Matlab... What I hoped for was something that also verified that the internal states of glm and the locking subsystem were as they should at every step of the test. Something that could certify that the hardware behaved as GFS expected it to when pushed more than test performance. -- birger From orcl.listas at gmail.com Thu Mar 16 00:20:45 2006 From: orcl.listas at gmail.com (Allyson - Listas) Date: Wed, 15 Mar 2006 21:20:45 -0300 Subject: [Linux-cluster] rhcs doubts. In-Reply-To: <1142439325.19535.31.camel@ayanami.boston.redhat.com> References: <44170B05.20805@gmail.com> <1142439325.19535.31.camel@ayanami.boston.redhat.com> Message-ID: <4418AF5D.5040009@gmail.com> Lon Hohberger wrote: >On Tue, 2006-03-14 at 15:27 -0300, Allyson - Listas wrote: > > >>Hi guys, >> >>I'm new at redhat cluster suite. Could Anybody help me in some questions? >> >>1st) I installed rhcs on 2 virtual machines and create a new cluster, >>setup a manual fence, a failvoer domain, create a IP resource and a >>service that uses just that IP for tests. Well, I'd like to know how can >>I force a failover of the service between nodes. This option is not >>available at system-config-cluster that allow just disable and enable >>the service. >> >> > >Drag it to the other node in the gui. > > --> I couldn't find this option at gui, but i find how to do it at command line.. [root at cs02 /]# clustat Member Status: Quorate Member Name Status ------ ---- ------ cs01.example.com Online, rgmanager cs02.example.com Online, Local, rgmanager Service Name Owner (Last) State ------- ---- ----- ------ ----- vip50 cs02.example.com started oracle-ha-fs cs02.example.com started [root at cs02 /]# clusvcadm -r oracle-ha-fs -m cs01.example.com Trying to relocate oracle-ha-fs to cs01.example.com...success [root at cs02 /]# clusvcadm -r vip50 -m cs01.example.com Trying to relocate vip50 to cs01.example.com...success [root at cs02 /]# clustat Member Status: Quorate Member Name Status ------ ---- ------ cs01.example.com Online, rgmanager cs02.example.com Online, Local, rgmanager Service Name Owner (Last) State ------- ---- ----- ------ ----- vip50 cs01.example.com started oracle-ha-fs cs01.example.com started > > > >>I noticed that the ip service created is not a virtual >>interface like eth0:1, but it was working because I could ping it, Is it >>Normal? >> >> > >Yes, try "/sbin/ip addr list", which is noted in the documentation. > > > > tks >>2nd) What is the real fuction of a fence device? >> >> > >Prevent data corruption in the event of a live-hang of a node with >outstanding dirty buffers. > > > > >>3rd) How can I setup a quorum device, and isn't necessary for a failover >>service? I read that it was needed at rhel3 but at rhel4 is not >>anymore, could you explain me that. >> >> > >What do you need to know? It's not needed because of the way CMAN >recovers - see > >http://people.redhat.com/teigland/sca.pdf > >-- Lon > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- Allyson A. Brito MSN: allysonbrito78 at hotmail.com SKYPE: allysonbrito RHCE / LPI-1 / SCSA OCP DBA 9i / OCA PL/SQL 9i From jparsons at redhat.com Thu Mar 16 00:28:10 2006 From: jparsons at redhat.com (James Parsons) Date: Wed, 15 Mar 2006 19:28:10 -0500 Subject: [Linux-cluster] rhcs doubts. In-Reply-To: <4418AF5D.5040009@gmail.com> References: <44170B05.20805@gmail.com> <1142439325.19535.31.camel@ayanami.boston.redhat.com> <4418AF5D.5040009@gmail.com> Message-ID: <4418B11A.5010704@redhat.com> Allyson - Listas wrote: > Lon Hohberger wrote: > >> On Tue, 2006-03-14 at 15:27 -0300, Allyson - Listas wrote: >> >> >>> Hi guys, >>> >>> I'm new at redhat cluster suite. Could Anybody help me in some >>> questions? >>> >>> 1st) I installed rhcs on 2 virtual machines and create a new >>> cluster, setup a manual fence, a failvoer domain, create a IP >>> resource and a service that uses just that IP for tests. Well, I'd >>> like to know how can I force a failover of the service between >>> nodes. This option is not available at system-config-cluster that >>> allow just disable and enable the service. >> >> >> Drag it to the other node in the gui. >> >> > --> I couldn't find this option at gui, but i find how to do it at > command line.. Just grab the service you want to move in the management view, and drag it to the node you want it to run on in the upper half of the GUI. :-) -j > > [root at cs02 /]# clustat > Member Status: Quorate > > Member Name Status > ------ ---- ------ > cs01.example.com Online, rgmanager > cs02.example.com Online, Local, rgmanager > > Service Name Owner (Last) State > ------- ---- ----- ------ ----- > vip50 cs02.example.com started > oracle-ha-fs cs02.example.com started > [root at cs02 /]# clusvcadm -r oracle-ha-fs -m cs01.example.com > Trying to relocate oracle-ha-fs to cs01.example.com...success > [root at cs02 /]# clusvcadm -r vip50 -m cs01.example.com > Trying to relocate vip50 to cs01.example.com...success > [root at cs02 /]# clustat > Member Status: Quorate > > Member Name Status > ------ ---- ------ > cs01.example.com Online, rgmanager > cs02.example.com Online, Local, rgmanager > > Service Name Owner (Last) State > ------- ---- ----- ------ ----- > vip50 cs01.example.com started > oracle-ha-fs cs01.example.com started > >> >> >> >>> I noticed that the ip service created is not a virtual interface >>> like eth0:1, but it was working because I could ping it, Is it Normal? >>> >> >> >> Yes, try "/sbin/ip addr list", which is noted in the documentation. >> >> >> >> > tks > >>> 2nd) What is the real fuction of a fence device? >>> >> >> >> Prevent data corruption in the event of a live-hang of a node with >> outstanding dirty buffers. >> >> >> >> >>> 3rd) How can I setup a quorum device, and isn't necessary for a >>> failover service? I read that it was needed at rhel3 but at rhel4 >>> is not anymore, could you explain me that. >>> >> >> >> What do you need to know? It's not needed because of the way CMAN >> recovers - see >> >> http://people.redhat.com/teigland/sca.pdf >> >> -- Lon >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> > > From orcl.listas at gmail.com Thu Mar 16 00:33:26 2006 From: orcl.listas at gmail.com (Allyson - Listas) Date: Wed, 15 Mar 2006 21:33:26 -0300 Subject: [Linux-cluster] Cluster Newbie Questions........ In-Reply-To: <17df45710603151444s51d1d34dpbdcad003f421a625@mail.gmail.com> References: <17df45710603151444s51d1d34dpbdcad003f421a625@mail.gmail.com> Message-ID: <4418B256.4050505@gmail.com> toxictux wrote: >hi all, > i am setting up a two node cluster with San based shared storage. >i have following questions regarding my setup. > >1. i am planning to use this cluster to setup a web based application. >i saw the example in redhat documentation on how to setup http service >in cluster. is it possible to have http and mysql services served by >same cluster?? > > > Yes, just make 2 diferent scripts to a better management. >2. can i set them up on the same LUN after i created 2 separate partitions?? > > > Yes, no problems. Just remember that devices on Linux can change depending on your scsi id, target, lun, etc... to avoid problems I mount filesystems using LABEL and not the device /dev/sdX. >3. which lock manager is recommended DLM or GuLM? >Thanks, >-F > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- Allyson A. Brito MSN: allysonbrito78 at hotmail.com SKYPE: allysonbrito RHCE / LPI-1 / SCSA OCP DBA 9i / OCA PL/SQL 9i From toxictux at gmail.com Thu Mar 16 02:01:40 2006 From: toxictux at gmail.com (toxictux) Date: Wed, 15 Mar 2006 20:01:40 -0600 Subject: [Linux-cluster] Cluster Newbie Questions........ In-Reply-To: <4418B256.4050505@gmail.com> References: <17df45710603151444s51d1d34dpbdcad003f421a625@mail.gmail.com> <4418B256.4050505@gmail.com> Message-ID: <17df45710603151801r16a35c9ejd539fded8ddb711f@mail.gmail.com> thanks allyson.... On 3/15/06, Allyson - Listas wrote: > toxictux wrote: > > >hi all, > > i am setting up a two node cluster with San based shared storage. > >i have following questions regarding my setup. > > > >1. i am planning to use this cluster to setup a web based application. > >i saw the example in redhat documentation on how to setup http service > >in cluster. is it possible to have http and mysql services served by > >same cluster?? > > > > > > > Yes, just make 2 diferent scripts to a better management. > > >2. can i set them up on the same LUN after i created 2 separate partitions?? > > > > > > > Yes, no problems. Just remember that devices on Linux can change > depending on your scsi id, target, lun, etc... to avoid problems I mount > filesystems using LABEL and not the device /dev/sdX. > > >3. which lock manager is recommended DLM or GuLM? > >Thanks, > >-F > > > >-- > >Linux-cluster mailing list > >Linux-cluster at redhat.com > >https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > > > > -- > Allyson A. Brito > MSN: allysonbrito78 at hotmail.com > SKYPE: allysonbrito > RHCE / LPI-1 / SCSA > OCP DBA 9i / OCA PL/SQL 9i > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From rohara at redhat.com Thu Mar 16 03:10:34 2006 From: rohara at redhat.com (Ryan O'Hara) Date: Wed, 15 Mar 2006 21:10:34 -0600 Subject: [Linux-cluster] Cluster Newbie Questions........ In-Reply-To: <17df45710603151444s51d1d34dpbdcad003f421a625@mail.gmail.com> References: <17df45710603151444s51d1d34dpbdcad003f421a625@mail.gmail.com> Message-ID: <4418D72A.2060603@redhat.com> toxictux wrote: > > 3. which lock manager is recommended DLM or GuLM? DLM is recommended. From suvankar_moitra at yahoo.com Thu Mar 16 05:07:09 2006 From: suvankar_moitra at yahoo.com (SUVANKAR MOITRA) Date: Wed, 15 Mar 2006 21:07:09 -0800 (PST) Subject: [Linux-cluster] rhcs doubts. In-Reply-To: <4418B11A.5010704@redhat.com> Message-ID: <20060316050709.70034.qmail@web52309.mail.yahoo.com> dear all, thanks for the document , its helps a lot. regards Suvankar --- James Parsons wrote: > Allyson - Listas wrote: > > > Lon Hohberger wrote: > > > >> On Tue, 2006-03-14 at 15:27 -0300, Allyson - > Listas wrote: > >> > >> > >>> Hi guys, > >>> > >>> I'm new at redhat cluster suite. Could Anybody > help me in some > >>> questions? > >>> > >>> 1st) I installed rhcs on 2 virtual machines and > create a new > >>> cluster, setup a manual fence, a failvoer > domain, create a IP > >>> resource and a service that uses just that IP > for tests. Well, I'd > >>> like to know how can I force a failover of the > service between > >>> nodes. This option is not available at > system-config-cluster that > >>> allow just disable and enable the service. > >> > >> > >> Drag it to the other node in the gui. > >> > >> > > --> I couldn't find this option at gui, but i > find how to do it at > > command line.. > > Just grab the service you want to move in the > management view, and drag > it to the node you want it to run on in the upper > half of the GUI. :-) > > -j > > > > > [root at cs02 /]# clustat > > Member Status: Quorate > > > > Member Name Status > > ------ ---- ------ > > cs01.example.com Online, > rgmanager > > cs02.example.com Online, > Local, rgmanager > > > > Service Name Owner (Last) > State > > ------- ---- ----- ------ > ----- > > vip50 cs02.example.com > started > > oracle-ha-fs cs02.example.com > started > > [root at cs02 /]# clusvcadm -r oracle-ha-fs -m > cs01.example.com > > Trying to relocate oracle-ha-fs to > cs01.example.com...success > > [root at cs02 /]# clusvcadm -r vip50 -m > cs01.example.com > > Trying to relocate vip50 to > cs01.example.com...success > > [root at cs02 /]# clustat > > Member Status: Quorate > > > > Member Name Status > > ------ ---- ------ > > cs01.example.com Online, > rgmanager > > cs02.example.com Online, > Local, rgmanager > > > > Service Name Owner (Last) > State > > ------- ---- ----- ------ > ----- > > vip50 cs01.example.com > started > > oracle-ha-fs cs01.example.com > started > > > >> > >> > >> > >>> I noticed that the ip service created is not a > virtual interface > >>> like eth0:1, but it was working because I could > ping it, Is it Normal? > >>> > >> > >> > >> Yes, try "/sbin/ip addr list", which is noted in > the documentation. > >> > >> > >> > >> > > tks > > > >>> 2nd) What is the real fuction of a fence device? > >>> > >> > >> > >> Prevent data corruption in the event of a > live-hang of a node with > >> outstanding dirty buffers. > >> > >> > >> > >> > >>> 3rd) How can I setup a quorum device, and isn't > necessary for a > >>> failover service? I read that it was needed at > rhel3 but at rhel4 > >>> is not anymore, could you explain me that. > >>> > >> > >> > >> What do you need to know? It's not needed > because of the way CMAN > >> recovers - see > >> > >> http://people.redhat.com/teigland/sca.pdf > >> > >> -- Lon > >> > >> -- > >> Linux-cluster mailing list > >> Linux-cluster at redhat.com > >> > https://www.redhat.com/mailman/listinfo/linux-cluster > >> > >> > >> > > > > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From Anthony.Assi at irisa.fr Thu Mar 16 08:58:21 2006 From: Anthony.Assi at irisa.fr (Anthony Assi) Date: Thu, 16 Mar 2006 09:58:21 +0100 Subject: [Linux-cluster] Cluster Newbie Questions........ In-Reply-To: <4418D72A.2060603@redhat.com> References: <17df45710603151444s51d1d34dpbdcad003f421a625@mail.gmail.com> <4418D72A.2060603@redhat.com> Message-ID: <441928AD.4010209@irisa.fr> Absolutely go for DLM; and not for GULM, we are facing small problems with the lock servers of GULM, and apperentley, it might not be supported with RHE5. Ryan O'Hara wrote: > toxictux wrote: > >> >> 3. which lock manager is recommended DLM or GuLM? > > > DLM is recommended. > From lhh at redhat.com Thu Mar 16 14:56:14 2006 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 16 Mar 2006 09:56:14 -0500 Subject: [Linux-cluster] Cluster Newbie Questions........ In-Reply-To: <4418B256.4050505@gmail.com> References: <17df45710603151444s51d1d34dpbdcad003f421a625@mail.gmail.com> <4418B256.4050505@gmail.com> Message-ID: <1142520974.19535.142.camel@ayanami.boston.redhat.com> On Wed, 2006-03-15 at 21:33 -0300, Allyson - Listas wrote: > >1. i am planning to use this cluster to setup a web based application. > >i saw the example in redhat documentation on how to setup http service > >in cluster. is it possible to have http and mysql services served by > >same cluster?? > > > Yes, just make 2 diferent scripts to a better management. Correct. > >2. can i set them up on the same LUN after i created 2 separate partitions?? > > > Yes, no problems. Just remember that devices on Linux can change > depending on your scsi id, target, lun, etc... to avoid problems I mount > filesystems using LABEL and not the device /dev/sdX. You can also look in to using CLVM for the device names, which will be consistent cluster-wide. If using the file system label, just type: LABEL=mylabel ...in the UI instead of "/dev/sda1". > >3. which lock manager is recommended DLM or GuLM? DLM for small node counts. -- Lon From theo at tkd.co.id Thu Mar 16 15:18:58 2006 From: theo at tkd.co.id (Theodorus) Date: Thu, 16 Mar 2006 22:18:58 +0700 Subject: [Linux-cluster] cluster suit 4.2 Message-ID: Hi all, We need your help. We have cluster suit 4.2 installed on RedHat AS 4.2. The cluster system has been run well. The resource group can be relocated when the one of the nodes is down. But, if we disconnect all network cables of one node on purpose, the cluster system stalled, why ? Thanks for your help. Rgds, Theo -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhh at redhat.com Thu Mar 16 15:18:14 2006 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 16 Mar 2006 10:18:14 -0500 Subject: [Linux-cluster] stress-testing GFS ? In-Reply-To: References: Message-ID: <1142522294.19535.146.camel@ayanami.boston.redhat.com> On Wed, 2006-03-15 at 18:14 -0500, Patton, Matthew F, CTR, OSD-PA&E wrote: > Classification: UNCLASSIFIED > > on a related note, should I anticipate an major gotcha's with respect > to a 30 nodes on a GFS volume? I intend to run GFS 6.1 with DLM. Using > RHEL4u2 at the moment. If there are any problems, it is probably a bug. If you're using rgmanager, you might not want to run it on more than 16 of the 30 (uncharted waters). Rgmanager is not needed for GFS at all, though. -- Lon From lhh at redhat.com Thu Mar 16 15:18:41 2006 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 16 Mar 2006 10:18:41 -0500 Subject: [Linux-cluster] cluster suit 4.2 In-Reply-To: References: Message-ID: <1142522321.19535.148.camel@ayanami.boston.redhat.com> On Thu, 2006-03-16 at 22:18 +0700, Theodorus wrote: > Hi all, > > > We need your help. > > > We have cluster suit 4.2 installed on RedHat AS 4.2. > > > The cluster system has been run well. The resource group can be > relocated when the one of the nodes is down. > > > But, if we disconnect all network cables of one node on purpose, the > cluster system stalled, why ? What kind of fencing are you using? -- Lon From philip.r.dana at nwp01.usace.army.mil Thu Mar 16 15:22:17 2006 From: philip.r.dana at nwp01.usace.army.mil (Philip R. Dana) Date: Thu, 16 Mar 2006 07:22:17 -0800 Subject: [Linux-cluster] RHCS4 rgmanager/clurmgrd problem Message-ID: <1142522537.12774.47.camel@nwp-wk-79033-l> We have a two node active/passive cluster running bind as our master DNS server. Shared storage is iSCSI on a NetApp Filer. The OS is CentOS 4.2. Whenever the rgmanager service on the passive node is started/restarted, the service resource on the active node fails in that named itself is shut down. The only way to recover, as near as I can tell, is to set autostart=0 in cluster.conf, reboot both nodes, then manually start the service on one of the nodes. Is this by design, or an "undocumented feature"? Any help will be greatly appreciated. TIA. From Matthew.Patton.ctr at osd.mil Thu Mar 16 15:27:15 2006 From: Matthew.Patton.ctr at osd.mil (Patton, Matthew F, CTR, OSD-PA&E) Date: Thu, 16 Mar 2006 10:27:15 -0500 Subject: [Linux-cluster] Cluster Newbie Questions........ Message-ID: Classification: UNCLASSIFIED > Lon Hohberger wrote: > > >3. which lock manager is recommended DLM or GuLM? > > DLM for small node counts. can you define "small" for us? less than a dozen? up to 50? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.yarwood at juno.co.uk Thu Mar 16 15:52:16 2006 From: ben.yarwood at juno.co.uk (Ben Yarwood) Date: Thu, 16 Mar 2006 15:52:16 -0000 Subject: [Linux-cluster] Fedora Updates Message-ID: <037201c64911$99b95710$3964a8c0@WS076> Our clusters currently run fedora 4 and we only update them using yum and the updates-released repository. It seems that apart from new kernel modules, the cluster components have not been updated since June last year. Other components seem to be updated more regularly. Can anyone shed any light on why this is the case? Cheers Ben From lhh at redhat.com Thu Mar 16 16:27:42 2006 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 16 Mar 2006 11:27:42 -0500 Subject: [Linux-cluster] cluster suit 4.2 In-Reply-To: <007d01c64914$9404bd70$4ee17bcb@golie> References: <1142522321.19535.148.camel@ayanami.boston.redhat.com> <007d01c64914$9404bd70$4ee17bcb@golie> Message-ID: <1142526462.19535.156.camel@ayanami.boston.redhat.com> On Thu, 2006-03-16 at 23:13 +0700, Paul wrote: > manual fence, because we have redundance PS, thx > You need to run fence_ack_manual on the surviving node. Note that running manual fencing in production environments is not supported. There is plenty of adequate remote power fencing hardware available which will handle multiple power supplies. -- Lon From mwill at penguincomputing.com Thu Mar 16 16:52:37 2006 From: mwill at penguincomputing.com (Michael Will) Date: Thu, 16 Mar 2006 08:52:37 -0800 Subject: [Linux-cluster] stress-testing GFS ? Message-ID: <433093DF7AD7444DA65EFAFE3987879C02A975@jellyfish.highlyscyld.com> openoffice 2.0 does support the graphs. I could not move them around like in excel but I could definitely see the default view of them. Michael -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Birger Wathne Sent: Wednesday, March 15, 2006 4:13 PM To: linux clustering Subject: Re: [Linux-cluster] stress-testing GFS ? Michael Will wrote: > iozone does test for a lot of different access patterns, and can > create nice spreadsheets including graphs from the point of view of a > single node. It also has a multiple node flag for running it across a > cluster. See -+m and -t options. It knows how to use 'rsh' and can > also be configured for any other remote execution command by setting > the enviroment variable RSH to say ssh or bpsh. > > Don't forget to post your benchmark results to this mailinglist ;-) > I used iozone and some homegrown scripts some years ago to test performance of various raid controllers as well as software raid on Sun systems. Always in single-node configurations. The easiest way to communicate the performance of a raid controller to other people was a series of 3d surface plots. Sadly, OpenOffice doesn't have those, so I had to switch to that commercial office package. I tried gnuplot, but frankly.... compare the readability of the final plot with excel and there was no comparison :-/ Perhaps Matlab... What I hoped for was something that also verified that the internal states of glm and the locking subsystem were as they should at every step of the test. Something that could certify that the hardware behaved as GFS expected it to when pushed more than test performance. -- birger -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From baostr at gmail.com Thu Mar 16 18:39:17 2006 From: baostr at gmail.com (Boris Ostrovsky) Date: Thu, 16 Mar 2006 13:39:17 -0500 Subject: [Linux-cluster] stress-testing GFS ? Message-ID: I have recently ran some very simple iozone tests on GFS (and OCFS2) and got somewhat disappointing results. I am attaching the spreadsheet. The first test was to measure single-node performance with ext3, GFS and OCFS2 partition that I mounted in a single node. The second was to use two nodes and run iozone in parallel (by hand, i.e. without -m/-t options). Single node performances were comparable in terms of wallclock time, although the benchmark values for ext3 were clearly better (so I am not sure I understand why wallclock times are so close). 2-node numbers show substantial performance degradation. Note, I didn't do any tuning, mostly because I didn't find much documentation on the subject (except that for OCFS2 I set cluster size to 1MB, which helped). The nodes were running FC4 with the disk connected to nodes via Emulex HBA. and cluster tools 1.01 I'd be very interested to hear comments on the numbers and hopefully some tuning suggestions. Thanks. -boris Date: Wed, 15 Mar 2006 14:20:28 -0800 > From: Michael Will > Subject: Re: [Linux-cluster] stress-testing GFS ? > To: linux clustering > Message-ID: <4418932C.9080001 at jellyfish.highlyscyld.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > iozone does test for a lot of different access patterns, and can create > nice spreadsheets including graphs > from the point of view of a single node. It also has a multiple node > flag for running it across a cluster. See -+m and -t > options. It knows how to use 'rsh' and can also be configured for any > other remote execution command by setting the > enviroment variable RSH to say ssh or bpsh. > > Don't forget to post your benchmark results to this mailinglist ;-) > > Michael > > Birger Wathne wrote: > > > I would like to put my cluster through a little controlled hell before > > declaring it ready for production. > > > > Is there any kind of stress-test/verification procedure to 'certify' > > shared storage with GFS? > > Ideally there would be some distributed software that could be run in > > a cluster to check that the shared storage behaves as expected under > > all kinds of load. Throughput, concurrent writing, GFS locking, file > > system locking, etc... > > Something that could interface with GFS internals to see that > > everything was 'right' at every step. > > > > Since I have seen nothing about the issue, I assume something like > > that doesn't exist, so... Any ideas on how to stress test GFS? > > Homegrown scripts? Known problems with hardware that a test should > > look for? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- random random bkwd record stride write rewrite read reread read write read rewrite read fwrite frewrite fread freread ext3 12.5min 113718 8335 91962 186143 4345 515 9612 258904 6002 112859 7230 76225 139576 gfs 13.5min 27217 8337 50117 62312 1611 604 8233 81180 5749 33633 7958 53301 40331 ocfs2 14.5min 42102 9345 65887 92481 1210 566 8136 155370 5605 41571 8699 78925 71724 gfs(n1) 46min 21467 5159 29705 35512 348 172 808 81188 4970 32680 8039 35667 58961 gfs(n2) 48min 40046 3493 29565 25093 504 327 906 81390 456 30035 4085 24953 22493 ocfs2 (n1) 38min 26813 4375 27406 27408 367 251 892 156194 5038 49998 8882 80288 111914 ocfs2 (n2) 35.5min 22756 5330 36728 29607 673 400 907 153949 953 45964 5117 34055 40158 From Matthew.Patton.ctr at osd.mil Thu Mar 16 19:16:31 2006 From: Matthew.Patton.ctr at osd.mil (Patton, Matthew F, CTR, OSD-PA&E) Date: Thu, 16 Mar 2006 14:16:31 -0500 Subject: [Linux-cluster] stress-testing GFS ? Message-ID: Classification: UNCLASSIFIED just idly wondering what the IO would be if NFS exporting the ext3 or gfs to the other node and running iozone on 2 such clients. There was no file contention was there? By that I mean each instance of iozone was writing to a different directory (both on GFS) so file-level read/write locking wasn't a factor. Presumably GFS locking is all about keeping the filesystem meta-data intact. BTW, has anyone applied the idea behind SoftUpdates to GFS? Say part of the heartbeat is a broadcast of the meta-data changes so while data blocks might by written syncronously, not every meta-change has to wait for the FC/array to commit it to disk before continuing? I'm thinking of what we did for firewall-farm syncronization which was 1xActive/NxPassive and they all could handle each other's network traffic at any time should the current master drop off with the only streams affected being those initiated since the last status update message was sent out to the passive nodes. Would it work such that the nodes vote on a meta-master and all meta-data is kept in memory and then periodically flushed? Because if each meta-change is broadcast and each node spools it to local storage, then when it's time to elect a new master the nodes can consult their transaction histories. Is there a good paper that describes the detailed inner-workings of GFS aside from having to read all the code? So far I've found this: https://open.datacore.ch/DCwiki.open/Wiki.jsp?page=GFS -------------- next part -------------- An HTML attachment was scrubbed... URL: From philip.r.dana at nwp01.usace.army.mil Thu Mar 16 20:16:41 2006 From: philip.r.dana at nwp01.usace.army.mil (Philip R. Dana) Date: Thu, 16 Mar 2006 12:16:41 -0800 Subject: [Linux-cluster] RHCS4 rgmanager/clurmgrd problem In-Reply-To: <1142522537.12774.47.camel@nwp-wk-79033-l> References: <1142522537.12774.47.camel@nwp-wk-79033-l> Message-ID: <1142540201.4163.8.camel@nwp-wk-79033-l> I found a work around. Like the gentleman with the mysql service problem a while back, I edited /etc/init.d/named on both nodes such than named stop returns 0, even though named is already stopped. I'm not smart enough, yet, to figure out why that works, but it does. On Thu, 2006-03-16 at 07:22 -0800, Philip R. Dana wrote: > We have a two node active/passive cluster running bind as our master DNS > server. Shared storage is iSCSI on a NetApp Filer. The OS is CentOS 4.2. > Whenever the rgmanager service on the passive node is started/restarted, > the service resource on the active node fails in that named itself is > shut down. The only way to recover, as near as I can tell, is to set > autostart=0 in cluster.conf, reboot both nodes, then manually start the > service on one of the nodes. Is this by design, or an "undocumented > feature"? > Any help will be greatly appreciated. TIA. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From gforte at leopard.us.udel.edu Thu Mar 16 20:26:43 2006 From: gforte at leopard.us.udel.edu (Greg Forte) Date: Thu, 16 Mar 2006 15:26:43 -0500 Subject: [Linux-cluster] RHCS4 rgmanager/clurmgrd problem In-Reply-To: <1142540201.4163.8.camel@nwp-wk-79033-l> References: <1142522537.12774.47.camel@nwp-wk-79033-l> <1142540201.4163.8.camel@nwp-wk-79033-l> Message-ID: <4419CA03.9090805@leopard.us.udel.edu> this has been covered, previously, but in brief: a) the cluster services try to stop a service before starting it when you enable it b) it expects the "/etc/init.d/service stop" command to return 0, indicating that there was no problem c) many of the stock service scripts return non-zero if you try to stop them when they're not running depending on your point of view, (c) is the "correct" behavior or not; in the case of cluster services, it's obviously not. For the purposes of cluster services, the script should only return non-zero on the 'stop' command if the service was, in fact, running, and the script failed to stop it. A better solution than simply returning 0 braindeadly would be to check the output of the script's 'status' command, and only attempt the stop if it's actually running, then return non-zero if the stop fails, 0 (success) if the stop succeeds OR it wasn't running in the first place. But that's a lot of work. ;-) -g Philip R. Dana wrote: > I found a work around. Like the gentleman with the mysql service problem > a while back, I edited /etc/init.d/named on both nodes such than named > stop returns 0, even though named is already stopped. I'm not smart > enough, yet, to figure out why that works, but it does. > > On Thu, 2006-03-16 at 07:22 -0800, Philip R. Dana wrote: >> We have a two node active/passive cluster running bind as our master DNS >> server. Shared storage is iSCSI on a NetApp Filer. The OS is CentOS 4.2. >> Whenever the rgmanager service on the passive node is started/restarted, >> the service resource on the active node fails in that named itself is >> shut down. The only way to recover, as near as I can tell, is to set >> autostart=0 in cluster.conf, reboot both nodes, then manually start the >> service on one of the nodes. Is this by design, or an "undocumented >> feature"? >> Any help will be greatly appreciated. TIA. >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Greg Forte gforte at udel.edu IT - User Services University of Delaware 302-831-1982 Newark, DE From philip.r.dana at nwp01.usace.army.mil Thu Mar 16 21:48:01 2006 From: philip.r.dana at nwp01.usace.army.mil (Philip R. Dana) Date: Thu, 16 Mar 2006 13:48:01 -0800 Subject: [Linux-cluster] RHCS4 rgmanager/clurmgrd problem In-Reply-To: <4419CA03.9090805@leopard.us.udel.edu> References: <1142522537.12774.47.camel@nwp-wk-79033-l> <1142540201.4163.8.camel@nwp-wk-79033-l> <4419CA03.9090805@leopard.us.udel.edu> Message-ID: <1142545681.4163.11.camel@nwp-wk-79033-l> Greg: Your explanation clarified for me what's happening and what needs to be done. Thanks much. On Thu, 2006-03-16 at 15:26 -0500, Greg Forte wrote: > this has been covered, previously, but in brief: > > a) the cluster services try to stop a service before starting it when > you enable it > b) it expects the "/etc/init.d/service stop" command to return 0, > indicating that there was no problem > c) many of the stock service scripts return non-zero if you try to stop > them when they're not running > > depending on your point of view, (c) is the "correct" behavior or not; > in the case of cluster services, it's obviously not. For the purposes > of cluster services, the script should only return non-zero on the > 'stop' command if the service was, in fact, running, and the script > failed to stop it. A better solution than simply returning 0 > braindeadly would be to check the output of the script's 'status' > command, and only attempt the stop if it's actually running, then return > non-zero if the stop fails, 0 (success) if the stop succeeds OR it > wasn't running in the first place. But that's a lot of work. ;-) > > -g > > Philip R. Dana wrote: > > I found a work around. Like the gentleman with the mysql service problem > > a while back, I edited /etc/init.d/named on both nodes such than named > > stop returns 0, even though named is already stopped. I'm not smart > > enough, yet, to figure out why that works, but it does. > > > > On Thu, 2006-03-16 at 07:22 -0800, Philip R. Dana wrote: > >> We have a two node active/passive cluster running bind as our master DNS > >> server. Shared storage is iSCSI on a NetApp Filer. The OS is CentOS 4.2. > >> Whenever the rgmanager service on the passive node is started/restarted, > >> the service resource on the active node fails in that named itself is > >> shut down. The only way to recover, as near as I can tell, is to set > >> autostart=0 in cluster.conf, reboot both nodes, then manually start the > >> service on one of the nodes. Is this by design, or an "undocumented > >> feature"? > >> Any help will be greatly appreciated. TIA. > >> > >> -- > >> Linux-cluster mailing list > >> Linux-cluster at redhat.com > >> https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > From zhendershot at cranel.com Thu Mar 16 21:51:21 2006 From: zhendershot at cranel.com (Hendershot, Zach) Date: Thu, 16 Mar 2006 16:51:21 -0500 Subject: [Linux-cluster] (no subject) Message-ID: All, I'm playing around with Red Hat Cluster Suite and I had a quick question. I was solely reading the manual for CS4 up until a few hours ago. Then I stumbled upon the CS3 manual. WOW, there is so much more detail (for example: how do you use the clusvcadm command). Why are people who only read the CS4 manual left out from all the good information. Additionally, of course some of the CS3 stuff is out of date so its additional information is a mixed bag. Also, I'm having trouble with service failover. I have a script that accepts {start, stop, status} for Apache. I manually fail Apache, and CS basically puts the service in the failed state because it can't stop the service (the stop command returns a 1 status, because it cant stop an already stopped service). The service never gets failed over to another node. And back to the documentation, why is something basic like script creation and API left out? Am I looking in the wrong place or overlooking some of this information? Maybe I'm just too used to VCS? Thank you all very much for your help. Have a great day. -------------- Zach Hendershot Software Engineer Cranel, Incorporated. Phone: 614.318.4288 Fax: 614.431.8388 Email: zhendershot at cranel.com Technology. Integrity. Focus. From zhendershot at cranel.com Thu Mar 16 22:03:25 2006 From: zhendershot at cranel.com (Hendershot, Zach) Date: Thu, 16 Mar 2006 17:03:25 -0500 Subject: [Linux-cluster] RE: Message-ID: All, Sorry about the missing subject, I was hasty. But, I just caught a recent conversation dealing with the service script information, so I've got that now. But I do still have questions about the inconsistencies with the documentation, and I would appreciate any enlightening anybody could shed. Thanks. Zach -----Original Message----- From: Hendershot, Zach Sent: Thursday, March 16, 2006 4:51 PM To: 'linux-cluster at redhat.com' Subject: All, I'm playing around with Red Hat Cluster Suite and I had a quick question. I was solely reading the manual for CS4 up until a few hours ago. Then I stumbled upon the CS3 manual. WOW, there is so much more detail (for example: how do you use the clusvcadm command). Why are people who only read the CS4 manual left out from all the good information. Additionally, of course some of the CS3 stuff is out of date so its additional information is a mixed bag. Also, I'm having trouble with service failover. I have a script that accepts {start, stop, status} for Apache. I manually fail Apache, and CS basically puts the service in the failed state because it can't stop the service (the stop command returns a 1 status, because it cant stop an already stopped service). The service never gets failed over to another node. And back to the documentation, why is something basic like script creation and API left out? Am I looking in the wrong place or overlooking some of this information? Maybe I'm just too used to VCS? Thank you all very much for your help. Have a great day. -------------- Zach Hendershot Software Engineer Cranel, Incorporated. Phone: 614.318.4288 Fax: 614.431.8388 Email: zhendershot at cranel.com Technology. Integrity. Focus. From lhh at redhat.com Thu Mar 16 23:59:06 2006 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 16 Mar 2006 18:59:06 -0500 Subject: [Linux-cluster] Cluster Newbie Questions........ In-Reply-To: References: Message-ID: <1142553546.19535.199.camel@ayanami.boston.redhat.com> On Thu, 2006-03-16 at 10:27 -0500, Patton, Matthew F, CTR, OSD-PA&E wrote: > Classification: UNCLASSIFIED > > > Lon Hohberger wrote: > > > > >3. which lock manager is recommended DLM or GuLM? > > > > DLM for small node counts. > > can you define "small" for us? less than a dozen? up to 50? DLM should theoretically be very scalable, but I do not know what the largest tested node count is at this point. I am pretty sure at least 32 should work fine. -- Lon From lhh at redhat.com Thu Mar 16 23:59:57 2006 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 16 Mar 2006 18:59:57 -0500 Subject: [Linux-cluster] RHCS4 rgmanager/clurmgrd problem In-Reply-To: <1142540201.4163.8.camel@nwp-wk-79033-l> References: <1142522537.12774.47.camel@nwp-wk-79033-l> <1142540201.4163.8.camel@nwp-wk-79033-l> Message-ID: <1142553597.19535.201.camel@ayanami.boston.redhat.com> On Thu, 2006-03-16 at 12:16 -0800, Philip R. Dana wrote: > I found a work around. Like the gentleman with the mysql service problem > a while back, I edited /etc/init.d/named on both nodes such than named > stop returns 0, even though named is already stopped. I'm not smart > enough, yet, to figure out why that works, but it does. *all* init scripts should do this. -- Lon From gforte at leopard.us.udel.edu Fri Mar 17 14:37:42 2006 From: gforte at leopard.us.udel.edu (Greg Forte) Date: Fri, 17 Mar 2006 09:37:42 -0500 Subject: [Linux-cluster] shared cluster.conf? Message-ID: <441AC9B6.2050403@leopard.us.udel.edu> Is there any reason not to put cluster.conf on a shared filesystem that's mounted in fstab, and symlink /etc/cluster/cluster.conf to that shared location? Then one would only have to run cman_tool version -r after updating the conf ... or am I missing some reason why this is a bad idea? -g Greg Forte gforte at udel.edu IT - User Services University of Delaware 302-831-1982 Newark, DE From zhendershot at cranel.com Fri Mar 17 14:41:43 2006 From: zhendershot at cranel.com (Hendershot, Zach) Date: Fri, 17 Mar 2006 09:41:43 -0500 Subject: [Linux-cluster] shared cluster.conf? Message-ID: If you are running CS4 at least, all you have to do is make your changes and then run "ccs_tool update /etc/cluster/cluster.conf" and it goes out to all the nodes automatically. With this functionality I wouldn't want to mess with a shared filesystem any potential mess you could get into. Zach Hendershot -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Greg Forte Sent: Friday, March 17, 2006 9:38 AM To: linux clustering Subject: [Linux-cluster] shared cluster.conf? Is there any reason not to put cluster.conf on a shared filesystem that's mounted in fstab, and symlink /etc/cluster/cluster.conf to that shared location? Then one would only have to run cman_tool version -r after updating the conf ... or am I missing some reason why this is a bad idea? -g Greg Forte gforte at udel.edu IT - User Services University of Delaware 302-831-1982 Newark, DE -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From carlopmart at gmail.com Fri Mar 17 17:14:52 2006 From: carlopmart at gmail.com (carlopmart) Date: Fri, 17 Mar 2006 18:14:52 +0100 Subject: [Linux-cluster] GFS at startup Message-ID: <441AEE8C.20404@gmail.com> Hi all, I have a simple doubt: where I can put gfs shared file system imported from gnbd's server: on fstab, on rc.local or on cluster.conf? i need this filesystem up before configured services on cluster.conf starts up. Thanks. -- CL Martinez carlopmart {at} gmail {d0t} com From Matthew.Patton.ctr at osd.mil Fri Mar 17 17:17:13 2006 From: Matthew.Patton.ctr at osd.mil (Patton, Matthew F, CTR, OSD-PA&E) Date: Fri, 17 Mar 2006 12:17:13 -0500 Subject: [Linux-cluster] shared cluster.conf? Message-ID: Classification: UNCLASSIFIED > Is there any reason not to put cluster.conf on a shared filesystem > that's mounted in fstab, and symlink /etc/cluster/cluster.conf to that wondered that myself. you can put the file on a shared volume (say /etc was actually on GFS for each node). "update" should ideally have a flag or better yet, a counterpart that says to "just reread the file and don't try to write a new one." -------------- next part -------------- An HTML attachment was scrubbed... URL: From grimme at atix.de Fri Mar 17 18:59:57 2006 From: grimme at atix.de (Marc Grimme) Date: Fri, 17 Mar 2006 19:59:57 +0100 Subject: [Linux-cluster] shared cluster.conf? In-Reply-To: References: Message-ID: <200603171959.57644.grimme@atix.de> On Friday 17 March 2006 18:17, Patton, Matthew F, CTR, OSD-PA&E wrote: > Classification: UNCLASSIFIED > > > Is there any reason not to put cluster.conf on a shared filesystem > > that's mounted in fstab, and symlink /etc/cluster/cluster.conf to that > > wondered that myself. you can put the file on a shared volume (say /etc was > actually on GFS for each node). "update" should ideally have a flag or > better yet, a counterpart that says to "just reread the file and don't try > to write a new one." until now we didn't have any problems with cluster.conf on GFS. We are using it for sharedroots and did not encounter any problems with it. Even the ccs_tool update and cman_tool version works. Regards Marc. -- Gruss / Regards, Marc Grimme Phone: +49-89 121 409-54 http://www.atix.de/ http://www.open-sharedroot.org/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From lhh at redhat.com Fri Mar 17 19:24:46 2006 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 17 Mar 2006 14:24:46 -0500 Subject: [Linux-cluster] shared cluster.conf? In-Reply-To: <441AC9B6.2050403@leopard.us.udel.edu> References: <441AC9B6.2050403@leopard.us.udel.edu> Message-ID: <1142623486.8266.54.camel@ayanami.boston.redhat.com> On Fri, 2006-03-17 at 09:37 -0500, Greg Forte wrote: > Is there any reason not to put cluster.conf on a shared filesystem > that's mounted in fstab, and symlink /etc/cluster/cluster.conf to that > shared location? Then one would only have to run cman_tool version -r > after updating the conf ... or am I missing some reason > why this is a bad idea? Circular dependency: +-> configuration | v | cluster infrastructure | v +-- gfs -- Lon From lhh at redhat.com Fri Mar 17 19:27:24 2006 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 17 Mar 2006 14:27:24 -0500 Subject: [Linux-cluster] GFS at startup In-Reply-To: <441AEE8C.20404@gmail.com> References: <441AEE8C.20404@gmail.com> Message-ID: <1142623644.8266.56.camel@ayanami.boston.redhat.com> On Fri, 2006-03-17 at 18:14 +0100, carlopmart wrote: > Hi all, > > I have a simple doubt: where I can put gfs shared file system imported > from gnbd's server: on fstab, on rc.local or on cluster.conf? i need > this filesystem up before configured services on cluster.conf starts up. If you put them in fstab, the GFS volumes should be mounted before rgmanager starts. -- Lon From ptader at fnal.gov Fri Mar 17 21:38:59 2006 From: ptader at fnal.gov (Paul Tader) Date: Fri, 17 Mar 2006 15:38:59 -0600 Subject: [Linux-cluster] lock_dlm kernel panics Message-ID: <441B2C73.3010202@fnal.gov> We're experiencing random kernel panics that all seem to be attributed to the lock_dlm module. (panic text from 3 different systems): /var/log/messages.1:Mar 9 10:01:45 node1 kernel: EIP is at do_dlm_lock+0x134/0x14e [lock_dlm] /var/log/messages.1:Mar 6 16:33:41 node1 kernel: EIP is at do_dlm_unlock+0x8b/0xa0 [lock_dlm] /var/log/messages.1:Mar 7 22:28:53 node1 kernel: EIP is at do_dlm_lock+0x134/0x14e [lock_dlm] /var/log/messages.3:Feb 22 13:35:07 node2 kernel: EIP is at do_dlm_lock+0x134/0x14e [lock_dlm] /var/log/messages.4:Feb 18 12:17:04 node2 kernel: EIP is at do_dlm_unlock+0x8b/0xa0 [lock_dlm] /var/log/messages.3:Feb 23 04:46:01 node3 kernel: EIP is at do_dlm_lock+0x134/0x14e [lock_dlm] On average, nodes stay up for about a week. The work load is steady and is mostly disk I/O. These nodes were running RHES3 with GFS 6.0. During that setup, we experienced much more frequent panics, even when the nodes weren't being used. My thought is that this is a hardware problem. Disk array, fibre switch or HBA? But in the hopes that there is some addition GFS turning or diagnostics I can perform that will either lead me to a hardware problem or GFS configuration change, I'm posting this message. Software: - RHES4 - GFS-6.1.2-0 - GFS-kernel-2.6.9-49.1 - One, 1Tb GFS partition Hardware: - 5 nodes total - Dual Xeon CPU's 2.66GHz - 2 Gb ram - 1 Gb eth0 - QLogic QLA2200 Latest complete panic message: --------------------------- Mar 17 11:38:02 nodename kernel: Mar 17 11:38:02 nodename kernel: d0 purged 0 requests Mar 17 11:38:02 nodename kernel: d0 mark waiting requests Mar 17 11:38:02 nodename kernel: d0 marked 0 requests Mar 17 11:38:02 nodename kernel: d0 recover event 17 done Mar 17 11:38:02 nodename kernel: d0 move flags 0,0,1 ids 14,17,17 Mar 17 11:38:02 nodename kernel: d0 process held requests Mar 17 11:38:02 nodename kernel: d0 processed 0 requests Mar 17 11:38:02 nodename kernel: d0 resend marked requests Mar 17 11:38:02 nodename kernel: d0 resent 0 requests Mar 17 11:38:02 nodename kernel: d0 recover event 17 finished Mar 17 11:38:02 nodename kernel: d0 send einval to 5 Mar 17 11:38:02 nodename kernel: d0 send einval to 5 Mar 17 11:38:02 nodename kernel: d0 (1983) req reply einval 2da2006d fr 2 r 2 5 9 Mar 17 11:38:02 nodename kernel: d0 send einval to 5 Mar 17 11:38:02 nodename kernel: d0 send einval to 3 Mar 17 11:38:02 nodename kernel: d0 (1983) req reply einval 410803b0 fr 5 r 5 5 a Mar 17 11:38:02 nodename kernel: d0 (1983) req reply einval 456f03d1 fr 2 r 2 5 1 Mar 17 11:38:02 nodename kernel: d0 send einval to 5 Mar 17 11:38:02 nodename kernel: d0 send einval to 5 Mar 17 11:38:02 nodename kernel: d0 send einval to 3 Mar 17 11:38:02 nodename kernel: d0 send einval to 3 Mar 17 11:38:02 nodename kernel: d0 (1983) req reply einval aca103f2 fr 5 r 5 5 2 Mar 17 11:38:02 nodename kernel: d0 grant lock on lockqueue 3 Mar 17 11:38:02 nodename kernel: d0 process_lockqueue_reply id bbfe0396 state 0 Mar 17 11:38:02 nodename kernel: d0 (1983) req reply einval d2d20215 fr 2 r 2 5 9 Mar 17 11:38:02 nodename kernel: d0 (1983) req reply einval d5a60059 fr 5 r 5 5 d Mar 17 11:38:02 nodename kernel: d0 (1983) req reply einval d886008f fr 3 r 3 5 e Mar 17 11:38:02 nodename kernel: d0 (1983) req reply einval 3130220 fr 2 r 2 5 c3 Mar 17 11:38:02 nodename kernel: d0 unlock fe20017a no id Mar 17 11:38:02 nodename kernel: 1976 pr_start last_stop 0 last_start 4 last_finish 0 Mar 17 11:38:02 nodename kernel: 1976 pr_start count 4 type 2 event 4 flags 250 Mar 17 11:38:02 nodename kernel: 1976 claim_jid 2 Mar 17 11:38:02 nodename kernel: 1976 pr_start 4 done 1 Mar 17 11:38:02 nodename kernel: 1976 pr_finish flags 5a Mar 17 11:38:02 nodename kernel: 1968 recovery_done jid 2 msg 309 a Mar 17 11:38:02 nodename kernel: 1968 recovery_done nodeid 4 flg 18 Mar 17 11:38:02 nodename kernel: 1976 pr_start last_stop 4 last_start 8 last_finish 4 Mar 17 11:38:02 nodename kernel: 1976 pr_start count 5 type 2 event 8 flags 21a Mar 17 11:38:02 nodename kernel: 1976 pr_start 8 done 1 Mar 17 11:38:02 nodename kernel: 1976 pr_finish flags 1a Mar 17 11:38:02 nodename kernel: 1976 rereq 3,624b610 id 7f1d022e 5,0 Mar 17 11:38:02 nodename kernel: 1976 pr_start last_stop 8 last_start 9 last_finish 8 Mar 17 11:38:02 nodename kernel: 1976 pr_start count 4 type 1 event 9 flags 21a Mar 17 11:38:02 nodename kernel: 1976 pr_start cb jid 0 id 2 Mar 17 11:38:02 nodename kernel: 1976 pr_start 9 done 0 Mar 17 11:38:02 nodename kernel: 1980 recovery_done jid 0 msg 308 11a Mar 17 11:38:02 nodename kernel: 1980 recovery_done nodeid 2 flg 1b Mar 17 11:38:02 nodename kernel: 1980 recovery_done start_done 9 Mar 17 11:38:02 nodename kernel: 1976 rereq 3,263e6dd id 7e2d01b9 3,0 Mar 17 11:38:02 nodename kernel: 1977 pr_finish flags 1a Mar 17 11:38:02 nodename kernel: 1976 pr_start last_stop 9 last_start 13 last_finish 9 Mar 17 11:38:02 nodename kernel: 1976 pr_start count 5 type 2 event 13 flags 21a Mar 17 11:38:02 nodename kernel: 1976 pr_start 13 done 1 Mar 17 11:38:02 nodename kernel: 1976 pr_finish flags 1a Mar 17 11:38:02 nodename kernel: 1976 pr_start last_stop 13 last_start 14 last_finish 13 Mar 17 11:38:02 nodename kernel: 1976 pr_start count 4 type 1 event 14 flags 21a Mar 17 11:38:02 nodename kernel: 1976 pr_start cb jid 4 id 5 Mar 17 11:38:02 nodename kernel: 1976 pr_start 14 done 0 Mar 17 11:38:02 nodename kernel: 1980 recovery_done jid 4 msg 308 11a Mar 17 11:38:02 nodename kernel: 1980 recovery_done nodeid 5 flg 1b Mar 17 11:38:02 nodename kernel: 1980 recovery_done start_done 14 Mar 17 11:38:02 nodename kernel: 1977 pr_finish flags 1a Mar 17 11:38:02 nodename kernel: 1976 pr_start last_stop 14 last_start 18 last_finish 14 Mar 17 11:38:02 nodename kernel: 1976 pr_start count 5 type 2 event 18 flags 21a Mar 17 11:38:02 nodename kernel: 1976 pr_start 18 done 1 Mar 17 11:38:02 nodename kernel: 1976 pr_finish flags 1a Mar 17 11:38:02 nodename kernel: Mar 17 11:38:02 nodename kernel: lock_dlm: Assertion failed on line 357 of file /mnt/src/4/BUILD/gfs-kernel-2.6.9-45/smp/src/dlm/lock.c Mar 17 11:38:02 nodename kernel: lock_dlm: assertion: "!error" Mar 17 11:38:02 nodename kernel: lock_dlm: time = 783572508 Mar 17 11:38:03 nodename kernel: d0: error=-22 num=3,a458688 lkf=9 flags=84 Mar 17 11:38:03 nodename kernel: Mar 17 11:38:03 nodename kernel: ------------[ cut here ]------------ Mar 17 11:38:03 nodename kernel: kernel BUG at /mnt/src/4/BUILD/gfs-kernel-2.6.9-45/smp/src/dlm/lock.c:357! Mar 17 11:38:03 nodename kernel: invalid operand: 0000 [#1] Mar 17 11:38:03 nodename kernel: SMP Mar 17 11:38:03 nodename kernel: Modules linked in: parport_pc lp parport autofs4 lock_dlm(U) gfs(U) lock_harness(U) nfs lockd dlm(U) cman(U) md5 ipv6 sunrpc dm_mirror button battery ac uhci_hcd ehci_hcd e100 mii e1000 floppy ext3 jbd dm_mod qla2200 qla2xxx scsi_transport_fc sd_mod scsi_mod Mar 17 11:38:03 nodename kernel: CPU: 1 Mar 17 11:38:03 nodename kernel: EIP: 0060:[] Not tainted VLI Mar 17 11:38:03 nodename kernel: EFLAGS: 00010246 (2.6.9-22.0.2.ELsmp) Mar 17 11:38:03 nodename kernel: EIP is at do_dlm_unlock+0x8b/0xa0 [lock_dlm] Mar 17 11:38:03 nodename kernel: eax: 00000001 ebx: f518d380 ecx: f5857f2c edx: f8bc0155Mar 17 11:38:03 nodename kernel: esi: ffffffea edi: f518d380 ebp: f8c3f000 esp: f5857f28Mar 17 11:38:03 nodename kernel: ds: 007b es: 007b ss: 0068 Mar 17 11:38:03 nodename kernel: Process gfs_glockd (pid: 1979, threadinfo=f5857000 task=f5b588b0) Mar 17 11:38:03 nodename kernel: Stack: f8bc0155 f8c3f000 00000003 f8bbb893 f8d19612 00000001 f514c268 f514c24c Mar 17 11:38:03 nodename kernel: f8d0f89e f8d44440 f4bf0cc0 f514c24c f8d44440 f514c24c f8d0ed97 f514c24c Mar 17 11:38:03 nodename kernel: 00000001 f514c2e0 f8d0ee4e f514c24c f514c268 f8d0ef71 00000001 f514c268 Mar 17 11:38:03 nodename kernel: Call Trace: Mar 17 11:38:03 nodename kernel: [] lm_dlm_unlock+0x14/0x1c [lock_dlm] Mar 17 11:38:03 nodename kernel: [] gfs_lm_unlock+0x2c/0x42 [gfs] Mar 17 11:38:03 nodename kernel: [] gfs_glock_drop_th+0xf3/0x12d [gfs] Mar 17 11:38:03 nodename kernel: [] rq_demote+0x7f/0x98 [gfs] Mar 17 11:38:03 nodename kernel: [] run_queue+0x5a/0xc1 [gfs] Mar 17 11:38:03 nodename kernel: [] unlock_on_glock+0x1f/0x28 [gfs] Mar 17 11:38:03 nodename kernel: [] gfs_reclaim_glock+0xc3/0x13c [gfs] Mar 17 11:38:03 nodename kernel: [] gfs_glockd+0x39/0xde [gfs] Mar 17 11:38:03 nodename kernel: [] default_wake_function+0x0/0xc Mar 17 11:38:03 nodename kernel: [] ret_from_fork+0x6/0x14 Mar 17 11:38:03 nodename kernel: [] default_wake_function+0x0/0xc Mar 17 11:38:03 nodename kernel: [] gfs_glockd+0x0/0xde [gfs] Mar 17 11:38:03 nodename kernel: [] kernel_thread_helper+0x5/0xb Mar 17 11:38:03 nodename kernel: Code: 73 34 8b 03 ff 73 2c ff 73 08 ff 73 04 ff 73 0c 56 ff 70 18 68 4d 02 bc f8 e8 84 6c 56 c7 83 c4 34 68 55 01 bc f8 e8 77 6c 56 c7 <0f> 0b 65 01 a2 00 bc f8 68 57 01 bc f8 e8 32 64 56 c7 5b 5e c3 Mar 17 11:38:03 nodename kernel: <0>Fatal exception: panic in 5 seconds Mar 17 13:08:01 nodename syslogd 1.4.1: restart. Thanks, Paul -- =========================================================================== Paul Tader Computing Div/CSS Dept Fermi National Accelerator Lab; PO Box 500 Batavia, IL 60510-0500 From teigland at redhat.com Fri Mar 17 22:40:43 2006 From: teigland at redhat.com (David Teigland) Date: Fri, 17 Mar 2006 16:40:43 -0600 Subject: [Linux-cluster] lock_dlm kernel panics In-Reply-To: <441B2C73.3010202@fnal.gov> References: <441B2C73.3010202@fnal.gov> Message-ID: <20060317224043.GC29244@redhat.com> On Fri, Mar 17, 2006 at 03:38:59PM -0600, Paul Tader wrote: > Mar 17 11:38:02 nodename kernel: d0 unlock fe20017a no id GFS is trying to unlock a lock that doesn't exist which causes the panic. We know this happens if cman shuts down the dlm while it's in use (cman does this if it's lost connection with the cluster.) There's some new output in the RHEL4U3 dlm that should tell us if that's in fact what's happening or if there's some other cause that we need to uncover. So, you should look on all nodes for any cman messages in /var/log/messages or the console. And when you're using the latest version look for the new dlm message "WARNING: dlm_emergency_shutdown". Dave From mag.andersen at gmail.com Fri Mar 17 22:53:44 2006 From: mag.andersen at gmail.com (Magnus Andersen) Date: Fri, 17 Mar 2006 17:53:44 -0500 Subject: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 Message-ID: <5ea165840603171453v2e8ba0f6m3630c2bd3eec7f7c@mail.gmail.com> Hi All, I've successfully installed and configured GFS on my three nodes, but when I try to mount the filesystem the prompt hangs until I kill the mount command. All servers are running RHEL 3 AS/ES U6 with the 2.4.21-37.0.1.ELsmp kernel and are connected to a MSA1500 SAN via FC. I've installed the following GFS rpms: [root at oradw root]# rpm -qa | grep -i gfs GFS-modules-6.0.2.27-0.1 GFS-modules-smp-6.0.2.27-0.1 GFS-6.0.2.27-0.1 Here is my pool configuration files and the output from pool_tool -s [root at backup gfs]# cat cluster_cca.cfg poolname cluster_cca subpools 1 subpool 0 0 1 pooldevice 0 0 /dev/sda1 [root at backup gfs]# cat pool0.cfg poolname pool_gfs1 subpools 1 subpool 0 0 1 pooldevice 0 0 /dev/sda2 [root at backup gfs]# cat pool1.cfg poolname pool_gfs2 subpools 1 subpool 0 0 1 pooldevice 0 0 /dev/sdb [root at backup gfs]# pool_tool -s Device Pool Label ====== ========== /dev/pool/cluster_cca <- CCA device -> /dev/pool/pool_gfs1 <- GFS filesystem -> /dev/pool/pool_gfs2 <- GFS filesystem -> /dev/cciss/c0d0 <- partition information -> /dev/cciss/c0d0p1 <- EXT2/3 filesystem -> /dev/cciss/c0d0p2 <- swap device -> /dev/cciss/c0d0p3 <- lvm1 subdevice -> /dev/sda <- partition information -> /dev/sda1 cluster_cca /dev/sda2 pool_gfs1 /dev/sdb pool_gfs2 Here are my ccs files. [root at backup cluster_cca]# cat cluster.ccs cluster { name = "cluster_cca" lock_gulm { servers = ["backup", "oradw", "gistest2"] } } [root at backup cluster_cca]# cat fence.ccs fence_devices { manual { agent = "fence_manual" } } [root at backup cluster_cca]# cat nodes.ccs nodes { backup { ip_interfaces { eth1 = "10.0.0.1" } fence { man { manual { ipaddr = "10.0.0.1" } } } } oradw { ip_interfaces { eth4 = "10.0.0.2" } fence { man { manual { ipaddr = "10.0.0.2" } } } } gistest2 { ip_interfaces { eth0 = "10.0.0.3" } fence { man { manual { ipaddr = "10.0.0.3" } } } } } Here is the command I used to create the filesystem: gfs_mkfs -p lock_gulm -t cluster_cca:pool_gfs2 -j 10 /dev/pool/pool_gfs2 Mount command that hangs: mount -t gfs /dev/pool/pool_gfs2 /gfs2 Here is the output I see in my messages log file. I see the last 5 lines repeated for each time I tried to mount the filesystem. Mar 17 15:47:05 backup ccsd[2645]: Starting ccsd 6.0.2.27: Mar 17 15:47:05 backup ccsd[2645]: Built: Jan 30 2006 15:28:33 Mar 17 15:47:05 backup ccsd[2645]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Mar 17 15:48:10 backup lock_gulmd[2652]: Starting lock_gulmd 6.0.2.27. (built Jan 30 2006 15:28:54) Copyright (C) 2004 Red Hat, Inc. All rights reserved. Mar 17 15:48:10 backup lock_gulmd[2652]: You are running in Fail-over mode. Mar 17 15:48:10 backup lock_gulmd[2652]: I am (backup) with ip (127.0.0.1) Mar 17 15:48:10 backup lock_gulmd[2652]: Forked core [2653]. Mar 17 15:48:11 backup lock_gulmd[2652]: Forked locktable [2654]. Mar 17 15:48:12 backup lock_gulmd[2652]: Forked ltpx [2655]. Mar 17 15:48:12 backup lock_gulmd_core[2653]: I see no Masters, So I am Arbitrating until enough Slaves talk to me. Mar 17 15:48:12 backup lock_gulmd_core[2653]: Could not send quorum update to slave backup Mar 17 15:48:12 backup lock_gulmd_core[2653]: New generation of server state. (1142628492484630) Mar 17 15:48:12 backup lock_gulmd_LTPX[2655]: New Master at backup:127.0.0.1 Mar 17 15:52:14 backup kernel: Lock_Harness 6.0.2.27 (built Jan 30 2006 15:32:58) installed Mar 17 15:52:14 backup kernel: GFS 6.0.2.27 (built Jan 30 2006 15:32:20) installed Mar 17 15:52:15 backup kernel: Gulm 6.0.2.27 (built Jan 30 2006 15:32:54) installed Mar 17 15:54:51 backup kernel: lock_gulm: ERROR cm_login failed. -512 Mar 17 15:54:51 backup kernel: lock_gulm: ERROR Got a -512 trying to start the threads. Mar 17 15:54:51 backup lock_gulmd_core[2653]: Error on xdr (GFS Kernel Interface:127.0.0.1 idx:3 fd:8): (-104:104:Connection reset by peer) Mar 17 15:54:51 backup kernel: lock_gulm: fsid=cluster_cca:gfs1: Exiting gulm_mount with errors -512 Mar 17 15:54:51 backup kernel: GFS: can't mount proto = lock_gulm, table = cluster_cca:gfs1, hostdata = Result from gulm_tool: [root at backup gfs]# gulm_tool nodelist backup Name: backup ip = 127.0.0.1 state = Logged in mode = Arbitrating missed beats = 0 last beat = 1142632189718986 delay avg = 10019686 max delay = 10019735 I'm a newbie to clusters and I have no clue where to look next. If any other information is needed let me know. Thanks, -- Magnus Andersen Systems Administrator / Oracle DBA Walker & Associates, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Britt.Treece at savvis.net Fri Mar 17 23:09:12 2006 From: Britt.Treece at savvis.net (Treece, Britt) Date: Fri, 17 Mar 2006 17:09:12 -0600 Subject: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B90152A1D4@s228130hz1ew08.apptix-01.savvis.net> Magnus, Try starting ccsd and lock_gulmd on all three servers. Once these start you should be able to see all three in gulm_tool nodelist localhost. At that point you should be able to mount your GFS pool vol's. Your lock cluster has to have a quorum of greater than half the servers configured in cluster.ccs, so at least 2 in your case before it will allow a GFS vol to be mounted. Regards, Britt ________________________________ From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Magnus Andersen Sent: Friday, March 17, 2006 4:54 PM To: linux-cluster at redhat.com Subject: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 Hi All, I've successfully installed and configured GFS on my three nodes, but when I try to mount the filesystem the prompt hangs until I kill the mount command. All servers are running RHEL 3 AS/ES U6 with the 2.4.21-37.0.1.ELsmp kernel and are connected to a MSA1500 SAN via FC. I've installed the following GFS rpms: [root at oradw root]# rpm -qa | grep -i gfs GFS-modules-6.0.2.27-0.1 GFS-modules-smp-6.0.2.27-0.1 GFS-6.0.2.27-0.1 Here is my pool configuration files and the output from pool_tool -s [root at backup gfs]# cat cluster_cca.cfg poolname cluster_cca subpools 1 subpool 0 0 1 pooldevice 0 0 /dev/sda1 [root at backup gfs]# cat pool0.cfg poolname pool_gfs1 subpools 1 subpool 0 0 1 pooldevice 0 0 /dev/sda2 [root at backup gfs]# cat pool1.cfg poolname pool_gfs2 subpools 1 subpool 0 0 1 pooldevice 0 0 /dev/sdb [root at backup gfs]# pool_tool -s Device Pool Label ====== ========== /dev/pool/cluster_cca <- CCA device -> /dev/pool/pool_gfs1 <- GFS filesystem -> /dev/pool/pool_gfs2 <- GFS filesystem -> /dev/cciss/c0d0 <- partition information -> /dev/cciss/c0d0p1 <- EXT2/3 filesystem -> /dev/cciss/c0d0p2 <- swap device -> /dev/cciss/c0d0p3 <- lvm1 subdevice -> /dev/sda <- partition information -> /dev/sda1 cluster_cca /dev/sda2 pool_gfs1 /dev/sdb pool_gfs2 Here are my ccs files. [root at backup cluster_cca]# cat cluster.ccs cluster { name = "cluster_cca" lock_gulm { servers = ["backup", "oradw", "gistest2"] } } [root at backup cluster_cca]# cat fence.ccs fence_devices { manual { agent = "fence_manual" } } [root at backup cluster_cca]# cat nodes.ccs nodes { backup { ip_interfaces { eth1 = "10.0.0.1" } fence { man { manual { ipaddr = " 10.0.0.1" } } } } oradw { ip_interfaces { eth4 = " 10.0.0.2" } fence { man { manual { ipaddr = " 10.0.0.2" } } } } gistest2 { ip_interfaces { eth0 = " 10.0.0.3" } fence { man { manual { ipaddr = " 10.0.0.3" } } } } } Here is the command I used to create the filesystem: gfs_mkfs -p lock_gulm -t cluster_cca:pool_gfs2 -j 10 /dev/pool/pool_gfs2 Mount command that hangs: mount -t gfs /dev/pool/pool_gfs2 /gfs2 Here is the output I see in my messages log file. I see the last 5 lines repeated for each time I tried to mount the filesystem. Mar 17 15:47:05 backup ccsd[2645]: Starting ccsd 6.0.2.27 : Mar 17 15:47:05 backup ccsd[2645]: Built: Jan 30 2006 15:28:33 Mar 17 15:47:05 backup ccsd[2645]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Mar 17 15:48:10 backup lock_gulmd[2652]: Starting lock_gulmd 6.0.2.27 . (built Jan 30 2006 15:28:54) Copyright (C) 2004 Red Hat, Inc. All rights reserved. Mar 17 15:48:10 backup lock_gulmd[2652]: You are running in Fail-over mode. Mar 17 15:48:10 backup lock_gulmd[2652]: I am (backup) with ip (127.0.0.1 ) Mar 17 15:48:10 backup lock_gulmd[2652]: Forked core [2653]. Mar 17 15:48:11 backup lock_gulmd[2652]: Forked locktable [2654]. Mar 17 15:48:12 backup lock_gulmd[2652]: Forked ltpx [2655]. Mar 17 15:48:12 backup lock_gulmd_core[2653]: I see no Masters, So I am Arbitrating until enough Slaves talk to me. Mar 17 15:48:12 backup lock_gulmd_core[2653]: Could not send quorum update to slave backup Mar 17 15:48:12 backup lock_gulmd_core[2653]: New generation of server state. (1142628492484630) Mar 17 15:48:12 backup lock_gulmd_LTPX[2655]: New Master at backup: 127.0.0.1 Mar 17 15:52:14 backup kernel: Lock_Harness 6.0.2.27 (built Jan 30 2006 15:32:58) installed Mar 17 15:52:14 backup kernel: GFS 6.0.2.27 (built Jan 30 2006 15:32:20) installed Mar 17 15:52:15 backup kernel: Gulm 6.0.2.27 (built Jan 30 2006 15:32:54) installed Mar 17 15:54:51 backup kernel: lock_gulm: ERROR cm_login failed. -512 Mar 17 15:54:51 backup kernel: lock_gulm: ERROR Got a -512 trying to start the threads. Mar 17 15:54:51 backup lock_gulmd_core[2653]: Error on xdr (GFS Kernel Interface:127.0.0.1 idx:3 fd:8): (-104:104:Connection reset by peer) Mar 17 15:54:51 backup kernel: lock_gulm: fsid=cluster_cca:gfs1: Exiting gulm_mount with errors -512 Mar 17 15:54:51 backup kernel: GFS: can't mount proto = lock_gulm, table = cluster_cca:gfs1, hostdata = Result from gulm_tool: [root at backup gfs]# gulm_tool nodelist backup Name: backup ip = 127.0.0.1 state = Logged in mode = Arbitrating missed beats = 0 last beat = 1142632189718986 delay avg = 10019686 max delay = 10019735 I'm a newbie to clusters and I have no clue where to look next. If any other information is needed let me know. Thanks, -- Magnus Andersen Systems Administrator / Oracle DBA Walker & Associates, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Britt.Treece at savvis.net Fri Mar 17 23:21:09 2006 From: Britt.Treece at savvis.net (Treece, Britt) Date: Fri, 17 Mar 2006 17:21:09 -0600 Subject: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B90152A1D5@s228130hz1ew08.apptix-01.savvis.net> Also, make sure your servers /etc/hosts file on all three servers looks similar to... 127.0.0.1 localhost.localdomain localhost 10.0.0.1 backup 10.0.0.2 oradw 10.0.0.3 gistest2 Britt ________________________________ From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Treece, Britt Sent: Friday, March 17, 2006 5:09 PM To: linux clustering Subject: RE: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 Magnus, Try starting ccsd and lock_gulmd on all three servers. Once these start you should be able to see all three in gulm_tool nodelist localhost. At that point you should be able to mount your GFS pool vol's. Your lock cluster has to have a quorum of greater than half the servers configured in cluster.ccs, so at least 2 in your case before it will allow a GFS vol to be mounted. Regards, Britt ________________________________ From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Magnus Andersen Sent: Friday, March 17, 2006 4:54 PM To: linux-cluster at redhat.com Subject: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 Hi All, I've successfully installed and configured GFS on my three nodes, but when I try to mount the filesystem the prompt hangs until I kill the mount command. All servers are running RHEL 3 AS/ES U6 with the 2.4.21-37.0.1.ELsmp kernel and are connected to a MSA1500 SAN via FC. I've installed the following GFS rpms: [root at oradw root]# rpm -qa | grep -i gfs GFS-modules-6.0.2.27-0.1 GFS-modules-smp-6.0.2.27-0.1 GFS-6.0.2.27-0.1 Here is my pool configuration files and the output from pool_tool -s [root at backup gfs]# cat cluster_cca.cfg poolname cluster_cca subpools 1 subpool 0 0 1 pooldevice 0 0 /dev/sda1 [root at backup gfs]# cat pool0.cfg poolname pool_gfs1 subpools 1 subpool 0 0 1 pooldevice 0 0 /dev/sda2 [root at backup gfs]# cat pool1.cfg poolname pool_gfs2 subpools 1 subpool 0 0 1 pooldevice 0 0 /dev/sdb [root at backup gfs]# pool_tool -s Device Pool Label ====== ========== /dev/pool/cluster_cca <- CCA device -> /dev/pool/pool_gfs1 <- GFS filesystem -> /dev/pool/pool_gfs2 <- GFS filesystem -> /dev/cciss/c0d0 <- partition information -> /dev/cciss/c0d0p1 <- EXT2/3 filesystem -> /dev/cciss/c0d0p2 <- swap device -> /dev/cciss/c0d0p3 <- lvm1 subdevice -> /dev/sda <- partition information -> /dev/sda1 cluster_cca /dev/sda2 pool_gfs1 /dev/sdb pool_gfs2 Here are my ccs files. [root at backup cluster_cca]# cat cluster.ccs cluster { name = "cluster_cca" lock_gulm { servers = ["backup", "oradw", "gistest2"] } } [root at backup cluster_cca]# cat fence.ccs fence_devices { manual { agent = "fence_manual" } } [root at backup cluster_cca]# cat nodes.ccs nodes { backup { ip_interfaces { eth1 = "10.0.0.1" } fence { man { manual { ipaddr = " 10.0.0.1" } } } } oradw { ip_interfaces { eth4 = " 10.0.0.2" } fence { man { manual { ipaddr = " 10.0.0.2" } } } } gistest2 { ip_interfaces { eth0 = " 10.0.0.3" } fence { man { manual { ipaddr = " 10.0.0.3" } } } } } Here is the command I used to create the filesystem: gfs_mkfs -p lock_gulm -t cluster_cca:pool_gfs2 -j 10 /dev/pool/pool_gfs2 Mount command that hangs: mount -t gfs /dev/pool/pool_gfs2 /gfs2 Here is the output I see in my messages log file. I see the last 5 lines repeated for each time I tried to mount the filesystem. Mar 17 15:47:05 backup ccsd[2645]: Starting ccsd 6.0.2.27 : Mar 17 15:47:05 backup ccsd[2645]: Built: Jan 30 2006 15:28:33 Mar 17 15:47:05 backup ccsd[2645]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Mar 17 15:48:10 backup lock_gulmd[2652]: Starting lock_gulmd 6.0.2.27 . (built Jan 30 2006 15:28:54) Copyright (C) 2004 Red Hat, Inc. All rights reserved. Mar 17 15:48:10 backup lock_gulmd[2652]: You are running in Fail-over mode. Mar 17 15:48:10 backup lock_gulmd[2652]: I am (backup) with ip (127.0.0.1 ) Mar 17 15:48:10 backup lock_gulmd[2652]: Forked core [2653]. Mar 17 15:48:11 backup lock_gulmd[2652]: Forked locktable [2654]. Mar 17 15:48:12 backup lock_gulmd[2652]: Forked ltpx [2655]. Mar 17 15:48:12 backup lock_gulmd_core[2653]: I see no Masters, So I am Arbitrating until enough Slaves talk to me. Mar 17 15:48:12 backup lock_gulmd_core[2653]: Could not send quorum update to slave backup Mar 17 15:48:12 backup lock_gulmd_core[2653]: New generation of server state. (1142628492484630) Mar 17 15:48:12 backup lock_gulmd_LTPX[2655]: New Master at backup: 127.0.0.1 Mar 17 15:52:14 backup kernel: Lock_Harness 6.0.2.27 (built Jan 30 2006 15:32:58) installed Mar 17 15:52:14 backup kernel: GFS 6.0.2.27 (built Jan 30 2006 15:32:20) installed Mar 17 15:52:15 backup kernel: Gulm 6.0.2.27 (built Jan 30 2006 15:32:54) installed Mar 17 15:54:51 backup kernel: lock_gulm: ERROR cm_login failed. -512 Mar 17 15:54:51 backup kernel: lock_gulm: ERROR Got a -512 trying to start the threads. Mar 17 15:54:51 backup lock_gulmd_core[2653]: Error on xdr (GFS Kernel Interface:127.0.0.1 idx:3 fd:8): (-104:104:Connection reset by peer) Mar 17 15:54:51 backup kernel: lock_gulm: fsid=cluster_cca:gfs1: Exiting gulm_mount with errors -512 Mar 17 15:54:51 backup kernel: GFS: can't mount proto = lock_gulm, table cluster_cca:gfs1, hostdata = Result from gulm_tool: [root at backup gfs]# gulm_tool nodelist backup Name: backup ip = 127.0.0.1 state = Logged in mode = Arbitrating missed beats = 0 last beat = 1142632189718986 delay avg = 10019686 max delay = 10019735 I'm a newbie to clusters and I have no clue where to look next. If any other information is needed let me know. Thanks, -- Magnus Andersen Systems Administrator / Oracle DBA Walker & Associates, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mag.andersen at gmail.com Fri Mar 17 23:35:09 2006 From: mag.andersen at gmail.com (Magnus Andersen) Date: Fri, 17 Mar 2006 18:35:09 -0500 Subject: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 In-Reply-To: <9A6FE0FCC2B29846824C5CD81C6647B90152A1D5@s228130hz1ew08.apptix-01.savvis.net> References: <9A6FE0FCC2B29846824C5CD81C6647B90152A1D5@s228130hz1ew08.apptix-01.savvis.net> Message-ID: <5ea165840603171535h399993fdmdb70847ecd5cfecb@mail.gmail.com> I ran the following on all the servers before I started to try and mount the share. ccsd -d /dev/pool/cluster_cca lock_gulmd They ran without errors. I did setup the hosts file, but when I looked at it agin I see that I called them backuphb, oradwhb, and gistest2hb. Do I need to set the ccs files backup with these names? Or, should I change the command switches? Thanks for your help, -- Magnus Andersen Systems Administrator / Oracle DBA Walker & Associates, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Britt.Treece at savvis.net Sat Mar 18 00:10:52 2006 From: Britt.Treece at savvis.net (Treece, Britt) Date: Fri, 17 Mar 2006 18:10:52 -0600 Subject: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B90152A1D6@s228130hz1ew08.apptix-01.savvis.net> Your cluster.ccs config needs to match what is in /etc/hosts or vice versa. If lock_gulmd is started are you seeing all three servers in gulm_tool nodelist? ________________________________ From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Magnus Andersen Sent: Friday, March 17, 2006 5:35 PM To: linux clustering Subject: Re: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 I ran the following on all the servers before I started to try and mount the share. ccsd -d /dev/pool/cluster_cca lock_gulmd They ran without errors. I did setup the hosts file, but when I looked at it agin I see that I called them backuphb, oradwhb, and gistest2hb. Do I need to set the ccs files backup with these names? Or, should I change the command switches? Thanks for your help, -- Magnus Andersen Systems Administrator / Oracle DBA Walker & Associates, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mag.andersen at gmail.com Sat Mar 18 01:37:12 2006 From: mag.andersen at gmail.com (Magnus Andersen) Date: Fri, 17 Mar 2006 20:37:12 -0500 Subject: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 In-Reply-To: <9A6FE0FCC2B29846824C5CD81C6647B90152A1D6@s228130hz1ew08.apptix-01.savvis.net> References: <9A6FE0FCC2B29846824C5CD81C6647B90152A1D6@s228130hz1ew08.apptix-01.savvis.net> Message-ID: <5ea165840603171737t34ff9798g8708de727ff01391@mail.gmail.com> This is what I have now. cluster.css [root at backup root]# cat cluster_cca/cluster.ccs cluster { name = "cluster_cca" lock_gulm { servers = ["backuphb", "oradwhb", "gistest2hb"] } } nodes.css [root at backup root]# cat cluster_cca/nodes.ccs nodes { backuphb { ip_interfaces { eth1 = "10.0.0.1" } fence { man { manual { ipaddr = "10.0.0.1" } } } } oradwhb { ip_interfaces { eth4 = "10.0.0.2" } fence { man { manual { ipaddr = "10.0.0.2" } } } } gistest2hb { ip_interfaces { eth0 = "10.0.0.3" } fence { man { manual { ipaddr = "10.0.0.3" } } } } } /etc/hosts [root at backup root]# cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 backup localhost.localdomain localhost 10.0.0.1 backuphb backuphb.walkerassoc.com backuphb 10.0.0.2 oradwhb oradwhb.walkerassoc.com oradwhb 10.0.0.3 gistest2hb gistest2hb.walkerassoc.com gistest2hb I ran this to update the cluster_cca pool after I modified the ccs files ccs_tool -O create /root/cluster_cca /dev/pool/cluster_cca Result out of my messages log ( this looks the same on all servers once I start lock_gulmd ) Mar 17 20:24:17 gistest2 lock_gulmd[2383]: Starting lock_gulmd 6.0.2.27. (built Jan 30 2006 15:28:54) Copyright (C) 2004 Red Hat, Inc. All rights reserved. Mar 17 20:24:17 gistest2 lock_gulmd[2383]: You are running in Fail-over mode. Mar 17 20:24:17 gistest2 lock_gulmd[2383]: I am (gistest2) with ip (127.0.0.1) Mar 17 20:24:17 gistest2 lock_gulmd[2383]: Forked core [2384]. Mar 17 20:24:18 gistest2 lock_gulmd_LT000[2385]: Not serving locks from this nod e. Mar 17 20:24:18 gistest2 lock_gulmd[2383]: Forked locktable [2385]. Mar 17 20:24:19 gistest2 lock_gulmd[2383]: Forked ltpx [2386]. Result from gulm_tool nodelist localhost [root at backup root]# gulm_tool nodelist localhost Name: backup ip = 127.0.0.1 state = Logged in mode = Pending missed beats = 0 last beat = 0 delay avg = 0 max delay = 0 Thanks, -- Magnus Andersen Systems Administrator / Oracle DBA Walker & Associates, Inc. From Britt.Treece at savvis.net Sat Mar 18 04:51:08 2006 From: Britt.Treece at savvis.net (Treece, Britt) Date: Fri, 17 Mar 2006 22:51:08 -0600 Subject: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 References: <9A6FE0FCC2B29846824C5CD81C6647B90152A1D6@s228130hz1ew08.apptix-01.savvis.net> <5ea165840603171737t34ff9798g8708de727ff01391@mail.gmail.com> Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B92D00FF@s228130hz1ew08.apptix-01.savvis.net> The nodename in nodes.ccs and cluster.ccs needs to match the hostname of each server. I'm getting the impression from the output below that it does not. >From the GFS 6.0 Admin guide... http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-sf-nodes.html Make sure that you specify Nodename as the Linux hostname and that the primary IP address of the node is associated with the hostname. Specifying NodeName other than the Linux hostname (for example the interface name) can cause unpredictable results - especially if the node is connected to multiple networks. To determine the hostname of a node, use the uname -n command on the node. To verify the IP address associated with the hostname, issue a ping command to the hostname. ________________________________ From: linux-cluster-bounces at redhat.com on behalf of Magnus Andersen Sent: Fri 3/17/2006 7:37 PM To: linux clustering Subject: Re: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 This is what I have now. cluster.css [root at backup root]# cat cluster_cca/cluster.ccs cluster { name = "cluster_cca" lock_gulm { servers = ["backuphb", "oradwhb", "gistest2hb"] } } nodes.css [root at backup root]# cat cluster_cca/nodes.ccs nodes { backuphb { ip_interfaces { eth1 = "10.0.0.1" } fence { man { manual { ipaddr = "10.0.0.1" } } } } oradwhb { ip_interfaces { eth4 = "10.0.0.2" } fence { man { manual { ipaddr = "10.0.0.2" } } } } gistest2hb { ip_interfaces { eth0 = "10.0.0.3" } fence { man { manual { ipaddr = "10.0.0.3" } } } } } /etc/hosts [root at backup root]# cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 backup localhost.localdomain localhost 10.0.0.1 backuphb backuphb.walkerassoc.com backuphb 10.0.0.2 oradwhb oradwhb.walkerassoc.com oradwhb 10.0.0.3 gistest2hb gistest2hb.walkerassoc.com gistest2hb I ran this to update the cluster_cca pool after I modified the ccs files ccs_tool -O create /root/cluster_cca /dev/pool/cluster_cca Result out of my messages log ( this looks the same on all servers once I start lock_gulmd ) Mar 17 20:24:17 gistest2 lock_gulmd[2383]: Starting lock_gulmd 6.0.2.27. (built Jan 30 2006 15:28:54) Copyright (C) 2004 Red Hat, Inc. All rights reserved. Mar 17 20:24:17 gistest2 lock_gulmd[2383]: You are running in Fail-over mode. Mar 17 20:24:17 gistest2 lock_gulmd[2383]: I am (gistest2) with ip (127.0.0.1) Mar 17 20:24:17 gistest2 lock_gulmd[2383]: Forked core [2384]. Mar 17 20:24:18 gistest2 lock_gulmd_LT000[2385]: Not serving locks from this nod e. Mar 17 20:24:18 gistest2 lock_gulmd[2383]: Forked locktable [2385]. Mar 17 20:24:19 gistest2 lock_gulmd[2383]: Forked ltpx [2386]. Result from gulm_tool nodelist localhost [root at backup root]# gulm_tool nodelist localhost Name: backup ip = 127.0.0.1 state = Logged in mode = Pending missed beats = 0 last beat = 0 delay avg = 0 max delay = 0 Thanks, -- Magnus Andersen Systems Administrator / Oracle DBA Walker & Associates, Inc. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 11493 bytes Desc: not available URL: From orcl.listas at gmail.com Sat Mar 18 04:55:00 2006 From: orcl.listas at gmail.com (Allyson - Listas) Date: Sat, 18 Mar 2006 01:55:00 -0300 Subject: [Linux-cluster] agent scripts Message-ID: <441B92A4.9040602@gmail.com> Hi Guys, I'd like your help in one question on rhcs 4 up 3. I'm working on my scripts to start/stop/monitoring a failover oracle 9.2.0.6 database. I created a new service and added a filesystem, ip and script resources. There is a cookbook or something that shows requirements of development of the cluster scripts? I'm having problems to start my service ... When I try to enable... [root at cs02 ora9i]# clustat Member Status: Quorate Member Name Status ------ ---- ------ cs02.example.com Online, Local, rgmanager cs01.example.com Online, rgmanager Service Name Owner (Last) State ------- ---- ----- ------ ----- vip50 cs02.example.com started oracle-ha-fs cs02.example.com started ora9i-ha (cs02.example.com) failed clu9i (none) stopped [root at cs02 ora9i]# clusvcadm -e clu9i Member cs02.example.com trying to enable clu9i...failed I receive these messages... Mar 18 01:59:12 cs02 clurgmgrd[2315]: Starting stopped service clu9i Mar 18 01:59:12 cs02 clurgmgrd[2315]: start on script "clu9i" returned 5 (program not installed) Mar 18 01:59:12 cs02 clurgmgrd[2315]: #68: Failed to start clu9i; return value: 1 Mar 18 01:59:12 cs02 clurgmgrd[2315]: Stopping service clu9i Mar 18 01:59:12 cs02 clurgmgrd[2315]: stop on script "clu9i" returned 5 (program not installed) Mar 18 01:59:12 cs02 clurgmgrd[2315]: Service clu9i is recovering Mar 18 01:59:12 cs02 clurgmgrd[2315]: #71: Relocating failed service clu9i Mar 18 01:59:13 cs02 clurgmgrd[2315]: Stopping service clu9i Mar 18 01:59:13 cs02 clurgmgrd[2315]: stop on script "clu9i" returned 5 (program not installed) Mar 18 01:59:13 cs02 clurgmgrd[2315]: Service clu9i is stopped I think strange the return code 5 logged on /var/log/messages, because my script works well manually... [root at cs02 ora9i]# ps -ef |grep pmon | grep -v grep [root at cs02 ora9i]# [root at cs02 ora9i]# [root at cs02 ora9i]# ./ora_clu9i start starting ora_clu9i... [root at cs02 ora9i]# ps -ef |grep pmon | grep -v grep ora9i 7934 1 0 02:06 ? 00:00:00 ora_pmon_clu9i [root at cs02 ora9i]# ./ora_clu9i status clu9i is running [root at cs02 ora9i]# ./ora_clu9i stop stopping ora_clu9i... [root at cs02 ora9i]# ps -ef |grep pmon | grep -v grep [root at cs02 ora9i]# ./ora_clu9i status clu9i is stopped Here's my script... #### Oracle Environment #### export LD_ASSUME_KERNEL=2.4.19 export ORACLE_BASE=/u01/ora9i export ORACLE_HOME=$ORACLE_BASE/product/9.2.0 export ORACLE_SID=clu9i export ORACLE_TERM=xterm export NLS_LANG=AMERICAN; export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib export LD_LIBRARY_PATH export PATH=$PATH:$ORACLE_HOME/bin #### prog="ora_clu9i" start () { echo "starting $prog..." su - ora9i -c "$ORACLE_HOME/bin/sqlplus '/ as sysdba' > /dev/null < /dev/null" return 0 } stop () { echo "stopping $prog..." su - ora9i -c "$ORACLE_HOME/bin/sqlplus '/ as sysdba' > /dev/null < /dev/null" return 0 } status() { if [ -r /tmp/orastat ]; then rm /tmp/orastat ; fi sqlplus /nolog < /tmp/orastat conn hr/hr quit eof DOWN=`grep -i error /tmp/orastat | grep -v grep | wc -l` if [ $DOWN -gt 0 ]; then echo $ORACLE_SID is stopped; return 0 else echo $ORACLE_SID is running; return 1 fi } case "$1" in start) start ;; stop) stop ;; status) status ;; *) echo $"Usage: $0 {start|stop|status}" exit 1 esac From orcl.listas at gmail.com Sat Mar 18 14:51:18 2006 From: orcl.listas at gmail.com (Allyson - Listas) Date: Sat, 18 Mar 2006 11:51:18 -0300 Subject: [Linux-cluster] agent scripts In-Reply-To: <441B92A4.9040602@gmail.com> References: <441B92A4.9040602@gmail.com> Message-ID: <441C1E66.80809@gmail.com> Hi Guys, I Solved my problem. I was having problems because when cluster start / stop services it follow the order of the resources are in the service. In my case i put IP, FILESYSTEM and SCRIPT, when cluster start the service it starts the ip , after filesystem and then script. And on *stop* it follows the same order, that was my problem, because my script checks the health of database doing a connection using a binary (sqlplus) that was on filesystgem that clusters umount!!! So, I change my status method and the script and it works. Change on Status made... status() { UP=`ps -ef |grep ora_pmon_clu9i |grep -v grep |wc -l` if [ $UP -gt 0 ]; then echo $ORACLE_SID is running; return 0 else echo $ORACLE_SID is stoped; return 1 fi } Regards, Allyson - Listas wrote: > Hi Guys, > > I'd like your help in one question on rhcs 4 up 3. > > I'm working on my scripts to start/stop/monitoring a failover oracle > 9.2.0.6 database. > I created a new service and added a filesystem, ip and script resources. > > There is a cookbook or something that shows requirements of > development of the cluster scripts? > > I'm having problems to start my service ... > When I try to enable... > > [root at cs02 ora9i]# clustat > Member Status: Quorate > > Member Name Status > ------ ---- ------ > cs02.example.com Online, Local, rgmanager > cs01.example.com Online, rgmanager > > Service Name Owner (Last) State > ------- ---- ----- ------ ----- > vip50 cs02.example.com started > oracle-ha-fs cs02.example.com started > ora9i-ha (cs02.example.com) failed > clu9i (none) stopped > [root at cs02 ora9i]# clusvcadm -e clu9i > Member cs02.example.com trying to enable clu9i...failed > > I receive these messages... > > Mar 18 01:59:12 cs02 clurgmgrd[2315]: Starting stopped > service clu9i > Mar 18 01:59:12 cs02 clurgmgrd[2315]: start on script "clu9i" > returned 5 (program not installed) > Mar 18 01:59:12 cs02 clurgmgrd[2315]: #68: Failed to start > clu9i; return value: 1 > Mar 18 01:59:12 cs02 clurgmgrd[2315]: Stopping service clu9i > Mar 18 01:59:12 cs02 clurgmgrd[2315]: stop on script "clu9i" > returned 5 (program not installed) > Mar 18 01:59:12 cs02 clurgmgrd[2315]: Service clu9i is > recovering > Mar 18 01:59:12 cs02 clurgmgrd[2315]: #71: Relocating failed > service clu9i > Mar 18 01:59:13 cs02 clurgmgrd[2315]: Stopping service clu9i > Mar 18 01:59:13 cs02 clurgmgrd[2315]: stop on script "clu9i" > returned 5 (program not installed) > Mar 18 01:59:13 cs02 clurgmgrd[2315]: Service clu9i is stopped > > I think strange the return code 5 logged on /var/log/messages, because > my script works well manually... > > [root at cs02 ora9i]# ps -ef |grep pmon | grep -v grep > [root at cs02 ora9i]# > [root at cs02 ora9i]# > [root at cs02 ora9i]# ./ora_clu9i start > starting ora_clu9i... > [root at cs02 ora9i]# ps -ef |grep pmon | grep -v grep > ora9i 7934 1 0 02:06 ? 00:00:00 ora_pmon_clu9i > [root at cs02 ora9i]# ./ora_clu9i status > clu9i is running > [root at cs02 ora9i]# ./ora_clu9i stop > stopping ora_clu9i... > [root at cs02 ora9i]# ps -ef |grep pmon | grep -v grep > [root at cs02 ora9i]# ./ora_clu9i status > clu9i is stopped > > > > Here's my script... > > #### Oracle Environment #### > export LD_ASSUME_KERNEL=2.4.19 > export ORACLE_BASE=/u01/ora9i > export ORACLE_HOME=$ORACLE_BASE/product/9.2.0 > export ORACLE_SID=clu9i > export ORACLE_TERM=xterm > export NLS_LANG=AMERICAN; > export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data > LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib > LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib > export LD_LIBRARY_PATH > export PATH=$PATH:$ORACLE_HOME/bin > #### > > prog="ora_clu9i" > > start () { > echo "starting $prog..." > > su - ora9i -c "$ORACLE_HOME/bin/sqlplus '/ as sysdba' > /dev/null < startup > quit > eof > " > su - ora9i -c "lsnrctl start > /dev/null" > > return 0 > > } > > stop () { > echo "stopping $prog..." > > su - ora9i -c "$ORACLE_HOME/bin/sqlplus '/ as sysdba' > /dev/null < shutdown immediate > quit > eof > " > su - ora9i -c "lsnrctl stop > /dev/null" > > return 0 > } > > status() { > if [ -r /tmp/orastat ]; then > rm /tmp/orastat ; > fi > > sqlplus /nolog < /tmp/orastat > conn hr/hr > quit > eof > > DOWN=`grep -i error /tmp/orastat | grep -v grep | wc -l` > > if [ $DOWN -gt 0 ]; then > echo $ORACLE_SID is stopped; > return 0 > else > echo $ORACLE_SID is running; > return 1 > fi > } > > case "$1" in > start) > start > ;; > stop) > stop > ;; > status) > status > ;; > *) > echo $"Usage: $0 {start|stop|status}" > exit 1 > esac > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Allyson A. Brito MSN: allysonbrito78 at hotmail.com SKYPE: allysonbrito RHCE / LPI-1 / SCSA OCP DBA 9i / OCA PL/SQL 9i From mag.andersen at gmail.com Sat Mar 18 18:19:06 2006 From: mag.andersen at gmail.com (Magnus Andersen) Date: Sat, 18 Mar 2006 13:19:06 -0500 Subject: [Linux-cluster] Unable to mount GFS on RHEL 3 U6 In-Reply-To: <9A6FE0FCC2B29846824C5CD81C6647B92D00FF@s228130hz1ew08.apptix-01.savvis.net> References: <9A6FE0FCC2B29846824C5CD81C6647B90152A1D6@s228130hz1ew08.apptix-01.savvis.net> <5ea165840603171737t34ff9798g8708de727ff01391@mail.gmail.com> <9A6FE0FCC2B29846824C5CD81C6647B92D00FF@s228130hz1ew08.apptix-01.savvis.net> Message-ID: <5ea165840603181019w45d92a06ncf873596ba08a880@mail.gmail.com> Britt, Thanks a million for your help. You got me to look at what was wrong. I changed my hosts file to look the same as the cluster.css and nodes.css file and I still didn't see all three servers on the gulm_tool nodelist. I went back and looked at my hosts file and saw that I also called localhost the same as the hostname. I removed the entry and left localhost.localdomain localhost and checked again. IT WORKS!!! ... :) I've sucessfully mounted the shares on all servers and created object on the shares it all works. Again, thanks for the help. Love these groups... :) Sincerely, -- Magnus Andersen Systems Administrator / Oracle DBA Walker & Associates, Inc. From carlopmart at gmail.com Sat Mar 18 19:58:01 2006 From: carlopmart at gmail.com (carlopmart) Date: Sat, 18 Mar 2006 20:58:01 +0100 Subject: [Linux-cluster] Load balancing on CS4? Message-ID: <441C6649.2070803@gmail.com> Hi all, I have configured two nodes on a vmware host. I would do load balancing for apache services (50-50). How can I do this? I didn't find anything about this on RedHat's documentation. Thanks. -- CL Martinez carlopmart {at} gmail {d0t} com From basv at sara.nl Sun Mar 19 10:41:40 2006 From: basv at sara.nl (Bas van der Vlies) Date: Sun, 19 Mar 2006 11:41:40 +0100 Subject: [Linux-cluster] lock_dlm kernel panics In-Reply-To: <20060317224043.GC29244@redhat.com> References: <441B2C73.3010202@fnal.gov> <20060317224043.GC29244@redhat.com> Message-ID: <52FFC5E9-FAE4-4414-B0B9-65103591CE59@sara.nl> On Mar 17, 2006, at 11:40 PM, David Teigland wrote: > On Fri, Mar 17, 2006 at 03:38:59PM -0600, Paul Tader wrote: >> Mar 17 11:38:02 nodename kernel: d0 unlock fe20017a no id > > GFS is trying to unlock a lock that doesn't exist which causes the > panic. > We know this happens if cman shuts down the dlm while it's in use > (cman > does this if it's lost connection with the cluster.) There's some new > output in the RHEL4U3 dlm that should tell us if that's in fact what's > happening or if there's some other cause that we need to uncover. > > So, you should look on all nodes for any cman messages in > /var/log/messages or the console. And when you're using the latest > version look for the new dlm message "WARNING: > dlm_emergency_shutdown". > We had a similiar problem on our 4 node GFS cluster. I have send the crash reports to the list as attachment for all 4 nodes. One cman crash and 3 dlm crashes. Can the list handle attachments or must i send it inline? nodes: 2.6.16-rc5 kernel GFS cvs STABLE -- Bas van der Vlies basv at sara.nl From filipe.miranda at gmail.com Sun Mar 19 15:00:38 2006 From: filipe.miranda at gmail.com (Filipe Miranda) Date: Sun, 19 Mar 2006 12:00:38 -0300 Subject: [Linux-cluster] Load balancing on CS4? In-Reply-To: <441C6649.2070803@gmail.com> References: <441C6649.2070803@gmail.com> Message-ID: Hello there, Here you can find the documentation on how to setup the LVS: http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/pt-lvs.html To try to setup the 50-50 ratio I guess the best shot is taking a look in this link bellow: http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/s1-lvs-scheduling.html#S2-LVS-SCHED I hope this helps you, Since you are testing, if it works please give us a feedback. Att. Filipe Miranda On 3/18/06, carlopmart wrote: > > Hi all, > > I have configured two nodes on a vmware host. I would do load > balancing for apache services (50-50). How can I do this? I didn't > find anything about this on RedHat's documentation. > > Thanks. > -- > CL Martinez > carlopmart {at} gmail {d0t} com > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Att. --- Filipe T Miranda RHCE - Red Hat Certified Engineer OCP8i - Oracle Certified Professional -------------- next part -------------- An HTML attachment was scrubbed... URL: From forums at daltonfirth.co.uk Sun Mar 19 21:07:03 2006 From: forums at daltonfirth.co.uk (James Firth) Date: Sun, 19 Mar 2006 21:07:03 +0000 Subject: [Linux-cluster] Node Failure Detection Problems Message-ID: <441DC7F7.2@daltonfirth.co.uk> Hi, I have some questions on configuring and tuning heartbeats and node-failure detection. I have a 2-node cluster. Whenever a node fails it seems to take a while to detect node failure. First question: I have reduced heartbeat hello_timer to 1 second, and deadnode_timeout to 5 seconds. Is there an elegant way to do this with cluster.conf? Currently I'm setting /proc/cluster/config/cman/hello_timer with an init script hack. Failure is detected by cman within 5 seconds, no problem, but clustat hangs during this time. Second question: clustat continues to hang for around 10 more seconds - 15 in total, before clurgmgrd does a state change. Does anyone know where this additional 10 seconds comes from? Is it configurable? Here is the system log for the transition: >>> Mar 19 21:01:33 firthy kernel: CMAN: removing node emsy from the cluster : Missed too many heartbeats Mar 19 21:01:33 firthy fenced[1878]: emsy not a cluster member after 0 sec post_fail_delay Mar 19 21:01:33 firthy fenced[1878]: fencing node "emsy" Mar 19 21:01:35 firthy fenced[1878]: fence "emsy" success Mar 19 21:01:44 firthy clurgmgrd[3347]: Magma Event: Membership Change Mar 19 21:01:44 firthy clurgmgrd[3347]: State change: emsy DOWN <<< Many thanks, James Firth From forums at daltonfirth.co.uk Sun Mar 19 21:38:12 2006 From: forums at daltonfirth.co.uk (James Firth) Date: Sun, 19 Mar 2006 21:38:12 +0000 Subject: [Linux-cluster] Node Failure Detection Problems In-Reply-To: <441DC7F7.2@daltonfirth.co.uk> References: <441DC7F7.2@daltonfirth.co.uk> Message-ID: <441DCF44.5020402@daltonfirth.co.uk> James Firth wrote: > Hi, > > I have some questions on configuring and tuning heartbeats and > node-failure detection. Further to my earlier mail - am also having problems with exported gnbd devices on node failure. I want to get gnbd to give up trying to reconnect on node failure, but it insists on retrying ad infinitum, causing services that are using imported gnbd volumes to lock. Regards, James Firth From toxictux at gmail.com Mon Mar 20 02:35:00 2006 From: toxictux at gmail.com (toxictux) Date: Sun, 19 Mar 2006 20:35:00 -0600 Subject: [Linux-cluster] fencing trouble with fence_wti Message-ID: <17df45710603191835p1fa8cfd5j59b574ccd18f12d@mail.gmail.com> hi all, i setup a 2 node cluster with http service. everything seems ok except for the fencing. i am using WTI ips 800. my problem is, whenever one of the nodes goes down. it does not get fenced. i get messages in my syslog ""fencing node "node2""" ""fence "node2" failed"" when i do $fence_node node2 it doesnt work either. i get following error message in my syslog Fence of "node2" was unsuccessful however, when i manually do it with fence_wti, it works, $fence_wti -a 216.xxx.xxx.xxx -p passwd -n 2 i am unable to see any other messages anywhere else. can anyone give any pointers?? or any suggestions on getting detailed debug message? Thanks -F From Alain.Moulle at bull.net Mon Mar 20 07:34:47 2006 From: Alain.Moulle at bull.net (Alain Moulle) Date: Mon, 20 Mar 2006 08:34:47 +0100 Subject: [Linux-cluster] CS4 Update 2/ Copy cluster.conf from one node on the peer one ? Message-ID: <441E5B17.90309@bull.net> Hi For a HA pair nodes with CS4 active on both nodes, is there any case where the CS4 decides by itself to copy the cluster.conf from one node on the peer one ? And if so, which cases ? Thanks Alain From thorsten.henrici at gfd.de Mon Mar 20 08:07:51 2006 From: thorsten.henrici at gfd.de (thorsten.henrici at gfd.de) Date: Mon, 20 Mar 2006 09:07:51 +0100 Subject: [Linux-cluster] Netmask of IP Address resource system-config-cluster 1.0.25 Message-ID: Hi, I'm a bit baffled, that I can't enter a netmask when configuring an IP Adress as a ressorce with the the system-config-cluster 1.0.25 tool. As a result, when starting rgmanager, the Service IP Adress gets a /32 netmask, which is not correct of cause. Since there won't be too many changes to the cluster.conf file in the long run, it would be alright to just edit it by hand. Unfortuneatly I wasn't able to figure out what the correct syntax is eg. or Is the DTD of cluster.conf generally available? My cluster.conf is attached below. Please feel free to suggest improvements, since I'm totally new to the RH ClusterSuite. For example, I don't want to use a fencing device, because I don't need any shared storage for the MySQL Cluster, which works with two seperate data nodes, that is the data itself will exist twice physically on seperate storage. Do I have to configure some kind of fencing anyways. If yes, how would this kind of dummy fencing look like in the cluster.conf? (Or is the entry all I need?) Many thanks in advance !