From m at alanny.ru Mon Feb 1 14:52:33 2010 From: m at alanny.ru (AlannY) Date: Mon, 1 Feb 2010 17:52:33 +0300 Subject: [Linux-cluster] [clvm] Volume group for uuid not found Message-ID: <20100201145231.GA796@alanny-pc.lcl.starlink.ru> Hi there. I'm trying to set up HA-Cluster on Fedora with Red Had Cluster Suite. I've already set up cman and it works. Now, I want to create shared strorage with clvm+gfs2. I've install lvm2-cluster. Made lvmconf --enable-cluster. And now: I have 2 nodes, clvmd started: on 1st node I've done: %# pvcreate /dev/sdb1 on 2nd node I've done: %# pvcreate /dev/sdb1 As you may see, there are /dev/sdb1 device, but it's not the same machine - it's 2 different devices. on 1st node I've done: %# vgcreate shared /dev/sdb1 "shared" is a name for volume group. In current step, I'm now created volume group on 2nd node. on 1st node I've executed: %# lvcreate -l 254 -n shared shared But, it failed with error: Error locking on node <2nd node name>: Volume group for uuid not found: QgSZ5Y8KWOxxbxYdk4Y0eOoJh8GwP1hv9GTrlwNycTk7xWBejIp0suFANorVEuKS Aborting. Failed to activate new LV to wipe the start of it. Then, I've created volume group "shared" on 2nd node: %# vgcreate shared /dev/sdb1 And then on 1st node, I've executed lvcreate once more, but have the same error. What am I doing wrong? I'm really newbie in clvm, but expierenced user of plain lvm, so, I have questions. My vgdisplay of 1st node: %# vgdisplay --- Volume group --- VG Name shared System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 27 VG Access read/write VG Status resizable Clustered yes Shared no MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 1016.00 MB PE Size 4.00 MB Total PE 254 Alloc PE / Size 0 / 0 Free PE / Size 254 / 1016.00 MB VG UUID QgSZ5Y-8KWO-xxbx-Ydk4-Y0eO-oJh8-GwP1hv On second node, there are either no volume groups or one named shared with just different VG UUID. -- )\._.,--....,'``. /, _.. \ _\ (`._ ,. `._.-(,_..'--(,_..'`-.;.' From ccaulfie at redhat.com Mon Feb 1 15:34:18 2010 From: ccaulfie at redhat.com (Christine Caulfield) Date: Mon, 01 Feb 2010 15:34:18 +0000 Subject: [Linux-cluster] [clvm] Volume group for uuid not found In-Reply-To: <20100201145231.GA796@alanny-pc.lcl.starlink.ru> References: <20100201145231.GA796@alanny-pc.lcl.starlink.ru> Message-ID: <4B66F47A.2000100@redhat.com> On 01/02/10 14:52, AlannY wrote: > Hi there. I'm trying to set up HA-Cluster on Fedora with Red Had Cluster Suite. > I've already set up cman and it works. > > Now, I want to create shared strorage with clvm+gfs2. I've install lvm2-cluster. Made > lvmconf --enable-cluster. And now: > > I have 2 nodes, clvmd started: > > on 1st node I've done: > > %# pvcreate /dev/sdb1 > > on 2nd node I've done: > > %# pvcreate /dev/sdb1 > > As you may see, there are /dev/sdb1 device, but it's not the same machine - it's 2 different devices. Is it really two different devices ? If so then clvmd will not work. It needs the same view of storage on all systems. Not necessarily the same names, but definitely the same actual storage. Chrissie > on 1st node I've done: > > %# vgcreate shared /dev/sdb1 > > "shared" is a name for volume group. > > In current step, I'm now created volume group on 2nd node. > > on 1st node I've executed: > > %# lvcreate -l 254 -n shared shared > > But, it failed with error: > > Error locking on node<2nd node name>: Volume group for uuid not found: QgSZ5Y8KWOxxbxYdk4Y0eOoJh8GwP1hv9GTrlwNycTk7xWBejIp0suFANorVEuKS > Aborting. Failed to activate new LV to wipe the start of it. > > Then, I've created volume group "shared" on 2nd node: > > %# vgcreate shared /dev/sdb1 > > And then on 1st node, I've executed lvcreate once more, but have the same error. > > What am I doing wrong? I'm really newbie in clvm, but expierenced user of plain lvm, so, I have questions. > > My vgdisplay of 1st node: > > %# vgdisplay > --- Volume group --- > VG Name shared > System ID > Format lvm2 > Metadata Areas 1 > Metadata Sequence No 27 > VG Access read/write > VG Status resizable > Clustered yes > Shared no > MAX LV 0 > Cur LV 0 > Open LV 0 > Max PV 0 > Cur PV 1 > Act PV 1 > VG Size 1016.00 MB > PE Size 4.00 MB > Total PE 254 > Alloc PE / Size 0 / 0 > Free PE / Size 254 / 1016.00 MB > VG UUID QgSZ5Y-8KWO-xxbx-Ydk4-Y0eO-oJh8-GwP1hv > > On second node, there are either no volume groups or one named shared with just different VG UUID. > From m at alanny.ru Mon Feb 1 16:09:53 2010 From: m at alanny.ru (AlannY) Date: Mon, 1 Feb 2010 19:09:53 +0300 Subject: [Linux-cluster] [clvm] Volume group for uuid not found In-Reply-To: <4B66F47A.2000100@redhat.com> References: <20100201145231.GA796@alanny-pc.lcl.starlink.ru> <4B66F47A.2000100@redhat.com> Message-ID: <20100201160952.GA9386@alanny-pc.lcl.starlink.ru> On Mon, Feb 01, 2010 at 03:34:18PM +0000, Christine Caulfield wrote: > Is it really two different devices ? If so then clvmd will not work. > It needs the same view of storage on all systems. Not necessarily > the same names, but definitely the same actual storage. So, should I use DRBD with clvm for data mirroring on both nodes? -- )\._.,--....,'``. /, _.. \ _\ (`._ ,. `._.-(,_..'--(,_..'`-.;.' From gordan at bobich.net Mon Feb 1 16:21:25 2010 From: gordan at bobich.net (Gordan Bobic) Date: Mon, 01 Feb 2010 16:21:25 +0000 Subject: [Linux-cluster] [clvm] Volume group for uuid not found In-Reply-To: <20100201160952.GA9386@alanny-pc.lcl.starlink.ru> References: <20100201145231.GA796@alanny-pc.lcl.starlink.ru> <4B66F47A.2000100@redhat.com> <20100201160952.GA9386@alanny-pc.lcl.starlink.ru> Message-ID: <4B66FF85.7070207@bobich.net> AlannY wrote: > On Mon, Feb 01, 2010 at 03:34:18PM +0000, Christine Caulfield wrote: >> Is it really two different devices ? If so then clvmd will not work. >> It needs the same view of storage on all systems. Not necessarily >> the same names, but definitely the same actual storage. > > So, should I use DRBD with clvm for data mirroring on both nodes? You could, indeed, use DRBD. Or you could cross-export the block devices with iSCSI (or ATAoE) and have them connected to all the nodes that way. I guess it depends on whether you prefer to use DRBD or CLVM. My preference is for DRBD, but don't let that bias your decision. :) Gordan From xml.devel at gmail.com Mon Feb 1 17:52:43 2010 From: xml.devel at gmail.com (Kumar, Ashish) Date: Mon, 1 Feb 2010 23:22:43 +0530 Subject: [Linux-cluster] heartbeat In-Reply-To: References: Message-ID: <633e02961002010952s32f7c749i5f830094fc81a55e@mail.gmail.com> > > I don?t have any nics that I can dedicate for a cross over cable for > heartbeat so I was thinking I could use the serial or the network itself. Is > it possible to just use the primary interface for heartbeat since it is my > understanding it is nothing more than a ping. > You don't need to dedicate a NIC for heartbeat with Red Hat Cluster. Unlike Linux-HA, RHCS makes use of multicast to communicate with other nodes. The multicast IP address is assigned by RHCS automatically, so you don't need to do any special configuration. Please have a look at this: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Administration/s1-multicast-considerations-CA.html For setup instructions, check this: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Administration/s1-general-prop-conga-CA.html- Section 3 deals with Multicast address settings -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ed.Sanborn at genband.com Mon Feb 1 19:37:45 2010 From: Ed.Sanborn at genband.com (Ed Sanborn) Date: Mon, 1 Feb 2010 13:37:45 -0600 Subject: [Linux-cluster] heartbeat In-Reply-To: <633e02961002010952s32f7c749i5f830094fc81a55e@mail.gmail.com> References: <633e02961002010952s32f7c749i5f830094fc81a55e@mail.gmail.com> Message-ID: <21E1652F6AA39A4B89489DCFC399C1FDAB9CA7@gbplamail.genband.com> I am intrigued by this thread. I am using RH Advanced Platform version 5.2. I have an 8-node cluster running on an IBM Bladecenter. For those that may not be familiar with the IBM Bladecenter the blades have 2 NICs on them and the Bladecenter chassis has a Nortel switch that the NIC's connect to. I am using both NIC's plugged into two segments of my LAN. One of them is being used for the multicast heartbeat traffic. It works BUT occasionally my network has serious disruptions due to this being a telecommunications company and the engineers are always doing something to loop the network and/or saturate the network with broadcasts taking down the subnet that my heartbeat traffic is on. Sometimes when this happens the network is so bad that the multicast traffic is affected and I lose quorum on the cluster. Once I fix the network problem I need to restart each node to get the cluster back up and running. Not good and not the correct design. I would love to NOT use a public NIC for the multicast/heartbeat traffic. I tried to setup virtual interfaces and VLAN those in the Nortel switch and moving the multicast/heartbeat traffic over to them, leaving the 2 NICS on the original LAN segments they were on but even with IBM assistance we were unsuccessful. Does anyone have any experience with this? I want to do this to truly isolate the heartbeat network to eliminate any possibility for network congestion taking down the cluster and also because I do not want the multicast traffic pumped all over the LAN. Thanks, Ed (978) 210-9855 From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Kumar, Ashish Sent: Monday, February 01, 2010 12:53 PM To: Fagnon Raymond; linux clustering Subject: Re: [Linux-cluster] heartbeat I don't have any nics that I can dedicate for a cross over cable for heartbeat so I was thinking I could use the serial or the network itself. Is it possible to just use the primary interface for heartbeat since it is my understanding it is nothing more than a ping. You don't need to dedicate a NIC for heartbeat with Red Hat Cluster. Unlike Linux-HA, RHCS makes use of multicast to communicate with other nodes. The multicast IP address is assigned by RHCS automatically, so you don't need to do any special configuration. Please have a look at this: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster _Administration/s1-multicast-considerations-CA.html For setup instructions, check this: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster _Administration/s1-general-prop-conga-CA.html - Section 3 deals with Multicast address settings -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.alawi at auckland.ac.nz Mon Feb 1 20:52:57 2010 From: a.alawi at auckland.ac.nz (Abraham Alawi) Date: Tue, 2 Feb 2010 09:52:57 +1300 Subject: [Linux-cluster] [clvm] Volume group for uuid not found In-Reply-To: <4B66FF85.7070207@bobich.net> References: <20100201145231.GA796@alanny-pc.lcl.starlink.ru> <4B66F47A.2000100@redhat.com> <20100201160952.GA9386@alanny-pc.lcl.starlink.ru> <4B66FF85.7070207@bobich.net> Message-ID: <0FA0A4F3-25D3-45AD-899F-80985BAFB21B@auckland.ac.nz> On 2/02/2010, at 5:21 AM, Gordan Bobic wrote: > AlannY wrote: >> On Mon, Feb 01, 2010 at 03:34:18PM +0000, Christine Caulfield wrote: >>> Is it really two different devices ? If so then clvmd will not work. >>> It needs the same view of storage on all systems. Not necessarily >>> the same names, but definitely the same actual storage. >> So, should I use DRBD with clvm for data mirroring on both nodes? > > You could, indeed, use DRBD. Or you could cross-export the block devices with iSCSI (or ATAoE) and have them connected to all the nodes that way. I guess it depends on whether you prefer to use DRBD or CLVM. My preference is for DRBD, but don't let that bias your decision. :) > > Gordan > I reckon GNBD will be a better solution in your case, DRBD is more suitable for non-cluster file systems (e.g. ext3, xfs .. ) in case of active-passive '''''''''''''''''''''''''''''''''''''''''''''''''''''' Abraham Alawi Unix/Linux Systems Administrator Science IT University of Auckland e: a.alawi at auckland.ac.nz p: +64-9-373 7599, ext#: 87572 '''''''''''''''''''''''''''''''''''''''''''''''''''''' From gordan at bobich.net Mon Feb 1 22:31:38 2010 From: gordan at bobich.net (Gordan Bobic) Date: Mon, 01 Feb 2010 22:31:38 +0000 Subject: [Linux-cluster] [clvm] Volume group for uuid not found In-Reply-To: <0FA0A4F3-25D3-45AD-899F-80985BAFB21B@auckland.ac.nz> References: <20100201145231.GA796@alanny-pc.lcl.starlink.ru> <4B66F47A.2000100@redhat.com> <20100201160952.GA9386@alanny-pc.lcl.starlink.ru> <4B66FF85.7070207@bobich.net> <0FA0A4F3-25D3-45AD-899F-80985BAFB21B@auckland.ac.nz> Message-ID: <4B67564A.7010503@bobich.net> Abraham Alawi wrote: > On 2/02/2010, at 5:21 AM, Gordan Bobic wrote: > >> AlannY wrote: >>> On Mon, Feb 01, 2010 at 03:34:18PM +0000, Christine Caulfield wrote: >>>> Is it really two different devices ? If so then clvmd will not work. >>>> It needs the same view of storage on all systems. Not necessarily >>>> the same names, but definitely the same actual storage. >>> So, should I use DRBD with clvm for data mirroring on both nodes? >> You could, indeed, use DRBD. Or you could cross-export the block devices with iSCSI (or ATAoE) and have them connected to all the nodes that way. I guess it depends on whether you prefer to use DRBD or CLVM. My preference is for DRBD, but don't let that bias your decision. :) >> >> Gordan >> > > I reckon GNBD will be a better solution in your case, DRBD is more > suitable for non-cluster file systems (e.g. ext3, xfs .. ) in case > of active-passive DRBD is specifically designed to also work in active-active mode. I've been running shared root GFS clusters on DRBD for years. DRBD is singularly _THE_ best solution for network RAID1, _especially_ in active-active mode with a clustered file system on top. It also handles resyncing after outages much more gracefully and transparently than other similar solutions. Gordan From a.alawi at auckland.ac.nz Mon Feb 1 23:22:50 2010 From: a.alawi at auckland.ac.nz (Abraham Alawi) Date: Tue, 2 Feb 2010 12:22:50 +1300 Subject: [Linux-cluster] [clvm] Volume group for uuid not found In-Reply-To: <4B67564A.7010503@bobich.net> References: <20100201145231.GA796@alanny-pc.lcl.starlink.ru> <4B66F47A.2000100@redhat.com> <20100201160952.GA9386@alanny-pc.lcl.starlink.ru> <4B66FF85.7070207@bobich.net> <0FA0A4F3-25D3-45AD-899F-80985BAFB21B@auckland.ac.nz> <4B67564A.7010503@bobich.net> Message-ID: <79CCC492-1AAE-4EAB-ADAF-713DD0D63498@auckland.ac.nz> On 2/02/2010, at 11:31 AM, Gordan Bobic wrote: > Abraham Alawi wrote: >> On 2/02/2010, at 5:21 AM, Gordan Bobic wrote: >>> AlannY wrote: >>>> On Mon, Feb 01, 2010 at 03:34:18PM +0000, Christine Caulfield wrote: >>>>> Is it really two different devices ? If so then clvmd will not work. >>>>> It needs the same view of storage on all systems. Not necessarily >>>>> the same names, but definitely the same actual storage. >>>> So, should I use DRBD with clvm for data mirroring on both nodes? >>> You could, indeed, use DRBD. Or you could cross-export the block devices with iSCSI (or ATAoE) and have them connected to all the nodes that way. I guess it depends on whether you prefer to use DRBD or CLVM. My preference is for DRBD, but don't let that bias your decision. :) >>> >>> Gordan >>> >> I reckon GNBD will be a better solution in your case, DRBD is more >> suitable for non-cluster file systems (e.g. ext3, xfs .. ) in case >> of active-passive > > DRBD is specifically designed to also work in active-active mode. I've been running shared root GFS clusters on DRBD for years. DRBD is singularly _THE_ best solution for network RAID1, _especially_ in active-active mode with a clustered file system on top. It also handles resyncing after outages much more gracefully and transparently than other similar solutions. > > Gordan > Yes, it does work in active-active but DRBD people themselves don't recommend running it in production active-active under cluster file system, I quote from their website: "DRBD's primary-primary mode with a shared disk file system (GFS, OCFS2). These systems are very sensitive to failures of the replication network. Currently we cannot generally recommend this for production use." http://www.drbd.org/home/mirroring/ In terms of production solution I reckon GNBD is designed more specifically for that purpose. I'm glad to know that DRBD is running well for you under GFS, honestly I have never tried this setup, it's worthwhile trying though. Cheers, -- Abraham '''''''''''''''''''''''''''''''''''''''''''''''''''''' Abraham Alawi Unix/Linux Systems Administrator Science IT University of Auckland e: a.alawi at auckland.ac.nz p: +64-9-373 7599, ext#: 87572 '''''''''''''''''''''''''''''''''''''''''''''''''''''' From gordan at bobich.net Mon Feb 1 23:42:18 2010 From: gordan at bobich.net (Gordan Bobic) Date: Mon, 01 Feb 2010 23:42:18 +0000 Subject: [Linux-cluster] [clvm] Volume group for uuid not found In-Reply-To: <79CCC492-1AAE-4EAB-ADAF-713DD0D63498@auckland.ac.nz> References: <20100201145231.GA796@alanny-pc.lcl.starlink.ru> <4B66F47A.2000100@redhat.com> <20100201160952.GA9386@alanny-pc.lcl.starlink.ru> <4B66FF85.7070207@bobich.net> <0FA0A4F3-25D3-45AD-899F-80985BAFB21B@auckland.ac.nz> <4B67564A.7010503@bobich.net> <79CCC492-1AAE-4EAB-ADAF-713DD0D63498@auckland.ac.nz> Message-ID: <4B6766DA.8070201@bobich.net> Abraham Alawi wrote: > On 2/02/2010, at 11:31 AM, Gordan Bobic wrote: > >> Abraham Alawi wrote: >>> On 2/02/2010, at 5:21 AM, Gordan Bobic wrote: >>>> AlannY wrote: >>>>> On Mon, Feb 01, 2010 at 03:34:18PM +0000, Christine Caulfield wrote: >>>>>> Is it really two different devices ? If so then clvmd will not work. >>>>>> It needs the same view of storage on all systems. Not necessarily >>>>>> the same names, but definitely the same actual storage. >>>>> So, should I use DRBD with clvm for data mirroring on both nodes? >>>> You could, indeed, use DRBD. Or you could cross-export the block devices with iSCSI (or ATAoE) and have them connected to all the nodes that way. I guess it depends on whether you prefer to use DRBD or CLVM. My preference is for DRBD, but don't let that bias your decision. :) >>>> >>>> Gordan >>>> >>> I reckon GNBD will be a better solution in your case, DRBD is more >>> suitable for non-cluster file systems (e.g. ext3, xfs .. ) in case >>> of active-passive >> DRBD is specifically designed to also work in active-active mode. I've been running shared root GFS clusters on DRBD for years. DRBD is singularly _THE_ best solution for network RAID1, _especially_ in active-active mode with a clustered file system on top. It also handles resyncing after outages much more gracefully and transparently than other similar solutions. >> >> Gordan >> > > Yes, it does work in active-active but DRBD people themselves don't > recommend running it in production active-active under cluster file > system, I quote from their website: > "DRBD's primary-primary mode with a shared disk file system (GFS, > OCFS2). These systems are very sensitive to failures of the > replication network. Currently we cannot generally recommend this > for production use." > http://www.drbd.org/home/mirroring/ That surprises me - it could just be an out of date page. I've used it in active-active mode with GFS on top in all sorts of harsh and abusive edge-case ways and never saw it skip a beat. > In terms of production solution I reckon GNBD is designed more > specifically for that purpose. Not really. GNBD is really paper thin and quite dumb. It doesn't actually have any feature overlap with DRBD. It's more akin to iSCSI, in the sense that it is for exporting a block device, not mirroring a block device. In other words it's a 1->many export feature. It won't provide mirroring on it's own. Features like mirroring and post-outage resync have to be taken higher by "some else". And these alternatives handle failures nowhere nearly as gracefully as DRBD. For example, if a DRBD mirror fails (failed disk), all access will get transparently redirected to the surviving mirror. If a node disconnects, upon reconnection it will sync on the blocks that changed since it was last connected, and do so transparently, as you would expect. Gordan From a.alawi at auckland.ac.nz Tue Feb 2 02:54:04 2010 From: a.alawi at auckland.ac.nz (Abraham Alawi) Date: Tue, 2 Feb 2010 15:54:04 +1300 Subject: [Linux-cluster] GFS problem In-Reply-To: <9e4c4ee71001270222j3f4664at4503694a0dd1c73d@mail.gmail.com> References: <9e4c4ee71001270222j3f4664at4503694a0dd1c73d@mail.gmail.com> Message-ID: <041FC704-EF91-4804-B5FD-5AC7940D8305@auckland.ac.nz> -The cluster.conf lists more than 12 nodes, if there're redundant nodes then you may need to clean up cluster.conf just in case -Why expected_votes="8"? expected_votes should be the total votes in a fully functioning cluster, in your case it should be '12' the quorum would be calculated automatically by the basic formula (1/2 expected_votes_number + 1), so in the case of 12 votes (1 vote/node) the quorum would be 7, in other words the cluster would be kept running as long as there's 7 nodes (because in your case 1 vote per node). -I'd change post_fail_delay="0" to 5 (seconds) If still no luck then try this line in your cluster.conf file: Good luck, -- Abraham On 27/01/2010, at 11:22 PM, Alex Urbanowicz wrote: > From: Jorge Palma > To: linux clustering > Subject: Re: [Linux-cluster] GFS problem > Message-ID: > <5b65f1b11001251343p659d3b96gf07dd2165adf521e at mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > Please send your fence configuration and cluster.conf > > cluster.conf: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fencing is done using fence_rsysrq so there is no configuration to speak of except the iptables/modprobe part: > > options ipt_SYSRQ passwd="fencepassword" tolerance=3720 > > -A INPUT -i bond0.108 -s 10.100.108.0/24 -d -p udp -m udp --dport 9 -j SYSRQ > > Alex. > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster '''''''''''''''''''''''''''''''''''''''''''''''''''''' Abraham Alawi Unix/Linux Systems Administrator Science IT University of Auckland e: a.alawi at auckland.ac.nz p: +64-9-373 7599, ext#: 87572 '''''''''''''''''''''''''''''''''''''''''''''''''''''' -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirk.schulz at kinzesberg.de Tue Feb 2 07:12:32 2010 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Tue, 02 Feb 2010 08:12:32 +0100 Subject: [Linux-cluster] heartbeat In-Reply-To: <21E1652F6AA39A4B89489DCFC399C1FDAB9CA7@gbplamail.genband.com> References: <633e02961002010952s32f7c749i5f830094fc81a55e@mail.gmail.com> <21E1652F6AA39A4B89489DCFC399C1FDAB9CA7@gbplamail.genband.com> Message-ID: <4B67D060.5060308@kinzesberg.de> Ed Sanborn schrieb: > > I am intrigued by this thread. I am using RH Advanced Platform version > 5.2. I have an 8-node cluster > > running on an IBM Bladecenter. For those that may not be familiar with > the IBM Bladecenter the blades have 2 NICs > > on them and the Bladecenter chassis has a Nortel switch that the NIC?s > connect to. I am using both NIC?s plugged > > into two segments of my LAN. One of them is being used for the > multicast heartbeat traffic. It works BUT occasionally > > my network has serious disruptions due to this being a > telecommunications company and the engineers are > > always doing something to loop the network and/or saturate the network > with broadcasts taking down the > > subnet that my heartbeat traffic is on. Sometimes when this happens > the network is so bad > > that the multicast traffic is affected and I lose quorum on the > cluster. Once I fix the network problem I need > > to restart each node to get the cluster back up and running. Not good > and not the correct design. > > I would love to NOT use a public NIC for the multicast/heartbeat > traffic. I tried to setup virtual interfaces and > > VLAN those in the Nortel switch and moving the multicast/heartbeat > traffic over to them, leaving the 2 NICS on the original LAN segments > they were on but even with IBM assistance we were unsuccessful. Does > anyone have any experience > > with this? I want to do this to truly isolate the heartbeat network to > eliminate any possibility for network congestion > > taking down the cluster and also because I do not want the multicast > traffic pumped all over the LAN. > What I do not understand at the moment: If you can afford to restrict one of every blade's two interfaces to cluster communication, why don't you put them into a VLAN (the real interfaces, not virtual ones) and see to it that the VLAN has not connection to any outside network? Then the engineers would have no means of flooding your cluster communication subnet. Dirk > > Thanks, > > Ed > > (978) 210-9855 > > *From:* linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] *On Behalf Of *Kumar, Ashish > *Sent:* Monday, February 01, 2010 12:53 PM > *To:* Fagnon Raymond; linux clustering > *Subject:* Re: [Linux-cluster] heartbeat > > I don?t have any nics that I can dedicate for a cross over cable > for heartbeat so I was thinking I could use the serial or the > network itself. Is it possible to just use the primary interface > for heartbeat since it is my understanding it is nothing more than > a ping. > > > You don't need to dedicate a NIC for heartbeat with Red Hat Cluster. > Unlike Linux-HA, RHCS makes use of multicast to communicate with other > nodes. The multicast IP address is assigned by RHCS automatically, so > you don't need to do any special configuration. > > Please have a look at this: > http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Administration/s1-multicast-considerations-CA.html > > For setup instructions, check this: > http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Administration/s1-general-prop-conga-CA.html > - Section 3 deals with Multicast address settings > > ------------------------------------------------------------------------ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From dirk.schulz at kinzesberg.de Tue Feb 2 07:38:55 2010 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Tue, 02 Feb 2010 08:38:55 +0100 Subject: [Linux-cluster] clvmd init script Message-ID: <4B67D68F.6090002@kinzesberg.de> I am running a 2 node cluster on CentOS 5.4. Until yesterday I had always started the cluster services manually (service cman start, service clvmd start ...) and already noticed that service clvmd start never finishes - so I used "service clvmd start &" to get a working shell back. Yesterday I tried autostarting the cluster services (just cman, clvmd, gfs; no rgmanager yet) with system boot up, and now clvmd is stuck the same. I did not look into the order of init scripts before; since clvmd is started long before sshd, I now cannot log in, neither remote nor local. I had started up both nodes for this test, the other one without autostarting the services, so I can log into this and check for cluster sanity. group_tool ls looks like this: > [root at pclus1cent5-01 ~]# group_tool ls > type level name id state > fence 0 default 00010002 none > [1 2] > dlm 1 clvmd 00020002 none > [1 2] > dlm 1 XenImages 00020001 none > [1] > dlm 1 XenConfigs 00040001 none > [1] > dlm 1 rgmanager 00050001 none > [1] > gfs 2 XenImages 00010001 none > [1] > gfs 2 XenConfigs 00030001 none > [1] XenConfigs and XenImages are GFS volumes - looks like they are mounted on both nodes, but clvmd only runs on one (which is weird since the GFS volumes are set up on clustered LVs). Or do I misinterpret the output? Now the question is: Can I determine from the working node what the problem on the stuck node is? Can I force clvmd to finish or give up? And do I have to do/can I do something to the clvmd init script to make it work as expected? Dirk From yvette at dbtgroup.com Tue Feb 2 07:47:42 2010 From: yvette at dbtgroup.com (yvette hirth) Date: Tue, 02 Feb 2010 07:47:42 +0000 Subject: [Linux-cluster] heartbeat In-Reply-To: <4B67D060.5060308@kinzesberg.de> References: <633e02961002010952s32f7c749i5f830094fc81a55e@mail.gmail.com> <21E1652F6AA39A4B89489DCFC399C1FDAB9CA7@gbplamail.genband.com> <4B67D060.5060308@kinzesberg.de> Message-ID: <4B67D89E.1080601@dbtgroup.com> Dirk H. Schulz wrote: > What I do not understand at the moment: If you can afford to restrict > one of every blade's two interfaces to cluster communication, why don't > you put them into a VLAN (the real interfaces, not virtual ones) and see > to it that the VLAN has not connection to any outside network? > Then the engineers would have no means of flooding your cluster > communication subnet. yes, like an old dell 5124 24-port gigE switch. i have about a bunch of them laying around, and you can find them for cheap on ebay (like $100 or so). connect each port on the switch to one nic per blade. make sure your hosts files on all blades list all blades so as to avoid dns (i'm sure it does if your cluster is working properly). you can block and log dns->out on your iptables and that way any unknown hosts will show up pronto. just don't connect it to your firewall or any other internal network and that'll work fine for a heartbeat-only subnet. i used something like this on a colo-hosted site for high-security sql-only (no outside) access and it worked fab. yvette hirth From fdinitto at redhat.com Tue Feb 2 10:10:43 2010 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Tue, 02 Feb 2010 11:10:43 +0100 Subject: [Linux-cluster] Organizing Bug Squash Party for Cluster 3.x, GFS2 and more Message-ID: <4B67FA23.2070800@redhat.com> Hi everybody, this is the first time that we are trying to organize this kind of event and we would like to hear opinions and ideas from everybody. Because this is a bit of uncharted territory for us (upstream), I don?t want to set too high expectations from the very first round, but hey.. let?s try and see what happens. The general idea is to have a 24 hours "work together with upstream" to find bugs, test quick fixes and provide all sort of feedback (even "HEY THIS SOFTWARE SUCKS" is good feedback if you can tell us why and where we need to improve). I don?t want to restrict this bug squash party to only Red Hat Cluster and GFS2. If other upstreams (pacemaker, OCFS2, drbd, corosync, openais, conga, you name it) wants to join, please do so. This is an open invitation. Feel free to forward this idea around as long as I am at least CC?ed to keep track of participants. Assuming there is interest in the event, let?s take a look at some simple practical details: - When is this going to happen? I am targeting the 2nd of March as candidate date. Starting at 00:00 UTC and finishing at 23:59 UTC. Some areas will have more coverages, others a bit less. If there will be a next time, we will move the window around to facilitate other time zones. - What should be tested? Anything really.. do whatever you think your cluster should be able to do. - What kind of hardware should be used? Again.. anything you want to put in the test cluster. Virtual machines? bare metal.. you decide. - We will get any help to configure the cluster? It depends. This is not a training session, but mostly a dedicate tour to fix bugs. We don?t want to exclude anyone but it?s best if you have already some experience with clustering. - As team we don?t have a 24h coverage around the world yet. We are working to plan shifts to have at least one maintainer for each component/subsystem available at any time. We will post what?s the best coverage later on. - In order to coordinate the event, we will use #linux-cluster IRC channel on freenode. - All issues found or features reported should be filed in the appropriate bug tracker. It?s important to have track in case a fix or a change cannot be provided within the 24h time frame. - Be ready to trash your test cluster. - Be ready to test and report back with information. As much as possible we will try to provide patches and packages (read below) that people can quickly install and use for testing. - Distribution of choice: as best as we can, we would like to help as many distributions as possible, but there are some practical limits. We can easily provide binary packages for Fedora 12 and Fedora rawhide. I am in the process to setup Debian and Ubuntu build machines, but whenever possible, it?s best if you can build your own binary packages to offload the team from this task. - We will assume that the base version to start testing is the latest upstream release version at the time (excluding kernel). It might be too time consuming to look into bugs that might have been already fixed. - Kernel bits.. This is clearly more complex to test and build if you are less familiar with kernel and distribution packaging. We will work on the base of the latest kernel available for your specific distribution. Please make sure to have a URL handy to the source code for us to look at. Please feel free to add anything I might have missed. Cheers Fabio From kkovachev at varna.net Tue Feb 2 10:18:18 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Tue, 2 Feb 2010 12:18:18 +0200 Subject: [Linux-cluster] heartbeat In-Reply-To: <4B67D89E.1080601@dbtgroup.com> References: <633e02961002010952s32f7c749i5f830094fc81a55e@mail.gmail.com> <21E1652F6AA39A4B89489DCFC399C1FDAB9CA7@gbplamail.genband.com> <4B67D060.5060308@kinzesberg.de> <4B67D89E.1080601@dbtgroup.com> Message-ID: <20100202101249.M5973@varna.net> On Tue, 02 Feb 2010 07:47:42 +0000, yvette hirth wrote > Dirk H. Schulz wrote: > > > What I do not understand at the moment: If you can afford to restrict > > one of every blade's two interfaces to cluster communication, why don't > > you put them into a VLAN (the real interfaces, not virtual ones) and see > > to it that the VLAN has not connection to any outside network? > > Then the engineers would have no means of flooding your cluster > > communication subnet. > > yes, like an old dell 5124 24-port gigE switch. i have about a bunch of > them laying around, and you can find them for cheap on ebay (like $100 > or so). connect each port on the switch to one nic per blade. > > make sure your hosts files on all blades list all blades so as to avoid > dns (i'm sure it does if your cluster is working properly). you can > block and log dns->out on your iptables and that way any unknown hosts > will show up pronto. > i would add when you separate the internal/comunication network, in host files to list node1.internal, node2.internal pointing to the node IP in that separated network and use those names in cluster.conf in order to move the multicast there > just don't connect it to your firewall or any other internal network and > that'll work fine for a heartbeat-only subnet. i used something like > this on a colo-hosted site for high-security sql-only (no outside) > access and it worked fab. > > yvette hirth > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From kkovachev at varna.net Tue Feb 2 10:34:07 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Tue, 2 Feb 2010 12:34:07 +0200 Subject: [Linux-cluster] [clvm] Volume group for uuid not found In-Reply-To: <4B6766DA.8070201@bobich.net> References: <20100201145231.GA796@alanny-pc.lcl.starlink.ru> <4B66F47A.2000100@redhat.com> <20100201160952.GA9386@alanny-pc.lcl.starlink.ru> <4B66FF85.7070207@bobich.net> <0FA0A4F3-25D3-45AD-899F-80985BAFB21B@auckland.ac.nz> <4B67564A.7010503@bobich.net> <79CCC492-1AAE-4EAB-ADAF-713DD0D63498@auckland.ac.nz> <4B6766DA.8070201@bobich.net> Message-ID: <20100202101833.M29087@varna.net> > > > > Yes, it does work in active-active but DRBD people themselves don't > > recommend running it in production active-active under cluster file > > system, I quote from their website: > > "DRBD's primary-primary mode with a shared disk file system (GFS, > > OCFS2). These systems are very sensitive to failures of the > > replication network. Currently we cannot generally recommend this > > for production use." > > http://www.drbd.org/home/mirroring/ > 'These systems are very sensitive to failures of the replication network' is the key problem here, but if you use the same interface for cluster communication the node would be fenced in such cases and then DRBD will gracefully recover > That surprises me - it could just be an out of date page. I've used it > in active-active mode with GFS on top in all sorts of harsh and abusive > edge-case ways and never saw it skip a beat. > if cman is not running on the node sometimes i get in the logs: block drbd0: istiod1[3173] Concurrent local write detected! [DISCARD L] new: 0s +1024; pending: 0s +1024 but this can be ignored and when cman is running all is fine in active-active mode even with multipath in multibus mode for both storages so in short DRBD is OK with shared GFS in active-active, but you should keep cman (fenced) running on the node, use the same (dedicated) network for DRBD and cluster communication and as an additional precaution you should wait for all DRBD resources to became Primary on both nodes before you start using the file system to avoid corruption. From dan.candea at quah.ro Tue Feb 2 10:48:12 2010 From: dan.candea at quah.ro (Dan Candea) Date: Tue, 02 Feb 2010 12:48:12 +0200 Subject: [Linux-cluster] unfencing in 2.03.10 Message-ID: <4B6802EC.5090203@quah.ro> hello I read on Cluster-devel list about unfencing. There were some modification made, fence_node -U, but this is on cluster3. What about cluster2 unfencing? I'm using fence_ifmib, the switch port through which the node is accessing the shared storage is put in down state. After reboot, the node joins the cluster but the port remains down, so I have to do manual unfencing. There something to do in this matter? or is the normal behavior? BR -- Dan C?ndea Does God Play Dice? From fdinitto at redhat.com Tue Feb 2 11:16:12 2010 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Tue, 02 Feb 2010 12:16:12 +0100 Subject: [Linux-cluster] unfencing in 2.03.10 In-Reply-To: <4B6802EC.5090203@quah.ro> References: <4B6802EC.5090203@quah.ro> Message-ID: <4B68097C.5070504@redhat.com> On 2/2/2010 11:48 AM, Dan Candea wrote: > hello > > I read on Cluster-devel list about unfencing. There were some > modification made, fence_node -U, but this is on cluster3. What about > cluster2 unfencing? cluster2 is not supported or developed any more. Unfencing was only implemented in cluster3. If you need unfencing operations, you need to switch to cluster3. Fabio From akhilesh.shastry at gmail.com Tue Feb 2 11:52:25 2010 From: akhilesh.shastry at gmail.com (Akhilesh Shastry) Date: Tue, 2 Feb 2010 17:22:25 +0530 Subject: [Linux-cluster] Query on fencing Message-ID: <648744101002020352t69a7a86bgfcf9afeafaafdb28@mail.gmail.com> Hello Experts, I have a issue wherein The cluster nodes were shutdown, then powered on one of them while keeping the other powered off. While the 1st node is booting, at some moment where fencing is getting started the other node gets powered on too and starts to boot. System and OS detail are: Operating System: RHEL5U4 Kernel Version: 2.6.18-164.el5 Server Model: BL460c G1 They are using HP iLO for fencing. The cluster is perfectly ok. I checked /sbin/fence_ilo and found that to be ok. Could anyone please help me with this. Thanks in advance. I will be waiting for your response Regards, Akhilesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ed.Sanborn at genband.com Tue Feb 2 16:42:39 2010 From: Ed.Sanborn at genband.com (Ed Sanborn) Date: Tue, 2 Feb 2010 10:42:39 -0600 Subject: [Linux-cluster] heartbeat In-Reply-To: <20100202101249.M5973@varna.net> References: <633e02961002010952s32f7c749i5f830094fc81a55e@mail.gmail.com> <21E1652F6AA39A4B89489DCFC399C1FDAB9CA7@gbplamail.genband.com><4B67D060.5060308@kinzesberg.de> <4B67D89E.1080601@dbtgroup.com> <20100202101249.M5973@varna.net> Message-ID: <21E1652F6AA39A4B89489DCFC399C1FDB06073@gbplamail.genband.com> OK, I was not clear enough. I can't take one of the 2 NIC's and dedicate it for the heartbeat traffic. That's the problem. BOTH of the NIC's subnets are needed for the engineers. I only have 2 physical NIC's. So I need to virtualize one of them so that I can get 2 networks shared on one physical link. In general I know how that is done. In this case, I'm struggling with the particulars on the RH OS, Nortel switch and also if memory serves me correctly last time I tried this, the heartbeat traffic could not be sent over a virtual interface only a real interface. Hopefully this makes my predicament clearer. I have plenty of spare switches... But only 2 NIC's. If I had a 3rd NIC on these blades I would be all set. Ed -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Kaloyan Kovachev Sent: Tuesday, February 02, 2010 5:18 AM To: linux clustering Subject: Re: [Linux-cluster] heartbeat On Tue, 02 Feb 2010 07:47:42 +0000, yvette hirth wrote > Dirk H. Schulz wrote: > > > What I do not understand at the moment: If you can afford to restrict > > one of every blade's two interfaces to cluster communication, why don't > > you put them into a VLAN (the real interfaces, not virtual ones) and see > > to it that the VLAN has not connection to any outside network? > > Then the engineers would have no means of flooding your cluster > > communication subnet. > > yes, like an old dell 5124 24-port gigE switch. i have about a bunch of > them laying around, and you can find them for cheap on ebay (like $100 > or so). connect each port on the switch to one nic per blade. > > make sure your hosts files on all blades list all blades so as to avoid > dns (i'm sure it does if your cluster is working properly). you can > block and log dns->out on your iptables and that way any unknown hosts > will show up pronto. > i would add when you separate the internal/comunication network, in host files to list node1.internal, node2.internal pointing to the node IP in that separated network and use those names in cluster.conf in order to move the multicast there > just don't connect it to your firewall or any other internal network and > that'll work fine for a heartbeat-only subnet. i used something like > this on a colo-hosted site for high-security sql-only (no outside) > access and it worked fab. > > yvette hirth > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From sklemer at gmail.com Tue Feb 2 18:05:02 2010 From: sklemer at gmail.com (=?UTF-8?B?16nXnNeV150g16fXnNee16g=?=) Date: Tue, 2 Feb 2010 20:05:02 +0200 Subject: [Linux-cluster] heuristic & score question Message-ID: <2746211a1002021005q59feb548x9c89f5f64d38ba30@mail.gmail.com> Hello. I build 2 members cluster with qdisk. I define 3 heuristic which each got score=1 . 1 to public network router & 2 to cross interconnect. Its seems that only the public heuristic works. Q1: 1. When i test the cross interconnect , nothing happen. 2. When i test the public heuristic i got insufficent score 0/1 & the system is rebooted , which means that the cluster dont recognize my 2 other heuristic. Any hints about this ??? Q2: Can i give the qdisk himself score=3 ????? Thanks in Advance !!!!!!!!!!!!! Sklemer. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ?ubierp2 /etc/cluster # cat cluster.conf From teigland at redhat.com Tue Feb 2 18:58:13 2010 From: teigland at redhat.com (David Teigland) Date: Tue, 2 Feb 2010 12:58:13 -0600 Subject: [Linux-cluster] unfencing in 2.03.10 In-Reply-To: <4B6802EC.5090203@quah.ro> References: <4B6802EC.5090203@quah.ro> Message-ID: <20100202185813.GB17418@redhat.com> On Tue, Feb 02, 2010 at 12:48:12PM +0200, Dan Candea wrote: > hello > > I read on Cluster-devel list about unfencing. There were some > modification made, fence_node -U, but this is on cluster3. What about > cluster2 unfencing? > I'm using fence_ifmib, the switch port through which the node is > accessing the shared storage is put in down state. After reboot, the > node joins the cluster but the port remains down, so I have to do manual > unfencing. > There something to do in this matter? or is the normal behavior? We never attempted to automate the unfencing prior to cluster3, so in cluster2 you need to set it up to run explicitly yourself, e.g. put a specific call to fence_ifmib in rc.local or something. Dave From a.alawi at auckland.ac.nz Tue Feb 2 20:21:03 2010 From: a.alawi at auckland.ac.nz (Abraham Alawi) Date: Wed, 3 Feb 2010 09:21:03 +1300 Subject: [Linux-cluster] clvmd init script In-Reply-To: <4B67D68F.6090002@kinzesberg.de> References: <4B67D68F.6090002@kinzesberg.de> Message-ID: <826FB581-C636-4B09-A9EF-723F601A9376@auckland.ac.nz> On 2/02/2010, at 8:38 PM, Dirk H. Schulz wrote: > I am running a 2 node cluster on CentOS 5.4. > > Until yesterday I had always started the cluster services manually (service cman start, service clvmd start ...) and already noticed that service clvmd start never finishes - so I used "service clvmd start &" to get a working shell back. > clvmd runs before the backend storage is ready? > Yesterday I tried autostarting the cluster services (just cman, clvmd, gfs; no rgmanager yet) with system boot up, and now clvmd is stuck the same. I did not look into the order of init scripts before; since clvmd is started long before sshd, I now cannot log in, neither remote nor local. > I had started up both nodes for this test, the other one without autostarting the services, so I can log into this and check for cluster sanity. > > group_tool ls looks like this: >> [root at pclus1cent5-01 ~]# group_tool ls >> type level name id state fence 0 default 00010002 none [1 2] >> dlm 1 clvmd 00020002 none [1 2] >> dlm 1 XenImages 00020001 none [1] >> dlm 1 XenConfigs 00040001 none [1] >> dlm 1 rgmanager 00050001 none [1] >> gfs 2 XenImages 00010001 none [1] >> gfs 2 XenConfigs 00030001 none [1] > XenConfigs and XenImages are GFS volumes - looks like they are mounted on both nodes, but clvmd only runs on one (which is weird since the GFS volumes are set up on clustered LVs). Or do I misinterpret the output? > The output of group_tool says the opposite, clvmd is running on both nodes, gfs volumes & rgmanager are running only on one. > Now the question is: > Can I determine from the working node what the problem on the stuck node is? Can I force clvmd to finish or give up? > > And do I have to do/can I do something to the clvmd init script to make it work as expected? > I think some startup scripts clvmd depend on runs before it runs, find that service and make sure it runs before clvmd, you can confirm that by changing the sequence of clvmd script to run at last > Dirk > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster '''''''''''''''''''''''''''''''''''''''''''''''''''''' Abraham Alawi Unix/Linux Systems Administrator Science IT University of Auckland e: a.alawi at auckland.ac.nz p: +64-9-373 7599, ext#: 87572 '''''''''''''''''''''''''''''''''''''''''''''''''''''' From pmdyer at ctgcentral2.com Tue Feb 2 22:01:27 2010 From: pmdyer at ctgcentral2.com (Paul M. Dyer) Date: Tue, 2 Feb 2010 16:01:27 -0600 (CST) Subject: [Linux-cluster] NFS clustering, with SAN In-Reply-To: <8ee061011001270653k744de6b7j6e9a76deb75a9b77@mail.gmail.com> Message-ID: <32499060.01265148087729.JavaMail.root@athena> I finally got around to reading the URL doc that I sent earlier. It is a good doc, but the approach I used was different. I wanted to post it here to see if it is useful, and to see if there are any problems with it. I have fs from the SAN, as ext3 that is always available to every node, but only mounted with the service below. When the nfsVolume service gets started on a node, it also brings it's own IP address. Clients mount based on the IP and NFS share. So the node offering the share is any node in the failover domain. Below is an abbreviated cluster.conf. Paul ----- Original Message ----- From: "Terry" To: "linux clustering" Sent: Wednesday, January 27, 2010 8:53:44 AM (GMT-0600) America/Chicago Subject: Re: [Linux-cluster] NFS clustering, with SAN On Tue, Jan 26, 2010 at 6:36 PM, Jonathan Horne wrote: > --- On Tue, 1/26/10, Terry wrote: > >> From: Terry >> Subject: Re: [Linux-cluster] NFS clustering, with SAN >> To: "linux clustering" >> Date: Tuesday, January 26, 2010, 5:57 PM >> On Tue, Jan 26, 2010 at 5:15 PM, >> Jonathan Horne >> wrote: >> > greetings, i am new to this list, my apologies if this >> topic has been touched on before (but so far searching has >> not yielded a direction for me to follow) >> > >> > i have 2 servers that share a file system over SAN. >> ?both servers export this file system as a NFS share. ?i >> need to implement some sort of clustering so that if one >> server goes down, clients dont lose their connection to the >> exported file system. ?im hoping to find some >> implementation that will have a pretty instantaneous >> failover, as there will be pretty constant filewrites/reads >> from multiple client servers. >> > >> > can i please get some recommendations on what i should >> research to make this happen? ill gladly accept tips, or >> documents to read, i can work with either. >> > >> > thanks, >> > Jonathan >> > >> >> Hi Jonathan, >> >> Welcome to the list.? I have a large nfs cluster I >> manage.? All of the >> configuration magic to reduce the downtime and headache is >> on the NFS >> client side.? When there is a failover event, there is >> a short outage >> period for the IP to switch over to the other node.? A >> lot of things >> affect this, including your switching (arp) >> environment.? Is it a >> single export?? If not, you can create each export as >> a separate >> service, with associating separate IP address and create >> an >> active/active type of NFS environment.? Just a >> thought. >> >> Thanks, >> Terry >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > Terry, thanks for the reply. > > my setup is like this: 2 servers, with a shared LUN. ?LUN is mounted on both as /opt/data, both have /opt/data listed in the /etc/exports file. ?so basically 2 servers, and showmount -e against both of them show the exact same thing (and of course, its a given that since its a shared LUN (ofcs2 file system), both exports contain the exact same data). ?right now were in the testing phase of this project, and there are 4 NFS clients that are connecting (2 oracle servers writing files to the export, and 2 weblogic servers reading the files from oracle from the export). > > ultimately trying to fix this up in an HA setup, so that if one NFS server drops off, the clients dont know the difference. ?environment is fully switched, and all nodes (NFS servers and clients) have bonded network interfaces connected to separate switches (which connect via port-channel). > > you mentioned that you have setup on the client side that takes care of your failover headaches, im interested in hearing more about how this works. > > thanks, > Jonathan > > > > > > -- The link Paul posted is a great resource. Start there. In short, you're going to have downtime in the event of a failover, short, but downtime. Increase nodes to reduce the chances. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From rmicmirregs at gmail.com Tue Feb 2 22:33:42 2010 From: rmicmirregs at gmail.com (Rafael =?ISO-8859-1?Q?Mic=F3?= Miranda) Date: Tue, 02 Feb 2010 23:33:42 +0100 Subject: [Linux-cluster] Query on fencing In-Reply-To: <648744101002020352t69a7a86bgfcf9afeafaafdb28@mail.gmail.com> References: <648744101002020352t69a7a86bgfcf9afeafaafdb28@mail.gmail.com> Message-ID: <1265150022.6543.4.camel@mecatol> Hi Akhilesh El mar, 02-02-2010 a las 17:22 +0530, Akhilesh Shastry escribi?: > Hello Experts, > > > > I have a issue wherein > > > > The cluster nodes were shutdown, then powered on one of them while > keeping the other powered off. While the 1st node is booting, at some > moment where fencing is getting started the other node gets powered on > too and starts to boot. > > > > System and OS detail are: > > Operating System: RHEL5U4 > > Kernel Version: 2.6.18-164.el5 > > Server Model: BL460c G1 > > > > They are using HP iLO for fencing. The cluster is perfectly ok. I > checked /sbin/fence_ilo and found that to be ok. > > Could anyone please help me with this. > > > > Thanks in advance. I will be waiting for your response > > > > Regards, > > Akhilesh > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Let me explain what I understood from your question: 1.- Both nodes are shutdown 2.- You start node A 3.- While node A is booting, at some moment it starts a fence operation over node B 4.- As a result of this fencing operation, node B starts to boot too. As i see this a perfectly normal situation if you have set the cluster services to autostart at system boot time. As node A boots and finds no communications with node B it will fence node B. The fencing operation will restart via iLO your server, so if the server is powered off it will get powered on as the result of the restart. You can see the log of your iLO messages off "Power on/off signal sent to host server by: XXXX" to check this. My personal recommendation is to avoid using automated boot of cluster services at system boot time. If a cluster node fails let an administrator check the node prior to adding it to the cluster again. I hope this helps. Cheers, Rafael -- Rafael Mic? Miranda From dan.candea at quah.ro Wed Feb 3 09:56:36 2010 From: dan.candea at quah.ro (Dan Candea) Date: Wed, 03 Feb 2010 11:56:36 +0200 Subject: [Linux-cluster] unfencing in 2.03.10 In-Reply-To: <20100202185813.GB17418@redhat.com> References: <4B6802EC.5090203@quah.ro> <20100202185813.GB17418@redhat.com> Message-ID: <4B694854.3000607@quah.ro> On 02/02/2010 08:58 PM, David Teigland wrote: > On Tue, Feb 02, 2010 at 12:48:12PM +0200, Dan Candea wrote: > >> hello >> >> I read on Cluster-devel list about unfencing. There were some >> modification made, fence_node -U, but this is on cluster3. What about >> cluster2 unfencing? >> I'm using fence_ifmib, the switch port through which the node is >> accessing the shared storage is put in down state. After reboot, the >> node joins the cluster but the port remains down, so I have to do manual >> unfencing. >> There something to do in this matter? or is the normal behavior? >> > We never attempted to automate the unfencing prior to cluster3, so in > cluster2 you need to set it up to run explicitly yourself, e.g. put a > specific call to fence_ifmib in rc.local or something. > > Dave > > ok. I'll take this approach. It's not something fancy to do it, but not elegant either :) thank you -- Dan C?ndea Does God Play Dice? From kitgerrits at gmail.com Wed Feb 3 11:24:05 2010 From: kitgerrits at gmail.com (Kit Gerrits) Date: Wed, 3 Feb 2010 12:24:05 +0100 Subject: [Linux-cluster] local node name "app-acc-006.app.rws.local" not found in cluster.conf Message-ID: <352eb1a51002030324t7a00e299y27f4150347a0ee48@mail.gmail.com> Hello, I am trying to move my cluster heartbeat to a different (interconnect) VLAN without changing the hostname. - Machines 1-5 have set their hostname to app-acc-00[1-5]-ic to enforce the setting. - Machine 6 has its regular hostname app-acc-006 - The alternate hostname app-acc-006-ic is registered to the bond1 interface through DNS and /etc/hosts. - O/S: Red Hat Enterprise Linux Server release 5.3 (Tikanga) - Cman: 2.0.98 The nodes with the changed hostname work fine; however our applications expect the regular hostname. So while this workaround would get our cluster working, it would break our application. What more do we need to keep the hostname as app-acc-00*, while using the IP attached to app-acc-00*-ic as heartbeat channel? I currently get: Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] local node name "app-acc-006.app.rws.local" not found in cluster.conf Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] Error reading CCS info, cannot start Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] Error reading config from CCS Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] AIS Executive exiting (reason: could not read the main configuration file). Details of the configuration are attached. -- He attacked everything in life with a mix of extraordinary genius and naive incompetence, and it was often difficult to tell which was which. - Douglas Adams -------------- next part -------------- Hello, I am trying to move my cluster heartbeat to a different (interconnect) VLAN without changing the hostname. - Machines 1-5 have set their hostname to app-acc-00[1-5]-ic to enforce the setting. - Machine 6 has its regular hostname app-acc-006 - The alternate hostname app-acc-006-ic is registered to the bond1 interface through DNS and /etc/hosts. - O/S: Red Hat Enterprise Linux Server release 5.3 (Tikanga) - Cman: 2.0.98 The nodes with the changed hostname work fine; however our applications expect the regular hostname. So while this workaround would get our cluster working, it would break our application. What more do we need to keep the hostname as app-acc-00*, while using the IP attached to app-acc-00*-ic as heartbeat channel? Feb 2 18:10:24 app-acc-006 ccsd[5168]: Starting ccsd 2.0.98: Feb 2 18:10:24 app-acc-006 ccsd[5168]: Built: Mar 5 2009 16:15:59 Feb 2 18:10:24 app-acc-006 ccsd[5168]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Feb 2 18:10:24 app-acc-006 ccsd[5168]: cluster.conf (cluster name = appacc, version = 32) found. Feb 2 18:10:24 app-acc-006 ccsd[5168]: Remote copy of cluster.conf is from quorate node. Feb 2 18:10:24 app-acc-006 ccsd[5168]: Local version # : 32 Feb 2 18:10:24 app-acc-006 ccsd[5168]: Remote version #: 32 Feb 2 18:10:24 app-acc-006 ccsd[5168]: Remote copy of cluster.conf is from quorate node. Feb 2 18:10:24 app-acc-006 ccsd[5168]: Local version # : 32 Feb 2 18:10:24 app-acc-006 ccsd[5168]: Remote version #: 32 Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3' Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] AIS Executive Service: started and ready to provide service. Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] local node name "app-acc-006.app.rws.local" not found in cluster.conf Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] Error reading CCS info, cannot start Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] Error reading config from CCS Feb 2 18:10:24 app-acc-006 openais[5174]: [MAIN ] AIS Executive exiting (reason: could not read the main configuration file). My cluster.conf: The hosts are registered in DNS: app-acc-001-ic.app.rws.local has address 10.47.125.97 app-acc-002-ic.app.rws.local has address 10.47.125.98 app-acc-003-ic.app.rws.local has address 10.47.125.100 app-acc-004-ic.app.rws.local has address 10.47.125.101 app-acc-005-ic.app.rws.local has address 10.47.125.103 app-acc-006-ic.app.rws.local has address 10.47.125.104 The hosts are also registered with their regular names in DNS app-acc-001-ic.app.rws.local has address 10.47.125.10 app-acc-002-ic.app.rws.local has address 10.47.125.11 app-acc-003-ic.app.rws.local has address 10.47.125.13 app-acc-004-ic.app.rws.local has address 10.47.125.14 app-acc-005-ic.app.rws.local has address 10.47.125.16 app-acc-006-ic.app.rws.local has address 10.47.125.17 Note: 10.47.125.0/27 and 10.47.125.96/27 are different subnets. The IP is locally registered: [root at app-acc-006 ~]# ip a 10: eth1: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:23:7d:55:46:ac brd ff:ff:ff:ff:ff:ff inet 10.47.125.17/27 brd 10.47.125.31 scope global eth1 inet6 fe80::223:7dff:fe55:46ac/64 scope link valid_lft forever preferred_lft forever 14: bond1: mtu 1500 qdisc noqueue link/ether 00:24:81:7d:b2:80 brd ff:ff:ff:ff:ff:ff inet 10.47.125.104/27 brd 10.47.125.127 scope global bond1 inet6 fe80::224:81ff:fe7d:b280/64 scope link valid_lft forever preferred_lft forever From kkovachev at varna.net Wed Feb 3 12:23:39 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Wed, 3 Feb 2010 14:23:39 +0200 Subject: [Linux-cluster] trivial fix Message-ID: <20100203121907.M46818@varna.net> Hi, if the pid file contains more than 1 line (like sendmail) the status_check_pid function returns an error. The attached patch replaces it with 'read pid' like it is done for stop_generic -------------- next part -------------- A non-text attachment was scrubbed... Name: ra-skelet.diff Type: application/octet-stream Size: 299 bytes Desc: not available URL: From mammadshah at hotmail.com Thu Feb 4 05:11:32 2010 From: mammadshah at hotmail.com (Muhammad Ammad Shah) Date: Thu, 4 Feb 2010 10:11:32 +0500 Subject: [Linux-cluster] Active Active Cluster Message-ID: HI, Is it possible to setup Active/Active cluster using two nodes, (i want to test this on MySQL and Oracle 11G). or it requires some other software? Thanks, Muhammad Ammad Shah _________________________________________________________________ Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. https://signup.live.com/signup.aspx?id=60969 -------------- next part -------------- An HTML attachment was scrubbed... URL: From raju.rajsand at gmail.com Thu Feb 4 07:50:23 2010 From: raju.rajsand at gmail.com (Rajagopal Swaminathan) Date: Thu, 4 Feb 2010 13:20:23 +0530 Subject: [Linux-cluster] Active Active Cluster In-Reply-To: References: Message-ID: <8786b91c1002032350t6083465dq9860a47f5e12978e@mail.gmail.com> Greetings, On Thu, Feb 4, 2010 at 10:41 AM, Muhammad Ammad Shah wrote: > > Is it possible to setup Active/Active cluster using two nodes, (i want to > test this on MySQL and Oracle 11G). or it requires some other software? > You will need Oracle RAC. Use the OS clustering for Active/passive DB. Never for Active/Active DB. Yes you can use Active/Active cluster for application such as NFS/Terminal server/Samba etc. You will need special software such as RAC Hope you have not committed to any customer :) Regards, Rajagopal From gordan at bobich.net Thu Feb 4 09:30:50 2010 From: gordan at bobich.net (Gordan Bobic) Date: Thu, 04 Feb 2010 09:30:50 +0000 Subject: [Linux-cluster] Active Active Cluster In-Reply-To: References: Message-ID: <4B6A93CA.2000603@bobich.net> Muhammad Ammad Shah wrote: > > HI, > > Is it possible to setup Active/Active cluster using two nodes, (i want > to test this on MySQL and Oracle 11G). or it requires some other software? Not sure how Oracle handles this use-case, but with MySQL you have three options: 1) Master-Master replication Pros: Mature, fast. Cons: There is an inherent race condition since the replication is asynchronous. You'll have to thing about your application and see whether that makes it an acceptable solution. 2) Shared data Pros: Relatively straightforward to set up, just point your database path to a shared GFS mount point and set locking type to external Cons: Extremely slow if it's not a read-mostly database 3) MySQL Cluster Pros: Faster than 2) Cons: While it's running all data has to be in RAM, which limits the size of the databases. Still slower than 1). You can then only use RHCS to fail over IPs around. Gordan From kprasad at cordys.com Thu Feb 4 10:42:35 2010 From: kprasad at cordys.com (Kukkala Prasad) Date: Thu, 4 Feb 2010 16:12:35 +0530 Subject: [Linux-cluster] Failover not happening when SAN connectivity failed Message-ID: Hi All, We have a cluster setup with Redhat Cluster Suite on RHEL5 OS. SAN is one of the resource configured in the cluster. We observed that if the SAN connectivity is lost on a given node then resources are not getting failovered to second node. How to resolve this problem? Regards, Prasad. ________________________________ The information contained in this communication is confidential, intended solely for the use of the individual or entity to whom it is addressed and may be legally privileged and protected by professional secrecy. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error. This email does not constitute any commitment from Cordys Holding BV or any of its subsidiaries except when expressly agreed in a written agreement between the intended recipient and Cordys Holding BV or its subsidiaries. Cordys is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. Cordys does not guarantee that the integrity of this communication has been maintained nor that the communication is free of viruses, interceptions or interference. If you are not the intended recipient of this communication please return the communication to the sender and delete and destroy all copies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From criley at erad.com Thu Feb 4 15:29:00 2010 From: criley at erad.com (Charles Riley) Date: Thu, 4 Feb 2010 10:29:00 -0500 (EST) Subject: [Linux-cluster] Failover not happening when SAN connectivity failed In-Reply-To: Message-ID: <25969557.390221265297340921.JavaMail.root@boardwalk2.erad.com> (you don't really expect someone to help you with no information regarding your configuration, do you?) Charles Charles Riley eRAD, Inc. ----- Original Message ----- From: "Kukkala Prasad" To: linux-cluster at redhat.com Sent: Thursday, February 4, 2010 5:42:35 AM GMT -05:00 US/Canada Eastern Subject: [Linux-cluster] Failover not happening when SAN connectivity failed Hi All, We have a cluster setup with Redhat Cluster Suite on RHEL5 OS. SAN is one of the resource configured in the cluster. We observed that if the SAN connectivity is lost on a given node then resources are not getting failovered to second node. How to resolve this problem? Regards, Prasad. The information contained in this communication is confidential, intended solely for the use of the individual or entity to whom it is addressed and may be legally privileged and protected by professional secrecy. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error. This email does not constitute any commitment from Cordys Holding BV or any of its subsidiaries except when expressly agreed in a written agreement between the intended recipient and Cordys Holding BV or its subsidiaries. Cordys is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. Cordys does not guarantee that the integrity of this communication has been maintained nor that the communication is free of viruses, interceptions or interference. If you are not the intended recipient of this communication please return the communication to the sender and delete and destroy all copies. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From jeff.sturm at eprize.com Thu Feb 4 15:56:32 2010 From: jeff.sturm at eprize.com (Jeff Sturm) Date: Thu, 4 Feb 2010 10:56:32 -0500 Subject: [Linux-cluster] Active Active Cluster In-Reply-To: <4B6A93CA.2000603@bobich.net> References: <4B6A93CA.2000603@bobich.net> Message-ID: <64D0546C5EBBD147B75DE133D798665F055D8713@hugo.eprize.local> > -----Original Message----- > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] > On Behalf Of Gordan Bobic > Sent: Thursday, February 04, 2010 4:31 AM > To: linux clustering > Subject: Re: [Linux-cluster] Active Active Cluster > > 3) MySQL Cluster > Pros: Faster than 2) > Cons: While it's running all data has to be in RAM, which limits the > size of the databases. Still slower than 1). MySQL Cluster supports disk data tables these days, which have their own caveats but can grow larger than available RAM. Indexes must still fit entirely in RAM. Plan on a lot of testing if anyone tries it--you can get good results if you have control over the queries and schemas, but a na?ve port from, say, InnoDB to NDB can be disastrous. MySQL Cluster is complex and has nothing to do with Red Hat Cluster (which is why I'll stop here--subscribe to cluster at lists.mysql.com for more info). -Jeff From crosa at redhat.com Thu Feb 4 17:20:13 2010 From: crosa at redhat.com (Cleber Rodrigues) Date: Thu, 04 Feb 2010 15:20:13 -0200 Subject: [Linux-cluster] Active Active Cluster In-Reply-To: <64D0546C5EBBD147B75DE133D798665F055D8713@hugo.eprize.local> References: <4B6A93CA.2000603@bobich.net> <64D0546C5EBBD147B75DE133D798665F055D8713@hugo.eprize.local> Message-ID: <1265304013.2245.4.camel@localhost> Folks, For the sole purpose of providing info: We had a very specific customer demand for high availability of a Web application, plus MySQL service and data. The machines holding theses services would be out in the field, with no access whatsoever to shared storage. We implemented a two node Red Hat Cluster configuration to allow for Web Application HA, and droped in a IP resource for the MySQL service. The MySQL service itself is always on, on both hosts, configured with master<->master replication. This has been in production use for something like 20 sites, for over two years. CR. On Thu, 2010-02-04 at 10:56 -0500, Jeff Sturm wrote: > > -----Original Message----- > > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] > > On Behalf Of Gordan Bobic > > Sent: Thursday, February 04, 2010 4:31 AM > > To: linux clustering > > Subject: Re: [Linux-cluster] Active Active Cluster > > > > 3) MySQL Cluster > > Pros: Faster than 2) > > Cons: While it's running all data has to be in RAM, which limits the > > size of the databases. Still slower than 1). > > MySQL Cluster supports disk data tables these days, which have their own caveats but can grow larger than available RAM. Indexes must still fit entirely in RAM. > > Plan on a lot of testing if anyone tries it--you can get good results if you have control over the queries and schemas, but a na?ve port from, say, InnoDB to NDB can be disastrous. MySQL Cluster is complex and has nothing to do with Red Hat Cluster (which is why I'll stop here--subscribe to cluster at lists.mysql.com for more info). > > -Jeff > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Cleber Rodrigues Solutions Architect - Red Hat, Inc. Mobile: +55 61 9185.3454 From thomas.meller at gmx.net Thu Feb 4 18:10:21 2010 From: thomas.meller at gmx.net (Thomas Meller) Date: Thu, 04 Feb 2010 19:10:21 +0100 Subject: [Linux-cluster] Quorum disk over RAID software device Message-ID: <20100204181021.212900@gmx.net> Just found this thread while searching for illumination. We are running a self-constructed cluster setup since 4 Years on a 2.4 kernel (RHAS3). It's a two-machine setup, nothing fancy. The fancy thing is that we boot the nodes in different computing centers from mirrored SAN devices which are run in separate storage controllers located in different computing centers over twice redundant FC fabrics, set up a mirrored root fs on these and don't need the local disks except for installation. In short: we survived everything. :-) Up to now, we used a vendor-specific multipath software (RDAC), which I want to get rid of. Today we could use RH5.3's dm-multipathing instead. Quorum is done by usage of Robin Miller's excellent scu utility and some scripting. To get there, I need to create an initrd which contains: - the newest qla-drivers - Qlogic's 'scli' utility - multipaths set up to create the raid-mirrors on - mdadm I am no RedHat-guru and not used to using mkinitrd. The Qlogic installer creates initrds automatically, but not usable for me. The mkinird script seems to know md-multipath, but no dm-multipath (as far as I understood it's code). Currently I am on the local disk, have multipath running, made the qla-drivers and copied scli, mdadm and the needed libraries manually to the initrd. What do I need in the initrd? I read something of a dm-multipathd. How can I include all modules needed (which?) by running mkinitrd? (do I need to have multipath and mirrors already running? Are there configfiles to write?) Is it possible to include specified tools (not modules) in a mkinitrd run? Why does mdadm -A tell me 'device or ressource busy' on the second of two raid1-devices? You see, I need the big picture as well as a short intro to mkinitrd and some troubleshooting. Can anyone help me find my way through? If there's documentation, I will gladly read it first. Up to now, I found only fuzzy things. Thanks a lot! Thomas -- http://www.openstreetmap.org/?lat=47.172&lon=7.4395&zoom=15&layers=B000FTFTT&mlat=47.16677&mlon=7.43513 Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser From gordan at bobich.net Thu Feb 4 18:59:52 2010 From: gordan at bobich.net (Gordan Bobic) Date: Thu, 04 Feb 2010 18:59:52 +0000 Subject: [Linux-cluster] Quorum disk over RAID software device In-Reply-To: <20100204181021.212900@gmx.net> References: <20100204181021.212900@gmx.net> Message-ID: <4B6B1928.2080200@bobich.net> Thomas, It sounds like what you're looking for is Open Shared Root: http://www.open-sharedroot.org/ Gordan Thomas Meller wrote: > Just found this thread while searching for illumination. > > We are running a self-constructed cluster setup since 4 Years on a 2.4 kernel (RHAS3). > It's a two-machine setup, nothing fancy. > The fancy thing is that we boot the nodes in different computing centers from mirrored SAN devices which are run in separate storage controllers located in different computing centers over twice redundant FC fabrics, set up a mirrored root fs on these and don't need the local disks except for installation. > In short: we survived everything. > :-) > > Up to now, we used a vendor-specific multipath software (RDAC), which I want to get rid of. Today we could use RH5.3's dm-multipathing instead. > Quorum is done by usage of Robin Miller's excellent scu utility and some scripting. > > To get there, I need to create an initrd which contains: > - the newest qla-drivers > - Qlogic's 'scli' utility > - multipaths set up to create the raid-mirrors on > - mdadm > > I am no RedHat-guru and not used to using mkinitrd. > The Qlogic installer creates initrds automatically, but not usable for me. > The mkinird script seems to know md-multipath, but no dm-multipath (as far as I understood it's code). > > Currently I am on the local disk, have multipath running, made the qla-drivers and copied scli, mdadm and the needed libraries manually to the initrd. > > What do I need in the initrd? I read something of a dm-multipathd. > How can I include all modules needed (which?) by running mkinitrd? (do I need to have multipath and mirrors already running? Are there configfiles to write?) > Is it possible to include specified tools (not modules) in a mkinitrd run? > Why does mdadm -A tell me 'device or ressource busy' on the second of two raid1-devices? > > You see, I need the big picture as well as a short intro to mkinitrd and some troubleshooting. > Can anyone help me find my way through? If there's documentation, I will gladly read it first. Up to now, I found only fuzzy things. > > Thanks a lot! > Thomas > From kprasad at cordys.com Fri Feb 5 04:40:14 2010 From: kprasad at cordys.com (Kukkala Prasad) Date: Fri, 5 Feb 2010 10:10:14 +0530 Subject: [Linux-cluster] Failover not happening when SAN connectivity failed In-Reply-To: <25969557.390221265297340921.JavaMail.root@boardwalk2.erad.com> References: <25969557.390221265297340921.JavaMail.root@boardwalk2.erad.com> Message-ID: Hi Charles, Here is the configuration details. Regards, Prasad. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Charles Riley Sent: Thursday, February 04, 2010 8:59 PM To: linux clustering Subject: Re: [Linux-cluster] Failover not happening when SAN connectivity failed (you don't really expect someone to help you with no information regarding your configuration, do you?) Charles Charles Riley eRAD, Inc. ----- Original Message ----- From: "Kukkala Prasad" To: linux-cluster at redhat.com Sent: Thursday, February 4, 2010 5:42:35 AM GMT -05:00 US/Canada Eastern Subject: [Linux-cluster] Failover not happening when SAN connectivity failed Hi All, We have a cluster setup with Redhat Cluster Suite on RHEL5 OS. SAN is one of the resource configured in the cluster. We observed that if the SAN connectivity is lost on a given node then resources are not getting failovered to second node. How to resolve this problem? Regards, Prasad. The information contained in this communication is confidential, intended solely for the use of the individual or entity to whom it is addressed and may be legally privileged and protected by professional secrecy. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error. This email does not constitute any commitment from Cordys Holding BV or any of its subsidiaries except when expressly agreed in a written agreement between the intended recipient and Cordys Holding BV or its subsidiaries. Cordys is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. Cordys does not guarantee that the integrity of this communication has been m! aintained nor that the communication is free of viruses, interceptions or interference. If you are not the intended recipient of this communication please return the communication to the sender and delete and destroy all copies. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From linuxcluster at msn.com Fri Feb 5 10:30:09 2010 From: linuxcluster at msn.com (H P) Date: Fri, 5 Feb 2010 16:00:09 +0530 Subject: [Linux-cluster] cron file Message-ID: Deal All, I am intrested in setting up a cron file for email notifications for the when job starts and ends on the cluster. Does anyone have an example of perl/shell script which can be configured with cron? Regards HP _________________________________________________________________ Hotmail: Free, trusted and rich email service. https://signup.live.com/signup.aspx?id=60969 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hicheerup at gmail.com Fri Feb 5 11:30:26 2010 From: hicheerup at gmail.com (linux-crazy) Date: Fri, 5 Feb 2010 17:00:26 +0530 Subject: [Linux-cluster] Failover not happening when SAN connectivity failed In-Reply-To: References: <25969557.390221265297340921.JavaMail.root@boardwalk2.erad.com> Message-ID: <29e045b81002050330s2e2fdb29i249c26c14bb8fa31@mail.gmail.com> Hi, Attach the /var/log/messages or rgmanager logs to get more info .. On Fri, Feb 5, 2010 at 10:10 AM, Kukkala Prasad wrote: > Hi Charles, > > Here is the configuration details. > > > > ? ? ? ? > ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? > ? ? ? ? > ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? > ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? > > > Regards, > Prasad. > > -----Original Message----- > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Charles Riley > Sent: Thursday, February 04, 2010 8:59 PM > To: linux clustering > Subject: Re: [Linux-cluster] Failover not happening when SAN connectivity failed > > > (you don't really expect someone to help you with no information regarding your configuration, do you?) > > Charles > > Charles Riley > eRAD, Inc. > > > > ----- Original Message ----- > From: "Kukkala Prasad" > To: linux-cluster at redhat.com > Sent: Thursday, February 4, 2010 5:42:35 AM GMT -05:00 US/Canada Eastern > Subject: [Linux-cluster] Failover not happening when SAN connectivity failed > > > > > > Hi All, > > > > We have a cluster setup with Redhat Cluster Suite on RHEL5 OS. > > SAN is one of the resource configured in the cluster. > > We observed that if the SAN connectivity is lost on a given node then resources are not getting failovered to second node. How to resolve this problem? > > > > Regards, > > Prasad. > > The information contained in this communication is confidential, intended solely for the use of the individual or entity to whom it is addressed and may be legally privileged and protected by professional secrecy. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error. This email does not constitute any commitment from Cordys Holding BV or any of its subsidiaries except when expressly agreed in a written agreement between the intended recipient and Cordys Holding BV or its subsidiaries. Cordys is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. Cordys does not guarantee that the integrity of this communication has been m! > ?aintained nor that the communication is free of viruses, interceptions or interference. If you are not the intended recipient of this communication please return the communication to the sender and delete and destroy all copies. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From thomas.meller at gmx.net Fri Feb 5 14:56:24 2010 From: thomas.meller at gmx.net (Thomas Meller) Date: Fri, 05 Feb 2010 15:56:24 +0100 Subject: [Linux-cluster] Quorum disk over RAID software device In-Reply-To: <4B6B1928.2080200@bobich.net> References: <20100204181021.212900@gmx.net> <4B6B1928.2080200@bobich.net> Message-ID: <20100205145624.280470@gmx.net> Many thanks, Gordan. This could nearly be the solution. But as I understand, it's not possible to mirror the root-(g)fs to another computing center despite for relying on a new SPOF (if at all possible) or on hardware-dependent solutions. We had storage problems in the past and could survive them. That's not possible any more, or else we will be bound to a new dependency on a working network infrastructure. Hmm... our current setup addressed this with half the effort. Anyway, does anyone in this forum use this technique and can send me the 'init'-script from the initrd? Thanks again, Thomas -------- Original-Nachricht -------- > Datum: Thu, 04 Feb 2010 18:59:52 +0000 > Von: Gordan Bobic > An: linux clustering > Betreff: Re: [Linux-cluster] Quorum disk over RAID software device > Thomas, > > It sounds like what you're looking for is Open Shared Root: > http://www.open-sharedroot.org/ > > Gordan > > Thomas Meller wrote: > > Just found this thread while searching for illumination. > > > > We are running a self-constructed cluster setup since 4 Years on a 2.4 > kernel (RHAS3). > > It's a two-machine setup, nothing fancy. > > The fancy thing is that we boot the nodes in different computing centers > from mirrored SAN devices which are run in separate storage controllers > located in different computing centers over twice redundant FC fabrics, set > up a mirrored root fs on these and don't need the local disks except for > installation. > > In short: we survived everything. > > :-) > > > > Up to now, we used a vendor-specific multipath software (RDAC), which I > want to get rid of. Today we could use RH5.3's dm-multipathing instead. > > Quorum is done by usage of Robin Miller's excellent scu utility and some > scripting. > > > > To get there, I need to create an initrd which contains: > > - the newest qla-drivers > > - Qlogic's 'scli' utility > > - multipaths set up to create the raid-mirrors on > > - mdadm > > > > I am no RedHat-guru and not used to using mkinitrd. > > The Qlogic installer creates initrds automatically, but not usable for > me. > > The mkinird script seems to know md-multipath, but no dm-multipath (as > far as I understood it's code). > > > > Currently I am on the local disk, have multipath running, made the > qla-drivers and copied scli, mdadm and the needed libraries manually to the > initrd. > > > > What do I need in the initrd? I read something of a dm-multipathd. > > How can I include all modules needed (which?) by running mkinitrd? (do I > need to have multipath and mirrors already running? Are there configfiles > to write?) > > Is it possible to include specified tools (not modules) in a mkinitrd > run? > > Why does mdadm -A tell me 'device or ressource busy' on the second of > two raid1-devices? > > > > You see, I need the big picture as well as a short intro to mkinitrd and > some troubleshooting. > > Can anyone help me find my way through? If there's documentation, I will > gladly read it first. Up to now, I found only fuzzy things. > > > > Thanks a lot! > > Thomas > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- http://www.openstreetmap.org/?lat=47.172&lon=7.4395&zoom=15&layers=B000FTFTT&mlat=47.16677&mlon=7.43513 Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/chbrowser From gordan at bobich.net Fri Feb 5 15:18:07 2010 From: gordan at bobich.net (Gordan Bobic) Date: Fri, 05 Feb 2010 15:18:07 +0000 Subject: [Linux-cluster] Quorum disk over RAID software device In-Reply-To: <20100205145624.280470@gmx.net> References: <20100204181021.212900@gmx.net> <4B6B1928.2080200@bobich.net> <20100205145624.280470@gmx.net> Message-ID: <4B6C36AF.8080901@bobich.net> Thomas Meller wrote: > Many thanks, Gordan. > > This could nearly be the solution. > But as I understand, it's not possible to mirror the root-(g)fs to another > computing center despite for relying on a new SPOF (if at all possible) > or on hardware-dependent solutions. Not sure I follow what you mean. If your underlying storage is synchronously mirrored, you should have no problem with GFS root on top of it. If things disconnect, of course, it'll split-brain, and one node will get panicked and need rebooting. If your mirrored SAN can't handle it gracefully, you could always use DRBD to handle the real-time mirroring. That will even re-direct to the remote storage node in case just the local storage fails. Since I added the DRBD functionality to OSR, I'm pretty sure it works. ;) > Hmm... our current setup addressed this with half the effort. If you need more graceful recovery from network outages (especially the cross-site interconnect), you could use something like GlusterFS + OSR. I contributed patches for that, too, but it's a bit experimental. > Anyway, does anyone in this forum use this technique and can send me > the 'init'-script from the initrd? I'm still not entirely sure what exactly you're looking for. Can you elaborate a bit more? Gordan From thomas.meller at gmx.net Sat Feb 6 00:29:58 2010 From: thomas.meller at gmx.net (Thomas Meller) Date: Sat, 06 Feb 2010 01:29:58 +0100 Subject: [Linux-cluster] Quorum disk over RAID software device In-Reply-To: <4B6C36AF.8080901@bobich.net> References: <20100204181021.212900@gmx.net> <4B6B1928.2080200@bobich.net> <20100205145624.280470@gmx.net> <4B6C36AF.8080901@bobich.net> Message-ID: <20100206002958.106730@gmx.net> You're right, I am unclear. Some years ago, we tried two versions: storage-based mirroring and host-based mirroring. As the processes were too complicated in our company we decided to mirror the disks host-based. So currently there is a /dev/md0 (simplified) consisting of sda (in Bern) and sdb (in Zurich), and each node has it's own root-fs exclusively. This cannot work with a shared GFS, as there are several machines doing updates on the FS and no central instance does always know the current state of the device's contents, thus no host-mirroring possible. You are talking of storage-based mirrors. In case of a failure, we would have to direct the storage system to use the second mirror as primary and direct our nodes to write on sdb instead of sda. That will involve controlling the storage from our machines (our storage people will love the idea) and installing the storage-specific software on them. If the Hardware in use changes, we need to re-engineer this solution and adapt to the new storage manufacturer's philosophy, if at all possible. I still have a third opportunity. I can use Qlocig's driver-based multipathing and keep using host-based mirroring instead of using dm-multipath, which currently prevents me from setting up raid-devices as root-fs. Well, that will work, but is somewhat ugly. So far, I had only a short glimpse on OSR. I think I will need to dive deeper. -------- Original-Nachricht -------- > Datum: Fri, 05 Feb 2010 15:18:07 +0000 > Von: Gordan Bobic > An: linux clustering > Betreff: Re: [Linux-cluster] Quorum disk over RAID software device > Thomas Meller wrote: > > Many thanks, Gordan. > > > > This could nearly be the solution. > > But as I understand, it's not possible to mirror the root-(g)fs to > another > > computing center despite for relying on a new SPOF (if at all possible) > > or on hardware-dependent solutions. > > Not sure I follow what you mean. If your underlying storage is > synchronously mirrored, you should have no problem with GFS root on top > of it. If things disconnect, of course, it'll split-brain, and one node > will get panicked and need rebooting. > > If your mirrored SAN can't handle it gracefully, you could always use > DRBD to handle the real-time mirroring. That will even re-direct to the > remote storage node in case just the local storage fails. > > Since I added the DRBD functionality to OSR, I'm pretty sure it works. ;) > > > Hmm... our current setup addressed this with half the effort. > > If you need more graceful recovery from network outages (especially the > cross-site interconnect), you could use something like GlusterFS + OSR. > I contributed patches for that, too, but it's a bit experimental. > > > Anyway, does anyone in this forum use this technique and can send me > > the 'init'-script from the initrd? > > I'm still not entirely sure what exactly you're looking for. Can you > elaborate a bit more? > > Gordan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- http://www.openstreetmap.org/?lat=47.172&lon=7.4395&zoom=15&layers=B000FTFTT&mlat=47.16677&mlon=7.43513 NEU: Mit GMX DSL ?ber 1000,- ? sparen! http://portal.gmx.net/de/go/dsl02 From gordan at bobich.net Sat Feb 6 02:27:55 2010 From: gordan at bobich.net (Gordan Bobic) Date: Sat, 06 Feb 2010 02:27:55 +0000 Subject: [Linux-cluster] Quorum disk over RAID software device In-Reply-To: <20100206002958.106730@gmx.net> References: <20100204181021.212900@gmx.net> <4B6B1928.2080200@bobich.net> <20100205145624.280470@gmx.net> <4B6C36AF.8080901@bobich.net> <20100206002958.106730@gmx.net> Message-ID: <4B6CD3AB.8090507@bobich.net> Thomas Meller wrote: > You're right, I am unclear. > > Some years ago, we tried two versions: storage-based > mirroring and host-based mirroring. As the processes were > too complicated in our company we decided to mirror the > disks host-based. So currently there is a /dev/md0 > (simplified) consisting of sda (in Bern) and sdb (in Zurich), > and each node has it's own root-fs exclusively. Since MD RAID cannot be mounted simultaneously from more than one place, I can only assume that you have a fail-over rather than an active-active solution. > This cannot work with a shared GFS, as there are several > machines doing updates on the FS and no central instance > does always know the current state of the device's contents, > thus no host-mirroring possible. I thought you have an MD device doing mirroring... > You are talking of storage-based mirrors. In case of a > failure, we would have to direct the storage system to use > the second mirror as primary and direct our nodes to write > on sdb instead of sda. Right - so you are using it as active-passive, then. > That will involve controlling the storage from our machines > (our storage people will love the idea) and installing the > storage-specific software on them. Or you can have DRBD do the mirroring and fail-over handling for you on whatever device(s) you have exposed to the servers. > If the Hardware in use changes, we need to re-engineer this > solution and adapt to the new storage manufacturer's > philosophy, if at all possible. Well, you'll always need to at least make sure you have a suitable SCSI driver available - unless you use something nice and open like iSCSI SANs. > I still have a third opportunity. I can use Qlocig's driver-based > multipathing and keep using host-based mirroring instead of > using dm-multipath, which currently prevents me from setting up > raid-devices as root-fs. I'm still not sure how all these relate in your setup. Are you saying that you are using the qlogic multi-path driver pointing at two different SANs while the SANs themselves are sorting out the synchronous real-time mirroring between them? > Well, that will work, but is somewhat ugly. > > So far, I had only a short glimpse on OSR. I think I will need > to dive deeper. It sounds like you'll need to add support for qlogic multi-path proprietary stuff to OSR before it'll do exactly what you want, but other than that, the idea behind it is to enable you have have a shared rootfs on a suitable cluster file system (GFS, OCFS, GlusterFS, etc.). It's generally useful when you need a big fat initrd (although there has been a significant effort to make the initrd go on a serious diet over time) to bootstrap things such as RHCS components, block device drivers (e.g. DRBD), or file systems that need a fuller environment to start up than a normal initrd (e.g. GlusterFS, things that need glibc, etc.). Gordan From mammadshah at hotmail.com Sun Feb 7 06:57:30 2010 From: mammadshah at hotmail.com (Muhammad Ammad Shah) Date: Sun, 7 Feb 2010 11:57:30 +0500 Subject: [Linux-cluster] Red Hat Cluster with VMware Server/Workstation Message-ID: Hi, Hope you are doing well, i need some help in regard to, How do i perform Red Hat clustering labs when using VMWare ? what would be fencing device. Thanks, Muhammad Ammad Shah _________________________________________________________________ Hotmail: Trusted email with powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mammadshah at hotmail.com Sun Feb 7 08:56:49 2010 From: mammadshah at hotmail.com (Muhammad Ammad Shah) Date: Sun, 7 Feb 2010 13:56:49 +0500 Subject: [Linux-cluster] fencing node and filesystem Message-ID: Hi, when a node(x) fenced by another node(y), will it corrupts the file system or no? and if the database was running and node(x) fenced by fenced node(y) will it corrupts the database or whats about the transaction ? Thanks, Muhammad Ammad Shah _________________________________________________________________ Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. https://signup.live.com/signup.aspx?id=60969 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpeterso at redhat.com Mon Feb 8 14:49:34 2010 From: rpeterso at redhat.com (Bob Peterson) Date: Mon, 8 Feb 2010 09:49:34 -0500 (EST) Subject: [Linux-cluster] fencing node and filesystem In-Reply-To: Message-ID: <266292776.1231251265640574215.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> ----- "Muhammad Ammad Shah" wrote: | Hi, | | when a node(x) fenced by another node(y), will it corrupts the file | system or no? and if the database was running and node(x) fenced by | fenced node(y) will it corrupts the database or whats about the | transaction ? | | | | Thanks, | Muhammad Ammad Shah With GFS and GFS2 there should be no metadata corruption, in theory. When quorum is regained (or if it's not lost) the fenced node's journal is replayed so the file system metadata should remain intact. That does not mean that data won't be lost. If the node was fenced while it had data in its cache, that data may be lost. In critical situations, you can mark files and/or directories with a flag that makes the data journaled as well, so the data should be protected. Of course, that has a performance penalty. There's a small section about this in the GFS faq. Regards, Bob Peterson Red Hat File Systems From jimbobpalmer at gmail.com Mon Feb 8 17:12:04 2010 From: jimbobpalmer at gmail.com (jimbob palmer) Date: Mon, 8 Feb 2010 18:12:04 +0100 Subject: [Linux-cluster] RHCS networking config (multicast/broadcast/unicast) Message-ID: Hello, I'd like to use RHCS but before I do, I have a question about how networking is handled. I have a network on which multicast is not enabled but is promoted to broadcast. I'd like to run multiple clusters on this network without them conflicting or causing lots of log noise. Can I configure RHCS so that this will work? Will different ports work, or perhaps is unicast support? I'd be interested if this is well supported or requires changing non config files. Thanks a lot. From yvette at dbtgroup.com Mon Feb 8 17:38:32 2010 From: yvette at dbtgroup.com (yvette hirth) Date: Mon, 08 Feb 2010 17:38:32 +0000 Subject: [Linux-cluster] non-RH/Suse systems? In-Reply-To: <266292776.1231251265640574215.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> References: <266292776.1231251265640574215.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> Message-ID: <4B704C18.8080104@dbtgroup.com> hi, i have three ubuntu 9.04 boxes i'd like to cluster. one is running on a DL380 G6 so i have ILO2 on that box. the other two boxes would need to be upgraded to hardware that has fencing. as i'm pretty happy with the DL380 G6 i'd upgrade those two boxes to G6's, but before we spend that cost, we'd like to know if we can cluster ubuntu. anyone using ubuntu? if so, problems? thanks for the info and your time. yvette hirth From Martin.Waite at datacash.com Mon Feb 8 17:51:26 2010 From: Martin.Waite at datacash.com (Martin Waite) Date: Mon, 8 Feb 2010 17:51:26 -0000 Subject: [Linux-cluster] non-RH/Suse systems? In-Reply-To: <4B704C18.8080104@dbtgroup.com> References: <266292776.1231251265640574215.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> <4B704C18.8080104@dbtgroup.com> Message-ID: Hi, There have been problems with old bugs in the rgmanager packages in Debian-based distributions that have prevented service failovers from completing cleanly. The fixes were recently pushed into the latest Lenny minor release. It is quite likely that Ubuntu packages have inherited these bugs. regards, Martin -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of yvette hirth Sent: 08 February 2010 17:39 To: linux clustering Subject: [Linux-cluster] non-RH/Suse systems? hi, i have three ubuntu 9.04 boxes i'd like to cluster. one is running on a DL380 G6 so i have ILO2 on that box. the other two boxes would need to be upgraded to hardware that has fencing. as i'm pretty happy with the DL380 G6 i'd upgrade those two boxes to G6's, but before we spend that cost, we'd like to know if we can cluster ubuntu. anyone using ubuntu? if so, problems? thanks for the info and your time. yvette hirth -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From raju.rajsand at gmail.com Tue Feb 9 01:48:38 2010 From: raju.rajsand at gmail.com (Rajagopal Swaminathan) Date: Tue, 9 Feb 2010 07:18:38 +0530 Subject: [Linux-cluster] fencing node and filesystem In-Reply-To: <266292776.1231251265640574215.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> References: <266292776.1231251265640574215.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> Message-ID: <8786b91c1002081748h4ae71da0h9d0ddd8ae6b6fc0@mail.gmail.com> Greetings, On Mon, Feb 8, 2010 at 8:19 PM, Bob Peterson wrote: > ----- "Muhammad Ammad Shah" wrote: > | when a node(x) fenced by another node(y), will it corrupts the file > | system or no? and if the database was running and node(x) fenced by > | fenced node(y) will it corrupts the database or whats about the > | transaction ? > > With GFS and GFS2 there should be no metadata corruption, in theory. > When quorum is regained (or if it's not lost) the fenced > node's journal is replayed so the file system metadata should > remain intact. ?That does not mean that data won't be lost. > If the node was fenced while it had data in its cache, that data > may be lost. ?In critical situations, you can mark files and/or > directories with a flag that makes the data journaled as well, > so the data should be protected. ?Of course, that has a performance > penalty. ?There's a small section about this in the GFS faq. Whatever Rob says is absolutely correct. Having said that, IIRC DBs like Oracle writes to disks every 3 seconds and Linux flushes to disk every five seconds anyways. So, one can say that the the data loss window could be at most 5 seconds. Am I right? Thanks and regards, Rajagopal From mammadshah at hotmail.com Tue Feb 9 06:04:06 2010 From: mammadshah at hotmail.com (Muhammad Ammad Shah) Date: Tue, 9 Feb 2010 11:04:06 +0500 Subject: [Linux-cluster] Linux-cluster Digest, Vol 70, Issue 9 In-Reply-To: References: Message-ID: Hi Bob, if we are running Database (oracle) on GFS on two node cluster, and if node(x) fenced, so this mean that there will be two problems. The node(x) File system (ext3) /, /var/, /usr,.... can be crashed if the insertion was being running, then its lost. if i am right on this then its big problem to setup power fencing in critical environments. i think it should be some thing different like SAN data path fencing. one more thing i want to know, if (oracle)node was running it was accessing the trace logs, control logs and ..... of oracle, then node(x) fenced and node(y) starts will it read from the old transaction point or it will be something else (i am not expert of oracle) . Thanks, Muhammad Ammad Shah > From: linux-cluster-request at redhat.com > Subject: Linux-cluster Digest, Vol 70, Issue 9 > To: linux-cluster at redhat.com > Date: Mon, 8 Feb 2010 12:00:05 -0500 > > Send Linux-cluster mailing list submissions to > linux-cluster at redhat.com > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.redhat.com/mailman/listinfo/linux-cluster > or, via email, send a message with subject or body 'help' to > linux-cluster-request at redhat.com > > You can reach the person managing the list at > linux-cluster-owner at redhat.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Linux-cluster digest..." > > > Today's Topics: > > 1. Re: fencing node and filesystem (Bob Peterson) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 8 Feb 2010 09:49:34 -0500 (EST) > From: Bob Peterson > To: linux clustering > Subject: Re: [Linux-cluster] fencing node and filesystem > Message-ID: > <266292776.1231251265640574215.JavaMail.root at zmail06.collab.prod.int.phx2.redhat.com> > > Content-Type: text/plain; charset=utf-8 > > ----- "Muhammad Ammad Shah" wrote: > | Hi, > | > | when a node(x) fenced by another node(y), will it corrupts the file > | system or no? and if the database was running and node(x) fenced by > | fenced node(y) will it corrupts the database or whats about the > | transaction ? > | > | > | > | Thanks, > | Muhammad Ammad Shah > > With GFS and GFS2 there should be no metadata corruption, in theory. > When quorum is regained (or if it's not lost) the fenced > node's journal is replayed so the file system metadata should > remain intact. That does not mean that data won't be lost. > If the node was fenced while it had data in its cache, that data > may be lost. In critical situations, you can mark files and/or > directories with a flag that makes the data journaled as well, > so the data should be protected. Of course, that has a performance > penalty. There's a small section about this in the GFS faq. > > Regards, > > Bob Peterson > Red Hat File Systems > > > > ------------------------------ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > End of Linux-cluster Digest, Vol 70, Issue 9 > ******************************************** _________________________________________________________________ Hotmail: Free, trusted and rich email service. https://signup.live.com/signup.aspx?id=60969 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpeterso at redhat.com Tue Feb 9 14:27:29 2010 From: rpeterso at redhat.com (Bob Peterson) Date: Tue, 9 Feb 2010 09:27:29 -0500 (EST) Subject: [Linux-cluster] Linux-cluster Digest, Vol 70, Issue 9 In-Reply-To: Message-ID: <1507008912.1329501265725649354.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> ----- "Muhammad Ammad Shah" wrote: | Hi Bob, | | if we are running Database (oracle) on GFS on two node cluster, and if | node(x) fenced, so this mean that there will be two problems. | | | 1. The node(x) File system (ext3) /, /var/, /usr,.... can be | crashed | 2. if the insertion was being running, then its lost. | | if i am right on this then its big problem to setup power fencing in | critical environments. i think it should be some thing different like | SAN data path fencing. | | one more thing i want to know, if (oracle)node was running it was | accessing the trace logs, control logs and ..... of oracle, then | node(x) fenced and node(y) starts will it read from the old | transaction point or it will be something else (i am not expert of | oracle) . | | | Thanks, | Muhammad Ammad Shah Hi Muhammad, Regarding #1: Each node should have its own root file system separate from the other node. Since ext3 has journals too, it should recover no problem from crashes and being fenced. Rest assured that I have fenced systems hundreds, maybe thousands of times with no file system corruption or data loss for their root ext3 file system. Regarding #2: I personally don't know oracle or how it manages its data. I only know that it uses direct io, so takes on the responsibility for maintaining data integrity. Regarding #3: I'm not sure what you're asking and I'm not an oracle expert. The journal replay feature of GFS should maintain integrity of the file system. But oracle's software has to take care of its own logs and journals within that file system. Regards, Bob Peterson Red Hat File Systems From jose.neto at liber4e.com Tue Feb 9 14:30:36 2010 From: jose.neto at liber4e.com (jose nuno neto) Date: Tue, 9 Feb 2010 14:30:36 -0000 (GMT) Subject: [Linux-cluster] 2node cluster + qdiskd for TieBreak Message-ID: <6b717372e2aee82ef4b27c835d7ad0f1.squirrel@fela.liber4e.com> Hellos Im running cman-2.0.115-1.el5_4.2 rgmanager-2.0.52-1.el5 kernel-2.6.18-164.el5 Trying to setup a 2node cluster that can keep up providing service on cluster network failure. I've tried simple 2node setups but got several 2nodes fenced, so now I'm trying with qdisk running. My problem is that qdisk does not seem to be doing any of kill or stop cman I have one heuristc for checking on IP, if I blocked it for one node, qdisk score comes down but no reboot or kill. If I stop cluster token communication qdisk verifies that the other node is down but doesn't issue a kill, and after 100secs ( post_fail_delay="100" ) cman is doing the fence. What do I ned to set so that qdisk does the kill? If put some pieces of the cluster.conf below Thanks in advance Jose -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From brem.belguebli at gmail.com Tue Feb 9 14:40:47 2010 From: brem.belguebli at gmail.com (brem belguebli) Date: Tue, 9 Feb 2010 15:40:47 +0100 Subject: [Linux-cluster] fencing node and filesystem In-Reply-To: <8786b91c1002081748h4ae71da0h9d0ddd8ae6b6fc0@mail.gmail.com> References: <266292776.1231251265640574215.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> <8786b91c1002081748h4ae71da0h9d0ddd8ae6b6fc0@mail.gmail.com> Message-ID: <29ae894c1002090640qd571fb2t5a8f39df3ab0c2a7@mail.gmail.com> Hi, What you have to look at before making assumptions on whether or not there will be corruption is how the application (in this case Oracle) opens its files (datafiles, redo logs etc...). >From what I remmember, Oracle defaults to O_DSYNC which tells to VFS to bypass the cache and write directly to the physical disks. This can be verifiable with lsof (+fg flag if it exists). Brem 2010/2/9 Rajagopal Swaminathan : > Greetings, > > On Mon, Feb 8, 2010 at 8:19 PM, Bob Peterson wrote: >> ----- "Muhammad Ammad Shah" wrote: >> | when a node(x) fenced by another node(y), will it corrupts the file >> | system or no? and if the database was running and node(x) fenced by >> | fenced node(y) will it corrupts the database or whats about the >> | transaction ? >> >> With GFS and GFS2 there should be no metadata corruption, in theory. >> When quorum is regained (or if it's not lost) the fenced >> node's journal is replayed so the file system metadata should >> remain intact. ?That does not mean that data won't be lost. >> If the node was fenced while it had data in its cache, that data >> may be lost. ?In critical situations, you can mark files and/or >> directories with a flag that makes the data journaled as well, >> so the data should be protected. ?Of course, that has a performance >> penalty. ?There's a small section about this in the GFS faq. > > Whatever Rob says is absolutely correct. > > Having said that, IIRC DBs like Oracle writes to disks every 3 seconds > and Linux flushes to disk every five seconds anyways. > > So, one can say that the the data loss window could be at most 5 > seconds. Am I right? > > Thanks and regards, > > Rajagopal > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From thomas at sjolshagen.net Tue Feb 9 15:01:23 2010 From: thomas at sjolshagen.net (Thomas Sjolshagen) Date: Tue, 9 Feb 2010 10:01:23 -0500 Subject: [Linux-cluster] Linux-cluster Digest, Vol 70, Issue 9 In-Reply-To: <1507008912.1329501265725649354.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> References: <1507008912.1329501265725649354.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> Message-ID: <05ACDEEB-7EF8-46B0-AEDC-5B3D16EC6BB4@sjolshagen.net> On Feb 9, 2010, at 9:27 AM, Bob Peterson wrote: > ----- "Muhammad Ammad Shah" wrote: > | Hi Bob, > | > | if we are running Database (oracle) on GFS on two node cluster, and if > | node(x) fenced, so this mean that there will be two problems. > | > | > | 1. The node(x) File system (ext3) /, /var/, /usr,.... can be > | crashed > | 2. if the insertion was being running, then its lost. > | And that is no different than if the power was lost (and the UPS backup failed). Oracle is designed to address unexpected power loss. That's what the rollback & redo logs (in part) are for (and the use of transactions in the applications, to group logical updates/inserts/deletes so as to ensure the database is in a predictable/expected state following a failed transaction). > | if i am right on this then its big problem to setup power fencing in > | critical environments. i think it should be some thing different like > | SAN data path fencing. > | Not really. SAN data path fencing would result in an IO error (after the DB IO timeout is exceeded) and thus the insert/update/delete fails or is only partially completed. Once again, the Oracle transactional and/or rollback/redo functionality would restore the database files to the point where the transaction register is 'complete'. > | one more thing i want to know, if (oracle)node was running it was > | accessing the trace logs, control logs and ..... of oracle, then > | node(x) fenced and node(y) starts will it read from the old > | transaction point or it will be something else (i am not expert of > | oracle) . All of the above goes out the window if you're hosting your data files on a file system without support for direct-IO or you've disabled directIO (and not enabled synchronous IO) in the init.ora configuration file (by default you have to take positive action to do so for modern releases or the Oracle RDBMS software). From m at alanny.ru Wed Feb 10 18:56:40 2010 From: m at alanny.ru (AlannY) Date: Wed, 10 Feb 2010 21:56:40 +0300 Subject: [Linux-cluster] [clvm] what happens when node is fenced/shutdown Message-ID: <20100210185639.GA2930@alanny-pc.lcl.starlink.ru> Hi there. I'm newbie in clustering, so I have one dummy question... I've create simple 2 node cluster configuration. One block device I've shared from one node to another with iSCSI. Then, I've created clvm volume group of 2 device (shared and local). The same, I've done on 2nd node. lvcreate, create filesystem, mount, no problems. But what will be happen if one node will be fenced or just shutdown. Will it allow to write data to volume group? Will it allow to read from it? Or, it will wait for 2nd node to started up and iSCSI exported device again? So on... Thanks for patience. -- )\._.,--....,'``. /, _.. \ _\ (`._ ,. `._.-(,_..'--(,_..'`-.;.' From dist-list at LEXUM.UMontreal.CA Wed Feb 10 22:31:21 2010 From: dist-list at LEXUM.UMontreal.CA (F M) Date: Wed, 10 Feb 2010 17:31:21 -0500 (EST) Subject: [Linux-cluster] Xen / Redhat cluster : 5.3 to 5.4 upgrade question and best practice In-Reply-To: <459199995.79901265840521327.JavaMail.root@vicenza.dmz.lexum.pri> Message-ID: <733013624.80021265841081441.JavaMail.root@vicenza.dmz.lexum.pri> Hello, I need to upgrade a Xen cluster using RH servers from 5.3 to 5.4. It is a 3 nodes cluster (+qdisk) and 1 luci server for management. All VM are in a GFS2 FS in /home/domU/FQDN folders. Each FQDN folder contains : -The xen config file -the VM FQDN.img file I read about some issues with rgmanager from 5.4 (unable to create VM config files that were not in /etc/xen). This bug has been fixed in rgmanager-2.0.52-1.el5_4.1 Can I safely apply the update without modifying any config made with 5.3 ? Do I need to tweak the cluster.conf between 5.3 and 5.4 ? Also, Is it better to upgrade the luci server before the nodes ? I am also curious about the nodes, what is the best practice : moving VM , removing the node from the cluster, upgrading and then reboot, see if everything is fine and go for the next one after that ? Can I stay with a mix of 5.3 and 5.4 for several days ? Regards, F From pradhanparas at gmail.com Wed Feb 10 22:44:09 2010 From: pradhanparas at gmail.com (Paras pradhan) Date: Wed, 10 Feb 2010 16:44:09 -0600 Subject: [Linux-cluster] Xen / Redhat cluster : 5.3 to 5.4 upgrade question and best practice In-Reply-To: <733013624.80021265841081441.JavaMail.root@vicenza.dmz.lexum.pri> References: <459199995.79901265840521327.JavaMail.root@vicenza.dmz.lexum.pri> <733013624.80021265841081441.JavaMail.root@vicenza.dmz.lexum.pri> Message-ID: <8b711df41002101444g42430182m81604bec535d0e8d@mail.gmail.com> On Wed, Feb 10, 2010 at 4:31 PM, F M wrote: > Hello, > I need to upgrade a Xen cluster using RH servers from 5.3 to 5.4. > > It is a 3 nodes cluster (+qdisk) and 1 luci server for management. > > All VM are in a GFS2 FS in /home/domU/FQDN folders. > > Each FQDN folder contains : > -The xen config file > -the VM FQDN.img file > > I read about some issues with rgmanager from 5.4 (unable to create VM > config files that were not in /etc/xen). This bug has been fixed in > rgmanager-2.0.52-1.el5_4.1 > Can I safely apply the update without modifying any config made with 5.3 ? > Do I need to tweak the cluster.conf between 5.3 and 5.4 ? > I had to add use_virsh="0" in cluster.conf to start vm as a cluster service. Like this: Thanks Paras > Also, > Is it better to upgrade the luci server before the nodes ? > > I am also curious about the nodes, what is the best practice : moving VM , > removing the node from the cluster, upgrading and then reboot, see if > everything is fine and go for the next one after that ? Can I stay with a > mix of 5.3 and 5.4 for several days ? > > Regards, > > F > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.meller at gmx.net Thu Feb 11 14:26:46 2010 From: thomas.meller at gmx.net (Thomas Meller) Date: Thu, 11 Feb 2010 15:26:46 +0100 Subject: [Linux-cluster] Quorum disk over RAID software device In-Reply-To: <4B6CD3AB.8090507@bobich.net> References: <20100204181021.212900@gmx.net> <4B6B1928.2080200@bobich.net> <20100205145624.280470@gmx.net> <4B6C36AF.8080901@bobich.net> <20100206002958.106730@gmx.net> <4B6CD3AB.8090507@bobich.net> Message-ID: <20100211142646.299360@gmx.net> Thanks Gordan! You put me to the right direction. Yes, we are driving an active/passive solution. Each node can see the storage LUNs of both nodes on startup. We sort them in an init script within the initrd (remove and re-add) to get a well-defined order. Then we start multipathing via vpath. Afterwards, we start root-raid1 via MD, then we mount the root fs and go on booting. This is what we did for the last 4 years. Driver-based multipathing was our first trial, before we found the current solution, but our colleagues rebelled. Therefore RDAC/vpath. Makes up for a big fat initrd, though. But ok, techniques evolve, and today dm-multipath is a supported solution. DRBD is unknown land for me. But it's really promising. The drawback is a resync when one node boots and longer resync times. (..if it works as I guess. link to documentation highly welcomed) We will see how this biest works when FC-communication fails on one or both nodes. > > I still have a third opportunity. I can use Qlocig's driver-based > > multipathing and keep using host-based mirroring instead of > > using dm-multipath, which currently prevents me from setting up > > raid-devices as root-fs. > > I'm still not sure how all these relate in your setup. Are you saying > that you are using the qlogic multi-path driver pointing at two > different SANs while the SANs themselves are sorting out the synchronous > real-time mirroring between them? No, both qlogic HBAs do see both Storage controllers via their own SAN. There are two switched SANs, crossovered for fault-tolerance, but else autonomous. (don't ask how) The driver does multipathing over both HBAs, so we see the same controllers/LUNs once instead of twice. But we don't use this anymore. It worked ok, though. Mirroring disk partitions from controllerA to controllerB was always done by MD-raid1. (simple&good) > > > Well, that will work, but is somewhat ugly. > > > > So far, I had only a short glimpse on OSR. I think I will need > > to dive deeper. > > It sounds like you'll need to add support for qlogic multi-path > proprietary stuff to OSR before it'll do exactly what you want, but > other than that, the idea behind it is to enable you have have a shared > rootfs on a suitable cluster file system (GFS, OCFS, GlusterFS, etc.). As far as I understood, no. I need to do multipathing and mirroring. Whether I do it by dm-multipath and MD-raid or by QLA-Multipath and DRBD, or any other combination, doesn't change the result. It only changes the way the whole thing behaves in case of a failure. Actually, I hope to be able to start a DRBD device in the initrd and mount it as root. That is the first hurdle to be taken. The rest is nice-to-have. > It's generally useful when you need a big fat initrd > to bootstrap things such as RHCS components, block device drivers > (e.g. DRBD), or file systems that need a fuller environment to start up > than a normal initrd (e.g. GlusterFS, things that need glibc, etc.). Nice-to have. I am glad to know there is such a thing. When we started, there was nothing. I will report back when I have some results. Thanks for your help! Thomas -- http://www.openstreetmap.org/?lat=47.172&lon=7.4395&zoom=15&layers=B000FTFTT&mlat=47.16677&mlon=7.43513 Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser From mammadshah at hotmail.com Fri Feb 12 07:36:31 2010 From: mammadshah at hotmail.com (Muhammad Ammad Shah) Date: Fri, 12 Feb 2010 12:36:31 +0500 Subject: [Linux-cluster] Active nodes Message-ID: HI, If i create 8 node cluster then how many nodes will be active and how many will be passive, or it depends upon application ? like in Microsoft exchange 2007, if there are 8 nodes in cluster one should be passive and 7 nodes will be performing as active nodes in cluster. Thanks, Muhammad Ammad Shah _________________________________________________________________ Hotmail: Trusted email with Microsoft?s powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969 -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at beekhof.net Fri Feb 12 07:43:48 2010 From: andrew at beekhof.net (Andrew Beekhof) Date: Fri, 12 Feb 2010 08:43:48 +0100 Subject: [Linux-cluster] Active nodes In-Reply-To: References: Message-ID: On Fri, Feb 12, 2010 at 8:36 AM, Muhammad Ammad Shah wrote: > HI, > > If i create 8 node cluster then how many nodes will be active and how many > will be passive, or it depends upon application ? It depends on where you tell the services to run. > like in Microsoft > exchange 2007, if there are 8 nodes in cluster one should be passive and 7 > nodes will be performing as active nodes in cluster. > > > > Thanks, > Muhammad Ammad Shah > > > > > ________________________________ > Hotmail: Trusted email with Microsoft?s powerful SPAM protection. Sign up > now. > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From siedler at hrd-asia.com Sat Feb 13 05:50:23 2010 From: siedler at hrd-asia.com (Wolf Siedler) Date: Sat, 13 Feb 2010 12:50:23 +0700 Subject: [Linux-cluster] Xen / Redhat cluster : 5.3 to 5.4 upgrade question and best practice In-Reply-To: <733013624.80021265841081441.JavaMail.root@vicenza.dmz.lexum.pri> References: <733013624.80021265841081441.JavaMail.root@vicenza.dmz.lexum.pri> Message-ID: <4B763D9F.4000901@hrd-asia.com> > I need to upgrade a Xen cluster using RH servers from 5.3 to 5.4. > > It is a 3 nodes cluster (+qdisk) and 1 luci server for management. > > All VM are in a GFS2 FS in /home/domU/FQDN folders. > > Each FQDN folder contains : > -The xen config file > -the VM FQDN.img file We run an apparently very similar setup, based on CentOS. A difference may well be that /etc/xen contains symbolic links to the VM config files. The files themselves are stored in the same location as the *.img files in the GFS2 file system. > I read about some issues with rgmanager from 5.4 (unable to create VM > config files that were not in /etc/xen). This bug has been fixed in > rgmanager-2.0.52-1.el5_4.1 > Can I safely apply the update without modifying any config made with > 5.3 ? Do I need to tweak the cluster.conf between 5.3 and 5.4 ? As Paras mentioned before, we added in cluster.conf for each VM service use_virsh="0", but we also added max_restarts="0" and restart_expire_time="0". > Also, > Is it better to upgrade the luci server before the nodes ? Did not do that. It is my understanding that you can do everything by modifying cluster.conf directly on the nodes as well. As a precaution, we always increased the version number manually and saved all cluster.conf within a short time interval. > I am also curious about the nodes, what is the best practice : moving > VM , removing the node from the cluster, upgrading and then reboot, > see if everything is fine and go for the next one after that ? Can I > stay with a mix of 5.3 and 5.4 for several days ? After a 5.3 update went all wrong, we had set to set vm autostart="0" and start them from the nodes directly. Luci then showed all VM services as disabled (they were running nevertheless). Our upgrade path went like this: The separate server running Luci was upgraded long time ago (independently). Upgraded each VM by simple "yum update" and shut them down properly from SSH terminal ("poweroff"). Afterward, the corresponding node was upgraded by "yum update" and then rebooted. The VMs on this node were then manually restarted (note that at this point cluster.conf still contained vm autostart="0"). Repeated the procedure with the next node. At this point, we started all VM services from the nodes (not through Luci). Up to here, Luci still considered the VM services as disabled. To re-integrate: Shut them down properly from SSH terminal. Modify each cluster.conf by adding use_virsh="0", max_restarts="0" and restart_expire_time="0". Also changed vm autostart="0" to vm autostart="1". (Do not forget to increase version number at the beginning of cluster.conf.) Then went to Luci and the corresponding VM service was listed afterwards (again) as available. They were not running, though. Afterwards, we used Luci command "Enable" for each VM service. Unfortunately, I don't recall whether the VM was started automatically at this point or if we had to restart it separately from Luci. Anyway, the communication Luci <-> cluster worked again. Even live migration (as ordered through Luci) worked flawlessly. Our update 5.3 -> 5.4 was done about a week ago, so it seemed to me as if the present state of CentOS 5.4 is stable and performing well. Final remarks: I understood from various sources that it seems to be advisable to have nodes and VMs running at the same OS level, at least at the same kernel version. Luci page "Storage" still fails. Seems to be a timeout problem. We came to consider Luci as a nice tool, but found it very reassuring to know how to perform the corresponding actions also directly from terminal on the nodes (i. e. by "xm ..." commands). Regards, Wolf From Martin.Waite at datacash.com Sat Feb 13 06:14:12 2010 From: Martin.Waite at datacash.com (Martin Waite) Date: Sat, 13 Feb 2010 06:14:12 -0000 Subject: [Linux-cluster] N+M support ? Message-ID: Hi, Suppose I have 3 services running on 5 nodes. Each node can run only 1 service, 2 nodes are reserved for failover. It is easy to configure rgmanager to cope with the first service node failure by including the 2 failover nodes in the failover domain for each service. However, is it possible to configure rgmanager such that on a second failure, only the failover node that is not currently running a service is considered for use ? Further, that if a third failure occurs, the affected service is not migrated at all ? Also, is it possible to rank the services such that if the failover nodes are occupied by low ranking services and the node running a higher ranking service fails, that the lowest ranking service is evicted so that the higher ranking service can be failed over ? regards, Martin From Martin.Waite at datacash.com Sat Feb 13 06:46:21 2010 From: Martin.Waite at datacash.com (Martin Waite) Date: Sat, 13 Feb 2010 06:46:21 -0000 Subject: [Linux-cluster] rhcs product management Message-ID: Hi, I have a few questions about how the RHCS product / project is managed. Is paid-for technical support available ? Are consultants available for hire ? What are the goals of the product ? How are these decided ? Is there a road map ? Are any Red Hat staff dedicated to this product ? If so, how many ? There is a surprising lack of this sort of sales information available - at least none that I have found outside of the scant RHEL AP documents. regards, Martin From sdake at redhat.com Sat Feb 13 07:10:07 2010 From: sdake at redhat.com (Steven Dake) Date: Sat, 13 Feb 2010 00:10:07 -0700 Subject: [Linux-cluster] rhcs product management In-Reply-To: References: Message-ID: <1266045007.2905.114.camel@localhost.localdomain> On Sat, 2010-02-13 at 06:46 +0000, Martin Waite wrote: > > I have a few questions about how the RHCS product / project is > managed. > > Is paid-for technical support available ? Are consultants available > for hire ? > > What are the goals of the product ? How are these decided ? > > Is there a road map ? > > Are any Red Hat staff dedicated to this product ? If so, how many ? > > There is a surprising lack of this sort of sales information available > - at least none that I have found outside of the scant RHEL AP > documents. > Martin, This list is for the discussion of community related issues for those people using cluster2/3/4 and gfs2 rather then company specific product issues. Company specific product issues should be directed at your vendor of choice. Distros are encouraged to adopt the upstream sources generated by this project to their liking and sell/support/give away however they choose. This is a community project first and foremost. The goals of the cluster project are to deliver a best of breed community driven cluster stack. The project objectives are decided by the community based upon typical open source process - write patches, submit them, they are merged, project increases quality/features, repeat. Influence over the project direction is directly related to effort done by contributors, as is typical with any type of community activity. Roadmaps are difficult to generate for community driven projects, because it always depends on what community members want to work on. There is no centralized "command and control" as is typical with proprietary software development. Regarding your business-related support questions, many distros have adopted the software provided by this project. If your looking for commercial support and consulting, I suggest you speak to your software vendor of choice - they likely have options for you. Regards -steve From brettcave at gmail.com Sat Feb 13 08:44:35 2010 From: brettcave at gmail.com (Brett Cave) Date: Sat, 13 Feb 2010 10:44:35 +0200 Subject: [Linux-cluster] shutdown (OS) of all GFS nodes Message-ID: hi, I have a GFS cluster (4 node + qdisk on SAN), and have problems shutting down cman service / unmount gfs mountpoints - it causes the shutdown to hang. I am running GFS & CLVM (lv's are xen guest drives). If i try and shut down cman service manually, i get an error that resources are still in use. 1 gfs directory is exported via NFS. I think it may be because of service stop order, specifically openais stopping before cman - could this be a valid reason? Init6 levels are: K00xendomains K01xend K03libvirtd K20nfs K20openais K74gfs K74gfs2 K76clvmd K78qdiskd K79cman K86nfslock If I manually run through the stopping of these services, gfs service hangs. This is the log: Feb 13 10:35:25 vmhost-01 gfs_controld[3227]: cluster is down, exiting Feb 13 10:35:25 vmhost-01 dlm_controld[3221]: cluster is down, exiting Feb 13 10:35:25 vmhost-01 fenced[3215]: cluster is down, exiting Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 4 Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 3 Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 2 Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 1 Feb 13 10:35:27 vmhost-01 qdiskd[3201]: cman_dispatch: Host is down Feb 13 10:35:27 vmhost-01 qdiskd[3201]: Halting qdisk operations Feb 13 10:35:51 vmhost-01 ccsd[3165]: Unable to connect to cluster infrastructure after 30 seconds. Feb 13 10:36:13 vmhost-01 mountd[3927]: Caught signal 15, un-registering and exiting. Feb 13 10:36:13 vmhost-01 kernel: nfsd: last server has exited Feb 13 10:36:13 vmhost-01 kernel: nfsd: unexporting all filesystems Feb 13 10:36:21 vmhost-01 ccsd[3165]: Unable to connect to cluster infrastructure after 60 seconds. ccsd continues to repeat the last message, increasing time: 60s, 90s, 120s, 180s, 210s, etc dmesg shows: dlm: closing connection to node 4 dlm: closing connection to node 3 dlm: closing connection to node 2 dlm: closing connection to node 1 There are no open files on GFS (from lsof) I am using gfs (1). The only workaround I have now is to reset the nodes via ILO once the shutdown process starts (and hangs on either gfs or cman service stop). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mammadshah at hotmail.com Sat Feb 13 10:34:28 2010 From: mammadshah at hotmail.com (Muhammad Ammad Shah) Date: Sat, 13 Feb 2010 15:34:28 +0500 Subject: [Linux-cluster] Active nodes In-Reply-To: References: Message-ID: Is there any way to tell the service to run on multiple nodes ? Thanks, Muhammad Ammad Shah > > HI, > > If i create 8 node cluster then how many nodes will be active and how many will be passive, or it depends upon application ? like in Microsoft exchange 2007, if there are 8 nodes in cluster one should be passive and 7 nodes will be performing as active nodes in cluster. > > > > Thanks, > Muhammad Ammad Shah > > > > > _________________________________________________________________ > Hotmail: Trusted email with Microsoft?s powerful SPAM protection. > https://signup.live.com/signup.aspx?id=60969 > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 2 > Date: Fri, 12 Feb 2010 08:43:48 +0100 > From: Andrew Beekhof > To: linux clustering > Subject: Re: [Linux-cluster] Active nodes > Message-ID: > > Content-Type: text/plain; charset=windows-1252 > > On Fri, Feb 12, 2010 at 8:36 AM, Muhammad Ammad Shah > wrote: > > HI, > > > > If i create 8 node cluster then how many nodes will be active and how many > > will be passive, or it depends upon application ? > > It depends on where you tell the services to run. > > > like in Microsoft > > exchange 2007, if there are 8 nodes in cluster one should be passive and 7 > > nodes will be performing as active nodes in cluster. > > > > > > > > Thanks, > > Muhammad Ammad Shah > > > > > > > > > > ________________________________ > > Hotmail: Trusted email with Microsoft?s powerful SPAM protection. Sign up > > now. > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > ------------------------------ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > End of Linux-cluster Digest, Vol 70, Issue 12 > ********************************************* _________________________________________________________________ Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. https://signup.live.com/signup.aspx?id=60969 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmicmirregs at gmail.com Sat Feb 13 14:01:20 2010 From: rmicmirregs at gmail.com (Rafael =?ISO-8859-1?Q?Mic=F3?= Miranda) Date: Sat, 13 Feb 2010 15:01:20 +0100 Subject: [Linux-cluster] N+M support ? In-Reply-To: References: Message-ID: <1266069680.6582.34.camel@mecatol> Hi Martin, Al your questions point to an advanced configuration using Failover Domains and Service propierties. El s?b, 13-02-2010 a las 06:14 +0000, Martin Waite escribi?: > Hi, > > Suppose I have 3 services running on 5 nodes. Each node can run only 1 service, 2 nodes are reserved for failover. Lets name the services S1, S2, S3, and the nodes N1, N2, N3 and N4, N5 for the failover ones. > > It is easy to configure rgmanager to cope with the first service node failure by including the 2 failover nodes in the failover domain for each service. Yes, you can configure it in the following way (name the failover domains F1, F2, F3) F1: service S1, nodes N1, N4, N5 F2: service S2, nodes N2, N4, N5 F3: service S3, nodes N3, N4, N5 You'll need to set the failover domains with the properties "restricted" and "ordered" for this. There is another property, "auto_failback", that will be of your interest. Keep it in mind. > > However, is it possible to configure rgmanager such that on a second failure, only the failover node that is not currently running a service is considered for use ? Yes. Services have the "run exclusive" option. That property will only allow a service to be run on a node that has no running services. With this option on, if N1 fails it will fail over the N4. After that if N2 fails the service S2 will be migrated to N5, and not to N4. > > Further, that if a third failure occurs, the affected service is not migrated at all ? Yes. If you have set the "run exclusive" option on the three services the service will not be migrated to any node according to: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Cluster_Administration/s1-add-service-CA.html > > Also, is it possible to rank the services such that if the failover nodes are occupied by low ranking services and the node running a higher ranking service fails, that the lowest ranking service is evicted so that the higher ranking service can be failed over ? > As far as I know, there is no direct function to do what you ask. You can play with all this options to get something similar to what you need. For example I propose you this configuration: Nodes: N1, N2, N3: service nodes N4, N5: failover nodes Services: S1: high ranking service. "Run exclusive" on. S2, S3: low ranking services. "Run exclusive" off. Fail Over Domains: F1: service S1. Nodes N1, N4, N5. "Restricted" on. "Ordered on". "Auto_failback" off. F2: service S2. Nodes N2, N5, N4. "Restricted" on. "Ordered on". "Auto_failback" on. F3: service S3. Nodes N3, N5, N4. "Restricted" on. "Ordered on". "Auto_failback" on. With this configuration you only penalise services S2 and S3 in case of N2 and N3 failure because both services will run on the same node, but you keep N4 free for your hing ranking S1 service. With "Auto_failback" on on faiolver domains F2 and F3 you will automatically migrate S2 or S3 back to its preferred node when they come back alive, so penalisation will be shorter in time. Remember that you are talking about a failure of up to 3 nodes in a cluster of 5 members. Maybe there is no sense in this because depending on the configuration given you can even lose Quorum before achieving this situation. > regards, > Martin > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Cheers, Rafael -- Rafael Mic? Miranda From kkovachev at varna.net Sat Feb 13 15:16:53 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Sat, 13 Feb 2010 17:16:53 +0200 Subject: [Linux-cluster] independent subtree Message-ID: <20100213145541.M89820@varna.net> Hello, when __independent_subtree is used and the resource fails the services is not relocated after max_restarts. Is this a bug or is by design? Example: the idea here is if MySQL have crashed mysqld_safe will restart it, but if there is a problem with Apache - to restart it without stopping MySQL. This works OK, but if there is a permanent problem with Apache (no access to the file system) it is restarted several times without the service being relocated to another node From Martin.Waite at datacash.com Sat Feb 13 15:45:13 2010 From: Martin.Waite at datacash.com (Martin Waite) Date: Sat, 13 Feb 2010 15:45:13 -0000 Subject: [Linux-cluster] rhcs product management References: <1266045007.2905.114.camel@localhost.localdomain> Message-ID: On 13 February 2010 07:10, Steven Dake wrote: > > On Sat, 2010-02-13 at 06:46 +0000, Martin Waite wrote: > > > > I have a few questions about how the RHCS product / project is > > managed. > > > > Is paid-for technical support available ? Are consultants available > > for hire ? > > > > What are the goals of the product ? How are these decided ? > > > > Is there a road map ? > > > > Are any Red Hat staff dedicated to this product ? If so, how many ? > > > > There is a surprising lack of this sort of sales information available > > - at least none that I have found outside of the scant RHEL AP > > documents. > > > > Martin, > > This list is for the discussion of community related issues for those > people using cluster2/3/4 and gfs2 rather then company specific product > issues. Company specific product issues should be directed at your > vendor of choice. Distros are encouraged to adopt the upstream sources > generated by this project to their liking and sell/support/give away > however they choose. This is a community project first and foremost. > > The goals of the cluster project are to deliver a best of breed > community driven cluster stack. The project objectives are decided by > the community based upon typical open source process - write patches, > submit them, they are merged, project increases quality/features, > repeat. Influence over the project direction is directly related to > effort done by contributors, as is typical with any type of community > activity. > > Roadmaps are difficult to generate for community driven projects, > because it always depends on what community members want to work on. > There is no centralized "command and control" as is typical with > proprietary software development. > > Regarding your business-related support questions, many distros have > adopted the software provided by this project. If your looking for > commercial support and consulting, I suggest you speak to your software > vendor of choice - they likely have options for you. > > Regards > -steve > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Hi Steve, I got the impression that this was a Red Hat product given that the mailing list, wiki, and documentation are all hosted and administered by Red Hat, and given that most commits to the source tree are from what I take to be employees of Red Hat. My only other experience of a community project is MySQL, which is very much a company product first and community project second. Please excuse my ignorance. It is clear that the source code is all available under the fedora Git repository, but the area I am struggling with is the documentation. Are the documents open source,or are these proprietry? Is it possible to contribute to the documentation set and the wiki ? Is there an SCM for these documents ? I would like to contribute to the project and I feel that a good way would be to help with documentation. For example, a recent question I asked (today,in fact) was resolved by a feature of failover domains that is present in the GUI documentation ("run exclusive") - but is nowhere to be found in the XML schema documentation. http://sources.redhat.com/cluster/doc/cluster_schema_rhel5.html In fact, that document seems to run out of steam in the final section. I want to know if it is worthwhile for me to update this document based on the most recent release. This would certainly be a useful exercise for me. regards, Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 7032 bytes Desc: not available URL: From Martin.Waite at datacash.com Sat Feb 13 15:55:29 2010 From: Martin.Waite at datacash.com (Martin Waite) Date: Sat, 13 Feb 2010 15:55:29 -0000 Subject: [Linux-cluster] N+M support ? References: <1266069680.6582.34.camel@mecatol> Message-ID: On 13 February 2010 14:01, Rafael Mic? Miranda wrote: > Hi Martin, > > Al your questions point to an advanced configuration using Failover > Domains and Service propierties. > > El s?b, 13-02-2010 a las 06:14 +0000, Martin Waite escribi?: >> Hi, >> >> Suppose I have 3 services running on 5 nodes. Each node can run only 1 service, 2 nodes are reserved for failover. > > Lets name the services S1, S2, S3, and the nodes N1, N2, N3 and N4, N5 > for the failover ones. > >> >> It is easy to configure rgmanager to cope with the first service node failure by including the 2 failover nodes in the failover domain for each service. > > Yes, you can configure it in the following way (name the failover > domains F1, F2, F3) > > F1: service S1, nodes N1, N4, N5 > F2: service S2, nodes N2, N4, N5 > F3: service S3, nodes N3, N4, N5 > > You'll need to set the failover domains with the properties "restricted" > and "ordered" for this. There is another property, "auto_failback", that > will be of your interest. Keep it in mind. > >> >> However, is it possible to configure rgmanager such that on a second failure, only the failover node that is not currently running a service is considered for use ? > > Yes. Services have the "run exclusive" option. That property will only > allow a service to be run on a node that has no running services. With > this option on, if N1 fails it will fail over the N4. After that if N2 > fails the service S2 will be migrated to N5, and not to N4. > >> >> Further, that if a third failure occurs, the affected service is not migrated at all ? > > Yes. If you have set the "run exclusive" option on the three services > the service will not be migrated to any node according to: > > http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Cluster_Administration/s1-add-service-CA.html > > >> >> Also, is it possible to rank the services such that if the failover nodes are occupied by low ranking services and the node running a higher ranking service fails, that the lowest ranking service is evicted so that the higher ranking service can be failed over ? >> > > As far as I know, there is no direct function to do what you ask. > > You can play with all this options to get something similar to what you > need. For example I propose you this configuration: > > Nodes: > > N1, N2, N3: service nodes > N4, N5: failover nodes > > Services: > > S1: high ranking service. "Run exclusive" on. > S2, S3: low ranking services. "Run exclusive" off. > > Fail Over Domains: > > F1: service S1. Nodes N1, N4, N5. "Restricted" on. "Ordered on". > "Auto_failback" off. > F2: service S2. Nodes N2, N5, N4. "Restricted" on. "Ordered on". > "Auto_failback" on. > F3: service S3. Nodes N3, N5, N4. "Restricted" on. "Ordered on". > "Auto_failback" on. > > With this configuration you only penalise services S2 and S3 in case of > N2 and N3 failure because both services will run on the same node, but > you keep N4 free for your hing ranking S1 service. With "Auto_failback" > on on faiolver domains F2 and F3 you will automatically migrate S2 or S3 > back to its preferred node when they come back alive, so penalisation > will be shorter in time. > > > Remember that you are talking about a failure of up to 3 nodes in a > cluster of 5 members. Maybe there is no sense in this because depending > on the configuration given you can even lose Quorum before achieving > this situation. > > >> regards, >> Martin >> >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > Cheers, > > Rafael > > -- > Rafael Mic? Miranda > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Hi Rafael, Thank you very much for the information. The "run exclusive" option does appear to do what I need. As for the service ranking and eviction scenario - you are correct that this only becomes necessary in the event of multiple failures, and perhaps I don't need to go that far. The easiest solution is to increase the "M" in the "N+M" set of nodes: I can survive one failure where M=1, two where M=2, etc. If more nodes fail than can be failed over, you are entering DR territory. regards, Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 7370 bytes Desc: not available URL: From andrew at beekhof.net Sat Feb 13 18:07:20 2010 From: andrew at beekhof.net (Andrew Beekhof) Date: Sat, 13 Feb 2010 19:07:20 +0100 Subject: [Linux-cluster] Active nodes In-Reply-To: References: Message-ID: On Sat, Feb 13, 2010 at 11:34 AM, Muhammad Ammad Shah wrote: > > Is there any way to tell the service to run on multiple nodes ? If you use Pacemaker, you can use clones. http://www.clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/s-resource-clone.html From cthulhucalling at gmail.com Sat Feb 13 18:17:56 2010 From: cthulhucalling at gmail.com (Ian Hayes) Date: Sat, 13 Feb 2010 10:17:56 -0800 Subject: [Linux-cluster] shutdown (OS) of all GFS nodes In-Reply-To: References: Message-ID: <36df569a1002131017j8f31232m22f71781528c418e@mail.gmail.com> I see things like this often if the gfs volume isn't unmounted before attempting to shut down the cluster daemons. On Feb 13, 2010 12:52 AM, "Brett Cave" wrote: hi, I have a GFS cluster (4 node + qdisk on SAN), and have problems shutting down cman service / unmount gfs mountpoints - it causes the shutdown to hang. I am running GFS & CLVM (lv's are xen guest drives). If i try and shut down cman service manually, i get an error that resources are still in use. 1 gfs directory is exported via NFS. I think it may be because of service stop order, specifically openais stopping before cman - could this be a valid reason? Init6 levels are: K00xendomains K01xend K03libvirtd K20nfs K20openais K74gfs K74gfs2 K76clvmd K78qdiskd K79cman K86nfslock If I manually run through the stopping of these services, gfs service hangs. This is the log: Feb 13 10:35:25 vmhost-01 gfs_controld[3227]: cluster is down, exiting Feb 13 10:35:25 vmhost-01 dlm_controld[3221]: cluster is down, exiting Feb 13 10:35:25 vmhost-01 fenced[3215]: cluster is down, exiting Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 4 Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 3 Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 2 Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 1 Feb 13 10:35:27 vmhost-01 qdiskd[3201]: cman_dispatch: Host is down Feb 13 10:35:27 vmhost-01 qdiskd[3201]: Halting qdisk operations Feb 13 10:35:51 vmhost-01 ccsd[3165]: Unable to connect to cluster infrastructure after 30 seconds. Feb 13 10:36:13 vmhost-01 mountd[3927]: Caught signal 15, un-registering and exiting. Feb 13 10:36:13 vmhost-01 kernel: nfsd: last server has exited Feb 13 10:36:13 vmhost-01 kernel: nfsd: unexporting all filesystems Feb 13 10:36:21 vmhost-01 ccsd[3165]: Unable to connect to cluster infrastructure after 60 seconds. ccsd continues to repeat the last message, increasing time: 60s, 90s, 120s, 180s, 210s, etc dmesg shows: dlm: closing connection to node 4 dlm: closing connection to node 3 dlm: closing connection to node 2 dlm: closing connection to node 1 There are no open files on GFS (from lsof) I am using gfs (1). The only workaround I have now is to reset the nodes via ILO once the shutdown process starts (and hangs on either gfs or cman service stop). -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From dhoffutt at gmail.com Sun Feb 14 16:11:28 2010 From: dhoffutt at gmail.com (Dustin Henry Offutt) Date: Sun, 14 Feb 2010 10:11:28 -0600 Subject: [Linux-cluster] Active nodes In-Reply-To: References: Message-ID: Could create multiple instances of the service using Failover Domains . On Sat, Feb 13, 2010 at 4:34 AM, Muhammad Ammad Shah wrote: > > Is there any way to tell the service to run on multiple nodes ? > > > Thanks, > Muhammad Ammad Shah > > > > > > > > HI, > > > > If i create 8 node cluster then how many nodes will be active and how > many will be passive, or it depends upon application ? like in Microsoft > exchange 2007, if there are 8 nodes in cluster one should be passive and 7 > nodes will be performing as active nodes in cluster. > > > > > > > > Thanks, > > Muhammad Ammad Shah > > > > > > > > > > _________________________________________________________________ > > Hotmail: Trusted email with Microsoft?s powerful SPAM protection. > > https://signup.live.com/signup.aspx?id=60969 > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > https://www.redhat.com/archives/linux-cluster/attachments/20100212/17299100/attachment.html > > > > > > ------------------------------ > > > > Message: 2 > > Date: Fri, 12 Feb 2010 08:43:48 +0100 > > From: Andrew Beekhof > > To: linux clustering > > Subject: Re: [Linux-cluster] Active nodes > > Message-ID: > > > > Content-Type: text/plain; charset=windows-1252 > > > > > On Fri, Feb 12, 2010 at 8:36 AM, Muhammad Ammad Shah > > wrote: > > > HI, > > > > > > If i create 8 node cluster then how many nodes will be active and how > many > > > will be passive, or it depends upon application ? > > > > It depends on where you tell the services to run. > > > > > like in Microsoft > > > exchange 2007, if there are 8 nodes in cluster one should be passive > and 7 > > > nodes will be performing as active nodes in cluster. > > > > > > > > > > > > Thanks, > > > Muhammad Ammad Shah > > > > > > > > > > > > > > > ________________________________ > > > Hotmail: Trusted email with Microsoft?s powerful SPAM protection. Sign > up > > > now. > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > > > > > > ------------------------------ > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > End of Linux-cluster Digest, Vol 70, Issue 12 > > ********************************************* > > ------------------------------ > Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. Sign up > now. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From extmaillist at linuxbox.cz Sun Feb 14 17:52:44 2010 From: extmaillist at linuxbox.cz (Nikola Ciprich) Date: Sun, 14 Feb 2010 18:52:44 +0100 Subject: [Linux-cluster] clvmd: BUG: lock held when returning to user space! Message-ID: <20100214175244.GB4466@nik-comp.linuxbox.cz> Hi, after each reboot of my cluster, I always see following warning while starting clvmd: [ 458.450539] ================================================ [ 458.458061] [ BUG: lock held when returning to user space! ] [ 458.463907] ------------------------------------------------ [ 458.469738] clvmd/7639 is leaving the kernel with locks still held! [ 458.476189] 1 lock held by clvmd/7639: [ 458.480086] #0: (&ls->ls_in_recovery){......}, at: [] dlm_new_lockspace+0x864/0xb80 [dlm] The problem seems to be here for quite a long time (sorry for not reporting sooner, since it doesn't seem to cause any real trouble, I never found time to report it, but I'm finally doing it now), currently I see it on recent 2.6.32 and 2.6.33 kernels, using lvm2-cluster-2.02.53. The system is x86_64 centos5, with self-compiled kernel, lvm, dm and cluster related stuff. If anybody would like to have look at the problem, I'll be glad to provide any help I can. thanks a lot in advance nik -- ------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis at linuxbox.cz ------------------------------------- From brettcave at gmail.com Mon Feb 15 07:10:28 2010 From: brettcave at gmail.com (Brett Cave) Date: Mon, 15 Feb 2010 09:10:28 +0200 Subject: [Linux-cluster] shutdown (OS) of all GFS nodes In-Reply-To: <36df569a1002131017j8f31232m22f71781528c418e@mail.gmail.com> References: <36df569a1002131017j8f31232m22f71781528c418e@mail.gmail.com> Message-ID: K74gfs = shutdown GFS service = unmount gfs volumes. Specifically, wouldn't that openais service cause issues with cman / gfs, and having openais shut down will result in gfs volumes not being able to unmount? i ran some service shutdowns manually over the weekend, and it seems that the following process works: 1) stop all services that have open files on gfs volumes or use clvm volumes (e.g. cutstom app referencing files, xendomains using cluster LV's) 2) service gfs stop (unmount gfs volumes) 3) service clvmd stop 4) service cman stop 5) init 6 this works without hanging. Definitely an indication that there maybe incorrect ordering of shutdown. anyone else see this happening? On Sat, Feb 13, 2010 at 8:17 PM, Ian Hayes wrote: > I see things like this often if the gfs volume isn't unmounted before > attempting to shut down the cluster daemons. > > On Feb 13, 2010 12:52 AM, "Brett Cave" wrote: > > hi, > > I have a GFS cluster (4 node + qdisk on SAN), and have problems shutting > down cman service / unmount gfs mountpoints - it causes the shutdown to > hang. I am running GFS & CLVM (lv's are xen guest drives). If i try and > shut down cman service manually, i get an error that resources are still in > use. 1 gfs directory is exported via NFS. > > I think it may be because of service stop order, specifically openais > stopping before cman - could this be a valid reason? > > Init6 levels are: > K00xendomains > K01xend > K03libvirtd > K20nfs > K20openais > K74gfs > K74gfs2 > K76clvmd > K78qdiskd > K79cman > K86nfslock > > > If I manually run through the stopping of these services, gfs service > hangs. This is the log: > Feb 13 10:35:25 vmhost-01 gfs_controld[3227]: cluster is down, exiting > Feb 13 10:35:25 vmhost-01 dlm_controld[3221]: cluster is down, exiting > Feb 13 10:35:25 vmhost-01 fenced[3215]: cluster is down, exiting > Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 4 > Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 3 > Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 2 > Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 1 > Feb 13 10:35:27 vmhost-01 qdiskd[3201]: cman_dispatch: Host is down > Feb 13 10:35:27 vmhost-01 qdiskd[3201]: Halting qdisk operations > Feb 13 10:35:51 vmhost-01 ccsd[3165]: Unable to connect to cluster > infrastructure after 30 seconds. > Feb 13 10:36:13 vmhost-01 mountd[3927]: Caught signal 15, un-registering > and exiting. > Feb 13 10:36:13 vmhost-01 kernel: nfsd: last server has exited > Feb 13 10:36:13 vmhost-01 kernel: nfsd: unexporting all filesystems > Feb 13 10:36:21 vmhost-01 ccsd[3165]: Unable to connect to cluster > infrastructure after 60 seconds. > > ccsd continues to repeat the last message, increasing time: 60s, 90s, 120s, > 180s, 210s, etc > > dmesg shows: > dlm: closing connection to node 4 > dlm: closing connection to node 3 > dlm: closing connection to node 2 > dlm: closing connection to node 1 > > There are no open files on GFS (from lsof) > > I am using gfs (1). > > The only workaround I have now is to reset the nodes via ILO once the > shutdown process starts (and hangs on either gfs or cman service stop). > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mgrac at redhat.com Mon Feb 15 15:45:34 2010 From: mgrac at redhat.com (Marek Grac) Date: Mon, 15 Feb 2010 16:45:34 +0100 Subject: [Linux-cluster] trivial fix In-Reply-To: <20100203121907.M46818@varna.net> References: <20100203121907.M46818@varna.net> Message-ID: <4B796C1E.7000802@redhat.com> On 02/03/2010 01:23 PM, Kaloyan Kovachev wrote: > Hi, > if the pid file contains more than 1 line (like sendmail) the > status_check_pid function returns an error. The attached patch replaces it with > 'read pid' like it is done for stop_generic > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Hi, thanks for a patch. I just added a test for empty pid file. Feel free to test it, I will add it in main tree in few days. m, diff --git a/rgmanager/src/resources/utils/ra-skelet.sh b/rgmanager/src/resources/utils/ra-skelet.sh index e892656..5baaaa9 100644 --- a/rgmanager/src/resources/utils/ra-skelet.sh +++ b/rgmanager/src/resources/utils/ra-skelet.sh @@ -14,7 +14,13 @@ status_check_pid() return $OCF_ERR_GENERIC fi - if [ ! -d /proc/`cat "$pid_file"` ]; then + read pid < "$pid_file" + + if [ -z "$pid" ]; then + return $OCF_ERR_GENERIC + fi + + if [ ! -d /proc/$pid ]; then return $OCF_ERR_GENERIC fi -------------- next part -------------- An HTML attachment was scrubbed... URL: From dist-list at LEXUM.UMontreal.CA Mon Feb 15 18:13:01 2010 From: dist-list at LEXUM.UMontreal.CA (F M) Date: Mon, 15 Feb 2010 13:13:01 -0500 (EST) Subject: [Linux-cluster] Xen / Redhat cluster : 5.3 to 5.4 upgrade question and best practice In-Reply-To: <4B763D9F.4000901@hrd-asia.com> Message-ID: <1431512389.92691266257581025.JavaMail.root@vicenza.dmz.lexum.pri> Tx a lot for your input. symlink seems to be a better long term solution. One last question, if I directly modified cluster.conf can I still use lucci after that or will it replace my custom edited cluster.conf? Once again , tx a lot for this great info. ----- Original Message ----- From: "Wolf Siedler" To: "linux clustering" Sent: Saturday, February 13, 2010 12:50:23 AM Subject: Re: [Linux-cluster] Xen / Redhat cluster : 5.3 to 5.4 upgrade question and best practice > I need to upgrade a Xen cluster using RH servers from 5.3 to 5.4. > > It is a 3 nodes cluster (+qdisk) and 1 luci server for management. > > All VM are in a GFS2 FS in /home/domU/FQDN folders. > > Each FQDN folder contains : > -The xen config file > -the VM FQDN.img file We run an apparently very similar setup, based on CentOS. A difference may well be that /etc/xen contains symbolic links to the VM config files. The files themselves are stored in the same location as the *.img files in the GFS2 file system. > I read about some issues with rgmanager from 5.4 (unable to create VM > config files that were not in /etc/xen). This bug has been fixed in > rgmanager-2.0.52-1.el5_4.1 > Can I safely apply the update without modifying any config made with > 5.3 ? Do I need to tweak the cluster.conf between 5.3 and 5.4 ? As Paras mentioned before, we added in cluster.conf for each VM service use_virsh="0", but we also added max_restarts="0" and restart_expire_time="0". > Also, > Is it better to upgrade the luci server before the nodes ? Did not do that. It is my understanding that you can do everything by modifying cluster.conf directly on the nodes as well. As a precaution, we always increased the version number manually and saved all cluster.conf within a short time interval. > I am also curious about the nodes, what is the best practice : moving > VM , removing the node from the cluster, upgrading and then reboot, > see if everything is fine and go for the next one after that ? Can I > stay with a mix of 5.3 and 5.4 for several days ? After a 5.3 update went all wrong, we had set to set vm autostart="0" and start them from the nodes directly. Luci then showed all VM services as disabled (they were running nevertheless). Our upgrade path went like this: The separate server running Luci was upgraded long time ago (independently). Upgraded each VM by simple "yum update" and shut them down properly from SSH terminal ("poweroff"). Afterward, the corresponding node was upgraded by "yum update" and then rebooted. The VMs on this node were then manually restarted (note that at this point cluster.conf still contained vm autostart="0"). Repeated the procedure with the next node. At this point, we started all VM services from the nodes (not through Luci). Up to here, Luci still considered the VM services as disabled. To re-integrate: Shut them down properly from SSH terminal. Modify each cluster.conf by adding use_virsh="0", max_restarts="0" and restart_expire_time="0". Also changed vm autostart="0" to vm autostart="1". (Do not forget to increase version number at the beginning of cluster.conf.) Then went to Luci and the corresponding VM service was listed afterwards (again) as available. They were not running, though. Afterwards, we used Luci command "Enable" for each VM service. Unfortunately, I don't recall whether the VM was started automatically at this point or if we had to restart it separately from Luci. Anyway, the communication Luci <-> cluster worked again. Even live migration (as ordered through Luci) worked flawlessly. Our update 5.3 -> 5.4 was done about a week ago, so it seemed to me as if the present state of CentOS 5.4 is stable and performing well. Final remarks: I understood from various sources that it seems to be advisable to have nodes and VMs running at the same OS level, at least at the same kernel version. Luci page "Storage" still fails. Seems to be a timeout problem. We came to consider Luci as a nice tool, but found it very reassuring to know how to perform the corresponding actions also directly from terminal on the nodes (i. e. by "xm ..." commands). Regards, Wolf -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From msalced at pucp.edu.pe Mon Feb 15 18:20:35 2010 From: msalced at pucp.edu.pe (Mario Salcedo) Date: Mon, 15 Feb 2010 13:20:35 -0500 Subject: [Linux-cluster] GFS2 and lvm Message-ID: <22344e331002151020o21ca6cf0naa1899ab1bb1cf9f@mail.gmail.com> Hi, I am configuring a Red hat cluster. In the guide say: "A GFS2 file system must be created on an LVM logical volume that is a linear or mirrored volume". Is that true. I create GFS2 without LVM. I think all ok. From pcs.fagnonr at pcsb.org Mon Feb 15 20:40:20 2010 From: pcs.fagnonr at pcsb.org (Fagnon Raymond) Date: Mon, 15 Feb 2010 15:40:20 -0500 Subject: [Linux-cluster] GFS and Cluster Message-ID: I have a hypothetical question. Will GFS work without the cluster software active? Here is what I would like to do. I have a two node nfs cluster. Currently the cluster is set up as active standby. What I propose is to shutdown the cluster software leaving GFS in place. Have both nodes NFS out /datadir. Put the virtual ip on an f5 load balancer and have NFS load balanced between the two nodes. This will allow me to leverage both servers instead of leaving one idle. Is this possible to accomplish. My understanding is that gfs is used to allow two nodes access to one file system whether they are running cluster software or not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jos at xos.nl Mon Feb 15 20:42:06 2010 From: jos at xos.nl (Jos Vos) Date: Mon, 15 Feb 2010 21:42:06 +0100 Subject: [Linux-cluster] GFS2 and lvm In-Reply-To: <22344e331002151020o21ca6cf0naa1899ab1bb1cf9f@mail.gmail.com> References: <22344e331002151020o21ca6cf0naa1899ab1bb1cf9f@mail.gmail.com> Message-ID: <20100215204206.GA9789@jasmine.xos.nl> On Mon, Feb 15, 2010 at 01:20:35PM -0500, Mario Salcedo wrote: > Hi, I am configuring a Red hat cluster. In the guide say: "A GFS2 > file system must be created on an LVM logical volume that is a linear > or mirrored volume". Is that true. > > I create GFS2 without LVM. I think all ok. Not really... How do you ensure then that the device paths of the GFS filesystems are the same on all nodes (as is required by the shared cluster.conf)? A shared disk may appear as /dev/sda in one node, but as /dev/sdb on another node, for example, dependent on hardware or configuration details. -- -- Jos Vos -- X/OS Experts in Open Systems BV | Phone: +31 20 6938364 -- Amsterdam, The Netherlands | Fax: +31 20 6948204 From scooter at cgl.ucsf.edu Mon Feb 15 21:33:42 2010 From: scooter at cgl.ucsf.edu (Scooter Morris) Date: Mon, 15 Feb 2010 13:33:42 -0800 Subject: [Linux-cluster] GFS2 and lvm In-Reply-To: <20100215204206.GA9789@jasmine.xos.nl> References: <22344e331002151020o21ca6cf0naa1899ab1bb1cf9f@mail.gmail.com> <20100215204206.GA9789@jasmine.xos.nl> Message-ID: <4B79BDB6.80207@cgl.ucsf.edu> On 02/15/2010 12:42 PM, Jos Vos wrote: > On Mon, Feb 15, 2010 at 01:20:35PM -0500, Mario Salcedo wrote: > > >> Hi, I am configuring a Red hat cluster. In the guide say: "A GFS2 >> file system must be created on an LVM logical volume that is a linear >> or mirrored volume". Is that true. >> >> I create GFS2 without LVM. I think all ok. >> > Not really... > > How do you ensure then that the device paths of the GFS filesystems > are the same on all nodes (as is required by the shared cluster.conf)? > > A shared disk may appear as /dev/sda in one node, but as /dev/sdb on > another node, for example, dependent on hardware or configuration > details. > > Actually, this is not really true. We have many GFS2 filesystems on a three node cluster and we don't use LVM. It really depends on how you are mounting the filesystems. You really need to mount them by LABEL, so the underlying disk identifiers becomes irrelevant. -- scooter From jeff.sturm at eprize.com Mon Feb 15 21:57:17 2010 From: jeff.sturm at eprize.com (Jeff Sturm) Date: Mon, 15 Feb 2010 16:57:17 -0500 Subject: [Linux-cluster] GFS and Cluster In-Reply-To: References: Message-ID: <64D0546C5EBBD147B75DE133D798665F055D884D@hugo.eprize.local> Mounting a GFS filesystem on two or more nodes concurrently requires that you have a lock manager in place. The cluster suite provides DLM for this, which absolutely requires an operational cluster. (You can mount GFS on a single node with no cluster, but that's not a very interesting way to use GFS.) From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Fagnon Raymond Sent: Monday, February 15, 2010 3:40 PM To: linux clustering Subject: [Linux-cluster] GFS and Cluster I have a hypothetical question. Will GFS work without the cluster software active? Here is what I would like to do. I have a two node nfs cluster. Currently the cluster is set up as active standby. What I propose is to shutdown the cluster software leaving GFS in place. Have both nodes NFS out /datadir. Put the virtual ip on an f5 load balancer and have NFS load balanced between the two nodes. This will allow me to leverage both servers instead of leaving one idle. Is this possible to accomplish. My understanding is that gfs is used to allow two nodes access to one file system whether they are running cluster software or not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jos at xos.nl Mon Feb 15 22:20:42 2010 From: jos at xos.nl (Jos Vos) Date: Mon, 15 Feb 2010 23:20:42 +0100 Subject: [Linux-cluster] GFS2 and lvm In-Reply-To: <4B79BDB6.80207@cgl.ucsf.edu> References: <22344e331002151020o21ca6cf0naa1899ab1bb1cf9f@mail.gmail.com> <20100215204206.GA9789@jasmine.xos.nl> <4B79BDB6.80207@cgl.ucsf.edu> Message-ID: <20100215222042.GA10725@jasmine.xos.nl> On Mon, Feb 15, 2010 at 01:33:42PM -0800, Scooter Morris wrote: > Actually, this is not really true. We have many GFS2 filesystems on a > three node cluster and we don't use LVM. It really depends on how you > are mounting the filesystems. You really need to mount them by LABEL, > so the underlying disk identifiers becomes irrelevant. Good point. I try to recall my own argumentation, but that probably was related to quorum disks/partitions: they can't be labeled (or at least couldn't in RHEL 5.0), so that was the final reason for me to choose for LVM. -- -- Jos Vos -- X/OS Experts in Open Systems BV | Phone: +31 20 6938364 -- Amsterdam, The Netherlands | Fax: +31 20 6948204 From linux at alteeve.com Mon Feb 15 22:26:04 2010 From: linux at alteeve.com (Digimer) Date: Mon, 15 Feb 2010 17:26:04 -0500 Subject: [Linux-cluster] Overview of openais.conf and amf.conf Message-ID: <4B79C9FC.3000407@alteeve.com> Hi all, I've asked this before on the OpenAIS mailing list, and again today, but without any luck. I hope I can be forgiven for cross-posting this question to the general clustering mailing list. :) I've been working on some docs for clustering, and have wanted to cover the options in openais.conf and amf.conf. I've had a heck of a time finding a list of what options are valid in each and how altering their values effects the cluster. Is there such a list? If not, would someone be willing to help me write an overview? Further, there are a lot of docs on the role of AIS in general, but I've not found a good doc explaining what role AIS plays in Linux clustering. The docs all seem to have the impression that the reader is already familiar with AIS in general and is just looking for the OpenAIS specifics. Any pointers or help will be much appreciated! Digi From yvette at dbtgroup.com Tue Feb 16 00:18:19 2010 From: yvette at dbtgroup.com (yvette hirth) Date: Tue, 16 Feb 2010 00:18:19 +0000 Subject: [Linux-cluster] GFS2 and lvm In-Reply-To: <20100215222042.GA10725@jasmine.xos.nl> References: <22344e331002151020o21ca6cf0naa1899ab1bb1cf9f@mail.gmail.com> <20100215204206.GA9789@jasmine.xos.nl> <4B79BDB6.80207@cgl.ucsf.edu> <20100215222042.GA10725@jasmine.xos.nl> Message-ID: <4B79E44B.2000908@dbtgroup.com> Jos Vos wrote: > On Mon, Feb 15, 2010 at 01:33:42PM -0800, Scooter Morris wrote: >> Actually, this is not really true. We have many GFS2 filesystems on a >> three node cluster and we don't use LVM. It really depends on how you >> are mounting the filesystems. You really need to mount them by LABEL, >> so the underlying disk identifiers becomes irrelevant. > Good point. > I try to recall my own argumentation, but that probably was related to > quorum disks/partitions: they can't be labeled (or at least couldn't > in RHEL 5.0), so that was the final reason for me to choose for LVM. but they can be referenced by UUID even if they cannot be labeled... yvette hirth From jumanjiman at gmail.com Tue Feb 16 02:40:02 2010 From: jumanjiman at gmail.com (Paul Morgan) Date: Mon, 15 Feb 2010 21:40:02 -0500 Subject: [Linux-cluster] GFS2 and lvm In-Reply-To: <4B79BDB6.80207@cgl.ucsf.edu> References: <22344e331002151020o21ca6cf0naa1899ab1bb1cf9f@mail.gmail.com> <20100215204206.GA9789@jasmine.xos.nl> <4B79BDB6.80207@cgl.ucsf.edu> Message-ID: <1b6fc7dd1002151840n6b17550elfd396a693719b040@mail.gmail.com> if you're not using LVM but ARE using multipath then you should set up user-friendly name or use a path from /dev/disk/by-* -paul On Feb 15, 2010 4:41 PM, "Scooter Morris" wrote: On 02/15/2010 12:42 PM, Jos Vos wrote: > > > > On Mon, Feb 15, 2010 at 01:20:35PM -0500, Mario Salcedo wrote: > > > > > >> > >> Hi, I am configuring... > Not really... > > How do you ensure then that the device paths of the GFS filesystems > are the same on all nodes (as is required by the shared cluster.conf)? > > A shared disk may appear as /dev/sda in one node, but as /dev/sdb on > another node, for example, dependent on hardware or configuration > details. > > > Actually, this is not really true. We have many GFS2 filesystems on a three node cluster and we don't use LVM. It really depends on how you are mounting the filesystems. You really need to mount them by LABEL, so the underlying disk identifiers becomes irrelevant. -- scooter -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/lin... -------------- next part -------------- An HTML attachment was scrubbed... URL: From siedler at hrd-asia.com Tue Feb 16 08:46:26 2010 From: siedler at hrd-asia.com (Wolf Siedler) Date: Tue, 16 Feb 2010 15:46:26 +0700 Subject: [Linux-cluster] Xen / Redhat cluster : 5.3 to 5.4 upgrade question and best practice In-Reply-To: <1431512389.92691266257581025.JavaMail.root@vicenza.dmz.lexum.pri> References: <1431512389.92691266257581025.JavaMail.root@vicenza.dmz.lexum.pri> Message-ID: <4B7A5B62.9000804@hrd-asia.com> > if I directly modified cluster.conf can I still use lucci after that > or will it replace my custom edited cluster.conf? Yes, absolutely. This is what we did. It may or may not be important, but some thoughts: - Make sure to increase version number when manually editing cluster.conf. - Make sure all version numbers on all nodes are identical. - As a precaution we explicitly logged off Luci when manually editing cluster.conf. Upon re-logon, the changes were present in Luci. However, I need to point out that I have not (yet?) gotten into Luci's internal workings. Regards, Wolf From fdinitto at redhat.com Tue Feb 16 11:55:16 2010 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Tue, 16 Feb 2010 12:55:16 +0100 Subject: [Linux-cluster] Upcoming changes to cluster releases Message-ID: <4B7A87A4.8060004@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi everybody, I just landed a set of changes to improve our release process for cluster 3. Here are the highlights: - - tarballs will be released in both .gz and bz2 formats. - - a Changelog-$VERSION will be available on the release area (same as the one available in the announce emails). - - a .sha256 file will also be available for each release to verify the downloads. - - for the more paranoid, the .sha256 file is signed with gpg (*). As of today, we will _not_ update tarballs on the old ftp://sources.redhat.com/pub/cluster/releases/ Please update your watchfiles to use: https://fedorahosted.org/releases/c/l/cluster/ Cheers Fabio (*) In order to verify the release signature (example): 1) $ gpg --recv-keys 0x6CE95CA7 2) $ gpg --fingerprint 0x6CE95CA7 pub 4096R/6CE95CA7 2010-02-08 Key fingerprint = 746F 4C0C A8C2 B82E CC0A 6948 08C8 1B2C 6CE9 5CA7 uid Cluster Release Team 3) $ gpg --check-sigs 0x6CE95CA7 pub 4096R/6CE95CA7 2010-02-08 uid Cluster Release Team sig!3 6CE95CA7 2010-02-08 Cluster Release Team sig! 63549F8E 2010-02-08 Fabio M. Di Nitto sig! 0B437A89 2010-02-08 Fabio M. Di Nitto sig! 4CF8CD0C 2010-02-08 Lon Hohberger The key will always carry only the UIDs of the release managers. 4) $ gpg --verify cluster-3.0.8.sha256.asc cluster-3.0.8.sha256 gpg: Signature made Tue Feb 16 12:37:42 2010 CET using RSA key ID 6CE95CA7 gpg: Good signature from "Cluster Release Team " Generally, step 1 to 3 are required only for a first time setup. Please always consult any extensive gpg how to for more information. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJLeoehAAoJEFA6oBJjVJ+OjA8P/iFhMjdVGYs/GUo0j/RSNzl+ a46BVyMzE/H6lPUDBZkyopuFNM0fgdMzHC5KNKcNyjO4gyVVp3/w16VoGjCfOzzk mmEFq4xr8XOfgUzYoRvRHVVn4zOFHLQlwQ+sw7zlZGbQXZiUP/ffiL4czeSZnztS 0S1uHkMSykZpMq0fAYPBFmf9ipsN3QQzIjL069e1ZYE5TqoZFVE8L9qV5MA92MSP Q+UAYUzRY7rsEguzCiM2oay/Xgi1cymYKQoUXEopRlPnebP/84Ntylkr5/w4mDqp E2uFT1DOmFCR6EdcVpMrGSbqgeaq3IfM5otJIm8jwfzHqI9FkXM4onCKYv2ww+11 /aAzTz4Zdu66wPBfBghk/AsuIMsOLFgyqoWZYobswrAWaqJhHtfRy/OXE5G4Mt8+ mJEOiR9e1AqPvrwuvKfkNu1CysDNUzGBIc8nm7JXY7qjRjpd1q/+CKPJHj98/S/g M0DkhCyfYh/oaDDC58RBzP1h3aKmFIRQ0AibvrQs6LHwWnfMznZ0kKENXvwXhV2A qSHr0oLKEGClBYkr/7VXkzJxwqOntVz6R6NT/t6RO6e+v6zimqYLYNpi+jj2YFgU 6W+XgYTi9MkmibI4lsopDX1iz7rVw9VujFanH0KnOtO94lteGgkqEmJzLXH1vYyM PZbO0x46MKn6xy0Ozf95 =tdgO -----END PGP SIGNATURE----- From fdinitto at redhat.com Tue Feb 16 12:15:04 2010 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Tue, 16 Feb 2010 13:15:04 +0100 Subject: [Linux-cluster] Organizing Bug Squash Party for Cluster 3.x, GFS2 and more In-Reply-To: <4B67FA23.2070800@redhat.com> References: <4B67FA23.2070800@redhat.com> Message-ID: <4B7A8C48.7010102@redhat.com> Given the close to 0 response to the initiative, I?d say we skip it. thanks anyway for your time. Fabio On 2/2/2010 11:10 AM, Fabio M. Di Nitto wrote: > Hi everybody, > > this is the first time that we are trying to organize this kind of event > and we would like to hear opinions and ideas from everybody. > > Because this is a bit of uncharted territory for us (upstream), I don?t > want to set too high expectations from the very first round, but hey.. > let?s try and see what happens. > > The general idea is to have a 24 hours "work together with upstream" to > find bugs, test quick fixes and provide all sort of feedback (even "HEY > THIS SOFTWARE SUCKS" is good feedback if you can tell us why and where > we need to improve). > > I don?t want to restrict this bug squash party to only Red Hat Cluster > and GFS2. If other upstreams (pacemaker, OCFS2, drbd, corosync, openais, > conga, you name it) wants to join, please do so. This is an open > invitation. Feel free to forward this idea around as long as I am at > least CC?ed to keep track of participants. > > Assuming there is interest in the event, let?s take a look at some > simple practical details: > > - When is this going to happen? I am targeting the 2nd of March as > candidate date. Starting at 00:00 UTC and finishing at 23:59 UTC. Some > areas will have more coverages, others a bit less. If there will be a > next time, we will move the window around to facilitate other time zones. > > - What should be tested? Anything really.. do whatever you think your > cluster should be able to do. > > - What kind of hardware should be used? Again.. anything you want to put > in the test cluster. Virtual machines? bare metal.. you decide. > > - We will get any help to configure the cluster? It depends. This is not > a training session, but mostly a dedicate tour to fix bugs. We don?t > want to exclude anyone but it?s best if you have already some experience > with clustering. > > - As team we don?t have a 24h coverage around the world yet. We are > working to plan shifts to have at least one maintainer for each > component/subsystem available at any time. We will post what?s the best > coverage later on. > > - In order to coordinate the event, we will use #linux-cluster IRC > channel on freenode. > > - All issues found or features reported should be filed in the > appropriate bug tracker. It?s important to have track in case a fix or a > change cannot be provided within the 24h time frame. > > - Be ready to trash your test cluster. > > - Be ready to test and report back with information. As much as possible > we will try to provide patches and packages (read below) that people can > quickly install and use for testing. > > - Distribution of choice: as best as we can, we would like to help as > many distributions as possible, but there are some practical limits. > We can easily provide binary packages for Fedora 12 and Fedora rawhide. > I am in the process to setup Debian and Ubuntu build machines, but > whenever possible, it?s best if you can build your own binary packages > to offload the team from this task. > > - We will assume that the base version to start testing is the latest > upstream release version at the time (excluding kernel). It might be too > time consuming to look into bugs that might have been already fixed. > > - Kernel bits.. This is clearly more complex to test and build if you > are less familiar with kernel and distribution packaging. We will work > on the base of the latest kernel available for your specific > distribution. Please make sure to have a URL handy to the source code > for us to look at. > > Please feel free to add anything I might have missed. > > Cheers > Fabio > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From swhiteho at redhat.com Tue Feb 16 13:06:22 2010 From: swhiteho at redhat.com (Steven Whitehouse) Date: Tue, 16 Feb 2010 13:06:22 +0000 Subject: [Linux-cluster] GFS2 and lvm In-Reply-To: <4B79BDB6.80207@cgl.ucsf.edu> References: <22344e331002151020o21ca6cf0naa1899ab1bb1cf9f@mail.gmail.com> <20100215204206.GA9789@jasmine.xos.nl> <4B79BDB6.80207@cgl.ucsf.edu> Message-ID: <1266325582.14393.363.camel@localhost.localdomain> Hi, On Mon, 2010-02-15 at 13:33 -0800, Scooter Morris wrote: > On 02/15/2010 12:42 PM, Jos Vos wrote: > > On Mon, Feb 15, 2010 at 01:20:35PM -0500, Mario Salcedo wrote: > > > > > >> Hi, I am configuring a Red hat cluster. In the guide say: "A GFS2 > >> file system must be created on an LVM logical volume that is a linear > >> or mirrored volume". Is that true. > >> > >> I create GFS2 without LVM. I think all ok. > >> > > Not really... > > > > How do you ensure then that the device paths of the GFS filesystems > > are the same on all nodes (as is required by the shared cluster.conf)? > > > > A shared disk may appear as /dev/sda in one node, but as /dev/sdb on > > another node, for example, dependent on hardware or configuration > > details. > > > > > Actually, this is not really true. We have many GFS2 filesystems on a > three node cluster and we don't use LVM. It really depends on how you > are mounting the filesystems. You really need to mount them by LABEL, > so the underlying disk identifiers becomes irrelevant. > > -- scooter > Any underlying storage system should be ok provided it provides the guarantees required by GFS2 in terms of caching and ordering of operations. The reason that the docs mention the use of CLVM is that it is a requirement for support, not because other solutions (including direct sharing of disks) will not work, Steve. From msalced at pucp.edu.pe Tue Feb 16 16:49:54 2010 From: msalced at pucp.edu.pe (Mario Salcedo) Date: Tue, 16 Feb 2010 11:49:54 -0500 Subject: [Linux-cluster] GFS2 and lvm In-Reply-To: <1266325582.14393.363.camel@localhost.localdomain> References: <22344e331002151020o21ca6cf0naa1899ab1bb1cf9f@mail.gmail.com> <20100215204206.GA9789@jasmine.xos.nl> <4B79BDB6.80207@cgl.ucsf.edu> <1266325582.14393.363.camel@localhost.localdomain> Message-ID: <22344e331002160849wf199c18kcc0bf913eba803be@mail.gmail.com> Thanks. I now understand better GFS2. I probe GFS2 in a cluster of 4 nodes identicals. Maybe for that i dont have problems when I mount /dev/sdb in all nodes. I think is best using LABEL or CLVM. 2010/2/16 Steven Whitehouse : > Hi, > > On Mon, 2010-02-15 at 13:33 -0800, Scooter Morris wrote: >> On 02/15/2010 12:42 PM, Jos Vos wrote: >> > On Mon, Feb 15, 2010 at 01:20:35PM -0500, Mario Salcedo wrote: >> > >> > >> >> Hi, I am configuring ?a Red hat cluster. In the guide say: "A GFS2 >> >> file system must be created on an LVM logical volume that is a linear >> >> or mirrored volume". Is that true. >> >> >> >> I create GFS2 without LVM. I think all ok. >> >> >> > Not really... >> > >> > How do you ensure then that the device paths of the GFS filesystems >> > are the same on all nodes (as is required by the shared cluster.conf)? >> > >> > A shared disk may appear as /dev/sda in one node, but as /dev/sdb on >> > another node, for example, dependent on hardware or configuration >> > details. >> > >> > >> Actually, this is not really true. ?We have many GFS2 filesystems on a >> three node cluster and we don't use LVM. ?It really depends on how you >> are mounting the filesystems. ?You really need to mount them by LABEL, >> so the underlying disk identifiers becomes irrelevant. >> >> -- scooter >> > Any underlying storage system should be ok provided it provides the > guarantees required by GFS2 in terms of caching and ordering of > operations. The reason that the docs mention the use of CLVM is that it > is a requirement for support, not because other solutions (including > direct sharing of disks) will not work, > > Steve. > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > From Martin.Waite at datacash.com Wed Feb 17 06:50:47 2010 From: Martin.Waite at datacash.com (Martin Waite) Date: Wed, 17 Feb 2010 06:50:47 -0000 Subject: [Linux-cluster] bad titles on the wiki Message-ID: Hi, A Google site search - "site://sources.redhat.com/cluster/wiki" - showed a few dozen links that obviously have nothing to do with linux-cluster, and in some cases contain unattributed copyright material, e.g.: http://sources.redhat.com/cluster/wiki/jacaodl contains text taken from the BBC News website - http://news.bbc.co.uk/nol/ukfs_news/hi/newsid_8350000/newsid_8356000/8356094.stm The MoinMoin site index http://sources.redhat.com/cluster/wiki/TitleIndex contains about 40 suspicious looking links. Is there a back-door open in the Wiki ? regards, Martin ========= The following titles should be removed: http://sources.redhat.com/cluster/wiki/134_cibao-channel-7-channel-7-programs-cibao-channel-7 http://sources.redhat.com/cluster/wiki/137_ars-moriendi http://sources.redhat.com/cluster/wiki/162_anarchy-postizquierda-conflicts-with-leftism http://sources.redhat.com/cluster/wiki/212_air-comet-history http://sources.redhat.com/cluster/wiki/257_bologna-sports http://sources.redhat.com/cluster/wiki/300_a%26%2339%3Barab-zaraq---lucid-dreaming-other-contributors http://sources.redhat.com/cluster/wiki/346_asset-allocation http://sources.redhat.com/cluster/wiki/383_1039-smoothed-out-slappy-hours http://sources.redhat.com/cluster/wiki/460_1005 http://sources.redhat.com/cluster/wiki/465_alberto-contador-career http://sources.redhat.com/cluster/wiki/497_142-bc http://sources.redhat.com/cluster/wiki/501_1001-died http://sources.redhat.com/cluster/wiki/535_albany-western-australia-history http://sources.redhat.com/cluster/wiki/577_asset-allocation http://sources.redhat.com/cluster/wiki/581_bikinis http://sources.redhat.com/cluster/wiki/640_compound-interest-calculator http://sources.redhat.com/cluster/wiki/673_avgustin-voloshin-independence-of-the-carpathian-ukraine http://sources.redhat.com/cluster/wiki/690_can-utility-and-the-coastliners-trivia http://sources.redhat.com/cluster/wiki/813_ali-canas-venezuelan-league http://sources.redhat.com/cluster/wiki/824_11-ss-volunteer-panzer-grenadier-division-quotnordlandquot http://sources.redhat.com/cluster/wiki/894_compound-interest-calculator http://sources.redhat.com/cluster/wiki/917_ambrosio-spinola-bibliography http://sources.redhat.com/cluster/wiki/931_asset-allocation http://sources.redhat.com/cluster/wiki/938_compound-interest-calculator http://sources.redhat.com/cluster/wiki/941_arrondissement http://sources.redhat.com/cluster/wiki/961_april-16-deaths http://sources.redhat.com/cluster/wiki/988_asset-allocation http://sources.redhat.com/cluster/wiki/998_1-fc-cologne-statistics http://sources.redhat.com/cluster/wiki/AirTicketsm http://sources.redhat.com/cluster/wiki/AirTicketsm/saeqpriezbjztuhpotavqfw http://sources.redhat.com/cluster/wiki/atlantic-city-ballys-casino-280 http://sources.redhat.com/cluster/wiki/blackjack-gambling-163 http://sources.redhat.com/cluster/wiki/carry.chen http://sources.redhat.com/cluster/wiki/jacaodl http://sources.redhat.com/cluster/wiki/learn-poker-220 http://sources.redhat.com/cluster/wiki/loki_681512 http://sources.redhat.com/cluster/wiki/loki_734955 http://sources.redhat.com/cluster/wiki/loki_750903 http://sources.redhat.com/cluster/wiki/make-money-with-online-casinos-113 http://sources.redhat.com/cluster/wiki/MoinMoin/fuoqtbtgmwlaimywckvu http://sources.redhat.com/cluster/wiki/MoinMoin/owjiglaqjupolmqmzhhbaxbujvavwa http://sources.redhat.com/cluster/wiki/online-casino-dubai-26 http://sources.redhat.com/cluster/wiki/online-gambling-slots-52 http://sources.redhat.com/cluster/wiki/pc-slot-machine-games-170 http://sources.redhat.com/cluster/wiki/play-poker-for-free-77 http://sources.redhat.com/cluster/wiki/standby-flights-from-toronto-393 From fdinitto at redhat.com Wed Feb 17 18:27:43 2010 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Wed, 17 Feb 2010 19:27:43 +0100 Subject: [Linux-cluster] bad titles on the wiki In-Reply-To: References: Message-ID: <4B7C351F.8050906@redhat.com> Hi Martin, thanks for reporting this issue. we will try to clean that up asap. Fabio On 2/17/2010 7:50 AM, Martin Waite wrote: > Hi, > > A Google site search - "site://sources.redhat.com/cluster/wiki" - showed a few dozen links that obviously have nothing to do with linux-cluster, and in some cases contain unattributed copyright material, e.g.: > > http://sources.redhat.com/cluster/wiki/jacaodl > > contains text taken from the BBC News website - > > http://news.bbc.co.uk/nol/ukfs_news/hi/newsid_8350000/newsid_8356000/8356094.stm > > The MoinMoin site index > > http://sources.redhat.com/cluster/wiki/TitleIndex > > contains about 40 suspicious looking links. Is there a back-door open in the Wiki ? > > regards, > Martin > > ========= > > The following titles should be removed: > http://sources.redhat.com/cluster/wiki/134_cibao-channel-7-channel-7-programs-cibao-channel-7 > http://sources.redhat.com/cluster/wiki/137_ars-moriendi > http://sources.redhat.com/cluster/wiki/162_anarchy-postizquierda-conflicts-with-leftism > http://sources.redhat.com/cluster/wiki/212_air-comet-history > http://sources.redhat.com/cluster/wiki/257_bologna-sports > http://sources.redhat.com/cluster/wiki/300_a%26%2339%3Barab-zaraq---lucid-dreaming-other-contributors > http://sources.redhat.com/cluster/wiki/346_asset-allocation > http://sources.redhat.com/cluster/wiki/383_1039-smoothed-out-slappy-hours > http://sources.redhat.com/cluster/wiki/460_1005 > http://sources.redhat.com/cluster/wiki/465_alberto-contador-career > http://sources.redhat.com/cluster/wiki/497_142-bc > http://sources.redhat.com/cluster/wiki/501_1001-died > http://sources.redhat.com/cluster/wiki/535_albany-western-australia-history > http://sources.redhat.com/cluster/wiki/577_asset-allocation > http://sources.redhat.com/cluster/wiki/581_bikinis > http://sources.redhat.com/cluster/wiki/640_compound-interest-calculator > http://sources.redhat.com/cluster/wiki/673_avgustin-voloshin-independence-of-the-carpathian-ukraine > http://sources.redhat.com/cluster/wiki/690_can-utility-and-the-coastliners-trivia > http://sources.redhat.com/cluster/wiki/813_ali-canas-venezuelan-league > http://sources.redhat.com/cluster/wiki/824_11-ss-volunteer-panzer-grenadier-division-quotnordlandquot > http://sources.redhat.com/cluster/wiki/894_compound-interest-calculator > http://sources.redhat.com/cluster/wiki/917_ambrosio-spinola-bibliography > http://sources.redhat.com/cluster/wiki/931_asset-allocation > http://sources.redhat.com/cluster/wiki/938_compound-interest-calculator > http://sources.redhat.com/cluster/wiki/941_arrondissement > http://sources.redhat.com/cluster/wiki/961_april-16-deaths > http://sources.redhat.com/cluster/wiki/988_asset-allocation > http://sources.redhat.com/cluster/wiki/998_1-fc-cologne-statistics > http://sources.redhat.com/cluster/wiki/AirTicketsm > http://sources.redhat.com/cluster/wiki/AirTicketsm/saeqpriezbjztuhpotavqfw > http://sources.redhat.com/cluster/wiki/atlantic-city-ballys-casino-280 > http://sources.redhat.com/cluster/wiki/blackjack-gambling-163 > http://sources.redhat.com/cluster/wiki/carry.chen > http://sources.redhat.com/cluster/wiki/jacaodl > http://sources.redhat.com/cluster/wiki/learn-poker-220 > http://sources.redhat.com/cluster/wiki/loki_681512 > http://sources.redhat.com/cluster/wiki/loki_734955 > http://sources.redhat.com/cluster/wiki/loki_750903 > http://sources.redhat.com/cluster/wiki/make-money-with-online-casinos-113 > http://sources.redhat.com/cluster/wiki/MoinMoin/fuoqtbtgmwlaimywckvu > http://sources.redhat.com/cluster/wiki/MoinMoin/owjiglaqjupolmqmzhhbaxbujvavwa > http://sources.redhat.com/cluster/wiki/online-casino-dubai-26 > http://sources.redhat.com/cluster/wiki/online-gambling-slots-52 > http://sources.redhat.com/cluster/wiki/pc-slot-machine-games-170 > http://sources.redhat.com/cluster/wiki/play-poker-for-free-77 > http://sources.redhat.com/cluster/wiki/standby-flights-from-toronto-393 > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From jose.neto at liber4e.com Thu Feb 18 15:12:33 2010 From: jose.neto at liber4e.com (jose nuno neto) Date: Thu, 18 Feb 2010 15:12:33 -0000 (GMT) Subject: [Linux-cluster] 2node cluster+qdiskd One Node link break to quorum disk gets rebooted In-Reply-To: <6b717372e2aee82ef4b27c835d7ad0f1.squirrel@fela.liber4e.com> References: <6b717372e2aee82ef4b27c835d7ad0f1.squirrel@fela.liber4e.com> Message-ID: Hi I have a 2node + qdisk ( on iSCSI) cluster conf. Seems be ok, except for one scenario: 1node link break at quorum disk ( iscsi break) On iSCSI link break node continues to work ok. On iSCSI recovery node puts eth ifaces down, looses hearbeats and gest fenced. Is this Ok? Thanks Jose Logs ########################### iSCSI FAIL Feb 18 11:44:49 node2 kernel: connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4297680222, last ping 4297685222, now 4297690222 Feb 18 11:44:49 node2 kernel: connection2:0: detected conn error (1011) Feb 18 11:44:50 node2 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3) Feb 18 11:44:50 node2 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3) Feb 18 11:44:52 node2 iscsid: connect to 172.26.244.4:3260 failed (Connection refused) Feb 18 11:44:52 node2 iscsid: connect to 172.26.244.4:3260 failed (Connection refused) Feb 18 11:44:56 node2 iscsid: connect to 172.26.244.4:3260 failed (Connection refused) Feb 18 11:44:56 node2 openais[6815]: [CMAN ] lost contact with quorum device Feb 18 11:44:59 node2 iscsid: connect to 172.26.244.4:3260 failed (Connection refused) iSCSI RECOVER Feb 18 11:52:47 node2 kernel: Synchronizing SCSI cache for disk sde: Feb 18 11:52:47 node2 kernel: Synchronizing SCSI cache for disk sdd: Feb 18 11:52:49 node2 kernel: FAILED Feb 18 11:52:49 node2 kernel: status = 1, message = 00, host = 0, driver = 08 Feb 18 11:52:49 node2 kernel: <6>sd: Current: sense key: Illegal Request Feb 18 11:52:49 node2 kernel: <> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1 ------------------------------ Feb 18 11:52:49 node2 kernel: bonding: bond0: backup interface eth5 is now down Feb 18 11:52:49 node2 kernel: ACPI: PCI interrupt for device 0000:1f:00.1 disabled Feb 18 11:52:49 node2 kernel: bonding: bond0: link status down for active interface eth0, disabling it Feb 18 11:52:49 node2 kernel: bonding: bond0: now running without any active interface ! Feb 18 11:52:49 node2 kernel: ACPI: PCI interrupt for device 0000:1f:00.0 disabled Feb 18 11:52:49 node2 kernel: bonding: bond2: link status down for interface eth1, disabling it in 2000 ms. Feb 18 11:52:49 node2 kernel: ACPI: PCI interrupt for device 0000:10:00.1 disabled Feb 18 11:52:49 node2 kernel: ACPI: PCI interrupt for device 0000:10:00.0 disabled Feb 18 11:52:49 node2 kernel: bonding: bond2: link status down for interface eth3, disabling it in 2000 ms. Feb 18 11:52:50 node2 kernel: bonding: bond1: backup interface eth4 is now down Feb 18 11:52:50 node2 kernel: ACPI: PCI interrupt for device 0000:0a:00.1 disabled Feb 18 11:52:50 node2 kernel: bonding: bond1: link status down for active interface eth2, disabling it Feb 18 11:52:50 node2 kernel: bonding: bond1: now running without any active interface ! Feb 18 11:52:50 node2 kernel: ACPI: PCI interrupt for device 0000:0a:00.0 disabled CleanNode Messages Feb 18 11:53:02 node1 kernel: dlm: closing connection to node 2 -------- Feb 18 11:54:42 node1 fenced[7310]: node2.lux.eib.org not a cluster member after 100 sec post_fail_delay Feb 18 11:54:42 node1 fenced[7310]: fencing node "node2.lux.eib.org" Feb 18 11:54:49 node1 fenced[7310]: fence "node2.lux.eib.org" success -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From lhh at redhat.com Thu Feb 18 20:28:13 2010 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 18 Feb 2010 15:28:13 -0500 Subject: [Linux-cluster] 2node cluster + qdiskd for TieBreak In-Reply-To: <6b717372e2aee82ef4b27c835d7ad0f1.squirrel@fela.liber4e.com> References: <6b717372e2aee82ef4b27c835d7ad0f1.squirrel@fela.liber4e.com> Message-ID: <1266524893.23427.1701.camel@localhost.localdomain> On Tue, 2010-02-09 at 14:30 +0000, jose nuno neto wrote: > master_wait="2" votes="1" log_level="7" min_score="1" reboot="0" > allow_kill="1" stop_cman="1" status_file > ="/tmp/QdiskStatus"> Get rid of status_file, set reboot to "1" to start. -- Lon From lhh at redhat.com Thu Feb 18 20:45:06 2010 From: lhh at redhat.com (Lon Hohberger) Date: Thu, 18 Feb 2010 15:45:06 -0500 Subject: [Linux-cluster] independent subtree In-Reply-To: <20100213145541.M89820@varna.net> References: <20100213145541.M89820@varna.net> Message-ID: <1266525906.23427.1718.camel@localhost.localdomain> On Sat, 2010-02-13 at 17:16 +0200, Kaloyan Kovachev wrote: > Hello, > when __independent_subtree is used and the resource fails the services is not > relocated after max_restarts. Is this a bug or is by design? > > Example: > > restart_expire_time="1800"> > > > > > > > the idea here is if MySQL have crashed mysqld_safe will restart it, but if > there is a problem with Apache - to restart it without stopping MySQL. This > works OK, but if there is a permanent problem with Apache (no access to the > file system) it is restarted several times without the service being relocated > to another node By design. There's a bugzilla open about adding per-resource restart counters, it has not been implemented. -- Lon From dhoffutt at gmail.com Fri Feb 19 00:52:16 2010 From: dhoffutt at gmail.com (Dustin Henry Offutt) Date: Thu, 18 Feb 2010 18:52:16 -0600 Subject: [Linux-cluster] independent subtree In-Reply-To: <1266525906.23427.1718.camel@localhost.localdomain> References: <20100213145541.M89820@varna.net> <1266525906.23427.1718.camel@localhost.localdomain> Message-ID: <4B7DE0C0.4060601@gmail.com> Hey there Lon, May I ask you what exactly it is that provisioning _independent_subtrees does? Haven't seen it documented (and have read voraciously...). A sincere thank you to you, Dusty Lon Hohberger wrote: > On Sat, 2010-02-13 at 17:16 +0200, Kaloyan Kovachev wrote: > >> Hello, >> when __independent_subtree is used and the resource fails the services is not >> relocated after max_restarts. Is this a bug or is by design? >> >> Example: >> >> > restart_expire_time="1800"> >> >> >> >> >> >> >> the idea here is if MySQL have crashed mysqld_safe will restart it, but if >> there is a problem with Apache - to restart it without stopping MySQL. This >> works OK, but if there is a permanent problem with Apache (no access to the >> file system) it is restarted several times without the service being relocated >> to another node >> > > By design. There's a bugzilla open about adding per-resource restart > counters, it has not been implemented. > > -- Lon > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dhoffutt at gmail.com Fri Feb 19 19:26:46 2010 From: dhoffutt at gmail.com (Dusty) Date: Sat, 20 Feb 2010 13:26:46 +1800 Subject: [Linux-cluster] cman not started: Can't find local node name in cluster.conf Message-ID: Hello, I had previously had several clusters running using the Cluster Suite software versions included in the RHEL5U3 ISO. The service that was being made highly available depends on the hostname being one static hostname, so I had made every node in the cluster the same name. In configuring the cluster.conf, was able to use the IP addresses instead of fully qualified hostnames, and this worked as of RHEL5U3. Just figured out, this seems not to be the case with the Cluster Suite version included in the RHEL5U4 x86_64 ISO. Using IPs instead of hostnames causes cman to say, "cman not started: Can't find local node name in cluster.conf". Does anyone know of a way around this? How would the Cluster Suite react to the hostname changing via a Cluster Suite service/resource script, at "script start" and then returning it to the original node-name as "script stop"? Cannot stress how absolutely enormous this issue is to me.... Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dhoffutt at gmail.com Fri Feb 19 19:44:26 2010 From: dhoffutt at gmail.com (Dusty) Date: Sat, 20 Feb 2010 13:44:26 +1800 Subject: [Linux-cluster] cman not started: Can't find local node name in cluster.conf In-Reply-To: References: Message-ID: Sorry, actually IP address are working as hostnames still in the cluster.conf (WHEW!) . This is just a sign of trying to work on NyquilD. Please accept my apologies for the unnecessary interruption... If I may, I would still like to leave the question open regarding, what if the hostnames were to change as part of the operation of the server? The software in question depends on the hostname being one hostname and not changing when it relocates to another cluster node. Thanks again! -------------- next part -------------- An HTML attachment was scrubbed... URL: From cbake at livemercial.com Sat Feb 20 04:13:37 2010 From: cbake at livemercial.com (Chris Bake) Date: Fri, 19 Feb 2010 22:13:37 -0600 Subject: [Linux-cluster] RHEL 5 - GFS2 Books Needed Message-ID: <1266639217.4742.5.camel@cbake-dev01> Hello, Can anyone recommend a complete GFS2 book? I've not been able to find anything online for sale that does more then touch on the subject. Desperate for knowledge here, Chris Bake cbake at livemercial.com From zachar at awst.at Sat Feb 20 09:34:22 2010 From: zachar at awst.at (zachar at awst.at) Date: Sat, 20 Feb 2010 10:34:22 +0100 (CET) Subject: [Linux-cluster] =?utf-8?q?RHEL_5_-_GFS2_Books_Needed?= Message-ID: Hi, Is this enough for you?: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Global_File_System_2/index.html The same in pdf: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/pdf/Global_File_System_2.pdf Regards, Balazs Chris Bake schrieb: > Hello, > > Can anyone recommend a complete GFS2 book? I've not been able to find > anything online for sale that does more then touch on the subject. > > Desperate for knowledge here, > Chris Bake > cbake at livemercial.com > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From jcasale at activenetwerx.com Sun Feb 21 20:30:38 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Sun, 21 Feb 2010 20:30:38 +0000 Subject: [Linux-cluster] Looking for guidance Message-ID: Hey, I am new to clusters, and have been reading up all the options but given I haven't any experience ever setting one up I don't know which option is best suited. I need to replicate a local volume between two servers, and export the data via nfs and rsyncd from either. They only need to be active/passive. I originally thought drbd and heartbeat v1 would be the simplest but looking at what's needed for nfs to work looks slightly hackish and I really don't know how well this will perform (It could very well be the best option). After looking at alternatives, I figured the cluster suite from rhel might be a good supported option but given the two node, active/passive setup that looks like it is way over complicated. The need for fencing can be slightly mitigated by requiring manual intervention to rejoin a node that has left, it's not active/active HA I am looking for, but redundant data and weak ha through failover. The nodes are in two buildings connected by gig fiber. Any experienced ops have any insight into an approach best suited that I could settle on and start researching? Thx, jlc From kkovachev at varna.net Mon Feb 22 10:55:21 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Mon, 22 Feb 2010 12:55:21 +0200 Subject: [Linux-cluster] Looking for guidance In-Reply-To: References: Message-ID: <20100222103350.M16636@varna.net> Hi, On Sun, 21 Feb 2010 20:30:38 +0000, Joseph L. Casale wrote > Hey, > I am new to clusters, and have been reading up all the options but > given I haven't any experience ever setting one up I don't know which > option is best suited. > > I need to replicate a local volume between two servers, and export the > data via nfs and rsyncd from either. They only need to be active/passive. > > I originally thought drbd and heartbeat v1 would be the simplest but looking > at what's needed for nfs to work looks slightly hackish and I really don't > know how well this will perform (It could very well be the best option). > For a simple HA with active/passive i would do it with just a cron script to check `drbdadm role ` and bring up the IP and services on the Primary if the other node is Secondary or Unknown (if not yet). I would still use some kind of fencing in DRBD's outdate-peer handler (at least set down the port(s) on the remote switch via SNMP) to avoid data corruption. Later when admin is about to rejoin the node (s)he should set the port back up after outdating the failed node's DRBD > After looking at alternatives, I figured the cluster suite from rhel might > be a good supported option but given the two node, active/passive setup > that looks like it is way over complicated. > > The need for fencing can be slightly mitigated by requiring manual intervention > to rejoin a node that has left, it's not active/active HA I am looking for, > but redundant data and weak ha through failover. The nodes are in two buildings > connected by gig fiber. > > Any experienced ops have any insight into an approach best suited that > I could settle on and start researching? > > Thx, > jlc > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From raju.rajsand at gmail.com Mon Feb 22 11:40:08 2010 From: raju.rajsand at gmail.com (Rajagopal Swaminathan) Date: Mon, 22 Feb 2010 17:10:08 +0530 Subject: [Linux-cluster] Looking for guidance In-Reply-To: References: Message-ID: <8786b91c1002220340s5521f8bh8863c71017123a5b@mail.gmail.com> Greetings, On Mon, Feb 22, 2010 at 2:00 AM, Joseph L. Casale wrote: > > Hey, > > I need to replicate a local volume between two servers, and export the > data via nfs and rsyncd from either. They only need to be active/passive. > psst.. wanna try this? sources.redhat.com/cluster/doc/nfscookbook.pdf and oh, DRBD in primary/primary mode works if you don't have (budget for) external shared storage. HTH Regards, Rajagopal From jcasale at activenetwerx.com Mon Feb 22 15:26:34 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Mon, 22 Feb 2010 15:26:34 +0000 Subject: [Linux-cluster] Looking for guidance In-Reply-To: <20100222103350.M16636@varna.net> References: <20100222103350.M16636@varna.net> Message-ID: >For a simple HA with active/passive i would do it with just a cron script to >check `drbdadm role ` and bring up the IP and services on the Primary if >the other node is Secondary or Unknown (if not yet). I would still use some >kind of fencing in DRBD's outdate-peer handler (at least set down the port(s) >on the remote switch via SNMP) to avoid data corruption. Later when admin is >about to rejoin the node (s)he should set the port back up after outdating the >failed node's DRBD Thanks for the thought. I like the fencing idea and no matter what choice I use I am pretty sure that's the route I will go. I do however want at least one failover possible, as I will likely be able to resolve the issue immediately after the first and if one node goes down, allowing it failover back and forth isn't logical. I'll keep this in mind while I read away. Thanks, jlc From jcasale at activenetwerx.com Mon Feb 22 15:35:49 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Mon, 22 Feb 2010 15:35:49 +0000 Subject: [Linux-cluster] Looking for guidance In-Reply-To: <8786b91c1002220340s5521f8bh8863c71017123a5b@mail.gmail.com> References: <8786b91c1002220340s5521f8bh8863c71017123a5b@mail.gmail.com> Message-ID: >psst.. wanna try this? > >sources.redhat.com/cluster/doc/nfscookbook.pdf > >and oh, DRBD in primary/primary mode works if you don't have (budget >for) external shared storage. Hi, I did find that doc while looking and gave it a read. Not knowing anything about how the suite operates, how does it mitigate the handling of stale filehandles in nfs? It doesn't mention anything about moving the nfs dirs into the shared storage? Seems so far from my weak grasp of what goes on that http://www.linux-ha.org/HaNFS covers the issue of failing over nfs. I haven't been able to find any docs that state how the rhel cluster suite handles this in a scenario like mine where I have a 2 node active/passive setup with one cluster ip for nfs that moves between them. Thanks! jlc Ps. No shared storage, sorry:) From tuckerd at lyle.smu.edu Mon Feb 22 16:15:49 2010 From: tuckerd at lyle.smu.edu (Doug Tucker) Date: Mon, 22 Feb 2010 10:15:49 -0600 Subject: [Linux-cluster] Repeated fencing Message-ID: <1266855349.16694.11.camel@thor.seas.smu.edu> We have a 2 4.x cluster that has developed an issue we are unable to resolve. Starting back in December, the nodes began fencing each other randomly, and as frequently as once a day. There is nothing at the console prior to it happening, and nothing in the logs. We have not been able to develop any pattern to this point, the 2 nodes appear to be functioning fine, and suddenly in the logs a message will appear about "node x missed too many heartbeats" and the next thing you see is it fencing the node. Thinking we possibly had a hardware issue, we replaced both nodes from scratch with new machines, the problem persists. The cluster communication is done via a crossover cable on eth1 on both devices with private ip's. We have a 2nd cluster that is not having this issue, and both nodes have been up for over 160 days. The configuration is basically identical to the problematic cluster. The only difference between the 2 now is the newer hardware on the problematic node (prior, that was identical), and the kernel. The non-problematic cluster is still running kernel 89.0.9 and the problematic cluster is on 89.0.11. We are afraid at this point to allow our non problematic cluster upgrade to the latest packages. Any insight or advice would be greatly appreciated, we have exhausted our ideas here. Sincerely, Doug From dist-list at LEXUM.UMontreal.CA Mon Feb 22 16:38:21 2010 From: dist-list at LEXUM.UMontreal.CA (F M) Date: Mon, 22 Feb 2010 11:38:21 -0500 (EST) Subject: [Linux-cluster] updating nodes and failback oprion In-Reply-To: <1345280262.389.1266856332156.JavaMail.root@vicenza.dmz.lexum.pri> Message-ID: <1566077.391.1266856701008.JavaMail.root@vicenza.dmz.lexum.pri> Hello, I have a 3 Xen nodes cluster + GFS2 (SAN). All VM are in the GFS2 FS. OS: just updated to RH 5.4 During the Dom0 update of one of the nodes I used live migration to move VMs to another node. After the update, I rebooted the server. When it came back VMs tried to go back to his preferred node. I'd like to move back the V.M. manually instead, is nofailback=1 would do the trick ? I looking for a wait to : - booting VM on a prefered node (This part is working) - If the node fails, the VM is restated on another node (this part is working) - If the VM is moved (live or relocate) by me or by the cluster because of a node failure. The VM stay on the new node. I will manually move it back after checking Regards, Here is some info abouth the cluster.conf Regards, (...) (...) From pmdyer at ctgcentral2.com Mon Feb 22 17:57:18 2010 From: pmdyer at ctgcentral2.com (Paul M. Dyer) Date: Mon, 22 Feb 2010 11:57:18 -0600 (CST) Subject: [Linux-cluster] Repeated fencing In-Reply-To: <1266855349.16694.11.camel@thor.seas.smu.edu> Message-ID: <3343105.11266861438483.JavaMail.root@athena> Crossover cable?????? With all the $$ spent, try putting a switch between the nodes. Paul ----- Original Message ----- From: "Doug Tucker" To: linux-cluster at redhat.com Sent: Monday, February 22, 2010 10:15:49 AM (GMT-0600) America/Chicago Subject: [Linux-cluster] Repeated fencing We have a 2 4.x cluster that has developed an issue we are unable to resolve. Starting back in December, the nodes began fencing each other randomly, and as frequently as once a day. There is nothing at the console prior to it happening, and nothing in the logs. We have not been able to develop any pattern to this point, the 2 nodes appear to be functioning fine, and suddenly in the logs a message will appear about "node x missed too many heartbeats" and the next thing you see is it fencing the node. Thinking we possibly had a hardware issue, we replaced both nodes from scratch with new machines, the problem persists. The cluster communication is done via a crossover cable on eth1 on both devices with private ip's. We have a 2nd cluster that is not having this issue, and both nodes have been up for over 160 days. The configuration is basically identical to the problematic cluster. The only difference between the 2 now is the newer hardware on the problematic node (prior, that was identical), and the kernel. The non-problematic cluster is still running kernel 89.0.9 and the problematic cluster is on 89.0.11. We are afraid at this point to allow our non problematic cluster upgrade to the latest packages. Any insight or advice would be greatly appreciated, we have exhausted our ideas here. Sincerely, Doug -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From andrew at beekhof.net Mon Feb 22 19:52:34 2010 From: andrew at beekhof.net (Andrew Beekhof) Date: Mon, 22 Feb 2010 20:52:34 +0100 Subject: [Linux-cluster] Looking for guidance In-Reply-To: References: Message-ID: On Sun, Feb 21, 2010 at 9:30 PM, Joseph L. Casale wrote: > Hey, > I am new to clusters, and have been reading up all the options but > given I haven't any experience ever setting one up I don't know which > option is best suited. > > I need to replicate a local volume between two servers, and export the > data via nfs and rsyncd from either. They only need to be active/passive. > > I originally thought drbd and heartbeat v1 would be the simplest but looking > at what's needed for nfs to work looks slightly hackish and I really don't > know how well this will perform (It could very well be the best option). I don;t think anyone on the project recommends Heartbeat v1 anymore. At least use the crm/pacemaker extensions (http://clusterlabs.org) if you're thinking about using Heartbeat in any way. > > After looking at alternatives, I figured the cluster suite from rhel might > be a good supported option but given the two node, active/passive setup > that looks like it is way over complicated. > > The need for fencing can be slightly mitigated by requiring manual intervention > to rejoin a node that has left, it's not active/active HA I am looking for, > but redundant data and weak ha through failover. The nodes are in two buildings > connected by gig fiber. > > Any experienced ops have any insight into an approach best suited that > I could settle on and start researching? > > Thx, > jlc > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From tuckerd at lyle.smu.edu Mon Feb 22 19:53:29 2010 From: tuckerd at lyle.smu.edu (Doug Tucker) Date: Mon, 22 Feb 2010 13:53:29 -0600 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <3343105.11266861438483.JavaMail.root@athena> References: <3343105.11266861438483.JavaMail.root@athena> Message-ID: <1266868409.16694.15.camel@thor.seas.smu.edu> We did. It's problematic when you need to reboot a switch or it goes down. They can't talk and try to fence each other. Crossover cable is a direct connection, actually far more efficient for what you are trying to accomplish. On Mon, 2010-02-22 at 11:57 -0600, Paul M. Dyer wrote: > Crossover cable?????? > > With all the $$ spent, try putting a switch between the nodes. > > Paul > > ----- Original Message ----- > From: "Doug Tucker" > To: linux-cluster at redhat.com > Sent: Monday, February 22, 2010 10:15:49 AM (GMT-0600) America/Chicago > Subject: [Linux-cluster] Repeated fencing > > We have a 2 4.x cluster that has developed an issue we are unable to > resolve. Starting back in December, the nodes began fencing each other > randomly, and as frequently as once a day. There is nothing at the > console prior to it happening, and nothing in the logs. We have not > been able to develop any pattern to this point, the 2 nodes appear to be > functioning fine, and suddenly in the logs a message will appear about > "node x missed too many heartbeats" and the next thing you see is it > fencing the node. Thinking we possibly had a hardware issue, we > replaced both nodes from scratch with new machines, the problem > persists. The cluster communication is done via a crossover cable on > eth1 on both devices with private ip's. We have a 2nd cluster that is > not having this issue, and both nodes have been up for over 160 days. > The configuration is basically identical to the problematic cluster. > The only difference between the 2 now is the newer hardware on the > problematic node (prior, that was identical), and the kernel. The > non-problematic cluster is still running kernel 89.0.9 and the > problematic cluster is on 89.0.11. We are afraid at this point to allow > our non problematic cluster upgrade to the latest packages. Any insight > or advice would be greatly appreciated, we have exhausted our ideas > here. > > Sincerely, > > Doug > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From jcasale at activenetwerx.com Mon Feb 22 20:28:01 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Mon, 22 Feb 2010 20:28:01 +0000 Subject: [Linux-cluster] Looking for guidance In-Reply-To: References: Message-ID: >I don;t think anyone on the project recommends Heartbeat v1 anymore. >At least use the crm/pacemaker extensions (http://clusterlabs.org) if >you're thinking about using Heartbeat in any way. Hi, That's a coincidence, I am just part way through your Fedora 11 doc using a couple of CentOS boxes. I stumbled on a few things but got them resolved: I see the conf file has changed, /etc/ais/openais.conf -> /etc/corosync/corosync.conf The init scripts need the export value changed for COROSYNC_DEFAULT_CONFIG_IFACE And it looks like the crm syntax might have changed a bit, I used crm configure location prefer-dev WebSite rule role=master 50: \#uname eq node-name instead of the shorter line you had. Lastly, the clusterlabs drbd rpm is a virtual package which yanks in xen? I just used the CentOS 8.3 packages. I am almost through it and intended on using ext3 instead of the clusterfs as only one node will ever be active, and I need to export this data via rsyncd and nfs. I am sure the rsyncd implementation will be trivial, but I am concerned about nfs. So far, the most complete coverage of this I have found is http://www.linux-ha.org/HaNFS . Is that a safe addition to the doc you wrote? Thanks very much for the effort on the doc, it certainly eased me into the field. jlc From lpleiman at redhat.com Mon Feb 22 20:42:08 2010 From: lpleiman at redhat.com (Leo Pleiman) Date: Mon, 22 Feb 2010 15:42:08 -0500 (EST) Subject: [Linux-cluster] Looking for guidance In-Reply-To: <1365448999.2035781266871295230.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Message-ID: <705642927.2035861266871328891.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Check this out... http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Configuration_Example_-_NFS_Over_GFS/index.html It provides nfs of the file system as a service to nodes outside the cluster. NFS from one node to the other is a real hack, you have nothing if the NFS server dies. The GFS(2) file system will need to be visible to both nodes (SAN, NAS, iSCSI). If you decide to use iSCSI, for the same reason as NFS, neither node should be the provider of the iSCSI target. You could further simplify by eliminating the GFS file system, use an ext3 (lvm) file system on a disk visible to both systems (SAN, NAS, or iSCSI) and managed by the cluster resource. You'll have to edit the lvm.conf to prevent clvd from starting the volume, allowing the cluster resource to manage it on only the active node. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Logical_Volume_Manager_Administration/index.html Appendix A from the link below (Apache example) illustrates shared storage managed by cluster resources. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Cluster_Administration/index.html Leo J Pleiman Senior Consultant Red Hat Consulting Services 410.688.3873 "Red Hat Ranked as #1 Software Vendor for Fifth Time in CIO Insight Study" ----- Original Message ----- From: "Andrew Beekhof" To: "linux clustering" Sent: Monday, February 22, 2010 2:52:34 PM GMT -05:00 US/Canada Eastern Subject: Re: [Linux-cluster] Looking for guidance On Sun, Feb 21, 2010 at 9:30 PM, Joseph L. Casale wrote: > Hey, > I am new to clusters, and have been reading up all the options but > given I haven't any experience ever setting one up I don't know which > option is best suited. > > I need to replicate a local volume between two servers, and export the > data via nfs and rsyncd from either. They only need to be active/passive. > > I originally thought drbd and heartbeat v1 would be the simplest but looking > at what's needed for nfs to work looks slightly hackish and I really don't > know how well this will perform (It could very well be the best option). I don;t think anyone on the project recommends Heartbeat v1 anymore. At least use the crm/pacemaker extensions (http://clusterlabs.org) if you're thinking about using Heartbeat in any way. > > After looking at alternatives, I figured the cluster suite from rhel might > be a good supported option but given the two node, active/passive setup > that looks like it is way over complicated. > > The need for fencing can be slightly mitigated by requiring manual intervention > to rejoin a node that has left, it's not active/active HA I am looking for, > but redundant data and weak ha through failover. The nodes are in two buildings > connected by gig fiber. > > Any experienced ops have any insight into an approach best suited that > I could settle on and start researching? > > Thx, > jlc > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From celsowebber at yahoo.com Mon Feb 22 21:40:37 2010 From: celsowebber at yahoo.com (Celso K. Webber) Date: Mon, 22 Feb 2010 13:40:37 -0800 (PST) Subject: [Linux-cluster] Repeated fencing In-Reply-To: <1266868409.16694.15.camel@thor.seas.smu.edu> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> Message-ID: <153982.12707.qm@web32107.mail.mud.yahoo.com> I endorse Doug's opinion. Although my opinion is empiric, I can afirm that a crossover (can be a straight cable in case of GigEthernet) is "more stable" than many Ethernet switches out there. Not to mention that sometimes the customer has only 100 Mbps ports, while using crossover cable you'll have GigE connections. So I'd like also to ask: is there officialy any known issues about using crossover cables instead of Ethernet switches for the private / heartbeat network? Thankks, Celso. ----- Original Message ---- From: Doug Tucker To: linux clustering Sent: Mon, February 22, 2010 4:53:29 PM Subject: Re: [Linux-cluster] Repeated fencing We did. It's problematic when you need to reboot a switch or it goes down. They can't talk and try to fence each other. Crossover cable is a direct connection, actually far more efficient for what you are trying to accomplish. On Mon, 2010-02-22 at 11:57 -0600, Paul M. Dyer wrote: > Crossover cable?????? > > With all the $$ spent, try putting a switch between the nodes. > > Paul > > ----- Original Message ----- > From: "Doug Tucker" > To: linux-cluster at redhat.com > Sent: Monday, February 22, 2010 10:15:49 AM (GMT-0600) America/Chicago > Subject: [Linux-cluster] Repeated fencing > > We have a 2 4.x cluster that has developed an issue we are unable to > resolve. Starting back in December, the nodes began fencing each other > randomly, and as frequently as once a day. There is nothing at the > console prior to it happening, and nothing in the logs. We have not > been able to develop any pattern to this point, the 2 nodes appear to be > functioning fine, and suddenly in the logs a message will appear about > "node x missed too many heartbeats" and the next thing you see is it > fencing the node. Thinking we possibly had a hardware issue, we > replaced both nodes from scratch with new machines, the problem > persists. The cluster communication is done via a crossover cable on > eth1 on both devices with private ip's. We have a 2nd cluster that is > not having this issue, and both nodes have been up for over 160 days. > The configuration is basically identical to the problematic cluster. > The only difference between the 2 now is the newer hardware on the > problematic node (prior, that was identical), and the kernel. The > non-problematic cluster is still running kernel 89.0.9 and the > problematic cluster is on 89.0.11. We are afraid at this point to allow > our non problematic cluster upgrade to the latest packages. Any insight > or advice would be greatly appreciated, we have exhausted our ideas > here. > > Sincerely, > > Doug > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From cmaiolino at redhat.com Mon Feb 22 22:33:43 2010 From: cmaiolino at redhat.com (Carlos Maiolino) Date: Mon, 22 Feb 2010 19:33:43 -0300 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <153982.12707.qm@web32107.mail.mud.yahoo.com> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> <153982.12707.qm@web32107.mail.mud.yahoo.com> Message-ID: <20100222223343.GA31846@andromeda.usersys.redhat.com> On Mon, Feb 22, 2010 at 01:40:37PM -0800, Celso K. Webber wrote: > I endorse Doug's opinion. > > Although my opinion is empiric, I can afirm that a crossover (can be a straight cable in case of GigEthernet) is "more stable" than many Ethernet switches out there. Not to mention that sometimes the customer has only 100 Mbps ports, while using crossover cable you'll have GigE connections. > > > So I'd like also to ask: is there officialy any known issues about using crossover cables instead of Ethernet switches for the private / heartbeat network? > > Thankks, Celso. > > > > ----- Original Message ---- > From: Doug Tucker > To: linux clustering > Sent: Mon, February 22, 2010 4:53:29 PM > Subject: Re: [Linux-cluster] Repeated fencing > > We did. It's problematic when you need to reboot a switch or it goes > down. They can't talk and try to fence each other. Crossover cable is > a direct connection, actually far more efficient for what you are trying > to accomplish. > > > On Mon, 2010-02-22 at 11:57 -0600, Paul M. Dyer wrote: > > Crossover cable?????? > > > > With all the $$ spent, try putting a switch between the nodes. > > > > Paul > > > > ----- Original Message ----- > > From: "Doug Tucker" > > To: linux-cluster at redhat.com > > Sent: Monday, February 22, 2010 10:15:49 AM (GMT-0600) America/Chicago > > Subject: [Linux-cluster] Repeated fencing > > > > We have a 2 4.x cluster that has developed an issue we are unable to > > resolve. Starting back in December, the nodes began fencing each other > > randomly, and as frequently as once a day. There is nothing at the > > console prior to it happening, and nothing in the logs. We have not > > been able to develop any pattern to this point, the 2 nodes appear to be > > functioning fine, and suddenly in the logs a message will appear about > > "node x missed too many heartbeats" and the next thing you see is it > > fencing the node. Thinking we possibly had a hardware issue, we > > replaced both nodes from scratch with new machines, the problem > > persists. The cluster communication is done via a crossover cable on > > eth1 on both devices with private ip's. We have a 2nd cluster that is > > not having this issue, and both nodes have been up for over 160 days. > > The configuration is basically identical to the problematic cluster. > > The only difference between the 2 now is the newer hardware on the > > problematic node (prior, that was identical), and the kernel. The > > non-problematic cluster is still running kernel 89.0.9 and the > > problematic cluster is on 89.0.11. We are afraid at this point to allow > > our non problematic cluster upgrade to the latest packages. Any insight > > or advice would be greatly appreciated, we have exhausted our ideas > > here. > > > > Sincerely, > > > > Doug > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Hi Doug, maybe you can avoid this kind of problem using a quorumdisk partition. a two node cluster is split-brain prone and with a quorumdisk partition you can avoid split-brain situations, which probably is causing this behavior. So, about use a cross-over (or straight) cable, I don't know any issue about it, but, try to check if it's using full-duplex mode. half-duplex mode on cross-over linked machines probably will cause heartbeat problems. cya.. -- --- Best Regards Carlos Eduardo Maiolino Support engineer Red Hat - Global Support Services From jcasale at activenetwerx.com Mon Feb 22 22:45:50 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Mon, 22 Feb 2010 22:45:50 +0000 Subject: [Linux-cluster] Looking for guidance In-Reply-To: <705642927.2035861266871328891.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> References: <1365448999.2035781266871295230.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> <705642927.2035861266871328891.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Message-ID: >Check this out... > >http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Configuration_Example_-_NFS_Over_GFS/index.html Great article, pretty much what I want to do with only two nodes though. >It provides nfs of the file system as a service to nodes outside the cluster. NFS from one node to the other is a real hack, you have nothing if the NFS server dies You got me here: So after reading Chp 9 I see my concerns are common. As this cluster has clients connect to only one ip that it moves around (Chp 1), not considering the situation where we move away and back, how does a client pick up where it left when the service is migrated from one cluster node to another? What modifications, if any, does the cluster suite do to nfs to mitigate this? All the nfs ha docs I see make changes to /var/lib/nfs which none of the RHEL docs corroborate. Is the re-exportation of the filesystem via nfs on a new node a potential data loss situation for the client such that any handles would no longer be valid, so a write in progress is certainly incomplete? I guess that's left to the client to mitigate, all we do is provide uptime at the cost of the momentary interruption? Thanks for all the patience and guidance while I wrap my head around this guys! jlc From raju.rajsand at gmail.com Tue Feb 23 01:17:04 2010 From: raju.rajsand at gmail.com (Rajagopal Swaminathan) Date: Tue, 23 Feb 2010 06:47:04 +0530 Subject: [Linux-cluster] Looking for guidance In-Reply-To: References: <8786b91c1002220340s5521f8bh8863c71017123a5b@mail.gmail.com> Message-ID: <8786b91c1002221717i3d6a61bevbd4ab5f7b9a34f08@mail.gmail.com> Greetings, On Mon, Feb 22, 2010 at 9:05 PM, Joseph L. Casale wrote: > how does it mitigate the > handling of stale filehandles in nfs? It doesn't mention anything about > moving the nfs dirs into the shared storage? > That's elementary, use udp than the default tcp for nfs share. ;) Regards Rajagopal From bernardchew at gmail.com Tue Feb 23 03:05:16 2010 From: bernardchew at gmail.com (Bernard Chew) Date: Tue, 23 Feb 2010 11:05:16 +0800 Subject: [Linux-cluster] Virtual machine fence fail question Message-ID: <95994e3c1002221905t7a6ccd8dg3eee1439979c62d3@mail.gmail.com> Hi, Given I have 2 Red Hat Clusters; 1 cluster consisting of physical hosts and another consisting of virtual guests which are hosted in the physical hosts. The physical host cluster uses DRAC fencing while the virtual guest cluster uses virtual machine fencing. If a physical host goes down, I saw that DRAC fencing takes place successfully but fencing fail for the virtual guests on the physical host which go down (together). Does the virtual machine fencing fails because the virtual guests are no longer available? How can I configure fencing so that both physical hosts and virtual guests are fenced correctly? Thank you in advance. Regards, Bernard From andrew at beekhof.net Tue Feb 23 07:27:10 2010 From: andrew at beekhof.net (Andrew Beekhof) Date: Tue, 23 Feb 2010 08:27:10 +0100 Subject: [Linux-cluster] Looking for guidance In-Reply-To: References: Message-ID: On Mon, Feb 22, 2010 at 9:28 PM, Joseph L. Casale wrote: >>I don;t think anyone on the project recommends Heartbeat v1 anymore. >>At least use the crm/pacemaker extensions (http://clusterlabs.org) if >>you're thinking about using Heartbeat in any way. > > Hi, > That's a coincidence, I am just part way through your Fedora 11 doc using > a couple of CentOS boxes. I stumbled on a few things but got them resolved: > > I see the conf file has changed, > /etc/ais/openais.conf -> /etc/corosync/corosync.conf Nod, I've got an F-12 version that I need to upload at some point (which uses corosync). We didn't support corosync at the time I wrote the F-11 version. > > The init scripts need the export value changed for COROSYNC_DEFAULT_CONFIG_IFACE > > And it looks like the crm syntax might have changed a bit, I used > crm configure location prefer-dev WebSite rule role=master 50: \#uname eq node-name > instead of the shorter line you had. This old form will still work though, same effect. > > Lastly, the clusterlabs drbd rpm is a virtual package which yanks in xen? > I just used the CentOS 8.3 packages. Yep, don't use the drbd packages from clusterlabs. They were an aborted attempt to make it easy for people to install but it didnt get very far. /me makes a note to remove them > > I am almost through it and intended on using ext3 instead of the clusterfs as > only one node will ever be active, and I need to export this data via rsyncd > and nfs. I am sure the rsyncd implementation will be trivial, but I am concerned > about nfs. So far, the most complete coverage of this I have found is > http://www.linux-ha.org/HaNFS . Is that a safe addition to the doc you wrote? [quote] crm_resource -r drbdgroup -U crm_mon; sleep 5; crm_mon [/quote] Seems to have been written for the crm, so yes, it is safe. > > Thanks very much for the effort on the doc, it certainly eased me into the field. Glad it helped :-) From kkovachev at varna.net Tue Feb 23 07:37:14 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Tue, 23 Feb 2010 09:37:14 +0200 Subject: [Linux-cluster] Looking for guidance In-Reply-To: <705642927.2035861266871328891.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> References: <1365448999.2035781266871295230.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> <705642927.2035861266871328891.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Message-ID: <20100223073024.M9764@varna.net> On Mon, 22 Feb 2010 15:42:08 -0500 (EST), Leo Pleiman wrote > Check this out... > > http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Configuration_Example_-_NFS_Over_GFS/index.html > > It provides nfs of the file system as a service to nodes outside the cluster. NFS from one node to the other is a real hack, you have nothing if the NFS server dies. The GFS(2) file system will need to be visible to both nodes (SAN, NAS, iSCSI). If you decide to use iSCSI, for the same reason as NFS, neither node should be the provider of the iSCSI target. neither or both with multipath :) and it actually works quite well. If the two iSCSI targets are cluster members it is OK to even use them as Active/Active and in both failover or multibus mode with GFS (not sure/tested about ext3) > You could further simplify by eliminating the GFS file system, use an ext3 (lvm) file system on a disk visible to both systems (SAN, NAS, or iSCSI) and managed by the cluster resource. You'll have to edit the lvm.conf to prevent clvd from starting the volume, allowing the cluster resource to manage it on only the active node. > > http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Logical_Volume_Manager_Administration/index.html > > Appendix A from the link below (Apache example) illustrates shared storage managed by cluster resources. > > http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Cluster_Administration/index.html > > Leo J Pleiman > Senior Consultant > Red Hat Consulting Services > 410.688.3873 > > "Red Hat Ranked as #1 Software Vendor for Fifth Time in CIO Insight Study" > > ----- Original Message ----- > From: "Andrew Beekhof" > To: "linux clustering" > Sent: Monday, February 22, 2010 2:52:34 PM GMT -05:00 US/Canada Eastern > Subject: Re: [Linux-cluster] Looking for guidance > > On Sun, Feb 21, 2010 at 9:30 PM, Joseph L. Casale > wrote: > > Hey, > > I am new to clusters, and have been reading up all the options but > > given I haven't any experience ever setting one up I don't know which > > option is best suited. > > > > I need to replicate a local volume between two servers, and export the > > data via nfs and rsyncd from either. They only need to be active/passive. > > > > I originally thought drbd and heartbeat v1 would be the simplest but looking > > at what's needed for nfs to work looks slightly hackish and I really don't > > know how well this will perform (It could very well be the best option). > > I don;t think anyone on the project recommends Heartbeat v1 anymore. > At least use the crm/pacemaker extensions (http://clusterlabs.org) if > you're thinking about using Heartbeat in any way. > > > > > After looking at alternatives, I figured the cluster suite from rhel might > > be a good supported option but given the two node, active/passive setup > > that looks like it is way over complicated. > > > > The need for fencing can be slightly mitigated by requiring manual intervention > > to rejoin a node that has left, it's not active/active HA I am looking for, > > but redundant data and weak ha through failover. The nodes are in two buildings > > connected by gig fiber. > > > > Any experienced ops have any insight into an approach best suited that > > I could settle on and start researching? > > > > Thx, > > jlc > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From fdinitto at redhat.com Tue Feb 23 08:22:27 2010 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Tue, 23 Feb 2010 09:22:27 +0100 Subject: [Linux-cluster] Cluster 3.0.8 stable release Message-ID: <4B839043.5080601@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The cluster team and its community are proud to announce the 3.0.8 stable release from the STABLE3 branch. This release contains several major bug fixes. We strongly recommend people to update your clusters. In order to build/run the 3.0.8 release you will need: - - corosync 1.2.0 - - openais 1.1.2 - - linux kernel 2.6.31 (only for GFS1 users) The new source tarball can be downloaded here: https://fedorahosted.org/releases/c/l/cluster/cluster-3.0.8.tar.bz2 As previously announced (https://www.redhat.com/archives/linux-cluster/2010-February/msg00090.html) Changelog, sha256, gpg signature and tar.gz files are also available. To report bugs or issues: https://bugzilla.redhat.com/ Would you like to meet the cluster team or members of its community? Join us on IRC (irc.freenode.net #linux-cluster) and share your experience with other sysadministrators or power users. Thanks/congratulations to all people that contributed to achieve this great milestone. Happy clustering, Fabio Under the hood (from 3.0.7): Abhijith Das (7): libgfs2: Bug 459630 - GFS2: changes needed to gfs2-utils due to gfs2meta fs changes in bz 457798 gfs_jadd: Bug 555363 - gfs_jadd does not resolve symbolic links gfs2_convert: gfs2_convert should fix statfs file mount.gfs2: Better error reporting when mounting a gfs fs without enough journals Merge branch 'STABLE3' of ssh://git.fedoraproject.org/git/cluster into mySTABLE3 gfs-kernel: Flock on GFS fs file will error with "Resource tempory unavailable" for EWOULDBLOCK gfs2_convert: gfs2_convert doesn't convert jdata files correctly Bob Peterson (61): Remove nvbuf_list and use fewer buffers Eliminate bad_block linked block list Simplify bitmap/block list structures Streamline the bitmap code by always using 4-bit size per block Misc blocklist optimizations Separate eattr_block list from the rest for efficiency gfs2: remove update_flags everywhere fsck.gfs2: give comfort when processing lots of data blocks fsck.gfs2: make query() count errors_found, errors_fixed Attach buffers to rgrp_list structs Make struct_out functions operate on bh's Attach bh's to inodes gfs2: Remove buf_lists fsck.gfs2: Verify rgrps free space against bitmap libgfs2: Consistent naming for blockmap functions Move duplicate code from libgfs2 to fsck.gfs2 libgfs2, fsck.gfs2: simplify block_query code gfs2: libgfs2 and fsck.gfs2 cleanups libgfs2: fs_bits speed up bitmap operations libgfs2: gfs2_log reform fsck.gfs2: convert dup_list to a rbtree fsck.gfs2: convert dir_info list to rbtree fsck.gfs2: convert inode hash to rbtree fsck.gfs2: pass1 should use gfs2_special_add not _set libgfs2: Remove unneeded sdp parameter in gfs2_block_set libgfs2: dir_split_leaf needs to zero out the new leaf libgfs2: dir_split_leaf needs to check for allocation failure libgfs2: Set block range based on rgrps, not device size fsck.gfs2: should use the libgfs2 is_system_directory fsck.gfs2: Journal replay should report what it's doing fsck.gfs2: fix directories that have odd number of pointers. libgfs2: Get rid of useless constants fsck.gfs2: link.c should log why it's making a change for debugging fsck.gfs2: Enforce consistent behavior in directory processing fsck.gfs2: enforce consistency between bitmap and blockmap fsck.gfs2: metawalk needs to check for no valid leaf blocks fsck.gfs2: metawalk was not checking many directories fsck.gfs2: separate check_data function in check_metatree lost+found link count and connections were not properly managed fsck.gfs2: reprocess lost+found and other inode metadata when blocks are added Misc cleanups fsck.gfs2: Check for massive amounts of pointer corruption fsck.gfs2: use gfs2_meta_inval vs. gfs2_inval_inode Eliminate unnecessary block_list from gfs2_edit fsck.gfs2: rename gfs2_meta_other to gfs2_meta_rgrp. Create a standard metadata delete interface fsck.gfs2: cleanup: refactor pass3 fsck.gfs2: Make pass1 undo its work for unrecoverable inodes fsck.gfs2: Overhaul duplicate reference processing fsck.gfs2: invalidate invalid mode inodes fsck.gfs2: Force intermediate lost+found inode updates fsck.gfs2: Free metadata list memory we don't need fsck.gfs2: Don't add extended attrib blocks to list twice fsck.gfs2: small parameter passing optimization fsck.gfs2: Free, don't invalidate, dinodes with bad depth Misc cleanups fsck.gfs2: If journal replay fails, give option to reinitialize journal Fix white space errors fsck.gfs2 fails on root fs: Device X is busy. gfs2_edit savemeta: Don't release indirect buffers too soon fsck.gfs2: Use fsck.ext3's method of dealing with root mounts Christine Caulfield (4): cman: use the typed objdb calls cman: don't set token_retransmits_before_loss_const cman: disable gfs plock_ownership when upgrading config: Add schema entry for clvmd David Teigland (15): man pages: cluster.conf cluster.rng: updates man pages: fence_node, fenced man pages: dlm_controld cluster.rng: dlm updates man pages: groupd man pages: fenced man pages: group_tool cluster.rng: group/groupd_compat man pages: gfs_controld cluster.rng: gfs_controld cluster.rng: fence, fencedevices man pages: dlm_tool man pages: gfs_control dlm_controld: check all messages against enable options Dyna Ares (1): config: Make broadcast attr reflect documentation Fabio M. Di Nitto (10): release: don't build gfs-utils tarball fence agents: man page clean up cman init: propagate errors from fence_tool operations gfs2: make init script LSB compliant fence agents: fix several agents build dlm_controld: fix linking nss: fix linking build: fix out-of-tree build of fence agents release script rework logrotate: fix logrotate default actions and set sane defaults Jonathan E. Brassow (1): rgmanager: halvm: Check ownership before stripping tags Lon Hohberger (17): config: Make nodeid attribute required config: Make nodeid required in ldif schema config: Fix license for value-list.[ch] config: Sync LDIF w/ cluster.rng rgmanager: isAlive error logging for file systems config: Add fence_virt to cluster.rng config: Update LDIF schema based on recent RelaxNG changes rgmanager: Make relocate-to-offline consistent qdisk: Fix logt_print which used to be perror() resource-agents: SAPDatabase: remove $TEMPFILE qdisk: Autoconfigure default timings Revert "qdisk: Autoconfigure default timings" rgmanager: Make VF timeout scale with token timeout rgmanager: Clean up build warnings qdisk: Autoconfigure default timings qdiskd: Autoconfigure votes based on node count qdisk: Fix uninitialized variable Marek 'marx' Grac (2): fencing: Add vendor URL to man pages resource agents: Handle multiline pid files Ryan O'Hara (2): Remove open3 calls and replace with simple qx commands. This avoids Always remove leading zeros from key value. Shane Bradley (1): resource-agents: Kill correct PIDs during force_unmount Tatsuo Kawasaki (1): qdisk: mkqdisk argument positioning cman/cman_tool/join.c | 17 +- cman/daemon/cman-preconfig.c | 200 ++-- cman/init.d/cman.in | 6 +- cman/man/qdisk.5 | 35 +- cman/qdisk/disk.c | 4 +- cman/qdisk/disk.h | 2 +- cman/qdisk/main.c | 81 ++- cman/qdisk/mkqdisk.c | 18 +- cman/qdisk/proc.c | 4 +- config/libs/libccsconfdb/libccs.c | 33 +- config/man/cluster.conf.5 | 327 ++++--- config/plugins/ldap/99cluster.ldif | 111 ++- config/plugins/ldap/configldap.c | 5 +- config/plugins/ldap/ldap-base.csv | 9 +- config/plugins/xml/config.c | 6 +- config/tools/ldap/rng2ldif/value-list.c | 1 - config/tools/ldap/rng2ldif/value-list.h | 1 - config/tools/xml/cluster.rng.in | 591 +++++++------ dlm/man/dlm_tool.8 | 97 ++- doc/cluster.logrotate.in | 7 +- fence/agents/alom/Makefile | 2 +- fence/agents/alom/fence_alom.py | 1 + fence/agents/apc/Makefile | 2 +- fence/agents/apc/fence_apc.py | 1 + fence/agents/apc_snmp/Makefile | 2 +- fence/agents/apc_snmp/fence_apc_snmp.py | 1 + fence/agents/baytech/fence_baytech.8 | 84 ++ fence/agents/bladecenter/Makefile | 2 +- fence/agents/bladecenter/fence_bladecenter.py | 1 + fence/agents/brocade/fence_brocade.8 | 84 ++ fence/agents/bullpap/fence_bullpap.8 | 73 ++ fence/agents/cisco_mds/Makefile | 2 +- fence/agents/cisco_mds/fence_cisco_mds.py | 1 + fence/agents/cpint/fence_cpint.8 | 54 ++ fence/agents/drac/fence_drac.8 | 99 ++ fence/agents/drac5/Makefile | 2 +- fence/agents/drac5/fence_drac5.py | 1 + fence/agents/egenera/fence_egenera.8 | 72 ++ fence/agents/eps/Makefile | 2 +- fence/agents/eps/fence_eps.py | 1 + fence/agents/ibmblade/Makefile | 2 +- fence/agents/ibmblade/fence_ibmblade.py | 1 + fence/agents/ifmib/Makefile | 2 +- fence/agents/ifmib/fence_ifmib.py | 1 + fence/agents/ilo/Makefile | 2 +- fence/agents/ilo/fence_ilo.py | 1 + fence/agents/ilo_mp/Makefile | 2 +- fence/agents/ilo_mp/fence_ilo_mp.py | 1 + fence/agents/intelmodular/Makefile | 2 +- fence/agents/intelmodular/fence_intelmodular.py | 1 + fence/agents/ldom/Makefile | 2 +- fence/agents/ldom/fence_ldom.py | 1 + fence/agents/lib/fence2man.xsl | 3 + fence/agents/lib/fencing.py.py | 2 + fence/agents/lpar/Makefile | 2 +- fence/agents/manual/Makefile | 2 + fence/agents/manual/fence_ack_manual.8 | 39 + fence/agents/mcdata/fence_mcdata.8 | 84 ++ fence/agents/nss_wrapper/Makefile | 2 +- fence/agents/rackswitch/Makefile | 2 + fence/agents/rackswitch/fence_rackswitch.8 | 70 ++ fence/agents/rsa/Makefile | 2 +- fence/agents/rsa/fence_rsa.py | 1 + fence/agents/rsb/fence_rsb.8 | 76 ++ fence/agents/sanbox2/Makefile | 2 +- fence/agents/sanbox2/fence_sanbox2.py | 1 + fence/agents/scsi/fence_scsi.8 | 109 +++ fence/agents/scsi/fence_scsi.pl | 145 +--- fence/agents/virsh/Makefile | 2 +- fence/agents/vixel/fence_vixel.8 | 72 ++ fence/agents/vmware/Makefile | 1 + fence/agents/vmware/fence_vmware.py | 1 + fence/agents/wti/Makefile | 2 +- fence/agents/wti/fence_wti.py | 1 + fence/agents/xcat/fence_xcat.8 | 63 ++ fence/agents/xvm/Makefile | 2 + fence/agents/xvm/fence_xvmd.8 | 124 +++ fence/agents/zvm/fence_zvm.8 | 64 ++ fence/man/Makefile | 20 - fence/man/fence_ack_manual.8 | 39 - fence/man/fence_baytech.8 | 82 -- fence/man/fence_brocade.8 | 82 -- fence/man/fence_bullpap.8 | 71 -- fence/man/fence_cpint.8 | 52 - fence/man/fence_drac.8 | 97 -- fence/man/fence_egenera.8 | 70 -- fence/man/fence_mcdata.8 | 82 -- fence/man/fence_node.8 | 34 +- fence/man/fence_rackswitch.8 | 68 -- fence/man/fence_rib.8 | 10 - fence/man/fence_rsb.8 | 75 -- fence/man/fence_scsi.8 | 109 --- fence/man/fence_vixel.8 | 70 -- fence/man/fence_xcat.8 | 61 -- fence/man/fence_xvmd.8 | 124 --- fence/man/fence_zvm.8 | 62 -- fence/man/fenced.8 | 196 +++-- gfs-kernel/src/gfs/glock.c | 74 ++- gfs-kernel/src/gfs/glock.h | 2 + gfs-kernel/src/gfs/incore.h | 2 + gfs-kernel/src/gfs/ops_file.c | 15 +- gfs-kernel/src/gfs/ops_fstype.c | 2 +- gfs/gfs_jadd/main.c | 15 +- gfs2/convert/gfs2_convert.c | 494 ++++++++--- gfs2/edit/gfs2hex.c | 58 +- gfs2/edit/gfs2hex.h | 2 +- gfs2/edit/hexedit.c | 324 ++++---- gfs2/edit/hexedit.h | 13 +- gfs2/edit/savemeta.c | 139 ++-- gfs2/fsck/eattr.c | 52 +- gfs2/fsck/eattr.h | 9 +- gfs2/fsck/fs_recovery.c | 155 ++-- gfs2/fsck/fsck.h | 87 ++- gfs2/fsck/initialize.c | 284 +++++-- gfs2/fsck/inode_hash.c | 95 +- gfs2/fsck/inode_hash.h | 9 +- gfs2/fsck/link.c | 92 +- gfs2/fsck/link.h | 8 +- gfs2/fsck/lost_n_found.c | 149 ++-- gfs2/fsck/main.c | 101 +- gfs2/fsck/metawalk.c | 1140 ++++++++++++++--------- gfs2/fsck/metawalk.h | 64 +- gfs2/fsck/pass1.c | 1019 ++++++++++++--------- gfs2/fsck/pass1b.c | 682 ++++++++------ gfs2/fsck/pass1c.c | 78 +- gfs2/fsck/pass2.c | 717 +++++++-------- gfs2/fsck/pass3.c | 252 +++--- gfs2/fsck/pass4.c | 143 ++-- gfs2/fsck/pass5.c | 78 +-- gfs2/fsck/rgrepair.c | 90 +-- gfs2/fsck/util.c | 342 +++++++- gfs2/fsck/util.h | 21 +- gfs2/include/osi_tree.h | 395 ++++++++ gfs2/init.d/gfs2.in | 155 ++-- gfs2/libgfs2/block_list.c | 290 +------ gfs2/libgfs2/buf.c | 224 +---- gfs2/libgfs2/fs_bits.c | 108 ++- gfs2/libgfs2/fs_geometry.c | 22 +- gfs2/libgfs2/fs_ops.c | 446 ++++++---- gfs2/libgfs2/gfs1.c | 82 ++- gfs2/libgfs2/gfs2_log.c | 34 +- gfs2/libgfs2/libgfs2.h | 340 +++---- gfs2/libgfs2/misc.c | 126 ++- gfs2/libgfs2/ondisk.c | 136 ++-- gfs2/libgfs2/recovery.c | 16 +- gfs2/libgfs2/rgrp.c | 44 +- gfs2/libgfs2/structures.c | 61 +- gfs2/libgfs2/super.c | 30 +- gfs2/mkfs/main_grow.c | 14 +- gfs2/mkfs/main_jadd.c | 8 +- gfs2/mkfs/main_mkfs.c | 14 +- gfs2/mount/mount.gfs2.c | 2 + gfs2/quota/main.c | 6 +- gfs2/tool/sb.c | 6 +- group/dlm_controld/Makefile | 1 + group/dlm_controld/cpg.c | 57 +- group/man/Makefile | 1 + group/man/dlm_controld.8 | 320 +++++-- group/man/gfs_control.8 | 44 + group/man/gfs_controld.8 | 236 +++-- group/man/group_tool.8 | 134 ++-- group/man/groupd.8 | 107 +-- make/fencebuild.mk | 11 + make/release.mk | 317 ++++--- rgmanager/include/vf.h | 2 +- rgmanager/src/clulib/vft.c | 7 +- rgmanager/src/daemons/main.c | 21 +- rgmanager/src/daemons/rg_state.c | 21 +- rgmanager/src/resources/SAPDatabase | 52 +- rgmanager/src/resources/clusterfs.sh | 15 +- rgmanager/src/resources/fs.sh.in | 40 +- rgmanager/src/resources/lvm_by_lv.sh | 9 + rgmanager/src/resources/utils/ra-skelet.sh | 8 +- scripts/fenceparse | 4 +- 174 files changed, 8600 insertions(+), 6511 deletions(-) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJLg5BAAAoJEFA6oBJjVJ+OHwkQAKtu2bZaf+cTumTkkKkjYBpX unuDCJ9Bi/Rn8nocBaye2ansJZQhelygh1VjvgVYqzseDSayBVv3k4+GOe/Xs6yN YUJwe3OPdBR5jLkjiXclLokmyplWG6RMQu2CpzxFGINnl1eGXEk/KRYGNMa2z2GG YGrc7cQ4kscNdi70SFnVfxhu4fUIfF16j32sve40NQyjBdJRTjRhvktl8E2oZr0D vNHu+z9hGkxFU1KxoOGBcX7MacyBSeEmIg/srnIV5Yzu9bqEtLzGvKQQ26aV83PG oKltGy+w3PRvsrPXzWGvzNzOKsS/BhT71IRuxgKzYTgyMoa+ZM0zWMLVSu0USPOH IDXAdck2MCwThgup5RQ8fEbojjpdvwpx1LY624H4nP03Gway1HLdN3DgWi3O7Jj7 dCQENP8vXAzOX3bmEK4Is0REdS+5Hu3QooGNbOfwgNFFqyzJ/1IMiEonRq/2WoI0 Hv61Hw1NOv4/VKTaZrknCK+S0wAjBryH6RHW6pJrsa79mM+FOoAJoD1/CuhfDWR4 QpNxQjRAyixAFPLMdR9aCh+3ybmyqhf/UsO0/18hd22UQ8kuKD05LssI0W3XIsIC NbtQZSXpYqH7xLFqoFV+5pFwoaGH5g6b0XDg68JeVglEzeAcJi4x9hfyntBU83cL qZvYLFS49oICRhUlS1aA =Edqm -----END PGP SIGNATURE----- From tuckerd at lyle.smu.edu Tue Feb 23 17:13:33 2010 From: tuckerd at lyle.smu.edu (Doug Tucker) Date: Tue, 23 Feb 2010 11:13:33 -0600 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <20100222223343.GA31846@andromeda.usersys.redhat.com> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> <153982.12707.qm@web32107.mail.mud.yahoo.com> <20100222223343.GA31846@andromeda.usersys.redhat.com> Message-ID: <1266945213.23528.0.camel@thor.seas.smu.edu> > Hi Doug, maybe you can avoid this kind of problem using a quorumdisk partition. a two node cluster is split-brain prone and with a quorumdisk partition you can avoid split-brain situations, which probably is causing this behavior. > > So, about use a cross-over (or straight) cable, I don't know any issue about it, but, try to check if it's using full-duplex mode. half-duplex mode on cross-over linked machines probably will cause heartbeat problems. > > cya.. Can you give me a little info about "split-brain" issues? I don't understand what you mean by that, and what I'm solving with a quorumdisk. And it has always worked fine, it just started happening after the install of this newer kernel/gfs module. The other 2 node cluster is still rock solid. Also, the network interfaces on the crossover are full duplex. Thanks for writing back, you're the first person who offered anything. From cmaiolino at redhat.com Tue Feb 23 17:39:39 2010 From: cmaiolino at redhat.com (Carlos Maiolino) Date: Tue, 23 Feb 2010 14:39:39 -0300 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <1266945213.23528.0.camel@thor.seas.smu.edu> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> <153982.12707.qm@web32107.mail.mud.yahoo.com> <20100222223343.GA31846@andromeda.usersys.redhat.com> <1266945213.23528.0.camel@thor.seas.smu.edu> Message-ID: <20100223173939.GA14976@andromeda.usersys.redhat.com> On Tue, Feb 23, 2010 at 11:13:33AM -0600, Doug Tucker wrote: > > Hi Doug, maybe you can avoid this kind of problem using a quorumdisk partition. a two node cluster is split-brain prone and with a quorumdisk partition you can avoid split-brain situations, which probably is causing this behavior. > > > > So, about use a cross-over (or straight) cable, I don't know any issue about it, but, try to check if it's using full-duplex mode. half-duplex mode on cross-over linked machines probably will cause heartbeat problems. > > > > cya.. > > Can you give me a little info about "split-brain" issues? I don't > understand what you mean by that, and what I'm solving with a > quorumdisk. And it has always worked fine, it just started happening > after the install of this newer kernel/gfs module. The other 2 node > cluster is still rock solid. Also, the network interfaces on the > crossover are full duplex. Thanks for writing back, you're the first > person who offered anything. > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Hi Doug, a split-brain is a situation when (in this case) the cluster communication channel are interrupted (dua any situation), and each node thinks that "it is the remaining node, and the another node is offline. So, with quroumdisk, you'll have a second "heartbeat" system, where both nodes will have a common partition and write metadata information to it. In case a heartbeat fail (cross-over cable in your environment), the cluster will still check if the quorumdisk is still available, and take decisions based on qdisk configuration. Sorry for this poor explainin, I really needed to write it as fast as I can, so, you can check these documents for help and a well understanding about qdisk http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Cluster_Administration/s1-qdisk-considerations-CA.html http://www.linuxdynasty.org/howto-setup-a-quorum-disk.html any doubts ask us.. -- --- Best Regards Carlos Eduardo Maiolino Support engineer Red Hat - Global Support Services From jcasale at activenetwerx.com Tue Feb 23 17:52:24 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Tue, 23 Feb 2010 17:52:24 +0000 Subject: [Linux-cluster] Distributed Storage in Cluster Suite Message-ID: With the vendor provided packages such as gfs etc, is it possible to accomplish a drbd like setup so that the nodes of a cluster house the shared storage themselves? I gather from my reading that is no, but just thought I would check. Thanks! jlc From gordan at bobich.net Tue Feb 23 19:01:58 2010 From: gordan at bobich.net (Gordan Bobic) Date: Tue, 23 Feb 2010 19:01:58 +0000 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: References: Message-ID: <4B842626.60800@bobich.net> Joseph L. Casale wrote: > With the vendor provided packages such as gfs etc, is it possible > to accomplish a drbd like setup so that the nodes of a cluster house > the shared storage themselves? > > I gather from my reading that is no, but just thought I would check. That largely depends on what exactly you want to do. You could export storage space via iSCSI and apply CLVM on top of that. IIRC CLVM can now handle mirroring. It's nowhere nearly as good as DRBD, though - that provides a very comprehensive solution for most eventualities you might face, including post-outage resync. Why, exactly, would you want to limit yourself to only what is in the distro? DRBD builds cleanly into RPM packages, I use it on a number of systems with GFS on top. Gordan From jcasale at activenetwerx.com Tue Feb 23 19:55:13 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Tue, 23 Feb 2010 19:55:13 +0000 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: <4B842626.60800@bobich.net> References: <4B842626.60800@bobich.net> Message-ID: >That largely depends on what exactly you want to do. >You could export storage space via iSCSI and apply CLVM on top of that. >IIRC CLVM can now handle mirroring. > >It's nowhere nearly as good as DRBD, though - that provides a very >comprehensive solution for most eventualities you might face, including >post-outage resync. > >Why, exactly, would you want to limit yourself to only what is in the >distro? DRBD builds cleanly into RPM packages, I use it on a number of >systems with GFS on top. Hi Gordan, I'm under the gun to get something operational asap and have been reading like mad but given the severity of getting it wrong, I want some experience under my belt before I step off the reservation. I don't have the infrastructure available to export an iscsi device, it's the backend I need redundant just the way drbd would provide. I am ok with drbd, but its integration with rhcs concerns me as I don't really know how the init scripts get constructed. I saw an archive post about the wrapper for /usr/share/cluster/drbd.sh to hand it the resource name. I can assume if a node tanks, drbd goes primary on the remaining node, everything is good. Looking through drbd.sh I see case's for promote/demote, I added this (the wrapper actually) as a resource "script" with a child "fs" in a mock cluster as the underlying device was primary on that node, it just started:) How does rhcs know when adding a "script" to pass the required cases? How do I configure this? Thanks guys, it's been a mountain of info the last few days so bear with me! jlc From lhh at redhat.com Tue Feb 23 20:02:27 2010 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 23 Feb 2010 15:02:27 -0500 Subject: [Linux-cluster] independent subtree In-Reply-To: <4B7DE0C0.4060601@gmail.com> References: <20100213145541.M89820@varna.net> <1266525906.23427.1718.camel@localhost.localdomain> <4B7DE0C0.4060601@gmail.com> Message-ID: <1266955347.23520.22.camel@localhost.localdomain> On Thu, 2010-02-18 at 18:52 -0600, Dustin Henry Offutt wrote: > Hey there Lon, > > May I ask you what exactly it is that provisioning > _independent_subtrees does? Haven't seen it documented (and have read > voraciously...). > > A sincere thank you to you, Sorry for the delay -- http://sources.redhat.com/cluster/wiki/ResourceTrees There's a section called "Failure Recovery & Independent Subtrees" -- Lon From lhh at redhat.com Tue Feb 23 20:03:13 2010 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 23 Feb 2010 15:03:13 -0500 Subject: [Linux-cluster] updating nodes and failback oprion In-Reply-To: <1566077.391.1266856701008.JavaMail.root@vicenza.dmz.lexum.pri> References: <1566077.391.1266856701008.JavaMail.root@vicenza.dmz.lexum.pri> Message-ID: <1266955393.23520.23.camel@localhost.localdomain> On Mon, 2010-02-22 at 11:38 -0500, F M wrote: > Hello, > I have a 3 Xen nodes cluster + GFS2 (SAN). All VM are in the GFS2 FS. > OS: just updated to RH 5.4 > > During the Dom0 update of one of the nodes I used live migration to move VMs to another node. After the update, I rebooted the server. When it came back VMs tried to go back to his preferred node. > > I'd like to move back the V.M. manually instead, is nofailback=1 would do the trick ? Yes. -- Lon From lhh at redhat.com Tue Feb 23 20:05:11 2010 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 23 Feb 2010 15:05:11 -0500 Subject: [Linux-cluster] Virtual machine fence fail question In-Reply-To: <95994e3c1002221905t7a6ccd8dg3eee1439979c62d3@mail.gmail.com> References: <95994e3c1002221905t7a6ccd8dg3eee1439979c62d3@mail.gmail.com> Message-ID: <1266955511.23520.25.camel@localhost.localdomain> On Tue, 2010-02-23 at 11:05 +0800, Bernard Chew wrote: > Hi, > > Given I have 2 Red Hat Clusters; 1 cluster consisting of physical > hosts and another consisting of virtual guests which are hosted in the > physical hosts. The physical host cluster uses DRAC fencing while the > virtual guest cluster uses virtual machine fencing. > > If a physical host goes down, I saw that DRAC fencing takes place > successfully but fencing fail for the virtual guests on the physical > host which go down (together). Does the virtual machine fencing fails > because the virtual guests are no longer available? How can I > configure fencing so that both physical hosts and virtual guests are > fenced correctly? Are you using fence_xvm/fence_xvmd or fence_virsh ? -- Lon From lhh at redhat.com Tue Feb 23 20:07:59 2010 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 23 Feb 2010 15:07:59 -0500 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: References: Message-ID: <1266955679.23520.26.camel@localhost.localdomain> On Tue, 2010-02-23 at 17:52 +0000, Joseph L. Casale wrote: > With the vendor provided packages such as gfs etc, is it possible > to accomplish a drbd like setup so that the nodes of a cluster house > the shared storage themselves? > > I gather from my reading that is no, but just thought I would check. http://sources.redhat.com/cluster/wiki/DRBD_Cookbook Also, there's: http://www.drbd.org/users-guide/ch-pacemaker.html -- Lon From gordan at bobich.net Tue Feb 23 20:40:57 2010 From: gordan at bobich.net (Gordan Bobic) Date: Tue, 23 Feb 2010 20:40:57 +0000 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: References: <4B842626.60800@bobich.net> Message-ID: <4B843D59.7000105@bobich.net> Joseph L. Casale wrote: > Hi Gordan, > I'm under the gun to get something operational asap and have been reading like > mad but given the severity of getting it wrong, I want some experience under my > belt before I step off the reservation. I don't have the infrastructure available > to export an iscsi device, it's the backend I need redundant just the way drbd > would provide. You can do that with iSCSI. Make a sparse file to represent the exported storage and export that via iSCSI. Do the same on both nodes. Then from both nodes connect to it, get CLVM on top and get it to mirror the two exported volumes. Put GFS on top. Nowhere nearly as clean and nice as DRBD, though, especially cometh time to recover from failure. > I am ok with drbd, but its integration with rhcs concerns me as I don't really > know how the init scripts get constructed. I saw an archive post about the wrapper > for /usr/share/cluster/drbd.sh to hand it the resource name. > > I can assume if a node tanks, drbd goes primary on the remaining node, everything > is good. Or you can go active-active with GFS. :) > Looking through drbd.sh I see case's for promote/demote, I added this (the wrapper actually) > as a resource "script" with a child "fs" in a mock cluster as the underlying device > was primary on that node, it just started:) How does rhcs know when adding a "script" > to pass the required cases? How do I configure this? I use it in active-active mode with GFS. In that case I just use the fencing agent in DRBD's "stonith" configuration so that when disconnection occurs, the failed node gets fenced. Gordan From yvette at dbtgroup.com Tue Feb 23 21:17:12 2010 From: yvette at dbtgroup.com (yvette hirth) Date: Tue, 23 Feb 2010 21:17:12 +0000 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: <4B843D59.7000105@bobich.net> References: <4B842626.60800@bobich.net> <4B843D59.7000105@bobich.net> Message-ID: <4B8445D8.1080907@dbtgroup.com> Gordan Bobic wrote: > I use it in active-active mode with GFS. In that case I just use the > fencing agent in DRBD's "stonith" configuration so that when > disconnection occurs, the failed node gets fenced. can you use FCP with active-active? we bought a bunch of FCP stuff (4gbps and 8gbps HBA's and an 8gbps switch) but not the CNA's needed for iSCSI. and active-active sounds like "the ticket". while we do have a few 10GBe's and are awaiting a switch, i hate to waste all that FCP stuff. originally the plan was to move block-stored data through FCP and NFS through the 10GBe's. if we go with this plan, can we use active-active? and if so, will our bandwidth improve? sorry i'm such a n00b on clusters but they're a lot different than clustered mainframes for this aging z/OS assembler programmer! thanks yvette From jcasale at activenetwerx.com Tue Feb 23 21:42:44 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Tue, 23 Feb 2010 21:42:44 +0000 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: <4B843D59.7000105@bobich.net> References: <4B842626.60800@bobich.net> <4B843D59.7000105@bobich.net> Message-ID: >Or you can go active-active with GFS. :) Ok, based on Lon Hohberger's and your suggestion, I'll mock this up. I presume the choice on Pri/Pri w/ GFS2 overcomes management issues imposed on the Pri/Sec role case's that would otherwise be involved. >> Looking through drbd.sh I see case's for promote/demote, I added this (the wrapper actually) >> as a resource "script" with a child "fs" in a mock cluster as the underlying device >> was primary on that node, it just started:) How does rhcs know when adding a "script" >> to pass the required cases? How do I configure this? > >I use it in active-active mode with GFS. In that case I just use the >fencing agent in DRBD's "stonith" configuration so that when >disconnection occurs, the failed node gets fenced. I see the ifmib fencing agent in git, am I ok to utilize this as my single fencing agent in this drbd pri/pri setup with gfs2? I suspect this would be effective in my environment given the equipment I have. Does this allow me to neglect the use of stonith and the config in the drbd.conf file and let rhcs handle fencing solely on its own? Or is this additional layer of fencing something needed to be done for drbd's sake, in that case I guess I would implement both? Thanks again everyone! jlc From gordan at bobich.net Tue Feb 23 21:47:20 2010 From: gordan at bobich.net (Gordan Bobic) Date: Tue, 23 Feb 2010 21:47:20 +0000 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: <4B8445D8.1080907@dbtgroup.com> References: <4B842626.60800@bobich.net> <4B843D59.7000105@bobich.net> <4B8445D8.1080907@dbtgroup.com> Message-ID: <4B844CE8.6030003@bobich.net> yvette hirth wrote: > Gordan Bobic wrote: > >> I use it in active-active mode with GFS. In that case I just use the >> fencing agent in DRBD's "stonith" configuration so that when >> disconnection occurs, the failed node gets fenced. > > can you use FCP with active-active? we bought a bunch of FCP stuff > (4gbps and 8gbps HBA's and an 8gbps switch) but not the CNA's needed for > iSCSI. and active-active sounds like "the ticket". while we do have a > few 10GBe's and are awaiting a switch, i hate to waste all that FCP stuff. You need something that will run TCP/IP. There is some support for IP-over-FC in Linux, though. You should be able to get iSCSI and DRBD running over that. Not tried it myself, though, so I cannot offer any guidance over and above what you can google. Gordan From gordan at bobich.net Tue Feb 23 21:52:29 2010 From: gordan at bobich.net (Gordan Bobic) Date: Tue, 23 Feb 2010 21:52:29 +0000 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: References: <4B842626.60800@bobich.net> <4B843D59.7000105@bobich.net> Message-ID: <4B844E1D.90504@bobich.net> Joseph L. Casale wrote: >> Or you can go active-active with GFS. :) > > Ok, based on Lon Hohberger's and your suggestion, I'll mock this up. > I presume the choice on Pri/Pri w/ GFS2 overcomes management issues > imposed on the Pri/Sec role case's that would otherwise be involved. Yes. Personally, I'm still using GFS1, but RedHat deem GFS2 stable enough for production. You may want to trial both and see what yields better performance. GFS1 is more tweakable, so some people are reporting much better performance with it for their use cases. >>> Looking through drbd.sh I see case's for promote/demote, I added this (the wrapper actually) >>> as a resource "script" with a child "fs" in a mock cluster as the underlying device >>> was primary on that node, it just started:) How does rhcs know when adding a "script" >>> to pass the required cases? How do I configure this? >> I use it in active-active mode with GFS. In that case I just use the >> fencing agent in DRBD's "stonith" configuration so that when >> disconnection occurs, the failed node gets fenced. > > I see the ifmib fencing agent in git, am I ok to utilize this as my single fencing > agent in this drbd pri/pri setup with gfs2? I suspect this would be effective > in my environment given the equipment I have. Sorry, never used that fencing agent. Perhaps someone else can answer that. > Does this allow me to neglect the use of stonith and the config in the drbd.conf > file and let rhcs handle fencing solely on its own? Or is this additional layer > of fencing something needed to be done for drbd's sake, in that case I guess I > would implement both? I implement both, just in case there is a weird transient outage and DRBD disconnects but RHCS doesn't. You lose nothing by putting it in stonith in drbd.conf, and it can save you problems later on. Gordan From yvette at dbtgroup.com Tue Feb 23 23:19:27 2010 From: yvette at dbtgroup.com (yvette hirth) Date: Tue, 23 Feb 2010 23:19:27 +0000 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: <4B844CE8.6030003@bobich.net> References: <4B842626.60800@bobich.net> <4B843D59.7000105@bobich.net> <4B8445D8.1080907@dbtgroup.com> <4B844CE8.6030003@bobich.net> Message-ID: <4B84627F.9010105@dbtgroup.com> Gordan Bobic wrote: > yvette hirth wrote: >> can you use FCP with active-active? we bought a bunch of FCP stuff >> (4gbps and 8gbps HBA's and an 8gbps switch) but not the CNA's needed >> for iSCSI. and active-active sounds like "the ticket". while we do >> have a few 10GBe's and are awaiting a switch, i hate to waste all that >> FCP stuff. > >> Gordan Bobic wrote: > You need something that will run TCP/IP. There is some support for > IP-over-FC in Linux, though. You should be able to get iSCSI and DRBD > running over that. Not tried it myself, though, so I cannot offer any > guidance over and above what you can google. thanks, Gordan, but perhaps i should explain a bit more. our main server has four ip gigE ports bonded for TCP/IP; we're using FCP for server:drive-array access; and we use NFS for server:workstation disk data movement. we are trying to do an HA config for now, because our workstations do not have any fencing gear, and while i'm sold on HP's ILO(2), more boxes are out-of-budget at this time. so, at this time we would like to "cluster" the servers only, into an HA "active-something" config. so the FCP will be used for server:drive-array access; and the 10GBe adapters are for server:workstation NFS disk access. as the drive arrays are 4gbps FCP, we'd like to make them more efficient, as the 10GBe network adapters are underutilized. what i'm asking is: does active-active support FCP for disk access (between the clustered servers) and will such an active-active HA server config provide higher bandwidth via FCP for block-disk access between the servers and the drive arrays? thanks again to all yvette From bernardchew at gmail.com Wed Feb 24 02:03:14 2010 From: bernardchew at gmail.com (Bernard Chew) Date: Wed, 24 Feb 2010 10:03:14 +0800 Subject: [Linux-cluster] Virtual machine fence fail question In-Reply-To: <1266955511.23520.25.camel@localhost.localdomain> References: <95994e3c1002221905t7a6ccd8dg3eee1439979c62d3@mail.gmail.com> <1266955511.23520.25.camel@localhost.localdomain> Message-ID: <95994e3c1002231803y7b08b74cq39202646983431e5@mail.gmail.com> > On Wed, Feb 24, 2010 at 4:05 AM, Lon Hohberger wrote: > On Tue, 2010-02-23 at 11:05 +0800, Bernard Chew wrote: >> Hi, >> >> Given I have 2 Red Hat Clusters; 1 cluster consisting of physical >> hosts and another consisting of virtual guests which are hosted in the >> physical hosts. The physical host cluster uses DRAC fencing while the >> virtual guest cluster uses virtual machine fencing. >> >> If a physical host goes down, I saw that DRAC fencing takes place >> successfully but fencing fail for the virtual guests on the physical >> host which go down (together). Does the virtual machine fencing fails >> because the virtual guests are no longer available? How can I >> configure fencing so that both physical hosts and virtual guests are >> fenced correctly? > > Are you using fence_xvm/fence_xvmd or fence_virsh ? > > -- Lon > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > Hi Lon, Thank you for looking into this. I am currently using fence_xvm/fence_xvmd. Regards, Bernard From raju.rajsand at gmail.com Wed Feb 24 04:35:47 2010 From: raju.rajsand at gmail.com (Rajagopal Swaminathan) Date: Wed, 24 Feb 2010 10:05:47 +0530 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: <4B84627F.9010105@dbtgroup.com> References: <4B842626.60800@bobich.net> <4B843D59.7000105@bobich.net> <4B8445D8.1080907@dbtgroup.com> <4B844CE8.6030003@bobich.net> <4B84627F.9010105@dbtgroup.com> Message-ID: <8786b91c1002232035t74b0c3edw7896b6915fac2f72@mail.gmail.com> Greetings, On Wed, Feb 24, 2010 at 4:49 AM, yvette hirth wrote: > we are trying to do an HA config for now, because our workstations do not > have any fencing gear, and while i'm sold on HP's ILO(2), more boxes are > out-of-budget at this time. It is a very bad idea to run a cluster without fencing. I have faced hard-acid-rock -- from PHBs (read MBAs) who did not approve the expense -- due to that. Have you checked out power fencing -- Like APC devices? They aren't expensive so to speak. I am in India so have no idea about what that will cost in your part of the world HTH Regards, Rajagopal From esggrupos at gmail.com Wed Feb 24 09:34:22 2010 From: esggrupos at gmail.com (ESGLinux) Date: Wed, 24 Feb 2010 10:34:22 +0100 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <1266945213.23528.0.camel@thor.seas.smu.edu> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> <153982.12707.qm@web32107.mail.mud.yahoo.com> <20100222223343.GA31846@andromeda.usersys.redhat.com> <1266945213.23528.0.camel@thor.seas.smu.edu> Message-ID: <3128ba141002240134o6f392edaid601c4ecfceb264c@mail.gmail.com> Hi Doug, the split brain is what is happening to you ;-) >From wikipedia: http://en.wikipedia.org/wiki/High-availability_cluster "HA clusters usually use a *heartbeat* private network connection which is used to monitor the health and status of each node in the cluster. One subtle, but serious condition every clustering software must be able to handle is split-brain. Split-brain occurs when all of the private links go down simultaneously, but the cluster nodes are still running. If that happens, each node in the cluster may mistakenly decide that every other node has gone down and attempt to start services that other nodes are still running. Having duplicate instances of services may cause data corruption on the shared storage." The qourum disk is a good choice to avoid it as they have told you. Good luck, Greetings, ESG 2010/2/23 Doug Tucker > > Hi Doug, maybe you can avoid this kind of problem using a quorumdisk > partition. a two node cluster is split-brain prone and with a quorumdisk > partition you can avoid split-brain situations, which probably is causing > this behavior. > > > > So, about use a cross-over (or straight) cable, I don't know any issue > about it, but, try to check if it's using full-duplex mode. half-duplex mode > on cross-over linked machines probably will cause heartbeat problems. > > > > cya.. > > Can you give me a little info about "split-brain" issues? I don't > understand what you mean by that, and what I'm solving with a > quorumdisk. And it has always worked fine, it just started happening > after the install of this newer kernel/gfs module. The other 2 node > cluster is still rock solid. Also, the network interfaces on the > crossover are full duplex. Thanks for writing back, you're the first > person who offered anything. > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkovachev at varna.net Wed Feb 24 10:49:47 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Wed, 24 Feb 2010 12:49:47 +0200 Subject: [Linux-cluster] Distributed Storage in Cluster Suite In-Reply-To: <4B843D59.7000105@bobich.net> References: <4B842626.60800@bobich.net> <4B843D59.7000105@bobich.net> Message-ID: <20100224103450.M99399@varna.net> Hi Gordan, On Tue, 23 Feb 2010 20:40:57 +0000, Gordan Bobic wrote > > I use it in active-active mode with GFS. In that case I just use the > fencing agent in DRBD's "stonith" configuration so that when > disconnection occurs, the failed node gets fenced. > I am also using it in active-active mode, but instead of CLVM i am exporting the DRBD resource via iSCSI and using multipath in multibus mode from the nodes. In this setup there are problems using stonith thought, because it blocks the access to both paths and multipath reports no active paths, which causes the GFS to be withdrawn on some of the nodes. So to avoid this i am using the same (bonded) interface for cluster and DRDBD communication with dont-care (for DRBD) and leave the fencing to the cluster instead of DRBD, which also avoids the double fencing from both. > Gordan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From dirk.schulz at kinzesberg.de Wed Feb 24 14:53:33 2010 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Wed, 24 Feb 2010 15:53:33 +0100 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <1266868409.16694.15.camel@thor.seas.smu.edu> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> Message-ID: <4B853D6D.9000802@kinzesberg.de> As far as I recall the docs say a switch is needed that is capable of IGMP etc. There must be issues with cross cables, if that is correct. I did not investigate further and took an old cisco 2950 (you can by masses of used ones on ebay very cheap) and configured it according to docs. To avoid the switch being the SPOF you can use 2 3750s with extended image, they can use LACP to bond links connected to both of them. That gets a bit more expensive then, admittedly, but even those should be available on ebay. Dirk Am 22.02.10 20:53, schrieb Doug Tucker: > We did. It's problematic when you need to reboot a switch or it goes > down. They can't talk and try to fence each other. Crossover cable is > a direct connection, actually far more efficient for what you are trying > to accomplish. > > > On Mon, 2010-02-22 at 11:57 -0600, Paul M. Dyer wrote: > >> Crossover cable?????? >> >> With all the $$ spent, try putting a switch between the nodes. >> >> Paul >> >> ----- Original Message ----- >> From: "Doug Tucker" >> To: linux-cluster at redhat.com >> Sent: Monday, February 22, 2010 10:15:49 AM (GMT-0600) America/Chicago >> Subject: [Linux-cluster] Repeated fencing >> >> We have a 2 4.x cluster that has developed an issue we are unable to >> resolve. Starting back in December, the nodes began fencing each other >> randomly, and as frequently as once a day. There is nothing at the >> console prior to it happening, and nothing in the logs. We have not >> been able to develop any pattern to this point, the 2 nodes appear to be >> functioning fine, and suddenly in the logs a message will appear about >> "node x missed too many heartbeats" and the next thing you see is it >> fencing the node. Thinking we possibly had a hardware issue, we >> replaced both nodes from scratch with new machines, the problem >> persists. The cluster communication is done via a crossover cable on >> eth1 on both devices with private ip's. We have a 2nd cluster that is >> not having this issue, and both nodes have been up for over 160 days. >> The configuration is basically identical to the problematic cluster. >> The only difference between the 2 now is the newer hardware on the >> problematic node (prior, that was identical), and the kernel. The >> non-problematic cluster is still running kernel 89.0.9 and the >> problematic cluster is on 89.0.11. We are afraid at this point to allow >> our non problematic cluster upgrade to the latest packages. Any insight >> or advice would be greatly appreciated, we have exhausted our ideas >> here. >> >> Sincerely, >> >> Doug >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From tuckerd at lyle.smu.edu Wed Feb 24 14:54:38 2010 From: tuckerd at lyle.smu.edu (Doug Tucker) Date: Wed, 24 Feb 2010 08:54:38 -0600 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <3128ba141002240134o6f392edaid601c4ecfceb264c@mail.gmail.com> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> <153982.12707.qm@web32107.mail.mud.yahoo.com> <20100222223343.GA31846@andromeda.usersys.redhat.com> <1266945213.23528.0.camel@thor.seas.smu.edu> <3128ba141002240134o6f392edaid601c4ecfceb264c@mail.gmail.com> Message-ID: <1267023278.9925.7.camel@thor.seas.smu.edu> Thanks to you and Carlos. I understand a bit better now what you are referring to, however, I don't believe that is the issue. The reason we went to the crossover cable was to avoid this issue, as we had a switch die once, and both then thought they were master and tried to fence the other. In my situation, there is no reason for the missed heartbeat that I can find. The interfaces have not gone down. We ran a test where I started a ping between the 2 that wrote out to a file until a "heartbeat" missed and a reboot occurred. There was not a single missed ping between the 2 nodes prior to the event. Also in a split brain, both machines should recognize the other one "gone" and try to become master. In this case, only 1 of the nodes at a time is seeing a "missed heartbeat" and then attempting to fence the other. We have replaced all hardware to include cables even to ensure it wasn't that. This appears to be some software bug of sorts. Again, we have another 2 node cluster that this doesn't occur on, but, they are running a different kernel and gfs module. On Wed, 2010-02-24 at 03:34 -0600, ESGLinux wrote: > Hi Doug, > > > the split brain is what is happening to you ;-) > > > From wikipedia: http://en.wikipedia.org/wiki/High-availability_cluster > "HA clusters usually use a heartbeat private network connection which > is used to monitor the health and status of each node in the cluster. > One subtle, but serious condition every clustering software must be > able to handle is split-brain. Split-brain occurs when all of the > private links go down simultaneously, but the cluster nodes are still > running. If that happens, each node in the cluster may mistakenly > decide that every other node has gone down and attempt to start > services that other nodes are still running. Having duplicate > instances of services may cause data corruption on the shared > storage." > > > The qourum disk is a good choice to avoid it as they have told you. > > > Good luck, > > > Greetings, > > > ESG > > > > > 2010/2/23 Doug Tucker > > Hi Doug, maybe you can avoid this kind of problem using a > quorumdisk partition. a two node cluster is split-brain prone > and with a quorumdisk partition you can avoid split-brain > situations, which probably is causing this behavior. > > > > So, about use a cross-over (or straight) cable, I don't know > any issue about it, but, try to check if it's using > full-duplex mode. half-duplex mode on cross-over linked > machines probably will cause heartbeat problems. > > > > cya.. > > > Can you give me a little info about "split-brain" issues? I > don't > understand what you mean by that, and what I'm solving with a > quorumdisk. And it has always worked fine, it just started > happening > after the install of this newer kernel/gfs module. The other > 2 node > cluster is still rock solid. Also, the network interfaces on > the > crossover are full duplex. Thanks for writing back, you're > the first > person who offered anything. > > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > From rpeterso at redhat.com Wed Feb 24 14:55:21 2010 From: rpeterso at redhat.com (Bob Peterson) Date: Wed, 24 Feb 2010 09:55:21 -0500 (EST) Subject: [Linux-cluster] Cluster 3.0.8 stable release In-Reply-To: <4B839043.5080601@redhat.com> Message-ID: <1555581768.2616731267023321291.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> ----- "Fabio M. Di Nitto" wrote: | -----BEGIN PGP SIGNED MESSAGE----- | Hash: SHA1 | | The cluster team and its community are proud to announce the 3.0.8 | stable release from the STABLE3 branch. | | This release contains several major bug fixes. We strongly recommend | people to update your clusters. Sorry, guys, but we've discovered a problem with mkfs.gfs2 that makes it impossible to create gfs2 file systems. The problem has been fixed in the STABLE3 and master branches of the git repos, and 3.0.9 will be released very soon. Regards, Bob Peterson Red Hat File Systems From kkovachev at varna.net Wed Feb 24 15:26:28 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Wed, 24 Feb 2010 17:26:28 +0200 Subject: [Linux-cluster] Cluster 3.0.8 stable release In-Reply-To: <1555581768.2616731267023321291.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> References: <4B839043.5080601@redhat.com> <1555581768.2616731267023321291.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> Message-ID: <20100224151717.M84482@varna.net> On Wed, 24 Feb 2010 09:55:21 -0500 (EST), Bob Peterson wrote > ----- "Fabio M. Di Nitto" wrote: > | -----BEGIN PGP SIGNED MESSAGE----- > | Hash: SHA1 > | > | The cluster team and its community are proud to announce the 3.0.8 > | stable release from the STABLE3 branch. > | > | This release contains several major bug fixes. We strongly recommend > | people to update your clusters. > > Sorry, guys, but we've discovered a problem with mkfs.gfs2 that makes > it impossible to create gfs2 file systems. The problem has been fixed > in the STABLE3 and master branches of the git repos, and 3.0.9 will be > released very soon. > Will this also fix the segfault of gfs2_quota? If i reset the quota and then init without adding any limits it segfaults and leaves the GFS2 locked on the node. I have not posted a bug yet, so if it is unrelated i should :) > Regards, > > Bob Peterson > Red Hat File Systems > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From jcasale at activenetwerx.com Wed Feb 24 15:54:51 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Wed, 24 Feb 2010 15:54:51 +0000 Subject: [Linux-cluster] Documentation on RHCS Message-ID: Looking at http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Cluster_Suite_Overview/index.html Gives a high level overview of what's going on, but doesn't explain fine grain details like what master process start all the other daemons as some docs suggest when using drbd with rhcs you need to adjust the init scripts of drbd to start inbetween some of those controlled by the cluster suite, but those are off, and obviously started by maybe cman? Also, http://sources.redhat.com/cluster/wiki/FAQ/CMAN#cman_shutdown suggests `cman_tool leave remove` is how to drop a node out of a cluster for any reason so it doesn't get fenced, but that doesn't work as other daemons are suggested by cman_tool to need to be shut down. Thanks guys, jlc From rpeterso at redhat.com Wed Feb 24 16:01:19 2010 From: rpeterso at redhat.com (Bob Peterson) Date: Wed, 24 Feb 2010 11:01:19 -0500 (EST) Subject: [Linux-cluster] Cluster 3.0.8 stable release In-Reply-To: <20100224151717.M84482@varna.net> Message-ID: <2036870551.2628711267027279944.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> ----- "Kaloyan Kovachev" wrote: | Will this also fix the segfault of gfs2_quota? | If i reset the quota and then init without adding any limits it | segfaults and | leaves the GFS2 locked on the node. I have not posted a bug yet, so if | it is | unrelated i should :) | Hi, It cannot be related to the patch I pushed last night. It may, however, be related to the large set of patches I recently pushed to STABLE3 for fsck.gfs2 issues; hard to say. Please open a bugzilla record on this and attach the pertinent info. You can assign it to me (rpeterso at redhat.com) or Abhi (adas at redhat.com) since he's our resident quota expert. Regards, Bob Peterson Red Hat File Systems From cmaiolino at redhat.com Wed Feb 24 18:38:50 2010 From: cmaiolino at redhat.com (Carlos Maiolino) Date: Wed, 24 Feb 2010 15:38:50 -0300 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <4B853D6D.9000802@kinzesberg.de> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> <4B853D6D.9000802@kinzesberg.de> Message-ID: <20100224183850.GC14976@andromeda.usersys.redhat.com> On Wed, Feb 24, 2010 at 03:53:33PM +0100, Dirk H. Schulz wrote: > As far as I recall the docs say a switch is needed that is capable > of IGMP etc. There must be issues with cross cables, if that is > correct. > > I did not investigate further and took an old cisco 2950 (you can by > masses of used ones on ebay very cheap) and configured it according > to docs. > > To avoid the switch being the SPOF you can use 2 3750s with extended > image, they can use LACP to bond links connected to both of them. > That gets a bit more expensive then, admittedly, but even those > should be available on ebay. > > Dirk > I don't know if IGMP is strict necessary, I don't know much about IGMP to be honest, but Is a must switch can work with multicast, it's how the corosync works.please someone corect me if I'm wrong =]. -- --- Best Regards Carlos Eduardo Maiolino Support engineer Red Hat - Global Support Services From cmaiolino at redhat.com Wed Feb 24 18:43:02 2010 From: cmaiolino at redhat.com (Carlos Maiolino) Date: Wed, 24 Feb 2010 15:43:02 -0300 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <1267023278.9925.7.camel@thor.seas.smu.edu> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> <153982.12707.qm@web32107.mail.mud.yahoo.com> <20100222223343.GA31846@andromeda.usersys.redhat.com> <1266945213.23528.0.camel@thor.seas.smu.edu> <3128ba141002240134o6f392edaid601c4ecfceb264c@mail.gmail.com> <1267023278.9925.7.camel@thor.seas.smu.edu> Message-ID: <20100224184302.GD14976@andromeda.usersys.redhat.com> On Wed, Feb 24, 2010 at 08:54:38AM -0600, Doug Tucker wrote: > Thanks to you and Carlos. I understand a bit better now what you are > referring to, however, I don't believe that is the issue. The reason we > went to the crossover cable was to avoid this issue, as we had a switch > die once, and both then thought they were master and tried to fence the > other. In my situation, there is no reason for the missed heartbeat > that I can find. The interfaces have not gone down. We ran a test > where I started a ping between the 2 that wrote out to a file until a > "heartbeat" missed and a reboot occurred. There was not a single missed > ping between the 2 nodes prior to the event. Also in a split brain, > both machines should recognize the other one "gone" and try to become > master. In this case, only 1 of the nodes at a time is seeing a "missed > heartbeat" and then attempting to fence the other. We have replaced all > hardware to include cables even to ensure it wasn't that. This appears > to be some software bug of sorts. Again, we have another 2 node cluster > that this doesn't occur on, but, they are running a different kernel and > gfs module. > Doug, did you search if there are any bugs in NIC's module that you are using ? Maybe try to look at kernel's changelog to see if there are any changes on these modules... cya -- --- Best Regards Carlos Eduardo Maiolino Support engineer Red Hat - Global Support Services From yvette at dbtgroup.com Wed Feb 24 18:45:41 2010 From: yvette at dbtgroup.com (yvette hirth) Date: Wed, 24 Feb 2010 18:45:41 +0000 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <1267023278.9925.7.camel@thor.seas.smu.edu> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> <153982.12707.qm@web32107.mail.mud.yahoo.com> <20100222223343.GA31846@andromeda.usersys.redhat.com> <1266945213.23528.0.camel@thor.seas.smu.edu> <3128ba141002240134o6f392edaid601c4ecfceb264c@mail.gmail.com> <1267023278.9925.7.camel@thor.seas.smu.edu> Message-ID: <4B8573D5.30009@dbtgroup.com> Doug Tucker wrote: > Thanks to you and Carlos. I understand a bit better now what you are > referring to, however, I don't believe that is the issue. The reason we > went to the crossover cable was to avoid this issue, as we had a switch > die once, and both then thought they were master and tried to fence the > other. In my situation, there is no reason for the missed heartbeat > that I can find. The interfaces have not gone down. We ran a test > where I started a ping between the 2 that wrote out to a file until a > "heartbeat" missed and a reboot occurred. There was not a single missed > ping between the 2 nodes prior to the event. Also in a split brain, > both machines should recognize the other one "gone" and try to become > master. In this case, only 1 of the nodes at a time is seeing a "missed > heartbeat" and then attempting to fence the other. We have replaced all > hardware to include cables even to ensure it wasn't that. This appears > to be some software bug of sorts. Again, we have another 2 node cluster > that this doesn't occur on, but, they are running a different kernel and > gfs module. ping is udp. is the heartbeat udp or tcp? perhaps you could ensure both servers have their clocks sync'ed and then run wireshark on each server capturing the crossover cable ethernet port and see which one is failing to signal the other... hth yvette hirth From crosa at redhat.com Wed Feb 24 18:54:30 2010 From: crosa at redhat.com (Cleber Rosa) Date: Wed, 24 Feb 2010 15:54:30 -0300 Subject: [Linux-cluster] Repeated fencing In-Reply-To: <20100224183850.GC14976@andromeda.usersys.redhat.com> References: <3343105.11266861438483.JavaMail.root@athena> <1266868409.16694.15.camel@thor.seas.smu.edu> <4B853D6D.9000802@kinzesberg.de> <20100224183850.GC14976@andromeda.usersys.redhat.com> Message-ID: <1267037670.14302.3.camel@localhost> If one feels brave enough, support for broadcast is back in RHEL 5.4 (if that's what is being used). I have used with very good results, although it is not officially supported. CR. On Wed, 2010-02-24 at 15:38 -0300, Carlos Maiolino wrote: > On Wed, Feb 24, 2010 at 03:53:33PM +0100, Dirk H. Schulz wrote: > > As far as I recall the docs say a switch is needed that is capable > > of IGMP etc. There must be issues with cross cables, if that is > > correct. > > > > I did not investigate further and took an old cisco 2950 (you can by > > masses of used ones on ebay very cheap) and configured it according > > to docs. > > > > To avoid the switch being the SPOF you can use 2 3750s with extended > > image, they can use LACP to bond links connected to both of them. > > That gets a bit more expensive then, admittedly, but even those > > should be available on ebay. > > > > Dirk > > > > I don't know if IGMP is strict necessary, I don't know much about IGMP to be honest, but Is a must switch can work with multicast, it's how the corosync works.please someone corect me if I'm wrong =]. > -- Cleber Rosa Solutions Architect - Red Hat, Inc. Mobile: +55 61 9185.3454 From jcasale at activenetwerx.com Fri Feb 26 00:09:25 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Fri, 26 Feb 2010 00:09:25 +0000 Subject: [Linux-cluster] Service not migrating Message-ID: Hi, Still testing around and learning the ins and outs of rhcs, I have an apache service with a mount etc configured that starts fine, migrates as well. If I drop the Ethernet interface on the node with the service active, the remaining node never starts the service, but if I gracefully take the running node out by rebooting it etc, the service migrates. Any ideas what I am missing? Thanks! jlc From alan.zg at gmail.com Fri Feb 26 02:37:12 2010 From: alan.zg at gmail.com (Alan A) Date: Thu, 25 Feb 2010 20:37:12 -0600 Subject: [Linux-cluster] GFS handling of executable files and I/O Message-ID: fromAlan A toGeneral Red Hat Linux discussion list < redhat-list at redhat.com> dateThu, Feb 25, 2010 at 7:56 PMsubjectGFS and I/Omailed-bygmail.com hide details 7:56 PM (39 minutes ago) I made /var/www gfs shared so I could share cgi-bin directory among cluster nodes. Yesterday we tested this out and failed. Users complained about slow page execution and I noticed with 'lsof /var/www/cgi-bin' command that many executable files are listing as open files. This brings me to a question - Does GFS treat executable files same as files open for read / write? Unrelated to previous question, what GFS parameters can I tune with 'gfs_tool settune /volumename' to get better I/O? I am dealing with single threaded application that has a lot of I/O, mostly opens files for writing. -- Alan A. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raju.rajsand at gmail.com Fri Feb 26 04:47:29 2010 From: raju.rajsand at gmail.com (Rajagopal Swaminathan) Date: Fri, 26 Feb 2010 10:17:29 +0530 Subject: [Linux-cluster] Service not migrating In-Reply-To: References: Message-ID: <8786b91c1002252047m3ac584d5k72536d2dbab3d9ec@mail.gmail.com> Greetings, On Fri, Feb 26, 2010 at 5:39 AM, Joseph L. Casale wrote: > Hi, > Still testing around and learning the ins and outs of rhcs, I have an > apache > service with a mount etc configured that starts fine, migrates as well. > but if I gracefully take the running node out by > rebooting it etc, the service migrates. > > Any ideas what I am missing? > Nothing. It is advertised to work that way. -------------- next part -------------- An HTML attachment was scrubbed... URL: From emilio at ugr.es Fri Feb 26 15:52:05 2010 From: emilio at ugr.es (Emilio Arjona) Date: Fri, 26 Feb 2010 16:52:05 +0100 Subject: [Linux-cluster] GFS2 and D state HTTPD processes Message-ID: Hi, we are experiencing some problems commented in an old thread: http://www.mail-archive.com/linux-cluster at redhat.com/msg07091.html We have 3 clustered servers under Red Hat 5.4 accessing a GFS2 resource. fstab options: /dev/vg_cluster/lv_cluster /opt/datacluster gfs2 defaults,noatime,nodiratime,noquota 0 0 GFS options: plock_rate_limit="0" plock_ownership=1 httpd processes run into D status sometimes and the only solution is hard reset the affected server. Can anyone give me some hints to diagnose the problem? Thanks :) -- Emilio Arjona. From jakov.sosic at srce.hr Fri Feb 26 17:29:44 2010 From: jakov.sosic at srce.hr (Jakov Sosic) Date: Fri, 26 Feb 2010 18:29:44 +0100 Subject: [Linux-cluster] Xen on RHEL cluster Message-ID: <4B880508.40808@srce.hr> Hi. I'm using Xen on RHEL cluster, and I have strange problems. I gave raw volumes from storage to Xen virtual machines. With windows, I have a problem that nodes don't see the volume as same one.... for example: clusternode1# clusvcadm -d vm:winxp clusternode1# dd if=/dev/mapper/winxp of=/node1winxp clusternode2# dd if=/dev/mapper/winxp of=/node2winxp clusternode3# dd if=/dev/mapper/winxp of=/node3winxp When I download these files and diff them, they all three differ. Also, sometimes very strange things happen. For example I download some file into winxp, shut it down, then start it on another node, and file is missing?!?!?!?! Should I use CLVM and not raw volumes from storage? Why is this happening? -- | Jakov Sosic | ICQ: 28410271 | PGP: 0x965CAE2D | ================================================================= | start fighting cancer -> http://www.worldcommunitygrid.org/ | From kkovachev at varna.net Fri Feb 26 21:59:30 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Fri, 26 Feb 2010 23:59:30 +0200 Subject: [Linux-cluster] Service not migrating In-Reply-To: References: Message-ID: <20100226215741.M41159@varna.net> On Fri, 26 Feb 2010 00:09:25 +0000, Joseph L. Casale wrote > Hi, > Still testing around and learning the ins and outs of rhcs, I have an apache > service with a mount etc configured that starts fine, migrates as well. If > I drop the Ethernet interface on the node with the service active, the remaining > node never starts the service, but if I gracefully take the running node out by > rebooting it etc, the service migrates. > > Any ideas what I am missing? Is the node successfully fenced? The service will not migrate before that and if you don't have proper fencing that could be the reason > > Thanks! > jlc > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From jcasale at activenetwerx.com Fri Feb 26 23:12:59 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Fri, 26 Feb 2010 23:12:59 +0000 Subject: [Linux-cluster] Service not migrating In-Reply-To: <20100226215741.M41159@varna.net> References: <20100226215741.M41159@varna.net> Message-ID: >Is the node successfully fenced? The service will not migrate before that and >if you don't have proper fencing that could be the reason Yes, this confirms what I thought. I did see that fencing was awry and I am working on that now. Ultimately I wanted to use fence_ifmib in production and fence_vmware_ng for my tests but it's not working with my esxi 4 hosts. Its reported to work, so I don't know what's wrong. It can get status, the "off" command goes unacknowledged. I have a low priv user on the box setup that can ssh in and run `vim-cmd`, is there an ssh fence script or any workarounds you might know of? Anyways, thanks for confirming the above! jlc From kkovachev at varna.net Sat Feb 27 13:36:08 2010 From: kkovachev at varna.net (Kaloyan Kovachev) Date: Sat, 27 Feb 2010 15:36:08 +0200 Subject: [Linux-cluster] Service not migrating In-Reply-To: References: <20100226215741.M41159@varna.net> Message-ID: <20100227133221.M96896@varna.net> On Fri, 26 Feb 2010 23:12:59 +0000, Joseph L. Casale wrote > >Is the node successfully fenced? The service will not migrate before that and > >if you don't have proper fencing that could be the reason > > Yes, this confirms what I thought. I did see that fencing was awry and I am working > on that now. Ultimately I wanted to use fence_ifmib in production and fence_vmware_ng Sorry i have no experience with those. > for my tests but it's not working with my esxi 4 hosts. Its reported to work, so I don't > know what's wrong. It can get status, the "off" command goes unacknowledged. I have a > low priv user on the box setup that can ssh in and run `vim-cmd`, is there an ssh fence > script or any workarounds you might know of? if there is no connection to the machine ssh is useless, but as a workaround (for the tests only) you may use manual fencing > > Anyways, thanks for confirming the above! > jlc > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From jcasale at activenetwerx.com Sat Feb 27 14:12:31 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Sat, 27 Feb 2010 14:12:31 +0000 Subject: [Linux-cluster] Service not migrating In-Reply-To: <20100227133221.M96896@varna.net> References: <20100226215741.M41159@varna.net> <20100227133221.M96896@varna.net> Message-ID: >if there is no connection to the machine ssh is useless, but as a workaround >(for the tests only) you may use manual fencing I meant ssh to the host, not vm. I did resolve the issue, it was a licensing issue and the fence_vmware_ng agent is working fine. The test setup is working well and failing over as predicted. Thanks for the help! jlc From jcasale at activenetwerx.com Sat Feb 27 14:36:19 2010 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Sat, 27 Feb 2010 14:36:19 +0000 Subject: [Linux-cluster] fence_ifmib problem Message-ID: Trying to run this script gives me the following error: Traceback (most recent call last): File "/usr/lib/fence/fence_ifmib.py", line 139, in ? main() File "/usr/lib/fence/fence_ifmib.py", line 112, in main atexit.register(atexit_handler) NameError: global name 'atexit' is not defined Anyone know what I am missing? I copied it from git and edited the sys.path.append("/usr/lib/fence") Thanks, jlc From dan at quah.ro Sat Feb 27 17:53:17 2010 From: dan at quah.ro (Dan Candea) Date: Sat, 27 Feb 2010 19:53:17 +0200 Subject: [Linux-cluster] fence_ifmib problem In-Reply-To: References: Message-ID: <4B895C0D.6070909@quah.ro> On 27.02.2010 16:36, Joseph L. Casale wrote: > Trying to run this script gives me the following error: > > Traceback (most recent call last): > File "/usr/lib/fence/fence_ifmib.py", line 139, in ? > main() > File "/usr/lib/fence/fence_ifmib.py", line 112, in main > atexit.register(atexit_handler) > NameError: global name 'atexit' is not defined > > Anyone know what I am missing? I copied it from git and edited the > sys.path.append("/usr/lib/fence") > > Thanks, > jlc > > Hi I didn't have the same error message but I made fence_ifmib to work by installing dev-python/pysnmp-2.0.9 on the OS. You could check which python modules it uses.Maybe this hint could help you. regards -- Dan C?ndea Does God Play Dice? From a.alawi at auckland.ac.nz Sun Feb 28 21:37:02 2010 From: a.alawi at auckland.ac.nz (Abraham Alawi) Date: Mon, 1 Mar 2010 10:37:02 +1300 Subject: [Linux-cluster] GFS handling of executable files and I/O In-Reply-To: References: Message-ID: <4E069A52-DCAE-4A1D-A25F-A07903AEA0AF@auckland.ac.nz> I reckon GFS is poorly documented, there are lots of things about it you won't be able to know until you deploy it and run into problems, I lately resized GFS and it badly effected NFS, it turned out to be a known issue but was not documented anywhere, adding journals is another example, they don't tell you would need to expand the file system if you decided to add more journals until you run gfs_jadd and complains of no enough space, gfs tuning options (gfs_tool gettune) are not documented either .. On 26/02/2010, at 3:37 PM, Alan A wrote: > > I made /var/www gfs shared so I could share cgi-bin directory among cluster nodes. Yesterday we tested this out and failed. Users complained about slow page execution and I noticed with 'lsof /var/www/cgi-bin' command that many executable files are listing as open files. This brings me to a question - Does GFS treat executable files same as files open for read / write? > > Unrelated to previous question, what GFS parameters can I tune with 'gfs_tool settune /volumename' to get better I/O? I am dealing with single threaded application that has a lot of I/O, mostly opens files for writing. > > -- > Alan A. > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster '''''''''''''''''''''''''''''''''''''''''''''''''''''' Abraham Alawi Unix/Linux Systems Administrator Science IT University of Auckland e: a.alawi at auckland.ac.nz p: +64-9-373 7599, ext#: 87572 '''''''''''''''''''''''''''''''''''''''''''''''''''''' -------------- next part -------------- An HTML attachment was scrubbed... URL: From jakov.sosic at srce.hr Sun Feb 28 23:26:15 2010 From: jakov.sosic at srce.hr (Jakov Sosic) Date: Mon, 01 Mar 2010 00:26:15 +0100 Subject: [Linux-cluster] Xen on RHEL cluster In-Reply-To: <4B880508.40808@srce.hr> References: <4B880508.40808@srce.hr> Message-ID: <4B8AFB97.90201@srce.hr> On 02/26/2010 06:29 PM, Jakov Sosic wrote: > Hi. > > I'm using Xen on RHEL cluster, and I have strange problems. I gave raw > volumes from storage to Xen virtual machines. With windows, I have a > problem that nodes don't see the volume as same one.... for example: > > clusternode1# clusvcadm -d vm:winxp > clusternode1# dd if=/dev/mapper/winxp of=/node1winxp > clusternode2# dd if=/dev/mapper/winxp of=/node2winxp > clusternode3# dd if=/dev/mapper/winxp of=/node3winxp > > When I download these files and diff them, they all three differ. > > Also, sometimes very strange things happen. For example I download some > file into winxp, shut it down, then start it on another node, and file > is missing?!?!?!?! > > Should I use CLVM and not raw volumes from storage? Why is this happening? It seems that CLVM is solution (migrated whole cluster to clvm today), and that the bug is double caching (both domU and dom0 caching), and that bugs like this occur when using raw LUNs: https://bugzilla.redhat.com/show_bug.cgi?id=466681 -- | Jakov Sosic | ICQ: 28410271 | PGP: 0x965CAE2D | ================================================================= | start fighting cancer -> http://www.worldcommunitygrid.org/ |