From fabbione at fabbione.net Mon Jan 2 09:59:53 2006 From: fabbione at fabbione.net (Fabio Massimo Di Nitto) Date: Mon, 02 Jan 2006 10:59:53 +0100 Subject: [Linux-cluster] Re: cluster/gfs-kernel/src/gfs ops_address.c ops_f ... In-Reply-To: <20051214203841.17983.qmail@sourceware.org> References: <20051214203841.17983.qmail@sourceware.org> Message-ID: <43B8F999.60602@fabbione.net> wcheng at sourceware.org wrote: > CVSROOT: /cvs/cluster > Module name: cluster > Branch: RHEL4 > Changes by: wcheng at sourceware.org 2005-12-14 20:38:41 > > Modified files: > gfs-kernel/src/gfs: ops_address.c ops_file.c > > Log message: > This gfs change (bugzilla 173913) pairs with kernel (bugzilla 173912) > 2.6.9-25 that requires an updated version of linux/fs.h. We move the > locking of i_sem and i_alloc_sem out of kernel directIO routine into > gfs read/write path. This is to re-arrange lock order so we don't get > into deadlock situation as described in the bugzilla. > > Patches: > http://sourceware.org/cgi-bin/cvsweb.cgi/cluster/gfs-kernel/src/gfs/ops_address.c.diff?cvsroot=cluster&only_with_tag=RHEL4&r1=1.5.2.2&r2=1.5.2.3 > http://sourceware.org/cgi-bin/cvsweb.cgi/cluster/gfs-kernel/src/gfs/ops_file.c.diff?cvsroot=cluster&only_with_tag=RHEL4&r1=1.16.2.6&r2=1.16.2.7 Hi Wendy, doesn't this patch require porting to the STABLE branch as well? I can't find a direct_IO_cluster_locking on vanilla kernels so i assume this is a specific redhat extension. if i understand the problem right, shouldn't the stable branch get the patch using blockdev_direct_IO_own_locking ? TIA Fabio -- I'm going to make him an offer he can't refuse. From carlopmart at gmail.com Mon Jan 2 13:55:07 2006 From: carlopmart at gmail.com (carlopmart) Date: Mon, 02 Jan 2006 14:55:07 +0100 Subject: [Linux-cluster] doubts about using clvm Message-ID: <43B930BB.5060800@gmail.com> Hi all, I am testing Cluster/GFS suite under my vmware lab. I have installed two RHEL 4 with CS and GFS support. Another machine acts as a GFS server (Ubuntu breezy server that has rhcs/gfs included). Ubuntu server has two phisycal disks exported by gnbd daemon and RHEL import these disks as a GFS clients. But I would like to export these disks as a unique disk using LVM. Then, I need to start clvm process under Ubuntu server or on RHEL clients??? Is it possible to accomplish this?. Thanks. -- CL Martinez carlopmart {at} gmail {d0t} com From wcheng at redhat.com Mon Jan 2 14:55:56 2006 From: wcheng at redhat.com (Wendy Cheng) Date: Mon, 02 Jan 2006 09:55:56 -0500 Subject: [Linux-cluster] Re: cluster/gfs-kernel/src/gfs ops_address.c ops_f ... In-Reply-To: <43B8F999.60602@fabbione.net> References: <20051214203841.17983.qmail@sourceware.org> <43B8F999.60602@fabbione.net> Message-ID: <1136213756.3280.7.camel@localhost.localdomain> On Mon, 2006-01-02 at 10:59 +0100, Fabio Massimo Di Nitto wrote: > wcheng at sourceware.org wrote: > > CVSROOT: /cvs/cluster > > Module name: cluster > > Branch: RHEL4 > > Changes by: wcheng at sourceware.org 2005-12-14 20:38:41 > > > > Modified files: > > gfs-kernel/src/gfs: ops_address.c ops_file.c > > > > Log message: > > This gfs change (bugzilla 173913) pairs with kernel (bugzilla 173912) > > 2.6.9-25 that requires an updated version of linux/fs.h. We move the > > locking of i_sem and i_alloc_sem out of kernel directIO routine into > > gfs read/write path. This is to re-arrange lock order so we don't get > > into deadlock situation as described in the bugzilla. > > > > Patches: > > http://sourceware.org/cgi-bin/cvsweb.cgi/cluster/gfs-kernel/src/gfs/ops_address.c.diff?cvsroot=cluster&only_with_tag=RHEL4&r1=1.5.2.2&r2=1.5.2.3 > > http://sourceware.org/cgi-bin/cvsweb.cgi/cluster/gfs-kernel/src/gfs/ops_file.c.diff?cvsroot=cluster&only_with_tag=RHEL4&r1=1.16.2.6&r2=1.16.2.7 > > Hi Wendy, > > doesn't this patch require porting to the STABLE branch as well? > > I can't find a direct_IO_cluster_locking on vanilla kernels so i assume this is > a specific redhat extension. > > if i understand the problem right, shouldn't the stable branch get the patch > using blockdev_direct_IO_own_locking ? > It requires base kernel changes and we haven't decided what to do with it yet. -- Wendy From erling.nygaard at gmail.com Mon Jan 2 15:26:18 2006 From: erling.nygaard at gmail.com (Erling Nygaard) Date: Mon, 2 Jan 2006 16:26:18 +0100 Subject: [Linux-cluster] doubts about using clvm In-Reply-To: <43B930BB.5060800@gmail.com> References: <43B930BB.5060800@gmail.com> Message-ID: Some nitpicking first :-) There is no such thing as a "GFS server" or "GFS client". The nodes that are mounting a GFS filesystem are usually called "GFS nodes" However, there are indeed "GNBD servers" and "GNBD clients that exports and imports blockdevices. So I understand your setup to be one node (the Ubuntu node) is acting as a GNBD server and exports two disk drives. And then there are two nodes (the RHEL4 nodes) that acts as GNBD clients and imports the two disk drives. On these disk drives is the GFS filesystem. I have not used CLVM with GFS yet, but with the old pool-driver (the volume manager that was shipped with GFS previously) this setup could be done in 2 different ways: 1. Combine the two disk drives on the GNBD server and export _one_ volume 2. Export _two_ volumes from the GNBD server and combine the volumes on the GNBD client / GFS nodes. I would assume both ways would work with CLVM. On 1/2/06, carlopmart wrote: > > Hi all, > > I am testing Cluster/GFS suite under my vmware lab. I have installed > two RHEL 4 with CS and GFS support. Another machine acts as a GFS > server (Ubuntu breezy server that has rhcs/gfs included). > > Ubuntu server has two phisycal disks exported by gnbd daemon and > RHEL import these disks as a GFS clients. But I would like to export > these disks as a unique disk using LVM. Then, I need to start clvm > process under Ubuntu server or on RHEL clients??? Is it possible to > accomplish this?. > > Thanks. > -- > CL Martinez > carlopmart {at} gmail {d0t} com > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- - Mac OS X. Because making Unix user-friendly is easier than debugging Windows -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlopmart at gmail.com Mon Jan 2 15:35:31 2006 From: carlopmart at gmail.com (carlopmart) Date: Mon, 02 Jan 2006 16:35:31 +0100 Subject: [Linux-cluster] doubts about using clvm In-Reply-To: References: <43B930BB.5060800@gmail.com> Message-ID: <43B94843.5010801@gmail.com> Sorry my poor vocabulary :) Erling Nygaard wrote: > Some nitpicking first :-) > There is no such thing as a "GFS server" or "GFS client". The nodes that > are mounting a GFS filesystem are usually called "GFS nodes" > However, there are indeed "GNBD servers" and "GNBD clients that exports > and imports blockdevices. > > So I understand your setup to be one node (the Ubuntu node) is acting as > a GNBD server and exports two disk drives. Correct. > And then there are two nodes (the RHEL4 nodes) that acts as GNBD clients > and imports the two disk drives. Correct. On these disk drives is the GFS filesystem. No, these are raw partiitons as describres in sources.redhat.com (min-gfs.txt). > > I have not used CLVM with GFS yet, but with the old pool-driver (the > volume manager that was shipped with GFS previously) this setup could be > done in 2 different ways: > 1. Combine the two disk drives on the GNBD server and export _one_ volume In this case I need to run clvm on ubuntu server, right?? > 2. Export _two_ volumes from the GNBD server and combine the volumes on > the GNBD client / GFS nodes. > > I would assume both ways would work with CLVM. > > > On 1/2/06, *carlopmart* > wrote: > > Hi all, > > I am testing Cluster/GFS suite under my vmware lab. I have installed > two RHEL 4 with CS and GFS support. Another machine acts as a GFS > server (Ubuntu breezy server that has rhcs/gfs included). > > Ubuntu server has two phisycal disks exported by gnbd daemon and > RHEL import these disks as a GFS clients. But I would like to export > these disks as a unique disk using LVM. Then, I need to start clvm > process under Ubuntu server or on RHEL clients??? Is it possible to > accomplish this?. > > Thanks. > -- > CL Martinez > carlopmart {at} gmail {d0t} com > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > -- > - > Mac OS X. Because making Unix user-friendly is easier than debugging > Windows > > > ------------------------------------------------------------------------ > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- CL Martinez carlopmart {at} gmail {d0t} com From erling.nygaard at gmail.com Mon Jan 2 17:18:08 2006 From: erling.nygaard at gmail.com (Erling Nygaard) Date: Mon, 2 Jan 2006 18:18:08 +0100 Subject: [Linux-cluster] doubts about using clvm In-Reply-To: <43B94843.5010801@gmail.com> References: <43B930BB.5060800@gmail.com> <43B94843.5010801@gmail.com> Message-ID: On 1/2/06, carlopmart wrote: > Sorry my poor vocabulary :) No problem. Common mistake :-) > > Erling Nygaard wrote: > > Some nitpicking first :-) > > There is no such thing as a "GFS server" or "GFS client". The nodes that > > are mounting a GFS filesystem are usually called "GFS nodes" > > However, there are indeed "GNBD servers" and "GNBD clients that exports > > and imports blockdevices. > > > > So I understand your setup to be one node (the Ubuntu node) is acting as > > a GNBD server and exports two disk drives. > > Correct. > > > And then there are two nodes (the RHEL4 nodes) that acts as GNBD clients > > and imports the two disk drives. > > Correct. > On these disk drives is the GFS filesystem. > > No, these are raw partiitons as describres in sources.redhat.com > (min-gfs.txt). Oh. Well, these are the partitions you intend to make GFS on then :-) > > > > I have not used CLVM with GFS yet, but with the old pool-driver (the > > volume manager that was shipped with GFS previously) this setup could be > > done in 2 different ways: > > 1. Combine the two disk drives on the GNBD server and export _one_ volume > In this case I need to run clvm on ubuntu server, right?? > Yes. In this case clvm daemon runs on the ubuntu server. There is one GNBD server exports the one LVM volume. In your setup I think this would be the easier way to do the setup. Your two disk drives will be combined on the GNBD server, so there is only one volume to be exported by the GNBD server. So on the GNBD clients you don't have to worry about having to combine two GNBD imports. With the second method you would have to run CLVM on both GFND clients (GFS nodes) to combine the exported disk drives. Other than somewhat simpler setup I don't thin there is any good reason for one over the other. As I said, I have only done this with pool, not CLVM. Other speak up if this for some bizarre reason would not work. > > 2. Export _two_ volumes from the GNBD server and combine the volumes on > > the GNBD client / GFS nodes. > > > > I would assume both ways would work with CLVM. > > > > > > On 1/2/06, *carlopmart* > > wrote: > > > > Hi all, > > > > I am testing Cluster/GFS suite under my vmware lab. I have installed > > two RHEL 4 with CS and GFS support. Another machine acts as a GFS > > server (Ubuntu breezy server that has rhcs/gfs included). > > > > Ubuntu server has two phisycal disks exported by gnbd daemon and > > RHEL import these disks as a GFS clients. But I would like to export > > these disks as a unique disk using LVM. Then, I need to start clvm > > process under Ubuntu server or on RHEL clients??? Is it possible to > > accomplish this?. > > > > Thanks. > > -- > > CL Martinez > > carlopmart {at} gmail {d0t} com -- - Mac OS X. Because making Unix user-friendly is easier than debugging Windows From carlopmart at gmail.com Mon Jan 2 18:00:52 2006 From: carlopmart at gmail.com (carlopmart) Date: Mon, 02 Jan 2006 19:00:52 +0100 Subject: [Linux-cluster] doubts about using clvm In-Reply-To: References: <43B930BB.5060800@gmail.com> <43B94843.5010801@gmail.com> Message-ID: <43B96A54.7070904@gmail.com> Thanks Erling. But I have last question. I will try to combine two disks on GFS clients side. If I understand correct, first i need to import gnbd devices on both GFS nodes, right??. And second, I need to setup lvm from GFS nodes and start clvm service on both nodes too. But, I need to create shared lvm disk on both GFS nodes or only on one node??? Erling Nygaard wrote: > On 1/2/06, carlopmart wrote: > >>Sorry my poor vocabulary :) > > > No problem. Common mistake :-) > >>Erling Nygaard wrote: >> >>>Some nitpicking first :-) >>>There is no such thing as a "GFS server" or "GFS client". The nodes that >>>are mounting a GFS filesystem are usually called "GFS nodes" >>>However, there are indeed "GNBD servers" and "GNBD clients that exports >>>and imports blockdevices. >>> >>>So I understand your setup to be one node (the Ubuntu node) is acting as >>>a GNBD server and exports two disk drives. >> >>Correct. >> >> >>>And then there are two nodes (the RHEL4 nodes) that acts as GNBD clients >>>and imports the two disk drives. >> >>Correct. >> On these disk drives is the GFS filesystem. >> >>No, these are raw partiitons as describres in sources.redhat.com >>(min-gfs.txt). > > > Oh. Well, these are the partitions you intend to make GFS on then :-) > > >>>I have not used CLVM with GFS yet, but with the old pool-driver (the >>>volume manager that was shipped with GFS previously) this setup could be >>>done in 2 different ways: >>>1. Combine the two disk drives on the GNBD server and export _one_ volume >> >>In this case I need to run clvm on ubuntu server, right?? >> > > Yes. In this case clvm daemon runs on the ubuntu server. There is one > GNBD server exports the one LVM volume. > > In your setup I think this would be the easier way to do the setup. > Your two disk drives will be combined on the GNBD server, so there is > only one volume to be exported by the GNBD server. > So on the GNBD clients you don't have to worry about having to combine > two GNBD imports. > > With the second method you would have to run CLVM on both GFND clients > (GFS nodes) to combine the exported disk drives. > > Other than somewhat simpler setup I don't thin there is any good > reason for one over the other. > > As I said, I have only done this with pool, not CLVM. Other speak up > if this for some bizarre reason would not work. > > > >>>2. Export _two_ volumes from the GNBD server and combine the volumes on >>>the GNBD client / GFS nodes. >>> >>>I would assume both ways would work with CLVM. >>> >>> >>>On 1/2/06, *carlopmart* >>> wrote: >>> >>> Hi all, >>> >>> I am testing Cluster/GFS suite under my vmware lab. I have installed >>> two RHEL 4 with CS and GFS support. Another machine acts as a GFS >>> server (Ubuntu breezy server that has rhcs/gfs included). >>> >>> Ubuntu server has two phisycal disks exported by gnbd daemon and >>> RHEL import these disks as a GFS clients. But I would like to export >>> these disks as a unique disk using LVM. Then, I need to start clvm >>> process under Ubuntu server or on RHEL clients??? Is it possible to >>> accomplish this?. >>> >>> Thanks. >>> -- >>> CL Martinez >>> carlopmart {at} gmail {d0t} com > > > -- > - > Mac OS X. Because making Unix user-friendly is easier than debugging Windows > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- CL Martinez carlopmart {at} gmail {d0t} com From sara_sodagar at yahoo.com Tue Jan 3 07:58:27 2006 From: sara_sodagar at yahoo.com (sara sodagar) Date: Mon, 2 Jan 2006 23:58:27 -0800 (PST) Subject: [Linux-cluster] urgenet question about ClusterSuit Message-ID: <20060103075827.21933.qmail@web31810.mail.mud.yahoo.com> Hi I have question about clustering as follows: we are planning to service 200000 customers for internet services such as webhosting,email,DNS.We have designed about 8 servers for each service to run a share of the load.I want to use Clustersuit as well as GFS as we are making use of SAN. As I have studied mannuals of clustersuit , it seems that there is only one service up in each clustersystems , my question is we this high number of users How can I rely just on one servers , I know that another server in failoverdomain comesup incase of first server failure , but what about load balancing? how can I have all of me servers running in my clustersystems ? I have studied about pirnaha for IP load balancing but it seems that I can not use GFS with Pirnaha? I appreciate If any one could help me reagrding this issue? Regards. Sara __________________________________________ Yahoo! DSL ? Something to write home about. Just $16.99/mo. or less. dsl.yahoo.com From gwood at dragonhold.org Tue Jan 3 08:19:34 2006 From: gwood at dragonhold.org (gwood at dragonhold.org) Date: Tue, 3 Jan 2006 08:19:34 -0000 (GMT) Subject: [Linux-cluster] urgenet question about ClusterSuit In-Reply-To: <20060103075827.21933.qmail@web31810.mail.mud.yahoo.com> References: <20060103075827.21933.qmail@web31810.mail.mud.yahoo.com> Message-ID: <5726.208.178.77.200.1136276374.squirrel@mnementh.dragonhold.org> > As I have studied mannuals of clustersuit , it seems > that there is only one service up in each > clustersystems , my question is we this high number of > users How can I rely just on one servers , I know that > another server in failoverdomain comesup incase of > first server failure , but what about load balancing? Load balancing is actually a totally different issue to the HA side of the cluster. You are probably best using GFS to provide shared access to the SAN, and then just running local copies of the apps on each machine - since you don't want the applications to failover to other servers and instead just stay where they are. > I have studied about pirnaha for IP load balancing but > it seems that I can not use GFS with Pirnaha? Depending on budget & expertise within your company, I think it would probably be easiest to use a pair of hardware load balancers in this case - since that way you can get things like session affinity very easily. If you'd rather stick with a Linux solution throughout, then piranha is probably the most appropriate (unless you want to just configure LVS yourself). I've only just quickly looked through the docs on the redhat site, and it looks like the piranha configuration can be totally separate to the application servers - the 'real' servers don't even need to be Linux if you need to run something else behind it... So using GFS/clustering on the application servers and piranha on the front end servers should work fine... However, you /might/ get better performance by using an active/passive configuration of machines attached to the SAN and other applications using NFS - depending on the usage of the data & locking performance of GFS (e.g. You could setup an HA NFS system using cluster suite and 2 nodes such that one shares out web pages and the other shares out mailboxes - but that they can each do both if the other fails. Then all the application servers can mount the NFS share and use that with possibly faster response times) From pcaulfie at redhat.com Tue Jan 3 08:44:44 2006 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 03 Jan 2006 08:44:44 +0000 Subject: [Linux-cluster] help on DLM In-Reply-To: <3beebd3bee07.3bee073beebd@vsnl.net> References: <3beebd3bee07.3bee073beebd@vsnl.net> Message-ID: <43BA397C.2030404@redhat.com> renapte at vsnl.net wrote: > I would like to know how to use DLM as a stand alone component without GFS. Is any document available on how to set it up by itself. Im using linux 2.6.9 > Thank you > Renuka > Setting up the DLM is just a subset of setting up the full GFS cluster. You don't need fencing or (obviously) GFS). The DLM itself needs no configuration so all you need to do is set up cman itself - this requires a cluster.conf file and ccsd running. So the procedure is roughly: - create a cluster.conf file - load cman & dlm kernel modules - start ccsd on all nodes - cman_tool join enjoy. -- patrick From pcaulfie at redhat.com Tue Jan 3 08:47:02 2006 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 03 Jan 2006 08:47:02 +0000 Subject: [Linux-cluster] Starting cman failed. In-Reply-To: <2315046d0512280404p59498d9ag6541433b5810c3ca@mail.gmail.com> References: <2315046d0512280404p59498d9ag6541433b5810c3ca@mail.gmail.com> Message-ID: <43BA3A06.6060809@redhat.com> Manilal K M wrote: > Hi all, > I am a newbie to Clusters and GFS. I have recently installed redhat > cluster on my FC-4 machine. The following packages are installed: > rgmanager-1.9.34-5 > system-config-cluster-1.0.12-1.0 > ccs-1.0.0-1 > magma-1.0.0-1 > magma-plugins-1.0.0-2 > cman-kernel-2.6.11.5-20050601.152643.FC4.18 > cman-1.0.0-1 > dlm-kernel-2.6.11.5-20050601.152643.FC4.17 > dlm-1.0.0-3 > fence-1.32.1-1 > gulm-1.0.0-2 > iddev-2.0.0-1 > GFS-6.1.0-3 > GFS-kernel-2.6.11.8-20050601.152643.FC4.20 > gnbd-1.0.0-1 > gnbd-kernel-2.6.11.2-20050420.133124.FC4.53 > The Kernel version is : 2.6.11-1.1369_FC4smp > > I have created the clusters with the graphical tool, and after that > started the service in this order: > > [root at triveni ~]# service ccsd start > Starting ccsd: [ OK ] > [root at triveni ~]# service cman start > Starting cman: [FAILED] > [root at triveni ~]# service fenced start > Starting fence domain: [FAILED] > [root at triveni ~]# service gfs start > [root at triveni ~]# service rgmanager start > Starting Cluster Service Manager: [ OK ] > [root at triveni ~]# > > The /var/log/messages displays the following errors: > [root at triveni ~]# tail -f /var/log/messages|grep clu > Dec 28 17:27:54 triveni ccsd[4319]: Unable to connect to cluster > infrastructure after 1980 seconds. > Dec 28 17:28:25 triveni ccsd[4319]: Unable to connect to cluster > infrastructure after 2010 seconds. > Dec 28 17:30:04 triveni ccsd[8842]: Unable to connect to cluster > infrastructure after 30 seconds. > Dec 28 17:30:34 triveni ccsd[8842]: Unable to connect to cluster > infrastructure after 60 seconds. > Dec 28 17:30:48 triveni clurgmgrd[9045]: Loading Service Data > Dec 28 17:30:48 triveni clurgmgrd[9045]: #5: Couldn't connect to ccsd! > Dec 28 17:30:48 triveni clurgmgrd[9045]: #8: Couldn't initialize services > Dec 28 17:31:05 triveni ccsd[8842]: Unable to connect to cluster > infrastructure after 90 seconds. > > What could be wrong?? Please help me. > Are there any CMAN messages in syslog? Try running cman_tool join from the command-line to see if it produces an error. And check your cluster.conf file too, a duff one is the most likely cause. patrick From pcaulfie at redhat.com Tue Jan 3 08:50:28 2006 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 03 Jan 2006 08:50:28 +0000 Subject: [Linux-cluster] doubts about using clvm In-Reply-To: <43B94843.5010801@gmail.com> References: <43B930BB.5060800@gmail.com> <43B94843.5010801@gmail.com> Message-ID: <43BA3AD4.9020103@redhat.com> carlopmart wrote: >> >> I have not used CLVM with GFS yet, but with the old pool-driver (the >> volume manager that was shipped with GFS previously) this setup could >> be done in 2 different ways: >> 1. Combine the two disk drives on the GNBD server and export _one_ volume > > In this case I need to run clvm on ubuntu server, right?? No, just normal LVM. If the server isn't part of the cluster, just serving disks to it, then you don't need the cluster metadata synchronisation. -- patrick From libregeek at gmail.com Tue Jan 3 09:25:59 2006 From: libregeek at gmail.com (Manilal K M) Date: Tue, 3 Jan 2006 14:55:59 +0530 Subject: [Linux-cluster] Starting cman failed. In-Reply-To: <43BA3A06.6060809@redhat.com> References: <2315046d0512280404p59498d9ag6541433b5810c3ca@mail.gmail.com> <43BA3A06.6060809@redhat.com> Message-ID: <2315046d0601030125w754b15f6g3d305b7005233ed7@mail.gmail.com> > > Are there any CMAN messages in syslog? > Try running cman_tool join from the command-line to see if it produces an error. > > And check your cluster.conf file too, a duff one is the most likely cause. Just now I reconfigured cluster suite in my machine and cman started without any error. I have followed the same steps that I have done earlier. This time I think that there is only problem GFS, which was not configured. I'm happy that CCS is working. However I will post the messages in /var/log/messages : Jan 3 14:46:58 triveni ccsd[7400]: Starting ccsd 1.0.0: Jan 3 14:46:58 triveni ccsd[7400]: Built: Jun 16 2005 10:45:39 Jan 3 14:46:58 triveni ccsd[7400]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Jan 3 14:47:09 triveni kernel: CMAN 2.6.11.5-20050601.152643.FC4.18 (built Dec 14 2005 11:44:37) installed Jan 3 14:47:09 triveni kernel: NET: Registered protocol family 30 Jan 3 14:47:09 triveni ccsd[7400]: cluster.conf (cluster name = alpha_cluster, version = 5) found. Jan 3 14:47:10 triveni kernel: CMAN: Waiting to join or form a Linux-cluster Jan 3 14:47:11 triveni ccsd[7400]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1.2 Jan 3 14:47:11 triveni ccsd[7400]: Initial status:: Inquorate Jan 3 14:47:42 triveni kernel: CMAN: forming a new cluster Jan 3 14:47:42 triveni kernel: CMAN: quorum regained, resuming activity Jan 3 14:47:42 triveni ccsd[7400]: Cluster is quorate. Allowing connections. Jan 3 14:47:42 triveni kernel: dlm: no version for "struct_module" found: kernel tainted. Jan 3 14:47:42 triveni kernel: DLM 2.6.11.5-20050601.152643.FC4.17 (built Dec 14 2005 11:48:32) installed Jan 3 14:48:32 triveni clurgmgrd[7502]: Loading Service Data Jan 3 14:48:32 triveni clurgmgrd[7502]: Initializing Services Jan 3 14:48:32 triveni clurgmgrd[7502]: Services Initialized Jan 3 14:48:32 triveni clurgmgrd[7502]: Logged in SG "usrm::manager" Jan 3 14:48:32 triveni clurgmgrd[7502]: Magma Event: Membership Change Jan 3 14:48:32 triveni clurgmgrd[7502]: State change: Local UP Jan 3 14:48:34 triveni clurgmgrd[7502]: Starting stopped service web server Jan 3 14:48:34 triveni clurgmgrd[7502]: Service web server started However it'll be nice if somebody can point to a step by step configuration of cluster & GFS in FC-4 other than http://www.redhat.com/docs/manuals/csgfs/. regards Manilal From Bowie_Bailey at BUC.com Tue Jan 3 16:30:52 2006 From: Bowie_Bailey at BUC.com (Bowie Bailey) Date: Tue, 3 Jan 2006 11:30:52 -0500 Subject: [Linux-cluster] doubts about using clvm Message-ID: <4766EEE585A6D311ADF500E018C154E302133482@bnifex.cis.buc.com> carlopmart wrote: > > Thanks Erling. But I have last question. I will try to combine two > disks on GFS clients side. If I understand correct, first i need to > import gnbd devices on both GFS nodes, right??. And second, I need to > setup lvm from GFS nodes and start clvm service on both nodes too. > But, I need to create shared lvm disk on both GFS nodes or only on one > node??? I am doing the same thing on my server using AoE drives rather than GNBD. You create the clvm volumes and GFS filesystem(s) from one node, and then use "vgscan" to load it all in on the second node. -- Bowie From carlopmart at gmail.com Tue Jan 3 16:37:24 2006 From: carlopmart at gmail.com (carlopmart) Date: Tue, 03 Jan 2006 17:37:24 +0100 Subject: [Linux-cluster] doubts about using clvm In-Reply-To: <4766EEE585A6D311ADF500E018C154E302133482@bnifex.cis.buc.com> References: <4766EEE585A6D311ADF500E018C154E302133482@bnifex.cis.buc.com> Message-ID: <43BAA844.5060708@gmail.com> Thanks Bowie. Bowie Bailey wrote: > carlopmart wrote: > >>Thanks Erling. But I have last question. I will try to combine two >>disks on GFS clients side. If I understand correct, first i need to >>import gnbd devices on both GFS nodes, right??. And second, I need to >>setup lvm from GFS nodes and start clvm service on both nodes too. >>But, I need to create shared lvm disk on both GFS nodes or only on one >>node??? > > > I am doing the same thing on my server using AoE drives rather than > GNBD. > > You create the clvm volumes and GFS filesystem(s) from one node, and then > use "vgscan" to load it all in on the second node. > -- CL Martinez carlopmart {at} gmail {d0t} com From ian at blenke.com Tue Jan 3 18:18:04 2006 From: ian at blenke.com (Ian Blenke) Date: Tue, 03 Jan 2006 13:18:04 -0500 Subject: [Linux-cluster] doubts about using clvm In-Reply-To: <4766EEE585A6D311ADF500E018C154E302133482@bnifex.cis.buc.com> References: <4766EEE585A6D311ADF500E018C154E302133482@bnifex.cis.buc.com> Message-ID: <43BABFDC.8020202@blenke.com> Bowie Bailey wrote: > carlopmart wrote: > >> Thanks Erling. But I have last question. I will try to combine two >> disks on GFS clients side. If I understand correct, first i need to >> import gnbd devices on both GFS nodes, right??. And second, I need to >> setup lvm from GFS nodes and start clvm service on both nodes too. >> But, I need to create shared lvm disk on both GFS nodes or only on one >> node??? >> > > I am doing the same thing on my server using AoE drives rather than > GNBD. > > You create the clvm volumes and GFS filesystem(s) from one node, and then > use "vgscan" to load it all in on the second node. > When a node goes down/is rebooted, how do you restore the "down, closewait" state on the remaining nodes that refer to that vblade/vblade-kernel? The "solution" appears to be stop lvm (to release open file handles to the /dev/etherd/e?.? devices), unload "aoe", and reload "aoe". On the remaining "good" nodes. This particular problem has me looking at gnbd devices again. If aoe were truly stateless, and the aoe clients could recover seamlessly on the restore of a vblade server, I'd have no issues. - Ian C. Blenke http://ian.blenke.com/ From Bowie_Bailey at BUC.com Tue Jan 3 19:44:50 2006 From: Bowie_Bailey at BUC.com (Bowie Bailey) Date: Tue, 3 Jan 2006 14:44:50 -0500 Subject: [Linux-cluster] doubts about using clvm Message-ID: <4766EEE585A6D311ADF500E018C154E302133484@bnifex.cis.buc.com> Ian Blenke wrote: > Bowie Bailey wrote: > > carlopmart wrote: > > > > > Thanks Erling. But I have last question. I will try to combine two > > > disks on GFS clients side. If I understand correct, first i need > > > to import gnbd devices on both GFS nodes, right??. And second, I > > > need to setup lvm from GFS nodes and start clvm service on both > > > nodes too. But, I need to create shared lvm disk on both GFS > > > nodes or only on one node??? > > > > > > > I am doing the same thing on my server using AoE drives rather than > > GNBD. > > > > You create the clvm volumes and GFS filesystem(s) from one node, > > and then use "vgscan" to load it all in on the second node. > > When a node goes down/is rebooted, how do you restore the "down, > closewait" state on the remaining nodes that refer to that > vblade/vblade-kernel? > > The "solution" appears to be stop lvm (to release open file handles to > the /dev/etherd/e?.? devices), unload "aoe", and reload "aoe". On the > remaining "good" nodes. > > This particular problem has me looking at gnbd devices again. > > If aoe were truly stateless, and the aoe clients could recover > seamlessly on the restore of a vblade server, I'd have no issues. I'm not sure what you mean. I have shutdown my GFS nodes several times now without any effect whatsoever on the remaining nodes. The only issue I have had has been with fencing since I am currently using manual fencing while trying to get my WTI power switches configured. We still need to do quite a bit of testing on this setup, so it's possible there are problems that I have not encountered yet, but so far it has worked very well for me. -- Bowie From ptader at fnal.gov Tue Jan 3 22:39:10 2006 From: ptader at fnal.gov (Paul Tader) Date: Tue, 03 Jan 2006 16:39:10 -0600 Subject: [Linux-cluster] Kernel panic: GFS: Assertion failed on line 1227 of file rgrp.c Message-ID: <43BAFD0E.2030702@fnal.gov> Were getting consistent kernel panics with most of our GFS nodes all pointing to the same line and file: Kernel panic: GFS: Assertion failed on line 1227 of file rgrp.c The longest uptime we've seen was a couple weeks, but for the most part the nodes will only stay up for a few days. Once one goes down, a couple more follow, not immediately, but within the hour. Configured are 5 nodes, three lock_gulm servers and two clients. Currently all are running the same version kernel, GFS and GFS modules as list below. - GFS-modules-smp-6.0.2.27-0 - GFS-6.0.2.27-0 - 2.4.21-37.ELsmp - Scientific Linux Release 303 (Fermi), RHES 3, update 3 really. - This is a new installation. From /var/log/messages: Jan 2 12:46:48 fnd0374 kernel: ce1bbbac f8bc7b72 00000246 00001000 db122da8 f8bf4000 db122da8 f8bc7d70 Jan 2 12:46:48 fnd0374 kernel: 00000246 000001f0 00000000 55b6bd08 f60fdd8c 00000005 00000001 ffffffff Jan 2 12:46:48 fnd0374 kernel: f8be16d0 f8be8c8f f8be8bc4 000004cb 00000016 f60f7c00 00000006 f8bf4000 Jan 2 12:46:48 fnd0374 kernel: Call Trace: [] gfs_asserti [gfs] 0x32 (0xce1bbbb0) Jan 2 12:46:48 fnd0374 kernel: [] gmalloc [gfs] 0x20 (0xce1bbbc8) Jan 2 12:46:48 fnd0374 kernel: [] blkalloc_internal [gfs] 0x130 (0xce1bbbec) Jan 2 12:46:48 fnd0374 kernel: [] .rodata.str1.1 [gfs] 0x1da3 (0xce1bbbf0) Jan 2 12:46:48 fnd0374 kernel: [] .rodata.str1.1 [gfs] 0x1cd8 (0xce1bbbf4) Jan 2 12:46:48 fnd0374 kernel: [] gfs_blkalloc [gfs] 0x7b (0xce1bbc20) Jan 2 12:46:48 fnd0374 kernel: [] get_datablock [gfs] 0xfc (0xce1bbc4c) Jan 2 12:46:48 fnd0374 kernel: [] gfs_block_map [gfs] 0x333 (0xce1bbc70) Jan 2 12:46:48 fnd0374 kernel: [] find_or_create_page [kernel] 0x63 (0xce1bbc9c) Jan 2 12:46:48 fnd0374 kernel: [] gfs_dgetblk [gfs] 0x3c (0xce1bbcec) Jan 2 12:46:48 fnd0374 kernel: [] get_block [gfs] 0xb9 (0xce1bbd28) Jan 2 12:46:48 fnd0374 kernel: [] __block_prepare_write [kernel] 0x1ab (0xce1bbd64) Jan 2 12:46:48 fnd0374 kernel: [] block_prepare_write [kernel] 0x39 (0xce1bbda8) Jan 2 12:46:48 fnd0374 kernel: [] get_block [gfs] 0x0 (0xce1bbdbc) Jan 2 12:46:48 fnd0374 kernel: [] gfs_prepare_write [gfs] 0x12c (0xce1bbdc8) Jan 2 12:46:48 fnd0374 kernel: [] get_block [gfs] 0x0 (0xce1bbdd8) Jan 2 12:46:48 fnd0374 kernel: [] do_generic_file_write [kernel] 0x1e3 (0xce1bbdf4) Jan 2 12:46:48 fnd0374 kernel: [] do_do_write [gfs] 0x2ab (0xce1bbe48) Jan 2 12:46:48 fnd0374 kernel: [] do_write [gfs] 0x18b (0xce1bbe94) Jan 2 12:46:48 fnd0374 kernel: [] gfs_walk_vma [gfs] 0x12e (0xce1bbed0) Jan 2 12:46:48 fnd0374 kernel: [] sock_read [kernel] 0x96 (0xce1bbf50) Jan 2 12:46:48 fnd0374 kernel: [] gfs_write [gfs] 0x91 (0xce1bbf6c) Jan 2 12:46:48 fnd0374 kernel: [] do_write [gfs] 0x0 (0xce1bbf80) Jan 2 12:46:48 fnd0374 kernel: [] sys_write [kernel] 0x97 (0xce1bbf94) Jan 2 12:46:48 fnd0374 kernel: Jan 2 12:46:48 fnd0374 kernel: Kernel panic: GFS: Assertion failed on line 1227 of file rgrp.c Jan 2 12:46:48 fnd0374 kernel: GFS: assertion: "x <= length" Jan 2 12:46:48 fnd0374 kernel: GFS: time = 1136227608 Jan 2 12:46:48 fnd0374 kernel: GFS: fsid=d0recon:d0.4: RG = 71028427 Appreciate any help. Thanks, Paul -- =========================================================================== Paul Tader Fermi National Accelerator Lab; PO Box 500 Batavia, IL 60510-0500 From libregeek at gmail.com Wed Jan 4 08:38:58 2006 From: libregeek at gmail.com (Manilal K M) Date: Wed, 4 Jan 2006 14:08:58 +0530 Subject: [Linux-cluster] Linux Virtual Server Vs RedHat Cluster Message-ID: <2315046d0601040038l496b7c8fj8ee86abf0bc96994@mail.gmail.com> Hello all, I am a beginner in the world of cluster and high availability. I have two mid-range servers (DELL) intended for web hosting. Each has it's own disk storage of some TB each. I don't have any shared storage for both of these servers. Both the servers are loaded with FC-4. Currently one of the server is active(other one is not functional). Now I need to cluster both these machines , so that I can make the servers high available and high performance. But still the priority is for High-Availability. I have tested different cluster suites including redhat clusters. But I'm confused in benchmarking these software. Yesterday I came across Linux Virtual Server. I have read the documents at http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/pt-lvs.html. Now I'm confused more .. I'll be greatly obliged if someone can help in making a decision between redhat cluster suite and linux virtual server. If somebody have experience in both these kindly provide a comparative study including their pros & cons. Applications that are intended to be deployed are: apache, openldap, mysql, postgresql, asterisk, openvpn. thanks in advance regards Manilal From irwan at magnifix.com.my Wed Jan 4 09:09:07 2006 From: irwan at magnifix.com.my (Mohd Irwan Jamaluddin) Date: Wed, 04 Jan 2006 17:09:07 +0800 Subject: [Linux-cluster] Linux Virtual Server Vs RedHat Cluster In-Reply-To: <2315046d0601040038l496b7c8fj8ee86abf0bc96994@mail.gmail.com> References: <2315046d0601040038l496b7c8fj8ee86abf0bc96994@mail.gmail.com> Message-ID: <1136365747.3272.40.camel@kuli.magnifix.com.my> On Wed, 2006-01-04 at 14:08 +0530, Manilal K M wrote: > Hello all, > I am a beginner in the world of cluster and high availability. I > have two mid-range servers (DELL) intended for web hosting. Each has > it's own disk storage of some TB each. I don't have any shared storage > for both of these servers. Both the servers are loaded with FC-4. > Currently one of the server is active(other one is not functional). > Now I need to cluster both these machines , so that I can make the > servers high available and high performance. But still the priority is > for High-Availability. I have tested different cluster suites > including redhat clusters. But I'm confused in benchmarking these > software. > Yesterday I came across Linux Virtual Server. I have read the > documents at http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/pt-lvs.html. > Now I'm confused more .. > I'll be greatly obliged if someone can help in making a decision > between redhat cluster suite and linux virtual server. If somebody > have experience in both these kindly provide a comparative study > including their pros & cons. > Applications that are intended to be deployed are: > apache, openldap, mysql, postgresql, asterisk, openvpn. You're so confused, young padawan. Red Hat Cluster Suite (RHCS) consists of 2 main features that are High- Availability & Load-Balancing. The High-Availability feature (always referred as Red Hat Cluster Manager) comes from Kimberlite (originally from Mission Critical Linux, Inc.) project while Load-Balancing feature (always referred as Piranha) comes from Linux Virtual Server (LVS) project. High-Availability feature in RHCS: http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/pt- clumanager.html Load-Balancing feature in RHCS: http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/pt-lvs.html If you want a so-called pure High-Availability feature, LVS seems not for you. You may love this project: http://www.linux-ha.org/ . If you require commercial-based application + commercial support, don't hesitate to use Red Hat Cluster Manager. -- Regards, Mohd Irwan Jamaluddin System Engineer, Magnifix Sdn. Bhd. [http://www.magnifix.com.my/] Tel: +603 42705073 Fax: +603 42701960 From fajar at telkom.co.id Wed Jan 4 09:18:55 2006 From: fajar at telkom.co.id (Fajar A. Nugraha) Date: Wed, 04 Jan 2006 16:18:55 +0700 Subject: [Linux-cluster] Linux Virtual Server Vs RedHat Cluster In-Reply-To: <2315046d0601040038l496b7c8fj8ee86abf0bc96994@mail.gmail.com> References: <2315046d0601040038l496b7c8fj8ee86abf0bc96994@mail.gmail.com> Message-ID: <43BB92FF.9010809@telkom.co.id> Manilal K M wrote: >Yesterday I came across Linux Virtual Server. I have read the >documents at http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/pt-lvs.html. >Now I'm confused more .. > > Think of Linux Virtual Server (LVS) as a load balancer. You have one (or more) machine functions as load balancer that distributes the load to multiple "real" servers. Usually you need some kind of frontend to LVS, such as piranha or keepalived. Each one of this "real" server needs to know how to handle the requests in the same way. Usually this means each real server needs to have identical data, either synchronized at application level (the way MySQL cluster does), manually synchronized (e.g. with rsync), or shared data (via NFS or cluster file system). The GFS-part of redhat cluster can be used to provide cluster file system for your real servers, but you should also make sure that your application can operate on shared data without causing corruption. For example, you can have multiple servers running apache accessing the same data/files provided by gfs and it should work OK, but you should not have multiple MySQL server accessing the same files. Since your priority is HA (which means active-standby configuration is OK), you can simply use redhat cluster without GFS. The cluster suite can provide mechanism for checking your services, and restarting or relocating the service if it fails. This means that if you have only one service, it will only run one node, leaving the other node idle. But if you have multiple services (apache, MySQL, etc) you can distribute the services on both nodes, and if one node fails the other node can take over the services. If you need HP and HA you can use redhat cluster with GFS to provide cluster file system and run your services on both nodes, but you'll also need some kind of load balancer (such as LVS) to distribute the load between those two nodes. -- Fajar From libregeek at gmail.com Wed Jan 4 09:50:50 2006 From: libregeek at gmail.com (Manilal K M) Date: Wed, 4 Jan 2006 15:20:50 +0530 Subject: [Linux-cluster] Linux Virtual Server Vs RedHat Cluster In-Reply-To: <43BB92FF.9010809@telkom.co.id> References: <2315046d0601040038l496b7c8fj8ee86abf0bc96994@mail.gmail.com> <43BB92FF.9010809@telkom.co.id> Message-ID: <2315046d0601040150q7dcb6daeh86fca399267ac248@mail.gmail.com> > Think of Linux Virtual Server (LVS) as a load balancer. You have one (or > more) machine functions as load balancer that distributes the load to > multiple "real" servers. Usually you need some kind of frontend to LVS, > such as piranha or keepalived. So this meand that apart from my real two servers, I need to have aother couple of machines for load balancing, right ? > > Since your priority is HA (which means active-standby configuration is > OK), you can simply use redhat cluster without GFS. The cluster suite But then how can I make sure that data availability is attained? Since I don't have a shared storage, I think GFS+GNBD should be used. > > If you need HP and HA you can use redhat cluster with GFS to provide > cluster file system and run your services on both nodes, but you'll also > need some kind of load balancer (such as LVS) to distribute the load > between those two nodes. Thanks, this made my confusion lesser. I was actually searching for the purpose of LVS. and unluckily couldn't find one. From halomoan at powere2e.com Wed Jan 4 10:02:50 2006 From: halomoan at powere2e.com (Halomoan Chow) Date: Wed, 4 Jan 2006 18:02:50 +0800 Subject: [Linux-cluster] Looking For Understanding GFS Processes Message-ID: <002401c61116$09c117c0$100fcc0a@pc002> Dear All I don't understand why it's very hard to find GFS documentation. The GFS documentation that I mean here not like the GFS documentation published on Redhat Official Website. They only publish documents how to use and implement the GFS. I'm a newbie in GFS so I'm looking for the basic or how of the GFS works documentation. If you guys know it, please help me. Thank you. Regards, Halomoan From pcaulfie at redhat.com Wed Jan 4 10:16:31 2006 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Wed, 04 Jan 2006 10:16:31 +0000 Subject: [Linux-cluster] Looking For Understanding GFS Processes In-Reply-To: <002401c61116$09c117c0$100fcc0a@pc002> References: <002401c61116$09c117c0$100fcc0a@pc002> Message-ID: <43BBA07F.50607@redhat.com> Halomoan Chow wrote: > Dear All > > I don't understand why it's very hard to find GFS documentation. > The GFS documentation that I mean here not like the GFS documentation > published on Redhat Official Website. They only publish documents how to use > and implement the GFS. > > I'm a newbie in GFS so I'm looking for the basic or how of the GFS works > documentation. > > If you guys know it, please help me. > http://people.redhat.com/%7Eteigland/sca.pdf -- patrick From hlawatschek at atix.de Wed Jan 4 10:36:13 2006 From: hlawatschek at atix.de (Mark Hlawatschek) Date: Wed, 4 Jan 2006 11:36:13 +0100 Subject: [Linux-cluster] urgenet question about ClusterSuit In-Reply-To: <20060103075827.21933.qmail@web31810.mail.mud.yahoo.com> References: <20060103075827.21933.qmail@web31810.mail.mud.yahoo.com> Message-ID: <200601041136.13858.hlawatschek@atix.de> Hi Sara, > we are planning to service 200000 customers for > internet services such as webhosting,email,DNS.We have > designed about 8 servers for each service to run a > share of the load.I want to use Clustersuit as well as > GFS as we are making use of SAN. Very good. We've built some solutions for internet services based on a GFS storage cluster with great success. We also created a diskless shared root cluster environment (cluster boots from SAN and shares the same root partition) for enhanced management and scalability. We determined that using a shared root cluster extremely reduces the effort for system administration. Besides, the scalability of the cluster and replacement of cluster nodes can be done within minimum timeframe and minimum installation work. As you want to create a cluster with 8 nodes and more, you should also think about that. We are in progress of releasing the shared root software at source forge site. The project's name will be open-sharedroot. > As I have studied mannuals of clustersuit , it seems > that there is only one service up in each > clustersystems , my question is we this high number of > users How can I rely just on one servers , I know that > another server in failoverdomain comesup incase of > first server failure , but what about load balancing? > how can I have all of me servers running in my > clustersystems ? > I have studied about pirnaha for IP load balancing but > it seems that I can not use GFS with Pirnaha? > I appreciate If any one could help me reagrding this > issue? As this topic has been already discussed, he're just a short sum-up: - You need an IP load-balancer in front of your cluster. You can use a hardware solution or a software solution like Piranha or other packages using LVS to implement this. To achieve highest availability you shoud create at least an active/passive HA cluster. The nicest solution would be an active/active loadbalancing. - To host the services you need to create the GFS storage cluster (also think about using shared root). Depending on the type of service, the applications are running in parallel on each cluster node (e.g. apache, tomcat, php), or are integrated into a failover configuration (e.g. mySQL). Best Regards, Mark -- Gruss / Regards, Dipl.-Ing. Mark Hlawatschek Phone: +49-89 121 409-55 http://www.atix.de/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From bmarzins at redhat.com Wed Jan 4 20:53:13 2006 From: bmarzins at redhat.com (Benjamin Marzinski) Date: Wed, 4 Jan 2006 14:53:13 -0600 Subject: [Linux-cluster] gnbd configuration (newbie) In-Reply-To: References: Message-ID: <20060104205312.GB6995@phlogiston.msp.redhat.com> On Tue, Dec 20, 2005 at 09:57:42AM -0600, Frazier, Darrell USA CRC (Contractor) wrote: > I am trying to set up a 2 node test cluster using the Cluster Suite and > GFS. Shared storage consists of three GFS formatted partitions on an iscsi > jbod. What I need to know is this: The documentation is a little vague in > the configuration of gnbd as a fencing device. How do I use the gnbd > device to fence the 2 nodes. Do I have to setup the existing GFS formatted > partitions to be exported as a GNBD device? Thanx in advance. GNBD is like iscsi. A regular node with local storage can use gnbd_serv to act like an iscsi target, exporting the local storage to the network. The GFS nodes access the storage via the GNBD driver, much like they would using the iSCSI driver. If you are using GNBD to create shared storage, you use fence_gnbd to disable access to that shared storage. Since you are using iSCSI for your shared storeage, you cannot use fence_gnbd to disable access to that storage. Sorry. You need to use another fencing method. -Ben > > > Darrell J. Frazier > > Unix System Administrator > > US Army Combat Readiness Center > > Fort Rucker, Alabama 36362 > > Com: (334)255-2676 > > DSN: 558-3879 > > Email: darrell.frazier at crc.army.mil > > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From Darrell.Frazier at crc.army.mil Wed Jan 4 20:56:30 2006 From: Darrell.Frazier at crc.army.mil (Frazier, Darrell USA CRC (Contractor)) Date: Wed, 4 Jan 2006 14:56:30 -0600 Subject: [Linux-cluster] gnbd configuration (newbie) Message-ID: Thank you so much for your reply. Do you happen to have a link on how to configure manual fencing? Thanx!! -----Original Message----- From: Benjamin Marzinski [mailto:bmarzins at redhat.com] Sent: Wednesday, January 04, 2006 2:53 PM To: linux clustering Subject: Re: [Linux-cluster] gnbd configuration (newbie) On Tue, Dec 20, 2005 at 09:57:42AM -0600, Frazier, Darrell USA CRC (Contractor) wrote: > I am trying to set up a 2 node test cluster using the Cluster Suite and > GFS. Shared storage consists of three GFS formatted partitions on an iscsi > jbod. What I need to know is this: The documentation is a little vague in > the configuration of gnbd as a fencing device. How do I use the gnbd > device to fence the 2 nodes. Do I have to setup the existing GFS formatted > partitions to be exported as a GNBD device? Thanx in advance. GNBD is like iscsi. A regular node with local storage can use gnbd_serv to act like an iscsi target, exporting the local storage to the network. The GFS nodes access the storage via the GNBD driver, much like they would using the iSCSI driver. If you are using GNBD to create shared storage, you use fence_gnbd to disable access to that shared storage. Since you are using iSCSI for your shared storeage, you cannot use fence_gnbd to disable access to that storage. Sorry. You need to use another fencing method. -Ben > > > Darrell J. Frazier > > Unix System Administrator > > US Army Combat Readiness Center > > Fort Rucker, Alabama 36362 > > Com: (334)255-2676 > > DSN: 558-3879 > > Email: darrell.frazier at crc.army.mil > > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -------------- next part -------------- An HTML attachment was scrubbed... URL: From bmarzins at redhat.com Wed Jan 4 21:36:11 2006 From: bmarzins at redhat.com (Benjamin Marzinski) Date: Wed, 4 Jan 2006 15:36:11 -0600 Subject: [Linux-cluster] Reduced performance question In-Reply-To: <5d96567b0512182237t1a78a16cl4777bbca6b80a357@mail.gmail.com> References: <5d96567b0512182237t1a78a16cl4777bbca6b80a357@mail.gmail.com> Message-ID: <20060104213611.GC6995@phlogiston.msp.redhat.com> On Mon, Dec 19, 2005 at 08:37:56AM +0200, Raz Ben-Jehuda(caro) wrote: > i have been measuring the performance of my cluster. > It is compsed of 4 machines, each machine exports 1 sata disk (maxtor > maxline III), > and theses machines are connected 1 Gbps ethernel drive. > > Peformance : > I am getting about 40% decrease in performance when i simply read a file. > > 1. Would switching to infiniband make it faster ? I couldn't tell you offhand. If you are saturating your network bandwidth, then anything to increase your bandwidth will definitely increase performance. If not, then it's probably a toss-up. > 2. What is GNBD IO model ? the gnbd driver works just like any block device driver, except that instead of writing the data to a locally attached device, it sends the data to the server over the network, with a small (28 byte) header attached. Once the server has written the data to the underlying storage, a reply header is sent back. For reads, only a header is send to the server, and the server sends back the data with a reply header attached. On the server, there is one thread for each client/device pair. This thread simply looks at the header, performs the necessary IO on the local device, and sends back a reply header, along with the data for read request. If you have cached mode on, the server will go through the cache, and use readahead. If you don't, the server will use direct IO to the device. At any rate, in order to guarantee consistency in case of a crash, the server will not send the reply until the data is on disk (the underlying device is opened O_SYNC). Most of the slowness comes from the server. There is only a single thread per client/device pair, so you will not start on the next request until the current one completes to disk. The best solution to this would probably be async IO. That way, the thread could pass the IO to the underlying device as quickly as it comes in (up till it runs short of memory) and then reply to the client as that IO comepletes to disk. > 2.1 Does it use async io ? nope. It should. > multithreaded IO ? nope. With async IO, I don't think multiple threads would be necessary. > sync IO ? Yup. > > 2.2 When a GNBD servs a node , does it know what is amount of data it > needs > to fetch right from the start ? > Meaning , if a node asks for 1MB buffer, would GNBD read 1 MB > buffer > or several small chunks ? Since the server only processes on request at a time, it only needs as much memory as the largest request that the client can send. The kernel only coalesces so many sectors into a single request. The gnbd device driver can handle that default limit, so this should not be a factor. If aysnc io was added, you could have arbitrarily many requests being processes by the server at once, so you would either need to preallocate a fixed amount of memory, and wait to process a request until there was some memory free, or you would need to allocate memory dynamically, possibly up to some limit. This would be a great performance enhancement, but since it's only a matter of time until there will be rock solid software based iSCSI targets, it may never happen. :( -Ben > Thanks > -- > Raz > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From robert at deakin.edu.au Fri Jan 6 02:49:21 2006 From: robert at deakin.edu.au (Robert Ruge) Date: Fri, 6 Jan 2006 13:49:21 +1100 Subject: [Linux-cluster] clurgmgrd failing Message-ID: <200601060249.k062nLuS004116@deakin.edu.au> Good new year to you all. I have two identical Redhat ES 4 clusters running cluster 1.01 and GFS. On both clusters I fairly regularly have the clurgmgrd die for no apparent reason and with no errors in the logs that I can see. Sometimes just restarting it will be ok but other times I have to reboot the node. It appears to happen on both clusters and on any of the four nodes involved. Is there any way of turning on debugging so that I can get some idea of what is causing the problem. Thanks. Robert From lhh at redhat.com Fri Jan 6 18:32:06 2006 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 06 Jan 2006 13:32:06 -0500 Subject: [Linux-cluster] clurgmgrd failing In-Reply-To: <200601060249.k062nLuS004116@deakin.edu.au> References: <200601060249.k062nLuS004116@deakin.edu.au> Message-ID: <1136572326.923.5.camel@ayanami.boston.redhat.com> On Fri, 2006-01-06 at 13:49 +1100, Robert Ruge wrote: > Good new year to you all. > > I have two identical Redhat ES 4 clusters running cluster 1.01 and > GFS. On both clusters I fairly regularly have the clurgmgrd die for no > apparent reason and with no errors in the logs that I can see. > Sometimes just restarting it will be ok but other times I have to > reboot the node. It appears to happen on both clusters and on any of > the four nodes involved. Is there any way of turning on debugging so > that I can get some idea of what is causing the problem. The current stable branch and the upcoming release have tons of additional logging, and the ability to change clurgmgrd's logging facility and log level (in the config file). In addition, you can start clurgmgrd with the "-d" option - you'll just want to edit syslog.conf to make sure you get all the messages. -- Lon From woytek+ at cmu.edu Mon Jan 9 19:41:36 2006 From: woytek+ at cmu.edu (Jonathan Woytek) Date: Mon, 09 Jan 2006 14:41:36 -0500 Subject: [Linux-cluster] GFS 6.0 lt_high_locks value in cluster.ccs Message-ID: <43C2BC70.7020107@cmu.edu> I've been trying to improve the performance of our GFS system since it went live a little over a year ago. Recently, while trying to improve gigabit network performance on another machine that seemed to be having some serious issues, I disabled hyperthreading in the BIOS and performance went through the roof. "Good!" said I. Seeing the performance increase there, I also disabled hyperthreading on two of our main GFS servers providing samba and NFS access to data on GFS filesystems (our backbone is iSCSI over GbE). Performance also generally increased there, but I started to notice two other issues. Issue #1 is with samba performance: While general access and read/write speeds are much faster now, there are odd delays when trying to copy a file to the filesystem over Samba and when browsing around the filesystem via Samba. The copy to the filesystem delay occurs as soon as the client system (Windows XP) is told to copy a file to the mapped network drive--the window involved will hang for a few seconds, then the copy will start and will move along at a good speed. The delays during browsing happen only occasionally, and don't seem to be related to folder contents. Issue #2 MAY be the cause of Issue #1. This is hard to determine right now. Issue #2 is that we are now hitting the highwater mark for locks in lock_gulmd almost all day long. This used to happen only occasionally, so we didn't worry about it too much. When it used to happen in the past, it would cause the user experience for Samba users to display the hangs during navigation (though nobody ever mentioned a problem copying files to the system). So, now to my question: I read on this list in a previous post about the lt_high_locks value in cluster.ccs. Is this a value that can be changed during runtime, or am I going to have to bring all the lock_gulmd's down to change this value? jonathan -- Jonathan Woytek w: 412-681-3463 woytek+ at cmu.edu NREC Computing Manager c: 412-401-1627 KB3HOZ PGP Key available upon request From woytek+ at cmu.edu Mon Jan 9 21:28:11 2006 From: woytek+ at cmu.edu (Jonathan Woytek) Date: Mon, 09 Jan 2006 16:28:11 -0500 Subject: [Linux-cluster] GFS 6.0 lt_high_locks value in cluster.ccs In-Reply-To: <43C2BC70.7020107@cmu.edu> References: <43C2BC70.7020107@cmu.edu> Message-ID: <43C2D56B.9010106@cmu.edu> Jonathan Woytek wrote: ... > So, now to my question: I read on this list in a previous post about > the lt_high_locks value in cluster.ccs. Is this a value that can be > changed during runtime, or am I going to have to bring all the > lock_gulmd's down to change this value? A related question: Is there a way to see what processes are holding locks in lock_gulmd? jonathan -- Jonathan Woytek w: 412-681-3463 woytek+ at cmu.edu NREC Computing Manager c: 412-401-1627 KB3HOZ PGP Key available upon request From cfeist at redhat.com Mon Jan 9 22:51:46 2006 From: cfeist at redhat.com (Chris Feist) Date: Mon, 09 Jan 2006 16:51:46 -0600 Subject: [Linux-cluster] GFS 6.0 lt_high_locks value in cluster.ccs In-Reply-To: <43C2BC70.7020107@cmu.edu> References: <43C2BC70.7020107@cmu.edu> Message-ID: <43C2E902.3030504@redhat.com> Yes, issue #2 could definitely be the cause of your first issue. Unfortunately you'll need to bring down your cluster to change the value of lt_high_locks. What is its value currently? And how much memory do you have on your gulm lock servers? You'll need about 256M of RAM for gulm for every 1 Million locks (plus enough for any other process and kernel). On each of the gulm clients you can also cat /proc/gulm/lockspace to see which client is using most of the locks. Let us know what you find out. Thanks! Chris Jonathan Woytek wrote: > Issue #2 MAY be the cause of Issue #1. This is hard to determine right > now. Issue #2 is that we are now hitting the highwater mark for locks > in lock_gulmd almost all day long. This used to happen only > occasionally, so we didn't worry about it too much. When it used to > happen in the past, it would cause the user experience for Samba users > to display the hangs during navigation (though nobody ever mentioned a > problem copying files to the system). > > So, now to my question: I read on this list in a previous post about > the lt_high_locks value in cluster.ccs. Is this a value that can be > changed during runtime, or am I going to have to bring all the > lock_gulmd's down to change this value? > > jonathan From cfeist at redhat.com Mon Jan 9 22:53:05 2006 From: cfeist at redhat.com (Chris Feist) Date: Mon, 09 Jan 2006 16:53:05 -0600 Subject: [Linux-cluster] GFS 6.0 lt_high_locks value in cluster.ccs In-Reply-To: <43C2D56B.9010106@cmu.edu> References: <43C2BC70.7020107@cmu.edu> <43C2D56B.9010106@cmu.edu> Message-ID: <43C2E951.9020002@redhat.com> Jonathan Woytek wrote: > A related question: Is there a way to see what processes are holding > locks in lock_gulmd? I don't think you can, but you can see the number of locks that each gfs node is holding by looking at /proc/gulm/lockspace on the gfs nodes. Thanks, Chris From woytek+ at cmu.edu Tue Jan 10 03:37:35 2006 From: woytek+ at cmu.edu (Jonathan Woytek) Date: Mon, 09 Jan 2006 22:37:35 -0500 Subject: [Linux-cluster] GFS 6.0 lt_high_locks value in cluster.ccs In-Reply-To: <43C2E902.3030504@redhat.com> References: <43C2BC70.7020107@cmu.edu> <43C2E902.3030504@redhat.com> Message-ID: <43C32BFF.6030608@cmu.edu> Chris Feist wrote: > Yes, issue #2 could definitely be the cause of your first issue. > Unfortunately you'll need to bring down your cluster to change the value > of lt_high_locks. What is its value currently? And how much memory do > you have on your gulm lock servers? You'll need about 256M of RAM for > gulm for every 1 Million locks (plus enough for any other process and > kernel). > > On each of the gulm clients you can also cat /proc/gulm/lockspace to see > which client is using most of the locks. Thanks for the response! I figured I would probably have to bring down the cluster to change the highwater setting, but I was hoping a bit that it could be changed dynamically. Oh well. The value is currently at the default, which I want to say is something like 1.04M. These machines are both lock servers and samba/NFS servers, and have 4GB of RAM available (I have three lock servers in the cluster, and all three have 4GB of RAM). A previous RedHat service call has me running the hugemem kernel on all three (the issue there was that, under just light activity loading, lowmem would be exhausted and the machines would enter an OOM spiral of death). Now that I have turned off hyperthreading, though, memory usage seems to be dramatically lower than it was prior to that change. For instance, the machine running samba services has been running since I turned off hyperthreading on Friday night. Today, the machine was under some pretty heavy load. On a normal day, prior to the hyperthreading change, I'd be down to maybe 500MB of lowmem free right now (out of 3GB). The only way to completely reclaim that memory would be to reboot. So, now I'm sitting here looking at this machine, and it has 3.02GB of 3.31GB free. I'm going to have to let this run for a while to determine if this is a red herring, but it looks much better than it ever has in the past. Here's the interesting output from the /proc/gulm gadgets (note that, at the time I grabbed these, I was seeing the "more than the max" message logged to syslog between once and twice per minute, but not at the 10-second rate that I read about previously): [root at xxxxx root]# cat /proc/gulm/filesystems/data0 Filesystem: data0 JID: 0 handler_queue_cur: 0 handler_queue_max: 26584 [root at xxxxx root]# cat /proc/gulm/filesystems/data1 Filesystem: data1 JID: 0 handler_queue_cur: 0 handler_queue_max: 4583 [root at xxxxx root]# cat /proc/gulm/filesystems/data2 Filesystem: data2 JID: 0 handler_queue_cur: 0 handler_queue_max: 11738 [root at xxxxx root]# cat /proc/gulm/lockspace lock counts: total: 41351 unl: 29215 exl: 3 shd: 12055 dfr: 0 pending: 0 lvbs: 16758 lops: 12597867 [root at xxxxx root]# From wcheng at redhat.com Tue Jan 10 16:51:58 2006 From: wcheng at redhat.com (Wendy Cheng) Date: Tue, 10 Jan 2006 11:51:58 -0500 Subject: [Linux-cluster] aysnc io to too slow over redhat cluster when reading In-Reply-To: <5d96567b0512200301k5d93857eh28571d02b10e39ee@mail.gmail.com> References: <5d96567b0512200301k5d93857eh28571d02b10e39ee@mail.gmail.com> Message-ID: <43C3E62E.2060602@redhat.com> Raz Ben-Jehuda(caro) wrote: > I have created an asyncio io test reader : > I ran it over linux cluster volume of 1TB, 4 machines > with raid0 of stripe size 512K over ethernet. > when issuing 50 conucurrent iops ( io operations ) the tester > hangs ( CTRL+C doesn't work ) and sometimes it simply far too slow. > I assume the filesystem you used was ext3 ? -- Wendy From ryan.turnbull at utilitran.com Tue Jan 10 20:22:26 2006 From: ryan.turnbull at utilitran.com (rturnbull) Date: Tue, 10 Jan 2006 13:22:26 -0700 Subject: [Linux-cluster] SAN file system to be shared to multiple nodes Message-ID: <43C41782.80004@utilitran.com> Hello to all, I have some questions about setting up a shared file system for a SAN environment with multiple nodes connecting to a shared disk. Here is what I have: DS4000 IBM FC SAN. I have created a storage array using the IBM software. The disk is connected to by multiple systems. Basically, lets say 5 systems are connecting through a SAN FC switch to the same array. As I discovered, the hard way, I can't just format this partition/disk ext3 as if one system writes, the other systems can't see the file until all have been unmounted and remounted again. So I need to have a shared file system that used by all the nodes that will allow reads/writes to be recognized by all the systems connecting to the shared SAN array. So my question is, what is the best file system available that is production worthy and capable of running on a Slackware 10.2 distro. Naturally, I would guess a lot of people would say GFS or OpenGFS, however is it stable and production ready. What about other shared file systems such as InterMezzo or Coda? Any suggestions would be greatly appreciated. Ryan Turnbull Calgary, Alberta From lhh at redhat.com Tue Jan 10 20:56:44 2006 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 10 Jan 2006 15:56:44 -0500 Subject: [Linux-cluster] SAN file system to be shared to multiple nodes In-Reply-To: <43C41782.80004@utilitran.com> References: <43C41782.80004@utilitran.com> Message-ID: <1136926604.4725.96.camel@ayanami.boston.redhat.com> On Tue, 2006-01-10 at 13:22 -0700, rturnbull wrote: > So my question is, what is the best file system available that is > production worthy and capable of running on a Slackware 10.2 distro. > Naturally, I would guess a lot of people would say GFS or OpenGFS, GFS is production ready and stable - and designed precisely for what you are trying to do. However, I do not know if anyone has packaged it for Slackware. I know it has been packaged for other distributions (apart from Red Hat Enterprise Linux), though, such as Ubuntu and Fedora Core. > however is it stable and production ready. What about other shared file > systems such as InterMezzo or Coda? These are distributed/disconnected file systems which solve a different paradigm, and they require a server (or replicated servers). GFS (6.1) does not require a server - for locks or data. There is a distributed lock manager for managing internal metadata and POSIX locks, and all clients have direct access to the SAN storage. -- Lon From ryan.turnbull at utilitran.com Tue Jan 10 22:03:40 2006 From: ryan.turnbull at utilitran.com (rturnbull) Date: Tue, 10 Jan 2006 15:03:40 -0700 Subject: [Linux-cluster] SAN file system to be shared to multiple nodes In-Reply-To: <1136926604.4725.96.camel@ayanami.boston.redhat.com> References: <43C41782.80004@utilitran.com> <1136926604.4725.96.camel@ayanami.boston.redhat.com> Message-ID: <43C42F3C.909@utilitran.com> Hello Lon, Ok. I'm fine with it not packages directly with Slackware. However there are a few concerns that I have in meeting this demand that I have this SAN working in 2 weeks time. ;) (Damn bosses). Anyways, I know there is kinda two versions of GFS. The Redhat GFS or the OpenGFS project. I'm fine with OpenGFS and getting that to work. However the first things that I come across are kernel patches to make this work. Is there any kernel patches for 2.4.26 kernels. I noticed its 2.4.20 and 2.4.22 but nothing for 2.4.26. Who is the best to contact for support on OpenGFS? Sounds like the project is well, dead at the moment..... Thanks Ryan Lon Hohberger wrote: >On Tue, 2006-01-10 at 13:22 -0700, rturnbull wrote: > > > >>So my question is, what is the best file system available that is >>production worthy and capable of running on a Slackware 10.2 distro. >>Naturally, I would guess a lot of people would say GFS or OpenGFS, >> >> > >GFS is production ready and stable - and designed precisely for what you >are trying to do. > >However, I do not know if anyone has packaged it for Slackware. I know >it has been packaged for other distributions (apart from Red Hat >Enterprise Linux), though, such as Ubuntu and Fedora Core. > > > >>however is it stable and production ready. What about other shared file >>systems such as InterMezzo or Coda? >> >> > >These are distributed/disconnected file systems which solve a different >paradigm, and they require a server (or replicated servers). > >GFS (6.1) does not require a server - for locks or data. There is a >distributed lock manager for managing internal metadata and POSIX locks, >and all clients have direct access to the SAN storage. > >-- Lon > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > > From mwill at penguincomputing.com Tue Jan 10 22:02:45 2006 From: mwill at penguincomputing.com (Michael Will) Date: Tue, 10 Jan 2006 14:02:45 -0800 Subject: [Linux-cluster] SAN file system to be shared to multiple nodes Message-ID: <9D61626573F86843908C9AE8BDB43905511E54@jellyfish.highlyscyld.com> You most likely want a 2.6 kernel based system. AFAIK redhat gfs in a developer version is free to download, all GPL? Michael -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of rturnbull Sent: Tuesday, January 10, 2006 2:04 PM To: linux clustering Subject: Re: [Linux-cluster] SAN file system to be shared to multiple nodes Hello Lon, Ok. I'm fine with it not packages directly with Slackware. However there are a few concerns that I have in meeting this demand that I have this SAN working in 2 weeks time. ;) (Damn bosses). Anyways, I know there is kinda two versions of GFS. The Redhat GFS or the OpenGFS project. I'm fine with OpenGFS and getting that to work. However the first things that I come across are kernel patches to make this work. Is there any kernel patches for 2.4.26 kernels. I noticed its 2.4.20 and 2.4.22 but nothing for 2.4.26. Who is the best to contact for support on OpenGFS? Sounds like the project is well, dead at the moment..... Thanks Ryan Lon Hohberger wrote: >On Tue, 2006-01-10 at 13:22 -0700, rturnbull wrote: > > > >>So my question is, what is the best file system available that is >>production worthy and capable of running on a Slackware 10.2 distro. >>Naturally, I would guess a lot of people would say GFS or OpenGFS, >> >> > >GFS is production ready and stable - and designed precisely for what you >are trying to do. > >However, I do not know if anyone has packaged it for Slackware. I know >it has been packaged for other distributions (apart from Red Hat >Enterprise Linux), though, such as Ubuntu and Fedora Core. > > > >>however is it stable and production ready. What about other shared file >>systems such as InterMezzo or Coda? >> >> > >These are distributed/disconnected file systems which solve a different >paradigm, and they require a server (or replicated servers). > >GFS (6.1) does not require a server - for locks or data. There is a >distributed lock manager for managing internal metadata and POSIX locks, >and all clients have direct access to the SAN storage. > >-- Lon > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > > -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From ryan.turnbull at utilitran.com Tue Jan 10 22:31:46 2006 From: ryan.turnbull at utilitran.com (rturnbull) Date: Tue, 10 Jan 2006 15:31:46 -0700 Subject: [Linux-cluster] GPFS?? Message-ID: <43C435D2.2060009@utilitran.com> Hello, I sent in a question before about SAN connecting multiple nodes to same shared disk. I have been doing some reading and came across GPFS (Global Parallel File System). Since I have all this shinny new ibm hardware, is this a "free" software filesystem? Anyone heard of or used this technology? Let me know. Ryan From sequel at neofreak.org Tue Jan 10 22:53:57 2006 From: sequel at neofreak.org (DeadManMoving) Date: Tue, 10 Jan 2006 17:53:57 -0500 Subject: [Linux-cluster] GPFS?? In-Reply-To: <43C435D2.2060009@utilitran.com> References: <43C435D2.2060009@utilitran.com> Message-ID: <1136933637.11886.1.camel@saloon.neofreak.org> GPFS is only for the AIX operating system AFAIK. On Tue, 2006-01-10 at 15:31 -0700, rturnbull wrote: > Hello, > I sent in a question before about SAN connecting multiple nodes to > same shared disk. > > I have been doing some reading and came across GPFS (Global Parallel > File System). Since I have all this shinny new ibm hardware, is this a > "free" software filesystem? Anyone heard of or used this technology? > > Let me know. > > Ryan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From mwill at penguincomputing.com Tue Jan 10 22:56:13 2006 From: mwill at penguincomputing.com (Michael Will) Date: Tue, 10 Jan 2006 14:56:13 -0800 Subject: [Linux-cluster] GPFS?? Message-ID: <9D61626573F86843908C9AE8BDB43905511E74@jellyfish.highlyscyld.com> google for GPFS and linux and you find a plentitude of references. It is not just for AIX anymore. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of DeadManMoving Sent: Tuesday, January 10, 2006 2:54 PM To: linux clustering Subject: Re: [Linux-cluster] GPFS?? GPFS is only for the AIX operating system AFAIK. On Tue, 2006-01-10 at 15:31 -0700, rturnbull wrote: > Hello, > I sent in a question before about SAN connecting multiple nodes to > same shared disk. > > I have been doing some reading and came across GPFS (Global > Parallel File System). Since I have all this shinny new ibm hardware, > is this a "free" software filesystem? Anyone heard of or used this technology? > > Let me know. > > Ryan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From tabmowzo at us.ibm.com Tue Jan 10 22:57:46 2006 From: tabmowzo at us.ibm.com (Peter R. Badovinatz) Date: Tue, 10 Jan 2006 14:57:46 -0800 Subject: [Linux-cluster] GPFS?? In-Reply-To: <43C435D2.2060009@utilitran.com> References: <43C435D2.2060009@utilitran.com> Message-ID: <43C43BEA.1080604@us.ibm.com> rturnbull wrote: > Hello, > I sent in a question before about SAN connecting multiple nodes to > same shared disk. > > I have been doing some reading and came across GPFS (Global Parallel > File System). Since I have all this shinny new ibm hardware, is this a > "free" software filesystem? Anyone heard of or used this technology? Ryan, GPFS is not free software (neither free as in beer nor freedom). It does run on a number of distros. All kinds of information are at: http://www-03.ibm.com/servers/eserver/clusters/software/gpfs.html The Frequently Asked Questions link on that page http://publib.boulder.ibm.com/infocenter/clresctr/index.jsp?topic=/com.ibm.cluster.gpfs.doc/gpfs_faqs/gpfs_faqs.html has pointers to distro, pricing, and other info, if at this point you're still interested. Lots of free GPFS tuning, performance and other information is available (free for download) via IBM Redbooks. Go to http://www.redbooks.ibm.com/ and put "GPFS" in the Search box and look around for what's interesting to you. > > Let me know. > > Ryan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > Peter -- Peter R. Badovinatz aka 'Wombat' -- IBM Linux Technology Center preferred: tabmowzo at us.ibm.com / alternate: wombat at us.ibm.com These are my opinions and absolutely not official opinions of IBM, Corp. From sequel at neofreak.org Wed Jan 11 04:05:01 2006 From: sequel at neofreak.org (DeadManMoving) Date: Tue, 10 Jan 2006 23:05:01 -0500 Subject: [Linux-cluster] GPFS?? In-Reply-To: <9D61626573F86843908C9AE8BDB43905511E74@jellyfish.highlyscyld.com> References: <9D61626573F86843908C9AE8BDB43905511E74@jellyfish.highlyscyld.com> Message-ID: <1136952301.11886.3.camel@saloon.neofreak.org> Sorry, the last time i've checked then. On Tue, 2006-01-10 at 14:56 -0800, Michael Will wrote: > google for GPFS and linux and you find a plentitude of references. It is > > not just for AIX anymore. > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of DeadManMoving > Sent: Tuesday, January 10, 2006 2:54 PM > To: linux clustering > Subject: Re: [Linux-cluster] GPFS?? > > GPFS is only for the AIX operating system AFAIK. > > On Tue, 2006-01-10 at 15:31 -0700, rturnbull wrote: > > Hello, > > I sent in a question before about SAN connecting multiple nodes to > > > same shared disk. > > > > I have been doing some reading and came across GPFS (Global > > Parallel File System). Since I have all this shinny new ibm hardware, > > > is this a "free" software filesystem? Anyone heard of or used this > technology? > > > > Let me know. > > > > Ryan > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From wim.coekaerts at oracle.com Wed Jan 11 04:07:36 2006 From: wim.coekaerts at oracle.com (Wim Coekaerts) Date: Tue, 10 Jan 2006 20:07:36 -0800 Subject: [Linux-cluster] GPFS?? In-Reply-To: <1136952301.11886.3.camel@saloon.neofreak.org> References: <9D61626573F86843908C9AE8BDB43905511E74@jellyfish.highlyscyld.com> <1136952301.11886.3.camel@saloon.neofreak.org> Message-ID: <20060111040736.GB27775@ca-server1.us.oracle.com> and it's not free and it's not open. they have some gluelayer code for gpfs and that's it , the rest is binary objects. (of course if you look at the webpage you 'd almost thinks its open but nope.)... On Tue, Jan 10, 2006 at 11:05:01PM -0500, DeadManMoving wrote: > Sorry, the last time i've checked then. > > On Tue, 2006-01-10 at 14:56 -0800, Michael Will wrote: > > google for GPFS and linux and you find a plentitude of references. It is > > > > not just for AIX anymore. > > > > -----Original Message----- > > From: linux-cluster-bounces at redhat.com > > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of DeadManMoving > > Sent: Tuesday, January 10, 2006 2:54 PM > > To: linux clustering > > Subject: Re: [Linux-cluster] GPFS?? > > > > GPFS is only for the AIX operating system AFAIK. > > > > On Tue, 2006-01-10 at 15:31 -0700, rturnbull wrote: > > > Hello, > > > I sent in a question before about SAN connecting multiple nodes to > > > > > same shared disk. > > > > > > I have been doing some reading and came across GPFS (Global > > > Parallel File System). Since I have all this shinny new ibm hardware, > > > > > is this a "free" software filesystem? Anyone heard of or used this > > technology? > > > > > > Let me know. > > > > > > Ryan > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From tristram at ubernet.co.nz Wed Jan 11 04:16:10 2006 From: tristram at ubernet.co.nz (Tristram Cheer) Date: Wed, 11 Jan 2006 17:16:10 +1300 Subject: [Linux-cluster] Software Iscsi with Redhat Cluster Message-ID: <43C4868A.2060508@ubernet.co.nz> Hi all, is anyone in a production setting using software Iscsi Targets and Initiators as a side options to GNBD? i'm exploring all our options for a Xen/Cluster Suite n+2 server setup for our ISP and would like to hear peoples thoughts on the best option, rather than using a SAN with FC we have decided to go with a Intel Raid Array with standard linux to reduce inital costs and need to find out what people are using in production to export block devices. Thanks in advance Tristram From mykleb at no.ibm.com Wed Jan 11 07:22:31 2006 From: mykleb at no.ibm.com (Jan-Frode Myklebust) Date: Wed, 11 Jan 2006 08:22:31 +0100 Subject: [Linux-cluster] Re: GPFS?? References: <43C435D2.2060009@utilitran.com> Message-ID: On 2006-01-10, rturnbull wrote: > > I have been doing some reading and came across GPFS (Global Parallel > File System). Since I have all this shinny new ibm hardware, is this a > "free" software filesystem? Anyone heard of or used this technology? It's not free, but it's *very* cool, and I think quite simple to use compared to the other cluster file systems. We GPFS-users hang out at https://lists.sdsc.edu/mailman/listinfo.cgi/gpfs-general -jf From thomsonr at ucalgary.ca Wed Jan 11 07:46:00 2006 From: thomsonr at ucalgary.ca (Ryan Thomson) Date: Wed, 11 Jan 2006 00:46:00 -0700 (MST) Subject: [Linux-cluster] Software Iscsi with Redhat Cluster In-Reply-To: <43C4868A.2060508@ubernet.co.nz> References: <43C4868A.2060508@ubernet.co.nz> Message-ID: <1080.68.146.207.245.1136965560.squirrel@68.146.207.245> Hi Tristram, I'm just about to put a completely Linux based software iSCSI RedHat Cluster with GFS into production. We have four RHEL4AS machines acting as cluster nodes and an 8TB RHEL4AS server is exporting disk as two arrays/iSCSI targets to the cluster nodes. Several more storage boxes we already own (previously Linux servers exporting large arrays over NFS) are to be added as iSCSI targets. I am using the iSCSI initiator that comes with RHEL4U2, the cisco open source one it is I believe. For targets, I'm using the iSCSI Enterprise Target (http://sourceforge.net/projects/iscsitarget/). The only thing I found so far that doesn't seem to work is the iSCSI alias. I can't seem to get the alias I set in the target to show up on the initiator. I don't know if the problem is the target or the initiator as I haven't found anything online yet about this issue. The currently available Linux iSCSI software seems to work pretty much flawlessly for me otherwise. So far it's been quite easy and painless setting up CLVM volumes and putting GFS on them, I even wrote a basic wrapper script to do all the work for me, streamlining the proceedure. Filesystem expansion seems to work as expected. I haven't played with snapshots. Initial numbers show transfer rates from end to end (NFS clients to Cluster NFS server to GFS) to be better for iSCSI than GNBD. Keep in mind these are initial tests using bonnie++ and using 'time' to time file copies of various sizes, nothing concrete. I suspected NFS to be a bottleneck but it seems that storage interconnect/fabric protocol still makes a difference even with NFS being crappy to the clients. >From cluster nodes to storage I found transfer rates to be near local max with iSCSI, again don't trust me though, do you own tests. My hardware doesn't have very high end disk, just SATA with 3ware 9500 cards. I didn't do the cluster node to storage test with GNBD :( Anyways, so far my initial experience has been great. I solved an issue causing my cluster nodes to kernel panic and ever since, it's been running very well serving Apache, MySQL, OpenLDAP and NFS exports. I haven't fully stress tested it yet as I don't have a workable means to do so right now, besides migrating users over slowly. I have zero experience with Xen so I can't help you there. I hope that helps. -- Ryan Thomson Systems Administrator University Of Calgary Biocomputing http://moose.bio.ucalgary.ca/ > Hi all, > > is anyone in a production setting using software Iscsi Targets and > Initiators as a side options to GNBD? i'm exploring all our options for > a Xen/Cluster Suite n+2 server setup for our ISP and would like to hear > peoples thoughts on the best option, rather than using a SAN with FC we > have decided to go with a Intel Raid Array with standard linux to reduce > inital costs and need to find out what people are using in production to > export block devices. > > Thanks in advance > > Tristram > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Ryan From tristram at ubernet.co.nz Wed Jan 11 07:57:39 2006 From: tristram at ubernet.co.nz (Tristram Cheer) Date: Wed, 11 Jan 2006 20:57:39 +1300 Subject: [Linux-cluster] Software Iscsi with Redhat Cluster In-Reply-To: <1080.68.146.207.245.1136965560.squirrel@68.146.207.245> References: <43C4868A.2060508@ubernet.co.nz> <1080.68.146.207.245.1136965560.squirrel@68.146.207.245> Message-ID: <43C4BA73.7080107@ubernet.co.nz> Thanks for the reply, So i take it that you are exporting LVM volume groups? i'm tring to avoid placing many services on the cluster so avoiding CLVM is a big one for me. I was planning on exporting the Logical Volumes, about 28 or so in total via iSCSI. The reason for avoiding Cluster Suite for anymore than GFS is that with XEN (3 real servers, 9 virtual ones) that the cluster can stop functioning correctly if all the virtual servers go down but the physical ones remain working fine. I'm yet to find a way to stop the virtual servers bringing down the whole cluster - this may be possable but i'm rather new to RHCS :) Cheers Tristram Ryan Thomson wrote: >Hi Tristram, > >I'm just about to put a completely Linux based software iSCSI RedHat >Cluster with GFS into production. We have four RHEL4AS machines acting as >cluster nodes and an 8TB RHEL4AS server is exporting disk as two >arrays/iSCSI targets to the cluster nodes. Several more storage boxes we >already own (previously Linux servers exporting large arrays over NFS) are >to be added as iSCSI targets. > >I am using the iSCSI initiator that comes with RHEL4U2, the cisco open >source one it is I believe. For targets, I'm using the iSCSI Enterprise >Target (http://sourceforge.net/projects/iscsitarget/). The only thing I >found so far that doesn't seem to work is the iSCSI alias. I can't seem to >get the alias I set in the target to show up on the initiator. I don't >know if the problem is the target or the initiator as I haven't found >anything online yet about this issue. The currently available Linux iSCSI >software seems to work pretty much flawlessly for me otherwise. > >So far it's been quite easy and painless setting up CLVM volumes and >putting GFS on them, I even wrote a basic wrapper script to do all the >work for me, streamlining the proceedure. Filesystem expansion seems to >work as expected. I haven't played with snapshots. > >Initial numbers show transfer rates from end to end (NFS clients to >Cluster NFS server to GFS) to be better for iSCSI than GNBD. Keep in mind >these are initial tests using bonnie++ and using 'time' to time file >copies of various sizes, nothing concrete. I suspected NFS to be a >bottleneck but it seems that storage interconnect/fabric protocol still >makes a difference even with NFS being crappy to the clients. > >>From cluster nodes to storage I found transfer rates to be near local max >with iSCSI, again don't trust me though, do you own tests. My hardware >doesn't have very high end disk, just SATA with 3ware 9500 cards. I didn't >do the cluster node to storage test with GNBD :( > >Anyways, so far my initial experience has been great. I solved an issue >causing my cluster nodes to kernel panic and ever since, it's been running >very well serving Apache, MySQL, OpenLDAP and NFS exports. I haven't fully >stress tested it yet as I don't have a workable means to do so right now, >besides migrating users over slowly. > >I have zero experience with Xen so I can't help you there. > >I hope that helps. > >-- >Ryan Thomson >Systems Administrator >University Of Calgary Biocomputing >http://moose.bio.ucalgary.ca/ > > > >>Hi all, >> >>is anyone in a production setting using software Iscsi Targets and >>Initiators as a side options to GNBD? i'm exploring all our options for >>a Xen/Cluster Suite n+2 server setup for our ISP and would like to hear >>peoples thoughts on the best option, rather than using a SAN with FC we >>have decided to go with a Intel Raid Array with standard linux to reduce >>inital costs and need to find out what people are using in production to >>export block devices. >> >>Thanks in advance >> >>Tristram >> >>-- >>Linux-cluster mailing list >>Linux-cluster at redhat.com >>https://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> > > > > From thomsonr at ucalgary.ca Wed Jan 11 08:46:45 2006 From: thomsonr at ucalgary.ca (Ryan Thomson) Date: Wed, 11 Jan 2006 01:46:45 -0700 (MST) Subject: [Linux-cluster] Software Iscsi with Redhat Cluster In-Reply-To: <43C4BA73.7080107@ubernet.co.nz> References: <43C4868A.2060508@ubernet.co.nz> <1080.68.146.207.245.1136965560.squirrel@68.146.207.245> <43C4BA73.7080107@ubernet.co.nz> Message-ID: <1114.68.146.207.245.1136969205.squirrel@68.146.207.245> Tristram, I'm mounting iSCSI targets on my cluster nodes. The iSCSI target machines are not cluster nodes but they do exist on the same private subnet as the cluster nodes. I then put my iSCSI mounts into a cluster aware volume group and create volumes inside that volume group. I then format my volumes with GFS and mount them on the appropriate nodes. Some volumes are mounted on all nodes, some are not. I'm doing it this way because I find it *much* easier to manage my volumes from the cluster nodes instead of the independent storage devices. I realize clvmd isn't for everyone though. Either way, my experience with the major Linux software iSCSI drivers is pretty good. I assume you get much better performance with high end disk if you use an iSCSI HBA instead of loading your server CPU but it works for me at near local speeds of my 3ware SATA RAID5 disk servers. The nice thing about this whole setup is I *can* use FC or GNBD or Infiniband->(I think, dont quote me) later if I want as anything I can mount on the cluster nodes as a block device can be utilized as shared storage. For us, this was a "selling" point for RHCS/GFS. We had disk we wanted to include in a SAN environment with the ability to add any kind of backend storage we want. RHCS/GFS delivered. -- Ryan Thomson > Thanks for the reply, > > So i take it that you are exporting LVM volume groups? i'm tring to > avoid placing many services on the cluster so avoiding CLVM is a big one > for me. I was planning on exporting the Logical Volumes, about 28 or so > in total via iSCSI. > > The reason for avoiding Cluster Suite for anymore than GFS is that with > XEN (3 real servers, 9 virtual ones) that the cluster can stop > functioning correctly if all the virtual servers go down but the > physical ones remain working fine. I'm yet to find a way to stop the > virtual servers bringing down the whole cluster - this may be possable > but i'm rather new to RHCS :) > > Cheers > > Tristram > > Ryan Thomson wrote: > >>Hi Tristram, >> >>I'm just about to put a completely Linux based software iSCSI RedHat >>Cluster with GFS into production. We have four RHEL4AS machines acting >> as >>cluster nodes and an 8TB RHEL4AS server is exporting disk as two >>arrays/iSCSI targets to the cluster nodes. Several more storage boxes we >>already own (previously Linux servers exporting large arrays over NFS) >> are >>to be added as iSCSI targets. >> >>I am using the iSCSI initiator that comes with RHEL4U2, the cisco open >>source one it is I believe. For targets, I'm using the iSCSI Enterprise >>Target (http://sourceforge.net/projects/iscsitarget/). The only thing I >>found so far that doesn't seem to work is the iSCSI alias. I can't seem >> to >>get the alias I set in the target to show up on the initiator. I don't >>know if the problem is the target or the initiator as I haven't found >>anything online yet about this issue. The currently available Linux >> iSCSI >>software seems to work pretty much flawlessly for me otherwise. >> >>So far it's been quite easy and painless setting up CLVM volumes and >>putting GFS on them, I even wrote a basic wrapper script to do all the >>work for me, streamlining the proceedure. Filesystem expansion seems to >>work as expected. I haven't played with snapshots. >> >>Initial numbers show transfer rates from end to end (NFS clients to >>Cluster NFS server to GFS) to be better for iSCSI than GNBD. Keep in >> mind >>these are initial tests using bonnie++ and using 'time' to time file >>copies of various sizes, nothing concrete. I suspected NFS to be a >>bottleneck but it seems that storage interconnect/fabric protocol still >>makes a difference even with NFS being crappy to the clients. >> >>>From cluster nodes to storage I found transfer rates to be near local >>> max >>with iSCSI, again don't trust me though, do you own tests. My hardware >>doesn't have very high end disk, just SATA with 3ware 9500 cards. I >> didn't >>do the cluster node to storage test with GNBD :( >> >>Anyways, so far my initial experience has been great. I solved an issue >>causing my cluster nodes to kernel panic and ever since, it's been >> running >>very well serving Apache, MySQL, OpenLDAP and NFS exports. I haven't >> fully >>stress tested it yet as I don't have a workable means to do so right >> now, >>besides migrating users over slowly. >> >>I have zero experience with Xen so I can't help you there. >> >>I hope that helps. >> >>-- >>Ryan Thomson >>Systems Administrator >>University Of Calgary Biocomputing >>http://moose.bio.ucalgary.ca/ >> >> >> >>>Hi all, >>> >>>is anyone in a production setting using software Iscsi Targets and >>>Initiators as a side options to GNBD? i'm exploring all our options for >>>a Xen/Cluster Suite n+2 server setup for our ISP and would like to hear >>>peoples thoughts on the best option, rather than using a SAN with FC we >>>have decided to go with a Intel Raid Array with standard linux to >>> reduce >>>inital costs and need to find out what people are using in production >>> to >>>export block devices. >>> >>>Thanks in advance >>> >>>Tristram >>> >>>-- >>>Linux-cluster mailing list >>>Linux-cluster at redhat.com >>>https://www.redhat.com/mailman/listinfo/linux-cluster >>> >>> >>> >> >> >> >> > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Ryan From lhh at redhat.com Wed Jan 11 15:35:15 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 11 Jan 2006 10:35:15 -0500 Subject: [Linux-cluster] SAN file system to be shared to multiple nodes In-Reply-To: <43C42F3C.909@utilitran.com> References: <43C41782.80004@utilitran.com> <1136926604.4725.96.camel@ayanami.boston.redhat.com> <43C42F3C.909@utilitran.com> Message-ID: <1136993715.5029.34.camel@ayanami.boston.redhat.com> On Tue, 2006-01-10 at 15:03 -0700, rturnbull wrote: > Anyways, I know there is kinda two versions of GFS. The Redhat GFS > or the OpenGFS project. > I'm fine with OpenGFS and getting that to work. > However the first things that I come across are kernel patches to make > this work. Is there any kernel patches for 2.4.26 kernels. I noticed > its 2.4.20 and 2.4.22 but nothing for 2.4.26. I'd see if it applies or not. > Who is the best to contact for support on OpenGFS? Sounds like the > project is well, dead at the moment..... I hope I have my facts straight here, so if I'm wrong, please correct me (anyone)... OpenGFS was created as a fork of then-current GFS code when Sistina decided to close up GFS and try to make money selling it. IIRC, it uses the IBM OpenDLM for locking. Eventually Sistina merged with Red Hat, and GFS was re-released under the GPL, thereby eliminating the original reason OpenGFS was created. The 6.0 code base is runs on 2.4.x kernels, but is really only in "maintenance mode" at this point -- no new features are being added. The 6.1 code base - which became the foundation of the linux-cluster project was an evolutionary step in decoupling GFS from the cluster infrastructure, thereby allowing both client-server and fully-distributed locking models. > >GFS (6.1) does not require a server - for locks or data. There is a > >distributed lock manager for managing internal metadata and POSIX locks, > >and all clients have direct access to the SAN storage. ... with that evolution comes a cost. GFS 6.1 only works on 2.6 kernels - :( So, unless Slack 10.2. has a 2.6 update, you might have trouble. With 6.0.x (and previous versions), you *need* to set up lock masters. -- Lon From lhh at redhat.com Wed Jan 11 17:30:58 2006 From: lhh at redhat.com (Lon Hohberger) Date: Wed, 11 Jan 2006 12:30:58 -0500 Subject: [Linux-cluster] SAN file system to be shared to multiple nodes In-Reply-To: <43C52A30.7020304@utilitran.com> References: <43C41782.80004@utilitran.com> <1136926604.4725.96.camel@ayanami.boston.redhat.com> <43C42F3C.909@utilitran.com> <1136993715.5029.34.camel@ayanami.boston.redhat.com> <43C52A30.7020304@utilitran.com> Message-ID: <1137000658.5029.45.camel@ayanami.boston.redhat.com> On Wed, 2006-01-11 at 08:54 -0700, rturnbull wrote: > Well Lon, > I'm prepared to upgrade to GFS 6.1 and update the kernel using a > standard kernel like 2.6 from http://www.kernel.org/ > > I have done it before with no problem at all. Do you have a web addy > to go to download all the components/packages for GFS 6.1? ftp://sources.redhat.com/pub/cluster/releases Though CVS from the STABLE branch is a good bet too... -- Lon From Hansjoerg.Maurer at dlr.de Thu Jan 12 09:04:10 2006 From: Hansjoerg.Maurer at dlr.de (=?ISO-8859-15?Q?Hansj=F6rg_Maurer?=) Date: Thu, 12 Jan 2006 10:04:10 +0100 Subject: [Linux-cluster] GPFS?? In-Reply-To: <43C435D2.2060009@utilitran.com> References: <43C435D2.2060009@utilitran.com> Message-ID: <43C61B8A.1010106@dlr.de> Hi GPFS runs very well on RHEL3 and RHEL4. It is officially supported from IBM only on selected Hardware Devices. We did a feature and performance comparison of GFS and GPFS as part of a cluster project. It has been published in the german computer magazine iX in 10/2005. Just a brief overview: Our impression was, that gpfs is very scalable (up to 2^99 Byte even on Linux Kernel 2.4 compared to 16 TByte with GFS) and has more features than gfs 6.1 has at the moment. It has cluster-wide snapshots and the possibility to store data transparently redundant (like raid1) on diffent storage systems. It could be expanded and rebalanced online, which means, that data are restriped over the added storage device which improves overall IO performance by addinge devices. It could be extended to LAN with Multipathing... We discoverd some gpfs issues with mmaped files and an undocumented limit in fcntl locks per file (problem with samba tdb files on gpfs) which could be increased. GFS could be used free and the integration into RHEL is much better (with GPFS you have to compile a kernel module on every kernel update or gpfs update). It is supported by Redhat with no special hardware requirements. If you need further information on GPFS you can contact me off list, because it might be off topic on a gfs list :-) Greetings Hansj?rg rturnbull schrieb: > Hello, > I sent in a question before about SAN connecting multiple nodes to > same shared disk. > > I have been doing some reading and came across GPFS (Global > Parallel File System). Since I have all this shinny new ibm hardware, > is this a "free" software filesystem? Anyone heard of or used this > technology? > > Let me know. > > Ryan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- _________________________________________________________________ Dr. Hansjoerg Maurer | LAN- & System-Manager | Deutsches Zentrum | DLR Oberpfaffenhofen f. Luft- und Raumfahrt e.V. | Institut f. Robotik | Postfach 1116 | Muenchner Strasse 20 82230 Wessling | 82234 Wessling Germany | | Tel: 08153/28-2431 | E-mail: Hansjoerg.Maurer at dlr.de Fax: 08153/28-1134 | WWW: http://www.robotic.dlr.de/ __________________________________________________________________ There are 10 types of people in this world, those who understand binary and those who don't. From suvankar_moitra at yahoo.com Thu Jan 12 13:41:45 2006 From: suvankar_moitra at yahoo.com (SUVANKAR MOITRA) Date: Thu, 12 Jan 2006 05:41:45 -0800 (PST) Subject: [Linux-cluster] SAN file system to be shared to multiple nodes In-Reply-To: <1136926604.4725.96.camel@ayanami.boston.redhat.com> Message-ID: <20060112134145.19530.qmail@web52310.mail.yahoo.com> dear Lon, This is my 1st Linux Cluster installation .My os is RHEL AS4 and Redhat Cluster Suite 4, can you give me a complite installation document on that so i can install it properly. My Hardware is as follows:------ Server :- 2 Nos HP DL380G4 Storage:-- 1 no Hp MSA500G2 regards Suvankar --- Lon Hohberger wrote: > On Tue, 2006-01-10 at 13:22 -0700, rturnbull wrote: > > > So my question is, what is the best file system > available that is > > production worthy and capable of running on a > Slackware 10.2 distro. > > Naturally, I would guess a lot of people would say > GFS or OpenGFS, > > GFS is production ready and stable - and designed > precisely for what you > are trying to do. > > However, I do not know if anyone has packaged it for > Slackware. I know > it has been packaged for other distributions (apart > from Red Hat > Enterprise Linux), though, such as Ubuntu and Fedora > Core. > > > however is it stable and production ready. What > about other shared file > > systems such as InterMezzo or Coda? > > These are distributed/disconnected file systems > which solve a different > paradigm, and they require a server (or replicated > servers). > > GFS (6.1) does not require a server - for locks or > data. There is a > distributed lock manager for managing internal > metadata and POSIX locks, > and all clients have direct access to the SAN > storage. > > -- Lon > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From suvankar_moitra at yahoo.com Thu Jan 12 13:49:29 2006 From: suvankar_moitra at yahoo.com (SUVANKAR MOITRA) Date: Thu, 12 Jan 2006 05:49:29 -0800 (PST) Subject: [Linux-cluster] GPFS?? In-Reply-To: <43C61B8A.1010106@dlr.de> Message-ID: <20060112134929.92303.qmail@web52308.mail.yahoo.com> dear Hansj?rg This is my 1st Linux Cluster installation .My os is RHEL AS4 and Redhat Cluster Suite 4, can you give me a complite installation document on that so i can install it properly. My Hardware is as follows:------ Server :- 2 Nos HP DL380G4 Storage:-- 1 no Hp MSA500G2 regards Suvankar --- Hansj?rg Maurer wrote: > Hi > > GPFS runs very well on RHEL3 and RHEL4. > It is officially supported from IBM only on selected > Hardware Devices. > > We did a feature and performance comparison of GFS > and GPFS as part of a > cluster project. > > It has been published in the german computer > magazine iX in 10/2005. > Just a brief overview: > > Our impression was, that gpfs is very scalable (up > to 2^99 Byte even on > Linux Kernel 2.4 compared to 16 TByte with GFS) > and has more features than gfs 6.1 has at the > moment. > > It has cluster-wide snapshots and the possibility to > store data > transparently redundant > (like raid1) on diffent storage systems. > It could be expanded and rebalanced online, which > means, that data are > restriped over > the added storage device which improves overall IO > performance by > addinge devices. > It could be extended to LAN with Multipathing... > > We discoverd some gpfs issues with mmaped files and > an undocumented > limit in fcntl locks per file > (problem with samba tdb files on gpfs) which could > be increased. > > GFS could be used free and the integration into RHEL > is much better > (with GPFS you have to compile a kernel module > on every kernel update or gpfs update). > It is supported by Redhat with no special hardware > requirements. > > If you need further information on GPFS you can > contact me off list, > because it might be off topic on a gfs list :-) > > > Greetings > > Hansj?rg > > > rturnbull schrieb: > > > Hello, > > I sent in a question before about SAN > connecting multiple nodes to > > same shared disk. > > > > I have been doing some reading and came across > GPFS (Global > > Parallel File System). Since I have all this > shinny new ibm hardware, > > is this a "free" software filesystem? Anyone > heard of or used this > > technology? > > > > Let me know. > > > > Ryan > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > _________________________________________________________________ > > Dr. Hansjoerg Maurer | LAN- & > System-Manager > | > Deutsches Zentrum | DLR > Oberpfaffenhofen > f. Luft- und Raumfahrt e.V. | > Institut f. Robotik | > Postfach 1116 | Muenchner Strasse > 20 > 82230 Wessling | 82234 Wessling > Germany | > | > Tel: 08153/28-2431 | E-mail: > Hansjoerg.Maurer at dlr.de > Fax: 08153/28-1134 | WWW: > http://www.robotic.dlr.de/ > __________________________________________________________________ > > > There are 10 types of people in this world, > those who understand binary and those who don't. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From dillo+cluster at seas.upenn.edu Thu Jan 12 18:35:40 2006 From: dillo+cluster at seas.upenn.edu (Bryan Cardillo) Date: Thu, 12 Jan 2006 13:35:40 -0500 Subject: [Linux-cluster] rgmanager startup script Message-ID: <20060112183540.GC3970@rover.pcbi.upenn.edu> I've been noticing a couple of things on a cluster I have setup to serve a gfs volume via nfs. first, I've been seeing a bunch of clurmtabd errors being logged, like this: Jan 12 13:18:46 sc02 clurmtabd[22381]: #29: rmtab_write_atomic: Invalid argument second, I've noticed the number of processes on the cluster members is growing (slowly). anyway, while searching out the cause of the first, I think I may have solved both. /etc/init.d/rgmanager should kill off all clurmtabd processes when it is shutdown, lines 83-84 from 'stop_cluster': 83 # Ensure all NFS rmtab daemons are dead. 84 killall $RMTABD &> /dev/null but, RMTABD is not defined. I've added RMTABD=clurmtabd and to /etc/init.d/rgmanager and it appears to have solved both issues. so, is this a bug, and if so should I file it in bugzilla, send an amusingly trivial patch, or let this email suffice? Cheers, Bryan Cardillo Penn Bioinformatics Core University of Pennsylvania dillo at seas.upenn.edu From libregeek at gmail.com Fri Jan 13 06:32:24 2006 From: libregeek at gmail.com (Manilal K M) Date: Fri, 13 Jan 2006 12:02:24 +0530 Subject: [Linux-cluster] NAS or SAN Message-ID: <2315046d0601122232rf92db0cv@mail.gmail.com> Hello all, I am planning to setup a small cluster with two servers. Each servers have an independent storage. My doubt is can i setup GFS+GNBD without actually installing RedHat Cluster suite ? Can I use linux-ha + GFS + GNBD ? My another doubt is that, for my requirement which will be more suitable - SAN / NAS ? please help regards Manilal From angela.p at spymac.com Fri Jan 13 12:40:29 2006 From: angela.p at spymac.com (Angela) Date: Fri, 13 Jan 2006 13:40:29 +0100 Subject: [Linux-cluster] GFS on Redhat Enterprise 3 ES Message-ID: <6.1.2.0.0.20060113133620.025b0908@datamail.datamat.it> Hi all, I have this configuration: two servers (both with Redhat Enterprise 3 ES, kernel 2.4.21-4.EL) and they are attached to a DiskArray with a HBA. Can I use RedHat GFS to share the disk between the two servers? If so which version I have to use and where can I download it? Unfortunately I can't upgrade the kernel. Thanks in advance, Angela From rkenna at redhat.com Fri Jan 13 13:47:16 2006 From: rkenna at redhat.com (Rob Kenna) Date: Fri, 13 Jan 2006 08:47:16 -0500 Subject: [Linux-cluster] GFS on Redhat Enterprise 3 ES In-Reply-To: <6.1.2.0.0.20060113133620.025b0908@datamail.datamat.it> References: <6.1.2.0.0.20060113133620.025b0908@datamail.datamat.it> Message-ID: <43C7AF64.9070601@redhat.com> Angela wrote: > Hi all, > I have this configuration: two servers (both with Redhat Enterprise 3 > ES, kernel 2.4.21-4.EL) and they are attached to a DiskArray with a HBA. > Can I use RedHat GFS to share the disk between the two servers? If so > which version I have to use and where can I download it? GFS 6.0 is available for Red Hat EL 3. You will need to move up to a newer kernel update though. I would recommend Update 6 which is the 2.4.21-37 kernel. GFS allows for the full sharing of a file system across the SAN. It's a layered product offering on top of Red Hat Enterprise Linux. Here's a link to learn more: http://www.redhat.com/en_us/USA/home/solutions/gfs/ - Rob > > Unfortunately I can't upgrade the kernel. > > Thanks in advance, > > Angela > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Robert Kenna / Red Hat Sr Product Mgr - Storage 10 Technology Park Drive Westford, MA 01886 o: (978) 392-2410 (x22410) f: (978) 392-1001 c: (978) 771-6314 rkenna at redhat.com From angela.p at spymac.com Fri Jan 13 13:58:19 2006 From: angela.p at spymac.com (Angela) Date: Fri, 13 Jan 2006 14:58:19 +0100 Subject: [Linux-cluster] GFS on Redhat Enterprise 3 ES In-Reply-To: <43C7AF64.9070601@redhat.com> References: <6.1.2.0.0.20060113133620.025b0908@datamail.datamat.it> <43C7AF64.9070601@redhat.com> Message-ID: <6.1.2.0.0.20060113145550.0266fb18@mail.spymac.com> Thanks Rob, but I can't upgrade or patch the kernel, do you think there is another solution? Maybe with GFS 5.2.1, but I can't find the sources or the rpms. Angela At 14:47 13/01/2006, Rob Kenna wrote: >Angela wrote: >>Hi all, >>I have this configuration: two servers (both with Redhat Enterprise 3 ES, >>kernel 2.4.21-4.EL) and they are attached to a DiskArray with a HBA. >>Can I use RedHat GFS to share the disk between the two servers? If so >>which version I have to use and where can I download it? > >GFS 6.0 is available for Red Hat EL 3. You will need to move up to a >newer kernel update though. I would recommend Update 6 which is the >2.4.21-37 kernel. GFS allows for the full sharing of a file system across >the SAN. It's a layered product offering on top of Red Hat Enterprise >Linux. Here's a link to learn more: > >http://www.redhat.com/en_us/USA/home/solutions/gfs/ > > >- Rob > >>Unfortunately I can't upgrade the kernel. >>Thanks in advance, >>Angela >> >>-- >>Linux-cluster mailing list >>Linux-cluster at redhat.com >>https://www.redhat.com/mailman/listinfo/linux-cluster > >-- >Robert Kenna / Red Hat >Sr Product Mgr - Storage >10 Technology Park Drive >Westford, MA 01886 >o: (978) 392-2410 (x22410) >f: (978) 392-1001 >c: (978) 771-6314 >rkenna at redhat.com > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster From Frank.Weyns at ordina.nl Fri Jan 13 14:26:52 2006 From: Frank.Weyns at ordina.nl (Weyns, Frank) Date: Fri, 13 Jan 2006 15:26:52 +0100 Subject: [Linux-cluster] Where I can find the light for GFS and Multipathing in RedHat clusters. Message-ID: <4D30FCF33FE1FC4DB79C18A73D46C67308598D@BA12-0013.work.local> Hi, I have HP cluster consisting of two nodes using as filesystem space two I-scsi servers. There are two packages: one production pkg , one dev/test pkg So the prod pkg runs on server 1 and and the dev/test on server 2 and in case of a failure everything goes to the node which survives. There are two buildings each containing a node and a I-scsi server, all servers and buildings are connected with gigabit. The filsystems are mirrored on a host-based manner. ( = importing iscsi disk and LVM/mirroring on top) So far so good, and I hope you get the picture. Now I want to implement this on a Redhat Cluster, and I have the four servers setup, iscsi working, cluster up running but ... The questions now: - Where does GFS in the picture ? Why should I use it ? Why not simple create /dev/md and LVMs/ext3 and import them when needed. - How do you implement multipathing in this picture ? I want redundancy on my gigabit NIC's I missed the boat somewhere but please direct where I can find the light for GFS and Multipathing in RedHat clusters. /Frank Disclaimer Dit bericht met eventuele bijlagen is vertrouwelijk en uitsluitend bestemd voor de geadresseerde. Indien u niet de bedoelde ontvanger bent, wordt u verzocht de afzender te waarschuwen en dit bericht met eventuele bijlagen direct te verwijderen en/of te vernietigen. Het is niet toegestaan dit bericht en eventuele bijlagen te vermenigvuldigen, door te sturen, openbaar te maken, op te slaan of op andere wijze te gebruiken. Ordina N.V. en/of haar groepsmaatschappijen accepteren geen verantwoordelijkheid of aansprakelijkheid voor schade die voortvloeit uit de inhoud en/of de verzending van dit bericht. This e-mail and any attachments are confidential and is solely intended for the addressee only. If you are not the intended recipient, please notify the sender and delete and/or destroy this message and any attachments immediately. It is prohibited to copy, to distribute, to disclose or to use this e-mail and any attachments in any other way. Ordina N.V. and/or its group companies do not accept any responsibility nor liability for any damage resulting from the content of and/or the transmission of this message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkenna at redhat.com Fri Jan 13 15:28:34 2006 From: rkenna at redhat.com (Rob Kenna) Date: Fri, 13 Jan 2006 10:28:34 -0500 Subject: [Linux-cluster] GFS on Redhat Enterprise 3 ES In-Reply-To: <6.1.2.0.0.20060113145550.0266fb18@mail.spymac.com> References: <6.1.2.0.0.20060113133620.025b0908@datamail.datamat.it> <43C7AF64.9070601@redhat.com> <6.1.2.0.0.20060113145550.0266fb18@mail.spymac.com> Message-ID: <43C7C722.4060500@redhat.com> Well, you're probably out of luck. We don't support new 5.2.1 users and 6.0 support started with RHEL 3, Update 2(2.4.21-15). I suppose you could download, build and support this yourself. Still, GFS is a filesystem and includes kernel code which you will need to load into the kernel. - Rob Angela wrote: > Thanks Rob, > but I can't upgrade or patch the kernel, do you think there is another > solution? Maybe with GFS 5.2.1, but I can't find the sources or the rpms. > > Angela > > At 14:47 13/01/2006, Rob Kenna wrote: > > >> Angela wrote: >>> Hi all, >>> I have this configuration: two servers (both with Redhat Enterprise 3 >>> ES, kernel 2.4.21-4.EL) and they are attached to a DiskArray with a HBA. >>> Can I use RedHat GFS to share the disk between the two servers? If so >>> which version I have to use and where can I download it? >> >> GFS 6.0 is available for Red Hat EL 3. You will need to move up to a >> newer kernel update though. I would recommend Update 6 which is the >> 2.4.21-37 kernel. GFS allows for the full sharing of a file system >> across the SAN. It's a layered product offering on top of Red Hat >> Enterprise Linux. Here's a link to learn more: >> >> http://www.redhat.com/en_us/USA/home/solutions/gfs/ >> >> >> - Rob >> >>> Unfortunately I can't upgrade the kernel. >>> Thanks in advance, >>> Angela >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> -- >> Robert Kenna / Red Hat >> Sr Product Mgr - Storage >> 10 Technology Park Drive >> Westford, MA 01886 >> o: (978) 392-2410 (x22410) >> f: (978) 392-1001 >> c: (978) 771-6314 >> rkenna at redhat.com >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Robert Kenna / Red Hat Sr Product Mgr - Storage 10 Technology Park Drive Westford, MA 01886 o: (978) 392-2410 (x22410) f: (978) 392-1001 c: (978) 771-6314 rkenna at redhat.com From irwan at magnifix.com.my Fri Jan 13 15:46:23 2006 From: irwan at magnifix.com.my (Mohd Irwan Jamaluddin) Date: Fri, 13 Jan 2006 23:46:23 +0800 Subject: [Linux-cluster] GFS on Redhat Enterprise 3 ES In-Reply-To: <6.1.2.0.0.20060113145550.0266fb18@mail.spymac.com> References: <6.1.2.0.0.20060113133620.025b0908@datamail.datamat.it> <43C7AF64.9070601@redhat.com> <6.1.2.0.0.20060113145550.0266fb18@mail.spymac.com> Message-ID: <1137167183.7069.4.camel@kuli.magnifix.com.my> On Fri, 2006-01-13 at 14:58 +0100, Angela wrote: > Thanks Rob, > but I can't upgrade or patch the kernel, do you think there is another > solution? Maybe with GFS 5.2.1, but I can't find the sources or the rpms. What's your constraint for you can't upgrade/patch the kernel? If you don't have an access to Red Hat Network, just grab the SRPMS & build it. Maybe this will help you, ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/3ES/en/os/SRPMS/kernel-2.4.21-37.EL.src.rpm -- Regards, +--------------------------------+ | Mohd Irwan Jamaluddin | | ## System Engineer, | | (o_ Magnifix Sdn. Bhd. | | //\ Tel: +603 42705073 | | V_/_ Fax: +603 42701960 | | http://www.magnifix.com/ | +--------------------------------+ From angela.p at spymac.com Fri Jan 13 16:22:56 2006 From: angela.p at spymac.com (Angela) Date: Fri, 13 Jan 2006 17:22:56 +0100 Subject: [Linux-cluster] GFS on Redhat Enterprise 3 ES In-Reply-To: <1137167183.7069.4.camel@kuli.magnifix.com.my> References: <6.1.2.0.0.20060113133620.025b0908@datamail.datamat.it> <43C7AF64.9070601@redhat.com> <6.1.2.0.0.20060113145550.0266fb18@mail.spymac.com> <1137167183.7069.4.camel@kuli.magnifix.com.my> Message-ID: <6.1.2.0.0.20060113172150.027090f0@mail.spymac.com> Hi, unfortunately is a contractual constraint, we are forced by our costumers to use the "original" kernel. Angela At 16:46 13/01/2006, Mohd Irwan Jamaluddin wrote: >On Fri, 2006-01-13 at 14:58 +0100, Angela wrote: > > Thanks Rob, > > but I can't upgrade or patch the kernel, do you think there is another > > solution? Maybe with GFS 5.2.1, but I can't find the sources or the rpms. > >What's your constraint for you can't upgrade/patch the kernel? >If you don't have an access to Red Hat Network, just grab the SRPMS & >build it. > >Maybe this will help you, >ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/3ES/en/os/SRPMS/kernel-2.4.21-37.EL.src.rpm >-- >Regards, >+--------------------------------+ >| Mohd Irwan Jamaluddin | >| ## System Engineer, | >| (o_ Magnifix Sdn. Bhd. | >| //\ Tel: +603 42705073 | >| V_/_ Fax: +603 42701960 | >| http://www.magnifix.com/ | >+--------------------------------+ > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster From mwill at penguincomputing.com Fri Jan 13 16:29:52 2006 From: mwill at penguincomputing.com (Michael Will) Date: Fri, 13 Jan 2006 08:29:52 -0800 Subject: [Linux-cluster] GFS on Redhat Enterprise 3 ES Message-ID: <9D61626573F86843908C9AE8BDB4390551208B@jellyfish.highlyscyld.com> A linux distribution like RHEL3 comes with updates for security fixes over time that you have to apply unless you want to be vulnerable to then well-published threats. That's the nature of the game. So while the customer can ask you not to apply your own patches at the very least they have to accept RHEL3u6 rather than the originally released RHEL3. Once you are there, if you then can then just load modules rather than patching the kernel specifically for GFS, then you are good to go. Michael -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Angela Sent: Friday, January 13, 2006 8:23 AM To: linux clustering Subject: Re: [Linux-cluster] GFS on Redhat Enterprise 3 ES Hi, unfortunately is a contractual constraint, we are forced by our costumers to use the "original" kernel. Angela At 16:46 13/01/2006, Mohd Irwan Jamaluddin wrote: >On Fri, 2006-01-13 at 14:58 +0100, Angela wrote: > > Thanks Rob, > > but I can't upgrade or patch the kernel, do you think there is > > another solution? Maybe with GFS 5.2.1, but I can't find the sources or the rpms. > >What's your constraint for you can't upgrade/patch the kernel? >If you don't have an access to Red Hat Network, just grab the SRPMS & >build it. > >Maybe this will help you, >ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/3ES/en/os/SRPM >S/kernel-2.4.21-37.EL.src.rpm >-- >Regards, >+--------------------------------+ >| Mohd Irwan Jamaluddin | >| ## System Engineer, | >| (o_ Magnifix Sdn. Bhd. | >| //\ Tel: +603 42705073 | >| V_/_ Fax: +603 42701960 | >| http://www.magnifix.com/ | >+--------------------------------+ > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From ptader at fnal.gov Fri Jan 13 23:11:59 2006 From: ptader at fnal.gov (Paul Tader) Date: Fri, 13 Jan 2006 17:11:59 -0600 Subject: [Linux-cluster] ccs_tool upgrade , fails to convert configs. Message-ID: <43C833BF.9050807@fnal.gov> I'm trying to a upgrade old CCS formated (GFS 6.0) archives to the new (GFS 6.1) xml format using the ccs_tool command but seem to be getting the syntax wrong. # ccs_tool upgrade archive/* (*.ccs files in dir) Unable to find cluster.ccs in the archive. Failed to upgrade CCS configuration information. I've tried different variations of the command: # ccs_tool upgrade . (while in the directory) # ccs_tool upgrade cluster.ccs ...all with no luck. The ccs_tool man page seems to suggest this is possible: " SYNOPSIS ccs_tool [OPTION].. ... upgrade This command is used to upgrade an old CCS format archive to the new xml format. is the location of the old archive, which can be either a block device archive or a file archive. The converted configuration will be printed to stdout. Thank You, Paul -- =========================================================================== Paul Tader Computing Div/CSS Dept Fermi National Accelerator Lab; PO Box 500 MS 369 Batavia, IL 60510-0500 From thaidn at gmail.com Sat Jan 14 08:05:37 2006 From: thaidn at gmail.com (Thai Duong) Date: Sat, 14 Jan 2006 15:05:37 +0700 Subject: [Linux-cluster] Unresolved symbols in gnbd.o, pool.o and gfs.o Message-ID: Hi folks, I downloaded the latest version of RHGFS for RHAS3 x86 from ftp.redhat.comand built it sucessfully at the first try. But when I tried to install GFS, GFS-modules RPM packages into the build machine, it kept reporting about Unresolved symbols in gnbd.o, pool.o and gfs.o. Some information [root at lock i386]# uname -a Linux lock.eab.com.vn 2.4.21-37.EL #1 Wed Sep 7 13:35:21 EDT 2005 i686 i686 i386 GNU/Linux Error message: [root at lock i386]# ls -l total 896 -rw-r--r-- 1 root root 645986 Jan 14 14:38 GFS-6.0.2.27-0.i386.rpm -rw-r--r-- 1 root root 30915 Jan 14 14:38 GFS-devel-6.0.2.27-0.i386.rpm -rw-r--r-- 1 root root 226037 Jan 14 14:38 GFS-modules-6.0.2.27-0.i386.rpm [root at lock i386]# rpm -Uvh GFS* Preparing... ########################################### [100%] 1:GFS ########################################### [ 33%] 2:GFS-devel ########################################### [ 67%] 3:GFS-modules ########################################### [100%] depmod: *** Unresolved symbols in /lib/modules/2.4.21-37.EL /kernel/drivers/block/gnbd/gnbd.o depmod: __down_read_Rbebf4a44 depmod: __up_read_R98a5b601 depmod: init_rwsem_R322cb358 depmod: __down_write_trylock_R92683f56 depmod: __up_write_R77fd5393 depmod: *** Unresolved symbols in /lib/modules/2.4.21-37.EL /kernel/drivers/md/pool/pool.o depmod: __down_read_Rbebf4a44 depmod: __up_read_R98a5b601 depmod: init_rwsem_R322cb358 depmod: __up_write_R77fd5393 depmod: __down_write_R716dfc70 depmod: *** Unresolved symbols in /lib/modules/2.4.21-37.EL /kernel/fs/gfs/gfs.o depmod: __down_read_Rbebf4a44 depmod: __up_read_R98a5b601 depmod: init_rwsem_R322cb358 depmod: __up_write_R77fd5393 depmod: __down_write_R716dfc70 Please help. TIA, --Thai Duong. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkenna at redhat.com Sat Jan 14 23:59:37 2006 From: rkenna at redhat.com (Rob Kenna) Date: Sat, 14 Jan 2006 18:59:37 -0500 Subject: [Linux-cluster] GFS on Redhat Enterprise 3 ES In-Reply-To: <9D61626573F86843908C9AE8BDB4390551208B@jellyfish.highlyscyld.com> References: <9D61626573F86843908C9AE8BDB4390551208B@jellyfish.highlyscyld.com> Message-ID: <43C99069.7080807@redhat.com> Michael Will wrote: > A linux distribution like RHEL3 comes with updates for security > fixes over time that you have to apply unless you want to be vulnerable > to then well-published threats. That's the nature of the game. So while > the customer can ask you not to apply your own patches at the very least > they have to accept RHEL3u6 rather than the originally released RHEL3. > > Once you are there, if you then can then just load modules rather than > patching > the kernel specifically for GFS, then you are good to go. > > Michael Seconding Michael's comment. A big part of Red Hat's value add is to fix bugs and deal with security issues. A lot of good work has gone into the updates for RHEL 3 since it's inception in Oct '03. We've pushed out 6 updates and number 7 is in the oven (beta.) - Rob > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Angela > Sent: Friday, January 13, 2006 8:23 AM > To: linux clustering > Subject: Re: [Linux-cluster] GFS on Redhat Enterprise 3 ES > > Hi, > unfortunately is a contractual constraint, we are forced by our > costumers to use the "original" kernel. > > Angela > > At 16:46 13/01/2006, Mohd Irwan Jamaluddin wrote: >> On Fri, 2006-01-13 at 14:58 +0100, Angela wrote: >>> Thanks Rob, >>> but I can't upgrade or patch the kernel, do you think there is >>> another solution? Maybe with GFS 5.2.1, but I can't find the sources > or the rpms. >> What's your constraint for you can't upgrade/patch the kernel? >> If you don't have an access to Red Hat Network, just grab the SRPMS & >> build it. >> >> Maybe this will help you, >> ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/3ES/en/os/SRPM >> S/kernel-2.4.21-37.EL.src.rpm >> -- >> Regards, >> +--------------------------------+ >> | Mohd Irwan Jamaluddin | >> | ## System Engineer, | >> | (o_ Magnifix Sdn. Bhd. | >> | //\ Tel: +603 42705073 | >> | V_/_ Fax: +603 42701960 | >> | http://www.magnifix.com/ | >> +--------------------------------+ >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Robert Kenna / Red Hat Sr Product Mgr - Storage 10 Technology Park Drive Westford, MA 01886 o: (978) 392-2410 (x22410) f: (978) 392-1001 c: (978) 771-6314 rkenna at redhat.com From troels at arvin.dk Sun Jan 15 19:59:28 2006 From: troels at arvin.dk (Troels Arvin) Date: Sun, 15 Jan 2006 20:59:28 +0100 Subject: [Linux-cluster] DLM or GULM? Message-ID: Hello, I'm setting up two RHEL 4 servers (HP DL380G4) in a fail-over solution, using Red Hat Cluster Suite. The two servers are both connected to a HP MSA 500G2 RAID storage system (which has two SCSI connectors). I don't have plans about using GFS at the moment: Only one server will be mounting the shared storage system at a time. I plan on using iLO fencing. The Red Hat Cluster Suite documentation mentions that either DLM or GULM should be used for lock management. However, the documentation doesn't cover on what basis the choice should be made. When would DLM be the best choice, and when would GULM be the lock manager to go for? -- Greetings from Troels Arvin From troels at arvin.dk Sun Jan 15 21:42:34 2006 From: troels at arvin.dk (Troels Arvin) Date: Sun, 15 Jan 2006 22:42:34 +0100 Subject: [Linux-cluster] Quorum partitions in Cluster Suite 4? Message-ID: Hello, I the documentation for older versions of RH Cluster Suite, quorum partitions are explicitly mentioned in the documentation. But that doesn't seem to be the case for the documentation for RH Cluster Suite 4. Why is that? - Have quorum partitions somehow become irrelevant? (I'm setting up a two-node cluster for fail-over; I don't plan to use GFS.) -- Greetings from Troels Arvin, Copenhagen, Denmark From kanderso at redhat.com Mon Jan 16 00:11:44 2006 From: kanderso at redhat.com (Kevin Anderson) Date: Sun, 15 Jan 2006 18:11:44 -0600 Subject: [Linux-cluster] DLM or GULM? In-Reply-To: References: Message-ID: <1137370304.3676.4.camel@localhost.localdomain> On Sun, 2006-01-15 at 20:59 +0100, Troels Arvin wrote: > Hello, > > I'm setting up two RHEL 4 servers (HP DL380G4) in a fail-over solution, > using Red Hat Cluster Suite. The two servers are both connected to a HP > MSA 500G2 RAID storage system (which has two SCSI connectors). > > I don't have plans about using GFS at the moment: Only one server will be > mounting the shared storage system at a time. I plan on using iLO fencing. > > The Red Hat Cluster Suite documentation mentions that either DLM or GULM > should be used for lock management. However, the documentation doesn't > cover on what basis the choice should be made. > > When would DLM be the best choice, and when would GULM be the lock manager > to go for? DLM is the best choice. GuLM requires at least 3 nodes in order to maintain proper quorum in the cluster. DLM has a special 2 node configuration setup that removes this requirement. Kevin From pegasus at nerv.eu.org Mon Jan 16 00:53:13 2006 From: pegasus at nerv.eu.org (Jure =?ISO-8859-2?Q?Pe=E8ar?=) Date: Mon, 16 Jan 2006 01:53:13 +0100 Subject: [Linux-cluster] Minimal requirements for two nodes to see eachother Message-ID: <20060116015313.14ceb7c3.pegasus@nerv.eu.org> Hi all, I'm setting up a simple two node cluster over a shared storage. I want to use gfs, but I'm hitting some problems much earlier. After setting up the basics with system-config-cluster and starting up ccsd, cman and rgmanager on both nodes, clustat says on node1: Member Status: Quorate Member Name Status ------ ---- ------ node1 Online, Local, rgmanager node2 Offline Service Name Owner (Last) State ------- ---- ----- ------ ----- http_test node1 started and on node2: Member Status: Quorate Member Name Status ------ ---- ------ node1 Offline node2 Online, Local, rgmanager Service Name Owner (Last) State ------- ---- ----- ------ ----- http_test node2 started For some reason nodes don't see eachother, which results in the above situation. I guess the same reason causes fence_tool join -w to wait forever, which on boot appears as a hung machine (a bit ugly, because fenced is started before sshd by default). Both nodes have their info in /etc/hosts as well as in DNS and network connectivity is OK. As the documentation is not much helpful with how to figure this out, I'm asking it here: what minimal requirements must be met for two machines to see eachother? How to debug a situation like this? -- Jure Pe?ar http://jure.pecar.org/ From rkenna at redhat.com Mon Jan 16 03:42:05 2006 From: rkenna at redhat.com (Rob Kenna) Date: Sun, 15 Jan 2006 22:42:05 -0500 Subject: [Linux-cluster] Quorum partitions in Cluster Suite 4? In-Reply-To: References: Message-ID: <43CB160D.2040802@redhat.com> RH Cluster Suite in RHEL 4 no longer requires a shared storage area for quorum, even in two node configs. - Rob Troels Arvin wrote: > Hello, > > I the documentation for older versions of RH Cluster Suite, quorum > partitions are explicitly mentioned in the documentation. But that doesn't > seem to be the case for the documentation for RH Cluster Suite 4. Why is > that? - Have quorum partitions somehow become irrelevant? > > (I'm setting up a two-node cluster for fail-over; I don't plan to use GFS.) > -- Robert Kenna / Red Hat Sr Product Mgr - Storage 10 Technology Park Drive Westford, MA 01886 o: (978) 392-2410 (x22410) f: (978) 392-1001 c: (978) 771-6314 rkenna at redhat.com From troels at arvin.dk Mon Jan 16 07:09:20 2006 From: troels at arvin.dk (Troels Arvin) Date: Mon, 16 Jan 2006 08:09:20 +0100 Subject: [Linux-cluster] Re: Quorum partitions in Cluster Suite 4? References: <43CB160D.2040802@redhat.com> Message-ID: On Sun, 15 Jan 2006 22:42:05 -0500, Rob Kenna wrote: > RH Cluster Suite in RHEL 4 no longer requires a shared storage area for > quorum, even in two node configs. OK, thanks. So Cluster Suite 4 doesn't _require_ a shared storage area. But is it possible to set one up, and in that case: Is it recommended? -- Greetings from Troels Arvin From sdake at mvista.com Tue Jan 3 19:01:39 2006 From: sdake at mvista.com (Steven Dake) Date: Tue, 03 Jan 2006 12:01:39 -0700 Subject: [Linux-cluster] Cluster Suite v3 software watchdog In-Reply-To: <43A9A412.3010309@webbertek.com.br> References: <43A99E27.5040300@webbertek.com.br> <1135190702.23999.13.camel@ayanami.boston.redhat.com> <43A9A412.3010309@webbertek.com.br> Message-ID: <1136314899.752.4.camel@unnamed.az.mvista.com> depends on arch and if standard kernel.org is modified. Kernel.org does the following: x86_64 has nmi_watchdog default to off i386 has nmi watchdog default to on no other arches have nmi watchdog that I am aware of. The nmi watchdog simply prints out a backtrace when interrupts are off for too long. This occurs because of a buggy software driver or kernel code that clears interrupts on a processor and doesn't reenable them. Hence, the nmi watchdog is not fed, and it triggers a stack backtrace (instead of a total lockup) which allows someone experienced in development to find the source of the offending lock and fix the kernel code. I really doubt if you are using any commercial vendor kernel with supplied drivers you will encounter this sort of failure; this feature is generally used during development of kernel code. Some vendor kernels do special things when an nmi watchdog occurs, like take a system memory dump and then reboot, to allow debugging of the crash by the vendor at a later time. Regards -steve On Wed, 2005-12-21 at 16:50 -0200, Celso K. Webber wrote: > Hi Lon, > > Thank you very much for your reply. I'll try your tips. > > Now another question: is it really necessary to pass on the > "nmi_watchdog=1" parameter to the kernel? Or is it enabled by default > under RHELv3 ou v4? > > Regards, > > Celso. > > Lon Hohberger escreveu: > > >On Wed, 2005-12-21 at 16:25 -0200, Celso K. Webber wrote: > > > > > > > >>Does anyone has had this issue before? Or am I missing any step on > >>configuring the software watchdog feature? > >> > >>Another question for the Red Hat people on the list: does this "software > >>watchdog" works ok? I ask because it's enabled by default when you add a > >>new member to the cluster. The Cluster Suite v3 manual tells nothing > >>about this resource either. > >> > >> > > > >Yes, it works fine. > > > >A few things could be happening: > > > >(1) The NMI watchdog will reboot the machine if it detects an NMI hang. > >This is only a few seconds. > > > >(2) The cluster is extremely paranoid because you are not using a > >STONITH device (power controller), and it's detecting internal hangs. > >Try increasing the failover time. > > > >(3) The cluster is not getting scheduled due to system load. See the > >man page for cludb(8) about clumembd%rtp - both may help. > > > > > >-- Lon > > > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From suvankar_moitra at pcstech.com Tue Jan 10 11:01:34 2006 From: suvankar_moitra at pcstech.com (Suvankar Moitra) Date: Tue, 10 Jan 2006 16:31:34 +0530 Subject: [Linux-cluster] Linux Cluster Message-ID: <000601c615d5$3bc98e00$c996a8c0@tsg> Hi, I'm looking for an installing and configuring procedure for RedHat Cluster suite . We have 2 servers HP Proliant DL380 G4 (RedHat Advanced Server 4) attached by HP MSA500 storage and we want to install and configure Cluster suite . Thanks. Suvankar -------------- next part -------------- An HTML attachment was scrubbed... URL: From wally_winchester at fastmail.fm Mon Jan 16 16:23:07 2006 From: wally_winchester at fastmail.fm (Wally Winchester) Date: Mon, 16 Jan 2006 16:23:07 +0000 Subject: [Linux-cluster] Command-line HOWTO? Message-ID: <1137428587.14059.251966492@webmail.messagingengine.com> Hello. I would like to setup some shared disk space using GFS. I have a SAN. The machines will be running in a VMWare setup. I think I would like to use a distributed lock manager to avoid a single point of failure. Is there a decent non-gui HOWTO that will walk me through the setup, and that doesn't assume prior knowledge? Thanks -- Wally Winchester wally_winchester at fastmail.fm -- http://www.fastmail.fm - The way an email service should be From clusterbuilder at gmail.com Mon Jan 16 18:53:21 2006 From: clusterbuilder at gmail.com (Nick I) Date: Mon, 16 Jan 2006 11:53:21 -0700 Subject: [Linux-cluster] loading Red Hat on multiple nodes Message-ID: Hi, I help maintain a Web site called www.clusterbuilder.org. We have a question and answer section to help those involved in clustering. We are developing a knowledgebase of cluster questions and responses so people with similar problems might be able to find answers to their question. I received a question concerning Red Hat and wanted to see what the opinions are of everyone here. "Which software is best to load the Redhat Hat Linux ES4 on 150-odd nodes from one cluster management node?" You can respond to this email or submit a response at www.clusterbuilder.org/FAQ Any response is greatly appreciated. Thanks, Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From pegasus at nerv.eu.org Mon Jan 16 23:24:25 2006 From: pegasus at nerv.eu.org (Jure =?ISO-8859-2?Q?Pe=E8ar?=) Date: Tue, 17 Jan 2006 00:24:25 +0100 Subject: [Linux-cluster] Minimal requirements for two nodes to see eachother In-Reply-To: <20060116015313.14ceb7c3.pegasus@nerv.eu.org> References: <20060116015313.14ceb7c3.pegasus@nerv.eu.org> Message-ID: <20060117002425.6808481a.pegasus@nerv.eu.org> On Mon, 16 Jan 2006 01:53:13 +0100 Jure Pe?ar wrote: > For some reason nodes don't see eachother, which results in the above > situation. This "some reason" turned out to be default iptables configuration that rhel instalation makes. Turning it off and later configuring it properly was all that was needed. -- Jure Pe?ar http://jure.pecar.org/ From libregeek at gmail.com Tue Jan 17 04:09:02 2006 From: libregeek at gmail.com (Manilal K M) Date: Tue, 17 Jan 2006 09:39:02 +0530 Subject: [Linux-cluster] loading Red Hat on multiple nodes In-Reply-To: References: Message-ID: <2315046d0601162009j24aaafbfs@mail.gmail.com> On 17/01/06, Nick I wrote: > Hi, > > I help maintain a Web site called www.clusterbuilder.org. We have a > question and answer section to help those involved in clustering. We are > developing a knowledgebase of cluster questions and responses so people with > similar problems might be able to find answers to their question. > > I received a question concerning Red Hat and wanted to see what the opinions > are of everyone here. > > "Which software is best to load the Redhat Hat Linux ES4 on 150-odd nodes > from one cluster management node?" > > You can respond to this email or submit a response at > www.clusterbuilder.org/FAQ > > Any response is greatly appreciated. Hi Nick, it's a wonderful project. I'l surely find time to contribute. regards Manilal From jharr at opsource.net Tue Jan 17 11:48:42 2006 From: jharr at opsource.net (Jeff Harr) Date: Tue, 17 Jan 2006 11:48:42 -0000 Subject: [Linux-cluster] Network hiccup + power-fencing = both nodes go down (redhat cluster 4) Message-ID: <38A48FA2F0103444906AD22E14F1B5A3029F57CD@mailxchg01.corp.opsource.net> Hi all, it has been a while since I posted anything. Once again, I'd appreciate anything anyone has to say regarding this latest issue. Basically, we have a situation where both nodes are suddenly unable to reach each other due to a "network hiccup", and they begin trying to fence each other (power fencing). Then suddenly, the network returns and they turn each other off. My need: make redhat cluster robust enough not to do this. It could be that my configurations are wrong, and I'm going to include them (attached). My idea/solution: I THINK I could increase the post-fail-delay to a higher number than 0, thus making it wait to see if things "come back up". Perhaps I make 1 node wait like 2 minutes for the other one to come up, and another node wait zero seconds. Thus insuring that nobody does anything at the same time? Some small proof that the dual-reboot happened: I know that both boxes fenced the other and "succeeded", and my ILO event logs show both servers being powered off. Thanks a lot, Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cluster_db2.conf Type: application/octet-stream Size: 1392 bytes Desc: cluster_db2.conf URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cluster_db1.conf Type: application/octet-stream Size: 1392 bytes Desc: cluster_db1.conf URL: From gwood at dragonhold.org Tue Jan 17 11:52:01 2006 From: gwood at dragonhold.org (gwood at dragonhold.org) Date: Tue, 17 Jan 2006 11:52:01 -0000 (GMT) Subject: [Linux-cluster] Network hiccup + power-fencing = both nodes go down (redhat cluster 4) In-Reply-To: <38A48FA2F0103444906AD22E14F1B5A3029F57CD@mailxchg01.corp.opsource.net > References: <38A48FA2F0103444906AD22E14F1B5A3029F57CD@mailxchg01.corp.opsource.net> Message-ID: <3335.208.178.77.200.1137498721.squirrel@mnementh.dragonhold.org> I think your best answer is to change your heartbeat configuration. The fact that a single 'hiccup' can cause them to lose connectivity is not a good thing - you are probably best using a crossover cable if possible, rather than using the switched lan. From jharr at opsource.net Tue Jan 17 11:59:03 2006 From: jharr at opsource.net (Jeff Harr) Date: Tue, 17 Jan 2006 11:59:03 -0000 Subject: [Linux-cluster] Network hiccup + power-fencing = both nodes go down (redhat cluster 4) Message-ID: <38A48FA2F0103444906AD22E14F1B5A3029F57D1@mailxchg01.corp.opsource.net> I thought about a crossover cable :) But I think that, these being blade servers, there's no connection in the back for that. I could be wrong though - I just don't have the hardware around to look at right this second (its in a datacenter). One thing on the heartbeats: the message log says "missed too many heartbeats". Off-hand, do you know where the number of heartbeats is configured? Thanks, Jeff -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of gwood at dragonhold.org Sent: Tuesday, January 17, 2006 6:52 AM To: linux clustering Subject: Re: [Linux-cluster] Network hiccup + power-fencing = both nodes go down (redhat cluster 4) I think your best answer is to change your heartbeat configuration. The fact that a single 'hiccup' can cause them to lose connectivity is not a good thing - you are probably best using a crossover cable if possible, rather than using the switched lan. -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From gwood at dragonhold.org Tue Jan 17 12:47:10 2006 From: gwood at dragonhold.org (gwood at dragonhold.org) Date: Tue, 17 Jan 2006 12:47:10 -0000 (GMT) Subject: [Linux-cluster] Network hiccup + power-fencing = both nodes go down (redhat cluster 4) In-Reply-To: <38A48FA2F0103444906AD22E14F1B5A3029F57D1@mailxchg01.corp.opsource.net > References: <38A48FA2F0103444906AD22E14F1B5A3029F57D1@mailxchg01.corp.opsource.net> Message-ID: <55598.208.178.77.200.1137502030.squirrel@mnementh.dragonhold.org> On Tue, January 17, 2006 11:59, Jeff Harr wrote: > I thought about a crossover cable :) But I think that, these being > blade servers, there's no connection in the back for that. I could be > wrong though - I just don't have the hardware around to look at right > this second (its in a datacenter). 'AH'. No, if it's blade servers the only network connectivity that you have is via the blade's management ports. A lot of chassis give you 2 network connections tho (the egenera & IBM that I've used both do) - but I can understand the physical limitations. :) > One thing on the heartbeats: the message log says "missed too many > heartbeats". Off-hand, do you know where the number of heartbeats is > configured? Not guilty I'm afraid - I've had to stop playing with clustering at the moment due to looking for a new job :) From pcaulfie at redhat.com Tue Jan 17 12:57:32 2006 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 17 Jan 2006 12:57:32 +0000 Subject: [Linux-cluster] Network hiccup + power-fencing = both nodes go down (redhat cluster 4) In-Reply-To: <38A48FA2F0103444906AD22E14F1B5A3029F57D1@mailxchg01.corp.opsource.net> References: <38A48FA2F0103444906AD22E14F1B5A3029F57D1@mailxchg01.corp.opsource.net> Message-ID: <43CCE9BC.2080109@redhat.com> Jeff Harr wrote: > I thought about a crossover cable :) But I think that, these being > blade servers, there's no connection in the back for that. I could be > wrong though - I just don't have the hardware around to look at right > this second (its in a datacenter). > > One thing on the heartbeats: the message log says "missed too many > heartbeats". Off-hand, do you know where the number of heartbeats is > configured? /proc/cluster/config/cman/hello_timer /proc/cluster/config/cman/deadnode_timeout -- patrick From jharr at opsource.net Tue Jan 17 12:59:05 2006 From: jharr at opsource.net (Jeff Harr) Date: Tue, 17 Jan 2006 12:59:05 -0000 Subject: [Linux-cluster] Network hiccup + power-fencing = both nodes go down (redhat cluster 4) Message-ID: <38A48FA2F0103444906AD22E14F1B5A3029F57E8@mailxchg01.corp.opsource.net> Well, thanks for your help thus far. I'm going to keep plugging away. I "seem to remember" that when I originally tested the config, I'd unplug the Ethernet from 1 box and the other would take about a minute before it determined that it had "missed too many heartbeats". Good luck with that job man, Jeff -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of gwood at dragonhold.org Sent: Tuesday, January 17, 2006 7:47 AM To: linux clustering Subject: RE: [Linux-cluster] Network hiccup + power-fencing = both nodes go down (redhat cluster 4) On Tue, January 17, 2006 11:59, Jeff Harr wrote: > I thought about a crossover cable :) But I think that, these being > blade servers, there's no connection in the back for that. I could be > wrong though - I just don't have the hardware around to look at right > this second (its in a datacenter). 'AH'. No, if it's blade servers the only network connectivity that you have is via the blade's management ports. A lot of chassis give you 2 network connections tho (the egenera & IBM that I've used both do) - but I can understand the physical limitations. :) > One thing on the heartbeats: the message log says "missed too many > heartbeats". Off-hand, do you know where the number of heartbeats is > configured? Not guilty I'm afraid - I've had to stop playing with clustering at the moment due to looking for a new job :) -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From jharr at opsource.net Tue Jan 17 13:06:09 2006 From: jharr at opsource.net (Jeff Harr) Date: Tue, 17 Jan 2006 13:06:09 -0000 Subject: [Linux-cluster] Network hiccup + power-fencing = both nodes godown (redhat cluster 4) Message-ID: <38A48FA2F0103444906AD22E14F1B5A3029F57EB@mailxchg01.corp.opsource.net> Patrick, this is awesome. Here are my numbers: [root at server1 ~]# cat /proc/cluster/config/cman/hello_timer 5 [root at server1 ~]# cat /proc/cluster/config/cman/deadnode_timeout 21 [root at server1 ~]# I'm assuming these are seconds. I think if I increase the deadnode_timeout to maybe 120, then a network hiccup or major glitch (like a reboot of a switch) could be ignored. Thanks again! Jeff -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Patrick Caulfield Sent: Tuesday, January 17, 2006 7:58 AM To: linux clustering Subject: Re: [Linux-cluster] Network hiccup + power-fencing = both nodes godown (redhat cluster 4) Jeff Harr wrote: > I thought about a crossover cable :) But I think that, these being > blade servers, there's no connection in the back for that. I could be > wrong though - I just don't have the hardware around to look at right > this second (its in a datacenter). > > One thing on the heartbeats: the message log says "missed too many > heartbeats". Off-hand, do you know where the number of heartbeats is > configured? /proc/cluster/config/cman/hello_timer /proc/cluster/config/cman/deadnode_timeout -- patrick -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From bijoy_kj at yahoo.com Tue Jan 17 12:52:13 2006 From: bijoy_kj at yahoo.com (Bijoy Joseph) Date: Tue, 17 Jan 2006 12:52:13 +0000 (GMT) Subject: [Linux-cluster] problem setting up cluster Message-ID: <20060117125213.26002.qmail@web31509.mail.mud.yahoo.com> after setting up the cluster on 2 nodes getting error on both system [root at test1 log]# clustat Cluster Status - db2_test Incarnation #0 (This member is not part of the cluster quorum) Member Status ------------------ ---------- test1 active <-- You are here test2 Inactive No Quorum - Service States Unknown error message form /var/log/syslog Jan 17 16:18:21 test1 cluquorumd[4819]: STONITH: No drivers c onfigured for host 'test1'! Jan 17 16:18:21 test1 cluquorumd[4819]: STONITH: Data integri ty may be compromised! Jan 17 16:18:21 test1 cluquorumd[4819]: STONITH: No drivers c onfigured for host 'test2'! Jan 17 16:18:21 test1 cluquorumd[4819]: STONITH: Data integri ty may be compromised! pls help ___________________________________________________________ To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com From pcaulfie at redhat.com Tue Jan 17 13:32:34 2006 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 17 Jan 2006 13:32:34 +0000 Subject: [Linux-cluster] Network hiccup + power-fencing = both nodes godown (redhat cluster 4) In-Reply-To: <38A48FA2F0103444906AD22E14F1B5A3029F57EB@mailxchg01.corp.opsource.net> References: <38A48FA2F0103444906AD22E14F1B5A3029F57EB@mailxchg01.corp.opsource.net> Message-ID: <43CCF1F2.1010707@redhat.com> Jeff Harr wrote: > Patrick, this is awesome. Here are my numbers: > > [root at server1 ~]# cat /proc/cluster/config/cman/hello_timer > 5 > [root at server1 ~]# cat /proc/cluster/config/cman/deadnode_timeout > 21 > [root at server1 ~]# > > I'm assuming these are seconds. I think if I increase the > deadnode_timeout to maybe 120, then a network hiccup or major glitch > (like a reboot of a switch) could be ignored. Yes, those are seconds. Sorry I should have said. -- patrick From jharr at opsource.net Tue Jan 17 13:49:55 2006 From: jharr at opsource.net (Jeff Harr) Date: Tue, 17 Jan 2006 13:49:55 -0000 Subject: [Linux-cluster] Network hiccup + power-fencing = both nodes godown(redhat cluster 4) Message-ID: <38A48FA2F0103444906AD22E14F1B5A3029F57EF@mailxchg01.corp.opsource.net> Thanks Patrick. I have upped my deadnode_timeouts to 120 each. My worry though is the box somehow rebooting and joining faster than the other can wait its 120 seconds and take over the cluster. Is there another timeout value that I can tweak to keep the original, crashed node from rebooting and joining too quickly? Unfortunately, when the boxes crash they seem to come right back up and not stay dead. I think this might be ILO behavior, but not sure. I know when I shutdown -hy now, they stay down, and when the power-fencing takes place they stay down too, but not for crashes. Thanks again, Jeff -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Patrick Caulfield Sent: Tuesday, January 17, 2006 8:33 AM To: linux clustering Subject: Re: [Linux-cluster] Network hiccup + power-fencing = both nodes godown(redhat cluster 4) Jeff Harr wrote: > Patrick, this is awesome. Here are my numbers: > > [root at server1 ~]# cat /proc/cluster/config/cman/hello_timer > 5 > [root at server1 ~]# cat /proc/cluster/config/cman/deadnode_timeout > 21 > [root at server1 ~]# > > I'm assuming these are seconds. I think if I increase the > deadnode_timeout to maybe 120, then a network hiccup or major glitch > (like a reboot of a switch) could be ignored. Yes, those are seconds. Sorry I should have said. -- patrick -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From pcaulfie at redhat.com Tue Jan 17 14:03:23 2006 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 17 Jan 2006 14:03:23 +0000 Subject: [Linux-cluster] Network hiccup + power-fencing = both nodes godown(redhat cluster 4) In-Reply-To: <38A48FA2F0103444906AD22E14F1B5A3029F57EF@mailxchg01.corp.opsource.net> References: <38A48FA2F0103444906AD22E14F1B5A3029F57EF@mailxchg01.corp.opsource.net> Message-ID: <43CCF92B.2040308@redhat.com> Jeff Harr wrote: > Thanks Patrick. I have upped my deadnode_timeouts to 120 each. > > My worry though is the box somehow rebooting and joining faster than the > other can wait its 120 seconds and take over the cluster. Is there > another timeout value that I can tweak to keep the original, crashed > node from rebooting and joining too quickly? Unfortunately, when the > boxes crash they seem to come right back up and not stay dead. I think > this might be ILO behavior, but not sure. I know when I shutdown -hy > now, they stay down, and when the power-fencing takes place they stay > down too, but not for crashes. > If the crashed node tries to join while the other node thinks it's still in the cluster then it will get rejected and its join should fail. Of course the other node will still think it's alive but won't be able to talk to it because it doesn't have any services running. When the remaining node notices it has gone then it should fence it (and cause another power cycle!). So things should be OK. Are you seeing actual problems ? -- patrick From jharr at opsource.net Tue Jan 17 14:34:09 2006 From: jharr at opsource.net (Jeff Harr) Date: Tue, 17 Jan 2006 14:34:09 -0000 Subject: [Linux-cluster] Network hiccup + power-fencing =both nodes godown(redhat cluster 4) Message-ID: <38A48FA2F0103444906AD22E14F1B5A3029F5803@mailxchg01.corp.opsource.net> Am I seeing actual problems: I was earlier :) I had the network blip, then both boxes fenced each-other after it recoverd. By blip, I'm guessing it was more than just a blip - more than 21 seconds. I think it was long enough that they both tried to fence the other, so that process was running when the network came back online, and then they did their thing. I don't mind a 2nd round of cycling on the bad box, really. I don't like it, but I care more about the one that's up and running with my database one it more than the one that failed. Thank you sir for all your help. Jeff -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Patrick Caulfield Sent: Tuesday, January 17, 2006 9:03 AM To: linux clustering Subject: Re: [Linux-cluster] Network hiccup + power-fencing =both nodes godown(redhat cluster 4) Jeff Harr wrote: > Thanks Patrick. I have upped my deadnode_timeouts to 120 each. > > My worry though is the box somehow rebooting and joining faster than the > other can wait its 120 seconds and take over the cluster. Is there > another timeout value that I can tweak to keep the original, crashed > node from rebooting and joining too quickly? Unfortunately, when the > boxes crash they seem to come right back up and not stay dead. I think > this might be ILO behavior, but not sure. I know when I shutdown -hy > now, they stay down, and when the power-fencing takes place they stay > down too, but not for crashes. > If the crashed node tries to join while the other node thinks it's still in the cluster then it will get rejected and its join should fail. Of course the other node will still think it's alive but won't be able to talk to it because it doesn't have any services running. When the remaining node notices it has gone then it should fence it (and cause another power cycle!). So things should be OK. Are you seeing actual problems ? -- patrick -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From damian.osullivan at hp.com Tue Jan 17 17:00:50 2006 From: damian.osullivan at hp.com (O'Sullivan, Damian) Date: Tue, 17 Jan 2006 17:00:50 -0000 Subject: [Linux-cluster] Cluster help Message-ID: <644A0966265D9D40AC7584FCE95611130212C5D5@dubexc01.emea.cpqcorp.net> Hi, I have been trying to set up a cluster that fails over (not brave enough for active/active yet!) samba and nfs. I have 2 DL-360G4 with a 2 SFS20 (dual port MSA20s) connected as shared storage. I have set up GFS and both servers can write/read from the shared storage. However I am not able to set up a working NFS setup here. I tried mounting /var/lib/nfs/rpc_pipefs on the shared storage but had no joy here. I also tried the cs-deploy tool but it does not recognise the storage. I hacked the script and it now sees the storage but it still fails trying to mount it. Has anyone a copy of a cluster.conf on a working NFS cluster or even better a active/active one? Thanks, Damian. From pegasus at nerv.eu.org Tue Jan 17 22:52:44 2006 From: pegasus at nerv.eu.org (Jure =?ISO-8859-2?Q?Pe=E8ar?=) Date: Tue, 17 Jan 2006 23:52:44 +0100 Subject: [Linux-cluster] HP iLO fencing - how many iLO variants are there? Message-ID: <20060117235244.532486c9.pegasus@nerv.eu.org> Hi all, I have two DL145 G1 servers with "Lights Out 100i Remote Management". I thought this is iLO but am not so sure anymore. The thing is accessible via telnet or http and works as expected. So I was kinda suprised when fence_ilo failed with no route to host at blahblahblah/SSL.pm line 104. Sure, it wants to connect to port 443 ... So I took a look at fence_ilo code and saw nothing that would talk telnet to this 100i thing ... I also checked its firmware version. It says 2.00, latest is 2.01 with no important updates ... So it looks like I'd have to write my own fencing agent. Are there any guidelines on how to do it "properly" to have it included in the mainline distro? Also, any docs/mailing list post about what parameters fencing agent needs to implement? -- Jure Pe?ar http://jure.pecar.org/ From pegasus at nerv.eu.org Tue Jan 17 23:38:18 2006 From: pegasus at nerv.eu.org (Jure =?ISO-8859-2?Q?Pe=E8ar?=) Date: Wed, 18 Jan 2006 00:38:18 +0100 Subject: [Linux-cluster] /usr on gfs Message-ID: <20060118003818.0216bc15.pegasus@nerv.eu.org> Another problem I have ... I want to have /usr on gfs, shared between two machines. Everything works fine, until shutdown - where the machines hangs with: Unmounting GFS filesystems: umount: /usr: device is busy [FAILED] INIT: no more processes left in this runlevel At this point keyboard is dead (expect numlock, which still works) and the machine must be hard reset. Any ideas? I think I have to study rc.d shutdown sequence a bit to figure out what holds /usr open ... -- Jure Pe?ar http://jure.pecar.org/ From hlawatschek at atix.de Wed Jan 18 08:32:16 2006 From: hlawatschek at atix.de (Mark Hlawatschek) Date: Wed, 18 Jan 2006 09:32:16 +0100 Subject: [Linux-cluster] HP iLO fencing - how many iLO variants are there? In-Reply-To: <20060117235244.532486c9.pegasus@nerv.eu.org> References: <20060117235244.532486c9.pegasus@nerv.eu.org> Message-ID: <200601180932.16510.hlawatschek@atix.de> Hi Jure, in my opinion the Lights Out 100i Remote Management" are IPMI interfaces. You could give fence_ipmilan a try. Mark On Tuesday 17 January 2006 23:52, Jure Pe?ar wrote: > Hi all, > > I have two DL145 G1 servers with "Lights Out 100i Remote Management". I > thought this is iLO but am not so sure anymore. > > The thing is accessible via telnet or http and works as expected. So I was > kinda suprised when fence_ilo failed with no route to host at > blahblahblah/SSL.pm line 104. Sure, it wants to connect to port 443 ... So > I took a look at fence_ilo code and saw nothing that would talk telnet to > this 100i thing ... > > I also checked its firmware version. It says 2.00, latest is 2.01 with no > important updates ... > > So it looks like I'd have to write my own fencing agent. Are there any > guidelines on how to do it "properly" to have it included in the mainline > distro? Also, any docs/mailing list post about what parameters fencing > agent needs to implement? -- Gruss / Regards, Dipl.-Ing. Mark Hlawatschek Phone: +49-89 121 409-55 http://www.atix.de/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From hlawatschek at atix.de Wed Jan 18 08:44:00 2006 From: hlawatschek at atix.de (Mark Hlawatschek) Date: Wed, 18 Jan 2006 09:44:00 +0100 Subject: [Linux-cluster] /usr on gfs In-Reply-To: <20060118003818.0216bc15.pegasus@nerv.eu.org> References: <20060118003818.0216bc15.pegasus@nerv.eu.org> Message-ID: <200601180944.00856.hlawatschek@atix.de> Hi Jure, make sure every process is stopped before you try to umount /usr. fuser -mv /usr returns a list with processes accessing the /usr mount. BTW: We will provide our shared root for gfs clusters soon. This will give you the possibility you to use gfs for the / mountpoint. Keep looking at http://www.open-sharedroot.org . Mark On Wednesday 18 January 2006 00:38, Jure Pe?ar wrote: > Another problem I have ... > > I want to have /usr on gfs, shared between two machines. Everything works > fine, until shutdown - where the machines hangs with: > > Unmounting GFS filesystems: umount: /usr: device is busy > [FAILED] > INIT: no more processes left in this runlevel > > At this point keyboard is dead (expect numlock, which still works) and the > machine must be hard reset. > > Any ideas? > > I think I have to study rc.d shutdown sequence a bit to figure out what > holds /usr open ... -- Gruss / Regards, Dipl.-Ing. Mark Hlawatschek Phone: +49-89 121 409-55 http://www.atix.de/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From brodriguezb at fujitsu.es Wed Jan 18 10:42:36 2006 From: brodriguezb at fujitsu.es (=?UTF-8?B?QmFydG9sb23DqSBSb2Ryw61ndWV6?=) Date: Wed, 18 Jan 2006 11:42:36 +0100 Subject: [Linux-cluster] GFS and httpd process with stat "D" Message-ID: <43CE1B9C.6020604@fujitsu.es> Hi list, Release: Red Hat Enterprise Linux ES release 4 (Nahant Update 2) GFS-6.1.2-0 GFS-kernel-smp-2.6.9-42.2 Kernel running: 2.6.9-22.0.1.ELsmp Apache: httpd-2.0.52-19.ent I have 4 servers with GFS. These servers have shared disk with GFS (The disks are in a san with fiberchannel). This scene is in production. 2 times in 2 weeks the httpd process become stat "D". We thinks GFS is blocking something (file or inode... i dont know). And when httpd try read or writhe there, then become stat "D": apache 14383 1.5 0.3 24004 13300 ? D 17:36 3:57 /usr/sbin/httpd apache 14503 1.6 0.3 23996 13296 ? D 17:49 3:53 /usr/sbin/httpd apache 14516 1.6 0.3 23988 13288 ? D 17:50 3:58 /usr/sbin/httpd apache 14522 1.6 0.3 24044 13340 ? D 17:50 3:54 /usr/sbin/httpd apache 14635 1.5 0.3 24092 13392 ? D 18:00 3:23 /usr/sbin/httpd apache 14661 1.6 0.3 24048 13344 ? D 18:03 3:33 /usr/sbin/httpd apache 14662 1.7 0.3 24096 13392 ? D 18:03 3:48 /usr/sbin/httpd apache 14671 1.4 0.3 24044 13340 ? D 18:03 3:12 /usr/sbin/httpd apache 14687 1.6 0.3 24020 13320 ? D 18:03 3:42 /usr/sbin/httpd apache 17329 0.8 0.3 24356 13640 ? D 19:42 1:04 /usr/sbin/httpd apache 17331 0.9 0.3 24000 13284 ? D 19:42 1:05 /usr/sbin/httpd apache 17332 1.2 0.3 24156 13456 ? D 19:43 1:29 /usr/sbin/httpd apache 17335 0.9 0.3 24128 13412 ? D 19:43 1:06 /usr/sbin/httpd apache 17345 0.9 0.3 23908 13120 ? D 19:43 1:07 /usr/sbin/httpd apache 17347 0.9 0.3 23608 12896 ? D 19:43 1:07 /usr/sbin/httpd apache 17385 0.8 0.3 24120 13416 ? D 19:45 1:01 /usr/sbin/httpd apache 17386 1.2 0.3 23932 13228 ? D 19:46 1:26 /usr/sbin/httpd apache 17387 1.1 0.3 23904 13200 ? D 19:46 1:17 /usr/sbin/httpd apache 17398 0.9 0.3 24064 13360 ? D 19:49 1:04 /usr/sbin/httpd apache 17599 1.3 0.3 24392 13676 ? D 20:41 0:52 /usr/sbin/httpd apache 17606 0.3 0.3 23472 12632 ? D 20:42 0:11 /usr/sbin/httpd apache 17607 0.6 0.3 24024 13312 ? D 20:42 0:23 /usr/sbin/httpd apache 17608 1.1 0.3 23880 13164 ? D 20:42 0:44 /usr/sbin/httpd apache 17609 0.3 0.3 23556 12736 ? D 20:42 0:11 /usr/sbin/httpd apache 17620 0.5 0.3 23928 13204 ? D 20:43 0:21 /usr/sbin/httpd apache 17632 0.0 0.2 23168 12268 ? D 20:46 0:03 /usr/sbin/httpd apache 17633 0.6 0.3 23448 12704 ? D 20:47 0:20 /usr/sbin/httpd apache 17635 0.0 0.3 23464 12592 ? D 20:48 0:01 /usr/sbin/httpd apache 17636 0.0 0.3 23448 12584 ? D 20:49 0:02 /usr/sbin/httpd apache 17637 0.2 0.3 23472 12640 ? D 20:49 0:07 /usr/sbin/httpd apache 17638 0.2 0.3 23508 12772 ? D 20:49 0:07 /usr/sbin/httpd apache 17639 0.2 0.3 23488 12672 ? D 20:49 0:07 /usr/sbin/httpd apache 17643 0.1 0.3 23860 13104 ? D 20:50 0:03 /usr/sbin/httpd apache 17644 0.0 0.2 23156 12376 ? D 20:51 0:02 /usr/sbin/httpd apache 17645 1.2 0.3 23472 12680 ? D 20:51 0:39 /usr/sbin/httpd apache 17647 0.6 0.2 23184 12428 ? D 20:52 0:19 /usr/sbin/httpd apache 17648 0.5 0.3 23496 12760 ? D 20:52 0:18 /usr/sbin/httpd apache 17665 0.4 0.3 23524 12700 ? D 20:54 0:11 /usr/sbin/httpd apache 17667 0.7 0.3 23972 13240 ? D 20:54 0:20 /usr/sbin/httpd apache 17668 0.1 0.3 23500 12652 ? D 20:54 0:03 /usr/sbin/httpd apache 17669 0.2 0.3 23488 12652 ? D 20:54 0:06 /usr/sbin/httpd apache 17671 0.8 0.3 23696 12888 ? D 20:55 0:25 /usr/sbin/httpd apache 17732 0.1 0.2 23188 12264 ? D 21:01 0:03 /usr/sbin/httpd apache 17755 0.4 0.3 23588 12768 ? D 21:02 0:11 /usr/sbin/httpd This httpd process appear in the 4 servers. The first time, we reboot all the cluster?s members, but the servers cant unmount GFS filesystem, therefore the only solution is a fisical poweroff (botton). The second time we saw the first httpd process with stat "D" and then reboot only this server. Command reboot dont work again and fisical poweroff is required. But then, the others 3 servers fix without reboot. In this 3 servers, the httpd process in stat "D" became stat "S", all ok. I had read this mail: https://www.redhat.com/archives/linux-cluster/2005-December/msg00054.html I think is very similar, but in my log dont appear explicitily "stuck in gfs" i dont know if i have the max logging for gfs. I only see in my logs the try of unmount or mount: Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: withdrawing from cluster at user's request Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: about to withdraw from the cluster Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: waiting for outstanding I/O Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: telling LM to withdraw Jan 16 21:53:55 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: withdrawn Jan 16 21:58:45 sf-03 gfs: Unmounting GFS filesystems: failed Jan 16 22:02:01 sf-03 gfs: Unmounting GFS filesystems: failed Jan 16 22:02:13 sf-03 gfs: Unmounting GFS filesystems: failed Jan 16 22:02:25 sf-03 gfs: Unmounting GFS filesystems: failed Jan 16 22:02:37 sf-03 gfs: Unmounting GFS filesystems: failed Jan 16 22:02:49 sf-03 gfs: Unmounting GFS filesystems: failed Jan 16 22:03:01 sf-03 gfs: Unmounting GFS filesystems: failed Jan 16 22:03:14 sf-03 vgchange: Can't deactivate volume group "vg_gfs1" with 2 open logical volume(s) Jan 16 22:03:14 sf-03 clvmd: Deactivating VG vg_gfs1: failed Jan 16 22:03:14 sf-03 vgchange: Can't deactivate volume group "vg_gfs2" with 1 open logical volume(s) Jan 16 22:03:14 sf-03 clvmd: Deactivating VG vg_gfs2: failed Jan 16 22:10:36 sf-03 ccsd[3229]: cluster.conf (cluster name = gfs_cluster, version = 2) found. Jan 16 22:11:59 sf-03 vgchange: 3 logical volume(s) in volume group "vg_gfs1" now active Jan 16 22:11:59 sf-03 vgchange: 3 logical volume(s) in volume group "vg_gfs2" now active Jan 16 22:12:20 sf-03 kernel: GFS 2.6.9-42.2 (built Oct 21 2005 11:57:26) installed Jan 16 22:12:20 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", "gfs_cluster:gfs_raiz" Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Joined cluster. Now mounting FS... Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: Trying to acquire journal lock... Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: Looking at journal... Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: Done Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Scanning for log elements... Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Found 0 unlinked inodes Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Found quota changes for 0 IDs Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Done Jan 16 22:12:22 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", "gfs_cluster:gfs_cache" Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Joined cluster. Now mounting FS... Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: jid=2: Trying to acquire journal lock... Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: jid=2: Looking at journal... Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: jid=2: Done Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Scanning for log elements... Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Found 3 unlinked inodes Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Found quota changes for 0 IDs Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Done Jan 16 22:12:26 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", "gfs_cluster:gfs_htdocs" Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: Joined cluster. Now mounting FS... Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: jid=2: Trying to acquire journal lock... Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: jid=2: Looking at journal... Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: jid=2: Done Jan 16 22:12:28 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", "gfs_cluster:lv_webcalle20" Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: Joined cluster. Now mounting FS... Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: jid=2: Trying to acquire journal lock... Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: jid=2: Looking at journal... Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: jid=2: Done Jan 16 22:12:30 sf-03 gfs: Mounting GFS filesystems: succeeded At the end we saw a gfs_tool option interesting "lockdump" and before reboot ran it. At the end there are some output of my mountpoint If you need more information about configuration or something please say me. gfs_tool lockdump /mountpoint: Glock (5, 2114727) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (2, 119206) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 119206/119206 type = 1 i_count = 1 i_flags = vnode = yes Glock (2, 228692) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 228692/228692 type = 1 i_count = 1 i_flags = vnode = yes Glock (5, 735842) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 Glock (5, 1418402) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (2, 1885491) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 1885491/1885491 type = 1 i_count = 1 i_flags = vnode = yes Glock (2, 399729) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 399729/399729 type = 1 i_count = 1 i_flags = vnode = yes Glock (5, 1386646) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (5, 241672) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (5, 207713) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (2, 946688) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 946688/946688 type = 1 i_count = 1 i_flags = vnode = yes Glock (5, 30184) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (2, 8340) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 8340/8340 type = 1 i_count = 1 i_flags = vnode = yes Glock (2, 2308745) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 2308745/2308745 type = 1 i_count = 1 i_flags = vnode = yes Glock (2, 949390) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 949390/949390 type = 1 i_count = 1 i_flags = vnode = yes Glock (5, 548987) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (2, 1437881) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 1437881/1437881 type = 1 i_count = 1 i_flags = vnode = yes Glock (5, 139108) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (2, 261765) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 261765/261765 type = 1 i_count = 1 i_flags = vnode = yes Glock (2, 2530374) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 2530374/2530374 type = 1 i_count = 1 i_flags = vnode = yes Glock (5, 33091) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (2, 863848) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 863848/863848 type = 1 i_count = 1 i_flags = vnode = yes Glock (5, 208549) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (2, 51708) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 51708/51708 type = 2 i_count = 1 i_flags = vnode = yes Glock (5, 1887878) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (5, 864369) gl_flags = gl_count = 2 gl_state = 3 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = no ail_bufs = no Holder owner = -1 gh_state = 3 gh_flags = 5 7 error = 0 gh_iflags = 1 6 7 Glock (2, 1436746) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 1436746/1436746 type = 1 i_count = 1 i_flags = vnode = yes Glock (2, 608211) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 608211/608211 type = 1 i_count = 1 i_flags = vnode = yes Glock (2, 609430) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 609430/609430 type = 1 i_count = 1 i_flags = vnode = yes Glock (2, 893532) gl_flags = gl_count = 2 gl_state = 0 req_gh = no req_bh = no lvb_count = 0 object = yes new_le = no incore_le = no reclaim = no aspace = 0 ail_bufs = no Inode: num = 893532/893532 type = 1 i_count = 1 i_flags = vnode = yes Thanks in advance. -- ________________________________________ Bartolom? Rodr?guez Bordallo Departamento de Explotaci?n de Servicios FUJITSU ESPA?A SERVICES, S.A.U. Camino Cerro de los Gamos, 1 28224 Pozuelo de Alarc?n, Madrid Tel.: 902 11 40 10 Mail: brodriguezb at fujitsu.es ________________________________________ La informaci?n contenida en este e-mail es confidencial y va dirigida ?nicamente al receptor que aparece como destinatario. Si ha recibido este e-mail por error, por favor, notif?quenoslo inmediatamente y b?rrelo de su sistema. Por favor, en tal caso, no lo copie ni lo use para ning?n prop?sito, ni revele sus contenidos a ninguna persona ni lo almacene ni copie esta informaci?n en ning?n medio. From brodriguezb at fujitsu.es Wed Jan 18 11:04:10 2006 From: brodriguezb at fujitsu.es (=?UTF-8?B?QmFydG9sb23DqSBSb2Ryw61ndWV6?=) Date: Wed, 18 Jan 2006 12:04:10 +0100 Subject: [Linux-cluster] GFS and httpd process with stat "D" In-Reply-To: <43CE1B9C.6020604@fujitsu.es> References: <43CE1B9C.6020604@fujitsu.es> Message-ID: <43CE20AA.70701@fujitsu.es> Im looking for answer in google and i have find another problem same mine: http://www.redhat.com/archives/linux-cluster/2005-April/msg00021.html This was 2005 April and it was send to this list, but there isnt answer :( Maybe the autor fix the problem. Regards, Tole. Bartolom? Rodr?guez wrote: > Hi list, > > Release: Red Hat Enterprise Linux ES release 4 (Nahant Update 2) > GFS-6.1.2-0 > GFS-kernel-smp-2.6.9-42.2 > Kernel running: 2.6.9-22.0.1.ELsmp > Apache: httpd-2.0.52-19.ent > I have 4 servers with GFS. These servers have shared disk with > GFS (The disks are in a san with fiberchannel). This scene is in > production. 2 times in 2 weeks the httpd process become stat "D". We > thinks GFS is blocking something (file or inode... i dont know). And > when httpd try read or writhe there, then become stat "D": > > apache 14383 1.5 0.3 24004 13300 ? D 17:36 3:57 > /usr/sbin/httpd > apache 14503 1.6 0.3 23996 13296 ? D 17:49 3:53 > /usr/sbin/httpd > apache 14516 1.6 0.3 23988 13288 ? D 17:50 3:58 > /usr/sbin/httpd > apache 14522 1.6 0.3 24044 13340 ? D 17:50 3:54 > /usr/sbin/httpd > apache 14635 1.5 0.3 24092 13392 ? D 18:00 3:23 > /usr/sbin/httpd > apache 14661 1.6 0.3 24048 13344 ? D 18:03 3:33 > /usr/sbin/httpd > apache 14662 1.7 0.3 24096 13392 ? D 18:03 3:48 > /usr/sbin/httpd > apache 14671 1.4 0.3 24044 13340 ? D 18:03 3:12 > /usr/sbin/httpd > apache 14687 1.6 0.3 24020 13320 ? D 18:03 3:42 > /usr/sbin/httpd > apache 17329 0.8 0.3 24356 13640 ? D 19:42 1:04 > /usr/sbin/httpd > apache 17331 0.9 0.3 24000 13284 ? D 19:42 1:05 > /usr/sbin/httpd > apache 17332 1.2 0.3 24156 13456 ? D 19:43 1:29 > /usr/sbin/httpd > apache 17335 0.9 0.3 24128 13412 ? D 19:43 1:06 > /usr/sbin/httpd > apache 17345 0.9 0.3 23908 13120 ? D 19:43 1:07 > /usr/sbin/httpd > apache 17347 0.9 0.3 23608 12896 ? D 19:43 1:07 > /usr/sbin/httpd > apache 17385 0.8 0.3 24120 13416 ? D 19:45 1:01 > /usr/sbin/httpd > apache 17386 1.2 0.3 23932 13228 ? D 19:46 1:26 > /usr/sbin/httpd > apache 17387 1.1 0.3 23904 13200 ? D 19:46 1:17 > /usr/sbin/httpd > apache 17398 0.9 0.3 24064 13360 ? D 19:49 1:04 > /usr/sbin/httpd > apache 17599 1.3 0.3 24392 13676 ? D 20:41 0:52 > /usr/sbin/httpd > apache 17606 0.3 0.3 23472 12632 ? D 20:42 0:11 > /usr/sbin/httpd > apache 17607 0.6 0.3 24024 13312 ? D 20:42 0:23 > /usr/sbin/httpd > apache 17608 1.1 0.3 23880 13164 ? D 20:42 0:44 > /usr/sbin/httpd > apache 17609 0.3 0.3 23556 12736 ? D 20:42 0:11 > /usr/sbin/httpd > apache 17620 0.5 0.3 23928 13204 ? D 20:43 0:21 > /usr/sbin/httpd > apache 17632 0.0 0.2 23168 12268 ? D 20:46 0:03 > /usr/sbin/httpd > apache 17633 0.6 0.3 23448 12704 ? D 20:47 0:20 > /usr/sbin/httpd > apache 17635 0.0 0.3 23464 12592 ? D 20:48 0:01 > /usr/sbin/httpd > apache 17636 0.0 0.3 23448 12584 ? D 20:49 0:02 > /usr/sbin/httpd > apache 17637 0.2 0.3 23472 12640 ? D 20:49 0:07 > /usr/sbin/httpd > apache 17638 0.2 0.3 23508 12772 ? D 20:49 0:07 > /usr/sbin/httpd > apache 17639 0.2 0.3 23488 12672 ? D 20:49 0:07 > /usr/sbin/httpd > apache 17643 0.1 0.3 23860 13104 ? D 20:50 0:03 > /usr/sbin/httpd > apache 17644 0.0 0.2 23156 12376 ? D 20:51 0:02 > /usr/sbin/httpd > apache 17645 1.2 0.3 23472 12680 ? D 20:51 0:39 > /usr/sbin/httpd > apache 17647 0.6 0.2 23184 12428 ? D 20:52 0:19 > /usr/sbin/httpd > apache 17648 0.5 0.3 23496 12760 ? D 20:52 0:18 > /usr/sbin/httpd > apache 17665 0.4 0.3 23524 12700 ? D 20:54 0:11 > /usr/sbin/httpd > apache 17667 0.7 0.3 23972 13240 ? D 20:54 0:20 > /usr/sbin/httpd > apache 17668 0.1 0.3 23500 12652 ? D 20:54 0:03 > /usr/sbin/httpd > apache 17669 0.2 0.3 23488 12652 ? D 20:54 0:06 > /usr/sbin/httpd > apache 17671 0.8 0.3 23696 12888 ? D 20:55 0:25 > /usr/sbin/httpd > apache 17732 0.1 0.2 23188 12264 ? D 21:01 0:03 > /usr/sbin/httpd > apache 17755 0.4 0.3 23588 12768 ? D 21:02 0:11 > /usr/sbin/httpd > > This httpd process appear in the 4 servers. The first time, we > reboot all the cluster?s members, but the servers cant unmount GFS > filesystem, therefore the only solution is a fisical poweroff > (botton). The second time we saw the first httpd process with stat "D" > and then reboot only this server. Command reboot dont work again and > fisical poweroff is required. But then, the others 3 servers fix > without reboot. In this 3 servers, the httpd process in stat "D" > became stat "S", all ok. > I had read this mail: > https://www.redhat.com/archives/linux-cluster/2005-December/msg00054.html > I think is very similar, but in my log dont appear explicitily > "stuck in gfs" i dont know if i have the max logging for gfs. I only > see in my logs the try of unmount or mount: > > Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > withdrawing from cluster at user's request > Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > about to withdraw from the cluster > Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > waiting for outstanding I/O > Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > telling LM to withdraw > Jan 16 21:53:55 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > withdrawn > Jan 16 21:58:45 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:02:01 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:02:13 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:02:25 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:02:37 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:02:49 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:03:01 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:03:14 sf-03 vgchange: Can't deactivate volume group > "vg_gfs1" with 2 open logical volume(s) > Jan 16 22:03:14 sf-03 clvmd: Deactivating VG vg_gfs1: failed > Jan 16 22:03:14 sf-03 vgchange: Can't deactivate volume group > "vg_gfs2" with 1 open logical volume(s) > Jan 16 22:03:14 sf-03 clvmd: Deactivating VG vg_gfs2: failed > Jan 16 22:10:36 sf-03 ccsd[3229]: cluster.conf (cluster name = > gfs_cluster, version = 2) found. > Jan 16 22:11:59 sf-03 vgchange: 3 logical volume(s) in volume group > "vg_gfs1" now active > Jan 16 22:11:59 sf-03 vgchange: 3 logical volume(s) in volume group > "vg_gfs2" now active > Jan 16 22:12:20 sf-03 kernel: GFS 2.6.9-42.2 (built Oct 21 2005 > 11:57:26) installed > Jan 16 22:12:20 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", > "gfs_cluster:gfs_raiz" > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Joined > cluster. Now mounting FS... > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: > Trying to acquire journal lock... > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: > Looking at journal... > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: > Done > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: > Scanning for log elements... > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Found > 0 unlinked inodes > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Found > quota changes for 0 IDs > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Done > Jan 16 22:12:22 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", > "gfs_cluster:gfs_cache" > Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: > Joined cluster. Now mounting FS... > Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: > jid=2: Trying to acquire journal lock... > Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: > jid=2: Looking at journal... > Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: > jid=2: Done > Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: > Scanning for log elements... > Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Found > 3 unlinked inodes > Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Found > quota changes for 0 IDs > Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Done > Jan 16 22:12:26 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", > "gfs_cluster:gfs_htdocs" > Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > Joined cluster. Now mounting FS... > Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > jid=2: Trying to acquire journal lock... > Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > jid=2: Looking at journal... > Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > jid=2: Done > Jan 16 22:12:28 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", > "gfs_cluster:lv_webcalle20" > Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: > Joined cluster. Now mounting FS... > Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: > jid=2: Trying to acquire journal lock... > Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: > jid=2: Looking at journal... > Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: > jid=2: Done > Jan 16 22:12:30 sf-03 gfs: Mounting GFS filesystems: succeeded > > At the end we saw a gfs_tool option interesting "lockdump" and > before reboot ran it. At the end there are some output of my mountpoint > If you need more information about configuration or something > please say me. > > gfs_tool lockdump /mountpoint: > > Glock (5, 2114727) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 119206) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 119206/119206 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 228692) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 228692/228692 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 735842) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 > Glock (5, 1418402) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 1885491) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 1885491/1885491 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 399729) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 399729/399729 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 1386646) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (5, 241672) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (5, 207713) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 946688) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 946688/946688 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 30184) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 8340) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 8340/8340 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 2308745) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 2308745/2308745 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 949390) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 949390/949390 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 548987) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 1437881) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 1437881/1437881 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 139108) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 261765) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 261765/261765 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 2530374) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 2530374/2530374 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 33091) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 863848) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 863848/863848 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 208549) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 51708) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 51708/51708 > type = 2 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 1887878) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (5, 864369) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 1436746) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 1436746/1436746 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 608211) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 608211/608211 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 609430) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 609430/609430 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 893532) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 893532/893532 > type = 1 > i_count = 1 > i_flags = > vnode = yes > > Thanks in advance. > -- ________________________________________ Bartolom? Rodr?guez Bordallo Departamento de Explotaci?n de Servicios FUJITSU ESPA?A SERVICES, S.A.U. Camino Cerro de los Gamos, 1 28224 Pozuelo de Alarc?n, Madrid Tel.: 902 11 40 10 Mail: brodriguezb at fujitsu.es ________________________________________ La informaci?n contenida en este e-mail es confidencial y va dirigida ?nicamente al receptor que aparece como destinatario. Si ha recibido este e-mail por error, por favor, notif?quenoslo inmediatamente y b?rrelo de su sistema. Por favor, en tal caso, no lo copie ni lo use para ning?n prop?sito, ni revele sus contenidos a ninguna persona ni lo almacene ni copie esta informaci?n en ning?n medio. From dgolden at cp.dias.ie Wed Jan 18 12:48:31 2006 From: dgolden at cp.dias.ie (David Golden) Date: Wed, 18 Jan 2006 12:48:31 +0000 Subject: [Linux-cluster] loading Red Hat on multiple nodes In-Reply-To: <2315046d0601162009j24aaafbfs@mail.gmail.com> References: <2315046d0601162009j24aaafbfs@mail.gmail.com> Message-ID: <20060118124831.GC2273@ariadne.cp.dias.ie> > > I received a question concerning Red Hat and wanted to see what the opinions > > are of everyone here. > > > > "Which software is best to load the Redhat Hat Linux ES4 on 150-odd nodes > >from one cluster management node?" > > SystemImager vs. Rocks Avalanche (bittorrent-enhanced kickstart), I guess. SystemImager's more general-purpose (e.g. not limited to redhatoids, easily used for updating with up-to-the-minute patched images), but its "Flamethrower" distribution method (to make it vaguely throughput-competitive with avalanche) uses multicast and therefore depends on multicast working right and is generally a bit more brittle (it's 'casting, so the installs have to be synchronised). There's nothing stopping someone writing a bittorrent patch for systemimager I guess, to get the best of both worlds. http://www.rocksclusters.org/ http://www.systemimager.org/ From hlawatschek at atix.de Wed Jan 18 13:53:01 2006 From: hlawatschek at atix.de (Mark Hlawatschek) Date: Wed, 18 Jan 2006 14:53:01 +0100 Subject: [Linux-cluster] GFS and httpd process with stat "D" In-Reply-To: <43CE1B9C.6020604@fujitsu.es> References: <43CE1B9C.6020604@fujitsu.es> Message-ID: <200601181453.02368.hlawatschek@atix.de> Hi Bartolom? , this can happen, if you are using php session ids. The httpd processes can run into a deadlock situation when two httpd processes try to flock the same session id file. In that case they stay in the "D" state which means waiting for I/O. To prevent this, we wrote a patch for php with the following features: - make transaction based file locking on session id files. - prevent php from flocking session ids (this is the NFS like behaviour, as NFS does not support flocks.) You can download the patch at http://open-sharedroot.org Have Fun, Mark On Wednesday 18 January 2006 11:42, Bartolom? Rodr?guez wrote: > Hi list, > > Release: Red Hat Enterprise Linux ES release 4 (Nahant Update 2) > GFS-6.1.2-0 > GFS-kernel-smp-2.6.9-42.2 > Kernel running: 2.6.9-22.0.1.ELsmp > Apache: httpd-2.0.52-19.ent > > I have 4 servers with GFS. These servers have shared disk with GFS > (The disks are in a san with fiberchannel). This scene is in production. > 2 times in 2 weeks the httpd process become stat "D". We thinks GFS is > blocking something (file or inode... i dont know). And when httpd try > read or writhe there, then become stat "D": > > apache 14383 1.5 0.3 24004 13300 ? D 17:36 3:57 > /usr/sbin/httpd > apache 14503 1.6 0.3 23996 13296 ? D 17:49 3:53 > /usr/sbin/httpd > apache 14516 1.6 0.3 23988 13288 ? D 17:50 3:58 > /usr/sbin/httpd > apache 14522 1.6 0.3 24044 13340 ? D 17:50 3:54 > /usr/sbin/httpd > apache 14635 1.5 0.3 24092 13392 ? D 18:00 3:23 > /usr/sbin/httpd > apache 14661 1.6 0.3 24048 13344 ? D 18:03 3:33 > /usr/sbin/httpd > apache 14662 1.7 0.3 24096 13392 ? D 18:03 3:48 > /usr/sbin/httpd > apache 14671 1.4 0.3 24044 13340 ? D 18:03 3:12 > /usr/sbin/httpd > apache 14687 1.6 0.3 24020 13320 ? D 18:03 3:42 > /usr/sbin/httpd > apache 17329 0.8 0.3 24356 13640 ? D 19:42 1:04 > /usr/sbin/httpd > apache 17331 0.9 0.3 24000 13284 ? D 19:42 1:05 > /usr/sbin/httpd > apache 17332 1.2 0.3 24156 13456 ? D 19:43 1:29 > /usr/sbin/httpd > apache 17335 0.9 0.3 24128 13412 ? D 19:43 1:06 > /usr/sbin/httpd > apache 17345 0.9 0.3 23908 13120 ? D 19:43 1:07 > /usr/sbin/httpd > apache 17347 0.9 0.3 23608 12896 ? D 19:43 1:07 > /usr/sbin/httpd > apache 17385 0.8 0.3 24120 13416 ? D 19:45 1:01 > /usr/sbin/httpd > apache 17386 1.2 0.3 23932 13228 ? D 19:46 1:26 > /usr/sbin/httpd > apache 17387 1.1 0.3 23904 13200 ? D 19:46 1:17 > /usr/sbin/httpd > apache 17398 0.9 0.3 24064 13360 ? D 19:49 1:04 > /usr/sbin/httpd > apache 17599 1.3 0.3 24392 13676 ? D 20:41 0:52 > /usr/sbin/httpd > apache 17606 0.3 0.3 23472 12632 ? D 20:42 0:11 > /usr/sbin/httpd > apache 17607 0.6 0.3 24024 13312 ? D 20:42 0:23 > /usr/sbin/httpd > apache 17608 1.1 0.3 23880 13164 ? D 20:42 0:44 > /usr/sbin/httpd > apache 17609 0.3 0.3 23556 12736 ? D 20:42 0:11 > /usr/sbin/httpd > apache 17620 0.5 0.3 23928 13204 ? D 20:43 0:21 > /usr/sbin/httpd > apache 17632 0.0 0.2 23168 12268 ? D 20:46 0:03 > /usr/sbin/httpd > apache 17633 0.6 0.3 23448 12704 ? D 20:47 0:20 > /usr/sbin/httpd > apache 17635 0.0 0.3 23464 12592 ? D 20:48 0:01 > /usr/sbin/httpd > apache 17636 0.0 0.3 23448 12584 ? D 20:49 0:02 > /usr/sbin/httpd > apache 17637 0.2 0.3 23472 12640 ? D 20:49 0:07 > /usr/sbin/httpd > apache 17638 0.2 0.3 23508 12772 ? D 20:49 0:07 > /usr/sbin/httpd > apache 17639 0.2 0.3 23488 12672 ? D 20:49 0:07 > /usr/sbin/httpd > apache 17643 0.1 0.3 23860 13104 ? D 20:50 0:03 > /usr/sbin/httpd > apache 17644 0.0 0.2 23156 12376 ? D 20:51 0:02 > /usr/sbin/httpd > apache 17645 1.2 0.3 23472 12680 ? D 20:51 0:39 > /usr/sbin/httpd > apache 17647 0.6 0.2 23184 12428 ? D 20:52 0:19 > /usr/sbin/httpd > apache 17648 0.5 0.3 23496 12760 ? D 20:52 0:18 > /usr/sbin/httpd > apache 17665 0.4 0.3 23524 12700 ? D 20:54 0:11 > /usr/sbin/httpd > apache 17667 0.7 0.3 23972 13240 ? D 20:54 0:20 > /usr/sbin/httpd > apache 17668 0.1 0.3 23500 12652 ? D 20:54 0:03 > /usr/sbin/httpd > apache 17669 0.2 0.3 23488 12652 ? D 20:54 0:06 > /usr/sbin/httpd > apache 17671 0.8 0.3 23696 12888 ? D 20:55 0:25 > /usr/sbin/httpd > apache 17732 0.1 0.2 23188 12264 ? D 21:01 0:03 > /usr/sbin/httpd > apache 17755 0.4 0.3 23588 12768 ? D 21:02 0:11 > /usr/sbin/httpd > > This httpd process appear in the 4 servers. The first time, we > reboot all the cluster?s members, but the servers cant unmount GFS > filesystem, therefore the only solution is a fisical poweroff (botton). > The second time we saw the first httpd process with stat "D" and then > reboot only this server. Command reboot dont work again and fisical > poweroff is required. But then, the others 3 servers fix without reboot. > In this 3 servers, the httpd process in stat "D" became stat "S", all ok. > I had read this mail: > https://www.redhat.com/archives/linux-cluster/2005-December/msg00054.html > I think is very similar, but in my log dont appear explicitily > "stuck in gfs" i dont know if i have the max logging for gfs. I only see > in my logs the try of unmount or mount: > > Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > withdrawing from cluster at user's request > Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: about > to withdraw from the cluster > Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > waiting for outstanding I/O > Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: > telling LM to withdraw > Jan 16 21:53:55 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: withdrawn > Jan 16 21:58:45 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:02:01 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:02:13 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:02:25 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:02:37 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:02:49 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:03:01 sf-03 gfs: Unmounting GFS filesystems: failed > Jan 16 22:03:14 sf-03 vgchange: Can't deactivate volume group > "vg_gfs1" with 2 open logical volume(s) > Jan 16 22:03:14 sf-03 clvmd: Deactivating VG vg_gfs1: failed > Jan 16 22:03:14 sf-03 vgchange: Can't deactivate volume group > "vg_gfs2" with 1 open logical volume(s) > Jan 16 22:03:14 sf-03 clvmd: Deactivating VG vg_gfs2: failed > Jan 16 22:10:36 sf-03 ccsd[3229]: cluster.conf (cluster name = > gfs_cluster, version = 2) found. > Jan 16 22:11:59 sf-03 vgchange: 3 logical volume(s) in volume group > "vg_gfs1" now active > Jan 16 22:11:59 sf-03 vgchange: 3 logical volume(s) in volume group > "vg_gfs2" now active > Jan 16 22:12:20 sf-03 kernel: GFS 2.6.9-42.2 (built Oct 21 2005 > 11:57:26) installed > Jan 16 22:12:20 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", > "gfs_cluster:gfs_raiz" > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Joined > cluster. Now mounting FS... > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: > Trying to acquire journal lock... > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: > Looking at journal... > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: Done > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Scanning > for log elements... > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Found 0 > unlinked inodes > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Found > quota changes for 0 IDs > Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Done > Jan 16 22:12:22 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", > "gfs_cluster:gfs_cache" > Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Joined > cluster. Now mounting FS... > Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: jid=2: > Trying to acquire journal lock... > Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: jid=2: > Looking at journal... > Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: jid=2: > Done Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: > Scanning for log elements... > Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Found 3 > unlinked inodes > Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Found > quota changes for 0 IDs > Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Done > Jan 16 22:12:26 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", > "gfs_cluster:gfs_htdocs" > Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: Joined > cluster. Now mounting FS... > Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: jid=2: > Trying to acquire journal lock... > Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: jid=2: > Looking at journal... > Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: jid=2: > Done > Jan 16 22:12:28 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", > "gfs_cluster:lv_webcalle20" > Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: > Joined cluster. Now mounting FS... > Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: > jid=2: Trying to acquire journal lock... > Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: > jid=2: Looking at journal... > Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: > jid=2: Done > Jan 16 22:12:30 sf-03 gfs: Mounting GFS filesystems: succeeded > > At the end we saw a gfs_tool option interesting "lockdump" and > before reboot ran it. At the end there are some output of my mountpoint > If you need more information about configuration or something please > say me. > > gfs_tool lockdump /mountpoint: > > Glock (5, 2114727) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 119206) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 119206/119206 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 228692) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 228692/228692 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 735842) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 > Glock (5, 1418402) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 1885491) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 1885491/1885491 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 399729) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 399729/399729 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 1386646) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (5, 241672) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (5, 207713) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 946688) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 946688/946688 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 30184) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 8340) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 8340/8340 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 2308745) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 2308745/2308745 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 949390) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 949390/949390 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 548987) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 1437881) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 1437881/1437881 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 139108) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 261765) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 261765/261765 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 2530374) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 2530374/2530374 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 33091) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 863848) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 863848/863848 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 208549) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 51708) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 51708/51708 > type = 2 > i_count = 1 > i_flags = > vnode = yes > Glock (5, 1887878) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (5, 864369) > gl_flags = > gl_count = 2 > gl_state = 3 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = no > ail_bufs = no > Holder > owner = -1 > gh_state = 3 > gh_flags = 5 7 > error = 0 > gh_iflags = 1 6 7 > Glock (2, 1436746) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 1436746/1436746 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 608211) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 608211/608211 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 609430) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 609430/609430 > type = 1 > i_count = 1 > i_flags = > vnode = yes > Glock (2, 893532) > gl_flags = > gl_count = 2 > gl_state = 0 > req_gh = no > req_bh = no > lvb_count = 0 > object = yes > new_le = no > incore_le = no > reclaim = no > aspace = 0 > ail_bufs = no > Inode: > num = 893532/893532 > type = 1 > i_count = 1 > i_flags = > vnode = yes > > Thanks in advance. -- Gruss / Regards, Dipl.-Ing. Mark Hlawatschek Phone: +49-89 121 409-55 http://www.atix.de/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From brodriguezb at fujitsu.es Wed Jan 18 14:33:19 2006 From: brodriguezb at fujitsu.es (=?UTF-8?B?QmFydG9sb23DqSBSb2Ryw61ndWV6?=) Date: Wed, 18 Jan 2006 15:33:19 +0100 Subject: [Linux-cluster] GFS and httpd process with stat "D" In-Reply-To: <200601181453.02368.hlawatschek@atix.de> References: <43CE1B9C.6020604@fujitsu.es> <200601181453.02368.hlawatschek@atix.de> Message-ID: <43CE51AF.5040403@fujitsu.es> Hello Mark and thanks for your answer. Im not using php session ids, this sessions are controlled by mysql. But all aplication (web) is php. It will be the same problem ? And there are some form for see which file is been blocked ? Thanks and regards. Tole. Mark Hlawatschek wrote: >Hi Bartolom? , > >this can happen, if you are using php session ids. The httpd processes can run >into a deadlock situation when two httpd processes try to flock the same >session id file. In that case they stay in the "D" state which means waiting >for I/O. >To prevent this, we wrote a patch for php with the following features: >- make transaction based file locking on session id files. >- prevent php from flocking session ids (this is the NFS like behaviour, as >NFS does not support flocks.) > >You can download the patch at http://open-sharedroot.org > >Have Fun, >Mark > > > >On Wednesday 18 January 2006 11:42, Bartolom? Rodr?guez wrote: > > >> Hi list, >> >> Release: Red Hat Enterprise Linux ES release 4 (Nahant Update 2) >> GFS-6.1.2-0 >> GFS-kernel-smp-2.6.9-42.2 >> Kernel running: 2.6.9-22.0.1.ELsmp >> Apache: httpd-2.0.52-19.ent >> >> I have 4 servers with GFS. These servers have shared disk with GFS >>(The disks are in a san with fiberchannel). This scene is in production. >>2 times in 2 weeks the httpd process become stat "D". We thinks GFS is >>blocking something (file or inode... i dont know). And when httpd try >>read or writhe there, then become stat "D": >> >>apache 14383 1.5 0.3 24004 13300 ? D 17:36 3:57 >>/usr/sbin/httpd >>apache 14503 1.6 0.3 23996 13296 ? D 17:49 3:53 >>/usr/sbin/httpd >>apache 14516 1.6 0.3 23988 13288 ? D 17:50 3:58 >>/usr/sbin/httpd >>apache 14522 1.6 0.3 24044 13340 ? D 17:50 3:54 >>/usr/sbin/httpd >>apache 14635 1.5 0.3 24092 13392 ? D 18:00 3:23 >>/usr/sbin/httpd >>apache 14661 1.6 0.3 24048 13344 ? D 18:03 3:33 >>/usr/sbin/httpd >>apache 14662 1.7 0.3 24096 13392 ? D 18:03 3:48 >>/usr/sbin/httpd >>apache 14671 1.4 0.3 24044 13340 ? D 18:03 3:12 >>/usr/sbin/httpd >>apache 14687 1.6 0.3 24020 13320 ? D 18:03 3:42 >>/usr/sbin/httpd >>apache 17329 0.8 0.3 24356 13640 ? D 19:42 1:04 >>/usr/sbin/httpd >>apache 17331 0.9 0.3 24000 13284 ? D 19:42 1:05 >>/usr/sbin/httpd >>apache 17332 1.2 0.3 24156 13456 ? D 19:43 1:29 >>/usr/sbin/httpd >>apache 17335 0.9 0.3 24128 13412 ? D 19:43 1:06 >>/usr/sbin/httpd >>apache 17345 0.9 0.3 23908 13120 ? D 19:43 1:07 >>/usr/sbin/httpd >>apache 17347 0.9 0.3 23608 12896 ? D 19:43 1:07 >>/usr/sbin/httpd >>apache 17385 0.8 0.3 24120 13416 ? D 19:45 1:01 >>/usr/sbin/httpd >>apache 17386 1.2 0.3 23932 13228 ? D 19:46 1:26 >>/usr/sbin/httpd >>apache 17387 1.1 0.3 23904 13200 ? D 19:46 1:17 >>/usr/sbin/httpd >>apache 17398 0.9 0.3 24064 13360 ? D 19:49 1:04 >>/usr/sbin/httpd >>apache 17599 1.3 0.3 24392 13676 ? D 20:41 0:52 >>/usr/sbin/httpd >>apache 17606 0.3 0.3 23472 12632 ? D 20:42 0:11 >>/usr/sbin/httpd >>apache 17607 0.6 0.3 24024 13312 ? D 20:42 0:23 >>/usr/sbin/httpd >>apache 17608 1.1 0.3 23880 13164 ? D 20:42 0:44 >>/usr/sbin/httpd >>apache 17609 0.3 0.3 23556 12736 ? D 20:42 0:11 >>/usr/sbin/httpd >>apache 17620 0.5 0.3 23928 13204 ? D 20:43 0:21 >>/usr/sbin/httpd >>apache 17632 0.0 0.2 23168 12268 ? D 20:46 0:03 >>/usr/sbin/httpd >>apache 17633 0.6 0.3 23448 12704 ? D 20:47 0:20 >>/usr/sbin/httpd >>apache 17635 0.0 0.3 23464 12592 ? D 20:48 0:01 >>/usr/sbin/httpd >>apache 17636 0.0 0.3 23448 12584 ? D 20:49 0:02 >>/usr/sbin/httpd >>apache 17637 0.2 0.3 23472 12640 ? D 20:49 0:07 >>/usr/sbin/httpd >>apache 17638 0.2 0.3 23508 12772 ? D 20:49 0:07 >>/usr/sbin/httpd >>apache 17639 0.2 0.3 23488 12672 ? D 20:49 0:07 >>/usr/sbin/httpd >>apache 17643 0.1 0.3 23860 13104 ? D 20:50 0:03 >>/usr/sbin/httpd >>apache 17644 0.0 0.2 23156 12376 ? D 20:51 0:02 >>/usr/sbin/httpd >>apache 17645 1.2 0.3 23472 12680 ? D 20:51 0:39 >>/usr/sbin/httpd >>apache 17647 0.6 0.2 23184 12428 ? D 20:52 0:19 >>/usr/sbin/httpd >>apache 17648 0.5 0.3 23496 12760 ? D 20:52 0:18 >>/usr/sbin/httpd >>apache 17665 0.4 0.3 23524 12700 ? D 20:54 0:11 >>/usr/sbin/httpd >>apache 17667 0.7 0.3 23972 13240 ? D 20:54 0:20 >>/usr/sbin/httpd >>apache 17668 0.1 0.3 23500 12652 ? D 20:54 0:03 >>/usr/sbin/httpd >>apache 17669 0.2 0.3 23488 12652 ? D 20:54 0:06 >>/usr/sbin/httpd >>apache 17671 0.8 0.3 23696 12888 ? D 20:55 0:25 >>/usr/sbin/httpd >>apache 17732 0.1 0.2 23188 12264 ? D 21:01 0:03 >>/usr/sbin/httpd >>apache 17755 0.4 0.3 23588 12768 ? D 21:02 0:11 >>/usr/sbin/httpd >> >> This httpd process appear in the 4 servers. The first time, we >>reboot all the cluster?s members, but the servers cant unmount GFS >>filesystem, therefore the only solution is a fisical poweroff (botton). >>The second time we saw the first httpd process with stat "D" and then >>reboot only this server. Command reboot dont work again and fisical >>poweroff is required. But then, the others 3 servers fix without reboot. >>In this 3 servers, the httpd process in stat "D" became stat "S", all ok. >> I had read this mail: >>https://www.redhat.com/archives/linux-cluster/2005-December/msg00054.html >> I think is very similar, but in my log dont appear explicitily >>"stuck in gfs" i dont know if i have the max logging for gfs. I only see >>in my logs the try of unmount or mount: >> >>Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: >>withdrawing from cluster at user's request >>Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: about >>to withdraw from the cluster >>Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: >>waiting for outstanding I/O >>Jan 16 21:53:54 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: >>telling LM to withdraw >>Jan 16 21:53:55 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: withdrawn >>Jan 16 21:58:45 sf-03 gfs: Unmounting GFS filesystems: failed >>Jan 16 22:02:01 sf-03 gfs: Unmounting GFS filesystems: failed >>Jan 16 22:02:13 sf-03 gfs: Unmounting GFS filesystems: failed >>Jan 16 22:02:25 sf-03 gfs: Unmounting GFS filesystems: failed >>Jan 16 22:02:37 sf-03 gfs: Unmounting GFS filesystems: failed >>Jan 16 22:02:49 sf-03 gfs: Unmounting GFS filesystems: failed >>Jan 16 22:03:01 sf-03 gfs: Unmounting GFS filesystems: failed >>Jan 16 22:03:14 sf-03 vgchange: Can't deactivate volume group >>"vg_gfs1" with 2 open logical volume(s) >>Jan 16 22:03:14 sf-03 clvmd: Deactivating VG vg_gfs1: failed >>Jan 16 22:03:14 sf-03 vgchange: Can't deactivate volume group >>"vg_gfs2" with 1 open logical volume(s) >>Jan 16 22:03:14 sf-03 clvmd: Deactivating VG vg_gfs2: failed >>Jan 16 22:10:36 sf-03 ccsd[3229]: cluster.conf (cluster name = >>gfs_cluster, version = 2) found. >>Jan 16 22:11:59 sf-03 vgchange: 3 logical volume(s) in volume group >>"vg_gfs1" now active >>Jan 16 22:11:59 sf-03 vgchange: 3 logical volume(s) in volume group >>"vg_gfs2" now active >>Jan 16 22:12:20 sf-03 kernel: GFS 2.6.9-42.2 (built Oct 21 2005 >>11:57:26) installed >>Jan 16 22:12:20 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", >>"gfs_cluster:gfs_raiz" >>Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Joined >>cluster. Now mounting FS... >>Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: >>Trying to acquire journal lock... >>Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: >>Looking at journal... >>Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: jid=2: Done >>Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Scanning >>for log elements... >>Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Found 0 >>unlinked inodes >>Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Found >>quota changes for 0 IDs >>Jan 16 22:12:22 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_raiz.2: Done >>Jan 16 22:12:22 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", >>"gfs_cluster:gfs_cache" >>Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Joined >>cluster. Now mounting FS... >>Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: jid=2: >>Trying to acquire journal lock... >>Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: jid=2: >>Looking at journal... >>Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: jid=2: >>Done Jan 16 22:12:25 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: >>Scanning for log elements... >>Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Found 3 >>unlinked inodes >>Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Found >>quota changes for 0 IDs >>Jan 16 22:12:26 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_cache.2: Done >>Jan 16 22:12:26 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", >>"gfs_cluster:gfs_htdocs" >>Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: Joined >>cluster. Now mounting FS... >>Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: jid=2: >>Trying to acquire journal lock... >>Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: jid=2: >>Looking at journal... >>Jan 16 22:12:28 sf-03 kernel: GFS: fsid=gfs_cluster:gfs_htdocs.2: jid=2: >>Done >>Jan 16 22:12:28 sf-03 kernel: GFS: Trying to join cluster "lock_dlm", >>"gfs_cluster:lv_webcalle20" >>Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: >>Joined cluster. Now mounting FS... >>Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: >>jid=2: Trying to acquire journal lock... >>Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: >>jid=2: Looking at journal... >>Jan 16 22:12:30 sf-03 kernel: GFS: fsid=gfs_cluster:lv_webcalle20.2: >>jid=2: Done >>Jan 16 22:12:30 sf-03 gfs: Mounting GFS filesystems: succeeded >> >> At the end we saw a gfs_tool option interesting "lockdump" and >>before reboot ran it. At the end there are some output of my mountpoint >> If you need more information about configuration or something please >>say me. >> >> gfs_tool lockdump /mountpoint: >> >>Glock (5, 2114727) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (2, 119206) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 119206/119206 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (2, 228692) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 228692/228692 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (5, 735842) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 >>Glock (5, 1418402) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (2, 1885491) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 1885491/1885491 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (2, 399729) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 399729/399729 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (5, 1386646) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (5, 241672) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (5, 207713) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (2, 946688) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 946688/946688 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (5, 30184) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (2, 8340) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 8340/8340 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (2, 2308745) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 2308745/2308745 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (2, 949390) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 949390/949390 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (5, 548987) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (2, 1437881) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 1437881/1437881 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (5, 139108) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (2, 261765) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 261765/261765 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (2, 2530374) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 2530374/2530374 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (5, 33091) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (2, 863848) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 863848/863848 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (5, 208549) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (2, 51708) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 51708/51708 >> type = 2 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (5, 1887878) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (5, 864369) >> gl_flags = >> gl_count = 2 >> gl_state = 3 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = no >> ail_bufs = no >> Holder >> owner = -1 >> gh_state = 3 >> gh_flags = 5 7 >> error = 0 >> gh_iflags = 1 6 7 >>Glock (2, 1436746) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 1436746/1436746 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (2, 608211) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 608211/608211 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (2, 609430) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 609430/609430 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >>Glock (2, 893532) >> gl_flags = >> gl_count = 2 >> gl_state = 0 >> req_gh = no >> req_bh = no >> lvb_count = 0 >> object = yes >> new_le = no >> incore_le = no >> reclaim = no >> aspace = 0 >> ail_bufs = no >> Inode: >> num = 893532/893532 >> type = 1 >> i_count = 1 >> i_flags = >> vnode = yes >> >>Thanks in advance. >> >> > > > -- ________________________________________ Bartolom? Rodr?guez Bordallo Departamento de Explotaci?n de Servicios FUJITSU ESPA?A SERVICES, S.A.U. Camino Cerro de los Gamos, 1 28224 Pozuelo de Alarc?n, Madrid Tel.: 902 11 40 10 Mail: brodriguezb at fujitsu.es ________________________________________ La informaci?n contenida en este e-mail es confidencial y va dirigida ?nicamente al receptor que aparece como destinatario. Si ha recibido este e-mail por error, por favor, notif?quenoslo inmediatamente y b?rrelo de su sistema. Por favor, en tal caso, no lo copie ni lo use para ning?n prop?sito, ni revele sus contenidos a ninguna persona ni lo almacene ni copie esta informaci?n en ning?n medio. From forgue at oakland.edu Wed Jan 18 17:18:46 2006 From: forgue at oakland.edu (Andrew J. Forgue) Date: Wed, 18 Jan 2006 12:18:46 -0500 Subject: [Linux-cluster] DLM error Message-ID: <43CE7876.1000408@oakland.edu> I have a 2-node cluster (briscoe & mccoy) running RHEL4U2 with associated CVS sources that I compiled yesterday (from that branch) and I'm getting an error when starting CLVMd: "DLM: connect from non cluster node" Everything else (cman, ccs, fenced) starts correctly. We have had this cluster up for a while and we put it behind a load balancer (Cisco CSS11500 or something) yesterday and changed the IP to rfc1918 addresses while having the old public IPs directly mapped to the private IPs. The DNS for the hostnames, however, resolve to the public IP so I added the private IPs into /etc/hosts: 10.0.3.10 briscoe briscoe.sys.oakland.edu 10.0.3.11 mccoy mccoy.sys.oakland.edu and their external IPs are: 141.210.8.xxx I've done a tcpdump between the nodes and the join source address is indeed the 10.0.3.11, but I get the error on whichever node is booted last. I didn't see a bug in bugzilla, so if this isn't my problem (which I think it is), I'll file a bug in bugzilla. But what I'm thinking is that for some reason it thinks that the other node is a 141.210 address, while the cluster resolves the node name to a 10.0.3.x address, causing this. Can anyone give me a tip or point me into the right direction? Thanks, Andrew -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 250 bytes Desc: OpenPGP digital signature URL: From teigland at redhat.com Wed Jan 18 17:39:21 2006 From: teigland at redhat.com (David Teigland) Date: Wed, 18 Jan 2006 11:39:21 -0600 Subject: [Linux-cluster] DLM error In-Reply-To: <43CE7876.1000408@oakland.edu> References: <43CE7876.1000408@oakland.edu> Message-ID: <20060118173921.GB8315@redhat.com> On Wed, Jan 18, 2006 at 12:18:46PM -0500, Andrew J. Forgue wrote: > I have a 2-node cluster (briscoe & mccoy) running RHEL4U2 with > associated CVS sources that I compiled yesterday (from that branch) and > I'm getting an error when starting CLVMd: > > "DLM: connect from non cluster node" Whatever IP the names in cluster.conf resolve to is what cman+dlm try to use. What does cman_tool nodes and status show? Dave From eric at bootseg.com Wed Jan 18 17:39:08 2006 From: eric at bootseg.com (Eric Kerin) Date: Wed, 18 Jan 2006 12:39:08 -0500 Subject: [Linux-cluster] DLM error In-Reply-To: <43CE7876.1000408@oakland.edu> References: <43CE7876.1000408@oakland.edu> Message-ID: <1137605949.4405.12.camel@auh5-0479.corp.jabil.org> On Wed, 2006-01-18 at 12:18 -0500, Andrew J. Forgue wrote: > I have a 2-node cluster (briscoe & mccoy) running RHEL4U2 with > associated CVS sources that I compiled yesterday (from that branch) and > I'm getting an error when starting CLVMd: > > "DLM: connect from non cluster node" > > Everything else (cman, ccs, fenced) starts correctly. We have had this > cluster up for a while and we put it behind a load balancer (Cisco > CSS11500 or something) yesterday and changed the IP to rfc1918 addresses > while having the old public IPs directly mapped to the private IPs. The > DNS for the hostnames, however, resolve to the public IP so I added the > private IPs into /etc/hosts: > > 10.0.3.10 briscoe briscoe.sys.oakland.edu > 10.0.3.11 mccoy mccoy.sys.oakland.edu > > and their external IPs are: 141.210.8.xxx > Can anyone give me a tip or point me into the right direction? > Run "cman_tool status" and make sure the node address it lists are what you expect to see. Also, check that the name used for the node in cluster.conf resolves to the IP address that it shows in "cman_tool status" for that node. Thanks, Eric Kerin eric at bootseg.com From jean-francois.larvoire at hp.com Wed Jan 18 18:56:01 2006 From: jean-francois.larvoire at hp.com (Larvoire, Jean-Francois) Date: Wed, 18 Jan 2006 19:56:01 +0100 Subject: [Linux-cluster] Two nodes don't see each other Message-ID: <6E74A51363EBE84381CD11B13AAF44A002204794@idaexc02.emea.cpqcorp.net> Hello, I'm experimenting with a 2-nodes cluster, based on RHEL4. The goal is to manage a service failover, I don't need GFS for now. I think the following packages are sufficient: ccs Manage the cluster configuration file cman Symmetric Cluster Manager cman-kernel Back end behind cman dlm Distributed Lock Manager dlm-kernel Back end behind dlm fence Power control magma cluster-abstraction library magma-plugins Maybe we need the sm plugin? Is this correct? Anyway I've built and installed these 1.01 packages on the two nodes. But when I start the cluster, both nodes seem to think they're the only node in town. Any clue welcome! # Startup procedure I used on both nodes: depmod -a modprobe cman modprobe dlm ccsd cman_tool join # What node 1 sees: [root at eiros3 ~]# cman_tool status Protocol version: 5.0.1 Config version: 2 Cluster name: eiros Cluster ID: 3249 Cluster Member: Yes Membership state: Cluster-Member Nodes: 1 Expected_votes: 1 Total_votes: 1 Quorum: 1 Active subsystems: 0 Node name: eiros3 Node addresses: 192.168.8.73 [root at eiros3 ~]# cman_tool nodes Node Votes Exp Sts Name 1 1 1 M eiros3 # What node 2 sees: [root at eiros4 ~]# cman_tool status Protocol version: 5.0.1 Config version: 2 Cluster name: eiros Cluster ID: 3249 Cluster Member: Yes Membership state: Cluster-Member Nodes: 1 Expected_votes: 1 Total_votes: 1 Quorum: 1 Active subsystems: 0 Node name: eiros4 Node addresses: 192.168.8.74 [root at eiros4 ~]# cman_tool nodes Node Votes Exp Sts Name 2 1 1 M eiros4 # Here's the cluster.conf file: ======================================================================== = Jean-Fran?ois Larvoire ========= _/ =========== = Hewlett-Packard ======= _/ ======= = 5 Avenue Raymond Chanas, Eybens ===== _/_/_/ _/_/_/ ===== = 38053 Grenoble Cedex 9, FRANCE ===== _/ _/ _/ _/ ===== = Phone: +33 476 14 13 38 ===== _/ _/ _/_/_/ ===== = Fax: +33 476 14 45 19 ======= _/ ======= = Email: jean-francois.larvoire at hp.com ========== _/ ========== ======================================================================== From forgue at oakland.edu Wed Jan 18 20:25:23 2006 From: forgue at oakland.edu (Andrew J. Forgue) Date: Wed, 18 Jan 2006 15:25:23 -0500 Subject: [Linux-cluster] DLM error In-Reply-To: <20060118173921.GB8315@redhat.com> References: <43CE7876.1000408@oakland.edu> <20060118173921.GB8315@redhat.com> Message-ID: <43CEA433.5040200@oakland.edu> David Teigland wrote: > On Wed, Jan 18, 2006 at 12:18:46PM -0500, Andrew J. Forgue wrote: >> I have a 2-node cluster (briscoe & mccoy) running RHEL4U2 with >> associated CVS sources that I compiled yesterday (from that branch) and >> I'm getting an error when starting CLVMd: >> >> "DLM: connect from non cluster node" > > Whatever IP the names in cluster.conf resolve to is what cman+dlm try to > use. What does cman_tool nodes and status show? > > Dave Thanks Dave, We created new hostnames for both machines (briscoe-i & mccoy-i) that resolve to their proper internal ip addresses. However, clvmd still gets that message and then goes into "Ds" state. Here's the output of cman_tool status from both nodes: --- [root at briscoe-i ~]# cman_tool nodes Node Votes Exp Sts Name 1 1 1 M mccoy-i 2 1 1 M briscoe-i --- [root at mccoy-i ~]# cman_tool status Protocol version: 5.0.1 Config version: 20 Cluster name: appsvr Cluster ID: 6602 Cluster Member: Yes Membership state: Cluster-Member Nodes: 2 Expected_votes: 1 Total_votes: 2 Quorum: 1 Active subsystems: 3 Node name: mccoy-i Node addresses: 10.0.3.11 --- [root at briscoe-i ~]# cman_tool status Protocol version: 5.0.1 Config version: 20 Cluster name: appsvr Cluster ID: 6602 Cluster Member: Yes Membership state: Cluster-Member Nodes: 2 Expected_votes: 1 Total_votes: 2 Quorum: 1 Active subsystems: 23 Node name: briscoe-i Node addresses: 10.0.3.10 --- --- DNS entries --- [root at briscoe-i ~]# host briscoe-i briscoe-i.sys.oakland.edu has address 10.0.3.10 [root at briscoe-i ~]# host mccoy-i mccoy-i.sys.oakland.edu has address 10.0.3.11 --- ... Some RGmanager stuff ... --- --- dmesg output --- ... CMAN: Waiting to join or form a Linux-cluster CMAN: forming a new cluster CMAN: quorum regained, resuming activity DLM (built Jan 17 2006 14:10:23) installed dlm: connect from non cluster node So now I'm at a loss as the IP addresses match their hostnames in our DNS system now. Andrew -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 250 bytes Desc: OpenPGP digital signature URL: From dist-list at LEXUM.UMontreal.CA Wed Jan 18 20:33:54 2006 From: dist-list at LEXUM.UMontreal.CA (FM) Date: Wed, 18 Jan 2006 15:33:54 -0500 Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? Message-ID: <43CEA632.7050107@lexum.umontreal.ca> Hello everybody, Is redhat cluster suite (RHEL 4 ) a good candidate for a webfarm ? My setup would be : several servers (1U AMd dual core) connected to iscsi storage array. Is Iscsi a good choice (SAN prices are too high for us) for hardware ? Our network is GB. We will have SCSI 10KRPM + read and write cache on the SCSI card + RAID5 thanks ! From mwill at penguincomputing.com Wed Jan 18 20:56:35 2006 From: mwill at penguincomputing.com (Michael Will) Date: Wed, 18 Jan 2006 12:56:35 -0800 Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? Message-ID: <9D61626573F86843908C9AE8BDB4390551243C@jellyfish.highlyscyld.com> I am surprised you use SCSI drives on the storage if you are price sensitive, usually SATA is the better bang for the buck unless you are doing databases with lots of small read and writes. Michael -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of FM Sent: Wednesday, January 18, 2006 12:34 PM To: Redhat Cluster Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? Hello everybody, Is redhat cluster suite (RHEL 4 ) a good candidate for a webfarm ? My setup would be : several servers (1U AMd dual core) connected to iscsi storage array. Is Iscsi a good choice (SAN prices are too high for us) for hardware ? Our network is GB. We will have SCSI 10KRPM + read and write cache on the SCSI card + RAID5 thanks ! -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From dist-list at LEXUM.UMontreal.CA Wed Jan 18 21:01:21 2006 From: dist-list at LEXUM.UMontreal.CA (FM) Date: Wed, 18 Jan 2006 16:01:21 -0500 Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? In-Reply-To: <9D61626573F86843908C9AE8BDB4390551243C@jellyfish.highlyscyld.com> References: <9D61626573F86843908C9AE8BDB4390551243C@jellyfish.highlyscyld.com> Message-ID: <43CEACA1.8000209@lexum.umontreal.ca> Thanks for the reply, I read about SATA storage but we sync from lan to dmz, so there is lots of r/w. Michael Will wrote: >I am surprised you use SCSI drives on the storage if >you are price sensitive, usually SATA is the better >bang for the buck unless you are doing databases with >lots of small read and writes. > >Michael > >-----Original Message----- >From: linux-cluster-bounces at redhat.com >[mailto:linux-cluster-bounces at redhat.com] On Behalf Of FM >Sent: Wednesday, January 18, 2006 12:34 PM >To: Redhat Cluster >Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? > >Hello everybody, > >Is redhat cluster suite (RHEL 4 ) a good candidate for a webfarm ? >My setup would be : several servers (1U AMd dual core) connected to >iscsi storage array. > >Is Iscsi a good choice (SAN prices are too high for us) for hardware ? >Our network is GB. >We will have SCSI 10KRPM + read and write cache on the SCSI card + RAID5 >thanks ! > > > > > > > > > > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > >-- >Linux-cluster mailing list >Linux-cluster at redhat.com >https://www.redhat.com/mailman/listinfo/linux-cluster > > From eric at bootseg.com Wed Jan 18 21:27:34 2006 From: eric at bootseg.com (Eric Kerin) Date: Wed, 18 Jan 2006 16:27:34 -0500 Subject: [Linux-cluster] Two nodes don't see each other In-Reply-To: <6E74A51363EBE84381CD11B13AAF44A002204794@idaexc02.emea.cpqcorp.net> References: <6E74A51363EBE84381CD11B13AAF44A002204794@idaexc02.emea.cpqcorp.net> Message-ID: <1137619655.4405.16.camel@auh5-0479.corp.jabil.org> On Wed, 2006-01-18 at 19:56 +0100, Larvoire, Jean-Francois wrote: > Hello, > > I'm experimenting with a 2-nodes cluster, based on RHEL4. > The goal is to manage a service failover, I don't need GFS for now. > I think the following packages are sufficient: > > Anyway I've built and installed these 1.01 packages on the two nodes. > But when I start the cluster, both nodes seem to think they're the only node in town. > Any clue welcome! > Check to see if the communication is being blocked by any of your iptables firewall rules. Thanks, Eric Kerin eric at bootseg.co From eric at bootseg.com Wed Jan 18 21:27:34 2006 From: eric at bootseg.com (Eric Kerin) Date: Wed, 18 Jan 2006 16:27:34 -0500 Subject: [Linux-cluster] Two nodes don't see each other In-Reply-To: <6E74A51363EBE84381CD11B13AAF44A002204794@idaexc02.emea.cpqcorp.net> References: <6E74A51363EBE84381CD11B13AAF44A002204794@idaexc02.emea.cpqcorp.net> Message-ID: <1137619655.4405.16.camel@auh5-0479.corp.jabil.org> On Wed, 2006-01-18 at 19:56 +0100, Larvoire, Jean-Francois wrote: > Hello, > > I'm experimenting with a 2-nodes cluster, based on RHEL4. > The goal is to manage a service failover, I don't need GFS for now. > I think the following packages are sufficient: > > Anyway I've built and installed these 1.01 packages on the two nodes. > But when I start the cluster, both nodes seem to think they're the only node in town. > Any clue welcome! > Check to see if the communication is being blocked by any of your iptables firewall rules. Thanks, Eric Kerin eric at bootseg.co From forgue at oakland.edu Thu Jan 19 00:53:57 2006 From: forgue at oakland.edu (Andrew J. Forgue) Date: Wed, 18 Jan 2006 19:53:57 -0500 Subject: [Linux-cluster] DLM error In-Reply-To: <43CEA433.5040200@oakland.edu> References: <43CE7876.1000408@oakland.edu> <20060118173921.GB8315@redhat.com> <43CEA433.5040200@oakland.edu> Message-ID: <43CEE325.5050908@oakland.edu> Andrew J. Forgue wrote: > David Teigland wrote: > >> On Wed, Jan 18, 2006 at 12:18:46PM -0500, Andrew J. Forgue wrote: >> >>> I have a 2-node cluster (briscoe & mccoy) running RHEL4U2 with >>> associated CVS sources that I compiled yesterday (from that branch) and >>> I'm getting an error when starting CLVMd: >>> >>> "DLM: connect from non cluster node" >>> >> Whatever IP the names in cluster.conf resolve to is what cman+dlm try to >> use. What does cman_tool nodes and status show? >> >> Dave >> > > Thanks Dave, > > We created new hostnames for both machines (briscoe-i & mccoy-i) that > resolve to their proper internal ip addresses. However, clvmd still > gets that message and then goes into "Ds" state. Here's the output of > cman_tool status from both nodes: > > --- > [root at briscoe-i ~]# cman_tool nodes > Node Votes Exp Sts Name > 1 1 1 M mccoy-i > 2 1 1 M briscoe-i > --- > [root at mccoy-i ~]# cman_tool status > Protocol version: 5.0.1 > Config version: 20 > Cluster name: appsvr > Cluster ID: 6602 > Cluster Member: Yes > Membership state: Cluster-Member > Nodes: 2 > Expected_votes: 1 > Total_votes: 2 > Quorum: 1 > Active subsystems: 3 > Node name: mccoy-i > Node addresses: 10.0.3.11 > --- > [root at briscoe-i ~]# cman_tool status > Protocol version: 5.0.1 > Config version: 20 > Cluster name: appsvr > Cluster ID: 6602 > Cluster Member: Yes > Membership state: Cluster-Member > Nodes: 2 > Expected_votes: 1 > Total_votes: 2 > Quorum: 1 > Active subsystems: 23 > Node name: briscoe-i > Node addresses: 10.0.3.10 > --- > --- DNS entries > --- > [root at briscoe-i ~]# host briscoe-i > briscoe-i.sys.oakland.edu has address 10.0.3.10 > [root at briscoe-i ~]# host mccoy-i > mccoy-i.sys.oakland.edu has address 10.0.3.11 > --- > > > > > > > > > > > > > > > > > > > > > > > ... Some RGmanager stuff ... > > --- > --- dmesg output > --- > ... > CMAN: Waiting to join or form a Linux-cluster > CMAN: forming a new cluster > CMAN: quorum regained, resuming activity > DLM (built Jan 17 2006 14:10:23) installed > dlm: connect from non cluster node > > So now I'm at a loss as the IP addresses match their hostnames in our > DNS system now. > > Andrew Hi Guys, We can ignore this, we figured it out. The Load balancer was rewriting the source IP address even when both nodes were on the same segment. Thanks for the help about checking IP addresses! Andrew -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 250 bytes Desc: OpenPGP digital signature URL: From grimme at atix.de Thu Jan 19 08:14:20 2006 From: grimme at atix.de (Marc Grimme) Date: Thu, 19 Jan 2006 09:14:20 +0100 Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? In-Reply-To: <43CEACA1.8000209@lexum.umontreal.ca> References: <9D61626573F86843908C9AE8BDB4390551243C@jellyfish.highlyscyld.com> <43CEACA1.8000209@lexum.umontreal.ca> Message-ID: <200601190914.20193.grimme@atix.de> Hello, I think the best way to tell what storage or infrastructure would be the best is to know more about your current setup and what issues with that you want to get ahead of. For example: if you really think about using iscsi, I don't think that SCSI or SATA drives make a big difference - depending on how many drives you use. But if all webservers currently have locally attached disk drives you want scale too linar with exchanging and IDE/parallel-SCSI Bus with an network topology using Ethernet. But my opinion is: if you have a lot of I/Os make yourself mostly independent from the latency of an ethernet and rethink about using Fibre-Channel with GFS. But the best advices could be made if you make your current setup and the things you want to achieve more clearly. Regards Marc. On Wednesday 18 January 2006 22:01, FM wrote: > Thanks for the reply, > I read about SATA storage but we sync from lan to dmz, so there is lots > of r/w. > > Michael Will wrote: > >I am surprised you use SCSI drives on the storage if > >you are price sensitive, usually SATA is the better > >bang for the buck unless you are doing databases with > >lots of small read and writes. > > > >Michael > > > >-----Original Message----- > >From: linux-cluster-bounces at redhat.com > >[mailto:linux-cluster-bounces at redhat.com] On Behalf Of FM > >Sent: Wednesday, January 18, 2006 12:34 PM > >To: Redhat Cluster > >Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? > > > >Hello everybody, > > > >Is redhat cluster suite (RHEL 4 ) a good candidate for a webfarm ? > >My setup would be : several servers (1U AMd dual core) connected to > >iscsi storage array. > > > >Is Iscsi a good choice (SAN prices are too high for us) for hardware ? > >Our network is GB. > >We will have SCSI 10KRPM + read and write cache on the SCSI card + RAID5 > >thanks ! > > > > > > > > > > > > > > > > > > > > > >-- > >Linux-cluster mailing list > >Linux-cluster at redhat.com > >https://www.redhat.com/mailman/listinfo/linux-cluster > > > >-- > >Linux-cluster mailing list > >Linux-cluster at redhat.com > >https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Gruss / Regards, Marc Grimme Phone: +49-89 121 409-54 http://www.atix.de/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From damian.osullivan at hp.com Thu Jan 19 09:25:38 2006 From: damian.osullivan at hp.com (O'Sullivan, Damian) Date: Thu, 19 Jan 2006 09:25:38 -0000 Subject: [Linux-cluster] Two nodes don't see each other Message-ID: <644A0966265D9D40AC7584FCE95611130216F049@dubexc01.emea.cpqcorp.net> JF, I used this one for the new panadol and it works fine : My cluter-start scripts and modules : service ccsd start service cman start service fenced start service clvmd start service gfs start service rgmanager start [root at panadola samba]# lsmod Module Size Used by lock_dlm 46196 2 gfs 321548 2 lock_harness 6960 2 lock_dlm,gfs nfsd 267105 9 exportfs 7745 1 nfsd lockd 77809 2 nfsd parport_pc 29185 0 lp 15089 0 parport 43981 2 parport_pc,lp autofs4 23241 0 i2c_dev 13633 0 i2c_core 28481 1 i2c_dev dlm 130180 9 lock_dlm cman 136480 19 lock_dlm,dlm md5 5697 1 ipv6 282657 28 sunrpc 170425 12 nfsd,lockd button 9057 0 battery 11209 0 ac 6729 0 uhci_hcd 34665 0 ehci_hcd 33349 0 hw_random 7137 0 tg3 91717 0 e1000 110381 0 floppy 65809 0 dm_snapshot 18561 0 dm_zero 3649 0 dm_mirror 28889 0 ext3 137681 3 jbd 68849 1 ext3 dm_mod 66433 9 dm_snapshot,dm_zero,dm_mirror cciss 59017 5 sd_mod 19392 0 scsi_mod 140177 2 cciss,sd_mod > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of > Larvoire, Jean-Francois > Sent: 18 January 2006 18:56 > To: linux-cluster at redhat.com > Subject: [Linux-cluster] Two nodes don't see each other > > Hello, > > I'm experimenting with a 2-nodes cluster, based on RHEL4. > The goal is to manage a service failover, I don't need GFS for now. > I think the following packages are sufficient: > > ccs Manage the cluster configuration file > cman Symmetric Cluster Manager > cman-kernel Back end behind cman > dlm Distributed Lock Manager > dlm-kernel Back end behind dlm > fence Power control > magma cluster-abstraction library > magma-plugins Maybe we need the sm plugin? > > Is this correct? > > Anyway I've built and installed these 1.01 packages on the two nodes. > But when I start the cluster, both nodes seem to think > they're the only node in town. > Any clue welcome! > > > # Startup procedure I used on both nodes: > depmod -a > modprobe cman > modprobe dlm > ccsd > cman_tool join > > # What node 1 sees: > [root at eiros3 ~]# cman_tool status > Protocol version: 5.0.1 > Config version: 2 > Cluster name: eiros > Cluster ID: 3249 > Cluster Member: Yes > Membership state: Cluster-Member > Nodes: 1 > Expected_votes: 1 > Total_votes: 1 > Quorum: 1 > Active subsystems: 0 > Node name: eiros3 > Node addresses: 192.168.8.73 > [root at eiros3 ~]# cman_tool nodes > Node Votes Exp Sts Name > 1 1 1 M eiros3 > > # What node 2 sees: > [root at eiros4 ~]# cman_tool status > Protocol version: 5.0.1 > Config version: 2 > Cluster name: eiros > Cluster ID: 3249 > Cluster Member: Yes > Membership state: Cluster-Member > Nodes: 1 > Expected_votes: 1 > Total_votes: 1 > Quorum: 1 > Active subsystems: 0 > Node name: eiros4 > Node addresses: 192.168.8.74 > [root at eiros4 ~]# cman_tool nodes > Node Votes Exp Sts Name > 2 1 1 M eiros4 > > # Here's the cluster.conf file: > > > > hostname="eiros3-mp" login="login" passwd="passwd"/> > hostname="eiros4-mp" login="login" passwd="passwd"/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > two_node="1" expected_votes="1"> > > > > ============================================================== > ========== > = Jean-Fran?ois Larvoire ========= _/ > =========== > = Hewlett-Packard ======= _/ > ======= > = 5 Avenue Raymond Chanas, Eybens ===== _/_/_/ > _/_/_/ ===== > = 38053 Grenoble Cedex 9, FRANCE ===== _/ _/ _/ > _/ ===== > = Phone: +33 476 14 13 38 ===== _/ _/ > _/_/_/ ===== > = Fax: +33 476 14 45 19 ======= _/ > ======= > = Email: jean-francois.larvoire at hp.com ========== _/ > ========== > ============================================================== > ========== > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From thaidn at gmail.com Thu Jan 19 09:43:34 2006 From: thaidn at gmail.com (Thai Duong) Date: Thu, 19 Jan 2006 16:43:34 +0700 Subject: [Linux-cluster] HP iLO fencing - how many iLO variants are there? In-Reply-To: <200601180932.16510.hlawatschek@atix.de> References: <20060117235244.532486c9.pegasus@nerv.eu.org> <200601180932.16510.hlawatschek@atix.de> Message-ID: I have the same problem. I have a bunch of HP servers ranging from ML370 to RX4640 that each of them has a ILO card bundled but I could not use fence_ilo or fence_rib. On 1/18/06, Mark Hlawatschek wrote: > > Hi Jure, > > in my opinion the Lights Out 100i Remote Management" are IPMI interfaces. > You > could give fence_ipmilan a try. > > Mark > > On Tuesday 17 January 2006 23:52, Jure Pe?ar wrote: > > Hi all, > > > > I have two DL145 G1 servers with "Lights Out 100i Remote Management". I > > thought this is iLO but am not so sure anymore. > > > > The thing is accessible via telnet or http and works as expected. So I > was > > kinda suprised when fence_ilo failed with no route to host at > > blahblahblah/SSL.pm line 104. Sure, it wants to connect to port 443 ... > So > > I took a look at fence_ilo code and saw nothing that would talk telnet > to > > this 100i thing ... > > > > I also checked its firmware version. It says 2.00, latest is 2.01 with > no > > important updates ... > > > > So it looks like I'd have to write my own fencing agent. Are there any > > guidelines on how to do it "properly" to have it included in the > mainline > > distro? Also, any docs/mailing list post about what parameters fencing > > agent needs to implement? > > -- > Gruss / Regards, > > Dipl.-Ing. Mark Hlawatschek > Phone: +49-89 121 409-55 > http://www.atix.de/ > > ** > ATIX - Ges. fuer Informationstechnologie und Consulting mbH > Einsteinstr. 10 - 85716 Unterschleissheim - Germany > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anderson at centtech.com Thu Jan 19 12:32:51 2006 From: anderson at centtech.com (Eric Anderson) Date: Thu, 19 Jan 2006 06:32:51 -0600 Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? In-Reply-To: <200601190914.20193.grimme@atix.de> References: <9D61626573F86843908C9AE8BDB4390551243C@jellyfish.highlyscyld.com> <43CEACA1.8000209@lexum.umontreal.ca> <200601190914.20193.grimme@atix.de> Message-ID: <43CF86F3.5040501@centtech.com> Marc Grimme wrote: > Hello, > I think the best way to tell what storage or infrastructure would be the best > is to know more about your current setup and what issues with that you want > to get ahead of. > For example: if you really think about using iscsi, I don't think that SCSI or > SATA drives make a big difference - depending on how many drives you use. But > if all webservers currently have locally attached disk drives you want scale > too linar with exchanging and IDE/parallel-SCSI Bus with an network topology > using Ethernet. > But my opinion is: if you have a lot of I/Os make yourself mostly independent > from the latency of an ethernet and rethink about using Fibre-Channel with > GFS. > Honestly, ethernet latencies (especially on gigabit ethernet) are lower than fiber channel latencies, so this statement doesn't really hold up. If you want very fast speeds, get an iSCSI array, populate it with 15k RPM scsi disks with big caches, max the cache out on the array, and set it up for a RAID0+1 (or RAID10 depending on the implementor). If you want fast speed, but not a big price, you can notch down to 10K scsi disks, or use 15k scsi disks and a raid 5, etc, and keep notching down until it fits your budget and needs. I agree here though that we really need to know a few things: - what kind of traffic is this? - size of the files most commonly used - total data size (how much space you need) - budget - demands (availability/performance/etc) Eric > But the best advices could be made if you make your current setup and the > things you want to achieve more clearly. > > Regards Marc. > > On Wednesday 18 January 2006 22:01, FM wrote: > >> Thanks for the reply, >> I read about SATA storage but we sync from lan to dmz, so there is lots >> of r/w. >> >> Michael Will wrote: >> >>> I am surprised you use SCSI drives on the storage if >>> you are price sensitive, usually SATA is the better >>> bang for the buck unless you are doing databases with >>> lots of small read and writes. >>> >>> Michael >>> >>> -----Original Message----- >>> From: linux-cluster-bounces at redhat.com >>> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of FM >>> Sent: Wednesday, January 18, 2006 12:34 PM >>> To: Redhat Cluster >>> Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? >>> >>> Hello everybody, >>> >>> Is redhat cluster suite (RHEL 4 ) a good candidate for a webfarm ? >>> My setup would be : several servers (1U AMd dual core) connected to >>> iscsi storage array. >>> >>> Is Iscsi a good choice (SAN prices are too high for us) for hardware ? >>> Our network is GB. >>> We will have SCSI 10KRPM + read and write cache on the SCSI card + RAID5 >>> thanks ! >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------ From omer at faruk.net Thu Jan 19 14:48:10 2006 From: omer at faruk.net (Omer Faruk Sen) Date: Thu, 19 Jan 2006 16:48:10 +0200 (EET) Subject: [Linux-cluster] redhat gfs evaluation subscription? Message-ID: <53921.193.140.74.2.1137682090.squirrel@193.140.74.2> Hi, Is it possible to download and evaluate Redhat GFS as a subscription evaluation? I want to implement it before investing money.. Best Regards. -- Omer Faruk Sen http://www.faruk.net From carlopmart at gmail.com Thu Jan 19 15:28:24 2006 From: carlopmart at gmail.com (carlopmart) Date: Thu, 19 Jan 2006 16:28:24 +0100 Subject: [Linux-cluster] Queston about rh cluster suite 3 Message-ID: <43CFB018.8010906@gmail.com> Hi all, I have a customer who needs to deploy RHCS 3 on two nodes using TrendMicro IMSS 5.7 with RHEL 3. And i have two questions: - Which type of power swithes I need to accomplish this?? i can not found anything about it on RedHat's website. Is it posible to use serial null cable instead a power switch? - Can I do active/active configuration for postfix and IMSS services?? Somebody has test it? Thanks. -- CL Martinez carlopmart {at} gmail {d0t} com From dist-list at LEXUM.UMontreal.CA Thu Jan 19 16:43:26 2006 From: dist-list at LEXUM.UMontreal.CA (FM) Date: Thu, 19 Jan 2006 11:43:26 -0500 Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? In-Reply-To: <43CF86F3.5040501@centtech.com> References: <9D61626573F86843908C9AE8BDB4390551243C@jellyfish.highlyscyld.com> <43CEACA1.8000209@lexum.umontreal.ca> <200601190914.20193.grimme@atix.de> <43CF86F3.5040501@centtech.com> Message-ID: <43CFC1AE.4070304@lexum.umontreal.ca> First, Thank you all for the great input ! Here is more output about our sites : Some static HTML web site. Sites are generated in Lan and then rsync to DMZ. And other with mod_perl + Postgresql Database (local) One of our most visited site (static html) is using nealy 200 GB of bandwidth / month and have 545368 hists a day. For decembre 2005 stats : http://stats.lexum.umontreal.ca/awstats.pl?month=12&year=2005&output=main&config=www.canlii.org&lang=en&framename=index Now we are using dual Xeon (2 Ghz), with 2,5 GB of RAM. with RAID 5 SCSI 10KRPM Network is GB Budget is 200K CA$ Thanks again Eric Anderson wrote: > Marc Grimme wrote: > >> Hello, >> I think the best way to tell what storage or infrastructure would be >> the best is to know more about your current setup and what issues >> with that you want to get ahead of. >> For example: if you really think about using iscsi, I don't think >> that SCSI or SATA drives make a big difference - depending on how >> many drives you use. But if all webservers currently have locally >> attached disk drives you want scale too linar with exchanging and >> IDE/parallel-SCSI Bus with an network topology using Ethernet. But my >> opinion is: if you have a lot of I/Os make yourself mostly >> independent from the latency of an ethernet and rethink about using >> Fibre-Channel with GFS. >> > > > Honestly, ethernet latencies (especially on gigabit ethernet) are > lower than fiber channel latencies, so this statement doesn't really > hold up. > If you want very fast speeds, get an iSCSI array, populate it with 15k > RPM scsi disks with big caches, max the cache out on the array, and > set it up for a RAID0+1 (or RAID10 depending on the implementor). If > you want fast speed, but not a big price, you can notch down to 10K > scsi disks, or use 15k scsi disks and a raid 5, etc, and keep notching > down until it fits your budget and needs. > I agree here though that we really need to know a few things: > > - what kind of traffic is this? > - size of the files most commonly used > - total data size (how much space you need) > - budget > - demands (availability/performance/etc) > > Eric > > > > > >> But the best advices could be made if you make your current setup and >> the things you want to achieve more clearly. >> >> Regards Marc. >> >> On Wednesday 18 January 2006 22:01, FM wrote: >> >> >>> Thanks for the reply, >>> I read about SATA storage but we sync from lan to dmz, so there is lots >>> of r/w. >>> >>> Michael Will wrote: >>> >>> >>>> I am surprised you use SCSI drives on the storage if >>>> you are price sensitive, usually SATA is the better >>>> bang for the buck unless you are doing databases with >>>> lots of small read and writes. >>>> >>>> Michael >>>> >>>> -----Original Message----- >>>> From: linux-cluster-bounces at redhat.com >>>> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of FM >>>> Sent: Wednesday, January 18, 2006 12:34 PM >>>> To: Redhat Cluster >>>> Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? >>>> >>>> Hello everybody, >>>> >>>> Is redhat cluster suite (RHEL 4 ) a good candidate for a webfarm ? >>>> My setup would be : several servers (1U AMd dual core) connected to >>>> iscsi storage array. >>>> >>>> Is Iscsi a good choice (SAN prices are too high for us) for hardware ? >>>> Our network is GB. >>>> We will have SCSI 10KRPM + read and write cache on the SCSI card + >>>> RAID5 >>>> thanks ! >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Linux-cluster mailing list >>>> Linux-cluster at redhat.com >>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>> >>>> -- >>>> Linux-cluster mailing list >>>> Linux-cluster at redhat.com >>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>>> >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >> >> >> > > > From damian.osullivan at hp.com Thu Jan 19 17:07:05 2006 From: damian.osullivan at hp.com (O'Sullivan, Damian) Date: Thu, 19 Jan 2006 17:07:05 -0000 Subject: [Linux-cluster] Kernel oops on GFS umount Message-ID: <644A0966265D9D40AC7584FCE95611130216F41F@dubexc01.emea.cpqcorp.net> If I ever reboot a node on a cluster it panics on the way down: Process umount (pid: 22814, threadinfo 00000100775fe000, task 000001007fb95030) Stack: 0000000000000084 ffffffff801f5163 0000000000000058 ffffff00102fe000 00000100276d7cf0 ffffffffa0295b96 0000000000000001 ffffffffa025c8c4 0000000000000001 ffffffffa02533c3 Call Trace:{vgacon_cursor+0} {:lock_dlm:lm_d lm_unlock+21} {:gfs:gfs_lm_unlock+41} {:gfs:gfs_glo ck_drop_th+290} {:gfs:run_queue+314} {:gfs:gfs_glock_ dq+323} {:gfs:gfs_glock_dq_uninit+9} {:gfs:gf s_make_fs_ro+118} {:gfs:gfs_put_super+654} {generic_shu tdown_super+202} {:gfs:gfs_kill_sb+42} {dummy_inode_pe rmission+0} {deactivate_super+95} {sys_umount+925 } {error_exit+0} {system_call+126} Code: 0f 0b 4e a1 29 a0 ff ff ff ff 65 01 48 c7 c7 53 a1 29 a0 31 RIP {:lock_dlm:do_dlm_unlock+167} RSP <00000100775ffca8> <0>Kernel panic - not syncing: Oops Kernel : 2.6.9-22.0.1.Elsmp GFS-6.1.2-0 D. From Bowie_Bailey at BUC.com Thu Jan 19 17:44:51 2006 From: Bowie_Bailey at BUC.com (Bowie Bailey) Date: Thu, 19 Jan 2006 12:44:51 -0500 Subject: [Linux-cluster] gfs_tool df question Message-ID: <4766EEE585A6D311ADF500E018C154E30213351C@bnifex.cis.buc.com> I am trying to understand the information provided by "gfs_tool df". When I run it on my system, I get this: Type Total Used Free use% ------------------------------------------------------------------------ inodes 2697580 2697580 0 100% metadata 104195 21410 82785 21% data 290040505 23167 290017338 0% I figured out that the data row is reporting used and free space in blocks and the inode row is self-explanatory, but what is "metadata"? Do I need to be concerned that metadata is 21% used while data hasn't even hit 1% yet? -- Bowie From Darrell.Frazier at crc.army.mil Thu Jan 19 20:26:22 2006 From: Darrell.Frazier at crc.army.mil (Frazier, Darrell USA CRC (Contractor)) Date: Thu, 19 Jan 2006 14:26:22 -0600 Subject: [Linux-cluster] GFS and Oracle Clusterware Message-ID: Hello. I'm sorry to post this here but after quite a bit of research, no one using Oracle can give me an answer I am hoping someone on this list may have gone through this and give me a pointer. I have a 2 node cluster running RHEL4, RHCS and GFS with shared storage being an iscsi jbod. Since I am unable to use OCFS2 I was hoping GFS would provide the cluster filesystem. My problem is this: Oracle Clusterware installs fine until I need to run the root.sh in which case I get the following: [root at opt2 crs]# ./root.sh WARNING: directory '/u02/app/oracle/product/10.2.0' is not owned by root WARNING: directory '/u02/app/oracle/product' is not owned by root WARNING: directory '/u02/app/oracle' is not owned by root WARNING: directory '/u02/app' is not owned by root WARNING: directory '/u02' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up NS directories PROT-1: Failed to initialize ocrconfig Failed to upgrade Oracle Cluster Registry configuration The GFS partitions are mounted as so: mount -O datavolume,nointr -t gfs /dev/vg00/vg00 /u04 (mounted this way for the OCR and Voting Disk file directories) Again, sorry to post this here, I am about at my wits end. Thanx in advance! Darrell J. Frazier Unix System Administrator US Army Combat Readiness Center Fort Rucker, Alabama 36362 Email: darrell.frazier at crc.army.mil -------------- next part -------------- An HTML attachment was scrubbed... URL: From grimme at atix.de Fri Jan 20 08:02:57 2006 From: grimme at atix.de (Marc Grimme) Date: Fri, 20 Jan 2006 09:02:57 +0100 Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? In-Reply-To: <43CFC1AE.4070304@lexum.umontreal.ca> References: <9D61626573F86843908C9AE8BDB4390551243C@jellyfish.highlyscyld.com> <43CF86F3.5040501@centtech.com> <43CFC1AE.4070304@lexum.umontreal.ca> Message-ID: <200601200902.57331.grimme@atix.de> On Thursday 19 January 2006 17:43, FM wrote: > First, Thank you all for the great input ! > Here is more output about our sites : > Some static HTML web site. Sites are generated in Lan and then rsync to > DMZ. And other with mod_perl + Postgresql Database (local) Don't use postgres clustered(parallel with multiple writer nodes, HA should not be a problem) on gfs. That is not supposed to work. As far as I know. You would need a dbms that supports parallel clustering itself. Like i.e. Oracle 9iRAC. > One of our most visited site (static html) is using nealy 200 GB of > bandwidth / month > and have 545368 hists a day. > For decembre 2005 stats : > http://stats.lexum.umontreal.ca/awstats.pl?month=12&year=2005&output=main&c >onfig=www.canlii.org&lang=en&framename=index > > Now we are using dual Xeon (2 Ghz), with 2,5 GB of RAM. with RAID 5 SCSI > 10KRPM > Network is GB > Budget is 200K CA$ Ok. You are having 500000 hit/day. One question would be how many I/Os does a hit issue. The other question is, is that the upper bound or do you expect it to grow, so that the infrastructure will have to grow in parallel. You should really be picky about the storage-infrastructure (includes storage system and locking network). Separate storage and locking network. Under normal circumstances 500000hits/day should not be a problem. But if you are right now using SCSI Disks stay with SCSI or FC-Disks they are way much faster and relyable then SATA. The more and faster Disks you use the more I/Os you will get. Besides for 200k CA$ you should also get a FC-Infrastructure and Storage. Also think about consolidating all data (even the os, share the root fs of the servers) on to the storage system. It saves you loads of management (scales within small amount of time) and disks in the servers. And you can easily replace a faulty server by pulling it out and powering a new one back on. We have made quite good experiences at webfarms with that concept. Check back a http://www.open-sharedroot.org/. We will have a howto for building up such a sharedroot within the next week. Hope that helps Regards Marc. > > Thanks again > > Eric Anderson wrote: > > Marc Grimme wrote: > >> Hello, > >> I think the best way to tell what storage or infrastructure would be > >> the best is to know more about your current setup and what issues > >> with that you want to get ahead of. > >> For example: if you really think about using iscsi, I don't think > >> that SCSI or SATA drives make a big difference - depending on how > >> many drives you use. But if all webservers currently have locally > >> attached disk drives you want scale too linar with exchanging and > >> IDE/parallel-SCSI Bus with an network topology using Ethernet. But my > >> opinion is: if you have a lot of I/Os make yourself mostly > >> independent from the latency of an ethernet and rethink about using > >> Fibre-Channel with GFS. > > > > Honestly, ethernet latencies (especially on gigabit ethernet) are > > lower than fiber channel latencies, so this statement doesn't really > > hold up. > > If you want very fast speeds, get an iSCSI array, populate it with 15k > > RPM scsi disks with big caches, max the cache out on the array, and > > set it up for a RAID0+1 (or RAID10 depending on the implementor). If > > you want fast speed, but not a big price, you can notch down to 10K > > scsi disks, or use 15k scsi disks and a raid 5, etc, and keep notching > > down until it fits your budget and needs. > > I agree here though that we really need to know a few things: > > > > - what kind of traffic is this? > > - size of the files most commonly used > > - total data size (how much space you need) > > - budget > > - demands (availability/performance/etc) > > > > Eric > > > >> But the best advices could be made if you make your current setup and > >> the things you want to achieve more clearly. > >> > >> Regards Marc. > >> > >> On Wednesday 18 January 2006 22:01, FM wrote: > >>> Thanks for the reply, > >>> I read about SATA storage but we sync from lan to dmz, so there is lots > >>> of r/w. > >>> > >>> Michael Will wrote: > >>>> I am surprised you use SCSI drives on the storage if > >>>> you are price sensitive, usually SATA is the better > >>>> bang for the buck unless you are doing databases with > >>>> lots of small read and writes. > >>>> > >>>> Michael > >>>> > >>>> -----Original Message----- > >>>> From: linux-cluster-bounces at redhat.com > >>>> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of FM > >>>> Sent: Wednesday, January 18, 2006 12:34 PM > >>>> To: Redhat Cluster > >>>> Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? > >>>> > >>>> Hello everybody, > >>>> > >>>> Is redhat cluster suite (RHEL 4 ) a good candidate for a webfarm ? > >>>> My setup would be : several servers (1U AMd dual core) connected to > >>>> iscsi storage array. > >>>> > >>>> Is Iscsi a good choice (SAN prices are too high for us) for hardware ? > >>>> Our network is GB. > >>>> We will have SCSI 10KRPM + read and write cache on the SCSI card + > >>>> RAID5 > >>>> thanks ! > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Linux-cluster mailing list > >>>> Linux-cluster at redhat.com > >>>> https://www.redhat.com/mailman/listinfo/linux-cluster > >>>> > >>>> -- > >>>> Linux-cluster mailing list > >>>> Linux-cluster at redhat.com > >>>> https://www.redhat.com/mailman/listinfo/linux-cluster > >>> > >>> -- > >>> Linux-cluster mailing list > >>> Linux-cluster at redhat.com > >>> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Gruss / Regards, Marc Grimme Phone: +49-89 121 409-54 http://www.atix.de/ ** ATIX - Ges. fuer Informationstechnologie und Consulting mbH Einsteinstr. 10 - 85716 Unterschleissheim - Germany From gforte at leopard.us.udel.edu Fri Jan 20 15:56:44 2006 From: gforte at leopard.us.udel.edu (Greg Forte) Date: Fri, 20 Jan 2006 10:56:44 -0500 Subject: [Linux-cluster] multiple issues Message-ID: <43D1083C.3040800@leopard.us.udel.edu> 1) the "easy" one first - the man page for clusvcadm lists a -l option for locking the service managers. Running clusvcadm shows this option is no longer available. The man page also references the command clushutdown, saying this is the preferred way of performing this action, but on my system I have a man page for clushutdown but no binary. So ... how does one go about doing this? 2) I was having trouble getting my services restarted - using 'clusvcadm -e httpd' (for example, where I have a service I've named httpd which sets up an IP address and starts httpd from the /etc/init.d/httpd script), it complained with the oh-so-informative message: #43: Service httpd has failed; can not start. I read somewhere that services had to be disabled and re-enabled after failure, so I tried -d instead and got the following: stop on script "httpd init script" returned 1 (generic error) #12: RG httpd failed to stop; intervention required. I finally figured out that I had to manually start the service on a node, then do clusvcadm -d, then do clusvcadm -e. Presumably the first step would not have been necessary if the httpd script didn't return an error status when you pass it stop and it's not already running. Any opinions on whether it makes sense to alter init scripts so that stop when the daemon is not running is not an error (and therefore doing clusvcadm -d on the not-running service would maybe work)? 3) the biggie: I have a GFS filesystem on a shared FC storage node (AX100). I haven't put any "real" data on it yet because I'm still testing, but yesterday I had the cluster up and running and the filesystem mounted on both nodes, and everything seemed peachy. I came back this morning to find that any attempted operations (e.g. 'ls') on the shared system came back with "Input/output error", and the following appeared in the logs: node 1: Jan 20 04:03:09 knob kernel: GFS: fsid=MAPS:shared_data.0: fatal: invalid metadata block Jan 20 04:03:09 knob kernel: GFS: fsid=MAPS:shared_data.0: bh = 352612748 (magic) Jan 20 04:03:09 knob kernel: GFS: fsid=MAPS:shared_data.0: function = gfs_rgrp_read Jan 20 04:03:09 knob kernel: GFS: fsid=MAPS:shared_data.0: file = /usr/src/build/648121-x86_64/BUILD/gfs-kernel-2.6.9-45/smp/src/gfs/rgrp.c, line = 830 Jan 20 04:03:09 knob kernel: GFS: fsid=MAPS:shared_data.0: time = 1137747789 Jan 20 04:03:09 knob kernel: GFS: fsid=MAPS:shared_data.0: about to withdraw from the cluster Jan 20 04:03:09 knob kernel: GFS: fsid=MAPS:shared_data.0: waiting for outstanding I/O Jan 20 04:03:09 knob kernel: GFS: fsid=MAPS:shared_data.0: telling LM to withdraw Jan 20 04:03:11 knob kernel: lock_dlm: withdraw abandoned memory Jan 20 04:03:11 knob kernel: GFS: fsid=MAPS:shared_data.0: withdrawn node 2: Jan 20 04:02:10 gully kernel: dlm: shared_data: process_lockqueue_reply id c0012 state 0 Jan 20 04:02:10 gully kernel: dlm: shared_data: process_lockqueue_reply id 90376 state 0 Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: jid=0: Trying to acquire journal lock... Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: jid=0: Looking at journal... Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: jid=0: Acquiring the transaction lock... Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: jid=0: Replaying journal... Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: jid=0: Replayed 0 of 0 blocks Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: jid=0: replays = 0, skips = 0, sames = 0 Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: jid=0: Journal replayed in 1s Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: jid=0: Done Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: fatal: invalid metadata block Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: bh = 352612748 (magic) Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: function = gfs_rgrp_read Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: file = /usr/src/build/648121-x86_64/BUILD/gfs-kernel-2.6.9-45/smp/src/gfs/rgrp.c, line = 830 Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: time = 1137747791 Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: about to withdraw from the cluster Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: waiting for outstanding I/O Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: telling LM to withdraw Jan 20 04:03:11 gully kernel: lock_dlm: withdraw abandoned memory Jan 20 04:03:11 gully kernel: GFS: fsid=MAPS:shared_data.1: withdrawn The time coincides with cron.daily firing, so I'm guessing the culprit is slocate (since that's the only job in cron.daily that would have touched that filesystem), but I'm not having any luck reproducing it. The only thing on that filesystem currently is the webroot, and there were no hits at the time. Any ideas? -g Greg Forte gforte at udel.edu IT - User Services University of Delaware 302-831-1982 Newark, DE From dist-list at LEXUM.UMontreal.CA Fri Jan 20 16:12:15 2006 From: dist-list at LEXUM.UMontreal.CA (FM) Date: Fri, 20 Jan 2006 11:12:15 -0500 Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? In-Reply-To: <200601200902.57331.grimme@atix.de> References: <9D61626573F86843908C9AE8BDB4390551243C@jellyfish.highlyscyld.com> <43CF86F3.5040501@centtech.com> <43CFC1AE.4070304@lexum.umontreal.ca> <200601200902.57331.grimme@atix.de> Message-ID: <43D10BDF.5040102@lexum.umontreal.ca> Thank you again for the input, We will use FC 2TB SAN (after checking budget) with 10K FC HD. Our Postgresql cluster will be active/passive (2 node connected to RAID 0+1 on one SAN group). Servers inside the webfarm will be connected to RAID 5 on another SAN group). Marc Grimme wrote: >On Thursday 19 January 2006 17:43, FM wrote: > > >>First, Thank you all for the great input ! >>Here is more output about our sites : >>Some static HTML web site. Sites are generated in Lan and then rsync to >>DMZ. And other with mod_perl + Postgresql Database (local) >> >> >Don't use postgres clustered(parallel with multiple writer nodes, HA should >not be a problem) on gfs. That is not supposed to work. As far as I know. You >would need a dbms that supports parallel clustering itself. Like i.e. Oracle >9iRAC. > > >>One of our most visited site (static html) is using nealy 200 GB of >>bandwidth / month >>and have 545368 hists a day. >>For decembre 2005 stats : >>http://stats.lexum.umontreal.ca/awstats.pl?month=12&year=2005&output=main&c >>onfig=www.canlii.org&lang=en&framename=index >> >>Now we are using dual Xeon (2 Ghz), with 2,5 GB of RAM. with RAID 5 SCSI >>10KRPM >>Network is GB >>Budget is 200K CA$ >> >> >Ok. You are having 500000 hit/day. One question would be how many I/Os does a >hit issue. The other question is, is that the upper bound or do you expect it >to grow, so that the infrastructure will have to grow in parallel. You should >really be picky about the storage-infrastructure (includes storage system and >locking network). Separate storage and locking network. Under normal >circumstances 500000hits/day should not be a problem. But if you are right >now using SCSI Disks stay with SCSI or FC-Disks they are way much faster and >relyable then SATA. The more and faster Disks you use the more I/Os you will >get. Besides for 200k CA$ you should also get a FC-Infrastructure and >Storage. >Also think about consolidating all data (even the os, share the root fs of the >servers) on to the storage system. It saves you loads of management (scales >within small amount of time) and disks in the servers. And you can easily >replace a faulty server by pulling it out and powering a new one back on. We >have made quite good experiences at webfarms with that concept. >Check back a http://www.open-sharedroot.org/. We will have a howto for >building up such a sharedroot within the next week. > >Hope that helps >Regards Marc. > > >>Thanks again >> >>Eric Anderson wrote: >> >> >>>Marc Grimme wrote: >>> >>> >>>>Hello, >>>>I think the best way to tell what storage or infrastructure would be >>>>the best is to know more about your current setup and what issues >>>>with that you want to get ahead of. >>>>For example: if you really think about using iscsi, I don't think >>>>that SCSI or SATA drives make a big difference - depending on how >>>>many drives you use. But if all webservers currently have locally >>>>attached disk drives you want scale too linar with exchanging and >>>>IDE/parallel-SCSI Bus with an network topology using Ethernet. But my >>>>opinion is: if you have a lot of I/Os make yourself mostly >>>>independent from the latency of an ethernet and rethink about using >>>>Fibre-Channel with GFS. >>>> >>>> >>>Honestly, ethernet latencies (especially on gigabit ethernet) are >>>lower than fiber channel latencies, so this statement doesn't really >>>hold up. >>>If you want very fast speeds, get an iSCSI array, populate it with 15k >>>RPM scsi disks with big caches, max the cache out on the array, and >>>set it up for a RAID0+1 (or RAID10 depending on the implementor). If >>>you want fast speed, but not a big price, you can notch down to 10K >>>scsi disks, or use 15k scsi disks and a raid 5, etc, and keep notching >>>down until it fits your budget and needs. >>>I agree here though that we really need to know a few things: >>> >>>- what kind of traffic is this? >>>- size of the files most commonly used >>>- total data size (how much space you need) >>>- budget >>>- demands (availability/performance/etc) >>> >>>Eric >>> >>> >>> >>>>But the best advices could be made if you make your current setup and >>>>the things you want to achieve more clearly. >>>> >>>>Regards Marc. >>>> >>>>On Wednesday 18 January 2006 22:01, FM wrote: >>>> >>>> >>>>>Thanks for the reply, >>>>>I read about SATA storage but we sync from lan to dmz, so there is lots >>>>>of r/w. >>>>> >>>>>Michael Will wrote: >>>>> >>>>> >>>>>>I am surprised you use SCSI drives on the storage if >>>>>>you are price sensitive, usually SATA is the better >>>>>>bang for the buck unless you are doing databases with >>>>>>lots of small read and writes. >>>>>> >>>>>>Michael >>>>>> >>>>>>-----Original Message----- >>>>>>From: linux-cluster-bounces at redhat.com >>>>>>[mailto:linux-cluster-bounces at redhat.com] On Behalf Of FM >>>>>>Sent: Wednesday, January 18, 2006 12:34 PM >>>>>>To: Redhat Cluster >>>>>>Subject: [Linux-cluster] WebFarm using RedHat cluster suite ? >>>>>> >>>>>>Hello everybody, >>>>>> >>>>>>Is redhat cluster suite (RHEL 4 ) a good candidate for a webfarm ? >>>>>>My setup would be : several servers (1U AMd dual core) connected to >>>>>>iscsi storage array. >>>>>> >>>>>>Is Iscsi a good choice (SAN prices are too high for us) for hardware ? >>>>>>Our network is GB. >>>>>>We will have SCSI 10KRPM + read and write cache on the SCSI card + >>>>>>RAID5 >>>>>>thanks ! >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>-- >>>>>>Linux-cluster mailing list >>>>>>Linux-cluster at redhat.com >>>>>>https://www.redhat.com/mailman/listinfo/linux-cluster >>>>>> >>>>>>-- >>>>>>Linux-cluster mailing list >>>>>>Linux-cluster at redhat.com >>>>>>https://www.redhat.com/mailman/listinfo/linux-cluster >>>>>> >>>>>> >>>>>-- >>>>>Linux-cluster mailing list >>>>>Linux-cluster at redhat.com >>>>>https://www.redhat.com/mailman/listinfo/linux-cluster >>>>> >>>>> >>-- >>Linux-cluster mailing list >>Linux-cluster at redhat.com >>https://www.redhat.com/mailman/listinfo/linux-cluster >> >> > > > From jbrassow at redhat.com Fri Jan 20 18:07:30 2006 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Fri, 20 Jan 2006 12:07:30 -0600 Subject: [Linux-cluster] Kernel oops on GFS umount In-Reply-To: <644A0966265D9D40AC7584FCE95611130216F41F@dubexc01.emea.cpqcorp.net> References: <644A0966265D9D40AC7584FCE95611130216F41F@dubexc01.emea.cpqcorp.net> Message-ID: <867876e7d1c3dc8425438bea88084802@redhat.com> Almost sounds like the network is being removed before the filesystem shuts down. Would this be the case? If so, we should have init scripts that take care of that. brassow On Jan 19, 2006, at 11:07 AM, O'Sullivan, Damian wrote: > If I ever reboot a node on a cluster it panics on the way down: > > Process umount (pid: 22814, threadinfo 00000100775fe000, task > 000001007fb95030) > Stack: 0000000000000084 ffffffff801f5163 0000000000000058 > ffffff00102fe000 > 00000100276d7cf0 ffffffffa0295b96 0000000000000001 > ffffffffa025c8c4 > 0000000000000001 ffffffffa02533c3 > Call Trace:{vgacon_cursor+0} > {:lock_dlm:lm_d > lm_unlock+21} > {:gfs:gfs_lm_unlock+41} > {:gfs:gfs_glo > ck_drop_th+290} > {:gfs:run_queue+314} > {:gfs:gfs_glock_ > dq+323} > {:gfs:gfs_glock_dq_uninit+9} > {:gfs:gf > s_make_fs_ro+118} > {:gfs:gfs_put_super+654} > {generic_shu > tdown_super+202} > {:gfs:gfs_kill_sb+42} > {dummy_inode_pe > rmission+0} > {deactivate_super+95} > {sys_umount+925 > } > {error_exit+0} > {system_call+126} > > > Code: 0f 0b 4e a1 29 a0 ff ff ff ff 65 01 48 c7 c7 53 a1 29 a0 31 > RIP {:lock_dlm:do_dlm_unlock+167} RSP > <00000100775ffca8> > <0>Kernel panic - not syncing: Oops > > Kernel : 2.6.9-22.0.1.Elsmp > GFS-6.1.2-0 > > D. > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From jbrassow at redhat.com Fri Jan 20 18:09:20 2006 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Fri, 20 Jan 2006 12:09:20 -0600 Subject: [Linux-cluster] gfs_tool df question In-Reply-To: <4766EEE585A6D311ADF500E018C154E30213351C@bnifex.cis.buc.com> References: <4766EEE585A6D311ADF500E018C154E30213351C@bnifex.cis.buc.com> Message-ID: <21bf2ec9d28eda0571262cdabef23c86@redhat.com> GFS will allocate more metadata blocks when it needs them. (Same thing with the inodes.) brassow On Jan 19, 2006, at 11:44 AM, Bowie Bailey wrote: > I am trying to understand the information provided by "gfs_tool df". > > When I run it on my system, I get this: > > Type Total Used Free use% > > ----------------------------------------------------------------------- > - > inodes 2697580 2697580 0 100% > metadata 104195 21410 82785 21% > data 290040505 23167 290017338 0% > > I figured out that the data row is reporting used and free space in > blocks and the inode row is self-explanatory, but what is "metadata"? > Do I need to be concerned that metadata is 21% used while data hasn't > even hit 1% yet? > > -- > Bowie > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From Bowie_Bailey at BUC.com Fri Jan 20 18:38:39 2006 From: Bowie_Bailey at BUC.com (Bowie Bailey) Date: Fri, 20 Jan 2006 13:38:39 -0500 Subject: [Linux-cluster] gfs_tool df question Message-ID: <4766EEE585A6D311ADF500E018C154E302133524@bnifex.cis.buc.com> Jonathan E Brassow wrote: > > On Jan 19, 2006, at 11:44 AM, Bowie Bailey wrote: > > > I am trying to understand the information provided by "gfs_tool df". > > > > When I run it on my system, I get this: > > > > Type Total Used Free use% > > > > ----------------------------------------------------------------------- > > - inodes 2697580 2697580 0 100% > > metadata 104195 21410 82785 21% > > data 290040505 23167 290017338 0% > > > > I figured out that the data row is reporting used and free space in > > blocks and the inode row is self-explanatory, but what is > > "metadata"? Do I need to be concerned that metadata is 21% used > > while data hasn't even hit 1% yet? > > GFS will allocate more metadata blocks when it needs them. (Same > thing with the inodes.) Ok. That answers one question, although I'm still not quite sure what metadata IS. Now, if I'm reading the above information correctly, it's showing that my data is taking up 23167 blocks which is about 90M and the metadata is taking up another 90M or so. However, when I check the usage of the filesystem with 'du', I get this: # du -sh mailboxes 11G mailboxes Now 180M is nowhere near 11G! Am I reading something wrong? Why am I getting different numbers? I tend to think that the du is more accurate as that is the amount of usage that I expected to see. Now I'm really confused... -- Bowie From teigland at redhat.com Fri Jan 20 18:50:26 2006 From: teigland at redhat.com (David Teigland) Date: Fri, 20 Jan 2006 12:50:26 -0600 Subject: [Linux-cluster] gfs_tool df question In-Reply-To: <4766EEE585A6D311ADF500E018C154E302133524@bnifex.cis.buc.com> References: <4766EEE585A6D311ADF500E018C154E302133524@bnifex.cis.buc.com> Message-ID: <20060120185026.GE11480@redhat.com> On Fri, Jan 20, 2006 at 01:38:39PM -0500, Bowie Bailey wrote: > > > Type Total Used Free use% > > > > > > ----------------------------------------------------------------------- > > > - inodes 2697580 2697580 0 100% > > > metadata 104195 21410 82785 21% > > > data 290040505 23167 290017338 0% > Now 180M is nowhere near 11G! Am I reading something wrong? Why am I > getting different numbers? Add the space used by the inode blocks. File/directory data is stuffed into the inode block if it's under around 3500 bytes. From Bowie_Bailey at BUC.com Fri Jan 20 20:18:54 2006 From: Bowie_Bailey at BUC.com (Bowie Bailey) Date: Fri, 20 Jan 2006 15:18:54 -0500 Subject: [Linux-cluster] gfs_tool df question Message-ID: <4766EEE585A6D311ADF500E018C154E302133526@bnifex.cis.buc.com> David Teigland wrote: > On Fri, Jan 20, 2006 at 01:38:39PM -0500, Bowie Bailey wrote: >>>> Type Total Used Free use% >>>> >>>> ---------------------------------------------------- >>>> inodes 2697580 2697580 0 100% >>>> metadata 104195 21410 82785 21% >>>> data 290040505 23167 290017338 0% > >> Now 180M is nowhere near 11G! Am I reading something wrong? Why >> am I getting different numbers? > > Add the space used by the inode blocks. File/directory data is > stuffed into the inode block if it's under around 3500 bytes. Ah...THAT's the piece of information I was missing! Now that you mention it, I remember reading about that somewhere. I guess it just didn't click. Based on what I've gathered so far, that chart is showing inodes, metadata, and data as a number of blocks, right? So to get disk usage, I add up the used inode, metadata, and data blocks and multiply by block size (4K)? That seems to show me a 1.1TB total with 10.5GB used, which sounds about right. -- Bowie From jbrassow at redhat.com Fri Jan 20 21:20:46 2006 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Fri, 20 Jan 2006 15:20:46 -0600 Subject: [Linux-cluster] gfs_tool df question In-Reply-To: <4766EEE585A6D311ADF500E018C154E302133526@bnifex.cis.buc.com> References: <4766EEE585A6D311ADF500E018C154E302133526@bnifex.cis.buc.com> Message-ID: <80814c9756a7dd9763d361ed9fbeccc0@redhat.com> On Jan 20, 2006, at 2:18 PM, Bowie Bailey wrote: > David Teigland wrote: >> On Fri, Jan 20, 2006 at 01:38:39PM -0500, Bowie Bailey wrote: >>>>> Type Total Used Free use% >>>>> >>>>> ---------------------------------------------------- >>>>> inodes 2697580 2697580 0 100% >>>>> metadata 104195 21410 82785 21% >>>>> data 290040505 23167 290017338 0% >> >>> Now 180M is nowhere near 11G! Am I reading something wrong? Why >>> am I getting different numbers? >> >> Add the space used by the inode blocks. File/directory data is >> stuffed into the inode block if it's under around 3500 bytes. > > Ah...THAT's the piece of information I was missing! Now that you > mention it, I remember reading about that somewhere. I guess it just > didn't click. > > Based on what I've gathered so far, that chart is showing inodes, > metadata, and data as a number of blocks, right? So to get disk usage, > I add up the used inode, metadata, and data blocks and multiply by > block > size (4K)? > > That seems to show me a 1.1TB total with 10.5GB used, which sounds > about > right. > yes. Also, to answer one of your earlier questions, metadata is file-system specific data (directories, indirect blocks, etc). brassow From omer at faruk.net Sun Jan 22 19:01:29 2006 From: omer at faruk.net (Omer Faruk Sen) Date: Sun, 22 Jan 2006 21:01:29 +0200 (EET) Subject: [Linux-cluster] cluster problem rhel as 4.2 and Message-ID: <37100.85.105.40.193.1137956489.squirrel@85.105.40.193> Hi, I have setup a simple cluster. Everything goes fine .. Cluster services get started and I can manually restart cluster service without any problem. But if I manually edit /etc/httpd/conf/httpd.conf (to make it not to work by rhcs) normally when I try to restart cluster it doesn?t work but when I re-edit httpd.conf so cluster can work I can?t start cluster. I always get error whatever I do until I reboot my system. I use manual fencing to test it but I think it makes no problem since I don?t use a shared storage. I have found that the problem seems to be /etc/rc.d/init.d/cman script. When I try to restart it it tries to reload dlm module. I use RHEL AS 4.2 (no up2date was done) and rhcs that can be downloaded from rhn (as iso). I th?nk this is a bug and I want to learn if this was resolved. By the way I have added different services (vsftpd, nfs ... and proved that it is not related with httpd ....) Here is my cluster.conf: