From daniel at osdl.org Tue Feb 1 19:03:57 2005 From: daniel at osdl.org (Daniel McNeil) Date: Tue, 01 Feb 2005 11:03:57 -0800 Subject: [Linux-cluster] ccsd does not run (latest cvs) Message-ID: <1107284637.27476.19.camel@ibm-c.pdx.osdl.net> I updated to the latest cvs and 2.6.10 and ccsd does not run. I ran make install to install everything, so it all should be up to date. I recompile ccsd with DEBUG defined and get this: [root at cl030 cluster]# ccsd [ccsd.c:287] Entering parse_cli_args [ccsd.c:447] Exiting parse_cli_args [ccsd.c:635] Entering daemonize [ccsd.c:648] Entering daemon mode. Failed to connect to cluster manager. Hint: Magma plugins are not in the right spot. Shouldn't make install put everything in the right spot? Where is the right spot? Is this running for everyone else? Thanks, Daniel From jbrassow at redhat.com Tue Feb 1 23:24:04 2005 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Tue, 1 Feb 2005 17:24:04 -0600 Subject: [Linux-cluster] ccsd does not run (latest cvs) In-Reply-To: <1107284637.27476.19.camel@ibm-c.pdx.osdl.net> References: <1107284637.27476.19.camel@ibm-c.pdx.osdl.net> Message-ID: <03893e3e7ff7070ba615d1d93cabb34a@redhat.com> It's not ccsd, it's magma... The plugins are not in the spot that libmagma is expecting them. There was a new tool added (magma_tool) which may help shed some light. # here's what mine says prompt> magma_tool config plugindir /usr/lib/magma prompt> ls `magma_tool config plugindir` magma_gulm.so magma_sm.so You may wish to 'cd cluster; make uninstall; make distclean; ./configure; make install'. brassow On Feb 1, 2005, at 1:03 PM, Daniel McNeil wrote: > I updated to the latest cvs and 2.6.10 and ccsd does not run. > > I ran make install to install everything, so it all should > be up to date. I recompile ccsd with DEBUG defined and get this: > > [root at cl030 cluster]# ccsd > [ccsd.c:287] Entering parse_cli_args > [ccsd.c:447] Exiting parse_cli_args > [ccsd.c:635] Entering daemonize > [ccsd.c:648] Entering daemon mode. > Failed to connect to cluster manager. > Hint: Magma plugins are not in the right spot. > > Shouldn't make install put everything in the right spot? > Where is the right spot? > Is this running for everyone else? > > Thanks, > > Daniel > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From daniel at osdl.org Wed Feb 2 00:39:34 2005 From: daniel at osdl.org (Daniel McNeil) Date: Tue, 01 Feb 2005 16:39:34 -0800 Subject: [Linux-cluster] ccsd does not run (latest cvs) In-Reply-To: <03893e3e7ff7070ba615d1d93cabb34a@redhat.com> References: <1107284637.27476.19.camel@ibm-c.pdx.osdl.net> <03893e3e7ff7070ba615d1d93cabb34a@redhat.com> Message-ID: <1107304773.10296.27.camel@ibm-c.pdx.osdl.net> On Tue, 2005-02-01 at 15:24, Jonathan E Brassow wrote: > It's not ccsd, it's magma... The plugins are not in the spot that > libmagma is expecting them. > > There was a new tool added (magma_tool) which may help shed some light. > > # here's what mine says > prompt> magma_tool config plugindir > /usr/lib/magma > > prompt> ls `magma_tool config plugindir` > magma_gulm.so magma_sm.so > > You may wish to 'cd cluster; make uninstall; make distclean; > ./configure; make install'. > > brassow > Every looks ok to me: # magma_tool config plugindir /usr/lib/magma # ls -l `magma_tool config plugindir` total 140 -rwxr-xr-x 2 root root 113399 Feb 1 15:38 magma_gulm.so -rwxr-xr-x 2 root root 19049 Feb 1 15:38 magma_sm.so I'll give the rebuilding from scratch a try. Daniel From daniel at osdl.org Wed Feb 2 01:31:11 2005 From: daniel at osdl.org (Daniel McNeil) Date: Tue, 01 Feb 2005 17:31:11 -0800 Subject: [Linux-cluster] ccsd does not run (latest cvs) In-Reply-To: <1107304773.10296.27.camel@ibm-c.pdx.osdl.net> References: <1107284637.27476.19.camel@ibm-c.pdx.osdl.net> <03893e3e7ff7070ba615d1d93cabb34a@redhat.com> <1107304773.10296.27.camel@ibm-c.pdx.osdl.net> Message-ID: <1107307871.10296.29.camel@ibm-c.pdx.osdl.net> On Tue, 2005-02-01 at 16:39, Daniel McNeil wrote: > On Tue, 2005-02-01 at 15:24, Jonathan E Brassow wrote: > > It's not ccsd, it's magma... The plugins are not in the spot that > > libmagma is expecting them. > > > > There was a new tool added (magma_tool) which may help shed some light. > > > > # here's what mine says > > prompt> magma_tool config plugindir > > /usr/lib/magma > > > > prompt> ls `magma_tool config plugindir` > > magma_gulm.so magma_sm.so > > > > You may wish to 'cd cluster; make uninstall; make distclean; > > ./configure; make install'. > > > > brassow > > > > Every looks ok to me: > > # magma_tool config plugindir > /usr/lib/magma > > # ls -l `magma_tool config plugindir` > total 140 > -rwxr-xr-x 2 root root 113399 Feb 1 15:38 magma_gulm.so > -rwxr-xr-x 2 root root 19049 Feb 1 15:38 magma_sm.so > > I'll give the rebuilding from scratch a try. > > Daniel Rebuilding everything from scratch worked. Thanks, Daniel From fmarchal at inf.ethz.ch Wed Feb 2 12:57:41 2005 From: fmarchal at inf.ethz.ch (Fabrice Marchal) Date: Wed, 02 Feb 2005 13:57:41 +0100 Subject: [Linux-cluster] Qs about GFS Message-ID: <4200CE45.2000807@inf.ethz.ch> Hi, I am planning to install GFS. Can it run without GNBD servers? (as in the examples provided in the admin doc in appendix). Is there some builtin redundancy (i.e. what if one of the nodes dies?) or is the redundancy totally delegated to the GNBD servers? Thanks, Fabrice -- ======================================================================== Fabrice Marchal http://www.inf.ethz.ch/~marchal fabrice.marchal at ieee.org marchal at inf.ethz.ch +41-(0)44-632-56-79 ETH Zurich, CoLab Computational Laboratory FAX:+41-(0)44-632-17-03 ======================================================================== From mingz at ele.uri.edu Tue Feb 1 01:17:19 2005 From: mingz at ele.uri.edu (Ming Zhang) Date: Mon, 31 Jan 2005 20:17:19 -0500 Subject: [Linux-cluster] [Fwd: any special requirement on scsi device?] Message-ID: <1107220639.3017.4.camel@localhost.localdomain> Hi folks I asked this question today, somebody there guided me to here. Can anybody help me on this? Thanks. ps, how to subscribe on this list? Thanks. Pls cc to me, Thanks. Ming -----Forwarded Message----- > From: Ming Zhang > To: opengfs-devel at lists.sourceforge.net > Subject: any special requirement on scsi device? > Date: Mon, 31 Jan 2005 16:41:00 -0500 > > Hi, folks > > I wonder if opengfs has any special requirements on scsi deivces. For > example, reserve/release scsi commands. > > the reason i ask this is we are developing a open source iscsi target > and would like to support opengfs. since we build all scsi response, we > need to know what extra scsi commands or feature we need to support. > > thanks. > > > ming From sunjw at onewaveinc.com Thu Feb 3 03:24:11 2005 From: sunjw at onewaveinc.com (=?GB2312?B?y++/oc6w?=) Date: Thu, 3 Feb 2005 11:24:11 +0800 Subject: [Linux-cluster] what's gfs_scand doing? Message-ID: Hi,all I got the gfs' code from cvs on 2004-12-12, which was for kernel 2.6.9, and the OS was FC3. I created three logic volumes by CLVM, then I test the read performance of the SAN storage. On one node, I got 150MB/s by reading 50 files on one LV concurrently; But when I readed the same 50 files on two nodes started at the same time, the result was 21 + 21 and totally 42 MB/s. So why the performance declined two much? I saw the result of command "top", the process "gfs_scand" got the No.1, so what's gfs_scand doing? And iowait got much cpu's idle time. Is it related to hardwares, such as the RAID controller, the HBA card, and the FC switch? Or related to hardware drivers? Thanks for any reply! Best regards! Luckey From mshk_00 at hotmail.com Thu Feb 3 11:51:04 2005 From: mshk_00 at hotmail.com (maria perez) Date: Thu, 03 Feb 2005 12:51:04 +0100 Subject: [Linux-cluster] GFS and HEARTBEAT Message-ID: Hi, Lon!! Thank you very much for your help!! I have the ideas more clear now. I hope have good luck with this, I tell you later. Maria _________________________________________________________________ Acepta el reto MSN Premium: Correos m?s divertidos con fotos y textos incre?bles en MSN Premium. Desc?rgalo y pru?balo 2 meses gratis. http://join.msn.com?XAPID=1697&DI=1055&HL=Footer_mailsenviados_correosmasdivertidos From rmayhew at mweb.com Thu Feb 3 12:53:08 2005 From: rmayhew at mweb.com (Richard Mayhew) Date: Thu, 3 Feb 2005 14:53:08 +0200 Subject: [Linux-cluster] 10 Node Installation - Loosing Heartbeat Message-ID: <91C4F1A7C418014D9F88E938C135545801274D4B@mwjdc2.mweb.com> Hi All, I have been able to successfully setup a 10 node GFS cluster, 4 Locking Servers and 6 Clients. Each server is a Dell PowerEdge 1750 with 1GB RAM, Dual P4 2.8 HT running RedHat Enterprise Server V3 Update 4. The first ethernet interface is used for normal network traffic and the second ethernet interface is used for GFS heatbeats only. Both interfaces run at 1GB-FD on 2 separate switches. I have installed the GFS-6.0.2-25 from http://bender.it.swin.edu.au/centos-3/RHGFS/i686/ on the latest RedHat ES Kernel. The storage is made available using a EMC CX600 SAN. (4 x 50GB) running over a McData Fiber Switch. The pool service uses the storage from the san over the EMC Powepath software using the emcpower pseudo devices. Pool is able to assemble the pools with no problem. The ccsd service is able to retrieve the gfs archive with no problems. The lock gulm server loads with no problem and communicates with the master lock server with out any errors or missed beats. GFS mounts with out any errors etc. I am able to access the storage on each server with no problem on throughput. The problem I am experiencing is as follows. Once the GFS system has been running for a few hours with some usage on each of the servers, some of the servers start missing beats. I increased the heartbeat rate to test every 60 seconds and to fail after 10 tries. This just prolonged the servers being fenced. The only thing I can come up with is that the locking server is buggy and stops responding to heartbeats. On the master server when it detects that the server has skipped the required number of beats, it tries to fence it and fails. I have setup the fencing to use the mcdata module and I have specified the correct login details. When the server that was fenced has had its lock server restarted it tries to relog in to the master lock server. This fails for obvious reasons as the master will refuse to allow it to reconnect due to the previous fencing failures. Manual fencing works without any problems but I have only tried this on the cmd line. Does anyone have an idea as to why the locking servers are hanging up when it comes to sending heartbeat beats and possibly why the fencing isnt working? Here are my configs with some of the privileged information changed. fence.css fence_devices { EMC_Switch_01 { agent = "fence_mcdata" ipaddr = "xxx.xxx.xxx.xxx" login = "XXXXXXXX" password = "xxxxxx" } I EMC_Switch_02 { agent = "fence_mcdata" ipaddr = "xxx.xxx.xxx.xxx" login = "XXXXXXXXXX" password = "xxxxxx" } } Cluster.ccs cluster { name = "mail" lock_gulm servers = ["store-01.mc.mweb.net","store-02.mc.mweb.net","store-03.mc.mweb.net","s tore-04.mc.mweb.net"] heartbeat_rate = 60 allowed_misses = 10 } } Nodes.ccs nodes { store-01.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_01 { port = 3 } } } } store-02.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_01 { port = 27 } } } } store-03.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_01 { port = 9 } } } } store-04.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_01 { port = 31 } } } } serv-01.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_01 { port = 19 } } } } serv-02.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_01 { port = 27 } } } } serv-03.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_01 { port = 31 } } } } serv-04.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_02 { port = 3 } } } } serv-05.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_02 { port = 9 } ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_02 { port = 3 } } } } serv-05.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_02 { port = 9 } } } } serv-06.mc.mweb.net { ip_interfaces { eth1 = "xxx.xxx.xxx.xxx" } fence { san { EMC_Switch_02 { port = 19 } } } } } -- Regards Richard Mayhew Unix Specialist MWEB Business Tel: + 27 11 340 7200 Fax: + 27 11 340 7288 Website: www.mwebbusiness.co.za From mtilstra at redhat.com Thu Feb 3 14:24:00 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Thu, 3 Feb 2005 08:24:00 -0600 Subject: [Linux-cluster] 10 Node Installation - Loosing Heartbeat In-Reply-To: <91C4F1A7C418014D9F88E938C135545801274D4B@mwjdc2.mweb.com> References: <91C4F1A7C418014D9F88E938C135545801274D4B@mwjdc2.mweb.com> Message-ID: <20050203142400.GA9385@redhat.com> On Thu, Feb 03, 2005 at 02:53:08PM +0200, Richard Mayhew wrote: > The problem I am experiencing is as follows. > > Once the GFS system has been running for a few hours with some usage on > each of the servers, some of the servers start missing beats. I > increased the heartbeat rate to test every 60 seconds and to fail after > 10 tries. This just prolonged the servers being fenced. The only thing I > can come up with is that the locking server is buggy and stops > responding to heartbeats. On the master server when it detects that the > server has skipped the required number of beats, it tries to fence it > and fails. I have setup the fencing to use the mcdata module and I have > specified the correct login details. When the server that was fenced has > had its lock server restarted it tries to relog in to the master lock > server. This fails for obvious reasons as the master will refuse to > allow it to reconnect due to the previous fencing failures. Manual > fencing works without any problems but I have only tried this on the cmd > line. > > Does anyone have an idea as to why the locking servers are hanging up > when it comes to sending heartbeat beats and possibly why the fencing > isnt working? First test the mcdata fencing. Get your configs installed and ccsd running. Then run `fence_node store-01.mc.mweb.net` (or one of the other nodes). Make sure that this has infact work. (look on the switch and what not. I don't know anything about the mcdata, so I cannot say much more here.) This will let you test and see how fencing is working. Don't continue until you can call fence_node for every node and it does in fact fence the node. So, on to the missed heartbeats. First, what is 'some usage'. Not that it should matter much, but just as an example, I can get missed heartbeats by syncing large (~1G or so) amounts of data to the internal drive. (but to miss 11 at 60s apiece, it probably not this.) Also, are there any messages from the Master or the Clients about the time they start missing heartbeats? (other than the missed heartbeat messages.) If so, might give some clues as to what's happening. Best thing to do when debugging heartbeats is to turn on those messages. So add cluster.ccs:cluster{ lock_gulm { verbosity = "Default,Heartbeat" } } to your config, and run things again. (might also want to turn the heartbeat rate back down for this.) There will now be messages in syslog for every heartbeat sent and received. Hopefully this will unveil something. -- Michael Conrad Tadpol Tilstra To understand recursion, you must first understand recursion. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From cristiano at develop.com.br Thu Feb 3 19:10:28 2005 From: cristiano at develop.com.br (Cristiano da Costa) Date: Thu, 3 Feb 2005 17:10:28 -0200 (BRST) Subject: [Linux-cluster] gnbd server as shared storage Message-ID: <2758704.1107457828765.JavaMail.SYSTEM@serraria> Hello list Im trying to configure Red Hat Cluster Suite in an enviroment without shared storage to learn how to admin and configure this. Then I using gnbd to simulate a shared storage and I mapped the exported devices in other two RHEL, that are members of the cluster, and bind gnbd devices imported to raw devices to use as quorum shared storage, but when start clumanager the machines reboot, informing the following error in log: Feb 3 14:40:55 parati clusvcmgrd[5133]: Couldn't read configuration from shared state: Success Feb 3 14:40:55 parati clusvcmgrd[5133]: Shared State Error: REBOOTING Then I make tests to simultaneos write data in a gnbd device mapped and I note that the files created by one node only appear to other node when the device is reimported and remounted. There are other way to configure RHCS without a shared storage like EMC storages or Sun storages? That are my configuration files for gnbd [root at ilhabela mnt]# cat /etc/sistina/ccs-build/angra.cca #CCA:angra #nigeb=cluster.ccs mtime=1107192495 size=147 cluster { name = "angra" lock_gulm { servers = ["angra"] heartbeat_rate = 0.3 allowed_misses = 1 } } #dne=cluster.ccs hash=BD5C824C #nigeb=nodes.ccs mtime=1107192495 size=177 nodes { angra { ip_interfaces { eth0 = "192.168.1.52" } fence { server { gnbd0 { ipaddr = "192.168.1.52" } } } } } #dne=nodes.ccs hash=72704281 #nigeb=fence.ccs mtime=1107192495 size=78 fence_devices { gnbd0 { agent = "fence_gnbd" server = "angra" } } #dne=fence.ccs hash=9E0446A4 [root at ilhabela mnt]# cat /root/angra/cluster.ccs cluster { name = "angra" lock_gulm { servers = ["angra"] heartbeat_rate = 0.3 allowed_misses = 1 } } [root at ilhabela mnt]# cat /root/angra/fence.ccs fence_devices { gnbd0 { agent = "fence_gnbd" server = "angra" } } [root at ilhabela mnt]# cat /root/angra/nodes.ccs nodes { angra { ip_interfaces { eth0 = "192.168.1.52" } fence { server { gnbd0 { ipaddr = "192.168.1.52" } } } } parati.localdomain { ip_interfaces { eth0 = "192.168.1.56" } fence { server { gnbd0 { ipaddr = "192.168.1.52" } } } } ilhabela.localdomain { ip_interfaces { eth0 = "192.168.1.57" } fence { server { gnbd0 { ipaddr = "192.168.1.52" } } } } } This is the output of gnbd_import -l in both nodes of the cluster [root at ilhabela root]# gnbd_import -l gnbd_import: Device name : hda10 ---------------------- Minor # : 1 Proc name : gnbdb IP : 192.168.1.52 Port : 14243 State : Close Disconnected Clear Readonly : No gnbd_import: Device name : hda9 ---------------------- Minor # : 2 Proc name : gnbdc IP : 192.168.1.52 Port : 14243 State : Close Disconnected Clear Readonly : No gnbd_import: Device name : hda11 ---------------------- Minor # : 3 Proc name : gnbdd IP : 192.168.1.52 Port : 14243 State : Open Connected Clear Readonly : No This is the content of /etc/sysconfig/rawdevices [root at ilhabela root]# cat /etc/sysconfig/rawdevices # raw device bindings # format: # # example: /dev/raw/raw1 /dev/sda1 # /dev/raw/raw2 8 5 /dev/raw/raw1 /dev/gnbd/hda9 /dev/raw/raw2 /dev/gnbd/hda10 Partition /dev/gnbd/had11 is ext3 and are mounted in both nodes, but the data write for one node only is acessible to other when the device is reimported and remounted. Grateful _________________________ Cristiano da Costa Consultor Develop IT Solutions www.develop.com.br Fone: (51) 3386-6620 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nigel.jewell at pixexcel.co.uk Fri Feb 4 10:26:36 2005 From: nigel.jewell at pixexcel.co.uk (Nigel Jewell) Date: Fri, 04 Feb 2005 10:26:36 +0000 Subject: [Linux-cluster] GNBD & Network Outage In-Reply-To: <20050125205538.GD13289@phlogiston.msp.redhat.com> References: <41F622D8.1060102@pixexcel.co.uk> <20050125205538.GD13289@phlogiston.msp.redhat.com> Message-ID: <42034DDC.9010002@pixexcel.co.uk> Dear Ben, Thank you for your detailed reply. It is always refreshing to get a decent response on a mailing list ;) . > Sure. You see the -c in you export line. Don't put it there. That puts > the device in (the very poorly named) uncached mode. This does two > things. > One: It causes the server to use direct IO to write to the exported > device, > so your read performance will take a hit. Two: It will time out after > a period (default to 10 sec). After gnbd times out, it must be able > to fence > the server before it will let the requests fail. This is so that you > know > that the server isn't simply stalled and might write out the requests > later > (if gnbd failed out, and the requests were rerouted to the backend > storage over > another gnbd server, if the first server wrote it's requests out > later, it > could cause data corruption). > > My understanding was that the "-c" put the device in cached mode, as described here: http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-gnbd-commands.html Or are you saying that by not putting the "-c" put its in uncached mode? > This means that to run in uncached mode, you need to have a cluster > manager and > fencing devices, which I'm not certain that you have. > > No we don't as we didn't really see the need, given what we want to do. > I've got some questions about your setup. Will this be part of a > clustered > filesystem setup? If it will, I see some problems with your mirror. When > other nodes (including the gnbd server node A) write to the exported > device, > these writes will not appear on the local partion of B. So won't your > mirror > get out of sync? If only B will write to the exported device, (and > that's > the only way I see this working) you can probably get by with nbd, which > simply fails out if it loses connection. > > The intention of the setup was to have two hosts both exporting an unmounted device, and the alternative device using it as a RAID-1 device. Then to use heartbeat to mount and unmount the partitions as required. For example: HOST A: /dev/hda1 (md0, ext3, mounted) /dev/hda2 (ext3, unmounted, gnbd_exported as A) /dev/gnbd/B (md0, ext3, mounted) HOST B: /dev/hda1 (ext3, unmounted, gnbd_exported as B) /dev/hda2 (md0, ext3, mounted) /dev/gnbd/A (md0, ext3, mounted) I hope that makes sense. If so, does what we are trying to achieve sound sensible? Any gotchas/advice? -- Nige. PixExcel Limited URL: http://www.pixexcel.co.uk From lhh at redhat.com Fri Feb 4 16:29:24 2005 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 04 Feb 2005 11:29:24 -0500 Subject: [Linux-cluster] ccsd does not run (latest cvs) In-Reply-To: <1107307871.10296.29.camel@ibm-c.pdx.osdl.net> References: <1107284637.27476.19.camel@ibm-c.pdx.osdl.net> <03893e3e7ff7070ba615d1d93cabb34a@redhat.com> <1107304773.10296.27.camel@ibm-c.pdx.osdl.net> <1107307871.10296.29.camel@ibm-c.pdx.osdl.net> Message-ID: <1107534564.14020.32.camel@ayanami.boston.redhat.com> On Tue, 2005-02-01 at 17:31 -0800, Daniel McNeil wrote: > > Daniel > > Rebuilding everything from scratch worked. Chances are that the plugins were built before the version of magma. I bumped the API version when I added magma_tool (just to be safe) because it required some changes to the build system. For reference, "magma_tool list" will tell you if the plugins were built against a different API version of magma. -- Lon From lhh at redhat.com Fri Feb 4 16:33:20 2005 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 04 Feb 2005 11:33:20 -0500 Subject: [Linux-cluster] gnbd server as shared storage In-Reply-To: <2758704.1107457828765.JavaMail.SYSTEM@serraria> References: <2758704.1107457828765.JavaMail.SYSTEM@serraria> Message-ID: <1107534800.14020.35.camel@ayanami.boston.redhat.com> On Thu, 2005-02-03 at 17:10 -0200, Cristiano da Costa wrote: > Feb 3 14:40:55 parati clusvcmgrd[5133]: Couldn't read > configuration from shared state: Success > Feb 3 14:40:55 parati clusvcmgrd[5133]: Shared State Error: > REBOOTING Try running 'shutil -i'. However, be sure you're running the latest version of clumanager. -- Lon From filip.sergeys at verzekeringen.be Fri Feb 4 16:52:31 2005 From: filip.sergeys at verzekeringen.be (Filip Sergeys) Date: 04 Feb 2005 17:52:31 +0100 Subject: [Linux-cluster] clvm mirroring target status Message-ID: <1107535951.14182.75.camel@wavebreaker.eccent.be> Hi, We are going to install a linux cluster with 2 gnbd servers (no SPOF) and gfs + clvm on the cluster nodes (4 nodes). I have two options, if I read the docs well, for duplicating data on the gnbd servers: 1) using clvm target mirroring on the cluster nodes 2) use drbd underneath to mirror discs. Basically two disks per machine: 1 live disk which is mirrored with drbd to the second disk in the second machine and the other way around in the second machine (so the second disk in the first machine is thus the mirror from the first (="live") disk in the second machine(sounds complicated, but it is just hard to write down)). Both live disks from each machine will be combined as one logical disk (If I understood well, this is possible). Question: what is the status of clvm mirroring? Is it stable? Suppose it is stable, so I have a choice: which one of the options would any of you choose? Reason? (Stability, performance, ...) I found two hits on google concerning clvm mirroring, but both say it is not finished yet. However the most recent one is from june 2004. I cannot test either because we have no spare machine. I'm going to buy two machine so I need to know which disk configuration I will be using. Thanks in advance, Regards, Filip Sergeys http://64.233.183.104/search?q=cache:r1Icx--aI2YJ:www.spinics.net/lists/gfs/msg03439.html+clvm+mirroring+gfs&hl=nl&start=12 https://www.redhat.com/archives/linux-cluster/2004-June/msg00028.html -- *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* * System Engineer, Verzekeringen NV * * www.verzekeringen.be * * Oostkaai 23 B-2170 Merksem * * 03/6416673 - 0477/340942 * *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -------------- next part -------------- An HTML attachment was scrubbed... URL: From bmarzins at redhat.com Fri Feb 4 17:34:43 2005 From: bmarzins at redhat.com (Benjamin Marzinski) Date: Fri, 4 Feb 2005 11:34:43 -0600 Subject: [Linux-cluster] GNBD & Network Outage In-Reply-To: <42034DDC.9010002@pixexcel.co.uk> References: <41F622D8.1060102@pixexcel.co.uk> <20050125205538.GD13289@phlogiston.msp.redhat.com> <42034DDC.9010002@pixexcel.co.uk> Message-ID: <20050204173443.GC3666@phlogiston.msp.redhat.com> On Fri, Feb 04, 2005 at 10:26:36AM +0000, Nigel Jewell wrote: > Dear Ben, > > Thank you for your detailed reply. It is always refreshing to get a > decent response on a mailing list ;) . > > My understanding was that the "-c" put the device in cached mode, as > described here: You are correct. > http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-gnbd-commands.html > > Or are you saying that by not putting the "-c" put its in uncached mode? Yes, that's what I meant to say. so much for decent responses :P > The intention of the setup was to have two hosts both exporting an > unmounted device, and the alternative device using it as a RAID-1 > device. Then to use heartbeat to mount and unmount the partitions as > required. For example: > > HOST A: > > /dev/hda1 (md0, ext3, mounted) > /dev/hda2 (ext3, unmounted, gnbd_exported as A) > /dev/gnbd/B (md0, ext3, mounted) > > HOST B: > > /dev/hda1 (ext3, unmounted, gnbd_exported as B) > /dev/hda2 (md0, ext3, mounted) > /dev/gnbd/A (md0, ext3, mounted) > > I hope that makes sense. O.k. let me see if I get this. hostA is setting up a mirror on /dev/hda1 and /dev/gnbd/B hostB is setting up a mirror on /dev/hda2 and /dev/gnbd/A So, if hostA goes down, you will be able to access its data on /dev/hda1 of hostB. If hostB goes down, you will be able to access its data on /dev/hda2 of hostA. > If so, does what we are trying to achieve sound sensible? Yeah, you can do that. >Any gotchas/advice? The obvious issue is, for example, if hostB goes down and you are accessing it's data through /dev/hda2 on hostA, when hostB comes back up, you must unmount /dev/hda2 on hostA before you export it. Alos, even if hostA never does any writing to /dev/hda2, the data on it may not be in sync with the data on /dev/hda2 of hostB, so you will need to resync them when hostB comes back. For this, there isn't any real reason to use gnbd over nbd... nbd will failout right away, which is annoying when it's caused by some transient network issue, but you don't need to have a clustermanager set up to use it. Another option which might work is to use gnbd in cached mode, but when you decide that the other node really isn't there, run # gnbd_import -rO This will flush the requests from the device. The /dev/gnbd/ file will also be removed, which may piss off your mirror. However, if this works, you get the benefit of retrying the connection until heartbeat decides the other node is really dead. Hope this helps. -Ben > > -- > Nige. > > PixExcel Limited > URL: http://www.pixexcel.co.uk > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From bmarzins at redhat.com Fri Feb 4 17:41:10 2005 From: bmarzins at redhat.com (Benjamin Marzinski) Date: Fri, 4 Feb 2005 11:41:10 -0600 Subject: [Linux-cluster] GNBD & Network Outage In-Reply-To: <42034DDC.9010002@pixexcel.co.uk> References: <41F622D8.1060102@pixexcel.co.uk> <20050125205538.GD13289@phlogiston.msp.redhat.com> <42034DDC.9010002@pixexcel.co.uk> Message-ID: <20050204174110.GD3666@phlogiston.msp.redhat.com> On Fri, Feb 04, 2005 at 10:26:36AM +0000, Nigel Jewell wrote: > Dear Ben, Oh yeah, or you could take a look at http://www.drbd.org/ -Ben > Thank you for your detailed reply. It is always refreshing to get a > decent response on a mailing list ;) . > > >Sure. You see the -c in you export line. Don't put it there. That puts > >the device in (the very poorly named) uncached mode. This does two > >things. > >One: It causes the server to use direct IO to write to the exported > >device, > >so your read performance will take a hit. Two: It will time out after > >a period (default to 10 sec). After gnbd times out, it must be able > >to fence > >the server before it will let the requests fail. This is so that you > >know > >that the server isn't simply stalled and might write out the requests > >later > >(if gnbd failed out, and the requests were rerouted to the backend > >storage over > >another gnbd server, if the first server wrote it's requests out > >later, it > >could cause data corruption). > > > > > > My understanding was that the "-c" put the device in cached mode, as > described here: > > http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-gnbd-commands.html > > Or are you saying that by not putting the "-c" put its in uncached mode? > > >This means that to run in uncached mode, you need to have a cluster > >manager and > >fencing devices, which I'm not certain that you have. > > > > > > No we don't as we didn't really see the need, given what we want to do. > > >I've got some questions about your setup. Will this be part of a > >clustered > >filesystem setup? If it will, I see some problems with your mirror. When > >other nodes (including the gnbd server node A) write to the exported > >device, > >these writes will not appear on the local partion of B. So won't your > >mirror > >get out of sync? If only B will write to the exported device, (and > >that's > >the only way I see this working) you can probably get by with nbd, which > >simply fails out if it loses connection. > > > > > > The intention of the setup was to have two hosts both exporting an > unmounted device, and the alternative device using it as a RAID-1 > device. Then to use heartbeat to mount and unmount the partitions as > required. For example: > > HOST A: > > /dev/hda1 (md0, ext3, mounted) > /dev/hda2 (ext3, unmounted, gnbd_exported as A) > /dev/gnbd/B (md0, ext3, mounted) > > HOST B: > > /dev/hda1 (ext3, unmounted, gnbd_exported as B) > /dev/hda2 (md0, ext3, mounted) > /dev/gnbd/A (md0, ext3, mounted) > > I hope that makes sense. > > If so, does what we are trying to achieve sound sensible? Any > gotchas/advice? > > -- > Nige. > > PixExcel Limited > URL: http://www.pixexcel.co.uk > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From bmarzins at redhat.com Fri Feb 4 17:53:20 2005 From: bmarzins at redhat.com (Benjamin Marzinski) Date: Fri, 4 Feb 2005 11:53:20 -0600 Subject: [Linux-cluster] gnbd server as shared storage In-Reply-To: <2758704.1107457828765.JavaMail.SYSTEM@serraria> References: <2758704.1107457828765.JavaMail.SYSTEM@serraria> Message-ID: <20050204175320.GE3666@phlogiston.msp.redhat.com> On Thu, Feb 03, 2005 at 05:10:28PM -0200, Cristiano da Costa wrote: > Hello list > Then I make tests to simultaneos write data in a gnbd device mapped and I note that the files created by one node only appear to other node when the device is reimported and remounted. > Partition /dev/gnbd/had11 is ext3 and are mounted in both nodes, but the data write for one node only is acessible to other when the device is reimported and remounted. If you actually had shared storage, and did this test, the results would be the same. ext3 is not a clustered filesystem. It never expects the data on disk to change out from under it, so it always trusts its cache. In order to share storage between multiple machines, you need a clustered filesystem, like GFS. -Ben > Grateful > > > > > _________________________ > Cristiano da Costa > Consultor > Develop IT Solutions > www.develop.com.br > Fone: (51) 3386-6620 > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From yazan at ccs.com.jo Fri Feb 4 18:04:39 2005 From: yazan at ccs.com.jo (Yazan Al-Sheyyab) Date: Fri, 4 Feb 2005 20:04:39 +0200 Subject: [Linux-cluster] a question about gfs file system Message-ID: <000901c50ae3$feb22970$69050364@yazanz> hello everybody, how much space that gfs file system can take from a partion ? i mean that if i have a 500 MB partition and want to format it as gfs file system , so how much space will remain for me in this partition after format it as gfs file system? Thanks. From cristiano at develop.com.br Fri Feb 4 18:21:15 2005 From: cristiano at develop.com.br (Cristiano da Costa) Date: Fri, 4 Feb 2005 16:21:15 -0200 (BRST) Subject: [Linux-cluster] gnbd server as shared storage Message-ID: <1002123.1107541275046.JavaMail.SYSTEM@serraria> Hi Ben Tanks in advance, really Im making wrong test, you are very right, but ... I update the clumager package to last version, and now the nodes aren't rebooting when start clumanager, but the quorum aren't not recognized and in this case Im using 2 raws that bind to 2 gnbd imported devices. There are the log of the cluster when start Feb 4 16:22:54 parati clumanager: [5283]: Starting Red Hat Cluster Manager... Feb 4 16:22:54 parati cluquorumd[5303]: changing loglevel from 5 to 7 Feb 4 16:22:54 parati cluquorumd[5303]: IPv4-TB: 192.168.1.52 Feb 4 16:22:54 parati cluquorumd[5303]: STONITH: No drivers configured for host '192.168.1.56'! Feb 4 16:22:54 parati cluquorumd[5303]: STONITH: Data integrity may be compromised! Feb 4 16:22:54 parati cluquorumd[5303]: STONITH: No drivers configured for host '192.168.1.57'! Feb 4 16:22:54 parati cluquorumd[5303]: STONITH: Data integrity may be compromised! Feb 4 16:22:54 parati cluquorumd[5303]: spawn_daemon: starting /usr/sbin/clumembd. Feb 4 16:22:54 parati cluquorumd[5303]: IP tie-breaker in use, not starting disk thread. Feb 4 16:22:54 parati clumembd[5313]: Changing loglevel from 6 to 7 Feb 4 16:22:54 parati clumembd[5313]: Transmit thread set to ON Feb 4 16:22:54 parati clumembd[5313]: Overriding TKO count to be 20 Feb 4 16:22:54 parati clumembd[5313]: Broadcast hearbeating set to ON Feb 4 16:22:54 parati clumembd[5313]: Multicast heartbeat OFF Feb 4 16:22:54 parati clumembd[5313]: I am member #0 Feb 4 16:22:54 parati clumembd[5313]: Interface IP is 127.0.0.1 Feb 4 16:22:54 parati cluquorumd[5303]: Cluster I/F: eth0 [192.168.1.56] Feb 4 16:22:54 parati clumembd[5313]: broadcast is 127.255.255.255 Feb 4 16:22:54 parati clumembd[5313]: Interface IP is 192.168.1.56 Feb 4 16:22:54 parati clumembd[5313]: broadcast is 192.168.1.255 Feb 4 16:22:54 parati clumembd[5313]: Interface IP is 10.1.1.1 Feb 4 16:22:54 parati clumembd[5313]: broadcast is 10.255.255.255 Feb 4 16:22:54 parati clumembd[5313]: Cluster I/F: eth0 [192.168.1.56] Feb 4 16:22:54 parati clumembd[5313]: clumembd_start_watchdog: set duration to 14. Feb 4 16:22:54 parati clumembd[5313]: Waiting for requests. Feb 4 16:22:54 parati clumembd[5313]: Transmit thread: pulsar Feb 4 16:22:55 parati clumembd[5313]: Verified connect from member #0 (127.0.0.1) Feb 4 16:22:55 parati clumembd[5313]: MB: New connect: fd9 Feb 4 16:22:55 parati clumembd[5313]: MB: Received EV_REGISTER, fd9 Feb 4 16:22:55 parati cluquorumd[5303]: spawn_daemon: starting /usr/sbin/clulockd. Feb 4 16:22:55 parati clulockd[5316]: /usr/sbin/clulockd starting Feb 4 16:22:55 parati clulockd[5316]: Cluster I/F: eth0 [192.168.1.56] Feb 4 16:22:55 parati cluquorumd[5303]: Verified connect from member #0 (127.0.0.1) Feb 4 16:22:55 parati cluquorumd[5303]: Q: Received EV_REGISTER, fd6 Feb 4 16:22:55 parati cluquorumd[5303]: Q: Received EV_MEMB_UPDATE, fd5 Feb 4 16:22:55 parati clulockd[5316]: Quorum Event: NO QUORUM Feb 4 16:22:55 parati clulockd[5316]: Lock Keeper = Member #-1 Feb 4 16:23:05 parati clumembd[5313]: Member 192.168.1.56 UP Feb 4 16:23:05 parati clumembd[5313]: MB: Initiating vote on: 0x00000001 Feb 4 16:23:05 parati clumembd[5320]: VF: Connecting to member #0 Feb 4 16:23:05 parati clumembd[5313]: Verified connect from member #0 (192.168.1.56) Feb 4 16:23:05 parati clumembd[5313]: MB: New connect: fd11 Feb 4 16:23:05 parati clumembd[5320]: VF: Push 0.5320 #1 Feb 4 16:23:05 parati clumembd[5320]: VF: Sending to member #0 Feb 4 16:23:05 parati clumembd[5320]: VF: Checking for consensus... Feb 4 16:23:05 parati clumembd[5313]: MB: Received VF_MESSAGE, fd11 Feb 4 16:23:05 parati clumembd[5313]: VF_JOIN_VIEW from member #0! Key: 0x27456381 #1 Feb 4 16:23:05 parati clumembd[5313]: VF: Voting YES Feb 4 16:23:05 parati clumembd[5320]: VF: Member #0 voted YES Feb 4 16:23:05 parati clumembd[5320]: VF: Broadcasting FORMED Feb 4 16:23:05 parati clumembd[5320]: VF: Converge Time: 0.000000 Feb 4 16:23:05 parati clumembd[5313]: MB: Received VF_MESSAGE, fd11 Feb 4 16:23:05 parati clumembd[5313]: VF: Received VF_VIEW_FORMED, fd11 Feb 4 16:23:05 parati clumembd[5313]: VF: Commit Key 0x27456381 #1 from member #0 Feb 4 16:23:05 parati clumembd[5313]: Membership View #1:0x00000001 Feb 4 16:23:06 parati clumembd[5313]: VF: pid 5320 exited, status 0 Feb 4 16:23:06 parati cluquorumd[5303]: Q: Received EV_MEMB_UPDATE, fd5 Feb 4 16:23:06 parati cluquorumd[5303]: Need 1 more members for quorum! Feb 4 16:23:06 parati cluquorumd[5303]: process_memb_update: Starting VF Feb 4 16:23:06 parati cluquorumd[5321]: VF: Connecting to member #0 Feb 4 16:23:06 parati cluquorumd[5303]: VF: Key 0x12345678 Still running Feb 4 16:23:06 parati cluquorumd[5303]: Verified connect from member #0 (192.168.1.56) Feb 4 16:23:06 parati cluquorumd[5303]: VF: Key 0x12345678 Still running Feb 4 16:23:06 parati cluquorumd[5321]: VF: Push 0.5321 #1 Feb 4 16:23:06 parati cluquorumd[5321]: VF: Sending to member #0 Feb 4 16:23:06 parati cluquorumd[5321]: VF: Checking for consensus... Feb 4 16:23:06 parati cluquorumd[5303]: Q: Received VF_MESSAGE, fd8 Feb 4 16:23:06 parati cluquorumd[5303]: VF_JOIN_VIEW from member #0! Key: 0x12345678 #1 Feb 4 16:23:06 parati cluquorumd[5303]: VF: Voting YES Feb 4 16:23:06 parati cluquorumd[5303]: VF: Key 0x12345678 Still running Feb 4 16:23:06 parati cluquorumd[5321]: VF: Member #0 voted YES Feb 4 16:23:06 parati cluquorumd[5321]: VF: Broadcasting FORMED Feb 4 16:23:06 parati cluquorumd[5321]: VF: Converge Time: 0.010000 Feb 4 16:23:06 parati cluquorumd[5303]: Q: Received VF_MESSAGE, fd8 Feb 4 16:23:06 parati cluquorumd[5303]: VF: Received VF_VIEW_FORMED, fd8 Feb 4 16:23:06 parati cluquorumd[5303]: VF: Commit Key 0x12345678 #1 from member #0 Feb 4 16:23:06 parati cluquorumd[5303]: Lack of Quorum Maintained Feb 4 16:23:06 parati clulockd[5316]: Quorum Event: NO QUORUM Feb 4 16:23:06 parati cluquorumd[5303]: VF: pid 5321 exited, status 0 Feb 4 16:23:06 parati clulockd[5316]: I am the new lock keeper Grateful Cristiano da Costa --- Original Message --- > On Thu, Feb 03, 2005 at 05:10:28PM -0200, > Cristiano da Costa wrote: > > Hello list > > > > > Then I make tests to simultaneos write data in > a gnbd device mapped and I note that the files > created by one node only appear to other node > when the device is reimported and remounted. > > > > > Partition /dev/gnbd/had11 is ext3 and are > mounted in both nodes, but the data write for > one node only is acessible to other when the > device is reimported and remounted. > > If you actually had shared storage, and did this > test, the results would be > the same. ext3 is not a clustered filesystem. > It never expects the data > on disk to change out from under it, so it > always trusts its cache. > In order to share storage between multiple > machines, you need a clustered > filesystem, like GFS. > > -Ben > > > Grateful > > > > > > > > > > _________________________ > > Cristiano da Costa > > Consultor > > Develop IT Solutions > > www.develop.com.br > > Fone: (51) 3386-6620 > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > From bmarzins at redhat.com Fri Feb 4 19:03:09 2005 From: bmarzins at redhat.com (Benjamin Marzinski) Date: Fri, 4 Feb 2005 13:03:09 -0600 Subject: [Linux-cluster] [Fwd: any special requirement on scsi device?] In-Reply-To: <1107220639.3017.4.camel@localhost.localdomain> References: <1107220639.3017.4.camel@localhost.localdomain> Message-ID: <20050204190309.GG3666@phlogiston.msp.redhat.com> On Mon, Jan 31, 2005 at 08:17:19PM -0500, Ming Zhang wrote: > Hi folks > > I asked this question today, somebody there guided me to here. Can > anybody help me on this? Thanks. > > ps, how to subscribe on this list? Thanks. I believe you can get to a subscription page from http://sources.redhat.com/cluster > Pls cc to me, Thanks. > > Ming > > -----Forwarded Message----- > > From: Ming Zhang > > To: opengfs-devel at lists.sourceforge.net > > Subject: any special requirement on scsi device? > > Date: Mon, 31 Jan 2005 16:41:00 -0500 > > > > Hi, folks > > > > I wonder if opengfs has any special requirements on scsi deivces. For > > example, reserve/release scsi commands. > > > > the reason i ask this is we are developing a open source iscsi target > > and would like to support opengfs. since we build all scsi response, we > > need to know what extra scsi commands or feature we need to support. Persistent reservations would be really useful for fencing. -Ben > > > > thanks. > > > > > > ming > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From mingz at ele.uri.edu Fri Feb 4 19:15:00 2005 From: mingz at ele.uri.edu (Ming Zhang) Date: Fri, 04 Feb 2005 14:15:00 -0500 Subject: [Linux-cluster] [Fwd: any special requirement on scsi device?] In-Reply-To: <20050204190309.GG3666@phlogiston.msp.redhat.com> References: <1107220639.3017.4.camel@localhost.localdomain> <20050204190309.GG3666@phlogiston.msp.redhat.com> Message-ID: <1107544500.3009.10.camel@localhost.localdomain> On Fri, 2005-02-04 at 14:03, Benjamin Marzinski wrote: > On Mon, Jan 31, 2005 at 08:17:19PM -0500, Ming Zhang wrote: > > Hi folks > > > > I asked this question today, somebody there guided me to here. Can > > anybody help me on this? Thanks. > > > > ps, how to subscribe on this list? Thanks. > > I believe you can get to a subscription page from > http://sources.redhat.com/cluster thx a lot. though it is very slow to connect. :P > > > Pls cc to me, Thanks. > > > > Ming > > > > -----Forwarded Message----- > > > From: Ming Zhang > > > To: opengfs-devel at lists.sourceforge.net > > > Subject: any special requirement on scsi device? > > > Date: Mon, 31 Jan 2005 16:41:00 -0500 > > > > > > Hi, folks > > > > > > I wonder if opengfs has any special requirements on scsi deivces. For > > > example, reserve/release scsi commands. > > > > > > the reason i ask this is we are developing a open source iscsi target > > > and would like to support opengfs. since we build all scsi response, we > > > need to know what extra scsi commands or feature we need to support. > > Persistent reservations would be really useful for fencing. u mean this is "useful". but is this "enough"? because many end users on our list ask for this feature. so i want to know what we need to do in order to get it done. thanks. ps, what is the minimum # of pc i need in order to install a gfs and test? ming > > -Ben > > > > > > > thanks. > > > > > > > > > ming > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster From tony at sybaspace.com Fri Feb 4 19:20:15 2005 From: tony at sybaspace.com (Tony Fraser) Date: Fri, 04 Feb 2005 11:20:15 -0800 Subject: [Linux-cluster] GNBD & Network Outage In-Reply-To: <42034DDC.9010002@pixexcel.co.uk> References: <41F622D8.1060102@pixexcel.co.uk> <20050125205538.GD13289@phlogiston.msp.redhat.com> <42034DDC.9010002@pixexcel.co.uk> Message-ID: <1107544814.1863.33.camel@sybaws1.office.sybaspace.com> On Fri, 2005-02-04 at 02:26, Nigel Jewell wrote: > The intention of the setup was to have two hosts both exporting an > unmounted device, and the alternative device using it as a RAID-1 > device. Then to use heartbeat to mount and unmount the partitions as > required. For example: > > HOST A: > > /dev/hda1 (md0, ext3, mounted) > /dev/hda2 (ext3, unmounted, gnbd_exported as A) > /dev/gnbd/B (md0, ext3, mounted) > > HOST B: > > /dev/hda1 (ext3, unmounted, gnbd_exported as B) > /dev/hda2 (md0, ext3, mounted) > /dev/gnbd/A (md0, ext3, mounted) > > I hope that makes sense. > > If so, does what we are trying to achieve sound sensible? Any > gotchas/advice? That sounds to me just like what DRBD (http://www.drbd.org/) was deigned to do. You might want to take a look at it. -- Tony Fraser tony at sybaspace.com Sybaspace Internet Solutions System Administrator phone: (250) 246-5368 fax: (250) 246-5398 From bmarzins at redhat.com Fri Feb 4 20:00:47 2005 From: bmarzins at redhat.com (Benjamin Marzinski) Date: Fri, 4 Feb 2005 14:00:47 -0600 Subject: [Linux-cluster] clvm mirroring target status In-Reply-To: <1107535951.14182.75.camel@wavebreaker.eccent.be> References: <1107535951.14182.75.camel@wavebreaker.eccent.be> Message-ID: <20050204200047.GH3666@phlogiston.msp.redhat.com> On Fri, Feb 04, 2005 at 05:52:31PM +0100, Filip Sergeys wrote: > Hi, > > We are going to install a linux cluster with 2 gnbd servers (no SPOF) > and gfs + clvm on the cluster nodes (4 nodes). I have two options, if I > read the docs well, for duplicating data on the gnbd servers: > 1) using clvm target mirroring on the cluster nodes > 2) use drbd underneath to mirror discs. Basically two disks per machine: > 1 live disk which is mirrored with drbd to the second disk in the second > machine and the other way around in the second machine > (so the second disk in the first machine is thus the mirror from the > first (="live") disk in the second machine(sounds complicated, but it is > just hard to write down)). > Both live disks from each machine will be combined as one logical disk > (If I understood well, this is possible). > > Question: what is the status of clvm mirroring? Is it stable? > Suppose it is stable, so I have a choice: which one of the options would > any of you choose? Reason? (Stability, performance, ...) I'm still not sure if cluster mirroring is available for testing (I don't think that it is). It's defintely not considered stable. I'm also sort of unsure about your drbd solution. As far as I know, drbd only allows write access on one node at a time. So, if the first machine uses drbd to write to a local device and one on the second machine, the second machine cannot write to that device. drbd is only useful for active passive setups. If you are using pool multipathing to multipath between the two gnbd servers, you could set it to failover mode, and modify the fencing agent that you are using to fence the gnbd_server, to make it tell drbd to fail over when you fence the server. I have never tried this, but it seems reasonable. One issue would be how to bring the failed server back up, since the devices are going to be out of sync. http://www.drbd.org/start.html says that drbd still only allows write access to one node at a time. sorry :( -Ben > I found two hits on google concerning clvm mirroring, but both say it is > not finished yet. However the most recent one is from june 2004. > I cannot test either because we have no spare machine. I'm going to buy > two machine so I need to know which disk configuration I will be using. > > Thanks in advance, > > Regards, > > Filip Sergeys > > > > http://64.233.183.104/search?q=cache:r1Icx--aI2YJ:www.spinics.net/lists/gfs/msg03439.html+clvm+mirroring+gfs&hl=nl&start=12 > https://www.redhat.com/archives/linux-cluster/2004-June/msg00028.html > > -- > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > * System Engineer, Verzekeringen NV * > * www.verzekeringen.be * > * Oostkaai 23 B-2170 Merksem * > * 03/6416673 - 0477/340942 * > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From lhh at redhat.com Fri Feb 4 20:02:16 2005 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 04 Feb 2005 15:02:16 -0500 Subject: [Linux-cluster] gnbd server as shared storage In-Reply-To: <1002123.1107541275046.JavaMail.SYSTEM@serraria> References: <1002123.1107541275046.JavaMail.SYSTEM@serraria> Message-ID: <1107547336.14020.155.camel@ayanami.boston.redhat.com> On Fri, 2005-02-04 at 16:21 -0200, Cristiano da Costa wrote: > I update the clumager package to last version, and now the nodes aren't rebooting when start clumanager, but the quorum aren't not recognized and in this case Im using 2 raws that bind to 2 gnbd imported devices. They're not used for quorum; you're using an IP tiebreaker. > Feb 4 16:23:06 parati cluquorumd[5303]: Lack of Quorum Maintained > Feb 4 16:23:06 parati clulockd[5316]: Quorum Event: NO QUORUM > Feb 4 16:23:06 parati cluquorumd[5303]: VF: pid 5321 exited, status 0 > Feb 4 16:23:06 parati clulockd[5316]: I am the new lock keeper Try running 'cluforce' or booting both nodes (see 'man cluforce' and 'man cludb' for more details). -- Lon From lhh at redhat.com Fri Feb 4 20:03:37 2005 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 04 Feb 2005 15:03:37 -0500 Subject: [Linux-cluster] a question about gfs file system In-Reply-To: <000901c50ae3$feb22970$69050364@yazanz> References: <000901c50ae3$feb22970$69050364@yazanz> Message-ID: <1107547417.14020.157.camel@ayanami.boston.redhat.com> On Fri, 2005-02-04 at 20:04 +0200, Yazan Al-Sheyyab wrote: > hello everybody, > > how much space that gfs file system can take from a partion ? > > i mean that if i have a 500 MB partition and want to format it as gfs file > system , so how much space will remain for me in this partition after format > it as gfs file system? Not sure of the exact metadata overhead, but it varies depending how many journals you create (i.e. how many nodes will access the file system). -- Lon From bmarzins at redhat.com Fri Feb 4 20:20:31 2005 From: bmarzins at redhat.com (Benjamin Marzinski) Date: Fri, 4 Feb 2005 14:20:31 -0600 Subject: [Linux-cluster] [Fwd: any special requirement on scsi device?] In-Reply-To: <1107544500.3009.10.camel@localhost.localdomain> References: <1107220639.3017.4.camel@localhost.localdomain> <20050204190309.GG3666@phlogiston.msp.redhat.com> <1107544500.3009.10.camel@localhost.localdomain> Message-ID: <20050204202031.GI3666@phlogiston.msp.redhat.com> On Fri, Feb 04, 2005 at 02:15:00PM -0500, Ming Zhang wrote: > On Fri, 2005-02-04 at 14:03, Benjamin Marzinski wrote: > > On Mon, Jan 31, 2005 at 08:17:19PM -0500, Ming Zhang wrote: > > > Hi folks > > > > > > I asked this question today, somebody there guided me to here. Can > > > anybody help me on this? Thanks. > > > > > > ps, how to subscribe on this list? Thanks. > > > > I believe you can get to a subscription page from > > http://sources.redhat.com/cluster > thx a lot. though it is very slow to connect. :P > > > > > > > Pls cc to me, Thanks. > > > > > > Ming > > > > > > -----Forwarded Message----- > > > > From: Ming Zhang > > > > To: opengfs-devel at lists.sourceforge.net > > > > Subject: any special requirement on scsi device? > > > > Date: Mon, 31 Jan 2005 16:41:00 -0500 > > > > > > > > Hi, folks > > > > > > > > I wonder if opengfs has any special requirements on scsi deivces. For > > > > example, reserve/release scsi commands. > > > > > > > > the reason i ask this is we are developing a open source iscsi target > > > > and would like to support opengfs. since we build all scsi response, we > > > > need to know what extra scsi commands or feature we need to support. > > > > Persistent reservations would be really useful for fencing. > u mean this is "useful". but is this "enough"? We have a fencing agent based on SCSI 3 persistent reservation. So if you implement persistent reservation, that should be enough for GFS fencing purposes. > because many end users on our list ask for this feature. so i want to > know what we need to do in order to get it done. thanks. > > ps, what is the minimum # of pc i need in order to install a gfs and > test? two machines. I'm not sure that we have made the persistent reservation fencing agent generally available, but if you would like to test it, we can defintely do that. -Ben > ming > > > > > -Ben > > > > > > > > > > thanks. > > > > > > > > > > > > ming > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > http://www.redhat.com/mailman/listinfo/linux-cluster From mingz at ele.uri.edu Fri Feb 4 21:11:29 2005 From: mingz at ele.uri.edu (Ming Zhang) Date: Fri, 04 Feb 2005 16:11:29 -0500 Subject: [Linux-cluster] [Fwd: any special requirement on scsi device?] In-Reply-To: <20050204202031.GI3666@phlogiston.msp.redhat.com> References: <1107220639.3017.4.camel@localhost.localdomain> <20050204190309.GG3666@phlogiston.msp.redhat.com> <1107544500.3009.10.camel@localhost.localdomain> <20050204202031.GI3666@phlogiston.msp.redhat.com> Message-ID: <1107551489.3003.9.camel@localhost.localdomain> Thanks so much for the information. This is really helpful. Ming On Fri, 2005-02-04 at 15:20, Benjamin Marzinski wrote: > On Fri, Feb 04, 2005 at 02:15:00PM -0500, Ming Zhang wrote: > > On Fri, 2005-02-04 at 14:03, Benjamin Marzinski wrote: > > > On Mon, Jan 31, 2005 at 08:17:19PM -0500, Ming Zhang wrote: > > > > Hi folks > > > > > > > > I asked this question today, somebody there guided me to here. Can > > > > anybody help me on this? Thanks. > > > > > > > > ps, how to subscribe on this list? Thanks. > > > > > > I believe you can get to a subscription page from > > > http://sources.redhat.com/cluster > > thx a lot. though it is very slow to connect. :P > > > > > > > > > > > Pls cc to me, Thanks. > > > > > > > > Ming > > > > > > > > -----Forwarded Message----- > > > > > From: Ming Zhang > > > > > To: opengfs-devel at lists.sourceforge.net > > > > > Subject: any special requirement on scsi device? > > > > > Date: Mon, 31 Jan 2005 16:41:00 -0500 > > > > > > > > > > Hi, folks > > > > > > > > > > I wonder if opengfs has any special requirements on scsi deivces. For > > > > > example, reserve/release scsi commands. > > > > > > > > > > the reason i ask this is we are developing a open source iscsi target > > > > > and would like to support opengfs. since we build all scsi response, we > > > > > need to know what extra scsi commands or feature we need to support. > > > > > > Persistent reservations would be really useful for fencing. > > u mean this is "useful". but is this "enough"? > > We have a fencing agent based on SCSI 3 persistent reservation. So if you > implement persistent reservation, that should be enough for GFS fencing > purposes. > > > because many end users on our list ask for this feature. so i want to > > know what we need to do in order to get it done. thanks. > > > > ps, what is the minimum # of pc i need in order to install a gfs and > > test? > > two machines. I'm not sure that we have made the persistent reservation > fencing agent generally available, but if you would like to test it, we can > defintely do that. > > -Ben > > > ming > > > > > > > > -Ben > > > > > > > > > > > > > thanks. > > > > > > > > > > > > > > > ming > > > > > > > > -- > > > > Linux-cluster mailing list > > > > Linux-cluster at redhat.com > > > > http://www.redhat.com/mailman/listinfo/linux-cluster From alewis at redhat.com Fri Feb 4 21:32:26 2005 From: alewis at redhat.com (AJ Lewis) Date: Fri, 04 Feb 2005 15:32:26 -0600 (CST) Subject: [Linux-cluster] Re: any special requirement on scsi device? In-Reply-To: <1107551942.3003.19.camel@localhost.localdomain> References: <1107551942.3003.19.camel@localhost.localdomain> Message-ID: <20050204.153226.59659031.alewis@redhat.com> From: Ming Zhang Subject: [Iscsitarget-devel] [Fwd: Re: [Linux-cluster] [Fwd: any special requirement on scsi device?]] Date: Fri, 04 Feb 2005 16:19:02 -0500 > This is what I got from linux-cluster list. > > We need to have the persistent reserve support in order to support GFS. Ming, Persistant reservations *can* be used for fencing with GFS, but it is not *required* for use with GFS. AFAIK, there are no SCSI specific commands that GFS requires at this time. Regards, - AJ Lewis Voice: 612-638-0500 Red Hat Inc. E-Mail: alewis at redhat.com 720 Washington Ave. SE, Suite 200 Minneapolis, MN 55414 Current GPG fingerprint = D9F8 EDCE 4242 855F A03D 9B63 F50C 54A8 578C 8715 Grab the key at: http://people.redhat.com/alewis/gpg.html or one of the many keyservers out there... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From mingz at ele.uri.edu Fri Feb 4 21:41:42 2005 From: mingz at ele.uri.edu (Ming Zhang) Date: Fri, 04 Feb 2005 16:41:42 -0500 Subject: [Linux-cluster] Re: any special requirement on scsi device? In-Reply-To: <20050204.153226.59659031.alewis@redhat.com> References: <1107551942.3003.19.camel@localhost.localdomain> <20050204.153226.59659031.alewis@redhat.com> Message-ID: <1107553301.3003.30.camel@localhost.localdomain> hehe. confused. so this is optional, but not mandatory. but how gfs decide whether to use it? seems u are on this list as well. so do u have experience on iet+gfs? ming On Fri, 2005-02-04 at 16:32, AJ Lewis wrote: > From: Ming Zhang > Subject: [Iscsitarget-devel] [Fwd: Re: [Linux-cluster] [Fwd: any special requirement on scsi device?]] > Date: Fri, 04 Feb 2005 16:19:02 -0500 > > > This is what I got from linux-cluster list. > > > > We need to have the persistent reserve support in order to support GFS. > > Ming, > > Persistant reservations *can* be used for fencing with GFS, but it is > not *required* for use with GFS. AFAIK, there are no SCSI specific > commands that GFS requires at this time. > > Regards, > - > AJ Lewis Voice: 612-638-0500 > Red Hat Inc. E-Mail: alewis at redhat.com > 720 Washington Ave. SE, Suite 200 > Minneapolis, MN 55414 > > Current GPG fingerprint = D9F8 EDCE 4242 855F A03D 9B63 F50C 54A8 578C 8715 > Grab the key at: http://people.redhat.com/alewis/gpg.html or one of the > many keyservers out there... From alewis at redhat.com Fri Feb 4 21:55:43 2005 From: alewis at redhat.com (AJ Lewis) Date: Fri, 04 Feb 2005 15:55:43 -0600 (CST) Subject: [Linux-cluster] Re: any special requirement on scsi device? In-Reply-To: <1107553301.3003.30.camel@localhost.localdomain> References: <1107551942.3003.19.camel@localhost.localdomain> <20050204.153226.59659031.alewis@redhat.com> <1107553301.3003.30.camel@localhost.localdomain> Message-ID: <20050204.155543.21920164.alewis@redhat.com> From: Ming Zhang Subject: Re: any special requirement on scsi device? Date: Fri, 04 Feb 2005 16:41:42 -0500 > hehe. confused. > > so this is optional, but not mandatory. but how gfs decide whether to > use it? There are cluster configuration file(s) that the cluster infrastructure on uses to handle this. GFS itself (the filesystem) relies on the cluster infrastructure to handle things like fencing. > seems u are on this list as well. so do u have experience on iet+gfs? I've not used iet yet, but I have run GFS on iSCSI with hardware iSCSI targets and the linux-iscsi initiator from sourceforge. Regards, -- AJ Lewis Voice: 612-638-0500 Red Hat Inc. E-Mail: alewis at redhat.com 720 Washington Ave. SE, Suite 200 Minneapolis, MN 55414 Current GPG fingerprint = D9F8 EDCE 4242 855F A03D 9B63 F50C 54A8 578C 8715 Grab the key at: http://people.redhat.com/alewis/gpg.html or one of the many keyservers out there... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From mingz at ele.uri.edu Fri Feb 4 22:01:27 2005 From: mingz at ele.uri.edu (Ming Zhang) Date: Fri, 04 Feb 2005 17:01:27 -0500 Subject: [Linux-cluster] Re: any special requirement on scsi device? In-Reply-To: <20050204.155543.21920164.alewis@redhat.com> References: <1107551942.3003.19.camel@localhost.localdomain> <20050204.153226.59659031.alewis@redhat.com> <1107553301.3003.30.camel@localhost.localdomain> <20050204.155543.21920164.alewis@redhat.com> Message-ID: <1107554486.3003.38.camel@localhost.localdomain> On Fri, 2005-02-04 at 16:55, AJ Lewis wrote: > From: Ming Zhang > Subject: Re: any special requirement on scsi device? > Date: Fri, 04 Feb 2005 16:41:42 -0500 > > > hehe. confused. > > > > so this is optional, but not mandatory. but how gfs decide whether to > > use it? > > There are cluster configuration file(s) that the cluster > infrastructure on uses to handle this. GFS itself (the filesystem) > relies on the cluster infrastructure to handle things like fencing. ic. thx. > > > seems u are on this list as well. so do u have experience on iet+gfs? > > I've not used iet yet, but I have run GFS on iSCSI with hardware iSCSI targets and the linux-iscsi initiator from sourceforge. it is fine. i am thinking if i can run GFS on 2 linux running in vmware. > > Regards, > -- > AJ Lewis Voice: 612-638-0500 > Red Hat Inc. E-Mail: alewis at redhat.com > 720 Washington Ave. SE, Suite 200 > Minneapolis, MN 55414 > > Current GPG fingerprint = D9F8 EDCE 4242 855F A03D 9B63 F50C 54A8 578C 8715 > Grab the key at: http://people.redhat.com/alewis/gpg.html or one of the > many keyservers out there... From yazan at ccs.com.jo Sat Feb 5 06:43:06 2005 From: yazan at ccs.com.jo (Yazan Al-Sheyyab) Date: Sat, 5 Feb 2005 08:43:06 +0200 Subject: [Linux-cluster] a question about gfs file system References: <000901c50ae3$feb22970$69050364@yazanz> <1107547417.14020.157.camel@ayanami.boston.redhat.com> Message-ID: <005a01c50b4d$f310d0c0$69050364@yazanz> Hi, OK if not sure , please can you tell me the minimum space for a partition to put if i want to format it as gfs ? because i am using ORACLE and i need to make raw devices and i will give each raw device a partition and some raw devices have a space of 50 MB and that in not enough for gfs , so if didi not sure about the space for gfs , please can you tell me the minimum space that i can use to be in a safe side for gfs file system. THANKS. ----- Original Message ----- From: "Lon Hohberger" To: "linux clistering" Sent: Friday, February 04, 2005 10:03 PM Subject: Re: [Linux-cluster] a question about gfs file system > On Fri, 2005-02-04 at 20:04 +0200, Yazan Al-Sheyyab wrote: > > hello everybody, > > > > how much space that gfs file system can take from a partion ? > > > > i mean that if i have a 500 MB partition and want to format it as gfs file > > system , so how much space will remain for me in this partition after format > > it as gfs file system? > > Not sure of the exact metadata overhead, but it varies depending how > many journals you create (i.e. how many nodes will access the file > system). > > -- Lon > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From filip.sergeys at verzekeringen.be Sat Feb 5 09:55:13 2005 From: filip.sergeys at verzekeringen.be (Filip Sergeys) Date: Sat, 5 Feb 2005 10:55:13 +0100 Subject: [Linux-cluster] clvm mirroring target status In-Reply-To: <20050204200047.GH3666@phlogiston.msp.redhat.com> References: <1107535951.14182.75.camel@wavebreaker.eccent.be> <20050204200047.GH3666@phlogiston.msp.redhat.com> Message-ID: <200502051055.13507.filip.sergeys@verzekeringen.be> I'll try to explain it a bit more structured Host A -------- Disk A Disk Bm (mirrored disk of disk B in host B, unmounted) Host B -------- Disk B Disk Am (mirrored disk of disk A in host A, unmounted) Normal working situation: --------------------------------- Disk A and Disk B are exported with GNBD. If I understood well, I can combine them into one logical disk for the clusternodes with clvm (striped maybe, don't know, need to read more about it). Disk Am and Bm are basically only used as mirroring for A and B. THis is done with drbd. So they are not taking part in rw actions in any way. Host B goes down: ------------------------ Heartbeat says it is down, I cut the power. This is what I think needs to be done: -Heartbeat moves the virtual IP address of host B to Host A. This is the IP address by which disk B was exported -Mount disk Bm read/write. -Export Bm with GNBD. The clusters should now be able to continue working, I think transparently (need to test that to know). Concequences: ------------------- Bringing host B back in the game needs a manual intervention. -Basically al services on the cluster nodes need to stop writing. -Sync the disk from Bm to B -Give host B back its virtual ip address -mount B read/write -umount Bm in host A -start all services again on the nodes. => I know this is not perfect. But we can live with that. This will need to happen after office hours. The thing is that we don't have the budget for shared storage and certainly not for a redundant shared storage solution because most entry level shared storages are SPOFs. I need to find out more about that multipathing. I am not sure how to use it in this configuration. If you have idea's for improvement, they are welcome. Regards, Filip PS. Thanx for your answer on the clvm mirroring state. On Friday 04 February 2005 21:00, Benjamin Marzinski wrote: > On Fri, Feb 04, 2005 at 05:52:31PM +0100, Filip Sergeys wrote: > > Hi, > > > > We are going to install a linux cluster with 2 gnbd servers (no SPOF) > > and gfs + clvm on the cluster nodes (4 nodes). I have two options, if I > > read the docs well, for duplicating data on the gnbd servers: > > 1) using clvm target mirroring on the cluster nodes > > 2) use drbd underneath to mirror discs. Basically two disks per machine: > > 1 live disk which is mirrored with drbd to the second disk in the second > > machine and the other way around in the second machine > > (so the second disk in the first machine is thus the mirror from the > > first (="live") disk in the second machine(sounds complicated, but it is > > just hard to write down)). > > Both live disks from each machine will be combined as one logical disk > > (If I understood well, this is possible). > > > > Question: what is the status of clvm mirroring? Is it stable? > > Suppose it is stable, so I have a choice: which one of the options would > > any of you choose? Reason? (Stability, performance, ...) > > I'm still not sure if cluster mirroring is available for testing (I don't > think that it is). It's defintely not considered stable. > > I'm also sort of unsure about your drbd solution. > As far as I know, drbd only allows write access on one node at a time. So, > if the first machine uses drbd to write to a local device and one on the > second machine, the second machine cannot write to that device. drbd is > only useful for active passive setups. If you are using pool multipathing > to multipath between the two gnbd servers, you could set it to failover > mode, and modify the fencing agent that you are using to fence the > gnbd_server, to make it tell drbd to fail over when you fence the server. > > I have never tried this, but it seems reasonable. One issue would be how to > bring the failed server back up, since the devices are going to be out of > sync. > > http://www.drbd.org/start.html says that drbd still only allows write > access to one node at a time. > > sorry :( > > -Ben > > > I found two hits on google concerning clvm mirroring, but both say it is > > not finished yet. However the most recent one is from june 2004. > > I cannot test either because we have no spare machine. I'm going to buy > > two machine so I need to know which disk configuration I will be using. > > > > Thanks in advance, > > > > Regards, > > > > Filip Sergeys > > > > > > > > http://64.233.183.104/search?q=cache:r1Icx--aI2YJ:www.spinics.net/lists/g > >fs/msg03439.html+clvm+mirroring+gfs&hl=nl&start=12 > > https://www.redhat.com/archives/linux-cluster/2004-June/msg00028.html > > > > -- > > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > > * System Engineer, Verzekeringen NV * > > * www.verzekeringen.be * > > * Oostkaai 23 B-2170 Merksem * > > * 03/6416673 - 0477/340942 * > > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From bastian at waldi.eu.org Mon Feb 7 10:09:49 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Mon, 7 Feb 2005 11:09:49 +0100 Subject: [Linux-cluster] [PATCH] dlm: Never link with ld -- Don't install headers executable Message-ID: <20050207100949.GA32586@wavehammer.waldi.eu.org> The following patch fixes two bugs in dlm. - ld is used for linking which regulary fails. - Headers are installed executable. Bastian -- Those who hate and fight must stop themselves -- otherwise it is not stopped. -- Spock, "Day of the Dove", stardate unknown -------------- next part -------------- --- dlm/lib/Makefile 2005-02-06 18:53:03.000000000 +0100 +++ dlm/lib/Makefile 2005-02-06 19:28:17.000000000 +0100 @@ -46,7 +46,7 @@ ${RANLIB} libdlm_lt.a $(LIBNAME).so.${RELEASE_MAJOR}.${RELEASE_MINOR}: libdlm.po libaislock.po - $(LD) -shared -o $@ -soname=$(LIBNAME).so.$(RELEASE_MAJOR) $^ + $(CC) -shared -o $@ -Wl,-soname=$(LIBNAME).so.$(RELEASE_MAJOR) $^ $(LIBNAME)_lt.so.${RELEASE_MAJOR}.${RELEASE_MINOR}: libdlm_lt.po $(CC) -shared -o $@ -Wl,-soname=$(LIBNAME)_lt.so.$(RELEASE_MAJOR) $^ @@ -74,7 +74,7 @@ install: all install -d ${incdir} - install libdlm.h ${incdir} + install -m644 libdlm.h ${incdir} install -d ${libdir} install $(LIBNAME).a ${libdir} install $(LIBNAME)_lt.a ${libdir} -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From pcaulfie at redhat.com Mon Feb 7 10:35:29 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Mon, 7 Feb 2005 10:35:29 +0000 Subject: [Linux-cluster] [PATCH] dlm: Never link with ld -- Don't install headers executable In-Reply-To: <20050207100949.GA32586@wavehammer.waldi.eu.org> References: <20050207100949.GA32586@wavehammer.waldi.eu.org> Message-ID: <20050207103529.GC25122@tykepenguin.com> On Mon, Feb 07, 2005 at 11:09:49AM +0100, Bastian Blank wrote: > The following patch fixes two bugs in dlm. > - ld is used for linking which regulary fails. > - Headers are installed executable. > Applied. Thanks. -- patrick From mtilstra at redhat.com Mon Feb 7 14:09:30 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Mon, 7 Feb 2005 08:09:30 -0600 Subject: [Linux-cluster] a question about gfs file system In-Reply-To: <005a01c50b4d$f310d0c0$69050364@yazanz> References: <000901c50ae3$feb22970$69050364@yazanz> <1107547417.14020.157.camel@ayanami.boston.redhat.com> <005a01c50b4d$f310d0c0$69050364@yazanz> Message-ID: <20050207140930.GA3020@redhat.com> On Sat, Feb 05, 2005 at 08:43:06AM +0200, Yazan Al-Sheyyab wrote: > Hi, > > OK if not sure , please can you tell me the minimum space for a partition > to put if i want to format it as gfs ? it varies. by default, each journal is 128M, but you can change that with mkfs.gfs. The minimum is 32M. The rest of the meta data? I duno, but it is rather dwarfed by the size of the journals. I'm not sure there has been much, if any, testing with devices smaller than a gigabyte. -- Michael Conrad Tadpol Tilstra Discretion will protect you, and understanding will guard you. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From alain at tessiot.info Mon Feb 7 08:56:20 2005 From: alain at tessiot.info (Alain TESSIOT) Date: Mon, 7 Feb 2005 09:56:20 +0100 Subject: [Linux-cluster] Cluster Suite an network interfaces Message-ID: <03ee01c50cf2$f90cc5f0$1b00a8c0@owtessiotalain> hi all, Excuse me for my bad english, I am french. I have two servers each one has four network interfaces. Server 1 : two bonded for the network : 222.10.3.1, two bonded for the cluster network : 192.168.0.1. Server 2 : two bonded for the network : 222.10.3.2, two bonded for the cluster network : 192.168.0.2. I have two rawdevices that works fine and a shared file system. I try to make a cluster for httpd which ip is 222.10.3.10. Everything works fine : if I stop the server 1, httpd, the virtual ip and the file system strats to run on the server 2. But, if httpd runs on the the Server 1and I unplug the 2 network interfaces (222.10.3.1), nothing happens .... Did I do something wrong ? Many thanks for your help Alain -------------- next part -------------- An HTML attachment was scrubbed... URL: From lhh at redhat.com Mon Feb 7 15:04:30 2005 From: lhh at redhat.com (Lon Hohberger) Date: Mon, 07 Feb 2005 10:04:30 -0500 Subject: [Linux-cluster] Cluster Suite an network interfaces In-Reply-To: <03ee01c50cf2$f90cc5f0$1b00a8c0@owtessiotalain> References: <03ee01c50cf2$f90cc5f0$1b00a8c0@owtessiotalain> Message-ID: <1107788670.9794.2.camel@ayanami.boston.redhat.com> On Mon, 2005-02-07 at 09:56 +0100, Alain TESSIOT wrote: > But, if httpd runs on the the Server 1and I unplug the 2 network > interfaces (222.10.3.1), nothing happens .... > Did I do something wrong ? See: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=144488 -- Lon From erwan at seanodes.com Mon Feb 7 16:59:01 2005 From: erwan at seanodes.com (Velu Erwan) Date: Mon, 07 Feb 2005 17:59:01 +0100 Subject: [Linux-cluster] About tuning GFS Message-ID: <1107795541.32581.28.camel@R1.seanodes.com> Hi folks, I've been trying to understand the way I can tune my GFS 6.0.2 configuration. I've found I can play using the setflag options which can change the exhash, jdata, unused, ea_indirect, directio, inherit_directio & inherit_jdata I know about direct_io and its inherit_directio regarding http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-manage-direct-io.html jdata are also decribes by http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-manage-data-journal.html But I didn't find anything for the other values. What are theses options ? I have the same questions regarding the gettune option which print a lots of value which seems undocumented. Could someone give me some help for understanding their behaviours ? Regards, -- Erwan Velu Consultant - Seanodes SA http://www.seanodes.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From danderso at redhat.com Mon Feb 7 17:36:54 2005 From: danderso at redhat.com (Derek Anderson) Date: Mon, 7 Feb 2005 11:36:54 -0600 Subject: [Linux-cluster] About tuning GFS In-Reply-To: <1107795541.32581.28.camel@R1.seanodes.com> References: <1107795541.32581.28.camel@R1.seanodes.com> Message-ID: <200502071136.54157.danderso@redhat.com> On Monday 07 February 2005 10:59, Velu Erwan wrote: > Hi folks, > I've been trying to understand the way I can tune my GFS 6.0.2 > configuration. > I've found I can play using the setflag options which can change the > exhash, jdata, unused, ea_indirect, directio, inherit_directio & > inherit_jdata > > I know about direct_io and its inherit_directio regarding > http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-manage-direct-io.ht >ml > > jdata are also decribes by > http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-manage-data-journal >.html > See also section 9.4: http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-manage-quota.html If you are not using quotas you can turn this off which should reduce overhead some. Another thing you can try is mounting with noatime (-o noatime): http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-manage-atimeconf.html#S2-MANAGE-MOUNTNOATIME > But I didn't find anything for the other values. > What are theses options ? > > I have the same questions regarding the gettune option which print a > lots of value which seems undocumented. > > Could someone give me some help for understanding their behaviours ? > Regards, From fmarchal at inf.ethz.ch Mon Feb 7 21:41:52 2005 From: fmarchal at inf.ethz.ch (Fabrice Marchal) Date: Mon, 07 Feb 2005 22:41:52 +0100 Subject: [Linux-cluster] GFS newbie question In-Reply-To: <200502071136.54157.danderso@redhat.com> References: <1107795541.32581.28.camel@R1.seanodes.com> <200502071136.54157.danderso@redhat.com> Message-ID: <4207E0A0.5000809@inf.ethz.ch> Hello, I would like to know if it is possible to run GFS on a cluster *without* external storage, i.e. without a NAS and with each GFS node a GNBD server itself. Is it possible to implement disk redundancy in such a configuration, as pictured on Fig 1.3 of the GFS manual? The simplest configuration being: node #1: Disk A, Disk B' (mirror of B on node #2) node #2: Disk B, Disk A' (mirror of A on node #1) Thanks, Fabrice -- ======================================================================== Fabrice Marchal http://www.inf.ethz.ch/~marchal fabrice.marchal at ieee.org marchal at inf.ethz.ch +41-(0)44-632-56-79 ETH Zurich, CoLab Computational Laboratory FAX:+41-(0)44-632-17-03 ======================================================================== From bmarzins at redhat.com Mon Feb 7 21:50:17 2005 From: bmarzins at redhat.com (Benjamin Marzinski) Date: Mon, 7 Feb 2005 15:50:17 -0600 Subject: [Linux-cluster] clvm mirroring target status In-Reply-To: <200502051055.13507.filip.sergeys@verzekeringen.be> References: <1107535951.14182.75.camel@wavebreaker.eccent.be> <20050204200047.GH3666@phlogiston.msp.redhat.com> <200502051055.13507.filip.sergeys@verzekeringen.be> Message-ID: <20050207215017.GK3666@phlogiston.msp.redhat.com> On Sat, Feb 05, 2005 at 10:55:13AM +0100, Filip Sergeys wrote: > I'll try to explain it a bit more structured > > Host A > -------- > Disk A > Disk Bm (mirrored disk of disk B in host B, unmounted) > > Host B > -------- > Disk B > Disk Am (mirrored disk of disk A in host A, unmounted) > > > Normal working situation: > --------------------------------- > Disk A and Disk B are exported with GNBD. If I understood well, I can combine > them into one logical disk for the clusternodes with clvm (striped maybe, > don't know, need to read more about it). > Disk Am and Bm are basically only used as mirroring for A and B. THis is done > with drbd. So they are not taking part in rw actions in any way. > > Host B goes down: > ------------------------ > Heartbeat says it is down, I cut the power. > This is what I think needs to be done: > -Heartbeat moves the virtual IP address of host B to Host A. This is the IP > address by which disk B was exported > -Mount disk Bm read/write. > -Export Bm with GNBD. The clusters should now be able to continue working, I > think transparently (need to test that to know). O.k. The way you plan to use drbd makes sense. The only issue is this: GFS doesn't use Heartbeat, the cluster manager does its own heartbeating. If you have two different heartbeating mechanisms controlling failover, things won't fail over all at once. Ideally, for all the stuff below the filesystem layer, including gnbd, you wouldn't use Heartbeat at all, but simply rely on the cluster manager. To do this, you would have to make drbd switch over when the cluster manager detected a node failure. This could be done by hacking a fencing agent, as I mentioned in my previous e-mail. If you must use heartbeat for the block device failover, you need to recoginze that this could happen before, during, or after the gfs failover, which may (probably will) cause problems occasionally. Unfortunately, I'm not sure that your multipathing setup will work. I am assuming that you are using pool for the multipathing. pool multipathing has two modes, round-robin, and failover. Obviously round-robin (which is where pool uses all the paths) won't work, because you only have one path available at once. However, failover mode probably won't work either, in the setup you explained. You would need to force pool to use Disk A from host A and Disk B from host B. Getting that to work right is probably possible, but not easy or reliable. The easiest way to do it is to make host A to have both disk A and disk B, and make host B have disk Am and Bm. To do this, GNBD import the disks from host A, assemble the pool, GNBD import the disks from host B, and use pool_mp to integrate them into the pool. This should automatically set you up in failover mode, with disks A and B as the primary disks and disks Am and Bm as the backups. I realize that this means that hostB is usually sitting idle. If you name your devices correctly, or import them in a specific order, you might be able to get pool to use the correct devices in the setup you described, but I'm not certain. What your design actually wants is for pool to not do multipathing at all, but to simply retry on failed IO. That way, when the virtual IP switches, gnbd will just automatically pick up the device at its new location. Unfortunately, pool and gnbd cannot do this. -Ben > Concequences: > ------------------- > Bringing host B back in the game needs a manual intervention. > -Basically al services on the cluster nodes need to stop writing. > -Sync the disk from Bm to B > -Give host B back its virtual ip address > -mount B read/write > -umount Bm in host A > -start all services again on the nodes. > => I know this is not perfect. But we can live with that. This will need to > happen after office hours. The thing is that we don't have the budget for > shared storage and certainly not for a redundant shared storage solution > because most entry level shared storages are SPOFs. > > I need to find out more about that multipathing. I am not sure how to use it > in this configuration. > If you have idea's for improvement, they are welcome. > > Regards, > > Filip > > PS. Thanx for your answer on the clvm mirroring state. > > > > > > > On Friday 04 February 2005 21:00, Benjamin Marzinski wrote: > > On Fri, Feb 04, 2005 at 05:52:31PM +0100, Filip Sergeys wrote: > > > Hi, > > > > > > We are going to install a linux cluster with 2 gnbd servers (no SPOF) > > > and gfs + clvm on the cluster nodes (4 nodes). I have two options, if I > > > read the docs well, for duplicating data on the gnbd servers: > > > 1) using clvm target mirroring on the cluster nodes > > > 2) use drbd underneath to mirror discs. Basically two disks per machine: > > > 1 live disk which is mirrored with drbd to the second disk in the second > > > machine and the other way around in the second machine > > > (so the second disk in the first machine is thus the mirror from the > > > first (="live") disk in the second machine(sounds complicated, but it is > > > just hard to write down)). > > > Both live disks from each machine will be combined as one logical disk > > > (If I understood well, this is possible). > > > > > > Question: what is the status of clvm mirroring? Is it stable? > > > Suppose it is stable, so I have a choice: which one of the options would > > > any of you choose? Reason? (Stability, performance, ...) > > > > I'm still not sure if cluster mirroring is available for testing (I don't > > think that it is). It's defintely not considered stable. > > > > I'm also sort of unsure about your drbd solution. > > As far as I know, drbd only allows write access on one node at a time. So, > > if the first machine uses drbd to write to a local device and one on the > > second machine, the second machine cannot write to that device. drbd is > > only useful for active passive setups. If you are using pool multipathing > > to multipath between the two gnbd servers, you could set it to failover > > mode, and modify the fencing agent that you are using to fence the > > gnbd_server, to make it tell drbd to fail over when you fence the server. > > > > I have never tried this, but it seems reasonable. One issue would be how to > > bring the failed server back up, since the devices are going to be out of > > sync. > > > > http://www.drbd.org/start.html says that drbd still only allows write > > access to one node at a time. > > > > sorry :( > > > > -Ben > > > > > I found two hits on google concerning clvm mirroring, but both say it is > > > not finished yet. However the most recent one is from june 2004. > > > I cannot test either because we have no spare machine. I'm going to buy > > > two machine so I need to know which disk configuration I will be using. > > > > > > Thanks in advance, > > > > > > Regards, > > > > > > Filip Sergeys > > > > > > > > > > > > http://64.233.183.104/search?q=cache:r1Icx--aI2YJ:www.spinics.net/lists/g > > >fs/msg03439.html+clvm+mirroring+gfs&hl=nl&start=12 > > > https://www.redhat.com/archives/linux-cluster/2004-June/msg00028.html > > > > > > -- > > > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > > > * System Engineer, Verzekeringen NV * > > > * www.verzekeringen.be * > > > * Oostkaai 23 B-2170 Merksem * > > > * 03/6416673 - 0477/340942 * > > > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From qwejohn at hotmail.com Tue Feb 8 07:58:12 2005 From: qwejohn at hotmail.com (John Que) Date: Tue, 08 Feb 2005 09:58:12 +0200 Subject: [Linux-cluster] page and buffer cache in Linux File Systems Message-ID: Hello, I am a newbie to Linux Clusters and to Clusters File Systems in particular. I am reading a book which was recently published, named: "Building Clustered Linux Systems" By Robert Lucke.(Published by Prentice Hall PTR /HP Professional Series.) (http://www.phptr.com/title/0131448536) I have a question regarding buffer or page cache on cluster file systems. I saw in Chapter 14 (page 429), the follwing text (in 14.4 Commercially Available Cluster section): "There is a set of common requirements that you'll find in all parallel file systems used in Linux Clusters... For datatbas use,you will find that the file systems needs to provide some form of direct access mode,in which the database software can read and write data blocks without using the Linux System's buffer or page cache. The database software performs this caching itself and removes the associated overhead". The author here talks about a subset of Linux Clusters File Systems, (parallel file systems);Specifically he mentioned PVFS1 and PVFS2. What I do not understand is this: Is this requirment specific to Cluster File Systems? Because as I understand, this same requirement should exist also for ordinary (non clustered) file systems like ext3. Does anybody know of a clustered file system which implements such requirement? Is there a way to configure / setup a clustered file system to avoid page/buffer cache ? (to enable what the author calls "direct access mode"). Any pointers to info on this are welcomed. And , I am not an expert in ext3 , but: in case such a requirement DOES exist for non clustered file systems (like ext3) - does ext3 implement this requirement ? How? Regards, John _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/ From filip.sergeys at verzekeringen.be Tue Feb 8 08:40:32 2005 From: filip.sergeys at verzekeringen.be (Filip Sergeys) Date: 08 Feb 2005 09:40:32 +0100 Subject: [Linux-cluster] clvm mirroring target status In-Reply-To: <20050207215017.GK3666@phlogiston.msp.redhat.com> References: <1107535951.14182.75.camel@wavebreaker.eccent.be> <20050204200047.GH3666@phlogiston.msp.redhat.com> <200502051055.13507.filip.sergeys@verzekeringen.be> <20050207215017.GK3666@phlogiston.msp.redhat.com> Message-ID: <1107852032.4838.53.camel@wavebreaker.eccent.be> On Mon, 2005-02-07 at 22:50, Benjamin Marzinski wrote: On Sat, Feb 05, 2005 at 10:55:13AM +0100, Filip Sergeys wrote: > I'll try to explain it a bit more structured > > Host A > -------- > Disk A > Disk Bm (mirrored disk of disk B in host B, unmounted) > > Host B > -------- > Disk B > Disk Am (mirrored disk of disk A in host A, unmounted) > > > Normal working situation: > --------------------------------- > Disk A and Disk B are exported with GNBD. If I understood well, I can combine > them into one logical disk for the clusternodes with clvm (striped maybe, > don't know, need to read more about it). > Disk Am and Bm are basically only used as mirroring for A and B. THis is done > with drbd. So they are not taking part in rw actions in any way. > > Host B goes down: > ------------------------ > Heartbeat says it is down, I cut the power. > This is what I think needs to be done: > -Heartbeat moves the virtual IP address of host B to Host A. This is the IP > address by which disk B was exported > -Mount disk Bm read/write. > -Export Bm with GNBD. The clusters should now be able to continue working, I > think transparently (need to test that to know). O.k. The way you plan to use drbd makes sense. The only issue is this: GFS doesn't use Heartbeat, the cluster manager does its own heartbeating. If you have two different heartbeating mechanisms controlling failover, things won't fail over all at once. Ideally, for all the stuff below the filesystem layer, including gnbd, you wouldn't use Heartbeat at all, but simply rely on the cluster manager. To do this, you would have to make drbd switch over when the cluster manager detected a node failure. This could be done by hacking a fencing agent, as I mentioned in my previous e-mail. If you must use heartbeat for the block device failover, you need to recoginze that this could happen before, during, or after the gfs failover, which may (probably will) cause problems occasionally. Unfortunately, I'm not sure that your multipathing setup will work. I am assuming that you are using pool for the multipathing. pool multipathing has two modes, round-robin, and failover. Obviously round-robin (which is where pool uses all the paths) won't work, because you only have one path available at once. However, failover mode probably won't work either, in the setup you explained. You would need to force pool to use Disk A from host A and Disk B from host B. Getting that to work right is probably possible, but not easy or reliable. The easiest way to do it is to make host A to have both disk A and disk B, and make host B have disk Am and Bm. To do this, GNBD import the disks from host A, assemble the pool, GNBD import the disks from host B, and use pool_mp to integrate them into the pool. This should automatically set you up in failover mode, with disks A and B as the primary disks and disks Am and Bm as the backups. I realize that this means that hostB is usually sitting idle. This sounds hopefull If you name your devices correctly, or import them in a specific order, you might be able to get pool to use the correct devices in the setup you described, but I'm not certain. "might be able..."-> Now I lost hope: I borrowed the idea of this setup from the GFS admin guide, "the economy and performance" setup. (http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-ov-perform.html#S2-OV-ECONOMY) Probably I misinterpreted the figure1-3, especially the disk part. Can you elaborate a bit more on: "if you name your devices correctly or import them in a specific order"? If I put Am and Bm on one machine, export them with gnbd and join them in the pool in failover mode, can I be sure there will be no writing on them? Because drbd won't let that happen. Thanx, Filip Sergeys What your design actually wants is for pool to not do multipathing at all, but to simply retry on failed IO. That way, when the virtual IP switches, gnbd will just automatically pick up the device at its new location. Unfortunately, pool and gnbd cannot do this. -Ben > Concequences: > ------------------- > Bringing host B back in the game needs a manual intervention. > -Basically al services on the cluster nodes need to stop writing. > -Sync the disk from Bm to B > -Give host B back its virtual ip address > -mount B read/write > -umount Bm in host A > -start all services again on the nodes. > => I know this is not perfect. But we can live with that. This will need to > happen after office hours. The thing is that we don't have the budget for > shared storage and certainly not for a redundant shared storage solution > because most entry level shared storages are SPOFs. > > I need to find out more about that multipathing. I am not sure how to use it > in this configuration. > If you have idea's for improvement, they are welcome. > > Regards, > > Filip > > PS. Thanx for your answer on the clvm mirroring state. > > > > > > > On Friday 04 February 2005 21:00, Benjamin Marzinski wrote: > > On Fri, Feb 04, 2005 at 05:52:31PM +0100, Filip Sergeys wrote: > > > Hi, > > > > > > We are going to install a linux cluster with 2 gnbd servers (no SPOF) > > > and gfs + clvm on the cluster nodes (4 nodes). I have two options, if I > > > read the docs well, for duplicating data on the gnbd servers: > > > 1) using clvm target mirroring on the cluster nodes > > > 2) use drbd underneath to mirror discs. Basically two disks per machine: > > > 1 live disk which is mirrored with drbd to the second disk in the second > > > machine and the other way around in the second machine > > > (so the second disk in the first machine is thus the mirror from the > > > first (="live") disk in the second machine(sounds complicated, but it is > > > just hard to write down)). > > > Both live disks from each machine will be combined as one logical disk > > > (If I understood well, this is possible). > > > > > > Question: what is the status of clvm mirroring? Is it stable? > > > Suppose it is stable, so I have a choice: which one of the options would > > > any of you choose? Reason? (Stability, performance, ...) > > > > I'm still not sure if cluster mirroring is available for testing (I don't > > think that it is). It's defintely not considered stable. > > > > I'm also sort of unsure about your drbd solution. > > As far as I know, drbd only allows write access on one node at a time. So, > > if the first machine uses drbd to write to a local device and one on the > > second machine, the second machine cannot write to that device. drbd is > > only useful for active passive setups. If you are using pool multipathing > > to multipath between the two gnbd servers, you could set it to failover > > mode, and modify the fencing agent that you are using to fence the > > gnbd_server, to make it tell drbd to fail over when you fence the server. > > > > I have never tried this, but it seems reasonable. One issue would be how to > > bring the failed server back up, since the devices are going to be out of > > sync. > > > > http://www.drbd.org/start.html says that drbd still only allows write > > access to one node at a time. > > > > sorry :( > > > > -Ben > > > > > I found two hits on google concerning clvm mirroring, but both say it is > > > not finished yet. However the most recent one is from june 2004. > > > I cannot test either because we have no spare machine. I'm going to buy > > > two machine so I need to know which disk configuration I will be using. > > > > > > Thanks in advance, > > > > > > Regards, > > > > > > Filip Sergeys > > > > > > > > > > > > http://64.233.183.104/search?q=cache:r1Icx--aI2YJ:www.spinics.net/lists/g > > >fs/msg03439.html+clvm+mirroring+gfs&hl=nl&start=12 > > > https://www.redhat.com/archives/linux-cluster/2004-June/msg00028.html > > > > > > -- > > > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > > > * System Engineer, Verzekeringen NV * > > > * www.verzekeringen.be * > > > * Oostkaai 23 B-2170 Merksem * > > > * 03/6416673 - 0477/340942 * > > > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com http://www.redhat.com/mailman/listinfo/linux-cluster -- *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* * System Engineer, Verzekeringen NV * * www.verzekeringen.be * * Oostkaai 23 B-2170 Merksem * * 03/6416673 - 0477/340942 * *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -------------- next part -------------- An HTML attachment was scrubbed... URL: From bastian at waldi.eu.org Tue Feb 8 13:54:33 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Tue, 8 Feb 2005 14:54:33 +0100 Subject: [Linux-cluster] static libs with PIC-code Message-ID: <20050208135433.GC31051@wavehammer.waldi.eu.org> cman, dlm and magma build static libraries with PIC-code, this makes it impossible to link that code into binaries. Bastian -- Killing is stupid; useless! -- McCoy, "A Private Little War", stardate 4211.8 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From pcaulfie at redhat.com Tue Feb 8 16:11:32 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 8 Feb 2005 16:11:32 +0000 Subject: [Linux-cluster] static libs with PIC-code In-Reply-To: <20050208135433.GC31051@wavehammer.waldi.eu.org> References: <20050208135433.GC31051@wavehammer.waldi.eu.org> Message-ID: <20050208161132.GA2354@tykepenguin.com> On Tue, Feb 08, 2005 at 02:54:33PM +0100, Bastian Blank wrote: > cman, dlm and magma build static libraries with PIC-code, this makes it > impossible to link that code into binaries. revision 1.7 date: 2004/10/25 17:52:29; author: lhh; state: Exp; lines: +3 -3 branches: 1.7.2; Make all the different libdlm targets use -fPIC during builds for proper symbol relocation on x86_64 lon ?? -- patrick From lhh at redhat.com Tue Feb 8 16:17:47 2005 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 08 Feb 2005 11:17:47 -0500 Subject: [Linux-cluster] static libs with PIC-code In-Reply-To: <20050208135433.GC31051@wavehammer.waldi.eu.org> References: <20050208135433.GC31051@wavehammer.waldi.eu.org> Message-ID: <1107879467.9794.45.camel@ayanami.boston.redhat.com> On Tue, 2005-02-08 at 14:54 +0100, Bastian Blank wrote: > cman, dlm and magma build static libraries with PIC-code, this makes it > impossible to link that code into binaries. Eh? On what architecture? -- Lon From lhh at redhat.com Tue Feb 8 16:58:23 2005 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 08 Feb 2005 11:58:23 -0500 Subject: [Linux-cluster] static libs with PIC-code In-Reply-To: <1107879467.9794.45.camel@ayanami.boston.redhat.com> References: <20050208135433.GC31051@wavehammer.waldi.eu.org> <1107879467.9794.45.camel@ayanami.boston.redhat.com> Message-ID: <1107881903.9794.59.camel@ayanami.boston.redhat.com> On Tue, 2005-02-08 at 11:17 -0500, Lon Hohberger wrote: > On Tue, 2005-02-08 at 14:54 +0100, Bastian Blank wrote: > > cman, dlm and magma build static libraries with PIC-code, this makes it > > impossible to link that code into binaries. > > Eh? On what architecture? Building -fPIC/-fpic generates position independent code, which is needed on some architectures in order to create dynamically-loadable plugins/libraries which use static libraries during linking. Building without -fPIC precludes this use. -- Lon From bmarzins at redhat.com Tue Feb 8 17:37:02 2005 From: bmarzins at redhat.com (Benjamin Marzinski) Date: Tue, 8 Feb 2005 11:37:02 -0600 Subject: [Linux-cluster] clvm mirroring target status In-Reply-To: <1107852032.4838.53.camel@wavebreaker.eccent.be> References: <1107535951.14182.75.camel@wavebreaker.eccent.be> <20050204200047.GH3666@phlogiston.msp.redhat.com> <200502051055.13507.filip.sergeys@verzekeringen.be> <20050207215017.GK3666@phlogiston.msp.redhat.com> <1107852032.4838.53.camel@wavebreaker.eccent.be> Message-ID: <20050208173702.GL3666@phlogiston.msp.redhat.com> > easy or reliable. The easiest way to do it is to make host A to have both > disk A and disk B, and make host B have disk Am and Bm. To do this, GNBD import > the disks from host A, assemble the pool, GNBD import the disks from host B, > and use pool_mp to integrate them into the pool. This should automatically > set you up in failover mode, with disks A and B as the primary disks and disks > Am and Bm as the backups. I realize that this means that hostB is usually > sitting idle. > > > This sounds hopefull > > If you name your devices correctly, or import them in a specific order, you > might be able to get pool to use the correct devices in the setup you > described, but I'm not certain. > > > "might be able..."-> Now I lost hope: I borrowed the idea of this setup > from the GFS admin guide, "the economy and performance" setup. > (http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-ov-perform.html#S2-OV-ECONOMY) > Probably I misinterpreted the figure1-3, especially the disk part. > Can you elaborate a bit more on: "if you name your devices correctly or > import them in a specific order"? > If I put Am and Bm on one machine, export them with gnbd and join them > in the pool in failover mode, can I be sure there will be no writing on > them? Because drbd won't let that happen. If you put Am and Bm on one machine, pool should be fine. That method should always work. The downside is that one machine is sitting idle. If you fink around with names and ordering of imports and stuff, you can probably get the setup you originally described to work. The benefit of this setup is that both machines are in use until one goes down. However, getting this setup to work may be trickier, and without looking at the pool code, I don't know exactly what you need to do. Sorry if my email wasn't clear. And about the admin-guide.... Um.... If you misinterpreted figure 1-3, then I did too. I wrote GNBD. I know all the testing that QA has done on it, and I have never heard of this setup being tested. I expect that somewhere, there is a marketing person to blame. That's not to say that it won't work. The tricky thing is to get pool to select the correct devices as primary ones and drbd to failover before pool does (which happens right after the failed node is fenced). Thanks for pointing this out to me. I was wondering why I was getting so many questions about drbd under gnbd. And this explains it. -Ben > Thanx, > > Filip Sergeys > > What your design actually wants is for pool to not do multipathing at all, but > to simply retry on failed IO. That way, when the virtual IP switches, gnbd > will just automatically pick up the device at its new location. Unfortunately, > pool and gnbd cannot do this. > > -Ben > > > Concequences: > > ------------------- > > Bringing host B back in the game needs a manual intervention. > > -Basically al services on the cluster nodes need to stop writing. > > -Sync the disk from Bm to B > > -Give host B back its virtual ip address > > -mount B read/write > > -umount Bm in host A > > -start all services again on the nodes. > > => I know this is not perfect. But we can live with that. This will need to > > happen after office hours. The thing is that we don't have the budget for > > shared storage and certainly not for a redundant shared storage solution > > because most entry level shared storages are SPOFs. > > > > I need to find out more about that multipathing. I am not sure how to use it > > in this configuration. > > If you have idea's for improvement, they are welcome. > > > > Regards, > > > > Filip > > > > PS. Thanx for your answer on the clvm mirroring state. > > > > > > > > > > > > > > On Friday 04 February 2005 21:00, Benjamin Marzinski wrote: > > > On Fri, Feb 04, 2005 at 05:52:31PM +0100, Filip Sergeys wrote: > > > > Hi, > > > > > > > > We are going to install a linux cluster with 2 gnbd servers (no SPOF) > > > > and gfs + clvm on the cluster nodes (4 nodes). I have two options, if I > > > > read the docs well, for duplicating data on the gnbd servers: > > > > 1) using clvm target mirroring on the cluster nodes > > > > 2) use drbd underneath to mirror discs. Basically two disks per machine: > > > > 1 live disk which is mirrored with drbd to the second disk in the second > > > > machine and the other way around in the second machine > > > > (so the second disk in the first machine is thus the mirror from the > > > > first (="live") disk in the second machine(sounds complicated, but it is > > > > just hard to write down)). > > > > Both live disks from each machine will be combined as one logical disk > > > > (If I understood well, this is possible). > > > > > > > > Question: what is the status of clvm mirroring? Is it stable? > > > > Suppose it is stable, so I have a choice: which one of the options would > > > > any of you choose? Reason? (Stability, performance, ...) > > > > > > I'm still not sure if cluster mirroring is available for testing (I don't > > > think that it is). It's defintely not considered stable. > > > > > > I'm also sort of unsure about your drbd solution. > > > As far as I know, drbd only allows write access on one node at a time. So, > > > if the first machine uses drbd to write to a local device and one on the > > > second machine, the second machine cannot write to that device. drbd is > > > only useful for active passive setups. If you are using pool multipathing > > > to multipath between the two gnbd servers, you could set it to failover > > > mode, and modify the fencing agent that you are using to fence the > > > gnbd_server, to make it tell drbd to fail over when you fence the server. > > > > > > I have never tried this, but it seems reasonable. One issue would be how to > > > bring the failed server back up, since the devices are going to be out of > > > sync. > > > > > > http://www.drbd.org/start.html says that drbd still only allows write > > > access to one node at a time. > > > > > > sorry :( > > > > > > -Ben > > > > > > > I found two hits on google concerning clvm mirroring, but both say it is > > > > not finished yet. However the most recent one is from june 2004. > > > > I cannot test either because we have no spare machine. I'm going to buy > > > > two machine so I need to know which disk configuration I will be using. > > > > > > > > Thanks in advance, > > > > > > > > Regards, > > > > > > > > Filip Sergeys > > > > > > > > > > > > > > > > http://64.233.183.104/search?q=cache:r1Icx--aI2YJ:www.spinics.net/lists/g > > > >fs/msg03439.html+clvm+mirroring+gfs&hl=nl&start=12 > > > > https://www.redhat.com/archives/linux-cluster/2004-June/msg00028.html > > > > > > > > -- > > > > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > > > > * System Engineer, Verzekeringen NV * > > > > * www.verzekeringen.be * > > > > * Oostkaai 23 B-2170 Merksem * > > > > * 03/6416673 - 0477/340942 * > > > > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > > > > > > > > -- > > > > Linux-cluster mailing list > > > > Linux-cluster at redhat.com > > > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > -- > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > * System Engineer, Verzekeringen NV * > * www.verzekeringen.be * > * Oostkaai 23 B-2170 Merksem * > * 03/6416673 - 0477/340942 * > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From lhh at redhat.com Tue Feb 8 18:02:39 2005 From: lhh at redhat.com (Lon Hohberger) Date: Tue, 08 Feb 2005 13:02:39 -0500 Subject: [Linux-cluster] page and buffer cache in Linux File Systems In-Reply-To: References: Message-ID: <1107885759.9794.65.camel@ayanami.boston.redhat.com> On Tue, 2005-02-08 at 09:58 +0200, John Que wrote: > And , I am not an expert in ext3 , but: > in case such a requirement DOES exist for non clustered file systems > (like ext3) - does ext3 implement this requirement ? How? I believe O_DIRECT is what you're looking for on ext3. -- Lon From mtilstra at redhat.com Tue Feb 8 18:38:31 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Tue, 8 Feb 2005 12:38:31 -0600 Subject: [Linux-cluster] page and buffer cache in Linux File Systems In-Reply-To: <1107885759.9794.65.camel@ayanami.boston.redhat.com> References: <1107885759.9794.65.camel@ayanami.boston.redhat.com> Message-ID: <20050208183831.GA3893@redhat.com> On Tue, Feb 08, 2005 at 01:02:39PM -0500, Lon Hohberger wrote: > On Tue, 2005-02-08 at 09:58 +0200, John Que wrote: > > > And , I am not an expert in ext3 , but: > > in case such a requirement DOES exist for non clustered file systems > > (like ext3) - does ext3 implement this requirement ? How? > > I believe O_DIRECT is what you're looking for on ext3. And gfs. -- Michael Conrad Tadpol Tilstra I've a memory like a sponge. It's soft, squishy and full of holes. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From vahram at Broadspire.com Tue Feb 8 21:43:19 2005 From: vahram at Broadspire.com (vahram) Date: Tue, 08 Feb 2005 13:43:19 -0800 Subject: [Linux-cluster] cluster architecture Message-ID: <42093277.8090906@broadspire.com> Hi all, I'm planning to put together a production web server farm that will consist of at least 6 servers. They will all be running Apache and Postfix, and will be sharing a 4+TB storage device. Horizontal scalability is a major issue for us. I just wanted to get some general recommendations on who to go with for our storage needs. We were considering a Netapp appliance, but the cost is extremely high and their solution is probably a bit overkill for our needs. Cost is a major issue for us. How does the performance of a Netapp appliance running NFS compare to a fibre-based storage device (such as an Apple XServe RAID or similar unit) running GFS? Is anyone here running GFS on a production server farm? Thanks! -vahram From bastian at waldi.eu.org Wed Feb 9 11:27:01 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Wed, 9 Feb 2005 12:27:01 +0100 Subject: [Linux-cluster] static libs with PIC-code In-Reply-To: <1107879467.9794.45.camel@ayanami.boston.redhat.com> References: <20050208135433.GC31051@wavehammer.waldi.eu.org> <1107879467.9794.45.camel@ayanami.boston.redhat.com> Message-ID: <20050209112701.GB14220@wavehammer.waldi.eu.org> On Tue, Feb 08, 2005 at 11:17:47AM -0500, Lon Hohberger wrote: > On Tue, 2005-02-08 at 14:54 +0100, Bastian Blank wrote: > > cman, dlm and magma build static libraries with PIC-code, this makes it > > impossible to link that code into binaries. > Eh? On what architecture? ARM for example. Bastian -- You're too beautiful to ignore. Too much woman. -- Kirk to Yeoman Rand, "The Enemy Within", stardate unknown -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From bastian at waldi.eu.org Wed Feb 9 11:28:38 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Wed, 9 Feb 2005 12:28:38 +0100 Subject: [Linux-cluster] static libs with PIC-code In-Reply-To: <1107881903.9794.59.camel@ayanami.boston.redhat.com> References: <20050208135433.GC31051@wavehammer.waldi.eu.org> <1107879467.9794.45.camel@ayanami.boston.redhat.com> <1107881903.9794.59.camel@ayanami.boston.redhat.com> Message-ID: <20050209112838.GC14220@wavehammer.waldi.eu.org> On Tue, Feb 08, 2005 at 11:58:23AM -0500, Lon Hohberger wrote: > Building -fPIC/-fpic generates position independent code, which is > needed on some architectures in order to create dynamically-loadable > plugins/libraries which use static libraries during linking. Building > without -fPIC precludes this use. Please take a look at X11 and/or the debian version of sdl, they build special pic static libs for this case. Bastian -- The heart is not a logical organ. -- Dr. Janet Wallace, "The Deadly Years", stardate 3479.4 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From filip.sergeys at verzekeringen.be Wed Feb 9 12:14:25 2005 From: filip.sergeys at verzekeringen.be (Filip Sergeys) Date: 09 Feb 2005 13:14:25 +0100 Subject: [Linux-cluster] clvm mirroring target status In-Reply-To: <20050208173702.GL3666@phlogiston.msp.redhat.com> References: <1107535951.14182.75.camel@wavebreaker.eccent.be> <20050204200047.GH3666@phlogiston.msp.redhat.com> <200502051055.13507.filip.sergeys@verzekeringen.be> <20050207215017.GK3666@phlogiston.msp.redhat.com> <1107852032.4838.53.camel@wavebreaker.eccent.be> <20050208173702.GL3666@phlogiston.msp.redhat.com> Message-ID: <1107951265.3919.92.camel@wavebreaker.eccent.be> Thank you for your clarification. If you are getting so many questions about this setup it's probably because people find it an appealing configuration. A good sign maybe for redhat to accomodate the pool a make it work that way. That would be true customer driven development. Just hinting ;) Thanx for your help so far, Regards, Filip Sergeys On Tue, 2005-02-08 at 18:37, Benjamin Marzinski wrote: > easy or reliable. The easiest way to do it is to make host A to have both > disk A and disk B, and make host B have disk Am and Bm. To do this, GNBD import > the disks from host A, assemble the pool, GNBD import the disks from host B, > and use pool_mp to integrate them into the pool. This should automatically > set you up in failover mode, with disks A and B as the primary disks and disks > Am and Bm as the backups. I realize that this means that hostB is usually > sitting idle. > > > This sounds hopefull > > If you name your devices correctly, or import them in a specific order, you > might be able to get pool to use the correct devices in the setup you > described, but I'm not certain. > > > "might be able..."-> Now I lost hope: I borrowed the idea of this setup > from the GFS admin guide, "the economy and performance" setup. > (http://www.redhat.com/docs/manuals/csgfs/admin-guide/s1-ov-perform.html#S2-OV-ECONOMY) > Probably I misinterpreted the figure1-3, especially the disk part. > Can you elaborate a bit more on: "if you name your devices correctly or > import them in a specific order"? > If I put Am and Bm on one machine, export them with gnbd and join them > in the pool in failover mode, can I be sure there will be no writing on > them? Because drbd won't let that happen. If you put Am and Bm on one machine, pool should be fine. That method should always work. The downside is that one machine is sitting idle. If you fink around with names and ordering of imports and stuff, you can probably get the setup you originally described to work. The benefit of this setup is that both machines are in use until one goes down. However, getting this setup to work may be trickier, and without looking at the pool code, I don't know exactly what you need to do. Sorry if my email wasn't clear. And about the admin-guide.... Um.... If you misinterpreted figure 1-3, then I did too. I wrote GNBD. I know all the testing that QA has done on it, and I have never heard of this setup being tested. I expect that somewhere, there is a marketing person to blame. That's not to say that it won't work. The tricky thing is to get pool to select the correct devices as primary ones and drbd to failover before pool does (which happens right after the failed node is fenced). Thanks for pointing this out to me. I was wondering why I was getting so many questions about drbd under gnbd. And this explains it. -Ben > Thanx, > > Filip Sergeys > > What your design actually wants is for pool to not do multipathing at all, but > to simply retry on failed IO. That way, when the virtual IP switches, gnbd > will just automatically pick up the device at its new location. Unfortunately, > pool and gnbd cannot do this. > > -Ben > > > Concequences: > > ------------------- > > Bringing host B back in the game needs a manual intervention. > > -Basically al services on the cluster nodes need to stop writing. > > -Sync the disk from Bm to B > > -Give host B back its virtual ip address > > -mount B read/write > > -umount Bm in host A > > -start all services again on the nodes. > > => I know this is not perfect. But we can live with that. This will need to > > happen after office hours. The thing is that we don't have the budget for > > shared storage and certainly not for a redundant shared storage solution > > because most entry level shared storages are SPOFs. > > > > I need to find out more about that multipathing. I am not sure how to use it > > in this configuration. > > If you have idea's for improvement, they are welcome. > > > > Regards, > > > > Filip > > > > PS. Thanx for your answer on the clvm mirroring state. > > > > > > > > > > > > > > On Friday 04 February 2005 21:00, Benjamin Marzinski wrote: > > > On Fri, Feb 04, 2005 at 05:52:31PM +0100, Filip Sergeys wrote: > > > > Hi, > > > > > > > > We are going to install a linux cluster with 2 gnbd servers (no SPOF) > > > > and gfs + clvm on the cluster nodes (4 nodes). I have two options, if I > > > > read the docs well, for duplicating data on the gnbd servers: > > > > 1) using clvm target mirroring on the cluster nodes > > > > 2) use drbd underneath to mirror discs. Basically two disks per machine: > > > > 1 live disk which is mirrored with drbd to the second disk in the second > > > > machine and the other way around in the second machine > > > > (so the second disk in the first machine is thus the mirror from the > > > > first (="live") disk in the second machine(sounds complicated, but it is > > > > just hard to write down)). > > > > Both live disks from each machine will be combined as one logical disk > > > > (If I understood well, this is possible). > > > > > > > > Question: what is the status of clvm mirroring? Is it stable? > > > > Suppose it is stable, so I have a choice: which one of the options would > > > > any of you choose? Reason? (Stability, performance, ...) > > > > > > I'm still not sure if cluster mirroring is available for testing (I don't > > > think that it is). It's defintely not considered stable. > > > > > > I'm also sort of unsure about your drbd solution. > > > As far as I know, drbd only allows write access on one node at a time. So, > > > if the first machine uses drbd to write to a local device and one on the > > > second machine, the second machine cannot write to that device. drbd is > > > only useful for active passive setups. If you are using pool multipathing > > > to multipath between the two gnbd servers, you could set it to failover > > > mode, and modify the fencing agent that you are using to fence the > > > gnbd_server, to make it tell drbd to fail over when you fence the server. > > > > > > I have never tried this, but it seems reasonable. One issue would be how to > > > bring the failed server back up, since the devices are going to be out of > > > sync. > > > > > > http://www.drbd.org/start.html says that drbd still only allows write > > > access to one node at a time. > > > > > > sorry :( > > > > > > -Ben > > > > > > > I found two hits on google concerning clvm mirroring, but both say it is > > > > not finished yet. However the most recent one is from june 2004. > > > > I cannot test either because we have no spare machine. I'm going to buy > > > > two machine so I need to know which disk configuration I will be using. > > > > > > > > Thanks in advance, > > > > > > > > Regards, > > > > > > > > Filip Sergeys > > > > > > > > > > > > > > > > http://64.233.183.104/search?q=cache:r1Icx--aI2YJ:www.spinics.net/lists/g > > > >fs/msg03439.html+clvm+mirroring+gfs&hl=nl&start=12 > > > > https://www.redhat.com/archives/linux-cluster/2004-June/msg00028.html > > > > > > > > -- > > > > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > > > > * System Engineer, Verzekeringen NV * > > > > * www.verzekeringen.be * > > > > * Oostkaai 23 B-2170 Merksem * > > > > * 03/6416673 - 0477/340942 * > > > > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > > > > > > > > -- > > > > Linux-cluster mailing list > > > > Linux-cluster at redhat.com > > > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster at redhat.com > > > http://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > > http://www.redhat.com/mailman/listinfo/linux-cluster > -- > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > * System Engineer, Verzekeringen NV * > * www.verzekeringen.be * > * Oostkaai 23 B-2170 Merksem * > * 03/6416673 - 0477/340942 * > *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster at redhat.com http://www.redhat.com/mailman/listinfo/linux-cluster -- *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* * System Engineer, Verzekeringen NV * * www.verzekeringen.be * * Oostkaai 23 B-2170 Merksem * * 03/6416673 - 0477/340942 * *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* -------------- next part -------------- An HTML attachment was scrubbed... URL: From rstevens at vitalstream.com Wed Feb 9 18:32:09 2005 From: rstevens at vitalstream.com (Rick Stevens) Date: Wed, 09 Feb 2005 10:32:09 -0800 Subject: [Linux-cluster] cluster architecture In-Reply-To: <42093277.8090906@broadspire.com> References: <42093277.8090906@broadspire.com> Message-ID: <420A5729.80403@vitalstream.com> vahram wrote: > Hi all, > > I'm planning to put together a production web server farm that will > consist of at least 6 servers. They will all be running Apache and > Postfix, and will be sharing a 4+TB storage device. Horizontal > scalability is a major issue for us. > > I just wanted to get some general recommendations on who to go with for > our storage needs. We were considering a Netapp appliance, but the cost > is extremely high and their solution is probably a bit overkill for our > needs. Cost is a major issue for us. > > How does the performance of a Netapp appliance running NFS compare to a > fibre-based storage device (such as an Apple XServe RAID or similar > unit) running GFS? Is anyone here running GFS on a production server > farm? Thanks! We use NetApps a lot. Their performance is terrific, but it is NFS over gigabit ethernet with all that entails and isn't as high as it would be on a SAN or other block-level device (this is true for any NAS). I will say that NetApps are bulletproof, easy to expand and software updates are very, very simple. Licensing is not cheap, but the fact you can run CIFS and NFS simultaneously is a plus. Yes, they cost money, but you get what you pay for. You could simulate a NetApp by getting a really beefy server with a FC or SCSI SAN attached to it and making it an NFS (and possibly Samba) server. I won't swear to what kind of performance you'd get, but you could possibly get 80% of wire speed, depending on your network architecture and other features. If you're using any NFS or NAS as a common file system, make sure you have "noac" set for the mounts or you may miss files put on the storage by other systems. Unfortunately, this eats into performance, but that's the nature of the beast. As far as SANs are concerned, you'll probably need a fiberchannel system for 6 nodes unless you can find a 6-port SCSI unit (doubtful). If you choose FC, you'll need to think about the switch fabric and whether you will have to deal with multipathing. If that's true, you have to make sure your vendor has multipathing modules for your kernel. You also need to look at bandwidth and whether the SAN you're looking at can sustain the I/O bandwidth you want. You also need to figure out how you're going to share that storage among the nodes in the cluster. We are evaluating several fairly large SANs for use with GFS, but our bandwidth needs are a bit, well, over-the-top. We need 9Gbps aggregate throughput. We're looking at IBM as well as Hitachi FC SAN solutions. ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - IGNORE that man behind the keyboard! - - - The Wizard of OS - ---------------------------------------------------------------------- From vahram at Broadspire.com Wed Feb 9 18:59:09 2005 From: vahram at Broadspire.com (vahram) Date: Wed, 09 Feb 2005 10:59:09 -0800 Subject: [Linux-cluster] cluster architecture In-Reply-To: <420A5729.80403@vitalstream.com> References: <42093277.8090906@broadspire.com> <420A5729.80403@vitalstream.com> Message-ID: <420A5D7D.8060200@broadspire.com> Raw throughput isn't really an issue for us. We're more interested in seek times. My biggest concern with GFS is stability and performance...any feedback in regards to that would be greatly appreciated. Thanks! Rick Stevens wrote: > vahram wrote: > >> Hi all, >> >> I'm planning to put together a production web server farm that will >> consist of at least 6 servers. They will all be running Apache and >> Postfix, and will be sharing a 4+TB storage device. Horizontal >> scalability is a major issue for us. >> >> I just wanted to get some general recommendations on who to go with >> for our storage needs. We were considering a Netapp appliance, but >> the cost is extremely high and their solution is probably a bit >> overkill for our needs. Cost is a major issue for us. >> >> How does the performance of a Netapp appliance running NFS compare to >> a fibre-based storage device (such as an Apple XServe RAID or similar >> unit) running GFS? Is anyone here running GFS on a production server >> farm? Thanks! > > > We use NetApps a lot. Their performance is terrific, but it is NFS over > gigabit ethernet with all that entails and isn't as high as it would be > on a SAN or other block-level device (this is true for any NAS). > > I will say that NetApps are bulletproof, easy to expand and software > updates are very, very simple. Licensing is not cheap, but the fact you > can run CIFS and NFS simultaneously is a plus. Yes, they cost money, > but you get what you pay for. You could simulate a NetApp by getting a > really beefy server with a FC or SCSI SAN attached to it and making it > an NFS (and possibly Samba) server. I won't swear to what kind of > performance you'd get, but you could possibly get 80% of wire speed, > depending on your network architecture and other features. > > If you're using any NFS or NAS as a common file system, make sure you > have "noac" set for the mounts or you may miss files put on the storage > by other systems. Unfortunately, this eats into performance, but that's > the nature of the beast. > > As far as SANs are concerned, you'll probably need a fiberchannel system > for 6 nodes unless you can find a 6-port SCSI unit (doubtful). If you > choose FC, you'll need to think about the switch fabric and whether you > will have to deal with multipathing. If that's true, you have to make > sure your vendor has multipathing modules for your kernel. You also > need to look at bandwidth and whether the SAN you're looking at can > sustain the I/O bandwidth you want. You also need to figure out how > you're going to share that storage among the nodes in the cluster. > > We are evaluating several fairly large SANs for use with GFS, but our > bandwidth needs are a bit, well, over-the-top. We need 9Gbps aggregate > throughput. We're looking at IBM as well as Hitachi FC SAN solutions. > ---------------------------------------------------------------------- > - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - > - VitalStream, Inc. http://www.vitalstream.com - > - - > - IGNORE that man behind the keyboard! - > - - The Wizard of OS - > ---------------------------------------------------------------------- > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From rstevens at vitalstream.com Wed Feb 9 19:21:34 2005 From: rstevens at vitalstream.com (Rick Stevens) Date: Wed, 09 Feb 2005 11:21:34 -0800 Subject: [Linux-cluster] cluster architecture In-Reply-To: <420A5D7D.8060200@broadspire.com> References: <42093277.8090906@broadspire.com> <420A5729.80403@vitalstream.com> <420A5D7D.8060200@broadspire.com> Message-ID: <420A62BE.2050804@vitalstream.com> vahram wrote: > Raw throughput isn't really an issue for us. We're more interested in > seek times. My biggest concern with GFS is stability and > performance...any feedback in regards to that would be greatly > appreciated. Thanks! So far, GFS has worked quite well under our tests. We have yet to have it break. Our current GFS implementation is only on two nodes with gulm running on a separate lock server. I intend to update the kernels on those nodes sometime this week (to the 2.6.11 variety) and change the locking from gulm to cman (since that seems to be fixed at this point). The SAN that's attached is an off-brand FC unit via a dual-port switch and the nodes are using QLogic QLA2300 HBAs. The application we've been using to test hasn't stressed it that badly (it's been pooping out long before the servers were stressed), so I can't say a whole lot. I can say that rsyncing the entire filesystem (20GB) didn't cause any problems, either pitching or catching. ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - You possess a mind not merely twisted, but actually sprained. - ---------------------------------------------------------------------- From ivan.ivanyi at isb-sib.ch Thu Feb 10 11:02:17 2005 From: ivan.ivanyi at isb-sib.ch (IVANYI Ivan) Date: Thu, 10 Feb 2005 12:02:17 +0100 Subject: [Linux-cluster] cluster architecture In-Reply-To: <420A62BE.2050804@vitalstream.com> References: <42093277.8090906@broadspire.com> <420A5729.80403@vitalstream.com> <420A5D7D.8060200@broadspire.com> <420A62BE.2050804@vitalstream.com> Message-ID: <420B3F39.4000204@isb-sib.ch> I've got 3 nodes direct attached to SAN. The performance of GFS has disappointed me a bit so far. Maybe I've got something wrong but then again documentation is lacking... unless I'm looking in the wrong places. Previously in a slightly different configuration I had only a slight performance hit with IBM's GPFS. Rick Stevens wrote: > vahram wrote: > >> Raw throughput isn't really an issue for us. We're more interested in >> seek times. My biggest concern with GFS is stability and >> performance...any feedback in regards to that would be greatly >> appreciated. Thanks! > > > So far, GFS has worked quite well under our tests. We have yet to have > it break. Our current GFS implementation is only on two nodes with gulm > running on a separate lock server. I intend to update the kernels on > those nodes sometime this week (to the 2.6.11 variety) and change the > locking from gulm to cman (since that seems to be fixed at this point). Again not too sure about the different locking mechanisms .. do you mean cman/dlm? will this work better for you? -- ************************************************************ Ivan Ivanyi Swiss Institute of Bioinformatics 1, rue Michel Servet CH-1211 Gen?ve 4 Switzerland Tel: (+41 22) 379 58 33 Fax: (+41 22) 379 58 58 E-mail: Ivan.Ivanyi at isb-sib.ch ************************************************************ PGP signature http://www.expasy.org/people/Ivan.Ivanyi.gpg From rstevens at vitalstream.com Thu Feb 10 17:22:14 2005 From: rstevens at vitalstream.com (Rick Stevens) Date: Thu, 10 Feb 2005 09:22:14 -0800 Subject: [Linux-cluster] cluster architecture In-Reply-To: <420B3F39.4000204@isb-sib.ch> References: <42093277.8090906@broadspire.com> <420A5729.80403@vitalstream.com> <420A5D7D.8060200@broadspire.com> <420A62BE.2050804@vitalstream.com> <420B3F39.4000204@isb-sib.ch> Message-ID: <420B9846.9@vitalstream.com> IVANYI Ivan wrote: > I've got 3 nodes direct attached to SAN. The performance of GFS has > disappointed me a bit so far. Maybe I've got something wrong but then > again documentation is lacking... unless I'm looking in the wrong places. > > Previously in a slightly different configuration I had only a slight > performance hit with IBM's GPFS. > > Rick Stevens wrote: > >> vahram wrote: >> >>> Raw throughput isn't really an issue for us. We're more interested >>> in seek times. My biggest concern with GFS is stability and >>> performance...any feedback in regards to that would be greatly >>> appreciated. Thanks! >> >> >> >> So far, GFS has worked quite well under our tests. We have yet to have >> it break. Our current GFS implementation is only on two nodes with gulm >> running on a separate lock server. I intend to update the kernels on >> those nodes sometime this week (to the 2.6.11 variety) and change the >> locking from gulm to cman (since that seems to be fixed at this point). > > > Again not too sure about the different locking mechanisms .. do you mean > cman/dlm? will this work better for you? Currently we use cman to do the LVM locking/clustering stuff and gulm to do the GFS locking as cman wasn't reliable handling GFS. The gods that write the stuff now tell me that cman can handle GFS properly, so I'm going to give it a whirl. > > -- ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens at vitalstream.com - - VitalStream, Inc. http://www.vitalstream.com - - - - You know the old saying--any technology sufficiently advanced is - - indistinguishable from a Perl script - - --Programming Perl, 2nd Edition - ---------------------------------------------------------------------- From tretkowski at inittab.de Fri Feb 11 14:21:38 2005 From: tretkowski at inittab.de (Norbert Tretkowski) Date: Fri, 11 Feb 2005 15:21:38 +0100 Subject: [Linux-cluster] GFS 6.0 patches for kernel 2.4 Message-ID: <20050211142138.GB1383@rollcage.inittab.de> Hi, I'm searching GFS 6.0 kernel patches for 2.4 kernels (to be honest, the 2.4.21 kernel from SuSE Linux Enterprise Server 8, but a patch against a vanilla kernel would also help). I found patches for RHEL3 kernels as SRPMs but these patches don't apply and/or don't build with the SLES8 kernel. Thanks, Norbert From daniel at osdl.org Sat Feb 12 00:47:38 2005 From: daniel at osdl.org (Daniel McNeil) Date: Fri, 11 Feb 2005 16:47:38 -0800 Subject: [Linux-cluster] cluster lost quorum after 11 hours Message-ID: <1108169257.5927.12.camel@ibm-c.pdx.osdl.net> I was running my test on a 3 node cluster and it died after 11 hours. cl030 lost quorum with the other 2 nodes kicked out of the cluster. cl031 also hit a bunch of asserts like lock_dlm: Assertion failed on line 352 of file /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c lock_dlm: assertion: "!error" lock_dlm: time = 291694516 stripefs: error=-22 num=2,19 I assume is caused by the cluster shutting down. /var/log/messages showed: cl030: Feb 11 02:44:33 cl030 kernel: CMAN: removing node cl032a from the cluster : No response to messages Feb 11 02:44:33 cl030 kernel: CMAN: removing node cl031a from the cluster : No response to messages Feb 11 02:44:33 cl030 kernel: CMAN: quorum lost, blocking activity Feb 11 14:40:33 cl030 sshd(pam_unix)[27323]: session opened for user root by (uid=0) cl031: Feb 11 02:44:33 cl031 kernel: CMAN: node cl032a has been removed from the cluster : No response to messages Feb 11 02:44:33 cl031 kernel: CMAN: node cl031a has been removed from the cluster : No response to messages Feb 11 02:44:33 cl031 kernel: CMAN: killed by NODEDOWN message Feb 11 02:44:33 cl031 kernel: CMAN: we are leaving the cluster. Feb 11 02:44:34 cl031 kernel: lowcomms_get_buffer: accepting is 0 Feb 11 02:44:34 cl031 kernel: dlm: stripefs: remote_stage error -105 2019c Feb 11 02:44:34 cl031 ccsd[3823]: [cluster_mgr.c:387] Cluster manager shutdown. Attemping to reconnect... Feb 11 02:44:34 cl031 kernel: SM: 00000001 sm_stop: SG still joined Feb 11 02:44:34 cl031 kernel: SM: 0100041e sm_stop: SG still joined Feb 11 02:44:34 cl031 kernel: SM: 0200041f sm_stop: SG still joined Feb 11 02:44:37 cl031 ccsd[3823]: [cluster_mgr.c:346] Unable to connect to cluster infrastructure after 30 seconds. Feb 11 02:45:07 cl031 ccsd[3823]: [cluster_mgr.c:346] Unable to connect to cluster infrastructure after 60 seconds. cl032: Feb 11 02:44:33 cl032 kernel: CMAN: node cl032a has been removed from the cluster : No response to messages Feb 11 02:44:33 cl032 kernel: CMAN: killed by NODEDOWN message Feb 11 02:44:33 cl032 kernel: CMAN: we are leaving the cluster. Feb 11 02:44:34 cl032 kernel: lowcomms_get_buffer: accepting is 0 Feb 11 02:44:34 cl032 kernel: dlm: stripefs: remote_stage error -105 102bd Feb 11 02:44:34 cl032 kernel: lowcomms_get_buffer: accepting is 0 Feb 11 02:44:34 cl032 ccsd[22909]: [cluster_mgr.c:387] Cluster manager shutdown. Attemping to reconnect... Feb 11 02:44:34 cl032 kernel: SM: 00000001 sm_stop: SG still joined Feb 11 02:44:34 cl032 kernel: SM: 0100041e sm_stop: SG still joined Feb 11 02:44:34 cl032 kernel: SM: 0200041f sm_stop: SG still joined Feb 11 02:44:53 cl032 ccsd[22909]: [cluster_mgr.c:346] Unable to connect to cluster infrastructure after 90 seconds. More info available here: http://developer.osdl.org/daniel/GFS/test.10feb2005/ I usually get closer to 50 hours before problems. Any ideas? Daniel From bastian at waldi.eu.org Sat Feb 12 15:42:16 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Sat, 12 Feb 2005 16:42:16 +0100 Subject: [Linux-cluster] config update kills cluster Message-ID: <20050212154216.GA3682@wavehammer.waldi.eu.org> Hi all I just tried config update via ccs_tool in a cman cluster. Each of the nodes got the new config but the kernel rejects joins with | CMAN: Join request from gfs1 rejected, config version local 1 remote 2 The cluster is running CVS from 2005-02-06. Bastian -- Vulcans believe peace should not depend on force. -- Amanda, "Journey to Babel", stardate 3842.3 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From bastian at waldi.eu.org Sat Feb 12 16:39:57 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Sat, 12 Feb 2005 17:39:57 +0100 Subject: [Linux-cluster] gfs_grow/_jadd don't accept mointpoint Message-ID: <20050212163957.GA17954@wavehammer.waldi.eu.org> Hi all gfs_grow/_jadd don't accept mountpoints as arguments. | # gfs_grow /mnt/ | GFS Filesystem /mnt/ not found | # grep /mnt /proc/mounts | /dev/sda2 /mnt gfs rw,noatime,nodiratime 0 0 This is CVS HEAD from 2005-02-06. Bastian -- Pain is a thing of the mind. The mind can be controlled. -- Spock, "Operation -- Annihilate!" stardate 3287.2 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From kpreslan at redhat.com Sat Feb 12 19:34:59 2005 From: kpreslan at redhat.com (Ken Preslan) Date: Sat, 12 Feb 2005 13:34:59 -0600 Subject: [Linux-cluster] gfs_grow/_jadd don't accept mointpoint In-Reply-To: <20050212163957.GA17954@wavehammer.waldi.eu.org> References: <20050212163957.GA17954@wavehammer.waldi.eu.org> Message-ID: <20050212193459.GA14807@potassium.msp.redhat.com> On Sat, Feb 12, 2005 at 05:39:57PM +0100, Bastian Blank wrote: > Hi all > > gfs_grow/_jadd don't accept mountpoints as arguments. > > | # gfs_grow /mnt/ > | GFS Filesystem /mnt/ not found > | # grep /mnt /proc/mounts > | /dev/sda2 /mnt gfs rw,noatime,nodiratime 0 0 > > This is CVS HEAD from 2005-02-06. Get rid of the trailing "/". i.e. gfs_grow /mnt -- Ken Preslan From bastian at waldi.eu.org Sun Feb 13 19:54:16 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Sun, 13 Feb 2005 20:54:16 +0100 Subject: [Linux-cluster] config update kills cluster In-Reply-To: <20050212154216.GA3682@wavehammer.waldi.eu.org> References: <20050212154216.GA3682@wavehammer.waldi.eu.org> Message-ID: <20050213195416.GD16192@wavehammer.waldi.eu.org> On Sat, Feb 12, 2005 at 04:42:16PM +0100, Bastian Blank wrote: > | CMAN: Join request from gfs1 rejected, config version local 1 remote 2 Bah, I should read correctly. Bastian -- You! What PLANET is this! -- McCoy, "The City on the Edge of Forever", stardate 3134.0 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From bastian at waldi.eu.org Sun Feb 13 19:59:55 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Sun, 13 Feb 2005 20:59:55 +0100 Subject: [Linux-cluster] possible to wait on fence domain startup Message-ID: <20050213195955.GE16192@wavehammer.waldi.eu.org> Hi folks Is it possible to wait for the startup of the fence domain? If I call "fence_tool join" and mount of a gfs volume without a sleep between, I get permission denied and the kernel log reports the missing fence domain. The time which is needed between this two calls seems to be related to the number of nodes in the fence domain. Bastian -- Without freedom of choice there is no creativity. -- Kirk, "The return of the Archons", stardate 3157.4 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From bastian at waldi.eu.org Sun Feb 13 22:26:59 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Sun, 13 Feb 2005 23:26:59 +0100 Subject: [Linux-cluster] ccs - fix -p parameter of ccsd Message-ID: <20050213222659.GA28716@wavehammer.waldi.eu.org> Hi folks The attached patch fixes the -p parameter of ccsd. Bastian -- It would be illogical to assume that all conditions remain stable. -- Spock, "The Enterprise Incident", stardate 5027.3 -------------- next part -------------- === daemon/ccsd.c ================================================================== --- daemon/ccsd.c (revision 315) +++ daemon/ccsd.c (local) @@ -296,7 +296,7 @@ memset(buff, 0, buff_size); - while((c = getopt(argc, argv, "46cdf:hlm:nP:t:sV")) != -1){ + while((c = getopt(argc, argv, "46cdf:hlm:np:P:t:sV")) != -1){ switch(c){ case '4': if(IPv6 == 1){ @@ -378,6 +378,7 @@ lockfile_location = optarg; buff_index += snprintf(buff+buff_index, buff_size-buff_index, " Lock file location:: %s\n", optarg); + break; case 'P': if(optarg[1] != ':'){ fprintf(stderr, "Bad argument to '-P' option.\n" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From bastian at waldi.eu.org Sun Feb 13 22:39:47 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Sun, 13 Feb 2005 23:39:47 +0100 Subject: [Linux-cluster] dlm - update dlm32 layer Message-ID: <20050213223947.GB28716@wavehammer.waldi.eu.org> Hi folks The attached patch adds biarch support to the dlm32 layer of libdlm. It is currently only enabled on s390 and sparc. Bastian -- Those who hate and fight must stop themselves -- otherwise it is not stopped. -- Spock, "Day of the Dove", stardate unknown -------------- next part -------------- === lib/Makefile ================================================================== --- lib/Makefile (revision 308) +++ lib/Makefile (local) @@ -22,7 +22,7 @@ include ${top_srcdir}/make/defines.mk -CFLAGS += -g -O -I. -fPIC +CFLAGS += -g -O -I. ifneq (${KERNEL_SRC}, ) # Use the kernel tree if patched, otherwise, look where cluster headers @@ -37,38 +37,35 @@ all: $(STATICLIB) $(SHAREDLIB) -$(LIBNAME).a: libdlm.o libaislock.o - ${AR} cr libdlm.a libdlm.o libaislock.o +lib_SOURCES = libdlm.c libaislock.c dlm32.c +lib_lt_SOURCES = libdlm_lt.c dlm32.c + +$(LIBNAME).a: $(lib_SOURCES:.c=.po) + ${AR} cr libdlm.a $^ ${RANLIB} libdlm.a -$(LIBNAME)_lt.a: libdlm_lt.o - ${AR} r libdlm_lt.a libdlm_lt.o +$(LIBNAME)_lt.a: $(lib_lt_SOURCES:.c=.po) + ${AR} r libdlm_lt.a $^ ${RANLIB} libdlm_lt.a -$(LIBNAME).so.${RELEASE_MAJOR}.${RELEASE_MINOR}: libdlm.po libaislock.po - $(LD) -shared -o $@ -soname=$(LIBNAME).so.$(RELEASE_MAJOR) $^ +$(LIBNAME).so.${RELEASE_MAJOR}.${RELEASE_MINOR}: $(lib_SOURCES:.c=.po) + $(CC) -shared -o $@ -Wl,-soname=$(LIBNAME).so.$(RELEASE_MAJOR) $^ -$(LIBNAME)_lt.so.${RELEASE_MAJOR}.${RELEASE_MINOR}: libdlm_lt.po +$(LIBNAME)_lt.so.${RELEASE_MAJOR}.${RELEASE_MINOR}: $(lib_lt_SOURCES:.c=.po) $(CC) -shared -o $@ -Wl,-soname=$(LIBNAME)_lt.so.$(RELEASE_MAJOR) $^ -libdlm.po: libdlm.c - $(CC) $(CFLAGS) -D_REENTRANT -c -o $@ $< +%_lt.o: %.c + $(CC) $(CFLAGS) -c -o $@ $< -libdlm.o: libdlm.c - $(CC) $(CFLAGS) -D_REENTRANT -c -o $@ $< +%_lt.po: %.c + $(CC) $(CFLAGS) -fPIC -c -o $@ $< -libaislock.po: libaislock.c +%.o: %.c $(CC) $(CFLAGS) -D_REENTRANT -c -o $@ $< -libaislock.o: libaislock.c - $(CC) $(CFLAGS) -D_REENTRANT -c -o $@ $< +%.po: %.c + $(CC) $(CFLAGS) -fPIC -D_REENTRANT -c -o $@ $< -libdlm_lt.po: libdlm.c - $(CC) $(CFLAGS) -c -o $@ $< - -libdlm_lt.o: libdlm.c - $(CC) $(CFLAGS) -c -o $@ $< - copytobin: all === lib/dlm32.c ================================================================== --- lib/dlm32.c (revision 308) +++ lib/dlm32.c (local) @@ -23,7 +23,23 @@ /* Convert 32 bit userland reads & writes to something suitable for a 64 bit kernel */ +#include +#include +#include +#include +#include +#include "dlm.h" +#include "dlm_device.h" +#if (defined(__s390__) || defined(__sparc__)) && __WORDSIZE == 32 +# define BUILD_BIARCH +#endif + +extern ssize_t dlm_read(int, struct dlm_lock_result *); +extern ssize_t dlm_read_data(int, struct dlm_lock_result *, size_t); +extern ssize_t dlm_write(int, struct dlm_write_request *, size_t); + +#ifdef BUILD_BIARCH /* 64 bit versions of the structs */ struct dlm_lock_params64 { uint8_t mode; @@ -32,10 +48,10 @@ uint32_t parent; struct dlm_range range; uint8_t namelen; - uint64_t castparam; + uint64_t castparam; uint64_t castaddr; uint64_t bastparam; - uint64_t bastaddr; + uint64_t bastaddr; uint64_t lksb; char lvb[DLM_LVB_LEN]; char name[1]; @@ -73,10 +89,10 @@ struct dlm_lksb64 { - int sb_status; - uint32_t sb_lkid; - char sb_flags; - uint64_t sb_lvbptr; + int sb_status; + uint32_t sb_lkid; + char sb_flags; + uint64_t sb_lvbptr; }; /* struct read from the "device" fd, @@ -104,16 +120,38 @@ int gqi_lockcount; /* output */ }; +static bool check_biarch_convert(void) +{ + static enum { undefined, native, convert } status; + if (status == undefined) + { + struct utsname buf; + if (uname(&buf) != 0) + status = native; + else if (strcmp(buf.machine, +#ifdef __s390__ + "s390x" +#endif +#ifdef __sparc__ + "sparc64" +#endif + ) == 0) + status = convert; + else + status = native; + } + if (status == convert) + return true; + return false; +} -int dlm_write(int fd, void *buf, int len) +static ssize_t _dlm_write_convert(int fd, struct dlm_write_request *req32, size_t len32) { char buf64[sizeof(struct dlm_write_request64) + DLM_RESNAME_MAXLEN]; struct dlm_write_request64 *req64; - struct dlm_write_request *req32; int len64; int ret; - req32 = (struct dlm_write_request *)buf; req64 = (struct dlm_write_request64 *)buf64; len64 = sizeof(struct dlm_write_request64); @@ -167,37 +205,33 @@ /* Fake the return length */ if (ret == len64) - ret = len; + ret = len32; return ret; } - -int dlm_read(int fd, void *buf, int len) +static ssize_t _dlm_read_convert(bool data, int fd, struct dlm_lock_result *res32, ssize_t len32) { - int ret; - int len64; - struct dlm_lock_result *res32; + ssize_t ret; + size_t len64; struct dlm_lock_result64 *res64; struct dlm_lock_result64 buf64; - res32 = (struct dlm_lock_result *)buf; - /* There are two types of read done here, the first just gets the structure, for that we need our own buffer because the 64 bit one is larger than the 32bit. When the user wants the extended information it has already been told the full (64bit) size of the buffer by the kernel so we can use that buffer for reading, that also avoids the need to copy the extended data blocks too. */ - if (len == sizeof(struct dlm_lock_result)) + if (!data) { - len64 = sizeof(struct dlm_lock_result64); + len64 = sizeof(buf64); res64 = &buf64; } else { - len64 = len; - res64 = (struct dlm_lock_result64 *)buf; + len64 = len32; + res64 = (struct dlm_lock_result64 *)res32; } ret = read(fd, res64, len64); @@ -222,10 +256,47 @@ struct dlm_queryinfo64 *qinfo64; struct dlm_queryinfo *qinfo32; - qinfo64 = (struct dlm_queryinfo64 *)(buf+res32->qinfo_offset); - qinfo32 = (struct dlm_queryinfo *)(buf+res32->qinfo_offset); + qinfo64 = (struct dlm_queryinfo64 *)(res32+res32->qinfo_offset); + qinfo32 = (struct dlm_queryinfo *)(res32+res32->qinfo_offset); qinfo32->gqi_lockcount = qinfo64->gqi_lockcount; } } return ret; } + +ssize_t dlm_read(int fd, struct dlm_lock_result *res) +{ + if (check_biarch_convert()) + return _dlm_read_convert(false, fd, res, 0); + return read(fd, res, sizeof(struct dlm_lock_result)); +} + +ssize_t dlm_read_data(int fd, struct dlm_lock_result *res, size_t len) +{ + if (check_biarch_convert()) + return _dlm_read_convert(true, fd, res, len); + return read(fd, res, len); +} + +ssize_t dlm_write(int fd, struct dlm_write_request *req, size_t len) +{ + if (check_biarch_convert()) + return _dlm_write_convert(fd, req, len); + return write(fd, req, len); +} +#else /* BUILD_BIARCH */ +ssize_t dlm_read(int fd, struct dlm_lock_result *res) +{ + return read(fd, res, sizeof(struct dlm_lock_result)); +} + +ssize_t dlm_read_data(int fd, struct dlm_lock_result *res, size_t len) +{ + return read(fd, res, len); +} + +ssize_t dlm_write(int fd, struct dlm_write_request *req, size_t len) +{ + return write(fd, req, len); +} +#endif /* BUILD_BIARCH */ === lib/libdlm.c ================================================================== --- lib/libdlm.c (revision 308) +++ lib/libdlm.c (local) @@ -46,15 +46,6 @@ #include "libdlm.h" #include "dlm_device.h" -/* Add other grotesqueries here as they arise */ -#if defined(__sparc__) && __WORDSIZE == 32 -#include "dlm32.c" -#else -#define dlm_write write -#define dlm_read read -#endif - - #define MISC_PREFIX "/dev/misc/" #define PROC_MISC "/proc/misc" #define DLM_PREFIX "dlm_" @@ -76,6 +67,10 @@ #endif }; +extern ssize_t dlm_read(int, struct dlm_lock_result *); +extern ssize_t dlm_read_data(int, struct dlm_lock_result *, size_t); +extern ssize_t dlm_write(int, struct dlm_write_request *, size_t); + /* The default lockspace. I've resisted putting locking around this as the user should be "sensible" and only do lockspace operations either in the @@ -399,14 +394,13 @@ static int do_dlm_dispatch(int fd) { - char resultbuf[sizeof(struct dlm_lock_result)]; - struct dlm_lock_result *result = (struct dlm_lock_result *)resultbuf; - char *fullresult=NULL; + struct dlm_lock_result resultbuf; + struct dlm_lock_result *result = &resultbuf, *fullresult = NULL; int status; void (*astaddr)(void *astarg); /* Just read the header first */ - status = dlm_read(fd, result, sizeof(struct dlm_lock_result)); + status = dlm_read(fd, result); if (status <= 0) return -1; @@ -418,7 +412,7 @@ if (!fullresult) return -1; - newstat = dlm_read(fd, fullresult, result->length); + newstat = dlm_read_data(fd, fullresult, result->length); /* If it read OK then use the new data. otherwise we can still deliver the AST, it just might not have all the -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From pcaulfie at redhat.com Mon Feb 14 08:46:52 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Mon, 14 Feb 2005 08:46:52 +0000 Subject: [Linux-cluster] dlm - update dlm32 layer In-Reply-To: <20050213223947.GB28716@wavehammer.waldi.eu.org> References: <20050213223947.GB28716@wavehammer.waldi.eu.org> Message-ID: <20050214084652.GC5724@tykepenguin.com> On Sun, Feb 13, 2005 at 11:39:47PM +0100, Bastian Blank wrote: > Hi folks > > The attached patch adds biarch support to the dlm32 layer of libdlm. It > is currently only enabled on s390 and sparc. I'm working on this, but the patch you sent last seems to break queries (even on i386) and I haven't got the bottom of why yet. -- patrick From adingman at cook-inc.com Mon Feb 14 14:37:37 2005 From: adingman at cook-inc.com (Andrew C. Dingman) Date: Mon, 14 Feb 2005 09:37:37 -0500 Subject: [Linux-cluster] fencing agent for Dell 1855 blades Message-ID: <1108391857.1075.10.camel@adingman.marcomm> Hi, all We're working with Dell 1855 blades in a GFS cluster. They can't be power fenced with an external power switch, and they don't use quite the same DRAC as other Dell systems. I've therefore written a fencing agent to use with them. So far, I've only tested it from the command line. If all goes well, I'll be able to test it with a live GFS cluster this afternoon. If anyone is interested, please take a look and let me know what you think. Ultimately, we'd like to get this included in the main fence distribution. Thanks in advance for any feedback. -- Andrew C. Dingman adingman at cook-inc dot com -------------- next part -------------- A non-text attachment was scrubbed... Name: fence_dracmc Type: application/x-perl Size: 6201 bytes Desc: not available URL: From bastian at waldi.eu.org Sun Feb 13 21:56:30 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Sun, 13 Feb 2005 22:56:30 +0100 Subject: [Linux-cluster] fence - convert manpages to the man macro package Message-ID: <20050213215630.GA9873@wavehammer.waldi.eu.org> Hi folks The current manpages are written in plain nroff which is not parsable by many scripts. The attached patch converts the manpages in the fence package to the man macro package. Bastian -- Schshschshchsch. -- The Gorn, "Arena", stardate 3046.2 -------------- next part -------------- === man/fence.8 ================================================================== --- man/fence.8 (revision 317) +++ man/fence.8 (local) @@ -5,45 +5,66 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence(8)''fence(8)' +.TH fence 8 -\fBNAME\fP -.in +7 +.SH NAME I/O Fencing reference guide -.in -\fBSYNOPSIS\fP -.in +7 +.SH SYNOPSIS Overview of related manual pages -.sp -.in -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION The I/O Fencing documentation has been split into a number of sections. Please refer to the table below to determine which man page coincides with the command/feature you are looking for. -.sp - fence I/O Fencing overview (this man page) - fenced I/O Fencing daemon -I/O Fencing agents +.TP 10 +fence +I/O Fencing overview (this man page) +.TP +fenced +I/O Fencing daemon - fence_apc for APC MasterSwitch and APC 79xx models - fence_bladecenter for IBM Bladecenters w/ telnet interface - fence_brocade for Brocade fibre channel switches (PortDisable) - fence_egenera for Egenera blades - fence_gnbd for GNBD-based GFS clusters - fence_ilo for HP ILO interfaces (formerly fence_rib) - fence_manual for manual intervention - fence_mcdata for McData fibre channel switches - fence_ack_manual for manual intervention - fence_sanbox2 for Qlogic SAN Box fibre channel switches - fence_vixel for Vixel switches (PortDisable) - fence_wti for WTI Network Power Switch - fence_node for use by lock_gulmd +.SS I/O Fencing agents -.sp -.in -\fBSEE ALSO\fP -.in +7 +.TP 20 +fence_apc +for APC MasterSwitch and APC 79xx models +.TP +fence_bladecenter +for IBM Bladecenters w/ telnet interface +.TP +fence_brocade +for Brocade fibre channel switches (PortDisable) +.TP +fence_egenera +for Egenera blades +.TP +fence_gnbd +for GNBD-based GFS clusters +.TP +fence_ilo +for HP ILO interfaces (formerly fence_rib) +.TP +fence_manual +for manual intervention +.TP +fence_mcdata +for McData fibre channel switches +.TP +fence_ack_manual +for manual intervention +.TP +fence_sanbox2 +for Qlogic SAN Box fibre channel switches +.TP +fence_vixel +for Vixel switches (PortDisable) +.TP +fence_wti +for WTI Network Power Switch +.TP +fence_node +for use by lock_gulmd + +.SH SEE ALSO gnbd(8), gfs(8) === man/fence_ack_manual.8 ================================================================== --- man/fence_ack_manual.8 (revision 317) +++ man/fence_ack_manual.8 (local) @@ -5,57 +5,39 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_ack_manual(8)''fence_ack_manual(8)' +.TH fence_ack_manual 8 -\fBNAME\fP -.in +7 +.SH NAME fence_ack_manual - program run by an operator as a part of manual I/O Fencing -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_ack_manual -n\fP \fInodename\fP +.SH SYNOPSIS +.B +fence_ack_manual +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_ack_manual is run by an operator on the same node that fence_manual(8) was run after the operator has reset a node which required fencing. A message in the system log indicates to the operator that they must reset a machine and then run fence_ack_manual. Running fence_ack_manual allows the cluster to continue with recovery of the fenced machine. The victim may be disconnected from storage rather than resetting it. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-O\fP -.in +7 Run without prompting for user confirmation. -.sp -.in +.TP \fB-n\fP \fInodename\fP -.in +7 Name of node that has been reset or disconnected from storage. -.sp -.in +.TP \fB-s\fP \fIIPaddress\fP -.in +7 IP address of the machine which has been reset or disconnected from storage. (Deprecated; use -n instead.) -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_apc.8 ================================================================== --- man/fence_apc.8 (revision 317) +++ man/fence_apc.8 (local) @@ -5,132 +5,85 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_apc(8)''fence_apc(8)' +.TH fence_apc 8 -\fBNAME\fP -.in +7 +.SH NAME fence_apc - I/O Fencing agent for APC MasterSwitch -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_apc -a\fP \fIIPaddress\fR \fB-l\fP \fIlogin\fR \fB-p\fP \fIpassword\fR \fB-n\fP \fIoutlet\fR -[\fB-o\fP action] [\fB-T\fP] [\fB-v\fP] +.SH SYNOPSIS +.B +Bfence_apc +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_apc is an I/O Fencing agent which can be used with the APC MasterSwitch network power switch. It logs into a MasterSwitch via telnet and reboots a specified outlet. Lengthy telnet connections to the MasterSwitch should be avoided while a GFS cluster is running because the connection will block any necessary fencing actions. -.sp + fence_apc accepts options on the command line as well as from stdin. Fenced sends parameters through stdin when it execs the agent. fence_apc can be run by itself with command line options. This is useful for testing and for turning outlets on or off from scripts. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress\fR -.in +7 IP address or hostname of the switch. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-l\fP \fIlogin\fR -.in +7 Login name. -.sp -.in +.TP \fB-n\fP \fI[:]outlet\fR -.in +7 The outlet number to act upon. -.sp -.in +.TP \fB-o\fP \fIaction\fR -.in +7 The action required. Reboot (default), Off or On. -.in -.sp +.TP \fB-p\fP \fIpassword\fR -.in +7 Password for login. -.sp -.in +.TP \fB-T\fP -.in +7 Test only. Answer NO to the confirmation prompt instead of YES. -.in -.sp +.TP \fB-v\fP -.in +7 Verbose. Record telnet session in /tmp/apclog. -.in -.sp +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.in -.sp -.in -7 -\fBSTDIN PARAMETERS\fP -.in -\fI agent = < param >\fR -.sp +.SH STDIN PARAMETERS +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_apc. -.sp - +.TP \fIipaddr = < hostname | ip >\fR -.sp IP address or hostname of the switch. -.sp - -\fI login = < param >\fR -.sp +.TP +\fIlogin = < param >\fR Login name. -.sp - -\fI option = < param >\fR -.sp +.TP +\fIoption = < param >\fR The action required. Reboot (default), Off or On. -.sp - -\fI passwd = < param >\fR -.sp +.TP +\fIpasswd = < param >\fR Password for login. -.sp - -\fI port = < param >\fR -.sp +.TP +\fIport = < param >\fR The outlet number to act upon. -.sp - -\fI switch = < param >\fR -.sp +.TP +\fIswitch = < param >\fR The switch to operate on. Defaults to "1" if not specified. -.sp - -\fI test = < param >\fR -.sp +.TP +\fItest = < param >\fR Test only. Answer NO to the confirmation prompt instead of YES. -.sp - -\fI verbose = < param >\fR -.sp +.TP +\fIverbose = < param >\fR Verbose. Record telnet session in /tmp/apclog. -.sp - -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_baytech.8 ================================================================== --- man/fence_baytech.8 (revision 317) +++ man/fence_baytech.8 (local) @@ -5,21 +5,17 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_baytech(8)''fence_baytech(8)' +.TH fence_baytech 8 -\fBNAME\fP -.in +7 +.SH NAME fence_baytech - I/O Fencing agent for Baytech RPC switches in combination with a Cyclades Terminal Server -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_baytech -a\fP \fIhost\fR \fB-l\fP \fIlogin\fR \fB-p\fP \fIpassword\fR \fB-n\fP \fIoutletname\fR \fB-o\fP \fIaction\fR +.SH SYNOPSIS +.B +fence_baytech +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION This fencing agent is written for the Baytech RPC27-20nc in combination with a Cyclades terminal server. The Cyclades TS exports the RPC's serial port @@ -27,95 +23,61 @@ However, this script relys upon the assumption that Telnet is used. Future features to this agent would allow the agent to work with a mulitude of different communication protocols such as Telnet, SSH or Kermit. -.sp + The other assumption that is made is that Outlet names do not end in space. The name "Foo" and "Foo " are identical when the RPC prints them with the status command. -.sp + fence_baytech accepts options on the command line as well as from stdin. fenced sends parameters through stdin when it execs the agent. fence_baytech can be run by itself with command line options which is useful for testing. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIhost\fP -.in +7 IP address or hostname to connect to. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-l\fP \fIlogin\fP -.in +7 Username name for the switch. -.sp -.in +.TP \fB-n\fP \fIport\fP -.in +7 The name of the outlet to act upon. -.in -.sp +.TP \fB-o\fP \fIaction\fP -.in +7 The action required. disable (default) or enable. -.in -.sp +.TP \fB-p\fP \fIpassword\fP -.in +7 Password for login. -.sp -.in +.TP \fB-q\fP -.in +7 Quiet mode: print only error messages. -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in +.SH STDIN PARAMETERS -\fI agent = < param >\fR -.sp +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_baytech. -.sp - -\fI host = < hostname | ip >\fR -.sp +.TP +\fIhost = < hostname | ip >\fR IP address or hostname to connect to. -.sp - -\fI login = < param >\fR -.sp +.TP +\fIlogin = < param >\fR Login name. -.sp - -\fI action = < param >\fR -.sp +.TP +\fIaction = < param >\fR The action required. On, Off, Status or Reboot (default) -.sp - -\fI passwd = < param >\fR -.sp +.TP +\fIpasswd = < param >\fR Password for login. -.sp - -\fI outlet = < param >\fR -.sp +.TP +\fIoutlet = < param >\fR The name of the outlet to act upon. -.sp - -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_bladecenter.8 ================================================================== --- man/fence_bladecenter.8 (revision 317) +++ man/fence_bladecenter.8 (local) @@ -5,115 +5,73 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_bladecenter(8)''fence_bladecenter(8)' +.TH fence_bladecenter 8 -\fBNAME\fP -.in +7 +.SH NAME fence_brocade - I/O Fencing agent for IBM Bladecenter -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_bladecenter-a\fP \fIIPaddress\fR \fB-l\fP \fIlogin\fR \fB-p\fP \fIpassword\fR \fB-n\fP \fIblade\fR \fB-o\fP \fIaction\fR +.SH SYNOPSIS +.B +fence_bladecenter +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_bladecenter is an I/O Fencing agent which can be used with IBM Bladecenters with recent enough firmware that includes telnet support. It logs into a Brocade chasis via telnet and uses the command line interface to power on and off blades. fence_bladecenter accepts options on the command line or from stdin. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress\fP -.in +7 IP address of the Bladecenter. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-l\fP \fIlogin\fP -.in +7 Login name for the Bladecenter. -.sp -.in +.TP \fB-n\fP \fIblade\fP -.in +7 The blade to operate on. -.in -.sp +.TP \fB-o\fP \fIaction\fP -.in +7 The action required. Valid actions are on, off, reboot (default) and status. -.in -.sp +.TP \fB-p\fP \fIpassword\fP -.in +7 Password for login. -.sp -.in +.TP \fB-q\fP -.in +7 Quiet mode: print only error messages. -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in +.TP \fB-v\fP \fIdebuglog\fP -.in +7 Log the telnet session to \fIdebuglog\fP for debugging purposes. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in - -\fI agent = < param >\fR -.sp -This option is used by fence_node(8) and is ignored by fence_brocade. -.sp - +.SH STDIN PARAMETERS +.TP +\fIagent = < param >\fR +This option is used by fence_node(8) and is ignored by fence_bladecenter. +.TP \fIipaddr = < hostname | ip >\fR -.sp IP address or hostname of the switch. -.sp - -\fI login = < param >\fR -.sp +.TP +\fIlogin = < param >\fR Login name. -.sp - -\fI option = < param >\fR -.sp +.TP +\fIoption = < param >\fR The action required. disable (default) or enable. -.sp - -\fI passwd = < param >\fR -.sp +.TP +\fIpasswd = < param >\fR Password for login. -.sp - -\fI blade = < param >\fR -.sp +.TP +\fIblade = < param >\fR The blade to operate on. -.sp - -\fI debuglog = < param>\fR -.sp +.TP +\fIdebuglog = < param>\fR Optional parameter to send debug transcript of the telnet session to a log file -.sp -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_brocade.8 ================================================================== --- man/fence_brocade.8 (revision 317) +++ man/fence_brocade.8 (local) @@ -5,118 +5,79 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_brocade(8)''fence_brocade(8)' +.TH fence_brocade 8 -\fBNAME\fP -.in +7 +.SH NAME fence_brocade - I/O Fencing agent for Brocade FC switches -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_brocade -a\fP \fIIPaddress\fR \fB-l\fP \fIlogin\fR \fB-p\fP \fIpassword\fR \fB-n\fP \fIport\fR \fB-o\fP \fIaction\fR +.SH SYNOPSIS +.B +fence_brocade +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_brocade is an I/O Fencing agent which can be used with Brocade FC switches. It logs into a Brocade switch via telnet and disables a specified port. Disabling the port which a machine is connected to effectively fences that machine. Lengthy telnet connections to the switch should be avoided while a GFS cluster is running because the connection will block any necessary fencing actions. -.sp + fence_brocade accepts options on the command line as well as from stdin. fenced sends parameters through stdin when it execs the agent. fence_brocade can be run by itself with command line options which is useful for testing. -.sp + After a fence operation has taken place the fenced machine can no longer connect to the Brocade FC switch. When the fenced machine is ready to be brought back into the GFS cluster (after reboot) the port on the Brocade FC switch needs to be enabled. This can be done by running fence_brocade and specifying the enable action. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress\fP -.in +7 IP address of the switch. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-l\fP \fIlogin\fP -.in +7 Login name for the switch. -.sp -.in +.TP \fB-n\fP \fIport\fP -.in +7 The port number to disable on the switch. -.in -.sp +.TP \fB-o\fP \fIaction\fP -.in +7 The action required. disable (default) or enable. -.in -.sp +.TP \fB-p\fP \fIpassword\fP -.in +7 Password for login. -.sp -.in +.TP \fB-q\fP -.in +7 Quiet mode: print only error messages. -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in - -\fI agent = < param >\fR -.sp +.SH STDIN PARAMETERS +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_brocade. -.sp - +.TP \fIipaddr = < hostname | ip >\fR -.sp IP address or hostname of the switch. -.sp - -\fI login = < param >\fR -.sp +.TP +\fIlogin = < param >\fR Login name. -.sp - -\fI option = < param >\fR -.sp +.TP +\fIoption = < param >\fR The action required. disable (default) or enable. -.sp - -\fI passwd = < param >\fR -.sp +.TP +\fIpasswd = < param >\fR Password for login. -.sp - -\fI port = < param >\fR -.sp +.TP +\fIport = < param >\fR The port number to disable on the switch. -.sp - -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_cpint.8 ================================================================== --- man/fence_cpint.8 (revision 317) +++ man/fence_cpint.8 (local) @@ -5,21 +5,17 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_cpint(8)''fence_cpint(8)' +.TH fence_cpint 8 -\fBNAME\fP -.in +7 +.SH NAME fence_cpint - I/O Fencing agent for GFS on s390 and zSeries VM clusters -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_cpint -u\fP \fIuserid\fP +.SH SYNOPSIS +.B +fence_cpint +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_cpint is an I/O Fencing agent used on a virtual machine running GFS in a s390 or zSeries VM cluster. It uses the cpint package to send a CP LOGOFF command to the specified virtual @@ -31,55 +27,33 @@ the secondary user of the virtual machine to be fenced. This means that unless all of you gulm server nodes are privilege class C, fence_cpint can only be used with SLM. -.sp fence_cpint accepts options on the command line as well as from stdin. fence_node sends the options through stdin when it execs the agent. fence_cpint can be run by itself with command line options which is useful for testing. -.sp -.in -\fBOPTIONS\fP -.in +.SH OPTIONS +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp - -.in +.TP \fB-u\fP \fIuserid\fP -.in +7 userid of the virtual machine to fence (required). -.sp - -.in +.TP \fB-q\fP -.in +7 quiet mode, no output. -.sp - -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in +.SH STDIN PARAMETERS +.TP \fIagent = < param >\fR -.sp This option is used by fence_node(8) and is ignored by fence_cpint. -.sp - +.TP \fIuserid = < parm >\fP -.sp userid of the virtual machine to fence (required). -.sp -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_egenera.8 ================================================================== --- man/fence_egenera.8 (revision 317) +++ man/fence_egenera.8 (local) @@ -5,22 +5,17 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_egenera(8)''fence_egenera(8)' +.TH fence_egenera 8 -\fBNAME\fP -.in +7 +.SH NAME fence_egenera - I/O Fencing agent for the Egenera BladeFrame -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_egenera -c\fP \fIcserver\fR \fB-l\fP \fIlpan\fR \fB-p\fP \fIpserver\fR [\fB-o\fP action] -[\fB-q\fP] +.SH SYNOPSIS +.B +fence_egenera +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_egenera is an I/O Fencing agent which can be used with the Egenera BladeFrame. It logs into a control blade (cserver) via ssh and operates on a processing blade (pserver) identified by the pserver name and the @@ -28,87 +23,55 @@ that ssh keys have been setup so that the fence_egenera does not require a password to authenticate. Refer to ssh(8) for more information on setting up ssh keys. -.sp + fence_egenera accepts options on the command line as well as from stdin. Fenced sends parameters through stdin when it execs the agent. fence_egenera can also be run by itself with command line options. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-c\fP \fIcserver\fR -.in +7 The cserver to ssh to. cserver can be in the form user at hostname to specify a different user to login as. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-l\fP \fIlpan\fR -.in +7 the lpan to operate on -.sp -.in +.TP \fB-o\fP \fIaction\fR -.in +7 The action required. reboot (default), off, on or status. -.in -.sp +.TP \fB-p\fP \fIpserver\fR -.in +7 the pserver to operate on -.sp -.in +.TP \fB-q\fP -.in +7 quite mode. supress output. -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.in -.sp -.in -7 -\fBSTDIN PARAMETERS\fP -.in -\fI action = < param >\fR -.sp +.SH STDIN PARAMETERS +.TP +\fIaction = < param >\fR The action required. reboot (default), off, on or status. -.sp - -\fI agent = < param >\fR -.sp +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_apc. -.sp - -\fI cserver = < param >\fR -.sp +.TP +\fIcserver = < param >\fR The cserver to ssh to. cserver can be in the form user at hostname to specify a different user to login as. -.sp - -\fI lpan = < param >\fR -.sp +.TP +\fIlpan = < param >\fR The lpan to operate on -.sp - -\fI pserver = < param >\fR -.sp +.TP +\fIpserver = < param >\fR The pserver to operate on -.sp +.TP +\fIesh = < param >\fR +The path to the esh command on the cserver (default is /opt/panmgr/bin/esh) -\fI esh = < param >\fR -.sp -The path to the esh command on the cserver (default is /opt/panmgr/bin/esh ) -.sp - - -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8), ssh(8) === man/fence_ibmblade.8 ================================================================== --- man/fence_ibmblade.8 (revision 317) +++ man/fence_ibmblade.8 (local) @@ -5,99 +5,64 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_ibmblade(8)''fence_ibmblade(8)' +.TH fence_ibmblade 8 -\fBNAME\fP -.in +7 +.SH NAME fence_ibmblade - I/O Fencing agent for IBM BladeCenter -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_ibmblade -a\fP \fIIPaddress\fR \fB-c\fP \fIcommunity\fR \fB-n\fP \fIport\fR \fB-o\fP \fIaction\fR +.SH SYNOPSIS +.B +fence_ibmblade +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_ibmblade is an I/O Fencing agent which can be used with IBM BladeCenter chassis. It issues SNMP Set request to BladeCenter chassins, rebooting, powering up or down the specified Blade Server. -.sp + fence_ibmblade accepts options on the command line as well as from stdin. fenced sends parameters through stdin when it execs the agent. fence_ibmblade can be run by itself with command line options which is useful for testing. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress\fP -.in +7 IP address of the BladeCenter chassis. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-c\fP \fIcommunity\fP -.in +7 SNMP community string to use. -.sp -.in +.TP \fB-n\fP \fIport\fP -.in +7 The Blade port number to disable. -.in -.sp +.TP \fB-o\fP \fIaction\fP -.in +7 The action required. Reboot (default), On or off. -.in -.sp +.TP \fB-q\fP -.in +7 Quiet mode: print only error messages. -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in - -\fI agent = < param >\fR -.sp +.SH STDIN PARAMETERS +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_ibmblade. -.sp - +.TP \fIipaddr = < hostname | ip >\fR -.sp IP address or hostname of the switch. -.sp - -\fI community = < param >\fR -.sp +.TP +\fIcommunity = < param >\fR SNMP community. -.sp - -\fI option = < param >\fR -.sp +.TP +\fIoption = < param >\fR The action required. reboot (default), on or off. -.sp - -\fI port = < param >\fR -.sp +.TP +\fIport = < param >\fR The Blade port number to disable. -.sp - -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_ilo.8 ================================================================== --- man/fence_ilo.8 (revision 317) +++ man/fence_ilo.8 (local) @@ -5,110 +5,72 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_ilo(8)''fence_ilo(8)' +.TH fence_ilo 8 -\fBNAME\fP -.in +7 +.SH NAME fence_ilo - I/O Fencing agent for HP Integrated Lights Out card -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_ilo -a\fP \fIIPaddress[:SSLport]\fR \fB-l\fP \fIlogin\fR \fB-p\fP \fIpassword\fR -[\fB-o\fP action] [\fB-vqVh\fP] +.SH SYNOPSIS +.B +fence_ilo +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_ilo is an I/O Fencing agent used for HP servers with the Integrated Light Out (iLO) PCI card. The agent opens an SSL connection to the iLO card. Once the SSL connection is established, the agent is able to communicate with the iLO card through an XML stream. -.sp + fence_ilo depends on the Net::SSLeay or Net::SSL perl module in order to establish the SSL connection to the iLO card. Net::SSL is available in the perl-Crypt-SSLeay package on RHN (http://rhn.redhat.com). Net::SSLeay is available at http://www.cpan.org. -.sp + NOTE: fence_ilo deprecates fence_rib. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress[:port]\fR -.in +7 IP address or hostname of the iLO card. If the SSL server of the card is not running on the default SSL port, 443, then [:port] will also need to be specified. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-l\fP \fIlogin\fR -.in +7 Login name. -.sp -.in +.TP \fB-o\fP \fIaction\fR -.in +7 The action required. reboot (default), off, on or status. -.in -.sp +.TP \fB-p\fP \fIpassword\fR -.in +7 Password for login. -.sp -.in +.TP \fB-v\fP -.in +7 Verbose. -.in -.sp +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.in -.sp -.in -7 -\fBSTDIN PARAMETERS\fP -.in - +.SH STDIN PARAMETERS +.TP \fIaction = < param >\fR -.sp The action required. reboot (default), off, on or status. -.sp - +.TP \fIagent = < param >\fR -.sp This option is used by fence_node(8) and is ignored by fence_ilo. -.sp - +.TP \fIhostname = < hostname | ip >\fR -.sp IP address or hostname of the iLO card. -.sp - +.TP \fIlogin = < param >\fR -.sp Login name. -.sp - +.TP \fIpasswd = < param >\fR -.sp Password for login. -.sp - +.TP \fIverbose = < param >\fR -.sp Verbose mode. -.sp - -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8), fence_rib(8) === man/fence_manual.8 ================================================================== --- man/fence_manual.8 (revision 317) +++ man/fence_manual.8 (local) @@ -5,76 +5,52 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_manual(8)''fence_manual(8)' +.TH fence_manual 8 -\fBNAME\fP -.in +7 +.SH NAME fence_manual - program run by fenced as a part of manual I/O Fencing -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_manual -n\fP \fInodename\fP +.SH SYNOPSIS +.B +fence_manual +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_manual is run by fenced. It creates a fifo and waits for its counter-part fence_ack_manual(8) to acknowledge that a failed node has been reset. fence_ack_manual(8) should only be run after the operator has reset the faulty node. While waiting for the manual acknowledgement, fence_manual also watches for the faulty node to rejoin the cluster; if it does, it's taken as an acknowledgement and completes. -.sp + Note: fence_manual is provided for use during testing and evaluation only. Sites should not use fence_manual as the primary fencing method on a production cluster. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-q\fP -.in +7 quiet mode, no output. -.sp -.in +.TP \fB-n\fP \fInodename\fP -.in +7 The node name (usually hostname) of the machine that needs to be reset or disconnected from shared storage. -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in - -\fI agent = < param >\fR -.sp +.SH STDIN PARAMETERS +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_manual. -.sp - -\fI nodename = < param >\fR -.sp +.TP +\fInodename = < param >\fR The node name (usually hostname) of the machine that needs to be reset or disconnected from storage. -.sp - -\fI ipaddr = < param >\fR -.sp +.TP +\fIipaddr = < param >\fR IP address or hostname of the machine that needs to be reset or disconnected from storage. (Deprecated; use nodename instead.) -.in -7 - - -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8), fence_ack_manual(8) === man/fence_mcdata.8 ================================================================== --- man/fence_mcdata.8 (revision 317) +++ man/fence_mcdata.8 (local) @@ -5,118 +5,79 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_mcdata(8)''fence_mcdata(8)' +.TH fence_mcdata 8 -\fBNAME\fP -.in +7 +.SH NAME fence_mcdata - I/O Fencing agent for McData FC switches -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_mcdata -a\fP \fIIPaddress\fR \fB-l\fP \fIlogin\fR \fB-p\fP \fIpassword\fR \fB-n\fP \fIport\fR \fB-o\fP \fIaction\fR +.SH SYNOPSIS +.B +fence_mcdata +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_mcdata is an I/O Fencing agent which can be used with McData FC switches. It logs into a McData switch via telnet and disables a specified port. Disabling the port which a machine is connected to effectively fences that machine. Lengthy telnet connections to the switch should be avoided while a GFS cluster is running because the connection will block any necessary fencing actions. -.sp + fence_mcdata accepts options on the command line as well as from stdin. fenced sends parameters through stdin when it execs the agent. fence_mcdata can be run by itself with command line options which is useful for testing. -.sp + After a fence operation has taken place the fenced machine can no longer connect to the McData FC switch. When the fenced machine is ready to be brought back into the GFS cluster (after reboot) the port on the McData FC switch needs to be enabled. This can be done by running fence_mcdata and specifying the enable action. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress\fP -.in +7 IP address of the switch. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-l\fP \fIlogin\fP -.in +7 Username name for the switch. -.sp -.in +.TP \fB-n\fP \fIport\fP -.in +7 The port number to disable on the switch. -.in -.sp +.TP \fB-o\fP \fIaction\fP -.in +7 The action required. disable (default) or enable. -.in -.sp +.TP \fB-p\fP \fIpassword\fP -.in +7 Password for login. -.sp -.in +.TP \fB-q\fP -.in +7 Quiet mode: print only error messages. -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in - -\fI agent = < param >\fR -.sp +.SH STDIN PARAMETERS +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_mcdata. -.sp - +.TP \fIipaddr = < hostname | ip >\fR -.sp IP address or hostname of the switch. -.sp - -\fI login = < param >\fR -.sp +.TP +\fIlogin = < param >\fR Login name. -.sp - -\fI option = < param >\fR -.sp +.TP +\fIoption = < param >\fR The action required. disable (default) or enable. -.sp - -\fI passwd = < param >\fR -.sp +.TP +\fIpasswd = < param >\fR Password for login. -.sp - -\fI port = < param >\fR -.sp +.TP +\fIport = < param >\fR The port number to disable on the switch. -.sp - -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_node.8 ================================================================== --- man/fence_node.8 (revision 317) +++ man/fence_node.8 (local) @@ -4,50 +4,35 @@ .\" This copyrighted material is made available to anyone wishing to use, .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_node(8)''fence_node(8)' -'\" View with 'groff -t -e -mandoc -Tlatin1 fence_node.8 | less' -\fBNAME\fP -.in +7 +.TH fence_node 8 + +.SH NAME fence_node - A program which performs I/O fencing on a single node. -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_node\fP [\fBoptions\fP] <\fBnode\fP> -.sp -.in -\fBDESCRIPTION\fP -.in +7 +.SH SYNOPSIS +.B +fence_node +[\fIOPTION\fR]... + +.SH DESCRIPTION \fBfence_node\fP is a program which accumulates all the necessary information for I/O fencing a particular node and then performs the fencing action by issuing a call to the proper fencing agent. \fBfence_node\fP gets the necessary information from the Cluster Configuration System (CCS). CCS must be running and properly configured for \fBfence_node\fP to work properly. -.in -\fBOPTIONS\fP -.in +7 +.SH OPTIONS +.TP \fB-h\fP -.in +7 Help. Print out the usage syntax. -.sp -.in - +.TP \fB-V\fP -.in +7 Print version information. -.sp -.in -7 -.in -7 -\fBEXAMPLES\fP -.in +7 + +.SH EXAMPLES To fence a node called ``bellerophon'': -.in +7 prompt> fence_node bellerophon -.in -7 -.sp -.in -7 -\fBSEE ALSO\fP -.in +7 + +.SH SEE ALSO fence(8), ccs(7) === man/fence_rackswitch.8 ================================================================== --- man/fence_rackswitch.8 (revision 317) +++ man/fence_rackswitch.8 (local) @@ -5,102 +5,65 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_rackswitch(8)''fence_rackswitch(8)' +.TH fence_rackswitch 8 -\fBNAME\fP -.in +7 +.SH NAME fence_rackswitch - I/O Fencing agent for RackSaver RackSwitch -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_rackswitch -a\fP \fIIPaddress\fR \fB-p\fP \fIpassword\fR \fB-l\fP \fIusername\fR \fB-n\fP \fIplug\fR +.SH SYNOPSIS +.B +fence_rackswitch +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_rackswitch is an I/O Fencing agent which can be used with the RackSaver RackSwitch. It logs into the RackSwitch and boots a specified plug. Using the http interface to the RackSwitch should be avoided while a GFS cluster is running because the connection may interfere with the operation of this agent. -.sp + fence_rackswitch accepts options on the command line as well as from stdin. fenced sends the options through stdin when it execs the agent. fence_rackswitch can be run by itself with command line options which is useful for testing. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress\fP -.in +7 IP address of the switch. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.in -.sp +.TP \fB-n\fP \fIplug\fP -.in +7 The plug number to power cycle. -.in -.sp +.TP \fB-p\fP \fIpassword\fP -.in +7 Password for login. -.sp -.in -.sp +.TP \fB-l\fP \fIusername\fP -.in +7 Username for login. -.in -.sp +.TP \fB-q\fP -.in +7 Quiet operation. Only print out error messages. -.in -.sp +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.in -.sp -.in -7 -\fBSTDIN PARAMETERS\fP -.in - -\fI agent = < param >\fR -.sp +.SH STDIN PARAMETERS +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_rackswitch. -.sp - +.TP \fIipaddr = < ip >\fR -.sp IP address of the switch. -.sp - -\fI username = < param >\fR -.sp +.TP +\fIusername = < param >\fR Username for login. -.sp - -\fI password = < param >\fR -.sp +.TP +\fIpassword = < param >\fR Password for login. -.sp - -\fI port = < param >\fR -.sp +.TP +\fIport = < param >\fR The port (outlet) number to act upon. -.sp -.in -.sp -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_rib.8 ================================================================== --- man/fence_rib.8 (revision 317) +++ man/fence_rib.8 (local) @@ -5,21 +5,13 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_rib(8)''fence_rib(8)' +.TH fence_rib 8 -\fBNAME\fP -.in +7 +.SH NAME fence_rib - I/O Fencing agent for Compaq Remote Insight Lights Out card -.in -.sp -\fBDESCRIPTION\fP -.in +7 -.sp +.SH DESCRIPTION fence_rib is deprecated. fence_ilo should be used instead -.sp -.in -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence_ilo(8) === man/fence_sanbox2.8 ================================================================== --- man/fence_sanbox2.8 (revision 317) +++ man/fence_sanbox2.8 (local) @@ -5,118 +5,79 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_sanbox2(8)''fence_sanbox2(8)' +.TH fence_sanbox2 8 -\fBNAME\fP -.in +7 +.SH NAME fence_sanbox2 - I/O Fencing agent for QLogic SANBox2 FC switches -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_sanbox2 -a\fP \fIIPaddress\fR \fB-l\fP \fIlogin\fR \fB-p\fP \fIpassword\fR \fB-n\fP \fIport\fR \fB-o\fP \fIaction\fR +.SH SYNOPSIS +.B +fence_sanbox2 +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_sanbox2 is an I/O Fencing agent which can be used with QLogic SANBox2 FC switches. It logs into a SANBox2 switch via telnet and disables a specified port. Disabling the port which a machine is connected to effectively fences that machine. Lengthy telnet connections to the switch should be avoided while a GFS cluster is running because the connection will block any necessary fencing actions. -.sp + fence_sanbox2 accepts options on the command line as well as from stdin. fenced sends parameters through stdin when it execs the agent. fence_sanbox2 can be run by itself with command line options which is useful for testing. -.sp + After a fence operation has taken place the fenced machine can no longer connect to the switch. When the fenced machine is ready to be brought back into the GFS cluster (after reboot) the port on the FC switch needs to be enabled. This can be done by running fence_sanbox2 and specifying the enable action. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress\fP -.in +7 IP address of the switch. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-l\fP \fIlogin\fP -.in +7 Login name for the switch. -.sp -.in +.TP \fB-n\fP \fIport\fP -.in +7 The port number to disable on the switch. -.in -.sp +.TP \fB-o\fP \fIaction\fP -.in +7 The action required. disable (default) or enable. -.in -.sp +.TP \fB-p\fP \fIpassword\fP -.in +7 Password for login. -.sp -.in +.TP \fB-q\fP -.in +7 Quiet mode: print only error messages. -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in - -\fI agent = < param >\fR -.sp +.SH STDIN PARAMETERS +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_sanbox2. -.sp - +.TP \fIipaddr = < hostname | ip >\fR -.sp IP address or hostname of the switch. -.sp - -\fI login = < param >\fR -.sp +.TP +\fIlogin = < param >\fR Login name. -.sp - -\fI option = < param >\fR -.sp +.TP +\fIoption = < param >\fR The action required. disable (default) or enable. -.sp - -\fI passwd = < param >\fR -.sp +.TP +\fIpasswd = < param >\fR Password for login. -.sp - -\fI port = < param >\fR -.sp +.TP +\fIport = < param >\fR The port number to disable on the switch. -.sp - -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_tool.8 ================================================================== --- man/fence_tool.8 (revision 317) +++ man/fence_tool.8 (local) @@ -4,21 +4,19 @@ .\" This copyrighted material is made available to anyone wishing to use, .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_tool(8)''fence_tool(8)' -'\" View with 'groff -t -e -mandoc -Tlatin1 fence_tool.8 | less' -\fBNAME\fP -.in +7 +.TH fence_tool 8 + +.SH NAME fence_tool - A program to join and leave the fence domain -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_tool\fP <\fBjoin | leave\fP> [\fBoptions\fP] -.sp -.in -\fBDESCRIPTION\fP -.in +7 +.SH SYNOPSIS +.B +fence_tool +<\fBjoin | leave\fP> +[\fIOPTION\fR]... + +.SH DESCRIPTION \fBfence_tool\fP is a program used to join or leave the default fence domain. Specifically, it starts the fence daemon (fenced) to join the domain and kills fenced to leave the domain. Fenced can be started @@ -42,54 +40,29 @@ A node must not leave the fence domain (fenced must not be terminated) while CLVM or GFS are in use. -.in -\fBOPTIONS\fP - -.in +7 - +.SH OPTIONS +.TP \fB-h\fP -.in +7 Help. Print out the usage syntax. -.sp -.in - +.TP \fB-V\fP -.in +7 Print version information. -.sp -.in - +.TP \fB-S\fP -.in +7 Skip self unfencing before joining. -.sp -.in - +.TP \fB-D\fP -.in +7 Enable debugging output and don't fork (also passed to fenced) -.sp -.in - +.TP \fB-j\fP \fIsecs\fP -.in +7 Post-join fencing delay (passed to fenced) -.sp -.in - +.TP \fB-f\fP \fIsecs\fP -.in +7 Post-fail fencing delay (passed to fenced) -.sp -.in - +.TP \fB-c\fP -.in +7 All nodes are in a clean state to start (passed to fenced) -.sp -.in -7 -.in -7 -\fBSEE ALSO\fP -.in +7 + +.SH SEE ALSO fenced(8), fence(8), fence_node(8) === man/fence_vixel.8 ================================================================== --- man/fence_vixel.8 (revision 317) +++ man/fence_vixel.8 (local) @@ -5,97 +5,67 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_vixel(8)''fence_vixel(8)' +.TH fence_vixel 8 -\fBNAME\fP -.in +7 +.SH NAME fence_vixel - I/O Fencing agent for Vixel FC switches -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_vixel [-hV] -a\fP \fIIPaddress\fR \fB-p\fP \fIpassword\fR \fB-n\fP \fIport\fR +.SH SYNOPSIS +.B +fence_vixel +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_vixel is an I/O Fencing agent which can be used with Vixel FC switches. It logs into a Vixel switch via telnet and removes the specified port from the zone. Removing the zone access from the port disables the port from being able to access the storage. -.sp + fence_vixel accepts options on the command line as well as from stdin. fenced sends parameters through stdin when it execs the agent. fence_vixel can be run by itself with command line options which is useful for testing. -.sp + After a fence operation has taken place the fenced machine can no longer connect to the Vixel FC switch. When the fenced machine is ready to be brought back into the GFS cluster (after reboot) the port on the Vixel FC switch needs to be enabled. In order to do this, log into the Vixel FC switch. Then go to: -.in -.in +7 + config->zones->config -.in -.in +7 + Then apply -.sp + Consult the Vixel manual for details -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress\fP -.in +7 IP address of the switch. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-n\fP \fIport\fP -.in +7 The port number to remove zoning from on the switch. -.in -.sp +.TP \fB-p\fP \fIpassword\fP -.in +7 Password for login. -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in - +.SH STDIN PARAMETERS +.TP \fIagent = < param >\fR -.sp This option is used by fence_node(8) and is ignored by fence_vixel. -.sp - +.TP \fIipaddr = < hostname | ip >\fR -.sp IP address or hostname of the switch. -.sp - +.TP \fIpasswd = < param >\fR -.sp Password for login. -.sp - +.TP \fIport = < param >\fR -.sp The port number to remove zoning from on the switch. -.sp -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH BSEE ALSO fence(8), fence_node(8) === man/fence_wti.8 ================================================================== --- man/fence_wti.8 (revision 317) +++ man/fence_wti.8 (local) @@ -5,99 +5,65 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_wti(8)''fence_wti(8)' +.TH fence_wti 8 -\fBNAME\fP -.in +7 +.SH NAME fence_wti - I/O Fencing agent for WTI Network Power Switch -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_wti -a\fP \fIIPaddress\fR \fB-p\fP \fIpassword\fR \fB-n\fP \fIplug\fR [\fB-T\fP] +.SH SYNOPSIS +.B +fence_wti +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_wti is an I/O Fencing agent which can be used with the WTI Network Power Switch (NPS). It logs into an NPS via telnet and boots a specified plug. Lengthy telnet connections to the NPS should be avoided while a GFS cluster is running because the connection will block any necessary fencing actions. -.sp + fence_wti accepts options on the command line as well as from stdin. fenced sends the options through stdin when it execs the agent. fence_wti can be run by itself with command line options which is useful for testing. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress\fP -.in +7 IP address of the switch. -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.in -.sp +.TP \fB-n\fP \fIplug\fP -.in +7 The plug number to power cycle. -.in -.sp +.TP \fB-p\fP \fIpassword\fP -.in +7 Password for login. -.sp -.in +.TP \fB-T\fP -.in +7 Test only. Do not power cycle. Reports state of the plug. -.in -.sp +.TP \fB-q\fP -.in +7 Quiet operation. Only print out error messages. -.in -.sp +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.in -.sp -.in -7 -\fBSTDIN PARAMETERS\fP -.in - -\fI agent = < param >\fR -.sp +.SH STDIN PARAMETERS +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_wti. -.sp - +.TP \fIipaddr = < hostname | ip >\fR -.sp IP address or hostname of the switch. -.sp - -\fI passwd = < param >\fR -.sp +.TP +\fIpasswd = < param >\fR Password for login. -.sp - -\fI port = < param >\fR -.sp +.TP +\fIport = < param >\fR The outlet number to act upon. -.sp - -\fI test = < param >\fR -.sp +.TP +\fItest = < param >\fR Test only. Answer NO to the confirmation prompt instead of YES. -.sp -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8) === man/fence_xcat.8 ================================================================== --- man/fence_xcat.8 (revision 317) +++ man/fence_xcat.8 (local) @@ -1,94 +1,63 @@ .\" Copyright (C) Sistina Software, Inc. 1997-2003 All rights reserved. -.tl 'fence_xcat(8)''fence_xcat(8)' +.TH fence_xcat 8 -\fBNAME\fP -.in +7 +.SH NAME fence_xcat - I/O Fencing agent for xcat environments -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_xcat -n\fP \fInodename\fR \fB -o\fP \fIaction\fR \fB -r\fP \fIrpower\fR +.SH SYNOPSIS +.B +fence_xcat +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_xcat is a wrapper to the rpower(1) command that is distributed with the xCAT project available at http://www.xcat.org. Use of fence_xcat requires that xcat has already been properlly configfured for your environment. Refer to xCAT(1) for more information on configuring xCAT. -.sp + fence_xcat accepts options on the command line as well as from stdin. fenced sends parameters through stdin when it execs the agent. fence_xcat can be run by itself with command line options which is useful for testing. -.sp + NOTE: It is recommended that fence_bladecenter(8) is used instead of fence_xcat if the bladecenter firmware supports telnet. This interface is much cleaner and easier to setup. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-n\fP \fInodename\fP -.in +7 The nodename as defined in nodelist.tab of the xCAT setup. -.in -.sp +.TP \fB-o\fP \fIaction\fP -.in +7 The action required. on, off, reset (default) or stat. -.in -.sp +.TP \fB-r\fP \fIrpower\fP -.in +7 The path to the rpower binary. -.sp -.in +.TP \fB-q\fP -.in +7 Quiet mode: print only error messages. -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in - -\fI agent = < param >\fR -.sp +.SH STDIN PARAMETERS +.TP +\fIagent = < param >\fR This option is used by fence_node(8) and is ignored by fence_xcat. -.sp - -\fI nodename = < param >\fR -.sp +.TP +\fInodename = < param >\fR The nodename as defined in nodelist.tab of the xCAT setup. -.sp - -\fI action = < param >\fR -.sp +.TP +\fIaction = < param >\fR The action required. on, off, reset (default) or stat. -.sp - -\fI rpower = < param >\fR -.sp +.TP +\fIrpower = < param >\fR The path to the rpower binary. -.sp - -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fence_node(8), fence_bladecenter(8), nodelist.tab(8), rpower(1), xCAT(1) === man/fence_zvm.8 ================================================================== --- man/fence_zvm.8 (revision 317) +++ man/fence_zvm.8 (local) @@ -5,89 +5,59 @@ .\" modify, copy, or redistribute it subject to the terms and conditions .\" of the GNU General Public License v.2. -.tl 'fence_zvm(8)''fence_zvm(8)' +.TH fence_zvm 8 -\fBNAME\fP -.in +7 +.SH NAME fence_zvm - I/O Fencing agent for GFS on s390 and zSeries VM clusters -.in -\fBSYNOPSIS\fP -.in +7 -\fBfence_zvm -a\fP \fIIPaddress\fP \fB-u\fP \fIuserid\fP \fB-p\fP \fIpassword\fP +.SH SYNOPSIS +.B +fence_zvm +[\fIOPTION\fR]... -.in -.sp -\fBDESCRIPTION\fP -.in +7 +.SH DESCRIPTION fence_zvm is an I/O Fencing agent used on a GFS virtual machine in a s390 or zSeries VM cluster. It uses the s3270 program to log the specified virtual machine out of VM. For fence_zvm to execute correctly, you must have s3270 in your PATH. -.sp fence_zvm accepts options on the command line as well as from stdin. fence_node sends the options through stdin when it execs the agent. fence_zvm can be run by itself with command line options which is useful for testing. -.sp -.in -\fBOPTIONS\fP -.in + +.SH OPTIONS +.TP \fB-a\fP \fIIPaddress\fP -.in +7 IP address or hostname of the Physical machine (required). -.sp -.in +.TP \fB-h\fP -.in +7 Print out a help message describing available options, then exit. -.sp -.in +.TP \fB-u\fP \fIuserid\fP -.in +7 userid of the virtual machine to fence (required). -.sp -.in +.TP \fB-p\fP \fIpassword\fP -.in +7 password of the virtual machine to fence (required). -.sp -.in +.TP \fB-q\fP -.in +7 quiet mode, no output. -.sp -.in +.TP \fB-V\fP -.in +7 Print out a version message, then exit. -.sp -.in -.in -7 -\fBSTDIN PARAMETERS\fP -.in +.SH STDIN PARAMETERS +.TP \fIagent = < param >\fP -.sp This option is used by fence_node(8) and is ignored by fence_zvm. -.sp - +.TP \fIipaddr = < hostname | ip >\fP -.sp IP address or hostname of the Physical machine (required). -.sp - +.TP \fIpasswd = < param >\fP -.sp password of the virtual machine to fence (required). -.sp - +.TP \fIuserid = < param >\fP -.sp userid of the virtual machine to fence (required). -.sp -.in -7 -\fBSEE ALSO\fP -.in +7 +.SH SEE ALSO fence(8), fenced(8), fence_node(8) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From pcaulfie at redhat.com Mon Feb 14 15:12:32 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Mon, 14 Feb 2005 15:12:32 +0000 Subject: [Linux-cluster] cluster lost quorum after 11 hours In-Reply-To: <1108169257.5927.12.camel@ibm-c.pdx.osdl.net> References: <1108169257.5927.12.camel@ibm-c.pdx.osdl.net> Message-ID: <20050214151232.GC13778@tykepenguin.com> On Fri, Feb 11, 2005 at 04:47:38PM -0800, Daniel McNeil wrote: > I was running my test on a 3 node cluster and it died > after 11 hours. cl030 lost quorum with the other 2 nodes > kicked out of the cluster. cl031 also hit a bunch of asserts > like > lock_dlm: Assertion failed on line 352 of file > /Views/redhat-cluster/cluster/gfs-kernel/src/dlm/lock.c > lock_dlm: assertion: "!error" > lock_dlm: time = 291694516 > stripefs: error=-22 num=2,19 > I assume is caused by the cluster shutting down. > > > /var/log/messages showed: > > cl030: > Feb 11 02:44:33 cl030 kernel: CMAN: removing node cl032a from the cluster : No response to messages > Feb 11 02:44:33 cl030 kernel: CMAN: removing node cl031a from the cluster : No response to messages > Feb 11 02:44:33 cl030 kernel: CMAN: quorum lost, blocking activity > Feb 11 14:40:33 cl030 sshd(pam_unix)[27323]: session opened for user root by (uid=0) You should only get nodes dying from "No response to messages" during a state transition of some sort (eg a node leaving or joining or possibly a GFS mount/dismount). In which case the DLM has to do recovery. I recently checked in a couple of changes that will stop the DLM recovery from taking over the machine when there are several thousand locks to recover, that might help. During a normal "steady" state, a node should not die from "No response to messages" because the only messages that are being sent are HELLO heartbeat messages and they are not acked. -- patrick From danderso at redhat.com Mon Feb 14 15:15:25 2005 From: danderso at redhat.com (Derek Anderson) Date: Mon, 14 Feb 2005 09:15:25 -0600 Subject: [Linux-cluster] config update kills cluster In-Reply-To: <20050212154216.GA3682@wavehammer.waldi.eu.org> References: <20050212154216.GA3682@wavehammer.waldi.eu.org> Message-ID: <200502140915.25156.danderso@redhat.com> On Saturday 12 February 2005 09:42, Bastian Blank wrote: > Hi all > > I just tried config update via ccs_tool in a cman cluster. Each of the > nodes got the new config but the kernel rejects joins with > > | CMAN: Join request from gfs1 rejected, config version local 1 remote 2 After updating the config file with ccs_tool you should notify cman of the new config version. So, like 'cman_tool version -r 2', where 2 is your updated version. Please try this and see if nodes can then join. > > The cluster is running CVS from 2005-02-06. > > Bastian From yazan at ccs.com.jo Mon Feb 14 15:36:09 2005 From: yazan at ccs.com.jo (Yazan Al-Sheyyab) Date: Mon, 14 Feb 2005 17:36:09 +0200 Subject: [Linux-cluster] mail on cluster Message-ID: <001901c512aa$e7d45600$69050364@yazanz> hi all, i want to begin a project about making a mail server on tow clustered servers. i want to know if i need the GFS during my work (because i dont have a shared storage here) and if any software from erdhat can i buy for mailserver or is the send mail service built on the system enough to do so in use? i want any to help me or to give me what i need to have and to do before i begin this task. what i need to begin? i know some of you are using this project , i really need to know what i need exactlly befor begin? (GFS,suites,raws........ ????). Regards. From ialberdi at histor.fr Mon Feb 14 16:59:05 2005 From: ialberdi at histor.fr (Ion Alberdi) Date: Mon, 14 Feb 2005 17:59:05 +0100 Subject: [Linux-cluster] Unique device on each node Message-ID: <4210D8D9.6060201@histor.fr> I'm working with rgmanager, and i have a question. Does a device needed by a service put in failover, must have the same name on each node the service could eventually run? More precisely is this configuration: a service that needs to mount */dev/hda1 in /mnt/fs when he runs on node1 */dev/hdb1 in /mnt/fs running on node2 impossible to implement with this cluster? The configuration file cluster.conf is the same for all the nodes and I haven't already found a way to say in the cluster.conf: in node1 do that,in node2 do that, so I assume it is imposible. I would like to have a confirmation for that. Thanks in advance for all the eventual answers. From bastian at waldi.eu.org Mon Feb 14 17:06:51 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Mon, 14 Feb 2005 18:06:51 +0100 Subject: [Linux-cluster] dlm - update dlm32 layer In-Reply-To: <20050214084652.GC5724@tykepenguin.com> References: <20050213223947.GB28716@wavehammer.waldi.eu.org> <20050214084652.GC5724@tykepenguin.com> Message-ID: <20050214170651.GD20443@wavehammer.waldi.eu.org> On Mon, Feb 14, 2005 at 08:46:52AM +0000, Patrick Caulfield wrote: > I'm working on this, but the patch you sent last seems to break queries (even on > i386) and I haven't got the bottom of why yet. My first patch was broken, I know. But how can I check if it works? clvmd does not report errors and seems to work correctly on i386. Bastian -- You! What PLANET is this! -- McCoy, "The City on the Edge of Forever", stardate 3134.0 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From bujan at isqsolutions.com Mon Feb 14 18:06:54 2005 From: bujan at isqsolutions.com (Manuel Bujan) Date: Mon, 14 Feb 2005 13:06:54 -0500 Subject: [Linux-cluster] ccsd error in new cvs References: <20050213223947.GB28716@wavehammer.waldi.eu.org><20050214084652.GC5724@tykepenguin.com> <20050214170651.GD20443@wavehammer.waldi.eu.org> Message-ID: <00ef01c512bf$f71c07b0$5001a8c0@pcbujan> Helo We are trying to install a gfs cluster from scratch based on current cvs code and we are getting the following when we tried to run ccsd: Unable to connect to cluster infrastructure after 30 seconds. Unable to connect to cluster infrastructure after 60 seconds. Unable to connect to cluster infrastructure after 90 seconds. Unable to connect to cluster infrastructure after 120 seconds. Unable to connect to cluster infrastructure after 150 seconds. ................ Any hints ? We are running kernel 2.6.10, and we succesfully compiled everything inside. Regards Bujan From yazan at ccs.com.jo Mon Feb 14 18:33:49 2005 From: yazan at ccs.com.jo (Yazan Al-Sheyyab) Date: Mon, 14 Feb 2005 20:33:49 +0200 Subject: [Linux-cluster] raw device Message-ID: <011701c512c3$ba1a84f0$69050364@yazanz> hi all, maybe i asked this question before So please execuse me about this. can i put more than one rawdevice in the same partition, i mean can i put in the /etc/sysconfig/rawdevices file more than one raw related with same partition as : /dev/raw/raw5 /dev/cciss/c0d0p3 /dev/raw/raw6 /dev/cciss/c0d0p3 .......... etc. ????????????????/ and if so or not can i use partition generated by LVM for rawdevices ??????? it maybe a rediculus question for you but i really need to know about. Thanks alot if you can help me. Regards. From jbrassow at redhat.com Mon Feb 14 18:41:46 2005 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Mon, 14 Feb 2005 12:41:46 -0600 Subject: [Linux-cluster] ccs - fix -p parameter of ccsd In-Reply-To: <20050213222659.GA28716@wavehammer.waldi.eu.org> References: <20050213222659.GA28716@wavehammer.waldi.eu.org> Message-ID: Did you need the -p parameter? Right now, the code is there to handle it, but it is disabled. It is also not mentioned in the man page or in the usage summary. This parameter was something that we thought about including but no consensus was ever reached - which is why the code is there, but the option is not handled. brassow On Feb 13, 2005, at 4:26 PM, Bastian Blank wrote: > Hi folks > > The attached patch fixes the -p parameter of ccsd. > > Bastian > > -- > It would be illogical to assume that all conditions remain stable. > -- Spock, "The Enterprise Incident", stardate 5027.3 > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From bastian at waldi.eu.org Mon Feb 14 18:46:20 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Mon, 14 Feb 2005 19:46:20 +0100 Subject: [Linux-cluster] ccs - fix -p parameter of ccsd In-Reply-To: References: <20050213222659.GA28716@wavehammer.waldi.eu.org> Message-ID: <20050214184620.GA28838@wavehammer.waldi.eu.org> On Mon, Feb 14, 2005 at 12:41:46PM -0600, Jonathan E Brassow wrote: > Did you need the -p parameter? I currently use it in the debian packages to overwrite the pid file location. Bastian -- In the strict scientific sense we all feed on death -- even vegetarians. -- Spock, "Wolf in the Fold", stardate 3615.4 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From jbrassow at redhat.com Mon Feb 14 18:48:14 2005 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Mon, 14 Feb 2005 12:48:14 -0600 Subject: [Linux-cluster] ccsd error in new cvs In-Reply-To: <00ef01c512bf$f71c07b0$5001a8c0@pcbujan> References: <20050213223947.GB28716@wavehammer.waldi.eu.org><20050214084652.GC5724@tykepenguin.com> <20050214170651.GD20443@wavehammer.waldi.eu.org> <00ef01c512bf$f71c07b0$5001a8c0@pcbujan> Message-ID: <3d015bc4b486a840262cc7e97ae4358f@redhat.com> This is not an error. It simply means that you haven't started the cluster manager yet. (That is, you haven't run 'cman_tool join -c ' yet -- or lock_gulmd if you are using gulm.) brassow On Feb 14, 2005, at 12:06 PM, Manuel Bujan wrote: > Helo > > We are trying to install a gfs cluster from scratch based on current > cvs code and we are getting the following when we tried to run ccsd: > > Unable to connect to cluster infrastructure after 30 seconds. > Unable to connect to cluster infrastructure after 60 seconds. > Unable to connect to cluster infrastructure after 90 seconds. > Unable to connect to cluster infrastructure after 120 seconds. > Unable to connect to cluster infrastructure after 150 seconds. > ................ > > Any hints ? > > We are running kernel 2.6.10, and we succesfully compiled everything > inside. > > Regards > Bujan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From bastian at waldi.eu.org Mon Feb 14 18:47:47 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Mon, 14 Feb 2005 19:47:47 +0100 Subject: [Linux-cluster] raw device In-Reply-To: <011701c512c3$ba1a84f0$69050364@yazanz> References: <011701c512c3$ba1a84f0$69050364@yazanz> Message-ID: <20050214184747.GB28838@wavehammer.waldi.eu.org> On Mon, Feb 14, 2005 at 08:33:49PM +0200, Yazan Al-Sheyyab wrote: > can i put more than one rawdevice in the same partition Raw devices are deprecated, use O_DIRECT on the standard device. Bastian -- If there are self-made purgatories, then we all have to live in them. -- Spock, "This Side of Paradise", stardate 3417.7 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From bujan at isqsolutions.com Mon Feb 14 19:14:17 2005 From: bujan at isqsolutions.com (Manuel Bujan) Date: Mon, 14 Feb 2005 14:14:17 -0500 Subject: [Linux-cluster] ccsd error in new cvs References: <20050213223947.GB28716@wavehammer.waldi.eu.org><20050214084652.GC5724@tykepenguin.com><20050214170651.GD20443@wavehammer.waldi.eu.org><00ef01c512bf$f71c07b0$5001a8c0@pcbujan> <3d015bc4b486a840262cc7e97ae4358f@redhat.com> Message-ID: <013c01c512c9$6297d010$5001a8c0@pcbujan> Hello, I'm sorry may be our problem is related with cman instead of with ccsd. Any way in any case when I tried to run cman I get a segmentation fault Regards Bujan ---------ccsd trace--------------------- Starting ccsd DEVEL.1108401572: Built: Feb 14 2005 12:20:28 Copyright (C) Red Hat, Inc. 2004 All rights reserved. No Daemon:: SET cluster.conf (cluster name = ISQCLUSTER, version = 14) found. Unable to connect to cluster infrastructure after 30 seconds. -------cman trace------------------------ # cman_tool join -c ISQCLUSTER multicast address 224.0.0.9 multicast address 224.0.0.1 multicast address 224.0.0.9 multicast address 224.0.0.1 multicast address 224.0.0.9 multicast address 224.0.0.1 .......... Segmentation fault That's what are we getting Regards #################### cluster.conf: #################### ######################################### ----- Original Message ----- From: "Jonathan E Brassow" To: "linux clistering" Sent: Monday, February 14, 2005 1:48 PM Subject: Re: [Linux-cluster] ccsd error in new cvs > This is not an error. It simply means that you haven't started the > cluster manager yet. (That is, you haven't run 'cman_tool join -c > ' yet -- or lock_gulmd if you are using gulm.) > > brassow > > On Feb 14, 2005, at 12:06 PM, Manuel Bujan wrote: > >> Helo >> >> We are trying to install a gfs cluster from scratch based on current cvs >> code and we are getting the following when we tried to run ccsd: >> >> Unable to connect to cluster infrastructure after 30 seconds. >> Unable to connect to cluster infrastructure after 60 seconds. >> Unable to connect to cluster infrastructure after 90 seconds. >> Unable to connect to cluster infrastructure after 120 seconds. >> Unable to connect to cluster infrastructure after 150 seconds. >> ................ >> >> Any hints ? >> >> We are running kernel 2.6.10, and we succesfully compiled everything >> inside. >> >> Regards >> Bujan >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> http://www.redhat.com/mailman/listinfo/linux-cluster >> > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From jbrassow at redhat.com Mon Feb 14 19:53:31 2005 From: jbrassow at redhat.com (Jonathan E Brassow) Date: Mon, 14 Feb 2005 13:53:31 -0600 Subject: [Linux-cluster] ccsd error in new cvs In-Reply-To: <013c01c512c9$6297d010$5001a8c0@pcbujan> References: <20050213223947.GB28716@wavehammer.waldi.eu.org><20050214084652.GC5724@tykepenguin.com><20050214170651.GD20443@wavehammer.waldi.eu.org><00ef01c512bf$f71c07b0$5001a8c0@pcbujan> <3d015bc4b486a840262cc7e97ae4358f@redhat.com> <013c01c512c9$6297d010$5001a8c0@pcbujan> Message-ID: Try the attached patch. This patch has been applied to the code in cvs brassow -------------- next part -------------- A non-text attachment was scrubbed... Name: join_ccs.c-patch Type: application/octet-stream Size: 637 bytes Desc: not available URL: -------------- next part -------------- On Feb 14, 2005, at 1:14 PM, Manuel Bujan wrote: > Hello, > > I'm sorry may be our problem is related with cman instead of with > ccsd. Any way in any case when I tried to run cman I get a > segmentation fault > > Regards > Bujan > > ---------ccsd trace--------------------- > Starting ccsd DEVEL.1108401572: > Built: Feb 14 2005 12:20:28 > Copyright (C) Red Hat, Inc. 2004 All rights reserved. > No Daemon:: SET > > cluster.conf (cluster name = ISQCLUSTER, version = 14) found. > Unable to connect to cluster infrastructure after 30 seconds. > > -------cman trace------------------------ > # cman_tool join -c ISQCLUSTER > multicast address 224.0.0.9 > multicast address 224.0.0.1 > multicast address 224.0.0.9 > multicast address 224.0.0.1 > multicast address 224.0.0.9 > multicast address 224.0.0.1 > .......... > > Segmentation fault > > That's what are we getting > > Regards > > #################### > cluster.conf: > #################### > > > > > > > > > > > > > > port="1"/> > > > > > > > > > > port="2"/> > > > > > > ipaddr="192.168. > 0.120" login="apc" passwd="r3hd3"/> > > > > ######################################### > > > > > ----- Original Message ----- From: "Jonathan E Brassow" > > To: "linux clistering" > Sent: Monday, February 14, 2005 1:48 PM > Subject: Re: [Linux-cluster] ccsd error in new cvs > > >> This is not an error. It simply means that you haven't started the >> cluster manager yet. (That is, you haven't run 'cman_tool join -c >> ' yet -- or lock_gulmd if you are using gulm.) >> >> brassow >> >> On Feb 14, 2005, at 12:06 PM, Manuel Bujan wrote: >> >>> Helo >>> >>> We are trying to install a gfs cluster from scratch based on current >>> cvs code and we are getting the following when we tried to run ccsd: >>> >>> Unable to connect to cluster infrastructure after 30 seconds. >>> Unable to connect to cluster infrastructure after 60 seconds. >>> Unable to connect to cluster infrastructure after 90 seconds. >>> Unable to connect to cluster infrastructure after 120 seconds. >>> Unable to connect to cluster infrastructure after 150 seconds. >>> ................ >>> >>> Any hints ? >>> >>> We are running kernel 2.6.10, and we succesfully compiled everything >>> inside. >>> >>> Regards >>> Bujan >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> http://www.redhat.com/mailman/listinfo/linux-cluster >>> >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> http://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster > From bastian at waldi.eu.org Mon Feb 14 20:35:06 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Mon, 14 Feb 2005 21:35:06 +0100 Subject: [Linux-cluster] fence - fence_ack_manual does not check nodename Message-ID: <20050214203506.GA7595@wavehammer.waldi.eu.org> Hi folks Is it expected that fence_ack_manual wants a nodename but do't checks them? Bastian -- I'm a soldier, not a diplomat. I can only tell the truth. -- Kirk, "Errand of Mercy", stardate 3198.9 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From bujan at isqsolutions.com Mon Feb 14 20:45:14 2005 From: bujan at isqsolutions.com (Manuel Bujan) Date: Mon, 14 Feb 2005 15:45:14 -0500 Subject: [Linux-cluster] ccsd error in new cvs References: <20050213223947.GB28716@wavehammer.waldi.eu.org><20050214084652.GC5724@tykepenguin.com><20050214170651.GD20443@wavehammer.waldi.eu.org><00ef01c512bf$f71c07b0$5001a8c0@pcbujan><3d015bc4b486a840262cc7e97ae4358f@redhat.com><013c01c512c9$6297d010$5001a8c0@pcbujan> Message-ID: <017c01c512d6$17e593b0$5001a8c0@pcbujan> Thanks for the patch, Now everything is working fine and we are able to start the cluster and mount a GFS partition of around 100 GB. I checked the logs and everything appears to be OK, except one line related to the LVM2 that state: Feb 14 15:31:36 atmail-1 lvm[4546]: locking_type not set correctly in lvm.conf, cluster operations will not work. Feb 14 15:34:15 atmail-1 /sbin/hotplug: no runnable /etc/hotplug/block.agent is installed I'm testing only with one of our nodes before upgrade to the second one because we one to be sure everything is working. What that message means ? Globally our lvm.conf has the following setup: > locking_type = 1 > locking_dir = "/var/lock/lvm" Which value of locking_type has to be used with a two node cman based cluster ? Regards Bujan ----- Original Message ----- From: "Jonathan E Brassow" To: "linux clistering" Sent: Monday, February 14, 2005 2:53 PM Subject: Re: [Linux-cluster] ccsd error in new cvs > Try the attached patch. > > This patch has been applied to the code in cvs > > brassow > > -------------------------------------------------------------------------------- > > > > On Feb 14, 2005, at 1:14 PM, Manuel Bujan wrote: > >> Hello, >> >> I'm sorry may be our problem is related with cman instead of with >> ccsd. Any way in any case when I tried to run cman I get a >> segmentation fault >> >> Regards >> Bujan >> >> ---------ccsd trace--------------------- >> Starting ccsd DEVEL.1108401572: >> Built: Feb 14 2005 12:20:28 >> Copyright (C) Red Hat, Inc. 2004 All rights reserved. >> No Daemon:: SET >> >> cluster.conf (cluster name = ISQCLUSTER, version = 14) found. >> Unable to connect to cluster infrastructure after 30 seconds. >> >> -------cman trace------------------------ >> # cman_tool join -c ISQCLUSTER >> multicast address 224.0.0.9 >> multicast address 224.0.0.1 >> multicast address 224.0.0.9 >> multicast address 224.0.0.1 >> multicast address 224.0.0.9 >> multicast address 224.0.0.1 >> .......... >> >> Segmentation fault >> >> That's what are we getting >> >> Regards >> >> #################### >> cluster.conf: >> #################### >> >> >> >> >> >> >> >> >> >> >> >> >> >> > port="1"/> >> >> >> >> >> >> >> >> >> >> > port="2"/> >> >> >> >> >> >> > ipaddr="192.168. >> 0.120" login="apc" passwd="r3hd3"/> >> >> >> >> ######################################### >> >> >> >> >> ----- Original Message ----- From: "Jonathan E Brassow" >> >> To: "linux clistering" >> Sent: Monday, February 14, 2005 1:48 PM >> Subject: Re: [Linux-cluster] ccsd error in new cvs >> >> >>> This is not an error. It simply means that you haven't started the >>> cluster manager yet. (That is, you haven't run 'cman_tool join -c >>> ' yet -- or lock_gulmd if you are using gulm.) >>> >>> brassow >>> >>> On Feb 14, 2005, at 12:06 PM, Manuel Bujan wrote: >>> >>>> Helo >>>> >>>> We are trying to install a gfs cluster from scratch based on current >>>> cvs code and we are getting the following when we tried to run ccsd: >>>> >>>> Unable to connect to cluster infrastructure after 30 seconds. >>>> Unable to connect to cluster infrastructure after 60 seconds. >>>> Unable to connect to cluster infrastructure after 90 seconds. >>>> Unable to connect to cluster infrastructure after 120 seconds. >>>> Unable to connect to cluster infrastructure after 150 seconds. >>>> ................ >>>> >>>> Any hints ? >>>> >>>> We are running kernel 2.6.10, and we succesfully compiled everything >>>> inside. >>>> >>>> Regards >>>> Bujan >>>> >>>> -- >>>> Linux-cluster mailing list >>>> Linux-cluster at redhat.com >>>> http://www.redhat.com/mailman/listinfo/linux-cluster >>>> >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> http://www.redhat.com/mailman/listinfo/linux-cluster >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> http://www.redhat.com/mailman/listinfo/linux-cluster >> > -------------------------------------------------------------------------------- > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > http://www.redhat.com/mailman/listinfo/linux-cluster From danderso at redhat.com Mon Feb 14 20:58:29 2005 From: danderso at redhat.com (Derek Anderson) Date: Mon, 14 Feb 2005 14:58:29 -0600 Subject: [Linux-cluster] ccsd error in new cvs In-Reply-To: <017c01c512d6$17e593b0$5001a8c0@pcbujan> References: <20050213223947.GB28716@wavehammer.waldi.eu.org> <017c01c512d6$17e593b0$5001a8c0@pcbujan> Message-ID: <200502141458.29093.danderso@redhat.com> On Monday 14 February 2005 14:45, Manuel Bujan wrote: > Thanks for the patch, > Now everything is working fine and we are able to start the cluster and > mount a GFS partition of around 100 GB. > > I checked the logs and everything appears to be OK, except one line related > to the LVM2 that state: > > Feb 14 15:31:36 atmail-1 lvm[4546]: locking_type not set correctly in > lvm.conf, cluster operations will not work. > Feb 14 15:34:15 atmail-1 /sbin/hotplug: no runnable > /etc/hotplug/block.agent is installed > > I'm testing only with one of our nodes before upgrade to the second one > because we one to be sure everything is working. > What that message means ? > > Globally our lvm.conf has the following setup: > > locking_type = 1 > > locking_dir = "/var/lock/lvm" > > Which value of locking_type has to be used with a two node cman based > cluster ? locking_type = 2 locking_library = "/usr/lib/liblvm2clusterlock.so" The locking_dir can be left the way it is. You will still get a warning when starting clvmd, but it can be ignored, and should be fixed for future builds: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=147819 From amanthei at redhat.com Mon Feb 14 21:09:05 2005 From: amanthei at redhat.com (Adam Manthei) Date: Mon, 14 Feb 2005 15:09:05 -0600 Subject: [Linux-cluster] fence - fence_ack_manual does not check nodename In-Reply-To: <20050214203506.GA7595@wavehammer.waldi.eu.org> References: <20050214203506.GA7595@wavehammer.waldi.eu.org> Message-ID: <20050214210905.GB29436@redhat.com> On Mon, Feb 14, 2005 at 09:35:06PM +0100, Bastian Blank wrote: > Hi folks > > Is it expected that fence_ack_manual wants a nodename but do't checks > them? yes. In fact, you can put whatever you want in "ipaddr" and I believe that it should work provided that a unique key is used for all nodes that are using fence_manual. I'm not 100% sure though since it's been a while since I looked at that code. The key "ipaddr" is a misleading name. -- Adam Manthei From pbruna at linuxcenterla.com Mon Feb 14 21:41:51 2005 From: pbruna at linuxcenterla.com (Patricio Bruna V) Date: Mon, 14 Feb 2005 18:41:51 -0300 Subject: [Linux-cluster] where i have to look? Message-ID: <1108417311.2733.17.camel@p.linuxcenter.cl> im seen a lot of ha projects, im a little miss, so im asking a bit of tips. what are the tecnologys that rules? openssi or ultramonkey? gfs, dlm or lustre? what book, site, etc, i have to study. thx -- Patricio Bruna http://www.linuxcenterla.com Ingeniero de Proyectos Mariano S?nchez Fontecilla 310 Red Hat Certified Engineer Las Condes, Santiago - CHILE Linux Center Latinoamerica Fono: 4834041 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From teigland at redhat.com Tue Feb 15 04:05:49 2005 From: teigland at redhat.com (David Teigland) Date: Tue, 15 Feb 2005 12:05:49 +0800 Subject: [Linux-cluster] possible to wait on fence domain startup In-Reply-To: <20050213195955.GE16192@wavehammer.waldi.eu.org> References: <20050213195955.GE16192@wavehammer.waldi.eu.org> Message-ID: <20050215040549.GA5395@redhat.com> On Sun, Feb 13, 2005 at 08:59:55PM +0100, Bastian Blank wrote: > Hi folks > > Is it possible to wait for the startup of the fence domain? > > If I call "fence_tool join" and mount of a gfs volume without a sleep > between, I get permission denied and the kernel log reports the missing > fence domain. > > The time which is needed between this two calls seems to be related to > the number of nodes in the fence domain. Now fixed by doing "fence_tool join -w" -- Dave Teigland From teigland at redhat.com Tue Feb 15 04:37:59 2005 From: teigland at redhat.com (David Teigland) Date: Tue, 15 Feb 2005 12:37:59 +0800 Subject: [Linux-cluster] fence - convert manpages to the man macro package In-Reply-To: <20050213215630.GA9873@wavehammer.waldi.eu.org> References: <20050213215630.GA9873@wavehammer.waldi.eu.org> Message-ID: <20050215043759.GB5395@redhat.com> On Sun, Feb 13, 2005 at 10:56:30PM +0100, Bastian Blank wrote: > Hi folks > > The current manpages are written in plain nroff which is not parsable by > many scripts. > > The attached patch converts the manpages in the fence package to the man Thanks, added. From fajar at telkom.co.id Tue Feb 15 06:52:21 2005 From: fajar at telkom.co.id (Fajar A. Nugraha) Date: Tue, 15 Feb 2005 13:52:21 +0700 Subject: [Linux-cluster] cluster latest cvs does not fence dead nodes automatically Message-ID: <42119C25.2000207@telkom.co.id> Hi, I'm building two-node cluster using today's cvs from sources.redhat.com:/cvs/cluster. Shared storage is located on FC shared disk. All work as expected up to using gfs. When I simulated a node crash (I did ifcfg eth0 down on node 2), node 1 simply says (on syslog): Feb 15 13:33:35 hosting-cl02-01 CMAN: removing node hosting-cl02-02 from the cluster : Missed too many heartbeats However, NO fencing occured. Not even a "fence failed" message. I use fence_ibmblade. After that, access to gfs device blocked (df -k still works though), and /proc/cluster/nodes show Node Votes Exp Sts Name 1 1 1 M node-01 2 1 1 X node-02 there's an "X" on node 2, but /proc/cluster/service shows Service Name GID LID State Code Fence Domain: "default" 1 2 run - [1 2] DLM Lock Space: "clvmd" 2 3 run - [1 2] DLM Lock Space: "data" 3 4 run - [1 2] DLM Lock Space: "config" 5 6 run - [1 2] GFS Mount Group: "data" 4 5 run - [1 2] GFS Mount Group: "config" 6 7 run - [1 2] which is the same content with before node 2 is dead. AFAIK, state should be "recover" or "waiting to recover" instead of run. If I reboot node 2 (which is the same thing if you exceute fence_ibmblade manually), and restart cluster services on that node, all is back to normal, and these messages show on syslog : Feb 15 13:38:40 node-01 CMAN: node node-02 rejoining Feb 15 13:38:40 node-01 fenced[25486]: node-02 not a cluster member after 0 sec post_fail_delay Feb 15 13:38:42 node-01 GFS: fsid=node:config.0: jid=1: Trying to acquire journal lock... Feb 15 13:38:42 node-01 GFS: fsid=node:data.0: jid=1: Trying to acquire journal lock... Feb 15 13:38:43 node-01 GFS: fsid=node:config.0: jid=1: Looking at journal... Feb 15 13:38:43 node-01 GFS: fsid=node:data.0: jid=1: Looking at journal... Feb 15 13:38:43 node-01 GFS: fsid=node:config.0: jid=1: Acquiring the transaction lock... Feb 15 13:38:43 node-01 GFS: fsid=node:config.0: jid=1: Replaying journal... Feb 15 13:38:43 node-01 GFS: fsid=node:config.0: jid=1: Replayed 0 of 0 blocks Feb 15 13:38:43 node-01 GFS: fsid=node:config.0: jid=1: replays = 0, skips = 0, sames = 0 Feb 15 13:38:43 node-01 GFS: fsid=node:data.0: jid=1: Acquiring the transaction lock... Feb 15 13:38:43 node-01 GFS: fsid=node:data.0: jid=1: Replaying journal... Feb 15 13:38:43 node-01 GFS: fsid=node:data.0: jid=1: Replayed 0 of 0 blocks Feb 15 13:38:43 node-01 GFS: fsid=node:data.0: jid=1: replays = 0, skips = 0, sames = 0 Feb 15 13:38:43 node-01 GFS: fsid=node:data.0: jid=1: Journal replayed in 1s Feb 15 13:38:43 node-01 GFS: fsid=node:data.0: jid=1: Done Feb 15 13:38:43 node-01 GFS: fsid=node:config.0: jid=1: Journal replayed in 1s Feb 15 13:38:43 node-01 GFS: fsid=node:config.0: jid=1: Done Any idea what's wrong? Regards, Fajar From fajar at telkom.co.id Tue Feb 15 06:58:12 2005 From: fajar at telkom.co.id (Fajar A. Nugraha) Date: Tue, 15 Feb 2005 13:58:12 +0700 Subject: [Linux-cluster] cluster latest cvs does not fence dead nodes automatically In-Reply-To: <42119C25.2000207@telkom.co.id> References: <42119C25.2000207@telkom.co.id> Message-ID: <42119D84.6030403@telkom.co.id> Fajar A. Nugraha wrote: > node 1 simply says (on syslog): > > Feb 15 13:33:35 hosting-cl02-01 CMAN: removing node hosting-cl02-02 > from the cluster : Missed too many heartbeats > sorry, wrong message (this was another pair of nodes) :) should be Feb 15 13:36:03 node-01 CMAN: removing node node-02 from the cluster : Missed too many heartbeats Other messages are correct. From teigland at redhat.com Tue Feb 15 07:13:59 2005 From: teigland at redhat.com (David Teigland) Date: Tue, 15 Feb 2005 15:13:59 +0800 Subject: [Linux-cluster] cluster latest cvs does not fence dead nodes automatically In-Reply-To: <42119C25.2000207@telkom.co.id> References: <42119C25.2000207@telkom.co.id> Message-ID: <20050215071359.GC5395@redhat.com> On Tue, Feb 15, 2005 at 01:52:21PM +0700, Fajar A. Nugraha wrote: > Hi, > > I'm building two-node cluster using today's cvs from > sources.redhat.com:/cvs/cluster. > Shared storage is located on FC shared disk. > All work as expected up to using gfs. > > When I simulated a node crash (I did ifcfg eth0 down on node 2), > node 1 simply says (on syslog): > > Feb 15 13:33:35 hosting-cl02-01 CMAN: removing node hosting-cl02-02 from > the cluster : Missed too many heartbeats > > However, NO fencing occured. Not even a "fence failed" message. I use > fence_ibmblade. > After that, access to gfs device blocked (df -k still works though), and > /proc/cluster/nodes show > > Node Votes Exp Sts Name > 1 1 1 M node-01 > 2 1 1 X node-02 It looks like the node names are wrong. There were some recent changes to how we deal with node names, but I don't see how this setup could ever have worked, even with previous code. The node names you put in cluster.conf must match the name on the network interface you want cman to use. It looks like the machine's name is "hosting-cl02-02". Is that the hostname on the network interface you want cman to use? If so, then that's the name you should enter in cluster.conf, and that's the name that should appear when you run "cman_tool status" and "cman_tool nodes". > there's an "X" on node 2, but /proc/cluster/service shows > > Service Name GID LID State Code > Fence Domain: "default" 1 2 run - > [1 2] > > DLM Lock Space: "clvmd" 2 3 run - > [1 2] > > DLM Lock Space: "data" 3 4 run - > [1 2] > > DLM Lock Space: "config" 5 6 run - > [1 2] > > GFS Mount Group: "data" 4 5 run - > [1 2] > > GFS Mount Group: "config" 6 7 run - > [1 2] > > which is the same content with before node 2 is dead. > AFAIK, state should be "recover" or "waiting to recover" instead of run. > > If I reboot node 2 (which is the same thing if you exceute > fence_ibmblade manually), and restart cluster services on that node, all > is back to normal, > and these messages show on syslog : > > Feb 15 13:38:40 node-01 CMAN: node node-02 rejoining > Feb 15 13:38:40 node-01 fenced[25486]: node-02 not a cluster member > after 0 sec post_fail_delay Above the names were "hosting-cl02-01" and "hosting-cl02-02". Could you clear that up and if there are still problems send your cluster.conf file? Thanks -- Dave Teigland From fajar at telkom.co.id Tue Feb 15 07:32:59 2005 From: fajar at telkom.co.id (Fajar A. Nugraha) Date: Tue, 15 Feb 2005 14:32:59 +0700 Subject: [Linux-cluster] cluster latest cvs does not fence dead nodes automatically In-Reply-To: <20050215071359.GC5395@redhat.com> References: <42119C25.2000207@telkom.co.id> <20050215071359.GC5395@redhat.com> Message-ID: <4211A5AB.6000701@telkom.co.id> David Teigland wrote: >It looks like the node names are wrong. There were some recent changes to >how we deal with node names, but I don't see how this setup could ever >have worked, even with previous code. > >The node names you put in cluster.conf must match the name on the network >interface you want cman to use. > It is. cluster node names is identical to each node's hostname, and those ip addresses are in /etc/hosts. > It looks like the machine's name is >"hosting-cl02-02". > My mistake. That particular line was from the wrong server :) > Is that the hostname on the network interface you want >cman to use? If so, then that's the name you should enter in >cluster.conf, and that's the name that should appear when you run >"cman_tool status" and "cman_tool nodes". > > > I'll make a new setup, using new node names (in cluster.conf too, just in case), try cman_tool and post the result in a few minutes. Regards, Fajar From fajar at telkom.co.id Tue Feb 15 08:34:44 2005 From: fajar at telkom.co.id (Fajar A. Nugraha) Date: Tue, 15 Feb 2005 15:34:44 +0700 Subject: [Linux-cluster] cluster latest cvs does not fence dead nodes automatically In-Reply-To: <20050215071359.GC5395@redhat.com> References: <42119C25.2000207@telkom.co.id> <20050215071359.GC5395@redhat.com> Message-ID: <4211B424.4090202@telkom.co.id> David Teigland wrote: >Above the names were "hosting-cl02-01" and "hosting-cl02-02". Could you >clear that up and if there are still problems send your cluster.conf file? >Thanks > > > Here's how it is now. Using new hostnames and cluster.conf (blade center's IP address and community string removed): ================================== =========================================== Commands and their output (console or syslog): # modprobe gfs # modprobe lock_dlm Feb 15 15:10:04 cluster-node1 Lock_Harness (built Feb 15 2005 12:00:38) installed Feb 15 15:10:04 cluster-node1 GFS (built Feb 15 2005 12:00:52) installed Feb 15 15:10:08 cluster-node1 CMAN (built Feb 15 2005 12:00:31) installed Feb 15 15:10:08 cluster-node1 NET: Registered protocol family 30 Feb 15 15:10:08 cluster-node1 DLM (built Feb 15 2005 12:00:34) installed Feb 15 15:10:08 cluster-node1 Lock_DLM (built Feb 15 2005 12:00:39) installed dm-mod is built-in in the kernel (not a module) # ccsd -V ccsd DEVEL.1108443619 (built Feb 15 2005 12:01:01) Copyright (C) Red Hat, Inc. 2004 All rights reserved. # ccsd -4 Feb 15 15:10:58 cluster-node1 ccsd[8556]: Starting ccsd DEVEL.1108443619: Feb 15 15:10:58 cluster-node1 ccsd[8556]: Built: Feb 15 2005 12:01:01 Feb 15 15:10:58 cluster-node1 ccsd[8556]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Feb 15 15:10:58 cluster-node1 ccsd[8556]: IP Protocol:: IPv4 only # cman_tool join Feb 15 15:12:27 cluster-node1 ccsd[8556]: cluster.conf (cluster name = cluster, version = 3) found. Feb 15 15:12:28 cluster-node1 CMAN: Waiting to join or form a Linux-cluster Feb 15 15:12:28 cluster-node1 ccsd[8558]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1 Feb 15 15:12:28 cluster-node1 ccsd[8558]: Initial status:: Inquorate Feb 15 15:13:00 cluster-node1 CMAN: forming a new cluster Feb 15 15:13:00 cluster-node1 CMAN: quorum regained, resuming activity Feb 15 15:13:00 cluster-node1 ccsd[8558]: Cluster is quorate. Allowing connections. # cman_tool status Protocol version: 5.0.1 Config version: 3 Cluster name: cluster Cluster ID: 13364 Membership state: Cluster-Member Nodes: 1 Expected_votes: 1 Total_votes: 1 Quorum: 1 Active subsystems: 0 Node name: cluster-node1 Node addresses: 192.168.192.146 # cman_tool nodes Node Votes Exp Sts Name 1 1 1 M cluster-node1 # fence_tool join Feb 15 15:14:26 cluster-node1 fenced[8847]: cluster-node2 not a cluster member after 6 sec post_join_delay Feb 15 15:14:26 cluster-node1 fenced[8847]: fencing node "cluster-node2" Feb 15 15:14:32 cluster-node1 fenced[8847]: fence "cluster-node2" success at this point "cluster-node2" was fenced and automatically rebooted, which is good. Now I join the cluster-node2 to the cluster : # modprobe gfs # modprobe lock_dlm # cman_tool join # fence_tool join Feb 15 15:18:30 cluster-node2 ccsd[8376]: Starting ccsd DEVEL.1108443619: Feb 15 15:18:30 cluster-node2 ccsd[8376]: Built: Feb 15 2005 12:01:01 Feb 15 15:18:30 cluster-node2 ccsd[8376]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Feb 15 15:18:30 cluster-node2 ccsd[8376]: IP Protocol:: IPv4 only Feb 15 15:18:34 cluster-node2 ccsd[8376]: cluster.conf (cluster name = cluster, version = 3) found. Feb 15 15:18:34 cluster-node2 ccsd[8376]: Remote copy of cluster.conf is from quorate node. Feb 15 15:18:34 cluster-node2 ccsd[8376]: Local version # : 3 Feb 15 15:18:34 cluster-node2 ccsd[8376]: Remote version #: 3 Feb 15 15:18:41 cluster-node2 Lock_Harness (built Feb 15 2005 12:00:38) installed Feb 15 15:18:41 cluster-node2 GFS (built Feb 15 2005 12:00:52) installed Feb 15 15:18:44 cluster-node2 CMAN (built Feb 15 2005 12:00:31) installed Feb 15 15:18:44 cluster-node2 NET: Registered protocol family 30 Feb 15 15:18:44 cluster-node2 DLM (built Feb 15 2005 12:00:34) installed Feb 15 15:18:44 cluster-node2 Lock_DLM (built Feb 15 2005 12:00:39) installed Feb 15 15:18:47 cluster-node2 ccsd[8376]: Remote copy of cluster.conf is from quorate node. Feb 15 15:18:47 cluster-node2 ccsd[8376]: Local version # : 3 Feb 15 15:18:47 cluster-node2 ccsd[8376]: Remote version #: 3 Feb 15 15:18:47 cluster-node2 CMAN: Waiting to join or form a Linux-cluster Feb 15 15:18:48 cluster-node2 ccsd[8378]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1 Feb 15 15:18:48 cluster-node2 ccsd[8378]: Initial status:: Inquorate Feb 15 15:18:50 cluster-node2 CMAN: sending membership request Feb 15 15:18:50 cluster-node2 CMAN: got node cluster-node1 Feb 15 15:18:50 cluster-node2 CMAN: quorum regained, resuming activity Feb 15 15:18:50 cluster-node2 ccsd[8378]: Cluster is quorate. Allowing connections. on node 1 : # clvmd Feb 15 15:24:56 cluster-node1 CMAN: WARNING no listener for port 11 on node cluster-node2 on node 2 : # clvmd Feb 15 15:25:03 cluster-node2 clvmd: Cluster LVM daemon started - connected to CMAN on node 1 : # cman_tool nodes Node Votes Exp Sts Name 1 1 1 M cluster-node1 2 1 1 M cluster-node2 # cman_tool services Service Name GID LID State Code Fence Domain: "default" 1 2 run - [1 2] DLM Lock Space: "clvmd" 3 3 run - [1 2] # cman_tool status Protocol version: 5.0.1 Config version: 3 Cluster name: cluster Cluster ID: 13364 Membership state: Cluster-Member Nodes: 2 Expected_votes: 1 Total_votes: 2 Quorum: 1 Active subsystems: 3 Node name: cluster-node1 Node addresses: 192.168.192.146 Now I shutdown node2's network interface. On node 2 : # ifconfig eth0 down On node 1 : Feb 15 15:29:50 cluster-node1 CMAN: removing node cluster-node2 from the cluster : Missed too many heartbeats # cman_tool status Protocol version: 5.0.1 Config version: 3 Cluster name: cluster Cluster ID: 13364 Membership state: Cluster-Member Nodes: 2 Expected_votes: 1 Total_votes: 2 Quorum: 1 Active subsystems: 3 Node name: cluster-node1 Node addresses: 192.168.192.146 # cman_tool status Protocol version: 5.0.1 Config version: 3 Cluster name: cluster Cluster ID: 13364 Membership state: Cluster-Member Nodes: 1 Expected_votes: 1 Total_votes: 1 Quorum: 1 Active subsystems: 3 Node name: cluster-node1 Node addresses: 192.168.192.146 # cman_tool nodes Node Votes Exp Sts Name 1 1 1 M cluster-node1 2 1 1 X cluster-node2 # cman_tool services Service Name GID LID State Code Fence Domain: "default" 1 2 run - [1 2] DLM Lock Space: "clvmd" 3 3 run - [1 2] No note about fencing whatsoever, and node 2 is not automatically rebooted. Shouldn't node 2 get fenced here? Regards, Fajar From pcaulfie at redhat.com Tue Feb 15 08:38:35 2005 From: pcaulfie at redhat.com (Patrick Caulfield) Date: Tue, 15 Feb 2005 08:38:35 +0000 Subject: [Linux-cluster] dlm - update dlm32 layer In-Reply-To: <20050214170651.GD20443@wavehammer.waldi.eu.org> References: <20050213223947.GB28716@wavehammer.waldi.eu.org> <20050214084652.GC5724@tykepenguin.com> <20050214170651.GD20443@wavehammer.waldi.eu.org> Message-ID: <20050215083835.GA11831@tykepenguin.com> On Mon, Feb 14, 2005 at 06:06:51PM +0100, Bastian Blank wrote: > On Mon, Feb 14, 2005 at 08:46:52AM +0000, Patrick Caulfield wrote: > > I'm working on this, but the patch you sent last seems to break queries (even on > > i386) and I haven't got the bottom of why yet. > > My first patch was broken, I know. But how can I check if it works? The second one was too - I've fixed it up and committed it now. > clvmd does not report errors and seems to work correctly on i386. > clvmd doesn't use the query interface so, yes, it will work fine. dlm/tests/usertest/dlmtest -Q exercises (in a limited way) the query facility. patrick From teigland at redhat.com Tue Feb 15 09:42:29 2005 From: teigland at redhat.com (David Teigland) Date: Tue, 15 Feb 2005 17:42:29 +0800 Subject: [Linux-cluster] cluster latest cvs does not fence dead nodes automatically In-Reply-To: <4211B424.4090202@telkom.co.id> References: <42119C25.2000207@telkom.co.id> <20050215071359.GC5395@redhat.com> <4211B424.4090202@telkom.co.id> Message-ID: <20050215094229.GG5395@redhat.com> On Tue, Feb 15, 2005 at 03:34:44PM +0700, Fajar A. Nugraha wrote: > # cman_tool nodes > Node Votes Exp Sts Name > 1 1 1 M cluster-node1 > 2 1 1 X cluster-node2 > > # cman_tool services > Service Name GID LID State Code > Fence Domain: "default" 1 2 run - > [1 2] > > DLM Lock Space: "clvmd" 3 3 run - > [1 2] > > No note about fencing whatsoever, and node 2 is not automatically rebooted. > Shouldn't node 2 get fenced here? Yes, a checkin last Friday left out a line to wake up cman_serviced. Fixed now. Thanks -- Dave Teigland From crsurf at terra.com.br Tue Feb 15 16:03:58 2005 From: crsurf at terra.com.br (crsurf) Date: Tue, 15 Feb 2005 13:03:58 -0300 Subject: [Linux-cluster] kernel 2.4.21-20.EL and GNBD Message-ID: Hello I'm trying to configure GNBD to use as shared device (raw) to simulate a shared storage to use with RHCS, but I'm having problems because the two nodes of the cluster are rebooting after start clumanager. I can use the packges in sources.redhat.com/cluster with kernel 2.4.21-20 to run GNBD and other packages requisiteds? Someone can help me to configure RHCS with GNBD as shared storage? Grateful Cristiano From ggilyeat at jhsph.edu Tue Feb 15 16:09:30 2005 From: ggilyeat at jhsph.edu (Gerald G. Gilyeat) Date: Tue, 15 Feb 2005 11:09:30 -0500 Subject: [Linux-cluster] GFS 6.0 Questions Message-ID: Good morning, y'all: A bit of background on our current setup before I get into the issues we've been having, and my questions for the list :) We've been running a 32node computer cluster with two head nodes attached to a 4TB EMC CX300 for ~9 months now. Attached to the storage device are two additional four-way systems. The 2 heads (call them f0 and1) and one of the four-ways (call it e0) are currently configured as GFS servers, and the second four-way is strictly a client to the GFS system (call it c0). The 4TB are divided into 4 RAID5 arrays: 2x 800GB, 1x730GB, 1x 1.4TB (roughly), each formatted and included in the GFS side of things. f0 and f1 are dual Opteron 248s w/4GB RAM, e0 is a quad Opty 848 w/16GB RAM and c0 is a . An additional system (t0) is planned to replace e0 as the third GFS server, relegating e0 to client-only status, as soon as I can finish building it (it's a dual Opty 248 w/6GB RAM). First, the GFS side of things is currently sharing the cluster's internal network for it's communications, mostly because we didn't have a second switch to dedicate to the task. While the cluster is currently lightly used, how sub-optimal is this? I'm currently searching for another switch that a partnering department has/had, but I don't know if they even know where it is at this point. Second: GFS likes to fence "e0" off on a fairly regular/common basis (once every other week or so, if not more often). This is really rather bad for us, from an operational standpoint - e0 is vital to the operation of our Biostatistics Department (Samba/NFS, user authentication, etc...). There is also some pretty nasty latency on occasion, with logins taking upwards of 30seconds to return to a prompt, providing it doesn't time out to begin with. In trying to figure out -why- it's constantly being fenced off, and in trying to solve the latency/performance issues, I've noticed a -very- large number of "notices" from GFS like the following: Feb 15 10:56:10 front-1 lock_gulmd_LT000[4073]: Lock count is at 1124832 which is more than the max 1048576. Sending Drop all req to clients Easy enough to gather that we're blowing away the current lock highwater mark. Is upping the highwater point a feasable thing to do -and- would it have an affect on performance, and what would that affect be? This weekend, we also noticed another weirdness (for us, anyways...) - e0 was fenced off on Saturday morning at 0504.09am, almost precisely 24 hours later e0 decided that the problem was the previous GFS master (f0), arbitrated itself to be Master, took over, fenced off F0 and then proceeded to hose the entire thing by the time I heard about things and was able to get on-site to bring it all back up (at 1am Monday morning). What is this apparent 24-hour timer, and is this expected behaviour? Finally - would increasing the heartbeat timer and the number of acceptable misses an appropriate and acceptable way to help decreases the frequency of e0 being fenced off? Sorry for the long post and the rambling. I've been fighting with this mess off and on for 6 months now. It's only now coming to a head because people are no longer able to get an "real" work done, computationally, on the system. And with researchers, not being able to get your computations done in time for your presentation next week is a really -=bad=- thing. Thanks! -- Jerry Gilyeat, RHCE Systems Administrator Molecular Microbiology and Immunology Johns Hopkins Bloomberg School of Public Health -------------- next part -------------- An HTML attachment was scrubbed... URL: From KaiSpeckmann at gmx.de Tue Feb 15 10:53:22 2005 From: KaiSpeckmann at gmx.de (KaiSpeckmann at gmx.de) Date: Tue, 15 Feb 2005 11:53:22 +0100 (MET) Subject: [Linux-cluster] mounting gfs fs on the first node locks all journals Message-ID: <17729.1108464802@www37.gmx.net> Hi, The cluster consists of two nodes: - a partition on node1 is configured as pool device. It should be used for cluster storage - node1 exports its device via gnbd and serves as lock_gulm master - node2 imports this device and is logged in as lock_gulm client on node1 - ccs config archives are stored locally on both machines. Until i try to mount the gfs fs on the first node everythings seems to be find to me. But when i try to mount the pool device (/dev/pool/pool1) on the first node, it acquires journal lock for all existing journals on the gfs fs, making it imposible to mount it from another node. Can someone tell me why this happens ? Here are my ccs config files: fence_devices { c1-locksrv { agent = "fence_gnbd" server = "cluster1" } manual-reset { agent = "fence_manual" } } nodes { cluster1 { ip_interfaces { eth0 = "192.168.0.1" } fence { fenceCluster1 { manual-reset { ipaddr = "192.168.0.1" } } } } cluster2 { ip_interfaces { eth0 = "192.168.0.2" } fence { fenceCluster2 { c1-locksrv { ipaddr = "192.168.0.2" } } } } } cluster { name = "rac" lock_gulm { servers = ["cluster1"] heartbeat_rate = 3 allowed_misses = 10 } } -- DSL Komplett von GMX +++ Superg?nstig und stressfrei einsteigen! AKTION "Kein Einrichtungspreis" nutzen: http://www.gmx.net/de/go/dsl From bastian at waldi.eu.org Tue Feb 15 16:47:14 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Tue, 15 Feb 2005 17:47:14 +0100 Subject: [Linux-cluster] Re: cluster/fence/fence_tool fence_tool.c In-Reply-To: <20050215035441.19043.qmail@sourceware.org> References: <20050215035441.19043.qmail@sourceware.org> Message-ID: <20050215164656.GB5824@wavehammer.waldi.eu.org> On Tue, Feb 15, 2005 at 03:54:41AM -0000, teigland at sourceware.org wrote: > Log message: > Add option to fence_tool to wait for the node to complete its join and > be a member of the fence domain. Two options: > fence_tool join -w > fence_tool join; fence_tool wait I think it may be better to implement the -w argument in fenced. It just need to fork after the kernel reports the successfull join not before. This will also remove one dependency to the not machine readable output of procfs. Bastian -- Intuition, however illogical, is recognized as a command prerogative. -- Kirk, "Obsession", stardate 3620.7 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From teigland at redhat.com Tue Feb 15 17:01:46 2005 From: teigland at redhat.com (David Teigland) Date: Wed, 16 Feb 2005 01:01:46 +0800 Subject: [Linux-cluster] Re: cluster/fence/fence_tool fence_tool.c In-Reply-To: <20050215164656.GB5824@wavehammer.waldi.eu.org> References: <20050215035441.19043.qmail@sourceware.org> <20050215164656.GB5824@wavehammer.waldi.eu.org> Message-ID: <20050215170146.GB17487@redhat.com> On Tue, Feb 15, 2005 at 05:47:14PM +0100, Bastian Blank wrote: > On Tue, Feb 15, 2005 at 03:54:41AM -0000, teigland at sourceware.org wrote: > > Log message: > > Add option to fence_tool to wait for the node to complete its join and > > be a member of the fence domain. Two options: > > fence_tool join -w > > fence_tool join; fence_tool wait > > I think it may be better to implement the -w argument in fenced. It just > need to fork after the kernel reports the successfull join not before. > > This will also remove one dependency to the not machine readable output > of procfs. No, the join (the ioctl performed by fenced) is asynchronous. The only way to really tell that it's complete is to monitor the proc file. (I know this may not be very nice, but it's not going to change in this generation of the code. If you don't want to use proc, don't use -w.) -- Dave Teigland From bastian at waldi.eu.org Tue Feb 15 16:59:32 2005 From: bastian at waldi.eu.org (Bastian Blank) Date: Tue, 15 Feb 2005 17:59:32 +0100 Subject: [Linux-cluster] commit mails Message-ID: <20050215165932.GC5824@wavehammer.waldi.eu.org> Hi folks Is it possible to get commit mails inclusive diffs? As I often want to know what really changed, I have to check each file by hand. Bastian -- I'm a soldier, not a diplomat. I can only tell the truth. -- Kirk, "Errand of Mercy", stardate 3198.9 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From mtilstra at redhat.com Tue Feb 15 17:43:26 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Tue, 15 Feb 2005 11:43:26 -0600 Subject: [Linux-cluster] GFS 6.0 Questions In-Reply-To: References: Message-ID: <421234BE.90207@redhat.com> Gerald G. Gilyeat wrote: [snip] > First, the GFS side of things is currently sharing the cluster's > internal network for it's communications, mostly because we didn't have > a second switch to dedicate to the task. While the cluster is currently > lightly used, how sub-optimal is this? I'm currently searching for > another switch that a partnering department has/had, but I don't know if > they even know where it is at this point. It really depends on how much the actual link is used. The more data that the other apps are pushing over the ethernet, the less of it gulm can use. It is also rather (unfortunately) difficult to tell gulm to use a different network device in the current releases. There is a fix pending for this, but its not out yet. > Second: GFS likes to fence "e0" off on a fairly regular/common basis > (once every other week or so, if not more often). This is really rather > bad for us, from an operational standpoint - e0 is vital to the > operation of our Biostatistics Department (Samba/NFS, user > authentication, etc...). There is also some pretty nasty latency on > occasion, with logins taking upwards of 30seconds to return to a prompt, > providing it doesn't time out to begin with. If the machine is getting this kind of delay, it is completely possible that the delay is also causing heartbeats to be missed. > In trying to figure out -why- it's constantly being fenced off, and in > trying to solve the latency/performance issues, I've noticed a -very- > large number of "notices" from GFS like the following: > Feb 15 10:56:10 front-1 lock_gulmd_LT000[4073]: Lock count is at 1124832 > which is more than the max 1048576. Sending Drop all req to clients > > Easy enough to gather that we're blowing away the current lock highwater > mark. > Is upping the highwater point a feasable thing to do -and- would it have > an affect on performance, and what would that affect be? cluster.ccs: cluster { lock_gulm { .... lt_high_locks = } } The highwater mark is an attempt to keep the amount of memory lock_gulmd uses down. When the highwater is hit, the lock server tells all gfs mounts to try and release locks. It does this every 10 seconds until the lock count falls below the highwater mark. This requires cycles, and so not doing it means less cycles used. The higher the highwater mark is, the more memory the gulm lock servers and gfs will use to store locks. The number is just the count of locks (in <=6.0) and not an actual representation of ram used. In short summery, in your case, a higher highwater mark may give some performance gained, at the loss of some memory available to other programs. > This weekend, we also noticed another weirdness (for us, anyways...) - > e0 was fenced off on Saturday morning at 0504.09am, almost precisely 24 > hours later e0 decided that the problem was the previous GFS master > (f0), arbitrated itself to be Master, took over, fenced off F0 and then > proceeded to hose the entire thing by the time I heard about things and > was able to get on-site to bring it all back up (at 1am Monday morning). > What is this apparent 24-hour timer, and is this expected behaviour? No, it sounds like some kind of freak chance. A very icky thing indeed. Very much sounds like a higher heartbeat_rate is needed. > Finally - would increasing the heartbeat timer and the number of > acceptable misses an appropriate and acceptable way to help decreases > the frequency of e0 being fenced off? Certainly. The default values for the heartbeat_rate and allowed_misses are just suggestions. Certain setups may require different values, and as far as I know the only way to figure this out is to try it. Sounds very much like you could use larger values. -- michael conrad tadpol tilstra -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 256 bytes Desc: OpenPGP digital signature URL: From ggilyeat at jhsph.edu Tue Feb 15 17:59:46 2005 From: ggilyeat at jhsph.edu (Gerald G. Gilyeat) Date: Tue, 15 Feb 2005 12:59:46 -0500 Subject: [Linux-cluster] GFS 6.0 Questions Message-ID: Thanks a bunch. The direction I was leaning on going, then, seems appropriate. I love it when things start coming together. Is there anyway to get some of these undocumented tunable features, well, documented? I couldn't for the life of me find anything indicating if the lock highwater mark was runtime tunable, for example. There is -some- concern about memory usage tanking things, but that will probably end up leading us to simply moving to dedicated locking servers instead of having them on the actual shared production machines (and really, we'd only need two...'f1' is strictly for management type work and backups...) Finally - so while it's -possible- to have the GFS "stuff" on a separate interface (and yes, it was a royal PITA getting it to work in the first place what with multiple NICs already...), it's not somthing that's at all easy to do, at least until the mentioned fix drops? bleh. Thanks! -- Jerry Gilyeat, RHCE Systems Administrator Molecular Microbiology and Immunology Johns Hopkins Bloomberg School of Public Health -----Original Message----- From: linux-cluster-bounces at redhat.com on behalf of Michael Conrad Tadpol Tilstra Sent: Tue 2/15/2005 12:43 PM To: linux clistering Subject: Re: [Linux-cluster] GFS 6.0 Questions Gerald G. Gilyeat wrote: [snip] > First, the GFS side of things is currently sharing the cluster's > internal network for it's communications, mostly because we didn't have > a second switch to dedicate to the task. While the cluster is currently > lightly used, how sub-optimal is this? I'm currently searching for > another switch that a partnering department has/had, but I don't know if > they even know where it is at this point. It really depends on how much the actual link is used. The more data that the other apps are pushing over the ethernet, the less of it gulm can use. It is also rather (unfortunately) difficult to tell gulm to use a different network device in the current releases. There is a fix pending for this, but its not out yet. > Second: GFS likes to fence "e0" off on a fairly regular/common basis > (once every other week or so, if not more often). This is really rather > bad for us, from an operational standpoint - e0 is vital to the > operation of our Biostatistics Department (Samba/NFS, user > authentication, etc...). There is also some pretty nasty latency on > occasion, with logins taking upwards of 30seconds to return to a prompt, > providing it doesn't time out to begin with. If the machine is getting this kind of delay, it is completely possible that the delay is also causing heartbeats to be missed. > In trying to figure out -why- it's constantly being fenced off, and in > trying to solve the latency/performance issues, I've noticed a -very- > large number of "notices" from GFS like the following: > Feb 15 10:56:10 front-1 lock_gulmd_LT000[4073]: Lock count is at 1124832 > which is more than the max 1048576. Sending Drop all req to clients > > Easy enough to gather that we're blowing away the current lock highwater > mark. > Is upping the highwater point a feasable thing to do -and- would it have > an affect on performance, and what would that affect be? cluster.ccs: cluster { lock_gulm { .... lt_high_locks = } } The highwater mark is an attempt to keep the amount of memory lock_gulmd uses down. When the highwater is hit, the lock server tells all gfs mounts to try and release locks. It does this every 10 seconds until the lock count falls below the highwater mark. This requires cycles, and so not doing it means less cycles used. The higher the highwater mark is, the more memory the gulm lock servers and gfs will use to store locks. The number is just the count of locks (in <=6.0) and not an actual representation of ram used. In short summery, in your case, a higher highwater mark may give some performance gained, at the loss of some memory available to other programs. > This weekend, we also noticed another weirdness (for us, anyways...) - > e0 was fenced off on Saturday morning at 0504.09am, almost precisely 24 > hours later e0 decided that the problem was the previous GFS master > (f0), arbitrated itself to be Master, took over, fenced off F0 and then > proceeded to hose the entire thing by the time I heard about things and > was able to get on-site to bring it all back up (at 1am Monday morning). > What is this apparent 24-hour timer, and is this expected behaviour? No, it sounds like some kind of freak chance. A very icky thing indeed. Very much sounds like a higher heartbeat_rate is needed. > Finally - would increasing the heartbeat timer and the number of > acceptable misses an appropriate and acceptable way to help decreases > the frequency of e0 being fenced off? Certainly. The default values for the heartbeat_rate and allowed_misses are just suggestions. Certain setups may require different values, and as far as I know the only way to figure this out is to try it. Sounds very much like you could use larger values. -- michael conrad tadpol tilstra -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 5574 bytes Desc: not available URL: From mtilstra at redhat.com Tue Feb 15 18:24:26 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Tue, 15 Feb 2005 12:24:26 -0600 Subject: [Linux-cluster] GFS 6.0 Questions In-Reply-To: References: Message-ID: <42123E5A.9070801@redhat.com> Gerald G. Gilyeat wrote: > Thanks a bunch. > The direction I was leaning on going, then, seems appropriate. I love > it when things start coming together. > > Is there anyway to get some of these undocumented tunable features, > well, documented? I couldn't for the life of me find anything indicating > if the lock highwater mark was runtime tunable, for example. erm, get me not busy enough that I have time to document stuff? its on my list of things todo, really. its just at the bottom somewhere. > There is -some- concern about memory usage tanking things, but that > will probably end up leading us to simply moving to dedicated locking > servers instead of having them on the actual shared production machines > (and really, we'd only need two...'f1' is strictly for management type > work and backups...) > Finally - so while it's -possible- to have the GFS "stuff" on a > separate interface (and yes, it was a royal PITA getting it to work in > the first place what with multiple NICs already...), it's not somthing > that's at all easy to do, at least until the mentioned fix drops? bleh. Work around is described here: https://bugzilla.redhat.com/beta/show_bug.cgi?id=131142 Not that difficult, just ugly. The problem is that gulm wants hostname==ip==node, and with multiple NICs, that's not the case any more. -- michael conrad tadpol tilstra -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 256 bytes Desc: OpenPGP digital signature URL: From ggilyeat at jhsph.edu Tue Feb 15 19:48:49 2005 From: ggilyeat at jhsph.edu (Gerald G. Gilyeat) Date: Tue, 15 Feb 2005 14:48:49 -0500 Subject: [Linux-cluster] GFS 6.0 Questions Message-ID: cluster.ccs: cluster { lock_gulm { .... lt_high_locks = } } The highwater mark is an attempt to keep the amount of memory lock_gulmd uses down. When the highwater is hit, the lock server tells all gfs mounts to try and release locks. It does this every 10 seconds until the lock count falls below the highwater mark. This requires cycles, and so not doing it means less cycles used. The higher the highwater mark is, the more memory the gulm lock servers and gfs will use to store locks. The number is just the count of locks (in <=6.0) and not an actual representation of ram used. In short summery, in your case, a higher highwater mark may give some performance gained, at the loss of some memory available to other programs. I just bounced the storage servers using the lt_high_locks directive as above. The cluster.ccs looks like the following: cluster { name = "hopkins" lock_gulm { servers = ["front-0", "front-1", "enigma"] } lt_high_locks = 2097152 heartbeat_rate = 30 allowed_misses = 4 } gulm_tool getstats front-1:lt000 returns the following: [root at front-0 root]# gulm_tool getstats front-1:lt000 I_am = Master run time = 831 pid = 4073 verbosity = Default id = 0 partitions = 1 out_queue = 0 drpb_queue = 0 locks = 80640 unlocked = 9267 exclusive = 19 shared = 71354 deferred = 0 lvbs = 9274 expired = 0 lock ops = 1805398 conflicts = 3 incomming_queue = 0 conflict_queue = 0 reply_queue = 0 free_locks = 87162 free_lkrqs = 60 used_lkrqs = 0 free_holders = 125909 used_holders = 81895 highwater = 1048576 Unless I'm mis-reading this, the lt_high_locks directive didn't do anything, unless the bottom number will change once it's breached? My apologies to the list for my verbosity, btw - I'm just under the gun trying to get this stable and working. -- Jerry Gilyeat, RHCE Systems Administrator Molecular Microbiology and Immunology Johns Hopkins Bloomberg School of Public Health -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3618 bytes Desc: not available URL: From mtilstra at redhat.com Tue Feb 15 20:00:42 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Tue, 15 Feb 2005 14:00:42 -0600 Subject: [Linux-cluster] GFS 6.0 Questions In-Reply-To: References: Message-ID: <421254EA.6010103@redhat.com> Gerald G. Gilyeat wrote: > Unless I'm mis-reading this, the lt_high_locks directive didn't do > anything, you read that right. damn. a bug. I'll look into it. Just to make sure I'm on the right page, which version are you running? -- michael conrad tadpol tilstra -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 256 bytes Desc: OpenPGP digital signature URL: From ggilyeat at jhsph.edu Tue Feb 15 20:07:28 2005 From: ggilyeat at jhsph.edu (Gerald G. Gilyeat) Date: Tue, 15 Feb 2005 15:07:28 -0500 Subject: [Linux-cluster] GFS 6.0 Questions Message-ID: GFS-6.0.0-7.1 At least, those are the RPMs currently on the box. We -are- planning an upgrade, but can move that forward if this bug is known to have been fixed in a newer version of 6.0. yay. I found a bug. -- Jerry Gilyeat, RHCE Systems Administrator Molecular Microbiology and Immunology Johns Hopkins Bloomberg School of Public Health -----Original Message----- From: linux-cluster-bounces at redhat.com on behalf of Michael Conrad Tadpol Tilstra Sent: Tue 2/15/2005 3:00 PM To: linux clistering Subject: Re: [Linux-cluster] GFS 6.0 Questions Gerald G. Gilyeat wrote: > Unless I'm mis-reading this, the lt_high_locks directive didn't do > anything, you read that right. damn. a bug. I'll look into it. Just to make sure I'm on the right page, which version are you running? -- michael conrad tadpol tilstra -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 2970 bytes Desc: not available URL: From mtilstra at redhat.com Tue Feb 15 20:14:52 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Tue, 15 Feb 2005 14:14:52 -0600 Subject: [Linux-cluster] GFS 6.0 Questions In-Reply-To: <421254EA.6010103@redhat.com> References: <421254EA.6010103@redhat.com> Message-ID: <4212583C.4020808@redhat.com> Michael Conrad Tadpol Tilstra wrote: > Gerald G. Gilyeat wrote: > >> Unless I'm mis-reading this, the lt_high_locks directive didn't do >> anything, > > > you read that right. damn. a bug. I'll look into it. > > Just to make sure I'm on the right page, which version are you running? oh, and adding the lt_high_locks value will require gulmd to be restarted before it notices. just so you know. -- michael conrad tadpol tilstra -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 256 bytes Desc: OpenPGP digital signature URL: From ggilyeat at jhsph.edu Tue Feb 15 20:15:44 2005 From: ggilyeat at jhsph.edu (Gerald G. Gilyeat) Date: Tue, 15 Feb 2005 15:15:44 -0500 Subject: [Linux-cluster] GFS 6.0 Questions Message-ID: One would think that rebooting the machines would do that... :) -- Jerry Gilyeat, RHCE Systems Administrator Molecular Microbiology and Immunology Johns Hopkins Bloomberg School of Public Health -----Original Message----- From: linux-cluster-bounces at redhat.com on behalf of Michael Conrad Tadpol Tilstra Sent: Tue 2/15/2005 3:14 PM To: linux clistering Subject: Re: [Linux-cluster] GFS 6.0 Questions Michael Conrad Tadpol Tilstra wrote: > Gerald G. Gilyeat wrote: > >> Unless I'm mis-reading this, the lt_high_locks directive didn't do >> anything, > > > you read that right. damn. a bug. I'll look into it. > > Just to make sure I'm on the right page, which version are you running? oh, and adding the lt_high_locks value will require gulmd to be restarted before it notices. just so you know. -- michael conrad tadpol tilstra -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 2938 bytes Desc: not available URL: From mtilstra at redhat.com Tue Feb 15 20:25:16 2005 From: mtilstra at redhat.com (Michael Conrad Tadpol Tilstra) Date: Tue, 15 Feb 2005 14:25:16 -0600 Subject: [Linux-cluster] GFS 6.0 Questions In-Reply-To: References: Message-ID: <42125AAC.4020008@redhat.com> Gerald G. Gilyeat wrote: > One would think that rebooting the machines would do that... > :) > right, sent that before I saw your next msg. k. i know this is a rather dumb question, but I'm gonna ask any ways. you rebuilt the cca device after changin cluster.ccs, right? -- michael conrad tadpol tilstra only dumber if not asked. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 256 bytes Desc: OpenPGP digital signature URL: From jhahm at yahoo.com Wed Feb 16 00:22:44 2005 From: jhahm at yahoo.com (Jiho Hahm) Date: Tue, 15 Feb 2005 16:22:44 -0800 (PST) Subject: [Linux-cluster] Specifying start/stop order of resources in a Message-ID: <20050216002244.22618.qmail@web50903.mail.yahoo.com> Hi, I'm having some trouble with configuring start/stop order of resources in a resource group. When I specify start and stop level values in resource elements, they are ignored. Resources are always started and stopped according to the type-specific level specified in cluster/rgmanager/src/resources/resourcegroup.sh (or /usr/share/cluster/resourcegroup.sh). What I basically want to do during startup of an RG is mount a couple of ext3 filesystems in a certain order, run a custom application, and finally bring up an IP address. During shutdown I want to do exactly the opposite: bring down IP address, stop application, and unmount volumes in reverse order. Here's what I have in cluster.conf: <...> ...