From stefan at lsd.co.za Sat Jan 1 15:00:07 2011 From: stefan at lsd.co.za (Stefan Lesicnik) Date: Sat, 1 Jan 2011 17:00:07 +0200 (SAST) Subject: [Linux-cluster] Multiple communication channels In-Reply-To: <4D1B92C1.2030900@alteeve.com> Message-ID: <1043183573.3.1293894007922.JavaMail.root@zimbra> ----- Original Message ----- > On 12/29/2010 02:49 PM, Kit Gerrits wrote: > > Hello, > > > > AFAIK, Multi-interface heartbeat is something that was only recently > > added to RHCS (earlier this year, if I recall correctly). > > > > Until then, the failover part was usually achieved by using a bonded > > interface as heartbeat interface. > > If possible, I would suggest using 2 (connected) Multicast switches > > and > > running a bond from each server to each switch. > > Or 2 regular switches and broadcast heartbeat (switches only > > connected > > to eachother) > > Otherwise, using an active-active bond (channel?) with 2 crossover > > cables might also work, but offers less protection against interface > > failures. > > > > > > Regards, > > > > Kit > > Hi, > > It was around in el5. Perhaps not in the early versions, I am not sure > exactly when it was added, but certainly by 5.4. > > In the recent 3.x branch, openais was replaced by corosync (for core > cluster communications), which is where rrp is controlled. > > Of course, I could always be wrong. :) > > Cheers. Thanks everybody for the replies. I am running 5.5 and followed the documentation on http://sources.redhat.com/cluster/wiki/MultiHome The config seems to be working with cman status showing both IP addresses. I will do proper testing to see if it actually works. Thanks for the assistance. Stefan From jacob.ishak at gmail.com Mon Jan 3 08:27:19 2011 From: jacob.ishak at gmail.com (jacob ishak) Date: Mon, 3 Jan 2011 10:27:19 +0200 Subject: [Linux-cluster] Multiple communication channels In-Reply-To: <1218306684.1.1293649373746.JavaMail.root@zimbra> References: <2119444257.0.1293649365513.JavaMail.root@zimbra> <1218306684.1.1293649373746.JavaMail.root@zimbra> Message-ID: Hi using the public interface for cluster communication is never recommended , you can add another crossover cable between the two servers , and apply bond configuration between the crossovers connections in this way you can have redundancy for cluster communication Br, jacob On Wed, Dec 29, 2010 at 9:02 PM, Stefan Lesicnik wrote: > Hi all, > > I am running RHCS 5 and have a two node cluster with a shared qdisk. I have > a bonded network bond0 and a back to back crossover eth1. > > Currently I have multicast cluster communication over the crossover, but > was wondering if it was possible to use bond0 as an alternative / failover. > So if eth1 was down, it could still communicate? > > I havent been able to find anything in the FAQ / documentation that would > suggest this, so I thought I would ask. > > Thanks alot and I hope everyone has a great new year :) > > Stefan > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hostmaster at inwx.de Mon Jan 3 23:54:45 2011 From: hostmaster at inwx.de (InterNetworX | Hostmaster) Date: Tue, 04 Jan 2011 00:54:45 +0100 Subject: [Linux-cluster] Processes in D state Message-ID: <4D2261C5.8000802@inwx.de> Hello, we are using GFS2 but sometimes there are processes hanging in D state: # ps axl | grep D F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND 0 0 14220 14219 20 0 19624 1916 - Ds ? 0:00 /usr/lib/postfix/master -t 0 0 14555 14498 20 0 16608 1716 - D+ /mnt/storage/openvz/root/129/dev/pts/0 0:00 apt-get install less 0 0 15068 15067 19 -1 36844 2156 - D /proc/sys/kernel/hung_task_timeout_secs" disables this message. [11142.334310] master D ffff88032b644800 0 14220 14219 0x00000000 [11142.334315] ffff88062dd40000 0000000000000086 0000000000000000 ffffffffa02628d9 [11142.334318] ffff88017a517ef8 000000000000fa40 ffff88017a517fd8 0000000000016940 [11142.334322] 0000000000016940 ffff88032b644800 ffff88032b644af8 0000000b7a517cd8 [11142.334325] Call Trace: [11142.334340] [] ? gfs2_glock_put+0xf9/0x118 [gfs2] [11142.334347] [] ? gfs2_glock_holder_wait+0x0/0xd [gfs2] [11142.334353] [] ? gfs2_glock_holder_wait+0x9/0xd [gfs2] [11142.334358] [] ? __wait_on_bit+0x41/0x70 [11142.334363] [] ? gfs2_glock_holder_wait+0x0/0xd [gfs2] [11142.334367] [] ? out_of_line_wait_on_bit+0x6b/0x77 [11142.334370] [] ? wake_bit_function+0x0/0x23 [11142.334376] [] ? gfs2_glock_wait+0x23/0x28 [gfs2] [11142.334383] [] ? gfs2_flock+0x17c/0x1f9 [gfs2] [11142.334386] [] ? virt_to_head_page+0x9/0x2a [11142.334389] [] ? ub_slab_ptr+0x22/0x65 [11142.334393] [] ? sys_flock+0xff/0x12a [11142.334396] [] ? system_call_fastpath+0x16/0x1b Any idea what is going wrong? Do you need any more informations? Mario From pradhanparas at gmail.com Tue Jan 4 01:23:25 2011 From: pradhanparas at gmail.com (Paras pradhan) Date: Mon, 3 Jan 2011 19:23:25 -0600 Subject: [Linux-cluster] Processes in D state In-Reply-To: <4D2261C5.8000802@inwx.de> References: <4D2261C5.8000802@inwx.de> Message-ID: I had the same problem. it locked the whole gfs cluster and had to reboot the node. after reboot all is fine now but still trying to find out what has caused it. Paras On Monday, January 3, 2011, InterNetworX | Hostmaster wrote: > Hello, > > we are using GFS2 but sometimes there are processes hanging in D state: > > # ps axl | grep D > F ? UID ? PID ?PPID PRI ?NI ? ?VSZ ? RSS WCHAN ?STAT TTY ? ? ? ?TIME COMMAND > 0 ? ? 0 14220 14219 ?20 ? 0 ?19624 ?1916 - ? ? ?Ds ? ? ? ? ? ? ?0:00 > /usr/lib/postfix/master -t > 0 ? ? 0 14555 14498 ?20 ? 0 ?16608 ?1716 - ? ? ?D+ > /mnt/storage/openvz/root/129/dev/pts/0 ? 0:00 apt-get install less > 0 ? ? 0 15068 15067 ?19 ?-1 ?36844 ?2156 - ? ? ?D /usr/lib/postfix/master -t > 0 ? ? 0 16603 16602 ?19 ?-1 ?36844 ?2156 - ? ? ?D /usr/lib/postfix/master -t > 4 ? 101 19534 13238 ?19 ?-1 ?33132 ?2984 - ? ? ?D< ? ? ? ? ? ? ?0:00 > smtpd -n smtp -t inet -u -c > 4 ? 101 19542 13238 ?19 ?-1 ?33116 ?2976 - ? ? ?D< ? ? ? ? ? ? ?0:00 > smtpd -n smtp -t inet -u -c > 0 ? ? 0 19735 13068 ?20 ? 0 ? 7548 ? 880 - ? ? ?S+ ? pts/0 ? ? ?0:00 grep D > > dmesg shows this message many times: > > [11142.334229] INFO: task master:14220 blocked for more than 120 seconds. > [11142.334266] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > disables this message. > [11142.334310] master ? ? ? ?D ffff88032b644800 ? ? 0 14220 ?14219 > 0x00000000 > [11142.334315] ?ffff88062dd40000 0000000000000086 0000000000000000 > ffffffffa02628d9 > [11142.334318] ?ffff88017a517ef8 000000000000fa40 ffff88017a517fd8 > 0000000000016940 > [11142.334322] ?0000000000016940 ffff88032b644800 ffff88032b644af8 > 0000000b7a517cd8 > [11142.334325] Call Trace: > [11142.334340] ?[] ? gfs2_glock_put+0xf9/0x118 [gfs2] > [11142.334347] ?[] ? gfs2_glock_holder_wait+0x0/0xd [gfs2] > [11142.334353] ?[] ? gfs2_glock_holder_wait+0x9/0xd [gfs2] > [11142.334358] ?[] ? __wait_on_bit+0x41/0x70 > [11142.334363] ?[] ? gfs2_glock_holder_wait+0x0/0xd [gfs2] > [11142.334367] ?[] ? out_of_line_wait_on_bit+0x6b/0x77 > [11142.334370] ?[] ? wake_bit_function+0x0/0x23 > [11142.334376] ?[] ? gfs2_glock_wait+0x23/0x28 [gfs2] > [11142.334383] ?[] ? gfs2_flock+0x17c/0x1f9 [gfs2] > [11142.334386] ?[] ? virt_to_head_page+0x9/0x2a > [11142.334389] ?[] ? ub_slab_ptr+0x22/0x65 > [11142.334393] ?[] ? sys_flock+0xff/0x12a > [11142.334396] ?[] ? system_call_fastpath+0x16/0x1b > > Any idea what is going wrong? Do you need any more informations? > > Mario > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From thomas at sjolshagen.net Tue Jan 4 10:00:56 2011 From: thomas at sjolshagen.net (Thomas Sjolshagen) Date: Tue, 04 Jan 2011 05:00:56 -0500 Subject: [Linux-cluster] GFS2 locking in a VM based cluster (KVM) Message-ID: <20110104050056.96196p5ratbwjie0@www.sjolshagen.net> Hi, Posted this on IRC just now, but was recommended to try another approach. I'm wondering if the locks/sec rate I'm seeing between two virtual machines is what I should expect - details below: I've got a 2-node cluster that is actually two KVM VMs (Fedora 14 in the guests/vms). Between the two VM's, I'm sharing - iSCSI based external array using virtio_net drivers in the guests/vms - two GFS2 file systems. After setting and as listed below in cluster.conf (and, of course, rebooting the whole cluster), I'm still only seeing 600-650 locks/sec when using the ping_pong utility. The same settings & configuration on the physical hosts (a separate cluster, but using the same iSCSI array) is seeing 5.5-6K locks/sec. I'm also curious as to why, when starting ping_pong on the virtualized node1, I see ~20-40K locks/sec, but if I start and then stop ping_pong on the virtualized node2, node1 never returns to anything close to the 20-40K locks/sec. Is that (also) expected? Both physical & virtual machines are running Fedora 14 w/gfs2-utils-3.1.0-3.fc14.x86_64 & gfs2-cluster-3.1.0-3.fc14.x86_64. // Thomas ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. From swhiteho at redhat.com Tue Jan 4 10:24:03 2011 From: swhiteho at redhat.com (Steven Whitehouse) Date: Tue, 04 Jan 2011 10:24:03 +0000 Subject: [Linux-cluster] GFS2 locking in a VM based cluster (KVM) In-Reply-To: <20110104050056.96196p5ratbwjie0@www.sjolshagen.net> References: <20110104050056.96196p5ratbwjie0@www.sjolshagen.net> Message-ID: <1294136643.2455.1.camel@dolmen> Hi, On Tue, 2011-01-04 at 05:00 -0500, Thomas Sjolshagen wrote: > Hi, > > Posted this on IRC just now, but was recommended to try another approach. > > I'm wondering if the locks/sec rate I'm seeing between two virtual > machines is what I should expect - details below: > > I've got a 2-node cluster that is actually two KVM VMs (Fedora 14 in > the guests/vms). Between the two VM's, I'm sharing - iSCSI based > external array using virtio_net drivers in the guests/vms - two GFS2 > file systems. After setting and as listed below > in cluster.conf (and, of course, rebooting the whole cluster), I'm > still only seeing 600-650 locks/sec when using the ping_pong utility. > > > > > The same settings & configuration on the physical hosts (a separate > cluster, but using the same iSCSI array) is seeing 5.5-6K locks/sec. > > I'm also curious as to why, when starting ping_pong on the virtualized > node1, I see ~20-40K locks/sec, but if I start and then stop ping_pong > on the virtualized node2, node1 never returns to anything close to the > 20-40K locks/sec. Is that (also) expected? > Did you have the second node mounted when you got the faster locking rates on the first node? Steve. From thomas at sjolshagen.net Tue Jan 4 10:45:35 2011 From: thomas at sjolshagen.net (Thomas Sjolshagen) Date: Tue, 04 Jan 2011 05:45:35 -0500 Subject: [Linux-cluster] GFS2 locking in a VM based cluster (KVM) In-Reply-To: <1294136643.2455.1.camel@dolmen> References: <20110104050056.96196p5ratbwjie0@www.sjolshagen.net> <1294136643.2455.1.camel@dolmen> Message-ID: On Tue, 04 Jan 2011 10:24:03 +0000, Steven Whitehouse wrote: > Hi, > > On Tue, 2011-01-04 at 05:00 -0500, Thomas Sjolshagen wrote: >> Hi, >> >> Posted this on IRC just now, but was recommended to try another >> approach. >> >> I'm wondering if the locks/sec rate I'm seeing between two virtual >> machines is what I should expect - details below: >> >> I've got a 2-node cluster that is actually two KVM VMs (Fedora 14 in >> the guests/vms). Between the two VM's, I'm sharing - iSCSI based >> external array using virtio_net drivers in the guests/vms - two GFS2 >> file systems. After setting and as listed below >> in cluster.conf (and, of course, rebooting the whole cluster), I'm >> still only seeing 600-650 locks/sec when using the ping_pong >> utility. >> >> >> >> >> The same settings & configuration on the physical hosts (a separate >> cluster, but using the same iSCSI array) is seeing 5.5-6K locks/sec. >> >> I'm also curious as to why, when starting ping_pong on the >> virtualized >> node1, I see ~20-40K locks/sec, but if I start and then stop >> ping_pong >> on the virtualized node2, node1 never returns to anything close to >> the >> 20-40K locks/sec. Is that (also) expected? >> > Did you have the second node mounted when you got the faster locking > rates on the first node? > Yes, both nodes mounted (before and after starting ping_pong on the 2nd node, no umount inbetween) // Thomas From emilio at ugr.es Tue Jan 4 11:27:52 2011 From: emilio at ugr.es (Emilio Arjona) Date: Tue, 4 Jan 2011 12:27:52 +0100 Subject: [Linux-cluster] Processes in D state In-Reply-To: References: <4D2261C5.8000802@inwx.de> Message-ID: Same problem here, in a webserver cluster httpd run into D state sometimes. I have to restart the node or even the whole cluster if there are more than one node locked. I'm using REDHAT 5.4 and HP hardware. Regards, 2011/1/4 Paras pradhan > I had the same problem. it locked the whole gfs cluster and had to > reboot the node. after reboot all is fine now but still trying to find > out what has caused it. > > Paras > > On Monday, January 3, 2011, InterNetworX | Hostmaster > wrote: > > Hello, > > > > we are using GFS2 but sometimes there are processes hanging in D state: > > > > # ps axl | grep D > > F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME > COMMAND > > 0 0 14220 14219 20 0 19624 1916 - Ds ? 0:00 > > /usr/lib/postfix/master -t > > 0 0 14555 14498 20 0 16608 1716 - D+ > > /mnt/storage/openvz/root/129/dev/pts/0 0:00 apt-get install less > > 0 0 15068 15067 19 -1 36844 2156 - D > /usr/lib/postfix/master -t > > 0 0 16603 16602 19 -1 36844 2156 - D > /usr/lib/postfix/master -t > > 4 101 19534 13238 19 -1 33132 2984 - D< ? 0:00 > > smtpd -n smtp -t inet -u -c > > 4 101 19542 13238 19 -1 33116 2976 - D< ? 0:00 > > smtpd -n smtp -t inet -u -c > > 0 0 19735 13068 20 0 7548 880 - S+ pts/0 0:00 grep > D > > > > dmesg shows this message many times: > > > > [11142.334229] INFO: task master:14220 blocked for more than 120 seconds. > > [11142.334266] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables this message. > > [11142.334310] master D ffff88032b644800 0 14220 14219 > > 0x00000000 > > [11142.334315] ffff88062dd40000 0000000000000086 0000000000000000 > > ffffffffa02628d9 > > [11142.334318] ffff88017a517ef8 000000000000fa40 ffff88017a517fd8 > > 0000000000016940 > > [11142.334322] 0000000000016940 ffff88032b644800 ffff88032b644af8 > > 0000000b7a517cd8 > > [11142.334325] Call Trace: > > [11142.334340] [] ? gfs2_glock_put+0xf9/0x118 [gfs2] > > [11142.334347] [] ? gfs2_glock_holder_wait+0x0/0xd > [gfs2] > > [11142.334353] [] ? gfs2_glock_holder_wait+0x9/0xd > [gfs2] > > [11142.334358] [] ? __wait_on_bit+0x41/0x70 > > [11142.334363] [] ? gfs2_glock_holder_wait+0x0/0xd > [gfs2] > > [11142.334367] [] ? out_of_line_wait_on_bit+0x6b/0x77 > > [11142.334370] [] ? wake_bit_function+0x0/0x23 > > [11142.334376] [] ? gfs2_glock_wait+0x23/0x28 [gfs2] > > [11142.334383] [] ? gfs2_flock+0x17c/0x1f9 [gfs2] > > [11142.334386] [] ? virt_to_head_page+0x9/0x2a > > [11142.334389] [] ? ub_slab_ptr+0x22/0x65 > > [11142.334393] [] ? sys_flock+0xff/0x12a > > [11142.334396] [] ? system_call_fastpath+0x16/0x1b > > > > Any idea what is going wrong? Do you need any more informations? > > > > Mario > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- ******************************************* Emilio Arjona Heredia Centro de Ense?anzas Virtuales de la Universidad de Granada C/ Real de Cartuja 36-38 http://cevug.ugr.es Tlfno.: 958-241000 ext. 20206 ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at sjolshagen.net Tue Jan 4 12:16:11 2011 From: thomas at sjolshagen.net (Thomas Sjolshagen) Date: Tue, 04 Jan 2011 07:16:11 -0500 Subject: [Linux-cluster] GFS2 locking in a VM based cluster (KVM) In-Reply-To: References: <20110104050056.96196p5ratbwjie0@www.sjolshagen.net> <1294136643.2455.1.camel@dolmen> Message-ID: <4e54d6ded4d62fa4fa0f8498384a4448@sjolshagen.net> On Tue, 04 Jan 2011 05:45:35 -0500, Thomas Sjolshagen wrote: >> > Yes, both nodes mounted (before and after starting ping_pong on the > 2nd node, no umount inbetween) > Well, disabling the 2nd corosync ring ( in cluster.conf) seems to have almost doubled the # of locks/sec (from 590-650 to 1000-1100ish) // Thomas From adrew at redhat.com Tue Jan 4 15:27:53 2011 From: adrew at redhat.com (Adam Drew) Date: Tue, 4 Jan 2011 10:27:53 -0500 (EST) Subject: [Linux-cluster] Processes in D state In-Reply-To: Message-ID: <1599288175.144252.1294154873625.JavaMail.root@zmail01.collab.prod.int.phx2.redhat.com> Hello, Processes accessing a GFS2 filesystem falling into D state is typically indicative of lock contention; however, other causes are also possible. D state is uninterruptable sleep waiting on IO. With regards to GFS2 this means that a PID has requested access to some object on disk and has not yet gained access to that object. As the PID cannot proceed until granted access it is hung in D state. The most common cause of D state PIDs on GFS2 is lock contention. GFS2's shared locking system is more complex than traditional single-node filesystems. You can run into a situation where a given PID is locking a resource but is waiting in line for a lock on another resource to be released where the holder of that second resource is waiting on the PID holding the first to release it as well. This causes a deadlock where neither process can make process, both end up in D state, and so will any process that requests access to either of those resources as well. In other cases PIDs requesting access to a resource on disk may build up faster than than they release them. In this case the queue of waiters will build and build until the filesystem grinds to a halt and appears to "hang." In other cases bugs or design issues may lead to locking bottlenecks. GFS2 locks are arbitrated in the glock (pronounced gee-lock) layer. The glock subsystem is exposed via debugfs. You can mount debugfs, look in the gfs2 directory, and view the glocks. You can then match up the glocks to the process list on the system and to the messages logs. Doing this for every node in the cluster can reveal problems. If you have Red Hat support I encourage you to engage them as learning to read glocks can be non-trivial process but it is not impossible. They are documented to a degree in the following documents: "Testing and verification of cluster filesystems" by Steven Whitehouse http://www.kernel.org/doc/ols/2009/ols2009-pages-311-318.pdf Global File System 2, Edition 7, section 1.4. "GFS2 Node Locking" http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Global_File_System_2/index.html#s1-ov-lockbounce More information is available out on the web. Regards, Adam Drew ----- Original Message ----- From: "Emilio Arjona" To: "linux clustering" Sent: Tuesday, January 4, 2011 6:27:52 AM Subject: Re: [Linux-cluster] Processes in D state Same problem here, in a webserver cluster httpd run into D state sometimes. I have to restart the node or even the whole cluster if there are more than one node locked. I'm using REDHAT 5.4 and HP hardware. Regards, 2011/1/4 Paras pradhan < pradhanparas at gmail.com > I had the same problem. it locked the whole gfs cluster and had to reboot the node. after reboot all is fine now but still trying to find out what has caused it. Paras On Monday, January 3, 2011, InterNetworX | Hostmaster < hostmaster at inwx.de > wrote: > Hello, > > we are using GFS2 but sometimes there are processes hanging in D state: > > # ps axl | grep D > F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND > 0 0 14220 14219 20 0 19624 1916 - Ds ? 0:00 > /usr/lib/postfix/master -t > 0 0 14555 14498 20 0 16608 1716 - D+ > /mnt/storage/openvz/root/129/dev/pts/0 0:00 apt-get install less > 0 0 15068 15067 19 -1 36844 2156 - D /usr/lib/postfix/master -t > 0 0 16603 16602 19 -1 36844 2156 - D /usr/lib/postfix/master -t > 4 101 19534 13238 19 -1 33132 2984 - D< ? 0:00 > smtpd -n smtp -t inet -u -c > 4 101 19542 13238 19 -1 33116 2976 - D< ? 0:00 > smtpd -n smtp -t inet -u -c > 0 0 19735 13068 20 0 7548 880 - S+ pts/0 0:00 grep D > > dmesg shows this message many times: > > [11142.334229] INFO: task master:14220 blocked for more than 120 seconds. > [11142.334266] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > disables this message. > [11142.334310] master D ffff88032b644800 0 14220 14219 > 0x00000000 > [11142.334315] ffff88062dd40000 0000000000000086 0000000000000000 > ffffffffa02628d9 > [11142.334318] ffff88017a517ef8 000000000000fa40 ffff88017a517fd8 > 0000000000016940 > [11142.334322] 0000000000016940 ffff88032b644800 ffff88032b644af8 > 0000000b7a517cd8 > [11142.334325] Call Trace: > [11142.334340] [] ? gfs2_glock_put+0xf9/0x118 [gfs2] > [11142.334347] [] ? gfs2_glock_holder_wait+0x0/0xd [gfs2] > [11142.334353] [] ? gfs2_glock_holder_wait+0x9/0xd [gfs2] > [11142.334358] [] ? __wait_on_bit+0x41/0x70 > [11142.334363] [] ? gfs2_glock_holder_wait+0x0/0xd [gfs2] > [11142.334367] [] ? out_of_line_wait_on_bit+0x6b/0x77 > [11142.334370] [] ? wake_bit_function+0x0/0x23 > [11142.334376] [] ? gfs2_glock_wait+0x23/0x28 [gfs2] > [11142.334383] [] ? gfs2_flock+0x17c/0x1f9 [gfs2] > [11142.334386] [] ? virt_to_head_page+0x9/0x2a > [11142.334389] [] ? ub_slab_ptr+0x22/0x65 > [11142.334393] [] ? sys_flock+0xff/0x12a > [11142.334396] [] ? system_call_fastpath+0x16/0x1b > > Any idea what is going wrong? Do you need any more informations? > > Mario > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster -- ******************************************* Emilio Arjona Heredia Centro de Ense?anzas Virtuales de la Universidad de Granada C/ Real de Cartuja 36-38 http://cevug.ugr.es Tlfno.: 958-241000 ext. 20206 ******************************************* -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From nukejun at gmail.com Tue Jan 4 18:42:45 2011 From: nukejun at gmail.com (juncheol park) Date: Tue, 4 Jan 2011 11:42:45 -0700 Subject: [Linux-cluster] GFS block size In-Reply-To: <64D0546C5EBBD147B75DE133D798665F06A12904@hugo.eprize.local> References: <64D0546C5EBBD147B75DE133D798665F06A12904@hugo.eprize.local> Message-ID: I also experimented 1k block size on GFS1. Although you can improve the disk usage using a smaller block size, typically it is recommended to use the block size same as the page size, which is 4k in Linux. I don't remember all the details of results. However, for large files, the overall performance of read/write operations with 1k block size was much worse than the one with 4k block size. This is obvious, though. If you don't care any performance degradation for large files, it would be fine for you to use 1k. Just my two cents, -Jun On Fri, Dec 17, 2010 at 3:53 PM, Jeff Sturm wrote: > One of our GFS filesystems tends to have a large number of very small files, > on average about 1000 bytes each. > > > > I realized this week we'd created our filesystems with default options.? As > an experiment on a test system, I've recreated a GFS filesystem with "-b > 1024" to reduce overall disk usage and disk bandwidth. > > > > Initially, tests look very good?single file creates are less than one > millisecond on average (down from about 5ms each).? Before I go very far > with this, I wanted to ask:? Has anyone else experimented with the block > size option, and are there any tricks or gotchas to report? > > > > (This is with CentOS 5.5, GFS 1.) > > > > -Jeff > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From adrew at redhat.com Tue Jan 4 19:17:38 2011 From: adrew at redhat.com (Adam Drew) Date: Tue, 4 Jan 2011 14:17:38 -0500 (EST) Subject: [Linux-cluster] GFS block size In-Reply-To: Message-ID: <102121344.150135.1294168658370.JavaMail.root@zmail01.collab.prod.int.phx2.redhat.com> If your average file size is less than 1K then using a block size of 1k may be a good option. If you can fit your data in a single block you get the minor performance boost of using a stuffed inode so you never have to walk a list from your inode to your data block. The performance boost should be small but could add up to larger gains over time with lots of transactions. If your average data payload is less than the default block-size however, you'll end up losing the delta. So, from a filesystem perspective, using a 1k blocksize to store mostly sub-1k files may be a good idea. You additionally may want to experiment with reducing your resource group size. Blocks are organized into resource groups. If you are using 1k blocks and sub-1k files then you'll end up with tons of stuffed inodes per resource group. Some operations in GFS require locking the resource group metadata (such as deletes) so you may start to experience performance bottle-necks depending on usage patterns and disk layout. All-in-all I'd be skeptical of the claim of large performance gains over time by changing rg size and block size but modest gains may be had. Still, some access patterns and filesystem layouts may experience greater performance gains with such tweaking. However, I would expect to see the most significant gains (in GFS1 at least) made by mount options and tuneables. Regards, Adam Drew ----- Original Message ----- From: "juncheol park" To: "linux clustering" Sent: Tuesday, January 4, 2011 1:42:45 PM Subject: Re: [Linux-cluster] GFS block size I also experimented 1k block size on GFS1. Although you can improve the disk usage using a smaller block size, typically it is recommended to use the block size same as the page size, which is 4k in Linux. I don't remember all the details of results. However, for large files, the overall performance of read/write operations with 1k block size was much worse than the one with 4k block size. This is obvious, though. If you don't care any performance degradation for large files, it would be fine for you to use 1k. Just my two cents, -Jun On Fri, Dec 17, 2010 at 3:53 PM, Jeff Sturm wrote: > One of our GFS filesystems tends to have a large number of very small files, > on average about 1000 bytes each. > > > > I realized this week we'd created our filesystems with default options.? As > an experiment on a test system, I've recreated a GFS filesystem with "-b > 1024" to reduce overall disk usage and disk bandwidth. > > > > Initially, tests look very good?single file creates are less than one > millisecond on average (down from about 5ms each).? Before I go very far > with this, I wanted to ask:? Has anyone else experimented with the block > size option, and are there any tricks or gotchas to report? > > > > (This is with CentOS 5.5, GFS 1.) > > > > -Jeff > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From sdake at redhat.com Tue Jan 4 20:59:29 2011 From: sdake at redhat.com (Steven Dake) Date: Tue, 04 Jan 2011 13:59:29 -0700 Subject: [Linux-cluster] [Openais] packet dissectors for totempg, cman, clvmd, rgmanager, cpg, In-Reply-To: <4D07A04C.9020703@redhat.com> References: <20100527.133950.593311767624382812.yamato@redhat.com> <20101214.210216.512133496326900668.yamato@redhat.com> <4D0776DC.9080003@redhat.com> <20101214.231525.648039044490713397.yamato@redhat.com> <4D078465.3020509@redhat.com> <4D079B49.8070009@redhat.com> <4D07A04C.9020703@redhat.com> Message-ID: <4D238A31.5040206@redhat.com> On 12/14/2010 09:50 AM, Jan Friesse wrote: > Steven Dake napsal(a): >> On 12/14/2010 07:51 AM, Jan Friesse wrote: >>> Masatake, >>> > > .... > >>>> Thank you. >>> Regards, >>> Honza >> >> >> I am not changing corosync license to GPL. I think the separate plugin >> works fine, and we can even take up packaging of it in fedora and Red >> Hat variants, if it is maintained in an upstream repo. >> >> Regards >> -steve > > Steve, > I'm not talking about relicensing corosync (it doesn't make any sense > and I would be first against that), but give permissions to that portion > of code (seems to be more or less header files) to use GPL (which also > seems to me like old version without support for NSS). It's same as what > we did for libqb. > > Separate plugin works fine for Fedora, but I'm not sure if it works also > for other distributions. > > > Regards, > Honza What headers are needed? We can likely provide a GPL version of the headers for third party projects to use. One issue is likely the use of libtomcrypt which has a "public domain" license which we did not write nor can re-license. Regards -steve From sdake at redhat.com Tue Jan 4 21:02:56 2011 From: sdake at redhat.com (Steven Dake) Date: Tue, 04 Jan 2011 14:02:56 -0700 Subject: [Linux-cluster] [Openais] packet dissectors for totempg, cman, clvmd, rgmanager, cpg, In-Reply-To: <20101215.000429.721897046580218183.yamato@redhat.com> References: <4D0776DC.9080003@redhat.com> <20101214.231525.648039044490713397.yamato@redhat.com> <4D078465.3020509@redhat.com> <20101215.000429.721897046580218183.yamato@redhat.com> Message-ID: <4D238B00.5010103@redhat.com> On 12/14/2010 08:04 AM, Masatake YAMATO wrote: > Thank you for replying. > >> Masatake, >> >> Masatake YAMATO napsal(a): >>> I'd like to your advice more detail seriously. >>> I've been developing this code for three years. >>> I don't want to make this code garbage. >>> >>>> Masatake, >>>> I'm pretty sure that biggest problem of your code was that it was >>>> licensed under BSD (three clause, same as Corosync has) >>>> license. Wireshark is licensed under GPL and even I like BSD licenses >>>> much more, I would recommend you to try to relicense code under GPL >>>> and send them this code. >>>> >>>> Regards, >>>> Honza >>> I got the similar comment from wireshark developer. >>> Please, read the discussion: >>> https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=3232 >>> >> >> I've read that thread long time before I've sent previous mail, so >> thats reason why I think that Wireshark developers just feel MUCH more >> comfortable with GPL and thats reason why they just ignoring it. > > I see. > >>> In my understanding there is no legal problem in putting 3-clause BSD >>> code into GPL code. Acutally wireshark includes some 3-clause BSD >>> code: >>> >> >> Actually there is really not. BSD to GPL works without problem, but >> many people just don't know it... > > ...it is too bad. I strongly believe FOSS developers should know the > intent behind of the both licenses. > >>> epan/dissectors/packet-radiotap-defs.h: >>> /*- >>> * Copyright (c) 2003, 2004 David Young. All rights reserved. >>> * >>> * $Id: packet-radiotap-defs.h 34554 2010-10-18 13:24:10Z morriss $ >>> * >>> * Redistribution and use in source and binary forms, with or without >>> * modification, are permitted provided that the following conditions >>> * are met: >>> * 1. Redistributions of source code must retain the above copyright >>> * notice, this list of conditions and the following disclaimer. >>> * 2. Redistributions in binary form must reproduce the above copyright >>> * notice, this list of conditions and the following disclaimer in the >>> * documentation and/or other materials provided with the distribution. >>> * 3. The name of David Young may not be used to endorse or promote >>> * products derived from this software without specific prior >>> * written permission. >>> * >>> * THIS SOFTWARE IS PROVIDED BY DAVID YOUNG ``AS IS'' AND ANY >>> * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >>> * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A >>> * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DAVID >>> * YOUNG BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, >>> * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED >>> * TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, >>> * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND >>> * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, >>> * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY >>> * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY >>> * OF SUCH DAMAGE. >>> */ >>> I'd like to separate the legal issue and preference. I think I >>> understand the importance of preference of upstream >>> developers. However, I'd like to clear the legal issue first. >>> >> >> Legally it's ok. But as you said, developers preference are >> different. And because you are trying to change THEIR code it's >> sometimes better to play they rules. > > I see. > >>> I can image there are people who prefer to GPL as the license covering >>> their software. But here I've taken some corosync code in my >>> dissector. It is essential part of my dissector. And corosync is >> >> ^^^ This may be problem. Question is how big is that part and if it >> can be possible to make exception there. Can you point that code? >> >> Steve, we were able to relicense HUGE portion of code in case of >> libqb, are we able to make the same for Wireshark dissector? > > Could you see https://github.com/masatake/wireshark-plugin-rhcs/blob/master/src/packet-corosync-totemnet.c#L156 > I refer totemnet.c to write dissect_corosynec_totemnet_with_decryption() function. > >>> licensed in 3-clause BSD, as you know. I'd like to change the license >>> to merge my code to upstream project. I cannot do it in this context. >>> See https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=3232#c13 >>> Thank you. >> >> Regards, >> Honza > > Masatake YAMATO Masatake, Red Hat is the author of the totemnet file and can provide that code under GPL if you like. We cannot modify the license for libtomcrypt as we are not the authors. Feel free to change the license for that particular code you rewrote in the link > Could you see https://github.com/masatake/wireshark-plugin-rhcs/blob/master/src/packet-corosync-totemnet.c#L156 under a GPL license if it helps move things along. Regards -steveu From yamato at redhat.com Wed Jan 5 05:56:35 2011 From: yamato at redhat.com (Masatake YAMATO) Date: Wed, 05 Jan 2011 14:56:35 +0900 (JST) Subject: [Linux-cluster] [Openais] packet dissectors for totempg, cman, clvmd, rgmanager, cpg, In-Reply-To: <4D238B00.5010103@redhat.com> References: <4D078465.3020509@redhat.com> <20101215.000429.721897046580218183.yamato@redhat.com> <4D238B00.5010103@redhat.com> Message-ID: <20110105.145635.1000595217822844587.yamato@redhat.com> Thank you very much. I'll push my patch again. Masatake YAMATO > On 12/14/2010 08:04 AM, Masatake YAMATO wrote: >> Thank you for replying. >> >>> Masatake, >>> >>> Masatake YAMATO napsal(a): >>>> I'd like to your advice more detail seriously. >>>> I've been developing this code for three years. >>>> I don't want to make this code garbage. >>>> >>>>> Masatake, >>>>> I'm pretty sure that biggest problem of your code was that it was >>>>> licensed under BSD (three clause, same as Corosync has) >>>>> license. Wireshark is licensed under GPL and even I like BSD licenses >>>>> much more, I would recommend you to try to relicense code under GPL >>>>> and send them this code. >>>>> >>>>> Regards, >>>>> Honza >>>> I got the similar comment from wireshark developer. >>>> Please, read the discussion: >>>> https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=3232 >>>> >>> >>> I've read that thread long time before I've sent previous mail, so >>> thats reason why I think that Wireshark developers just feel MUCH more >>> comfortable with GPL and thats reason why they just ignoring it. >> >> I see. >> >>>> In my understanding there is no legal problem in putting 3-clause BSD >>>> code into GPL code. Acutally wireshark includes some 3-clause BSD >>>> code: >>>> >>> >>> Actually there is really not. BSD to GPL works without problem, but >>> many people just don't know it... >> >> ...it is too bad. I strongly believe FOSS developers should know the >> intent behind of the both licenses. >> >>>> epan/dissectors/packet-radiotap-defs.h: >>>> /*- >>>> * Copyright (c) 2003, 2004 David Young. All rights reserved. >>>> * >>>> * $Id: packet-radiotap-defs.h 34554 2010-10-18 13:24:10Z morriss $ >>>> * >>>> * Redistribution and use in source and binary forms, with or without >>>> * modification, are permitted provided that the following conditions >>>> * are met: >>>> * 1. Redistributions of source code must retain the above copyright >>>> * notice, this list of conditions and the following disclaimer. >>>> * 2. Redistributions in binary form must reproduce the above copyright >>>> * notice, this list of conditions and the following disclaimer in the >>>> * documentation and/or other materials provided with the distribution. >>>> * 3. The name of David Young may not be used to endorse or promote >>>> * products derived from this software without specific prior >>>> * written permission. >>>> * >>>> * THIS SOFTWARE IS PROVIDED BY DAVID YOUNG ``AS IS'' AND ANY >>>> * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >>>> * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A >>>> * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DAVID >>>> * YOUNG BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, >>>> * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED >>>> * TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, >>>> * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND >>>> * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, >>>> * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY >>>> * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY >>>> * OF SUCH DAMAGE. >>>> */ >>>> I'd like to separate the legal issue and preference. I think I >>>> understand the importance of preference of upstream >>>> developers. However, I'd like to clear the legal issue first. >>>> >>> >>> Legally it's ok. But as you said, developers preference are >>> different. And because you are trying to change THEIR code it's >>> sometimes better to play they rules. >> >> I see. >> >>>> I can image there are people who prefer to GPL as the license covering >>>> their software. But here I've taken some corosync code in my >>>> dissector. It is essential part of my dissector. And corosync is >>> >>> ^^^ This may be problem. Question is how big is that part and if it >>> can be possible to make exception there. Can you point that code? >>> >>> Steve, we were able to relicense HUGE portion of code in case of >>> libqb, are we able to make the same for Wireshark dissector? >> >> Could you see https://github.com/masatake/wireshark-plugin-rhcs/blob/master/src/packet-corosync-totemnet.c#L156 >> I refer totemnet.c to write dissect_corosynec_totemnet_with_decryption() function. >> >>>> licensed in 3-clause BSD, as you know. I'd like to change the license >>>> to merge my code to upstream project. I cannot do it in this context. >>>> See https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=3232#c13 >>>> Thank you. >>> >>> Regards, >>> Honza >> >> Masatake YAMATO > > Masatake, > > Red Hat is the author of the totemnet file and can provide that code > under GPL if you like. We cannot modify the license for libtomcrypt as > we are not the authors. Feel free to change the license for that > particular code you rewrote in the link > >> Could you see > https://github.com/masatake/wireshark-plugin-rhcs/blob/master/src/packet-corosync-totemnet.c#L156 > > under a GPL license if it helps move things along. > > Regards > -steveu From hostmaster at inwx.de Wed Jan 5 12:56:25 2011 From: hostmaster at inwx.de (InterNetworX | Hostmaster) Date: Wed, 05 Jan 2011 13:56:25 +0100 Subject: [Linux-cluster] Processes in D state In-Reply-To: <1599288175.144252.1294154873625.JavaMail.root@zmail01.collab.prod.int.phx2.redhat.com> References: <1599288175.144252.1294154873625.JavaMail.root@zmail01.collab.prod.int.phx2.redhat.com> Message-ID: <4D246A79.7020807@inwx.de> Hi Adam, thanks for your help. One problem was, that we did not mounted the GFS2 file system with no noatime and nodiratime options. We still have a problem with postfix. The gfs2 hang analyzer says: There is 1 glock with waiters. node4, pid 20902 is waiting for glock 6/11486739, which is held by pid 12382 Both PIDs are on the some node: root 12382 0.0 0.0 36844 2300 ? Ss 12:39 0:00 /usr/lib/postfix/master root 20902 0.0 0.0 36844 2156 ? Ds 12:45 0:00 /usr/lib/postfix/master -t I have no idea what Postfix is trying to do here?! Mario Am 04.01.11 16:27, schrieb Adam Drew: > Hello, > > Processes accessing a GFS2 filesystem falling into D state is typically indicative of lock contention; however, other causes are also possible. D state is uninterruptable sleep waiting on IO. With regards to GFS2 this means that a PID has requested access to some object on disk and has not yet gained access to that object. As the PID cannot proceed until granted access it is hung in D state. > > The most common cause of D state PIDs on GFS2 is lock contention. GFS2's shared locking system is more complex than traditional single-node filesystems. You can run into a situation where a given PID is locking a resource but is waiting in line for a lock on another resource to be released where the holder of that second resource is waiting on the PID holding the first to release it as well. This causes a deadlock where neither process can make process, both end up in D state, and so will any process that requests access to either of those resources as well. In other cases PIDs requesting access to a resource on disk may build up faster than than they release them. In this case the queue of waiters will build and build until the filesystem grinds to a halt and appears to "hang." In other cases bugs or design issues may lead to locking bottlenecks. > > GFS2 locks are arbitrated in the glock (pronounced gee-lock) layer. The glock subsystem is exposed via debugfs. You can mount debugfs, look in the gfs2 directory, and view the glocks. You can then match up the glocks to the process list on the system and to the messages logs. Doing this for every node in the cluster can reveal problems. If you have Red Hat support I encourage you to engage them as learning to read glocks can be non-trivial process but it is not impossible. They are documented to a degree in the following documents: > > "Testing and verification of cluster filesystems" by Steven Whitehouse > http://www.kernel.org/doc/ols/2009/ols2009-pages-311-318.pdf > > Global File System 2, Edition 7, section 1.4. "GFS2 Node Locking" > http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Global_File_System_2/index.html#s1-ov-lockbounce > > More information is available out on the web. > > Regards, > Adam Drew > > ----- Original Message ----- > From: "Emilio Arjona" > To: "linux clustering" > Sent: Tuesday, January 4, 2011 6:27:52 AM > Subject: Re: [Linux-cluster] Processes in D state > > > Same problem here, > > > in a webserver cluster httpd run into D state sometimes. I have to restart the node or even the whole cluster if there are more than one node locked. I'm using REDHAT 5.4 and HP hardware. > > > Regards, > > > 2011/1/4 Paras pradhan < pradhanparas at gmail.com > > > > I had the same problem. it locked the whole gfs cluster and had to > reboot the node. after reboot all is fine now but still trying to find > out what has caused it. > > Paras > > On Monday, January 3, 2011, InterNetworX | Hostmaster > > > > < hostmaster at inwx.de > wrote: >> Hello, >> >> we are using GFS2 but sometimes there are processes hanging in D state: >> >> # ps axl | grep D >> F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND >> 0 0 14220 14219 20 0 19624 1916 - Ds ? 0:00 >> /usr/lib/postfix/master -t >> 0 0 14555 14498 20 0 16608 1716 - D+ >> /mnt/storage/openvz/root/129/dev/pts/0 0:00 apt-get install less >> 0 0 15068 15067 19 -1 36844 2156 - D> /usr/lib/postfix/master -t >> 0 0 16603 16602 19 -1 36844 2156 - D> /usr/lib/postfix/master -t >> 4 101 19534 13238 19 -1 33132 2984 - D< ? 0:00 >> smtpd -n smtp -t inet -u -c >> 4 101 19542 13238 19 -1 33116 2976 - D< ? 0:00 >> smtpd -n smtp -t inet -u -c >> 0 0 19735 13068 20 0 7548 880 - S+ pts/0 0:00 grep D >> >> dmesg shows this message many times: >> >> [11142.334229] INFO: task master:14220 blocked for more than 120 seconds. >> [11142.334266] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >> disables this message. >> [11142.334310] master D ffff88032b644800 0 14220 14219 >> 0x00000000 >> [11142.334315] ffff88062dd40000 0000000000000086 0000000000000000 >> ffffffffa02628d9 >> [11142.334318] ffff88017a517ef8 000000000000fa40 ffff88017a517fd8 >> 0000000000016940 >> [11142.334322] 0000000000016940 ffff88032b644800 ffff88032b644af8 >> 0000000b7a517cd8 >> [11142.334325] Call Trace: >> [11142.334340] [] ? gfs2_glock_put+0xf9/0x118 [gfs2] >> [11142.334347] [] ? gfs2_glock_holder_wait+0x0/0xd [gfs2] >> [11142.334353] [] ? gfs2_glock_holder_wait+0x9/0xd [gfs2] >> [11142.334358] [] ? __wait_on_bit+0x41/0x70 >> [11142.334363] [] ? gfs2_glock_holder_wait+0x0/0xd [gfs2] >> [11142.334367] [] ? out_of_line_wait_on_bit+0x6b/0x77 >> [11142.334370] [] ? wake_bit_function+0x0/0x23 >> [11142.334376] [] ? gfs2_glock_wait+0x23/0x28 [gfs2] >> [11142.334383] [] ? gfs2_flock+0x17c/0x1f9 [gfs2] >> [11142.334386] [] ? virt_to_head_page+0x9/0x2a >> [11142.334389] [] ? ub_slab_ptr+0x22/0x65 >> [11142.334393] [] ? sys_flock+0xff/0x12a >> [11142.334396] [] ? system_call_fastpath+0x16/0x1b >> >> Any idea what is going wrong? Do you need any more informations? >> >> Mario >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > From jeff.sturm at eprize.com Wed Jan 5 13:47:44 2011 From: jeff.sturm at eprize.com (Jeff Sturm) Date: Wed, 5 Jan 2011 08:47:44 -0500 Subject: [Linux-cluster] GFS block size In-Reply-To: <102121344.150135.1294168658370.JavaMail.root@zmail01.collab.prod.int.phx2.redhat.com> References: <102121344.150135.1294168658370.JavaMail.root@zmail01.collab.prod.int.phx2.redhat.com> Message-ID: <64D0546C5EBBD147B75DE133D798665F06A1296E@hugo.eprize.local> Adam, Thank you for the background on stuffed inodes and resource groups, it is much appreciated. For this specific application most files are under 1k. A few are larger (20-30k) but they are rare and so I think we can accommodate a small performance hit for these. Overall the file system may contain 500,000 or more of these small files at a time. The improvement we measured is a bit more than "modest". Our benchmark finishes about 30% faster with the 1k block size compared to 4k. That's a nice win for a simple change. Disk bandwidth to/from shared storage might be a factor--we have 12 nodes accessing this storage, so the aggregate bandwidth is considerable. It has been suggested to me that NFS would yield more performance gains, but I have not attempted this. RHCS has so far met our expectations of high availability. Given that NFS is not a cluster file system I'm nervous that such a setup could introduce new points of failure. (I realize that NFS could be coupled with e.g. DRBD+pacemaker for failover purposes.) We implemented the typical GFS1 tuneables long ago (noatime, noquota, statfs_fast). Disabling SELinux also helped. Checking block size was truly an afterthought, and we had not given any consideration to resource group size either. I've learned a ton about disk storage by implementing shared storage and clustered filesystems over the past 3 years. Block devices are a bit "magical" in general, and widely misunderstood by system administrators and software engineers. (For example, I've heard some fantastic performance claims on ext3 file systems that turned out to demonstrate how effective Linux is at hiding disk latency.) Thanks again to you and this list for providing continued insight. -Jeff > -----Original Message----- > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] > On Behalf Of Adam Drew > Sent: Tuesday, January 04, 2011 2:18 PM > To: linux clustering > Subject: Re: [Linux-cluster] GFS block size > > If your average file size is less than 1K then using a block size of 1k may be a good > option. If you can fit your data in a single block you get the minor performance boost > of using a stuffed inode so you never have to walk a list from your inode to your data > block. The performance boost should be small but could add up to larger gains over > time with lots of transactions. If your average data payload is less than the default > block-size however, you'll end up losing the delta. So, from a filesystem perspective, > using a 1k blocksize to store mostly sub-1k files may be a good idea. > > You additionally may want to experiment with reducing your resource group size. > Blocks are organized into resource groups. If you are using 1k blocks and sub-1k files > then you'll end up with tons of stuffed inodes per resource group. Some operations in > GFS require locking the resource group metadata (such as deletes) so you may start > to experience performance bottle-necks depending on usage patterns and disk layout. > > All-in-all I'd be skeptical of the claim of large performance gains over time by changing > rg size and block size but modest gains may be had. Still, some access patterns and > filesystem layouts may experience greater performance gains with such tweaking. > However, I would expect to see the most significant gains (in GFS1 at least) made by > mount options and tuneables. > > Regards, > Adam Drew > > ----- Original Message ----- > From: "juncheol park" > To: "linux clustering" > Sent: Tuesday, January 4, 2011 1:42:45 PM > Subject: Re: [Linux-cluster] GFS block size > > I also experimented 1k block size on GFS1. Although you can improve the disk usage > using a smaller block size, typically it is recommended to use the block size same as > the page size, which is 4k in Linux. > > I don't remember all the details of results. However, for large files, the overall > performance of read/write operations with 1k block size was much worse than the one > with 4k block size. This is obvious, though. If you don't care any performance > degradation for large files, it would be fine for you to use 1k. > > Just my two cents, > > -Jun > > > On Fri, Dec 17, 2010 at 3:53 PM, Jeff Sturm wrote: > > One of our GFS filesystems tends to have a large number of very small > > files, on average about 1000 bytes each. > > > > > > > > I realized this week we'd created our filesystems with default > > options.? As an experiment on a test system, I've recreated a GFS > > filesystem with "-b 1024" to reduce overall disk usage and disk bandwidth. > > > > > > > > Initially, tests look very good?single file creates are less than one > > millisecond on average (down from about 5ms each).? Before I go very > > far with this, I wanted to ask:? Has anyone else experimented with the > > block size option, and are there any tricks or gotchas to report? > > > > > > > > (This is with CentOS 5.5, GFS 1.) > > > > > > > > -Jeff > > > > > > > > -- > > Linux-cluster mailing list > > Linux-cluster at redhat.com > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From luiceur at gmail.com Wed Jan 5 16:08:28 2011 From: luiceur at gmail.com (Luis Cebamanos) Date: Wed, 05 Jan 2011 16:08:28 +0000 Subject: [Linux-cluster] How to re-store my lost data Message-ID: <4D24977C.1080301@gmail.com> Dear everyone! we have recently had an unknown problem with our cluster and we have lost some data, including the latest user accounts created. Does anyone have any idea of how to recover those user accounts and data? The data haven't been deleted so it should be in somewhere in the disk!!! Please, any help would be much more appreciated. Best, Luis From linux at alteeve.com Wed Jan 5 16:28:24 2011 From: linux at alteeve.com (Digimer) Date: Wed, 05 Jan 2011 11:28:24 -0500 Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D24977C.1080301@gmail.com> References: <4D24977C.1080301@gmail.com> Message-ID: <4D249C28.7050406@alteeve.com> On 01/05/2011 11:08 AM, Luis Cebamanos wrote: > Dear everyone! > > we have recently had an unknown problem with our cluster and we have > lost some data, including the latest user accounts created. > Does anyone have any idea of how to recover those user accounts and data? > The data haven't been deleted so it should be in somewhere in the disk!!! > > Please, any help would be much more appreciated. > > > Best, > > Luis Please provide more details. Specifically, what file system? How did the data loss occur (as best as you know)? What versions of what cluster applications? What has been done since then? Where and how was the data stored? etc. -- Digimer E-Mail: digimer at alteeve.com AN!Whitepapers: http://alteeve.com Node Assassin: http://nodeassassin.org From rpeterso at redhat.com Wed Jan 5 16:55:08 2011 From: rpeterso at redhat.com (Bob Peterson) Date: Wed, 5 Jan 2011 11:55:08 -0500 (EST) Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D24977C.1080301@gmail.com> Message-ID: <1776791552.141901.1294246508264.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> ----- Original Message ----- | Dear everyone! | | we have recently had an unknown problem with our cluster and we have | lost some data, including the latest user accounts created. | Does anyone have any idea of how to recover those user accounts and | data? | The data haven't been deleted so it should be in somewhere in the | disk!!! | | Please, any help would be much more appreciated. | | | Best, | | Luis We need lots of more details before we can help. For example: Is this clustered storage? Is it your root mount point? What file system is this? GFS? GFS2? ext3? ext4? Other? What happened to it? Did you lose drives in a RAID array? Did you notice the missing data before or after fsck was run? We need lots of details. Regards, Bob Peterson Red Hat File Systems From jcasale at activenetwerx.com Wed Jan 5 23:37:07 2011 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Wed, 5 Jan 2011 23:37:07 +0000 Subject: [Linux-cluster] Service State via snmp Message-ID: Is there any way to query if a service is frozen with snmp, it doesn't appear that "running" is distinguished from "frozen" in rhcServiceStatusCode and I was hoping to avoid anything not native to snmp in order to deduce this. I could use an extend or exec for example but that's not really desirable... Thanks, jlc From parvez.h.shaikh at gmail.com Thu Jan 6 05:24:40 2011 From: parvez.h.shaikh at gmail.com (Parvez Shaikh) Date: Thu, 6 Jan 2011 10:54:40 +0530 Subject: [Linux-cluster] Determining red hat cluster version Message-ID: Hi all, Is there any command which states Red Hat cluster version? I tried cman_tool version, and ccs_tool -V both produce different results, most likely reporting version of their own (not of Cluster suite) yum list installed *Cluster* produces following - Installed Packages Cluster_Administration-en-US.noarch 5.2-1 installed ..... Does it mean cluster version is 5.2-1? Any help will be appreciated. Thanks From fdinitto at redhat.com Thu Jan 6 07:04:53 2011 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Thu, 06 Jan 2011 08:04:53 +0100 Subject: [Linux-cluster] Determining red hat cluster version In-Reply-To: References: Message-ID: <4D256995.5040706@redhat.com> On 1/6/2011 6:24 AM, Parvez Shaikh wrote: > Hi all, > > Is there any command which states Red Hat cluster version? > > I tried cman_tool version, and ccs_tool -V both produce different > results, most likely reporting version of their own (not of Cluster > suite) > rpm -q -f $(which cman_tool) is one option, otherwise you need to parse cman_tool protocol version manually. Fabio From parvez.h.shaikh at gmail.com Thu Jan 6 07:28:29 2011 From: parvez.h.shaikh at gmail.com (Parvez Shaikh) Date: Thu, 6 Jan 2011 12:58:29 +0530 Subject: [Linux-cluster] Determining red hat cluster version In-Reply-To: <4D256995.5040706@redhat.com> References: <4D256995.5040706@redhat.com> Message-ID: Hi Fabio This produces output - cman-2.0.115-29.el5 So does it indicate 2.0.115-29 is version? On Thu, Jan 6, 2011 at 12:34 PM, Fabio M. Di Nitto wrote: > On 1/6/2011 6:24 AM, Parvez Shaikh wrote: >> Hi all, >> >> Is there any command which states Red Hat cluster version? >> >> I tried cman_tool version, and ccs_tool -V both produce different >> results, most likely reporting version of their own (not of Cluster >> suite) >> > > rpm -q -f $(which cman_tool) is one option, otherwise you need to parse > cman_tool protocol version manually. > > Fabio > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From fdinitto at redhat.com Thu Jan 6 07:44:52 2011 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Thu, 06 Jan 2011 08:44:52 +0100 Subject: [Linux-cluster] Determining red hat cluster version In-Reply-To: References: <4D256995.5040706@redhat.com> Message-ID: <4D2572F4.8070903@redhat.com> On 1/6/2011 8:28 AM, Parvez Shaikh wrote: > Hi Fabio > > This produces output - > > cman-2.0.115-29.el5 > > So does it indicate 2.0.115-29 is version? yes Fabio From parvez.h.shaikh at gmail.com Thu Jan 6 08:02:52 2011 From: parvez.h.shaikh at gmail.com (Parvez Shaikh) Date: Thu, 6 Jan 2011 13:32:52 +0530 Subject: [Linux-cluster] Determining red hat cluster version In-Reply-To: <4D2572F4.8070903@redhat.com> References: <4D256995.5040706@redhat.com> <4D2572F4.8070903@redhat.com> Message-ID: Thanks Fabio Is this version same as what can be referred as version of "Red Hat Cluster Suite"? The reason I am asking is, as a part of RHCS there are various components (Cluster_Administration-en-US, cluster-cim, cluster-snmp, cman, rgmanager, luci, ricci etc etc) and each of which shows its own version - cman has version as below, rgmanager as version 2.0.52. cluster-cim and cluster-snmp,modcluster has version 0.12.1, system-config-cluster has 1.0.57 version. Is there one version number referring to "Cluster Suite" which would have encompassed entire set of components (with their own versions may be) Gratefully yours On Thu, Jan 6, 2011 at 1:14 PM, Fabio M. Di Nitto wrote: > On 1/6/2011 8:28 AM, Parvez Shaikh wrote: >> Hi Fabio >> >> This produces output - >> >> cman-2.0.115-29.el5 >> >> So does it indicate 2.0.115-29 is version? > > yes > > Fabio > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From fdinitto at redhat.com Thu Jan 6 08:18:37 2011 From: fdinitto at redhat.com (Fabio M. Di Nitto) Date: Thu, 06 Jan 2011 09:18:37 +0100 Subject: [Linux-cluster] Determining red hat cluster version In-Reply-To: References: <4D256995.5040706@redhat.com> <4D2572F4.8070903@redhat.com> Message-ID: <4D257ADD.3080306@redhat.com> On 1/6/2011 9:02 AM, Parvez Shaikh wrote: > Thanks Fabio > > Is this version same as what can be referred as version of "Red Hat > Cluster Suite"? > > The reason I am asking is, as a part of RHCS there are various > components (Cluster_Administration-en-US, cluster-cim, cluster-snmp, > cman, rgmanager, luci, ricci etc etc) and each of which shows its own > version - Each component is separate for a reason. In general it?s safe enough to refer to RHCS to the cman version, but when filing bugs, it?s always best to get the correct version for a specific component. Fabio From parvez.h.shaikh at gmail.com Thu Jan 6 13:53:20 2011 From: parvez.h.shaikh at gmail.com (Parvez Shaikh) Date: Thu, 6 Jan 2011 19:23:20 +0530 Subject: [Linux-cluster] configuring bladecenter fence device Message-ID: Hi all, >From RHCS documentation, I could see that bladecenter is one of the fence devices - http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Administration/ap-fence-device-param-CA.html Table B.9. IBM Blade Center Field Description Name A name for the IBM BladeCenter device connected to the cluster. IP Address The IP address assigned to the device. Login The login name used to access the device. Password The password used to authenticate the connection to the device. Password Script (optional) The script that supplies a password for access to the fence device. Using this supersedes the Password parameter. Blade The blade of the device. Use SSH (Rhel 5.4 and later) Indicates that system will use SSH to access the device. As per my understanding, IP address is IP address of management module of IBM blade center, login/password represent credentials to access the same. However did not get the parameter 'Blade'. How does it play role in fencing? In a situation where there are two blades - Blade-1 and Blade-2 and if Blade-1 goes down(hardware node failure), Blade-2 should fence out Blade-1, in that situation fenced on Blade-2 should power off(?) blade-2 using fence_bladecenter, so how should below sniplet of cluster.conf file should look like? - In which situation fence_bladecenter would be used to power on the blade? Your gratefully From bturner at redhat.com Thu Jan 6 16:06:41 2011 From: bturner at redhat.com (Ben Turner) Date: Thu, 6 Jan 2011 11:06:41 -0500 (EST) Subject: [Linux-cluster] configuring bladecenter fence device In-Reply-To: Message-ID: <752752065.118339.1294330001557.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> To address: As per my understanding, IP address is IP address of management module of IBM blade center, login/password represent credentials to access the same. >> Correct. However did not get the parameter 'Blade'. How does it play role in fencing? >> If I recall correctly the blade= is the identifier used to identify the blade in the AMM. I can't remember if it is a number of a slot or a user defined name. It corresponds to # fence_bladecenter -h -n, --plug= Physical plug number on device or name of virtual machine If the fencing code: "port" : { "getopt" : "n:", "longopt" : "plug", "help" : "-n, --plug= Physical plug number on device or\n" + " name of virtual machine", "required" : "1", "shortdesc" : "Physical plug number or name of virtual machine", "order" : 1 }, To test this try running: /sbin/fence_bladecenter -a -l -p -n -o status -v An example cluster.conf looks like: -Ben ----- Original Message ----- > Hi all, > > >From RHCS documentation, I could see that bladecenter is one of the > fence devices - > http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Administration/ap-fence-device-param-CA.html > > Table B.9. IBM Blade Center > Field Description > Name A name for the IBM BladeCenter device connected to the cluster. > IP Address The IP address assigned to the device. > Login The login name used to access the device. > Password The password used to authenticate the connection to the > device. > Password Script (optional) The script that supplies a password for > access to the fence device. Using this supersedes the Password > parameter. > Blade The blade of the device. > Use SSH (Rhel 5.4 and later) Indicates that system will use SSH to > access the device. > > As per my understanding, IP address is IP address of management module > of IBM blade center, login/password represent credentials to access > the same. > > However did not get the parameter 'Blade'. How does it play role in > fencing? > > In a situation where there are two blades - Blade-1 and Blade-2 and > if Blade-1 goes down(hardware node failure), Blade-2 should fence out > Blade-1, in that situation fenced on Blade-2 should power off(?) > blade-2 using fence_bladecenter, so how should below sniplet of > cluster.conf file should look like? - > > > > > > > name="BLADECENTER"/> > > > > > > > name="BLADECENTER"/> > > > > > > In which situation fence_bladecenter would be used to power on the > blade? > > Your gratefully > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From bradley.a.morrison at jpmchase.com Thu Jan 6 17:01:32 2011 From: bradley.a.morrison at jpmchase.com (Morrison, Bradley A) Date: Thu, 6 Jan 2011 12:01:32 -0500 Subject: [Linux-cluster] reboot to rejoin RAC cluster? Message-ID: Q: Can I reboot one node in a two-node cluster and have it rejoin the cluster? I've a two-node cluster which recently had HBAs replaced on both cluster nodes. Node 1 was ejected sometime after its latest reboot, and now won't mount its OCFS volumes. The volumes' headers are verified from n1, i.e., it can see the volumes, but mounting fails. Restarting o2cb yields "modprobe: FATAL: Module ocfs2_stackglue not found." I want to reboot n1 IFF this will have it rejoin the cluster - unless there's another way to have n1 rejoin the cluster w/o reboot. Status for n1: n1# service o2cb status Driver for "configfs": Loaded Filesystem "configfs": Mounted Driver for "ocfs2_dlmfs": Loaded Filesystem "ocfs2_dlmfs": Mounted Checking O2CB cluster ocfs2: Online Heartbeat dead threshold = 61 Network idle timeout: 30000 Network keepalive delay: 2000 Network reconnect delay: 2000 Checking O2CB heartbeat: Not active n1# Status for n2: n2# service o2cb status service o2cb status Driver for "configfs": Loaded Filesystem "configfs": Mounted Driver for "ocfs2_dlmfs": Loaded Filesystem "ocfs2_dlmfs": Mounted Checking O2CB cluster ocfs2: Online Heartbeat dead threshold = 61 Network idle timeout: 30000 Network keepalive delay: 2000 Network reconnect delay: 2000 Checking O2CB heartbeat: Active n2# This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is privileged, confidential, legally privileged, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities. -------------- next part -------------- An HTML attachment was scrubbed... URL: From james.hofmeister at hp.com Thu Jan 6 18:00:30 2011 From: james.hofmeister at hp.com (Hofmeister, James (WTEC Linux)) Date: Thu, 6 Jan 2011 18:00:30 +0000 Subject: [Linux-cluster] What is the current recommendation concerning shutting down a cluster node? Message-ID: What is the current recommendation concerning shutting down a cluster node? Is it acceptable to use shutdown or init to reboot an active/running cluster node? I have reviewed the RHCS admin guide and it does not state to *not* use shutdown or init. RHCS RH436 training manual (page 305) says the way to remove a node from the cluster is: umount gfs service rgmanager stop service gfs stop service clvmd stop service cman stop ...and I have found other references that say: Use the "leave cluster" functionality from luci. Also "cman_tool leave". Also fence_ command. In the RH436 class it was verbally discussed during the fencing chapter that it was a bad thing to shutdown a node, but to instead power down the machine which has been implemented in many of the fence_ scripts. So the question is: Is it currently acceptable to use shutdown or init to reboot an active/running cluster node? ~and~ if not, is this documented? Regards, ????? James Hofmeister? Hewlett Packard Linux Solutions Engineer From linux at alteeve.com Thu Jan 6 18:51:21 2011 From: linux at alteeve.com (Digimer) Date: Thu, 06 Jan 2011 13:51:21 -0500 Subject: [Linux-cluster] What is the current recommendation concerning shutting down a cluster node? In-Reply-To: References: Message-ID: <4D260F29.2070706@alteeve.com> On 01/06/2011 01:00 PM, Hofmeister, James (WTEC Linux) wrote: > umount gfs > service rgmanager stop > service gfs stop > service clvmd stop > service cman stop This is my method. However, note that stopping GFS unmounts the volumes, so you can skip the manual unmount. -- Digimer E-Mail: digimer at alteeve.com AN!Whitepapers: http://alteeve.com Node Assassin: http://nodeassassin.org From yvette at dbtgroup.com Thu Jan 6 19:30:16 2011 From: yvette at dbtgroup.com (yvette hirth) Date: Thu, 06 Jan 2011 19:30:16 +0000 Subject: [Linux-cluster] [DRBD-user] DRBD and KVM for a HA-Cluster ? In-Reply-To: <4D2611DC.10205@alteeve.com> References: <15785B7E063D464C86DD482FCAE4EBA5012E8F22C05E@XCH11.scidom.de> <4D24BC39.50102@alteeve.com> <15785B7E063D464C86DD482FCAE4EBA5012E8FCD44A9@XCH11.scidom.de> <4D25C92F.3020104@alteeve.com> <15785B7E063D464C86DD482FCAE4EBA5012E8FCD44AB@XCH11.scidom.de> <4D25E7F8.6090101@alteeve.com> <15785B7E063D464C86DD482FCAE4EBA5012E8FD78613@XCH11.scidom.de> <4D2611DC.10205@alteeve.com> Message-ID: <4D261848.9030109@dbtgroup.com> Digimer wrote: (snippage) > In fact, it's a benefit because, last I checked, snapshot'ing of > clvm was not possible. and it still isn't. i tried to slapshot a gfs2 volume and it refused, which makes tar'ing a gfs2 directory - without getting "source volume changed during processing" messages - impossible, at least in my setup (RHEL/Centos 5.5, five-way clustah). so i created xfs filesystems, rsync the gfs2 -> xfs stuff hourly, and backup from the xfs. it's "not-really-wonderful" but works fine. yvette From luiceur at gmail.com Thu Jan 6 19:36:16 2011 From: luiceur at gmail.com (Luis Cebamanos) Date: Thu, 06 Jan 2011 19:36:16 +0000 Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D249C28.7050406@alteeve.com> References: <4D24977C.1080301@gmail.com> <4D249C28.7050406@alteeve.com> Message-ID: <4D2619B0.2040001@gmail.com> Is a cluster with 16 nodes and I suspect the problem is in the head node: $cat /proc/version Linux version 2.6.11.4-21.11-smp (geeko at buildhost) (gcc version 3.3.5 20050117 (prerelease) (SUSE Linux)) #1 SMP Thu Feb 2 20:54:26 GMT 2006 $cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 37 model name : AMD Opteron(tm) Processor 246 stepping : 1 cpu MHz : 1994.349 cache size : 1024 KB physical id : 255 siblings : 1 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 pni syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm bogomips : 3923.96 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp processor : 1 vendor_id : AuthenticAMD cpu family : 15 model : 37 model name : AMD Opteron(tm) Processor 246 stepping : 1 cpu MHz : 1994.349 cache size : 1024 KB physical id : 255 siblings : 1 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 pni syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm bogomips : 3981.31 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp cat /proc/meminfo # MemTotal: 2055264 kB MemFree: 1781708 kB Buffers: 96696 kB Cached: 99892 kB SwapCached: 0 kB Active: 114836 kB Inactive: 98388 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 2055264 kB LowFree: 1781708 kB SwapTotal: 4192924 kB SwapFree: 4192924 kB Dirty: 52 kB Writeback: 0 kB Mapped: 28104 kB Slab: 43688 kB CommitLimit: 5220556 kB Committed_AS: 232052 kB PageTables: 1512 kB VmallocTotal: 34359738367 kB VmallocUsed: 2412 kB VmallocChunk: 34359735867 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB df -T Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sda2 ext3 33032228 19971720 11382520 64% / tmpfs tmpfs 1027632 0 1027632 0% /dev/shm /dev/sda1 ext3 124427 9412 108591 8% /boot /dev/sda6 ext3 2063504 33820 1924864 2% /tmp /dev/sda9 ext3 166698068 119970708 38259504 76% /users /dev/sda8 ext3 32250392 966320 29645848 4% /usr/local /dev/sda7 ext3 2063504 901804 1056880 47% /var We were trying to install new hard drives to the system but something that we don't know went wrong and it ended up in almost 4 years of work lost!!! Please, let me know what else can I do to be able to get the data back! Best On 01/05/2011 04:28 PM, Digimer wrote: > On 01/05/2011 11:08 AM, Luis Cebamanos wrote: >> Dear everyone! >> >> we have recently had an unknown problem with our cluster and we have >> lost some data, including the latest user accounts created. >> Does anyone have any idea of how to recover those user accounts and data? >> The data haven't been deleted so it should be in somewhere in the disk!!! >> >> Please, any help would be much more appreciated. >> >> >> Best, >> >> Luis > Please provide more details. Specifically, what file system? How did the > data loss occur (as best as you know)? What versions of what cluster > applications? What has been done since then? Where and how was the data > stored? etc. > From thomas at sjolshagen.net Thu Jan 6 19:42:37 2011 From: thomas at sjolshagen.net (Thomas Sjolshagen) Date: Thu, 06 Jan 2011 14:42:37 -0500 Subject: [Linux-cluster] =?utf-8?q?What_is_the_current_recommendation_conc?= =?utf-8?q?erning_shutting_down_a_cluster_node=3F?= In-Reply-To: <4D260F29.2070706@alteeve.com> References: <4D260F29.2070706@alteeve.com> Message-ID: On Thu, 06 Jan 2011 13:51:21 -0500, Digimer wrote: > On 01/06/2011 01:00 PM, Hofmeister, James (WTEC Linux) wrote: >> umount gfs >> service rgmanager stop >> service gfs stop >> service clvmd stop >> service cman stop > > This is my method. However, note that stopping GFS unmounts the > volumes, > so you can skip the manual unmount. Being a Fedora 14 user (at this point), rebooting the cluster node(s) with shutdown -r|h now has been working fine for me since transitioning to F14 (should be the same cluster stack as what RHEL 6 uses now, I believe) Migrating any rgmanager services off of the node I'm bringing down first seems to be required (httpd/apache is pretty slow at shutting down and thus causes my shutdown -r generated umount of the gfs|gfs2 file systems to fail) // Thomas PS: is there a specific reason why the GFS/GFS2 service shutdown option doesn't do something akin to "fuser -k" for the mountpoints in order to kill off any processes that would cause the umount to fail? Yes it's brutal, and certain databases would be a little unhappy, but I'd argue any db shutdown script that doesn't stall until the DB is actually down are buggy by definition. From linux at alteeve.com Thu Jan 6 19:44:17 2011 From: linux at alteeve.com (Digimer) Date: Thu, 06 Jan 2011 14:44:17 -0500 Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D2619B0.2040001@gmail.com> References: <4D24977C.1080301@gmail.com> <4D249C28.7050406@alteeve.com> <4D2619B0.2040001@gmail.com> Message-ID: <4D261B91.3050308@alteeve.com> On 01/06/2011 02:36 PM, Luis Cebamanos wrote: > Is a cluster with 16 nodes and I suspect the problem is in the head node: > $cat /proc/version > Linux version 2.6.11.4-21.11-smp (geeko at buildhost) (gcc version 3.3.5 > 20050117 (prerelease) (SUSE Linux)) #1 SMP Thu Feb 2 20:54:26 GMT 2006 > > df -T > Filesystem Type 1K-blocks Used Available Use% Mounted on > /dev/sda2 ext3 33032228 19971720 11382520 64% / > tmpfs tmpfs 1027632 0 1027632 0% /dev/shm > /dev/sda1 ext3 124427 9412 108591 8% /boot > /dev/sda6 ext3 2063504 33820 1924864 2% /tmp > /dev/sda9 ext3 166698068 119970708 38259504 76% /users > /dev/sda8 ext3 32250392 966320 29645848 4% /usr/local > /dev/sda7 ext3 2063504 901804 1056880 47% /var > > We were trying to install new hard drives to the system but something > that we don't know went wrong and it ended up in almost 4 years of work > lost!!! > Please, let me know what else can I do to be able to get the data back! > > Best It's a bit late now, but I suppose you don't have backups? As for how to help, what you provided was only marginally helpful. We need a much more extensive overview of your setup and configuration, versions, etc. before we can have any idea if we can help. In the short term, don't try anything yourself without careful thought. If the data is very valuable, consider hiring a data recovery firm near you who can come and look at your setup. -- Digimer E-Mail: digimer at alteeve.com AN!Whitepapers: http://alteeve.com Node Assassin: http://nodeassassin.org From luiceur at gmail.com Thu Jan 6 19:51:18 2011 From: luiceur at gmail.com (Luis Cebamanos) Date: Thu, 06 Jan 2011 19:51:18 +0000 Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D261B91.3050308@alteeve.com> References: <4D24977C.1080301@gmail.com> <4D249C28.7050406@alteeve.com> <4D2619B0.2040001@gmail.com> <4D261B91.3050308@alteeve.com> Message-ID: <4D261D36.7080008@gmail.com> Well, there are not backups of that valuable lost data but the cluster was using disk mirroring but no one has a clue of how to take advantage of that. Nobody here is a cluster expert and that has been the problem I guess. We haven't really "touch" any important system file, after physically installed the hard disk, we realized that the cluster wasn't properly working. We rebooted the cluster without the old disks and that has been the result. Worst scenario, we will need to call an expert, but we think it can not be a big deal as I said, we haven't modified the previous configuration... On 01/06/2011 07:44 PM, Digimer wrote: > On 01/06/2011 02:36 PM, Luis Cebamanos wrote: >> Is a cluster with 16 nodes and I suspect the problem is in the head node: >> $cat /proc/version >> Linux version 2.6.11.4-21.11-smp (geeko at buildhost) (gcc version 3.3.5 >> 20050117 (prerelease) (SUSE Linux)) #1 SMP Thu Feb 2 20:54:26 GMT 2006 >> >> df -T >> Filesystem Type 1K-blocks Used Available Use% Mounted on >> /dev/sda2 ext3 33032228 19971720 11382520 64% / >> tmpfs tmpfs 1027632 0 1027632 0% /dev/shm >> /dev/sda1 ext3 124427 9412 108591 8% /boot >> /dev/sda6 ext3 2063504 33820 1924864 2% /tmp >> /dev/sda9 ext3 166698068 119970708 38259504 76% /users >> /dev/sda8 ext3 32250392 966320 29645848 4% /usr/local >> /dev/sda7 ext3 2063504 901804 1056880 47% /var >> >> We were trying to install new hard drives to the system but something >> that we don't know went wrong and it ended up in almost 4 years of work >> lost!!! >> Please, let me know what else can I do to be able to get the data back! >> >> Best > It's a bit late now, but I suppose you don't have backups? > > As for how to help, what you provided was only marginally helpful. We > need a much more extensive overview of your setup and configuration, > versions, etc. before we can have any idea if we can help. > > In the short term, don't try anything yourself without careful thought. > If the data is very valuable, consider hiring a data recovery firm near > you who can come and look at your setup. > From gordan at bobich.net Thu Jan 6 20:03:04 2011 From: gordan at bobich.net (Gordan Bobic) Date: Thu, 06 Jan 2011 20:03:04 +0000 Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D261D36.7080008@gmail.com> References: <4D24977C.1080301@gmail.com> <4D249C28.7050406@alteeve.com> <4D2619B0.2040001@gmail.com> <4D261B91.3050308@alteeve.com> <4D261D36.7080008@gmail.com> Message-ID: <4D261FF8.4050704@bobich.net> Luis, You still haven't provided the relevant details of your configuration. The df output you provided isn't relevant in terms of the data recovery. You haven't even mentioned what file system the lost data was on. You mention mirroring - what was doing the mirroring? Hardware RAID? Software MD RAID? Software DM RAID? LVM? Gordan On 01/06/2011 07:51 PM, Luis Cebamanos wrote: > Well, there are not backups of that valuable lost data but the cluster > was using disk mirroring but no one has a clue of how to take advantage > of that. Nobody here is a cluster expert and that has been the problem I > guess. > We haven't really "touch" any important system file, after physically > installed the hard disk, we realized that the cluster wasn't properly > working. We rebooted the cluster without the old disks and that has been > the result. > Worst scenario, we will need to call an expert, but we think it can not > be a big deal as I said, we haven't modified the previous configuration... > On 01/06/2011 07:44 PM, Digimer wrote: >> On 01/06/2011 02:36 PM, Luis Cebamanos wrote: >>> Is a cluster with 16 nodes and I suspect the problem is in the head >>> node: >>> $cat /proc/version >>> Linux version 2.6.11.4-21.11-smp (geeko at buildhost) (gcc version 3.3.5 >>> 20050117 (prerelease) (SUSE Linux)) #1 SMP Thu Feb 2 20:54:26 GMT 2006 >>> >>> df -T >>> Filesystem Type 1K-blocks Used Available Use% Mounted on >>> /dev/sda2 ext3 33032228 19971720 11382520 64% / >>> tmpfs tmpfs 1027632 0 1027632 0% /dev/shm >>> /dev/sda1 ext3 124427 9412 108591 8% /boot >>> /dev/sda6 ext3 2063504 33820 1924864 2% /tmp >>> /dev/sda9 ext3 166698068 119970708 38259504 76% /users >>> /dev/sda8 ext3 32250392 966320 29645848 4% /usr/local >>> /dev/sda7 ext3 2063504 901804 1056880 47% /var >>> >>> We were trying to install new hard drives to the system but something >>> that we don't know went wrong and it ended up in almost 4 years of work >>> lost!!! >>> Please, let me know what else can I do to be able to get the data back! >>> >>> Best >> It's a bit late now, but I suppose you don't have backups? >> >> As for how to help, what you provided was only marginally helpful. We >> need a much more extensive overview of your setup and configuration, >> versions, etc. before we can have any idea if we can help. >> >> In the short term, don't try anything yourself without careful thought. >> If the data is very valuable, consider hiring a data recovery firm near >> you who can come and look at your setup. >> > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From work at fajar.net Thu Jan 6 20:05:34 2011 From: work at fajar.net (Fajar A. Nugraha) Date: Fri, 7 Jan 2011 03:05:34 +0700 Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D2619B0.2040001@gmail.com> References: <4D24977C.1080301@gmail.com> <4D249C28.7050406@alteeve.com> <4D2619B0.2040001@gmail.com> Message-ID: On Fri, Jan 7, 2011 at 2:36 AM, Luis Cebamanos wrote: > Is a cluster with 16 nodes and I suspect the problem is in the head node: > We were trying to install new hard drives to the system but something that > we don't know went wrong and it ended up in almost 4 years of work lost!!! > Please, let me know what else can I do to be able to get the data back! You still haven't answered these questions >> Please provide more details. Specifically, what file system? If only ext3 is involved, and you're sharing it to other nodes from the head node via nfs, then this list is probably the wrong place to ask. >> How did the >> data loss occur (as best as you know)? What versions of what cluster >> applications? ... and if you don't even know what cluster application you use, no one will be able to help even if they want to. >> What has been done since then? Where and how was the data >> stored? etc. ... continuing the part of "how was the data stored", since you df output only shows sda mounted, you can start by checking : - how many disks you have - are they all detected (e.g 4 disks usually show up as sda - sdd). - do you use LVM - how are the disks mounted. Is it through fstab, or does the cluster takes care of mounting the resource as well /etc/fstab and /var/log/messages should be a good place to start looking. Simply saying "something that we don't know went wrong" won't be getting you anywhere. If you think you need more expertise, getting help from a local expert who can get hands-on to your servers and know what to look for would be a good start. -- Fajar From linux at alteeve.com Thu Jan 6 20:09:52 2011 From: linux at alteeve.com (Digimer) Date: Thu, 06 Jan 2011 15:09:52 -0500 Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D261D36.7080008@gmail.com> References: <4D24977C.1080301@gmail.com> <4D249C28.7050406@alteeve.com> <4D2619B0.2040001@gmail.com> <4D261B91.3050308@alteeve.com> <4D261D36.7080008@gmail.com> Message-ID: <4D262190.1090609@alteeve.com> On 01/06/2011 02:51 PM, Luis Cebamanos wrote: > Well, there are not backups of that valuable lost data but the cluster > was using disk mirroring but no one has a clue of how to take advantage > of that. Nobody here is a cluster expert and that has been the problem I > guess. > We haven't really "touch" any important system file, after physically > installed the hard disk, we realized that the cluster wasn't properly > working. We rebooted the cluster without the old disks and that has been > the result. > Worst scenario, we will need to call an expert, but we think it can not > be a big deal as I said, we haven't modified the previous configuration... Luis, Stop doing anything and call an expert right away. If your data is valuable and you are not familiar with clustering and/or systems administration, you will very likely remove any remaining chance for data recovery. In the future, *always* have external backups! -- Digimer E-Mail: digimer at alteeve.com AN!Whitepapers: http://alteeve.com Node Assassin: http://nodeassassin.org From luiceur at gmail.com Thu Jan 6 20:07:45 2011 From: luiceur at gmail.com (Luis Cebamanos) Date: Thu, 06 Jan 2011 20:07:45 +0000 Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D261B91.3050308@alteeve.com> References: <4D24977C.1080301@gmail.com> <4D249C28.7050406@alteeve.com> <4D2619B0.2040001@gmail.com> <4D261B91.3050308@alteeve.com> Message-ID: <4D262111.6040903@gmail.com> It is not only data, it is like the system were booting from a 4 years time configuration as everything that we have done, included other kind of configuration files, just disappeared!!! On 01/06/2011 07:44 PM, Digimer wrote: > On 01/06/2011 02:36 PM, Luis Cebamanos wrote: >> Is a cluster with 16 nodes and I suspect the problem is in the head node: >> $cat /proc/version >> Linux version 2.6.11.4-21.11-smp (geeko at buildhost) (gcc version 3.3.5 >> 20050117 (prerelease) (SUSE Linux)) #1 SMP Thu Feb 2 20:54:26 GMT 2006 >> >> df -T >> Filesystem Type 1K-blocks Used Available Use% Mounted on >> /dev/sda2 ext3 33032228 19971720 11382520 64% / >> tmpfs tmpfs 1027632 0 1027632 0% /dev/shm >> /dev/sda1 ext3 124427 9412 108591 8% /boot >> /dev/sda6 ext3 2063504 33820 1924864 2% /tmp >> /dev/sda9 ext3 166698068 119970708 38259504 76% /users >> /dev/sda8 ext3 32250392 966320 29645848 4% /usr/local >> /dev/sda7 ext3 2063504 901804 1056880 47% /var >> >> We were trying to install new hard drives to the system but something >> that we don't know went wrong and it ended up in almost 4 years of work >> lost!!! >> Please, let me know what else can I do to be able to get the data back! >> >> Best > It's a bit late now, but I suppose you don't have backups? > > As for how to help, what you provided was only marginally helpful. We > need a much more extensive overview of your setup and configuration, > versions, etc. before we can have any idea if we can help. > > In the short term, don't try anything yourself without careful thought. > If the data is very valuable, consider hiring a data recovery firm near > you who can come and look at your setup. > From thomas at sjolshagen.net Thu Jan 6 20:18:17 2011 From: thomas at sjolshagen.net (Thomas Sjolshagen) Date: Thu, 06 Jan 2011 15:18:17 -0500 Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D261D36.7080008@gmail.com> References: <4D24977C.1080301@gmail.com> <4D249C28.7050406@alteeve.com> <4D2619B0.2040001@gmail.com> <4D261B91.3050308@alteeve.com> <4D261D36.7080008@gmail.com> Message-ID: <099b08dbe663c998c8b14a7bf84bd0ad@sjolshagen.net> We'd need to know how your storage is configured. Everything from how storage is connected to the system (if it's external) or if you're using an internal RAID controller. What sort of mirroring are you using, how many locally connected HDD drives do you have, how are the clustered file systems connected to this system (iSCSI, Fibre Channel, something else), the contents of /etc/fstab, /etc/cluster/cluster.conf, output from the lvm commands (IIRC, #pvs, #lvs and #vgs will work on RHEL to display any LVM configuration hightlights). Also, what version of the cluster stack are you using (and what components - things like chkconfig --list, rpm -qa | grep cman, etc, etc). There are a couple of documents on the web that will provide you with examples/suggestions for the types of data you can/should collect before reporting a problem with the Red Hat Cluster Stack. Also, what caused you to change the drive, how did it get changed and was anything done as part of that change in order to get the system back online again. If you don't know what you need to look at, I'm not sure you're the optimal person to try and fix this problem. Data recovery can be a fairly low-level thing in the right (wrong) set of circumstances. // Thomas On Thu, 06 Jan 2011 19:51:18 +0000, Luis Cebamanos wrote: > Well, there are not backups of that valuable lost data but the > cluster was using disk mirroring but no one has a clue of how to take > advantage of that. Nobody here is a cluster expert and that has been > the problem I guess. > We haven't really "touch" any important system file, after physically > installed the hard disk, we realized that the cluster wasn't properly > working. We rebooted the cluster without the old disks and that has > been the result. > Worst scenario, we will need to call an expert, but we think it can > not be a big deal as I said, we haven't modified the previous > configuration... > On 01/06/2011 07:44 PM, Digimer wrote: >> On 01/06/2011 02:36 PM, Luis Cebamanos wrote: >>> Is a cluster with 16 nodes and I suspect the problem is in the head >>> node: >>> $cat /proc/version >>> Linux version 2.6.11.4-21.11-smp (geeko at buildhost) (gcc version >>> 3.3.5 >>> 20050117 (prerelease) (SUSE Linux)) #1 SMP Thu Feb 2 20:54:26 GMT >>> 2006 >>> >>> df -T >>> Filesystem Type 1K-blocks Used Available Use% Mounted on >>> /dev/sda2 ext3 33032228 19971720 11382520 64% / >>> tmpfs tmpfs 1027632 0 1027632 0% /dev/shm >>> /dev/sda1 ext3 124427 9412 108591 8% /boot >>> /dev/sda6 ext3 2063504 33820 1924864 2% /tmp >>> /dev/sda9 ext3 166698068 119970708 38259504 76% /users >>> /dev/sda8 ext3 32250392 966320 29645848 4% /usr/local >>> /dev/sda7 ext3 2063504 901804 1056880 47% /var >>> >>> We were trying to install new hard drives to the system but >>> something >>> that we don't know went wrong and it ended up in almost 4 years of >>> work >>> lost!!! >>> Please, let me know what else can I do to be able to get the data >>> back! >>> >>> Best >> It's a bit late now, but I suppose you don't have backups? >> >> As for how to help, what you provided was only marginally helpful. >> We >> need a much more extensive overview of your setup and configuration, >> versions, etc. before we can have any idea if we can help. >> >> In the short term, don't try anything yourself without careful >> thought. >> If the data is very valuable, consider hiring a data recovery firm >> near >> you who can come and look at your setup. >> > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From linux at alteeve.com Thu Jan 6 20:19:50 2011 From: linux at alteeve.com (Digimer) Date: Thu, 06 Jan 2011 15:19:50 -0500 Subject: [Linux-cluster] How to re-store my lost data In-Reply-To: <4D262111.6040903@gmail.com> References: <4D24977C.1080301@gmail.com> <4D249C28.7050406@alteeve.com> <4D2619B0.2040001@gmail.com> <4D261B91.3050308@alteeve.com> <4D262111.6040903@gmail.com> Message-ID: <4D2623E6.4080708@alteeve.com> On 01/06/2011 03:07 PM, Luis Cebamanos wrote: > It is not only data, it is like the system were booting from a 4 years > time configuration as everything that we have done, included other kind > of configuration files, just disappeared!!! I apologize now for being blunt; None of that matters. What you are telling us is useless information. You have not even told us yet what operating system you are running, never mind other applications or configuration information. Step back, take a deep breath, and come back with actual application names, their versions and how they are configured. If you can not do this then you should not be attempting to recover the data as you will only make things worse. -- Digimer E-Mail: digimer at alteeve.com AN!Whitepapers: http://alteeve.com Node Assassin: http://nodeassassin.org From jlbec at evilplan.org Thu Jan 6 22:03:29 2011 From: jlbec at evilplan.org (Joel Becker) Date: Thu, 6 Jan 2011 14:03:29 -0800 Subject: [Linux-cluster] reboot to rejoin RAC cluster? In-Reply-To: References: Message-ID: <20110106220329.GA3312@mail.oracle.com> On Thu, Jan 06, 2011 at 12:01:32PM -0500, Morrison, Bradley A wrote: > Q: Can I reboot one node in a two-node cluster and have it rejoin the cluster? You certainly should be able to. You should not need a reboot either if you just want to rejoin. > I've a two-node cluster which recently had HBAs replaced on both cluster nodes. > Node 1 was ejected sometime after its latest reboot, and now won't mount its OCFS volumes. The volumes' headers are verified from n1, i.e., it can see the volumes, but mounting fails. Restarting o2cb yields "modprobe: FATAL: Module ocfs2_stackglue not found." This sounds like a kernel configuration problem. If you can't load ocfs2_stackglue, you can't get the filesystem started. What kernel are you running? Joel -- "When arrows don't penetrate, see... Cupid grabs the pistol." http://www.jlbec.org/ jlbec at evilplan.org From rhel-cluster at feystorm.net Fri Jan 7 01:58:18 2011 From: rhel-cluster at feystorm.net (Patrick H.) Date: Thu, 06 Jan 2011 18:58:18 -0700 Subject: [Linux-cluster] RHEL6 & IP load balancing Message-ID: <4D26733A.9010109@feystorm.net> So I just started setting up a RHEL6 box for use in a load balanced cluster and have run across a problem. The way you set up a virtual IP on the back end realhost side is to add an interface alias to the loopback device (such as lo:0). Well the ifup-eth script in RHEL6 refuses to add aliases to the loopback interface. Additionally if you try to add the alias to the real ethX device instead it fails because the arping check it does finds that the IP is already running on the IPVS director. So, how is one supposed to setup a realhost now? The difference from RHEL5 is that RHEL5 doesnt check to see if youre adding an alias to the loopback device or not. Why does RHEL6 even care about that anyway? Theres nothing wrong with it... From bradley.a.morrison at jpmchase.com Fri Jan 7 04:37:17 2011 From: bradley.a.morrison at jpmchase.com (Morrison, Bradley A) Date: Thu, 6 Jan 2011 23:37:17 -0500 Subject: [Linux-cluster] reboot to rejoin RAC cluster? In-Reply-To: <20110106220329.GA3312@mail.oracle.com> References: <20110106220329.GA3312@mail.oracle.com> Message-ID: Thanks for responding, Joel. It turned out to be a VLAN change which required a change with the bonded NIC. Immediately after issuing ifenslave -c bond-hb eth2 on the problem node, it joined the cluster and the OCFS volumes could be mounted. -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Joel Becker Sent: Thursday, January 06, 2011 4:03 PM To: linux clustering Subject: Re: [Linux-cluster] reboot to rejoin RAC cluster? On Thu, Jan 06, 2011 at 12:01:32PM -0500, Morrison, Bradley A wrote: > Q: Can I reboot one node in a two-node cluster and have it rejoin the cluster? You certainly should be able to. You should not need a reboot either if you just want to rejoin. > I've a two-node cluster which recently had HBAs replaced on both cluster nodes. > Node 1 was ejected sometime after its latest reboot, and now won't mount its OCFS volumes. The volumes' headers are verified from n1, i.e., it can see the volumes, but mounting fails. Restarting o2cb yields "modprobe: FATAL: Module ocfs2_stackglue not found." This sounds like a kernel configuration problem. If you can't load ocfs2_stackglue, you can't get the filesystem started. What kernel are you running? Joel -- "When arrows don't penetrate, see... Cupid grabs the pistol." http://www.jlbec.org/ jlbec at evilplan.org -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is privileged, confidential, legally privileged, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities. From parvez.h.shaikh at gmail.com Fri Jan 7 04:42:16 2011 From: parvez.h.shaikh at gmail.com (Parvez Shaikh) Date: Fri, 7 Jan 2011 10:12:16 +0530 Subject: [Linux-cluster] configuring bladecenter fence device In-Reply-To: <752752065.118339.1294330001557.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> References: <752752065.118339.1294330001557.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Message-ID: Hi Ben Thanks a ton for below information. But I have doubt on cluster.conf file snippet below - Here for "node1" device blade is "2". Does it mean node1 is blade[2] from AMM perspective? So in order to fence out node1 fence_bladecenter would turn off blade[2]? Thanks On Thu, Jan 6, 2011 at 9:36 PM, Ben Turner wrote: > To address: > > As per my understanding, IP address is IP address of management module > of IBM blade center, login/password represent credentials to access > the same. > >>> Correct. > > However did not get the parameter 'Blade'. How does it play role in > fencing? > >>> If I recall correctly the blade= is the identifier used to identify the blade in the AMM. ?I can't remember if it is a number of a slot or a user defined name. ?It corresponds to > > # fence_bladecenter -h > > ? -n, --plug= ? ? ? ? ? ? ? ?Physical plug number on device or > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?name of virtual machine > If the fencing code: > > ? ? ? ?"port" : { > ? ? ? ? ? ? ? ?"getopt" : "n:", > ? ? ? ? ? ? ? ?"longopt" : "plug", > ? ? ? ? ? ? ? ?"help" : "-n, --plug= ? ? ? ? ? ? ? ?Physical plug number on device or\n" + > ? ? ? ? ? ? ? ?" ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?name of virtual machine", > ? ? ? ? ? ? ? ?"required" : "1", > ? ? ? ? ? ? ? ?"shortdesc" : "Physical plug number or name of virtual machine", > ? ? ? ? ? ? ? ?"order" : 1 }, > > To test this try running: > > /sbin/fence_bladecenter -a -l -p -n -o status -v > > An example cluster.conf looks like: > > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? > > ? ? ? > ? ? ? ? ? ? ? ? passwd="XXXXXXX"/> > ? ? ? ? > > -Ben > > > > > ----- Original Message ----- >> Hi all, >> >> >From RHCS documentation, I could see that bladecenter is one of the >> fence devices - >> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Administration/ap-fence-device-param-CA.html >> >> Table B.9. IBM Blade Center >> Field Description >> Name A name for the IBM BladeCenter device connected to the cluster. >> IP Address The IP address assigned to the device. >> Login The login name used to access the device. >> Password The password used to authenticate the connection to the >> device. >> Password Script (optional) The script that supplies a password for >> access to the fence device. Using this supersedes the Password >> parameter. >> Blade The blade of the device. >> Use SSH (Rhel 5.4 and later) Indicates that system will use SSH to >> access the device. >> >> As per my understanding, IP address is IP address of management module >> of IBM blade center, login/password represent credentials to access >> the same. >> >> However did not get the parameter 'Blade'. How does it play role in >> fencing? >> >> In a situation where there are two blades - Blade-1 and Blade-2 and >> if Blade-1 goes down(hardware node failure), Blade-2 should fence out >> Blade-1, in that situation fenced on Blade-2 should power off(?) >> blade-2 using fence_bladecenter, so how should below sniplet of >> cluster.conf file should look like? - >> >> >> >> >> >> >> > name="BLADECENTER"/> >> >> >> >> >> >> >> > name="BLADECENTER"/> >> >> >> >> >> >> In which situation fence_bladecenter would be used to power on the >> blade? >> >> Your gratefully >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From hal at elizium.za.net Fri Jan 7 05:39:28 2011 From: hal at elizium.za.net (Hugo Lombard) Date: Fri, 7 Jan 2011 07:39:28 +0200 Subject: [Linux-cluster] configuring bladecenter fence device In-Reply-To: References: <752752065.118339.1294330001557.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Message-ID: <20110107053928.GZ16798@squishy.elizium.za.net> On Fri, Jan 07, 2011 at 10:12:16AM +0530, Parvez Shaikh wrote: > > > > > > > > > > Here for "node1" device blade is "2". Does it mean node1 is blade[2] > from AMM perspective? So in order to fence out node1 fence_bladecenter > would turn off blade[2]? > Hi Parvez We use BladeCenters for our clusters, and I can confirm that the 'blade="2"' parameter will translate to 'blade[2]' on the AMM. IOW, the '2' is the slot number that the blade is in. Two more things that might be of help: - The user specified in the 'login' parameter under the fencedevice should be a 'Blade Administrator' for the slots in question. - If you're running with SELinux enabled, check that the 'fenced_can_network_connect' boolean is set to 'on'. Regards -- Hugo Lombard From parvez.h.shaikh at gmail.com Fri Jan 7 07:14:19 2011 From: parvez.h.shaikh at gmail.com (Parvez Shaikh) Date: Fri, 7 Jan 2011 12:44:19 +0530 Subject: [Linux-cluster] configuring bladecenter fence device In-Reply-To: <20110107053928.GZ16798@squishy.elizium.za.net> References: <752752065.118339.1294330001557.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> <20110107053928.GZ16798@squishy.elizium.za.net> Message-ID: Thanks Hugo Your gratefully On Fri, Jan 7, 2011 at 11:09 AM, Hugo Lombard wrote: > On Fri, Jan 07, 2011 at 10:12:16AM +0530, Parvez Shaikh wrote: >> >> ? ? ? ? ? ? ? ? >> ? ? ? ? ? ? ? ? ? ? ? ? >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? >> ? ? ? ? ? ? ? ? ? ? ? ? >> ? ? ? ? ? ? ? ? >> >> Here for "node1" device blade is "2". Does it mean node1 is blade[2] >> from AMM perspective? So in order to fence out node1 fence_bladecenter >> would turn off blade[2]? >> > > Hi Parvez > > We use BladeCenters for our clusters, and I can confirm that the > 'blade="2"' parameter will translate to 'blade[2]' on the AMM. ?IOW, the > '2' is the slot number that the blade is in. > > Two more things that might be of help: > > - The user specified in the 'login' parameter under the fencedevice > ?should be a 'Blade Administrator' for the slots in question. > > - If you're running with SELinux enabled, check that the > ?'fenced_can_network_connect' boolean is set to 'on'. > > Regards > > -- > Hugo Lombard > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > From hal at elizium.za.net Fri Jan 7 10:27:41 2011 From: hal at elizium.za.net (Hugo Lombard) Date: Fri, 7 Jan 2011 12:27:41 +0200 Subject: [Linux-cluster] What is the current recommendation concerning shutting down a cluster node? In-Reply-To: References: Message-ID: <20110107102741.GA16798@squishy.elizium.za.net> On Thu, Jan 06, 2011 at 06:00:30PM +0000, Hofmeister, James (WTEC Linux) wrote: > > In the RH436 class it was verbally discussed during the fencing > chapter that it was a bad thing to shutdown a node, but to instead > power down the machine which has been implemented in many of the > fence_ scripts. > FWIW, I suspect the context of that comment is different from that of your question. In the context of fencing, the cluster has already decided that the box being fenced is in a bad state, and should go down as quickly as possible. Seeing that the box is being fenced, it likely means that it's not communicating with the cluster, so when an orderly shutdown tries and stop the cluster and associated services on the node that's being fenced, it will hang, and the shutdown won't continue, so the fencing won't be successful, and so your cluster services like GFS will remain hung. Better to power kill the node so that the cluster can continue as soon as possible. -- Hugo Lombard From rossnick-lists at cybercat.ca Fri Jan 7 14:14:12 2011 From: rossnick-lists at cybercat.ca (Nicolas Ross) Date: Fri, 7 Jan 2011 09:14:12 -0500 Subject: [Linux-cluster] What is the current recommendation concerning shutting down a cluster node? References: <4D260F29.2070706@alteeve.com> Message-ID: <39D54A755A74441FA5FEDCF5AC9EC4A1@versa> > On 01/06/2011 01:00 PM, Hofmeister, James (WTEC Linux) wrote: >> umount gfs >> service rgmanager stop >> service gfs stop >> service clvmd stop >> service cman stop > > This is my method. However, note that stopping GFS unmounts the volumes, > so you can skip the manual unmount. Would a regular shutdown (-r/-h) now do the exact same thing ? From linux at alteeve.com Fri Jan 7 15:23:36 2011 From: linux at alteeve.com (Digimer) Date: Fri, 07 Jan 2011 10:23:36 -0500 Subject: [Linux-cluster] What is the current recommendation concerning shutting down a cluster node? In-Reply-To: <39D54A755A74441FA5FEDCF5AC9EC4A1@versa> References: <4D260F29.2070706@alteeve.com> <39D54A755A74441FA5FEDCF5AC9EC4A1@versa> Message-ID: <4D272FF8.7080002@alteeve.com> On 01/07/2011 09:14 AM, Nicolas Ross wrote: >> On 01/06/2011 01:00 PM, Hofmeister, James (WTEC Linux) wrote: >>> umount gfs >>> service rgmanager stop >>> service gfs stop >>> service clvmd stop >>> service cman stop >> >> This is my method. However, note that stopping GFS unmounts the volumes, >> so you can skip the manual unmount. > > Would a regular shutdown (-r/-h) now do the exact same thing ? Technically it should, I suppose, but I've had it not work enough times that I've now created a little 'stop_cluster.sh' and 'start_cluster.sh' scripts that I run. -- Digimer E-Mail: digimer at alteeve.com AN!Whitepapers: http://alteeve.com Node Assassin: http://nodeassassin.org From parvez.h.shaikh at gmail.com Mon Jan 10 08:51:14 2011 From: parvez.h.shaikh at gmail.com (Parvez Shaikh) Date: Mon, 10 Jan 2011 14:21:14 +0530 Subject: [Linux-cluster] Error while manual fencing and output of clustat Message-ID: Dear experts, I have two node cluster(node1 and node2), and manual fencing is configured. Service S2 is running on node2. To ensure failover happen, I shutdown node2.. I see following messages in /var/log/messages - agent "fence_manual" reports: failed: fence_manual no node name fence_ack_manual -n node2 doesn't work saying there is no FIFO in /tmp. fence_ack_manual -n node2 -e do work and then service S2 fails over to node2. Trying to find out why fence_manual is reporting error? node2 is pingable hostname and its entry is in /etc/hosts of node1 (and vice versa). I also see that after failover when I do "clustat -x" I get cluster status (in XML format) with - I was expecting last_owner would correspond to node2(because this is node which was running service S and has failed); which would indicate that service is failing over FROM node2. Is there a way that node in cluster (a node on which service is failing over) could determine from which node the given service is failing over? Any inputs would be greatly appreciated. Thanks Yours gratefully From hostmaster at inwx.de Mon Jan 10 09:05:38 2011 From: hostmaster at inwx.de (InterNetworX | Hostmaster) Date: Mon, 10 Jan 2011 10:05:38 +0100 Subject: [Linux-cluster] Virtualization software with GFS2 Message-ID: <4D2ACBE2.9090202@inwx.de> Hello, what virtualization software are you using with GFS2? KVM, XEN, OpenVZ? I think we are the first one who tries to use OpenVZ. We have many lock problems. We can not recommend to use. Regards, Mario From ableisch at redhat.com Mon Jan 10 10:24:59 2011 From: ableisch at redhat.com (Andreas Bleischwitz) Date: Mon, 10 Jan 2011 11:24:59 +0100 Subject: [Linux-cluster] Howto define two-node cluster in enterprise environment Message-ID: <4D2ADE7B.3010500@redhat.com> Hello list, I recently ran into some questions regarding a two-node cluster in an enterprise environment, where single-point-of-failures were tried to be eliminated whenever possible. The situation is the following: Two-node cluster, SAN-based shared storage - multipathed; host-based mirrored, bonded NICS, Quorum device as tie-breaker. Problem: The quorum device is the single-point-of-failure as the SAN-device could fail and hence the quorum-disc wouldn't be accessible. The quorum-disc can't be host-based mirrored, as this would require cmirror - which depends on a quorate cluster. One solution: use storage-based mirroring - with extra costs, limited to no support with mixed storage vendors. Another solution: Use a third - no service - node which has to have the same SAN-connections as the other two nodes out of cluster reasons. This node will idle most of the time and therefore be very uneconomic. How are such situations usually solved using RHCS? There must be a way of configuring a two-nodecluster without having a SPOF defined. HP had a quorum-host with their no longer maintained Service Guard, which could do quorum for more than on cluster at once. Any suggestions appreciated. Best regrads, Andreas Bleischwitz From gcharles at ups.com Mon Jan 10 12:38:25 2011 From: gcharles at ups.com (gcharles at ups.com) Date: Mon, 10 Jan 2011 07:38:25 -0500 Subject: [Linux-cluster] Howto define two-node cluster in enterprise environment In-Reply-To: <4D2ADE7B.3010500@redhat.com> References: <4D2ADE7B.3010500@redhat.com> Message-ID: <49CCA172B74C1B4D916CB9B71FB952DA27D409CB73@njrarsvr3bef.us.ups.com> While a third idle node in the cluster is a way to regulate the quorum votes, you're right in that it's not very economical. A way to keep the quorum device from being an SPOF is to assure it is multipathed as well. However, by default, the quorum code does not define the device from its multipathed name. Instead, it defaults to the dm-# which we've proven in the past does not retain its name through reboots or rescans. What you need to do is get the disk ID number of the shared quorum disk itself, and create an alias name for it in multipath.conf: ... multipaths { multipath { wwid 36006016338602300ca974b4b1b7edf11 alias qdisk } ... ...then define it in cluster.conf with a device="/dev/mapper/qdisk" in the quorumd stanza. When you enter a clustat, your qdisk should show up like this: /dev/mapper/qdisk 0 Online, Quorum Disk You can test this by disconnecting one of your SAN connections, and watch the cluster log. It will show a loss of communication with the quorum disk for a few seconds and then return to normal. Regards; Greg Charles -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Andreas Bleischwitz Sent: Monday, January 10, 2011 5:25 AM To: linux-cluster at redhat.com Subject: [Linux-cluster] Howto define two-node cluster in enterprise environment Hello list, I recently ran into some questions regarding a two-node cluster in an enterprise environment, where single-point-of-failures were tried to be eliminated whenever possible. The situation is the following: Two-node cluster, SAN-based shared storage - multipathed; host-based mirrored, bonded NICS, Quorum device as tie-breaker. Problem: The quorum device is the single-point-of-failure as the SAN-device could fail and hence the quorum-disc wouldn't be accessible. The quorum-disc can't be host-based mirrored, as this would require cmirror - which depends on a quorate cluster. One solution: use storage-based mirroring - with extra costs, limited to no support with mixed storage vendors. Another solution: Use a third - no service - node which has to have the same SAN-connections as the other two nodes out of cluster reasons. This node will idle most of the time and therefore be very uneconomic. How are such situations usually solved using RHCS? There must be a way of configuring a two-nodecluster without having a SPOF defined. HP had a quorum-host with their no longer maintained Service Guard, which could do quorum for more than on cluster at once. Any suggestions appreciated. Best regrads, Andreas Bleischwitz -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From hostmaster at inwx.de Mon Jan 10 12:48:05 2011 From: hostmaster at inwx.de (InterNetworX | Hostmaster) Date: Mon, 10 Jan 2011 13:48:05 +0100 Subject: [Linux-cluster] waiting for glock: pid does not exists Message-ID: <4D2B0005.1050807@inwx.de> Hello, we are trying to run OpenVZ on a GFS2. We copied a virtual machine to the GFS2 storage (on node1) and added the service to cluster.conf. After reloading the configuration on all nodes, rgmanager was trying to start the virtual machine on node3. That is not working and now the machine is hanging with a lock. This is the result of the gfs2 hang analyzer: There are 4 glocks with waiters. node1, pid 2674 is waiting for glock 3/8389396, which is held by pid 6821 node3, pid 7024 is waiting for glock 3/8389396, which is held by pid 6821 node1, pid 10188 is waiting for glock 2/1857345, which is held by pid 6821 node3, pid 6772 is waiting for glock 2/1857345, which is held by pid 6821 node3, pid 7251 is waiting for glock 2/1857345, which is held by pid 6821 node3, pid 7289 is waiting for glock 2/1857345, which is held by pid 6821 carl, pid 23817 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 4243 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 7055 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 7090 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 7129 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 7176 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 7230 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 7270 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 7306 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 7345 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 7369 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 7402 is waiting for glock 2/394135, which is held by pid 7024 node3, pid 6821 is waiting for glock 5/8425127, which is held by pid 7258 The pid 6821 is still running on node3: root 6821 0.0 0.0 12216 696 ? D< 08:29 0:00 /bin/cp -fp /etc/hosts /etc/hosts.12 The problem pid is 7258 - but I can not find this process running on any node. Any idea what is the problem here? Mario From ableisch at redhat.com Mon Jan 10 12:54:41 2011 From: ableisch at redhat.com (Andreas Bleischwitz) Date: Mon, 10 Jan 2011 13:54:41 +0100 Subject: [Linux-cluster] Howto define two-node cluster in enterprise environment In-Reply-To: <49CCA172B74C1B4D916CB9B71FB952DA27D409CB73@njrarsvr3bef.us.ups.com> References: <4D2ADE7B.3010500@redhat.com> <49CCA172B74C1B4D916CB9B71FB952DA27D409CB73@njrarsvr3bef.us.ups.com> Message-ID: <4D2B0191.10607@redhat.com> Hello Greg, On 01/10/2011 01:38 PM, gcharles at ups.com wrote: > While a third idle node in the cluster is a way to regulate the quorum votes, you're right in that it's not very economical. > > A way to keep the quorum device from being an SPOF is to assure it is multipathed as well. However, by default, the quorum code does not define the device from its multipathed name. Instead, it defaults to the dm-# which we've proven in the past does not retain its name through reboots or rescans. What you need to do is get the disk ID number of the shared quorum disk itself, and create an alias name for it in multipath.conf: > ... > multipaths { > multipath { > wwid 36006016338602300ca974b4b1b7edf11 > alias qdisk > } > ... This will not eleminate the SPOF using one ONE storage... What I meant is using at least two storage devices in different locations i.e. DCs. I'll assume serious SAN connections are always using multipath ;) And by the way, I would prefer using the qdisc-label - which should be unique and is scanned during start of qdiscd. > > ...then define it in cluster.conf with a device="/dev/mapper/qdisk" in the quorumd stanza. When you enter a clustat, your qdisk should show up like this: > > /dev/mapper/qdisk 0 Online, Quorum Disk > > You can test this by disconnecting one of your SAN connections, and watch the cluster log. It will show a loss of communication with the quorum disk for a few seconds and then return to normal. ... and qdisc will fail if the underlaying SAN device got lost - f.e. power failure in one DC. As already said, SAN-storage will be mirrored to two DCs using cmirorror, but this is not possible for the qdisc. Thanks for your answer, anyways. > > > Regards; > Greg Charles > > -----Original Message----- > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Andreas Bleischwitz > Sent: Monday, January 10, 2011 5:25 AM > To: linux-cluster at redhat.com > Subject: [Linux-cluster] Howto define two-node cluster in enterprise environment > > Hello list, > > I recently ran into some questions regarding a two-node cluster in an enterprise environment, where single-point-of-failures were tried to be eliminated whenever possible. > > The situation is the following: > Two-node cluster, SAN-based shared storage - multipathed; host-based mirrored, bonded NICS, Quorum device as tie-breaker. > > Problem: > The quorum device is the single-point-of-failure as the SAN-device could fail and hence the quorum-disc wouldn't be accessible. > The quorum-disc can't be host-based mirrored, as this would require cmirror - which depends on a quorate cluster. > One solution: use storage-based mirroring - with extra costs, limited to no support with mixed storage vendors. > Another solution: Use a third - no service - node which has to have the same SAN-connections as the other two nodes out of cluster reasons. This node will idle most of the time and therefore be very uneconomic. > > How are such situations usually solved using RHCS? There must be a way of configuring a two-nodecluster without having a SPOF defined. > > HP had a quorum-host with their no longer maintained Service Guard, which could do quorum for more than on cluster at once. > > Any suggestions appreciated. > > Best regrads, > > Andreas Bleischwitz > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From xavier.montagutelli at unilim.fr Mon Jan 10 13:28:19 2011 From: xavier.montagutelli at unilim.fr (Xavier Montagutelli) Date: Mon, 10 Jan 2011 14:28:19 +0100 Subject: [Linux-cluster] Error while manual fencing and output of clustat In-Reply-To: References: Message-ID: <201101101428.19680.xavier.montagutelli@unilim.fr> Hello Parvez, On Monday 10 January 2011 09:51:14 Parvez Shaikh wrote: > Dear experts, > > I have two node cluster(node1 and node2), and manual fencing is > configured. Service S2 is running on node2. To ensure failover happen, > I shutdown node2.. I see following messages in /var/log/messages - > > agent "fence_manual" reports: failed: fence_manual > no node name I am not an expert, but could you show us your cluster.conf file ? You need to give a "nodename" attribute to the fence_manual agent somewhere, the error message makes me think it's missing. For example : ... > > fence_ack_manual -n node2 doesn't work saying there is no FIFO in > /tmp. fence_ack_manual -n node2 -e do work and then service S2 fails > over to node2. > > Trying to find out why fence_manual is reporting error? node2 is > pingable hostname and its entry is in /etc/hosts of node1 (and vice > versa). I also see that after failover when I do "clustat -x" I get > cluster status (in XML format) with - > > > > > flags_str="" owner="node1" last_owner="node1" restarts="0" > last_transition="1294676678" last_transition_str="xxxxxxxxxx"/> > > > > I was expecting last_owner would correspond to node2(because this is > node which was running service S and has failed); which would indicate > that service is failing over FROM node2. Is there a way that node in > cluster (a node on which service is failing over) could determine from > which node the given service is failing over? > > Any inputs would be greatly appreciated. > > Thanks > > Yours gratefully > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Xavier Montagutelli Tel : +33 (0)5 55 45 77 20 Service Commun Informatique Fax : +33 (0)5 55 45 75 95 Universite de Limoges 123, avenue Albert Thomas 87060 Limoges cedex From swhiteho at redhat.com Mon Jan 10 14:07:46 2011 From: swhiteho at redhat.com (Steven Whitehouse) Date: Mon, 10 Jan 2011 14:07:46 +0000 Subject: [Linux-cluster] waiting for glock: pid does not exists In-Reply-To: <4D2B0005.1050807@inwx.de> References: <4D2B0005.1050807@inwx.de> Message-ID: <1294668466.2450.59.camel@dolmen> Hi, On Mon, 2011-01-10 at 13:48 +0100, InterNetworX | Hostmaster wrote: > Hello, > > we are trying to run OpenVZ on a GFS2. We copied a virtual machine to > the GFS2 storage (on node1) and added the service to cluster.conf. After > reloading the configuration on all nodes, rgmanager was trying to start > the virtual machine on node3. That is not working and now the machine is > hanging with a lock. > > This is the result of the gfs2 hang analyzer: > > There are 4 glocks with waiters. > node1, pid 2674 is waiting for glock 3/8389396, which is held by pid 6821 > node3, pid 7024 is waiting for glock 3/8389396, which is held by pid 6821 > > > node1, pid 10188 is waiting for glock 2/1857345, which is held by pid 6821 > node3, pid 6772 is waiting for glock 2/1857345, which is held by pid 6821 > node3, pid 7251 is waiting for glock 2/1857345, which is held by pid 6821 > node3, pid 7289 is waiting for glock 2/1857345, which is held by pid 6821 > > > carl, pid 23817 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 4243 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 7055 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 7090 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 7129 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 7176 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 7230 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 7270 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 7306 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 7345 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 7369 is waiting for glock 2/394135, which is held by pid 7024 > node3, pid 7402 is waiting for glock 2/394135, which is held by pid 7024 > > > node3, pid 6821 is waiting for glock 5/8425127, which is held by pid 7258 > > > > The pid 6821 is still running on node3: > > root 6821 0.0 0.0 12216 696 ? D< 08:29 0:00 /bin/cp > -fp /etc/hosts /etc/hosts.12 > > The problem pid is 7258 - but I can not find this process running on any > node. Any idea what is the problem here? > > Mario > If pid 7528 has exited, then it is almost certainly not a problem. What makes you think that this is the issue? Since it is a type 5 glock, it should not be blocking access to anything, Steve. > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From kitgerrits at gmail.com Mon Jan 10 20:17:43 2011 From: kitgerrits at gmail.com (Kit Gerrits) Date: Mon, 10 Jan 2011 21:17:43 +0100 Subject: [Linux-cluster] What is the current recommendation concerning shutting down a cluster node? In-Reply-To: <4D272FF8.7080002@alteeve.com> Message-ID: <4d2b6967.0607cc0a.7087.ffff8fa2@mx.google.com> Hello, I have found the same behaviour. Shutting down a cluster node without first stopping the cluster services results in a node that has not logged out from the cluster. The cluster may even fence the node and bring it back on-line. (it won't stay dead!) You can also leave the cluster with 'cman_tool leave': http://linux.die.net/man/8/cman_tool Tells CMAN to leave the cluster. You cannot do this if there are subsystems (eg DLM, GFS) active. You should dismount all GFS filesystems, shutdown CLVM, fenced and anything else using the cluster manager before using cman_tool leave. Look at 'cman_tool status|services' to see how many (and which) services are running. When a node leaves the cluster, the remaining nodes recalculate quorum and this may block cluster activity if the required number of votes is not present. If this node is to be down for an extended period of time and you need to keep the cluster running, add the remove option, and the remaining nodes will recalculate quorum such that activity can continue. Regards, Kit -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Digimer Sent: vrijdag 7 januari 2011 16:24 To: linux clustering Subject: Re: [Linux-cluster] What is the current recommendation concerning shutting down a cluster node? On 01/07/2011 09:14 AM, Nicolas Ross wrote: >> On 01/06/2011 01:00 PM, Hofmeister, James (WTEC Linux) wrote: >>> umount gfs >>> service rgmanager stop >>> service gfs stop >>> service clvmd stop >>> service cman stop >> >> This is my method. However, note that stopping GFS unmounts the >> volumes, so you can skip the manual unmount. > > Would a regular shutdown (-r/-h) now do the exact same thing ? Technically it should, I suppose, but I've had it not work enough times that I've now created a little 'stop_cluster.sh' and 'start_cluster.sh' scripts that I run. -- Digimer E-Mail: digimer at alteeve.com AN!Whitepapers: http://alteeve.com Node Assassin: http://nodeassassin.org -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From kitgerrits at gmail.com Mon Jan 10 20:21:45 2011 From: kitgerrits at gmail.com (Kit Gerrits) Date: Mon, 10 Jan 2011 21:21:45 +0100 Subject: [Linux-cluster] Howto define two-node cluster in enterpriseenvironment In-Reply-To: <4D2ADE7B.3010500@redhat.com> Message-ID: <4d2b6a59.0607cc0a.7080.ffff8f43@mx.google.com> Hello fellow administrator, If you have a SAN... Why can't you have the SAN publish the same LUN to the two cluster nodes simultaneously? It is only used as a raw device, so there should be no ugly filesystem side-effects. Regards, Kit -----Original Message----- From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Andreas Bleischwitz Sent: maandag 10 januari 2011 11:25 To: linux-cluster at redhat.com Subject: [Linux-cluster] Howto define two-node cluster in enterpriseenvironment Hello list, I recently ran into some questions regarding a two-node cluster in an enterprise environment, where single-point-of-failures were tried to be eliminated whenever possible. The situation is the following: Two-node cluster, SAN-based shared storage - multipathed; host-based mirrored, bonded NICS, Quorum device as tie-breaker. Problem: The quorum device is the single-point-of-failure as the SAN-device could fail and hence the quorum-disc wouldn't be accessible. The quorum-disc can't be host-based mirrored, as this would require cmirror - which depends on a quorate cluster. One solution: use storage-based mirroring - with extra costs, limited to no support with mixed storage vendors. Another solution: Use a third - no service - node which has to have the same SAN-connections as the other two nodes out of cluster reasons. This node will idle most of the time and therefore be very uneconomic. How are such situations usually solved using RHCS? There must be a way of configuring a two-nodecluster without having a SPOF defined. HP had a quorum-host with their no longer maintained Service Guard, which could do quorum for more than on cluster at once. Any suggestions appreciated. Best regrads, Andreas Bleischwitz -- Linux-cluster mailing list Linux-cluster at redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster From thomas at sjolshagen.net Mon Jan 10 20:38:12 2011 From: thomas at sjolshagen.net (Thomas Sjolshagen) Date: Mon, 10 Jan 2011 15:38:12 -0500 Subject: [Linux-cluster] Howto define two-node cluster in enterpriseenvironment In-Reply-To: <4d2b6a59.0607cc0a.7080.ffff8f43@mx.google.com> References: <4d2b6a59.0607cc0a.7080.ffff8f43@mx.google.com> Message-ID: <4a614ae3080a5ce8f908a9b877081986@sjolshagen.net> On Mon, 10 Jan 2011 21:21:45 +0100, "Kit Gerrits" wrote: > Hello fellow administrator, > > If you have a SAN... > Why can't you have the SAN publish the same LUN to the two cluster > nodes > simultaneously? You can, but you minimally need to guarantee (not believe or think, but guarantee!) that both nodes do not a) write to the same sectors, file systems or LVM volumes at the same time (this is actually a whole lot more difficult to do than most people think) - including boot sectors, partition tables, LVM metadata, etc, etc, b) think they're exclusively accessing the LUN I.e. there must be something on the nodes - an application, OS tool or something else - that understands that there is more than one reader & writer to a LUN and thus synchronizes this. > It is only used as a raw device, so there should be no ugly > filesystem > side-effects. File systems only serve to make this a lot more obvious to the end user or administrator since it's integrity tends to get shot fairly quickly and there are integrity checks in place. On raw devices, you get the "benefit" of ignorance about the fact that your data is corrupt, unless b) above is true. Hth, // Thomas > > > Regards, > > Kit > > -----Original Message----- > From: linux-cluster-bounces at redhat.com > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Andreas > Bleischwitz > Sent: maandag 10 januari 2011 11:25 > To: linux-cluster at redhat.com > Subject: [Linux-cluster] Howto define two-node cluster in > enterpriseenvironment > > Hello list, > > I recently ran into some questions regarding a two-node cluster in an > enterprise environment, where single-point-of-failures were tried to > be > eliminated whenever possible. > > The situation is the following: > Two-node cluster, SAN-based shared storage - multipathed; host-based > mirrored, bonded NICS, Quorum device as tie-breaker. > > Problem: > The quorum device is the single-point-of-failure as the SAN-device > could > fail and hence the quorum-disc wouldn't be accessible. > The quorum-disc can't be host-based mirrored, as this would require > cmirror > - which depends on a quorate cluster. > One solution: use storage-based mirroring - with extra costs, limited > to no > support with mixed storage vendors. > Another solution: Use a third - no service - node which has to have > the same > SAN-connections as the other two nodes out of cluster reasons. This > node > will idle most of the time and therefore be very uneconomic. > > How are such situations usually solved using RHCS? There must be a > way of > configuring a two-nodecluster without having a SPOF defined. > > HP had a quorum-host with their no longer maintained Service Guard, > which > could do quorum for more than on cluster at once. > > Any suggestions appreciated. > > Best regrads, > > Andreas Bleischwitz > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster From carlopmart at gmail.com Mon Jan 10 21:16:51 2011 From: carlopmart at gmail.com (carlopmart) Date: Mon, 10 Jan 2011 22:16:51 +0100 Subject: [Linux-cluster] Problems with a script when is launched via rgmanager Message-ID: <4D2B7743.2060800@gmail.com> Hi all, I am trying to set up a splunk cluster service on two RHEL5.5 hosts (fully updated). My problems becomes when I trying to setup this service under rgmanager: script ever fails. If I launch the script manually, all works as expected. If I test the service using rg_test comand, all works ok as expected. This is the error when rgmanager tries to launch the service: Jan 10 17:50:55 lorien clurgmgrd[25394]: Starting disabled service service:siemmgmt-svc Jan 10 17:50:55 lorien clurgmgrd: [25394]: Link for eth0: Detected Jan 10 17:50:55 lorien clurgmgrd: [25394]: Adding IPv4 address 172.25.70.22/28 to eth0 Jan 10 17:50:55 lorien clurgmgrd: [25394]: Pinging addr 172.25.70.22 from dev eth0 Jan 10 17:50:57 lorien clurgmgrd: [25394]: Sending gratuitous ARP: 172.25.70.22 00:50:56:14:5a:1e brd ff:ff:ff:ff:ff:ff Jan 10 17:50:58 lorien clurgmgrd: [25394]: Unknown file system type 'ext4' for device /dev/inasvol/splunkvol. Assuming fsck is required. Jan 10 17:50:58 lorien clurgmgrd: [25394]: Running fsck on /dev/inasvol/splunkvol Jan 10 17:50:58 lorien clurgmgrd: [25394]: mounting /dev/inasvol/splunkvol on /data/services/siem/splunk Jan 10 17:50:58 lorien clurgmgrd: [25394]: mount -t ext4 -o rw /dev/inasvol/splunkvol /data/services/siem/splunk Jan 10 17:50:58 lorien clurgmgrd: [25394]: Executing /data/config/etc/init.d/splunk-cluster start Jan 10 17:50:58 lorien clurgmgrd: [25394]: script:splunk-cluster: start of /data/config/etc/init.d/splunk-cluster failed (returned 1) Jan 10 17:50:58 lorien clurgmgrd[25394]: start on script "splunk-cluster" returned 1 (generic error) Jan 10 17:50:58 lorien clurgmgrd[25394]: #68: Failed to start service:siemmgmt-svc; return value: 1 Jan 10 17:50:58 lorien clurgmgrd[25394]: Stopping failed service service:siemmgmt-svc Jan 10 17:50:58 lorien clurgmgrd[25394]: Stopping service service:siemmgmt-svc Jan 10 17:50:58 lorien clurgmgrd: [25394]: Executing /data/config/etc/init.d/splunk-cluster stop Jan 10 17:50:58 lorien clurgmgrd: [25394]: script:splunk-cluster: stop of /data/config/etc/init.d/splunk-cluster failed (returned 1) Jan 10 17:50:58 lorien clurgmgrd[25394]: stop on script "splunk-cluster" returned 1 (generic error) Jan 10 17:50:59 lorien clurgmgrd: [25394]: unmounting /data/services/siem/splunk Jan 10 17:50:59 lorien clurgmgrd: [25394]: Removing IPv4 address 172.25.70.22/28 from eth0 Jan 10 17:51:09 lorien clurgmgrd[25394]: #12: RG service:siemmgmt-svc failed to stop; intervention required Jan 10 17:51:09 lorien clurgmgrd[25394]: Service service:siemmgmt-svc is failed Jan 10 17:51:09 lorien clurgmgrd[25394]: #13: Service service:siemmgmt-svc failed to stop cleanly Jan 10 17:51:09 lorien clurgmgrd[25394]: Handling failure request for RG service:siemmgmt-svc Jan 10 17:51:19 lorien clurgmgrd[25394]: 2 events processed And this is the output using rg_test command: [root at lorien ~]# rg_test test /etc/cluster/cluster.conf start service siemmgmt-svc Running in test mode. Starting siemmgmt-svc... Unknown file system type 'ext4' for device /dev/inasvol/splunkvol. Assuming fsck is required. Running fsck on /dev/inasvol/splunkvol mounting /dev/inasvol/splunkvol on /data/services/siem/splunk mount -t ext4 -o rw /dev/inasvol/splunkvol /data/services/siem/splunk Link for eth0: Detected Adding IPv4 address 172.25.70.22/28 to eth0 Pinging addr 172.25.70.22 from dev eth0 Sending gratuitous ARP: 172.25.70.22 00:50:56:14:5a:1e brd ff:ff:ff:ff:ff:ff Executing /data/config/etc/init.d/splunk-cluster start + . /etc/init.d/functions ++ TEXTDOMAIN=initscripts ++ umask 022 ++ PATH=/sbin:/usr/sbin:/bin:/usr/bin ++ export PATH ++ '[' -z '' ']' ++ COLUMNS=80 ++ '[' -z '' ']' +++ /sbin/consoletype ++ CONSOLETYPE=pty ++ '[' -f /etc/sysconfig/i18n -a -z '' ']' ++ . /etc/profile.d/lang.sh +++ sourced=0 +++ for langfile in /etc/sysconfig/i18n '$HOME/.i18n' +++ '[' -f /etc/sysconfig/i18n ']' +++ . /etc/sysconfig/i18n ++++ LANG=en_US.UTF-8 ++++ SYSFONT=latarcyrheb-sun16 +++ sourced=1 +++ for langfile in /etc/sysconfig/i18n '$HOME/.i18n' +++ '[' -f /.i18n ']' +++ '[' -n '' ']' +++ '[' 1 = 1 ']' +++ '[' -n en_US.UTF-8 ']' +++ export LANG +++ '[' -n '' ']' +++ unset LC_ADDRESS +++ '[' -n '' ']' +++ unset LC_CTYPE +++ '[' -n '' ']' +++ unset LC_COLLATE +++ '[' -n '' ']' +++ unset LC_IDENTIFICATION +++ '[' -n '' ']' +++ unset LC_MEASUREMENT +++ '[' -n '' ']' +++ unset LC_MESSAGES +++ '[' -n '' ']' +++ unset LC_MONETARY +++ '[' -n '' ']' +++ unset LC_NAME +++ '[' -n '' ']' +++ unset LC_NUMERIC +++ '[' -n '' ']' +++ unset LC_PAPER +++ '[' -n '' ']' +++ unset LC_TELEPHONE +++ '[' -n '' ']' +++ unset LC_TIME +++ '[' -n C ']' +++ '[' C '!=' en_US.UTF-8 ']' +++ export LC_ALL +++ '[' -n '' ']' +++ unset LANGUAGE +++ '[' -n '' ']' +++ unset LINGUAS +++ '[' -n '' ']' +++ unset _XKB_CHARSET +++ consoletype=pty +++ '[' -z pty ']' +++ '[' -n '' ']' +++ '[' -n '' ']' +++ '[' -n en_US.UTF-8 ']' +++ case $LANG in +++ '[' dumb = linux ']' +++ unset SYSFONTACM SYSFONT +++ unset sourced +++ unset langfile ++ '[' -z '' ']' ++ '[' -f /etc/sysconfig/init ']' ++ . /etc/sysconfig/init +++ BOOTUP=color +++ GRAPHICAL=yes +++ RES_COL=60 +++ MOVE_TO_COL='echo -en \033[60G' +++ SETCOLOR_SUCCESS='echo -en \033[0;32m' +++ SETCOLOR_FAILURE='echo -en \033[0;31m' +++ SETCOLOR_WARNING='echo -en \033[0;33m' +++ SETCOLOR_NORMAL='echo -en \033[0;39m' +++ LOGLEVEL=3 +++ PROMPT=yes +++ AUTOSWAP=no ++ '[' pty = serial ']' ++ '[' color '!=' verbose ']' ++ INITLOG_ARGS=-q ++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d' + '[' '!' -d /data/services/siem/splunk/etc ']' + HOME=/data/services/siem/splunk + DIRECTORY=/data/services/siem/splunk + export HOME + case "$1" in + start + echo -n 'Starting Splunk: ' Starting Splunk: + sudo -H -u splunk /data/services/siem/splunk/bin/splunk start Splunk> The IT Search Engine. Checking prerequisites... Checking http port [172.25.70.22:9000]: open Checking mgmt port [172.25.70.22:9089]: open Checking configuration... Done. Checking index directory... Done. Checking databases... Validated databases: _audit, _blocksignature, _internal, _thefishbucket, history, main, sample, summary Checking for SELinux. All preliminary checks passed. [ OK ] [ OK ] Starting splunk server daemon (splunkd)... Done.Starting splunkweb... Done. If you get stuck, we're here to help. Look for answers here: http://www.splunk.com/base/Documentation The Splunk web interface is at https://172.25.70.22:9000 + RETVAL=0 + '[' 0 -eq 0 ']' + success + '[' color '!=' verbose -a -z '' ']' + echo_success + '[' color = color ']' + echo -en '\033[60G' + echo -n '[' [+ '[' color = color ']' + echo -en '\033[0;32m' + echo -n ' OK ' OK + '[' color = color ']' + echo -en '\033[0;39m' + echo -n ']' ]+ echo -ne '\r' + return 0 + return 0 + echo + return 0 + exit 0 Start of siemmgmt-svc complete As you can see, all works ok. Service configuration under cluster.conf: