From anoop_rajkumar at merck.com  Thu Jul  1 14:11:36 2010
From: anoop_rajkumar at merck.com (Rajkumar, Anoop)
Date: Thu, 1 Jul 2010 10:11:36 -0400
Subject: [Linux-cluster] RHEL Cluster node fencing and cluster
Message-ID: <C651C3AA2A6A1D4980D35451DDE3F96B721520@usctmx1160.merck.com>

Hi

Gfs file system I have created is not being used for services running on
tis cluster. I was planning to use it for a database but I was getting
into stale cluster problem as soon as file system was created. After
changing to gfs instead of gfs2 everything is fine.

Thanks
Anoop


Message: 2
Date: Tue, 29 Jun 2010 13:16:42 +0300
From: jacob ishak <jacob.ishak at gmail.com>
To: linux clustering <linux-cluster at redhat.com>
Subject: Re: [Linux-cluster] RHEL Cluster node fencing and cluster
Message-ID:
	<AANLkTik2y7_oRHENLq-Z6shLwcV-AsBtB6zsENHm2LQh at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

in your cluster.conf

fstype="ext3"

it should be fstype="gfs" or gfs2

BR

On Mon, Jun 28, 2010 at 8:42 PM, Rajkumar, Anoop
<anoop_rajkumar at merck.com>wrote:

>  Hi
>
> I am not getting into the problem now of cluster getting staled after
I
> create gfs file system instaed of gfs2. Here is my cluster.conf file.
>
> [root at system1 cluster]# more cluster.conf
> <?xml version="1.0"?>
> <cluster alias="cluster1" config_version="33" name="cluster1">
>         <fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="100"/>
>         <clusternodes>
>                 <clusternode name="system1.merck.com" nodeid="1"
> votes="1">
>                         <fence>
>                                 <method name="1">
>                                         <device name="system1r"/>
>                                 </method>
>                         </fence>
>                 </clusternode>
>                 <clusternode name="system2.merck.com" nodeid="2"
> votes="1">
>                         <fence>
>                                 <method name="1">
>                                         <device name="system2r"/>
>                                 </method>
>                         </fence>
>                 </clusternode>
>         </clusternodes>
>         <cman expected_votes="1" two_node="1"/>
>         <fencedevices>
>                 <fencedevice agent="fence_ilo" hostname="
> system1r.merck.com" login="admin
> " name="system1r" passwd="Anwyccdfy57"/>
>                 <fencedevice agent="fence_ilo" hostname="
> system2r.merck.com" login="admin
> " name="system1r" passwd="Anwyccdfy57"/>
>         </fencedevices>
>         <rm>
>                 <failoverdomains>
>                         <failoverdomain name="webdomain"
nofailback="0"
> ordered="1" restricte
> d="1">
>                                 <failoverdomainnode name="
> system1.merck.com" priority="
> 1"/>
>                                 <failoverdomainnode name="
> system2.merck.com" priority="
> 2"/>
>                         </failoverdomain>
>                 </failoverdomains>
>                 <resources>
>                         <ip address="54.3.xyz.abc" monitor_link="1"/>
>                         <script file="/etc/init.d/orig.httpd"
name="http
> startup script"/>
>                         <fs device="/dev/sda2" force_fsck="0"
> force_unmount="0" fsid="6443" f
> stype="ext3" mountpoint="/var/www/html" name="httpd-content"
options=""
> self_fence="0"/>
>                         <fs device="/dev/sda1" force_fsck="0"
> force_unmount="0" fsid="30579"
> fstype="ext3" mountpoint="/var/lib/mysql" name="mysql-content"
options=""
> self_fence="0"/>
>                         <script file="/etc/init.d/mysqld" name="mysql
> startup script"/>
>                         <ip address="192.168.0.3" monitor_link="1"/>
>                 </resources>
>                 <service autostart="1" domain="webdomain"
> name="http-service" recovery="resta
> rt">
>                         <script ref="http startup script"/>
>                         <fs ref="httpd-content"/>
>                         <ip ref="54.3.xyz.abc"/>
>                 </service>
>                 <service autostart="1" domain="webdomain"
exclusive="0"
> name="mysql" recovery
> ="disable">
>                         <fs ref="mysql-content"/>
>                         <script ref="mysql startup script"/>
>                         <ip ref="192.168.0.3"/>
>                 </service>
>         </rm>
> </cluster>
>
> Thanks
> Anoop

Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100701/8f01d923/attachment.htm>

From christoph at macht-blau.org  Fri Jul  2 13:59:00 2010
From: christoph at macht-blau.org (C. Handel)
Date: Fri, 2 Jul 2010 15:59:00 +0200
Subject: [Linux-cluster] postgres cluster without shared storage
	(ESGLinux)
Message-ID: <AANLkTimxnSoYGVy2DiGU3Cl9ODYgoiWtywfnUcOVQoyS@mail.gmail.com>

> I need to mount a two nodes cluster with postgres as service. I have mounted
> it in the past but with a shared storage and using GFS but now I don?t have
> this element.
>
> The idea is to have a master node with all the data in its own disk and have
> a mechanism to replicate this data to the slave node in its own disk. If the
> master goes down the slave begin to give the service and the flow of data
> will go from this node to the other one. (the slave node becomes the master
> one)
>
> Is it possible to do something like this?

To get a good HA solution it is always necessary to know all edge
cases of the clients. If you can modify the clients/application it is
much easier to get a working HA solution.

Also you need to define what is HA? Failover in seconds? minutes?
seamless? Client reconnect?

With postgresql 8.3+ (i think it started with 8.3) you can do
logshipping. That results in an active server accepting connections
and a standby. The standby does not accept connections. It follows the
original data pretty close but can lag behind (same lag goes for mysql
replication, except mysql allows readonly on the slave). You need to
script the failover. That is, detect that the master missbehaves,
shoot it (important!) and switch the standby from replication to
service. You can't have a switchback to the master. Clients need to
detect that they lost connection and reconnect. This solution is
straight forward, easy to understand and holds no surprises (so why
did it do this? How did i loose this transaction?). Failover time
depends on timeouts (how long do you wait before you decide that the
master is dead?) but can be done sub-five-minutes.

Starting with postgesql 9 you can have hot-standby that accepts
read-only connections. And the standby can upgrade the connections to
read-write without a reconnect. You still need to work your way
through the ip failover. Haven't used this (and 9 is still beta).


Greetings
   Christoph



From m at alanny.ru  Sun Jul  4 15:24:03 2010
From: m at alanny.ru (AlannY)
Date: Sun, 04 Jul 2010 19:24:03 +0400
Subject: [Linux-cluster] Cross mount of block devices on 2 nodes
Message-ID: <91a3cb66e9de92dcff1c625ca5944b7c@imap.hc.ru>

Hi there. I'm new in clustering, so I have a question.

I have 2 nodes. For example, ONE and TWO. On ONE, I can export disk block
device
(via GNBD or iSCSI) to TWO. On TWO, I can export another disk block device
and import
it in ONE.

Some, ASCII art:

  +-------------+          +-----------+
  | ONE +-------+          | TWO       |
  |     | disk1 | -------> |           |
  |     +-------+          +-------+   |
  |             | <------- | disk2 |   |
  |             |          +-------+   |
  +-------------+          +-------+---+

Is it possible? Can I append this 2 block devices in one LVM volume group?
Can I format this volume group in GFS2 (or any other cluster filesystem)?

Is there any pros and cons for this method?

Thanks for patience.

P.S. I'm not dumb, just newbie.



From jcasale at activenetwerx.com  Sun Jul  4 18:39:45 2010
From: jcasale at activenetwerx.com (Joseph L. Casale)
Date: Sun, 4 Jul 2010 18:39:45 +0000
Subject: [Linux-cluster] Cross mount of block devices on 2 nodes
In-Reply-To: <91a3cb66e9de92dcff1c625ca5944b7c@imap.hc.ru>
References: <91a3cb66e9de92dcff1c625ca5944b7c@imap.hc.ru>
Message-ID: <CA5A491E9DEFBE4CB777DE97E21575E906933891@prato.activenetwerx.local>

>Is it possible? Can I append this 2 block devices in one LVM volume group?
>Can I format this volume group in GFS2 (or any other cluster filesystem)?
>
>Is there any pros and cons for this method?

Ugh, why reinvent the wheel...
Look into DRBD, this essentially does what you want or just use the tools
RH provides to create shared storage, if that's in fact what you actually
need.

What exactly are you trying to achieve?

jlc



From kkovachev at varna.net  Sun Jul  4 18:55:36 2010
From: kkovachev at varna.net (Kaloyan Kovachev)
Date: Sun, 04 Jul 2010 21:55:36 +0300
Subject: [Linux-cluster] Cross mount of block devices on 2 nodes
In-Reply-To: <91a3cb66e9de92dcff1c625ca5944b7c@imap.hc.ru>
References: <91a3cb66e9de92dcff1c625ca5944b7c@imap.hc.ru>
Message-ID: <eefee4e2eb762abc4e72e67947f4b678@mx.varna.net>

Hi,

On Sun, 04 Jul 2010 19:24:03 +0400, AlannY <m at alanny.ru> wrote:
> Hi there. I'm new in clustering, so I have a question.
> 
> I have 2 nodes. For example, ONE and TWO. On ONE, I can export disk
block
> device
> (via GNBD or iSCSI) to TWO. On TWO, I can export another disk block
device
> and import
> it in ONE.
> 
> Some, ASCII art:
> 
>   +-------------+          +-----------+
>   | ONE +-------+          | TWO       |
>   |     | disk1 | -------> |           |
>   |     +-------+          +-------+   |
>   |             | <------- | disk2 |   |
>   |             |          +-------+   |
>   +-------------+          +-------+---+
> 
> Is it possible? Can I append this 2 block devices in one LVM volume
group?
> Can I format this volume group in GFS2 (or any other cluster
filesystem)?
> 

Yes, it is possible, but i guess you want to make a mirror between the two
... take a look at DRBD in this case

> Is there any pros and cons for this method?
> 

It depends on your usage case, but you haven't provided any info about it.

> Thanks for patience.
> 
> P.S. I'm not dumb, just newbie.
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From linux-cluster at redhat.com  Mon Jul  5 11:04:59 2010
From: linux-cluster at redhat.com (Mailbot for etexusa.com)
Date: Mon, 5 Jul 2010 04:04:59 -0700
Subject: [Linux-cluster] DSN: failed (error)
Message-ID: <mAWtxIeCDcrRuC8PO02@etexusa.com>


This is a Delivery Status Notification (DSN).

I was unable to deliver your message to
webgate at vancouver-webpages.com.

I said 
  (end of message)

And they gave me the error;
  554 5.7.1 Virus Windows.EXE.Is.Blocked.UNOFFICIAL detected by the ClamAV AntiVirus

 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/rfc822-headers
Size: 500 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100705/949fe9a8/attachment.bin>

From torintino1 at hotmail.com  Mon Jul  5 20:46:41 2010
From: torintino1 at hotmail.com (Torintino T)
Date: Mon, 5 Jul 2010 23:46:41 +0300
Subject: [Linux-cluster] Redhat Cluster
Message-ID: <BAY110-W4829A1231F308D80A23AB68CB10@phx.gbl>


 

Dear All,

I am newbie to Linux Clustering, i have  2 standalone Redhat servers, i want to setup a cluster on those servers,
to synchronize between each other, and to make a one as standby to the other, if a one fails the other will switchover.

I will mostly use Apache, Mysql, and PHP.

I have read the "Cluster Administration" document, i found that there are multiple methods to setup the cluster,
actually i want to ask expert people in the clustering, which method is the most proper one,
and should i have use a fence device, which one i will preferably use.

Thanks for assistance.
 		 	   		  
_________________________________________________________________
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
https://signup.live.com/signup.aspx?id=60969
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100705/52878c05/attachment.htm>

From a.alawi at auckland.ac.nz  Tue Jul  6 01:22:47 2010
From: a.alawi at auckland.ac.nz (Abraham Alawi)
Date: Tue, 6 Jul 2010 13:22:47 +1200
Subject: [Linux-cluster] CLVM/GFS hangs while rebuilding an AoE RAID5 LUN,
	anyone experienced that before? any clue?
Message-ID: <3AA10161-7D76-4CC6-8D85-FC49FF9C2B36@auckland.ac.nz>

The system was running well for a while but lately we had a flaky disk in the RAID array which we replaced with a healthy one but suddenly the CLVM/GFS became unusable, we can mount GFS but while listing it recursively 'ls -R' it hangs with Input/output error, can't even access the c/LVM LUN rawly using 'dd' BUT we still can access the LVM PV devices using 'dd'. Reconfiguring the LVM volume as a local one and accessing it exclusively from one node doesn't make a difference. 

RHEL5: 2.6.18-164.11.1.el5
# modinfo gfs
filename:       /lib/modules/2.6.18-164.11.1.el5/weak-updates/gfs/gfs.ko
license:        GPL
author:         Red Hat, Inc.
description:    Global File System 0.1.34-2.el5
srcversion:     3B1BAC4069F1A4B556A958A
depends:        dlm
vermagic:       2.6.18-159.el5 SMP mod_unload gcc-4.1

# uname -r
2.6.18-164.11.1.el5

# modinfo /lib/modules/2.6.18-164.11.1.el5/kernel/drivers/block/aoe/aoe.ko
filename:       /lib/modules/2.6.18-164.11.1.el5/kernel/drivers/block/aoe/aoe.ko
description:    AoE block/char driver for 2.6.2 and newer 2.6 kernels
author:         Sam Hopkins <sah at coraid.com>
license:        GPL
srcversion:     42BF122979AC807F2BB50E6
depends:        
vermagic:       2.6.18-164.11.1.el5 SMP mod_unload gcc-4.1
parm:           aoe_iflist:aoe_iflist=dev1[,dev2...]
 (string)
parm:           version:aoe module version 74
 (string)
parm:           aoe_dyndevs:Use dynamic minor numbers for devices. (int)
parm:           aoe_deadsecs:After aoe_deadsecs seconds, give up and fail dev. (int)
parm:           aoe_maxout:Only aoe_maxout outstanding packets for every MAC on eX.Y. (int)
parm:           aoe_maxsectors:When nonzero, set the maximum number of sectors per I/O request in new devices. (int)

# modinfo dlm
filename:       /lib/modules/2.6.18-164.11.1.el5/kernel/fs/dlm/dlm.ko
license:        GPL
author:         Red Hat, Inc.
description:    Distributed Lock Manager
srcversion:     E768995007648CA8DB078AE
depends:        configfs
vermagic:       2.6.18-164.11.1.el5 SMP mod_unload gcc-4.1
module_sig:	883f3504b56fe19c59c69348c13cf1f1126a509f6ddaee3965ee8b5fcd04163669647a889a9801e09f722187d1de068c0d52cd2b99bc3d475cb6ca1a0



Herein what the kernel spits out:

Jul  6 11:27:36 kiwiland kernel: GFS 0.1.34-2.el5 (built Sep  9 2009 06:54:42) installed
Jul  6 11:27:36 kiwiland kernel: Lock_DLM (built Sep  9 2009 06:54:38) installed
Jul  6 11:27:36 kiwiland kernel: Lock_Nolock (built Sep  9 2009 06:54:37) installed
Jul  6 11:27:36 kiwiland kernel: Trying to join cluster "lock_dlm", "FSC:files"
Jul  6 11:27:36 kiwiland kernel: Joined cluster. Now mounting FS...
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Trying to acquire journal lock...
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Looking at journal...
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Acquiring the transaction lock...
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Replaying journal...
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Replayed 0 of 11 blocks
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: replays = 0, skips = 4, sames = 7
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Journal replayed in 1s
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Done
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=1: Trying to acquire journal lock...
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=1: Looking at journal...
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=1: Done
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Scanning for log elements...
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Found 2 unlinked inodes
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Found quota changes for 2 IDs
Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Done
Jul  6 11:27:36 kiwiland kernel: Trying to join cluster "lock_dlm", "FSC:webcluster"
Jul  6 11:27:36 kiwiland kernel: Joined cluster. Now mounting FS...
Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=1: Trying to acquire journal lock...
Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=1: Looking at journal...
Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=1: Done
Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Scanning for log elements...
Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Found 0 unlinked inodes
Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Found quota changes for 0 IDs
Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Done
Jul  6 11:27:37 kiwiland kernel: Installing knfsd (copyright (C) 1996 okir at monad.swb.de).
Jul  6 11:27:39 kiwiland kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Jul  6 11:27:39 kiwiland kernel: NFSD: starting 90-second grace period
Jul  6 11:32:21 kiwiland kernel: dlm: closing connection to node 1
Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Trying to acquire journal lock...
Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0: fatal: invalid metadata block
Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   bh = 1432543247 (magic)
Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   function = gfs_rgrp_read
Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   file = /builddir/build/BUILD/gfs-kmod-0.1.34/_kmod_build_/src/gfs/rgrp.c, line = 830
Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   time = 1278372781
Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0: about to withdraw from the cluster
Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0: telling LM to withdraw
Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Looking at journal...
Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Acquiring the transaction lock...
Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Replaying journal...
Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Replayed 0 of 0 blocks
Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: replays = 0, skips = 0, sames = 0
Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Journal replayed in 1s
Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Done
Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:files.0: withdrawn
Jul  6 11:33:02 kiwiland kernel: 
Jul  6 11:33:02 kiwiland kernel: Call Trace:
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff88805018>] :gfs:gfs_lm_withdraw+0xc4/0xd3
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff80063a36>] __wait_on_bit+0x60/0x6e
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8001538b>] sync_buffer+0x0/0x3f
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff800a00e5>] wake_bit_function+0x0/0x23
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8881cc97>] :gfs:gfs_meta_check_ii+0x32/0x3e
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff88819439>] :gfs:gfs_rgrp_read+0x139/0x225
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff887fb8e8>] :gfs:glock_wait_internal+0x229/0x2c3
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff887fbd17>] :gfs:gfs_glock_nq+0x395/0x3d6
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff887fbd6e>] :gfs:gfs_glock_nq_init+0x16/0x2a
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff88817466>] :gfs:gfs_rgrp_lvb_init+0x1e/0x3f
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8881a46f>] :gfs:gfs_stat_gfs+0x213/0x273
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8881353d>] :gfs:gfs_statfs+0x67/0xea
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff800deba3>] vfs_statfs+0x63/0x7f
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886d2ce>] :nfsd:nfsd_statfs+0x28/0x38
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff888745f8>] :nfsd:nfsd3_proc_fsstat+0x3f/0x54
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a1db>] :nfsd:nfsd_dispatch+0xd8/0x1d6
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff886e0529>] :sunrpc:svc_process+0x454/0x71b
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff80064644>] __down_read+0x12/0x92
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a5a1>] :nfsd:nfsd+0x0/0x2cb
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a746>] :nfsd:nfsd+0x1a5/0x2cb
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a5a1>] :nfsd:nfsd+0x0/0x2cb
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a5a1>] :nfsd:nfsd+0x0/0x2cb
Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul  6 11:33:02 kiwiland kernel: 


Another kernel spit out:
Jul  5 02:01:19 Hercules kernel: GFS: fsid=FSC:files.0: fast statfs start time = 1278252079
Jul  5 03:01:16 Hercules kernel: GFS: fsid=FSC:files.0: fast statfs start time = 1278255676
Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0: fatal: invalid metadata block
Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   bh = 86700288 (magic)
Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   function = gfs_get_meta_buffer
Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   file = /builddir/build/BUILD/gfs-kmod-0.1.34/_kmod_build_/src/gfs/dio.c, line = 1225
Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   time = 1278255737
Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0: about to withdraw from the cluster
Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0: telling LM to withdraw
Jul  5 03:02:21 Hercules kernel: GFS: fsid=FSC:files.0: withdrawn
Jul  5 03:02:21 Hercules kernel: 
Jul  5 03:02:21 Hercules kernel: Call Trace:
Jul  5 03:02:21 Hercules kernel:  [<ffffffff8880a018>] :gfs:gfs_lm_withdraw+0xc4/0xd3
Jul  5 03:02:21 Hercules kernel:  [<ffffffff8001538b>] sync_buffer+0x0/0x3f
Jul  5 03:02:21 Hercules kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Jul  5 03:02:21 Hercules kernel:  [<ffffffff800a00e5>] wake_bit_function+0x0/0x23
Jul  5 03:02:21 Hercules kernel:  [<ffffffff88821c97>] :gfs:gfs_meta_check_ii+0x32/0x3e
Jul  5 03:02:21 Hercules kernel:  [<ffffffff887f7717>] :gfs:gfs_get_meta_buffer+0x1d1/0x247
Jul  5 03:02:21 Hercules kernel:  [<ffffffff88804193>] :gfs:gfs_copyin_dinode+0x1d/0x12f
Jul  5 03:02:21 Hercules kernel:  [<ffffffff88800d6e>] :gfs:gfs_glock_nq_init+0x16/0x2a
Jul  5 03:02:21 Hercules kernel:  [<ffffffff888043e3>] :gfs:inode_create+0x13e/0x1df
Jul  5 03:02:21 Hercules kernel:  [<ffffffff88804a5d>] :gfs:gfs_inode_get+0x9d/0xba
Jul  5 03:02:21 Hercules kernel:  [<ffffffff888053bb>] :gfs:gfs_lookupi+0x33d/0x3df
Jul  5 03:02:21 Hercules kernel:  [<ffffffff887fce57>] :gfs:ea_find_i+0x0/0x6b
Jul  5 03:02:21 Hercules kernel:  [<ffffffff888172af>] :gfs:gfs_lookup+0x363/0x41a
Jul  5 03:02:21 Hercules kernel:  [<ffffffff80025426>] igrab+0x25/0x34
Jul  5 03:02:21 Hercules kernel:  [<ffffffff888055a0>] :gfs:gfs_iget+0x3d/0x1f1
Jul  5 03:02:21 Hercules kernel:  [<ffffffff88801224>] :gfs:gfs_glock_dq+0x13c/0x14b
Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000cf01>] do_lookup+0xe5/0x1e6
Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000a22b>] __link_path_walk+0xa01/0xf42
Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
Jul  5 03:02:21 Hercules kernel:  [<ffffffff80012752>] getname+0x15b/0x1c2
Jul  5 03:02:21 Hercules kernel:  [<ffffffff800236ba>] __user_walk_fd+0x37/0x4c
Jul  5 03:02:21 Hercules kernel:  [<ffffffff8003f235>] vfs_lstat_fd+0x18/0x47
Jul  5 03:02:21 Hercules kernel:  [<ffffffff8002a95a>] sys_newlstat+0x19/0x31
Jul  5 03:02:21 Hercules kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
Jul  5 03:02:21 Hercules kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83


Thanks in advance,

  -- Abraham

''''''''''''''''''''''''''''''''''''''''''''''''''''''
Abraham Alawi

Unix/Linux Systems Administrator
Science IT
University of Auckland
e: a.alawi at auckland.ac.nz
p: +64-9-373 7599, ext#: 87572

''''''''''''''''''''''''''''''''''''''''''''''''''''''




From sakect at gmail.com  Tue Jul  6 03:44:21 2010
From: sakect at gmail.com (POWERBALL ONLINE)
Date: Tue, 6 Jul 2010 10:44:21 +0700
Subject: [Linux-cluster] Redhat Cluster
In-Reply-To: <BAY110-W4829A1231F308D80A23AB68CB10@phx.gbl>
References: <BAY110-W4829A1231F308D80A23AB68CB10@phx.gbl>
Message-ID: <AANLkTinG_sNwYuM3uaeWJs4mCNZNpjd6HyvH_hvpAd6d@mail.gmail.com>

I thing you should use conga because it very easy

2010/7/6 Torintino T <torintino1 at hotmail.com>

>
> Dear All,
>
> I am newbie to Linux Clustering, i have  2 standalone Redhat servers, i
> want to setup a cluster on those servers,
> to synchronize between each other, and to make a one as standby to the
> other, if a one fails the other will switchover.
>
> I will mostly use Apache, Mysql, and PHP.
>
> I have read the "Cluster Administration" document, i found that there are
> multiple methods to setup the cluster,
> actually i want to ask expert people in the clustering, which method is the
> most proper one,
> and should i have use a fence device, which one i will preferably use.
>
> Thanks for assistance.
>
> ------------------------------
> Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. Sign up
> now. <https://signup.live.com/signup.aspx?id=60969>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100706/eaae2cb5/attachment.htm>

From swhiteho at redhat.com  Tue Jul  6 08:22:01 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Tue, 06 Jul 2010 09:22:01 +0100
Subject: [Linux-cluster] CLVM/GFS hangs while rebuilding an AoE RAID5
 LUN, anyone experienced that before? any clue?
In-Reply-To: <3AA10161-7D76-4CC6-8D85-FC49FF9C2B36@auckland.ac.nz>
References: <3AA10161-7D76-4CC6-8D85-FC49FF9C2B36@auckland.ac.nz>
Message-ID: <1278404521.2514.2.camel@localhost>

Hi,

It looks to me as if the fs is corrupt in some manner. Try unmounting on
all nodes and running fsck on one node on the filesystem. Make sure you
save the output of fsck in case that is useful for future debugging and
make sure you have a backup of the data in question first.

Its tricky to say exactly what might have gone wrong (the fsck output
might give a clue) but you will certainly need fsck to fix whatever the
problem is,

Steve.

On Tue, 2010-07-06 at 13:22 +1200, Abraham Alawi wrote:
> The system was running well for a while but lately we had a flaky disk in the RAID array which we replaced with a healthy one but suddenly the CLVM/GFS became unusable, we can mount GFS but while listing it recursively 'ls -R' it hangs with Input/output error, can't even access the c/LVM LUN rawly using 'dd' BUT we still can access the LVM PV devices using 'dd'. Reconfiguring the LVM volume as a local one and accessing it exclusively from one node doesn't make a difference. 
> 
> RHEL5: 2.6.18-164.11.1.el5
> # modinfo gfs
> filename:       /lib/modules/2.6.18-164.11.1.el5/weak-updates/gfs/gfs.ko
> license:        GPL
> author:         Red Hat, Inc.
> description:    Global File System 0.1.34-2.el5
> srcversion:     3B1BAC4069F1A4B556A958A
> depends:        dlm
> vermagic:       2.6.18-159.el5 SMP mod_unload gcc-4.1
> 
> # uname -r
> 2.6.18-164.11.1.el5
> 
> # modinfo /lib/modules/2.6.18-164.11.1.el5/kernel/drivers/block/aoe/aoe.ko
> filename:       /lib/modules/2.6.18-164.11.1.el5/kernel/drivers/block/aoe/aoe.ko
> description:    AoE block/char driver for 2.6.2 and newer 2.6 kernels
> author:         Sam Hopkins <sah at coraid.com>
> license:        GPL
> srcversion:     42BF122979AC807F2BB50E6
> depends:        
> vermagic:       2.6.18-164.11.1.el5 SMP mod_unload gcc-4.1
> parm:           aoe_iflist:aoe_iflist=dev1[,dev2...]
>  (string)
> parm:           version:aoe module version 74
>  (string)
> parm:           aoe_dyndevs:Use dynamic minor numbers for devices. (int)
> parm:           aoe_deadsecs:After aoe_deadsecs seconds, give up and fail dev. (int)
> parm:           aoe_maxout:Only aoe_maxout outstanding packets for every MAC on eX.Y. (int)
> parm:           aoe_maxsectors:When nonzero, set the maximum number of sectors per I/O request in new devices. (int)
> 
> # modinfo dlm
> filename:       /lib/modules/2.6.18-164.11.1.el5/kernel/fs/dlm/dlm.ko
> license:        GPL
> author:         Red Hat, Inc.
> description:    Distributed Lock Manager
> srcversion:     E768995007648CA8DB078AE
> depends:        configfs
> vermagic:       2.6.18-164.11.1.el5 SMP mod_unload gcc-4.1
> module_sig:	883f3504b56fe19c59c69348c13cf1f1126a509f6ddaee3965ee8b5fcd04163669647a889a9801e09f722187d1de068c0d52cd2b99bc3d475cb6ca1a0
> 
> 
> 
> Herein what the kernel spits out:
> 
> Jul  6 11:27:36 kiwiland kernel: GFS 0.1.34-2.el5 (built Sep  9 2009 06:54:42) installed
> Jul  6 11:27:36 kiwiland kernel: Lock_DLM (built Sep  9 2009 06:54:38) installed
> Jul  6 11:27:36 kiwiland kernel: Lock_Nolock (built Sep  9 2009 06:54:37) installed
> Jul  6 11:27:36 kiwiland kernel: Trying to join cluster "lock_dlm", "FSC:files"
> Jul  6 11:27:36 kiwiland kernel: Joined cluster. Now mounting FS...
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Trying to acquire journal lock...
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Looking at journal...
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Acquiring the transaction lock...
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Replaying journal...
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Replayed 0 of 11 blocks
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: replays = 0, skips = 4, sames = 7
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Journal replayed in 1s
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Done
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=1: Trying to acquire journal lock...
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=1: Looking at journal...
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=1: Done
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Scanning for log elements...
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Found 2 unlinked inodes
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Found quota changes for 2 IDs
> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Done
> Jul  6 11:27:36 kiwiland kernel: Trying to join cluster "lock_dlm", "FSC:webcluster"
> Jul  6 11:27:36 kiwiland kernel: Joined cluster. Now mounting FS...
> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=1: Trying to acquire journal lock...
> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=1: Looking at journal...
> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=1: Done
> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Scanning for log elements...
> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Found 0 unlinked inodes
> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Found quota changes for 0 IDs
> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Done
> Jul  6 11:27:37 kiwiland kernel: Installing knfsd (copyright (C) 1996 okir at monad.swb.de).
> Jul  6 11:27:39 kiwiland kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
> Jul  6 11:27:39 kiwiland kernel: NFSD: starting 90-second grace period
> Jul  6 11:32:21 kiwiland kernel: dlm: closing connection to node 1
> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Trying to acquire journal lock...
> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0: fatal: invalid metadata block
> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   bh = 1432543247 (magic)
> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   function = gfs_rgrp_read
> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   file = /builddir/build/BUILD/gfs-kmod-0.1.34/_kmod_build_/src/gfs/rgrp.c, line = 830
> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   time = 1278372781
> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0: about to withdraw from the cluster
> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0: telling LM to withdraw
> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Looking at journal...
> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Acquiring the transaction lock...
> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Replaying journal...
> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Replayed 0 of 0 blocks
> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: replays = 0, skips = 0, sames = 0
> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Journal replayed in 1s
> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Done
> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:files.0: withdrawn
> Jul  6 11:33:02 kiwiland kernel: 
> Jul  6 11:33:02 kiwiland kernel: Call Trace:
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff88805018>] :gfs:gfs_lm_withdraw+0xc4/0xd3
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff80063a36>] __wait_on_bit+0x60/0x6e
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8001538b>] sync_buffer+0x0/0x3f
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff800a00e5>] wake_bit_function+0x0/0x23
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8881cc97>] :gfs:gfs_meta_check_ii+0x32/0x3e
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff88819439>] :gfs:gfs_rgrp_read+0x139/0x225
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff887fb8e8>] :gfs:glock_wait_internal+0x229/0x2c3
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff887fbd17>] :gfs:gfs_glock_nq+0x395/0x3d6
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff887fbd6e>] :gfs:gfs_glock_nq_init+0x16/0x2a
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff88817466>] :gfs:gfs_rgrp_lvb_init+0x1e/0x3f
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8881a46f>] :gfs:gfs_stat_gfs+0x213/0x273
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8881353d>] :gfs:gfs_statfs+0x67/0xea
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff800deba3>] vfs_statfs+0x63/0x7f
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886d2ce>] :nfsd:nfsd_statfs+0x28/0x38
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff888745f8>] :nfsd:nfsd3_proc_fsstat+0x3f/0x54
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a1db>] :nfsd:nfsd_dispatch+0xd8/0x1d6
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff886e0529>] :sunrpc:svc_process+0x454/0x71b
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff80064644>] __down_read+0x12/0x92
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a5a1>] :nfsd:nfsd+0x0/0x2cb
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a746>] :nfsd:nfsd+0x1a5/0x2cb
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a5a1>] :nfsd:nfsd+0x0/0x2cb
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a5a1>] :nfsd:nfsd+0x0/0x2cb
> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
> Jul  6 11:33:02 kiwiland kernel: 
> 
> 
> Another kernel spit out:
> Jul  5 02:01:19 Hercules kernel: GFS: fsid=FSC:files.0: fast statfs start time = 1278252079
> Jul  5 03:01:16 Hercules kernel: GFS: fsid=FSC:files.0: fast statfs start time = 1278255676
> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0: fatal: invalid metadata block
> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   bh = 86700288 (magic)
> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   function = gfs_get_meta_buffer
> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   file = /builddir/build/BUILD/gfs-kmod-0.1.34/_kmod_build_/src/gfs/dio.c, line = 1225
> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   time = 1278255737
> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0: about to withdraw from the cluster
> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0: telling LM to withdraw
> Jul  5 03:02:21 Hercules kernel: GFS: fsid=FSC:files.0: withdrawn
> Jul  5 03:02:21 Hercules kernel: 
> Jul  5 03:02:21 Hercules kernel: Call Trace:
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8880a018>] :gfs:gfs_lm_withdraw+0xc4/0xd3
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8001538b>] sync_buffer+0x0/0x3f
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff800a00e5>] wake_bit_function+0x0/0x23
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff88821c97>] :gfs:gfs_meta_check_ii+0x32/0x3e
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff887f7717>] :gfs:gfs_get_meta_buffer+0x1d1/0x247
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff88804193>] :gfs:gfs_copyin_dinode+0x1d/0x12f
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff88800d6e>] :gfs:gfs_glock_nq_init+0x16/0x2a
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff888043e3>] :gfs:inode_create+0x13e/0x1df
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff88804a5d>] :gfs:gfs_inode_get+0x9d/0xba
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff888053bb>] :gfs:gfs_lookupi+0x33d/0x3df
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff887fce57>] :gfs:ea_find_i+0x0/0x6b
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff888172af>] :gfs:gfs_lookup+0x363/0x41a
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff80025426>] igrab+0x25/0x34
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff888055a0>] :gfs:gfs_iget+0x3d/0x1f1
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff88801224>] :gfs:gfs_glock_dq+0x13c/0x14b
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000cf01>] do_lookup+0xe5/0x1e6
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000a22b>] __link_path_walk+0xa01/0xf42
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff80012752>] getname+0x15b/0x1c2
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff800236ba>] __user_walk_fd+0x37/0x4c
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8003f235>] vfs_lstat_fd+0x18/0x47
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8002a95a>] sys_newlstat+0x19/0x31
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
> 
> 
> Thanks in advance,
> 
>   -- Abraham
> 
> ''''''''''''''''''''''''''''''''''''''''''''''''''''''
> Abraham Alawi
> 
> Unix/Linux Systems Administrator
> Science IT
> University of Auckland
> e: a.alawi at auckland.ac.nz
> p: +64-9-373 7599, ext#: 87572
> 
> ''''''''''''''''''''''''''''''''''''''''''''''''''''''
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From peter.mcgowan at bit63.com  Tue Jul  6 13:12:41 2010
From: peter.mcgowan at bit63.com (Peter McGowan)
Date: Tue, 6 Jul 2010 14:12:41 +0100
Subject: [Linux-cluster] Two node cluster with iLO fencing.
Message-ID: <45A3F572-02C5-4C58-94A6-4938DE4602C4@bit63.com>

Hi

I'm looking to put together a two-node cluster that will use HP iLOs for fencing. The systems have their bond0 interfaces connected to the public LAN, and I was intending to use a private non-routable network (bond1) purely for cluster communication traffic.

What's confused me is the line in the CMAN FAQ which says "If you are using per-node power management of any sort where the device is not shared between cluster nodes, it must be connected to the same network used by CMAN for cluster communication"

Is anyone able to define "same network" in this context? My iLOs will be reachable through bond0 (public LAN), but not through bond1 (cluster network) - does this count as "same network"?

Thanks for any clarification,
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100706/3dd17361/attachment.htm>

From ccaulfie at redhat.com  Tue Jul  6 13:30:38 2010
From: ccaulfie at redhat.com (Christine Caulfield)
Date: Tue, 06 Jul 2010 14:30:38 +0100
Subject: [Linux-cluster] Two node cluster with iLO fencing.
In-Reply-To: <45A3F572-02C5-4C58-94A6-4938DE4602C4@bit63.com>
References: <45A3F572-02C5-4C58-94A6-4938DE4602C4@bit63.com>
Message-ID: <4C332FFE.8070107@redhat.com>

On 06/07/10 14:12, Peter McGowan wrote:
> Hi
>
> I'm looking to put together a two-node cluster that will use HP iLOs for
> fencing. The systems have their bond0 interfaces connected to the public
> LAN, and I was intending to use a private non-routable network (bond1)
> purely for cluster communication traffic.
>
> What's confused me is the line in the CMAN FAQ which says "If you are
> using per-node power management of any sort where the device is not
> shared between cluster nodes, it must be connected to the same network
> used by CMAN for cluster communication"
>
> Is anyone able to define "same network" in this context? My iLOs will be
> reachable through bond0 (public LAN), but not through bond1 (cluster
> network) - does this count as "same network"?
>


No it doesn't. "Same network" means the same (set of) interfaces that 
the cluster traffic goes over.

The reason for this is so that in the event of a network outage, only 
one node has access to the fence device and can fence the other. If the 
fence device is on a different network (as yours is) then both nodes 
will be able to fence each other in the event that the cluster 
communications network breaks. When this happens both nodes will be 
rebooted, rather than just one.

Chrissie



From peter.mcgowan at bit63.com  Tue Jul  6 13:50:08 2010
From: peter.mcgowan at bit63.com (Peter McGowan)
Date: Tue, 6 Jul 2010 14:50:08 +0100
Subject: [Linux-cluster] Two node cluster with iLO fencing.
In-Reply-To: <4C332FFE.8070107@redhat.com>
References: <45A3F572-02C5-4C58-94A6-4938DE4602C4@bit63.com>
	<4C332FFE.8070107@redhat.com>
Message-ID: <CF0513AF-A121-4CDD-8854-4A8E7C6238B6@bit63.com>


> On 06/07/10 14:12, Peter McGowan wrote:
>> Hi
>> 
>> I'm looking to put together a two-node cluster that will use HP iLOs for
>> fencing. The systems have their bond0 interfaces connected to the public
>> LAN, and I was intending to use a private non-routable network (bond1)
>> purely for cluster communication traffic.
>> 
>> What's confused me is the line in the CMAN FAQ which says "If you are
>> using per-node power management of any sort where the device is not
>> shared between cluster nodes, it must be connected to the same network
>> used by CMAN for cluster communication"
>> 
>> Is anyone able to define "same network" in this context? My iLOs will be
>> reachable through bond0 (public LAN), but not through bond1 (cluster
>> network) - does this count as "same network"?
>> 
> 
> 
> No it doesn't. "Same network" means the same (set of) interfaces that the cluster traffic goes over.
> 
> The reason for this is so that in the event of a network outage, only one node has access to the fence device and can fence the other. If the fence device is on a different network (as yours is) then both nodes will be able to fence each other in the event that the cluster communications network breaks. When this happens both nodes will be rebooted, rather than just one.
> 
> Chrissie


OK, that makes sense, thanks for the answer. It sounds like using a separate network for cluster communication is generally not such a good idea, as I wouldn't be able to use ping type heuristics either if I decide to use a quorum disk. 

Thanks,
Peter

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100706/8f89dc5a/attachment.htm>

From rajatjpatel at gmail.com  Tue Jul  6 16:18:19 2010
From: rajatjpatel at gmail.com (rajatjpatel)
Date: Tue, 6 Jul 2010 21:48:19 +0530
Subject: [Linux-cluster] cluster step
Message-ID: <AANLkTinmNNsxILabHffBWtRc6E6H_Ad6C842ps5LcwFQ@mail.gmail.com>

http://studyhat.blogspot.com/2010/01/cluster-hp-ilo.html
http://studyhat.blogspot.com/2009/11/clustering-linux-ha.html

there 2 way setup cluster for redhat

one u can use system-config-cluster

two u can use luci

both the way above link will help to setup
Regards,

Rajat J Patel

FIRST THEY IGNORE YOU...
THEN THEY LAUGH AT YOU...
THEN THEY FIGHT YOU...
THEN YOU WIN...

Skype rajatjpatel
AIM    rajatjpatel
yahoo rajatjpatel
msn    rajatjpatel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100706/66553498/attachment.htm>

From rajatjpatel at gmail.com  Tue Jul  6 16:25:25 2010
From: rajatjpatel at gmail.com (rajatjpatel)
Date: Tue, 6 Jul 2010 21:55:25 +0530
Subject: [Linux-cluster] hp ilo
Message-ID: <AANLkTil_yifeBhHDt1OhlAXDjXqDLaL4jkPElLEqB_kY@mail.gmail.com>

http://studyhat.blogspot.com/2010/01/cluster-hp-ilo.html

http://studyhat.blogspot.com/2009/11/clustering-linux-ha.html

fenc_ilo -l login name that is administrator -p password -i ipaddress -o -v
status

# fence_ilo -a 10.163.16.31 -l Administrator -p password -o status -v

Regards,

Rajat J Patel

FIRST THEY IGNORE YOU...
THEN THEY LAUGH AT YOU...
THEN THEY FIGHT YOU...
THEN YOU WIN...

Skype rajatjpatel
AIM    rajatjpatel
yahoo rajatjpatel
msn    rajatjpatel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100706/6b2af9ab/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fence_ilo
Type: application/octet-stream
Size: 3304 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100706/6b2af9ab/attachment.obj>

From a.alawi at auckland.ac.nz  Tue Jul  6 23:57:58 2010
From: a.alawi at auckland.ac.nz (Abraham Alawi)
Date: Wed, 7 Jul 2010 11:57:58 +1200
Subject: [Linux-cluster] CLVM/GFS hangs while rebuilding an AoE RAID5
	LUN, anyone experienced that before? any clue?
In-Reply-To: <1278404521.2514.2.camel@localhost>
References: <3AA10161-7D76-4CC6-8D85-FC49FF9C2B36@auckland.ac.nz>
	<1278404521.2514.2.camel@localhost>
Message-ID: <8299E832-4699-4F13-B4F8-A56F0D2C4F88@auckland.ac.nz>

Thanks Steve, Yes, sadly I can confirm the file system has been corrupted but I still don't understand why I/O will stop flowing at the LVM level (& doesn't fence it either) and why fsck keeps crashing without a useful error message, is there any signal I can send to gfs_fsck to by pass certain stages? Also to speed up the fsck process, I was thinking of utilizing the RAM and increase the read_ahead parameter (hdparm -a) of the PV device (an AoE device) by 1GB since that will hugely optimize the sequential read and fscking is mostly a sequential read process and very bit of writings, what do you think?

Herein the tail of the last fsck log file:
(metawalk.c:516)	Extended attributes exist for inode #34020861.
(metawalk.c:413)	Checking EA leaf block #34020862.
(pass1.c:485)	Setting block #34020862 to eattr block
(pass1.c:907)	Checking metadata block 34020862
(pass1.c:923)	Metadata block 34020862 not an inode or free metadata
(pass1.c:907)	Checking metadata block 34020863
(link.c:22)	Setting link count to 1 for 34020863
(metawalk.c:516)	Extended attributes exist for inode #34020863.
(metawalk.c:413)	Checking EA leaf block #34020864.
(pass1.c:485)	Setting block #34020864 to eattr block
(pass1.c:907)	Checking metadata block 34020864
(pass1.c:923)	Metadata block 34020864 not an inode or free metadata
(pass1.c:907)	Checking metadata block 34020865
(link.c:22)	Setting link count to 1 for 34020865
(pass1.c:213)	Setting 34020917 to data block
(pass1.c:213)	Setting 34020918 to data block
(pass1.c:213)	Setting 34020919 to data block
(pass1.c:213)	Setting 34020920 to data block
(metawalk.c:516)	Extended attributes exist for inode #34020865.
(metawalk.c:413)	Checking EA leaf block #34020866.
(pass1.c:485)	Setting block #34020866 to eattr block
(pass1.c:907)	Checking metadata block 34020866
(pass1.c:923)	Metadata block 34020866 not an inode or free metadata
(pass1.c:907)	Checking metadata block 34020867


Thanks,

  -- Abraham

On 6/07/2010, at 8:22 PM, Steven Whitehouse wrote:

> Hi,
> 
> It looks to me as if the fs is corrupt in some manner. Try unmounting on
> all nodes and running fsck on one node on the filesystem. Make sure you
> save the output of fsck in case that is useful for future debugging and
> make sure you have a backup of the data in question first.
> 
> Its tricky to say exactly what might have gone wrong (the fsck output
> might give a clue) but you will certainly need fsck to fix whatever the
> problem is,
> 
> Steve.
> 
> On Tue, 2010-07-06 at 13:22 +1200, Abraham Alawi wrote:
>> The system was running well for a while but lately we had a flaky disk in the RAID array which we replaced with a healthy one but suddenly the CLVM/GFS became unusable, we can mount GFS but while listing it recursively 'ls -R' it hangs with Input/output error, can't even access the c/LVM LUN rawly using 'dd' BUT we still can access the LVM PV devices using 'dd'. Reconfiguring the LVM volume as a local one and accessing it exclusively from one node doesn't make a difference. 
>> 
>> RHEL5: 2.6.18-164.11.1.el5
>> # modinfo gfs
>> filename:       /lib/modules/2.6.18-164.11.1.el5/weak-updates/gfs/gfs.ko
>> license:        GPL
>> author:         Red Hat, Inc.
>> description:    Global File System 0.1.34-2.el5
>> srcversion:     3B1BAC4069F1A4B556A958A
>> depends:        dlm
>> vermagic:       2.6.18-159.el5 SMP mod_unload gcc-4.1
>> 
>> # uname -r
>> 2.6.18-164.11.1.el5
>> 
>> # modinfo /lib/modules/2.6.18-164.11.1.el5/kernel/drivers/block/aoe/aoe.ko
>> filename:       /lib/modules/2.6.18-164.11.1.el5/kernel/drivers/block/aoe/aoe.ko
>> description:    AoE block/char driver for 2.6.2 and newer 2.6 kernels
>> author:         Sam Hopkins <sah at coraid.com>
>> license:        GPL
>> srcversion:     42BF122979AC807F2BB50E6
>> depends:        
>> vermagic:       2.6.18-164.11.1.el5 SMP mod_unload gcc-4.1
>> parm:           aoe_iflist:aoe_iflist=dev1[,dev2...]
>> (string)
>> parm:           version:aoe module version 74
>> (string)
>> parm:           aoe_dyndevs:Use dynamic minor numbers for devices. (int)
>> parm:           aoe_deadsecs:After aoe_deadsecs seconds, give up and fail dev. (int)
>> parm:           aoe_maxout:Only aoe_maxout outstanding packets for every MAC on eX.Y. (int)
>> parm:           aoe_maxsectors:When nonzero, set the maximum number of sectors per I/O request in new devices. (int)
>> 
>> # modinfo dlm
>> filename:       /lib/modules/2.6.18-164.11.1.el5/kernel/fs/dlm/dlm.ko
>> license:        GPL
>> author:         Red Hat, Inc.
>> description:    Distributed Lock Manager
>> srcversion:     E768995007648CA8DB078AE
>> depends:        configfs
>> vermagic:       2.6.18-164.11.1.el5 SMP mod_unload gcc-4.1
>> module_sig:	883f3504b56fe19c59c69348c13cf1f1126a509f6ddaee3965ee8b5fcd04163669647a889a9801e09f722187d1de068c0d52cd2b99bc3d475cb6ca1a0
>> 
>> 
>> 
>> Herein what the kernel spits out:
>> 
>> Jul  6 11:27:36 kiwiland kernel: GFS 0.1.34-2.el5 (built Sep  9 2009 06:54:42) installed
>> Jul  6 11:27:36 kiwiland kernel: Lock_DLM (built Sep  9 2009 06:54:38) installed
>> Jul  6 11:27:36 kiwiland kernel: Lock_Nolock (built Sep  9 2009 06:54:37) installed
>> Jul  6 11:27:36 kiwiland kernel: Trying to join cluster "lock_dlm", "FSC:files"
>> Jul  6 11:27:36 kiwiland kernel: Joined cluster. Now mounting FS...
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Trying to acquire journal lock...
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Looking at journal...
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Acquiring the transaction lock...
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Replaying journal...
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Replayed 0 of 11 blocks
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: replays = 0, skips = 4, sames = 7
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Journal replayed in 1s
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=0: Done
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=1: Trying to acquire journal lock...
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=1: Looking at journal...
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: jid=1: Done
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Scanning for log elements...
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Found 2 unlinked inodes
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Found quota changes for 2 IDs
>> Jul  6 11:27:36 kiwiland kernel: GFS: fsid=FSC:files.0: Done
>> Jul  6 11:27:36 kiwiland kernel: Trying to join cluster "lock_dlm", "FSC:webcluster"
>> Jul  6 11:27:36 kiwiland kernel: Joined cluster. Now mounting FS...
>> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=1: Trying to acquire journal lock...
>> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=1: Looking at journal...
>> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=1: Done
>> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Scanning for log elements...
>> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Found 0 unlinked inodes
>> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Found quota changes for 0 IDs
>> Jul  6 11:27:37 kiwiland kernel: GFS: fsid=FSC:webcluster.1: Done
>> Jul  6 11:27:37 kiwiland kernel: Installing knfsd (copyright (C) 1996 okir at monad.swb.de).
>> Jul  6 11:27:39 kiwiland kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
>> Jul  6 11:27:39 kiwiland kernel: NFSD: starting 90-second grace period
>> Jul  6 11:32:21 kiwiland kernel: dlm: closing connection to node 1
>> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Trying to acquire journal lock...
>> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0: fatal: invalid metadata block
>> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   bh = 1432543247 (magic)
>> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   function = gfs_rgrp_read
>> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   file = /builddir/build/BUILD/gfs-kmod-0.1.34/_kmod_build_/src/gfs/rgrp.c, line = 830
>> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0:   time = 1278372781
>> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0: about to withdraw from the cluster
>> Jul  6 11:33:01 kiwiland kernel: GFS: fsid=FSC:files.0: telling LM to withdraw
>> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Looking at journal...
>> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Acquiring the transaction lock...
>> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Replaying journal...
>> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Replayed 0 of 0 blocks
>> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: replays = 0, skips = 0, sames = 0
>> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Journal replayed in 1s
>> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:webcluster.1: jid=0: Done
>> Jul  6 11:33:02 kiwiland kernel: GFS: fsid=FSC:files.0: withdrawn
>> Jul  6 11:33:02 kiwiland kernel: 
>> Jul  6 11:33:02 kiwiland kernel: Call Trace:
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff88805018>] :gfs:gfs_lm_withdraw+0xc4/0xd3
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff80063a36>] __wait_on_bit+0x60/0x6e
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8001538b>] sync_buffer+0x0/0x3f
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff800a00e5>] wake_bit_function+0x0/0x23
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8881cc97>] :gfs:gfs_meta_check_ii+0x32/0x3e
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff88819439>] :gfs:gfs_rgrp_read+0x139/0x225
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff887fb8e8>] :gfs:glock_wait_internal+0x229/0x2c3
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff887fbd17>] :gfs:gfs_glock_nq+0x395/0x3d6
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff887fbd6e>] :gfs:gfs_glock_nq_init+0x16/0x2a
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff88817466>] :gfs:gfs_rgrp_lvb_init+0x1e/0x3f
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8881a46f>] :gfs:gfs_stat_gfs+0x213/0x273
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8881353d>] :gfs:gfs_statfs+0x67/0xea
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff800deba3>] vfs_statfs+0x63/0x7f
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886d2ce>] :nfsd:nfsd_statfs+0x28/0x38
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff888745f8>] :nfsd:nfsd3_proc_fsstat+0x3f/0x54
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a1db>] :nfsd:nfsd_dispatch+0xd8/0x1d6
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff886e0529>] :sunrpc:svc_process+0x454/0x71b
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff80064644>] __down_read+0x12/0x92
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a5a1>] :nfsd:nfsd+0x0/0x2cb
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a746>] :nfsd:nfsd+0x1a5/0x2cb
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a5a1>] :nfsd:nfsd+0x0/0x2cb
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8886a5a1>] :nfsd:nfsd+0x0/0x2cb
>> Jul  6 11:33:02 kiwiland kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
>> Jul  6 11:33:02 kiwiland kernel: 
>> 
>> 
>> Another kernel spit out:
>> Jul  5 02:01:19 Hercules kernel: GFS: fsid=FSC:files.0: fast statfs start time = 1278252079
>> Jul  5 03:01:16 Hercules kernel: GFS: fsid=FSC:files.0: fast statfs start time = 1278255676
>> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0: fatal: invalid metadata block
>> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   bh = 86700288 (magic)
>> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   function = gfs_get_meta_buffer
>> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   file = /builddir/build/BUILD/gfs-kmod-0.1.34/_kmod_build_/src/gfs/dio.c, line = 1225
>> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0:   time = 1278255737
>> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0: about to withdraw from the cluster
>> Jul  5 03:02:17 Hercules kernel: GFS: fsid=FSC:files.0: telling LM to withdraw
>> Jul  5 03:02:21 Hercules kernel: GFS: fsid=FSC:files.0: withdrawn
>> Jul  5 03:02:21 Hercules kernel: 
>> Jul  5 03:02:21 Hercules kernel: Call Trace:
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8880a018>] :gfs:gfs_lm_withdraw+0xc4/0xd3
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8001538b>] sync_buffer+0x0/0x3f
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff800a00e5>] wake_bit_function+0x0/0x23
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff88821c97>] :gfs:gfs_meta_check_ii+0x32/0x3e
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff887f7717>] :gfs:gfs_get_meta_buffer+0x1d1/0x247
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff88804193>] :gfs:gfs_copyin_dinode+0x1d/0x12f
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff88800d6e>] :gfs:gfs_glock_nq_init+0x16/0x2a
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff888043e3>] :gfs:inode_create+0x13e/0x1df
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff88804a5d>] :gfs:gfs_inode_get+0x9d/0xba
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff888053bb>] :gfs:gfs_lookupi+0x33d/0x3df
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff887fce57>] :gfs:ea_find_i+0x0/0x6b
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff888172af>] :gfs:gfs_lookup+0x363/0x41a
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff80025426>] igrab+0x25/0x34
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff888055a0>] :gfs:gfs_iget+0x3d/0x1f1
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff88801224>] :gfs:gfs_glock_dq+0x13c/0x14b
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000cf01>] do_lookup+0xe5/0x1e6
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000a22b>] __link_path_walk+0xa01/0xf42
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff80012752>] getname+0x15b/0x1c2
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff800236ba>] __user_walk_fd+0x37/0x4c
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8003f235>] vfs_lstat_fd+0x18/0x47
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8002a95a>] sys_newlstat+0x19/0x31
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
>> Jul  5 03:02:21 Hercules kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
>> 
>> 
>> Thanks in advance,
>> 
>>  -- Abraham
>> 
>> ''''''''''''''''''''''''''''''''''''''''''''''''''''''
>> Abraham Alawi
>> 
>> Unix/Linux Systems Administrator
>> Science IT
>> University of Auckland
>> e: a.alawi at auckland.ac.nz
>> p: +64-9-373 7599, ext#: 87572
>> 
>> ''''''''''''''''''''''''''''''''''''''''''''''''''''''
>> 
>> 
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

''''''''''''''''''''''''''''''''''''''''''''''''''''''
Abraham Alawi

Unix/Linux Systems Administrator
Science IT
University of Auckland
e: a.alawi at auckland.ac.nz
p: +64-9-373 7599, ext#: 87572

''''''''''''''''''''''''''''''''''''''''''''''''''''''




From sroy.tcs at gmail.com  Wed Jul  7 01:53:10 2010
From: sroy.tcs at gmail.com (=?ISO-8859-1?Q?S=E9bastien_Roy?=)
Date: Wed, 7 Jul 2010 03:53:10 +0200
Subject: [Linux-cluster] hp ilo
In-Reply-To: <AANLkTil_yifeBhHDt1OhlAXDjXqDLaL4jkPElLEqB_kY@mail.gmail.com>
References: <AANLkTil_yifeBhHDt1OhlAXDjXqDLaL4jkPElLEqB_kY@mail.gmail.com>
Message-ID: <AANLkTinPMncNycqMnl2Iy70qcKUyxFa67jO8IwB5Y8Br@mail.gmail.com>

please reply on original threads, not new emails

2010/7/6 rajatjpatel <rajatjpatel at gmail.com>

>
>
> http://studyhat.blogspot.com/2010/01/cluster-hp-ilo.html
>
> http://studyhat.blogspot.com/2009/11/clustering-linux-ha.html
>
> fenc_ilo -l login name that is administrator -p password -i ipaddress -o -v
> status
>
> # fence_ilo -a 10.163.16.31 -l Administrator -p password -o status -v
>
> Regards,
>
> Rajat J Patel
>
> FIRST THEY IGNORE YOU...
> THEN THEY LAUGH AT YOU...
> THEN THEY FIGHT YOU...
> THEN YOU WIN...
>
> Skype rajatjpatel
> AIM    rajatjpatel
> yahoo rajatjpatel
> msn    rajatjpatel
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100707/15e320f8/attachment.htm>

From Bennie_R_Thomas at raytheon.com  Wed Jul  7 07:44:54 2010
From: Bennie_R_Thomas at raytheon.com (Bennie Thomas)
Date: Wed, 07 Jul 2010 02:44:54 -0500
Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes
	randomly	reboot
In-Reply-To: <AANLkTilnu133h5o9e-nrol_ZWQDY7_U8pWayWjn_eHQX@mail.gmail.com>
References: <4C290BE0.3090707@raytheon.com>
	<AANLkTilnu133h5o9e-nrol_ZWQDY7_U8pWayWjn_eHQX@mail.gmail.com>
Message-ID: <4C343076.3060204@raytheon.com>

Nothing in the log files. Just a syslogd restart 
message(/var/log/messages). Checking the iLo/IML logs; it just shows 
that the node was fenced by the other node.

Paul Morgan wrote:
>
> You don't mention whether you checked your logs. If there is nothing 
> to indicate a problem, it may be a bios issue.
>
> At my client we recently updated our g6 bios for this exact behavior. 
> Sorry I don't have the hp link at the moment, but it's a known problem 
> with a fix.
>
> -paul
>
> --
> Top-posted from gmail on android 2.2
>
> On Jun 28, 2010 5:04 PM, "Bennie Thomas" <Bennie_R_Thomas at raytheon.com 
> <mailto:Bennie_R_Thomas at raytheon.com>> wrote:
> > I currently have 2 DL380 G6 with and HP MSA2312 disk array. Running
> > Redhat 5u4 64bit. I have a quorum disk. I use the Cluster as an
> > Active/passive NFS Cluster
> > The problem I am having is one or both of the nodes will randomly
> > reboot. Has anyone experienced this problem
> >
> > --
> > Bennie Thomas
> > Sr. Information Systems Technologist II
> > Raytheon Company
> >
> > 972.205.4126
> > 972.205.6363 fax
> > 888.347.1660 pager
> > Bennie_R_Thomas at raytheon.com <mailto:Bennie_R_Thomas at raytheon.com>
> >
> >
> > DISCLAIMER: This message contains information that may be 
> confidential and privileged. Unless you are the addressee (or 
> authorized to receive mail for the addressee), you should not use, 
> copy or disclose to anyone this message or any information contained 
> in this message. If you have received this message in error, please so 
> advise the sender by reply e-mail and delete this message. Thank you 
> for your cooperation.
> >
> > Any views or opinions presented are solely those of the author and 
> do not necessarily represent those of Raytheon unless specifically 
> stated.
> > Electronic communications including email may be monitored by Raytheon
> > for operational or business reasons.
> >
> >
> >
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com <mailto:Linux-cluster at redhat.com>
> > https://www.redhat.com/mailman/listinfo/linux-cluster
>
> ------------------------------------------------------------------------
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

-- 
Bennie Thomas
Sr. Information Systems Technologist II
Raytheon Company

972.205.4126
972.205.6363 fax
888.347.1660 pager
Bennie_R_Thomas at raytheon.com


DISCLAIMER: This message contains information that may be confidential and privileged. Unless you are the addressee (or authorized to receive mail for the addressee), you should not use, copy or disclose to anyone this message or any information contained in this message. If you have received this message in error, please so advise the sender by reply e-mail and delete this message. Thank you for your cooperation.

Any views or opinions presented are solely those of the author and do not necessarily represent those of Raytheon unless specifically stated. 
Electronic communications including email may be monitored by Raytheon
for operational or business reasons.






From Bennie_R_Thomas at raytheon.com  Wed Jul  7 07:45:49 2010
From: Bennie_R_Thomas at raytheon.com (Bennie Thomas)
Date: Wed, 07 Jul 2010 02:45:49 -0500
Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes
	randomly	reboot
In-Reply-To: <4C2B5322.9030305@arlut.utexas.edu>
References: <mailman.20241.1277763744.8372.linux-cluster@redhat.com>
	<4C2B5322.9030305@arlut.utexas.edu>
Message-ID: <4C3430AD.7090804@raytheon.com>


Sorry, I am using ext3 file systems.

Randy Zagar wrote:
> Yes.  My experience is that you can't currently nfs-export *any* GFS 
> or GFS2 filesystems.
>
> Exporting EXT3/EXT4 filesystems, however, doesn't appear to be a problem.
>
> -Randy Zagar <zagar at arlut.utexas.edu>
>
> On 06/28/2010 05:22 PM, linux-cluster-request at redhat.com wrote:
>> From: Bennie Thomas<Bennie_R_Thomas at raytheon.com>
>> Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes randomly
>>     reboot
>> Message-ID:<4C290BE0.3090707 at raytheon.com>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> I currently have 2 DL380 G6 with and HP MSA2312 disk array. Running
>> Redhat 5u4 64bit.  I have a quorum disk.  I use the Cluster as an
>> Active/passive NFS Cluster
>> The problem I am having is one or both of the nodes will randomly
>> reboot. Has anyone experienced this problem
>>
>>    
>
> ------------------------------------------------------------------------
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

-- 
Bennie Thomas
Sr. Information Systems Technologist II
Raytheon Company

972.205.4126
972.205.6363 fax
888.347.1660 pager
Bennie_R_Thomas at raytheon.com


DISCLAIMER: This message contains information that may be confidential and privileged. Unless you are the addressee (or authorized to receive mail for the addressee), you should not use, copy or disclose to anyone this message or any information contained in this message. If you have received this message in error, please so advise the sender by reply e-mail and delete this message. Thank you for your cooperation.

Any views or opinions presented are solely those of the author and do not necessarily represent those of Raytheon unless specifically stated. 
Electronic communications including email may be monitored by Raytheon
for operational or business reasons.






From Bennie_R_Thomas at raytheon.com  Wed Jul  7 07:52:34 2010
From: Bennie_R_Thomas at raytheon.com (Bennie Thomas)
Date: Wed, 07 Jul 2010 02:52:34 -0500
Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes randomly
	reboot
In-Reply-To: <1277908521.3158.441.camel@localhost.localdomain>
References: <mailman.20241.1277763744.8372.linux-cluster@redhat.com>	<4C2B5322.9030305@arlut.utexas.edu>
	<1277908521.3158.441.camel@localhost.localdomain>
Message-ID: <4C343242.6000804@raytheon.com>

Hello Steve,

I have the cluster traffic going over a private interface. I know have 
the nodes connected point-to-point.
I have thought about the network traffic, However, with a dedicated 
interface, I should not have this problem.

I have built, numerous Clusters from different vendors. This particular 
cluster is an active/passive. I have noticed
that the passive node in the cluster is the one that reboots the most.

Thanks for any/all input....



Steven Whitehouse wrote:
> Hi,
>
> On Wed, 2010-06-30 at 09:22 -0500, Randy Zagar wrote:
>   
>> Yes.  My experience is that you can't currently nfs-export *any* GFS or 
>> GFS2 filesystems.
>>
>>     
> You can, but there are only a fairly small number of configurations
> which will actually work from the larger number of possible
> configurations. We do hope to expand that a bit in the future, but for
> the time being its best to stick to a active/passive failover export
> which is not mixed with any other protocol (Samba) or any local
> applications.
>
>   
>> Exporting EXT3/EXT4 filesystems, however, doesn't appear to be a problem.
>>
>> -Randy Zagar <zagar at arlut.utexas.edu>
>>
>> On 06/28/2010 05:22 PM, linux-cluster-request at redhat.com wrote:
>>     
>>> From: Bennie Thomas<Bennie_R_Thomas at raytheon.com>
>>> Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes randomly
>>> 	reboot
>>> Message-ID:<4C290BE0.3090707 at raytheon.com>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> I currently have 2 DL380 G6 with and HP MSA2312 disk array. Running
>>> Redhat 5u4 64bit.  I have a quorum disk.  I use the Cluster as an
>>> Active/passive NFS Cluster
>>> The problem I am having is one or both of the nodes will randomly
>>> reboot. Has anyone experienced this problem
>>>
>>>       
> Is the node being fenced? This might be down to excessive network
> traffic blocking the cluster traffic and making it appear as if the node
> is down when it isn't, or something similar to that. Do you get any log
> messages?
>
> Steve.
>
>   
>>>    
>>>       
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>     
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>   

-- 
Bennie Thomas
Sr. Information Systems Technologist II
Raytheon Company

972.205.4126
972.205.6363 fax
888.347.1660 pager
Bennie_R_Thomas at raytheon.com


DISCLAIMER: This message contains information that may be confidential and privileged. Unless you are the addressee (or authorized to receive mail for the addressee), you should not use, copy or disclose to anyone this message or any information contained in this message. If you have received this message in error, please so advise the sender by reply e-mail and delete this message. Thank you for your cooperation.

Any views or opinions presented are solely those of the author and do not necessarily represent those of Raytheon unless specifically stated. 
Electronic communications including email may be monitored by Raytheon
for operational or business reasons.






From rpeterso at redhat.com  Wed Jul  7 13:59:16 2010
From: rpeterso at redhat.com (Bob Peterson)
Date: Wed, 7 Jul 2010 09:59:16 -0400 (EDT)
Subject: [Linux-cluster] CLVM/GFS hangs while rebuilding an AoE RAID5
 LUN, anyone experienced that before? any clue?
In-Reply-To: <8299E832-4699-4F13-B4F8-A56F0D2C4F88@auckland.ac.nz>
Message-ID: <1242538746.62831278511156951.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com>

----- "Abraham Alawi" <a.alawi at auckland.ac.nz> wrote:
| Thanks Steve, Yes, sadly I can confirm the file system has been
| corrupted but I still don't understand why I/O will stop flowing at
| the LVM level (& doesn't fence it either) and why fsck keeps crashing
| without a useful error message, is there any signal I can send to
| gfs_fsck to by pass certain stages? Also to speed up the fsck process,
| I was thinking of utilizing the RAM and increase the read_ahead
| parameter (hdparm -a) of the PV device (an AoE device) by 1GB since
| that will hugely optimize the sequential read and fscking is mostly a
| sequential read process and very bit of writings, what do you think?
| 
| Herein the tail of the last fsck log file:
| (metawalk.c:516)	Extended attributes exist for inode #34020861.
| (metawalk.c:413)	Checking EA leaf block #34020862.
| (pass1.c:485)	Setting block #34020862 to eattr block
| (pass1.c:907)	Checking metadata block 34020862
| (pass1.c:923)	Metadata block 34020862 not an inode or free metadata
| (pass1.c:907)	Checking metadata block 34020863
| (link.c:22)	Setting link count to 1 for 34020863
| (metawalk.c:516)	Extended attributes exist for inode #34020863.
| (metawalk.c:413)	Checking EA leaf block #34020864.
| (pass1.c:485)	Setting block #34020864 to eattr block
| (pass1.c:907)	Checking metadata block 34020864
| (pass1.c:923)	Metadata block 34020864 not an inode or free metadata
| (pass1.c:907)	Checking metadata block 34020865
| (link.c:22)	Setting link count to 1 for 34020865
| (pass1.c:213)	Setting 34020917 to data block
| (pass1.c:213)	Setting 34020918 to data block
| (pass1.c:213)	Setting 34020919 to data block
| (pass1.c:213)	Setting 34020920 to data block
| (metawalk.c:516)	Extended attributes exist for inode #34020865.
| (metawalk.c:413)	Checking EA leaf block #34020866.
| (pass1.c:485)	Setting block #34020866 to eattr block
| (pass1.c:907)	Checking metadata block 34020866
| (pass1.c:923)	Metadata block 34020866 not an inode or free metadata
| (pass1.c:907)	Checking metadata block 34020867
| 
| 
| Thanks,
| 
|   -- Abraham

Hi Abraham,

The only way to tell that is to examine the metadata.  Can you
use "gfs2_edit savemeta" to save the gfs metadata to a file, and
send it to me?  (gfs2_edit works on gfs1 and gfs2 metadata).
I'm willing to take a look at it to see what's wrong.

Be sure you use a very recent version of gfs2_edit though, because
some older versions don't always save everything they should.

If gfs2_edit crashes during the save, there's no good way to solve
this other than to debug gfs_fsck with the gdb debugger.  The gfs_fsck
program would have to be compiled with the -g option for debugging.

Regards,

Bob Peterson
Red Hat File Systems



From rhurst at bidmc.harvard.edu  Thu Jul  8 14:17:31 2010
From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu)
Date: Thu, 8 Jul 2010 10:17:31 -0400
Subject: [Linux-cluster] RHEL 5.5 fail-over policy with shared resources
In-Reply-To: <1242538746.62831278511156951.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com>
References: <8299E832-4699-4F13-B4F8-A56F0D2C4F88@auckland.ac.nz>
	<1242538746.62831278511156951.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com>
Message-ID: <50168EC934B8D64AA8D8DD37F840F3DE0562E63334@EVS2CCR.its.caregroup.org>

An (unverified) issue occur to us yesterday when moving a service from a RHEL 4.7 cluster into a new, but existing, RHEL 5.5 cluster:

There was an oversight in regards to one of its GFS Shared Resources; its superblock was not re-written with the new cluster name, so it could not mount on the new cluster.  Starting the cluster service would fail, however, its policy is to remain "disabled", and not fail-over to another domain member.  But it attempted to startup anyways.

That's where we ran into some unexpected trouble.

We attempted to enable the package again and it went along happily mounting private resources, ext3 filesystems, that were now partially mounting on a fail-over domain.  Fortunately for us, the Linux kernel detected some underlying blocks being modifed and immediately switched those mounts to read-only.  We discovered the issue, umount'ed all the filesystems and ran e2fsck -- which happily repaired a few mishaps.

Fixing the GFS superblock solved our problem, but we are curious if this is a "feature" or a "bug" with the fail-over attempt.  With private resources failing, we don't get this recovery effort on another server -- it just fails which is the way we want it to do.

This is still our observation, and we will at some point stage this scenario at our D.R. site for testing, but I thought I bounce it off this list.

Pertinent log information follows:

Jul  7 18:59:16 columbia clurgmgrd[20213]: <notice> Starting disabled service service:TOBY 
Jul  7 19:00:04 columbia clurgmgrd: [20213]: <err> 'mount -t gfs -o noatime /dev/mapper/VGCCC-lvoltobywav /toby/wav' failed, error=1 
Jul  7 19:00:13 columbia clurgmgrd[20213]: <notice> start on clusterfs "CCC-lvoltobywav" returned 2 (invalid argument(s)) 
Jul  7 19:00:13 columbia clurgmgrd[20213]: <warning> #68: Failed to start service:TOBY; return value: 1 
Jul  7 19:00:13 columbia clurgmgrd[20213]: <notice> Stopping service service:TOBY 

Jul  7 19:00:46 columbia clurgmgrd[20213]: <notice> Service service:TOBY is recovering 
Jul  7 19:00:46 columbia clurgmgrd[20213]: <warning> #71: Relocating failed service service:TOBY 
Jul  7 19:01:44 columbia clurgmgrd[20213]: <alert> #2: Service service:TOBY returned failure code.  Last Owner: zodiac 
Jul  7 19:01:44 columbia clurgmgrd[20213]: <alert> #4: Administrator intervention required. 
Jul  7 19:01:45 columbia clurgmgrd[20213]: <alert> #2: Service service:TOBY returned failure code.  Last Owner: zodiac 
Jul  7 19:01:45 columbia clurgmgrd[20213]: <alert> #4: Administrator intervention required. 

Jul  7 19:04:05 columbia clurgmgrd[20213]: <notice> Stopping service service:TOBY 
Jul  7 19:04:05 columbia clurgmgrd[20213]: <notice> Service service:TOBY is disabled



From virginian at blueyonder.co.uk  Fri Jul  9 20:43:00 2010
From: virginian at blueyonder.co.uk (Virginian)
Date: Fri, 9 Jul 2010 21:43:00 +0100
Subject: [Linux-cluster] Fedora 13 Clustering
Message-ID: <37F244B6767942AD9908B56800D317C8@laptop1>

Hi All,

I've played around with Red Hat Cluster Suite on Centos 5 and found plenty of information on how to install and configure the cluster and nodes. I have been trawling Google and other search engines trying to find accurate instructions for running Cluster Suite on Fedora 13. So far I have pretty much drawn a blank. What I am looking for is instructions on how to configure a 2 node cluster with iSCSI shared storage for GFS2, the aim being to run virtualised Linux guests under KVM and have the ability to migrate the guests from one node to the other in the event of a node failure. From what I have read, what I understood to be RHCS in RHEL 5 / Centos 5 has changed quite significantly. It looks like openais / corosync are new components and ccs has largely disappeared. If I could find some information on how to configure corosync, openais, amf etc. to achieve a fairly simple 2 node cluster as described I think I would be ok. I've tried googling for this under Fedora 13 but so far drawn a blank. Has anybody else set up a cluster similar to the above under Fedora 13?

Regards

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100709/2aa0d1c2/attachment.htm>

From linux at alteeve.com  Sun Jul 11 05:08:38 2010
From: linux at alteeve.com (Digimer)
Date: Sun, 11 Jul 2010 01:08:38 -0400
Subject: [Linux-cluster] The purpose of /usr/lib64/stonith/plugins/external
Message-ID: <4C3951D6.40807@alteeve.com>

Hi all,

   I've got a home-grown fence device I am trying to add support for 
Fedora 13 (specifically, corosync 1.2.3, cluster-glue 1.0.2 et. al.).

   Under CentOS, I wrote an agent and placed it in /usr/sbin/ and that 
worked fine. Now though, there is the 
/usr/lib64/stonith/plugins/external/ directory which seems to also be 
used. I noticed this when I used 'stonith -L'.

   I'm still getting familiar with the changes to the cluster stack, so 
I apologize if this has been asked before. :)

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From linux at alteeve.com  Mon Jul 12 04:57:15 2010
From: linux at alteeve.com (Digimer)
Date: Mon, 12 Jul 2010 00:57:15 -0400
Subject: [Linux-cluster] corosync issue with two interface directives
Message-ID: <4C3AA0AB.4060800@alteeve.com>

Hi all,

   I've got two 'interface {}' directives in corosync. When I pull the 
cable from the interface on ring 1, nothing happens (no log entries, 
zilch). When I put it back and the pull the cable on ring 0, it fails 
completely and fences. Is there something I am missing?

Here is the corosync.conf file I am using:

------------------------------------------------------------------------
compatibility: whitetank

# Totem Protocol options.
totem {
	version: 2
	secauth: off
	threads: 0
	rrp_mode: passive
	interface {
		# This is the back-channel subnet, which is the primary
		# network for the totem protocol.
		ringnumber: 0
		bindnetaddr: 10.0.1.0
		mcastaddr: 226.94.1.1
		mcastport: 5405
	}
	interface {
		# This is the storage network, which acts as a
		# secondary, backup network for the totem protocol.
		ringnumber: 1
		bindnetaddr: 10.0.0.0
		mcastaddr: 227.94.1.1
		mcastport: 5406
	}
}

logging {
	to_syslog: yes
         fileline: off
         to_stderr: yes
         to_logfile: yes
         to_syslog: yes
         logfile: /var/log/corosync.log
         debug: off
         timestamp: on
}

amf {
         mode: disabled
}
------------------------------------------------------------------------

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From sdake at redhat.com  Mon Jul 12 05:12:10 2010
From: sdake at redhat.com (Steven Dake)
Date: Sun, 11 Jul 2010 22:12:10 -0700
Subject: [Linux-cluster] corosync issue with two interface directives
In-Reply-To: <4C3AA0AB.4060800@alteeve.com>
References: <4C3AA0AB.4060800@alteeve.com>
Message-ID: <4C3AA42A.7020408@redhat.com>

On 07/11/2010 09:57 PM, Digimer wrote:
> Hi all,
>
> I've got two 'interface {}' directives in corosync. When I pull the
> cable from the interface on ring 1, nothing happens (no log entries,
> zilch). When I put it back and the pull the cable on ring 0, it fails
> completely and fences. Is there something I am missing?
>
> Here is the corosync.conf file I am using:
>
> ------------------------------------------------------------------------
> compatibility: whitetank
>
> # Totem Protocol options.
> totem {
> version: 2
> secauth: off
> threads: 0
> rrp_mode: passive
> interface {
> # This is the back-channel subnet, which is the primary
> # network for the totem protocol.
> ringnumber: 0
> bindnetaddr: 10.0.1.0
> mcastaddr: 226.94.1.1
> mcastport: 5405
> }
> interface {
> # This is the storage network, which acts as a
> # secondary, backup network for the totem protocol.
> ringnumber: 1
> bindnetaddr: 10.0.0.0
> mcastaddr: 227.94.1.1
> mcastport: 5406
> }
> }

What are the interface netmasks?

>
> logging {
> to_syslog: yes
> fileline: off
> to_stderr: yes
> to_logfile: yes
> to_syslog: yes
> logfile: /var/log/corosync.log
> debug: off
> timestamp: on
> }
>
> amf {
> mode: disabled
> }
> ------------------------------------------------------------------------
>



From andrew at beekhof.net  Mon Jul 12 08:47:57 2010
From: andrew at beekhof.net (Andrew Beekhof)
Date: Mon, 12 Jul 2010 10:47:57 +0200
Subject: [Linux-cluster] The purpose of
	/usr/lib64/stonith/plugins/external
In-Reply-To: <4C3951D6.40807@alteeve.com>
References: <4C3951D6.40807@alteeve.com>
Message-ID: <AANLkTikoXAlARw73PKhEMkQ1S7zwx2QPtyaHM-KetYjH@mail.gmail.com>

On Sun, Jul 11, 2010 at 7:08 AM, Digimer <linux at alteeve.com> wrote:
> Hi all,
>
> ?I've got a home-grown fence device I am trying to add support for Fedora 13
> (specifically, corosync 1.2.3, cluster-glue 1.0.2 et. al.).
>
> ?Under CentOS, I wrote an agent and placed it in /usr/sbin/ and that worked
> fine.

Thats fine then.  Pacemaker can use RHCS devices too.

> Now though, there is the /usr/lib64/stonith/plugins/external/
> directory which seems to also be used. I noticed this when I used 'stonith
> -L'.

This is for the old heartbeat-style fencing devices.  You can ignore it.

>
> ?I'm still getting familiar with the changes to the cluster stack, so I
> apologize if this has been asked before. :)
>
> --
> Digimer
> E-Mail: ? ? ? ? linux at alteeve.com
> AN!Whitepapers: http://alteeve.com
> Node Assassin: ?http://nodeassassin.org
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From linux at alteeve.com  Mon Jul 12 13:13:04 2010
From: linux at alteeve.com (Digimer)
Date: Mon, 12 Jul 2010 09:13:04 -0400
Subject: [Linux-cluster] The purpose
	of	/usr/lib64/stonith/plugins/external
In-Reply-To: <AANLkTikoXAlARw73PKhEMkQ1S7zwx2QPtyaHM-KetYjH@mail.gmail.com>
References: <4C3951D6.40807@alteeve.com>
	<AANLkTikoXAlARw73PKhEMkQ1S7zwx2QPtyaHM-KetYjH@mail.gmail.com>
Message-ID: <4C3B14E0.8060007@alteeve.com>

On 10-07-12 04:47 AM, Andrew Beekhof wrote:
> On Sun, Jul 11, 2010 at 7:08 AM, Digimer<linux at alteeve.com>  wrote:
>> Hi all,
>>
>>   I've got a home-grown fence device I am trying to add support for Fedora 13
>> (specifically, corosync 1.2.3, cluster-glue 1.0.2 et. al.).
>>
>>   Under CentOS, I wrote an agent and placed it in /usr/sbin/ and that worked
>> fine.
>
> Thats fine then.  Pacemaker can use RHCS devices too.
>
>> Now though, there is the /usr/lib64/stonith/plugins/external/
>> directory which seems to also be used. I noticed this when I used 'stonith
>> -L'.
>
> This is for the old heartbeat-style fencing devices.  You can ignore it.

That's what I was starting to think, and ended up doing just that, 
ignoring it.

If I may ask; The "Cluster from Scratch" manual for Fedora 11/12 
suggests starting corosync directly. This kept failing because it didn't 
find a configured stonith device. However, I went back and created a 
cluster.conf and then started cman and everything was (largely) fine.

Is there a document or similar that more clearly defines what is the old 
HA-Linux components versus the RHCS? Or further, what is the proper 
method of building a cluster based on RHCS using corosync/pacemaker?

Thanks kindly!

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From linux at alteeve.com  Mon Jul 12 13:20:11 2010
From: linux at alteeve.com (Digimer)
Date: Mon, 12 Jul 2010 09:20:11 -0400
Subject: [Linux-cluster] corosync issue with two interface directives
In-Reply-To: <4C3AA42A.7020408@redhat.com>
References: <4C3AA0AB.4060800@alteeve.com> <4C3AA42A.7020408@redhat.com>
Message-ID: <4C3B168B.7080706@alteeve.com>

On 10-07-12 01:12 AM, Steven Dake wrote:
> On 07/11/2010 09:57 PM, Digimer wrote:
>> Hi all,
>>
>> I've got two 'interface {}' directives in corosync. When I pull the
>> cable from the interface on ring 1, nothing happens (no log entries,
>> zilch). When I put it back and the pull the cable on ring 0, it fails
>> completely and fences. Is there something I am missing?
>>
>> Here is the corosync.conf file I am using:
>>
>> ------------------------------------------------------------------------
>> compatibility: whitetank
>>
>> # Totem Protocol options.
>> totem {
>> version: 2
>> secauth: off
>> threads: 0
>> rrp_mode: passive
>> interface {
>> # This is the back-channel subnet, which is the primary
>> # network for the totem protocol.
>> ringnumber: 0
>> bindnetaddr: 10.0.1.0
>> mcastaddr: 226.94.1.1
>> mcastport: 5405
>> }
>> interface {
>> # This is the storage network, which acts as a
>> # secondary, backup network for the totem protocol.
>> ringnumber: 1
>> bindnetaddr: 10.0.0.0
>> mcastaddr: 227.94.1.1
>> mcastport: 5406
>> }
>> }
>
> What are the interface netmasks?

/24 (255.255.255.0). I thought they had to be on separate networks. Or 
does the netmask need to be specified in the corosync.conf file as well?

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From jayfitzpatrick at gmail.com  Mon Jul 12 13:37:00 2010
From: jayfitzpatrick at gmail.com (Jason Fitzpatrick)
Date: Mon, 12 Jul 2010 14:37:00 +0100
Subject: [Linux-cluster] Fedora 13 Clustering
In-Reply-To: <37F244B6767942AD9908B56800D317C8@laptop1>
References: <37F244B6767942AD9908B56800D317C8@laptop1>
Message-ID: <AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>

Hi John..

Normally I do not agree with replying to posts without anything
constructive to say but I thought that I should in this case.

I have to admit that I ran into the same issues as you, I built a RHEL
5.5 cluster and decided to give the FC13 path a try, Tried to use Luci
/ Ricci to configure the nodes and had no luck even managing to log
into Luci, no documents / man pages included with the packages and the
readme file that was included was the same one that was included with
the old source files, and had not been updated since.

In the end I gave up and went back to RHEL.

(that said you can go the heartbeat route, this I got to work when
using oracles ocfs as a shared data store on drbd)

Jay

On 9 July 2010 21:43, Virginian <virginian at blueyonder.co.uk> wrote:
> Hi All,
>
> I've played around with Red Hat Cluster Suite on Centos 5 and found plenty
> of information on how to install and configure the cluster and nodes. I have
> been trawling Google and other search engines trying to find accurate
> instructions for running Cluster Suite on Fedora 13. So far I have pretty
> much drawn a blank. What I am looking for is instructions on how to
> configure a 2 node cluster with iSCSI shared storage for GFS2, the aim being
> to run virtualised Linux guests under KVM and have the ability to migrate
> the guests from one node to the other in the event of a node failure. From
> what I have read, what I understood to be RHCS in RHEL 5 / Centos 5 has
> changed quite significantly. It looks like openais / corosync are new
> components and ccs has largely disappeared. If I could find some information
> on how to configure corosync, openais, amf etc. to achieve a fairly simple 2
> node cluster as described I think I would be ok. I've tried googling for
> this under Fedora 13 but so far drawn a blank. Has anybody else set up a
> cluster similar to the above under Fedora 13?
>
> Regards
>
> John
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 

"The only difference between saints and sinners is that every saint
has a past while every sinner has a future. "
? Oscar Wilde



From gordan at bobich.net  Mon Jul 12 13:47:28 2010
From: gordan at bobich.net (Gordan Bobic)
Date: Mon, 12 Jul 2010 14:47:28 +0100
Subject: [Linux-cluster] Fedora 13 Clustering
In-Reply-To: <AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>
References: <37F244B6767942AD9908B56800D317C8@laptop1>
	<AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>
Message-ID: <1278942448.2679.2.camel@thinkpad>

Maybe I'm missing what exactly you guys are talking about, but
everything I can think of in RHCS is configured via cluster.conf. My
cluster.conf files from RHEL5 work just fine on RHEL6 beta, which is
virtually identical to F12/F13. I don't see why you would have any
problems getting RHCS to work on F13 using the same instructions you
would follow for RHEL/CentOS 5.

Gordan

On Mon, 2010-07-12 at 14:37 +0100, Jason Fitzpatrick wrote:
> Hi John..
> 
> Normally I do not agree with replying to posts without anything
> constructive to say but I thought that I should in this case.
> 
> I have to admit that I ran into the same issues as you, I built a RHEL
> 5.5 cluster and decided to give the FC13 path a try, Tried to use Luci
> / Ricci to configure the nodes and had no luck even managing to log
> into Luci, no documents / man pages included with the packages and the
> readme file that was included was the same one that was included with
> the old source files, and had not been updated since.
> 
> In the end I gave up and went back to RHEL.
> 
> (that said you can go the heartbeat route, this I got to work when
> using oracles ocfs as a shared data store on drbd)
> 
> Jay
> 
> On 9 July 2010 21:43, Virginian <virginian at blueyonder.co.uk> wrote:
> > Hi All,
> >
> > I've played around with Red Hat Cluster Suite on Centos 5 and found plenty
> > of information on how to install and configure the cluster and nodes. I have
> > been trawling Google and other search engines trying to find accurate
> > instructions for running Cluster Suite on Fedora 13. So far I have pretty
> > much drawn a blank. What I am looking for is instructions on how to
> > configure a 2 node cluster with iSCSI shared storage for GFS2, the aim being
> > to run virtualised Linux guests under KVM and have the ability to migrate
> > the guests from one node to the other in the event of a node failure. From
> > what I have read, what I understood to be RHCS in RHEL 5 / Centos 5 has
> > changed quite significantly. It looks like openais / corosync are new
> > components and ccs has largely disappeared. If I could find some information
> > on how to configure corosync, openais, amf etc. to achieve a fairly simple 2
> > node cluster as described I think I would be ok. I've tried googling for
> > this under Fedora 13 but so far drawn a blank. Has anybody else set up a
> > cluster similar to the above under Fedora 13?
> >
> > Regards
> >
> > John
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> 
> 
> 




From virginian at blueyonder.co.uk  Mon Jul 12 14:06:03 2010
From: virginian at blueyonder.co.uk (Virginian)
Date: Mon, 12 Jul 2010 15:06:03 +0100
Subject: [Linux-cluster] Fedora 13 Clustering
References: <37F244B6767942AD9908B56800D317C8@laptop1><AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>
	<1278942448.2679.2.camel@thinkpad>
Message-ID: <134A6BB8530549C8828D857491790777@Desktop>

Hi Gordon,

It looks to me like things have changed quite a bit in F13 compared to RHEL 
/ Centos 5. I agree, cluster.conf  looks the same but there is a new 
component called Corosync and ccs seems to have gone.

Traditionally I have used Conga to configure the Cluster (although not 
without problems at times it has to be said). However, in F13 there is no 
"luci_admin" utility to set the password and Luci isn't accessible from a 
web browser. I don't mind configuring things by hand if that's the only way 
to do it, I was just looking to be pointed in the right direction really. I 
have received some help offline from a member of this list which has proved 
invaluable. I have now managed to get a basic cluster up and running on F13. 
Our fellow list member is working on a document which I think will be of 
enormous help to others when completed (in fact it is already a useful doc).

The old docs specified which ports / protocols needed to be opened up in 
iptables but as I haven't sussed out what's needed under F13 I am running 
with iptables stopped for now. If anybody has this information I would be 
very grateful to receive it.

Thanks & regards

John

----- Original Message ----- 
From: "Gordan Bobic" <gordan at bobich.net>
To: "linux clustering" <linux-cluster at redhat.com>
Sent: Monday, July 12, 2010 2:47 PM
Subject: Re: [Linux-cluster] Fedora 13 Clustering


> Maybe I'm missing what exactly you guys are talking about, but
> everything I can think of in RHCS is configured via cluster.conf. My
> cluster.conf files from RHEL5 work just fine on RHEL6 beta, which is
> virtually identical to F12/F13. I don't see why you would have any
> problems getting RHCS to work on F13 using the same instructions you
> would follow for RHEL/CentOS 5.
>
> Gordan
>
> On Mon, 2010-07-12 at 14:37 +0100, Jason Fitzpatrick wrote:
>> Hi John..
>>
>> Normally I do not agree with replying to posts without anything
>> constructive to say but I thought that I should in this case.
>>
>> I have to admit that I ran into the same issues as you, I built a RHEL
>> 5.5 cluster and decided to give the FC13 path a try, Tried to use Luci
>> / Ricci to configure the nodes and had no luck even managing to log
>> into Luci, no documents / man pages included with the packages and the
>> readme file that was included was the same one that was included with
>> the old source files, and had not been updated since.
>>
>> In the end I gave up and went back to RHEL.
>>
>> (that said you can go the heartbeat route, this I got to work when
>> using oracles ocfs as a shared data store on drbd)
>>
>> Jay
>>
>> On 9 July 2010 21:43, Virginian <virginian at blueyonder.co.uk> wrote:
>> > Hi All,
>> >
>> > I've played around with Red Hat Cluster Suite on Centos 5 and found 
>> > plenty
>> > of information on how to install and configure the cluster and nodes. I 
>> > have
>> > been trawling Google and other search engines trying to find accurate
>> > instructions for running Cluster Suite on Fedora 13. So far I have 
>> > pretty
>> > much drawn a blank. What I am looking for is instructions on how to
>> > configure a 2 node cluster with iSCSI shared storage for GFS2, the aim 
>> > being
>> > to run virtualised Linux guests under KVM and have the ability to 
>> > migrate
>> > the guests from one node to the other in the event of a node failure. 
>> > From
>> > what I have read, what I understood to be RHCS in RHEL 5 / Centos 5 has
>> > changed quite significantly. It looks like openais / corosync are new
>> > components and ccs has largely disappeared. If I could find some 
>> > information
>> > on how to configure corosync, openais, amf etc. to achieve a fairly 
>> > simple 2
>> > node cluster as described I think I would be ok. I've tried googling 
>> > for
>> > this under Fedora 13 but so far drawn a blank. Has anybody else set up 
>> > a
>> > cluster similar to the above under Fedora 13?
>> >
>> > Regards
>> >
>> > John
>> > --
>> > Linux-cluster mailing list
>> > Linux-cluster at redhat.com
>> > https://www.redhat.com/mailman/listinfo/linux-cluster
>> >
>>
>>
>>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster 




From gordan at bobich.net  Mon Jul 12 14:25:11 2010
From: gordan at bobich.net (Gordan Bobic)
Date: Mon, 12 Jul 2010 15:25:11 +0100
Subject: [Linux-cluster] Fedora 13 Clustering
In-Reply-To: <134A6BB8530549C8828D857491790777@Desktop>
References: <37F244B6767942AD9908B56800D317C8@laptop1>
	<AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>
	<1278942448.2679.2.camel@thinkpad>
	<134A6BB8530549C8828D857491790777@Desktop>
Message-ID: <1278944711.2679.12.camel@thinkpad>

On Mon, 2010-07-12 at 15:06 +0100, Virginian wrote:
> Hi Gordon,
> 
> It looks to me like things have changed quite a bit in F13 compared to RHEL 
> / Centos 5. I agree, cluster.conf  looks the same but there is a new 
> component called Corosync and ccs seems to have gone.

According to this, ccs_tool & co. are still in the F13 cman package:
http://rpm.pbone.net/index.php3/stat/4/idpl/14125020/dir/fedora_13/com/cman-3.0.12-2.fc13.x86_64.rpm.html

> Traditionally I have used Conga to configure the Cluster (although not 
> without problems at times it has to be said). However, in F13 there is no 
> "luci_admin" utility to set the password and Luci isn't accessible from a 
> web browser. I don't mind configuring things by hand if that's the only way 
> to do it, I was just looking to be pointed in the right direction really.

I can't say I've ever used the GUIs to configure things - I've always
found they are at best buggy, and more often than not just get in the
way. I for one am glad to see the back of the all system-config
packages.

> The old docs specified which ports / protocols needed to be opened up in 
> iptables but as I haven't sussed out what's needed under F13 I am running 
> with iptables stopped for now. If anybody has this information I would be 
> very grateful to receive it.

I haven't looked, but a few minutes with tcpdump or iptables -j LOG on a
quiet network should yield the information you seek.

Gordan



From cmaiolino at redhat.com  Mon Jul 12 14:44:31 2010
From: cmaiolino at redhat.com (Carlos Maiolino)
Date: Mon, 12 Jul 2010 11:44:31 -0300
Subject: [Linux-cluster] Fedora 13 Clustering
In-Reply-To: <1278944711.2679.12.camel@thinkpad>
References: <37F244B6767942AD9908B56800D317C8@laptop1>
	<AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>
	<1278942448.2679.2.camel@thinkpad>
	<134A6BB8530549C8828D857491790777@Desktop>
	<1278944711.2679.12.camel@thinkpad>
Message-ID: <20100712144431.GC2343@andromeda.usersys.redhat.com>

On Mon, Jul 12, 2010 at 03:25:11PM +0100, Gordan Bobic wrote:
> On Mon, 2010-07-12 at 15:06 +0100, Virginian wrote:
> > Hi Gordon,
> > 
> > It looks to me like things have changed quite a bit in F13 compared to RHEL 
> > / Centos 5. I agree, cluster.conf  looks the same but there is a new 
> > component called Corosync and ccs seems to have gone.
> 
> According to this, ccs_tool & co. are still in the F13 cman package:
> http://rpm.pbone.net/index.php3/stat/4/idpl/14125020/dir/fedora_13/com/cman-3.0.12-2.fc13.x86_64.rpm.html
> 
> > Traditionally I have used Conga to configure the Cluster (although not 
> > without problems at times it has to be said). However, in F13 there is no 
> > "luci_admin" utility to set the password and Luci isn't accessible from a 
> > web browser. I don't mind configuring things by hand if that's the only way 
> > to do it, I was just looking to be pointed in the right direction really.
> 
> I can't say I've ever used the GUIs to configure things - I've always
> found they are at best buggy, and more often than not just get in the
> way. I for one am glad to see the back of the all system-config
> packages.
> 
> > The old docs specified which ports / protocols needed to be opened up in 
> > iptables but as I haven't sussed out what's needed under F13 I am running 
> > with iptables stopped for now. If anybody has this information I would be 
> > very grateful to receive it.
> 
> I haven't looked, but a few minutes with tcpdump or iptables -j LOG on a
> quiet network should yield the information you seek.
> 
> Gordan
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

Hi guys,

most of things of cluster in Fedora has been changed, comparing with RHEL5.

We are using corosync for membership now, and (IIRC) openais is must to maintain some compatibility with APIs. CMAN still controls the quorum system, and cluster.conf continues to be the configuration file.

If you want to learn more about how to create a cluster using Fedora 13, I believe a good idea is to learn about corosync at http://www.corosync.org, and cman at http://sources.redhat.com/cluster/wiki/

-- 
---

Best Regards

Carlos Eduardo Maiolino
Software Maintenance Engineer
Red Hat - Global Support Services



From mad at wol.de  Mon Jul 12 16:20:24 2010
From: mad at wol.de (Marc - A. Dahlhaus)
Date: Mon, 12 Jul 2010 18:20:24 +0200
Subject: [Linux-cluster] Fedora 13 Clustering
In-Reply-To: <134A6BB8530549C8828D857491790777@Desktop>
References: <37F244B6767942AD9908B56800D317C8@laptop1>
	<AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>
	<1278942448.2679.2.camel@thinkpad>
	<134A6BB8530549C8828D857491790777@Desktop>
Message-ID: <1278951624.19544.387.camel@marc>

Am Montag, den 12.07.2010, 15:06 +0100 schrieb Virginian:
> Hi Gordon,
> 
> It looks to me like things have changed quite a bit in F13 compared to RHEL 
> / Centos 5. I agree, cluster.conf  looks the same but there is a new 
> component called Corosync and ccs seems to have gone.

Take a look here:

http://people.redhat.com/ccaulfie/docs/Whither%20cman.pdf

>From RHEL5 to RHEL6 you change from cluster 2.X to cluster 3.X

Documentation can be found here:

http://sources.redhat.com/cluster/wiki/

> Traditionally I have used Conga to configure the Cluster (although not 
> without problems at times it has to be said). However, in F13 there is no 
> "luci_admin" utility to set the password and Luci isn't accessible from a 
> web browser. I don't mind configuring things by hand if that's the only way 
> to do it, I was just looking to be pointed in the right direction really. I 
> have received some help offline from a member of this list which has proved 
> invaluable. I have now managed to get a basic cluster up and running on F13. 
> Our fellow list member is working on a document which I think will be of 
> enormous help to others when completed (in fact it is already a useful doc).

Docs on Conga (Luci and Ricci) are here:

http://sources.redhat.com/cluster/conga/

I don't know if this is newer because i made a large way around even
trying to use conga after i spotted that it utilizes turbogears (which
has a huge dependency tree just to get anything out of it).

> The old docs specified which ports / protocols needed to be opened up in 
> iptables but as I haven't sussed out what's needed under F13 I am running 
> with iptables stopped for now. If anybody has this information I would be 
> very grateful to receive it.

The FAQs on http://corosync.org/ and http://openais.org/ could help
here. Unfortunately the FAQ entry about firewalls has no content.

Bug?!

A "netstat -tulpen" on a root shell should show the ports that corosync
and openais listening on. They use a single multicast port if i'm not
mistaken.

> Thanks & regards
> 
> John

Marc



From sdake at redhat.com  Mon Jul 12 17:55:49 2010
From: sdake at redhat.com (Steven Dake)
Date: Mon, 12 Jul 2010 10:55:49 -0700
Subject: [Linux-cluster] Fedora 13 Clustering
In-Reply-To: <134A6BB8530549C8828D857491790777@Desktop>
References: <37F244B6767942AD9908B56800D317C8@laptop1><AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>	<1278942448.2679.2.camel@thinkpad>
	<134A6BB8530549C8828D857491790777@Desktop>
Message-ID: <4C3B5725.50306@redhat.com>

On 07/12/2010 07:06 AM, Virginian wrote:
> Hi Gordon,
>
> It looks to me like things have changed quite a bit in F13 compared to
> RHEL / Centos 5. I agree, cluster.conf looks the same but there is a new
> component called Corosync and ccs seems to have gone.
>

These changes should be essentially invisible to the typical fedora user 
of the software.



From dfrincu at streamwide.ro  Mon Jul 12 19:17:42 2010
From: dfrincu at streamwide.ro (Dan Frincu)
Date: Mon, 12 Jul 2010 22:17:42 +0300
Subject: [Linux-cluster] Fedora 13 Clustering
In-Reply-To: <1278951624.19544.387.camel@marc>
References: <37F244B6767942AD9908B56800D317C8@laptop1>	<AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>	<1278942448.2679.2.camel@thinkpad>	<134A6BB8530549C8828D857491790777@Desktop>
	<1278951624.19544.387.camel@marc>
Message-ID: <4C3B6A56.1020105@streamwide.ro>

Hi,

Marc - A. Dahlhaus wrote:
> Am Montag, den 12.07.2010, 15:06 +0100 schrieb Virginian:
>   
>> Hi Gordon,
>>
>> It looks to me like things have changed quite a bit in F13 compared to RHEL 
>> / Centos 5. I agree, cluster.conf  looks the same but there is a new 
>> component called Corosync and ccs seems to have gone.
>>     
>
> Take a look here:
>
> http://people.redhat.com/ccaulfie/docs/Whither%20cman.pdf
>
> >From RHEL5 to RHEL6 you change from cluster 2.X to cluster 3.X
>
> Documentation can be found here:
>
> http://sources.redhat.com/cluster/wiki/
>
>   
>> Traditionally I have used Conga to configure the Cluster (although not 
>> without problems at times it has to be said). However, in F13 there is no 
>> "luci_admin" utility to set the password and Luci isn't accessible from a 
>> web browser. I don't mind configuring things by hand if that's the only way 
>> to do it, I was just looking to be pointed in the right direction really. I 
>> have received some help offline from a member of this list which has proved 
>> invaluable. I have now managed to get a basic cluster up and running on F13. 
>> Our fellow list member is working on a document which I think will be of 
>> enormous help to others when completed (in fact it is already a useful doc).
>>     
>
> Docs on Conga (Luci and Ricci) are here:
>
> http://sources.redhat.com/cluster/conga/
>
> I don't know if this is newer because i made a large way around even
> trying to use conga after i spotted that it utilizes turbogears (which
> has a huge dependency tree just to get anything out of it).
>
>   
>> The old docs specified which ports / protocols needed to be opened up in 
>> iptables but as I haven't sussed out what's needed under F13 I am running 
>> with iptables stopped for now. If anybody has this information I would be 
>> very grateful to receive it.
>>     
>
> The FAQs on http://corosync.org/ and http://openais.org/ could help
> here. Unfortunately the FAQ entry about firewalls has no content.
>
> Bug?!
>
> A "netstat -tulpen" on a root shell should show the ports that corosync
> and openais listening on. They use a single multicast port if i'm not
> mistaken.
>
>   
In both corosync and openais, you define the ports and addresses it 
listens on. It's usually recommended that you have redundant links 
between the cluster nodes. Each link is represented by an interface 
declaration, in which you bind to a network card's IP address, you 
define a multicast group (might even require Protocol Independent 
Multicast routing to be enabled in some cases) and a multicast port. All 
nodes that should be on the same cluster should also communicate on the 
same multicast address and port, when using redundant links, both 
multicast addresses and ports must match, meaning each node will have 2 
unique IP addresses, and 2 multicast addresses and ports shared between 
all nodes. Multicast is used for messaging and propagation of 
configuration changes. You can access any node and configure the cluster 
from it, changes are propagated automagically upon committing updates to 
the cluster resource manager, usually Pacemaker. Syntax is as follows.

        interface {
                ringnumber: 0
                bindnetaddr: 192.168.1.1
                mcastaddr: 239.1.1.1
                mcastport: 5405
        }

        interface {
                ringnumber: 1
                bindnetaddr: 192.168.1.2
                mcastaddr: 239.1.1.2
                mcastport: 5405
        }

Regards.
>> Thanks & regards
>>
>> John
>>     
>
> Marc
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>   

-- 
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100712/812186cc/attachment.htm>

From dfrincu at streamwide.ro  Mon Jul 12 19:38:44 2010
From: dfrincu at streamwide.ro (Dan Frincu)
Date: Mon, 12 Jul 2010 22:38:44 +0300
Subject: [Linux-cluster] corosync issue with two interface directives
In-Reply-To: <4C3B168B.7080706@alteeve.com>
References: <4C3AA0AB.4060800@alteeve.com> <4C3AA42A.7020408@redhat.com>
	<4C3B168B.7080706@alteeve.com>
Message-ID: <4C3B6F44.2020002@streamwide.ro>

Just a wild stab in the dark, but both your multicast groups result in 
the same MAC address 01:00:5e:5e:01:01. Now these are supposed to be 
redundant connections, which means different IP/port/MAC's, different 
routes, in case one fails the other one takes over. Try changing the 
multicast groups so that the they differ at the end of the address, 
rather than at the beginning.

You could use 226.94.1.1 for ring 0 and 226.94.1.2 for ring 1. The key 
thing to remember here is that when building a multicast MAC address, 
you only use the low order 23 bits of the multicast IP address, thus 
allowing for overlapping of close high order bits in the multicast IP 
address.

Regards.

Digimer wrote:
> On 10-07-12 01:12 AM, Steven Dake wrote:
>> On 07/11/2010 09:57 PM, Digimer wrote:
>>> Hi all,
>>>
>>> I've got two 'interface {}' directives in corosync. When I pull the
>>> cable from the interface on ring 1, nothing happens (no log entries,
>>> zilch). When I put it back and the pull the cable on ring 0, it fails
>>> completely and fences. Is there something I am missing?
>>>
>>> Here is the corosync.conf file I am using:
>>>
>>> ------------------------------------------------------------------------ 
>>>
>>> compatibility: whitetank
>>>
>>> # Totem Protocol options.
>>> totem {
>>> version: 2
>>> secauth: off
>>> threads: 0
>>> rrp_mode: passive
>>> interface {
>>> # This is the back-channel subnet, which is the primary
>>> # network for the totem protocol.
>>> ringnumber: 0
>>> bindnetaddr: 10.0.1.0
>>> mcastaddr: 226.94.1.1
>>> mcastport: 5405
>>> }
>>> interface {
>>> # This is the storage network, which acts as a
>>> # secondary, backup network for the totem protocol.
>>> ringnumber: 1
>>> bindnetaddr: 10.0.0.0
>>> mcastaddr: 227.94.1.1
>>> mcastport: 5406
>>> }
>>> }
>>
>> What are the interface netmasks?
>
> /24 (255.255.255.0). I thought they had to be on separate networks. Or 
> does the netmask need to be specified in the corosync.conf file as well?
>

-- 
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania



From cfeist at redhat.com  Mon Jul 12 19:41:17 2010
From: cfeist at redhat.com (Chris Feist)
Date: Mon, 12 Jul 2010 14:41:17 -0500
Subject: [Linux-cluster] Fedora 13 Clustering
In-Reply-To: <1278951624.19544.387.camel@marc>
References: <37F244B6767942AD9908B56800D317C8@laptop1>	<AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>	<1278942448.2679.2.camel@thinkpad>	<134A6BB8530549C8828D857491790777@Desktop>
	<1278951624.19544.387.camel@marc>
Message-ID: <4C3B6FDD.7000505@redhat.com>

On 07/12/10 11:20, Marc - A. Dahlhaus wrote:
> Am Montag, den 12.07.2010, 15:06 +0100 schrieb Virginian:
>> Hi Gordon,
>>
>> It looks to me like things have changed quite a bit in F13 compared to RHEL
>> / Centos 5. I agree, cluster.conf  looks the same but there is a new
>> component called Corosync and ccs seems to have gone.
>
> Take a look here:
>
> http://people.redhat.com/ccaulfie/docs/Whither%20cman.pdf
>
>> From RHEL5 to RHEL6 you change from cluster 2.X to cluster 3.X
>
> Documentation can be found here:
>
> http://sources.redhat.com/cluster/wiki/
>
>> Traditionally I have used Conga to configure the Cluster (although not
>> without problems at times it has to be said). However, in F13 there is no
>> "luci_admin" utility to set the password and Luci isn't accessible from a
>> web browser. I don't mind configuring things by hand if that's the only way
>> to do it, I was just looking to be pointed in the right direction really. I
>> have received some help offline from a member of this list which has proved
>> invaluable. I have now managed to get a basic cluster up and running on F13.
>> Our fellow list member is working on a document which I think will be of
>> enormous help to others when completed (in fact it is already a useful doc).
>
> Docs on Conga (Luci and Ricci) are here:
>
> http://sources.redhat.com/cluster/conga/

If you want to use a GUI to configure clustering you will most likely have to 
wait at least until Fedora 14.  Luci is not functional in the current fedora 
(and hasn't worked in the last several releases).

>
> I don't know if this is newer because i made a large way around even
> trying to use conga after i spotted that it utilizes turbogears (which
> has a huge dependency tree just to get anything out of it).
>
>> The old docs specified which ports / protocols needed to be opened up in
>> iptables but as I haven't sussed out what's needed under F13 I am running
>> with iptables stopped for now. If anybody has this information I would be
>> very grateful to receive it.
>
> The FAQs on http://corosync.org/ and http://openais.org/ could help
> here. Unfortunately the FAQ entry about firewalls has no content.
>
> Bug?!
>
> A "netstat -tulpen" on a root shell should show the ports that corosync
> and openais listening on. They use a single multicast port if i'm not
> mistaken.
>
>> Thanks&  regards
>>
>> John
>
> Marc
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From virginian at blueyonder.co.uk  Mon Jul 12 20:14:12 2010
From: virginian at blueyonder.co.uk (Virginian)
Date: Mon, 12 Jul 2010 21:14:12 +0100
Subject: [Linux-cluster] Fedora 13 Clustering
References: <37F244B6767942AD9908B56800D317C8@laptop1>	<AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>	<1278942448.2679.2.camel@thinkpad>	<134A6BB8530549C8828D857491790777@Desktop><1278951624.19544.387.camel@marc>
	<4C3B6FDD.7000505@redhat.com>
Message-ID: <3F87A99EEB2A4974B175A95A391F3BBA@Desktop>

Thanks very much for all the replies. I will read up on the documentation 
links that have been posted....


----- Original Message ----- 
From: "Chris Feist" <cfeist at redhat.com>
To: "linux clustering" <linux-cluster at redhat.com>
Sent: Monday, July 12, 2010 8:41 PM
Subject: Re: [Linux-cluster] Fedora 13 Clustering


> On 07/12/10 11:20, Marc - A. Dahlhaus wrote:
>> Am Montag, den 12.07.2010, 15:06 +0100 schrieb Virginian:
>>> Hi Gordon,
>>>
>>> It looks to me like things have changed quite a bit in F13 compared to 
>>> RHEL
>>> / Centos 5. I agree, cluster.conf  looks the same but there is a new
>>> component called Corosync and ccs seems to have gone.
>>
>> Take a look here:
>>
>> http://people.redhat.com/ccaulfie/docs/Whither%20cman.pdf
>>
>>> From RHEL5 to RHEL6 you change from cluster 2.X to cluster 3.X
>>
>> Documentation can be found here:
>>
>> http://sources.redhat.com/cluster/wiki/
>>
>>> Traditionally I have used Conga to configure the Cluster (although not
>>> without problems at times it has to be said). However, in F13 there is 
>>> no
>>> "luci_admin" utility to set the password and Luci isn't accessible from 
>>> a
>>> web browser. I don't mind configuring things by hand if that's the only 
>>> way
>>> to do it, I was just looking to be pointed in the right direction 
>>> really. I
>>> have received some help offline from a member of this list which has 
>>> proved
>>> invaluable. I have now managed to get a basic cluster up and running on 
>>> F13.
>>> Our fellow list member is working on a document which I think will be of
>>> enormous help to others when completed (in fact it is already a useful 
>>> doc).
>>
>> Docs on Conga (Luci and Ricci) are here:
>>
>> http://sources.redhat.com/cluster/conga/
>
> If you want to use a GUI to configure clustering you will most likely have 
> to wait at least until Fedora 14.  Luci is not functional in the current 
> fedora (and hasn't worked in the last several releases).
>
>>
>> I don't know if this is newer because i made a large way around even
>> trying to use conga after i spotted that it utilizes turbogears (which
>> has a huge dependency tree just to get anything out of it).
>>
>>> The old docs specified which ports / protocols needed to be opened up in
>>> iptables but as I haven't sussed out what's needed under F13 I am 
>>> running
>>> with iptables stopped for now. If anybody has this information I would 
>>> be
>>> very grateful to receive it.
>>
>> The FAQs on http://corosync.org/ and http://openais.org/ could help
>> here. Unfortunately the FAQ entry about firewalls has no content.
>>
>> Bug?!
>>
>> A "netstat -tulpen" on a root shell should show the ports that corosync
>> and openais listening on. They use a single multicast port if i'm not
>> mistaken.
>>
>>> Thanks&  regards
>>>
>>> John
>>
>> Marc
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster 




From linux at alteeve.com  Tue Jul 13 02:45:45 2010
From: linux at alteeve.com (Digimer)
Date: Mon, 12 Jul 2010 22:45:45 -0400
Subject: [Linux-cluster] corosync issue with two interface directives
In-Reply-To: <4C3B6F44.2020002@streamwide.ro>
References: <4C3AA0AB.4060800@alteeve.com>
	<4C3AA42A.7020408@redhat.com>	<4C3B168B.7080706@alteeve.com>
	<4C3B6F44.2020002@streamwide.ro>
Message-ID: <4C3BD359.9070703@alteeve.com>

On 10-07-12 03:38 PM, Dan Frincu wrote:
> Just a wild stab in the dark, but both your multicast groups result in
> the same MAC address 01:00:5e:5e:01:01. Now these are supposed to be
> redundant connections, which means different IP/port/MAC's, different
> routes, in case one fails the other one takes over. Try changing the
> multicast groups so that the they differ at the end of the address,
> rather than at the beginning.
>
> You could use 226.94.1.1 for ring 0 and 226.94.1.2 for ring 1. The key
> thing to remember here is that when building a multicast MAC address,
> you only use the low order 23 bits of the multicast IP address, thus
> allowing for overlapping of close high order bits in the multicast IP
> address.
>
> Regards.

Hi Dan,

   I tried changing my config file as per your recommendation, but it 
didn't seem to sort out the problem. The 'corosync-objctl' output only 
shows one ring. Oddly though, when I don't start corosync via cman, and 
instead start corosync by itself, both rings are in fact shown...

   Also, I must claim ignorance - I am not familiar with how multicast 
addresses map to MAC addresses. Perhaps I am missing something 
fundamental here? I originally extrapolated the second ring's config by 
adapting the first ring which, itself, was adapted from example configs 
available in the docs.

Here is my current corosync.conf file:

------------------------------------------------------------------
# This is a skeleton example configuration file.
compatibility: whitetank

# Totem Protocol options.
totem {
	version: 2
	secauth: off
	threads: 0
	rrp_mode: passive
	interface {
		# This is the back-channel subnet, which is the primary network
		# for the totem protocol.
		ringnumber: 0
		bindnetaddr: 10.0.1.0
		mcastaddr: 226.94.1.1
		mcastport: 5405
	}
	interface {
		# This is the storage network, which acts as a secondary, backup
		# network for the totem protocol.
		ringnumber: 1
		bindnetaddr: 10.0.0.0
		mcastaddr: 227.94.1.2
		mcastport: 5405
	}
}

logging {
	to_syslog: yes
         fileline: off
         to_stderr: yes
         to_logfile: off
         to_syslog: yes
         #logfile: /var/log/corosync.log
         debug: on
         timestamp: on
}

amf {
	mode: disabled
}
------------------------------------------------------------------

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From dfrincu at streamwide.ro  Tue Jul 13 04:33:19 2010
From: dfrincu at streamwide.ro (Dan Frincu)
Date: Tue, 13 Jul 2010 07:33:19 +0300
Subject: [Linux-cluster] corosync issue with two interface directives
In-Reply-To: <4C3BD359.9070703@alteeve.com>
References: <4C3AA0AB.4060800@alteeve.com>
	<4C3AA42A.7020408@redhat.com>	<4C3B168B.7080706@alteeve.com>
	<4C3B6F44.2020002@streamwide.ro> <4C3BD359.9070703@alteeve.com>
Message-ID: <4C3BEC8F.5030200@streamwide.ro>

It seems that the issue could also be caused because Corosync looks at 
the classfull network masks of a bindnetaddr directive, in this case, 
even if you have addresses with /24 netmask, Corosync thinks they're /8. 
Try changing to a private class C addressing scheme, which also uses /24.

Read more here http://www.corosync.org/doku.php?id=faq:configure_openais

Regards.

Digimer wrote:
> On 10-07-12 03:38 PM, Dan Frincu wrote:
>> Just a wild stab in the dark, but both your multicast groups result in
>> the same MAC address 01:00:5e:5e:01:01. Now these are supposed to be
>> redundant connections, which means different IP/port/MAC's, different
>> routes, in case one fails the other one takes over. Try changing the
>> multicast groups so that the they differ at the end of the address,
>> rather than at the beginning.
>>
>> You could use 226.94.1.1 for ring 0 and 226.94.1.2 for ring 1. The key
>> thing to remember here is that when building a multicast MAC address,
>> you only use the low order 23 bits of the multicast IP address, thus
>> allowing for overlapping of close high order bits in the multicast IP
>> address.
>>
>> Regards.
>
> Hi Dan,
>
>   I tried changing my config file as per your recommendation, but it 
> didn't seem to sort out the problem. The 'corosync-objctl' output only 
> shows one ring. Oddly though, when I don't start corosync via cman, 
> and instead start corosync by itself, both rings are in fact shown...
>
>   Also, I must claim ignorance - I am not familiar with how multicast 
> addresses map to MAC addresses. Perhaps I am missing something 
> fundamental here? I originally extrapolated the second ring's config 
> by adapting the first ring which, itself, was adapted from example 
> configs available in the docs.
>
> Here is my current corosync.conf file:
>
> ------------------------------------------------------------------
> # This is a skeleton example configuration file.
> compatibility: whitetank
>
> # Totem Protocol options.
> totem {
>     version: 2
>     secauth: off
>     threads: 0
>     rrp_mode: passive
>     interface {
>         # This is the back-channel subnet, which is the primary network
>         # for the totem protocol.
>         ringnumber: 0
>         bindnetaddr: 10.0.1.0
>         mcastaddr: 226.94.1.1
>         mcastport: 5405
>     }
>     interface {
>         # This is the storage network, which acts as a secondary, backup
>         # network for the totem protocol.
>         ringnumber: 1
>         bindnetaddr: 10.0.0.0
>         mcastaddr: 227.94.1.2
>         mcastport: 5405
>     }
> }
>
> logging {
>     to_syslog: yes
>         fileline: off
>         to_stderr: yes
>         to_logfile: off
>         to_syslog: yes
>         #logfile: /var/log/corosync.log
>         debug: on
>         timestamp: on
> }
>
> amf {
>     mode: disabled
> }
> ------------------------------------------------------------------
>

-- 
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100713/2f6d14e0/attachment.htm>

From linux at alteeve.com  Tue Jul 13 05:00:51 2010
From: linux at alteeve.com (Digimer)
Date: Tue, 13 Jul 2010 01:00:51 -0400
Subject: [Linux-cluster] corosync issue with two interface directives
In-Reply-To: <4C3BEC8F.5030200@streamwide.ro>
References: <4C3AA0AB.4060800@alteeve.com>	<4C3AA42A.7020408@redhat.com>	<4C3B168B.7080706@alteeve.com>	<4C3B6F44.2020002@streamwide.ro>
	<4C3BD359.9070703@alteeve.com> <4C3BEC8F.5030200@streamwide.ro>
Message-ID: <4C3BF303.70702@alteeve.com>

On 10-07-13 12:33 AM, Dan Frincu wrote:
> It seems that the issue could also be caused because Corosync looks at
> the classfull network masks of a bindnetaddr directive, in this case,
> even if you have addresses with /24 netmask, Corosync thinks they're /8.
> Try changing to a private class C addressing scheme, which also uses /24.
>
> Read more here http://www.corosync.org/doku.php?id=faq:configure_openais
>
> Regards.

That's an interesting assumption. Is there no way to specify the 
netmask? If not, I'll try that change.

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From sdake at redhat.com  Tue Jul 13 05:23:57 2010
From: sdake at redhat.com (Steven Dake)
Date: Mon, 12 Jul 2010 22:23:57 -0700
Subject: [Linux-cluster] corosync issue with two interface directives
In-Reply-To: <4C3BD359.9070703@alteeve.com>
References: <4C3AA0AB.4060800@alteeve.com>	<4C3AA42A.7020408@redhat.com>	<4C3B168B.7080706@alteeve.com>	<4C3B6F44.2020002@streamwide.ro>
	<4C3BD359.9070703@alteeve.com>
Message-ID: <4C3BF86D.7050406@redhat.com>

On 07/12/2010 07:45 PM, Digimer wrote:
> On 10-07-12 03:38 PM, Dan Frincu wrote:
>> Just a wild stab in the dark, but both your multicast groups result in
>> the same MAC address 01:00:5e:5e:01:01. Now these are supposed to be
>> redundant connections, which means different IP/port/MAC's, different
>> routes, in case one fails the other one takes over. Try changing the
>> multicast groups so that the they differ at the end of the address,
>> rather than at the beginning.
>>
>> You could use 226.94.1.1 for ring 0 and 226.94.1.2 for ring 1. The key
>> thing to remember here is that when building a multicast MAC address,
>> you only use the low order 23 bits of the multicast IP address, thus
>> allowing for overlapping of close high order bits in the multicast IP
>> address.
>>
>> Regards.
>
> Hi Dan,
>
> I tried changing my config file as per your recommendation, but it
> didn't seem to sort out the problem. The 'corosync-objctl' output only
> shows one ring. Oddly though, when I don't start corosync via cman, and
> instead start corosync by itself, both rings are in fact shown...
>
> Also, I must claim ignorance - I am not familiar with how multicast
> addresses map to MAC addresses. Perhaps I am missing something
> fundamental here? I originally extrapolated the second ring's config by
> adapting the first ring which, itself, was adapted from example configs
> available in the docs.
>

Corosync doesn't use /etc/corosync/corosync.conf when started from cman. 
  It uses /etc/cluster/cluster.conf.

Regards
-steve

> Here is my current corosync.conf file:
>
> ------------------------------------------------------------------
> # This is a skeleton example configuration file.
> compatibility: whitetank
>
> # Totem Protocol options.
> totem {
> version: 2
> secauth: off
> threads: 0
> rrp_mode: passive
> interface {
> # This is the back-channel subnet, which is the primary network
> # for the totem protocol.
> ringnumber: 0
> bindnetaddr: 10.0.1.0
> mcastaddr: 226.94.1.1
> mcastport: 5405
> }
> interface {
> # This is the storage network, which acts as a secondary, backup
> # network for the totem protocol.
> ringnumber: 1
> bindnetaddr: 10.0.0.0
> mcastaddr: 227.94.1.2
> mcastport: 5405
> }
> }
>
> logging {
> to_syslog: yes
> fileline: off
> to_stderr: yes
> to_logfile: off
> to_syslog: yes
> #logfile: /var/log/corosync.log
> debug: on
> timestamp: on
> }
>
> amf {
> mode: disabled
> }
> ------------------------------------------------------------------
>



From andrew at beekhof.net  Tue Jul 13 08:04:43 2010
From: andrew at beekhof.net (Andrew Beekhof)
Date: Tue, 13 Jul 2010 10:04:43 +0200
Subject: [Linux-cluster] The purpose of
	/usr/lib64/stonith/plugins/external
In-Reply-To: <4C3B14E0.8060007@alteeve.com>
References: <4C3951D6.40807@alteeve.com>
	<AANLkTikoXAlARw73PKhEMkQ1S7zwx2QPtyaHM-KetYjH@mail.gmail.com>
	<4C3B14E0.8060007@alteeve.com>
Message-ID: <AANLkTimm6U30Jo7arVkSVQollMH015j38SuoZxzDSD5G@mail.gmail.com>

On Mon, Jul 12, 2010 at 3:13 PM, Digimer <linux at alteeve.com> wrote:
> On 10-07-12 04:47 AM, Andrew Beekhof wrote:
>>
>> On Sun, Jul 11, 2010 at 7:08 AM, Digimer<linux at alteeve.com> ?wrote:
>>>
>>> Hi all,
>>>
>>> ?I've got a home-grown fence device I am trying to add support for Fedora
>>> 13
>>> (specifically, corosync 1.2.3, cluster-glue 1.0.2 et. al.).
>>>
>>> ?Under CentOS, I wrote an agent and placed it in /usr/sbin/ and that
>>> worked
>>> fine.
>>
>> Thats fine then. ?Pacemaker can use RHCS devices too.
>>
>>> Now though, there is the /usr/lib64/stonith/plugins/external/
>>> directory which seems to also be used. I noticed this when I used
>>> 'stonith
>>> -L'.
>>
>> This is for the old heartbeat-style fencing devices. ?You can ignore it.
>
> That's what I was starting to think, and ended up doing just that, ignoring
> it.
>
> If I may ask; The "Cluster from Scratch" manual for Fedora 11/12 suggests
> starting corosync directly. This kept failing because it didn't find a
> configured stonith device.

Define "this"?
What couldn't find a fencing device?

> However, I went back and created a cluster.conf
> and then started cman and everything was (largely) fine.

1.1.3 will actually be able to sit on top of a regular cman cluster.
It can start and stop independently of corosync or cman.

Its also not that hard to configure corosync.conf to start cman .

>
> Is there a document or similar that more clearly defines what is the old
> HA-Linux components versus the RHCS? Or further, what is the proper method
> of building a cluster based on RHCS using corosync/pacemaker?
>
> Thanks kindly!
>
> --
> Digimer
> E-Mail: ? ? ? ? linux at alteeve.com
> AN!Whitepapers: http://alteeve.com
> Node Assassin: ?http://nodeassassin.org
>



From tserong at novell.com  Tue Jul 13 13:14:01 2010
From: tserong at novell.com (Tim Serong)
Date: Tue, 13 Jul 2010 07:14:01 -0600
Subject: [Linux-cluster] corosync issue with two interface directives
In-Reply-To: <4C3BF303.70702@alteeve.com>
References: <4C3AA0AB.4060800@alteeve.com> <4C3AA42A.7020408@redhat.com>
	<4C3B168B.7080706@alteeve.com> <4C3B6F44.2020002@streamwide.ro>
	<4C3BD359.9070703@alteeve.com> <4C3BEC8F.5030200@streamwide.ro>
	<4C3BF303.70702@alteeve.com>
Message-ID: <4C3CF3390200000A00017F50@novprvlin0050.provo.novell.com>

On 7/13/2010 at 03:00 PM, Digimer <linux at alteeve.com> wrote: 
> On 10-07-13 12:33 AM, Dan Frincu wrote: 
> > It seems that the issue could also be caused because Corosync looks at 
> > the classfull network masks of a bindnetaddr directive, in this case, 
> > even if you have addresses with /24 netmask, Corosync thinks they're /8. 
> > Try changing to a private class C addressing scheme, which also uses /24. 
> > 
> > Read more here http://www.corosync.org/doku.php?id=faq:configure_openais 
> > 
> > Regards. 
>  
> That's an interesting assumption. Is there no way to specify the  
> netmask? If not, I'll try that change. 
 
Corosync looks at the netmask of the *interface* matching the bindnetaddr,
so if you've configured eth0 as (for example) 10.1.2.3/24, the appropriate
bindnetaddr is 10.1.2.0.  If you'd configured eth0 as a /8, the bindnetaddr
would need to be 10.0.0.0, and so forth.

Hope that helps to make everything even more confusing :)

Regards,

Tim


-- 
Tim Serong <tserong at novell.com>
Senior Clustering Engineer, OPS Engineering, Novell Inc.






From dfrincu at streamwide.ro  Tue Jul 13 14:35:34 2010
From: dfrincu at streamwide.ro (Dan Frincu)
Date: Tue, 13 Jul 2010 17:35:34 +0300
Subject: [Linux-cluster] corosync issue with two interface directives
In-Reply-To: <4C3CF3390200000A00017F50@novprvlin0050.provo.novell.com>
References: <4C3AA0AB.4060800@alteeve.com>
	<4C3AA42A.7020408@redhat.com>	<4C3B168B.7080706@alteeve.com>
	<4C3B6F44.2020002@streamwide.ro>	<4C3BD359.9070703@alteeve.com>
	<4C3BEC8F.5030200@streamwide.ro>	<4C3BF303.70702@alteeve.com>
	<4C3CF3390200000A00017F50@novprvlin0050.provo.novell.com>
Message-ID: <4C3C79B6.9080507@streamwide.ro>

Tim Serong wrote:
> On 7/13/2010 at 03:00 PM, Digimer <linux at alteeve.com> wrote: 
>   
>> On 10-07-13 12:33 AM, Dan Frincu wrote: 
>>     
>>> It seems that the issue could also be caused because Corosync looks at 
>>> the classfull network masks of a bindnetaddr directive, in this case, 
>>> even if you have addresses with /24 netmask, Corosync thinks they're /8. 
>>> Try changing to a private class C addressing scheme, which also uses /24. 
>>>
>>> Read more here http://www.corosync.org/doku.php?id=faq:configure_openais 
>>>
>>> Regards. 
>>>       
>>  
>> That's an interesting assumption. Is there no way to specify the  
>> netmask? If not, I'll try that change. 
>>     
>  
> Corosync looks at the netmask of the *interface* matching the bindnetaddr,
> so if you've configured eth0 as (for example) 10.1.2.3/24, the appropriate
> bindnetaddr is 10.1.2.0.  If you'd configured eth0 as a /8, the bindnetaddr
> would need to be 10.0.0.0, and so forth.
>
> Hope that helps to make everything even more confusing :)
>   
Indeed. Just to keep the confusion level high, what happens if you have 
a bindnetaddr that matches 2 or more interfaces, for whatever reason? 
Which interface does it bind to, or does it bind to any interface? If 
you place 10.1.2.3/32, would the matching bindnetaddr be 10.1.2.3? Or 
does it have to end in 0 (mandatory) as specified in the docs?

Regards,
Dan
> Regards,
>
> Tim
>
>
>   

-- 
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100713/4bfc8ed1/attachment.htm>

From sdake at redhat.com  Tue Jul 13 16:42:39 2010
From: sdake at redhat.com (Steven Dake)
Date: Tue, 13 Jul 2010 09:42:39 -0700
Subject: [Linux-cluster] corosync issue with two interface directives
In-Reply-To: <4C3C79B6.9080507@streamwide.ro>
References: <4C3AA0AB.4060800@alteeve.com>	<4C3AA42A.7020408@redhat.com>	<4C3B168B.7080706@alteeve.com>	<4C3B6F44.2020002@streamwide.ro>	<4C3BD359.9070703@alteeve.com>	<4C3BEC8F.5030200@streamwide.ro>	<4C3BF303.70702@alteeve.com>	<4C3CF3390200000A00017F50@novprvlin0050.provo.novell.com>
	<4C3C79B6.9080507@streamwide.ro>
Message-ID: <4C3C977F.6070502@redhat.com>

On 07/13/2010 07:35 AM, Dan Frincu wrote:
> Tim Serong wrote:
>> On 7/13/2010 at 03:00 PM, Digimer<linux at alteeve.com>  wrote:
>>
>>> On 10-07-13 12:33 AM, Dan Frincu wrote:
>>>
>>>> It seems that the issue could also be caused because Corosync looks at
>>>> the classfull network masks of a bindnetaddr directive, in this case,
>>>> even if you have addresses with /24 netmask, Corosync thinks they're /8.
>>>> Try changing to a private class C addressing scheme, which also uses /24.
>>>>
>>>> Read more herehttp://www.corosync.org/doku.php?id=faq:configure_openais
>>>>
>>>> Regards.
>>>>
>>>
>>> That's an interesting assumption. Is there no way to specify the
>>> netmask? If not, I'll try that change.
>>>
>>
>> Corosync looks at the netmask of the *interface* matching the bindnetaddr,
>> so if you've configured eth0 as (for example) 10.1.2.3/24, the appropriate
>> bindnetaddr is 10.1.2.0.  If you'd configured eth0 as a /8, the bindnetaddr
>> would need to be 10.0.0.0, and so forth.
>>
>> Hope that helps to make everything even more confusing :)
>>
> Indeed. Just to keep the confusion level high, what happens if you have
> a bindnetaddr that matches 2 or more interfaces, for whatever reason?

If your environment includes such a configuration, the specific IP 
address that corosync should bind to is set in bindnetaddr to force a 
binding to a specific IP.  ie: ip = 192.168.1.1, bindnetaddr=192.168.1.1.

The purpose of using the NIC's netmask in the determination of Ethernet 
interfaces to bind to allow one configuration file to be used in a 
specific group of clusters on the same netmask.

> Which interface does it bind to, or does it bind to any interface? If
> you place 10.1.2.3/32, would the matching bindnetaddr be 10.1.2.3? Or

can netmask be set to 0?
[root at cast sdake]# ifconfig eth0 netmask 0
SIOCSIFNETMASK: Cannot assign requested address
SIOCGIFADDR: Cannot assign requested address
SIOCSIFBROADCAST: Cannot assign requested address

> does it have to end in 0 (mandatory) as specified in the docs?
>

The docs were recently changed to the following:

bindnetaddr
This specifies the network address the corosync executive should
bind to.  For example, if the local  interface  is  192.168.5.92
with  netmask 255.255.255.0, set bindnetaddr to 192.168.5.0.  If
the   local   interface    is    192.168.5.92    with    netmask
255.255.255.192,  set bindnetaddr to 192.168.5.64, and so forth.


> Regards,
> Dan
>> Regards,
>>
>> Tim
>>
>>
>>
>
> --
> Dan FRINCU
> Systems Engineer
> CCNA, RHCE
> Streamwide Romania
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From jonathan.barber at gmail.com  Tue Jul 13 17:23:46 2010
From: jonathan.barber at gmail.com (Jonathan Barber)
Date: Tue, 13 Jul 2010 18:23:46 +0100
Subject: [Linux-cluster] Updating <fence_daemon/> parameters
Message-ID: <AANLkTin0EHPlwIBh2gdoIoz9wT3BUK_W6PXs8ymrSEhG@mail.gmail.com>

I'm experimenting with changing the post_fence_delay and
post_join_delay parameters in the /etc/cluster/cluster.conf
<fence_daemon/> element, but I'm having to use "service cman restart"
to get the changes picked up (after doing "ccs_tool update
/etc/cluster/cluster.conf"). Sending the HUP signal to fenced just
kills it, is there a way to get fenced to have it's config refreshed?

It's okay to do this in our test environment, but I don't really want
to have to migrate services that are in production to get these values
updated.

This is on RHEL5u4 x86_64 with cman-2.0.115-1.el5.

Cheers
-- 
Jonathan Barber <jonathan.barber at gmail.com>



From Bennie_R_Thomas at raytheon.com  Tue Jul 13 22:39:30 2010
From: Bennie_R_Thomas at raytheon.com (Bennie Thomas)
Date: Tue, 13 Jul 2010 17:39:30 -0500
Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes
	randomly	reboot
In-Reply-To: <4C343242.6000804@raytheon.com>
References: <mailman.20241.1277763744.8372.linux-cluster@redhat.com>	<4C2B5322.9030305@arlut.utexas.edu>	<1277908521.3158.441.camel@localhost.localdomain>
	<4C343242.6000804@raytheon.com>
Message-ID: <4C3CEB22.2060002@raytheon.com>

I think I may have found my problem. However, I am not sure how to fix it.

I noticed that node one shows the quorum disk to be /dev/dm-2 and node 2 
shows the quorum disk to be /dev/dm-3

How can I change these ?


Bennie Thomas wrote:
> Hello Steve,
>
> I have the cluster traffic going over a private interface. I know have 
> the nodes connected point-to-point.
> I have thought about the network traffic, However, with a dedicated 
> interface, I should not have this problem.
>
> I have built, numerous Clusters from different vendors. This 
> particular cluster is an active/passive. I have noticed
> that the passive node in the cluster is the one that reboots the most.
>
> Thanks for any/all input....
>
>
>
> Steven Whitehouse wrote:
>> Hi,
>>
>> On Wed, 2010-06-30 at 09:22 -0500, Randy Zagar wrote:
>>  
>>> Yes.  My experience is that you can't currently nfs-export *any* GFS 
>>> or GFS2 filesystems.
>>>
>>>     
>> You can, but there are only a fairly small number of configurations
>> which will actually work from the larger number of possible
>> configurations. We do hope to expand that a bit in the future, but for
>> the time being its best to stick to a active/passive failover export
>> which is not mixed with any other protocol (Samba) or any local
>> applications.
>>
>>  
>>> Exporting EXT3/EXT4 filesystems, however, doesn't appear to be a 
>>> problem.
>>>
>>> -Randy Zagar <zagar at arlut.utexas.edu>
>>>
>>> On 06/28/2010 05:22 PM, linux-cluster-request at redhat.com wrote:
>>>    
>>>> From: Bennie Thomas<Bennie_R_Thomas at raytheon.com>
>>>> Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes randomly
>>>>     reboot
>>>> Message-ID:<4C290BE0.3090707 at raytheon.com>
>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>>
>>>> I currently have 2 DL380 G6 with and HP MSA2312 disk array. Running
>>>> Redhat 5u4 64bit.  I have a quorum disk.  I use the Cluster as an
>>>> Active/passive NFS Cluster
>>>> The problem I am having is one or both of the nodes will randomly
>>>> reboot. Has anyone experienced this problem
>>>>
>>>>       
>> Is the node being fenced? This might be down to excessive network
>> traffic blocking the cluster traffic and making it appear as if the node
>> is down when it isn't, or something similar to that. Do you get any log
>> messages?
>>
>> Steve.
>>
>>  
>>>>          
>>> -- 
>>> Linux-cluster mailing list
>>> Linux-cluster at redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>     
>>
>> -- 
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>>   
>

-- 
Bennie Thomas
Sr. Information Systems Technologist II
Raytheon Company

972.205.4126
972.205.6363 fax
888.347.1660 pager
Bennie_R_Thomas at raytheon.com


DISCLAIMER: This message contains information that may be confidential and privileged. Unless you are the addressee (or authorized to receive mail for the addressee), you should not use, copy or disclose to anyone this message or any information contained in this message. If you have received this message in error, please so advise the sender by reply e-mail and delete this message. Thank you for your cooperation.

Any views or opinions presented are solely those of the author and do not necessarily represent those of Raytheon unless specifically stated. 
Electronic communications including email may be monitored by Raytheon
for operational or business reasons.






From jumanjiman at gmail.com  Tue Jul 13 23:17:25 2010
From: jumanjiman at gmail.com (Paul Morgan)
Date: Tue, 13 Jul 2010 19:17:25 -0400
Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes randomly
	reboot
In-Reply-To: <4C3CEB22.2060002@raytheon.com>
References: <mailman.20241.1277763744.8372.linux-cluster@redhat.com>
	<4C2B5322.9030305@arlut.utexas.edu>
	<1277908521.3158.441.camel@localhost.localdomain>
	<4C343242.6000804@raytheon.com> <4C3CEB22.2060002@raytheon.com>
Message-ID: <AANLkTikOgoER8LUXKwsLcdyu-Xo8zzSp2aYe79zYhIOA@mail.gmail.com>

/dev/mapper/friendly-name,  Label,  /dev/disk/by*, or  symlink  created by
custom  udev rule

-paul

--
Top-posted from gmail on android 2.2

On Jul 13, 2010 6:55 PM, "Bennie Thomas" <Bennie_R_Thomas at raytheon.com>
wrote:
> I think I may have found my problem. However, I am not sure how to fix it.
>
> I noticed that node one shows the quorum disk to be /dev/dm-2 and node 2
> shows the quorum disk to be /dev/dm-3
>
> How can I change these ?
>
>
> Bennie Thomas wrote:
>> Hello Steve,
>>
>> I have the cluster traffic going over a private interface. I know have
>> the nodes connected point-to-point.
>> I have thought about the network traffic, However, with a dedicated
>> interface, I should not have this problem.
>>
>> I have built, numerous Clusters from different vendors. This
>> particular cluster is an active/passive. I have noticed
>> that the passive node in the cluster is the one that reboots the most.
>>
>> Thanks for any/all input....
>>
>>
>>
>> Steven Whitehouse wrote:
>>> Hi,
>>>
>>> On Wed, 2010-06-30 at 09:22 -0500, Randy Zagar wrote:
>>>
>>>> Yes. My experience is that you can't currently nfs-export *any* GFS
>>>> or GFS2 filesystems.
>>>>
>>>>
>>> You can, but there are only a fairly small number of configurations
>>> which will actually work from the larger number of possible
>>> configurations. We do hope to expand that a bit in the future, but for
>>> the time being its best to stick to a active/passive failover export
>>> which is not mixed with any other protocol (Samba) or any local
>>> applications.
>>>
>>>
>>>> Exporting EXT3/EXT4 filesystems, however, doesn't appear to be a
>>>> problem.
>>>>
>>>> -Randy Zagar <zagar at arlut.utexas.edu>
>>>>
>>>> On 06/28/2010 05:22 PM, linux-cluster-request at redhat.com wrote:
>>>>
>>>>> From: Bennie Thomas<Bennie_R_Thomas at raytheon.com>
>>>>> Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes randomly
>>>>> reboot
>>>>> Message-ID:<4C290BE0.3090707 at raytheon.com>
>>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>>>
>>>>> I currently have 2 DL380 G6 with and HP MSA2312 disk array. Running
>>>>> Redhat 5u4 64bit. I have a quorum disk. I use the Cluster as an
>>>>> Active/passive NFS Cluster
>>>>> The problem I am having is one or both of the nodes will randomly
>>>>> reboot. Has anyone experienced this problem
>>>>>
>>>>>
>>> Is the node being fenced? This might be down to excessive network
>>> traffic blocking the cluster traffic and making it appear as if the node
>>> is down when it isn't, or something similar to that. Do you get any log
>>> messages?
>>>
>>> Steve.
>>>
>>>
>>>>>
>>>> --
>>>> Linux-cluster mailing list
>>>> Linux-cluster at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>
>>>
>>> --
>>> Linux-cluster mailing list
>>> Linux-cluster at redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>
>>>
>>
>
> --
> Bennie Thomas
> Sr. Information Systems Technologist II
> Raytheon Company
>
> 972.205.4126
> 972.205.6363 fax
> 888.347.1660 pager
> Bennie_R_Thomas at raytheon.com
>
>
> DISCLAIMER: This message contains information that may be confidential and
privileged. Unless you are the addressee (or authorized to receive mail for
the addressee), you should not use, copy or disclose to anyone this message
or any information contained in this message. If you have received this
message in error, please so advise the sender by reply e-mail and delete
this message. Thank you for your cooperation.
>
> Any views or opinions presented are solely those of the author and do not
necessarily represent those of Raytheon unless specifically stated.
> Electronic communications including email may be monitored by Raytheon
> for operational or business reasons.
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100713/2180f993/attachment.htm>

From jayfitzpatrick at gmail.com  Wed Jul 14 08:55:36 2010
From: jayfitzpatrick at gmail.com (Jason Fitzpatrick)
Date: Wed, 14 Jul 2010 09:55:36 +0100
Subject: [Linux-cluster] Fedora 13 Clustering
In-Reply-To: <134A6BB8530549C8828D857491790777@Desktop>
References: <37F244B6767942AD9908B56800D317C8@laptop1>
	<AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>
	<1278942448.2679.2.camel@thinkpad>
	<134A6BB8530549C8828D857491790777@Desktop>
Message-ID: <AANLkTikA9jP0i-ztO7vm6d9xI4m3PrX3u4MmH2gcKTXh@mail.gmail.com>

Hi John..

re: "However, in F13 there is no "luci_admin" utility to set the
password and Luci isn't accessible from a web browser."

I found the same myself when trying to configure, using the luci_admin
from RHEL 5 did not work either, I even went as far as e-mailing the
dev from Redhat who compiled the FC13 version to try and get the
default password but have had no reply (e-mail sent from work account
and I do not have access to it at the moment)

as for the Luci not being available, I believe that it is now running
on a different port, from what I can remember it is now on 8080 ( a
netstat -an| grep luci should get the port)

When you load this up you get a demo version of Luci with systems
populated, and it looks nice!

I also tried the RHEL version of Luci against the FC13 cluster with
mixed results (if you count it not working as a mixed result!)

Jay

On 12 July 2010 15:06, Virginian <virginian at blueyonder.co.uk> wrote:
> Hi Gordon,
>
> It looks to me like things have changed quite a bit in F13 compared to RHEL
> / Centos 5. I agree, cluster.conf ?looks the same but there is a new
> component called Corosync and ccs seems to have gone.
>
> Traditionally I have used Conga to configure the Cluster (although not
> without problems at times it has to be said). However, in F13 there is no
> "luci_admin" utility to set the password and Luci isn't accessible from a
> web browser. I don't mind configuring things by hand if that's the only way
> to do it, I was just looking to be pointed in the right direction really. I
> have received some help offline from a member of this list which has proved
> invaluable. I have now managed to get a basic cluster up and running on F13.
> Our fellow list member is working on a document which I think will be of
> enormous help to others when completed (in fact it is already a useful doc).
>
> The old docs specified which ports / protocols needed to be opened up in
> iptables but as I haven't sussed out what's needed under F13 I am running
> with iptables stopped for now. If anybody has this information I would be
> very grateful to receive it.
>
> Thanks & regards
>
> John
>
> ----- Original Message ----- From: "Gordan Bobic" <gordan at bobich.net>
> To: "linux clustering" <linux-cluster at redhat.com>
> Sent: Monday, July 12, 2010 2:47 PM
> Subject: Re: [Linux-cluster] Fedora 13 Clustering
>
>
>> Maybe I'm missing what exactly you guys are talking about, but
>> everything I can think of in RHCS is configured via cluster.conf. My
>> cluster.conf files from RHEL5 work just fine on RHEL6 beta, which is
>> virtually identical to F12/F13. I don't see why you would have any
>> problems getting RHCS to work on F13 using the same instructions you
>> would follow for RHEL/CentOS 5.
>>
>> Gordan
>>
>> On Mon, 2010-07-12 at 14:37 +0100, Jason Fitzpatrick wrote:
>>>
>>> Hi John..
>>>
>>> Normally I do not agree with replying to posts without anything
>>> constructive to say but I thought that I should in this case.
>>>
>>> I have to admit that I ran into the same issues as you, I built a RHEL
>>> 5.5 cluster and decided to give the FC13 path a try, Tried to use Luci
>>> / Ricci to configure the nodes and had no luck even managing to log
>>> into Luci, no documents / man pages included with the packages and the
>>> readme file that was included was the same one that was included with
>>> the old source files, and had not been updated since.
>>>
>>> In the end I gave up and went back to RHEL.
>>>
>>> (that said you can go the heartbeat route, this I got to work when
>>> using oracles ocfs as a shared data store on drbd)
>>>
>>> Jay
>>>
>>> On 9 July 2010 21:43, Virginian <virginian at blueyonder.co.uk> wrote:
>>> > Hi All,
>>> >
>>> > I've played around with Red Hat Cluster Suite on Centos 5 and found >
>>> > plenty
>>> > of information on how to install and configure the cluster and nodes. I
>>> > > have
>>> > been trawling Google and other search engines trying to find accurate
>>> > instructions for running Cluster Suite on Fedora 13. So far I have >
>>> > pretty
>>> > much drawn a blank. What I am looking for is instructions on how to
>>> > configure a 2 node cluster with iSCSI shared storage for GFS2, the aim
>>> > > being
>>> > to run virtualised Linux guests under KVM and have the ability to >
>>> > migrate
>>> > the guests from one node to the other in the event of a node failure. >
>>> > From
>>> > what I have read, what I understood to be RHCS in RHEL 5 / Centos 5 has
>>> > changed quite significantly. It looks like openais / corosync are new
>>> > components and ccs has largely disappeared. If I could find some >
>>> > information
>>> > on how to configure corosync, openais, amf etc. to achieve a fairly >
>>> > simple 2
>>> > node cluster as described I think I would be ok. I've tried googling >
>>> > for
>>> > this under Fedora 13 but so far drawn a blank. Has anybody else set up
>>> > > a
>>> > cluster similar to the above under Fedora 13?
>>> >
>>> > Regards
>>> >
>>> > John
>>> > --
>>> > Linux-cluster mailing list
>>> > Linux-cluster at redhat.com
>>> > https://www.redhat.com/mailman/listinfo/linux-cluster
>>> >
>>>
>>>
>>>
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 

"The only difference between saints and sinners is that every saint
has a past while every sinner has a future. "
? Oscar Wilde



From volker at ixolution.de  Wed Jul 14 10:02:15 2010
From: volker at ixolution.de (Volker Dormeyer)
Date: Wed, 14 Jul 2010 12:02:15 +0200
Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes randomly
 reboot
In-Reply-To: <4C3CEB22.2060002@raytheon.com>
References: <mailman.20241.1277763744.8372.linux-cluster@redhat.com>
	<4C2B5322.9030305@arlut.utexas.edu>
	<1277908521.3158.441.camel@localhost.localdomain>
	<4C343242.6000804@raytheon.com> <4C3CEB22.2060002@raytheon.com>
Message-ID: <20100714100215.GC5334@dijkstra>

Hi,

On Tue, Jul 13, 2010 at 05:39:30PM -0500,
Bennie Thomas <Bennie_R_Thomas at raytheon.com> wrote:
> I think I may have found my problem. However, I am not sure how to fix it.
> 
> I noticed that node one shows the quorum disk to be /dev/dm-2 and
> node 2 shows the quorum disk to be /dev/dm-3

How do you configured the cluster to use the Quorum? By Label or by Device?
If you used the device settings and specified a dm-device in cluster.conf,
this would be your problem.

If you used the Quorum-label, it shouldn't. The underlying dm-device-name
doesn't matter as long as you do not specify it explicitly. The devices will
be mapped to the appropriate dm-device, if you use /dev/mapper/... devices.
Hence, I don't beleive this to be your problem.

Do you have cman killed messages in the log of the remaining node before it is
being fenced? May be you can post the appropriate part of the log and the
config.

It might be helpful to run a debug-log for qdiskd - just to see whether your
qdisk is the root cause. Could you add the options log_level="7" and
log_facility="local3" to your qdisk configuration part? You need to setup
an appropriate entry in /etc/syslog.conf then, as well. Qdiskd logs some debug
messages, if it is not able to access the qdisk, etc.

> How can I change these ?

However, I think your Quorum Device is located on a logical volume. Hence, it
makes sense to force LVM to use descriptive device names by setting the device
filter. I. e.:

preferred_names = [ "^/dev/mapper/mpath", "^/dev/[hs]d" ]
filter = [ "a|/dev/mapper/mpath.*|", "a|/dev/cciss/.*|", "r|.*|" ]

Afterwards, perfrom "pvscan" and LVM should be using the descriptive names.
"pvs" should now list "/dev/mapper/mpath*" devices. Of course, you have to
change the settings if you do not use multipathing. You local storage
controller should be listed in the filter, as well...

The filter will not solve your problem, but makes the configuration more
transparent.

If your quorum device is located on a self-made dm-device instead, just create
it with a unique name. 

Regards,
Volker



From cfeist at redhat.com  Wed Jul 14 12:28:15 2010
From: cfeist at redhat.com (Chris Feist)
Date: Wed, 14 Jul 2010 07:28:15 -0500
Subject: [Linux-cluster] Fedora 13 Clustering
In-Reply-To: <AANLkTikA9jP0i-ztO7vm6d9xI4m3PrX3u4MmH2gcKTXh@mail.gmail.com>
References: <37F244B6767942AD9908B56800D317C8@laptop1>	<AANLkTimpOd3aCt9qR6F3z-LdBeTWPNUOgTqbGdYiQySC@mail.gmail.com>	<1278942448.2679.2.camel@thinkpad>	<134A6BB8530549C8828D857491790777@Desktop>
	<AANLkTikA9jP0i-ztO7vm6d9xI4m3PrX3u4MmH2gcKTXh@mail.gmail.com>
Message-ID: <4C3DAD5F.7010806@redhat.com>

On 07/14/10 03:55, Jason Fitzpatrick wrote:
> Hi John..
>
> re: "However, in F13 there is no "luci_admin" utility to set the
> password and Luci isn't accessible from a web browser."

Just a note, luci in F11,12,13,etc. Is completely non-functional.  It is missing 
large sections of code to even make it work.  There is the possibility that a 
working luci will be present in F14, but right now the only way to configure the 
cluster is by editing the cluster.conf file.

>
> I found the same myself when trying to configure, using the luci_admin
> from RHEL 5 did not work either, I even went as far as e-mailing the
> dev from Redhat who compiled the FC13 version to try and get the
> default password but have had no reply (e-mail sent from work account
> and I do not have access to it at the moment)
>
> as for the Luci not being available, I believe that it is now running
> on a different port, from what I can remember it is now on 8080 ( a
> netstat -an| grep luci should get the port)
>
> When you load this up you get a demo version of Luci with systems
> populated, and it looks nice!
>
> I also tried the RHEL version of Luci against the FC13 cluster with
> mixed results (if you count it not working as a mixed result!)
>
> Jay
>
> On 12 July 2010 15:06, Virginian<virginian at blueyonder.co.uk>  wrote:
>> Hi Gordon,
>>
>> It looks to me like things have changed quite a bit in F13 compared to RHEL
>> / Centos 5. I agree, cluster.conf  looks the same but there is a new
>> component called Corosync and ccs seems to have gone.
>>
>> Traditionally I have used Conga to configure the Cluster (although not
>> without problems at times it has to be said). However, in F13 there is no
>> "luci_admin" utility to set the password and Luci isn't accessible from a
>> web browser. I don't mind configuring things by hand if that's the only way
>> to do it, I was just looking to be pointed in the right direction really. I
>> have received some help offline from a member of this list which has proved
>> invaluable. I have now managed to get a basic cluster up and running on F13.
>> Our fellow list member is working on a document which I think will be of
>> enormous help to others when completed (in fact it is already a useful doc).
>>
>> The old docs specified which ports / protocols needed to be opened up in
>> iptables but as I haven't sussed out what's needed under F13 I am running
>> with iptables stopped for now. If anybody has this information I would be
>> very grateful to receive it.
>>
>> Thanks&  regards
>>
>> John
>>
>> ----- Original Message ----- From: "Gordan Bobic"<gordan at bobich.net>
>> To: "linux clustering"<linux-cluster at redhat.com>
>> Sent: Monday, July 12, 2010 2:47 PM
>> Subject: Re: [Linux-cluster] Fedora 13 Clustering
>>
>>
>>> Maybe I'm missing what exactly you guys are talking about, but
>>> everything I can think of in RHCS is configured via cluster.conf. My
>>> cluster.conf files from RHEL5 work just fine on RHEL6 beta, which is
>>> virtually identical to F12/F13. I don't see why you would have any
>>> problems getting RHCS to work on F13 using the same instructions you
>>> would follow for RHEL/CentOS 5.
>>>
>>> Gordan
>>>
>>> On Mon, 2010-07-12 at 14:37 +0100, Jason Fitzpatrick wrote:
>>>>
>>>> Hi John..
>>>>
>>>> Normally I do not agree with replying to posts without anything
>>>> constructive to say but I thought that I should in this case.
>>>>
>>>> I have to admit that I ran into the same issues as you, I built a RHEL
>>>> 5.5 cluster and decided to give the FC13 path a try, Tried to use Luci
>>>> / Ricci to configure the nodes and had no luck even managing to log
>>>> into Luci, no documents / man pages included with the packages and the
>>>> readme file that was included was the same one that was included with
>>>> the old source files, and had not been updated since.
>>>>
>>>> In the end I gave up and went back to RHEL.
>>>>
>>>> (that said you can go the heartbeat route, this I got to work when
>>>> using oracles ocfs as a shared data store on drbd)
>>>>
>>>> Jay
>>>>
>>>> On 9 July 2010 21:43, Virginian<virginian at blueyonder.co.uk>  wrote:
>>>>> Hi All,
>>>>>
>>>>> I've played around with Red Hat Cluster Suite on Centos 5 and found>
>>>>> plenty
>>>>> of information on how to install and configure the cluster and nodes. I
>>>>>> have
>>>>> been trawling Google and other search engines trying to find accurate
>>>>> instructions for running Cluster Suite on Fedora 13. So far I have>
>>>>> pretty
>>>>> much drawn a blank. What I am looking for is instructions on how to
>>>>> configure a 2 node cluster with iSCSI shared storage for GFS2, the aim
>>>>>> being
>>>>> to run virtualised Linux guests under KVM and have the ability to>
>>>>> migrate
>>>>> the guests from one node to the other in the event of a node failure.>
>>>>> From
>>>>> what I have read, what I understood to be RHCS in RHEL 5 / Centos 5 has
>>>>> changed quite significantly. It looks like openais / corosync are new
>>>>> components and ccs has largely disappeared. If I could find some>
>>>>> information
>>>>> on how to configure corosync, openais, amf etc. to achieve a fairly>
>>>>> simple 2
>>>>> node cluster as described I think I would be ok. I've tried googling>
>>>>> for
>>>>> this under Fedora 13 but so far drawn a blank. Has anybody else set up
>>>>>> a
>>>>> cluster similar to the above under Fedora 13?
>>>>>
>>>>> Regards
>>>>>
>>>>> John
>>>>> --
>>>>> Linux-cluster mailing list
>>>>> Linux-cluster at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Linux-cluster mailing list
>>> Linux-cluster at redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>
>
>



From anoop_rajkumar at merck.com  Wed Jul 14 14:58:28 2010
From: anoop_rajkumar at merck.com (Rajkumar, Anoop)
Date: Wed, 14 Jul 2010 10:58:28 -0400
Subject: [Linux-cluster] Adding third node to existing cluster problem
Message-ID: <C651C3AA2A6A1D4980D35451DDE3F96B7216C6@usctmx1160.merck.com>

Hi

I have a two node cluster and I was adding third node to it but its
failing with below error message.

Jul 14 10:44:56 usrylxap235 dlm_controld[5472]: connect to ccs error
-111, check ccsd or cluster status
Jul 14 10:44:56 usrylxap235 ccsd[5758]: Initial status:: Inquorate
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] HeartBeat is
Disabled. To enable set heartbeat_failures_allowed > 0
Jul 14 10:44:56 usrylxap235 ccsd[5758]: Cluster is not quorate.
Refusing connection.
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jul 14 10:44:56 usrylxap235 ccsd[5758]: Error while processing connect:
Connection refused
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] The network interface
[192.168.0.7] is now up.
Jul 14 10:44:56 usrylxap235 gfs_controld[5478]: connect to ccs error
-111, check ccsd or cluster status
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] Created or loaded
sequence id 68.192.168.0.7 for this ring.
Jul 14 10:44:56 usrylxap235 ccsd[5758]: Cluster is not quorate.
Refusing connection.
Jul 14 10:44:56 usrylxap235 openais[5764]: [TOTEM] entering GATHER state
from 15.
Jul 14 10:44:56 usrylxap235 ccsd[5758]: Error while processing connect:
Connection refused

Clustat output from other two nodes shows third node as offline.

[root at usrylxap237 ~]# clustat
Cluster Status for cluster1 @ Wed Jul 14 10:50:57 2010
Member Status: Quorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 usrylxap237-1.merck.com                     1 Online, Local, rgmanager
 usrylxap238-1.merck.com                     2 Online, rgmanager
 usrylxap235-1.merck.com                     3 Offline

 Service Name                   Owner (Last)                   State
 ------- ----                   ----- ------                   -----
 service:http-service           usrylxap237-1.merck.com        started
 service:mysql                  usrylxap237-1.merck.com        started


Clustat output from third node shows other two nodes offline.

[root at usrylxap235 sysconfig]# clustat
Cluster Status for cluster1 @ Wed Jul 14 10:52:47 2010
Member Status: Inquorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 usrylxap237-1.merck.com                     1 Offline
 usrylxap238-1.merck.com                     2 Offline
 usrylxap235-1.merck.com                     3 Online, Local


I will appreciate any help to resolv this.

Thanks
Anoop

Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100714/d99e8086/attachment.htm>

From laszlo at beres.me  Wed Jul 14 15:16:34 2010
From: laszlo at beres.me (Laszlo Beres)
Date: Wed, 14 Jul 2010 17:16:34 +0200
Subject: [Linux-cluster] Adding third node to existing cluster problem
In-Reply-To: <C651C3AA2A6A1D4980D35451DDE3F96B7216C6@usctmx1160.merck.com>
References: <C651C3AA2A6A1D4980D35451DDE3F96B7216C6@usctmx1160.merck.com>
Message-ID: <AANLkTikTp8qsKgUsDjcEZwJReIHxhpNSSlwwMWzc6L3T@mail.gmail.com>

On Wed, Jul 14, 2010 at 4:58 PM, Rajkumar, Anoop
<anoop_rajkumar at merck.com> wrote:

> I have a two node cluster and I was adding third node to it but its failing
> with below error message.

Have you done this way?

http://sources.redhat.com/cluster/wiki/FAQ/CMAN#cman_2to3

-- 
L?szl? B?res? ? ? ? ? ? Unix system engineer
http://www.google.com/profiles/beres.laszlo



From Bennie_R_Thomas at raytheon.com  Wed Jul 14 15:49:59 2010
From: Bennie_R_Thomas at raytheon.com (Bennie Thomas)
Date: Wed, 14 Jul 2010 10:49:59 -0500
Subject: [Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes randomly
	reboot
In-Reply-To: <20100714100215.GC5334@dijkstra>
References: <mailman.20241.1277763744.8372.linux-cluster@redhat.com>	<4C2B5322.9030305@arlut.utexas.edu>	<1277908521.3158.441.camel@localhost.localdomain>	<4C343242.6000804@raytheon.com>
	<4C3CEB22.2060002@raytheon.com> <20100714100215.GC5334@dijkstra>
Message-ID: <4C3DDCA7.6070104@raytheon.com>

The device names turned out to not be a problem.. I did configure my 
systems to use "label" vice device.
I have turned on debug to try and isolate the problem.  I am waiting for 
the node to reboot again.

I am not seeing any cman messages prior to node being fenced. This is 
the puzzling thing. As there are
no indications of the node having trouble..

Thanks

Volker Dormeyer wrote:
> Hi,
>
> On Tue, Jul 13, 2010 at 05:39:30PM -0500,
> Bennie Thomas <Bennie_R_Thomas at raytheon.com> wrote:
>   
>> I think I may have found my problem. However, I am not sure how to fix it.
>>
>> I noticed that node one shows the quorum disk to be /dev/dm-2 and
>> node 2 shows the quorum disk to be /dev/dm-3
>>     
>
> How do you configured the cluster to use the Quorum? By Label or by Device?
> If you used the device settings and specified a dm-device in cluster.conf,
> this would be your problem.
>
> If you used the Quorum-label, it shouldn't. The underlying dm-device-name
> doesn't matter as long as you do not specify it explicitly. The devices will
> be mapped to the appropriate dm-device, if you use /dev/mapper/... devices.
> Hence, I don't beleive this to be your problem.
>
> Do you have cman killed messages in the log of the remaining node before it is
> being fenced? May be you can post the appropriate part of the log and the
> config.
>
> It might be helpful to run a debug-log for qdiskd - just to see whether your
> qdisk is the root cause. Could you add the options log_level="7" and
> log_facility="local3" to your qdisk configuration part? You need to setup
> an appropriate entry in /etc/syslog.conf then, as well. Qdiskd logs some debug
> messages, if it is not able to access the qdisk, etc.
>
>   
>> How can I change these ?
>>     
>
> However, I think your Quorum Device is located on a logical volume. Hence, it
> makes sense to force LVM to use descriptive device names by setting the device
> filter. I. e.:
>
> preferred_names = [ "^/dev/mapper/mpath", "^/dev/[hs]d" ]
> filter = [ "a|/dev/mapper/mpath.*|", "a|/dev/cciss/.*|", "r|.*|" ]
>
> Afterwards, perfrom "pvscan" and LVM should be using the descriptive names.
> "pvs" should now list "/dev/mapper/mpath*" devices. Of course, you have to
> change the settings if you do not use multipathing. You local storage
> controller should be listed in the filter, as well...
>
> The filter will not solve your problem, but makes the configuration more
> transparent.
>
> If your quorum device is located on a self-made dm-device instead, just create
> it with a unique name. 
>
> Regards,
> Volker
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>   

-- 
Bennie Thomas
Sr. Information Systems Technologist II
Raytheon Company

972.205.4126
972.205.6363 fax
888.347.1660 pager
Bennie_R_Thomas at raytheon.com


DISCLAIMER: This message contains information that may be confidential and privileged. Unless you are the addressee (or authorized to receive mail for the addressee), you should not use, copy or disclose to anyone this message or any information contained in this message. If you have received this message in error, please so advise the sender by reply e-mail and delete this message. Thank you for your cooperation.

Any views or opinions presented are solely those of the author and do not necessarily represent those of Raytheon unless specifically stated. 
Electronic communications including email may be monitored by Raytheon
for operational or business reasons.






From linux at alteeve.com  Wed Jul 14 16:12:22 2010
From: linux at alteeve.com (Digimer)
Date: Wed, 14 Jul 2010 12:12:22 -0400
Subject: [Linux-cluster] cluster.conf totem argument problems
Message-ID: <4C3DE1E6.4070508@alteeve.com>

Hi all,

   I'm trying to migrate my corosync.conf file's settings into 
cluster.conf. I discussed this with people on IRC, but I wanted to start 
a thread as I think this is a bigger issue that would benefit from a 
consistent thread.

   As I understand, any arguments in cluster.conf should be fed to the 
corosync preconfig. From there, defaults should be applied and the whole 
shebang should go into corosync-objdb. However. when I put a ticket[1] 
in for a related problem, Lon mentioned that some arguments are not yet 
supported and thus do not have support in cluster.rng. These two 
statements seem to contradict each other. Can this be clarified?

   Next, I stripped out the questionable arguments and tried to load 
cman with the cluster.conf having the totem arguments. Specifically:

	<totem rpp_mode="passive">
		<interface ringnumber="0" bindnetaddr="10.0.1.0"
			mcastaddr="226.94.1.1" mcastport="5405"/>
	</totem>

   With the above argument in my cluster.conf, it passed validation but 
then failed to start with:

Starting cluster:
    Checking Network Manager...                             [  OK  ]
    Global setup...                                         [  OK  ]
    Loading kernel modules...                               [  OK  ]
    Mounting configfs...                                    [  OK  ]
    Starting cman... corosync died: Error, reason code is 8
                                                            [FAILED]

   In the log file, I saw this error:

Jul 14 00:00:46 an-node01 corosync[2227]:   [MAIN  ] Corosync Cluster 
Engine ('1.2.3'): started and ready to provide service.
Jul 14 00:00:46 an-node01 corosync[2227]:   [MAIN  ] Corosync built-in 
features: nss rdma
Jul 14 00:00:46 an-node01 corosync[2227]:   [MAIN  ] Successfully read 
config from /etc/cluster/cluster.conf
Jul 14 00:00:46 an-node01 corosync[2227]:   [MAIN  ] Successfully parsed 
cman config
Jul 14 00:00:46 an-node01 corosync[2227]:   [MAIN  ] Successfully 
configured openais services to load
Jul 14 00:00:46 an-node01 corosync[2227]:   [MAIN  ] parse error in 
config: No multicast port specified
Jul 14 00:00:46 an-node01 corosync[2227]:   [MAIN  ] Corosync Cluster 
Engine exiting with status 8 at main.c:1430.

   I don't think this is a syntax problem. When I remove the <totem /> 
section and start cman, it comes up fine. Further, there is a totem 
ring, which I assume the adapter created out of defaults. I can see this 
via corosync-objctl (sorry, I don't have the exact output at the 
moment). There are several 'totem.X' entries, of which, one is 'mcastport'.

   Finally, when I shutdown cman, setup my corosync.conf file and start 
it by itself, the configuration loads. In fact, there I can specify a 
redundant ring and corosync-objctl shows both rings.

   Any idea what's going on here? Should I submit a new bug?

   Thanks in advance!


1. https://bugzilla.redhat.com/show_bug.cgi?id=613880

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From anoop_rajkumar at merck.com  Wed Jul 14 20:47:34 2010
From: anoop_rajkumar at merck.com (Rajkumar, Anoop)
Date: Wed, 14 Jul 2010 16:47:34 -0400
Subject: [Linux-cluster] Adding third node to existing cluster problem
Message-ID: <C651C3AA2A6A1D4980D35451DDE3F96B7216ED@usctmx1160.merck.com>

Hi

Yes I have done this way. When I try to start on third node cman start
fails and cman_tool shows that it is not recognizing other two nodes
which are working properly..

[root at usrylxap235 ~]# service cman start
Starting cluster:
   Loading modules... done
   Mounting configfs... done
   Starting ccsd... done
   Starting cman... done
   Starting daemons... done
   Starting fencing...




failed

                                                           [FAILED]

[root at usrylxap235 ~]# cman_tool status
Version: 6.1.0
Config Version: 44
Cluster Name: cluster1
Cluster Id: 26777
Cluster Member: Yes
Cluster Generation: 84
Membership state: Cluster-Member
Nodes: 1
Expected votes: 3
Total votes: 1
Quorum: 2 Activity blocked
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: usrylxap235-1.merck.com
Node ID: 3
Multicast addresses: 239.192.104.2
Node addresses: 192.168.0.7


Attached is cluster.conf file.
 <<clusterconfig.txt>> 
Thanks
Anoop


Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100714/c75f5076/attachment.htm>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: clusterconfig.txt
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100714/c75f5076/attachment.txt>

From linux at alteeve.com  Thu Jul 15 02:54:00 2010
From: linux at alteeve.com (Digimer)
Date: Wed, 14 Jul 2010 22:54:00 -0400
Subject: [Linux-cluster] cluster.conf totem argument problems
In-Reply-To: <4C3DE1E6.4070508@alteeve.com>
References: <4C3DE1E6.4070508@alteeve.com>
Message-ID: <4C3E7848.4040107@alteeve.com>

I've submitted this issue as a bug:

https://bugzilla.redhat.com/show_bug.cgi?id=614697

Cheers,

Madi

On 10-07-14 12:12 PM, Digimer wrote:
> Hi all,
>
> I'm trying to migrate my corosync.conf file's settings into
> cluster.conf. I discussed this with people on IRC, but I wanted to start
> a thread as I think this is a bigger issue that would benefit from a
> consistent thread.
>
> As I understand, any arguments in cluster.conf should be fed to the
> corosync preconfig. From there, defaults should be applied and the whole
> shebang should go into corosync-objdb. However. when I put a ticket[1]
> in for a related problem, Lon mentioned that some arguments are not yet
> supported and thus do not have support in cluster.rng. These two
> statements seem to contradict each other. Can this be clarified?
>
> Next, I stripped out the questionable arguments and tried to load cman
> with the cluster.conf having the totem arguments. Specifically:
>
> <totem rpp_mode="passive">
> <interface ringnumber="0" bindnetaddr="10.0.1.0"
> mcastaddr="226.94.1.1" mcastport="5405"/>
> </totem>
>
> With the above argument in my cluster.conf, it passed validation but
> then failed to start with:
>
> Starting cluster:
> Checking Network Manager... [ OK ]
> Global setup... [ OK ]
> Loading kernel modules... [ OK ]
> Mounting configfs... [ OK ]
> Starting cman... corosync died: Error, reason code is 8
> [FAILED]
>
> In the log file, I saw this error:
>
> Jul 14 00:00:46 an-node01 corosync[2227]: [MAIN ] Corosync Cluster
> Engine ('1.2.3'): started and ready to provide service.
> Jul 14 00:00:46 an-node01 corosync[2227]: [MAIN ] Corosync built-in
> features: nss rdma
> Jul 14 00:00:46 an-node01 corosync[2227]: [MAIN ] Successfully read
> config from /etc/cluster/cluster.conf
> Jul 14 00:00:46 an-node01 corosync[2227]: [MAIN ] Successfully parsed
> cman config
> Jul 14 00:00:46 an-node01 corosync[2227]: [MAIN ] Successfully
> configured openais services to load
> Jul 14 00:00:46 an-node01 corosync[2227]: [MAIN ] parse error in config:
> No multicast port specified
> Jul 14 00:00:46 an-node01 corosync[2227]: [MAIN ] Corosync Cluster
> Engine exiting with status 8 at main.c:1430.
>
> I don't think this is a syntax problem. When I remove the <totem />
> section and start cman, it comes up fine. Further, there is a totem
> ring, which I assume the adapter created out of defaults. I can see this
> via corosync-objctl (sorry, I don't have the exact output at the
> moment). There are several 'totem.X' entries, of which, one is 'mcastport'.
>
> Finally, when I shutdown cman, setup my corosync.conf file and start it
> by itself, the configuration loads. In fact, there I can specify a
> redundant ring and corosync-objctl shows both rings.
>
> Any idea what's going on here? Should I submit a new bug?
>
> Thanks in advance!
>
> 1. https://bugzilla.redhat.com/show_bug.cgi?id=613880


-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From bernardchew at gmail.com  Thu Jul 15 03:21:12 2010
From: bernardchew at gmail.com (Bernard Chew)
Date: Thu, 15 Jul 2010 11:21:12 +0800
Subject: [Linux-cluster] Adding third node to existing cluster problem
In-Reply-To: <C651C3AA2A6A1D4980D35451DDE3F96B7216ED@usctmx1160.merck.com>
References: <C651C3AA2A6A1D4980D35451DDE3F96B7216ED@usctmx1160.merck.com>
Message-ID: <AANLkTintfM8Sea-B1xbakxPm8iheuDY1hFmxpi-c2S-r@mail.gmail.com>

> On Thu, Jul 15, 2010 at 4:47 AM, Rajkumar, Anoop <anoop_rajkumar at merck.com> wrote:
> Hi
>
> Yes I have done this way. When I try to start on third node cman start fails
> and cman_tool shows that it is not recognizing other two nodes which are
> working properly..
>
> [root at usrylxap235 ~]# service cman start
> Starting cluster:
> ?? Loading modules... done
> ?? Mounting configfs... done
> ?? Starting ccsd... done
> ?? Starting cman... done
> ?? Starting daemons... done
> ?? Starting fencing...
>
>
> failed
>
> ?????????????????????????????????????????????????????????? [FAILED]
>
> [root at usrylxap235 ~]# cman_tool status
> Version: 6.1.0
> Config Version: 44
> Cluster Name: cluster1
> Cluster Id: 26777
> Cluster Member: Yes
> Cluster Generation: 84
> Membership state: Cluster-Member
> Nodes: 1
> Expected votes: 3
> Total votes: 1
> Quorum: 2 Activity blocked
> Active subsystems: 5
> Flags:
> Ports Bound: 0
> Node name: usrylxap235-1.merck.com
> Node ID: 3
> Multicast addresses: 239.192.104.2
> Node addresses: 192.168.0.7
>
> Attached is cluster.conf file.
> <<clusterconfig.txt>>
> Thanks
> Anoop
>
> Notice:  This e-mail message, together with any attachments, contains
> information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
> New Jersey, USA 08889), and/or its affiliates Direct contact information
> for affiliates is available at
> http://www.merck.com/contact/contacts.html) that may be confidential,
> proprietary copyrighted and/or legally privileged. It is intended solely
> for the use of the individual or entity named on this message. If you are
> not the intended recipient, and have received this message in error,
> please notify us immediately by reply e-mail and then delete it from
> your system.
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>

Hi,

Have you checked the firewall settings?

Regards,
Bernard



From mchiniwar at gmail.com  Thu Jul 15 04:22:07 2010
From: mchiniwar at gmail.com (Mahantesh Chiniwar)
Date: Thu, 15 Jul 2010 09:52:07 +0530
Subject: [Linux-cluster]  system-config-cluster to conga
Message-ID: <AANLkTimxaSh3u8CyuHIMHuUlE0OsQNy9SgoS5gU_AU3Y@mail.gmail.com>

Hi,

I have configured cluster using system-config-cluster on RHEL-5.4.

Could you let me know if i can do the following

1. Can I move from system-config-cluster to conga?

2. Can I use the same cluster.conf in conga?

3. Can both cman and conga run together?

4. Currently in my setup cman service is running and I have started ricci
service and have initilized the luci server on one of the cluster nodes but
when i tried to start the luci service it fails. var/log/messages show just
shows   "luci: Luci startup failed An error occurred while starting luci" I
dont have any other information that this. What might be the reason?

5. I tried stopping cman service even this did not help.


This information will help me.



-- 
Thanks,
Mahantesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100715/f89fb8c3/attachment.htm>

From jacob.ishak at gmail.com  Thu Jul 15 14:11:24 2010
From: jacob.ishak at gmail.com (jacob ishak)
Date: Thu, 15 Jul 2010 17:11:24 +0300
Subject: [Linux-cluster] system-config-cluster to conga
In-Reply-To: <AANLkTimxaSh3u8CyuHIMHuUlE0OsQNy9SgoS5gU_AU3Y@mail.gmail.com>
References: <AANLkTimxaSh3u8CyuHIMHuUlE0OsQNy9SgoS5gU_AU3Y@mail.gmail.com>
Message-ID: <AANLkTinBg26g0ckZ85MFURm2Eh3fvDlpekluHmURYXTU@mail.gmail.com>

Hi

1. yes you can move from system-config-cluster to conga , after conga is up
from conga Web GUI you can discover already installed clusters and import
configuration
2. yes off course , everything relating to cluster configuration is the same
, the only thing that differs is that you are managing the cluster from
conga , you are not altering with the cluster config itself
3. off course cman have to be running all the time in order for ur cluster
to be up
4. conga configuration is straight forward and you can find it cluster admin
guides
5. never stop cman you cluster will crash



On Thu, Jul 15, 2010 at 7:22 AM, Mahantesh Chiniwar <mchiniwar at gmail.com>wrote:

> Hi,
>
> I have configured cluster using system-config-cluster on RHEL-5.4.
>
> Could you let me know if i can do the following
>
> 1. Can I move from system-config-cluster to conga?
>
> 2. Can I use the same cluster.conf in conga?
>
> 3. Can both cman and conga run together?
>
> 4. Currently in my setup cman service is running and I have started ricci
> service and have initilized the luci server on one of the cluster nodes but
> when i tried to start the luci service it fails. var/log/messages show just
> shows   "luci: Luci startup failed An error occurred while starting luci" I
> dont have any other information that this. What might be the reason?
>
> 5. I tried stopping cman service even this did not help.
>
>
> This information will help me.
>
>
>
> --
> Thanks,
> Mahantesh
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100715/e930c989/attachment.htm>

From mchiniwar at gmail.com  Thu Jul 15 15:38:51 2010
From: mchiniwar at gmail.com (Mahantesh Chiniwar)
Date: Thu, 15 Jul 2010 21:08:51 +0530
Subject: [Linux-cluster] system-config-cluster to conga
In-Reply-To: <AANLkTinBg26g0ckZ85MFURm2Eh3fvDlpekluHmURYXTU@mail.gmail.com>
References: <AANLkTimxaSh3u8CyuHIMHuUlE0OsQNy9SgoS5gU_AU3Y@mail.gmail.com>
	<AANLkTinBg26g0ckZ85MFURm2Eh3fvDlpekluHmURYXTU@mail.gmail.com>
Message-ID: <AANLkTinTdjvEPFGbqdLp_LLUNbHTdw8jMXXlL1FVJWnZ@mail.gmail.com>

Thank you for the responses..


On Thu, Jul 15, 2010 at 7:41 PM, jacob ishak <jacob.ishak at gmail.com> wrote:

> Hi
>
> 1. yes you can move from system-config-cluster to conga , after conga is up
> from conga Web GUI you can discover already installed clusters and import
> configuration
> 2. yes off course , everything relating to cluster configuration is the
> same , the only thing that differs is that you are managing the cluster from
> conga , you are not altering with the cluster config itself
> 3. off course cman have to be running all the time in order for ur cluster
> to be up
> 4. conga configuration is straight forward and you can find it cluster
> admin guides
> 5. never stop cman you cluster will crash
>
>
>
> On Thu, Jul 15, 2010 at 7:22 AM, Mahantesh Chiniwar <mchiniwar at gmail.com>wrote:
>
>> Hi,
>>
>> I have configured cluster using system-config-cluster on RHEL-5.4.
>>
>> Could you let me know if i can do the following
>>
>> 1. Can I move from system-config-cluster to conga?
>>
>> 2. Can I use the same cluster.conf in conga?
>>
>> 3. Can both cman and conga run together?
>>
>> 4. Currently in my setup cman service is running and I have started ricci
>> service and have initilized the luci server on one of the cluster nodes but
>> when i tried to start the luci service it fails. var/log/messages show just
>> shows   "luci: Luci startup failed An error occurred while starting luci" I
>> dont have any other information that this. What might be the reason?
>>
>> 5. I tried stopping cman service even this did not help.
>>
>>
>> This information will help me.
>>
>>
>>
>> --
>> Thanks,
>> Mahantesh
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 
Cheers,
Mahantesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100715/a972daba/attachment.htm>

From anoop_rajkumar at merck.com  Thu Jul 15 18:06:25 2010
From: anoop_rajkumar at merck.com (Rajkumar, Anoop)
Date: Thu, 15 Jul 2010 14:06:25 -0400
Subject: [Linux-cluster] Adding third node to existing cluster
Message-ID: <C651C3AA2A6A1D4980D35451DDE3F96B72170B@usctmx1160.merck.com>

Hi

Here are the firewall settings

#more /etc/sysconfig/iptables
# Generated by iptables-save v1.3.5 on Wed Jul 14 10:13:12 2010
*filter
:INPUT ACCEPT [2:186]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [4:486]
-A INPUT -i eth5 -p udp -m udp --dport 5405 -j ACCEPT
-A INPUT -i eth5 -p udp -m udp --sport 5405 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
-A INPUT -i eth5 -p udp -m udp --dport 50007 -j ACCEPT
-A INPUT -i eth5 -p udp -m udp --sport 50007 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
-A INPUT -i eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
-A OUTPUT -o eth5 -p udp -m udp --sport 5405 -j ACCEPT
-A OUTPUT -o eth5 -p udp -m udp --dport 5405 -j ACCEPT
-A OUTPUT -o eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
-A OUTPUT -o eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
-A OUTPUT -o eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
-A OUTPUT -o eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
-A OUTPUT -o eth5 -p udp -m udp --sport 50007 -j ACCEPT
-A OUTPUT -o eth5 -p udp -m udp --dport 50007 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
-A OUTPUT -o eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
COMMIT

Here are the IP in the system,I am using eth5 (Which is in private
network with other two nodes, connected to switch)

[root at usrylxap235 ~]# ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
    link/ether 00:1c:c4:f0:bd:d8 brd ff:ff:ff:ff:ff:ff
    inet 54.3.254.235/24 brd 54.3.254.255 scope global eth0
    inet6 fe80::21c:c4ff:fef0:bdd8/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:1c:c4:f0:bd:da brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:1c:c4:5e:f8:d8 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:1c:c4:5e:f8:da brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:1e:0b:71:ac:6c brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
    link/ether 00:1e:0b:71:ac:6e brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.7/24 brd 192.168.0.255 scope global eth5
    inet6 fe80::21e:bff:fe71:ac6e/64 scope link
       valid_lft forever preferred_lft forever
8: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0


Thanks
Anoop

>Date: Thu, 15 Jul 2010 11:21:12 +0800
>From: Bernard Chew <bernardchew at gmail.com>
>To: linux clustering <linux-cluster at redhat.com>
>Subject: Re: [Linux-cluster] Adding third node to existing cluster
	problem
Message-ID:
	<AANLkTintfM8Sea-B1xbakxPm8iheuDY1hFmxpi-c2S-r at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

> On Thu, Jul 15, 2010 at 4:47 AM, Rajkumar, Anoop
<anoop_rajkumar at merck.com> wrote:
> Hi
>
> Yes I have done this way. When I try to start on third node cman start
fails
> and cman_tool shows that it is not recognizing other two nodes which
are
> working properly..
>
> [root at usrylxap235 ~]# service cman start
> Starting cluster:
> ?? Loading modules... done
> ?? Mounting configfs... done
> ?? Starting ccsd... done
> ?? Starting cman... done
> ?? Starting daemons... done
> ?? Starting fencing...
>
>
> failed
>
> ?????????????????????????????????????????????????????????? [FAILED]
>
> [root at usrylxap235 ~]# cman_tool status
> Version: 6.1.0
> Config Version: 44
> Cluster Name: cluster1
> Cluster Id: 26777
> Cluster Member: Yes
> Cluster Generation: 84
> Membership state: Cluster-Member
> Nodes: 1
> Expected votes: 3
> Total votes: 1
> Quorum: 2 Activity blocked
> Active subsystems: 5
> Flags:
> Ports Bound: 0
> Node name: usrylxap235-1.merck.com
> Node ID: 3
> Multicast addresses: 239.192.104.2
> Node addresses: 192.168.0.7
>
> Attached is cluster.conf file.
> <<clusterconfig.txt>>
> Thanks
> Anoop
>
> Notice:  This e-mail message, together with any attachments, contains
> information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
> New Jersey, USA 08889), and/or its affiliates Direct contact
information
> for affiliates is available at
> http://www.merck.com/contact/contacts.html) that may be confidential,
> proprietary copyrighted and/or legally privileged. It is intended
solely
> for the use of the individual or entity named on this message. If you
are
> not the intended recipient, and have received this message in error,
> please notify us immediately by reply e-mail and then delete it from
> your system.
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>

Hi,

Have you checked the firewall settings?

Regards,
Bernard


Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100715/1a04a66e/attachment.htm>

From justinshafer at gmail.com  Fri Jul 16 06:02:34 2010
From: justinshafer at gmail.com (Justin Shafer)
Date: Fri, 16 Jul 2010 01:02:34 -0500
Subject: [Linux-cluster] Stateful Samba\CTDB Failover
Message-ID: <000501cb24ac$7d0f2330$772d6990$@com>

I have read this on the mailing list..
http://www.mail-archive.com/linux-cluster at redhat.com/msg08757.html

 

Basically I want a program called Dentrix which only does SMB1 before
migrating to SQL, I want it to have a stateful failover which right now
doesn't seem possible with Samba. I read in that archive, that you guys are
waiting for SMB2 and Durable File Handles for stateful failover.

 

Just one question. Microsoft can do this perfectly with failover server
2008, steel-eye, and xp as a client, and Dentrix doesn't make durable file
requests.. But somehow the failover is perfect with Microsoft.

 

With Samba, my mapped drive is always there during failover, no data loss,
etc. But Dentrix will gripe and say "another file is open at a workstation"
and I have to close and open the program. If Microsoft can do failover with
Dentrix, Samba should be able to.. But of course saying it is easier then
doing it, Im sure. It costs a lot to do it with MS.

 

-Justin Shafer

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100716/507b6bca/attachment.htm>

From mchiniwar at gmail.com  Fri Jul 16 09:19:46 2010
From: mchiniwar at gmail.com (Mahantesh Chiniwar)
Date: Fri, 16 Jul 2010 14:49:46 +0530
Subject: [Linux-cluster]  unable to start luci service
Message-ID: <AANLkTinAaQPqGTbsTAqXm7qRdfdD3ap2iXh0EOfeYZYF@mail.gmail.com>

Hi,

I am trying to setup a Conga cluster on Redhat 5.4 and i am unable to start
the luci service.


Details:

[root at RHEL-N1 ~]# rpm -qa | grep luci
luci-0.12.2-6.el5

[root at RHEL-N1 ~]# rpm -qa | grep ricci
ricci-0.12.2-6.el5

When I start ricci service i get the following message in /var/log/message,
but the service starts.

Jul 16 14:31:47 RHEL-N1 saslauthd[2413]: detach_tty      : could not lock
pid file /var/run/saslauthd/saslauthd.pid: Resource temporarily unavailable
Jul 16 14:31:47 RHEL-N1 saslauthd[2412]: detach_tty      : Cannot start
saslauthd
Jul 16 14:31:47 RHEL-N1 saslauthd[2412]: detach_tty      : Another instance
of saslauthd is currently running
Jul 16 14:31:47 RHEL-N1 ricci: startup succeeded

When I try to start luci service it fails to start. /var/log/message shows
"luci: Luci startup failed An error occurred while starting luci"


Could you let me know what is causing this issue?



-- 
Mahantesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100716/70201d85/attachment.htm>

From jayfitzpatrick at gmail.com  Fri Jul 16 12:55:42 2010
From: jayfitzpatrick at gmail.com (Jason Fitzpatrick)
Date: Fri, 16 Jul 2010 13:55:42 +0100
Subject: [Linux-cluster] Stateful Samba\CTDB Failover
In-Reply-To: <000501cb24ac$7d0f2330$772d6990$@com>
References: <000501cb24ac$7d0f2330$772d6990$@com>
Message-ID: <AANLkTimpRBa7ppskw0NWuoDy-x3Z0SkSMTYNzepC8JOl@mail.gmail.com>

Hi Justin.

My understanding of all this is that SMB2 was only introduced with
Vista (http://blogs.technet.com/b/josebda/archive/2008/12/05/smb2-a-complete-redesign-of-the-main-remote-file-protocol-for-windows.aspx)
and as a result your client has to be using SMB1

I was looking into SMBv4 for RHEL and it looks like you are going to
have to pull from the testing tree, which is not something that I am
willing to do as this was planned for a production environment, hard
enough to sell Linux to a Windows audience without having to explain
why I am using an unstable version of SAMBA

Long story short, due to the use of DRBD to replicate across our
datacenters, and the fact that I am using CTDB to cluster the IP
addresses of the clustered resource I found that the failover between
nodes was faster than the Windows cluster (no need to fail over the
disk as it was active active - saving a couple of seconds) but no
statefull, so the cluster has been dropped down from being a high
visability system to a proof of concept, I will be revisiting the
statefull failover when SMB2 is available via Redhat repos

Jay

On 16 July 2010 07:02, Justin Shafer <justinshafer at gmail.com> wrote:
> I have read this on the mailing list..
> http://www.mail-archive.com/linux-cluster at redhat.com/msg08757.html
>
>
>
> Basically I want a program called Dentrix which only does SMB1 before
> migrating to SQL, I want it to have a stateful failover which right now
> doesn?t seem possible with Samba. I read in that archive, that you guys are
> waiting for SMB2 and Durable File Handles for stateful failover.
>
>
>
> Just one question. Microsoft can do this perfectly with failover server
> 2008, steel-eye, and xp as a client, and Dentrix doesn?t make durable file
> requests.. But somehow the failover is perfect with Microsoft.
>
>
>
> With Samba, my mapped drive is always there during failover, no data loss,
> etc. But Dentrix will gripe and say ?another file is open at a workstation?
> and I have to close and open the program. If Microsoft can do failover with
> Dentrix, Samba should be able to.. But of course saying it is easier then
> doing it, Im sure. It costs a lot to do it with MS.
>
>
>
> -Justin Shafer
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 

"The only difference between saints and sinners is that every saint
has a past while every sinner has a future. "
? Oscar Wilde



From Martin.Waite at datacash.com  Fri Jul 16 13:18:22 2010
From: Martin.Waite at datacash.com (Martin Waite)
Date: Fri, 16 Jul 2010 14:18:22 +0100
Subject: [Linux-cluster] NTP time steps causes cluster reconfiguration
Message-ID: <A78DB34D00374344A0AB65B6523C05DC05BD6B81@marsden.win.datacash.com>

Hi,

 

During testing, I noticed that a time step caused by ntpd caused the
cluster to drop into GATHER state:

 

Jun 16 12:13:16 cp1edidbm001 ntpd[30917]: time reset -16.332117 s

Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering GATHER
state from 12.

Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Creating commit
token because I am the rep.

Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Saving state aru 9e
high seq received 9e

Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Storing new
sequence id for ring 328

Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering COMMIT
state.

Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering RECOVERY
state.

...

 

This is easily repeatable through setting the clock forwards by 20
seconds using /bin/date.  This probably causes comms timeouts to expire
prematurely, and almost every time causes the cluster to reconfigure -
luckily without affecting running services.

 

Stepping the clock backwards also causes a similar disruption, but there
is a long lag between changing the time and the cluster reconfiguring:
perhaps this extends a timeout or sleep on the affected node, causing
genuine timeouts on the other nodes.

 

All I am looking for is some reassurance that clock changes are not
going to crash the cluster.  Is anyone able to confirm this please ?

 

regards,

Martin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100716/80dfbadc/attachment.htm>

From justinshafer at gmail.com  Fri Jul 16 13:41:04 2010
From: justinshafer at gmail.com (Justin Shafer)
Date: Fri, 16 Jul 2010 08:41:04 -0500
Subject: [Linux-cluster] Stateful Samba\CTDB Failover
In-Reply-To: <AANLkTimpRBa7ppskw0NWuoDy-x3Z0SkSMTYNzepC8JOl@mail.gmail.com>
References: <000501cb24ac$7d0f2330$772d6990$@com>
	<AANLkTimpRBa7ppskw0NWuoDy-x3Z0SkSMTYNzepC8JOl@mail.gmail.com>
Message-ID: <002001cb24ec$8a5c55e0$9f1501a0$@com>

Yep I agree. The failover itself is almost instant.. I too am running samba
ctdb in active/active with corosync/drbd/ocfs2.. though I haven't added ctdb
as a resource to corosync, I have just been starting it manually. Corosync
wants to know there the SMB private directory is and I was told by someone
at samba that ctdb can figure all that out for itself... so I am wondering
why its still needed in the resource... But I figure by the time I would use
it in production, all that stuff will be answered. =)

Like novell says that the smb_private_dir with sysconfig/ctdb should have it
in there, and the private directory should be on shared storage.. along with
your filesystem your serving out.

But I was told differently by someone at samba...  so maybe the
smb_private_dir is just old....  Eitherway when I failover my mapped drive
is still in tact.. If I was in an excel spreadsheet, I would be a happy guy.
But Im trying Dentrix... sigh. (they should be on sql by now anyways).

-Justin

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Jason Fitzpatrick
Sent: Friday, July 16, 2010 7:56 AM
To: linux clustering
Subject: Re: [Linux-cluster] Stateful Samba\CTDB Failover

Hi Justin.

My understanding of all this is that SMB2 was only introduced with
Vista
(http://blogs.technet.com/b/josebda/archive/2008/12/05/smb2-a-complete-redes
ign-of-the-main-remote-file-protocol-for-windows.aspx)
and as a result your client has to be using SMB1

I was looking into SMBv4 for RHEL and it looks like you are going to
have to pull from the testing tree, which is not something that I am
willing to do as this was planned for a production environment, hard
enough to sell Linux to a Windows audience without having to explain
why I am using an unstable version of SAMBA

Long story short, due to the use of DRBD to replicate across our
datacenters, and the fact that I am using CTDB to cluster the IP
addresses of the clustered resource I found that the failover between
nodes was faster than the Windows cluster (no need to fail over the
disk as it was active active - saving a couple of seconds) but no
statefull, so the cluster has been dropped down from being a high
visability system to a proof of concept, I will be revisiting the
statefull failover when SMB2 is available via Redhat repos

Jay

On 16 July 2010 07:02, Justin Shafer <justinshafer at gmail.com> wrote:
> I have read this on the mailing list..
> http://www.mail-archive.com/linux-cluster at redhat.com/msg08757.html
>
>
>
> Basically I want a program called Dentrix which only does SMB1 before
> migrating to SQL, I want it to have a stateful failover which right now
> doesn't seem possible with Samba. I read in that archive, that you guys
are
> waiting for SMB2 and Durable File Handles for stateful failover.
>
>
>
> Just one question. Microsoft can do this perfectly with failover server
> 2008, steel-eye, and xp as a client, and Dentrix doesn't make durable file
> requests.. But somehow the failover is perfect with Microsoft.
>
>
>
> With Samba, my mapped drive is always there during failover, no data loss,
> etc. But Dentrix will gripe and say "another file is open at a
workstation"
> and I have to close and open the program. If Microsoft can do failover
with
> Dentrix, Samba should be able to.. But of course saying it is easier then
> doing it, Im sure. It costs a lot to do it with MS.
>
>
>
> -Justin Shafer
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 

"The only difference between saints and sinners is that every saint
has a past while every sinner has a future. "
- Oscar Wilde

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From justinshafer at gmail.com  Fri Jul 16 13:49:42 2010
From: justinshafer at gmail.com (Justin Shafer)
Date: Fri, 16 Jul 2010 08:49:42 -0500
Subject: [Linux-cluster] Stateful Samba\CTDB Failover
In-Reply-To: <AANLkTimpRBa7ppskw0NWuoDy-x3Z0SkSMTYNzepC8JOl@mail.gmail.com>
References: <000501cb24ac$7d0f2330$772d6990$@com>
	<AANLkTimpRBa7ppskw0NWuoDy-x3Z0SkSMTYNzepC8JOl@mail.gmail.com>
Message-ID: <002501cb24ed$bf55c730$3e015590$@com>

Also, Im not using the round-robin method for failover with DNS records, Im
using the LVS method. Don't think that matters.... And I don't know if
Corosync expects to use LVS or go off the public_addresses file as corosync
is supposed to take your ctdb file and reconfigure it.. Like should I have
corosync do the ip failover or CTDB. So I just have ctdb do it all, since
that seems good. Maybe its time to go play with cman, but without that cool
transparent failover, I feel like I have pretty much hit the end of the
road. This is as cool as its going to get, for today.

-J

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Jason Fitzpatrick
Sent: Friday, July 16, 2010 7:56 AM
To: linux clustering
Subject: Re: [Linux-cluster] Stateful Samba\CTDB Failover

Hi Justin.

My understanding of all this is that SMB2 was only introduced with
Vista
(http://blogs.technet.com/b/josebda/archive/2008/12/05/smb2-a-complete-redes
ign-of-the-main-remote-file-protocol-for-windows.aspx)
and as a result your client has to be using SMB1

I was looking into SMBv4 for RHEL and it looks like you are going to
have to pull from the testing tree, which is not something that I am
willing to do as this was planned for a production environment, hard
enough to sell Linux to a Windows audience without having to explain
why I am using an unstable version of SAMBA

Long story short, due to the use of DRBD to replicate across our
datacenters, and the fact that I am using CTDB to cluster the IP
addresses of the clustered resource I found that the failover between
nodes was faster than the Windows cluster (no need to fail over the
disk as it was active active - saving a couple of seconds) but no
statefull, so the cluster has been dropped down from being a high
visability system to a proof of concept, I will be revisiting the
statefull failover when SMB2 is available via Redhat repos

Jay

On 16 July 2010 07:02, Justin Shafer <justinshafer at gmail.com> wrote:
> I have read this on the mailing list..
> http://www.mail-archive.com/linux-cluster at redhat.com/msg08757.html
>
>
>
> Basically I want a program called Dentrix which only does SMB1 before
> migrating to SQL, I want it to have a stateful failover which right now
> doesn't seem possible with Samba. I read in that archive, that you guys
are
> waiting for SMB2 and Durable File Handles for stateful failover.
>
>
>
> Just one question. Microsoft can do this perfectly with failover server
> 2008, steel-eye, and xp as a client, and Dentrix doesn't make durable file
> requests.. But somehow the failover is perfect with Microsoft.
>
>
>
> With Samba, my mapped drive is always there during failover, no data loss,
> etc. But Dentrix will gripe and say "another file is open at a
workstation"
> and I have to close and open the program. If Microsoft can do failover
with
> Dentrix, Samba should be able to.. But of course saying it is easier then
> doing it, Im sure. It costs a lot to do it with MS.
>
>
>
> -Justin Shafer
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 

"The only difference between saints and sinners is that every saint
has a past while every sinner has a future. "
- Oscar Wilde

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From kkovachev at varna.net  Fri Jul 16 14:36:17 2010
From: kkovachev at varna.net (Kaloyan Kovachev)
Date: Fri, 16 Jul 2010 17:36:17 +0300
Subject: [Linux-cluster] NTP time steps causes cluster reconfiguration
In-Reply-To: <A78DB34D00374344A0AB65B6523C05DC05BD6B81@marsden.win.datacash.com>
References: <A78DB34D00374344A0AB65B6523C05DC05BD6B81@marsden.win.datacash.com>
Message-ID: <4c44d9e646f5ab99e9f6c02fe566adb5@mx.varna.net>

Hi,
 i can confirm, that time steps do cause reconfiguration. Not sure if this
was the reason, but one of my nodes was fenced from time to time
(previously) after several reconfigurations and also it caused some
problems with gfs being withdrawn.
 ntpdate running as cron job does step changes, but ntpd should not cause
step changes. It should instead speed-up or slow-down the clock until it is
synchronized. However using the -g option you may ask that the clock jumps
once at the start of ntpd.
 I have configured all cluster nodes to synchronize from each other via
ntpd (configured as peers) and each from one (different) additional
(startum 1 or 2) source as server. Since then i don't see reconfiguration
in the logs.

On Fri, 16 Jul 2010 14:18:22 +0100, "Martin Waite"
<Martin.Waite at datacash.com> wrote:
> Hi,
> 
>  
> 
> During testing, I noticed that a time step caused by ntpd caused the
> cluster to drop into GATHER state:
> 
>  
> 
> Jun 16 12:13:16 cp1edidbm001 ntpd[30917]: time reset -16.332117 s
> 
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering GATHER
> state from 12.
> 
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Creating commit
> token because I am the rep.
> 
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Saving state aru 9e
> high seq received 9e
> 
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Storing new
> sequence id for ring 328
> 
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering COMMIT
> state.
> 
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering RECOVERY
> state.
> 
> ...
> 
>  
> 
> This is easily repeatable through setting the clock forwards by 20
> seconds using /bin/date.  This probably causes comms timeouts to expire
> prematurely, and almost every time causes the cluster to reconfigure -
> luckily without affecting running services.
> 
>  
> 
> Stepping the clock backwards also causes a similar disruption, but there
> is a long lag between changing the time and the cluster reconfiguring:
> perhaps this extends a timeout or sleep on the affected node, causing
> genuine timeouts on the other nodes.
> 
>  
> 
> All I am looking for is some reassurance that clock changes are not
> going to crash the cluster.  Is anyone able to confirm this please ?
> 
>  
> 
> regards,
> 
> Martin



From Martin.Waite at datacash.com  Fri Jul 16 15:11:38 2010
From: Martin.Waite at datacash.com (Martin Waite)
Date: Fri, 16 Jul 2010 16:11:38 +0100
Subject: [Linux-cluster] NTP time steps causes cluster reconfiguration
In-Reply-To: <4c44d9e646f5ab99e9f6c02fe566adb5@mx.varna.net>
References: <A78DB34D00374344A0AB65B6523C05DC05BD6B81@marsden.win.datacash.com>
	<4c44d9e646f5ab99e9f6c02fe566adb5@mx.varna.net>
Message-ID: <A78DB34D00374344A0AB65B6523C05DC05BD6C5A@marsden.win.datacash.com>

Hi,

NTP has a step-threshold - if the time difference is greater than the
threshold, it will step the time rather than speeding it up or down.  So
even using ntpd can cause clock steps (especially in our test
environment where our crappy overloaded NTP servers sometimes lose 30
seconds).

On some VMware test hosts, I did manage to make the cluster fence some
nodes through changing the time backwards and forwards, but I could not
reproduce the effect on physical hosts.     I was hoping that the
fencing was caused by a combination of clock changes and VM guest timing
flakiness, but from your description, it sounds like this might be a
real risk on physical servers too.

I had better do some more testing.

Thanks for the input.

regards,
Martin

> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com]
> On Behalf Of Kaloyan Kovachev
> Sent: 16 July 2010 15:36
> To: linux clustering
> Subject: Re: [Linux-cluster] NTP time steps causes cluster
reconfiguration
> 
> Hi,
>  i can confirm, that time steps do cause reconfiguration. Not sure if
this
> was the reason, but one of my nodes was fenced from time to time
> (previously) after several reconfigurations and also it caused some
> problems with gfs being withdrawn.
>  ntpdate running as cron job does step changes, but ntpd should not
cause
> step changes. It should instead speed-up or slow-down the clock until
it is
> synchronized. However using the -g option you may ask that the clock
jumps
> once at the start of ntpd.
>  I have configured all cluster nodes to synchronize from each other
via
> ntpd (configured as peers) and each from one (different) additional
> (startum 1 or 2) source as server. Since then i don't see
reconfiguration
> in the logs.
> 
> On Fri, 16 Jul 2010 14:18:22 +0100, "Martin Waite"
> <Martin.Waite at datacash.com> wrote:
> > Hi,
> >
> >
> >
> > During testing, I noticed that a time step caused by ntpd caused the
> > cluster to drop into GATHER state:
> >
> >
> >
> > Jun 16 12:13:16 cp1edidbm001 ntpd[30917]: time reset -16.332117 s
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering GATHER
> > state from 12.
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Creating commit
> > token because I am the rep.
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Saving state
aru 9e
> > high seq received 9e
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Storing new
> > sequence id for ring 328
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering COMMIT
> > state.
> >
> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering
RECOVERY
> > state.
> >
> > ...
> >
> >
> >
> > This is easily repeatable through setting the clock forwards by 20
> > seconds using /bin/date.  This probably causes comms timeouts to
expire
> > prematurely, and almost every time causes the cluster to reconfigure
-
> > luckily without affecting running services.
> >
> >
> >
> > Stepping the clock backwards also causes a similar disruption, but
there
> > is a long lag between changing the time and the cluster
reconfiguring:
> > perhaps this extends a timeout or sleep on the affected node,
causing
> > genuine timeouts on the other nodes.
> >
> >
> >
> > All I am looking for is some reassurance that clock changes are not
> > going to crash the cluster.  Is anyone able to confirm this please ?
> >
> >
> >
> > regards,
> >
> > Martin
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From kkovachev at varna.net  Fri Jul 16 16:27:09 2010
From: kkovachev at varna.net (Kaloyan Kovachev)
Date: Fri, 16 Jul 2010 19:27:09 +0300
Subject: [Linux-cluster] NTP time steps causes cluster reconfiguration
In-Reply-To: <A78DB34D00374344A0AB65B6523C05DC05BD6C5A@marsden.win.datacash.com>
References: <A78DB34D00374344A0AB65B6523C05DC05BD6B81@marsden.win.datacash.com>
	<4c44d9e646f5ab99e9f6c02fe566adb5@mx.varna.net>
	<A78DB34D00374344A0AB65B6523C05DC05BD6C5A@marsden.win.datacash.com>
Message-ID: <7747fd70a949aacb6b320857b471e725@mx.varna.net>

On Fri, 16 Jul 2010 16:11:38 +0100, "Martin Waite"
<Martin.Waite at datacash.com> wrote:
> Hi,
> 
> NTP has a step-threshold - if the time difference is greater than the
> threshold, it will step the time rather than speeding it up or down.  So
> even using ntpd can cause clock steps (especially in our test
> environment where our crappy overloaded NTP servers sometimes lose 30
> seconds).

That's why i have setup all nodes as peers to each other - they will try
to synchronize each other and as the delay is minimal between the hosts no
big changes in their clocks would be possible except on reboot (before cman
is started). One or two additional time servers (nearest to you from
pool.ntp.org), but different for each host would avoid having the time
offset on the cluster too far from the real world

> 
> On some VMware test hosts, I did manage to make the cluster fence some
> nodes through changing the time backwards and forwards, but I could not
> reproduce the effect on physical hosts.     I was hoping that the
> fencing was caused by a combination of clock changes and VM guest timing
> flakiness, but from your description, it sounds like this might be a
> real risk on physical servers too.
> 
> I had better do some more testing.
> 
> Thanks for the input.
> 
> regards,
> Martin
> 
>> -----Original Message-----
>> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com]
>> On Behalf Of Kaloyan Kovachev
>> Sent: 16 July 2010 15:36
>> To: linux clustering
>> Subject: Re: [Linux-cluster] NTP time steps causes cluster
> reconfiguration
>> 
>> Hi,
>>  i can confirm, that time steps do cause reconfiguration. Not sure if
> this
>> was the reason, but one of my nodes was fenced from time to time
>> (previously) after several reconfigurations and also it caused some
>> problems with gfs being withdrawn.
>>  ntpdate running as cron job does step changes, but ntpd should not
> cause
>> step changes. It should instead speed-up or slow-down the clock until
> it is
>> synchronized. However using the -g option you may ask that the clock
> jumps
>> once at the start of ntpd.
>>  I have configured all cluster nodes to synchronize from each other
> via
>> ntpd (configured as peers) and each from one (different) additional
>> (startum 1 or 2) source as server. Since then i don't see
> reconfiguration
>> in the logs.
>> 
>> On Fri, 16 Jul 2010 14:18:22 +0100, "Martin Waite"
>> <Martin.Waite at datacash.com> wrote:
>> > Hi,
>> >
>> >
>> >
>> > During testing, I noticed that a time step caused by ntpd caused the
>> > cluster to drop into GATHER state:
>> >
>> >
>> >
>> > Jun 16 12:13:16 cp1edidbm001 ntpd[30917]: time reset -16.332117 s
>> >
>> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering GATHER
>> > state from 12.
>> >
>> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Creating commit
>> > token because I am the rep.
>> >
>> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Saving state
> aru 9e
>> > high seq received 9e
>> >
>> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Storing new
>> > sequence id for ring 328
>> >
>> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering COMMIT
>> > state.
>> >
>> > Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering
> RECOVERY
>> > state.
>> >
>> > ...
>> >
>> >
>> >
>> > This is easily repeatable through setting the clock forwards by 20
>> > seconds using /bin/date.  This probably causes comms timeouts to
> expire
>> > prematurely, and almost every time causes the cluster to reconfigure
> -
>> > luckily without affecting running services.
>> >
>> >
>> >
>> > Stepping the clock backwards also causes a similar disruption, but
> there
>> > is a long lag between changing the time and the cluster
> reconfiguring:
>> > perhaps this extends a timeout or sleep on the affected node,
> causing
>> > genuine timeouts on the other nodes.
>> >
>> >
>> >
>> > All I am looking for is some reassurance that clock changes are not
>> > going to crash the cluster.  Is anyone able to confirm this please ?
>> >
>> >
>> >
>> > regards,
>> >
>> > Martin
>> 
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From lagern at lafayette.edu  Fri Jul 16 17:21:57 2010
From: lagern at lafayette.edu (Nathan Lager)
Date: Fri, 16 Jul 2010 13:21:57 -0400
Subject: [Linux-cluster] How do i stop VM's on a failed node?
Message-ID: <4C409535.4030603@lafayette.edu>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have a 4-node cluster, running KVM, and a host of VM's.

One of the nodes failed unexpectedly.  I'm having two issues.  First,
the VM's which were on that node are still reported as up and running on
that node in clustat, and second, i'm unable to interact with the
cluster using clusvcadm.  Every command hangs.  I've tried disabling
these VM's, and the commands hang.

How can i clear out this dead node, without having to forceably restart
my entire cluster?


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkxAlTUACgkQsZqG4IN3sukarACgqcL1wQrjc5PR1rDfSAX6ojJP
RWUAn2KFAmKYf5MX8+oC5aws+v1eAxts
=fNb8
-----END PGP SIGNATURE-----



From crh at ubiqx.mn.org  Fri Jul 16 17:28:28 2010
From: crh at ubiqx.mn.org (Christopher R. Hertel)
Date: Fri, 16 Jul 2010 12:28:28 -0500
Subject: [Linux-cluster] Stateful Samba\CTDB Failover
In-Reply-To: <000501cb24ac$7d0f2330$772d6990$@com>
References: <000501cb24ac$7d0f2330$772d6990$@com>
Message-ID: <4C4096BC.4010004@ubiqx.mn.org>

Are you running Samba with CTDB on a cluster?

This question really belongs on the samba-technical mailing list since it
probably has more to do with Samba's state management than the underlying
cluster file system.

Chris -)-----

Justin Shafer wrote:
> I have read this on the mailing list..
> http://www.mail-archive.com/linux-cluster at redhat.com/msg08757.html
> 
>  
> 
> Basically I want a program called Dentrix which only does SMB1 before
> migrating to SQL, I want it to have a stateful failover which right now
> doesn?t seem possible with Samba. I read in that archive, that you guys
> are waiting for SMB2 and Durable File Handles for stateful failover.
> 
>  
> 
> Just one question. Microsoft can do this perfectly with failover server
> 2008, steel-eye, and xp as a client, and Dentrix doesn?t make durable
> file requests.. But somehow the failover is perfect with Microsoft.
> 
>  
> 
> With Samba, my mapped drive is always there during failover, no data
> loss, etc. But Dentrix will gripe and say ?another file is open at a
> workstation? and I have to close and open the program. If Microsoft can
> do failover with Dentrix, Samba should be able to.. But of course saying
> it is easier then doing it, Im sure. It costs a lot to do it with MS.
> 
>  
> 
> -Justin Shafer
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh at ubiqx.mn.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh at ubiqx.org



From sdake at redhat.com  Fri Jul 16 20:36:32 2010
From: sdake at redhat.com (Steven Dake)
Date: Fri, 16 Jul 2010 13:36:32 -0700
Subject: [Linux-cluster] NTP time steps causes cluster reconfiguration
In-Reply-To: <A78DB34D00374344A0AB65B6523C05DC05BD6B81@marsden.win.datacash.com>
References: <A78DB34D00374344A0AB65B6523C05DC05BD6B81@marsden.win.datacash.com>
Message-ID: <4C40C2D0.5010802@redhat.com>

On 07/16/2010 06:18 AM, Martin Waite wrote:
> Hi,
>
> During testing, I noticed that a time step caused by ntpd caused the
> cluster to drop into GATHER state:
>
> Jun 16 12:13:16 cp1edidbm001 ntpd[30917]: time reset -16.332117 s
>
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering GATHER
> state from 12.
>
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Creating commit
> token because I am the rep.
>
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Saving state aru 9e
> high seq received 9e
>
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] Storing new
> sequence id for ring 328
>
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering COMMIT state.
>
> Jun 16 12:13:26 cp1edidbm001 openais[15929]: [TOTEM] entering RECOVERY
> state.
>
> ...
>
> This is easily repeatable through setting the clock forwards by 20
> seconds using /bin/date. This probably causes comms timeouts to expire
> prematurely, and almost every time causes the cluster to reconfigure -
> luckily without affecting running services.
>
> Stepping the clock backwards also causes a similar disruption, but there
> is a long lag between changing the time and the cluster reconfiguring:
> perhaps this extends a timeout or sleep on the affected node, causing
> genuine timeouts on the other nodes.
>
> All I am looking for is some reassurance that clock changes are not
> going to crash the cluster. Is anyone able to confirm this please ?
>
> regards,
>
> Martin
>

Martin,

NTP integration is not available with openais.  That feature has been 
introduced into corosync.

Regards
-steve

>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From bernardchew at gmail.com  Sat Jul 17 15:35:40 2010
From: bernardchew at gmail.com (Bernard Chew)
Date: Sat, 17 Jul 2010 23:35:40 +0800
Subject: [Linux-cluster] Adding third node to existing cluster
In-Reply-To: <C651C3AA2A6A1D4980D35451DDE3F96B72170B@usctmx1160.merck.com>
References: <C651C3AA2A6A1D4980D35451DDE3F96B72170B@usctmx1160.merck.com>
Message-ID: <AANLkTin8o2OU-UQM2fWpAM7Qqvgsb2WKc4T6uFrNdTqE@mail.gmail.com>

> On Fri, Jul 16, 2010 at 2:06 AM, Rajkumar, Anoop <anoop_rajkumar at merck.com> wrote:
> Hi
>
> Here are the firewall settings
>
> #more /etc/sysconfig/iptables
> # Generated by iptables-save v1.3.5 on Wed Jul 14 10:13:12 2010
> *filter
> :INPUT ACCEPT [2:186]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [4:486]
> -A INPUT -i eth5 -p udp -m udp --dport 5405 -j ACCEPT
> -A INPUT -i eth5 -p udp -m udp --sport 5405 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
> -A INPUT -i eth5 -p udp -m udp --dport 50007 -j ACCEPT
> -A INPUT -i eth5 -p udp -m udp --sport 50007 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --sport 5405 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --dport 5405 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --sport 50007 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --dport 50007 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
> COMMIT
>
> Here are the IP in the system,I am using eth5 (Which is in private network
> with other two nodes, connected to switch)
>
> [root at usrylxap235 ~]# ip addr list
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
> ??? link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> ??? inet 127.0.0.1/8 scope host lo
> ??? inet6 ::1/128 scope host
> ?????? valid_lft forever preferred_lft forever
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
> 1000
> ??? link/ether 00:1c:c4:f0:bd:d8 brd ff:ff:ff:ff:ff:ff
> ??? inet 54.3.254.235/24 brd 54.3.254.255 scope global eth0
> ??? inet6 fe80::21c:c4ff:fef0:bdd8/64 scope link
> ?????? valid_lft forever preferred_lft forever
> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
> ??? link/ether 00:1c:c4:f0:bd:da brd ff:ff:ff:ff:ff:ff
> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
> ??? link/ether 00:1c:c4:5e:f8:d8 brd ff:ff:ff:ff:ff:ff
> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
> ??? link/ether 00:1c:c4:5e:f8:da brd ff:ff:ff:ff:ff:ff
> 6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
> ??? link/ether 00:1e:0b:71:ac:6c brd ff:ff:ff:ff:ff:ff
> 7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
> 1000
> ??? link/ether 00:1e:0b:71:ac:6e brd ff:ff:ff:ff:ff:ff
> ??? inet 192.168.0.7/24 brd 192.168.0.255 scope global eth5
> ??? inet6 fe80::21e:bff:fe71:ac6e/64 scope link
> ?????? valid_lft forever preferred_lft forever
> 8: sit0: <NOARP> mtu 1480 qdisc noop
> ??? link/sit 0.0.0.0 brd 0.0.0.0
>
> Thanks
> Anoop

Hi,

Let's see if we can rule out security or network issues here. Are you
able to allow all traffic for eth5 on all nodes? In addition, you may
want to add a static multicast route to ensure the multicast traffic
is going through eth5 on all nodes.

Regards,
Bernard



From RJM002 at shsu.edu  Sun Jul 18 03:10:52 2010
From: RJM002 at shsu.edu (Marti, Robert)
Date: Sat, 17 Jul 2010 22:10:52 -0500
Subject: [Linux-cluster] Adding third node to existing cluster
In-Reply-To: <AANLkTin8o2OU-UQM2fWpAM7Qqvgsb2WKc4T6uFrNdTqE@mail.gmail.com>
References: <C651C3AA2A6A1D4980D35451DDE3F96B72170B@usctmx1160.merck.com>,
	<AANLkTin8o2OU-UQM2fWpAM7Qqvgsb2WKc4T6uFrNdTqE@mail.gmail.com>
Message-ID: <8FAC1E47484E43469AA28DBF35C955E4BDE0B47EAC@EXMBX.SHSU.EDU>


> On Fri, Jul 16, 2010 at 2:06 AM, Rajkumar, Anoop <anoop_rajkumar at merck.com> wrote:
> Hi
>
> Here are the firewall settings
>
> #more /etc/sysconfig/iptables
> # Generated by iptables-save v1.3.5 on Wed Jul 14 10:13:12 2010
> *filter
> :INPUT ACCEPT [2:186]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [4:486]
> -A INPUT -i eth5 -p udp -m udp --dport 5405 -j ACCEPT
> -A INPUT -i eth5 -p udp -m udp --sport 5405 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
> -A INPUT -i eth5 -p udp -m udp --dport 50007 -j ACCEPT
> -A INPUT -i eth5 -p udp -m udp --sport 50007 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --sport 5405 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --dport 5405 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --sport 50007 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --dport 50007 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
> COMMIT
>
> Here are the IP in the system,I am using eth5 (Which is in private network
> with other two nodes, connected to switch)
>
> [root at usrylxap235 ~]# ip addr list
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
>     inet6 ::1/128 scope host
>        valid_lft forever preferred_lft forever
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
> 1000
>     link/ether 00:1c:c4:f0:bd:d8 brd ff:ff:ff:ff:ff:ff
>     inet 54.3.254.235/24 brd 54.3.254.255 scope global eth0
>     inet6 fe80::21c:c4ff:fef0:bdd8/64 scope link
>        valid_lft forever preferred_lft forever
> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>     link/ether 00:1c:c4:f0:bd:da brd ff:ff:ff:ff:ff:ff
> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>     link/ether 00:1c:c4:5e:f8:d8 brd ff:ff:ff:ff:ff:ff
> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>     link/ether 00:1c:c4:5e:f8:da brd ff:ff:ff:ff:ff:ff
> 6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>     link/ether 00:1e:0b:71:ac:6c brd ff:ff:ff:ff:ff:ff
> 7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
> 1000
>     link/ether 00:1e:0b:71:ac:6e brd ff:ff:ff:ff:ff:ff
>     inet 192.168.0.7/24 brd 192.168.0.255 scope global eth5
>     inet6 fe80::21e:bff:fe71:ac6e/64 scope link
>        valid_lft forever preferred_lft forever
> 8: sit0: <NOARP> mtu 1480 qdisc noop
>     link/sit 0.0.0.0 brd 0.0.0.0
>
> Thanks
> Anoop

Hi,

Let's see if we can rule out security or network issues here. Are you
able to allow all traffic for eth5 on all nodes? In addition, you may
want to add a static multicast route to ensure the multicast traffic
is going through eth5 on all nodes.

Regards,
Bernard

-----------------------------------------------------------------------

He's essentially running a wide open firewall with those rules.  Default ACCEPT on all chains, and no DROPs or REJECTs.  It's not a firewall issue.  It may be multicast not going out the right interface, however.



From volker at ixolution.de  Sun Jul 18 09:28:53 2010
From: volker at ixolution.de (Volker Dormeyer)
Date: Sun, 18 Jul 2010 11:28:53 +0200
Subject: [Linux-cluster] How do i stop VM's on a failed node?
In-Reply-To: <4C409535.4030603@lafayette.edu>
References: <4C409535.4030603@lafayette.edu>
Message-ID: <20100718092853.GA4159@dijkstra>

Hi,

On Fri, Jul 16, 2010 at 01:21:57PM -0400,
Nathan Lager <lagern at lafayette.edu> wrote:
> I have a 4-node cluster, running KVM, and a host of VM's.
> 
> One of the nodes failed unexpectedly.  I'm having two issues.  First,
> the VM's which were on that node are still reported as up and running on
> that node in clustat, and second, i'm unable to interact with the
> cluster using clusvcadm.  Every command hangs.  I've tried disabling
> these VM's, and the commands hang.
> 
> How can i clear out this dead node, without having to forceably restart
> my entire cluster?

To me, it sounds like fencing is not configured, properly.

What has been logged on the remaining nodes? I would assume, they tried to
fence the failed node - but are not able to.

Log-Snippets and Config would be helpful.

Regards,
Volker



From anoop_rajkumar at merck.com  Mon Jul 19 15:26:52 2010
From: anoop_rajkumar at merck.com (Rajkumar, Anoop)
Date: Mon, 19 Jul 2010 11:26:52 -0400
Subject: [Linux-cluster] Linux-cluster Digest, Vol 75, Issue 19
In-Reply-To: <mailman.9.1279468803.7579.linux-cluster@redhat.com>
References: <mailman.9.1279468803.7579.linux-cluster@redhat.com>
Message-ID: <C651C3AA2A6A1D4980D35451DDE3F96B3097F3@usctmx1160.merck.com>

 
Hi

Cluster is using private subnet address as node name and cman shows
following information on thir node.

[root at usrylxap235 ~]# cman_tool status
Version: 6.1.0
Config Version: 44
Cluster Name: cluster1
Cluster Id: 26777
Cluster Member: Yes
Cluster Generation: 88
Membership state: Cluster-Member
Nodes: 1
Expected votes: 3
Total votes: 1
Quorum: 2 Activity blocked
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: usrylxap235-1.merck.com
Node ID: 3
Multicast addresses: 239.192.104.2
Node addresses: 192.168.0.7

Two active nodes have following info.

[root at usrylxap237 net]# cman_tool status
Version: 6.1.0
Config Version: 44
Cluster Name: cluster1
Cluster Id: 26777
Cluster Member: Yes
Cluster Generation: 296
Membership state: Cluster-Member
Nodes: 2
Expected votes: 3
Total votes: 2
Quorum: 2
Active subsystems: 8
Flags: Dirty
Ports Bound: 0 177
Node name: usrylxap237-1.merck.com
Node ID: 1
Multicast addresses: 239.192.104.2
Node addresses: 192.168.0.2

All the nodes have private network connected to foundry switch. How I
make sure that multicast traffic is going through private network which
is eth5 for third node, eth0 for first and second node?

Thanks
Anoop 
-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of
linux-cluster-request at redhat.com
Sent: Sunday, July 18, 2010 12:00 PM
To: linux-cluster at redhat.com
Subject: Linux-cluster Digest, Vol 75, Issue 19

Send Linux-cluster mailing list submissions to
	linux-cluster at redhat.com

To subscribe or unsubscribe via the World Wide Web, visit
	https://www.redhat.com/mailman/listinfo/linux-cluster
or, via email, send a message with subject or body 'help' to
	linux-cluster-request at redhat.com

You can reach the person managing the list at
	linux-cluster-owner at redhat.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Linux-cluster digest..."


Today's Topics:

   1. Re: Adding third node to existing cluster (Marti, Robert)
   2. Re: How do i stop VM's on a failed node? (Volker Dormeyer)


----------------------------------------------------------------------

Message: 1
Date: Sat, 17 Jul 2010 22:10:52 -0500
From: "Marti, Robert" <RJM002 at shsu.edu>
To: linux clustering <linux-cluster at redhat.com>
Subject: Re: [Linux-cluster] Adding third node to existing cluster
Message-ID:
	<8FAC1E47484E43469AA28DBF35C955E4BDE0B47EAC at EXMBX.SHSU.EDU>
Content-Type: text/plain; charset="us-ascii"


> On Fri, Jul 16, 2010 at 2:06 AM, Rajkumar, Anoop
<anoop_rajkumar at merck.com> wrote:
> Hi
>
> Here are the firewall settings
>
> #more /etc/sysconfig/iptables
> # Generated by iptables-save v1.3.5 on Wed Jul 14 10:13:12 2010
> *filter
> :INPUT ACCEPT [2:186]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [4:486]
> -A INPUT -i eth5 -p udp -m udp --dport 5405 -j ACCEPT
> -A INPUT -i eth5 -p udp -m udp --sport 5405 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
> -A INPUT -i eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
> -A INPUT -i eth5 -p udp -m udp --dport 50007 -j ACCEPT
> -A INPUT -i eth5 -p udp -m udp --sport 50007 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
> -A INPUT -i eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --sport 5405 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --dport 5405 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
> -A OUTPUT -o eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --sport 50007 -j ACCEPT
> -A OUTPUT -o eth5 -p udp -m udp --dport 50007 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
> COMMIT
>
> Here are the IP in the system,I am using eth5 (Which is in private
network
> with other two nodes, connected to switch)
>
> [root at usrylxap235 ~]# ip addr list
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
>     inet6 ::1/128 scope host
>        valid_lft forever preferred_lft forever
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen
> 1000
>     link/ether 00:1c:c4:f0:bd:d8 brd ff:ff:ff:ff:ff:ff
>     inet 54.3.254.235/24 brd 54.3.254.255 scope global eth0
>     inet6 fe80::21c:c4ff:fef0:bdd8/64 scope link
>        valid_lft forever preferred_lft forever
> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>     link/ether 00:1c:c4:f0:bd:da brd ff:ff:ff:ff:ff:ff
> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>     link/ether 00:1c:c4:5e:f8:d8 brd ff:ff:ff:ff:ff:ff
> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>     link/ether 00:1c:c4:5e:f8:da brd ff:ff:ff:ff:ff:ff
> 6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>     link/ether 00:1e:0b:71:ac:6c brd ff:ff:ff:ff:ff:ff
> 7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen
> 1000
>     link/ether 00:1e:0b:71:ac:6e brd ff:ff:ff:ff:ff:ff
>     inet 192.168.0.7/24 brd 192.168.0.255 scope global eth5
>     inet6 fe80::21e:bff:fe71:ac6e/64 scope link
>        valid_lft forever preferred_lft forever
> 8: sit0: <NOARP> mtu 1480 qdisc noop
>     link/sit 0.0.0.0 brd 0.0.0.0
>
> Thanks
> Anoop

Hi,

Let's see if we can rule out security or network issues here. Are you
able to allow all traffic for eth5 on all nodes? In addition, you may
want to add a static multicast route to ensure the multicast traffic
is going through eth5 on all nodes.

Regards,
Bernard

-----------------------------------------------------------------------

He's essentially running a wide open firewall with those rules.  Default
ACCEPT on all chains, and no DROPs or REJECTs.  It's not a firewall
issue.  It may be multicast not going out the right interface, however.



------------------------------

Message: 2
Date: Sun, 18 Jul 2010 11:28:53 +0200
From: Volker Dormeyer <volker at ixolution.de>
To: linux clustering <linux-cluster at redhat.com>
Subject: Re: [Linux-cluster] How do i stop VM's on a failed node?
Message-ID: <20100718092853.GA4159 at dijkstra>
Content-Type: text/plain; charset=us-ascii

Hi,

On Fri, Jul 16, 2010 at 01:21:57PM -0400,
Nathan Lager <lagern at lafayette.edu> wrote:
> I have a 4-node cluster, running KVM, and a host of VM's.
> 
> One of the nodes failed unexpectedly.  I'm having two issues.  First,
> the VM's which were on that node are still reported as up and running
on
> that node in clustat, and second, i'm unable to interact with the
> cluster using clusvcadm.  Every command hangs.  I've tried disabling
> these VM's, and the commands hang.
> 
> How can i clear out this dead node, without having to forceably
restart
> my entire cluster?

To me, it sounds like fencing is not configured, properly.

What has been logged on the remaining nodes? I would assume, they tried
to
fence the failed node - but are not able to.

Log-Snippets and Config would be helpful.

Regards,
Volker



------------------------------

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

End of Linux-cluster Digest, Vol 75, Issue 19
*********************************************
Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.




From dfrincu at streamwide.ro  Tue Jul 20 08:57:13 2010
From: dfrincu at streamwide.ro (Dan Frincu)
Date: Tue, 20 Jul 2010 11:57:13 +0300
Subject: [Linux-cluster] Linux-cluster Digest, Vol 75, Issue 19
In-Reply-To: <C651C3AA2A6A1D4980D35451DDE3F96B3097F3@usctmx1160.merck.com>
References: <mailman.9.1279468803.7579.linux-cluster@redhat.com>
	<C651C3AA2A6A1D4980D35451DDE3F96B3097F3@usctmx1160.merck.com>
Message-ID: <4C4564E9.1010404@streamwide.ro>


Rajkumar, Anoop wrote:
>  
> Hi
>
> Cluster is using private subnet address as node name and cman shows
> following information on thir node.
>
> [root at usrylxap235 ~]# cman_tool status
> Version: 6.1.0
> Config Version: 44
> Cluster Name: cluster1
> Cluster Id: 26777
> Cluster Member: Yes
> Cluster Generation: 88
> Membership state: Cluster-Member
> Nodes: 1
> Expected votes: 3
> Total votes: 1
> Quorum: 2 Activity blocked
> Active subsystems: 5
> Flags:
> Ports Bound: 0
> Node name: usrylxap235-1.merck.com
> Node ID: 3
> Multicast addresses: 239.192.104.2
> Node addresses: 192.168.0.7
>
> Two active nodes have following info.
>
> [root at usrylxap237 net]# cman_tool status
> Version: 6.1.0
> Config Version: 44
> Cluster Name: cluster1
> Cluster Id: 26777
> Cluster Member: Yes
> Cluster Generation: 296
> Membership state: Cluster-Member
> Nodes: 2
> Expected votes: 3
> Total votes: 2
> Quorum: 2
> Active subsystems: 8
> Flags: Dirty
> Ports Bound: 0 177
> Node name: usrylxap237-1.merck.com
> Node ID: 1
> Multicast addresses: 239.192.104.2
> Node addresses: 192.168.0.2
>
> All the nodes have private network connected to foundry switch. How I
> make sure that multicast traffic is going through private network which
> is eth5 for third node, eth0 for first and second node?
>   
netstat -gn shows multicast groups and the interfaces they're linked to.
Ping the multicast address from one node and run "tcpdump -ni ethx icmp" 
on the other node to see if multicast traffic is reaching the node. Then 
repeat the process in the other direction.

Must have icmp traffic allowed between nodes first. If you don't want to 
test icmp and only want to test TCP/UDP, use netcat.

Regards,
Dan.
> Thanks
> Anoop 
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of
> linux-cluster-request at redhat.com
> Sent: Sunday, July 18, 2010 12:00 PM
> To: linux-cluster at redhat.com
> Subject: Linux-cluster Digest, Vol 75, Issue 19
>
> Send Linux-cluster mailing list submissions to
> 	linux-cluster at redhat.com
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://www.redhat.com/mailman/listinfo/linux-cluster
> or, via email, send a message with subject or body 'help' to
> 	linux-cluster-request at redhat.com
>
> You can reach the person managing the list at
> 	linux-cluster-owner at redhat.com
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Linux-cluster digest..."
>
>
> Today's Topics:
>
>    1. Re: Adding third node to existing cluster (Marti, Robert)
>    2. Re: How do i stop VM's on a failed node? (Volker Dormeyer)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 17 Jul 2010 22:10:52 -0500
> From: "Marti, Robert" <RJM002 at shsu.edu>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: Re: [Linux-cluster] Adding third node to existing cluster
> Message-ID:
> 	<8FAC1E47484E43469AA28DBF35C955E4BDE0B47EAC at EXMBX.SHSU.EDU>
> Content-Type: text/plain; charset="us-ascii"
>
>
>   
>> On Fri, Jul 16, 2010 at 2:06 AM, Rajkumar, Anoop
>>     
> <anoop_rajkumar at merck.com> wrote:
>   
>> Hi
>>
>> Here are the firewall settings
>>
>> #more /etc/sysconfig/iptables
>> # Generated by iptables-save v1.3.5 on Wed Jul 14 10:13:12 2010
>> *filter
>> :INPUT ACCEPT [2:186]
>> :FORWARD ACCEPT [0:0]
>> :OUTPUT ACCEPT [4:486]
>> -A INPUT -i eth5 -p udp -m udp --dport 5405 -j ACCEPT
>> -A INPUT -i eth5 -p udp -m udp --sport 5405 -j ACCEPT
>> -A INPUT -i eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
>> -A INPUT -i eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
>> -A INPUT -i eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
>> -A INPUT -i eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
>> -A INPUT -i eth5 -p udp -m udp --dport 50007 -j ACCEPT
>> -A INPUT -i eth5 -p udp -m udp --sport 50007 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
>> -A OUTPUT -o eth5 -p udp -m udp --sport 5405 -j ACCEPT
>> -A OUTPUT -o eth5 -p udp -m udp --dport 5405 -j ACCEPT
>> -A OUTPUT -o eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
>> -A OUTPUT -o eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
>> -A OUTPUT -o eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
>> -A OUTPUT -o eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
>> -A OUTPUT -o eth5 -p udp -m udp --sport 50007 -j ACCEPT
>> -A OUTPUT -o eth5 -p udp -m udp --dport 50007 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
>> COMMIT
>>
>> Here are the IP in the system,I am using eth5 (Which is in private
>>     
> network
>   
>> with other two nodes, connected to switch)
>>
>> [root at usrylxap235 ~]# ip addr list
>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>     inet 127.0.0.1/8 scope host lo
>>     inet6 ::1/128 scope host
>>        valid_lft forever preferred_lft forever
>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>>     
> qlen
>   
>> 1000
>>     link/ether 00:1c:c4:f0:bd:d8 brd ff:ff:ff:ff:ff:ff
>>     inet 54.3.254.235/24 brd 54.3.254.255 scope global eth0
>>     inet6 fe80::21c:c4ff:fef0:bdd8/64 scope link
>>        valid_lft forever preferred_lft forever
>> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>     link/ether 00:1c:c4:f0:bd:da brd ff:ff:ff:ff:ff:ff
>> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>     link/ether 00:1c:c4:5e:f8:d8 brd ff:ff:ff:ff:ff:ff
>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>     link/ether 00:1c:c4:5e:f8:da brd ff:ff:ff:ff:ff:ff
>> 6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>     link/ether 00:1e:0b:71:ac:6c brd ff:ff:ff:ff:ff:ff
>> 7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>>     
> qlen
>   
>> 1000
>>     link/ether 00:1e:0b:71:ac:6e brd ff:ff:ff:ff:ff:ff
>>     inet 192.168.0.7/24 brd 192.168.0.255 scope global eth5
>>     inet6 fe80::21e:bff:fe71:ac6e/64 scope link
>>        valid_lft forever preferred_lft forever
>> 8: sit0: <NOARP> mtu 1480 qdisc noop
>>     link/sit 0.0.0.0 brd 0.0.0.0
>>
>> Thanks
>> Anoop
>>     
>
> Hi,
>
> Let's see if we can rule out security or network issues here. Are you
> able to allow all traffic for eth5 on all nodes? In addition, you may
> want to add a static multicast route to ensure the multicast traffic
> is going through eth5 on all nodes.
>
> Regards,
> Bernard
>
> -----------------------------------------------------------------------
>
> He's essentially running a wide open firewall with those rules.  Default
> ACCEPT on all chains, and no DROPs or REJECTs.  It's not a firewall
> issue.  It may be multicast not going out the right interface, however.
>
>
>
> ------------------------------
>
> Message: 2
> Date: Sun, 18 Jul 2010 11:28:53 +0200
> From: Volker Dormeyer <volker at ixolution.de>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: Re: [Linux-cluster] How do i stop VM's on a failed node?
> Message-ID: <20100718092853.GA4159 at dijkstra>
> Content-Type: text/plain; charset=us-ascii
>
> Hi,
>
> On Fri, Jul 16, 2010 at 01:21:57PM -0400,
> Nathan Lager <lagern at lafayette.edu> wrote:
>   
>> I have a 4-node cluster, running KVM, and a host of VM's.
>>
>> One of the nodes failed unexpectedly.  I'm having two issues.  First,
>> the VM's which were on that node are still reported as up and running
>>     
> on
>   
>> that node in clustat, and second, i'm unable to interact with the
>> cluster using clusvcadm.  Every command hangs.  I've tried disabling
>> these VM's, and the commands hang.
>>
>> How can i clear out this dead node, without having to forceably
>>     
> restart
>   
>> my entire cluster?
>>     
>
> To me, it sounds like fencing is not configured, properly.
>
> What has been logged on the remaining nodes? I would assume, they tried
> to
> fence the failed node - but are not able to.
>
> Log-Snippets and Config would be helpful.
>
> Regards,
> Volker
>
>
>
> ------------------------------
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> End of Linux-cluster Digest, Vol 75, Issue 19
> *********************************************
> Notice:  This e-mail message, together with any attachments, contains
> information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
> New Jersey, USA 08889), and/or its affiliates Direct contact information
> for affiliates is available at 
> http://www.merck.com/contact/contacts.html) that may be confidential,
> proprietary copyrighted and/or legally privileged. It is intended solely
> for the use of the individual or entity named on this message. If you are
> not the intended recipient, and have received this message in error,
> please notify us immediately by reply e-mail and then delete it from 
> your system.
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>   

-- 
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100720/c1659fc0/attachment.htm>

From anoop_rajkumar at merck.com  Tue Jul 20 21:57:32 2010
From: anoop_rajkumar at merck.com (Rajkumar, Anoop)
Date: Tue, 20 Jul 2010 17:57:32 -0400
Subject: [Linux-cluster] Linux-cluster Digest, Vol 75, Issue 21
In-Reply-To: <mailman.39.1279641607.9065.linux-cluster@redhat.com>
References: <mailman.39.1279641607.9065.linux-cluster@redhat.com>
Message-ID: <C651C3AA2A6A1D4980D35451DDE3F96B3097F4@usctmx1160.merck.com>


 You are right. I was not having multicast group in third node when I
did netstat -gn, have added below config in cluster.conf and restarted
cluster and now I have multicast group on all nodes. Problem is packets
are being received at public network of third node unlike other two
nodes where they are being received at private network IP. Any inputs
how to change that?

Added to cluster.conf

<multicast addr="239.192.104.2"/>
                <cman/>

>From third node.

[root at usrylxap235 ~]# ip addr list|grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
    inet 192.168.0.7/24 brd 192.168.0.255 scope global eth0
[root at usrylxap235 ~]# ip addr list|grep eth5
7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
    inet 54.3.254.235/24 brd 54.3.254.255 scope global eth5

[root at usrylxap235 ~]# tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

0 packets captured
1 packets received by filter
0 packets dropped by kernel

[root at usrylxap235 ~]# tcpdump -i eth5 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on eth5, link-type EN10MB (Ethernet), capture size 96 bytes
17:48:19.292089 IP usrylxap237.merck.com > 239.192.104.2: ICMP echo
request, id 26924, seq 23, length 64
22 packets captured
22 packets received by filter
0 packets dropped by kernel

Appreciate your help.

Anoop

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of
linux-cluster-request at redhat.com
Sent: Tuesday, July 20, 2010 12:00 PM
To: linux-cluster at redhat.com
Subject: Linux-cluster Digest, Vol 75, Issue 21

Send Linux-cluster mailing list submissions to
	linux-cluster at redhat.com

To subscribe or unsubscribe via the World Wide Web, visit
	https://www.redhat.com/mailman/listinfo/linux-cluster
or, via email, send a message with subject or body 'help' to
	linux-cluster-request at redhat.com

You can reach the person managing the list at
	linux-cluster-owner at redhat.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Linux-cluster digest..."


Today's Topics:

   1. Re: Linux-cluster Digest, Vol 75, Issue 19 (Dan Frincu)


----------------------------------------------------------------------

Message: 1
Date: Tue, 20 Jul 2010 11:57:13 +0300
From: Dan Frincu <dfrincu at streamwide.ro>
To: linux clustering <linux-cluster at redhat.com>
Subject: Re: [Linux-cluster] Linux-cluster Digest, Vol 75, Issue 19
Message-ID: <4C4564E9.1010404 at streamwide.ro>
Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"


Rajkumar, Anoop wrote:
>  
> Hi
>
> Cluster is using private subnet address as node name and cman shows
> following information on thir node.
>
> [root at usrylxap235 ~]# cman_tool status
> Version: 6.1.0
> Config Version: 44
> Cluster Name: cluster1
> Cluster Id: 26777
> Cluster Member: Yes
> Cluster Generation: 88
> Membership state: Cluster-Member
> Nodes: 1
> Expected votes: 3
> Total votes: 1
> Quorum: 2 Activity blocked
> Active subsystems: 5
> Flags:
> Ports Bound: 0
> Node name: usrylxap235-1.merck.com
> Node ID: 3
> Multicast addresses: 239.192.104.2
> Node addresses: 192.168.0.7
>
> Two active nodes have following info.
>
> [root at usrylxap237 net]# cman_tool status
> Version: 6.1.0
> Config Version: 44
> Cluster Name: cluster1
> Cluster Id: 26777
> Cluster Member: Yes
> Cluster Generation: 296
> Membership state: Cluster-Member
> Nodes: 2
> Expected votes: 3
> Total votes: 2
> Quorum: 2
> Active subsystems: 8
> Flags: Dirty
> Ports Bound: 0 177
> Node name: usrylxap237-1.merck.com
> Node ID: 1
> Multicast addresses: 239.192.104.2
> Node addresses: 192.168.0.2
>
> All the nodes have private network connected to foundry switch. How I
> make sure that multicast traffic is going through private network
which
> is eth5 for third node, eth0 for first and second node?
>   
netstat -gn shows multicast groups and the interfaces they're linked to.
Ping the multicast address from one node and run "tcpdump -ni ethx icmp"

on the other node to see if multicast traffic is reaching the node. Then

repeat the process in the other direction.

Must have icmp traffic allowed between nodes first. If you don't want to

test icmp and only want to test TCP/UDP, use netcat.

Regards,
Dan.
> Thanks
> Anoop 
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of
> linux-cluster-request at redhat.com
> Sent: Sunday, July 18, 2010 12:00 PM
> To: linux-cluster at redhat.com
> Subject: Linux-cluster Digest, Vol 75, Issue 19
>
> Send Linux-cluster mailing list submissions to
> 	linux-cluster at redhat.com
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://www.redhat.com/mailman/listinfo/linux-cluster
> or, via email, send a message with subject or body 'help' to
> 	linux-cluster-request at redhat.com
>
> You can reach the person managing the list at
> 	linux-cluster-owner at redhat.com
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Linux-cluster digest..."
>
>
> Today's Topics:
>
>    1. Re: Adding third node to existing cluster (Marti, Robert)
>    2. Re: How do i stop VM's on a failed node? (Volker Dormeyer)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 17 Jul 2010 22:10:52 -0500
> From: "Marti, Robert" <RJM002 at shsu.edu>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: Re: [Linux-cluster] Adding third node to existing cluster
> Message-ID:
> 	<8FAC1E47484E43469AA28DBF35C955E4BDE0B47EAC at EXMBX.SHSU.EDU>
> Content-Type: text/plain; charset="us-ascii"
>
>
>   
>> On Fri, Jul 16, 2010 at 2:06 AM, Rajkumar, Anoop
>>     
> <anoop_rajkumar at merck.com> wrote:
>   
>> Hi
>>
>> Here are the firewall settings
>>
>> #more /etc/sysconfig/iptables
>> # Generated by iptables-save v1.3.5 on Wed Jul 14 10:13:12 2010
>> *filter
>> :INPUT ACCEPT [2:186]
>> :FORWARD ACCEPT [0:0]
>> :OUTPUT ACCEPT [4:486]
>> -A INPUT -i eth5 -p udp -m udp --dport 5405 -j ACCEPT
>> -A INPUT -i eth5 -p udp -m udp --sport 5405 -j ACCEPT
>> -A INPUT -i eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
>> -A INPUT -i eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
>> -A INPUT -i eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
>> -A INPUT -i eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
>> -A INPUT -i eth5 -p udp -m udp --dport 50007 -j ACCEPT
>> -A INPUT -i eth5 -p udp -m udp --sport 50007 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
>> -A INPUT -i eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
>> -A OUTPUT -o eth5 -p udp -m udp --sport 5405 -j ACCEPT
>> -A OUTPUT -o eth5 -p udp -m udp --dport 5405 -j ACCEPT
>> -A OUTPUT -o eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
>> -A OUTPUT -o eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
>> -A OUTPUT -o eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
>> -A OUTPUT -o eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
>> -A OUTPUT -o eth5 -p udp -m udp --sport 50007 -j ACCEPT
>> -A OUTPUT -o eth5 -p udp -m udp --dport 50007 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
>> COMMIT
>>
>> Here are the IP in the system,I am using eth5 (Which is in private
>>     
> network
>   
>> with other two nodes, connected to switch)
>>
>> [root at usrylxap235 ~]# ip addr list
>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>     inet 127.0.0.1/8 scope host lo
>>     inet6 ::1/128 scope host
>>        valid_lft forever preferred_lft forever
>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>>     
> qlen
>   
>> 1000
>>     link/ether 00:1c:c4:f0:bd:d8 brd ff:ff:ff:ff:ff:ff
>>     inet 54.3.254.235/24 brd 54.3.254.255 scope global eth0
>>     inet6 fe80::21c:c4ff:fef0:bdd8/64 scope link
>>        valid_lft forever preferred_lft forever
>> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>     link/ether 00:1c:c4:f0:bd:da brd ff:ff:ff:ff:ff:ff
>> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>     link/ether 00:1c:c4:5e:f8:d8 brd ff:ff:ff:ff:ff:ff
>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>     link/ether 00:1c:c4:5e:f8:da brd ff:ff:ff:ff:ff:ff
>> 6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>     link/ether 00:1e:0b:71:ac:6c brd ff:ff:ff:ff:ff:ff
>> 7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>>     
> qlen
>   
>> 1000
>>     link/ether 00:1e:0b:71:ac:6e brd ff:ff:ff:ff:ff:ff
>>     inet 192.168.0.7/24 brd 192.168.0.255 scope global eth5
>>     inet6 fe80::21e:bff:fe71:ac6e/64 scope link
>>        valid_lft forever preferred_lft forever
>> 8: sit0: <NOARP> mtu 1480 qdisc noop
>>     link/sit 0.0.0.0 brd 0.0.0.0
>>
>> Thanks
>> Anoop
>>     
>
> Hi,
>
> Let's see if we can rule out security or network issues here. Are you
> able to allow all traffic for eth5 on all nodes? In addition, you may
> want to add a static multicast route to ensure the multicast traffic
> is going through eth5 on all nodes.
>
> Regards,
> Bernard
>
>
-----------------------------------------------------------------------
>
> He's essentially running a wide open firewall with those rules.
Default
> ACCEPT on all chains, and no DROPs or REJECTs.  It's not a firewall
> issue.  It may be multicast not going out the right interface,
however.
>
>
>
> ------------------------------
>
> Message: 2
> Date: Sun, 18 Jul 2010 11:28:53 +0200
> From: Volker Dormeyer <volker at ixolution.de>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: Re: [Linux-cluster] How do i stop VM's on a failed node?
> Message-ID: <20100718092853.GA4159 at dijkstra>
> Content-Type: text/plain; charset=us-ascii
>
> Hi,
>
> On Fri, Jul 16, 2010 at 01:21:57PM -0400,
> Nathan Lager <lagern at lafayette.edu> wrote:
>   
>> I have a 4-node cluster, running KVM, and a host of VM's.
>>
>> One of the nodes failed unexpectedly.  I'm having two issues.  First,
>> the VM's which were on that node are still reported as up and running
>>     
> on
>   
>> that node in clustat, and second, i'm unable to interact with the
>> cluster using clusvcadm.  Every command hangs.  I've tried disabling
>> these VM's, and the commands hang.
>>
>> How can i clear out this dead node, without having to forceably
>>     
> restart
>   
>> my entire cluster?
>>     
>
> To me, it sounds like fencing is not configured, properly.
>
> What has been logged on the remaining nodes? I would assume, they
tried
> to
> fence the failed node - but are not able to.
>
> Log-Snippets and Config would be helpful.
>
> Regards,
> Volker
>
>
>
> ------------------------------
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> End of Linux-cluster Digest, Vol 75, Issue 19
> *********************************************
> Notice:  This e-mail message, together with any attachments, contains
> information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
> New Jersey, USA 08889), and/or its affiliates Direct contact
information
> for affiliates is available at 
> http://www.merck.com/contact/contacts.html) that may be confidential,
> proprietary copyrighted and/or legally privileged. It is intended
solely
> for the use of the individual or entity named on this message. If you
are
> not the intended recipient, and have received this message in error,
> please notify us immediately by reply e-mail and then delete it from 
> your system.
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>   

-- 
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://www.redhat.com/archives/linux-cluster/attachments/20100720/c165
9fc0/attachment.html>

------------------------------

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

End of Linux-cluster Digest, Vol 75, Issue 21
*********************************************
Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.




From dfrincu at streamwide.ro  Wed Jul 21 07:13:14 2010
From: dfrincu at streamwide.ro (Dan Frincu)
Date: Wed, 21 Jul 2010 10:13:14 +0300
Subject: [Linux-cluster] Linux-cluster Digest, Vol 75, Issue 21
In-Reply-To: <C651C3AA2A6A1D4980D35451DDE3F96B3097F4@usctmx1160.merck.com>
References: <mailman.39.1279641607.9065.linux-cluster@redhat.com>
	<C651C3AA2A6A1D4980D35451DDE3F96B3097F4@usctmx1160.merck.com>
Message-ID: <4C469E0A.20806@streamwide.ro>

 From the man page.
/
Multicast network configuration/

Cman can be configured to use multicast instead of broadcast (broadcast 
is used by default if no multicast parameters are given.) To configure 
multicast add one line under the <cman> section *and another under the 
<clusternode> section*:

<cman> <multicast addr="224.0.0.1"/> </cman>

<clusternode name="nd1"> <multicast addr="224.0.0.1" 
*interface="eth0"*/> </clusternode>

The multicast addresses must match and the address must be usable on the 
interface name given for the node.

When running netstat -gn you have something like this:

IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      224.0.0.1
eth0            1      224.0.0.1
br0             1      224.0.0.1
tap0            1      224.0.0.1

On the third node check to see a matching pair between eth0 and 
239.192.104.2 in the output of netstat -gn. If you don't see it or it 
matches another interface, such as eth5, change according to the man 
page reference above.

Regards,
Dan.

Rajkumar, Anoop wrote:
>  You are right. I was not having multicast group in third node when I
> did netstat -gn, have added below config in cluster.conf and restarted
> cluster and now I have multicast group on all nodes. Problem is packets
> are being received at public network of third node unlike other two
> nodes where they are being received at private network IP. Any inputs
> how to change that?
>
> Added to cluster.conf
>
> <multicast addr="239.192.104.2"/>
>                 <cman/>
>
> From third node.
>
> [root at usrylxap235 ~]# ip addr list|grep eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> qlen 1000
>     inet 192.168.0.7/24 brd 192.168.0.255 scope global eth0
> [root at usrylxap235 ~]# ip addr list|grep eth5
> 7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> qlen 1000
>     inet 54.3.254.235/24 brd 54.3.254.255 scope global eth5
>
> [root at usrylxap235 ~]# tcpdump -i eth0 icmp
> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
>
> 0 packets captured
> 1 packets received by filter
> 0 packets dropped by kernel
>
> [root at usrylxap235 ~]# tcpdump -i eth5 icmp
> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> decode
> listening on eth5, link-type EN10MB (Ethernet), capture size 96 bytes
> 17:48:19.292089 IP usrylxap237.merck.com > 239.192.104.2: ICMP echo
> request, id 26924, seq 23, length 64
> 22 packets captured
> 22 packets received by filter
> 0 packets dropped by kernel
>
> Appreciate your help.
>
> Anoop
>
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of
> linux-cluster-request at redhat.com
> Sent: Tuesday, July 20, 2010 12:00 PM
> To: linux-cluster at redhat.com
> Subject: Linux-cluster Digest, Vol 75, Issue 21
>
> Send Linux-cluster mailing list submissions to
> 	linux-cluster at redhat.com
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://www.redhat.com/mailman/listinfo/linux-cluster
> or, via email, send a message with subject or body 'help' to
> 	linux-cluster-request at redhat.com
>
> You can reach the person managing the list at
> 	linux-cluster-owner at redhat.com
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Linux-cluster digest..."
>
>
> Today's Topics:
>
>    1. Re: Linux-cluster Digest, Vol 75, Issue 19 (Dan Frincu)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 20 Jul 2010 11:57:13 +0300
> From: Dan Frincu <dfrincu at streamwide.ro>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: Re: [Linux-cluster] Linux-cluster Digest, Vol 75, Issue 19
> Message-ID: <4C4564E9.1010404 at streamwide.ro>
> Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
>
>
> Rajkumar, Anoop wrote:
>   
>>  
>> Hi
>>
>> Cluster is using private subnet address as node name and cman shows
>> following information on thir node.
>>
>> [root at usrylxap235 ~]# cman_tool status
>> Version: 6.1.0
>> Config Version: 44
>> Cluster Name: cluster1
>> Cluster Id: 26777
>> Cluster Member: Yes
>> Cluster Generation: 88
>> Membership state: Cluster-Member
>> Nodes: 1
>> Expected votes: 3
>> Total votes: 1
>> Quorum: 2 Activity blocked
>> Active subsystems: 5
>> Flags:
>> Ports Bound: 0
>> Node name: usrylxap235-1.merck.com
>> Node ID: 3
>> Multicast addresses: 239.192.104.2
>> Node addresses: 192.168.0.7
>>
>> Two active nodes have following info.
>>
>> [root at usrylxap237 net]# cman_tool status
>> Version: 6.1.0
>> Config Version: 44
>> Cluster Name: cluster1
>> Cluster Id: 26777
>> Cluster Member: Yes
>> Cluster Generation: 296
>> Membership state: Cluster-Member
>> Nodes: 2
>> Expected votes: 3
>> Total votes: 2
>> Quorum: 2
>> Active subsystems: 8
>> Flags: Dirty
>> Ports Bound: 0 177
>> Node name: usrylxap237-1.merck.com
>> Node ID: 1
>> Multicast addresses: 239.192.104.2
>> Node addresses: 192.168.0.2
>>
>> All the nodes have private network connected to foundry switch. How I
>> make sure that multicast traffic is going through private network
>>     
> which
>   
>> is eth5 for third node, eth0 for first and second node?
>>   
>>     
> netstat -gn shows multicast groups and the interfaces they're linked to.
> Ping the multicast address from one node and run "tcpdump -ni ethx icmp"
>
> on the other node to see if multicast traffic is reaching the node. Then
>
> repeat the process in the other direction.
>
> Must have icmp traffic allowed between nodes first. If you don't want to
>
> test icmp and only want to test TCP/UDP, use netcat.
>
> Regards,
> Dan.
>   
>> Thanks
>> Anoop 
>> -----Original Message-----
>> From: linux-cluster-bounces at redhat.com
>> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of
>> linux-cluster-request at redhat.com
>> Sent: Sunday, July 18, 2010 12:00 PM
>> To: linux-cluster at redhat.com
>> Subject: Linux-cluster Digest, Vol 75, Issue 19
>>
>> Send Linux-cluster mailing list submissions to
>> 	linux-cluster at redhat.com
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> 	https://www.redhat.com/mailman/listinfo/linux-cluster
>> or, via email, send a message with subject or body 'help' to
>> 	linux-cluster-request at redhat.com
>>
>> You can reach the person managing the list at
>> 	linux-cluster-owner at redhat.com
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Linux-cluster digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: Adding third node to existing cluster (Marti, Robert)
>>    2. Re: How do i stop VM's on a failed node? (Volker Dormeyer)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sat, 17 Jul 2010 22:10:52 -0500
>> From: "Marti, Robert" <RJM002 at shsu.edu>
>> To: linux clustering <linux-cluster at redhat.com>
>> Subject: Re: [Linux-cluster] Adding third node to existing cluster
>> Message-ID:
>> 	<8FAC1E47484E43469AA28DBF35C955E4BDE0B47EAC at EXMBX.SHSU.EDU>
>> Content-Type: text/plain; charset="us-ascii"
>>
>>
>>   
>>     
>>> On Fri, Jul 16, 2010 at 2:06 AM, Rajkumar, Anoop
>>>     
>>>       
>> <anoop_rajkumar at merck.com> wrote:
>>   
>>     
>>> Hi
>>>
>>> Here are the firewall settings
>>>
>>> #more /etc/sysconfig/iptables
>>> # Generated by iptables-save v1.3.5 on Wed Jul 14 10:13:12 2010
>>> *filter
>>> :INPUT ACCEPT [2:186]
>>> :FORWARD ACCEPT [0:0]
>>> :OUTPUT ACCEPT [4:486]
>>> -A INPUT -i eth5 -p udp -m udp --dport 5405 -j ACCEPT
>>> -A INPUT -i eth5 -p udp -m udp --sport 5405 -j ACCEPT
>>> -A INPUT -i eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
>>> -A INPUT -i eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
>>> -A INPUT -i eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
>>> -A INPUT -i eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
>>> -A INPUT -i eth5 -p udp -m udp --dport 50007 -j ACCEPT
>>> -A INPUT -i eth5 -p udp -m udp --sport 50007 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
>>> -A INPUT -i eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
>>> -A OUTPUT -o eth5 -p udp -m udp --sport 5405 -j ACCEPT
>>> -A OUTPUT -o eth5 -p udp -m udp --dport 5405 -j ACCEPT
>>> -A OUTPUT -o eth0 -p tcp -m tcp --sport 14567 -j ACCEPT
>>> -A OUTPUT -o eth0 -p tcp -m tcp --dport 14567 -j ACCEPT
>>> -A OUTPUT -o eth0 -p tcp -m tcp --sport 16851 -j ACCEPT
>>> -A OUTPUT -o eth0 -p tcp -m tcp --dport 16851 -j ACCEPT
>>> -A OUTPUT -o eth5 -p udp -m udp --sport 50007 -j ACCEPT
>>> -A OUTPUT -o eth5 -p udp -m udp --dport 50007 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 11111 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 11111 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 21064 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 21064 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50009 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50009 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50008 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50008 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 50006 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 50006 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41969 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41969 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41968 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41968 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41967 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41967 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --sport 41966 -j ACCEPT
>>> -A OUTPUT -o eth5 -p tcp -m tcp --dport 41966 -j ACCEPT
>>> COMMIT
>>>
>>> Here are the IP in the system,I am using eth5 (Which is in private
>>>     
>>>       
>> network
>>   
>>     
>>> with other two nodes, connected to switch)
>>>
>>> [root at usrylxap235 ~]# ip addr list
>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
>>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>>     inet 127.0.0.1/8 scope host lo
>>>     inet6 ::1/128 scope host
>>>        valid_lft forever preferred_lft forever
>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>>>     
>>>       
>> qlen
>>   
>>     
>>> 1000
>>>     link/ether 00:1c:c4:f0:bd:d8 brd ff:ff:ff:ff:ff:ff
>>>     inet 54.3.254.235/24 brd 54.3.254.255 scope global eth0
>>>     inet6 fe80::21c:c4ff:fef0:bdd8/64 scope link
>>>        valid_lft forever preferred_lft forever
>>> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>>     link/ether 00:1c:c4:f0:bd:da brd ff:ff:ff:ff:ff:ff
>>> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>>     link/ether 00:1c:c4:5e:f8:d8 brd ff:ff:ff:ff:ff:ff
>>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>>     link/ether 00:1c:c4:5e:f8:da brd ff:ff:ff:ff:ff:ff
>>> 6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>>>     link/ether 00:1e:0b:71:ac:6c brd ff:ff:ff:ff:ff:ff
>>> 7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>>>     
>>>       
>> qlen
>>   
>>     
>>> 1000
>>>     link/ether 00:1e:0b:71:ac:6e brd ff:ff:ff:ff:ff:ff
>>>     inet 192.168.0.7/24 brd 192.168.0.255 scope global eth5
>>>     inet6 fe80::21e:bff:fe71:ac6e/64 scope link
>>>        valid_lft forever preferred_lft forever
>>> 8: sit0: <NOARP> mtu 1480 qdisc noop
>>>     link/sit 0.0.0.0 brd 0.0.0.0
>>>
>>> Thanks
>>> Anoop
>>>     
>>>       
>> Hi,
>>
>> Let's see if we can rule out security or network issues here. Are you
>> able to allow all traffic for eth5 on all nodes? In addition, you may
>> want to add a static multicast route to ensure the multicast traffic
>> is going through eth5 on all nodes.
>>
>> Regards,
>> Bernard
>>
>>
>>     
> -----------------------------------------------------------------------
>   
>> He's essentially running a wide open firewall with those rules.
>>     
> Default
>   
>> ACCEPT on all chains, and no DROPs or REJECTs.  It's not a firewall
>> issue.  It may be multicast not going out the right interface,
>>     
> however.
>   
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Sun, 18 Jul 2010 11:28:53 +0200
>> From: Volker Dormeyer <volker at ixolution.de>
>> To: linux clustering <linux-cluster at redhat.com>
>> Subject: Re: [Linux-cluster] How do i stop VM's on a failed node?
>> Message-ID: <20100718092853.GA4159 at dijkstra>
>> Content-Type: text/plain; charset=us-ascii
>>
>> Hi,
>>
>> On Fri, Jul 16, 2010 at 01:21:57PM -0400,
>> Nathan Lager <lagern at lafayette.edu> wrote:
>>   
>>     
>>> I have a 4-node cluster, running KVM, and a host of VM's.
>>>
>>> One of the nodes failed unexpectedly.  I'm having two issues.  First,
>>> the VM's which were on that node are still reported as up and running
>>>     
>>>       
>> on
>>   
>>     
>>> that node in clustat, and second, i'm unable to interact with the
>>> cluster using clusvcadm.  Every command hangs.  I've tried disabling
>>> these VM's, and the commands hang.
>>>
>>> How can i clear out this dead node, without having to forceably
>>>     
>>>       
>> restart
>>   
>>     
>>> my entire cluster?
>>>     
>>>       
>> To me, it sounds like fencing is not configured, properly.
>>
>> What has been logged on the remaining nodes? I would assume, they
>>     
> tried
>   
>> to
>> fence the failed node - but are not able to.
>>
>> Log-Snippets and Config would be helpful.
>>
>> Regards,
>> Volker
>>
>>
>>
>> ------------------------------
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>> End of Linux-cluster Digest, Vol 75, Issue 19
>> *********************************************
>> Notice:  This e-mail message, together with any attachments, contains
>> information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
>> New Jersey, USA 08889), and/or its affiliates Direct contact
>>     
> information
>   
>> for affiliates is available at 
>> http://www.merck.com/contact/contacts.html) that may be confidential,
>> proprietary copyrighted and/or legally privileged. It is intended
>>     
> solely
>   
>> for the use of the individual or entity named on this message. If you
>>     
> are
>   
>> not the intended recipient, and have received this message in error,
>> please notify us immediately by reply e-mail and then delete it from 
>> your system.
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>   
>>     
>
>   

-- 
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100721/beffe1c8/attachment.htm>

From anoop_rajkumar at merck.com  Wed Jul 21 20:56:56 2010
From: anoop_rajkumar at merck.com (Rajkumar, Anoop)
Date: Wed, 21 Jul 2010 16:56:56 -0400
Subject: [Linux-cluster] Linux-cluster Digest, Vol 75, Issue 21
In-Reply-To: <4C469E0A.20806@streamwide.ro>
References: <mailman.39.1279641607.9065.linux-cluster@redhat.com>
	<C651C3AA2A6A1D4980D35451DDE3F96B3097F4@usctmx1160.merck.com>
	<4C469E0A.20806@streamwide.ro>
Message-ID: <C651C3AA2A6A1D4980D35451DDE3F96B721808@usctmx1160.merck.com>

Hi Dan
 
Thanks
 
That resolved it.
 
Anoop

________________________________

From: Dan Frincu [mailto:dfrincu at streamwide.ro] 
Sent: Wednesday, July 21, 2010 3:13 AM
To: Rajkumar, Anoop
Cc: linux-cluster at redhat.com
Subject: Re: Linux-cluster Digest, Vol 75, Issue 21


>From the man page.

Multicast network configuration 

Cman can be configured to use multicast instead of broadcast (broadcast
is used by default if no multicast parameters are given.) To configure
multicast add one line under the <cman> section and another under the
<clusternode> section:

<cman> <multicast addr="224.0.0.1"/> </cman>

<clusternode name="nd1"> <multicast addr="224.0.0.1" interface="eth0"/>
</clusternode>

The multicast addresses must match and the address must be usable on the
interface name given for the node.

When running netstat -gn you have something like this:

IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      224.0.0.1
eth0            1      224.0.0.1
br0             1      224.0.0.1
tap0            1      224.0.0.1

On the third node check to see a matching pair between eth0 and
239.192.104.2 in the output of netstat -gn. If you don't see it or it
matches another interface, such as eth5, change according to the man
page reference above.

Regards,
Dan.

Rajkumar, Anoop wrote: 

	 You are right. I was not having multicast group in third node
when I
	did netstat -gn, have added below config in cluster.conf and
restarted
	cluster and now I have multicast group on all nodes. Problem is
packets
	are being received at public network of third node unlike other
two
	nodes where they are being received at private network IP. Any
inputs
	how to change that?
	
	Added to cluster.conf
	
	<multicast addr="239.192.104.2"/>
	                <cman/>
	
	From third node.
	
	[root at usrylxap235 ~]# ip addr list|grep eth0
	2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast
	qlen 1000
	    inet 192.168.0.7/24 brd 192.168.0.255 scope global eth0
	[root at usrylxap235 ~]# ip addr list|grep eth5
	7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast
	qlen 1000
	    inet 54.3.254.235/24 brd 54.3.254.255 scope global eth5
	
	[root at usrylxap235 ~]# tcpdump -i eth0 icmp
	tcpdump: verbose output suppressed, use -v or -vv for full
protocol
	decode
	listening on eth0, link-type EN10MB (Ethernet), capture size 96
bytes
	
	0 packets captured
	1 packets received by filter
	0 packets dropped by kernel
	
	[root at usrylxap235 ~]# tcpdump -i eth5 icmp
	tcpdump: verbose output suppressed, use -v or -vv for full
protocol
	decode
	listening on eth5, link-type EN10MB (Ethernet), capture size 96
bytes
	17:48:19.292089 IP usrylxap237.merck.com > 239.192.104.2: ICMP
echo
	request, id 26924, seq 23, length 64
	22 packets captured
	22 packets received by filter
	0 packets dropped by kernel
	
	Appreciate your help.
	
	Anoop
	
	-----Original Message-----
	From: linux-cluster-bounces at redhat.com
	[mailto:linux-cluster-bounces at redhat.com] On Behalf Of
	linux-cluster-request at redhat.com
	Sent: Tuesday, July 20, 2010 12:00 PM
	To: linux-cluster at redhat.com
	Subject: Linux-cluster Digest, Vol 75, Issue 21
	
	Send Linux-cluster mailing list submissions to
		linux-cluster at redhat.com
	
	To subscribe or unsubscribe via the World Wide Web, visit
		https://www.redhat.com/mailman/listinfo/linux-cluster
	or, via email, send a message with subject or body 'help' to
		linux-cluster-request at redhat.com
	
	You can reach the person managing the list at
		linux-cluster-owner at redhat.com
	
	When replying, please edit your Subject line so it is more
specific
	than "Re: Contents of Linux-cluster digest..."
	
	
	Today's Topics:
	
	   1. Re: Linux-cluster Digest, Vol 75, Issue 19 (Dan Frincu)
	
	
	
----------------------------------------------------------------------
	
	Message: 1
	Date: Tue, 20 Jul 2010 11:57:13 +0300
	From: Dan Frincu <dfrincu at streamwide.ro>
<mailto:dfrincu at streamwide.ro> 
	To: linux clustering <linux-cluster at redhat.com>
<mailto:linux-cluster at redhat.com> 
	Subject: Re: [Linux-cluster] Linux-cluster Digest, Vol 75, Issue
19
	Message-ID: <4C4564E9.1010404 at streamwide.ro>
<mailto:4C4564E9.1010404 at streamwide.ro> 
	Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
	
	
	Rajkumar, Anoop wrote:
	  

		 
		Hi
		
		Cluster is using private subnet address as node name and
cman shows
		following information on thir node.
		
		[root at usrylxap235 ~]# cman_tool status
		Version: 6.1.0
		Config Version: 44
		Cluster Name: cluster1
		Cluster Id: 26777
		Cluster Member: Yes
		Cluster Generation: 88
		Membership state: Cluster-Member
		Nodes: 1
		Expected votes: 3
		Total votes: 1
		Quorum: 2 Activity blocked
		Active subsystems: 5
		Flags:
		Ports Bound: 0
		Node name: usrylxap235-1.merck.com
		Node ID: 3
		Multicast addresses: 239.192.104.2
		Node addresses: 192.168.0.7
		
		Two active nodes have following info.
		
		[root at usrylxap237 net]# cman_tool status
		Version: 6.1.0
		Config Version: 44
		Cluster Name: cluster1
		Cluster Id: 26777
		Cluster Member: Yes
		Cluster Generation: 296
		Membership state: Cluster-Member
		Nodes: 2
		Expected votes: 3
		Total votes: 2
		Quorum: 2
		Active subsystems: 8
		Flags: Dirty
		Ports Bound: 0 177
		Node name: usrylxap237-1.merck.com
		Node ID: 1
		Multicast addresses: 239.192.104.2
		Node addresses: 192.168.0.2
		
		All the nodes have private network connected to foundry
switch. How I
		make sure that multicast traffic is going through
private network
		    

	which
	  

		is eth5 for third node, eth0 for first and second node?
		  
		    

	netstat -gn shows multicast groups and the interfaces they're
linked to.
	Ping the multicast address from one node and run "tcpdump -ni
ethx icmp"
	
	on the other node to see if multicast traffic is reaching the
node. Then
	
	repeat the process in the other direction.
	
	Must have icmp traffic allowed between nodes first. If you don't
want to
	
	test icmp and only want to test TCP/UDP, use netcat.
	
	Regards,
	Dan.
	  

		Thanks
		Anoop 
		-----Original Message-----
		From: linux-cluster-bounces at redhat.com
		[mailto:linux-cluster-bounces at redhat.com] On Behalf Of
		linux-cluster-request at redhat.com
		Sent: Sunday, July 18, 2010 12:00 PM
		To: linux-cluster at redhat.com
		Subject: Linux-cluster Digest, Vol 75, Issue 19
		
		Send Linux-cluster mailing list submissions to
			linux-cluster at redhat.com
		
		To subscribe or unsubscribe via the World Wide Web,
visit
	
https://www.redhat.com/mailman/listinfo/linux-cluster
		or, via email, send a message with subject or body
'help' to
			linux-cluster-request at redhat.com
		
		You can reach the person managing the list at
			linux-cluster-owner at redhat.com
		
		When replying, please edit your Subject line so it is
more specific
		than "Re: Contents of Linux-cluster digest..."
		
		
		Today's Topics:
		
		   1. Re: Adding third node to existing cluster (Marti,
Robert)
		   2. Re: How do i stop VM's on a failed node? (Volker
Dormeyer)
		
		
	
----------------------------------------------------------------------
		
		Message: 1
		Date: Sat, 17 Jul 2010 22:10:52 -0500
		From: "Marti, Robert" <RJM002 at shsu.edu>
<mailto:RJM002 at shsu.edu> 
		To: linux clustering <linux-cluster at redhat.com>
<mailto:linux-cluster at redhat.com> 
		Subject: Re: [Linux-cluster] Adding third node to
existing cluster
		Message-ID:
	
<8FAC1E47484E43469AA28DBF35C955E4BDE0B47EAC at EXMBX.SHSU.EDU>
<mailto:8FAC1E47484E43469AA28DBF35C955E4BDE0B47EAC at EXMBX.SHSU.EDU> 
		Content-Type: text/plain; charset="us-ascii"
		
		
		  
		    

			On Fri, Jul 16, 2010 at 2:06 AM, Rajkumar, Anoop
			    
			      

		<anoop_rajkumar at merck.com>
<mailto:anoop_rajkumar at merck.com>  wrote:
		  
		    

			Hi
			
			Here are the firewall settings
			
			#more /etc/sysconfig/iptables
			# Generated by iptables-save v1.3.5 on Wed Jul
14 10:13:12 2010
			*filter
			:INPUT ACCEPT [2:186]
			:FORWARD ACCEPT [0:0]
			:OUTPUT ACCEPT [4:486]
			-A INPUT -i eth5 -p udp -m udp --dport 5405 -j
ACCEPT
			-A INPUT -i eth5 -p udp -m udp --sport 5405 -j
ACCEPT
			-A INPUT -i eth0 -p tcp -m tcp --dport 14567 -j
ACCEPT
			-A INPUT -i eth0 -p tcp -m tcp --sport 14567 -j
ACCEPT
			-A INPUT -i eth0 -p tcp -m tcp --dport 16851 -j
ACCEPT
			-A INPUT -i eth0 -p tcp -m tcp --sport 16851 -j
ACCEPT
			-A INPUT -i eth5 -p udp -m udp --dport 50007 -j
ACCEPT
			-A INPUT -i eth5 -p udp -m udp --sport 50007 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --dport 11111 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --sport 11111 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --dport 21064 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --sport 21064 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --dport 50009 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --sport 50009 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --dport 50008 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --sport 50008 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --dport 50006 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --sport 50006 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --dport 41969 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --sport 41969 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --dport 41968 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --sport 41968 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --dport 41967 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --sport 41967 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --dport 41966 -j
ACCEPT
			-A INPUT -i eth5 -p tcp -m tcp --sport 41966 -j
ACCEPT
			-A OUTPUT -o eth5 -p udp -m udp --sport 5405 -j
ACCEPT
			-A OUTPUT -o eth5 -p udp -m udp --dport 5405 -j
ACCEPT
			-A OUTPUT -o eth0 -p tcp -m tcp --sport 14567 -j
ACCEPT
			-A OUTPUT -o eth0 -p tcp -m tcp --dport 14567 -j
ACCEPT
			-A OUTPUT -o eth0 -p tcp -m tcp --sport 16851 -j
ACCEPT
			-A OUTPUT -o eth0 -p tcp -m tcp --dport 16851 -j
ACCEPT
			-A OUTPUT -o eth5 -p udp -m udp --sport 50007 -j
ACCEPT
			-A OUTPUT -o eth5 -p udp -m udp --dport 50007 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --sport 11111 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --dport 11111 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --sport 21064 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --dport 21064 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --sport 50009 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --dport 50009 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --sport 50008 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --dport 50008 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --sport 50006 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --dport 50006 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --sport 41969 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --dport 41969 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --sport 41968 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --dport 41968 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --sport 41967 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --dport 41967 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --sport 41966 -j
ACCEPT
			-A OUTPUT -o eth5 -p tcp -m tcp --dport 41966 -j
ACCEPT
			COMMIT
			
			Here are the IP in the system,I am using eth5
(Which is in private
			    
			      

		network
		  
		    

			with other two nodes, connected to switch)
			
			[root at usrylxap235 ~]# ip addr list
			1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc
noqueue
			    link/loopback 00:00:00:00:00:00 brd
00:00:00:00:00:00
			    inet 127.0.0.1/8 scope host lo
			    inet6 ::1/128 scope host
			       valid_lft forever preferred_lft forever
			2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu
1500 qdisc pfifo_fast
			    
			      

		qlen
		  
		    

			1000
			    link/ether 00:1c:c4:f0:bd:d8 brd
ff:ff:ff:ff:ff:ff
			    inet 54.3.254.235/24 brd 54.3.254.255 scope
global eth0
			    inet6 fe80::21c:c4ff:fef0:bdd8/64 scope link
			       valid_lft forever preferred_lft forever
			3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc
noop qlen 1000
			    link/ether 00:1c:c4:f0:bd:da brd
ff:ff:ff:ff:ff:ff
			4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc
noop qlen 1000
			    link/ether 00:1c:c4:5e:f8:d8 brd
ff:ff:ff:ff:ff:ff
			5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc
noop qlen 1000
			    link/ether 00:1c:c4:5e:f8:da brd
ff:ff:ff:ff:ff:ff
			6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc
noop qlen 1000
			    link/ether 00:1e:0b:71:ac:6c brd
ff:ff:ff:ff:ff:ff
			7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu
1500 qdisc pfifo_fast
			    
			      

		qlen
		  
		    

			1000
			    link/ether 00:1e:0b:71:ac:6e brd
ff:ff:ff:ff:ff:ff
			    inet 192.168.0.7/24 brd 192.168.0.255 scope
global eth5
			    inet6 fe80::21e:bff:fe71:ac6e/64 scope link
			       valid_lft forever preferred_lft forever
			8: sit0: <NOARP> mtu 1480 qdisc noop
			    link/sit 0.0.0.0 brd 0.0.0.0
			
			Thanks
			Anoop
			    
			      

		Hi,
		
		Let's see if we can rule out security or network issues
here. Are you
		able to allow all traffic for eth5 on all nodes? In
addition, you may
		want to add a static multicast route to ensure the
multicast traffic
		is going through eth5 on all nodes.
		
		Regards,
		Bernard
		
		
		    

	
-----------------------------------------------------------------------
	  

		He's essentially running a wide open firewall with those
rules.
		    

	Default
	  

		ACCEPT on all chains, and no DROPs or REJECTs.  It's not
a firewall
		issue.  It may be multicast not going out the right
interface,
		    

	however.
	  

		
		------------------------------
		
		Message: 2
		Date: Sun, 18 Jul 2010 11:28:53 +0200
		From: Volker Dormeyer <volker at ixolution.de>
<mailto:volker at ixolution.de> 
		To: linux clustering <linux-cluster at redhat.com>
<mailto:linux-cluster at redhat.com> 
		Subject: Re: [Linux-cluster] How do i stop VM's on a
failed node?
		Message-ID: <20100718092853.GA4159 at dijkstra>
		Content-Type: text/plain; charset=us-ascii
		
		Hi,
		
		On Fri, Jul 16, 2010 at 01:21:57PM -0400,
		Nathan Lager <lagern at lafayette.edu>
<mailto:lagern at lafayette.edu>  wrote:
		  
		    

			I have a 4-node cluster, running KVM, and a host
of VM's.
			
			One of the nodes failed unexpectedly.  I'm
having two issues.  First,
			the VM's which were on that node are still
reported as up and running
			    
			      

		on
		  
		    

			that node in clustat, and second, i'm unable to
interact with the
			cluster using clusvcadm.  Every command hangs.
I've tried disabling
			these VM's, and the commands hang.
			
			How can i clear out this dead node, without
having to forceably
			    
			      

		restart
		  
		    

			my entire cluster?
			    
			      

		To me, it sounds like fencing is not configured,
properly.
		
		What has been logged on the remaining nodes? I would
assume, they
		    

	tried
	  

		to
		fence the failed node - but are not able to.
		
		Log-Snippets and Config would be helpful.
		
		Regards,
		Volker
		
		
		
		------------------------------
		
		--
		Linux-cluster mailing list
		Linux-cluster at redhat.com
		https://www.redhat.com/mailman/listinfo/linux-cluster
		
		End of Linux-cluster Digest, Vol 75, Issue 19
		*********************************************
		Notice:  This e-mail message, together with any
attachments, contains
		information of Merck & Co., Inc. (One Merck Drive,
Whitehouse Station,
		New Jersey, USA 08889), and/or its affiliates Direct
contact
		    

	information
	  

		for affiliates is available at 
		http://www.merck.com/contact/contacts.html) that may be
confidential,
		proprietary copyrighted and/or legally privileged. It is
intended
		    

	solely
	  

		for the use of the individual or entity named on this
message. If you
		    

	are
	  

		not the intended recipient, and have received this
message in error,
		please notify us immediately by reply e-mail and then
delete it from 
		your system.
		
		
		--
		Linux-cluster mailing list
		Linux-cluster at redhat.com
		https://www.redhat.com/mailman/listinfo/linux-cluster
		  
		    

	
	  


-- 
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania
Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100721/35f24245/attachment.htm>

From Wahyu.Darmawan at ag-it.com  Thu Jul 22 05:29:06 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Thu, 22 Jul 2010 12:29:06 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
Message-ID: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>

Hi all,
I have problem with my cluster. I have 2 nodes of my cluster.
On July 12 the nodes missed  connection from storage and lost 1 disk partition on both nodes. Then I decided to assigned a new of partition from storage, and use it on my nodes.
But until today, the quorum is not going to quorate.
FYI, I use GFS for my filesystem.
So, what do I have to do to make quorum disk back to quorate status?

I appreciate for your answer.

Thanks,
Wahyu

Note, I attached the cluster log messages below:
Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is disabled
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service Oracle_DBrepwm
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is disabled
Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBahmcm
Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm started
Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBportal
Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal started
Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBcard5
Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5 started
Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBhrdtxn
Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn started
Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBrepwm
Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm started
Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)
Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.310000)
Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.400000)
Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.690000)
Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.620000)
Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.020000)
Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.550000)
Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.050000)
Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.710000)
Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:05:30 t0175 last message repeated 2 times
Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.530000)
Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.200000)
Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.820000)
Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.420000)
Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.820000)
Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.540000)
Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.740000)
Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.020000)
Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.900000)
Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.270000)
Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.980000)
Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.470000)
Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.870000)
Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.940000)
Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.780000)
Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.360000)
Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.790000)
Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.730000)
Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.100000)
Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.580000)
Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.460000)
Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.790000)
Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.240000)
Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.670000)
Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.570000)
Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.000000)
Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.670000)
Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 18 02:04:31 t0175 last message repeated 3 times
Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.870000)
Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.920000)
Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.040000)
Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.350000)
Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.450000)
Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.450000)
Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.120000)
Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.480000)
Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.960000)
Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.030000)
Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.940000)
Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.560000)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100722/3a83e27e/attachment.htm>

From Wahyu.Darmawan at ag-it.com  Thu Jul 22 12:44:20 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Thu, 22 Jul 2010 19:44:20 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
Message-ID: <0FE1B519D219B64A87E99E36202385450517247FA8@svr-e2k7.ag-it.com>

Hi all,
Any update?

Thank you.



From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Wahyu Darmawan
Sent: Thursday, July 22, 2010 12:29 PM
To: linux-cluster at redhat.com
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi all,
I have problem with my cluster. I have 2 nodes of my cluster.
On July 12 the nodes missed  connection from storage and lost 1 disk partition on both nodes. Then I decided to assigned a new of partition from storage, and use it on my nodes.
But until today, the quorum is not going to quorate.
FYI, I use GFS for my filesystem.
So, what do I have to do to make quorum disk back to quorate status?

I appreciate for your answer.

Thanks,
Wahyu

Note, I attached the cluster log messages below:
Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is disabled
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service Oracle_DBrepwm
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is disabled
Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBahmcm
Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm started
Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBportal
Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal started
Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBcard5
Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5 started
Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBhrdtxn
Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn started
Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBrepwm
Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm started
Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)
Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.310000)
Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.400000)
Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.690000)
Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.620000)
Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.020000)
Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.550000)
Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.050000)
Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.710000)
Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:05:30 t0175 last message repeated 2 times
Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.530000)
Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.200000)
Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.820000)
Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.420000)
Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.820000)
Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.540000)
Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.740000)
Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.020000)
Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.900000)
Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.270000)
Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.980000)
Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.470000)
Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.870000)
Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.940000)
Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.780000)
Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.360000)
Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.790000)
Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.730000)
Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.100000)
Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.580000)
Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.460000)
Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.790000)
Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.240000)
Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.670000)
Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.570000)
Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.000000)
Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.670000)
Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 18 02:04:31 t0175 last message repeated 3 times
Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.870000)
Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.920000)
Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.040000)
Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.350000)
Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.450000)
Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.450000)
Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.120000)
Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.480000)
Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.960000)
Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.030000)
Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.940000)
Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.560000)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100722/cd561e7e/attachment.htm>

From sakect at gmail.com  Thu Jul 22 15:53:53 2010
From: sakect at gmail.com (POWERBALL ONLINE)
Date: Thu, 22 Jul 2010 22:53:53 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
Message-ID: <AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>

Hi

What tool you use for create cluster? Is it Conga ?



On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan
<Wahyu.Darmawan at ag-it.com>wrote:

>  Hi all,
>
> I have problem with my cluster. I have 2 nodes of my cluster.
>
> On July 12 the nodes missed  connection from storage and lost 1 disk
> partition on both nodes. Then I decided to assigned a new of partition from
> storage, and use it on my nodes.
>
> But until today, the quorum is not going to quorate.
>
> FYI, I use GFS for my filesystem.
>
> So, what do I have to do to make quorum disk back to quorate status?
>
>
>
> I appreciate for your answer.
>
>
>
> Thanks,
>
> Wahyu
>
>
>
> Note, I attached the cluster log messages below:
>
> Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is
> disabled
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service
> Oracle_DBrepwm
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is
> disabled
>
> Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBahmcm
>
> Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm
> started
>
> Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBportal
>
> Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal
> started
>
> Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBcard5
>
> Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5
> started
>
> Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBhrdtxn
>
> Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn
> started
>
> Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBrepwm
>
> Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm
> started
>
> Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.120000)
>
> Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.070000)
>
> Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.310000)
>
> Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.400000)
>
> Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.690000)
>
> Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.620000)
>
> Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.020000)
>
> Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.550000)
>
> Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.050000)
>
> Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.710000)
>
> Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:05:30 t0175 last message repeated 2 times
>
> Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.530000)
>
> Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.200000)
>
> Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.820000)
>
> Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.420000)
>
> Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.820000)
>
> Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.540000)
>
> Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.740000)
>
> Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.020000)
>
> Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.900000)
>
> Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.270000)
>
> Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.980000)
>
> Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.470000)
>
> Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.870000)
>
> Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.940000)
>
> Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.780000)
>
> Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.360000)
>
> Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.790000)
>
> Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.730000)
>
> Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.100000)
>
> Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.580000)
>
> Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.460000)
>
> Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.790000)
>
> Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.240000)
>
> Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.670000)
>
> Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.570000)
>
> Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.000000)
>
> Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.670000)
>
> Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 18 02:04:31 t0175 last message repeated 3 times
>
> Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.870000)
>
> Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.920000)
>
> Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.040000)
>
> Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.350000)
>
> Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.450000)
>
> Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.450000)
>
> Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.120000)
>
> Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.480000)
>
> Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.960000)
>
> Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.030000)
>
> Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.940000)
>
> Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.560000)
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100722/dbe1cbfd/attachment.htm>

From Wahyu.Darmawan at ag-it.com  Thu Jul 22 16:02:25 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Thu, 22 Jul 2010 23:02:25 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
Message-ID: <0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>

Yes correct. I use Conga for my clusterware.

Thanks,


From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of POWERBALL ONLINE
Sent: Thursday, July 22, 2010 10:54 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi

What tool you use for create cluster? Is it Conga ?



On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi all,
I have problem with my cluster. I have 2 nodes of my cluster.
On July 12 the nodes missed  connection from storage and lost 1 disk partition on both nodes. Then I decided to assigned a new of partition from storage, and use it on my nodes.
But until today, the quorum is not going to quorate.
FYI, I use GFS for my filesystem.
So, what do I have to do to make quorum disk back to quorate status?

I appreciate for your answer.

Thanks,
Wahyu

Note, I attached the cluster log messages below:
Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is disabled
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service Oracle_DBrepwm
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is disabled
Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBahmcm
Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm started
Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBportal
Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal started
Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBcard5
Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5 started
Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBhrdtxn
Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn started
Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBrepwm
Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm started
Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)
Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.310000)
Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.400000)
Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.690000)
Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.620000)
Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.020000)
Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.550000)
Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.050000)
Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.710000)
Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:05:30 t0175 last message repeated 2 times
Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.530000)
Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.200000)
Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.820000)
Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.420000)
Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.820000)
Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.540000)
Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.740000)
Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.020000)
Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.900000)
Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.270000)
Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.980000)
Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.470000)
Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.870000)
Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.940000)
Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.780000)
Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.360000)
Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.790000)
Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.730000)
Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.100000)
Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.580000)
Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.460000)
Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.790000)
Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.240000)
Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.670000)
Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.570000)
Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.000000)
Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.670000)
Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 18 02:04:31 t0175 last message repeated 3 times
Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.870000)
Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.920000)
Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.040000)
Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.350000)
Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.450000)
Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.450000)
Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.120000)
Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.480000)
Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.960000)
Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.030000)
Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.940000)
Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.560000)


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100722/5e4ee696/attachment.htm>

From Wahyu.Darmawan at ag-it.com  Thu Jul 22 16:43:30 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Thu, 22 Jul 2010 23:43:30 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
Message-ID: <0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>

Hi,
So, what your suggestion?

Thanks,


From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Wahyu Darmawan
Sent: Thursday, July 22, 2010 11:02 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Yes correct. I use Conga for my clusterware.

Thanks,


From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of POWERBALL ONLINE
Sent: Thursday, July 22, 2010 10:54 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi

What tool you use for create cluster? Is it Conga ?



On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi all,
I have problem with my cluster. I have 2 nodes of my cluster.
On July 12 the nodes missed  connection from storage and lost 1 disk partition on both nodes. Then I decided to assigned a new of partition from storage, and use it on my nodes.
But until today, the quorum is not going to quorate.
FYI, I use GFS for my filesystem.
So, what do I have to do to make quorum disk back to quorate status?

I appreciate for your answer.

Thanks,
Wahyu

Note, I attached the cluster log messages below:
Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is disabled
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service Oracle_DBrepwm
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is disabled
Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBahmcm
Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm started
Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBportal
Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal started
Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBcard5
Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5 started
Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBhrdtxn
Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn started
Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBrepwm
Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm started
Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)
Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.310000)
Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.400000)
Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.690000)
Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.620000)
Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.020000)
Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.550000)
Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.050000)
Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.710000)
Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:05:30 t0175 last message repeated 2 times
Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.530000)
Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.200000)
Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.820000)
Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.420000)
Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.820000)
Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.540000)
Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.740000)
Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.020000)
Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.900000)
Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.270000)
Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.980000)
Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.470000)
Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.870000)
Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.940000)
Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.780000)
Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.360000)
Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.790000)
Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.730000)
Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.100000)
Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.580000)
Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.460000)
Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.790000)
Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.240000)
Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.670000)
Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.570000)
Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.000000)
Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.670000)
Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 18 02:04:31 t0175 last message repeated 3 times
Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.870000)
Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.920000)
Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.040000)
Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.350000)
Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.450000)
Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.450000)
Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.120000)
Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.480000)
Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.960000)
Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.030000)
Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.940000)
Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.560000)


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100722/bf0dc2ef/attachment.htm>

From sakect at gmail.com  Thu Jul 22 17:22:14 2010
From: sakect at gmail.com (POWERBALL ONLINE)
Date: Fri, 23 Jul 2010 00:22:14 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
Message-ID: <AANLkTinITjz2fXUE61PxgI9pZnK4nJ1jhikzI_F-9j5w@mail.gmail.com>

Please give me the information after you run the mkqdisk -L command

On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan
<Wahyu.Darmawan at ag-it.com>wrote:

>  Hi,
>
> So, what your suggestion?
>
>
>
> Thanks,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *Wahyu Darmawan
> *Sent:* Thursday, July 22, 2010 11:02 PM
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Yes correct. I use Conga for my clusterware.
>
>
>
> Thanks,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *POWERBALL ONLINE
> *Sent:* Thursday, July 22, 2010 10:54 PM
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Hi
>
>
>
> What tool you use for create cluster? Is it Conga ?
>
>
>
>
>
> On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi all,
>
> I have problem with my cluster. I have 2 nodes of my cluster.
>
> On July 12 the nodes missed  connection from storage and lost 1 disk
> partition on both nodes. Then I decided to assigned a new of partition from
> storage, and use it on my nodes.
>
> But until today, the quorum is not going to quorate.
>
> FYI, I use GFS for my filesystem.
>
> So, what do I have to do to make quorum disk back to quorate status?
>
>
>
> I appreciate for your answer.
>
>
>
> Thanks,
>
> Wahyu
>
>
>
> Note, I attached the cluster log messages below:
>
> Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is
> disabled
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service
> Oracle_DBrepwm
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is
> disabled
>
> Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBahmcm
>
> Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm
> started
>
> Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBportal
>
> Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal
> started
>
> Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBcard5
>
> Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5
> started
>
> Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBhrdtxn
>
> Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn
> started
>
> Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBrepwm
>
> Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm
> started
>
> Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.120000)
>
> Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.070000)
>
> Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.310000)
>
> Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.400000)
>
> Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.690000)
>
> Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.620000)
>
> Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.020000)
>
> Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.550000)
>
> Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.050000)
>
> Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.710000)
>
> Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:05:30 t0175 last message repeated 2 times
>
> Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.530000)
>
> Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.200000)
>
> Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.820000)
>
> Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.420000)
>
> Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.820000)
>
> Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.540000)
>
> Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.740000)
>
> Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.020000)
>
> Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.900000)
>
> Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.270000)
>
> Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.980000)
>
> Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.470000)
>
> Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.870000)
>
> Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.940000)
>
> Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.780000)
>
> Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.360000)
>
> Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.790000)
>
> Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.730000)
>
> Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.100000)
>
> Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.580000)
>
> Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.460000)
>
> Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.790000)
>
> Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.240000)
>
> Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.670000)
>
> Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.570000)
>
> Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.000000)
>
> Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.670000)
>
> Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 18 02:04:31 t0175 last message repeated 3 times
>
> Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.870000)
>
> Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.920000)
>
> Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.040000)
>
> Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.350000)
>
> Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.450000)
>
> Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.450000)
>
> Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.120000)
>
> Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.480000)
>
> Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.960000)
>
> Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.030000)
>
> Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.940000)
>
> Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.560000)
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100723/78071438/attachment.htm>

From sakect at gmail.com  Thu Jul 22 17:28:37 2010
From: sakect at gmail.com (POWERBALL ONLINE)
Date: Fri, 23 Jul 2010 00:28:37 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
Message-ID: <AANLkTikdqYmBW3xDvraban3CIdexrZMqtUQtKIbXNtCO@mail.gmail.com>

Please give me the information after you run mkqdisk -L and clustat -i
command

On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan
<Wahyu.Darmawan at ag-it.com>wrote:

>  Hi,
>
> So, what your suggestion?
>
>
>
> Thanks,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *Wahyu Darmawan
> *Sent:* Thursday, July 22, 2010 11:02 PM
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Yes correct. I use Conga for my clusterware.
>
>
>
> Thanks,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *POWERBALL ONLINE
> *Sent:* Thursday, July 22, 2010 10:54 PM
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Hi
>
>
>
> What tool you use for create cluster? Is it Conga ?
>
>
>
>
>
> On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi all,
>
> I have problem with my cluster. I have 2 nodes of my cluster.
>
> On July 12 the nodes missed  connection from storage and lost 1 disk
> partition on both nodes. Then I decided to assigned a new of partition from
> storage, and use it on my nodes.
>
> But until today, the quorum is not going to quorate.
>
> FYI, I use GFS for my filesystem.
>
> So, what do I have to do to make quorum disk back to quorate status?
>
>
>
> I appreciate for your answer.
>
>
>
> Thanks,
>
> Wahyu
>
>
>
> Note, I attached the cluster log messages below:
>
> Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is
> disabled
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service
> Oracle_DBrepwm
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is
> disabled
>
> Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBahmcm
>
> Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm
> started
>
> Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBportal
>
> Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal
> started
>
> Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBcard5
>
> Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5
> started
>
> Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBhrdtxn
>
> Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn
> started
>
> Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBrepwm
>
> Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm
> started
>
> Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.120000)
>
> Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.070000)
>
> Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.310000)
>
> Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.400000)
>
> Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.690000)
>
> Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.620000)
>
> Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.020000)
>
> Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.550000)
>
> Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.050000)
>
> Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.710000)
>
> Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:05:30 t0175 last message repeated 2 times
>
> Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.530000)
>
> Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.200000)
>
> Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.820000)
>
> Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.420000)
>
> Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.820000)
>
> Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.540000)
>
> Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.740000)
>
> Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.020000)
>
> Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.900000)
>
> Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.270000)
>
> Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.980000)
>
> Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.470000)
>
> Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.870000)
>
> Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.940000)
>
> Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.780000)
>
> Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.360000)
>
> Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.790000)
>
> Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.730000)
>
> Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.100000)
>
> Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.580000)
>
> Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.460000)
>
> Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.790000)
>
> Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.240000)
>
> Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.670000)
>
> Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.570000)
>
> Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.000000)
>
> Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.670000)
>
> Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 18 02:04:31 t0175 last message repeated 3 times
>
> Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.870000)
>
> Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.920000)
>
> Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.040000)
>
> Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.350000)
>
> Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.450000)
>
> Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.450000)
>
> Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.120000)
>
> Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.480000)
>
> Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.960000)
>
> Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.030000)
>
> Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.940000)
>
> Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.560000)
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100723/7aaa5f97/attachment.htm>

From jayfitzpatrick at gmail.com  Thu Jul 22 20:35:46 2010
From: jayfitzpatrick at gmail.com (Jason Fitzpatrick)
Date: Thu, 22 Jul 2010 21:35:46 +0100
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <AANLkTinITjz2fXUE61PxgI9pZnK4nJ1jhikzI_F-9j5w@mail.gmail.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
	<AANLkTinITjz2fXUE61PxgI9pZnK4nJ1jhikzI_F-9j5w@mail.gmail.com>
Message-ID: <AANLkTilv5XMZ4_V2zUqQjX0tJDQ3V2FugPe4k7SV2qGP@mail.gmail.com>

Hi..

I had a quick look around and this seems to be listed as a bug here:
https://bugzilla.redhat.com/show_bug.cgi?id=490147

also can you check your CPU usage as per here:
http://markmail.org/message/2a4e5nkxywane3sz

if it is GFS that is causing the problems you may need to unmount the
drive and run a fsck on it (I have had similar after attempting to
mount a corrupt GFS volume)

Hope this helps

Jay

On 22 July 2010 18:22, POWERBALL ONLINE <sakect at gmail.com> wrote:
> Please give me the information after you run the mkqdisk -L command
>
> On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>>
>> Hi,
>>
>> So, what your suggestion?
>>
>>
>>
>> Thanks,
>>
>>
>>
>>
>>
>> From: linux-cluster-bounces at redhat.com
>> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Wahyu Darmawan
>> Sent: Thursday, July 22, 2010 11:02 PM
>> To: linux clustering
>> Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>>
>>
>>
>> Yes correct. I use Conga for my clusterware.
>>
>>
>>
>> Thanks,
>>
>>
>>
>>
>>
>> From: linux-cluster-bounces at redhat.com
>> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of POWERBALL ONLINE
>> Sent: Thursday, July 22, 2010 10:54 PM
>> To: linux clustering
>> Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>>
>>
>>
>> Hi
>>
>>
>>
>> What tool you use for create cluster? Is it Conga ?
>>
>>
>>
>> On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan
>> <Wahyu.Darmawan at ag-it.com> wrote:
>>
>> Hi all,
>>
>> I have problem with my cluster. I have 2 nodes of my cluster.
>>
>> On July 12 the nodes missed? connection from storage and lost 1 disk
>> partition on both nodes. Then I decided to assigned a new of partition from
>> storage, and use it on my nodes.
>>
>> But until today, the quorum is not going to quorate.
>>
>> FYI, I use GFS for my filesystem.
>>
>> So, what do I have to do to make quorum disk back to quorate status?
>>
>>
>>
>> I appreciate for your answer.
>>
>>
>>
>> Thanks,
>>
>> Wahyu
>>
>>
>>
>> Note, I attached the cluster log messages below:
>>
>> Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn
>> is disabled
>>
>> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service
>> Oracle_DBrepwm
>>
>> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is
>> disabled
>>
>> Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service
>> Oracle_DBahmcm
>>
>> Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm
>> started
>>
>> Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service
>> Oracle_DBportal
>>
>> Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal
>> started
>>
>> Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service
>> Oracle_DBcard5
>>
>> Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5
>> started
>>
>> Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service
>> Oracle_DBhrdtxn
>>
>> Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn
>> started
>>
>> Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service
>> Oracle_DBrepwm
>>
>> Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm
>> started
>>
>> Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (12.120000)
>>
>> Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.900000)
>>
>> Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.010000)
>>
>> Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.010000)
>>
>> Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.070000)
>>
>> Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.310000)
>>
>> Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (12.400000)
>>
>> Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.690000)
>>
>> Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.000000)
>>
>> Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.620000)
>>
>> Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.020000)
>>
>> Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.550000)
>>
>> Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (13.050000)
>>
>> Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.710000)
>>
>> Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.000000)
>>
>> Jul 13 02:05:30 t0175 last message repeated 2 times
>>
>> Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.530000)
>>
>> Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.200000)
>>
>> Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (11.820000)
>>
>> Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.720000)
>>
>> Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.000000)
>>
>> Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.420000)
>>
>> Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.820000)
>>
>> Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (11.540000)
>>
>> Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.740000)
>>
>> Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.010000)
>>
>> Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.000000)
>>
>> Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.980000)
>>
>> Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.020000)
>>
>> Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.900000)
>>
>> Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.270000)
>>
>> Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (11.980000)
>>
>> Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.470000)
>>
>> Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.000000)
>>
>> Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.000000)
>>
>> Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.010000)
>>
>> Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.980000)
>>
>> Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.870000)
>>
>> Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (12.940000)
>>
>> Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.780000)
>>
>> Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.010000)
>>
>> Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.000000)
>>
>> Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.360000)
>>
>> Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.790000)
>>
>> Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.730000)
>>
>> Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (12.100000)
>>
>> Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.720000)
>>
>> Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.000000)
>>
>> Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.580000)
>>
>> Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.720000)
>>
>> Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (12.460000)
>>
>> Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.790000)
>>
>> Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.000000)
>>
>> Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.000000)
>>
>> Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.000000)
>>
>> Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.240000)
>>
>> Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.670000)
>>
>> Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.570000)
>>
>> Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (13.000000)
>>
>> Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.670000)
>>
>> Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.000000)
>>
>> Jul 18 02:04:31 t0175 last message repeated 3 times
>>
>> Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (11.870000)
>>
>> Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.930000)
>>
>> Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.000000)
>>
>> Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.010000)
>>
>> Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.980000)
>>
>> Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.900000)
>>
>> Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (11.920000)
>>
>> Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.930000)
>>
>> Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.010000)
>>
>> Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.000000)
>>
>> Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.040000)
>>
>> Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (2.350000)
>>
>> Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (12.450000)
>>
>> Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.450000)
>>
>> Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.010000)
>>
>> Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.120000)
>>
>> Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.480000)
>>
>> Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (11.960000)
>>
>> Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.030000)
>>
>> Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.000000)
>>
>> Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.990000)
>>
>> Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.000000)
>>
>> Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.000000)
>>
>> Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.940000)
>>
>> Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than
>> 1 second to complete (1.560000)
>>
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 

"The only difference between saints and sinners is that every saint
has a past while every sinner has a future. "
? Oscar Wilde



From gianluca.cecchi at gmail.com  Thu Jul 22 20:37:48 2010
From: gianluca.cecchi at gmail.com (Gianluca Cecchi)
Date: Thu, 22 Jul 2010 22:37:48 +0200
Subject: [Linux-cluster] question about stop service and then shutdown node
Message-ID: <AANLkTim5R7Ym822JDDD9KfIszUNKs4bsRrwSlpKM0kJP@mail.gmail.com>

Hello,
based on cluster version 2 architecture, and in particular on a two nodes
cluster with quorum disk and rh el 5.5 + updates operating systems, is this
below the expected behavior?

baseline
service X set in autostart=1
node A + node B up and running with service X started on node A

first scenario :
1) clusvcadm -s X
2) shutdown -h now on node A

==> node B during its reconfiguration phase starts service X

second scenario :
1) clusvcadm -s X
2) shutdown -h now on node B

==> node A during its reconfiguration phase starts service X

Thanks in advance
Gianluca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100722/0827a96f/attachment.htm>

From gianluca.cecchi at gmail.com  Thu Jul 22 20:58:14 2010
From: gianluca.cecchi at gmail.com (Gianluca Cecchi)
Date: Thu, 22 Jul 2010 22:58:14 +0200
Subject: [Linux-cluster] rhcs upgrade from rh el 5.4 to rh el 5.5 and alb
	bonding on production lan
Message-ID: <AANLkTimDLUgpw1MT7Vt8kDlY3Aclv5_eWa64-CkkFfCW@mail.gmail.com>

Hello,

target is upgrade from rh el 5.4 to 5.5 in a two nodes cluster with quorum
disk.
Part of the upgrade is also replacement of the nodes themselves.

Current nodes are using alb bonding for the production lan and the driver
for the underlying eth adapters is tg3
Target nodes I'm testing are again with alb bonding for the production lan
and the driver for the underlying eth adapters is bnx2

During relocation of service I can see that the node that has released the
virtual ip continues to send arp replies to some particular clients on the
same lan telling that the virtual ip "is-at" its mac address....
(done via tcpdump on the client itself)

This way these clients are then unable to ping and reach the cluster ip.
Instead it seems that clients on other lans are not influenced (aka the
routers are ok... as I also verified in arp table of the router on the lan
of nodes)

Even if I do
arp -d vip

suddenly the arp cache of the client is filled again with the vip associated
to the mac of this node that doesn't have the vip anymore
(on this node an "arp -an" correctly says that the mac is the other node
one...)

If I shutdown the (now) passive node, the arp replies stop and after an "arp
-d" on the client, it can then reach the vip...

Current workaround is to set active-backup on the production lan bonding.

I don't experimented the problem on the production cluster, but many changes
are involved, such as:
hardware
eth driver from tg3 to bnx2
kernel version updates
rhcs sw updates

Is there any suggested bonding strategy for the production lan of a cluster
that has to manage a vip?
Anyone having problems with bnx2, if this one can be the cause..?

What is the program called by the cluster stack when it has to start/stop a
vip?
in general I can see the vip with the command
ip addr list
but not with
ifconfig -a

I don't know if this is due to ifconfig being deprecated or to the way the
cluster enables the vip on the bonding interface...

Thanks in advance,
Gianluca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100722/dd0b097b/attachment.htm>

From rom at twister.dyndns.org  Thu Jul 22 15:55:38 2010
From: rom at twister.dyndns.org (Fred Wittekind)
Date: Thu, 22 Jul 2010 11:55:38 -0400
Subject: [Linux-cluster] GFS performance issue on CentOS 5.5
Message-ID: <4C4869FA.8020006@twister.dyndns.org>



Two web servers, both virtualized with CentOS Xen servers as host
(residing on two different physical servers). 
GFS used to store home directories containing web document roots.

Shared block device used by GFS is an ISCSI target with the ISCSI
initiator residing on the Dom-0, and presented to Dom-U webservers as
drives.
Also, providing a second shared block device for quorum disk.

If I hit the web site on just one of the nodes, it behaves as expected. 
If I try to load the websites from both nodes at the same time (two web
browser instances), the load average on both nodes spikes, and page
loads very slowly.  The site I am trying to host is very high traffic,
and if the servers can be nearly brought to there knees with two web
browser instances running on a single workstation, that's not going to work.

I am not seeing any error messages in the logs regarding the cluster.

Any help or suggestions on how to troubleshoot this issue would be
greatly appreciated.


[root at www3 www]# find ./ | wc -l
64815

[root at www3 www]# gfs2_tool df /home/www
/home/www:
  SB lock proto = "lock_dlm"
  SB lock table = "Web:homewww"
  SB ondisk format = 1801
  SB multihost format = 1900
  Block size = 4096
  Journals = 2
  Resource Groups = 316
  Mounted lock proto = "lock_dlm"
  Mounted lock table = "Web:homewww"
  Mounted host data = "jid=1:id=393217:first=0"
  Journal number = 1
  Lock module flags = 0
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total Blocks   Used Blocks    Free Blocks    use%
  ------------------------------------------------------------------------
  data           20707148       4347237        16359911       21%
  inodes         16426386       66475          16359911       0%

[root at www3 www]# gfs2_tool gettune /home/www
new_files_directio = 0
new_files_jdata = 0
quota_scale = 1.0000   (1, 1)
logd_secs = 1
recoverd_secs = 60
statfs_quantum = 30
stall_secs = 600
quota_cache_secs = 300
quota_simul_sync = 64
statfs_slow = 0
complain_secs = 10
max_readahead = 262144
quota_quantum = 60
quota_warn_period = 10
jindex_refresh_secs = 60
log_flush_secs = 60
incore_log_blocks = 1024

[root at www3 www]# cat /etc/cluster/cluster.conf |egrep '(dlm)|(gfs)'
        <dlm plock_ownership="1" plock_rate_limit="0"/>
        <gfs_controld plock_rate_limit="0"/>


Fred Wittekind



From swhiteho at redhat.com  Fri Jul 23 15:20:59 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Fri, 23 Jul 2010 16:20:59 +0100
Subject: [Linux-cluster] GFS performance issue on CentOS 5.5
In-Reply-To: <4C4869FA.8020006@twister.dyndns.org>
References: <4C4869FA.8020006@twister.dyndns.org>
Message-ID: <1279898459.2465.39.camel@localhost>

Hi,

Did you mount the fs noatime? Are there any writes to the fs?

Steve.

On Thu, 2010-07-22 at 11:55 -0400, Fred Wittekind wrote:
> 
> Two web servers, both virtualized with CentOS Xen servers as host
> (residing on two different physical servers). 
> GFS used to store home directories containing web document roots.
> 
> Shared block device used by GFS is an ISCSI target with the ISCSI
> initiator residing on the Dom-0, and presented to Dom-U webservers as
> drives.
> Also, providing a second shared block device for quorum disk.
> 
> If I hit the web site on just one of the nodes, it behaves as expected. 
> If I try to load the websites from both nodes at the same time (two web
> browser instances), the load average on both nodes spikes, and page
> loads very slowly.  The site I am trying to host is very high traffic,
> and if the servers can be nearly brought to there knees with two web
> browser instances running on a single workstation, that's not going to work.
> 
> I am not seeing any error messages in the logs regarding the cluster.
> 
> Any help or suggestions on how to troubleshoot this issue would be
> greatly appreciated.
> 
> 
> [root at www3 www]# find ./ | wc -l
> 64815
> 
> [root at www3 www]# gfs2_tool df /home/www
> /home/www:
>   SB lock proto = "lock_dlm"
>   SB lock table = "Web:homewww"
>   SB ondisk format = 1801
>   SB multihost format = 1900
>   Block size = 4096
>   Journals = 2
>   Resource Groups = 316
>   Mounted lock proto = "lock_dlm"
>   Mounted lock table = "Web:homewww"
>   Mounted host data = "jid=1:id=393217:first=0"
>   Journal number = 1
>   Lock module flags = 0
>   Local flocks = FALSE
>   Local caching = FALSE
> 
>   Type           Total Blocks   Used Blocks    Free Blocks    use%
>   ------------------------------------------------------------------------
>   data           20707148       4347237        16359911       21%
>   inodes         16426386       66475          16359911       0%
> 
> [root at www3 www]# gfs2_tool gettune /home/www
> new_files_directio = 0
> new_files_jdata = 0
> quota_scale = 1.0000   (1, 1)
> logd_secs = 1
> recoverd_secs = 60
> statfs_quantum = 30
> stall_secs = 600
> quota_cache_secs = 300
> quota_simul_sync = 64
> statfs_slow = 0
> complain_secs = 10
> max_readahead = 262144
> quota_quantum = 60
> quota_warn_period = 10
> jindex_refresh_secs = 60
> log_flush_secs = 60
> incore_log_blocks = 1024
> 
> [root at www3 www]# cat /etc/cluster/cluster.conf |egrep '(dlm)|(gfs)'
>         <dlm plock_ownership="1" plock_rate_limit="0"/>
>         <gfs_controld plock_rate_limit="0"/>
> 
> 
> Fred Wittekind
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From gordan at bobich.net  Fri Jul 23 15:15:49 2010
From: gordan at bobich.net (Gordan Bobic)
Date: Fri, 23 Jul 2010 16:15:49 +0100
Subject: [Linux-cluster] GFS performance issue on CentOS 5.5
In-Reply-To: <4C4869FA.8020006@twister.dyndns.org>
References: <4C4869FA.8020006@twister.dyndns.org>
Message-ID: <1279898149.2749.9.camel@thinkpad>

Sounds like you are suffering from extreme lock bouncing between the
nodes. This is a FAQ. I suggest you have a read through the mail
archives of this list for similar discussions, e.g.:

http://www.mail-archive.com/linux-cluster at redhat.com/msg04412.html
http://www.mail-archive.com/linux-cluster at redhat.com/msg08177.html

Gordan

On Thu, 2010-07-22 at 11:55 -0400, Fred Wittekind wrote:
> 
> Two web servers, both virtualized with CentOS Xen servers as host
> (residing on two different physical servers). 
> GFS used to store home directories containing web document roots.
> 
> Shared block device used by GFS is an ISCSI target with the ISCSI
> initiator residing on the Dom-0, and presented to Dom-U webservers as
> drives.
> Also, providing a second shared block device for quorum disk.
> 
> If I hit the web site on just one of the nodes, it behaves as expected. 
> If I try to load the websites from both nodes at the same time (two web
> browser instances), the load average on both nodes spikes, and page
> loads very slowly.  The site I am trying to host is very high traffic,
> and if the servers can be nearly brought to there knees with two web
> browser instances running on a single workstation, that's not going to work.
> 
> I am not seeing any error messages in the logs regarding the cluster.
> 
> Any help or suggestions on how to troubleshoot this issue would be
> greatly appreciated.
> 
> 
> [root at www3 www]# find ./ | wc -l
> 64815
> 
> [root at www3 www]# gfs2_tool df /home/www
> /home/www:
>   SB lock proto = "lock_dlm"
>   SB lock table = "Web:homewww"
>   SB ondisk format = 1801
>   SB multihost format = 1900
>   Block size = 4096
>   Journals = 2
>   Resource Groups = 316
>   Mounted lock proto = "lock_dlm"
>   Mounted lock table = "Web:homewww"
>   Mounted host data = "jid=1:id=393217:first=0"
>   Journal number = 1
>   Lock module flags = 0
>   Local flocks = FALSE
>   Local caching = FALSE
> 
>   Type           Total Blocks   Used Blocks    Free Blocks    use%
>   ------------------------------------------------------------------------
>   data           20707148       4347237        16359911       21%
>   inodes         16426386       66475          16359911       0%
> 
> [root at www3 www]# gfs2_tool gettune /home/www
> new_files_directio = 0
> new_files_jdata = 0
> quota_scale = 1.0000   (1, 1)
> logd_secs = 1
> recoverd_secs = 60
> statfs_quantum = 30
> stall_secs = 600
> quota_cache_secs = 300
> quota_simul_sync = 64
> statfs_slow = 0
> complain_secs = 10
> max_readahead = 262144
> quota_quantum = 60
> quota_warn_period = 10
> jindex_refresh_secs = 60
> log_flush_secs = 60
> incore_log_blocks = 1024
> 
> [root at www3 www]# cat /etc/cluster/cluster.conf |egrep '(dlm)|(gfs)'
>         <dlm plock_ownership="1" plock_rate_limit="0"/>
>         <gfs_controld plock_rate_limit="0"/>
> 
> 
> Fred Wittekind
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From Wahyu.Darmawan at ag-it.com  Fri Jul 23 15:18:12 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Fri, 23 Jul 2010 22:18:12 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <AANLkTikdqYmBW3xDvraban3CIdexrZMqtUQtKIbXNtCO@mail.gmail.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
	<AANLkTikdqYmBW3xDvraban3CIdexrZMqtUQtKIbXNtCO@mail.gmail.com>
Message-ID: <0FE1B519D219B64A87E99E362023854505172482F1@svr-e2k7.ag-it.com>

Hi,
Thanks for your response.
Here is the info on below.


T0171
[root at t0171 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  t0171.example.com                    Online, Local, rgmanager
  t0175.example.com                    Online, rgmanager

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  IP_Virtual           t0175.example.com          started
  GFS_Mounting         t0175.example.com          started
  Listener_Service     t0175.example.com          started
  Oracle_DBahmcm       t0175.example.com          started
  Oracle_DBportal      t0175.example.com          started
  Oracle_DBcard5       t0175.example.com          started
  Oracle_DBhrdtxn      t0175.example.com          started
  Oracle_DBrepwm       t0175.example.com          started
[root at t0171 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171.example.com
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/sdi2:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-18:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171.example.com
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-19:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com
                Kernel Sector Size:   512
                Recorded Sector Size: 512


T0175
[root at t0175 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  t0171.example.com                    Online, rgmanager
  t0175.example.com                    Online, Local, rgmanager

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  IP_Virtual           t0175. example.com          started
  GFS_Mounting         t0175. example.com          started
  Listener_Service     t0175. example.com          started
  Oracle_DBahmcm       t0175. example.com          started
  Oracle_DBportal      t0175. example.com          started
  Oracle_DBcard5       t0175. example.com          started
  Oracle_DBhrdtxn      t0175. example.com          started
  Oracle_DBrepwm       t0175. example.com          started
[root at t0175 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171. example.com
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/sdi2:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175. example.com
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-19:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171. example.com
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-20:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com
                Kernel Sector Size:   512
                Recorded Sector Size: 512


Thanks & Regards,
WD

From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of POWERBALL ONLINE
Sent: Friday, July 23, 2010 12:29 AM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Please give me the information after you run mkqdisk -L and clustat -i command
On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi,
So, what your suggestion?

Thanks,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Wahyu Darmawan
Sent: Thursday, July 22, 2010 11:02 PM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Yes correct. I use Conga for my clusterware.

Thanks,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Thursday, July 22, 2010 10:54 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi

What tool you use for create cluster? Is it Conga ?



On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi all,
I have problem with my cluster. I have 2 nodes of my cluster.
On July 12 the nodes missed  connection from storage and lost 1 disk partition on both nodes. Then I decided to assigned a new of partition from storage, and use it on my nodes.
But until today, the quorum is not going to quorate.
FYI, I use GFS for my filesystem.
So, what do I have to do to make quorum disk back to quorate status?

I appreciate for your answer.

Thanks,
Wahyu

Note, I attached the cluster log messages below:
Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is disabled
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service Oracle_DBrepwm
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is disabled
Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBahmcm
Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm started
Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBportal
Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal started
Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBcard5
Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5 started
Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBhrdtxn
Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn started
Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBrepwm
Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm started
Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)
Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.310000)
Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.400000)
Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.690000)
Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.620000)
Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.020000)
Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.550000)
Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.050000)
Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.710000)
Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:05:30 t0175 last message repeated 2 times
Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.530000)
Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.200000)
Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.820000)
Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.420000)
Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.820000)
Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.540000)
Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.740000)
Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.020000)
Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.900000)
Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.270000)
Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.980000)
Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.470000)
Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.870000)
Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.940000)
Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.780000)
Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.360000)
Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.790000)
Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.730000)
Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.100000)
Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.580000)
Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.460000)
Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.790000)
Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.240000)
Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.670000)
Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.570000)
Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.000000)
Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.670000)
Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 18 02:04:31 t0175 last message repeated 3 times
Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.870000)
Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.920000)
Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.040000)
Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.350000)
Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.450000)
Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.450000)
Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.120000)
Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.480000)
Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.960000)
Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.030000)
Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.940000)
Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.560000)


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100723/96268315/attachment.htm>

From rom at twister.dyndns.org  Fri Jul 23 17:34:02 2010
From: rom at twister.dyndns.org (Fred Wittekind)
Date: Fri, 23 Jul 2010 13:34:02 -0400
Subject: [Linux-cluster] GFS performance issue on CentOS 5.5
In-Reply-To: <1279898459.2465.39.camel@localhost>
References: <4C4869FA.8020006@twister.dyndns.org>
	<1279898459.2465.39.camel@localhost>
Message-ID: <4C49D28A.5020704@twister.dyndns.org>

 I do have noatime, need to add nodiratime.  There are PHP session files
on the gfs2 volume.  There used to be Zend Cache file on it, but I moved
those to tmpfs.  There is one other set of cache files that where put in
a bad directory, so they are hard to move to tmpfs without changing the
client's website code (although I may do this anyways).

I had considered moving the PHP session files to a memcache server, but
then I observed very long load times on static website content
(images).  Since images don't involve the session files, I had decided
that it looked like there was more of a problem than the session files. 
Although, could lock issues with the session files effect load times of
files in a completely different area of the directory tree?

I am not aware of any other files that are written to frequently.

On 7/23/2010 11:20 AM, Steven Whitehouse wrote:
> Hi,
>
> Did you mount the fs noatime? Are there any writes to the fs?
>
> Steve.
>
> On Thu, 2010-07-22 at 11:55 -0400, Fred Wittekind wrote:
>> Two web servers, both virtualized with CentOS Xen servers as host
>> (residing on two different physical servers). 
>> GFS used to store home directories containing web document roots.
>>
>> Shared block device used by GFS is an ISCSI target with the ISCSI
>> initiator residing on the Dom-0, and presented to Dom-U webservers as
>> drives.
>> Also, providing a second shared block device for quorum disk.
>>
>> If I hit the web site on just one of the nodes, it behaves as expected. 
>> If I try to load the websites from both nodes at the same time (two web
>> browser instances), the load average on both nodes spikes, and page
>> loads very slowly.  The site I am trying to host is very high traffic,
>> and if the servers can be nearly brought to there knees with two web
>> browser instances running on a single workstation, that's not going to work.
>>
>> I am not seeing any error messages in the logs regarding the cluster.
>>
>> Any help or suggestions on how to troubleshoot this issue would be
>> greatly appreciated.
>>
>>
>> [root at www3 www]# find ./ | wc -l
>> 64815
>>
>> [root at www3 www]# gfs2_tool df /home/www
>> /home/www:
>>   SB lock proto = "lock_dlm"
>>   SB lock table = "Web:homewww"
>>   SB ondisk format = 1801
>>   SB multihost format = 1900
>>   Block size = 4096
>>   Journals = 2
>>   Resource Groups = 316
>>   Mounted lock proto = "lock_dlm"
>>   Mounted lock table = "Web:homewww"
>>   Mounted host data = "jid=1:id=393217:first=0"
>>   Journal number = 1
>>   Lock module flags = 0
>>   Local flocks = FALSE
>>   Local caching = FALSE
>>
>>   Type           Total Blocks   Used Blocks    Free Blocks    use%
>>   ------------------------------------------------------------------------
>>   data           20707148       4347237        16359911       21%
>>   inodes         16426386       66475          16359911       0%
>>
>> [root at www3 www]# gfs2_tool gettune /home/www
>> new_files_directio = 0
>> new_files_jdata = 0
>> quota_scale = 1.0000   (1, 1)
>> logd_secs = 1
>> recoverd_secs = 60
>> statfs_quantum = 30
>> stall_secs = 600
>> quota_cache_secs = 300
>> quota_simul_sync = 64
>> statfs_slow = 0
>> complain_secs = 10
>> max_readahead = 262144
>> quota_quantum = 60
>> quota_warn_period = 10
>> jindex_refresh_secs = 60
>> log_flush_secs = 60
>> incore_log_blocks = 1024
>>
>> [root at www3 www]# cat /etc/cluster/cluster.conf |egrep '(dlm)|(gfs)'
>>         <dlm plock_ownership="1" plock_rate_limit="0"/>
>>         <gfs_controld plock_rate_limit="0"/>
>>
>>
>> Fred Wittekind
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From sakect at gmail.com  Fri Jul 23 17:35:13 2010
From: sakect at gmail.com (POWERBALL ONLINE)
Date: Sat, 24 Jul 2010 00:35:13 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <0FE1B519D219B64A87E99E362023854505172482F1@svr-e2k7.ag-it.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
	<AANLkTikdqYmBW3xDvraban3CIdexrZMqtUQtKIbXNtCO@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854505172482F1@svr-e2k7.ag-it.com>
Message-ID: <AANLkTi=Wj4=2m+ufnNAMiOUxh8WqnbCDZY_ySK5dNL2o@mail.gmail.com>

I though that your configuration is wrong because when you run clustat it
should display quorum disk is online too.
If your configuration wrong and quorum not online it will have the error
that you meet.
Please give me the file /etc/cluster/cluster.conf to me too.



On Fri, Jul 23, 2010 at 10:18 PM, Wahyu Darmawan
<Wahyu.Darmawan at ag-it.com>wrote:

>  Hi,
>
> Thanks for your response.
>
> Here is the info on below.
>
>
>
>
>
> T0171
>
> [root at t0171 ~]# clustat
>
> Member Status: Quorate
>
>
>
>   Member Name                              Status
>
>   ------ ----                              ------
>
>   t0171.example.com                    Online, Local, rgmanager
>
>   t0175.example.com                    Online, rgmanager
>
>
>
>   Service Name         Owner (Last)                   State
>
>   ------- ----         ----- ------                   -----
>
>   IP_Virtual           t0175.example.com          started
>
>   GFS_Mounting         t0175.example.com          started
>
>   Listener_Service     t0175.example.com          started
>
>   Oracle_DBahmcm       t0175.example.com          started
>
>   Oracle_DBportal      t0175.example.com          started
>
>   Oracle_DBcard5       t0175.example.com          started
>
>   Oracle_DBhrdtxn      t0175.example.com          started
>
>   Oracle_DBrepwm       t0175.example.com          started
>
> [root at t0171 ~]# mkqdisk -L
>
> mkqdisk v0.5.2
>
> /dev/sdi1:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/sdi2:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-18:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-19:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
>
>
> T0175
>
> [root at t0175 ~]# clustat
>
> Member Status: Quorate
>
>
>
>   Member Name                              Status
>
>   ------ ----                              ------
>
>   t0171.example.com                    Online, rgmanager
>
>   t0175.example.com                    Online, Local, rgmanager
>
>
>
>   Service Name         Owner (Last)                   State
>
>   ------- ----         ----- ------                   -----
>
>   IP_Virtual           t0175. example.com          started
>
>   GFS_Mounting         t0175. example.com          started
>
>   Listener_Service     t0175. example.com          started
>
>   Oracle_DBahmcm       t0175. example.com          started
>
>   Oracle_DBportal      t0175. example.com          started
>
>   Oracle_DBcard5       t0175. example.com          started
>
>   Oracle_DBhrdtxn      t0175. example.com          started
>
>   Oracle_DBrepwm       t0175. example.com          started
>
> [root at t0175 ~]# mkqdisk -L
>
> mkqdisk v0.5.2
>
> /dev/sdi1:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171. example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/sdi2:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175. example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-19:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171. example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-20:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
>
>
> Thanks & Regards,
>
> WD
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *POWERBALL ONLINE
> *Sent:* Friday, July 23, 2010 12:29 AM
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Please give me the information after you run mkqdisk -L and clustat -i
> command
>
> On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi,
>
> So, what your suggestion?
>
>
>
> Thanks,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *Wahyu Darmawan
> *Sent:* Thursday, July 22, 2010 11:02 PM
>
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Yes correct. I use Conga for my clusterware.
>
>
>
> Thanks,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *POWERBALL ONLINE
> *Sent:* Thursday, July 22, 2010 10:54 PM
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Hi
>
>
>
> What tool you use for create cluster? Is it Conga ?
>
>
>
>
>
> On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi all,
>
> I have problem with my cluster. I have 2 nodes of my cluster.
>
> On July 12 the nodes missed  connection from storage and lost 1 disk
> partition on both nodes. Then I decided to assigned a new of partition from
> storage, and use it on my nodes.
>
> But until today, the quorum is not going to quorate.
>
> FYI, I use GFS for my filesystem.
>
> So, what do I have to do to make quorum disk back to quorate status?
>
>
>
> I appreciate for your answer.
>
>
>
> Thanks,
>
> Wahyu
>
>
>
> Note, I attached the cluster log messages below:
>
> Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is
> disabled
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service
> Oracle_DBrepwm
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is
> disabled
>
> Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBahmcm
>
> Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm
> started
>
> Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBportal
>
> Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal
> started
>
> Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBcard5
>
> Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5
> started
>
> Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBhrdtxn
>
> Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn
> started
>
> Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBrepwm
>
> Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm
> started
>
> Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.120000)
>
> Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.070000)
>
> Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.310000)
>
> Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.400000)
>
> Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.690000)
>
> Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.620000)
>
> Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.020000)
>
> Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.550000)
>
> Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.050000)
>
> Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.710000)
>
> Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:05:30 t0175 last message repeated 2 times
>
> Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.530000)
>
> Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.200000)
>
> Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.820000)
>
> Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.420000)
>
> Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.820000)
>
> Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.540000)
>
> Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.740000)
>
> Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.020000)
>
> Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.900000)
>
> Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.270000)
>
> Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.980000)
>
> Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.470000)
>
> Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.870000)
>
> Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.940000)
>
> Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.780000)
>
> Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.360000)
>
> Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.790000)
>
> Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.730000)
>
> Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.100000)
>
> Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.580000)
>
> Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.460000)
>
> Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.790000)
>
> Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.240000)
>
> Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.670000)
>
> Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.570000)
>
> Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.000000)
>
> Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.670000)
>
> Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 18 02:04:31 t0175 last message repeated 3 times
>
> Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.870000)
>
> Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.920000)
>
> Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.040000)
>
> Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.350000)
>
> Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.450000)
>
> Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.450000)
>
> Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.120000)
>
> Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.480000)
>
> Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.960000)
>
> Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.030000)
>
> Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.940000)
>
> Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.560000)
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100724/2740389a/attachment.htm>

From gordan at bobich.net  Fri Jul 23 17:40:50 2010
From: gordan at bobich.net (Gordan Bobic)
Date: Fri, 23 Jul 2010 18:40:50 +0100
Subject: [Linux-cluster] GFS performance issue on CentOS 5.5
In-Reply-To: <4C49D28A.5020704@twister.dyndns.org>
References: <4C4869FA.8020006@twister.dyndns.org>	<1279898459.2465.39.camel@localhost>
	<4C49D28A.5020704@twister.dyndns.org>
Message-ID: <4C49D422.1020805@bobich.net>

Fred Wittekind wrote:
>  I do have noatime, need to add nodiratime.

noatime is a superset of nodiratime.

> There are PHP session files
> on the gfs2 volume.  There used to be Zend Cache file on it, but I moved
> those to tmpfs.  There is one other set of cache files that where put in
> a bad directory, so they are hard to move to tmpfs without changing the
> client's website code (although I may do this anyways).

Indeed you may, that sounds like a good place to start looking for your 
performance problems.

> I had considered moving the PHP session files to a memcache server, but
> then I observed very long load times on static website content
> (images).  Since images don't involve the session files, I had decided
> that it looked like there was more of a problem than the session files. 
> Although, could lock issues with the session files effect load times of
> files in a completely different area of the directory tree?

Session files will be accessed for ever page hit, will they not?

Gordan



From rom at twister.dyndns.org  Fri Jul 23 17:52:32 2010
From: rom at twister.dyndns.org (Fred Wittekind)
Date: Fri, 23 Jul 2010 13:52:32 -0400
Subject: [Linux-cluster] GFS performance issue on CentOS 5.5
In-Reply-To: <1279898149.2749.9.camel@thinkpad>
References: <4C4869FA.8020006@twister.dyndns.org>
	<1279898149.2749.9.camel@thinkpad>
Message-ID: <4C49D6E0.8040103@twister.dyndns.org>

 Does manually tuning glock trimming still apply to GFS2?

I have seen references that have talked about increasing resource
groups, and also resources saying that if increased to high it could
hurt performance.  From what I can figure out looks like the only way to
change this is a reformat, is this correct?  I am not opposed to doing
this, as this cluster is not yet in production, although, reformating,
and restoring all the content to the file system will take a pretty
decent amount of time, so I don't really want to do it to many times. 
Would increasing the resource groups be a worth while thing to try, and
if so, is there any good way to figure out (short of trial and error)
what value I should use for the resource group allocation?

I found this tread from a search using a search links I found in one of
the threads you gave me a link to:
http://www.mail-archive.com/linux-cluster at redhat.com/msg00990.html

On 7/23/2010 11:15 AM, Gordan Bobic wrote:
> Sounds like you are suffering from extreme lock bouncing between the
> nodes. This is a FAQ. I suggest you have a read through the mail
> archives of this list for similar discussions, e.g.:
>
> http://www.mail-archive.com/linux-cluster at redhat.com/msg04412.html
> http://www.mail-archive.com/linux-cluster at redhat.com/msg08177.html
>
> Gordan
>
> On Thu, 2010-07-22 at 11:55 -0400, Fred Wittekind wrote:
>> Two web servers, both virtualized with CentOS Xen servers as host
>> (residing on two different physical servers). 
>> GFS used to store home directories containing web document roots.
>>
>> Shared block device used by GFS is an ISCSI target with the ISCSI
>> initiator residing on the Dom-0, and presented to Dom-U webservers as
>> drives.
>> Also, providing a second shared block device for quorum disk.
>>
>> If I hit the web site on just one of the nodes, it behaves as expected. 
>> If I try to load the websites from both nodes at the same time (two web
>> browser instances), the load average on both nodes spikes, and page
>> loads very slowly.  The site I am trying to host is very high traffic,
>> and if the servers can be nearly brought to there knees with two web
>> browser instances running on a single workstation, that's not going to work.
>>
>> I am not seeing any error messages in the logs regarding the cluster.
>>
>> Any help or suggestions on how to troubleshoot this issue would be
>> greatly appreciated.
>>
>>
>> [root at www3 www]# find ./ | wc -l
>> 64815
>>
>> [root at www3 www]# gfs2_tool df /home/www
>> /home/www:
>>   SB lock proto = "lock_dlm"
>>   SB lock table = "Web:homewww"
>>   SB ondisk format = 1801
>>   SB multihost format = 1900
>>   Block size = 4096
>>   Journals = 2
>>   Resource Groups = 316
>>   Mounted lock proto = "lock_dlm"
>>   Mounted lock table = "Web:homewww"
>>   Mounted host data = "jid=1:id=393217:first=0"
>>   Journal number = 1
>>   Lock module flags = 0
>>   Local flocks = FALSE
>>   Local caching = FALSE
>>
>>   Type           Total Blocks   Used Blocks    Free Blocks    use%
>>   ------------------------------------------------------------------------
>>   data           20707148       4347237        16359911       21%
>>   inodes         16426386       66475          16359911       0%
>>
>> [root at www3 www]# gfs2_tool gettune /home/www
>> new_files_directio = 0
>> new_files_jdata = 0
>> quota_scale = 1.0000   (1, 1)
>> logd_secs = 1
>> recoverd_secs = 60
>> statfs_quantum = 30
>> stall_secs = 600
>> quota_cache_secs = 300
>> quota_simul_sync = 64
>> statfs_slow = 0
>> complain_secs = 10
>> max_readahead = 262144
>> quota_quantum = 60
>> quota_warn_period = 10
>> jindex_refresh_secs = 60
>> log_flush_secs = 60
>> incore_log_blocks = 1024
>>
>> [root at www3 www]# cat /etc/cluster/cluster.conf |egrep '(dlm)|(gfs)'
>>         <dlm plock_ownership="1" plock_rate_limit="0"/>
>>         <gfs_controld plock_rate_limit="0"/>
>>
>>
>> Fred Wittekind
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From rom at twister.dyndns.org  Fri Jul 23 17:56:43 2010
From: rom at twister.dyndns.org (Fred Wittekind)
Date: Fri, 23 Jul 2010 13:56:43 -0400
Subject: [Linux-cluster] GFS performance issue on CentOS 5.5
In-Reply-To: <4C49D422.1020805@bobich.net>
References: <4C4869FA.8020006@twister.dyndns.org>	<1279898459.2465.39.camel@localhost>	<4C49D28A.5020704@twister.dyndns.org>
	<4C49D422.1020805@bobich.net>
Message-ID: <4C49D7DB.9050707@twister.dyndns.org>

 On 7/23/2010 1:40 PM, Gordan Bobic wrote:
> Fred Wittekind wrote:
>>  I do have noatime, need to add nodiratime.
>
> noatime is a superset of nodiratime.
Thanks, that's good to know.
>
>> There are PHP session files
>> on the gfs2 volume.  There used to be Zend Cache file on it, but I moved
>> those to tmpfs.  There is one other set of cache files that where put in
>> a bad directory, so they are hard to move to tmpfs without changing the
>> client's website code (although I may do this anyways).
>
> Indeed you may, that sounds like a good place to start looking for
> your performance problems.
I will try this.
>
>> I had considered moving the PHP session files to a memcache server, but
>> then I observed very long load times on static website content
>> (images).  Since images don't involve the session files, I had decided
>> that it looked like there was more of a problem than the session
>> files. Although, could lock issues with the session files effect load
>> times of
>> files in a completely different area of the directory tree?
>
> Session files will be accessed for ever page hit, will they not?
No.  PHP session files are only hit when accessing a PHP page that does
a session_start call.  So, if for instance you load a jpg file (that is
not served by a php script), there is no session file access.
>
> Gordan
>
> -- 
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From eschneid at uccs.edu  Fri Jul 23 22:20:32 2010
From: eschneid at uccs.edu (Eric Schneider)
Date: Fri, 23 Jul 2010 16:20:32 -0600
Subject: [Linux-cluster] Cluster services die when nonactive node is rebooted
Message-ID: <00e901cb2ab5$43bedc60$cb3c9520$@uccs.edu>

I have a few 2 node clusters and I notice that recently the clusters lose
quorum when I reboot the node without running services.  I could do this in
the past without any problems.  CentOS 5.5 on ESX 4.0 u1.  Maybe a bug with
a new kernel or cman software?

 

I get the following right away when the node reboots:

Jul 23 16:02:32 happy5 clurgmgrd[4269]: <notice> Member 2 shutting down

Jul 23 16:02:52 happy5 qdiskd[3562]: <info> Node 2 shutdown

Jul 23 16:03:02 happy5 qdiskd[3562]: <info> Assuming master role

Jul 23 16:03:03 happy5 clurgmgrd[4269]: <emerg> #1: Quorum Dissolved

Jul 23 16:03:03 happy5 openais[3533]: [CMAN ] lost contact with quorum
device

Jul 23 16:03:03 happy5 openais[3533]: [CMAN ] quorum lost, blocking activity

Jul 23 16:03:03 happy5 ccsd[3493]: Cluster is not quorate.  Refusing
connection.

Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing connect:
Connection refused

Jul 23 16:03:03 happy5 ccsd[3493]: Cluster is not quorate.  Refusing
connection.

Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing connect:
Connection refused

Jul 23 16:03:03 happy5 ccsd[3493]: Invalid descriptor specified (-111).

Jul 23 16:03:03 happy5 ccsd[3493]: Someone may be attempting something evil.

Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing get: Invalid
request descriptor

Jul 23 16:03:03 happy5 ccsd[3493]: Invalid descriptor specified (-111).

Jul 23 16:03:03 happy5 ccsd[3493]: Someone may be attempting something evil.

Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing get: Invalid
request descriptor

 

<?xml version="1.0"?>

<cluster alias="delta_cluster" config_version="40" name="delta_cluster">

        <fence_daemon post_fail_delay="5" post_join_delay="120"/>

        <quorumd interval="5" label="delta_qdisk" min_score="1" tko="6"
votes="1">

                <heuristic interval="5" program="ping -t1 -c1 192.168.1.1"
score="1"/>

        </quorumd>

        <clusternodes>

                <clusternode name="node1" nodeid="1" votes="1">

                        <fence>

                                <method name="1">

                                        <device name="node1"/>

                                </method>

                        </fence>

                </clusternode>

                <clusternode name="node2" nodeid="2" votes="1">

                        <fence>

                                <method name="1">

                                        <device name="node2"/>

                                </method>

                        </fence>

                </clusternode>

        </clusternodes>

        <cman expected_votes="3">

                <multicast addr="224.0.0.1" interface="eth0"/>

        </cman>

        <fencedevices>

                <fencedevice agent="fence_manual" name="fence_manual"/>

                <fencedevice agent="fence_vmware" ipaddr="bob"
login="username" name="node1" passwd="password" port="node1"/>

                <fencedevice agent="fence_vmware" ipaddr="bob"
login="username" name="node2" passwd="password" port="node2"/>

        </fencedevices>

        <rm>

                <failoverdomains>

                        <failoverdomain name="node1" ordered="0"
restricted="1">

                                <failoverdomainnode name="node1"
priority="1"/>

                        </failoverdomain>

                        <failoverdomain name="node2" restricted="1">

                                <failoverdomainnode name="node2"
priority="1"/>

                        </failoverdomain>

                        <failoverdomain name="failover_pro-http"
restricted="0">

                                <failoverdomainnode name="node1"
priority="1"/>

                                <failoverdomainnode name="node2"
priority="1"/>

                        </failoverdomain>

                </failoverdomains>

             

        </rm>

        <totem token="21000"/>

</cluster>

 

Thanks,

 

Eric 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100723/c1475d75/attachment.htm>

From susvirkar.3616 at gmail.com  Sun Jul 25 06:17:52 2010
From: susvirkar.3616 at gmail.com (umesh susvirkar)
Date: Sun, 25 Jul 2010 11:47:52 +0530
Subject: [Linux-cluster] Cluster services die when nonactive node is
	rebooted
In-Reply-To: <00e901cb2ab5$43bedc60$cb3c9520$@uccs.edu>
References: <00e901cb2ab5$43bedc60$cb3c9520$@uccs.edu>
Message-ID: <AANLkTin1Ph3LB=qpEgs6qnhKKBQR5bGNu+SGfOk7HnA3@mail.gmail.com>

Try to set following in you cluster.conf file

<cman expected_votes="3" quorum_dev_poll="35000" >

                <multicast addr="224.0.0.1" interface="eth0"/>

        </cman>

---
cal for
quorum_dev_poll > (interval * tko )

as per below 5*6 = 30 so 35
<quorumd i*nterval="5"* label="delta_qdisk" min_score="1" *tko="6"*votes="1">

                <heuristic interval="5" program="ping -t1 -c1 192.168.1.1"
score="1"/>

        </quorumd>


for more info read following doc
https://access.redhat.com/kb/docs/DOC-2882
http://people.redhat.com/lhh/cmanvsqdisk.png


On Sat, Jul 24, 2010 at 3:50 AM, Eric Schneider <eschneid at uccs.edu> wrote:

> I have a few 2 node clusters and I notice that recently the clusters lose
> quorum when I reboot the node without running services.  I could do this in
> the past without any problems.  CentOS 5.5 on ESX 4.0 u1.  Maybe a bug with
> a new kernel or cman software?
>
>
>
> I get the following right away when the node reboots:
>
> Jul 23 16:02:32 happy5 clurgmgrd[4269]: <notice> Member 2 shutting down
>
> Jul 23 16:02:52 happy5 qdiskd[3562]: <info> Node 2 shutdown
>
> Jul 23 16:03:02 happy5 qdiskd[3562]: <info> Assuming master role
>
> Jul 23 16:03:03 happy5 clurgmgrd[4269]: <emerg> #1: Quorum Dissolved
>
> Jul 23 16:03:03 happy5 openais[3533]: [CMAN ] lost contact with quorum
> device
>
> Jul 23 16:03:03 happy5 openais[3533]: [CMAN ] quorum lost, blocking
> activity
>
> Jul 23 16:03:03 happy5 ccsd[3493]: Cluster is not quorate.  Refusing
> connection.
>
> Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing connect:
> Connection refused
>
> Jul 23 16:03:03 happy5 ccsd[3493]: Cluster is not quorate.  Refusing
> connection.
>
> Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing connect:
> Connection refused
>
> Jul 23 16:03:03 happy5 ccsd[3493]: Invalid descriptor specified (-111).
>
> Jul 23 16:03:03 happy5 ccsd[3493]: Someone may be attempting something
> evil.
>
> Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing get: Invalid
> request descriptor
>
> Jul 23 16:03:03 happy5 ccsd[3493]: Invalid descriptor specified (-111).
>
> Jul 23 16:03:03 happy5 ccsd[3493]: Someone may be attempting something
> evil.
>
> Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing get: Invalid
> request descriptor
>
>
>
> <?xml version="1.0"?>
>
> <cluster alias="delta_cluster" config_version="40" name="delta_cluster">
>
>         <fence_daemon post_fail_delay="5" post_join_delay="120"/>
>
>         <quorumd interval="5" label="delta_qdisk" min_score="1" tko="6"
> votes="1">
>
>                 <heuristic interval="5" program="ping -t1 -c1 192.168.1.1"
> score="1"/>
>
>         </quorumd>
>
>         <clusternodes>
>
>                 <clusternode name="node1" nodeid="1" votes="1">
>
>                         <fence>
>
>                                 <method name="1">
>
>                                         <device name="node1"/>
>
>                                 </method>
>
>                         </fence>
>
>                 </clusternode>
>
>                 <clusternode name="node2" nodeid="2" votes="1">
>
>                         <fence>
>
>                                 <method name="1">
>
>                                         <device name="node2"/>
>
>                                 </method>
>
>                         </fence>
>
>                 </clusternode>
>
>         </clusternodes>
>
>         <cman expected_votes="3">
>
>                 <multicast addr="224.0.0.1" interface="eth0"/>
>
>         </cman>
>
>         <fencedevices>
>
>                 <fencedevice agent="fence_manual" name="fence_manual"/>
>
>                 <fencedevice agent="fence_vmware" ipaddr="bob"
> login="username" name="node1" passwd="password" port="node1"/>
>
>                 <fencedevice agent="fence_vmware" ipaddr="bob"
> login="username" name="node2" passwd="password" port="node2"/>
>
>         </fencedevices>
>
>         <rm>
>
>                 <failoverdomains>
>
>                         <failoverdomain name="node1" ordered="0"
> restricted="1">
>
>                                 <failoverdomainnode name="node1"
> priority="1"/>
>
>                         </failoverdomain>
>
>                         <failoverdomain name="node2" restricted="1">
>
>                                 <failoverdomainnode name="node2"
> priority="1"/>
>
>                         </failoverdomain>
>
>                         <failoverdomain name="failover_pro-http"
> restricted="0">
>
>                                 <failoverdomainnode name="node1"
> priority="1"/>
>
>                                 <failoverdomainnode name="node2"
> priority="1"/>
>
>                         </failoverdomain>
>
>                 </failoverdomains>
>
>
>
>         </rm>
>
>         <totem token="21000"/>
>
> </cluster>
>
>
>
> Thanks,
>
>
>
> Eric
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100725/d19bdc6b/attachment.htm>

From swhiteho at redhat.com  Mon Jul 26 09:23:26 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Mon, 26 Jul 2010 10:23:26 +0100
Subject: [Linux-cluster] GFS performance issue on CentOS 5.5
In-Reply-To: <4C49D6E0.8040103@twister.dyndns.org>
References: <4C4869FA.8020006@twister.dyndns.org>
	<1279898149.2749.9.camel@thinkpad>
	<4C49D6E0.8040103@twister.dyndns.org>
Message-ID: <1280136206.2513.15.camel@localhost>

Hi,

On Fri, 2010-07-23 at 13:52 -0400, Fred Wittekind wrote:
> Does manually tuning glock trimming still apply to GFS2?
> 
No, it is done automatically according to memory pressure on an LRU
basis.

> I have seen references that have talked about increasing resource
> groups, and also resources saying that if increased to high it could
> hurt performance.  From what I can figure out looks like the only way to
> change this is a reformat, is this correct?  I am not opposed to doing
> this, as this cluster is not yet in production, although, reformating,
> and restoring all the content to the file system will take a pretty
> decent amount of time, so I don't really want to do it to many times. 
> Would increasing the resource groups be a worth while thing to try, and
> if so, is there any good way to figure out (short of trial and error)
> what value I should use for the resource group allocation?
> 
You can, depending on workload, get better performance by altering the
balance between resource group size and the number of resource groups.
This usually takes some experimentation due to differences in hardware &
workload in any particular situation. As you say, it does require a
reformat, but mkfs.gfs2 usually isn't too far wrong so the defaults are
usually reasonably good anyway.

> I found this tread from a search using a search links I found in one of
> the threads you gave me a link to:
> http://www.mail-archive.com/linux-cluster at redhat.com/msg00990.html
> 
This doesn't really apply to gfs2. That deamon is gone and the cpu time
used is much reduced as a consequence,

Steve.

> On 7/23/2010 11:15 AM, Gordan Bobic wrote:
> > Sounds like you are suffering from extreme lock bouncing between the
> > nodes. This is a FAQ. I suggest you have a read through the mail
> > archives of this list for similar discussions, e.g.:
> >
> > http://www.mail-archive.com/linux-cluster at redhat.com/msg04412.html
> > http://www.mail-archive.com/linux-cluster at redhat.com/msg08177.html
> >
> > Gordan
> >
> > On Thu, 2010-07-22 at 11:55 -0400, Fred Wittekind wrote:
> >> Two web servers, both virtualized with CentOS Xen servers as host
> >> (residing on two different physical servers). 
> >> GFS used to store home directories containing web document roots.
> >>
> >> Shared block device used by GFS is an ISCSI target with the ISCSI
> >> initiator residing on the Dom-0, and presented to Dom-U webservers as
> >> drives.
> >> Also, providing a second shared block device for quorum disk.
> >>
> >> If I hit the web site on just one of the nodes, it behaves as expected. 
> >> If I try to load the websites from both nodes at the same time (two web
> >> browser instances), the load average on both nodes spikes, and page
> >> loads very slowly.  The site I am trying to host is very high traffic,
> >> and if the servers can be nearly brought to there knees with two web
> >> browser instances running on a single workstation, that's not going to work.
> >>
> >> I am not seeing any error messages in the logs regarding the cluster.
> >>
> >> Any help or suggestions on how to troubleshoot this issue would be
> >> greatly appreciated.
> >>
> >>
> >> [root at www3 www]# find ./ | wc -l
> >> 64815
> >>
> >> [root at www3 www]# gfs2_tool df /home/www
> >> /home/www:
> >>   SB lock proto = "lock_dlm"
> >>   SB lock table = "Web:homewww"
> >>   SB ondisk format = 1801
> >>   SB multihost format = 1900
> >>   Block size = 4096
> >>   Journals = 2
> >>   Resource Groups = 316
> >>   Mounted lock proto = "lock_dlm"
> >>   Mounted lock table = "Web:homewww"
> >>   Mounted host data = "jid=1:id=393217:first=0"
> >>   Journal number = 1
> >>   Lock module flags = 0
> >>   Local flocks = FALSE
> >>   Local caching = FALSE
> >>
> >>   Type           Total Blocks   Used Blocks    Free Blocks    use%
> >>   ------------------------------------------------------------------------
> >>   data           20707148       4347237        16359911       21%
> >>   inodes         16426386       66475          16359911       0%
> >>
> >> [root at www3 www]# gfs2_tool gettune /home/www
> >> new_files_directio = 0
> >> new_files_jdata = 0
> >> quota_scale = 1.0000   (1, 1)
> >> logd_secs = 1
> >> recoverd_secs = 60
> >> statfs_quantum = 30
> >> stall_secs = 600
> >> quota_cache_secs = 300
> >> quota_simul_sync = 64
> >> statfs_slow = 0
> >> complain_secs = 10
> >> max_readahead = 262144
> >> quota_quantum = 60
> >> quota_warn_period = 10
> >> jindex_refresh_secs = 60
> >> log_flush_secs = 60
> >> incore_log_blocks = 1024
> >>
> >> [root at www3 www]# cat /etc/cluster/cluster.conf |egrep '(dlm)|(gfs)'
> >>         <dlm plock_ownership="1" plock_rate_limit="0"/>
> >>         <gfs_controld plock_rate_limit="0"/>
> >>
> >>
> >> Fred Wittekind
> >>
> >> --
> >> Linux-cluster mailing list
> >> Linux-cluster at redhat.com
> >> https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From pengyu.ut at gmail.com  Mon Jul 26 19:40:34 2010
From: pengyu.ut at gmail.com (Peng Yu)
Date: Mon, 26 Jul 2010 14:40:34 -0500
Subject: [Linux-cluster] Recommendation of SAN devices
In-Reply-To: <AANLkTinSoYMPyJVatsrOycXoevreXhxoKBiYXzC4XhWH@mail.gmail.com>
References: <AANLkTinSoYMPyJVatsrOycXoevreXhxoKBiYXzC4XhWH@mail.gmail.com>
Message-ID: <AANLkTimgUQnpE2q+E0Dh+1AHLbyhvOzkeYbDs-nSuEQX@mail.gmail.com>

Hi,

Essentially, I want to build a cluster where multiple machines
(multi-core) access the same cluster file system (say using GFS). Each
machine might
concurrently access the same file system for write or read. The
processes running on these machines are I/O intensity. I think that if
each machine could use hundreds of MB/s I/O. I want the whole system
be scalable in the sense that
more machines can be added without making the I/O the bottleneck.

The SRX model in the following webpage is a choice. However, I'd like
to see all the possible choices from different vendors so that I can
make a more informative decision. Could you please share me some
insights that you have or point me to a mailing list where I might get
some answers?

http://www.coraid.com/products/index.html

--
Regards,
Peng



From arunkp1987 at gmail.com  Mon Jul 26 19:56:29 2010
From: arunkp1987 at gmail.com (Arun Kp)
Date: Tue, 27 Jul 2010 01:26:29 +0530
Subject: [Linux-cluster] Postgres-cluster with shared storage.
Message-ID: <AANLkTi=jaVWP_Qb0tFOCE2_i51dQEzjTDAh66BVzot1x@mail.gmail.com>

Dear ALL,

We have tried postgresql High availability cluster using Redhat Cluster Suit
and successfully completed.But we are facing small issue that is,

When the cluster try to relocate (Stopping in the Active node and starting
in the Passive node) the Postgresql service  works fine if there is no
client connection .But when one client is connected it will take 5 minutes
for relocating.Here we have given 5 minutes as shutdown wait time in
cluster.Some times relocating getting failed due to stopping of postgresql
server taking too much time
We have given the following options for postgresql in the cluster

Name :DB-igvtdc
Configdir:/Storage   where our data directory resides(In SAN).
Postmaster options : -p 5433 -h 172.31.1.20 -m
shutdown wait time :300 s

So kindly help me to resolve this problem.


Thank you
Regards
Arun K P
HCL Infosystems Ltd
Kolkata 26
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100727/f2c0e6ac/attachment.htm>

From gordan at bobich.net  Mon Jul 26 21:25:19 2010
From: gordan at bobich.net (Gordan Bobic)
Date: Mon, 26 Jul 2010 22:25:19 +0100
Subject: [Linux-cluster] Postgres-cluster with shared storage.
In-Reply-To: <AANLkTi=jaVWP_Qb0tFOCE2_i51dQEzjTDAh66BVzot1x@mail.gmail.com>
References: <AANLkTi=jaVWP_Qb0tFOCE2_i51dQEzjTDAh66BVzot1x@mail.gmail.com>
Message-ID: <4C4DFD3F.5080502@bobich.net>

I suspect you'll find this is due to postgresql taking a long time to 
stop. Nothing to do with RHCS as such.

If you issue "/etc/init.d/postgresql stop" does that take a long time to 
return? If you forcefully power off the node, does the fail-over happen 
without delay? If so, try asking on the postgresql mailing list how to 
make it shut down quickly.

Gordan

Arun Kp wrote:
> Dear ALL,
>  
> We have tried postgresql High availability cluster using Redhat Cluster 
> Suit and successfully completed.But we are facing small issue that is,
> 
> When the cluster try to relocate (Stopping in the Active node and 
> starting in the Passive node) the Postgresql service  works fine if 
> there is no client connection .But when one client is connected it will 
> take 5 minutes for relocating.Here we have given 5 minutes as shutdown 
> wait time in cluster.Some times relocating getting failed due to 
> stopping of postgresql server taking too much time    
> We have given the following options for postgresql in the cluster
> 
> Name :DB-igvtdc
> Configdir:/Storage   where our data directory resides(In SAN).
> Postmaster options : -p 5433 -h 172.31.1.20 -m
> shutdown wait time :300 s
> 
> So kindly help me to resolve this problem.  
>  
> 
> Thank you
> Regards
> Arun K P
> HCL Infosystems Ltd
> Kolkata 26
> 
> 
> ------------------------------------------------------------------------
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From scooter at cgl.ucsf.edu  Mon Jul 26 23:54:59 2010
From: scooter at cgl.ucsf.edu (Scooter Morris)
Date: Mon, 26 Jul 2010 16:54:59 -0700
Subject: [Linux-cluster] Clearing a glock
Message-ID: <4C4E2053.7050404@cgl.ucsf.edu>

  We've got two nodes of a three node gfs2 cluster that seem to be in 
some sort of deadlock.  We're seeing a number of gfs2-related stack 
traces in dmesg:

INFO: task igtcpython.sh:22945 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
igtcpython.sh D ffff81011cabd218     0 22945  22874                     
(NOTLB)
  ffff8101f9fedcc8 0000000000000086 ffff8101f9fede38 ffffffff8000a604
  ffff8102b257f778 0000000000000006 ffff8105819d37a0 ffff8107a7f93820
  0004084c69951298 00000000000365c6 ffff8105819d3988 000000042d07a810
Call Trace:
  [<ffffffff8000a604>] __link_path_walk+0xdf8/0xf42
  [<ffffffff8002c9e4>] mntput_no_expire+0x19/0x89
  [<ffffffff8000ea46>] link_path_walk+0xa6/0xb2
  [<ffffffff887d6ee7>] :gfs2:just_schedule+0x0/0xe
  [<ffffffff887d6ef0>] :gfs2:just_schedule+0x9/0xe
  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
  [<ffffffff887d6ee7>] :gfs2:just_schedule+0x0/0xe
  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
  [<ffffffff800a0aec>] wake_bit_function+0x0/0x23
  [<ffffffff887d6ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
  [<ffffffff887e679e>] :gfs2:gfs2_permission+0x83/0xd5
  [<ffffffff887e6796>] :gfs2:gfs2_permission+0x7b/0xd5
  [<ffffffff8000d918>] permission+0x81/0xc8
  [<ffffffff8003c0c0>] open_exec+0x60/0xc0
  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
  [<ffffffff8003ed4d>] do_execve+0x46/0x1f7
  [<ffffffff8005516d>] sys_execve+0x36/0x4c
  [<ffffffff8005d4d3>] stub_execve+0x67/0xb0

and gfs2_hangalyzer is saying:

./gfs2_hangalyzer -n wilkins-pi -a
wilkins-pi: UsrLocal: G:  s:UN n:2/a5b67f f:l t:SH d:EX/0 l:0 a:0 r:58
wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:21810 [python] 
gfs2_readpage+0x61/0x199 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:21809 [python] 
gfs2_readpage+0x61/0x199 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:1897 [python] 
gfs2_readpage+0x61/0x199 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:6307 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:10436 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:11000 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:11003 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:12140 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21499 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:32601 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:653 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:10078 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:14436 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:7500 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:22815 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26056 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26062 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26122 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26124 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26128 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31825 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:32125 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:2441 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:2444 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21792 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21793 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21794 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31941 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31970 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8584 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8590 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8821 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8822 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9487 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9488 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9489 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9490 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18878 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:325 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:3139 [ls] 
gfs2_getattr+0x7d/0xc4 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:3144 [ls] 
gfs2_getattr+0x7d/0xc4 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:7450 [ls] 
gfs2_getattr+0x7d/0xc4 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31741 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:4982 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18258 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18262 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18263 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18265 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18269 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20039 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20042 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20043 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20044 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20046 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20763 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
                         lkb_id N RemoteID  pid exflg lkbflgs stat gr 
rq    waiting n ln             resource name
wilkins-pi: UsrLocal:  26513a1 3  1385218 21810     0       0 wait -1  
3          0 3 24 "       2          a5b67f"



There is 1 glock with waiters.
wilkins-pi.compbio.ucsf.edu, pid 21810 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 21809 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 1897 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 6307 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 10436 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 11000 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 11003 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 12140 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 21499 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 32601 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 653 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 10078 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 14436 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 7500 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 22815 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 26056 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 26062 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 26122 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 26124 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 26128 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 31825 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 32125 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 2441 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 2444 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 21792 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 21793 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 21794 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 31941 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 31970 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 8584 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 8590 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 8821 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 8822 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 9487 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 9488 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 9489 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 9490 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18878 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 325 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 3139 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 3144 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 7450 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 31741 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 4982 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18258 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18262 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18263 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18265 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18269 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20039 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20042 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20043 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20044 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20046 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20763 is waiting for glock 2/a5b67f, 
but no holder was found.
          The dlm has granted lkb "       2          a5b67f" to pid 
391344724

Clearly, I've got a hung lock of some sort.  Is there any way to clear 
the glock to free up all of these processes?  I really hate to reboot 
the cluster to clear this up since it's only effecting one pipeline....

Thanks in advance!

-- scooter



From mpartio at gmail.com  Tue Jul 27 09:29:30 2010
From: mpartio at gmail.com (Mikko Partio)
Date: Tue, 27 Jul 2010 12:29:30 +0300
Subject: [Linux-cluster] Postgres-cluster with shared storage.
In-Reply-To: <AANLkTi=jaVWP_Qb0tFOCE2_i51dQEzjTDAh66BVzot1x@mail.gmail.com>
References: <AANLkTi=jaVWP_Qb0tFOCE2_i51dQEzjTDAh66BVzot1x@mail.gmail.com>
Message-ID: <AANLkTimtC923x5J1izq1YgFB23Ou7ZZR+O6FU76JE1V-@mail.gmail.com>

On Mon, Jul 26, 2010 at 10:56 PM, Arun Kp <arunkp1987 at gmail.com> wrote:
>
> When the cluster try to relocate (Stopping in the Active node and starting
> in the Passive node) the Postgresql service  works fine if there is no
> client connection .But when one client is connected it will take 5 minutes
> for relocating.Here we have given 5 minutes as shutdown wait time in
> cluster.Some times relocating getting failed due to stopping of postgresql
> server taking too much time
> We have given the following options for postgresql in the cluster
>
> Name :DB-igvtdc
> Configdir:/Storage   where our data directory resides(In SAN).
> Postmaster options : -p 5433 -h 172.31.1.20 -m
> shutdown wait time :300 s
>

In your init script (/etc/init.d/postgresql or similar) check what is the
actual command that is used to shut down the database. My guess is that it's
something like

pg_ctl -D /path/to/database stop

Then just add switch "-m immediate" so that the database is not waiting for
all clients to disconnect before shutting done. More information on pg_ctl
syntax: http://www.postgresql.org/docs/8.4/static/app-pg-ctl.html

Regards

Mikko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100727/204ac09c/attachment.htm>

From swhiteho at redhat.com  Tue Jul 27 12:15:15 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Tue, 27 Jul 2010 13:15:15 +0100
Subject: [Linux-cluster] Clearing a glock
In-Reply-To: <4C4E2053.7050404@cgl.ucsf.edu>
References: <4C4E2053.7050404@cgl.ucsf.edu>
Message-ID: <1280232915.2543.5.camel@localhost>

Hi,

If you translate a5b67f into decimal, then that is the inode number of
the inode which is causing a problem. It looks to me as if you have too
many processes trying to access this one inode from multiple nodes.

Its not obvious from the traces that anything is actually stuck, but if
you take two traces, a few seconds or minutes apart, then it should
become more obvious whether the cluster is making progress or whether it
really is stuck,

Steve.




From scooter at cgl.ucsf.edu  Tue Jul 27 12:57:06 2010
From: scooter at cgl.ucsf.edu (Scooter Morris)
Date: Tue, 27 Jul 2010 05:57:06 -0700
Subject: [Linux-cluster] Clearing a glock
In-Reply-To: <1280232915.2543.5.camel@localhost>
References: <4C4E2053.7050404@cgl.ucsf.edu> <1280232915.2543.5.camel@localhost>
Message-ID: <4C4ED7A2.9030706@cgl.ucsf.edu>

  On 7/27/10 5:15 AM, Steven Whitehouse wrote:
> Hi,
>
> If you translate a5b67f into decimal, then that is the inode number of
> the inode which is causing a problem. It looks to me as if you have too
> many processes trying to access this one inode from multiple nodes.
>
> Its not obvious from the traces that anything is actually stuck, but if
> you take two traces, a few seconds or minutes apart, then it should
> become more obvious whether the cluster is making progress or whether it
> really is stuck,
>
> Steve.
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
Hi Steve,
     As always, thanks for the reply.  The cluster was, indeed, truly 
stuck.  I rebooted it last night to clear everything out.  I never did 
figure out which file was the problem.  I did a find -inum, but the find 
hung too.  By that point the load average was up to 80 and climbing.  
Any ideas on how to avoid this?  Are there tunable values I need to 
increase to allow more processes to access any individual inode?

Thanks!

-- scooter



From swhiteho at redhat.com  Tue Jul 27 13:18:50 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Tue, 27 Jul 2010 14:18:50 +0100
Subject: [Linux-cluster] Clearing a glock
In-Reply-To: <4C4ED7A2.9030706@cgl.ucsf.edu>
References: <4C4E2053.7050404@cgl.ucsf.edu>
	<1280232915.2543.5.camel@localhost>  <4C4ED7A2.9030706@cgl.ucsf.edu>
Message-ID: <1280236730.2543.14.camel@localhost>

Hi,

On Tue, 2010-07-27 at 05:57 -0700, Scooter Morris wrote:
> On 7/27/10 5:15 AM, Steven Whitehouse wrote:
> > Hi,
> >
> > If you translate a5b67f into decimal, then that is the inode number of
> > the inode which is causing a problem. It looks to me as if you have too
> > many processes trying to access this one inode from multiple nodes.
> >
> > Its not obvious from the traces that anything is actually stuck, but if
> > you take two traces, a few seconds or minutes apart, then it should
> > become more obvious whether the cluster is making progress or whether it
> > really is stuck,
> >
> > Steve.
> >
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> Hi Steve,
>      As always, thanks for the reply.  The cluster was, indeed, truly 
> stuck.  I rebooted it last night to clear everything out.  I never did 
> figure out which file was the problem.  I did a find -inum, but the find 
> hung too.  By that point the load average was up to 80 and climbing.  
> Any ideas on how to avoid this?  Are there tunable values I need to 
> increase to allow more processes to access any individual inode?
> 
The LA includes processes waiting for glocks since that is an
uninterruptible wait, so thats where most of the LA came from.

The find is unlikely to work while the cluster is stuck, since if it
does find the cuplrit inode, it is, by definition already stuck so the
find process would just join the queue. If a find fails to discover the
inode when the cluster has been rebooted and is back working again, then
it was probably a temporary file of some kind.

There are no tunable values since the limitation on the access to the
inode is the speed of the hardware in terms of how many times a given
inode can be synced, invalidated and the glock passed on to another node
in a given time period. It is a limitation of the hardware and the
architecture of the filesystem.

There are a few things which can probably be improved in due course, but
in the main the best way to avoid problems with congestion on inodes is
just to be careful about the access pattern across nodes.

That said, if it really was completely stuck, that is a real bug and not
the result of the access pattern since the code is designed such that
progress should always be made, even if its painfully slow,

Steve.




From Dave.Jones at maritz.com  Tue Jul 27 13:58:13 2010
From: Dave.Jones at maritz.com (Jones, Dave)
Date: Tue, 27 Jul 2010 08:58:13 -0500
Subject: [Linux-cluster] Clearing a glock
In-Reply-To: <4C4E2053.7050404@cgl.ucsf.edu>
References: <4C4E2053.7050404@cgl.ucsf.edu>
Message-ID: <3B29E77570056E408AB74D923453D55902C00F3E@FENEXCH1602C.us.maritz.net>



Maybe a bit off topic, but IMO Red Hat should really consider changing
the name of this process.
Thought I somehow crossed mailing lists and was about to read a post
about a jammed polymer-framed 9mm pistol. 

D

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Scooter Morris
Sent: Monday, July 26, 2010 6:55 PM
To: linux clustering
Subject: [Linux-cluster] Clearing a glock


  We've got two nodes of a three node gfs2 cluster that seem to be in 
some sort of deadlock.  We're seeing a number of gfs2-related stack 
traces in dmesg:

INFO: task igtcpython.sh:22945 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
igtcpython.sh D ffff81011cabd218     0 22945  22874                     
(NOTLB)
  ffff8101f9fedcc8 0000000000000086 ffff8101f9fede38 ffffffff8000a604
  ffff8102b257f778 0000000000000006 ffff8105819d37a0 ffff8107a7f93820
  0004084c69951298 00000000000365c6 ffff8105819d3988 000000042d07a810
Call Trace:
  [<ffffffff8000a604>] __link_path_walk+0xdf8/0xf42
  [<ffffffff8002c9e4>] mntput_no_expire+0x19/0x89
  [<ffffffff8000ea46>] link_path_walk+0xa6/0xb2
  [<ffffffff887d6ee7>] :gfs2:just_schedule+0x0/0xe
  [<ffffffff887d6ef0>] :gfs2:just_schedule+0x9/0xe
  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
  [<ffffffff887d6ee7>] :gfs2:just_schedule+0x0/0xe
  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
  [<ffffffff800a0aec>] wake_bit_function+0x0/0x23
  [<ffffffff887d6ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
  [<ffffffff887e679e>] :gfs2:gfs2_permission+0x83/0xd5
  [<ffffffff887e6796>] :gfs2:gfs2_permission+0x7b/0xd5
  [<ffffffff8000d918>] permission+0x81/0xc8
  [<ffffffff8003c0c0>] open_exec+0x60/0xc0
  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
  [<ffffffff8003ed4d>] do_execve+0x46/0x1f7
  [<ffffffff8005516d>] sys_execve+0x36/0x4c
  [<ffffffff8005d4d3>] stub_execve+0x67/0xb0

and gfs2_hangalyzer is saying:

./gfs2_hangalyzer -n wilkins-pi -a
wilkins-pi: UsrLocal: G:  s:UN n:2/a5b67f f:l t:SH d:EX/0 l:0 a:0 r:58
wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:21810 [python] 
gfs2_readpage+0x61/0x199 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:21809 [python] 
gfs2_readpage+0x61/0x199 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:1897 [python] 
gfs2_readpage+0x61/0x199 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:6307 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:10436 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:11000 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:11003 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:12140 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21499 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:32601 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:653 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:10078 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:14436 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:7500 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:22815 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26056 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26062 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26122 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26124 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26128 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31825 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:32125 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:2441 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:2444 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21792 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21793 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21794 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31941 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31970 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8584 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8590 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8821 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8822 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9487 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9488 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9489 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9490 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18878 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:325 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:3139 [ls] 
gfs2_getattr+0x7d/0xc4 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:3144 [ls] 
gfs2_getattr+0x7d/0xc4 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:7450 [ls] 
gfs2_getattr+0x7d/0xc4 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31741 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:4982 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18258 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18262 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18263 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18265 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18269 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20039 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20042 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20043 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20044 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20046 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20763 [igtcpython.sh] 
gfs2_permission+0x7b/0xd5 [gfs2]
                         lkb_id N RemoteID  pid exflg lkbflgs stat gr 
rq    waiting n ln             resource name
wilkins-pi: UsrLocal:  26513a1 3  1385218 21810     0       0 wait -1  
3          0 3 24 "       2          a5b67f"



There is 1 glock with waiters.
wilkins-pi.compbio.ucsf.edu, pid 21810 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 21809 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 1897 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 6307 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 10436 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 11000 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 11003 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 12140 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 21499 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 32601 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 653 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 10078 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 14436 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 7500 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 22815 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 26056 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 26062 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 26122 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 26124 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 26128 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 31825 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 32125 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 2441 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 2444 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 21792 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 21793 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 21794 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 31941 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 31970 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 8584 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 8590 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 8821 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 8822 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 9487 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 9488 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 9489 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 9490 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18878 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 325 is waiting for glock 2/a5b67f, but 
no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 3139 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 3144 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 7450 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 31741 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 4982 is waiting for glock 2/a5b67f, but

no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18258 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18262 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18263 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18265 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 18269 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20039 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20042 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20043 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20044 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20046 is waiting for glock 2/a5b67f, 
but no holder was found.
wilkins-pi.compbio.ucsf.edu, pid 20763 is waiting for glock 2/a5b67f, 
but no holder was found.
          The dlm has granted lkb "       2          a5b67f" to pid 
391344724

Clearly, I've got a hung lock of some sort.  Is there any way to clear 
the glock to free up all of these processes?  I really hate to reboot 
the cluster to clear this up since it's only effecting one pipeline....

Thanks in advance!

-- scooter

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

Confidentiality Warning:  This e-mail contains information intended only for the use of the individual or entity named above.  If the reader of this e-mail is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, any dissemination, publication or copying of this e-mail is strictly prohibited.  The sender does not accept any responsibility for any loss, disruption or damage to your data or computer system that may occur while using data contained in, or transmitted with, this e-mail.  
If you have received this e-mail in error, please immediately notify us by return e-mail.  Thank you.






From swhiteho at redhat.com  Tue Jul 27 14:24:41 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Tue, 27 Jul 2010 15:24:41 +0100
Subject: [Linux-cluster] Clearing a glock
In-Reply-To: <3B29E77570056E408AB74D923453D55902C00F3E@FENEXCH1602C.us.maritz.net>
References: <4C4E2053.7050404@cgl.ucsf.edu>
	<3B29E77570056E408AB74D923453D55902C00F3E@FENEXCH1602C.us.maritz.net>
Message-ID: <1280240681.2543.19.camel@localhost>

Hi,

On Tue, 2010-07-27 at 08:58 -0500, Jones, Dave wrote:
> 
> Maybe a bit off topic, but IMO Red Hat should really consider changing
> the name of this process.
> Thought I somehow crossed mailing lists and was about to read a post
> about a jammed polymer-framed 9mm pistol. 
> 
> D
> 
:-)

The name is not perhaps ideal, and its origins are somewhat lost in the
mists of time. The main issue is that if we did change it, it would make
things even more confusing at least while the transistion was in effect,

Steve.

> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Scooter Morris
> Sent: Monday, July 26, 2010 6:55 PM
> To: linux clustering
> Subject: [Linux-cluster] Clearing a glock
> 
> 
>   We've got two nodes of a three node gfs2 cluster that seem to be in 
> some sort of deadlock.  We're seeing a number of gfs2-related stack 
> traces in dmesg:
> 
> INFO: task igtcpython.sh:22945 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
> message.
> igtcpython.sh D ffff81011cabd218     0 22945  22874                     
> (NOTLB)
>   ffff8101f9fedcc8 0000000000000086 ffff8101f9fede38 ffffffff8000a604
>   ffff8102b257f778 0000000000000006 ffff8105819d37a0 ffff8107a7f93820
>   0004084c69951298 00000000000365c6 ffff8105819d3988 000000042d07a810
> Call Trace:
>   [<ffffffff8000a604>] __link_path_walk+0xdf8/0xf42
>   [<ffffffff8002c9e4>] mntput_no_expire+0x19/0x89
>   [<ffffffff8000ea46>] link_path_walk+0xa6/0xb2
>   [<ffffffff887d6ee7>] :gfs2:just_schedule+0x0/0xe
>   [<ffffffff887d6ef0>] :gfs2:just_schedule+0x9/0xe
>   [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
>   [<ffffffff887d6ee7>] :gfs2:just_schedule+0x0/0xe
>   [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
>   [<ffffffff800a0aec>] wake_bit_function+0x0/0x23
>   [<ffffffff887d6ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
>   [<ffffffff887e679e>] :gfs2:gfs2_permission+0x83/0xd5
>   [<ffffffff887e6796>] :gfs2:gfs2_permission+0x7b/0xd5
>   [<ffffffff8000d918>] permission+0x81/0xc8
>   [<ffffffff8003c0c0>] open_exec+0x60/0xc0
>   [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>   [<ffffffff8003ed4d>] do_execve+0x46/0x1f7
>   [<ffffffff8005516d>] sys_execve+0x36/0x4c
>   [<ffffffff8005d4d3>] stub_execve+0x67/0xb0
> 
> and gfs2_hangalyzer is saying:
> 
> ./gfs2_hangalyzer -n wilkins-pi -a
> wilkins-pi: UsrLocal: G:  s:UN n:2/a5b67f f:l t:SH d:EX/0 l:0 a:0 r:58
> wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:21810 [python] 
> gfs2_readpage+0x61/0x199 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:21809 [python] 
> gfs2_readpage+0x61/0x199 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:1897 [python] 
> gfs2_readpage+0x61/0x199 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:6307 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:10436 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:11000 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:11003 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:12140 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21499 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:32601 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:653 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:10078 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:14436 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:7500 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:22815 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26056 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26062 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26122 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26124 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26128 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31825 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:32125 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:2441 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:2444 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21792 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21793 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21794 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31941 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31970 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8584 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8590 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8821 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8822 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9487 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9488 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9489 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9490 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18878 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:325 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:3139 [ls] 
> gfs2_getattr+0x7d/0xc4 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:3144 [ls] 
> gfs2_getattr+0x7d/0xc4 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:7450 [ls] 
> gfs2_getattr+0x7d/0xc4 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31741 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:4982 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18258 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18262 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18263 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18265 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18269 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20039 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20042 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20043 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20044 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20046 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20763 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
>                          lkb_id N RemoteID  pid exflg lkbflgs stat gr 
> rq    waiting n ln             resource name
> wilkins-pi: UsrLocal:  26513a1 3  1385218 21810     0       0 wait -1  
> 3          0 3 24 "       2          a5b67f"
> 
> 
> 
> There is 1 glock with waiters.
> wilkins-pi.compbio.ucsf.edu, pid 21810 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 21809 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 1897 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 6307 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 10436 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 11000 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 11003 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 12140 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 21499 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 32601 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 653 is waiting for glock 2/a5b67f, but 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 10078 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 14436 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 7500 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 22815 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 26056 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 26062 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 26122 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 26124 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 26128 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 31825 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 32125 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 2441 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 2444 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 21792 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 21793 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 21794 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 31941 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 31970 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 8584 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 8590 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 8821 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 8822 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 9487 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 9488 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 9489 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 9490 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18878 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 325 is waiting for glock 2/a5b67f, but 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 3139 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 3144 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 7450 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 31741 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 4982 is waiting for glock 2/a5b67f, but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18258 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18262 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18263 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18265 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18269 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20039 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20042 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20043 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20044 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20046 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20763 is waiting for glock 2/a5b67f, 
> but no holder was found.
>           The dlm has granted lkb "       2          a5b67f" to pid 
> 391344724
> 
> Clearly, I've got a hung lock of some sort.  Is there any way to clear 
> the glock to free up all of these processes?  I really hate to reboot 
> the cluster to clear this up since it's only effecting one pipeline....
> 
> Thanks in advance!
> 
> -- scooter
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> Confidentiality Warning:  This e-mail contains information intended only for the use of the individual or entity named above.  If the reader of this e-mail is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, any dissemination, publication or copying of this e-mail is strictly prohibited.  The sender does not accept any responsibility for any loss, disruption or damage to your data or computer system that may occur while using data contained in, or transmitted with, this e-mail.  
> If you have received this e-mail in error, please immediately notify us by return e-mail.  Thank you.
> 
> 
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From Dave.Jones at maritz.com  Tue Jul 27 15:00:36 2010
From: Dave.Jones at maritz.com (Jones, Dave)
Date: Tue, 27 Jul 2010 10:00:36 -0500
Subject: [Linux-cluster] Clearing a glock
In-Reply-To: <1280240681.2543.19.camel@localhost>
References: <4C4E2053.7050404@cgl.ucsf.edu><3B29E77570056E408AB74D923453D55902C00F3E@FENEXCH1602C.us.maritz.net>
	<1280240681.2543.19.camel@localhost>
Message-ID: <3B29E77570056E408AB74D923453D55902C00F3F@FENEXCH1602C.us.maritz.net>

Understood
 

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse
Sent: Tuesday, July 27, 2010 9:25 AM
To: linux clustering
Subject: Re: [Linux-cluster] Clearing a glock

Hi,

On Tue, 2010-07-27 at 08:58 -0500, Jones, Dave wrote:
> 
> Maybe a bit off topic, but IMO Red Hat should really consider changing
> the name of this process.
> Thought I somehow crossed mailing lists and was about to read a post
> about a jammed polymer-framed 9mm pistol. 
> 
> D
> 
:-)

The name is not perhaps ideal, and its origins are somewhat lost in the
mists of time. The main issue is that if we did change it, it would make
things even more confusing at least while the transistion was in effect,

Steve.

> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Scooter Morris
> Sent: Monday, July 26, 2010 6:55 PM
> To: linux clustering
> Subject: [Linux-cluster] Clearing a glock
> 
> 
>   We've got two nodes of a three node gfs2 cluster that seem to be in 
> some sort of deadlock.  We're seeing a number of gfs2-related stack 
> traces in dmesg:
> 
> INFO: task igtcpython.sh:22945 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
> message.
> igtcpython.sh D ffff81011cabd218     0 22945  22874

> (NOTLB)
>   ffff8101f9fedcc8 0000000000000086 ffff8101f9fede38 ffffffff8000a604
>   ffff8102b257f778 0000000000000006 ffff8105819d37a0 ffff8107a7f93820
>   0004084c69951298 00000000000365c6 ffff8105819d3988 000000042d07a810
> Call Trace:
>   [<ffffffff8000a604>] __link_path_walk+0xdf8/0xf42
>   [<ffffffff8002c9e4>] mntput_no_expire+0x19/0x89
>   [<ffffffff8000ea46>] link_path_walk+0xa6/0xb2
>   [<ffffffff887d6ee7>] :gfs2:just_schedule+0x0/0xe
>   [<ffffffff887d6ef0>] :gfs2:just_schedule+0x9/0xe
>   [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
>   [<ffffffff887d6ee7>] :gfs2:just_schedule+0x0/0xe
>   [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
>   [<ffffffff800a0aec>] wake_bit_function+0x0/0x23
>   [<ffffffff887d6ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
>   [<ffffffff887e679e>] :gfs2:gfs2_permission+0x83/0xd5
>   [<ffffffff887e6796>] :gfs2:gfs2_permission+0x7b/0xd5
>   [<ffffffff8000d918>] permission+0x81/0xc8
>   [<ffffffff8003c0c0>] open_exec+0x60/0xc0
>   [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>   [<ffffffff8003ed4d>] do_execve+0x46/0x1f7
>   [<ffffffff8005516d>] sys_execve+0x36/0x4c
>   [<ffffffff8005d4d3>] stub_execve+0x67/0xb0
> 
> and gfs2_hangalyzer is saying:
> 
> ./gfs2_hangalyzer -n wilkins-pi -a
> wilkins-pi: UsrLocal: G:  s:UN n:2/a5b67f f:l t:SH d:EX/0 l:0 a:0 r:58
> wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:21810 [python] 
> gfs2_readpage+0x61/0x199 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:21809 [python] 
> gfs2_readpage+0x61/0x199 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:W e:0 p:1897 [python] 
> gfs2_readpage+0x61/0x199 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:6307 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:10436 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:11000 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:11003 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:12140 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21499 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:32601 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:653 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:10078 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:14436 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:7500 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:22815 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26056 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26062 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26122 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26124 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:26128 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31825 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:32125 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:2441 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:2444 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21792 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21793 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:21794 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31941 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31970 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8584 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8590 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8821 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:8822 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9487 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9488 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9489 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:9490 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18878 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:325 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:3139 [ls] 
> gfs2_getattr+0x7d/0xc4 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:3144 [ls] 
> gfs2_getattr+0x7d/0xc4 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:7450 [ls] 
> gfs2_getattr+0x7d/0xc4 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:31741 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:4982 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18258 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18262 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18263 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18265 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:18269 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20039 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20042 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20043 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20044 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20046 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
> wilkins-pi: UsrLocal:  H: s:SH f:aW e:0 p:20763 [igtcpython.sh] 
> gfs2_permission+0x7b/0xd5 [gfs2]
>                          lkb_id N RemoteID  pid exflg lkbflgs stat gr 
> rq    waiting n ln             resource name
> wilkins-pi: UsrLocal:  26513a1 3  1385218 21810     0       0 wait -1

> 3          0 3 24 "       2          a5b67f"
> 
> 
> 
> There is 1 glock with waiters.
> wilkins-pi.compbio.ucsf.edu, pid 21810 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 21809 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 1897 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 6307 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 10436 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 11000 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 11003 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 12140 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 21499 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 32601 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 653 is waiting for glock 2/a5b67f,
but 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 10078 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 14436 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 7500 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 22815 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 26056 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 26062 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 26122 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 26124 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 26128 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 31825 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 32125 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 2441 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 2444 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 21792 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 21793 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 21794 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 31941 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 31970 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 8584 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 8590 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 8821 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 8822 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 9487 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 9488 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 9489 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 9490 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18878 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 325 is waiting for glock 2/a5b67f,
but 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 3139 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 3144 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 7450 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 31741 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 4982 is waiting for glock 2/a5b67f,
but
> 
> no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18258 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18262 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18263 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18265 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 18269 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20039 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20042 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20043 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20044 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20046 is waiting for glock 2/a5b67f, 
> but no holder was found.
> wilkins-pi.compbio.ucsf.edu, pid 20763 is waiting for glock 2/a5b67f, 
> but no holder was found.
>           The dlm has granted lkb "       2          a5b67f" to pid 
> 391344724
> 
> Clearly, I've got a hung lock of some sort.  Is there any way to clear

> the glock to free up all of these processes?  I really hate to reboot 
> the cluster to clear this up since it's only effecting one
pipeline....
> 
> Thanks in advance!
> 
> -- scooter
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> Confidentiality Warning:  This e-mail contains information intended
only for the use of the individual or entity named above.  If the reader
of this e-mail is not the intended recipient or the employee or agent
responsible for delivering it to the intended recipient, any
dissemination, publication or copying of this e-mail is strictly
prohibited.  The sender does not accept any responsibility for any loss,
disruption or damage to your data or computer system that may occur
while using data contained in, or transmitted with, this e-mail.  
> If you have received this e-mail in error, please immediately notify
us by return e-mail.  Thank you.
> 
> 
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

Confidentiality Warning:  This e-mail contains information intended only for the use of the individual or entity named above.  If the reader of this e-mail is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, any dissemination, publication or copying of this e-mail is strictly prohibited.  The sender does not accept any responsibility for any loss, disruption or damage to your data or computer system that may occur while using data contained in, or transmitted with, this e-mail.  
If you have received this e-mail in error, please immediately notify us by return e-mail.  Thank you.






From m.watts at eris.qinetiq.com  Tue Jul 27 15:30:10 2010
From: m.watts at eris.qinetiq.com (Mark Watts)
Date: Tue, 27 Jul 2010 16:30:10 +0100
Subject: [Linux-cluster] Network Interface Binding
Message-ID: <4C4EFB82.8070907@eris.qinetiq.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have a working CentOS 5.5/RHCS cluster in order to support a simple
GFS2 filesystem (ontop of DRBD) between 2 nodes.

Since the two nodes have multiple network interfaces, I'd like to tie
all cluster communication to ETH2 as ETH0 is my production-facing interface.

I understand I can partially achieve this by using a hostname mapped
(via /etc/hosts) to the IP address of ETH2 in cluster.conf.
This seems to work for OpenAIS; the multicast address it uses is bound
to to eth2 according to "ip maddress show dev eth2", but ccsd is still
listening on 0.0.0.0 for various ports according to netstat.

How can I force ccsd to use ETH2?

Regards,

Mark.

- -- 
Mark Watts BSc RHCE MBCS
Senior Systems Engineer, IPR Secure Managed Hosting
www.QinetiQ.com
QinetiQ - Delivering customer-focused solutions
GPG Key: http://www.linux-corner.info/mwatts.gpg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkxO+4IACgkQBn4EFUVUIO3+5QCgwctCm6Xd4kohggq8RsvPim5+
iYUAn2jLHQAUejGof9VDCRr3t/HYS9iH
=0XH3
-----END PGP SIGNATURE-----



From swhiteho at redhat.com  Tue Jul 27 16:02:26 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Tue, 27 Jul 2010 17:02:26 +0100
Subject: [Linux-cluster] Network Interface Binding
In-Reply-To: <4C4EFB82.8070907@eris.qinetiq.com>
References: <4C4EFB82.8070907@eris.qinetiq.com>
Message-ID: <1280246546.2543.23.camel@localhost>

Hi,

On Tue, 2010-07-27 at 16:30 +0100, Mark Watts wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I have a working CentOS 5.5/RHCS cluster in order to support a simple
> GFS2 filesystem (ontop of DRBD) between 2 nodes.
> 
> Since the two nodes have multiple network interfaces, I'd like to tie
> all cluster communication to ETH2 as ETH0 is my production-facing interface.
> 
> I understand I can partially achieve this by using a hostname mapped
> (via /etc/hosts) to the IP address of ETH2 in cluster.conf.
> This seems to work for OpenAIS; the multicast address it uses is bound
> to to eth2 according to "ip maddress show dev eth2", but ccsd is still
> listening on 0.0.0.0 for various ports according to netstat.
> 
> How can I force ccsd to use ETH2?
> 
> Regards,
> 
> Mark.
> 
Assuming that you have different subnets on the interfaces, setting the
routing table correctly should be enough. Just be careful that any names
which are being resolved point to the correct ip addresses.

If you don't have a different subnet, you can still do it if you mark
the traffic with iptables and route according to the mark.

You may need multiple routing tables (again depending on the exact
configuration) and a couple of routing rules to ensure that replies are
always sent out of the same interfaces on which the queries came in on,

Steve.

> - -- 
> Mark Watts BSc RHCE MBCS
> Senior Systems Engineer, IPR Secure Managed Hosting
> www.QinetiQ.com
> QinetiQ - Delivering customer-focused solutions
> GPG Key: http://www.linux-corner.info/mwatts.gpg
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.14 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAkxO+4IACgkQBn4EFUVUIO3+5QCgwctCm6Xd4kohggq8RsvPim5+
> iYUAn2jLHQAUejGof9VDCRr3t/HYS9iH
> =0XH3
> -----END PGP SIGNATURE-----
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From m.watts at eris.qinetiq.com  Tue Jul 27 16:20:17 2010
From: m.watts at eris.qinetiq.com (Mark Watts)
Date: Tue, 27 Jul 2010 17:20:17 +0100
Subject: [Linux-cluster] Network Interface Binding
In-Reply-To: <1280246546.2543.23.camel@localhost>
References: <4C4EFB82.8070907@eris.qinetiq.com>
	<1280246546.2543.23.camel@localhost>
Message-ID: <4C4F0741.5090307@eris.qinetiq.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 27/07/10 17:02, Steven Whitehouse wrote:
> Hi,
> 
> On Tue, 2010-07-27 at 16:30 +0100, Mark Watts wrote:
> I have a working CentOS 5.5/RHCS cluster in order to support a simple
> GFS2 filesystem (ontop of DRBD) between 2 nodes.
> 
> Since the two nodes have multiple network interfaces, I'd like to tie
> all cluster communication to ETH2 as ETH0 is my production-facing interface.
> 
> I understand I can partially achieve this by using a hostname mapped
> (via /etc/hosts) to the IP address of ETH2 in cluster.conf.
> This seems to work for OpenAIS; the multicast address it uses is bound
> to to eth2 according to "ip maddress show dev eth2", but ccsd is still
> listening on 0.0.0.0 for various ports according to netstat.
> 
> How can I force ccsd to use ETH2?
> 
> Regards,
> 
> Mark.
> 
>> Assuming that you have different subnets on the interfaces, setting the
>> routing table correctly should be enough. Just be careful that any names
>> which are being resolved point to the correct ip addresses.
> 
>> If you don't have a different subnet, you can still do it if you mark
>> the traffic with iptables and route according to the mark.
> 
>> You may need multiple routing tables (again depending on the exact
>> configuration) and a couple of routing rules to ensure that replies are
>> always sent out of the same interfaces on which the queries came in on,
> 
>> Steve.

Well, My interfaces are setup as follows:

ETH0	192.168.1.1/24
ETH2	172.16.1.1/24

cluster.conf references "node1.cluster" which is in /etc/hosts as
172.16.1.1 (node2.cluster follows as .2)

I have no explicit routing on the box, other than a default gateway out
on ETH3's subnet, which is different to the two I've listed.

I suppose my main issue is that I don't want to expose anything other
than TCP/80 on ETH0. Usually I'd do this as a combination of binding
services to specific IP's and doing both inbound and outbound iptables
rules for each interface. These would typically cover every service/port
that I'd use, but multicast makes this a slightly different beast.

Not having done iptables rules for multicast before, I'm a little wary
of doing something without fully understanding how something works
before I firewall it!

Regards,

Mark.

- -- 
Mark Watts BSc RHCE MBCS
Senior Systems Engineer, IPR Secure Managed Hosting
www.QinetiQ.com
QinetiQ - Delivering customer-focused solutions
GPG Key: http://www.linux-corner.info/mwatts.gpg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkxPB0EACgkQBn4EFUVUIO3ndQCeK5DZp3WKMmZNuYFFdWG8Nph+
Qu4AoMSnzIMFsSqEW/G7CsL28Psnv/kZ
=SwPT
-----END PGP SIGNATURE-----



From scooter at cgl.ucsf.edu  Tue Jul 27 17:14:47 2010
From: scooter at cgl.ucsf.edu (Scooter Morris)
Date: Tue, 27 Jul 2010 10:14:47 -0700
Subject: [Linux-cluster] Clearing a glock
In-Reply-To: <1280236730.2543.14.camel@localhost>
References: <4C4E2053.7050404@cgl.ucsf.edu>	<1280232915.2543.5.camel@localhost>
	<4C4ED7A2.9030706@cgl.ucsf.edu>
	<1280236730.2543.14.camel@localhost>
Message-ID: <4C4F1407.9070609@cgl.ucsf.edu>

  Hi Steve,
     More information.  The offending file was /usr/local/bin/python2.6, 
which we use heavily on all nodes.  Our general use is through the #! 
mechanism in .py files.  Does this offer any clues as to why we had all 
of those processes waiting on a lock with no holder?

-- scooter

On 07/27/2010 06:18 AM, Steven Whitehouse wrote:
> Hi,
>
> On Tue, 2010-07-27 at 05:57 -0700, Scooter Morris wrote:
>> On 7/27/10 5:15 AM, Steven Whitehouse wrote:
>>> Hi,
>>>
>>> If you translate a5b67f into decimal, then that is the inode number of
>>> the inode which is causing a problem. It looks to me as if you have too
>>> many processes trying to access this one inode from multiple nodes.
>>>
>>> Its not obvious from the traces that anything is actually stuck, but if
>>> you take two traces, a few seconds or minutes apart, then it should
>>> become more obvious whether the cluster is making progress or whether it
>>> really is stuck,
>>>
>>> Steve.
>>>
>>>
>>> --
>>> Linux-cluster mailing list
>>> Linux-cluster at redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>> Hi Steve,
>>       As always, thanks for the reply.  The cluster was, indeed, truly
>> stuck.  I rebooted it last night to clear everything out.  I never did
>> figure out which file was the problem.  I did a find -inum, but the find
>> hung too.  By that point the load average was up to 80 and climbing.
>> Any ideas on how to avoid this?  Are there tunable values I need to
>> increase to allow more processes to access any individual inode?
>>
> The LA includes processes waiting for glocks since that is an
> uninterruptible wait, so thats where most of the LA came from.
>
> The find is unlikely to work while the cluster is stuck, since if it
> does find the cuplrit inode, it is, by definition already stuck so the
> find process would just join the queue. If a find fails to discover the
> inode when the cluster has been rebooted and is back working again, then
> it was probably a temporary file of some kind.
>
> There are no tunable values since the limitation on the access to the
> inode is the speed of the hardware in terms of how many times a given
> inode can be synced, invalidated and the glock passed on to another node
> in a given time period. It is a limitation of the hardware and the
> architecture of the filesystem.
>
> There are a few things which can probably be improved in due course, but
> in the main the best way to avoid problems with congestion on inodes is
> just to be careful about the access pattern across nodes.
>
> That said, if it really was completely stuck, that is a real bug and not
> the result of the access pattern since the code is designed such that
> progress should always be made, even if its painfully slow,
>
> Steve.
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From swhiteho at redhat.com  Tue Jul 27 17:35:28 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Tue, 27 Jul 2010 18:35:28 +0100
Subject: [Linux-cluster] Clearing a glock
In-Reply-To: <4C4F1407.9070609@cgl.ucsf.edu>
References: <4C4E2053.7050404@cgl.ucsf.edu>
	<1280232915.2543.5.camel@localhost> <4C4ED7A2.9030706@cgl.ucsf.edu>
	<1280236730.2543.14.camel@localhost>  <4C4F1407.9070609@cgl.ucsf.edu>
Message-ID: <1280252128.2543.51.camel@localhost>

Hi,

On Tue, 2010-07-27 at 10:14 -0700, Scooter Morris wrote:
> Hi Steve,
>      More information.  The offending file was /usr/local/bin/python2.6, 
> which we use heavily on all nodes.  Our general use is through the #! 
> mechanism in .py files.  Does this offer any clues as to why we had all 
> of those processes waiting on a lock with no holder?
> 
> -- scooter
> 
Not really. I'd have expected that to be mapped read-only on the nodes
and there should be no write activity to it at all, so it should scale
very well. Did you set noatime?

I can't think of any other reason why that should have been an issue,

Steve.

> On 07/27/2010 06:18 AM, Steven Whitehouse wrote:
> > Hi,
> >
> > On Tue, 2010-07-27 at 05:57 -0700, Scooter Morris wrote:
> >> On 7/27/10 5:15 AM, Steven Whitehouse wrote:
> >>> Hi,
> >>>
> >>> If you translate a5b67f into decimal, then that is the inode number of
> >>> the inode which is causing a problem. It looks to me as if you have too
> >>> many processes trying to access this one inode from multiple nodes.
> >>>
> >>> Its not obvious from the traces that anything is actually stuck, but if
> >>> you take two traces, a few seconds or minutes apart, then it should
> >>> become more obvious whether the cluster is making progress or whether it
> >>> really is stuck,
> >>>
> >>> Steve.
> >>>
> >>>
> >>> --
> >>> Linux-cluster mailing list
> >>> Linux-cluster at redhat.com
> >>> https://www.redhat.com/mailman/listinfo/linux-cluster
> >> Hi Steve,
> >>       As always, thanks for the reply.  The cluster was, indeed, truly
> >> stuck.  I rebooted it last night to clear everything out.  I never did
> >> figure out which file was the problem.  I did a find -inum, but the find
> >> hung too.  By that point the load average was up to 80 and climbing.
> >> Any ideas on how to avoid this?  Are there tunable values I need to
> >> increase to allow more processes to access any individual inode?
> >>
> > The LA includes processes waiting for glocks since that is an
> > uninterruptible wait, so thats where most of the LA came from.
> >
> > The find is unlikely to work while the cluster is stuck, since if it
> > does find the cuplrit inode, it is, by definition already stuck so the
> > find process would just join the queue. If a find fails to discover the
> > inode when the cluster has been rebooted and is back working again, then
> > it was probably a temporary file of some kind.
> >
> > There are no tunable values since the limitation on the access to the
> > inode is the speed of the hardware in terms of how many times a given
> > inode can be synced, invalidated and the glock passed on to another node
> > in a given time period. It is a limitation of the hardware and the
> > architecture of the filesystem.
> >
> > There are a few things which can probably be improved in due course, but
> > in the main the best way to avoid problems with congestion on inodes is
> > just to be careful about the access pattern across nodes.
> >
> > That said, if it really was completely stuck, that is a real bug and not
> > the result of the access pattern since the code is designed such that
> > progress should always be made, even if its painfully slow,
> >
> > Steve.
> >
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From scooter at cgl.ucsf.edu  Tue Jul 27 17:45:33 2010
From: scooter at cgl.ucsf.edu (Scooter Morris)
Date: Tue, 27 Jul 2010 10:45:33 -0700
Subject: [Linux-cluster] Clearing a glock
In-Reply-To: <1280252128.2543.51.camel@localhost>
References: <4C4E2053.7050404@cgl.ucsf.edu>	<1280232915.2543.5.camel@localhost>
	<4C4ED7A2.9030706@cgl.ucsf.edu>	<1280236730.2543.14.camel@localhost>
	<4C4F1407.9070609@cgl.ucsf.edu>
	<1280252128.2543.51.camel@localhost>
Message-ID: <4C4F1B3D.1050204@cgl.ucsf.edu>

  On 07/27/2010 10:35 AM, Steven Whitehouse wrote:
> Hi,
>
> On Tue, 2010-07-27 at 10:14 -0700, Scooter Morris wrote:
>> Hi Steve,
>>       More information.  The offending file was /usr/local/bin/python2.6,
>> which we use heavily on all nodes.  Our general use is through the #!
>> mechanism in .py files.  Does this offer any clues as to why we had all
>> of those processes waiting on a lock with no holder?
>>
>> -- scooter
>>
> Not really. I'd have expected that to be mapped read-only on the nodes
> and there should be no write activity to it at all, so it should scale
> very well. Did you set noatime?
Yes.
> I can't think of any other reason why that should have been an issue,
Neither could I.  Well, we'll let it ride for now, but if it repeats, I 
file a bug and open a case with RedHat support (and move the binary off 
of gfs2).

-- scooter
> Steve.
>
>> On 07/27/2010 06:18 AM, Steven Whitehouse wrote:
>>> Hi,
>>>
>>> On Tue, 2010-07-27 at 05:57 -0700, Scooter Morris wrote:
>>>> On 7/27/10 5:15 AM, Steven Whitehouse wrote:
>>>>> Hi,
>>>>>
>>>>> If you translate a5b67f into decimal, then that is the inode number of
>>>>> the inode which is causing a problem. It looks to me as if you have too
>>>>> many processes trying to access this one inode from multiple nodes.
>>>>>
>>>>> Its not obvious from the traces that anything is actually stuck, but if
>>>>> you take two traces, a few seconds or minutes apart, then it should
>>>>> become more obvious whether the cluster is making progress or whether it
>>>>> really is stuck,
>>>>>
>>>>> Steve.
>>>>>
>>>>>
>>>>> --
>>>>> Linux-cluster mailing list
>>>>> Linux-cluster at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>> Hi Steve,
>>>>        As always, thanks for the reply.  The cluster was, indeed, truly
>>>> stuck.  I rebooted it last night to clear everything out.  I never did
>>>> figure out which file was the problem.  I did a find -inum, but the find
>>>> hung too.  By that point the load average was up to 80 and climbing.
>>>> Any ideas on how to avoid this?  Are there tunable values I need to
>>>> increase to allow more processes to access any individual inode?
>>>>
>>> The LA includes processes waiting for glocks since that is an
>>> uninterruptible wait, so thats where most of the LA came from.
>>>
>>> The find is unlikely to work while the cluster is stuck, since if it
>>> does find the cuplrit inode, it is, by definition already stuck so the
>>> find process would just join the queue. If a find fails to discover the
>>> inode when the cluster has been rebooted and is back working again, then
>>> it was probably a temporary file of some kind.
>>>
>>> There are no tunable values since the limitation on the access to the
>>> inode is the speed of the hardware in terms of how many times a given
>>> inode can be synced, invalidated and the glock passed on to another node
>>> in a given time period. It is a limitation of the hardware and the
>>> architecture of the filesystem.
>>>
>>> There are a few things which can probably be improved in due course, but
>>> in the main the best way to avoid problems with congestion on inodes is
>>> just to be careful about the access pattern across nodes.
>>>
>>> That said, if it really was completely stuck, that is a real bug and not
>>> the result of the access pattern since the code is designed such that
>>> progress should always be made, even if its painfully slow,
>>>
>>> Steve.
>>>
>>>
>>> --
>>> Linux-cluster mailing list
>>> Linux-cluster at redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From Wahyu.Darmawan at ag-it.com  Wed Jul 28 01:49:55 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Wed, 28 Jul 2010 08:49:55 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <AANLkTi=Wj4=2m+ufnNAMiOUxh8WqnbCDZY_ySK5dNL2o@mail.gmail.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
	<AANLkTikdqYmBW3xDvraban3CIdexrZMqtUQtKIbXNtCO@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854505172482F1@svr-e2k7.ag-it.com>
	<AANLkTi=Wj4=2m+ufnNAMiOUxh8WqnbCDZY_ySK5dNL2o@mail.gmail.com>
Message-ID: <0FE1B519D219B64A87E99E3620238545052C1AE4CB@svr-e2k7.ag-it.com>

Hi Ball ;),
Thanks for your response. Please find the cluster.conf below.

<?xml version="1.0"?>
<cluster alias="DBAHMPS" config_version="2" name="TAI123">
                <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
                <clusternodes>
                                <clusternode name="node1.example.com" nodeid="1" votes="1">
                                                <fence>
                                                                <method name="1">
                                                                                <device domain="RHEL" name="VM_Fence"/>
                                                                </method>
                                                </fence>
                                </clusternode>
                                <clusternode name="node2.example.com" nodeid="2" votes="1">
                                                <fence>
                                                                <method name="1">
                                                                                <device domain="RHEL2" name="VM_Fence"/>
                                                                </method>
                                                </fence>
                                </clusternode>
                </clusternodes>
                <cman expected_votes="1" two_node="1"/>
                <fencedevices>
                                <fencedevice agent="fence_xvm" name="VM_Fence"/>
                </fencedevices>
                <rm>
                                <failoverdomains>
                                                <failoverdomain name="IP_Virtual" ordered="1" restricted="1">
                                                                <failoverdomainnode name="node1.example.com" priority="1"/>
                                                                <failoverdomainnode name="node2.example.com" priority="1"/>
                                                </failoverdomain>
                                </failoverdomains>
                                <resources>
                                                <ip address="192.168.122.100/24" monitor_link="1"/>
                                                <fs device="/dev/mapper/VolGroup00-LogVol00" force_fsck="0" force_unmount="0" fsid="43719" fstype="ext3" mountpoint="/var/www/html/" name="HTML Doc" options="" self_fence="0"/>
                                                <script file="/etc/init.d/httpd" name="Web"/>
                                                <apache config_file="/etc/httpd/conf/httpd.conf" httpd_options="" name="httpd" server_root="/etc/httpd" shutdown_wait="3"/>
                                                <oracledb home="/u01/app/oracle/product/9.2.0.8/" name="DBAHMPS" type="10g" user="oracle" vhost="dbahmps.example.com"/>
                                </resources>
                                <service autostart="1" exclusive="1" name="HTTPD_Service" recovery="relocate">
                                                <ip ref="192.168.122.100/24"/>
                                                <fs ref="HTML Doc"/>
                                                <script ref="Web"/>
                                </service>
                                <service autostart="1" domain="IP_Virtual" name="Web" recovery="restart">
                                                <fs ref="HTML Doc"/>
                                                <script ref="Web"/>
                                </service>
                                <service autostart="1" domain="IP_Virtual" exclusive="1" name="Apache Service" nfslock="1">
                                                <apache ref="httpd"/>
                                </service>
                </rm>
</cluster>

Appreciating for your help Ball.


Rgds,
Wahyu

From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of POWERBALL ONLINE
Sent: Saturday, July 24, 2010 12:35 AM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

I though that your configuration is wrong because when you run clustat it should display quorum disk is online too.
If your configuration wrong and quorum not online it will have the error that you meet.
Please give me the file /etc/cluster/cluster.conf to me too.



On Fri, Jul 23, 2010 at 10:18 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi,
Thanks for your response.
Here is the info on below.


T0171
[root at t0171 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  t0171.example.com<http://t0171.example.com/>                    Online, Local, rgmanager
  t0175.example.com<http://t0175.example.com/>                    Online, rgmanager

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  IP_Virtual           t0175.example.com<http://t0175.example.com/>          started
  GFS_Mounting         t0175.example.com<http://t0175.example.com/>          started
  Listener_Service     t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBahmcm       t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBportal      t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBcard5       t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBhrdtxn      t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBrepwm       t0175.example.com<http://t0175.example.com/>          started
[root at t0171 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171.example.com<http://t0171.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/sdi2:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-18:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171.example.com<http://t0171.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-19:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512


T0175
[root at t0175 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  t0171.example.com<http://t0171.example.com/>                    Online, rgmanager
  t0175.example.com<http://t0175.example.com/>                    Online, Local, rgmanager

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  IP_Virtual           t0175. example.com<http://example.com/>          started
  GFS_Mounting         t0175. example.com<http://example.com/>          started
  Listener_Service     t0175. example.com<http://example.com/>          started
  Oracle_DBahmcm       t0175. example.com<http://example.com/>          started
  Oracle_DBportal      t0175. example.com<http://example.com/>          started
  Oracle_DBcard5       t0175. example.com<http://example.com/>          started
  Oracle_DBhrdtxn      t0175. example.com<http://example.com/>          started
  Oracle_DBrepwm       t0175. example.com<http://example.com/>          started
[root at t0175 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/sdi2:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-19:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-20:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512


Thanks & Regards,
WD

From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Friday, July 23, 2010 12:29 AM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Please give me the information after you run mkqdisk -L and clustat -i command
On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi,
So, what your suggestion?

Thanks,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Wahyu Darmawan
Sent: Thursday, July 22, 2010 11:02 PM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Yes correct. I use Conga for my clusterware.

Thanks,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Thursday, July 22, 2010 10:54 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi

What tool you use for create cluster? Is it Conga ?



On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi all,
I have problem with my cluster. I have 2 nodes of my cluster.
On July 12 the nodes missed  connection from storage and lost 1 disk partition on both nodes. Then I decided to assigned a new of partition from storage, and use it on my nodes.
But until today, the quorum is not going to quorate.
FYI, I use GFS for my filesystem.
So, what do I have to do to make quorum disk back to quorate status?

I appreciate for your answer.

Thanks,
Wahyu

Note, I attached the cluster log messages below:
Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is disabled
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service Oracle_DBrepwm
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is disabled
Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBahmcm
Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm started
Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBportal
Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal started
Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBcard5
Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5 started
Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBhrdtxn
Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn started
Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBrepwm
Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm started
Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)
Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.310000)
Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.400000)
Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.690000)
Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.620000)
Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.020000)
Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.550000)
Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.050000)
Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.710000)
Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:05:30 t0175 last message repeated 2 times
Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.530000)
Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.200000)
Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.820000)
Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.420000)
Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.820000)
Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.540000)
Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.740000)
Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.020000)
Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.900000)
Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.270000)
Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.980000)
Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.470000)
Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.870000)
Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.940000)
Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.780000)
Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.360000)
Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.790000)
Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.730000)
Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.100000)
Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.580000)
Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.460000)
Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.790000)
Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.240000)
Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.670000)
Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.570000)
Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.000000)
Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.670000)
Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 18 02:04:31 t0175 last message repeated 3 times
Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.870000)
Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.920000)
Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.040000)
Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.350000)
Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.450000)
Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.450000)
Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.120000)
Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.480000)
Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.960000)
Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.030000)
Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.940000)
Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.560000)


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100728/b712b8b2/attachment.htm>

From benedikt.koeppel at let.ethz.ch  Wed Jul 28 07:58:21 2010
From: benedikt.koeppel at let.ethz.ch (=?iso-8859-1?Q?K=F6ppel__Benedikt_=28LET=29?=)
Date: Wed, 28 Jul 2010 07:58:21 +0000
Subject: [Linux-cluster] SAN with GFS2 on RHEL 6 beta: STONITH right after
	start
Message-ID: <ABD9AEFD-FAC6-4E01-A799-B312BD7BE0EF@let.ethz.ch>

Hello

I have two nodes running RHEL 6 beta 2  and configured corosync as follows. 
Both nodes have access to a SAN disk. The disk is partitioned into /dev/sdb1 
for SBD STONITH and /dev/sdb2 for data. /dev/sdb2 has a GFS2 filesystem on the 
LVM (vg01/lv00). For the configuration, I followed the Cluster from Scratch PDF 
from clusterlabs.

As soon as I start the two nodes, one of them gets immediately fenced and shut 
down. I see in the logs, that the fenced node tries to mount the FS when he 
gets shot down. I have no clue why this happens. Can anyone give me a hint how 
to fix my cluster?


Configuration:
[root at pcmknode-1 ~]# crm configure show
node pcmknode-1
node pcmknode-2
primitive WebFS ocf:heartbeat:Filesystem \
	params device="/dev/vg01/lv00" directory="/data_1" fstype="gfs2"
primitive dlm ocf:pacemaker:controld \
	params configdir="/config" \
	op monitor interval="120s"
primitive gfs-control ocf:pacemaker:controld \
	params daemon="gfs_controld.pcmk" args="-g 0" \
	op monitor interval="120s"
primitive resSBD stonith:external/sbd \
	params sbd_device="/dev/sdb1"
clone WebFSClone WebFS
clone dlm-clone dlm \
	meta interleave="true" target-role="Started"
clone gfs-clone gfs-control \
	meta interleave="true" target-role="Started"
location cli-prefer-WebFS WebFSClone \
	rule $id="cli-prefer-rule-WebFS" inf: #uname eq pcmknode-1 and date lt "2010-07-27 21:53:10Z"
colocation WebFS-with-gfs-control inf: WebFSClone gfs-clone
colocation gfs-with-dlm inf: gfs-clone dlm-clone
order start-WebFS-after-gfs-control inf: gfs-clone WebFSClone
order start-gfs-after-dlm inf: dlm-clone gfs-clone
property $id="cib-bootstrap-options" \
	dc-version="1.1.2-f059ec7ced7a86f18e5490b67ebf4a0b963bccfe" \
	cluster-infrastructure="openais" \
	expected-quorum-votes="2" \
	stonith-enabled="true" \
	stonith-timeout="30s" \
	no-quorum-policy="ignore"



These are the logs:
pcmknode-1: /var/log/messages
~ snip ~
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: native_color: Resource WebFS:0 cannot run anywhere
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: custom_action: Action dlm:0_stop_0 on pcmknode-2 is unrunnable (offline)
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: custom_action: Marking node pcmknode-2 unclean
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: custom_action: Action gfs-control:0_stop_0 on pcmknode-2 is unrunnable (offline)
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: custom_action: Marking node pcmknode-2 unclean
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: stage6: Scheduling Node pcmknode-2 for STONITH
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: native_stop_constraints: dlm:0_stop_0 is implicit after pcmknode-2 is fenced
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: native_stop_constraints: gfs-control:0_stop_0 is implicit after pcmknode-2 is fenced
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: find_compatible_child: Colocating gfs-control:1 with dlm:1 on pcmknode-1
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: clone_rsc_order_lh: Interleaving dlm:1 with gfs-control:1
Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: te_fence_node: Executing reboot fencing operation (30) on pcmknode-2 (timeout=30000)
Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: te_rsc_command: Initiating action 22: stop WebFS:1_stop_0 on pcmknode-1 (local)
Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: do_lrm_rsc_op: Performing key=22:3:0:2419bb70-dce6-4a0e-b649-fae2b0f21b8d op=WebFS:1_stop_0 )
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: find_compatible_child: Colocating dlm:0 with gfs-control:0 on pcmknode-2
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: clone_rsc_order_lh: Interleaving gfs-control:0 with dlm:0
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: find_compatible_child: Colocating dlm:1 with gfs-control:1 on pcmknode-1
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: clone_rsc_order_lh: Interleaving gfs-control:1 with dlm:1
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: find_compatible_child: Colocating WebFS:1 with gfs-control:1 on pcmknode-1
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: clone_rsc_order_lh: Interleaving gfs-control:1 with WebFS:1
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Leave resource resSBD	(Started pcmknode-1)
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Stop resource dlm:0	(pcmknode-2)
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Leave resource dlm:1	(Started pcmknode-1)
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Stop resource gfs-control:0	(pcmknode-2)
Jul 28 00:46:32 pcmknode-1 cib: [2900]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-23.raw
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Leave resource gfs-control:1	(Started pcmknode-1)
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Leave resource WebFS:0	(Stopped)
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Restart resource WebFS:1	(Started pcmknode-1)
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: process_pe_message: Transition 3: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-4.bz2
Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Jul 28 00:46:32 pcmknode-1 cib: [2900]: info: write_cib_contents: Wrote version 0.127.0 of the CIB to disk (digest: af7f98fa70bd2ef644e8e70d6f2ceea9)
Jul 28 00:46:32 pcmknode-1 cib: [2900]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.vx6ONO (digest: /var/lib/heartbeat/crm/cib.kGkBp3)
Jul 28 00:46:32 pcmknode-1 Filesystem[2902]: INFO: Running stop for /dev/vg01/lv00 on /data_1
Jul 28 00:46:32 pcmknode-1 Filesystem[2902]: INFO: Trying to unmount /data_1
Jul 28 00:46:35 pcmknode-1 stonith-ng: [2624]: ERROR: remote_op_query_timeout: Query 8f1eeecf-4832-4430-b8e8-41a645675c58 for pcmknode-2 timed out
Jul 28 00:46:35 pcmknode-1 stonith-ng: [2624]: ERROR: remote_op_timeout: Action reboot (8f1eeecf-4832-4430-b8e8-41a645675c58) for pcmknode-2 timed out
Jul 28 00:46:35 pcmknode-1 stonith-ng: [2624]: info: remote_op_done: Notifing clients of 8f1eeecf-4832-4430-b8e8-41a645675c58 (reboot of pcmknode-2 from 540c61a4-d351-40c7-aa60-efd445097180 by (null)): 0, rc=-7
Jul 28 00:46:35 pcmknode-1 stonith-ng: [2624]: info: stonith_notify_client: Sending st_fence-notification to client 2629/cc57856c-5357-4343-95a9-712771f711ae
Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: log_data_element: tengine_stonith_callback: StonithOp <remote-op state="0" st_target="pcmknode-2" st_op="reboot" />
Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: tengine_stonith_callback: Stonith operation 2/30:3:0:2419bb70-dce6-4a0e-b649-fae2b0f21b8d: Operation timed out (-7)
Jul 28 00:46:35 pcmknode-1 crmd: [2629]: ERROR: tengine_stonith_callback: Stonith of pcmknode-2 failed (-7)... aborting transition.
Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: abort_transition_graph: tengine_stonith_callback:402 - Triggered transition abort (complete=0) : Stonith failed
Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: update_abort_priority: Abort action done superceeded by restart
Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: tengine_stonith_notify: Peer pcmknode-2 was terminated (reboot) by (null) for pcmknode-1 (ref=8f1eeecf-4832-4430-b8e8-41a645675c58): Operation timed out
~ snip ~



pcmknode-2: /var/log/messages
~ snip ~
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Received ringid(192.168.1.186:620) seq 91
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Delivering 90 to 91
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Delivering MCAST message with seq 91 to pending delivery queue
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Received ringid(192.168.1.186:620) seq 92
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Delivering 91 to 92
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Delivering MCAST message with seq 92 to pending delivery queue
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] mcasted message added to pending queue
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Delivering 92 to 93
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Delivering MCAST message with seq 93 to pending delivery queue
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [CPG   ] got procjoin message from cluster node -1147763583
Jul 28 00:46:29 pcmknode-2 cib: [2542]: debug: cib_process_xpath: cib_query: //nvpar[@name='terminate'] does not exist
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Received ringid(192.168.1.186:620) seq 93
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] releasing messages up to and including 92
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [CPG   ] got mcast request on 0x1b072a0
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Received ringid(192.168.1.186:620) seq 94
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Delivering 93 to 94
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Delivering MCAST message with seq 94 to pending delivery queue
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] mcasted message added to pending queue
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] releasing messages up to and including 93
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Delivering 94 to 95
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Delivering MCAST message with seq 95 to pending delivery queue
Jul 28 00:46:29 pcmknode-2 kernel: : dlm: got connection from -1164540799
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] Received ringid(192.168.1.186:620) seq 95
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] releasing messages up to and including 94
Jul 28 00:46:29 pcmknode-2 corosync[2535]:   [TOTEM ] releasing messages up to and including 95
Jul 28 00:46:29 pcmknode-2 kernel: GFS2: fsid=pcmknode:data1s.0: Joined cluster. Now mounting FS...
-- that was the last message in the log.




So, how can I fix my cluster? What exactly is the problem?


Thanks,
Benedikt



From andrew at beekhof.net  Wed Jul 28 08:25:09 2010
From: andrew at beekhof.net (Andrew Beekhof)
Date: Wed, 28 Jul 2010 10:25:09 +0200
Subject: [Linux-cluster] SAN with GFS2 on RHEL 6 beta: STONITH right
	after start
In-Reply-To: <ABD9AEFD-FAC6-4E01-A799-B312BD7BE0EF@let.ethz.ch>
References: <ABD9AEFD-FAC6-4E01-A799-B312BD7BE0EF@let.ethz.ch>
Message-ID: <AANLkTi=VLpogRZF2mBUbXhmm1HXJ1Zs9tQxs_y_Q1dc=@mail.gmail.com>

On Wed, Jul 28, 2010 at 9:58 AM, K?ppel  Benedikt (LET)
<benedikt.koeppel at let.ethz.ch> wrote:
> Hello
>
> I have two nodes running RHEL 6 beta 2 ?and configured corosync as follows.
> Both nodes have access to a SAN disk. The disk is partitioned into /dev/sdb1
> for SBD STONITH and /dev/sdb2 for data. /dev/sdb2 has a GFS2 filesystem on the
> LVM (vg01/lv00). For the configuration, I followed the Cluster from Scratch PDF
> from clusterlabs.
>
> As soon as I start the two nodes, one of them gets immediately fenced and shut
> down.

Are these the Pacemaker packages that ship with RHEL?
Because external/sbd isnt a fencing device we ship on RHEL so I've
having trouble imagining how you're nodes are being fenced.

> I see in the logs, that the fenced node tries to mount the FS when he
> gets shot down. I have no clue why this happens. Can anyone give me a hint how
> to fix my cluster?
>
>
> Configuration:
> [root at pcmknode-1 ~]# crm configure show
> node pcmknode-1
> node pcmknode-2
> primitive WebFS ocf:heartbeat:Filesystem \
> ? ? ? ?params device="/dev/vg01/lv00" directory="/data_1" fstype="gfs2"
> primitive dlm ocf:pacemaker:controld \
> ? ? ? ?params configdir="/config" \
> ? ? ? ?op monitor interval="120s"
> primitive gfs-control ocf:pacemaker:controld \
> ? ? ? ?params daemon="gfs_controld.pcmk" args="-g 0" \
> ? ? ? ?op monitor interval="120s"
> primitive resSBD stonith:external/sbd \
> ? ? ? ?params sbd_device="/dev/sdb1"
> clone WebFSClone WebFS
> clone dlm-clone dlm \
> ? ? ? ?meta interleave="true" target-role="Started"
> clone gfs-clone gfs-control \
> ? ? ? ?meta interleave="true" target-role="Started"
> location cli-prefer-WebFS WebFSClone \
> ? ? ? ?rule $id="cli-prefer-rule-WebFS" inf: #uname eq pcmknode-1 and date lt "2010-07-27 21:53:10Z"
> colocation WebFS-with-gfs-control inf: WebFSClone gfs-clone
> colocation gfs-with-dlm inf: gfs-clone dlm-clone
> order start-WebFS-after-gfs-control inf: gfs-clone WebFSClone
> order start-gfs-after-dlm inf: dlm-clone gfs-clone
> property $id="cib-bootstrap-options" \
> ? ? ? ?dc-version="1.1.2-f059ec7ced7a86f18e5490b67ebf4a0b963bccfe" \
> ? ? ? ?cluster-infrastructure="openais" \
> ? ? ? ?expected-quorum-votes="2" \
> ? ? ? ?stonith-enabled="true" \
> ? ? ? ?stonith-timeout="30s" \
> ? ? ? ?no-quorum-policy="ignore"
>
>
>
> These are the logs:
> pcmknode-1: /var/log/messages
> ~ snip ~
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: native_color: Resource WebFS:0 cannot run anywhere
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: custom_action: Action dlm:0_stop_0 on pcmknode-2 is unrunnable (offline)
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: custom_action: Marking node pcmknode-2 unclean
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: custom_action: Action gfs-control:0_stop_0 on pcmknode-2 is unrunnable (offline)
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: custom_action: Marking node pcmknode-2 unclean
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: stage6: Scheduling Node pcmknode-2 for STONITH
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: native_stop_constraints: dlm:0_stop_0 is implicit after pcmknode-2 is fenced
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: native_stop_constraints: gfs-control:0_stop_0 is implicit after pcmknode-2 is fenced
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: find_compatible_child: Colocating gfs-control:1 with dlm:1 on pcmknode-1
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: clone_rsc_order_lh: Interleaving dlm:1 with gfs-control:1
> Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: te_fence_node: Executing reboot fencing operation (30) on pcmknode-2 (timeout=30000)
> Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: te_rsc_command: Initiating action 22: stop WebFS:1_stop_0 on pcmknode-1 (local)
> Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: do_lrm_rsc_op: Performing key=22:3:0:2419bb70-dce6-4a0e-b649-fae2b0f21b8d op=WebFS:1_stop_0 )
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: find_compatible_child: Colocating dlm:0 with gfs-control:0 on pcmknode-2
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: clone_rsc_order_lh: Interleaving gfs-control:0 with dlm:0
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: find_compatible_child: Colocating dlm:1 with gfs-control:1 on pcmknode-1
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: clone_rsc_order_lh: Interleaving gfs-control:1 with dlm:1
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: find_compatible_child: Colocating WebFS:1 with gfs-control:1 on pcmknode-1
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: clone_rsc_order_lh: Interleaving gfs-control:1 with WebFS:1
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Leave resource resSBD ? (Started pcmknode-1)
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Stop resource dlm:0 ? ? (pcmknode-2)
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Leave resource dlm:1 ? ?(Started pcmknode-1)
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Stop resource gfs-control:0 ? ? (pcmknode-2)
> Jul 28 00:46:32 pcmknode-1 cib: [2900]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-23.raw
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Leave resource gfs-control:1 ? ?(Started pcmknode-1)
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Leave resource WebFS:0 ?(Stopped)
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: notice: LogActions: Restart resource WebFS:1 ? ? ? ?(Started pcmknode-1)
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: process_pe_message: Transition 3: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-4.bz2
> Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: process_pe_message: Configuration WARNINGs found during PE processing. ?Please run "crm_verify -L" to identify issues.
> Jul 28 00:46:32 pcmknode-1 cib: [2900]: info: write_cib_contents: Wrote version 0.127.0 of the CIB to disk (digest: af7f98fa70bd2ef644e8e70d6f2ceea9)
> Jul 28 00:46:32 pcmknode-1 cib: [2900]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.vx6ONO (digest: /var/lib/heartbeat/crm/cib.kGkBp3)
> Jul 28 00:46:32 pcmknode-1 Filesystem[2902]: INFO: Running stop for /dev/vg01/lv00 on /data_1
> Jul 28 00:46:32 pcmknode-1 Filesystem[2902]: INFO: Trying to unmount /data_1
> Jul 28 00:46:35 pcmknode-1 stonith-ng: [2624]: ERROR: remote_op_query_timeout: Query 8f1eeecf-4832-4430-b8e8-41a645675c58 for pcmknode-2 timed out
> Jul 28 00:46:35 pcmknode-1 stonith-ng: [2624]: ERROR: remote_op_timeout: Action reboot (8f1eeecf-4832-4430-b8e8-41a645675c58) for pcmknode-2 timed out
> Jul 28 00:46:35 pcmknode-1 stonith-ng: [2624]: info: remote_op_done: Notifing clients of 8f1eeecf-4832-4430-b8e8-41a645675c58 (reboot of pcmknode-2 from 540c61a4-d351-40c7-aa60-efd445097180 by (null)): 0, rc=-7
> Jul 28 00:46:35 pcmknode-1 stonith-ng: [2624]: info: stonith_notify_client: Sending st_fence-notification to client 2629/cc57856c-5357-4343-95a9-712771f711ae
> Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: log_data_element: tengine_stonith_callback: StonithOp <remote-op state="0" st_target="pcmknode-2" st_op="reboot" />
> Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: tengine_stonith_callback: Stonith operation 2/30:3:0:2419bb70-dce6-4a0e-b649-fae2b0f21b8d: Operation timed out (-7)
> Jul 28 00:46:35 pcmknode-1 crmd: [2629]: ERROR: tengine_stonith_callback: Stonith of pcmknode-2 failed (-7)... aborting transition.
> Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: abort_transition_graph: tengine_stonith_callback:402 - Triggered transition abort (complete=0) : Stonith failed
> Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
> Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: update_abort_priority: Abort action done superceeded by restart
> Jul 28 00:46:35 pcmknode-1 crmd: [2629]: info: tengine_stonith_notify: Peer pcmknode-2 was terminated (reboot) by (null) for pcmknode-1 (ref=8f1eeecf-4832-4430-b8e8-41a645675c58): Operation timed out
> ~ snip ~
>
>
>
> pcmknode-2: /var/log/messages
> ~ snip ~
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Received ringid(192.168.1.186:620) seq 91
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Delivering 90 to 91
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Delivering MCAST message with seq 91 to pending delivery queue
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Received ringid(192.168.1.186:620) seq 92
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Delivering 91 to 92
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Delivering MCAST message with seq 92 to pending delivery queue
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] mcasted message added to pending queue
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Delivering 92 to 93
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Delivering MCAST message with seq 93 to pending delivery queue
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [CPG ? ] got procjoin message from cluster node -1147763583
> Jul 28 00:46:29 pcmknode-2 cib: [2542]: debug: cib_process_xpath: cib_query: //nvpar[@name='terminate'] does not exist
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Received ringid(192.168.1.186:620) seq 93
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] releasing messages up to and including 92
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [CPG ? ] got mcast request on 0x1b072a0
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Received ringid(192.168.1.186:620) seq 94
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Delivering 93 to 94
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Delivering MCAST message with seq 94 to pending delivery queue
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] mcasted message added to pending queue
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] releasing messages up to and including 93
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Delivering 94 to 95
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Delivering MCAST message with seq 95 to pending delivery queue
> Jul 28 00:46:29 pcmknode-2 kernel: : dlm: got connection from -1164540799
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] Received ringid(192.168.1.186:620) seq 95
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] releasing messages up to and including 94
> Jul 28 00:46:29 pcmknode-2 corosync[2535]: ? [TOTEM ] releasing messages up to and including 95
> Jul 28 00:46:29 pcmknode-2 kernel: GFS2: fsid=pcmknode:data1s.0: Joined cluster. Now mounting FS...
> -- that was the last message in the log.
>
>
>
>
> So, how can I fix my cluster? What exactly is the problem?
>
>
> Thanks,
> Benedikt
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From benedikt.koeppel at let.ethz.ch  Wed Jul 28 08:55:32 2010
From: benedikt.koeppel at let.ethz.ch (=?iso-8859-1?Q?K=F6ppel__Benedikt_=28LET=29?=)
Date: Wed, 28 Jul 2010 08:55:32 +0000
Subject: [Linux-cluster] SAN with GFS2 on RHEL 6 beta: STONITH
	right	after start
In-Reply-To: <AANLkTi=VLpogRZF2mBUbXhmm1HXJ1Zs9tQxs_y_Q1dc=@mail.gmail.com>
References: <ABD9AEFD-FAC6-4E01-A799-B312BD7BE0EF@let.ethz.ch>
	<AANLkTi=VLpogRZF2mBUbXhmm1HXJ1Zs9tQxs_y_Q1dc=@mail.gmail.com>
Message-ID: <40D20CDC-34B8-4638-A84F-F3655D71EB3C@let.ethz.ch>

Hello

> 
> Are these the Pacemaker packages that ship with RHEL?
> Because external/sbd isnt a fencing device we ship on RHEL so I've
> having trouble imagining how you're nodes are being fenced.

Yes, I use the packages from RHEL, but compiled the source RPM separately to have SBD. 



From pengyu.ut at gmail.com  Wed Jul 28 13:13:27 2010
From: pengyu.ut at gmail.com (Peng Yu)
Date: Wed, 28 Jul 2010 08:13:27 -0500
Subject: [Linux-cluster] Recommendation of SAN devices
In-Reply-To: <AANLkTimgUQnpE2q+E0Dh+1AHLbyhvOzkeYbDs-nSuEQX@mail.gmail.com>
References: <AANLkTinSoYMPyJVatsrOycXoevreXhxoKBiYXzC4XhWH@mail.gmail.com>
	<AANLkTimgUQnpE2q+E0Dh+1AHLbyhvOzkeYbDs-nSuEQX@mail.gmail.com>
Message-ID: <AANLkTimtfODwZ-mxv3EjTDFt2JL_qPbckWPZ8LbBAHwz@mail.gmail.com>

Hi,

Does anybody have any experience with any vendors of storage devices
(for performance) for cluster file system? Or could you please
recommend where I should look for relevant information? I googled, but
there are just so much information that I don't know where to start
with.

On Mon, Jul 26, 2010 at 2:40 PM, Peng Yu <pengyu.ut at gmail.com> wrote:
> Hi,
>
> Essentially, I want to build a cluster where multiple machines
> (multi-core) access the same cluster file system (say using GFS). Each
> machine might
> concurrently access the same file system for write or read. The
> processes running on these machines are I/O intensity. I think that if
> each machine could use hundreds of MB/s I/O. I want the whole system
> be scalable in the sense that
> more machines can be added without making the I/O the bottleneck.
>
> The SRX model in the following webpage is a choice. However, I'd like
> to see all the possible choices from different vendors so that I can
> make a more informative decision. Could you please share me some
> insights that you have or point me to a mailing list where I might get
> some answers?
>
> http://www.coraid.com/products/index.html
>
> --
> Regards,
> Peng
>



-- 
Regards,
Peng



From gordan at bobich.net  Wed Jul 28 13:28:39 2010
From: gordan at bobich.net (Gordan Bobic)
Date: Wed, 28 Jul 2010 14:28:39 +0100
Subject: [Linux-cluster] Recommendation of SAN devices
In-Reply-To: <AANLkTimtfODwZ-mxv3EjTDFt2JL_qPbckWPZ8LbBAHwz@mail.gmail.com>
References: <AANLkTinSoYMPyJVatsrOycXoevreXhxoKBiYXzC4XhWH@mail.gmail.com>
	<AANLkTimgUQnpE2q+E0Dh+1AHLbyhvOzkeYbDs-nSuEQX@mail.gmail.com>
	<AANLkTimtfODwZ-mxv3EjTDFt2JL_qPbckWPZ8LbBAHwz@mail.gmail.com>
Message-ID: <1280323719.2724.4.camel@thinkpad>

Unless you are working with very few very large files with low frequency
of opening/closing those files, your bottleneck isn't going to be the
SAN or the latency of the disk array, it'll be the latency between your
cluster nodes.

If you want high performance, you'll have to design your application to
play nice with performance bottlenecks of cluster file systems. Here are
a couple threads from the archives of this mailing lists to get you
started.

http://www.mail-archive.com/linux-cluster at redhat.com/msg04412.html
http://www.mail-archive.com/linux-cluster at redhat.com/msg08177.html

Gordan

On Wed, 2010-07-28 at 08:13 -0500, Peng Yu wrote:
> Hi,
> 
> Does anybody have any experience with any vendors of storage devices
> (for performance) for cluster file system? Or could you please
> recommend where I should look for relevant information? I googled, but
> there are just so much information that I don't know where to start
> with.
> 
> On Mon, Jul 26, 2010 at 2:40 PM, Peng Yu <pengyu.ut at gmail.com> wrote:
> > Hi,
> >
> > Essentially, I want to build a cluster where multiple machines
> > (multi-core) access the same cluster file system (say using GFS). Each
> > machine might
> > concurrently access the same file system for write or read. The
> > processes running on these machines are I/O intensity. I think that if
> > each machine could use hundreds of MB/s I/O. I want the whole system
> > be scalable in the sense that
> > more machines can be added without making the I/O the bottleneck.
> >
> > The SRX model in the following webpage is a choice. However, I'd like
> > to see all the possible choices from different vendors so that I can
> > make a more informative decision. Could you please share me some
> > insights that you have or point me to a mailing list where I might get
> > some answers?
> >
> > http://www.coraid.com/products/index.html
> >
> > --
> > Regards,
> > Peng
> >
> 
> 
> 




From jeff.sturm at eprize.com  Wed Jul 28 13:43:47 2010
From: jeff.sturm at eprize.com (Jeff Sturm)
Date: Wed, 28 Jul 2010 09:43:47 -0400
Subject: [Linux-cluster] Recommendation of SAN devices
In-Reply-To: <AANLkTimtfODwZ-mxv3EjTDFt2JL_qPbckWPZ8LbBAHwz@mail.gmail.com>
References: <AANLkTinSoYMPyJVatsrOycXoevreXhxoKBiYXzC4XhWH@mail.gmail.com><AANLkTimgUQnpE2q+E0Dh+1AHLbyhvOzkeYbDs-nSuEQX@mail.gmail.com>
	<AANLkTimtfODwZ-mxv3EjTDFt2JL_qPbckWPZ8LbBAHwz@mail.gmail.com>
Message-ID: <64D0546C5EBBD147B75DE133D798665F06A11D8A@hugo.eprize.local>

> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com]
> On Behalf Of Peng Yu
> Sent: Wednesday, July 28, 2010 9:13 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] Recommendation of SAN devices
> 
> Hi,
> 
> Does anybody have any experience with any vendors of storage devices
> (for performance) for cluster file system? Or could you please
> recommend where I should look for relevant information? I googled, but
> there are just so much information that I don't know where to start
> with.

Be prepared to do a lot of testing.  The Coraid SRX may be a good choice
(we have a few of those) but there are many to choose from.  Dell
EqualLogic iSCSI enclosures also work well.

Your results may depend as much or more on what drives you choose than
the enclosure.  10K SATA disks might give you the most bang for the
buck, 15K SAS should give you more performance.  Depending on how much
capacity you need, SSD might also be a good choice.

Don't take any vendor's word for it.  See if you can get a few on loan
and perform your disk benchmarks before you buy.

-Jeff





From agk at redhat.com  Wed Jul 28 16:41:12 2010
From: agk at redhat.com (Alasdair G Kergon)
Date: Wed, 28 Jul 2010 17:41:12 +0100
Subject: [Linux-cluster] lvm2-cluster (clvmd) security fix (Moderate)
Message-ID: <20100728164112.GG15417@agk-dp.fab.redhat.com>

I have released version 2.02.72 of LVM2 today to address a security problem.

  ftp://sources.redhat.com/pub/lvm2/LVM2.2.02.72.tgz
  ftp://sources.redhat.com/pub/lvm2/LVM2.2.02.72.tgz.asc

If you are running clvmd on a machine where you also have non-root users
you should seriously consider applying the patch,   The problem has been
in the code base since 2004.  LVM2 on its own is not vulnerable to this
problem.

Vendors were notified last week.

Red Hat's Security Advisories are here:

   https://rhn.redhat.com/errata/RHSA-2010-0567.html
   https://rhn.redhat.com/errata/RHSA-2010-0568.html

Fuller details are in the Red Hat Bugzilla:
   https://bugzilla.redhat.com/CVE-2010-2526

The essential part of the patch is attached there and it should apply to
older releases too.

Alasdair
-- 
agk at redhat.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100728/f939fb45/attachment.sig>

From eschneid at uccs.edu  Wed Jul 28 18:47:53 2010
From: eschneid at uccs.edu (Eric Schneider)
Date: Wed, 28 Jul 2010 12:47:53 -0600
Subject: [Linux-cluster] Cluster services die when nonactive node
	is	rebooted
In-Reply-To: <AANLkTin1Ph3LB=qpEgs6qnhKKBQR5bGNu+SGfOk7HnA3@mail.gmail.com>
References: <00e901cb2ab5$43bedc60$cb3c9520$@uccs.edu>
	<AANLkTin1Ph3LB=qpEgs6qnhKKBQR5bGNu+SGfOk7HnA3@mail.gmail.com>
Message-ID: <015c01cb2e85$6352a600$29f7f200$@uccs.edu>

I made the change and I will try it today during our scheduled maintenance.

 

Thanks,

 

Eric 

 

From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of umesh susvirkar
Sent: Sunday, July 25, 2010 12:18 AM
To: linux clustering
Subject: Re: [Linux-cluster] Cluster services die when nonactive node is
rebooted

 

 

Try to set following in you cluster.conf file

 

<cman expected_votes="3" quorum_dev_poll="35000" >

 

                <multicast addr="224.0.0.1" interface="eth0"/>

 

        </cman>

 

---

cal for 

quorum_dev_poll > (interval * tko ) 

 

as per below 5*6 = 30 so 35

<quorumd interval="5" label="delta_qdisk" min_score="1" tko="6" votes="1">

 

                <heuristic interval="5" program="ping -t1 -c1 192.168.1.1"
score="1"/>

 

        </quorumd>

 

 

for more info read following doc

https://access.redhat.com/kb/docs/DOC-2882

http://people.redhat.com/lhh/cmanvsqdisk.png

 

 

On Sat, Jul 24, 2010 at 3:50 AM, Eric Schneider <eschneid at uccs.edu> wrote:

I have a few 2 node clusters and I notice that recently the clusters lose
quorum when I reboot the node without running services.  I could do this in
the past without any problems.  CentOS 5.5 on ESX 4.0 u1.  Maybe a bug with
a new kernel or cman software?

 

I get the following right away when the node reboots:

Jul 23 16:02:32 happy5 clurgmgrd[4269]: <notice> Member 2 shutting down

Jul 23 16:02:52 happy5 qdiskd[3562]: <info> Node 2 shutdown

Jul 23 16:03:02 happy5 qdiskd[3562]: <info> Assuming master role

Jul 23 16:03:03 happy5 clurgmgrd[4269]: <emerg> #1: Quorum Dissolved

Jul 23 16:03:03 happy5 openais[3533]: [CMAN ] lost contact with quorum
device

Jul 23 16:03:03 happy5 openais[3533]: [CMAN ] quorum lost, blocking activity

Jul 23 16:03:03 happy5 ccsd[3493]: Cluster is not quorate.  Refusing
connection.

Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing connect:
Connection refused

Jul 23 16:03:03 happy5 ccsd[3493]: Cluster is not quorate.  Refusing
connection.

Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing connect:
Connection refused

Jul 23 16:03:03 happy5 ccsd[3493]: Invalid descriptor specified (-111).

Jul 23 16:03:03 happy5 ccsd[3493]: Someone may be attempting something evil.

Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing get: Invalid
request descriptor

Jul 23 16:03:03 happy5 ccsd[3493]: Invalid descriptor specified (-111).

Jul 23 16:03:03 happy5 ccsd[3493]: Someone may be attempting something evil.

Jul 23 16:03:03 happy5 ccsd[3493]: Error while processing get: Invalid
request descriptor

 

<?xml version="1.0"?>

<cluster alias="delta_cluster" config_version="40" name="delta_cluster">

        <fence_daemon post_fail_delay="5" post_join_delay="120"/>

        <quorumd interval="5" label="delta_qdisk" min_score="1" tko="6"
votes="1">

                <heuristic interval="5" program="ping -t1 -c1 192.168.1.1"
score="1"/>

        </quorumd>

        <clusternodes>

                <clusternode name="node1" nodeid="1" votes="1">

                        <fence>

                                <method name="1">

                                        <device name="node1"/>

                                </method>

                        </fence>

                </clusternode>

                <clusternode name="node2" nodeid="2" votes="1">

                        <fence>

                                <method name="1">

                                        <device name="node2"/>

                                </method>

                        </fence>

                </clusternode>

        </clusternodes>

        <cman expected_votes="3">

                <multicast addr="224.0.0.1" interface="eth0"/>

        </cman>

        <fencedevices>

                <fencedevice agent="fence_manual" name="fence_manual"/>

                <fencedevice agent="fence_vmware" ipaddr="bob"
login="username" name="node1" passwd="password" port="node1"/>

                <fencedevice agent="fence_vmware" ipaddr="bob"
login="username" name="node2" passwd="password" port="node2"/>

        </fencedevices>

        <rm>

                <failoverdomains>

                        <failoverdomain name="node1" ordered="0"
restricted="1">

                                <failoverdomainnode name="node1"
priority="1"/>

                        </failoverdomain>

                        <failoverdomain name="node2" restricted="1">

                                <failoverdomainnode name="node2"
priority="1"/>

                        </failoverdomain>

                        <failoverdomain name="failover_pro-http"
restricted="0">

                                <failoverdomainnode name="node1"
priority="1"/>

                                <failoverdomainnode name="node2"
priority="1"/>

                        </failoverdomain>

                </failoverdomains>

             

        </rm>

        <totem token="21000"/>

</cluster>

 

Thanks,

 

Eric 

 


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100728/c189a1f2/attachment.htm>

From benedikt.koeppel at let.ethz.ch  Wed Jul 28 19:44:00 2010
From: benedikt.koeppel at let.ethz.ch (=?iso-8859-1?Q?K=F6ppel__Benedikt_=28LET=29?=)
Date: Wed, 28 Jul 2010 19:44:00 +0000
Subject: [Linux-cluster] SAN with GFS2 on RHEL 6 beta:
	STONITH	right	after start
In-Reply-To: <40D20CDC-34B8-4638-A84F-F3655D71EB3C@let.ethz.ch>
References: <ABD9AEFD-FAC6-4E01-A799-B312BD7BE0EF@let.ethz.ch>
	<AANLkTi=VLpogRZF2mBUbXhmm1HXJ1Zs9tQxs_y_Q1dc=@mail.gmail.com>
	<40D20CDC-34B8-4638-A84F-F3655D71EB3C@let.ethz.ch>
Message-ID: <E6A90FC4-46C2-400E-92DC-4A66B484BB90@let.ethz.ch>

OK, with the help of Andrew, I tried it again.

Some important logs from the problem:

~snip~

1317 Jul 28 00:46:31 pcmknode-1 corosync[2618]: [TOTEM ] A processor failed, forming new configuration.
1318 Jul 28 00:46:32 pcmknode-1 kernel: dlm: closing connection to node -1147763583
1319 Jul 28 00:46:32 pcmknode-1 corosync[2618]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 624: memb=1, new=0, lost=1
1320 Jul 28 00:46:32 pcmknode-1 corosync[2618]: [pcmk ] info: pcmk_peer_update: memb: pcmknode-1 3130426497
1321 Jul 28 00:46:32 pcmknode-1 corosync[2618]: [pcmk ] info: pcmk_peer_update: lost: pcmknode-2 3147203713

~snip~

1338 Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: crm_update_peer: Node pcmknode-2: id=3147203713 state=lost (new) addr=r(0) ip(192.168.150.187) votes=1 born=620 seen=620 proc=00000000000000000000000000111312
1339 Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: erase_node_from_join: Removed node pcmknode-2 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1
1340 Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: crm_update_quorum: Updating quorum status to false (call=45)

~snip~

1351 Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: pe_fence_node: Node pcmknode-2 will be fenced because it is un-expectedly down
1352 Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: determine_online_status_fencing: ha_state=active, ccm_state=false, crm_state=online, join_state=member, expected=member
1353 Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: determine_online_status: Node pcmknode-2 is unclean

~snip~




I then removed the LVM from /dev/sdb2 and created the GFS2 right on /dev/sdb2 
(without LVM). That does not solve the problem.

Starting corosync on only one node works fine, even the GFS2 disk can get 
mounted. But as soon as the GFS2 disk will be mounted on the second node, the 
node gets fenced immediately. I set WebFSClone to target Stopped, and as soon 
as I manually started it again, the node got fenced.
Manually mounting the GFS2 disk (with mount -t gfs2...) on the second node also 
causes the STONITH.

One word about my STONITH: It is SBD which runs via /dev/sdb1. I got the 
cluster-glue SRPM from clusterlabs and extracted it, to manually compile SBD 
(only SBD, nothing else). I then installed SBD. So my system runs all packages 
from those repositories: RHEL 6 beta, EPEL, Clusterlabs.



I monitored the network traffic with tcpdump and analyzed it afterwards with 
Wireshark. The two DLMs are communicating, but I don't know if probably 
something goes wrong there. I see there a packet going from pcmknode-2 to 
pcmknode-1, with that content (by Wireshark): (some lines omitted which I think 
are not interesting, can provide them if needed)
Command: message (1)
Message Type: lookup message (11)
External Flags: 0x08, Return the contents of the lock value block
Status: Unknown (0)
Granted Mode: invalid (-1)
Request Mode: exclusive (5)

And then the response from pcmknode-1 to pcmknode-2:
Command: message (1)
Message Type: request reply (5)
External Flags: 0x08, Return the contents of the lock value block
Status: granted (2)
Granted Mode: exclusive (5)
Request Mode: invalid (-1)

I wonder why pcmknode-1 says "Granted: exclusive" to pcmknode-2. 
Immediately after the request reply, the pcmknode-2 writes to log "Now mounting 
FS..." and gets fenced and shut down.


So, is there probably something wrong with the DLM?



Regards,
Benedikt



From Wahyu.Darmawan at ag-it.com  Thu Jul 29 06:20:39 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Thu, 29 Jul 2010 13:20:39 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <0FE1B519D219B64A87E99E3620238545052C1AE4CB@svr-e2k7.ag-it.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
	<AANLkTikdqYmBW3xDvraban3CIdexrZMqtUQtKIbXNtCO@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854505172482F1@svr-e2k7.ag-it.com>
	<AANLkTi=Wj4=2m+ufnNAMiOUxh8WqnbCDZY_ySK5dNL2o@mail.gmail.com>
	<0FE1B519D219B64A87E99E3620238545052C1AE4CB@svr-e2k7.ag-it.com>
Message-ID: <0FE1B519D219B64A87E99E3620238545052C1AE97D@svr-e2k7.ag-it.com>

Hi all,
Does anyone can help me?
Many thanks for your help.

Regards,


From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Wahyu Darmawan
Sent: Wednesday, July 28, 2010 8:50 AM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi Ball ;),
Thanks for your response. Please find the cluster.conf below.

<?xml version="1.0"?>
<cluster alias="DBAHMPS" config_version="2" name="TAI123">
                <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
                <clusternodes>
                                <clusternode name="node1.example.com" nodeid="1" votes="1">
                                                <fence>
                                                                <method name="1">
                                                                                <device domain="RHEL" name="VM_Fence"/>
                                                                </method>
                                                </fence>
                                </clusternode>
                                <clusternode name="node2.example.com" nodeid="2" votes="1">
                                                <fence>
                                                                <method name="1">
                                                                                <device domain="RHEL2" name="VM_Fence"/>
                                                                </method>
                                                </fence>
                                </clusternode>
                </clusternodes>
                <cman expected_votes="1" two_node="1"/>
                <fencedevices>
                                <fencedevice agent="fence_xvm" name="VM_Fence"/>
                </fencedevices>
                <rm>
                                <failoverdomains>
                                                <failoverdomain name="IP_Virtual" ordered="1" restricted="1">
                                                                <failoverdomainnode name="node1.example.com" priority="1"/>
                                                                <failoverdomainnode name="node2.example.com" priority="1"/>
                                                </failoverdomain>
                                </failoverdomains>
                                <resources>
                                                <ip address="192.168.122.100/24" monitor_link="1"/>
                                                <fs device="/dev/mapper/VolGroup00-LogVol00" force_fsck="0" force_unmount="0" fsid="43719" fstype="ext3" mountpoint="/var/www/html/" name="HTML Doc" options="" self_fence="0"/>
                                                <script file="/etc/init.d/httpd" name="Web"/>
                                                <apache config_file="/etc/httpd/conf/httpd.conf" httpd_options="" name="httpd" server_root="/etc/httpd" shutdown_wait="3"/>
                                                <oracledb home="/u01/app/oracle/product/9.2.0.8/" name="DBAHMPS" type="10g" user="oracle" vhost="dbahmps.example.com"/>
                                </resources>
                                <service autostart="1" exclusive="1" name="HTTPD_Service" recovery="relocate">
                                                <ip ref="192.168.122.100/24"/>
                                                <fs ref="HTML Doc"/>
                                                <script ref="Web"/>
                                </service>
                                <service autostart="1" domain="IP_Virtual" name="Web" recovery="restart">
                                                <fs ref="HTML Doc"/>
                                                <script ref="Web"/>
                                </service>
                                <service autostart="1" domain="IP_Virtual" exclusive="1" name="Apache Service" nfslock="1">
                                                <apache ref="httpd"/>
                                </service>
                </rm>
</cluster>

Appreciating for your help Ball.


Rgds,
Wahyu

From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of POWERBALL ONLINE
Sent: Saturday, July 24, 2010 12:35 AM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

I though that your configuration is wrong because when you run clustat it should display quorum disk is online too.
If your configuration wrong and quorum not online it will have the error that you meet.
Please give me the file /etc/cluster/cluster.conf to me too.



On Fri, Jul 23, 2010 at 10:18 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi,
Thanks for your response.
Here is the info on below.


T0171
[root at t0171 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  t0171.example.com<http://t0171.example.com/>                    Online, Local, rgmanager
  t0175.example.com<http://t0175.example.com/>                    Online, rgmanager

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  IP_Virtual           t0175.example.com<http://t0175.example.com/>          started
  GFS_Mounting         t0175.example.com<http://t0175.example.com/>          started
  Listener_Service     t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBahmcm       t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBportal      t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBcard5       t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBhrdtxn      t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBrepwm       t0175.example.com<http://t0175.example.com/>          started
[root at t0171 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171.example.com<http://t0171.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/sdi2:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-18:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171.example.com<http://t0171.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-19:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512


T0175
[root at t0175 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  t0171.example.com<http://t0171.example.com/>                    Online, rgmanager
  t0175.example.com<http://t0175.example.com/>                    Online, Local, rgmanager

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  IP_Virtual           t0175. example.com<http://example.com/>          started
  GFS_Mounting         t0175. example.com<http://example.com/>          started
  Listener_Service     t0175. example.com<http://example.com/>          started
  Oracle_DBahmcm       t0175. example.com<http://example.com/>          started
  Oracle_DBportal      t0175. example.com<http://example.com/>          started
  Oracle_DBcard5       t0175. example.com<http://example.com/>          started
  Oracle_DBhrdtxn      t0175. example.com<http://example.com/>          started
  Oracle_DBrepwm       t0175. example.com<http://example.com/>          started
[root at t0175 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/sdi2:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-19:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-20:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512


Thanks & Regards,
WD

From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Friday, July 23, 2010 12:29 AM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Please give me the information after you run mkqdisk -L and clustat -i command
On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi,
So, what your suggestion?

Thanks,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Wahyu Darmawan
Sent: Thursday, July 22, 2010 11:02 PM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Yes correct. I use Conga for my clusterware.

Thanks,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Thursday, July 22, 2010 10:54 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi

What tool you use for create cluster? Is it Conga ?



On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi all,
I have problem with my cluster. I have 2 nodes of my cluster.
On July 12 the nodes missed  connection from storage and lost 1 disk partition on both nodes. Then I decided to assigned a new of partition from storage, and use it on my nodes.
But until today, the quorum is not going to quorate.
FYI, I use GFS for my filesystem.
So, what do I have to do to make quorum disk back to quorate status?

I appreciate for your answer.

Thanks,
Wahyu

Note, I attached the cluster log messages below:
Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is disabled
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service Oracle_DBrepwm
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is disabled
Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBahmcm
Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm started
Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBportal
Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal started
Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBcard5
Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5 started
Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBhrdtxn
Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn started
Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBrepwm
Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm started
Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)
Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.310000)
Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.400000)
Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.690000)
Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.620000)
Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.020000)
Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.550000)
Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.050000)
Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.710000)
Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:05:30 t0175 last message repeated 2 times
Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.530000)
Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.200000)
Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.820000)
Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.420000)
Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.820000)
Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.540000)
Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.740000)
Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.020000)
Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.900000)
Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.270000)
Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.980000)
Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.470000)
Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.870000)
Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.940000)
Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.780000)
Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.360000)
Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.790000)
Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.730000)
Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.100000)
Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.580000)
Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.460000)
Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.790000)
Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.240000)
Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.670000)
Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.570000)
Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.000000)
Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.670000)
Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 18 02:04:31 t0175 last message repeated 3 times
Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.870000)
Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.920000)
Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.040000)
Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.350000)
Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.450000)
Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.450000)
Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.120000)
Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.480000)
Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.960000)
Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.030000)
Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.940000)
Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.560000)


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100729/a7432807/attachment.htm>

From andrew at beekhof.net  Thu Jul 29 09:01:09 2010
From: andrew at beekhof.net (Andrew Beekhof)
Date: Thu, 29 Jul 2010 11:01:09 +0200
Subject: [Linux-cluster] SAN with GFS2 on RHEL 6 beta: STONITH right
	after start
In-Reply-To: <E6A90FC4-46C2-400E-92DC-4A66B484BB90@let.ethz.ch>
References: <ABD9AEFD-FAC6-4E01-A799-B312BD7BE0EF@let.ethz.ch>
	<AANLkTi=VLpogRZF2mBUbXhmm1HXJ1Zs9tQxs_y_Q1dc=@mail.gmail.com>
	<40D20CDC-34B8-4638-A84F-F3655D71EB3C@let.ethz.ch>
	<E6A90FC4-46C2-400E-92DC-4A66B484BB90@let.ethz.ch>
Message-ID: <AANLkTi=anZ6aru5rRS15BNVo3Rrz=uojyv7=kME2Zqrv@mail.gmail.com>

On Wed, Jul 28, 2010 at 9:44 PM, K?ppel  Benedikt (LET)
<benedikt.koeppel at let.ethz.ch> wrote:
> OK, with the help of Andrew, I tried it again.
>
> Some important logs from the problem:
>
> ~snip~
>
> 1317 Jul 28 00:46:31 pcmknode-1 corosync[2618]: [TOTEM ] A processor failed, forming new configuration.
> 1318 Jul 28 00:46:32 pcmknode-1 kernel: dlm: closing connection to node -1147763583
> 1319 Jul 28 00:46:32 pcmknode-1 corosync[2618]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 624: memb=1, new=0, lost=1
> 1320 Jul 28 00:46:32 pcmknode-1 corosync[2618]: [pcmk ] info: pcmk_peer_update: memb: pcmknode-1 3130426497
> 1321 Jul 28 00:46:32 pcmknode-1 corosync[2618]: [pcmk ] info: pcmk_peer_update: lost: pcmknode-2 3147203713
>
> ~snip~
>
> 1338 Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: crm_update_peer: Node pcmknode-2: id=3147203713 state=lost (new) addr=r(0) ip(192.168.150.187) votes=1 born=620 seen=620 proc=00000000000000000000000000111312
> 1339 Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: erase_node_from_join: Removed node pcmknode-2 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1
> 1340 Jul 28 00:46:32 pcmknode-1 crmd: [2629]: info: crm_update_quorum: Updating quorum status to false (call=45)
>
> ~snip~
>
> 1351 Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: pe_fence_node: Node pcmknode-2 will be fenced because it is un-expectedly down
> 1352 Jul 28 00:46:32 pcmknode-1 pengine: [2628]: info: determine_online_status_fencing: ha_state=active, ccm_state=false, crm_state=online, join_state=member, expected=member
> 1353 Jul 28 00:46:32 pcmknode-1 pengine: [2628]: WARN: determine_online_status: Node pcmknode-2 is unclean
>
> ~snip~
>
>
>
>
> I then removed the LVM from /dev/sdb2 and created the GFS2 right on /dev/sdb2
> (without LVM). That does not solve the problem.
>
> Starting corosync on only one node works fine, even the GFS2 disk can get
> mounted. But as soon as the GFS2 disk will be mounted on the second node, the
> node gets fenced immediately. I set WebFSClone to target Stopped, and as soon
> as I manually started it again, the node got fenced.
> Manually mounting the GFS2 disk (with mount -t gfs2...) on the second node also
> causes the STONITH.
>
> One word about my STONITH: It is SBD which runs via /dev/sdb1. I got the
> cluster-glue SRPM from clusterlabs and extracted it, to manually compile SBD
> (only SBD, nothing else). I then installed SBD. So my system runs all packages
> from those repositories: RHEL 6 beta, EPEL, Clusterlabs.
>
>
>
> I monitored the network traffic with tcpdump and analyzed it afterwards with
> Wireshark. The two DLMs are communicating, but I don't know if probably
> something goes wrong there. I see there a packet going from pcmknode-2 to
> pcmknode-1, with that content (by Wireshark): (some lines omitted which I think
> are not interesting, can provide them if needed)
> Command: message (1)
> Message Type: lookup message (11)
> External Flags: 0x08, Return the contents of the lock value block
> Status: Unknown (0)
> Granted Mode: invalid (-1)
> Request Mode: exclusive (5)
>
> And then the response from pcmknode-1 to pcmknode-2:
> Command: message (1)
> Message Type: request reply (5)
> External Flags: 0x08, Return the contents of the lock value block
> Status: granted (2)
> Granted Mode: exclusive (5)
> Request Mode: invalid (-1)
>
> I wonder why pcmknode-1 says "Granted: exclusive" to pcmknode-2.

No idea, I don't have much to do with the DLM.

> Immediately after the request reply, the pcmknode-2 writes to log "Now mounting
> FS..." and gets fenced and shut down.

As I explained on IRC yesterday, the node getting fenced is not the issue here.

For some reason mounting the GFS volume is causing the node to fail -
this is the root cause.
Almost all the gfs and dlm code is shared between cman and pacemaker -
so its quite possible that the dlm has an issue.

Perhaps file a bug against the dlm.
>
>
> So, is there probably something wrong with the DLM?
>
>
>
> Regards,
> Benedikt
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From oheinz at fbihome.de  Thu Jul 29 10:52:30 2010
From: oheinz at fbihome.de (Oliver Heinz)
Date: Thu, 29 Jul 2010 12:52:30 +0200
Subject: [Linux-cluster] GFS2 lockf(): Function not implemented
Message-ID: <201007291252.30522.oheinz@fbihome.de>

Hi,

I'm trying to get a pacemaker/gfs2 setup working. Everything works good so 
far, but when applications try to do locks on the shared gfs2 volumes I get 
"function not implemented".


For testing purposes I used the locktest script here: 
http://nfsv4.bullopensource.org/tools/tests/locktest.php

root at server-c:~# mount 
...
/dev/dm-12 on /srv/data/shared type gfs2 (rw,relatime,hostdata=jid=1)

root at server-c:~# gfs_control -n ls
gfs mountgroups
name          pcmk
id            0xecab71b1
flags         0x00000008 mounted
change        member 2 joined 1 remove 0 failed 0 seq 1,1
members       50331658 67108874 
all nodes
nodeid 50331658 jid 1 member 1 failed 0 start 1 seq_add 1 seq_rem 0 mount done
nodeid 67108874 jid 0 member 1 failed 0 start 1 seq_add 1 seq_rem 0 mount done

root at server-c:~# gfs2_tool sb /dev/mapper/data-shared all
  mh_magic = 0x01161970
  mh_type = 1
  mh_format = 100
  sb_fs_format = 1801
  sb_multihost_format = 1900
  sb_bsize = 4096
  sb_bsize_shift = 12
  no_formal_ino = 2
  no_addr = 23
  no_formal_ino = 1
  no_addr = 22
  sb_lockproto = lock_dlm
  sb_locktable = pcmk:pcmk
  uuid = 0E4C5945-EDAB-761C-4259-D0AA2C011189



syslog:
[74368.912580] GFS2: fsid=: Trying to join cluster "lock_dlm", "pcmk:pcmk"
[74368.989608] GFS2: fsid=pcmk:pcmk.1: Joined cluster. Now mounting FS...
[74369.293334] GFS2: fsid=pcmk:pcmk.1: jid=1, already locked for use
[74369.361891] GFS2: fsid=pcmk:pcmk.1: jid=1: Looking at journal...
[74369.434188] GFS2: fsid=pcmk:pcmk.1: jid=1: Done


So everything seems to be fine, but locking files fith lockf/fcntl seems to fail. 
as far as I understand whether distributed locking or lokal locking is used 
should be unnoticed by the application, right?

Kernel is right now a vserver and gresecurity patched 2.6.32.16-grsec2.2.0-
vs2.3.0.36.29.4 but the problems occur with with the current Debian 
squeeze/sid Kernels.
redhat-cluster/gfs2 version is 3.0.12.
communication stack is corosync 1.2.7 and the device is a LVs


Any ideas what I did wrong?


TIA,
Oliver






From swhiteho at redhat.com  Thu Jul 29 11:27:09 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Thu, 29 Jul 2010 12:27:09 +0100
Subject: [Linux-cluster] GFS2 lockf(): Function not implemented
In-Reply-To: <201007291252.30522.oheinz@fbihome.de>
References: <201007291252.30522.oheinz@fbihome.de>
Message-ID: <1280402829.2518.11.camel@localhost>

Hi,

On Thu, 2010-07-29 at 12:52 +0200, Oliver Heinz wrote:
> Hi,
> 
> I'm trying to get a pacemaker/gfs2 setup working. Everything works good so 
> far, but when applications try to do locks on the shared gfs2 volumes I get 
> "function not implemented".
> 
> 
> For testing purposes I used the locktest script here: 
> http://nfsv4.bullopensource.org/tools/tests/locktest.php
> 
Was it trying to do mandatory locking? We don't support that. Can you
strace the failing process and also show the exact error message that
you are getting?

Steve.




From oheinz at fbihome.de  Thu Jul 29 12:55:46 2010
From: oheinz at fbihome.de (Oliver Heinz)
Date: Thu, 29 Jul 2010 14:55:46 +0200
Subject: [Linux-cluster] GFS2 lockf(): Function not implemented
In-Reply-To: <201007291345.44349.oheinz@fbihome.de>
References: <201007291252.30522.oheinz@fbihome.de>
	<1280402829.2518.11.camel@localhost>
	<201007291345.44349.oheinz@fbihome.de>
Message-ID: <201007291455.47146.oheinz@fbihome.de>


Here's the 'strace -f' which might be more useful than the plain strace 
originally attached.


Oliver


Am Donnerstag, 29. Juli 2010 13:45:43 schrieb Oliver Heinz:
> Am Donnerstag, 29. Juli 2010 13:27:09 schrieb Steven Whitehouse:
> > Hi,
> > 
> > On Thu, 2010-07-29 at 12:52 +0200, Oliver Heinz wrote:
> > > Hi,
> > > 
> > > I'm trying to get a pacemaker/gfs2 setup working. Everything works good
> > > so far, but when applications try to do locks on the shared gfs2
> > > volumes I get "function not implemented".
> > > 
> > > 
> > > For testing purposes I used the locktest script here:
> > > http://nfsv4.bullopensource.org/tools/tests/locktest.php
> > 
> > Was it trying to do mandatory locking? We don't support that. Can you
> > strace the failing process and also show the exact error message that
> > you are getting?
> 
> That's what I get:
> 
> root at server-c:~# ./locktests  -n 2 -f /srv/data/shared/locktest
> Init
> process initalization
> ....
> --------------------------------------
> 
> TEST : TRY TO WRITE ON A READ  LOCK:Master: can't set lock
> 
> : Function not implemented
> 
> Echec
> 
> : Function not implemented
> 
> strace output is attached.
> 
> 
> The real world problem I'm facing is when I try to start vservers that
> reside on the gfs2 Filesystem, strace output is also attached.
> 
> root at server-c:~# vserver -v testshared start
> secure-mount: lockf(): Function not implemented
> Failed to update mtab-file
> secure-mount: lockf(): Function not implemented
> Failed to update mtab-file
> secure-mount: lockf(): Function not implemented
> Failed to update mtab-file
> 
> 
> 
> 
> TIA,
> Oliver


-------------- next part --------------
execve("./locktests", ["./locktests", "-n", "2", "-f", "/srv/data/shared/locktest"], [/* 16 vars */]) = 0
brk(0)                                  = 0x6034d8
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3eedef5000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=21068, ...}) = 0
mmap(NULL, 21068, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3eedeef000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libpthread.so.0", O_RDONLY)  = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340Y\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=131260, ...}) = 0
mmap(NULL, 2208640, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3eedabe000
mprotect(0x7f3eedad5000, 2093056, PROT_NONE) = 0
mmap(0x7f3eedcd4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f3eedcd4000
mmap(0x7f3eedcd6000, 13184, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3eedcd6000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\355\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1432968, ...}) = 0
mmap(NULL, 3541032, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3eed75d000
mprotect(0x7f3eed8b5000, 2093056, PROT_NONE) = 0
mmap(0x7f3eedab4000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x157000) = 0x7f3eedab4000
mmap(0x7f3eedab9000, 18472, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3eedab9000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3eedeee000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3eedeed000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3eedeec000
arch_prctl(ARCH_SET_FS, 0x7f3eedeed700) = 0
mprotect(0x7f3eedab4000, 16384, PROT_READ) = 0
mprotect(0x7f3eedcd4000, 4096, PROT_READ) = 0
mprotect(0x7f3eedef7000, 4096, PROT_READ) = 0
munmap(0x7f3eedeef000, 21068)           = 0
set_tid_address(0x7f3eedeed9d0)         = 14628
set_robust_list(0x7f3eedeed9e0, 0x18)   = 0
futex(0x7fff5fd07b7c, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fff5fd07b7c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7f3eedeed700) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x7f3eedac3860, [], SA_RESTORER|SA_SIGINFO, 0x7f3eedaccf60}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7f3eedac38f0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f3eedaccf60}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
fstat(1, {st_mode=S_IFREG|0600, st_size=3104, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3eedef4000
brk(0)                                  = 0x6034d8
brk(0x6244d8)                           = 0x6244d8
brk(0x625000)                           = 0x625000
pipe([3, 4])                            = 0
pipe([5, 6])                            = 0
write(0, ".", 1)                        = 1
clone(Process 14629 attached
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f3eedeed9d0) = 14629
[pid 14628] pipe([7, 8])                = 0
[pid 14629] write(0, ".", 1 <unfinished ...>
[pid 14628] write(0, ".", 1 <unfinished ...>
[pid 14629] <... write resumed> )       = 1
[pid 14628] <... write resumed> )       = 1
[pid 14628] clone( <unfinished ...>
[pid 14629] read(5, Process 14630 attached (waiting for parent)
Process 14630 resumed (parent 14628 ready)
 <unfinished ...>
[pid 14628] <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f3eedeed9d0) = 14630
[pid 14630] write(0, ".", 1 <unfinished ...>
[pid 14628] open("/srv/data/shared/locktest", O_RDWR|O_CREAT|O_SYNC, 0600 <unfinished ...>
[pid 14630] <... write resumed> )       = 1
[pid 14628] <... open resumed> )        = 9
[pid 14630] read(7,  <unfinished ...>
[pid 14628] write(0, "TEST : TRY TO WRITE ON A READ  L"..., 36) = 36
[pid 14628] write(9, "Ceci est une phrase test \303\251crite"..., 62) = 62
[pid 14628] fcntl(9, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=0, len=0}) = -1 ENOSYS (Function not implemented)
[pid 14628] dup(2)                      = 10
[pid 14628] fcntl(10, F_GETFL)          = 0x8401 (flags O_WRONLY|O_APPEND|O_LARGEFILE)
[pid 14628] close(10)                   = 0
[pid 14628] write(2, "Master: can't set lock\n: Functio"..., 50Master: can't set lock
: Function not implemented
) = 50
[pid 14628] write(2, "Echec\n: Invalid argument\n", 25Echec
: Invalid argument
) = 25
[pid 14628] write(1, "Init\nprocess initalization\n\n----"..., 68Init
process initalization

--------------------------------------

) = 68
[pid 14628] exit_group(0)               = ?
Process 14629 detached
Process 14630 detached

From svictor.titus at gmail.com  Thu Jul 29 13:33:59 2010
From: svictor.titus at gmail.com (victor titus)
Date: Thu, 29 Jul 2010 06:33:59 -0700 (PDT)
Subject: [Linux-cluster] Invitation to connect on LinkedIn
Message-ID: <1826950122.752623.1280410439506.JavaMail.app@ech3-cdn13.prod>

LinkedIn
------------victor titus requested to add you as a connection on LinkedIn:
------------------------------------------

Marian,

I'd like to add you to my professional network on LinkedIn.

- victor

Accept invitation from victor titus
http://www.linkedin.com/e/-odgn7o-gc7nbe4w-6q/ulDuieLaAX544oVCOYcgj_GaXIys4TuLMXGmOx/blk/I2229551135_2/1BpC5vrmRLoRZcjkkZt5YCpnlOt3RApnhMpmdzgmhxrSNBszYOnPkPcj4RdjAOcz99bSlJlSdgsCtzbPoTczwQdj8Ndj4LrCBxbOYWrSlI/EML_comm_afe/

View invitation from victor titus
http://www.linkedin.com/e/-odgn7o-gc7nbe4w-6q/ulDuieLaAX544oVCOYcgj_GaXIys4TuLMXGmOx/blk/I2229551135_2/39vdjcNcjkRej8OcAALqnpPbOYWrSlI/svi/
------------------------------------------

DID YOU KNOW you can be the first to know when a trusted member of your network changes jobs? With Network Updates on your LinkedIn home page, you'll be notified as members of your network change their current position. Be the first to know and reach out!
http://www.linkedin.com/

 
------
(c) 2010, LinkedIn Corporation
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100729/7340b0a3/attachment.htm>

From paul at dugas.cc  Thu Jul 29 14:14:37 2010
From: paul at dugas.cc (Paul Dugas)
Date: Thu, 29 Jul 2010 10:14:37 -0400
Subject: [Linux-cluster] blocking dlm_lock
Message-ID: <AANLkTikofK7A4Qbiy9ydNSCpOTGDO2qDCYR=Hj7OSJBu@mail.gmail.com>

Morning all,

I have a cluster of three RHEL5-x86_64 machines (all up to date)
sharing a GFS filesystem on a Coraid AoE unit.  Last night, I shut the
whole thing down to replace batteries in a couple UPS units and bought
things back up without issue.  About an hour later, access to the
shared filesystem stalled from all three machines.  It was late so I
figured I missed something so I brought it back down an up again and
it was fine.  About 4am this morning, it did it again.  By the time I
got to the site, people were already screaming so I simply restarted
it again.  I've had some time (and coffee) now to look through the
logs and am finding little of value.  I see two anomalies but I don't
know what they mean.

The first thing I found is a number of lines like so:

  openais[3953]: [TOTEM] Retransmit List: 3eb233

The second this is a set off messages like this:

  kernel: INFO: task nfsd:3523 blocked for more than 120 seconds.

These are followed by stack dumps where dlm_lock is on top.

Some searching suggests this may be an issue with my switch.  Is that
reasonable?  Is there a way to get further diagnostics?  This cluster
has been in service for a couple years so I'm leaning toward something
being broken instead of configured wrong.  Any help would be
appreciated.

Paul
--
Paul Dugas ? 522 Black Canyon Park, Canton GA 30114 USA ?
Paul at Dugas.cc ? +1.404.932.1355



From susvirkar.3616 at gmail.com  Thu Jul 29 16:35:25 2010
From: susvirkar.3616 at gmail.com (umesh susvirkar)
Date: Thu, 29 Jul 2010 22:05:25 +0530
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <0FE1B519D219B64A87E99E3620238545052C1AE97D@svr-e2k7.ag-it.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
	<AANLkTikdqYmBW3xDvraban3CIdexrZMqtUQtKIbXNtCO@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854505172482F1@svr-e2k7.ag-it.com>
	<AANLkTi=Wj4=2m+ufnNAMiOUxh8WqnbCDZY_ySK5dNL2o@mail.gmail.com>
	<0FE1B519D219B64A87E99E3620238545052C1AE4CB@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E3620238545052C1AE97D@svr-e2k7.ag-it.com>
Message-ID: <AANLkTikD0x3yW3P7ueCfO0W17mgQbyHdAMTih-3vv7qW@mail.gmail.com>

You have attach cluster.conf file for virtual guest where you have  form
"cluster without quorum disk"

Kindly attach cluster.conf file where you have used quorum disk
Also log file where your are getting  not quorate message.
attach log file from both node of cluster.




On Thu, Jul 29, 2010 at 11:50 AM, Wahyu Darmawan
<Wahyu.Darmawan at ag-it.com>wrote:

>  Hi all,
>
> Does anyone can help me?
>
> Many thanks for your help.
>
>
>
> Regards,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *Wahyu Darmawan
> *Sent:* Wednesday, July 28, 2010 8:50 AM
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Hi Ball ;),
>
> Thanks for your response. Please find the cluster.conf below.
>
>
>
> <?xml version="1.0"?>
>
> <cluster alias="DBAHMPS" config_version="2" name="TAI123">
>
>                 <fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="3"/>
>
>                 <clusternodes>
>
>                                 <clusternode name="node1.example.com"
> nodeid="1" votes="1">
>
>                                                 <fence>
>
>                                                                 <method
> name="1">
>
>
> <device domain="RHEL" name="VM_Fence"/>
>
>                                                                 </method>
>
>                                                 </fence>
>
>                                 </clusternode>
>
>                                 <clusternode name="node2.example.com"
> nodeid="2" votes="1">
>
>                                                 <fence>
>
>                                                                 <method
> name="1">
>
>
> <device domain="RHEL2" name="VM_Fence"/>
>
>                                                                 </method>
>
>                                                 </fence>
>
>                                 </clusternode>
>
>                 </clusternodes>
>
>                 <cman expected_votes="1" two_node="1"/>
>
>                 <fencedevices>
>
>                                 <fencedevice agent="fence_xvm"
> name="VM_Fence"/>
>
>                 </fencedevices>
>
>                 <rm>
>
>                                 <failoverdomains>
>
>                                                 <failoverdomain
> name="IP_Virtual" ordered="1" restricted="1">
>
>
> <failoverdomainnode name="node1.example.com" priority="1"/>
>
>
> <failoverdomainnode name="node2.example.com" priority="1"/>
>
>                                                 </failoverdomain>
>
>                                 </failoverdomains>
>
>                                 <resources>
>
>                                                 <ip address="
> 192.168.122.100/24" monitor_link="1"/>
>
>                                                 <fs
> device="/dev/mapper/VolGroup00-LogVol00" force_fsck="0" force_unmount="0"
> fsid="43719" fstype="ext3" mountpoint="/var/www/html/" name="HTML Doc"
> options="" self_fence="0"/>
>
>                                                 <script
> file="/etc/init.d/httpd" name="Web"/>
>
>                                                 <apache
> config_file="/etc/httpd/conf/httpd.conf" httpd_options="" name="httpd"
> server_root="/etc/httpd" shutdown_wait="3"/>
>
>                                                 <oracledb
> home="/u01/app/oracle/product/9.2.0.8/" name="DBAHMPS" type="10g"
> user="oracle" vhost="dbahmps.example.com"/>
>
>                                 </resources>
>
>                                 <service autostart="1" exclusive="1"
> name="HTTPD_Service" recovery="relocate">
>
>                                                 <ip ref="
> 192.168.122.100/24"/>
>
>                                                 <fs ref="HTML Doc"/>
>
>                                                 <script ref="Web"/>
>
>                                 </service>
>
>                                 <service autostart="1" domain="IP_Virtual"
> name="Web" recovery="restart">
>
>                                                 <fs ref="HTML Doc"/>
>
>                                                 <script ref="Web"/>
>
>                                 </service>
>
>                                 <service autostart="1" domain="IP_Virtual"
> exclusive="1" name="Apache Service" nfslock="1">
>
>                                                 <apache ref="httpd"/>
>
>                                 </service>
>
>                 </rm>
>
> </cluster>
>
>
>
> Appreciating for your help Ball.
>
>
>
>
>
> Rgds,
>
> Wahyu
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *POWERBALL ONLINE
> *Sent:* Saturday, July 24, 2010 12:35 AM
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> I though that your configuration is wrong because when you run clustat it
> should display quorum disk is online too.
>
> If your configuration wrong and quorum not online it will have the error
> that you meet.
>
> Please give me the file /etc/cluster/cluster.conf to me too.
>
>
>
>
>
> On Fri, Jul 23, 2010 at 10:18 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi,
>
> Thanks for your response.
>
> Here is the info on below.
>
>
>
>
>
> T0171
>
> [root at t0171 ~]# clustat
>
> Member Status: Quorate
>
>
>
>   Member Name                              Status
>
>   ------ ----                              ------
>
>   t0171.example.com                    Online, Local, rgmanager
>
>   t0175.example.com                    Online, rgmanager
>
>
>
>   Service Name         Owner (Last)                   State
>
>   ------- ----         ----- ------                   -----
>
>   IP_Virtual           t0175.example.com          started
>
>   GFS_Mounting         t0175.example.com          started
>
>   Listener_Service     t0175.example.com          started
>
>   Oracle_DBahmcm       t0175.example.com          started
>
>   Oracle_DBportal      t0175.example.com          started
>
>   Oracle_DBcard5       t0175.example.com          started
>
>   Oracle_DBhrdtxn      t0175.example.com          started
>
>   Oracle_DBrepwm       t0175.example.com          started
>
> [root at t0171 ~]# mkqdisk -L
>
> mkqdisk v0.5.2
>
> /dev/sdi1:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/sdi2:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-18:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-19:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
>
>
> T0175
>
> [root at t0175 ~]# clustat
>
> Member Status: Quorate
>
>
>
>   Member Name                              Status
>
>   ------ ----                              ------
>
>   t0171.example.com                    Online, rgmanager
>
>   t0175.example.com                    Online, Local, rgmanager
>
>
>
>   Service Name         Owner (Last)                   State
>
>   ------- ----         ----- ------                   -----
>
>   IP_Virtual           t0175. example.com          started
>
>   GFS_Mounting         t0175. example.com          started
>
>   Listener_Service     t0175. example.com          started
>
>   Oracle_DBahmcm       t0175. example.com          started
>
>   Oracle_DBportal      t0175. example.com          started
>
>   Oracle_DBcard5       t0175. example.com          started
>
>   Oracle_DBhrdtxn      t0175. example.com          started
>
>   Oracle_DBrepwm       t0175. example.com          started
>
> [root at t0175 ~]# mkqdisk -L
>
> mkqdisk v0.5.2
>
> /dev/sdi1:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171. example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/sdi2:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175. example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-19:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171. example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-20:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
>
>
> Thanks & Regards,
>
> WD
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *POWERBALL ONLINE
> *Sent:* Friday, July 23, 2010 12:29 AM
>
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Please give me the information after you run mkqdisk -L and clustat -i
> command
>
> On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi,
>
> So, what your suggestion?
>
>
>
> Thanks,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *Wahyu Darmawan
> *Sent:* Thursday, July 22, 2010 11:02 PM
>
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Yes correct. I use Conga for my clusterware.
>
>
>
> Thanks,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *POWERBALL ONLINE
> *Sent:* Thursday, July 22, 2010 10:54 PM
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Hi
>
>
>
> What tool you use for create cluster? Is it Conga ?
>
>
>
>
>
> On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi all,
>
> I have problem with my cluster. I have 2 nodes of my cluster.
>
> On July 12 the nodes missed  connection from storage and lost 1 disk
> partition on both nodes. Then I decided to assigned a new of partition from
> storage, and use it on my nodes.
>
> But until today, the quorum is not going to quorate.
>
> FYI, I use GFS for my filesystem.
>
> So, what do I have to do to make quorum disk back to quorate status?
>
>
>
> I appreciate for your answer.
>
>
>
> Thanks,
>
> Wahyu
>
>
>
> Note, I attached the cluster log messages below:
>
> Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is
> disabled
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service
> Oracle_DBrepwm
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is
> disabled
>
> Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBahmcm
>
> Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm
> started
>
> Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBportal
>
> Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal
> started
>
> Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBcard5
>
> Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5
> started
>
> Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBhrdtxn
>
> Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn
> started
>
> Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBrepwm
>
> Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm
> started
>
> Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.120000)
>
> Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.070000)
>
> Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.310000)
>
> Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.400000)
>
> Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.690000)
>
> Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.620000)
>
> Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.020000)
>
> Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.550000)
>
> Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.050000)
>
> Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.710000)
>
> Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:05:30 t0175 last message repeated 2 times
>
> Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.530000)
>
> Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.200000)
>
> Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.820000)
>
> Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.420000)
>
> Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.820000)
>
> Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.540000)
>
> Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.740000)
>
> Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.020000)
>
> Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.900000)
>
> Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.270000)
>
> Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.980000)
>
> Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.470000)
>
> Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.870000)
>
> Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.940000)
>
> Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.780000)
>
> Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.360000)
>
> Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.790000)
>
> Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.730000)
>
> Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.100000)
>
> Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.580000)
>
> Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.460000)
>
> Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.790000)
>
> Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.240000)
>
> Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.670000)
>
> Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.570000)
>
> Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.000000)
>
> Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.670000)
>
> Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 18 02:04:31 t0175 last message repeated 3 times
>
> Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.870000)
>
> Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.920000)
>
> Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.040000)
>
> Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.350000)
>
> Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.450000)
>
> Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.450000)
>
> Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.120000)
>
> Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.480000)
>
> Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.960000)
>
> Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.030000)
>
> Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.940000)
>
> Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.560000)
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100729/88ace646/attachment.htm>

From rhurst at bidmc.harvard.edu  Thu Jul 29 17:07:12 2010
From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu)
Date: Thu, 29 Jul 2010 13:07:12 -0400
Subject: [Linux-cluster] GFS2 lockf(): Function not implemented
In-Reply-To: <201007291455.47146.oheinz@fbihome.de>
References: <201007291252.30522.oheinz@fbihome.de>
	<1280402829.2518.11.camel@localhost>	<201007291345.44349.oheinz@fbihome.de>
	<201007291455.47146.oheinz@fbihome.de>
Message-ID: <50168EC934B8D64AA8D8DD37F840F3DE0562E635C8@EVS2CCR.its.caregroup.org>

This may be a red herring, but can you check your permission bits on the file, /srv/data/shared/locktest

If an 'ls -al locktest' shows it like 'rwxr-Sr-x' with the 'S' in the permissions for the group, that indicates mandatory locking -- I know GFS does/did not support that; I have not checked since my first encounter with that limitation from October 2006, so it may not be true today or with GFS2.  Perhaps you can 'chmod g-s locktest' and try it again -- that had worked for me back then.


-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Oliver Heinz
Sent: Thursday, July 29, 2010 8:56 AM
To: linux-cluster at redhat.com
Subject: Re: [Linux-cluster] GFS2 lockf(): Function not implemented


Here's the 'strace -f' which might be more useful than the plain strace originally attached.


Oliver


Am Donnerstag, 29. Juli 2010 13:45:43 schrieb Oliver Heinz:
> Am Donnerstag, 29. Juli 2010 13:27:09 schrieb Steven Whitehouse:
> > Hi,
> > 
> > On Thu, 2010-07-29 at 12:52 +0200, Oliver Heinz wrote:
> > > Hi,
> > > 
> > > I'm trying to get a pacemaker/gfs2 setup working. Everything works good
> > > so far, but when applications try to do locks on the shared gfs2
> > > volumes I get "function not implemented".
> > > 
> > > 
> > > For testing purposes I used the locktest script here:
> > > http://nfsv4.bullopensource.org/tools/tests/locktest.php
> > 
> > Was it trying to do mandatory locking? We don't support that. Can you
> > strace the failing process and also show the exact error message that
> > you are getting?
> 
> That's what I get:
> 
> root at server-c:~# ./locktests  -n 2 -f /srv/data/shared/locktest
> Init
> process initalization
> ....
> --------------------------------------
> 
> TEST : TRY TO WRITE ON A READ  LOCK:Master: can't set lock
> 
> : Function not implemented
> 
> Echec
> 
> : Function not implemented
> 
> strace output is attached.
> 
> 
> The real world problem I'm facing is when I try to start vservers that
> reside on the gfs2 Filesystem, strace output is also attached.
> 
> root at server-c:~# vserver -v testshared start
> secure-mount: lockf(): Function not implemented
> Failed to update mtab-file
> secure-mount: lockf(): Function not implemented
> Failed to update mtab-file
> secure-mount: lockf(): Function not implemented
> Failed to update mtab-file
> 
> 
> 
> 
> TIA,
> Oliver





From dhoffutt at gmail.com  Thu Jul 29 19:18:58 2010
From: dhoffutt at gmail.com (Dustin Henry Offutt)
Date: Thu, 29 Jul 2010 14:18:58 -0500
Subject: [Linux-cluster] Creating custom OCF_RESKEY_ variables,
	now how and what to refresh?
Message-ID: <AANLkTim+WEBDoEGeqAO6h75srW_v_dgUMuE9XxgZD6=m@mail.gmail.com>

Hello,

Does anyone know how to force a cluster (the "Cluster Suite" as released
with the RHEL5.4 ISO, cman, rgmanager, et.al.) to recognize that new
OCF_RESKEY variables have been introduced in a /usr/share/cluster/ script?

On one cluster the new variables are recognized and used by all nodes. Same
exact script, another cluster, just put the updated /usr/share/cluster/
script in today, and it's like it hasn't had something "refreshed", and
doesn't see the new "rules," if that makes any sense - despite bouncing the
cluster suite and the cluster nodes.

Thank you...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100729/9e06c704/attachment.htm>

From Wahyu.Darmawan at ag-it.com  Fri Jul 30 01:31:31 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Fri, 30 Jul 2010 08:31:31 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <AANLkTikD0x3yW3P7ueCfO0W17mgQbyHdAMTih-3vv7qW@mail.gmail.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
	<AANLkTikdqYmBW3xDvraban3CIdexrZMqtUQtKIbXNtCO@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854505172482F1@svr-e2k7.ag-it.com>
	<AANLkTi=Wj4=2m+ufnNAMiOUxh8WqnbCDZY_ySK5dNL2o@mail.gmail.com>
	<0FE1B519D219B64A87E99E3620238545052C1AE4CB@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E3620238545052C1AE97D@svr-e2k7.ag-it.com>
	<AANLkTikD0x3yW3P7ueCfO0W17mgQbyHdAMTih-3vv7qW@mail.gmail.com>
Message-ID: <0FE1B519D219B64A87E99E3620238545052C1AEB0A@svr-e2k7.ag-it.com>

Hi Umesh,
Please find another history from bottom of this mail. Because I have attached log message on the bottom.

Thanks,
Wahyu

From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of umesh susvirkar
Sent: Thursday, July 29, 2010 11:35 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

You have attach cluster.conf file for virtual guest where you have  form "cluster without quorum disk"

Kindly attach cluster.conf file where you have used quorum disk
Also log file where your are getting  not quorate message.
attach log file from both node of cluster.




On Thu, Jul 29, 2010 at 11:50 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi all,
Does anyone can help me?
Many thanks for your help.

Regards,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Wahyu Darmawan
Sent: Wednesday, July 28, 2010 8:50 AM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi Ball ;),
Thanks for your response. Please find the cluster.conf below.

<?xml version="1.0"?>
<cluster alias="DBAHMPS" config_version="2" name="TAI123">
                <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
                <clusternodes>
                                <clusternode name="node1.example.com<http://node1.example.com>" nodeid="1" votes="1">
                                                <fence>
                                                                <method name="1">
                                                                                <device domain="RHEL" name="VM_Fence"/>
                                                                </method>
                                                </fence>
                                </clusternode>
                                <clusternode name="node2.example.com<http://node2.example.com>" nodeid="2" votes="1">
                                                <fence>
                                                                <method name="1">
                                                                                <device domain="RHEL2" name="VM_Fence"/>
                                                                </method>
                                                </fence>
                                </clusternode>
                </clusternodes>
                <cman expected_votes="1" two_node="1"/>
                <fencedevices>
                                <fencedevice agent="fence_xvm" name="VM_Fence"/>
                </fencedevices>
                <rm>
                                <failoverdomains>
                                                <failoverdomain name="IP_Virtual" ordered="1" restricted="1">
                                                                <failoverdomainnode name="node1.example.com<http://node1.example.com>" priority="1"/>
                                                                <failoverdomainnode name="node2.example.com<http://node2.example.com>" priority="1"/>
                                                </failoverdomain>
                                </failoverdomains>
                                <resources>
                                                <ip address="192.168.122.100/24<http://192.168.122.100/24>" monitor_link="1"/>
                                                <fs device="/dev/mapper/VolGroup00-LogVol00" force_fsck="0" force_unmount="0" fsid="43719" fstype="ext3" mountpoint="/var/www/html/" name="HTML Doc" options="" self_fence="0"/>
                                                <script file="/etc/init.d/httpd" name="Web"/>
                                                <apache config_file="/etc/httpd/conf/httpd.conf" httpd_options="" name="httpd" server_root="/etc/httpd" shutdown_wait="3"/>
                                                <oracledb home="/u01/app/oracle/product/9.2.0.8/<http://9.2.0.8/>" name="DBAHMPS" type="10g" user="oracle" vhost="dbahmps.example.com<http://dbahmps.example.com>"/>
                                </resources>
                                <service autostart="1" exclusive="1" name="HTTPD_Service" recovery="relocate">
                                                <ip ref="192.168.122.100/24<http://192.168.122.100/24>"/>
                                                <fs ref="HTML Doc"/>
                                                <script ref="Web"/>
                                </service>
                                <service autostart="1" domain="IP_Virtual" name="Web" recovery="restart">
                                                <fs ref="HTML Doc"/>
                                                <script ref="Web"/>
                                </service>
                                <service autostart="1" domain="IP_Virtual" exclusive="1" name="Apache Service" nfslock="1">
                                                <apache ref="httpd"/>
                                </service>
                </rm>
</cluster>

Appreciating for your help Ball.


Rgds,
Wahyu

From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Saturday, July 24, 2010 12:35 AM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

I though that your configuration is wrong because when you run clustat it should display quorum disk is online too.
If your configuration wrong and quorum not online it will have the error that you meet.
Please give me the file /etc/cluster/cluster.conf to me too.



On Fri, Jul 23, 2010 at 10:18 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi,
Thanks for your response.
Here is the info on below.


T0171
[root at t0171 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  t0171.example.com<http://t0171.example.com/>                    Online, Local, rgmanager
  t0175.example.com<http://t0175.example.com/>                    Online, rgmanager

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  IP_Virtual           t0175.example.com<http://t0175.example.com/>          started
  GFS_Mounting         t0175.example.com<http://t0175.example.com/>          started
  Listener_Service     t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBahmcm       t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBportal      t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBcard5       t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBhrdtxn      t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBrepwm       t0175.example.com<http://t0175.example.com/>          started
[root at t0171 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171.example.com<http://t0171.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/sdi2:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-18:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171.example.com<http://t0171.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-19:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512


T0175
[root at t0175 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  t0171.example.com<http://t0171.example.com/>                    Online, rgmanager
  t0175.example.com<http://t0175.example.com/>                    Online, Local, rgmanager

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  IP_Virtual           t0175. example.com<http://example.com/>          started
  GFS_Mounting         t0175. example.com<http://example.com/>          started
  Listener_Service     t0175. example.com<http://example.com/>          started
  Oracle_DBahmcm       t0175. example.com<http://example.com/>          started
  Oracle_DBportal      t0175. example.com<http://example.com/>          started
  Oracle_DBcard5       t0175. example.com<http://example.com/>          started
  Oracle_DBhrdtxn      t0175. example.com<http://example.com/>          started
  Oracle_DBrepwm       t0175. example.com<http://example.com/>          started
[root at t0175 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/sdi2:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-19:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-20:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512


Thanks & Regards,
WD

From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Friday, July 23, 2010 12:29 AM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Please give me the information after you run mkqdisk -L and clustat -i command
On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi,
So, what your suggestion?

Thanks,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Wahyu Darmawan
Sent: Thursday, July 22, 2010 11:02 PM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Yes correct. I use Conga for my clusterware.

Thanks,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Thursday, July 22, 2010 10:54 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi

What tool you use for create cluster? Is it Conga ?



On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi all,
I have problem with my cluster. I have 2 nodes of my cluster.
On July 12 the nodes missed  connection from storage and lost 1 disk partition on both nodes. Then I decided to assigned a new of partition from storage, and use it on my nodes.
But until today, the quorum is not going to quorate.
FYI, I use GFS for my filesystem.
So, what do I have to do to make quorum disk back to quorate status?

I appreciate for your answer.

Thanks,
Wahyu

Note, I attached the cluster log messages below:
Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is disabled
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service Oracle_DBrepwm
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is disabled
Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBahmcm
Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm started
Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBportal
Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal started
Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBcard5
Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5 started
Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBhrdtxn
Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn started
Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBrepwm
Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm started
Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)
Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.310000)
Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.400000)
Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.690000)
Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.620000)
Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.020000)
Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.550000)
Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.050000)
Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.710000)
Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:05:30 t0175 last message repeated 2 times
Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.530000)
Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.200000)
Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.820000)
Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.420000)
Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.820000)
Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.540000)
Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.740000)
Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.020000)
Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.900000)
Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.270000)
Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.980000)
Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.470000)
Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.870000)
Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.940000)
Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.780000)
Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.360000)
Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.790000)
Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.730000)
Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.100000)
Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.580000)
Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.460000)
Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.790000)
Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.240000)
Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.670000)
Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.570000)
Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.000000)
Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.670000)
Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 18 02:04:31 t0175 last message repeated 3 times
Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.870000)
Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.920000)
Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.040000)
Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.350000)
Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.450000)
Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.450000)
Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.120000)
Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.480000)
Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.960000)
Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.030000)
Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.940000)
Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.560000)


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100730/f0a3069b/attachment.htm>

From kkovachev at varna.net  Fri Jul 30 08:16:45 2010
From: kkovachev at varna.net (Kaloyan Kovachev)
Date: Fri, 30 Jul 2010 11:16:45 +0300
Subject: [Linux-cluster]
 =?utf-8?q?Creating_custom_OCF=5FRESKEY=5F_variabl?=
 =?utf-8?q?es=2C_now_how_and_what_to_refresh=3F?=
In-Reply-To: <AANLkTim+WEBDoEGeqAO6h75srW_v_dgUMuE9XxgZD6=m@mail.gmail.com>
References: <AANLkTim+WEBDoEGeqAO6h75srW_v_dgUMuE9XxgZD6=m@mail.gmail.com>
Message-ID: <0b55638253d93971945bd8eb50992eb3@mx.varna.net>

Hi,
 you should update /usr/share/cluster/cluster.rng to validate the new
options and your 'new' service/script

On Thu, 29 Jul 2010 14:18:58 -0500, Dustin Henry Offutt
<dhoffutt at gmail.com> wrote:
> Hello,
> 
> Does anyone know how to force a cluster (the "Cluster Suite" as released
> with the RHEL5.4 ISO, cman, rgmanager, et.al.) to recognize that new
> OCF_RESKEY variables have been introduced in a /usr/share/cluster/
script?
> 
> On one cluster the new variables are recognized and used by all nodes.
Same
> exact script, another cluster, just put the updated /usr/share/cluster/
> script in today, and it's like it hasn't had something "refreshed", and
> doesn't see the new "rules," if that makes any sense - despite bouncing
the
> cluster suite and the cluster nodes.
> 
> Thank you...



From fdinitto at redhat.com  Fri Jul 30 11:55:29 2010
From: fdinitto at redhat.com (Fabio M. Di Nitto)
Date: Fri, 30 Jul 2010 13:55:29 +0200
Subject: [Linux-cluster] Cluster 3.0.14 stable release
Message-ID: <4C52BDB1.4000900@redhat.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

The cluster team and its community are proud to announce the 3.0.14
stable release from the STABLE3 branch.

This release contains a few major bug fixes. We strongly recommend
people to update their clusters.

In order to build/run the 3.0.14 release you will need:

- - corosync 1.2.7
- - openais 1.1.3
- - linux kernel 2.6.31 (only for GFS1 users)

IMPORTANT: cluster 3.0.14 absolutely requires corosync 1.2.7 for 2 nodes
cluster or cman will not start, reporting an error from corosync.

The new source tarball can be downloaded here:

https://fedorahosted.org/releases/c/l/cluster/cluster-3.0.14.tar.bz2

To report bugs or issues:

   https://bugzilla.redhat.com/

Would you like to meet the cluster team or members of its community?

   Join us on IRC (irc.freenode.net #linux-cluster) and share your
   experience  with other sysadministrators or power users.

Thanks/congratulations to all people that contributed to achieve this
great milestone.

Happy clustering,
Fabio

Under the hood (from 3.0.13):

Abhijith Das (1):
      gfs2 manual pages: gfs2_convert manpage and documentation updates

Bob Peterson (4):
      gfs2_edit restoremeta should not return 0 on failure
      fsck.gfs2: unaligned access on ia64
      GFS2: libgfs2 bitfit algorithm using wrong shift point
      Make gfs2_edit show bit-to-block translation when viewing bitmaps

Carlos Eduardo Maiolino (1):
      resource-agents: Remove nfs service temp directories

Christine Caulfield (2):
      cman: Recalculate expected_votes on a config reload.
      cman: Mention cman_tool version -S in man page

David Teigland (2):
      dlm_controld: fix plock checkpoint signatures
      dlm_controld: fix plock owner in checkpoints

Fabio M. Di Nitto (17):
      resource agents: Remove bashisms from resource scripts
      cman init: more LSB compliance
      rgmanger init: more LSB complaint bits
      cman: add default config snippet for cman init script
      gfs2 init: make the init script LSB compliant
      cman config: copy all logging objects to the top level tree
      fence: rename ibmblade to bladecenter_snmp
      fence agents: add compatibility symlink for ibmblade /
bladecenter_snmp
      cman: allow init script to pass options to fenced
      config: fix several issues with reload operation
      cman-preconfig: better handle of logging reload operation
      cman-preconfig: better handle of logging reload operation (part 2)
      config: free new config if we cannot find the config_version
      config: more cman_tool config reload cleanup
      cman: simply message broadcasting handling
      config: better error report when autodetecting config version errors
      cman: fix consensus calculation

Federico Simoncelli (1):
      config: Add missing cman_label

Guido Gunther (1):
      fence_rsb: Raise exceptions not strings

Jan Friesse (1):
      cman: check for new config only once per second

Lon Hohberger (37):
      config: Add missing resource docs to cluster.rng
      config: Clean up recursion and documentation
      config: Add documentation for interface tag
      Revert "config: Clean up recursion and documentation"
      Revert "config: Add missing resource docs to cluster.rng"
      resource-agents: Add missing resource docs
      config: Add missing resource docs to cluster.rng
      config: Clean up recursion and documentation
      resource-agents: Clean up recursion and documentation
      rgmanager: Pass timeouts to resource agents
      resource-agents: Make vm.sh use stop/start timeouts
      rgmanager: Use sysrq-b to reboot
      Revert "resource-agents: Make vm.sh use stop/start timeouts"
      resource-agents: Make vm.sh use stop/start timeouts
      resource-agents: fix incorrect link resolution in fs-lib
      rgmanager: Make clustat -f not query CCS/objdb
      resource-agents: Add NFSv4 support
      resource-agents: Add NFSv4 agent to installation
      config: Update cluster schema
      config: Update cluster LDIF schema
      doc: Add auto-generated cluster.conf reference
      doc: Install cluster.conf reference
      rgmanager: Make clulog filter correctly
      resource-agents: Add resource type to logging
      rgmanager: Man page improvements
      resource-agents: Allow other values for "yes"
      config: Allow multiple logging_daemon tags
      config: Add doc for cman_label attribute
      config: Update LDIF schema
      cman: Recalculate quorum on config change
      config: Add tomcat-6 resource agent to schema
      config: Add tomcat-6 to ldif schema
      doc: Add tomcat-6 to cluster_conf.html
      cman: Recalculate quorum on quorum device vote changes
      cman: Deprecate specifying config version to cman_tool
      config: Regenerate fencing definitions for XML schema
      config: Exclude fence_scsi from generated section

Marek 'marx' Grac (8):
      fence_wti: support non-default TCP ports
      fence_apc: fence_apc fails for some port numbers
      resource-agents: Use SIGQUIT if SIGTERM was not fast enough
      resource-agents: new agent for tomcat 6
      resource-agents: change build system to include tomcat6 RA
      resource-agents: RA for psql does not work correctly with netmask
      fence_ilo: will throw exception if user does not have power
priviledges
      fencing: Not all parameters appear in metadata

Masahiro Matsuya (1):
      resource-agents: Fix migration mapping behavior w/ virsh

Ryan O'Hara (1):
      Fix open flags so that logfile won't be truncated each time we
open it.

 cman/cman_tool/main.c                              |   40 +-
 cman/daemon/cman-preconfig.c                       |  259 +++--
 cman/daemon/cmanconfig.c                           |    8 +-
 cman/daemon/commands.c                             |  194 ++--
 cman/init.d/Makefile                               |   10 +-
 cman/init.d/cman.in                                |   26 +-
 cman/init.d/cman.init.defaults.in                  |   97 ++
 cman/man/cman_tool.8                               |    7 +
 config/plugins/ldap/99cluster.ldif                 |   34 +-
 config/plugins/ldap/ldap-base.csv                  |    8 +-
 config/tools/xml/cluster.rng.in                    |  895 ++++++++-------
 doc/Makefile                                       |    3 +-
 doc/cluster_conf.html                              | 1184
++++++++++++++++++++
 fence/agents/apc/fence_apc.py                      |    4 +-
 fence/agents/bladecenter_snmp/Makefile             |    7 +
 .../bladecenter_snmp/fence_bladecenter_snmp.py     |   88 ++
 fence/agents/ibmblade/Makefile                     |   15 +-
 fence/agents/ibmblade/fence_ibmblade.8             |    1 +
 fence/agents/ibmblade/fence_ibmblade.py            |   88 --
 fence/agents/ilo/fence_ilo.py                      |    7 +-
 fence/agents/lib/fencing.py.py                     |   12 +
 fence/agents/rsb/fence_rsb.py                      |    6 +-
 fence/agents/scsi/fence_scsi.pl                    |    2 +-
 fence/agents/wti/fence_wti.py                      |    6 +-
 gfs2/edit/hexedit.c                                |   63 +-
 gfs2/edit/savemeta.c                               |    2 +-
 gfs2/fsck/fs_bits.h                                |   11 -
 gfs2/fsck/pass5.c                                  |    1 -
 gfs2/init.d/gfs2.in                                |   63 +-
 gfs2/libgfs2/fs_bits.c                             |  115 ++-
 gfs2/libgfs2/libgfs2.h                             |    5 +-
 gfs2/libgfs2/rgrp.c                                |    6 +
 gfs2/man/gfs2_convert.8                            |   38 +-
 group/dlm_controld/plock.c                         |   10 +-
 rgmanager/include/res-ocf.h                        |    1 +
 rgmanager/init.d/rgmanager.in                      |    5 +
 rgmanager/man/clulog.8                             |   18 +-
 rgmanager/man/rgmanager.8                          |  346 ++++++-
 rgmanager/src/daemons/restree.c                    |   29 +-
 rgmanager/src/daemons/watchdog.c                   |   20 +
 rgmanager/src/resources/Makefile                   |    6 +-
 rgmanager/src/resources/SAPDatabase                |    2 +-
 rgmanager/src/resources/SAPInstance                |    2 +-
 rgmanager/src/resources/nfsserver.sh               |  180 +++-
 rgmanager/src/resources/ocf-shellfuncs             |    2 +-
 rgmanager/src/resources/postgres-8.sh              |    4 +-
 rgmanager/src/resources/ra2rng.xsl                 |   17 +-
 rgmanager/src/resources/resources.rng.mid          |    3 +-
 rgmanager/src/resources/resources.rng.tail         |   18 +-
 rgmanager/src/resources/service.sh                 |   52 +-
 rgmanager/src/resources/svclib_nfslock             |    4 +-
 rgmanager/src/resources/tomcat-6.metadata          |   74 ++
 rgmanager/src/resources/tomcat-6.sh                |  222 ++++
 rgmanager/src/resources/utils/config-utils.sh.in   |    4 +-
 rgmanager/src/resources/utils/fs-lib.sh            |   16 +-
 rgmanager/src/resources/utils/messages.sh          |    8 +
 rgmanager/src/resources/utils/ra-skelet.sh         |   40 +-
 rgmanager/src/resources/vm.sh                      |  150 ++-
 rgmanager/src/utils/clulog.c                       |   41 +-
 rgmanager/src/utils/clustat.c                      |    6 +-
 60 files changed, 3559 insertions(+), 1026 deletions(-)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJMUr2vAAoJEFA6oBJjVJ+OohQP/2O51DLeNcIhI9n8NltsoARx
knNtfuGJLu8H/PoIN0WiAeM/S96o9L/WKZzF4/JWHvZ9aQIbv8i/VGwAMJWmsrpl
jU9VE5JqoF1fvWxJ2yEWsApfRsJxwHDtpIwOq74HDRbF4PMnrEU//WCl+NjZfdn5
kSAJm4VHuj6XnPl0SReA27dy2InOA+5rJpGfzy3zj9FTMlgQFD8KY78XIEBO1DkB
DvG7lAjbxJD3xoN59HuwTimzMxFy8hrkyWKmCg1S1bSRE9iacv+gowksYyrJAjbd
wUnuvHKGIoTV0jIKyRF/zMu/6ERZxblNcwtWNA4snMPkTcXaLcp9qrqwMjy21feC
cHHIdtsGq0N4/D3syFTz2ebXBKsk85+dZWrKtYMhxSgy+W8vooaPWRCH+UnOto94
VdLCn3kET2UeHPu1ITZpuUGxF/ZZNnmYOiZ31T9u8RxCfG/HiDWfNPhbH1eWrFhb
SOgOxfD4LCz7lUflXDICvDqRLBIR8153nQVZ589d7sl/YzhTMssJPAoKZApkzVHB
PwE60sSp//mPpMJVV1EYzhm1i4S2viVTkFKQy7Xy7f6jk/TcNBkP93B6zXDNsCXq
y1TfRI9jsJ2lundgpJslPY/syt00/j1YINbNPpFk5ohO/v4L6apepqrKU4+H6frP
y4ck+yaQYo+xbL1yIRJf
=FOvR
-----END PGP SIGNATURE-----



From susvirkar.3616 at gmail.com  Fri Jul 30 15:58:55 2010
From: susvirkar.3616 at gmail.com (umesh susvirkar)
Date: Fri, 30 Jul 2010 21:28:55 +0530
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <0FE1B519D219B64A87E99E3620238545052C1AEB0A@svr-e2k7.ag-it.com>
References: <0FE1B519D219B64A87E99E36202385450517247E53@svr-e2k7.ag-it.com>
	<AANLkTilezWEKrCobljJbxNJSV_VHbvAqUB6c7BZWVJLQ@mail.gmail.com>
	<0FE1B519D219B64A87E99E36202385450517247FCC@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E36202385450517247FCF@svr-e2k7.ag-it.com>
	<AANLkTikdqYmBW3xDvraban3CIdexrZMqtUQtKIbXNtCO@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854505172482F1@svr-e2k7.ag-it.com>
	<AANLkTi=Wj4=2m+ufnNAMiOUxh8WqnbCDZY_ySK5dNL2o@mail.gmail.com>
	<0FE1B519D219B64A87E99E3620238545052C1AE4CB@svr-e2k7.ag-it.com>
	<0FE1B519D219B64A87E99E3620238545052C1AE97D@svr-e2k7.ag-it.com>
	<AANLkTikD0x3yW3P7ueCfO0W17mgQbyHdAMTih-3vv7qW@mail.gmail.com>
	<0FE1B519D219B64A87E99E3620238545052C1AEB0A@svr-e2k7.ag-it.com>
Message-ID: <AANLkTi=3FLA5Dm9940YEAXwM+J67MiM_ukgmHwN09jBb@mail.gmail.com>

I  cannot find any log message /cluster.conf file other than following
message in mail


Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
second to complete (12.120000)

Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
second to complete (1.900000)

Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
second to complete (1.010000)

Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
second to complete (1.990000)

Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
second to complete (1.010000)

Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
second to complete (2.070000)




On Fri, Jul 30, 2010 at 7:01 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>wrote:

>  Hi Umesh,
>
> Please find another history from bottom of this mail. Because I have
> attached log message on the bottom.
>
>
>
> Thanks,
>
> Wahyu
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *umesh susvirkar
> *Sent:* Thursday, July 29, 2010 11:35 PM
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> You have attach cluster.conf file for virtual guest where you have  form
> "cluster without quorum disk"
>
>
>
> Kindly attach cluster.conf file where you have used quorum disk
>
> Also log file where your are getting  not quorate message.
>
> attach log file from both node of cluster.
>
>
>
>
>
>
>
>
>
> On Thu, Jul 29, 2010 at 11:50 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi all,
>
> Does anyone can help me?
>
> Many thanks for your help.
>
>
>
> Regards,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *Wahyu Darmawan
> *Sent:* Wednesday, July 28, 2010 8:50 AM
>
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Hi Ball ;),
>
> Thanks for your response. Please find the cluster.conf below.
>
>
>
> <?xml version="1.0"?>
>
> <cluster alias="DBAHMPS" config_version="2" name="TAI123">
>
>                 <fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="3"/>
>
>                 <clusternodes>
>
>                                 <clusternode name="node1.example.com"
> nodeid="1" votes="1">
>
>                                                 <fence>
>
>                                                                 <method
> name="1">
>
>
> <device domain="RHEL" name="VM_Fence"/>
>
>                                                                 </method>
>
>                                                 </fence>
>
>                                 </clusternode>
>
>                                 <clusternode name="node2.example.com"
> nodeid="2" votes="1">
>
>                                                 <fence>
>
>                                                                 <method
> name="1">
>
>
> <device domain="RHEL2" name="VM_Fence"/>
>
>                                                                 </method>
>
>                                                 </fence>
>
>                                 </clusternode>
>
>                 </clusternodes>
>
>                 <cman expected_votes="1" two_node="1"/>
>
>                 <fencedevices>
>
>                                 <fencedevice agent="fence_xvm"
> name="VM_Fence"/>
>
>                 </fencedevices>
>
>                 <rm>
>
>                                 <failoverdomains>
>
>                                                 <failoverdomain
> name="IP_Virtual" ordered="1" restricted="1">
>
>
> <failoverdomainnode name="node1.example.com" priority="1"/>
>
>
> <failoverdomainnode name="node2.example.com" priority="1"/>
>
>                                                 </failoverdomain>
>
>                                 </failoverdomains>
>
>                                 <resources>
>
>                                                 <ip address="
> 192.168.122.100/24" monitor_link="1"/>
>
>                                                 <fs
> device="/dev/mapper/VolGroup00-LogVol00" force_fsck="0" force_unmount="0"
> fsid="43719" fstype="ext3" mountpoint="/var/www/html/" name="HTML Doc"
> options="" self_fence="0"/>
>
>                                                 <script
> file="/etc/init.d/httpd" name="Web"/>
>
>                                                 <apache
> config_file="/etc/httpd/conf/httpd.conf" httpd_options="" name="httpd"
> server_root="/etc/httpd" shutdown_wait="3"/>
>
>                                                 <oracledb
> home="/u01/app/oracle/product/9.2.0.8/" name="DBAHMPS" type="10g"
> user="oracle" vhost="dbahmps.example.com"/>
>
>                                 </resources>
>
>                                 <service autostart="1" exclusive="1"
> name="HTTPD_Service" recovery="relocate">
>
>                                                 <ip ref="
> 192.168.122.100/24"/>
>
>                                                 <fs ref="HTML Doc"/>
>
>                                                 <script ref="Web"/>
>
>                                 </service>
>
>                                 <service autostart="1" domain="IP_Virtual"
> name="Web" recovery="restart">
>
>                                                 <fs ref="HTML Doc"/>
>
>                                                 <script ref="Web"/>
>
>                                 </service>
>
>                                 <service autostart="1" domain="IP_Virtual"
> exclusive="1" name="Apache Service" nfslock="1">
>
>                                                 <apache ref="httpd"/>
>
>                                 </service>
>
>                 </rm>
>
> </cluster>
>
>
>
> Appreciating for your help Ball.
>
>
>
>
>
> Rgds,
>
> Wahyu
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *POWERBALL ONLINE
> *Sent:* Saturday, July 24, 2010 12:35 AM
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> I though that your configuration is wrong because when you run clustat it
> should display quorum disk is online too.
>
> If your configuration wrong and quorum not online it will have the error
> that you meet.
>
> Please give me the file /etc/cluster/cluster.conf to me too.
>
>
>
>
>
> On Fri, Jul 23, 2010 at 10:18 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi,
>
> Thanks for your response.
>
> Here is the info on below.
>
>
>
>
>
> T0171
>
> [root at t0171 ~]# clustat
>
> Member Status: Quorate
>
>
>
>   Member Name                              Status
>
>   ------ ----                              ------
>
>   t0171.example.com                    Online, Local, rgmanager
>
>   t0175.example.com                    Online, rgmanager
>
>
>
>   Service Name         Owner (Last)                   State
>
>   ------- ----         ----- ------                   -----
>
>   IP_Virtual           t0175.example.com          started
>
>   GFS_Mounting         t0175.example.com          started
>
>   Listener_Service     t0175.example.com          started
>
>   Oracle_DBahmcm       t0175.example.com          started
>
>   Oracle_DBportal      t0175.example.com          started
>
>   Oracle_DBcard5       t0175.example.com          started
>
>   Oracle_DBhrdtxn      t0175.example.com          started
>
>   Oracle_DBrepwm       t0175.example.com          started
>
> [root at t0171 ~]# mkqdisk -L
>
> mkqdisk v0.5.2
>
> /dev/sdi1:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/sdi2:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-18:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-19:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
>
>
> T0175
>
> [root at t0175 ~]# clustat
>
> Member Status: Quorate
>
>
>
>   Member Name                              Status
>
>   ------ ----                              ------
>
>   t0171.example.com                    Online, rgmanager
>
>   t0175.example.com                    Online, Local, rgmanager
>
>
>
>   Service Name         Owner (Last)                   State
>
>   ------- ----         ----- ------                   -----
>
>   IP_Virtual           t0175. example.com          started
>
>   GFS_Mounting         t0175. example.com          started
>
>   Listener_Service     t0175. example.com          started
>
>   Oracle_DBahmcm       t0175. example.com          started
>
>   Oracle_DBportal      t0175. example.com          started
>
>   Oracle_DBcard5       t0175. example.com          started
>
>   Oracle_DBhrdtxn      t0175. example.com          started
>
>   Oracle_DBrepwm       t0175. example.com          started
>
> [root at t0175 ~]# mkqdisk -L
>
> mkqdisk v0.5.2
>
> /dev/sdi1:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171. example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/sdi2:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175. example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-19:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum1
>
>                 Created:              Mon Mar 29 18:44:37 2010
>
>                 Host:                 t0171. example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
> /dev/dm-20:
>
>                 Magic:                eb7a62c2
>
>                 Label:                dbquorum2
>
>                 Created:              Mon Mar 29 18:26:28 2010
>
>                 Host:                 t0175.example.com
>
>                 Kernel Sector Size:   512
>
>                 Recorded Sector Size: 512
>
>
>
>
>
> Thanks & Regards,
>
> WD
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *POWERBALL ONLINE
> *Sent:* Friday, July 23, 2010 12:29 AM
>
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Please give me the information after you run mkqdisk -L and clustat -i
> command
>
> On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi,
>
> So, what your suggestion?
>
>
>
> Thanks,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *Wahyu Darmawan
> *Sent:* Thursday, July 22, 2010 11:02 PM
>
>
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Yes correct. I use Conga for my clusterware.
>
>
>
> Thanks,
>
>
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *POWERBALL ONLINE
> *Sent:* Thursday, July 22, 2010 10:54 PM
> *To:* linux clustering
> *Subject:* Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
>
>
> Hi
>
>
>
> What tool you use for create cluster? Is it Conga ?
>
>
>
>
>
> On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi all,
>
> I have problem with my cluster. I have 2 nodes of my cluster.
>
> On July 12 the nodes missed  connection from storage and lost 1 disk
> partition on both nodes. Then I decided to assigned a new of partition from
> storage, and use it on my nodes.
>
> But until today, the quorum is not going to quorate.
>
> FYI, I use GFS for my filesystem.
>
> So, what do I have to do to make quorum disk back to quorate status?
>
>
>
> I appreciate for your answer.
>
>
>
> Thanks,
>
> Wahyu
>
>
>
> Note, I attached the cluster log messages below:
>
> Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is
> disabled
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service
> Oracle_DBrepwm
>
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is
> disabled
>
> Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBahmcm
>
> Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm
> started
>
> Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBportal
>
> Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal
> started
>
> Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBcard5
>
> Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5
> started
>
> Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBhrdtxn
>
> Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn
> started
>
> Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBrepwm
>
> Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm
> started
>
> Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.120000)
>
> Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.070000)
>
> Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.310000)
>
> Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.400000)
>
> Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.690000)
>
> Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.620000)
>
> Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.020000)
>
> Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.550000)
>
> Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.050000)
>
> Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.710000)
>
> Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 13 02:05:30 t0175 last message repeated 2 times
>
> Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.530000)
>
> Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.200000)
>
> Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.820000)
>
> Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.420000)
>
> Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.820000)
>
> Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.540000)
>
> Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.740000)
>
> Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.020000)
>
> Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.900000)
>
> Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.270000)
>
> Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.980000)
>
> Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.470000)
>
> Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.870000)
>
> Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.940000)
>
> Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.780000)
>
> Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.360000)
>
> Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.790000)
>
> Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.730000)
>
> Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.100000)
>
> Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.580000)
>
> Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
>
> Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.460000)
>
> Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.790000)
>
> Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.240000)
>
> Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.670000)
>
> Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.570000)
>
> Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.000000)
>
> Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.670000)
>
> Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 18 02:04:31 t0175 last message repeated 3 times
>
> Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.870000)
>
> Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
>
> Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
>
> Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
>
> Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.920000)
>
> Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
>
> Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.040000)
>
> Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.350000)
>
> Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.450000)
>
> Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.450000)
>
> Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
>
> Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.120000)
>
> Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.480000)
>
> Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.960000)
>
> Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.030000)
>
> Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
>
> Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
>
> Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.940000)
>
> Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.560000)
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100730/f29c9e4d/attachment.htm>

From dhoffutt at gmail.com  Fri Jul 30 16:00:40 2010
From: dhoffutt at gmail.com (Dustin Henry Offutt)
Date: Fri, 30 Jul 2010 11:00:40 -0500
Subject: [Linux-cluster] Creating custom OCF_RESKEY_ variables,
	now how 	and what to refresh?
In-Reply-To: <0b55638253d93971945bd8eb50992eb3@mx.varna.net>
References: <AANLkTim+WEBDoEGeqAO6h75srW_v_dgUMuE9XxgZD6=m@mail.gmail.com>
	<0b55638253d93971945bd8eb50992eb3@mx.varna.net>
Message-ID: <AANLkTi=hoop_swSFAD-y9mLtsrzWRAHGXWxVAqS+nsOW@mail.gmail.com>

Hello there Kaloyan,

There is no "cluster.rng" anywhere on my system, and no other .rgn file that
looks related....

Any idea of an alias for this file?

[root at node1 ~]# ls -l /usr/share/cluster/cluster.rng
ls: /usr/share/cluster/cluster.rng: No such file or directory
[root at node1 ~]# ls /usr/share/cluster/*.rng
ls: /usr/share/cluster/*.rng: No such file or directory
[root at node1 ~]# locate cluster.rng
[root at node1 ~]# locate .rng
/usr/lib64/luci/zope/lib/python/Products/Archetypes/www/registry.rng
/usr/share/doc/python-virtinst-0.400.3/image.rng
/usr/share/hal/fdi/fdi.rng
/usr/share/libvirt/schemas/capability.rng
/usr/share/libvirt/schemas/domain.rng
/usr/share/libvirt/schemas/network.rng
/usr/share/libvirt/schemas/nodedev.rng
/usr/share/libvirt/schemas/storagepool.rng
/usr/share/libvirt/schemas/storagevol.rng
/usr/share/xml/docbook/slides/3.3.1/schema/rng/slides.rng
/usr/share/xml/docbook/slides/3.3.1/schema/rng/full/calstblx.rng
/usr/share/xml/docbook/slides/3.3.1/schema/rng/full/dbhierx.rng
/usr/share/xml/docbook/slides/3.3.1/schema/rng/full/dbnotnx.rng
/usr/share/xml/docbook/slides/3.3.1/schema/rng/full/dbpoolx.rng
/usr/share/xml/docbook/slides/3.3.1/schema/rng/full/docbookx.rng
/usr/share/xml/docbook/slides/3.3.1/schema/rng/full/htmltblx.rng
/usr/share/xml/docbook/slides/3.3.1/schema/rng/full/slides-full.rng
/usr/share/xml/docbook/slides/3.3.1/schema/rng/full/slides.rng
[root at node1 ~]#


On Fri, Jul 30, 2010 at 3:16 AM, Kaloyan Kovachev <kkovachev at varna.net>wrote:

> Hi,
>  you should update /usr/share/cluster/cluster.rng to validate the new
> options and your 'new' service/script
>
> On Thu, 29 Jul 2010 14:18:58 -0500, Dustin Henry Offutt
> <dhoffutt at gmail.com> wrote:
> > Hello,
> >
> > Does anyone know how to force a cluster (the "Cluster Suite" as released
> > with the RHEL5.4 ISO, cman, rgmanager, et.al.) to recognize that new
> > OCF_RESKEY variables have been introduced in a /usr/share/cluster/
> script?
> >
> > On one cluster the new variables are recognized and used by all nodes.
> Same
> > exact script, another cluster, just put the updated /usr/share/cluster/
> > script in today, and it's like it hasn't had something "refreshed", and
> > doesn't see the new "rules," if that makes any sense - despite bouncing
> the
> > cluster suite and the cluster nodes.
> >
> > Thank you...
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100730/69087f35/attachment.htm>

From Wahyu.Darmawan at ag-it.com  Fri Jul 30 17:35:31 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Sat, 31 Jul 2010 00:35:31 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
Message-ID: <fv6wv3tbcjyghwjnhd2atnfi.1280512091478@email.android.com>

Hi experts,
Does anyone else can help me?
Many thanks.

Rgds,
Wahyu

umesh susvirkar <susvirkar.3616 at gmail.com> wrote:



I  cannot find any log message /cluster.conf file other than following message in mail


Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)



On Fri, Jul 30, 2010 at 7:01 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi Umesh,
Please find another history from bottom of this mail. Because I have attached log message on the bottom.

Thanks,
Wahyu

From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of umesh susvirkar
Sent: Thursday, July 29, 2010 11:35 PM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

You have attach cluster.conf file for virtual guest where you have  form "cluster without quorum disk"

Kindly attach cluster.conf file where you have used quorum disk
Also log file where your are getting  not quorate message.
attach log file from both node of cluster.




On Thu, Jul 29, 2010 at 11:50 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi all,
Does anyone can help me?
Many thanks for your help.

Regards,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Wahyu Darmawan
Sent: Wednesday, July 28, 2010 8:50 AM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi Ball ;),
Thanks for your response. Please find the cluster.conf below.

<?xml version="1.0"?>
<cluster alias="DBAHMPS" config_version="2" name="TAI123">
                <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
                <clusternodes>
                                <clusternode name="node1.example.com<http://node1.example.com>" nodeid="1" votes="1">
                                                <fence>
                                                                <method name="1">
                                                                                <device domain="RHEL" name="VM_Fence"/>
                                                                </method>
                                                </fence>
                                </clusternode>
                                <clusternode name="node2.example.com<http://node2.example.com>" nodeid="2" votes="1">
                                                <fence>
                                                                <method name="1">
                                                                                <device domain="RHEL2" name="VM_Fence"/>
                                                                </method>
                                                </fence>
                                </clusternode>
                </clusternodes>
                <cman expected_votes="1" two_node="1"/>
                <fencedevices>
                                <fencedevice agent="fence_xvm" name="VM_Fence"/>
                </fencedevices>
                <rm>
                                <failoverdomains>
                                                <failoverdomain name="IP_Virtual" ordered="1" restricted="1">
                                                                <failoverdomainnode name="node1.example.com<http://node1.example.com>" priority="1"/>
                                                                <failoverdomainnode name="node2.example.com<http://node2.example.com>" priority="1"/>
                                                </failoverdomain>
                                </failoverdomains>
                                <resources>
                                                <ip address="192.168.122.100/24<http://192.168.122.100/24>" monitor_link="1"/>
                                                <fs device="/dev/mapper/VolGroup00-LogVol00" force_fsck="0" force_unmount="0" fsid="43719" fstype="ext3" mountpoint="/var/www/html/" name="HTML Doc" options="" self_fence="0"/>
                                                <script file="/etc/init.d/httpd" name="Web"/>
                                                <apache config_file="/etc/httpd/conf/httpd.conf" httpd_options="" name="httpd" server_root="/etc/httpd" shutdown_wait="3"/>
                                                <oracledb home="/u01/app/oracle/product/9.2.0.8/<http://9.2.0.8/>" name="DBAHMPS" type="10g" user="oracle" vhost="dbahmps.example.com<http://dbahmps.example.com>"/>
                                </resources>
                                <service autostart="1" exclusive="1" name="HTTPD_Service" recovery="relocate">
                                                <ip ref="192.168.122.100/24<http://192.168.122.100/24>"/>
                                                <fs ref="HTML Doc"/>
                                                <script ref="Web"/>
                                </service>
                                <service autostart="1" domain="IP_Virtual" name="Web" recovery="restart">
                                                <fs ref="HTML Doc"/>
                                                <script ref="Web"/>
                                </service>
                                <service autostart="1" domain="IP_Virtual" exclusive="1" name="Apache Service" nfslock="1">
                                                <apache ref="httpd"/>
                                </service>
                </rm>
</cluster>

Appreciating for your help Ball.


Rgds,
Wahyu

From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Saturday, July 24, 2010 12:35 AM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

I though that your configuration is wrong because when you run clustat it should display quorum disk is online too.
If your configuration wrong and quorum not online it will have the error that you meet.
Please give me the file /etc/cluster/cluster.conf to me too.



On Fri, Jul 23, 2010 at 10:18 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi,
Thanks for your response.
Here is the info on below.


T0171
[root at t0171 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  t0171.example.com<http://t0171.example.com/>                    Online, Local, rgmanager
  t0175.example.com<http://t0175.example.com/>                    Online, rgmanager

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  IP_Virtual           t0175.example.com<http://t0175.example.com/>          started
  GFS_Mounting         t0175.example.com<http://t0175.example.com/>          started
  Listener_Service     t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBahmcm       t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBportal      t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBcard5       t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBhrdtxn      t0175.example.com<http://t0175.example.com/>          started
  Oracle_DBrepwm       t0175.example.com<http://t0175.example.com/>          started
[root at t0171 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171.example.com<http://t0171.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/sdi2:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-18:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171.example.com<http://t0171.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-19:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512


T0175
[root at t0175 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  t0171.example.com<http://t0171.example.com/>                    Online, rgmanager
  t0175.example.com<http://t0175.example.com/>                    Online, Local, rgmanager

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  IP_Virtual           t0175. example.com<http://example.com/>          started
  GFS_Mounting         t0175. example.com<http://example.com/>          started
  Listener_Service     t0175. example.com<http://example.com/>          started
  Oracle_DBahmcm       t0175. example.com<http://example.com/>          started
  Oracle_DBportal      t0175. example.com<http://example.com/>          started
  Oracle_DBcard5       t0175. example.com<http://example.com/>          started
  Oracle_DBhrdtxn      t0175. example.com<http://example.com/>          started
  Oracle_DBrepwm       t0175. example.com<http://example.com/>          started
[root at t0175 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/sdi2:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-19:
                Magic:                eb7a62c2
                Label:                dbquorum1
                Created:              Mon Mar 29 18:44:37 2010
                Host:                 t0171. example.com<http://example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512

/dev/dm-20:
                Magic:                eb7a62c2
                Label:                dbquorum2
                Created:              Mon Mar 29 18:26:28 2010
                Host:                 t0175.example.com<http://t0175.example.com/>
                Kernel Sector Size:   512
                Recorded Sector Size: 512


Thanks & Regards,
WD

From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Friday, July 23, 2010 12:29 AM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Please give me the information after you run mkqdisk -L and clustat -i command
On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi,
So, what your suggestion?

Thanks,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Wahyu Darmawan
Sent: Thursday, July 22, 2010 11:02 PM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Yes correct. I use Conga for my clusterware.

Thanks,


From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
Sent: Thursday, July 22, 2010 10:54 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi

What tool you use for create cluster? Is it Conga ?



On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi all,
I have problem with my cluster. I have 2 nodes of my cluster.
On July 12 the nodes missed  connection from storage and lost 1 disk partition on both nodes. Then I decided to assigned a new of partition from storage, and use it on my nodes.
But until today, the quorum is not going to quorate.
FYI, I use GFS for my filesystem.
So, what do I have to do to make quorum disk back to quorate status?

I appreciate for your answer.

Thanks,
Wahyu

Note, I attached the cluster log messages below:
Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is disabled
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service Oracle_DBrepwm
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is disabled
Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBahmcm
Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm started
Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBportal
Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal started
Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBcard5
Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5 started
Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBhrdtxn
Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn started
Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBrepwm
Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm started
Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)
Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.310000)
Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.400000)
Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.690000)
Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.620000)
Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.020000)
Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.550000)
Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.050000)
Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.710000)
Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:05:30 t0175 last message repeated 2 times
Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.530000)
Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.200000)
Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.820000)
Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.420000)
Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.820000)
Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.540000)
Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.740000)
Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.020000)
Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.900000)
Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.270000)
Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.980000)
Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.470000)
Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.870000)
Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.940000)
Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.780000)
Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.360000)
Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.790000)
Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.730000)
Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.100000)
Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.580000)
Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.460000)
Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.790000)
Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.240000)
Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.670000)
Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.570000)
Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.000000)
Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.670000)
Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 18 02:04:31 t0175 last message repeated 3 times
Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.870000)
Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.920000)
Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.040000)
Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.350000)
Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.450000)
Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.450000)
Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.120000)
Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.480000)
Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.960000)
Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.030000)
Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.940000)
Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.560000)


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster




From member at linkedin.com  Sat Jul 31 10:31:35 2010
From: member at linkedin.com (wenxiao li via LinkedIn)
Date: Sat, 31 Jul 2010 03:31:35 -0700 (PDT)
Subject: [Linux-cluster] wenxiao li wants to stay in touch on LinkedIn
Message-ID: <2033861795.184816.1280572295754.JavaMail.app@ech3-cdn10.prod>

LinkedIn
------------wenxiao li requested to add you as a connection on LinkedIn:
------------------------------------------

Marian,

I'd like to add you to my professional network on LinkedIn.

- wenxiao li

Accept invitation from wenxiao li
http://www.linkedin.com/e/-odgn7o-gcaboj7r-5h/ulDuieLaAX544oVCOYcgj_GaXIys4TuLMXGmOx/blk/I2233986645_2/1BpC5vrmRLoRZcjkkZt5YCpnlOt3RApnhMpmdzgmhxrSNBszYOnPkQdzoUejcPcz99bP1gom5IkR5ObPwSe34RejsNdj4LrCBxbOYWrSlI/EML_comm_afe/

View invitation from wenxiao li
http://www.linkedin.com/e/-odgn7o-gcaboj7r-5h/ulDuieLaAX544oVCOYcgj_GaXIys4TuLMXGmOx/blk/I2233986645_2/39vdjgSdzwVcPcOcAALqnpPbOYWrSlI/svi/ 

------------------------------------------
DID YOU KNOW you can use your LinkedIn profile as your website? Select a vanity URL and then promote this address on your business cards, email signatures, website, etc
http://www.linkedin.com/e/-odgn7o-gcaboj7r-5h/ewp/inv-21/


 
------
(c) 2010, LinkedIn Corporation
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100731/76dd2ce9/attachment.htm>

From sakect at gmail.com  Sat Jul 31 13:17:47 2010
From: sakect at gmail.com (POWERBALL ONLINE)
Date: Sat, 31 Jul 2010 20:17:47 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <fv6wv3tbcjyghwjnhd2atnfi.1280512091478@email.android.com>
References: <fv6wv3tbcjyghwjnhd2atnfi.1280512091478@email.android.com>
Message-ID: <AANLkTim-_81=eSKf6eEgMsV3vGkLWY08j7JX=vMYGhhH@mail.gmail.com>

Hi

Refer your cluster configuration file I can not see that you are already
configure about quorum disk and voting please recheck your confiuration
again.

On Sat, Jul 31, 2010 at 12:35 AM, Wahyu Darmawan
<Wahyu.Darmawan at ag-it.com>wrote:

> Hi experts,
> Does anyone else can help me?
> Many thanks.
>
> Rgds,
> Wahyu
>
> umesh susvirkar <susvirkar.3616 at gmail.com> wrote:
>
>
>
> I  cannot find any log message /cluster.conf file other than following
> message in mail
>
>
> Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.120000)
> Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
> Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
> Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
> Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.070000)
>
>
>
> On Fri, Jul 30, 2010 at 7:01 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com
> <mailto:Wahyu.Darmawan at ag-it.com>> wrote:
> Hi Umesh,
> Please find another history from bottom of this mail. Because I have
> attached log message on the bottom.
>
> Thanks,
> Wahyu
>
> From: linux-cluster-bounces at redhat.com<mailto:
> linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com
> <mailto:linux-cluster-bounces at redhat.com>] On Behalf Of umesh susvirkar
> Sent: Thursday, July 29, 2010 11:35 PM
>
> To: linux clustering
> Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
> You have attach cluster.conf file for virtual guest where you have  form
> "cluster without quorum disk"
>
> Kindly attach cluster.conf file where you have used quorum disk
> Also log file where your are getting  not quorate message.
> attach log file from both node of cluster.
>
>
>
>
> On Thu, Jul 29, 2010 at 11:50 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com
> <mailto:Wahyu.Darmawan at ag-it.com>> wrote:
> Hi all,
> Does anyone can help me?
> Many thanks for your help.
>
> Regards,
>
>
> From: linux-cluster-bounces at redhat.com<mailto:
> linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com
> <mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Wahyu Darmawan
> Sent: Wednesday, July 28, 2010 8:50 AM
>
> To: linux clustering
> Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
> Hi Ball ;),
> Thanks for your response. Please find the cluster.conf below.
>
> <?xml version="1.0"?>
> <cluster alias="DBAHMPS" config_version="2" name="TAI123">
>                <fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="3"/>
>                <clusternodes>
>                                <clusternode name="node1.example.com<
> http://node1.example.com>" nodeid="1" votes="1">
>                                                <fence>
>                                                                <method
> name="1">
>
>    <device domain="RHEL" name="VM_Fence"/>
>                                                                </method>
>                                                </fence>
>                                </clusternode>
>                                <clusternode name="node2.example.com<
> http://node2.example.com>" nodeid="2" votes="1">
>                                                <fence>
>                                                                <method
> name="1">
>
>    <device domain="RHEL2" name="VM_Fence"/>
>                                                                </method>
>                                                </fence>
>                                </clusternode>
>                </clusternodes>
>                <cman expected_votes="1" two_node="1"/>
>                <fencedevices>
>                                <fencedevice agent="fence_xvm"
> name="VM_Fence"/>
>                </fencedevices>
>                <rm>
>                                <failoverdomains>
>                                                <failoverdomain
> name="IP_Virtual" ordered="1" restricted="1">
>
>  <failoverdomainnode name="node1.example.com<http://node1.example.com>"
> priority="1"/>
>
>  <failoverdomainnode name="node2.example.com<http://node2.example.com>"
> priority="1"/>
>                                                </failoverdomain>
>                                </failoverdomains>
>                                <resources>
>                                                <ip address="
> 192.168.122.100/24<http://192.168.122.100/24>" monitor_link="1"/>
>                                                <fs
> device="/dev/mapper/VolGroup00-LogVol00" force_fsck="0" force_unmount="0"
> fsid="43719" fstype="ext3" mountpoint="/var/www/html/" name="HTML Doc"
> options="" self_fence="0"/>
>                                                <script
> file="/etc/init.d/httpd" name="Web"/>
>                                                <apache
> config_file="/etc/httpd/conf/httpd.conf" httpd_options="" name="httpd"
> server_root="/etc/httpd" shutdown_wait="3"/>
>                                                <oracledb
> home="/u01/app/oracle/product/9.2.0.8/<http://9.2.0.8/>" name="DBAHMPS"
> type="10g" user="oracle" vhost="dbahmps.example.com<
> http://dbahmps.example.com>"/>
>                                </resources>
>                                <service autostart="1" exclusive="1"
> name="HTTPD_Service" recovery="relocate">
>                                                <ip ref="192.168.122.100/24
> <http://192.168.122.100/24>"/>
>                                                <fs ref="HTML Doc"/>
>                                                <script ref="Web"/>
>                                </service>
>                                <service autostart="1" domain="IP_Virtual"
> name="Web" recovery="restart">
>                                                <fs ref="HTML Doc"/>
>                                                <script ref="Web"/>
>                                </service>
>                                <service autostart="1" domain="IP_Virtual"
> exclusive="1" name="Apache Service" nfslock="1">
>                                                <apache ref="httpd"/>
>                                </service>
>                </rm>
> </cluster>
>
> Appreciating for your help Ball.
>
>
> Rgds,
> Wahyu
>
> From: linux-cluster-bounces at redhat.com<mailto:
> linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com
> <mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
> Sent: Saturday, July 24, 2010 12:35 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
> I though that your configuration is wrong because when you run clustat it
> should display quorum disk is online too.
> If your configuration wrong and quorum not online it will have the error
> that you meet.
> Please give me the file /etc/cluster/cluster.conf to me too.
>
>
>
> On Fri, Jul 23, 2010 at 10:18 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com
> <mailto:Wahyu.Darmawan at ag-it.com>> wrote:
> Hi,
> Thanks for your response.
> Here is the info on below.
>
>
> T0171
> [root at t0171 ~]# clustat
> Member Status: Quorate
>
>  Member Name                              Status
>  ------ ----                              ------
>  t0171.example.com<http://t0171.example.com/>                    Online,
> Local, rgmanager
>  t0175.example.com<http://t0175.example.com/>                    Online,
> rgmanager
>
>  Service Name         Owner (Last)                   State
>  ------- ----         ----- ------                   -----
>  IP_Virtual           t0175.example.com<http://t0175.example.com/>
>  started
>  GFS_Mounting         t0175.example.com<http://t0175.example.com/>
>  started
>  Listener_Service     t0175.example.com<http://t0175.example.com/>
>  started
>  Oracle_DBahmcm       t0175.example.com<http://t0175.example.com/>
>  started
>  Oracle_DBportal      t0175.example.com<http://t0175.example.com/>
>  started
>  Oracle_DBcard5       t0175.example.com<http://t0175.example.com/>
>  started
>  Oracle_DBhrdtxn      t0175.example.com<http://t0175.example.com/>
>  started
>  Oracle_DBrepwm       t0175.example.com<http://t0175.example.com/>
>  started
> [root at t0171 ~]# mkqdisk -L
> mkqdisk v0.5.2
> /dev/sdi1:
>                Magic:                eb7a62c2
>                Label:                dbquorum1
>                Created:              Mon Mar 29 18:44:37 2010
>                Host:                 t0171.example.com<
> http://t0171.example.com/>
>                Kernel Sector Size:   512
>                Recorded Sector Size: 512
>
> /dev/sdi2:
>                Magic:                eb7a62c2
>                Label:                dbquorum2
>                Created:              Mon Mar 29 18:26:28 2010
>                Host:                 t0175.example.com<
> http://t0175.example.com/>
>                Kernel Sector Size:   512
>                Recorded Sector Size: 512
>
> /dev/dm-18:
>                Magic:                eb7a62c2
>                Label:                dbquorum1
>                Created:              Mon Mar 29 18:44:37 2010
>                Host:                 t0171.example.com<
> http://t0171.example.com/>
>                Kernel Sector Size:   512
>                Recorded Sector Size: 512
>
> /dev/dm-19:
>                Magic:                eb7a62c2
>                Label:                dbquorum2
>                Created:              Mon Mar 29 18:26:28 2010
>                Host:                 t0175.example.com<
> http://t0175.example.com/>
>                Kernel Sector Size:   512
>                Recorded Sector Size: 512
>
>
> T0175
> [root at t0175 ~]# clustat
> Member Status: Quorate
>
>  Member Name                              Status
>  ------ ----                              ------
>  t0171.example.com<http://t0171.example.com/>                    Online,
> rgmanager
>  t0175.example.com<http://t0175.example.com/>                    Online,
> Local, rgmanager
>
>  Service Name         Owner (Last)                   State
>  ------- ----         ----- ------                   -----
>  IP_Virtual           t0175. example.com<http://example.com/>
>  started
>  GFS_Mounting         t0175. example.com<http://example.com/>
>  started
>  Listener_Service     t0175. example.com<http://example.com/>
>  started
>  Oracle_DBahmcm       t0175. example.com<http://example.com/>
>  started
>  Oracle_DBportal      t0175. example.com<http://example.com/>
>  started
>  Oracle_DBcard5       t0175. example.com<http://example.com/>
>  started
>  Oracle_DBhrdtxn      t0175. example.com<http://example.com/>
>  started
>  Oracle_DBrepwm       t0175. example.com<http://example.com/>
>  started
> [root at t0175 ~]# mkqdisk -L
> mkqdisk v0.5.2
> /dev/sdi1:
>                Magic:                eb7a62c2
>                Label:                dbquorum1
>                Created:              Mon Mar 29 18:44:37 2010
>                Host:                 t0171. example.com<
> http://example.com/>
>                Kernel Sector Size:   512
>                Recorded Sector Size: 512
>
> /dev/sdi2:
>                Magic:                eb7a62c2
>                Label:                dbquorum2
>                Created:              Mon Mar 29 18:26:28 2010
>                Host:                 t0175. example.com<
> http://example.com/>
>                Kernel Sector Size:   512
>                Recorded Sector Size: 512
>
> /dev/dm-19:
>                Magic:                eb7a62c2
>                Label:                dbquorum1
>                Created:              Mon Mar 29 18:44:37 2010
>                Host:                 t0171. example.com<
> http://example.com/>
>                Kernel Sector Size:   512
>                Recorded Sector Size: 512
>
> /dev/dm-20:
>                Magic:                eb7a62c2
>                Label:                dbquorum2
>                Created:              Mon Mar 29 18:26:28 2010
>                Host:                 t0175.example.com<
> http://t0175.example.com/>
>                Kernel Sector Size:   512
>                Recorded Sector Size: 512
>
>
> Thanks & Regards,
> WD
>
> From: linux-cluster-bounces at redhat.com<mailto:
> linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com
> <mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
> Sent: Friday, July 23, 2010 12:29 AM
>
> To: linux clustering
> Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
> Please give me the information after you run mkqdisk -L and clustat -i
> command
> On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com
> <mailto:Wahyu.Darmawan at ag-it.com>> wrote:
> Hi,
> So, what your suggestion?
>
> Thanks,
>
>
> From: linux-cluster-bounces at redhat.com<mailto:
> linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com
> <mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Wahyu Darmawan
> Sent: Thursday, July 22, 2010 11:02 PM
>
> To: linux clustering
> Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
> Yes correct. I use Conga for my clusterware.
>
> Thanks,
>
>
> From: linux-cluster-bounces at redhat.com<mailto:
> linux-cluster-bounces at redhat.com> [mailto:linux-cluster-bounces at redhat.com
> <mailto:linux-cluster-bounces at redhat.com>] On Behalf Of POWERBALL ONLINE
> Sent: Thursday, July 22, 2010 10:54 PM
> To: linux clustering
> Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
>
> Hi
>
> What tool you use for create cluster? Is it Conga ?
>
>
>
>  On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <
> Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
> Hi all,
> I have problem with my cluster. I have 2 nodes of my cluster.
> On July 12 the nodes missed  connection from storage and lost 1 disk
> partition on both nodes. Then I decided to assigned a new of partition from
> storage, and use it on my nodes.
> But until today, the quorum is not going to quorate.
> FYI, I use GFS for my filesystem.
> So, what do I have to do to make quorum disk back to quorate status?
>
> I appreciate for your answer.
>
> Thanks,
> Wahyu
>
> Note, I attached the cluster log messages below:
> Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is
> disabled
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service
> Oracle_DBrepwm
> Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is
> disabled
> Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBahmcm
> Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm
> started
> Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBportal
> Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal
> started
> Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBcard5
> Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5
> started
> Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBhrdtxn
> Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn
> started
> Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service
> Oracle_DBrepwm
> Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm
> started
> Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.120000)
> Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
> Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
> Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
> Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.070000)
> Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.310000)
> Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.400000)
> Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.690000)
> Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
> Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.620000)
> Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.020000)
> Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.550000)
> Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.050000)
> Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.710000)
> Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
> Jul 13 02:05:30 t0175 last message repeated 2 times
> Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.530000)
> Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.200000)
> Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.820000)
> Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
> Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
> Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.420000)
> Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.820000)
> Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.540000)
> Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.740000)
> Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
> Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
> Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
> Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.020000)
> Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.900000)
> Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.270000)
> Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.980000)
> Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.470000)
> Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
> Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
> Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
> Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
> Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.870000)
> Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.940000)
> Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.780000)
> Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
> Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
> Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.360000)
> Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.790000)
> Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.730000)
> Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.100000)
> Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
> Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
> Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.580000)
> Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.720000)
> Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.460000)
> Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.790000)
> Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
> Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
> Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
> Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.240000)
> Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.670000)
> Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.570000)
> Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (13.000000)
> Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.670000)
> Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
> Jul 18 02:04:31 t0175 last message repeated 3 times
> Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.870000)
> Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
> Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.000000)
> Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
> Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.980000)
> Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.900000)
> Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.920000)
> Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.930000)
> Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
> Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
> Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.040000)
> Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (2.350000)
> Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (12.450000)
> Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.450000)
> Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.010000)
> Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.120000)
> Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.480000)
> Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (11.960000)
> Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.030000)
> Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
> Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.990000)
> Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
> Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.000000)
> Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.940000)
> Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1
> second to complete (1.560000)
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
>  https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100731/79ab517e/attachment.htm>